Chinese Encoding with Python Requests Library

1. Introduction

Working with Chinese text can be challenging, especially when handling encoding and decoding properly. In this blog post, we will demonstrate how to handle Chinese encoding correctly using the Python Requests library, ensuring that you can seamlessly work with Chinese text in your web applications.

2. Install Requests Library

First, you need to install the Requests library if you haven’t already. You can do this using pip:

pip install requests

Once you have the Requests library installed, you can start working with Chinese text in your HTTP requests.

3. Handling Chinese Encoding in Requests Library

When making HTTP requests using the Requests library, it’s essential to handle encoding correctly to avoid garbled text or “mojibake.” Here’s how to do that:

3.1 Set the Correct Encoding for Requests Response

When receiving a response containing Chinese text, you should set the response encoding to “utf-8.” This ensures that the content is properly decoded before you work with it. Here’s an example:

import requests

url = "https://www.baidu.com"
response = requests.get(url)

At this point, if you check the output by either response.text or response.content, it’ll show garbled text for the Chinese characters.

We’ll need to set the response’s encoding to ensure that it is properly decoded.

# Ensure the response content is properly decoded
response.encoding = "utf-8"
content = response.text

print(content)

Another way is to use decode upon the content of the response. Similarly, you can view the Chinese Characters correctly.

response.content.decode('utf-8')

3.2 Sending Chinese in Request Data

When sending Chinese text in your HTTP requests (e.g., POST requests), you should ensure that the data is properly encoded. Here’s an example of sending Chinese text as JSON data in a POST request:

In this example, we use the json.dumps() function with the ensure_ascii=False parameter to generate a JSON string containing the Chinese text. We then encode the string as “utf-8” before sending it in the POST request. We also set the “Content-Type” header to indicate that the data is in JSON format and uses the “utf-8” charset.

import requests
import json

url = "https://example.com/your_endpoint"
data = {
    "message": "你好, 世界!"
}

headers = {
    "Content-Type": "application/json; charset=utf-8"
}

response = requests.post(url, data=json.dumps(data, ensure_ascii=False).encode('utf-8'), headers=headers)

# Handle the response as needed

4. Conclusion

Handling Chinese encoding correctly when working with the Python Requests library is essential to avoid garbled text or “mojibake.” By setting the proper encoding for the response and sending correctly encoded data in your requests, you can seamlessly work with Chinese text in your web applications. Follow these best practices to ensure your Python applications can handle Chinese text without issues. You may explore the Requests library on its official website and our other Python tutorials.