The Fix
pip install requests==2.27.0
Based on closed psf/requests issue #5667 · PR/commit linked
@@ -503,6 +503,10 @@ def get_encoding_from_headers(headers):
return 'ISO-8859-1'
+ if 'application/json' in content_type:
+ # Assume UTF-8 based on RFC 4627: https://www.ietf.org/rfc/rfc4627.txt since the charset was unset
+ return 'utf-8'
repro.py
import requests
import json
r = requests.get("https://api.covidtracking.com/v1/us/current.json")
r._content = b'{"name":"rd\xce\xba"}' # This is utf-8
r.headers = {
"Content-Type": "application/json",
}
print(r.json())
print(json.loads(r.text))
verify
Re-run the minimal reproduction on your broken version, then apply the fix and re-run.
fix.md
Option A — Upgrade to fixed release\npip install requests==2.27.0\nWhen NOT to use: This fix should not be applied if the content type is not application/json.\n\n
Why This Fix Works in Production
- Trigger: This might be an interesting one... I found that `r.text` and `r.json()` _can_ return different results in some specific cases. I don't understand why the…
- Mechanism: The encoding detection for JSON responses was incorrectly handled by the library
- Why the fix works: Fixes the issue by updating the `get_encoding_from_headers` function to return 'utf-8' if the content type is set to application/json and charset is not specified. (first fixed release: 2.27.0).
Production impact:
- If left unfixed, this can cause silent data inconsistencies that propagate (bad cache entries, incorrect downstream decisions).
Why This Breaks in Prod
- The encoding detection for JSON responses was incorrectly handled by the library
- Production symptom (often without a traceback): This might be an interesting one... I found that `r.text` and `r.json()` _can_ return different results in some specific cases. I don't understand why the difference between the two cases is changing the result of `r.text`.
Proof / Evidence
- GitHub issue: #5667
- Fix PR: https://github.com/psf/requests/pull/5673
- First fixed release: 2.27.0
- Reproduced locally: No (not executed)
- Last verified: 2026-02-09
- Confidence: 0.85
- Did this fix it?: Yes (upstream fix exists)
- Own content ratio: 0.59
Discussion
High-signal excerpts from the issue thread (symptoms, repros, edge-cases).
“That's too bad about the issues with .text. If only it were possible to drop stuff like that. I totally agree with where you're coming…”
“So there's a semantic difference between .json() and .text”
“I looked into this some more and the reason for the difference is that chardet is guessing different text encodings for the two different strings”
Failure Signature (Search String)
- This might be an interesting one... I found that `r.text` and `r.json()` _can_ return different results in some specific cases. I don't understand why the difference between the
- The `name` is different even though it did not change at all! The existence of `"uuid":"1234"` in the response's contents somehow changes the decoding. I have no clue why.
Copy-friendly signature
signature.txt
Failure Signature
-----------------
This might be an interesting one... I found that `r.text` and `r.json()` _can_ return different results in some specific cases. I don't understand why the difference between the two cases is changing the result of `r.text`.
The `name` is different even though it did not change at all! The existence of `"uuid":"1234"` in the response's contents somehow changes the decoding. I have no clue why.
Error Message
Signature-only (no traceback captured)
error.txt
Error Message
-------------
This might be an interesting one... I found that `r.text` and `r.json()` _can_ return different results in some specific cases. I don't understand why the difference between the two cases is changing the result of `r.text`.
The `name` is different even though it did not change at all! The existence of `"uuid":"1234"` in the response's contents somehow changes the decoding. I have no clue why.
Minimal Reproduction
repro.py
import requests
import json
r = requests.get("https://api.covidtracking.com/v1/us/current.json")
r._content = b'{"name":"rd\xce\xba"}' # This is utf-8
r.headers = {
"Content-Type": "application/json",
}
print(r.json())
print(json.loads(r.text))
What Broke
Inconsistent data returned when accessing JSON response via different methods.
Why It Broke
The encoding detection for JSON responses was incorrectly handled by the library
Fix Options (Details)
Option A — Upgrade to fixed release Safe default (recommended)
pip install requests==2.27.0
When NOT to use: This fix should not be applied if the content type is not application/json.
Use when you can deploy the upstream fix. It is usually lower-risk than long-lived workarounds.
Fix reference: https://github.com/psf/requests/pull/5673
First fixed release: 2.27.0
Last verified: 2026-02-09. Validate in your environment.
When NOT to Use This Fix
- This fix should not be applied if the content type is not application/json.
Verify Fix
verify
Re-run the minimal reproduction on your broken version, then apply the fix and re-run.
Did This Fix Work in Your Case?
Quick signal helps us prioritize which fixes to verify and improve.
Prevention
- Add a TLS smoke test that performs a real handshake in CI (include CA bundle validation and hostname checks).
- Alert on handshake failures by error string and endpoint to catch cert/CA changes quickly.
Version Compatibility Table
| Version | Status |
|---|---|
| 2.27.0 | Fixed |
Related Issues
No related fixes found.
Sources
We don’t republish the full GitHub discussion text. Use the links above for context.