aiohttp (Fix)

The Fix

Disables writelines for the test_write_large_payload_deflate_compression_data_in_eof test to prevent failures with Python 3.12.9+ or 3.13.2+.

Based on closed aio-libs/aiohttp issue #10421 · PR/commit linked

Production note: This tends to surface only under concurrency. Reproduce with load tests and watch for lock contention/cancellation paths.

Jump to Verify Open PR/Commit

@@ -0,0 +1 @@
@@ -0,0 +1 @@
+Fixed test ``test_write_large_payload_deflate_compression_data_in_eof_writelines`` failing with Python 3.12.9+ or 3.13.2+ -- by :user:`bdraco`.
diff --git a/tests/test_http_writer.py b/tests/test_http_writer.py
index 95dabeab377..b76cac0fe58 100644

repro.py

$ pytest -n1 tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof 
========================================================= test session starts =========================================================
platform linux -- Python 3.12.9, pytest-8.1.1, pluggy-1.5.0 -- /tmp/aiohttp/.venv/bin/python
codspeed: 3.2.0 (disabled, mode: walltime, timer_resolution: 1.0ns)
cachedir: .pytest_cache
rootdir: /tmp/aiohttp
configfile: setup.cfg
plugins: cov-6.0.0, codspeed-3.2.0, mock-3.14.0, xdist-3.6.1
1 worker [1 item]      
scheduling tests via LoadScheduling

tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof[pyloop] 
[gw0] [100%] FAILED tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof[pyloop] 

============================================================== FAILURES ===============================================================
__________________________________ test_write_large_payload_deflate_compression_data_in_eof[pyloop] ___________________________________
[gw0] linux -- Python 3.12.9 /tmp/aiohttp/.venv/bin/python

protocol = <NonCallableMagicMock spec_set='BaseProtocol' id='139909514918992'>
transport = <NonCallableMagicMock name='mock.transport' spec_set='Transport' id='139909512338736'>
loop = <_UnixSelectorEventLoop running=False closed=False debug=False>

    async def test_write_large_payload_deflate_compression_data_in_eof(
        protocol: BaseProtocol,
        transport: asyncio.Transport,
        loop: asyncio.AbstractEventLoop,
    ) -> None:
        msg = http.StreamWriter(protocol, loop)
        msg.enable_compression("deflate")
    
        await msg.write(b"data" * 4096)
        assert transport.write.called  # type: ignore[attr-defined]
        chunks = [c[1][0] for c in list(transport.write.mock_calls)]  # type: ignore[attr-defined]
        transport.write.reset_mock()  # type: ignore[attr-defined]
    
        # This payload compresses to 20447 bytes
        payload = b"".join(
            [bytes((*range(0, i), *range(i, 0, -1))) for i in range(255) for _ in range(64)]
        )
        await msg.write_eof(payload)
        chunks.extend([c[1][0] for c in list(transport.write.mock_calls)])  # type: ignore[attr-defined]
    
        assert all(chunks)
        content = b"".join(chunks)
      assert zlib.decompress(content) == (b"data" * 4096) + payload
E       zlib.error: Error -5 while decompressing data: incomplete or truncated stream

chunks     = [b'x\x9c']
content    = b'x\x9c'
loop       = <_UnixSelectorEventLoop running=False closed=False debug=False>
msg        = <aiohttp.http_writer.StreamWriter object at 0x7f3f38c896a0>
payload    = b'\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00...\x1f\x1e\x1d\x1c\x1b\x1a\x19\x18\x17\x16\x15\x14\x13\x12\x11\x10\x0f\x0e\r\x0c\x0b\n\t\x08\x07\x06\x05\x04\x03\x02\x01'
protocol   = <NonCallableMagicMock spec_set='BaseProtocol' id='139909514918992'>
transport  = <NonCallableMagicMock name='mock.transport' spec_set='Transport' id='139909512338736'>

tests/test_http_writer.py:148: error

verify

Re-run the minimal reproduction on your broken version, then apply the fix and re-run.

fix.md

Option A — Apply the official fix\nDisables writelines for the test_write_large_payload_deflate_compression_data_in_eof test to prevent failures with Python 3.12.9+ or 3.13.2+.\nWhen NOT to use: Do not apply this fix if the test is expected to run with writelines enabled.\n\n

Why This Fix Works in Production

Trigger: `tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof[pyloop]` fails with Python 3.12.9
Mechanism: Disables writelines for the test_write_large_payload_deflate_compression_data_in_eof test to prevent failures with Python 3.12.9+ or 3.13.2+.

Production impact:

If left unfixed, failures can be intermittent under concurrency (hard to reproduce; shows up as sporadic 5xx/timeouts).

Why This Breaks in Prod

Shows up under Python 3.12.9 in real deployments (not just unit tests).
Production symptom (often without a traceback): `tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof[pyloop]` fails with Python 3.12.9

Proof / Evidence

GitHub issue: #10421
Fix PR: https://github.com/aio-libs/aiohttp/pull/10423
Reproduced locally: No (not executed)
Last verified: 2026-02-09
Confidence: 0.80
Did this fix it?: Yes (upstream fix exists)
Own content ratio: 0.38

Discussion

High-signal excerpts from the issue thread (symptoms, repros, edge-cases).

Jump to Sources Open on GitHub

“I would assume that chunks contained more than just [b'x\x9c'] on older releases, so maybe it's an issue with the mocking or something?”

@Dreamsorcerer · 2025-02-06 · source

“Does it have anything to do with the zero-copy writes, @bdraco? I'm guessing the chunk there is from the eof call, so the writes are…”

@Dreamsorcerer · 2025-02-06 · source

“Its probably a race and the additional data is in transport.writelines.mock_calls”

@bdraco · 2025-02-06 · source

“By the way, I've bisected CPython and aiohttp's test suite literally starts failing on the commit increasing the version, so I think the SKIP_WRITELINES condition…”

@mgorny · 2025-02-06 · source

Failure Signature (Search String)

`tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof[pyloop]` fails with Python 3.12.9
[gw0] [100%] FAILED tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof[pyloop]

Copy-friendly signature

signature.txt

Failure Signature
-----------------
`tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof[pyloop]` fails with Python 3.12.9
[gw0] [100%] FAILED tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof[pyloop]

Error Message

Signature-only (no traceback captured)

error.txt

Error Message
-------------
`tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof[pyloop]` fails with Python 3.12.9
[gw0] [100%] FAILED tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof[pyloop]

Minimal Reproduction

repro.py

$ pytest -n1 tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof 
========================================================= test session starts =========================================================
platform linux -- Python 3.12.9, pytest-8.1.1, pluggy-1.5.0 -- /tmp/aiohttp/.venv/bin/python
codspeed: 3.2.0 (disabled, mode: walltime, timer_resolution: 1.0ns)
cachedir: .pytest_cache
rootdir: /tmp/aiohttp
configfile: setup.cfg
plugins: cov-6.0.0, codspeed-3.2.0, mock-3.14.0, xdist-3.6.1
1 worker [1 item]      
scheduling tests via LoadScheduling

tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof[pyloop] 
[gw0] [100%] FAILED tests/test_http_writer.py::test_write_large_payload_deflate_compression_data_in_eof[pyloop] 

============================================================== FAILURES ===============================================================
__________________________________ test_write_large_payload_deflate_compression_data_in_eof[pyloop] ___________________________________
[gw0] linux -- Python 3.12.9 /tmp/aiohttp/.venv/bin/python

protocol = <NonCallableMagicMock spec_set='BaseProtocol' id='139909514918992'>
transport = <NonCallableMagicMock name='mock.transport' spec_set='Transport' id='139909512338736'>
loop = <_UnixSelectorEventLoop running=False closed=False debug=False>

    async def test_write_large_payload_deflate_compression_data_in_eof(
        protocol: BaseProtocol,
        transport: asyncio.Transport,
        loop: asyncio.AbstractEventLoop,
    ) -> None:
        msg = http.StreamWriter(protocol, loop)
        msg.enable_compression("deflate")
    
        await msg.write(b"data" * 4096)
        assert transport.write.called  # type: ignore[attr-defined]
        chunks = [c[1][0] for c in list(transport.write.mock_calls)]  # type: ignore[attr-defined]
        transport.write.reset_mock()  # type: ignore[attr-defined]
    
        # This payload compresses to 20447 bytes
        payload = b"".join(
            [bytes((*range(0, i), *range(i, 0, -1))) for i in range(255) for _ in range(64)]
        )
        await msg.write_eof(payload)
        chunks.extend([c[1][0] for c in list(transport.write.mock_calls)])  # type: ignore[attr-defined]
    
        assert all(chunks)
        content = b"".join(chunks)
      assert zlib.decompress(content) == (b"data" * 4096) + payload
E       zlib.error: Error -5 while decompressing data: incomplete or truncated stream

chunks     = [b'x\x9c']
content    = b'x\x9c'
loop       = <_UnixSelectorEventLoop running=False closed=False debug=False>
msg        = <aiohttp.http_writer.StreamWriter object at 0x7f3f38c896a0>
payload    = b'\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00...\x1f\x1e\x1d\x1c\x1b\x1a\x19\x18\x17\x16\x15\x14\x13\x12\x11\x10\x0f\x0e\r\x0c\x0b\n\t\x08\x07\x06\x05\x04\x03\x02\x01'
protocol   = <NonCallableMagicMock spec_set='BaseProtocol' id='139909514918992'>
transport  = <NonCallableMagicMock name='mock.transport' spec_set='Transport' id='139909512338736'>

tests/test_http_writer.py:148: error

Environment

Python: 3.12.9

What Broke

Test failures lead to unreliable CI/CD pipelines and potential undetected issues in production.

Fix Options (Details)

Option A — Apply the official fix

Disables writelines for the test_write_large_payload_deflate_compression_data_in_eof test to prevent failures with Python 3.12.9+ or 3.13.2+.

When NOT to use: Do not apply this fix if the test is expected to run with writelines enabled.

Fix reference: https://github.com/aio-libs/aiohttp/pull/10423

Last verified: 2026-02-09. Validate in your environment.

When NOT to Use This Fix

Do not apply this fix if the test is expected to run with writelines enabled.

Verify Fix

verify

Re-run the minimal reproduction on your broken version, then apply the fix and re-run.

Did This Fix Work in Your Case?

Quick signal helps us prioritize which fixes to verify and improve.

Prevention

Add a stress test that runs high-concurrency workloads and fails on thread dumps / blocked locks.
Enable watchdog dumps in prod (faulthandler, thread dump endpoint) to capture deadlocks quickly.

Version Compatibility Table

Version	Status
3.12.9	Broken
3.13.2	Broken

Related Issues

No related fixes found.

Cluster: aiohttp:race-condition All hubs All clusters

Related clusters: Configuration error Data consistency Auth

Sources

We don’t republish the full GitHub discussion text. Use the links above for context.