Celery Tasks completed during a cold shutdown incorrectly

The Fix

pip install celery==5.6.0

Based on closed celery/celery issue #9505 · PR/commit linked

Production note: This usually shows up under retries/timeouts. Treat it as a side-effect risk until you can verify behavior with a canary + real traffic.

Jump to Verify Open PR/Commit

@@ -350,7 +350,7 @@ def during_soft_shutdown(worker: Worker):
 
     # Cancel all unacked requests and allow the worker to terminate naturally
-    worker.consumer.cancel_all_unacked_requests()
+    worker.consumer.cancel_active_requests()

repro.py

tox -e 3.13-smoke -- -n auto -k "test_REMAP_SIGTERM and test_cold_shutdown"
ROOT: will run in automatically provisioned tox, host /opt/homebrew/opt/[email protected]/bin/python3.10 is missing [requires (has)]: tox-gh-actions
ROOT: provision> .tox/.tox/bin/python -m tox -e 3.13-smoke -- -n auto -k 'test_REMAP_SIGTERM and test_cold_shutdown'
ROOT: tox-gh-actions won't override envlist because tox is not running in GitHub Actions
.pkg-cpython313: _optional_hooks> python /Users/nusnus/dev/GitHub/celery/.tox/.tox/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg-cpython313: get_requires_for_build_editable> python /Users/nusnus/dev/GitHub/celery/.tox/.tox/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg-cpython313: build_editable> python /Users/nusnus/dev/GitHub/celery/.tox/.tox/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
3.13-smoke: install_package> python -I -m pip install --force-reinstall --no-deps /Users/nusnus/dev/GitHub/celery/.tox/.tmp/package/33/celery-5.5.0rc4-0.editable-py3-none-any.whl
3.13-smoke: commands[0]> pytest -xsvv t/smoke --dist=loadscope --reruns 5 --reruns-delay 10 -n auto -k 'test_REMAP_SIGTERM and test_cold_shutdown'
======================================================================================= test session starts ========================================================================================
platform darwin -- Python 3.13.2, pytest-8.3.4, pluggy-1.5.0 -- /Users/nusnus/dev/GitHub/celery/.tox/3.13-smoke/bin/python
cachedir: .tox/3.13-smoke/.pytest_cache
rootdir: /Users/nusnus/dev/GitHub/celery
configfile: pyproject.toml
plugins: cov-6.0.0, docker-tools-3.1.3, subtests-0.14.1, click-1.1.0, order-1.3.0, github-actions-annotate-failures-0.3.0, anyio-4.8.0, rerunfailures-15.0, timeout-2.3.1, celery-1.2.0, xdist-3.6.1
10 workers [2 items]      
scheduling tests via LoadScopeScheduling

t/smoke/tests/test_worker.py::test_worker_shutdown::test_REMAP_SIGTERM::test_cold_shutdown[celery_setup_worker-celery_redis_broker-celery_redis_backend] 
[gw0] PASSED t/smoke/tests/test_worker.py::test_worker_shutdown::test_REMAP_SIGTERM::test_cold_shutdown[celery_setup_worker-celery_redis_broker-celery_redis_backend] 
t/smoke/tests/test_worker.py::test_worker_shutdown::test_REMAP_SIGTERM::test_cold_shutdown[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend] 
[gw0] PASSED t/smoke/tests/test_worker.py::test_worker_shutdown::test_REMAP_SIGTERM::test_cold_shutdown[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend] 

=================================================================================== 2 passed in 65.83s (0:01:05) ===================================================================================
  3.13-smoke: OK (67.81=setup[1.29]+cmd[66.52] seconds)
  congratulations :) (74.85 seconds)

verify

Re-run the minimal reproduction on your broken version, then apply the fix and re-run.

fix.md

Option A — Upgrade to fixed release\npip install celery==5.6.0\nWhen NOT to use: Do not apply this fix if using a version prior to 5.5.\n\n

Why This Fix Works in Production

Trigger: [2025-01-21 02:29:20,211: WARNING/ForkPoolWorker-1] Value: 8
Mechanism: Tasks completed during a cold shutdown incorrectly timeout due to improper cancellation logic
Why the fix works: Fixes the issue where tasks completed during a cold shutdown incorrectly timeout. (first fixed release: 5.6.0).

Production impact:

If left unfixed, the same config can fail only in production (env differences), causing startup failures or partial feature outages.

Why This Breaks in Prod

Shows up under Python 3.9 in real deployments (not just unit tests).
Tasks completed during a cold shutdown incorrectly timeout due to improper cancellation logic
Surfaces as: [2025-01-21 02:29:20,211: WARNING/ForkPoolWorker-1] Value: 8

Proof / Evidence

GitHub issue: #9505
Fix PR: https://github.com/celery/celery/pull/9678
First fixed release: 5.6.0
Reproduced locally: No (not executed)
Last verified: 2026-02-09
Confidence: 0.95
Did this fix it?: Yes (upstream fix exists)
Own content ratio: 0.20

Discussion

High-signal excerpts from the issue thread (symptoms, repros, edge-cases).

Jump to Sources Open on GitHub

“@ojaswa1942 > Hi everyone, thank you for actively addressing this issue”

@Nusnus · 2025-11-17 · confirmation · source

“> Will be released in the upcoming 5.6.0rc1 ! Love it! Thanks for all of your work to get this working!”

@daveisfera · 2025-11-17 · confirmation · source

“Yes, this is new in 5.5 and is using the new shutdown functionality of 5.5, so I don't believe this was an issue before because…”

@daveisfera · 2025-02-24 · source

“> Hi everyone, thank you for actively addressing this issue”

@Nusnus · 2025-10-26 · source

Failure Signature (Search String)

[2025-01-21 02:29:20,211: WARNING/ForkPoolWorker-1] Value: 8

Error Message

Stack trace

error.txt

Error Message
-------------
[2025-01-21 02:29:20,211: WARNING/ForkPoolWorker-1] Value: 8

worker: Hitting Ctrl+C again will terminate all running tasks!
[2025-01-21 02:29:21,132: WARNING/MainProcess] Initiating Soft Shutdown, terminating in 16 seconds
[2025-01-21 02:29:28,220: WARNING/ForkPoolWorker-1] Done: 8
[2025-01-21 02:29:28,228: INFO/ForkPoolWorker-1] Task mysite.celery.long_task[97636c13-9c46-4880-8f31-76b277a37bf5] succeeded in 8.017255892998946s: None
[2025-01-21 02:29:37,167: ERROR/MainProcess] Task handler raised error: TimeLimitExceeded(15)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 684, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
billiard.einfo.ExceptionWithTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 684, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
billiard.exceptions.TimeLimitExceeded: TimeLimitExceeded(15,)
"""
[2025-01-21 02:29:37,167: ERROR/MainProcess] Hard time limit (15s) exceeded for mysite.celery.long_task[97636c13-9c46-4880-8f31-76b277a37bf5]

worker: Cold shutdown (MainProcess)

Stack trace

error.txt

Error Message
-------------
...
[2025-01-21 02:29:37,167: ERROR/MainProcess] Task handler raised error: TimeLimitExceeded(15)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 684, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
billiard.einfo.ExceptionWithTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 684, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
billiard.exceptions.TimeLimitExceeded: TimeLimitExceeded(15,)
"""
[2025-01-21 02:29:37,167: ERROR/MainProcess] Hard time limit (15s) exceeded for mysite.celery.long_task[97636c13-9c46-4880-8f31-76b277a37bf5]
...

Stack trace

error.txt

Error Message
-------------
[2025-04-28 20:20:06,433: INFO/MainProcess] Connected to amqp://local_user:**@rabbitmq_cold_shutdown:5672//
[2025-04-28 20:20:06,457: INFO/MainProcess] celery@celery ready.
[2025-04-28 20:20:13,916: INFO/MainProcess] Task mysite.celery.long_task[e94d6bc5-d7d3-488d-a101-0557754a4fab] received
[2025-04-28 20:20:13,918: WARNING/ForkPoolWorker-1] Value: 8

worker: Hitting Ctrl+C again will terminate all running tasks!
[2025-04-28 20:20:14,124: WARNING/MainProcess] Initiating Soft Shutdown, terminating in 16 seconds
[2025-04-28 20:20:21,928: WARNING/ForkPoolWorker-1] Done: 8
[2025-04-28 20:20:21,947: INFO/ForkPoolWorker-1] Task mysite.celery.long_task[e94d6bc5-d7d3-488d-a101-0557754a4fab] succeeded in 8.028847295092419s: None
[2025-04-28 20:20:30,147: ERROR/MainProcess] Task handler raised error: TimeLimitExceeded(15)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 684, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
billiard.einfo.ExceptionWithTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 684, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
billiard.exceptions.TimeLimitExceeded: TimeLimitExceeded(15,)
"""
[2025-04-28 20:20:30,148: ERROR/MainProcess] Hard time limit (15s) exceeded for mysite.celery.long_task[e94d6bc5-d7d3-488
... (truncated) ...

Minimal Reproduction

repro.py

tox -e 3.13-smoke -- -n auto -k "test_REMAP_SIGTERM and test_cold_shutdown"
ROOT: will run in automatically provisioned tox, host /opt/homebrew/opt/[email protected]/bin/python3.10 is missing [requires (has)]: tox-gh-actions
ROOT: provision> .tox/.tox/bin/python -m tox -e 3.13-smoke -- -n auto -k 'test_REMAP_SIGTERM and test_cold_shutdown'
ROOT: tox-gh-actions won't override envlist because tox is not running in GitHub Actions
.pkg-cpython313: _optional_hooks> python /Users/nusnus/dev/GitHub/celery/.tox/.tox/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg-cpython313: get_requires_for_build_editable> python /Users/nusnus/dev/GitHub/celery/.tox/.tox/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg-cpython313: build_editable> python /Users/nusnus/dev/GitHub/celery/.tox/.tox/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
3.13-smoke: install_package> python -I -m pip install --force-reinstall --no-deps /Users/nusnus/dev/GitHub/celery/.tox/.tmp/package/33/celery-5.5.0rc4-0.editable-py3-none-any.whl
3.13-smoke: commands[0]> pytest -xsvv t/smoke --dist=loadscope --reruns 5 --reruns-delay 10 -n auto -k 'test_REMAP_SIGTERM and test_cold_shutdown'
======================================================================================= test session starts ========================================================================================
platform darwin -- Python 3.13.2, pytest-8.3.4, pluggy-1.5.0 -- /Users/nusnus/dev/GitHub/celery/.tox/3.13-smoke/bin/python
cachedir: .tox/3.13-smoke/.pytest_cache
rootdir: /Users/nusnus/dev/GitHub/celery
configfile: pyproject.toml
plugins: cov-6.0.0, docker-tools-3.1.3, subtests-0.14.1, click-1.1.0, order-1.3.0, github-actions-annotate-failures-0.3.0, anyio-4.8.0, rerunfailures-15.0, timeout-2.3.1, celery-1.2.0, xdist-3.6.1
10 workers [2 items]      
scheduling tests via LoadScopeScheduling

t/smoke/tests/test_worker.py::test_worker_shutdown::test_REMAP_SIGTERM::test_cold_shutdown[celery_setup_worker-celery_redis_broker-celery_redis_backend] 
[gw0] PASSED t/smoke/tests/test_worker.py::test_worker_shutdown::test_REMAP_SIGTERM::test_cold_shutdown[celery_setup_worker-celery_redis_broker-celery_redis_backend] 
t/smoke/tests/test_worker.py::test_worker_shutdown::test_REMAP_SIGTERM::test_cold_shutdown[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend] 
[gw0] PASSED t/smoke/tests/test_worker.py::test_worker_shutdown::test_REMAP_SIGTERM::test_cold_shutdown[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend] 

=================================================================================== 2 passed in 65.83s (0:01:05) ===================================================================================
  3.13-smoke: OK (67.81=setup[1.29]+cmd[66.52] seconds)
  congratulations :) (74.85 seconds)

Environment

Python: 3.9

What Broke

Tasks report failure instead of success during cold shutdown, causing incorrect task status.

Why It Broke

Tasks completed during a cold shutdown incorrectly timeout due to improper cancellation logic

Fix Options (Details)

Option A — Upgrade to fixed release Safe default (recommended)

pip install celery==5.6.0

When NOT to use: Do not apply this fix if using a version prior to 5.5.

Use when you can deploy the upstream fix. It is usually lower-risk than long-lived workarounds.

Fix reference: https://github.com/celery/celery/pull/9678

First fixed release: 5.6.0

Last verified: 2026-02-09. Validate in your environment.

When NOT to Use This Fix

Do not apply this fix if using a version prior to 5.5.

Verify Fix

verify

Re-run the minimal reproduction on your broken version, then apply the fix and re-run.

Did This Fix Work in Your Case?

Quick signal helps us prioritize which fixes to verify and improve.

Prevention

Make timeouts explicit and test them (unit + integration) to avoid silent behavior changes.
Instrument retries (attempt count + reason) and alert on spikes to catch dependency slowdowns.

Version Compatibility Table

Version	Status
5.6.0	Fixed

Related Issues

No related fixes found.

Cluster: celery:configuration-error Celery hub Celery best practices All hubs All clusters

Related clusters: Data consistency Duplicates Duplicate processing

Sources

We don’t republish the full GitHub discussion text. Use the links above for context.

Celery Tasks completed during a cold shutdown incorrectly timeout with 5.5 (Fix)

The Fix

Why This Fix Works in Production

Why This Breaks in Prod

Proof / Evidence

Discussion

Failure Signature (Search String)

Error Message

Minimal Reproduction

Environment

What Broke

Why It Broke

Fix Options (Details)

Option A — Upgrade to fixed release Safe default (recommended)

Get updates

When NOT to Use This Fix

Verify Fix

Did This Fix Work in Your Case?

Prevention

Version Compatibility Table

Related Issues

Sources