The Fix
pip install celery==5.5.0
Based on closed celery/celery issue #4842 · PR/commit linked
@@ -9,7 +9,7 @@
import warnings
from collections import namedtuple
-from datetime import datetime, timedelta, timezone
+from datetime import timedelta
from functools import partial
[2018-06-22 14:38:03,737: ERROR/MainProcess] Received task: message,
on_ack=<promise@0x7f4147022ee8 --> <bound method Consumer.call_soon of <Consumer: w_perforce@HOST (running)>>>,
on_reject=<promise@0x7f4147022dc8 --> <bound method Consumer.call_soon of <Consumer: w_perforce@HOST (running)>>>,
app=<Celery octo at 0x7f41496cea20>,
hostname=w_perforce@HOST,
eventer=<celery.events.dispatcher.EventDispatcher object at 0x7f414700fa20>,
task=<@task: works.tasks.task_p4_info of octo at 0x7f41496cea20>,
connection_errors=(<class 'amqp.exceptions.ConnectionError'>, <class 'OSError'>, <class 'OSError'>, <class 'OSError'>),
body=b'\x80\x02]q\x00}q\x01(X\x06\x00\x00\x00branchq\x02X\x08\x00\x00\x00tkn_mainq\x03X\t\x00\x00\x00user_nameq\x04X\t\x00\x00\x00DEV_ADMINq\x05X\x03\x00\x00\x00runq\x06X\x07\x00\x00\x00p4_infoq\x07ua}q\x08}q\t(X\x05\x00\x00\x00chainq\nNX\t\x00\x00\x00callbacksq\x0bNX\x08\x00\x00\x00errbacksq\x0cNX\x05\x00\x00\x00chordq\rNu\x87q\x0e.',
headers={'parent_id': None, 'retries': 0, 'id': '77c56aa8-eee1-404f-942e-3c850dc960ea', 'expires': None, 'reply_to': '5db87f93-e390-3679-b2ab-e7a0aaf48510', 'delivery_info': {'exchange': '', 'priority': None, 'redelivered': False, 'routing_key': 'perforce'}, 'argsrepr': "[{'branch': 'tkn_main', 'user_name': 'DEV_ADMIN', 'run': 'p4_info'}]", 'correlation_id': '77c56aa8-eee1-404f-942e-3c850dc960ea', 'kwargsrepr': '{}', 'root_id': '77c56aa8-eee1-404f-942e-3c850dc960ea', 'timelimit': [None, None], 'lang': 'py', 'task': 'works.tasks.task_p4_info', 'shadow': None, 'origin': 'gen31414@HOST', 'eta': None, 'group': None},
decoded=False,
utc=False
Re-run the minimal reproduction on your broken version, then apply the fix and re-run.
Option A — Upgrade to fixed release\npip install celery==5.5.0\nWhen NOT to use: Do not use this fix if your application requires strict UTC timestamps for task execution.\n\n
Why This Fix Works in Production
- Trigger: I'm still struggling with actual task time, which is use UTC time, but inspecting task obj content doesn't show me anything:
- Mechanism: Celery was using UTC for the date_done field instead of the configured timezone
- Why the fix works: Fixes the timezone issue in Celery where the date_done field was being saved using UTC instead of the configured timezone. (first fixed release: 5.5.0).
- If left unfixed, this can cause silent data inconsistencies that propagate (bad cache entries, incorrect downstream decisions).
Why This Breaks in Prod
- Triggered by an upgrade/regression window: 03.625825–03.630514 breaks; 5.5.0 is the first fixed release.
- Celery was using UTC for the date_done field instead of the configured timezone
- Production symptom (often without a traceback): I'm still struggling with actual task time, which is use UTC time, but inspecting task obj content doesn't show me anything:
Proof / Evidence
- GitHub issue: #4842
- Fix PR: https://github.com/celery/celery/pull/8385
- Fix commit: https://github.com/celery/celery/commit/f436f10303b25291c43329704d98a906095f08b8
- First fixed release: 5.5.0
- Affected versions: 03.625825–03.630514
- Reproduced locally: No (not executed)
- Last verified: 2026-02-09
- Confidence: 0.75
- Did this fix it?: Yes (upstream fix exists)
- Own content ratio: 0.44
Discussion
High-signal excerpts from the issue thread (symptoms, repros, edge-cases).
“I hope this is fixed with newer releases”
“Here is really detailed log with references where we can see TZ was set and now it's London summer time . celery_tz_debug.log task_tz_debug.log But task…”
“Looks like I was wrong, this is flower change time of execution, DB and real time of task is still 1 hr in the past…”
“Next step: Hardcode time.py -> @cached_property def utc(self): get_tz = pytz_timezone('Europe/London') Flower Tasks board: Flower "Advanced task options" django_celery_results_taskresult 2018-06-22 19:53:52.124279 This is total mess…”
Failure Signature (Search String)
- I'm still struggling with actual task time, which is use UTC time, but inspecting task obj content doesn't show me anything:
- [2018-06-22 14:38:03,737: ERROR/MainProcess] Received task: message,
Copy-friendly signature
Failure Signature
-----------------
I'm still struggling with actual task time, which is use UTC time, but inspecting task obj content doesn't show me anything:
[2018-06-22 14:38:03,737: ERROR/MainProcess] Received task: message,
Error Message
Signature-only (no traceback captured)
Error Message
-------------
I'm still struggling with actual task time, which is use UTC time, but inspecting task obj content doesn't show me anything:
[2018-06-22 14:38:03,737: ERROR/MainProcess] Received task: message,
Minimal Reproduction
[2018-06-22 14:38:03,737: ERROR/MainProcess] Received task: message,
on_ack=<promise@0x7f4147022ee8 --> <bound method Consumer.call_soon of <Consumer: w_perforce@HOST (running)>>>,
on_reject=<promise@0x7f4147022dc8 --> <bound method Consumer.call_soon of <Consumer: w_perforce@HOST (running)>>>,
app=<Celery octo at 0x7f41496cea20>,
hostname=w_perforce@HOST,
eventer=<celery.events.dispatcher.EventDispatcher object at 0x7f414700fa20>,
task=<@task: works.tasks.task_p4_info of octo at 0x7f41496cea20>,
connection_errors=(<class 'amqp.exceptions.ConnectionError'>, <class 'OSError'>, <class 'OSError'>, <class 'OSError'>),
body=b'\x80\x02]q\x00}q\x01(X\x06\x00\x00\x00branchq\x02X\x08\x00\x00\x00tkn_mainq\x03X\t\x00\x00\x00user_nameq\x04X\t\x00\x00\x00DEV_ADMINq\x05X\x03\x00\x00\x00runq\x06X\x07\x00\x00\x00p4_infoq\x07ua}q\x08}q\t(X\x05\x00\x00\x00chainq\nNX\t\x00\x00\x00callbacksq\x0bNX\x08\x00\x00\x00errbacksq\x0cNX\x05\x00\x00\x00chordq\rNu\x87q\x0e.',
headers={'parent_id': None, 'retries': 0, 'id': '77c56aa8-eee1-404f-942e-3c850dc960ea', 'expires': None, 'reply_to': '5db87f93-e390-3679-b2ab-e7a0aaf48510', 'delivery_info': {'exchange': '', 'priority': None, 'redelivered': False, 'routing_key': 'perforce'}, 'argsrepr': "[{'branch': 'tkn_main', 'user_name': 'DEV_ADMIN', 'run': 'p4_info'}]", 'correlation_id': '77c56aa8-eee1-404f-942e-3c850dc960ea', 'kwargsrepr': '{}', 'root_id': '77c56aa8-eee1-404f-942e-3c850dc960ea', 'timelimit': [None, None], 'lang': 'py', 'task': 'works.tasks.task_p4_info', 'shadow': None, 'origin': 'gen31414@HOST', 'eta': None, 'group': None},
decoded=False,
utc=False
What Broke
Tasks were recorded with UTC timestamps instead of the expected local timezone.
Why It Broke
Celery was using UTC for the date_done field instead of the configured timezone
Fix Options (Details)
Option A — Upgrade to fixed release Safe default (recommended)
pip install celery==5.5.0
Use when you can deploy the upstream fix. It is usually lower-risk than long-lived workarounds.
Option D — Guard side-effects with OnceOnly Guardrail for side-effects
Mitigate duplicate external side-effects under retries/timeouts/agent loops by gating the operation before calling external systems.
- Place OnceOnly between your code/agent and real side-effects (Stripe, emails, CRM, APIs).
- Use a stable key per side-effect (e.g., customer_id + action + idempotency_key).
- Fail-safe: configure fail-open vs fail-closed based on blast radius and spend risk.
- This does NOT fix data corruption; it only prevents duplicate side-effects.
Show example snippet (optional)
from onceonly import OnceOnly
import os
once = OnceOnly(api_key=os.environ["ONCEONLY_API_KEY"], fail_open=True)
# Stable idempotency key per real side-effect.
# Use a request id / job id / webhook delivery id / Stripe event id, etc.
event_id = "evt_..." # replace
key = f"stripe:webhook:{event_id}"
res = once.check_lock(key=key, ttl=3600)
if res.duplicate:
return {"status": "already_processed"}
# Safe to execute the side-effect exactly once.
handle_event(event_id)
Fix reference: https://github.com/celery/celery/pull/8385
First fixed release: 5.5.0
Last verified: 2026-02-09. Validate in your environment.
When NOT to Use This Fix
- Do not use this fix if your application requires strict UTC timestamps for task execution.
- Do not use this to hide logic bugs or data corruption. Use it to block duplicate external side-effects and enforce tool permissions/spend caps.
Verify Fix
Re-run the minimal reproduction on your broken version, then apply the fix and re-run.
Did This Fix Work in Your Case?
Quick signal helps us prioritize which fixes to verify and improve.
Prevention
- Capture the exact failing error string in logs and tests so you can reproduce via a minimal script.
- Pin production dependencies and upgrade only with a reproducible test that hits the failing path.
Version Compatibility Table
| Version | Status |
|---|---|
| 03.625825 | Broken |
| 03.629900 | Broken |
| 03.630299 | Broken |
| 03.630514 | Broken |
| 5.5.0 | Fixed |
Related Issues
No related fixes found.
Sources
We don’t republish the full GitHub discussion text. Use the links above for context.