Jump to solution
Verify

The Fix

pip install celery==5.5.0

Based on closed celery/celery issue #4842 · PR/commit linked

Jump to Verify Open PR/Commit
@@ -9,7 +9,7 @@ import warnings from collections import namedtuple -from datetime import datetime, timedelta, timezone +from datetime import timedelta from functools import partial
repro.py
[2018-06-22 14:38:03,737: ERROR/MainProcess] Received task: message, on_ack=<promise@0x7f4147022ee8 --> <bound method Consumer.call_soon of <Consumer: w_perforce@HOST (running)>>>, on_reject=<promise@0x7f4147022dc8 --> <bound method Consumer.call_soon of <Consumer: w_perforce@HOST (running)>>>, app=<Celery octo at 0x7f41496cea20>, hostname=w_perforce@HOST, eventer=<celery.events.dispatcher.EventDispatcher object at 0x7f414700fa20>, task=<@task: works.tasks.task_p4_info of octo at 0x7f41496cea20>, connection_errors=(<class 'amqp.exceptions.ConnectionError'>, <class 'OSError'>, <class 'OSError'>, <class 'OSError'>), body=b'\x80\x02]q\x00}q\x01(X\x06\x00\x00\x00branchq\x02X\x08\x00\x00\x00tkn_mainq\x03X\t\x00\x00\x00user_nameq\x04X\t\x00\x00\x00DEV_ADMINq\x05X\x03\x00\x00\x00runq\x06X\x07\x00\x00\x00p4_infoq\x07ua}q\x08}q\t(X\x05\x00\x00\x00chainq\nNX\t\x00\x00\x00callbacksq\x0bNX\x08\x00\x00\x00errbacksq\x0cNX\x05\x00\x00\x00chordq\rNu\x87q\x0e.', headers={'parent_id': None, 'retries': 0, 'id': '77c56aa8-eee1-404f-942e-3c850dc960ea', 'expires': None, 'reply_to': '5db87f93-e390-3679-b2ab-e7a0aaf48510', 'delivery_info': {'exchange': '', 'priority': None, 'redelivered': False, 'routing_key': 'perforce'}, 'argsrepr': "[{'branch': 'tkn_main', 'user_name': 'DEV_ADMIN', 'run': 'p4_info'}]", 'correlation_id': '77c56aa8-eee1-404f-942e-3c850dc960ea', 'kwargsrepr': '{}', 'root_id': '77c56aa8-eee1-404f-942e-3c850dc960ea', 'timelimit': [None, None], 'lang': 'py', 'task': 'works.tasks.task_p4_info', 'shadow': None, 'origin': 'gen31414@HOST', 'eta': None, 'group': None}, decoded=False, utc=False
verify
Re-run the minimal reproduction on your broken version, then apply the fix and re-run.
fix.md
Option A — Upgrade to fixed release\npip install celery==5.5.0\nWhen NOT to use: Do not use this fix if your application requires strict UTC timestamps for task execution.\n\n

Why This Fix Works in Production

  • Trigger: I'm still struggling with actual task time, which is use UTC time, but inspecting task obj content doesn't show me anything:
  • Mechanism: Celery was using UTC for the date_done field instead of the configured timezone
  • Why the fix works: Fixes the timezone issue in Celery where the date_done field was being saved using UTC instead of the configured timezone. (first fixed release: 5.5.0).
Production impact:
  • If left unfixed, this can cause silent data inconsistencies that propagate (bad cache entries, incorrect downstream decisions).

Why This Breaks in Prod

  • Triggered by an upgrade/regression window: 03.625825–03.630514 breaks; 5.5.0 is the first fixed release.
  • Celery was using UTC for the date_done field instead of the configured timezone
  • Production symptom (often without a traceback): I'm still struggling with actual task time, which is use UTC time, but inspecting task obj content doesn't show me anything:

Proof / Evidence

Discussion

High-signal excerpts from the issue thread (symptoms, repros, edge-cases).

“I hope this is fixed with newer releases”
@auvipy · 2019-07-06 · confirmation · source
“Here is really detailed log with references where we can see TZ was set and now it's London summer time . celery_tz_debug.log task_tz_debug.log But task…”
@trianglesis · 2018-06-22 · source
“Looks like I was wrong, this is flower change time of execution, DB and real time of task is still 1 hr in the past…”
@trianglesis · 2018-06-22 · source
“Next step: Hardcode time.py -> @cached_property def utc(self): get_tz = pytz_timezone('Europe/London') Flower Tasks board: Flower "Advanced task options" django_celery_results_taskresult 2018-06-22 19:53:52.124279 This is total mess…”
@trianglesis · 2018-06-22 · source

Failure Signature (Search String)

  • I'm still struggling with actual task time, which is use UTC time, but inspecting task obj content doesn't show me anything:
  • [2018-06-22 14:38:03,737: ERROR/MainProcess] Received task: message,
Copy-friendly signature
signature.txt
Failure Signature ----------------- I'm still struggling with actual task time, which is use UTC time, but inspecting task obj content doesn't show me anything: [2018-06-22 14:38:03,737: ERROR/MainProcess] Received task: message,

Error Message

Signature-only (no traceback captured)
error.txt
Error Message ------------- I'm still struggling with actual task time, which is use UTC time, but inspecting task obj content doesn't show me anything: [2018-06-22 14:38:03,737: ERROR/MainProcess] Received task: message,

Minimal Reproduction

repro.py
[2018-06-22 14:38:03,737: ERROR/MainProcess] Received task: message, on_ack=<promise@0x7f4147022ee8 --> <bound method Consumer.call_soon of <Consumer: w_perforce@HOST (running)>>>, on_reject=<promise@0x7f4147022dc8 --> <bound method Consumer.call_soon of <Consumer: w_perforce@HOST (running)>>>, app=<Celery octo at 0x7f41496cea20>, hostname=w_perforce@HOST, eventer=<celery.events.dispatcher.EventDispatcher object at 0x7f414700fa20>, task=<@task: works.tasks.task_p4_info of octo at 0x7f41496cea20>, connection_errors=(<class 'amqp.exceptions.ConnectionError'>, <class 'OSError'>, <class 'OSError'>, <class 'OSError'>), body=b'\x80\x02]q\x00}q\x01(X\x06\x00\x00\x00branchq\x02X\x08\x00\x00\x00tkn_mainq\x03X\t\x00\x00\x00user_nameq\x04X\t\x00\x00\x00DEV_ADMINq\x05X\x03\x00\x00\x00runq\x06X\x07\x00\x00\x00p4_infoq\x07ua}q\x08}q\t(X\x05\x00\x00\x00chainq\nNX\t\x00\x00\x00callbacksq\x0bNX\x08\x00\x00\x00errbacksq\x0cNX\x05\x00\x00\x00chordq\rNu\x87q\x0e.', headers={'parent_id': None, 'retries': 0, 'id': '77c56aa8-eee1-404f-942e-3c850dc960ea', 'expires': None, 'reply_to': '5db87f93-e390-3679-b2ab-e7a0aaf48510', 'delivery_info': {'exchange': '', 'priority': None, 'redelivered': False, 'routing_key': 'perforce'}, 'argsrepr': "[{'branch': 'tkn_main', 'user_name': 'DEV_ADMIN', 'run': 'p4_info'}]", 'correlation_id': '77c56aa8-eee1-404f-942e-3c850dc960ea', 'kwargsrepr': '{}', 'root_id': '77c56aa8-eee1-404f-942e-3c850dc960ea', 'timelimit': [None, None], 'lang': 'py', 'task': 'works.tasks.task_p4_info', 'shadow': None, 'origin': 'gen31414@HOST', 'eta': None, 'group': None}, decoded=False, utc=False

What Broke

Tasks were recorded with UTC timestamps instead of the expected local timezone.

Why It Broke

Celery was using UTC for the date_done field instead of the configured timezone

Fix Options (Details)

Option A — Upgrade to fixed release Safe default (recommended)

pip install celery==5.5.0

When NOT to use: Do not use this fix if your application requires strict UTC timestamps for task execution.

Use when you can deploy the upstream fix. It is usually lower-risk than long-lived workarounds.

Option D — Guard side-effects with OnceOnly Guardrail for side-effects

Mitigate duplicate external side-effects under retries/timeouts/agent loops by gating the operation before calling external systems.

  • Place OnceOnly between your code/agent and real side-effects (Stripe, emails, CRM, APIs).
  • Use a stable key per side-effect (e.g., customer_id + action + idempotency_key).
  • Fail-safe: configure fail-open vs fail-closed based on blast radius and spend risk.
  • This does NOT fix data corruption; it only prevents duplicate side-effects.
Show example snippet (optional)
onceonly.py
from onceonly import OnceOnly import os once = OnceOnly(api_key=os.environ["ONCEONLY_API_KEY"], fail_open=True) # Stable idempotency key per real side-effect. # Use a request id / job id / webhook delivery id / Stripe event id, etc. event_id = "evt_..." # replace key = f"stripe:webhook:{event_id}" res = once.check_lock(key=key, ttl=3600) if res.duplicate: return {"status": "already_processed"} # Safe to execute the side-effect exactly once. handle_event(event_id)

See OnceOnly SDK

When NOT to use: Do not use this to hide logic bugs or data corruption. Use it to block duplicate external side-effects and enforce tool permissions/spend caps.

Fix reference: https://github.com/celery/celery/pull/8385

First fixed release: 5.5.0

Last verified: 2026-02-09. Validate in your environment.

Get updates

We publish verified fixes weekly. No spam.

Subscribe

When NOT to Use This Fix

  • Do not use this fix if your application requires strict UTC timestamps for task execution.
  • Do not use this to hide logic bugs or data corruption. Use it to block duplicate external side-effects and enforce tool permissions/spend caps.

Verify Fix

verify
Re-run the minimal reproduction on your broken version, then apply the fix and re-run.

Did This Fix Work in Your Case?

Quick signal helps us prioritize which fixes to verify and improve.

Prevention

  • Capture the exact failing error string in logs and tests so you can reproduce via a minimal script.
  • Pin production dependencies and upgrade only with a reproducible test that hits the failing path.

Version Compatibility Table

VersionStatus
03.625825 Broken
03.629900 Broken
03.630299 Broken
03.630514 Broken
5.5.0 Fixed

Related Issues

No related fixes found.

Sources

We don’t republish the full GitHub discussion text. Use the links above for context.