Celery Regard backend redis's switching to replica as

The Fix

pip install celery==5.3.0b2

Based on closed celery/celery issue #7946 · PR/commit linked

Production note: This usually shows up under retries/timeouts. Treat it as a side-effect risk until you can verify behavior with a canary + real traffic.

Jump to Verify Open PR/Commit

@@ -89,6 +89,7 @@ def __repr__(self):
         connection_retry_on_startup=Option(None, type='bool'),
         connection_max_retries=Option(100, type='int'),
+        channel_error_retry=Option(False, type='bool'),
         failover_strategy=Option(None, type='string'),
         heartbeat=Option(120, type='int'),

repro.py

[CRITICAL - 2022-11-04 10:45:27 - 21 worker:207] Unrecoverable error: ReadOnlyError("You can't write against a read only replica.")
Traceback (most recent call last):
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 326, in start
    blueprint.start(self)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 618, in start
    c.loop(*c.loop_args())
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/loops.py", line 97, in asynloop
    next(loop)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
    cb(*cbargs)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 1266, in on_readable
    self.cycle.on_readable(fileno)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 504, in on_readable
    chan.handlers[type]()
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 898, in _brpop_read
    **options)
  File "/var/www/venv/lib/python3.7/site-packages/redis/client.py", line 1189, in parse_response
    response = connection.read_response()
  File "/var/www/venv/lib/python3.7/site-packages/redis/connection.py", line 817, in read_response
    raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica.

verify

Re-run the minimal reproduction on your broken version, then apply the fix and re-run.

fix.md

Option A — Upgrade to fixed release\npip install celery==5.3.0b2\nWhen NOT to use: This fix should not be used if the application cannot tolerate retries for channel errors.\n\n

Why This Fix Works in Production

Trigger: [CRITICAL - 2022-11-04 10:45:27 - 21 worker:207] Unrecoverable error: ReadOnlyError("You can't write against a read only replica.")
Mechanism: Celery does not handle Redis failover correctly, leading to ReadOnlyError during write attempts
Why the fix works: Added a new option `broker_channel_error_retry` to handle Redis failover errors more gracefully. (first fixed release: 5.3.0b2).

Production impact:

If left unfixed, the same config can fail only in production (env differences), causing startup failures or partial feature outages.

Why This Breaks in Prod

Shows up under Python 3.7 in real deployments (not just unit tests).
Celery does not handle Redis failover correctly, leading to ReadOnlyError during write attempts
Surfaces as: [CRITICAL - 2022-11-04 10:45:27 - 21 worker:207] Unrecoverable error: ReadOnlyError("You can't write against a read only replica.")

Proof / Evidence

GitHub issue: #7946
Fix PR: https://github.com/celery/celery/pull/7951
First fixed release: 5.3.0b2
Reproduced locally: No (not executed)
Last verified: 2026-02-09
Confidence: 0.85
Did this fix it?: Yes (upstream fix exists)
Own content ratio: 0.32

Discussion

High-signal excerpts from the issue thread (symptoms, repros, edge-cases).

Jump to Sources Open on GitHub

“Hey @nkns165 :wave:, Thank you for opening an issue”

@open-collective-bot · 2022-12-07 · source

Failure Signature (Search String)

[CRITICAL - 2022-11-04 10:45:27 - 21 worker:207] Unrecoverable error: ReadOnlyError("You can't write against a read only replica.")

Error Message

Stack trace

error.txt

Error Message
-------------
[CRITICAL - 2022-11-04 10:45:27 - 21 worker:207] Unrecoverable error: ReadOnlyError("You can't write against a read only replica.")
Traceback (most recent call last):
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 326, in start
    blueprint.start(self)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 618, in start
    c.loop(*c.loop_args())
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/loops.py", line 97, in asynloop
    next(loop)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
    cb(*cbargs)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 1266, in on_readable
    self.cycle.on_readable(fileno)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 504, in on_readable
    chan.handlers[type]()
... (truncated) ...

Minimal Reproduction

repro.py

[CRITICAL - 2022-11-04 10:45:27 - 21 worker:207] Unrecoverable error: ReadOnlyError("You can't write against a read only replica.")
Traceback (most recent call last):
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 326, in start
    blueprint.start(self)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 618, in start
    c.loop(*c.loop_args())
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/loops.py", line 97, in asynloop
    next(loop)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
    cb(*cbargs)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 1266, in on_readable
    self.cycle.on_readable(fileno)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 504, in on_readable
    chan.handlers[type]()
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 898, in _brpop_read
    **options)
  File "/var/www/venv/lib/python3.7/site-packages/redis/client.py", line 1189, in parse_response
    response = connection.read_response()
  File "/var/www/venv/lib/python3.7/site-packages/redis/connection.py", line 817, in read_response
    raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica.

Environment

Python: 3.7

What Broke

Main process hangs or exits unexpectedly during Redis failover, impacting long-running tasks.

Why It Broke

Celery does not handle Redis failover correctly, leading to ReadOnlyError during write attempts

Fix Options (Details)

Option A — Upgrade to fixed release Safe default (recommended)

pip install celery==5.3.0b2

When NOT to use: This fix should not be used if the application cannot tolerate retries for channel errors.

Use when you can deploy the upstream fix. It is usually lower-risk than long-lived workarounds.

Option D — Guard side-effects with OnceOnly Guardrail for side-effects

Mitigate duplicate external side-effects under retries/timeouts/agent loops by gating the operation before calling external systems.

Place OnceOnly between your code/agent and real side-effects (Stripe, emails, CRM, APIs).
Use a stable key per side-effect (e.g., customer_id + action + idempotency_key).
Fail-safe: configure fail-open vs fail-closed based on blast radius and spend risk.

Show example snippet (optional)

onceonly.py

from onceonly import OnceOnly
import os

once = OnceOnly(api_key=os.environ["ONCEONLY_API_KEY"], fail_open=True)

# Stable idempotency key per real side-effect.
# Use a request id / job id / webhook delivery id / Stripe event id, etc.
event_id = "evt_..."  # replace
key = f"stripe:webhook:{event_id}"

res = once.check_lock(key=key, ttl=3600)
if res.duplicate:
    return {"status": "already_processed"}

# Safe to execute the side-effect exactly once.
handle_event(event_id)

See OnceOnly SDK

When NOT to use: Do not use this to hide logic bugs or data corruption. Use it to block duplicate external side-effects and enforce tool permissions/spend caps.

Fix reference: https://github.com/celery/celery/pull/7951

First fixed release: 5.3.0b2

Last verified: 2026-02-09. Validate in your environment.

When NOT to Use This Fix

This fix should not be used if the application cannot tolerate retries for channel errors.
Do not use this to hide logic bugs or data corruption. Use it to block duplicate external side-effects and enforce tool permissions/spend caps.

Verify Fix

verify

Re-run the minimal reproduction on your broken version, then apply the fix and re-run.

Did This Fix Work in Your Case?

Quick signal helps us prioritize which fixes to verify and improve.

Prevention

Capture the exact failing error string in logs and tests so you can reproduce via a minimal script.
Pin production dependencies and upgrade only with a reproducible test that hits the failing path.

Version Compatibility Table

Version	Status
5.3.0b2	Fixed

Related Issues

No related fixes found.

Cluster: celery:configuration-error Celery hub Celery best practices All hubs All clusters

Related clusters: Data consistency Duplicates Duplicate processing

Sources

We don’t republish the full GitHub discussion text. Use the links above for context.

Celery Regard backend redis's switching to replica as recoverable error (Fix)