Celery Chain of groups causing duplicate tasks (Fix)

The Fix

pip install celery==5.5.0

Based on closed celery/celery issue #8180 · PR/commit linked

Production note: This usually shows up under retries/timeouts. Treat it as a side-effect risk until you can verify behavior with a canary + real traffic.

Jump to Verify Open PR/Commit

@@ -974,9 +974,7 @@ def __or__(self, other):
         elif isinstance(other, _chain):
             # chain | chain -> chain
-            # use type(self) for _chain subclasses
-            return type(self)(seq_concat_seq(
-                self.unchain_tasks(), other.unchain_tasks()), app=self._app)

repro.py

from celery.utils.log import get_task_logger
task_logger = get_task_logger(__name__)

import celery
assert celery.__version__ == '5.3.0b2'

@celery_app.task(bind=True)
def dummy_task(self, wn):
    task_logger.info(f'worker_num: {wn}')

@celery_app.task(bind=True)
def dummy_launch(self):
    # structure with apparently no issue
    # worker_structure = [
    #     [0, 1],
    #     [2, 3],
    #     [4],
    # ]

    # structure which causes issue. e.g., logs give 0,1,2,3,4,5,6,4,5,6
    worker_structure = [
        [0, 1],
        [2, 3],
        [4, 5],
        [6]
    ]

    # issue.
    worker_tasks = chain(group(dummy_task.si(wn) for wn in worker_nums)
        for worker_nums in worker_structure)
    
    start = [dummy_task.si('start')]
    end = [dummy_task.si('end')]

    # testing to see whether dummy tasks fix:

    # issue
    # worker_tasks = chain([group(dummy_task.si(wn) for wn in worker_nums)
    #     for worker_nums in worker_structure] + end)


    # no issue
    # worker_tasks = chain(start + [group(dummy_task.si(wn) for wn in worker_nums)
    #     for worker_nums in worker_structure])

    return self.replace(worker_tasks)

verify

Re-run the minimal reproduction on your broken version, then apply the fix and re-run.

fix.md

Option A — Upgrade to fixed release\npip install celery==5.5.0\nWhen NOT to use: Do not apply this fix if the task chaining logic is intentionally designed to allow duplicates.\n\n

Why This Fix Works in Production

Trigger: Chain of groups causing duplicate tasks
Mechanism: The issue arises when chaining chains with groups, leading to multiple task executions
Why the fix works: Fixes the issue where a task runs multiple times when chaining chains with groups by ensuring proper checks for upgrades to chords. (first fixed release: 5.5.0).

Production impact:

If left unfixed, retries/timeouts can trigger duplicate external side-effects (double charges, duplicate emails, repeated writes).

Why This Breaks in Prod

The issue arises when chaining chains with groups, leading to multiple task executions
Production symptom (often without a traceback): Chain of groups causing duplicate tasks

Proof / Evidence

GitHub issue: #8180
Fix PR: https://github.com/celery/celery/pull/9021
First fixed release: 5.5.0
Reproduced locally: No (not executed)
Last verified: 2026-02-09
Confidence: 0.75
Did this fix it?: Yes (upstream fix exists)
Own content ratio: 0.42

Discussion

High-signal excerpts from the issue thread (symptoms, repros, edge-cases).

Jump to Sources Open on GitHub

“Fixed in https://github.com/celery/celery/pull/9021”

@Nusnus · 2024-06-25 · confirmation · source

“I just ran into this as well and found this bug report after hours of debugging and questioning my sanity. @deklanw Thank you for sharing…”

@pencil · 2023-04-27 · source

“Hey @deklanw :wave:, Thank you for opening an issue”

@open-collective-bot · 2023-04-04 · source

“I am still experiencing this problem with Celery 5.4.0. I'll do more testing and report back later this week.”

@pencil · 2024-05-07 · source

Failure Signature (Search String)

Chain of groups causing duplicate tasks
- [x] I have included all related issues and possible duplicate issues

Copy-friendly signature

signature.txt

Failure Signature
-----------------
Chain of groups causing duplicate tasks
- [x] I have included all related issues and possible duplicate issues

Error Message

Signature-only (no traceback captured)

error.txt

Error Message
-------------
Chain of groups causing duplicate tasks
- [x] I have included all related issues and possible duplicate issues

Minimal Reproduction

repro.py

from celery.utils.log import get_task_logger
task_logger = get_task_logger(__name__)

import celery
assert celery.__version__ == '5.3.0b2'

@celery_app.task(bind=True)
def dummy_task(self, wn):
    task_logger.info(f'worker_num: {wn}')

@celery_app.task(bind=True)
def dummy_launch(self):
    # structure with apparently no issue
    # worker_structure = [
    #     [0, 1],
    #     [2, 3],
    #     [4],
    # ]

    # structure which causes issue. e.g., logs give 0,1,2,3,4,5,6,4,5,6
    worker_structure = [
        [0, 1],
        [2, 3],
        [4, 5],
        [6]
    ]

    # issue.
    worker_tasks = chain(group(dummy_task.si(wn) for wn in worker_nums)
        for worker_nums in worker_structure)
    
    start = [dummy_task.si('start')]
    end = [dummy_task.si('end')]

    # testing to see whether dummy tasks fix:

    # issue
    # worker_tasks = chain([group(dummy_task.si(wn) for wn in worker_nums)
    #     for worker_nums in worker_structure] + end)


    # no issue
    # worker_tasks = chain(start + [group(dummy_task.si(wn) for wn in worker_nums)
    #     for worker_nums in worker_structure])

    return self.replace(worker_tasks)

What Broke

Tasks are executed multiple times, causing unexpected behavior and potential data inconsistencies.

Why It Broke

The issue arises when chaining chains with groups, leading to multiple task executions

Fix Options (Details)

Option A — Upgrade to fixed release Safe default (recommended)

pip install celery==5.5.0

When NOT to use: Do not apply this fix if the task chaining logic is intentionally designed to allow duplicates.

Use when you can deploy the upstream fix. It is usually lower-risk than long-lived workarounds.

Fix reference: https://github.com/celery/celery/pull/9021

First fixed release: 5.5.0

Last verified: 2026-02-09. Validate in your environment.

When NOT to Use This Fix

Do not apply this fix if the task chaining logic is intentionally designed to allow duplicates.

Verify Fix

verify

Re-run the minimal reproduction on your broken version, then apply the fix and re-run.

Did This Fix Work in Your Case?

Quick signal helps us prioritize which fixes to verify and improve.

Prevention

Capture the exact failing error string in logs and tests so you can reproduce via a minimal script.
Pin production dependencies and upgrade only with a reproducible test that hits the failing path.

Version Compatibility Table

Version	Status
5.5.0	Fixed

Related Issues

No related fixes found.

Cluster: celery:duplicate-processing Celery hub Celery best practices All hubs All clusters

Related clusters: Configuration error Data consistency Duplicates

Sources

We don’t republish the full GitHub discussion text. Use the links above for context.