Celery Exception Serialization error? Tasks raising

The Fix

pip install celery==4.4.0rc5

Based on closed celery/celery issue #4860 · PR/commit linked

Jump to Verify Open PR/Commit

@@ -260,3 +260,4 @@ Igor Kasianov, 2018/01/20
 Chris Mitchell, 2018/02/27
 Josue Balandrano Coronel, 2018/05/24
+Tom Booth, 2018/07/06
diff --git a/celery/backends/base.py b/celery/backends/base.py
index eed17a2582c..2837d522bcb 100644

repro.py

Client caught exception in _post(). Unable to retry.
Traceback (most recent call last):
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), sel
f.timeout, **extra_kw)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection
    raise err
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the 
above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/va
r/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connection.py", line 314, in connect
    conn = self._new_conn()
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
    self, "F
ailed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f695118a4a8>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/celery/venv/lib
/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, 
error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='sjc1.solvedirect.com', port=443): Max retries exceeded with url: /ws/rest/oauth/token?grant_type=client_credentials (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f695118a4a8>: Failed to establish a new connection: [Errno 110] Connection timed out
',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/celery/venv/lib/python3.6/site-packages/cisco_tac/ciscotac.py", line 67, in _make_post_request
    result = requests.request('POST', url, json=payload, headers=header, verify=True, auth=auth)
  File "/var/lib/celery/venv/lib/python3.6
/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adap
ter.send(request, **kwargs)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/requests/adapters.py", line 513, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='sjc1.solvedirect.com', port=443): Max retries exceeded with url: /ws/rest/oauth/token?grant_type=client_credentials (Caused by NewConnecti
onError('<urllib3.connection.VerifiedHTTPSConnection object at
... (truncated) ...

verify

Re-run the minimal reproduction on your broken version, then apply the fix and re-run.

fix.md

Option A — Upgrade to fixed release\npip install celery==4.4.0rc5\nWhen NOT to use: This fix should not be used if the application does not require JSON serialization for exceptions.\n\n

Why This Fix Works in Production

Trigger: Client caught exception in _post(). Unable to retry.
Mechanism: Exception arguments that are not JSON-serializable caused tasks to get stuck in a 'PENDING' state
Why the fix works: Fixes an issue where exception arguments that are not JSON-serializable caused tasks to get stuck in a 'PENDING' state. (first fixed release: 4.4.0rc5).

Production impact:

If left unfixed, this can cause silent data inconsistencies that propagate (bad cache entries, incorrect downstream decisions).

Why This Breaks in Prod

Shows up under Python 3.6 in real deployments (not just unit tests).
Exception arguments that are not JSON-serializable caused tasks to get stuck in a 'PENDING' state
Surfaces as: Client caught exception in _post(). Unable to retry.

Proof / Evidence

GitHub issue: #4860
Fix PR: https://github.com/celery/celery/pull/4864
First fixed release: 4.4.0rc5
Reproduced locally: No (not executed)
Last verified: 2026-02-09
Confidence: 0.85
Did this fix it?: Yes (upstream fix exists)
Own content ratio: 0.30

Discussion

High-signal excerpts from the issue thread (symptoms, repros, edge-cases).

Jump to Sources Open on GitHub

“This looks like an exception serialization error. trying to repro the problem, we caught this nasty traceback on the worker: https://pastebin.com/F4m8gzie”

@johnarnold · 2018-06-28 · repro detail · source

“Possibly related -- kombu catches all connection errors defined by the provider (redis in my case): https://github.com/celery/kombu/issues/802”

@johnarnold · 2018-06-28 · source

“It seems like tasks should not be able to bubble up exceptions which will break kombu....”

@johnarnold · 2018-06-28 · source

“Acutally the original issue may be different than those Kombu issues..”

@johnarnold · 2018-06-28 · source

Failure Signature (Search String)

Client caught exception in _post(). Unable to retry.

Error Message

Stack trace

error.txt

Error Message
-------------
Client caught exception in _post(). Unable to retry.
Traceback (most recent call last):
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), sel
f.timeout, **extra_kw)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection
    raise err
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the 
above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/va
r/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connection.py", line 314, in connect
    conn = self._new_conn()
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
    self, "F
ailed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionE
... (truncated) ...

Minimal Reproduction

repro.py

Client caught exception in _post(). Unable to retry.
Traceback (most recent call last):
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), sel
f.timeout, **extra_kw)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection
    raise err
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the 
above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/va
r/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connection.py", line 314, in connect
    conn = self._new_conn()
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
    self, "F
ailed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f695118a4a8>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/celery/venv/lib
/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, 
error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='sjc1.solvedirect.com', port=443): Max retries exceeded with url: /ws/rest/oauth/token?grant_type=client_credentials (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f695118a4a8>: Failed to establish a new connection: [Errno 110] Connection timed out
',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/celery/venv/lib/python3.6/site-packages/cisco_tac/ciscotac.py", line 67, in _make_post_request
    result = requests.request('POST', url, json=payload, headers=header, verify=True, auth=auth)
  File "/var/lib/celery/venv/lib/python3.6
/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adap
ter.send(request, **kwargs)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/requests/adapters.py", line 513, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='sjc1.solvedirect.com', port=443): Max retries exceeded with url: /ws/rest/oauth/token?grant_type=client_credentials (Caused by NewConnecti
onError('<urllib3.connection.VerifiedHTTPSConnection object at
... (truncated) ...

Environment

Python: 3.6

What Broke

Tasks raising exceptions remain in 'PENDING' state, leading to unprocessed tasks and potential data loss.

Why It Broke

Exception arguments that are not JSON-serializable caused tasks to get stuck in a 'PENDING' state

Fix Options (Details)

Option A — Upgrade to fixed release Safe default (recommended)

pip install celery==4.4.0rc5

When NOT to use: This fix should not be used if the application does not require JSON serialization for exceptions.

Use when you can deploy the upstream fix. It is usually lower-risk than long-lived workarounds.

Fix reference: https://github.com/celery/celery/pull/4864

First fixed release: 4.4.0rc5

Last verified: 2026-02-09. Validate in your environment.

When NOT to Use This Fix

This fix should not be used if the application does not require JSON serialization for exceptions.

Verify Fix

verify

Re-run the minimal reproduction on your broken version, then apply the fix and re-run.

Did This Fix Work in Your Case?

Quick signal helps us prioritize which fixes to verify and improve.

Prevention

Track RSS + object counts after deployments; alert on monotonic growth and GC pressure.
Add a long-running test that repeats the failing call path and asserts stable memory.

Version Compatibility Table

Version	Status
4.4.0rc5	Fixed

Related Issues

No related fixes found.

Cluster: celery:data-consistency Celery hub Celery best practices All hubs All clusters

Related clusters: Configuration error Duplicates Duplicate processing

Sources

We don’t republish the full GitHub discussion text. Use the links above for context.

Celery Exception Serialization error? Tasks raising exceptions stuck in "PENDING" state (Fix)

The Fix

Why This Fix Works in Production

Why This Breaks in Prod

Proof / Evidence

Discussion

Failure Signature (Search String)

Error Message

Minimal Reproduction

Environment

What Broke

Why It Broke

Fix Options (Details)

Option A — Upgrade to fixed release Safe default (recommended)

Get updates

When NOT to Use This Fix

Verify Fix

Did This Fix Work in Your Case?

Prevention

Version Compatibility Table

Related Issues

Sources