psycopg Support binary protocol for hstore (Fix)

The Fix

Adds support for the binary protocol for hstore, addressing issues with decoding hstore columns when using the binary protocol.

Based on closed psycopg/psycopg issue #1030 · PR/commit linked

Jump to Verify Open PR/Commit

@@ -20,6 +20,7 @@ Psycopg 3.2.7 (unreleased)
 
 - Add SRID support to shapely dumpers/loaders (:ticket:`#1028`).
+- Add support for binary hstore (:ticket:`#1030`).

repro.py

_U32_STRUCT = Struct('!I')
_I2B = {i: i.to_bytes(4) for i in range(256)}

class HstoreBinaryLoader(RecursiveLoader):
    format = Format.BINARY
    _encoding: str

    def load(self, data: Buffer) -> dict[str, str | None]:
        if len(data) < 12:  # Fast-path if too small to contain any data.
            return {}

        unpack_from = _U32_STRUCT.unpack_from
        encoding = self._encoding
        result = {}

        view = bytes(data)
        size, = unpack_from(view)
        pos = 4

        for _ in range(size):
            key_size, = unpack_from(view, pos)
            pos += 4

            key = view[pos : pos + key_size].decode(encoding)
            pos += key_size

            value_size, = unpack_from(view, pos)
            pos += 4

            if value_size == 0xFFFFFFFF:
                value = None
            else:
                value = view[pos : pos + value_size].decode(encoding)
                pos += value_size

            result[key] = value

        return result


class HstoreBinaryDumper(RecursiveDumper):
    format = Format.BINARY
    _encoding: str

    def dump(self, obj: dict[str, str | None]) -> Buffer:
        if not obj:
            return b'\x00\x00\x00\x00'

        pack = _U32_STRUCT.pack
        i2b = _I2B
        encoding = self._encoding
        buffer: list[bytes] = [i2b.get(l := len(obj)) or pack(l)]

        for key, value in obj.items():
            key_bytes = key.encode(encoding)
            buffer.append(i2b.get(l := len(key_bytes)) or pack(l))
            buffer.append(key_bytes)

            if value is None:
                buffer.append(b'\xFF\xFF\xFF\xFF')
            else:
                value_bytes = value.encode(encoding)
                buffer.append(i2b.get(l := len(value_bytes)) or pack(l))
                buffer.append(value_bytes)

        return b''.join(buffer)

verify

Re-run the minimal reproduction on your broken version, then apply the fix and re-run.

fix.md

Option A — Apply the official fix\nAdds support for the binary protocol for hstore, addressing issues with decoding hstore columns when using the binary protocol.\nWhen NOT to use: This fix should not be used if the application relies on the previous behavior of hstore decoding.\n\n

Why This Fix Works in Production

Trigger: I was a fan of the `pack_into=struct.pack_into` hack, but from some measurements I took not many years ago it doesn't make anymore a substantial difference, in…
Mechanism: Hstore columns are not correctly decoded when using the binary protocol, leading to type errors

Production impact:

If left unfixed, this can cause silent data inconsistencies that propagate (bad cache entries, incorrect downstream decisions).

Why This Breaks in Prod

Hstore columns are not correctly decoded when using the binary protocol, leading to type errors
Production symptom (often without a traceback): I was a fan of the `pack_into=struct.pack_into` hack, but from some measurements I took not many years ago it doesn't make anymore a substantial difference, in modern versions of Python.

Proof / Evidence

GitHub issue: #1030
Fix PR: https://github.com/psycopg/psycopg/pull/1031
Reproduced locally: No (not executed)
Last verified: 2026-02-09
Confidence: 0.70
Did this fix it?: Yes (upstream fix exists)
Own content ratio: 0.43

Discussion

High-signal excerpts from the issue thread (symptoms, repros, edge-cases).

Jump to Sources Open on GitHub

“You can speed up the execution 20x by writing this stuff in Cython, just sayin'. Using pack_into() from a single Struct(">I") instance is more than…”

@dvarrazzo · 2025-03-21 · source

“TIL there exists Struct class 😮”

@Zaczero · 2025-03-21 · source

“Also, to clarify, what I meant for > it doesn't make anymore a substantial difference, in modern versions of Python”

@dvarrazzo · 2025-03-21 · source

“Continued in https://github.com/psycopg/psycopg/pull/1031”

@Zaczero · 2025-03-21 · source

Failure Signature (Search String)

I was a fan of the `pack_into=struct.pack_into` hack, but from some measurements I took not many years ago it doesn't make anymore a substantial difference, in modern versions of
it doesn't make anymore a substantial difference, in modern versions of Python.

Copy-friendly signature

signature.txt

Failure Signature
-----------------
I was a fan of the `pack_into=struct.pack_into` hack, but from some measurements I took not many years ago it doesn't make anymore a substantial difference, in modern versions of Python.
it doesn't make anymore a substantial difference, in modern versions of Python.

Error Message

Signature-only (no traceback captured)

error.txt

Error Message
-------------
I was a fan of the `pack_into=struct.pack_into` hack, but from some measurements I took not many years ago it doesn't make anymore a substantial difference, in modern versions of Python.
it doesn't make anymore a substantial difference, in modern versions of Python.

Minimal Reproduction

repro.py

_U32_STRUCT = Struct('!I')
_I2B = {i: i.to_bytes(4) for i in range(256)}

class HstoreBinaryLoader(RecursiveLoader):
    format = Format.BINARY
    _encoding: str

    def load(self, data: Buffer) -> dict[str, str | None]:
        if len(data) < 12:  # Fast-path if too small to contain any data.
            return {}

        unpack_from = _U32_STRUCT.unpack_from
        encoding = self._encoding
        result = {}

        view = bytes(data)
        size, = unpack_from(view)
        pos = 4

        for _ in range(size):
            key_size, = unpack_from(view, pos)
            pos += 4

            key = view[pos : pos + key_size].decode(encoding)
            pos += key_size

            value_size, = unpack_from(view, pos)
            pos += 4

            if value_size == 0xFFFFFFFF:
                value = None
            else:
                value = view[pos : pos + value_size].decode(encoding)
                pos += value_size

            result[key] = value

        return result


class HstoreBinaryDumper(RecursiveDumper):
    format = Format.BINARY
    _encoding: str

    def dump(self, obj: dict[str, str | None]) -> Buffer:
        if not obj:
            return b'\x00\x00\x00\x00'

        pack = _U32_STRUCT.pack
        i2b = _I2B
        encoding = self._encoding
        buffer: list[bytes] = [i2b.get(l := len(obj)) or pack(l)]

        for key, value in obj.items():
            key_bytes = key.encode(encoding)
            buffer.append(i2b.get(l := len(key_bytes)) or pack(l))
            buffer.append(key_bytes)

            if value is None:
                buffer.append(b'\xFF\xFF\xFF\xFF')
            else:
                value_bytes = value.encode(encoding)
                buffer.append(i2b.get(l := len(value_bytes)) or pack(l))
                buffer.append(value_bytes)

        return b''.join(buffer)

What Broke

Data is received as raw bytes instead of a Python dictionary, causing type errors.

Why It Broke

Hstore columns are not correctly decoded when using the binary protocol, leading to type errors

Fix Options (Details)

Option A — Apply the official fix

Adds support for the binary protocol for hstore, addressing issues with decoding hstore columns when using the binary protocol.

When NOT to use: This fix should not be used if the application relies on the previous behavior of hstore decoding.

Fix reference: https://github.com/psycopg/psycopg/pull/1031

Last verified: 2026-02-09. Validate in your environment.

When NOT to Use This Fix

This fix should not be used if the application relies on the previous behavior of hstore decoding.

Verify Fix

verify

Re-run the minimal reproduction on your broken version, then apply the fix and re-run.

Did This Fix Work in Your Case?

Quick signal helps us prioritize which fixes to verify and improve.

Prevention

Capture the exact failing error string in logs and tests so you can reproduce via a minimal script.
Pin production dependencies and upgrade only with a reproducible test that hits the failing path.

Related Issues

No related fixes found.

Cluster: psycopg:data-consistency All hubs All clusters

Related clusters: Configuration error Timeout Race condition

Sources

We don’t republish the full GitHub discussion text. Use the links above for context.