Question about pyzstd.compress() determinism vs compression.zstd

[datahack00]

I hope this message finds you well. First of all, I sincerely apologize for reaching out this way. I couldn't find a better channel, and I hope you don't mind.

I also want to take a moment to thank you for your work on pyzstd. It has served us reliably, and we genuinely appreciate the effort you've put into the library.

We are currently in the process of migrating our compression layer from pyzstd to Python's built-in compression.zstd module (introduced in Python 3.14), and during that transition, we noticed an interesting behavioral difference that we haven't been able to fully explain.

With pyzstd, the following call always produced an identical byte output across multiple invocations:

pyzstd.compress(
    data=content,
    zstd_dict=training_dict.as_digested_dict
)

However, with the new built-in module, the equivalent call:

zstd.compress(
    data=content,
    zstd_dict=training_dict.as_digested_dict
)

...produces byte objects of different sizes across two consecutive calls with the same input.

One additional detail that may be relevant: between the two compress() calls, we reload the dictionary from disk using:

training_dict = zstd.ZstdDict(zst_file)

It is the exact same .zst file each time, we simply call this line again before the second compress() call. We're not sure whether re-instantiating ZstdDict from the same file could have any effect on the resulting compressed output, but we wanted to mention it in case it's a factor.

Would you happen to know why this might be the case?

Thank you so much for your time, and apologies again for the interruption.

[datahack00]

input = {
  "test_1": 1283998000000,
  "test_2": 0,
  "test_3": "test"
}

CONTENT = json.dumps(input).encode('utf-8')

def _load_dict(path: str) -> zstd.ZstdDict:
    with open(path, "rb") as f:
        return zstd.ZstdDict(f.read())

def test_compress_after_reloading_dict_from_disk_produces_identical_output():
    training_dict = _load_dict(DICT_PATH)
    result_1 = zstd.compress(data=CONTENT, zstd_dict=training_dict.as_digested_dict)

    training_dict = _load_dict(DICT_PATH)
    result_2 = zstd.compress(data=CONTENT, zstd_dict=training_dict.as_digested_dict)

    assert result_1 == result_2, (
        f"Reloading ZstdDict from disk changed the compressed output.\n"
        f"  Before reload: {len(result_1)} bytes\n"
        f"  After reload:  {len(result_2)} bytes"
    )

When invoked multiple times, sometimes test_compress_after_reloading_dict_from_disk_produces_identical_output fails, sometimes passes.

[Rogdham]

Hello and no worries, this is an appropriate channel to ask such questions!

You report is quite complete already, it's only missing the dictionary file to be able to reproduce it entirely on my side and troubleshoot. Could you share the dictionary file, or a way to generate it that triggers the issue?

Also please share your Python version and OS, as well as the version of pyzstd.

For the record, pyzstd uses compression.zstd internally since version 0.19.0.

[datahack00]

Hey @Rogdham, we've been using pyzstd==0.17.0 and now Python 3.14 with compression.zstd. My local OS is ProductName: macOS, ProductVersion: 26.5, BuildVersion: 25F71. Unfortunately, I can not send the zst dict file as it may contain sensitive data, but I think the behaviour is the same regardless of the specific .zst file.

We had some legacy unit tests that were directly comparing byte values, and they started failing after I switched from pyzstd to compression.zstd. Since this area is considered quite sensitive on our side, others wanted to better understand what might be the reason behind this behavior before giving the green light for the migration to compression.zstd.

[Rogdham]

Reproducer:

import pyzstd as zstd

DICT = b"7\xa40\xec/<\xabJ\t\x10\x10\xdf033\xb3w\n3\xf1x<\x1e\x8f\xc7\xe3\xf1x<\xcf\xf3\xbc\xf7\xd4BAAAAAAAAAAAAAAAAAAAAAAAAA\xa1P(\x14\n\x85B\xa1P(\x14\n\x85\xa2(\x8a\xa2(J)}t\xe1\xe1\xe1\xe1\xe1\xe1\xe1\xe1\xe1\xe1\xe1\xe1\xe1\xe1\xe1\xe1\xe1\xe1\xe1\xf1x<\x1e\x8f\xc7\xe3\xf1x\x9e\xe7y\xef\x01\x01\x00\x00\x00\x04\x00\x00\x00\x08\x00\x00\x00helloworld test data sample foo bar baz hello w"
DATA = b"hello world test"

last = None
for i in range(10000):
    value = zstd.compress(DATA, zstd_dict=zstd.ZstdDict(DICT).as_digested_dict)
    if value != last:
        print(f"{value.hex()} on run {i}")
        last = value

Output:

• With version 0.18.0: seems to only have a single value
• With version 0.19.0 (uses stdlib internally or its backport): several values

$ uv run -p python3.13 --with 'pyzstd==0.18.0' reproducer.py
28b52ffd232f3cab4a106500003068656c6c6f200100735345 on run 0

$ uv run -p python3.13 --with 'pyzstd==0.19.0' reproducer.py
28b52ffd232f3cab4a1081000068656c6c6f20776f726c642074657374 on run 0
28b52ffd232f3cab4a106500001868207702005315ca40b101 on run 3

Note: the actual values changes from one run to the other.

[Rogdham]

After some analysis, this seems like a bug in CPython to me.

I have opened a ticket there and opened a PR for a fix!