Fix transactions with compressed files - #2070
Conversation
Compressed wrappers do not implement commit or discard, so tracking the wrapper caused transaction exit to raise AttributeError. Track the raw filesystem file before applying compression so it can be committed or discarded after the wrapper flushes. Add a local filesystem regression test for compressed text writes. Fixes fsspec#1584
|
The windows failure looks genuine |
A compression wrapper such as GzipFile does not close the underlying file object it wraps, so on transaction commit the temp file could still hold buffered bytes and a pending gzip trailer. On Linux/macOS the subsequent rename happened to work and the data was flushed on interpreter cleanup, but on Windows the move ran against a still-open handle and produced a truncated archive (EOFError: Compressed file ended before the end-of-stream marker was reached). Close the temp file, if still open, before moving it into place.
|
You were right, it was genuine. The Windows job failed in Cause: Fix is one hunk in Verified locally on macOS (test_local.py + test_memory.py green, transaction tests green) — I do not have a Windows box, so I am relying on the pytest-win job here for confirmation. |
Compressed transactions currently queue the compression wrapper, so transaction exit calls
commit()onGzipFileand raisesAttributeError. Queue the raw filesystem file before applying compression; the wrapper still flushes on context exit, then the raw file commits normally.The regression test reproduces the reported local gzip text write and verifies that the committed file decompresses to the original data.
Fixes #1584