Repository navigation
LZ4 and lazy decompression fixes - #277
Merged
Merged
Conversation
- Subsequent calls to ``asdf_ndarray_read_tile_2d`` on the same array accidentally reused the raw, compressed data pointer, rather than reusing an existing decompressed data pointer. - Bug causing early exit from a the lazy decompression handle_pagefault loop, if the pagefault occurred past the first page of a multi-page ``decomp.chunk_size``. This was missed in prior testing partly due to asdf-format#276 which obscuring possible issues with lazy decompression, and partly due to poor test coverage of different ``decomp.chunk_size`` settings. Better documented the lazy decompression/UFFD code since I had started to forget how some of it works myself. asdf-formatgh-274
This is explained more in the code, but the basic idea is that if multiple threads are reading simultaneously from a lazy-decompressed memory range, it could result in multiple attempts to decompress the same chunk of compressed data, which resulted in an ioctl error causing the fault handler thread to exit (thus causing any further page faults to *not* be handled and block and/or segfault the application.
macOS apparently doesn't have the pthread_barrier API, a problem I also ran into on libasdf-gwcs but forgot about. Also go ahead and commit some TRACE log messages used while debugging this. Since they are TRACE level they won't add any sig overhead by default.
embray
marked this pull request as ready for review
October 14, 2026 15:23
This was referenced Oct 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Resolves #276 and will resolve #274 when done.
AI Disclosure
No AI used.