Skip to content

Support random-access ZIP input without buffering the whole archive #66

Closed

Description

@andrew

Browsing a large ZIP that is already stored in a local file currently requires buffering the entire compressed archive. OpenStream and OpenBrowser accept io.Reader and read ZIP input into a byte slice even when the original input also supports io.ReaderAt.

To reproduce, wrap a ZIP input in a reader that counts Read and ReadAt calls, then call OpenBrowser and List. The current implementation reads the entire input sequentially and never calls ReadAt.

Add an explicit constructor accepting io.ReaderAt and an archive size, allowing ZIP metadata and member bodies to be read directly from the source. Keep the existing forward-only input API available. mholt/archives uses random-access inputs and section readers for this purpose.

Preserve input-size checks, entry-count checks before allocating ZIP metadata, expanded-size limits, and cancellation. Define ownership of the underlying input and how whole-artifact hashing works if this is exposed through Reader. Conda's outer ZIP could also use this input path.

Tests should exercise the public API with a real file and an instrumented random-access reader, including malformed central directories and limits. A benchmark should compare allocations when listing a large ZIP and extracting one small member.

Source inspected: origin/main at dc26815591e2353cf40dc890953483664f882bdf. This is a library, so there is no git-pkgs --version output for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions