Skip to content

fix(filetransfer): recover completed downloads after EOF resume - #1

Draft
elezar wants to merge 1 commit into
masterfrom
codex/fix-resume-at-eof
Draft

elezar wants to merge 1 commit into
masterfrom
codex/fix-resume-at-eof

Conversation

@elezar

@elezar elezar commented Oct 2, 2026

Copy link
Copy Markdown
Owner

Personal-fork draft for sharing and human review. This is not an upstream submission.

An interrupted streaming download can deliver the entire payload and still report a transport error. Nix then resumes at the payload length, receives HTTP 416, and fails the download even though all bytes have already arrived.

This change accepts a resumed GET's 416 as completion only when libcurl reports success, Content-Range: bytes */N exactly matches the resume offset and streamed byte count, the response has no non-identity content encoding, and any ETags supplied on both the streamed response and the 416 agree. Missing or inconsistent length evidence still fails. Existing downstream hash validation remains in place; the change does not restart or replay bytes into the streaming sink.

The regression fixture sends a chunked response without the final chunk, records the actual Range request, and supplies an expected SHA-256 hash to nix-prefetch-url. It is wired into the existing filetransfer-retry-backoff integration derivation.

Validation:

  • Built the patched executable from master (reports Nix 2.36.0) on aarch64-darwin.
  • All nine focused cases passed: complete download, partial resume, EOF resume, missing/malformed Content-Range, shorter/longer object size, changed ETag, and corrupted same-length content rejected by hash validation.
  • Installed Nix 2.34.6 reproduced the EOF failure using the same fixture in baseline mode.
  • Changed-file pre-commit checks and git diff --check passed.
  • The existing six-case retry/backoff integration script passed against the patched executable with deterministic retry jitter settings.

The full upstream unit/functional suites and the Hydra integration derivation have not been run.

Context: the investigation started with cache.nixos.org failures in NVIDIA/OpenShell PR 4066. The CI logs show Nix 2.35.2 retrying at offsets equal to narinfo FileSize, followed by HTTP 416. The synthetic test uses a different transport error and does not establish the cause of those original failures. Related cache report: NixOS/infra#1106.

AI assistance: Codex assisted with investigation, implementation, tests, and this description. Human review and human-authored contribution text are still required before any upstream submission under Nix's contribution policy.

Assisted-by: Codex (unspecified)
Signed-off-by: Evan Lezar <elezar@nvidia.com>
if (writtenToSink)
curl_easy_setopt(req, CURLOPT_RESUME_FROM_LARGE, writtenToSink);
resumeOffset = writtenToSink;
if (resumeOffset)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't written a lot of cpp since university and some small envoy PRs, but I remember having the if () {} curly braces being heavily encouraged for even one-line statements. though now I'm cross-referencing against the google style guide and they're okay with it, caltech's guide is against it. as long as it lints against nix's style guide ¯_(ツ)_/¯

(it feels weird to be giving a nit: comment on code style these days, almost nostalgic)

@matthewgrossman matthewgrossman left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense to me, as you saw in the openshell PR it seems like a semi-common issue around the community so I'm almost surprised we were hitting it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants