Repository navigation
fork fleet: desk seed fetch round-5 nits (#258) - #729
Conversation
- An early try (a new copy on the host during a back-off wait) keeps `last`: it still counts, but never moves the end of the wait; the log line says the real time left. - Every `_proc_gnusparse_*` handler of this Python's tarfile is refused, and each run reads a small sparse tar of each pax form (0.0, 0.1, 1.0) to prove tarfile still reaches the refusal; if not, it fetches nothing (exit 2). - Only a block of zeros ends the tar (any other bad header tears the seed), and only zeros may follow it, read through tarfile's own buffer. - Docs: a fix made right after another new copy waits for the step (up to 24 h); the early-try rule; the tar end; the sparse check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review lens A (untrusted tar), head 9660496: SAFERead-only review. I read #258, #237's design, and this PR's script, docs and tests. I also ran my own crafted tars through Blocking: none. What I tried (all torn or harmless on both Pythons)
Can a crafted seed get the retry schedule to fetch more often, or stop a fix being retried?No new route. Minor
Note (not part of this PR)A host can still reset the back-off by leaving the seed out of its list for one run ( State question
|
Lens B (state and tests) on 9660496: SAFENo blocking findings. One major doc finding: the docs promise a retry timing the code does not have. It is a wording fix, not a safety problem. BlockingNone. Major
Minor
State question
Tests (Linux, Python 3.12, run on this machine)
|
- Docs: an early try does move the next normal try one step later (the wait doubles from the last try that waited; only at the 24 h cap not). Five lines now say so; the early-try test is renamed to match and also checks the 24 h cap. - sparse_read: the probe tar is made inside its try, so a tarfile that cannot write it gives exit 2 and old seeds are still dropped. Only the probe's own refusal counts as refused; another Torn, a tarfile error or a member read whole is named and fails closed. New probes: an old-GNU sparse member and a header over HEADER_MOST, which prove tarfile still calls Info._proc_member. - The tar-end test also accepts the size -1 message of Python 3.13. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix (head d666cd6) for lens A and lens B on 9660496Changed
Tests: Left, with why
|
* fork fleet: desk seed fetch round-5 nits (#258) - An early try (a new copy on the host during a back-off wait) keeps `last`: it still counts, but never moves the end of the wait; the log line says the real time left. - Every `_proc_gnusparse_*` handler of this Python's tarfile is refused, and each run reads a small sparse tar of each pax form (0.0, 0.1, 1.0) to prove tarfile still reaches the refusal; if not, it fetches nothing (exit 2). - Only a block of zeros ends the tar (any other bad header tears the seed), and only zeros may follow it, read through tarfile's own buffer. - Docs: a fix made right after another new copy waits for the step (up to 24 h); the early-try rule; the tar end; the sparse check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fork fleet: desk seed fetch, fixes from the #729 reviews (#258) - Docs: an early try does move the next normal try one step later (the wait doubles from the last try that waited; only at the 24 h cap not). Five lines now say so; the early-try test is renamed to match and also checks the 24 h cap. - sparse_read: the probe tar is made inside its try, so a tarfile that cannot write it gives exit 2 and old seeds are still dropped. Only the probe's own refusal counts as refused; another Torn, a tarfile error or a member read whole is named and fails closed. New probes: an old-GNU sparse member and a header over HEADER_MOST, which prove tarfile still calls Info._proc_member. - The tar-end test also accepts the size -1 message of Python 3.13. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fork fleet: desk seed fetch round-5 nits (#258) - An early try (a new copy on the host during a back-off wait) keeps `last`: it still counts, but never moves the end of the wait; the log line says the real time left. - Every `_proc_gnusparse_*` handler of this Python's tarfile is refused, and each run reads a small sparse tar of each pax form (0.0, 0.1, 1.0) to prove tarfile still reaches the refusal; if not, it fetches nothing (exit 2). - Only a block of zeros ends the tar (any other bad header tears the seed), and only zeros may follow it, read through tarfile's own buffer. - Docs: a fix made right after another new copy waits for the step (up to 24 h); the early-try rule; the tar end; the sparse check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fork fleet: desk seed fetch, fixes from the #729 reviews (#258) - Docs: an early try does move the next normal try one step later (the wait doubles from the last try that waited; only at the 24 h cap not). Five lines now say so; the early-try test is renamed to match and also checks the 24 h cap. - sparse_read: the probe tar is made inside its try, so a tarfile that cannot write it gives exit 2 and old seeds are still dropped. Only the probe's own refusal counts as refused; another Torn, a tarfile error or a member read whole is named and fails closed. New probes: an old-GNU sparse member and a header over HEADER_MOST, which prove tarfile still calls Info._proc_member. - The tar-end test also accepts the size -1 message of Python 3.13. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fork fleet: desk seed fetch round-5 nits (#258) - An early try (a new copy on the host during a back-off wait) keeps `last`: it still counts, but never moves the end of the wait; the log line says the real time left. - Every `_proc_gnusparse_*` handler of this Python's tarfile is refused, and each run reads a small sparse tar of each pax form (0.0, 0.1, 1.0) to prove tarfile still reaches the refusal; if not, it fetches nothing (exit 2). - Only a block of zeros ends the tar (any other bad header tears the seed), and only zeros may follow it, read through tarfile's own buffer. - Docs: a fix made right after another new copy waits for the step (up to 24 h); the early-try rule; the tar end; the sparse check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fork fleet: desk seed fetch, fixes from the #729 reviews (#258) - Docs: an early try does move the next normal try one step later (the wait doubles from the last try that waited; only at the 24 h cap not). Five lines now say so; the early-try test is renamed to match and also checks the 24 h cap. - sparse_read: the probe tar is made inside its try, so a tarfile that cannot write it gives exit 2 and old seeds are still dropped. Only the probe's own refusal counts as refused; another Torn, a tarfile error or a member read whole is named and fails closed. New probes: an old-GNU sparse member and a header over HEADER_MOST, which prove tarfile still calls Info._proc_member. - The tar-end test also accepts the size -1 message of Python 3.13. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Follow-ups from the round-5 lenses on #237 (desk seed fetch): security lens237sr5 and correctness lens237cr5. All four items in #258 are fixed in code, docs, or both.
What and why
TornLogdocstring,fork-fleet.mdand the desk README now say that a fix made right after another new copy waits for the end of the step, up to 24 h. This trade-off is intended.last, so the wait runs from the last try that waited, not from the early try. So an early try moves the next normal try at most one step later (h 5, not h 3, after an early try at h 1.5), and not at all at the 24 h cap. Before, each early try also restarted the wait from itself. The log line after a torn fetch gives the real time left (TornLog.due), not2^(tries-1)from now. An hourly broken remake now gets tries at h 0, 1, 2, 5, 6 (before: 0, 1, 2, 6, 7). The test that pins this schedule is updated.Infonow refuses every_proc_gnusparse_*that this Python'starfile.TarInfohas, found by name at import. Before, only 00, 01 and 10 were listed by hand, so a later tarfile's new handler would not be overridden.Tar/Info(sparse_read): one pax sparse tar of each form (0.0, 0.1, 1.0), an old-GNU sparse member, and a header overHEADER_MOST. The last two prove tarfile still callsInfo._proc_member. Each probe must be torn by its own refusal. Anything else (read as a sparse member, read whole, torn by another limit, a tarfile error, or the probe cannot even be made) is named, and the run fetches nothing and exits 2, like a missingPRIVATEpart (checkshows it). Seeds past their age are still dropped.Info.fromtarfilelets onlyEOFHeaderError(a block of zeros) end the tar. Any other header error now tears the seed: a bad checksum, a paxsize<0, a stream that ends with no end blocks, or a bad header as the last block. Before,TarFile.next()ended the tar quietly at any of these after the first header, and a short seed was swapped in.t.fileobj), and only zero bytes are accepted. These bytes still count toward the trailer's size and SHA-256. Normal end blocks and record padding pass.Not changed: the host-down-for-good row (cr4 2, documented in #237) is not part of #258.
State question
lastand setsearly, so a second new copy in the same step waits until the step ends. Tested:test_an_early_try_keeps_the_wait_running_from_the_last_try_that_waited(also: at the 24 h cap an early try moves nothing)..torn.jsonis saved before the fetch. The test reads it inside the fetch, which is what a run killed there leaves:lastunchanged, tries + 1,earlytrue. The next run waits for the step that was already running.last, is read as it is. It waits from thatlastand is tried once the step ends (test_an_old_record_whose_early_try_moved_last_still_waits). A record withoutearly, or withoutcopy, behaves as before.ReadErroror a quiet end. A tar of only end blocks still unpacks as an empty seed.Tests
.github/scripts/fork-fleet-desk-seed.test.py: 7 new tests, plus the hourly-remake test now expects h 0, 1, 2, 5, 6 and checks the hours in the log line. 77 tests, OK on Linux, Python 3.12 (CI's) and 3.13.5.sparse_readmissing_proc_gnusparse_11not overriddentest_an_old_record_whose_early_try_moved_last_still_waitspasses on both. It checks old state, not a change.fork-gate-trust-docs.test.py(86 OK) andfork-owned-reads.test.py(4 OK).fork-fleet-desk.test.pyandfork-kit/desk/tests/install.test.pyare Linux-only; they fail the same way on origin/main on macOS, and CI runs them.node .github/scripts/fork-seam-check.cjs --config-ref origin/main origin/main HEAD: passes. Only fork files change.Spot-checks: each fix line was removed alone, and its test went red:
lastkept on an early try ->test_an_early_try_never_moves_the_next_try_laterdue->test_a_host_that_remakes_each_hour_never_meets_the_back_off_proc_gnusparse_*(back to the 3 fixed names) ->test_a_sparse_form_a_later_tarfile_adds_is_refused_toosparse_readinrun->test_a_python_that_reads_a_sparse_member_past_info_fetches_nothing(all 3 forms)refusal not in str(e)removed) ->test_only_its_own_refusal_proves_a_probefromtarfileturning header errors intoTorn->test_only_a_block_of_zeros_ends_the_tar_and_only_zeros_follow. The bad-header-as-last-block and no-end-blocks cases are caught only by this line; the bad checksum followed by more members is also caught by the zeros check.Review round (lens A and B on 9660496, both SAFE)
Fixed in d666cd6:
test_an_early_try_keeps_the_wait_running_from_the_last_try_that_waited.try. A tarfile that cannot write it now gives exit 2 and still drops seeds past their age (test_a_probe_that_cannot_be_made_still_drops_an_old_seed_and_is_exit_2; on the old script: a rawValueError, exit 1).Torn, a tarfile error, or a member read whole is named and fails closed (test_only_its_own_refusal_proves_a_probe; fails on the old script).Smember and a header overHEADER_MOST, prove tarfile still callsInfo._proc_member(same test,_proc_membercase).Left: lens A's note that a host can reset the back-off by leaving a seed out of its list for one run. #237 documents it, and the host already decides what the seed holds, so it is not part of #258. On Python 3.13.5,
test_a_link_named_with_a_slash_over_a_file_is_refused(an older test this PR does not touch) fails on its message only; CI runs 3.12.General vs project
General. This changes how the desktop reads any seed tar and paces retries. No project's seed names, sizes or makers are involved. Only fork files change:
.github/scripts/fork-fleet-desk-seed*,.github/fork-fleet.md,fork-kit/desk/README.md.Closes #258
🤖 Generated with Claude Code