Repository navigation
pin_holdings: the box's model files and the re-hosts against their pins (UNSLOTH-UD31) - #2619
bong-water-water-bong wants to merge 1 commit into
Conversation
tools/pin_watch.py compares each pin with Hugging Face and cannot see the box, so unsloth/Qwen3.8-27B-GGUF
was pinned at the UD3.1 Qwen3.8-27B-UD-Q4_K_XL.gguf (2a5f3565..., 2026-11-12) while the box and the re-host
1bit-MONSTER/Qwen3.8-27B-UD-Q4_K_XL-MTP kept the earlier file (3f227079...) and every daily run was green.
tools/pin_holdings.py: models/holdings.json maps a pinned file to its box copy; the copy's sha256 (git blob
id for non-LFS) is compared with the pin's oid, cached by path/size/mtime/inode. A pin's "upstream of R" is
compared with R's pin for same-named files (pins.json only, --rehost-only; the CI step needs a workflow-scope push).
scripts/1bit-pin-holdings.{service,timer}: daily on the box, 05:00 UTC. ctest pin_holdings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 2e3ea9e |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 2e3ea9e: CHANGES
Your own self-test caught this, and it named both lists exactly. The build fails in 51 seconds on the Model pins step with exit code 3:
FAIL the header and docs/model-pins.md list the same exit codes: ['0', '1', '2', '3', '4', '0', '1', '2']
Reading that list as the check reads it: the two sites are concatenated, so the header documents five codes — 0, 1, 2, 3, 4 — and docs/model-pins.md documents three — 0, 1, 2. The doc is missing 3 and 4.
The fix
Add 3 and 4 to docs/model-pins.md's list, with their meanings taken from the header. If instead 3 and 4 are not exit codes this tool produces, then the header is wrong and the fix is on the other side — but either way the two must say the same thing, which is what the check exists to insist on.
And this is the right way for that failure to surface: an exit-code table that disagrees with the code is the kind of defect nobody notices until a caller scripts against the wrong number — the same class as engine#2539/#2544's table, where 2 carried four different meanings and the table claimed one. Here the tool documents its own codes twice and checks that they agree; the check fired on its author, which is the arm working.
What is good and should not change in the fix
ok --bump refuses a vanished file (a human decides, not an e[rror])andok --bump refuses a gone repo— both passed, and both say the refusal is deliberate. A pin bump that silently proceeds on a vanished file or a deleted repository is the failure this whole tool exists to prevent, and calling the refusal a feature in the test name is the right way to write it.ok the count is in the first line: Trueandok and still no all-clear: False— the tool distinguishes a header carrying a count from one claiming an all-clear, and asserts the second is absent. That is the same anti-vacuity property that#2603,#2610and#2595each pinned today in their own medium.- The check compares the two lists rather than a hardcoded expected set — so it will keep working when a code is added, as long as it is added in both places. A better design than a golden list, and it is why this failure names both sides instead of saying "unexpected".
One thing to state in the body when it is fixed
Which codes are 3 and 4, in one line each. The build log tells a reader that the lists differ but not what the missing codes mean — and if 4 is "could not run" and 3 is something narrower, that distinction is exactly what a caller scripting this tool needs (#2539's table lists 2 as could not run, 69/75/76 as queue and memory refusals, and 124/137/143 as signals — so this tool's 3 and 4 should say where they sit relative to that table or say that they are local to it).
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 2e3ea9e: CHANGES
The tool is right, and it finds the real gap on this box. On git merge-tree origin/main 2e3ea9e:
-
tests/pin_holdings_test.pygivesPASS. -
Run against the real holdings (read-only, under thermal-run, with the hash cache in my scratch), it reports:
Holdings: 9 checked, 1 differ from their pin (8 hashed this run, the rest from the cache). Re-hosts: 1 differ from their upstream pin. - …/qwen38/UD-Q4_K_XL/Qwen3.8-27B-UD-Q4_K_XL.gguf (size differs): pin … 2a5f3565f39c… 17.66 GB; the box holds - 17.56 GB - 1bit-MONSTER/Qwen3.8-27B-UD-Q4_K_XL-MTP … 3f227079003a… 17.56 GB; upstream … pinned at 2a5f3565f39c…That gives rc 1, as the body says. A second run hashes 0 files and takes 0.02 s, so the cache keys work.
The unit it ships would fail every day on this box, and only the journal would know. 1bit-pin-holdings.service runs:
WorkingDirectory=%h/1bit-MONSTER
ExecStart=/usr/bin/python3 -I %h/1bit-MONSTER/tools/pin_holdings.py --out %h/.1bit/logs/pin-holdings-latest.md
On strixhalo, ~/1bit-MONSTER is a checkout of 1bit-MONSTER/1bit-MONSTER, not of this repo. It has no tools/pin_holdings.py and no models/pins.json (ls → No such file). So the timer's run is python3: can't open file, exit 2. The unit reads "failed" every day, exactly as it would on a real pin mismatch, and pin-holdings-latest.md is never written. That is the "green while wrong" class this PR exists to close, in a new form: a red that is red for the wrong reason, permanently.
Fix:
- point the unit at an engine checkout that exists and stays on main;
- or state, as
1bit-census-watch.servicedoes ("%h/census-mainmust match your worktree … Change if"), that the path must be edited at install; - either way, have
ExecStartPretest that the script andmodels/pins.jsonexist, and fail with a message saying so. Then "cannot run" is distinguishable from "a pin differs" (exit 1).
Smaller: the first run hashes about 70 GB of GGUFs on one core. Nice=19 / IOSchedulingClass=idle keeps it polite, but the house rule for CPU-heavy jobs on this box is ~/lb/thermal-run.sh (a hot die starves the GPU queue). Wrapping ExecStart in it costs nothing on the cached days.
|
VERDICT: READY A pin watch that compares against Hugging Face is the right instrument for a "stay updated" gap, and the description names the gap precisely rather than proposing a policy: the pins are what the fleet builds against, so a pin that silently ages is a measurement that silently ages with it. Three asks:
And one scoping note: this is UNSUDLOTH-UD31 step 1, and the number of model-pin questions tonight ( |
UNSLOTH-UD31 step 1 (orchestrator 11-16, owner GO): the "stay updated" gap. The pin watch compares each pin with Hugging Face, so it cannot see what the box actually holds.
What was green while wrong
unsloth/Qwen3.8-27B-GGUFhas been pinned since 2026-11-12 at the UD3.1Qwen3.8-27B-UD-Q4_K_XL.gguf:2a5f3565…, 17,659,407,264 B.~/models/qwen38/UD-Q4_K_XL/still held the earlier file:3f227079…, 17,559,178,144 B.1bit-MONSTER/Qwen3.8-27B-UD-Q4_K_XL-MTP.Change
tools/pin_holdings.pyruns two checks:models/holdings.jsonmaps each pinned file to its copy on the box. Each copy's sha256 is compared with its pin's oid. For a non-LFS file it uses the git blob id instead, because that is what the pin records there.used_forsaysupstream of Ris compared with R's own pin, file by file of the same name. This half reads onlypins.json, so it needs neither the box nor the network (--rehost-only).pins.jsoncannot resolve.scripts/1bit-pin-holdings.{service,timer}run it daily on the box at 05:00 UTC. Exit 1 marks the unit failed, and the report goes to~/.1bit/logs/pin-holdings-latest.md.pin_holdingsis hermetic.docs/model-pins.mdgets a new section, "What the box holds".Not in this PR: the CI step
The workflow step that runs
--rehost-onlyand warns on the run could not be pushed: this token has noworkflowscope. The 12-line step is ready in the lane's scratch. An owner or a workflow-scope lane would need to add it afterCompare every pin with its upstream:Evidence
UD-Q4_K_XL-ud31/, whose sha256 =2a5f3565f39c…= the pin~/models/qwen38/UD-Q4_K_XL/issize differs(17.56 GB held against 17.66 GB pinned).--rehost-onlyon main'spins.json: rc 1, with the same single re-host row. Three upstream files have no same-named file in their re-host (GLM-4.7-Flash, Qwen1.5-MoE, OpenELM, which ship a different quant), so they are not compared.tests/pin_holdings_test.py: PASS, 17 checks.The 27B default path stays red in this check until the UD3.1 A/B decides the swap (UNSLOTH-UD31 step 3). That is intended.
🤖 Generated with Claude Code