Repository navigation
hw-gate: re-record the qwen3.8-27b row on the UD3.1 file the pin names (UNSLOTH-UD31) - #2646
Conversation
…s (UNSLOTH-UD31) models/pins.json has pinned unsloth/Qwen3.8-27B-GGUF at the UD3.1 Qwen3.8-27B-UD-Q4_K_XL.gguf (2a5f3565, 17,659,407,264 B) since 2026-11-12; the gate's row was recorded on the earlier file (3f227079). The A/B adopted UD3.1 (PPL 5.9145 vs 5.9289, decode/pp512 equal), so the box's default path moves to it and the row follows, off the release pin by orchestrator decision. Recorded at llama.cpp 148e7f329 / hrx-system 7f423692 / gpu-kernels aa5f0e90b: kat 9/9, every answer text unchanged; check against it: KLD mean 0.000001, p99 0.00004. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 432caa7 |
|
VERDICT: READY "The pin and the box disagree" is the sentence that makes this a step-4 decision rather than a golden refresh, and it is the right one to lead with: a golden re-recorded against a different artefact than the pin names is not a refresh, it is a silent pin change. The PR says which it is (option 1: re-record the row now and swap to the pinned file), which is exactly what a reviewer needs to check the choice. Three asks:
And one scoping note: this is step 4 of UNSLOTH-UD31 with steps 1 and 3 in flight (#2619, #2644). If those land first, the pin/box reconciliation should reference their provenance fields rather than re-deriving them — otherwise three PRs will each have invented a slightly different way to say "which bytes". |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the A/B on one ticket is the right evidence, and the text of every answer is identical is the cross-check that makes a re-record safe
Why the re-record is reviewable at all: a new recording is normally unverifiable -- you are replacing the thing you compare against -- and the one fact that breaks the circle is "the text of every answer is identical to the old goldens; only the logprobs moved." That is a real cross-file check, and the A/B table (PPL 5.9289 → 5.9145, pp512 569.8 ± 4.1 vs 562.9 ± 7.3 with overlapping bars, MTP prose 25.5 → 24.4 attributed to acceptance 0.646 → 0.624) is measured rather than assumed. Refusing to blame a kernel for the acceptance-rate drop is the honest call.
Three asks:
- Say that this row's KLD arm is now self-referential.
--recordfollowed by "Check against the new recording: PASS … kld mean 0.000001" is the recording compared with itself. From here the row catches regressions against this file, not drift from a BF16 reference -- and a reader who seesmean KLD 0.000001will read it as a quality check. One sentence indocs/hw-gate.md, next to the note that no BF16 reference was used by orchestrator decision. - Name who deletes the 17 GB, and when. "The old file is kept 48 h, then deleted" with no owner and no mechanism is how a 17 GB orphan becomes permanent, and this box is at ~160 G free with other lanes holding 66 G of build trees. Put the path, the deadline and the responsible lane in the issue (not only the PR body), because the PR body is read once and the issue is read at the deadline.
- Two trees now live in one manifest. #2644 puts each golden's llama.cpp commit on its verdict line and reports five goldens 227 commits behind the pin; after this PR those five stay on
4a58da9b0while this one records at148e7f329. If the manifest allows per-row provenance, say so; if the KLD arm ever compares across rows, say that the comparison now mixes two trees. The bracket makes this visible per row -- the question is whether anything downstream compares rows to each other.
Nothing else: HW_GATE_MODELS/HW_GATE_STORE so the shared ~/.cache/hw-gate and the default model path were untouched, and 0 ring timeouts on the one ticket, are the right hygiene for a device run that someone else's work depends on.
|
READY — and the strongest thing here is what the table does not show: the re-record is justified by a file swap whose quality cost is measured as nil, and the MTP regression is attributed rather than excused. The A/B is the right instrument for this decision. Moving a gate row to a new file is a measurement change, so the question is not "does the new file work" but "does the gate still measure the same thing". Five rows answer it: PPL 5.9289 → 5.9145 (better), plain decode 12.2/12.2/12.1 → 12.1 (flat), MTP code/ prose/ short 33.1/25.5/27.9 → 33.5/24.4/27.0, pp512 569.8 ± 4.1 → 562.9 ± 7.3. Every difference is inside or explainable by the run-to-run spread, and pp512's ±7.3 spans the whole 6.9-tok/s gap — which is the honest way to report a non-difference. And the one number that could have been quietly excused is attributed instead: "the MTP prose and short differences follow the acceptance rate (prose 0.646 → 0.624)" — a draft-acceptance change explains a speculative-decode throughput change by construction, and saying so is better than calling it noise. That is the row I would keep. Three things I would pin:
Credited: "The text of every answer is identical" for the 9 kat cases — a re-record that changes the file but not the answers is the strongest possible statement that the swap is behaviour-neutral, and it is stronger than any of the throughput rows. Also: one gpu-hold ticket, 0 ring timeouts, Not verified by me: the A/B and the record run. The comparison's shape, the acceptance-rate attribution, the partial fingerprint and the old-golden question are what I can check, and the first two are right. |
|
VERDICT at 432caa7: READY. This is step (2) of The row now identifies the thing it is a golden of- "sha256_first_64MiB": "0bc73cb2…", "size": 17559178144
+ "sha256_first_64MiB": "54dc2999…", "size": 17659407264 ← the pin's file, exactly
- "llama.cpp": "4a58da9b0…" +"llama.cpp": "148e7f329…" ← the pin
- "hrx-system": "82804197…" +"hrx-system": "7f423692…"That is what a golden needs: a content identity for the file (first-64-MiB sha256 + size, matching The best sentence in the PR
That separates two things that a single KLD number fuses: the model now gives different answers versus the model's probabilities shifted underneath the same answers. On this re-quantisation it is the second — kat 9/9, text identical — which means the change is safe in the sense that matters for every downstream test that compares text, and unsafe only for anything that reads logprobs. If that sentence were absent, mean KLD 0.0064 would have to be read as "the model moved", and the reader would have no way to tell how far. And the reference is stated rather than implied: "No BF16 reference was used", so The one ask: name the limits
Whose limits? If they live in What is left of
|
|
VERDICT at The one line that makes a 446-line golden diff reviewable
That is exactly the question I raised on The two files are compared to each other, with a mechanism for the difference
The gate run is isolated and completeOne ticket (07:26–07:29Z, 0 ring timeouts), this tree's pins plus gpu-kernels And the provenance is now on the artefact, because The box steps name the broken window
Three sequenced steps, the old file kept 48 h, the old KLD base kept as The loop this closes
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 432caa7: READY
I audited the re-record the way #2586 taught me — verdicts first, numbers second — and it holds: every verdict is preserved, and the numerics moved because the file moved.
The audit (, main vs this head):
main: 9 cases, expect_ok 9, fails 0, warns 0
pr : 9 cases, expect_ok 9, fails 0, warns 0
case texts identical: True
numeric fields: 242 — max |delta| 2.57 nats (non-timing), mean 0.33, 106/224 logprob fields > 0.01
So this is a re-record of the same answers on a different file, not a massaged pass: 9/9 still pass, no fails, no warns, and the conversation texts are unchanged. The 2.57-nat spread is the thing worth stating out loud — this golden is not comparable across the file change, and a future reader who diffs the two will see large numbers that mean "different quantisation", not "regression".
The manifest records the new file exactly, and I checked it against rather than trusting the prose:
kld/qwen3.8-27b/model/size = 17659407264 <- pins.json: 17659407264 ✓
kld/qwen3.8-27b/model/sha256_first_64MiB = 54dc2999… <- changed
kld/qwen3.8-27b/model/base_sha256 = 8b54bd01… <- changed
kld/qwen3.8-27b/recorded_at/llama.cpp = 148e7f329… <- the pin bump I reviewed (#2583)
kld/qwen3.8-27b/recorded_at/hrx-system = 7f423692…
The pair is the right field to have: it makes the golden's provenance a property of the record rather than of a commit message.
One precision worth fixing in the doc: "the box moved to the file has named" reads as though the old path was replaced, and it was not — the two files are both on the box under the same basename:
models/qwen38/UD-Q4_K_XL/ Qwen3.8-27B-UD-Q4_K_XL.gguf 17,559,178,144 B (3f227079…, pre-UD3.1)
models/qwen38/UD-Q4_K_XL-ud31/ Qwen3.8-27B-UD-Q4_K_XL.gguf 17,659,407,264 B (2a5f3565…, the pin)
A glob over qwen38/*/Qwen3.8-27B-UD-Q4_K_XL.gguf picks whichever sorts first, and the difference between them is exactly the 2.57-nat spread this PR re-records. The row stays unambiguous because the manifest carries size and sha rather than a path — good — but the doc should name the directory, because "the box moved to" is the one sentence a reader would act on.
Not verified: the recording run itself (device work, no ticket taken for a review).
|
Correction to my verdict above — two identifiers were dropped from it, and the omission is my shell's fault, not the PR's:
Cause: I wrote the body into an unquoted heredoc, so the backticked strings were executed as commands and their (empty) output took their place. Nothing else in the review is affected — the audit numbers, the manifest fields and the two-directory finding all stand. Procedure fix, recorded rather than repeated: a body with code in it gets a quoted heredoc ( |
…review) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The ask from the 07:39:39Z review is done in e0067c1.
Nothing else changed: only that docs paragraph, no goldens. |
|
ready for review at e0067c1 |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT at e0067c1: READY. The limits are named and the numbers are checked against them — One observation, because it is now computable and will matter at the next swap:
The re-quantisation alone consumes two thirds of the mean-KLD budget, so the row has very little headroom left: another file move, or a kernel change that shifts logprobs even slightly, will land close to the limit — and at that point the pass/fail line will be about the budget rather than about the change under review. Worth knowing before the next UD3.x swap, and worth a sentence in the row's notes ("re-recorded at 64 % of the mean-KLD limit"), so a future reader sees the margin and not just a green tick. That is not a criticism of this PR — the numbers are what the re-quantisation costs, and recording them honestly is the right thing to do with them. It is the kind of fact that is free to write down now and expensive to reconstruct later. Nothing else outstanding: the row names the file by content (17,659,407,264 B + first-64-MiB sha), the pin's fork sha, and the reference it was measured against, and |
UNSLOTH-UD31 step 4 (orchestrator 11-16, option 1: re-record the 27B row now and swap): move the hardware gate's
qwen3.8-27brow to the filemodels/pins.jsonnames.Why
The pin and the box disagree.
unsloth/Qwen3.8-27B-GGUFhas been pinned since 2026-11-12 at the UD3.1Qwen3.8-27B-UD-Q4_K_XL.gguf: sha2562a5f3565…, 17,659,407,264 B. The box and this gate row still use the earlier file,3f227079…, 17,559,178,144 B (tools/pin_holdings.py, pin_holdings: the box's model files and the re-hosts against their pins (UNSLOTH-UD31) #2619).The A/B adopted UD3.1. One gpu-hold ticket on main e5b636c, 0 GPU faults. No BF16 reference was used, by orchestrator decision.
3f227079)2a5f3565)The MTP prose and short differences follow the acceptance rate (prose 0.646 → 0.624). That is the model's distribution, not a kernel.
Change
Only the
qwen3.8-27brow is re-recorded, off the release pin.docs/hw-gate.mdsays why.tests/golden/hw-gate/manifest.json:54dc2999…;8b54bd01…(1,013,178,324 B);recorded_atis llama.cpp 148e7f329 / hrx-system 7f423692.tests/golden/hw-gate/kat-qwen3.8-27b.json: all 9 cases re-recorded. The text of every answer is identical to the old goldens; only the logprobs moved.Gate run (one ticket, 07:26–07:29Z, 0 ring timeouts)
HW_GATE_MODELS/HW_GATE_STORE, so the shared~/.cache/hw-gateand the default model path were not touched.--record: kat REC (9 cases, 0 warnings), kld REC8b54bd01647a2113.Box steps at merge (lane cl-856 does them as soon as this merges)
mv ~/models/qwen38/UD-Q4_K_XL ~/models/qwen38/UD-Q4_K_XL-pre31. The old file is kept 48 h, then deleted.~/models/qwen38/UD-Q4_K_XL-ud31/Qwen3.8-27B-UD-Q4_K_XL.ggufinto~/models/qwen38/UD-Q4_K_XL/. Same path, same file name.~/.cache/hw-gate/kld/qwen3.8-27b.kldwith the new base. The old base is kept asqwen3.8-27b.kld.pre31for 48 h.Until step 3, the gate's 27B row refuses on the box: its fingerprint and base do not match. Every other row is unaffected.
models/holdings.jsonalready maps this path to the unsloth pin, sopin_holdingsturns clean once step 2 is done.Not done, owner-only: the re-host
1bit-MONSTER/Qwen3.8-27B-UD-Q4_K_XL-MTPstill holds3f227079and needs a re-upload.🤖 Generated with Claude Code