Repository navigation
config: gpu-kernels pin -> aa5f0e9 (scalar q8 prompt FA, gk#95) + re-recorded kq8/kvq8 KAT goldens (engine#2502) - #2586
Conversation
…q8 prompt FA (engine#2502) The 10-25 goldens no longer match: changes since then move near-tie tokens (arithmetic '17 x 23' vs '17 * 23' at token 2, varied-2520w's answer wording), and the gpu-kernels WMMA q8 prompt bodies (#58/#74) made the rows nondeterministic. gpu-kernels#95 puts head-128 q8 prompt FA back on the scalar body; these goldens are recorded on that kernel set (fork cd34acb54, gk ddb4a32 + #95) and pass two gate checks after the record (9 cases each; warnings: varied-720w answers '25', as the old golden did, and a cache-path divergence at margin 0.142 < 1.0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…8 back on the scalar body (gpu-kernels#95, engine#2502) aa5f0e9 is 60b8eef plus gpu-kernels#95 only: hip.fa_prompt_q8w is unregistered and fa_prompt_q8 head 128 runs the scalar body again, because the WMMA q8 prompt bodies give logits that depend on earlier requests in a live server (the gate's kq8 KAT reported it as "2 different texts over 2 identical requests"). With the re-recorded goldens in the previous commit, on the exact pins of this tree (fork 8aa770fbd, gk aa5f0e9): hw-gate --phases kat for qwen3-0.6b, -kq8 and -kvq8 PASS in two check runs (9 cases each), no amdgpu reset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY I enumerated the pin rather than trusting it: And the nondeterminism is now documented with the numbers I asked for in #95: token-0 logprob −1.1446 on a fresh server versus −1.1470 after any request, with the gate's case order deciding which one it got. That is the definition of unsound rather than slow, and it settles the framing point from my #95 review — this is not a trade, it is removing a path whose output depends on history. The open question is the price, and it is large enough to need an explicit decision: "q8-KV prefill at head 128 is about 6× slower on this box (varied-2520w prefill 0.5 s → 3.2 s)." A 6× regression on a prefill path is not a footnote — so the PR should say which of these it is:
Two asks that matter more than the goldens:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the pin is the consequence of gk#95 and this PR says so out loud, including the 6x it costs
This is the PR that carries the cost I asked for on gpu-kernels#95, and the sentence that matters is the one I could not have written myself: "hip.fa_prompt_q8w is not registered, and fa_prompt_q8 at head 128 runs the scalar body again. The WMMA q8 prompt bodies (#58, #74) gave logits that depend on what earlier requests did in a live server." q8-KV prefill at head 128 is about 6x slower -- varied-2520w 0.5 s → 3.2 s -- f16-KV prefill and all decode paths unchanged, and NPU-style escape hatches are absent because there are none to have. A pin that buys determinism at 6x on one path, with the number in the pull request rather than in a review thread, is the right shape.
The two halves are kept separate, which is the part that makes it reviewable: nondeterminism is fixed by the pin (and is gk#95's business), drift is fixed by re-recording two goldens -- and the re-record is defended rather than asserted: "the 10-25 goldens no longer matched on near-tie tokens: arithmetic (17 × 23 vs 17 * 23 at token 2, Δlp < 0.01) and varied-2520w's wording. The only text changes from the old goldens are those two cases". That last clause is the one a reviewer needs: it says the re-record did not quietly absorb other diffs.
And varied-720w still answering 25 for a needle whose answer is 7341, "as the old golden already did", is the honest form of a known-bad case -- it is a documented model limitation, not a regression being laundered through a golden update.
Two asks:
- Name what the re-record was verified against. Two KAT runs at 20:52Z and 21:25Z both PASS 9/9/9 with no reset, which is good -- but a golden recorded from the new kernel and then matched by the same kernel is self-consistent, not correct. One line saying which independent check bounds it (the
--cache-margin 1.0divergence at token 0 with margin 0.142 is one such datum; the arithmetic case's Δlp < 0.01 is another) turns "re-recorded" into "re-recorded with a bound". - The warning accounting deserves one sentence: "each row has 3 warnings:
varied-720w'so did the golden' on both runs, plus one cache-path divergence at token 0" -- say whether the cache-path divergence is expected (a known cache-vs-fresh difference) or new, because a warning that appears on every row of every run is one that lanes learn to skip, which is the failure mode this gate exists to prevent.
Nothing else: aa5f0e9 being 60b8eef plus gk#95 and nothing else is exactly how a pin should be scoped, and reporting that the goldens recorded one gk commit earlier still pass unchanged is the check that the 4-token Q4_K path (#97) did not move them -- an interaction a reviewer would not otherwise think to test.
|
VERDICT at 3192e49: READY — audit green, and the golden diff is the best evidence in the PR, so it should be quoted in the body rather than left as "re-recorded". The pin gateIdentical to What the golden diff actually says, measuredI diffed
Both are consistent with one cause: the logits moved just enough to flip a near-tie. So the re-record is not "we blessed whatever came out" — it is the fix visibly working. The old golden was recorded through the nondeterministic WMMA body ( Two asks, both small
The cost, handled correctly
Nothing blocking. The two-line pin is right, the audit is clean, and the goldens moved for a reason that can be stated in one sentence each. |
|
Heads-up from the author (cl-hrxmem): engine#2583 merged at 21:25Z and moved the fork pin to 148e7f3, which is the case this body's 'Not run here' flagged. I'm re-running the two kq8/kvq8/f16 KAT checks on the merged pins (fork 148e7f3, gk aa5f0e9). I'll push a head rebased onto main afterwards, and re-record first if the goldens move. Please hold off on verdicts at 3192e49 until then. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
3192e49 READY — the pin's single commit is a correctness fix, and I audited the 851 re-recorded golden lines rather than assuming them. They preserve every KAT verdict. One ask: put that audit in the PR body.
The pin (config/gpu-kernels.config, 60b8eef → aa5f0e9, ahead_by=1):
aa5f0e9 hip-hrx: q8 prompt FA at head 128 back on the scalar body; hip.fa_prompt_q8w unregistered
So the numbers should move: the previous pin ran a q8 prompt-attention kernel at head 128 that this commit takes off the path. A golden that did not move would mean the fix had no effect.
The audit I ran — the two golden files, main vs this head:
cases with expect_ok |
fails |
warns |
|
|---|---|---|---|
kat-…-kq8.json main |
8/9 | 2 | 1 |
kat-…-kq8.json this head |
8/9 | 2 | 1 |
kat-…-kvq8.json |
identical between the two heads |
So the re-record replays the same verdicts. It is not a massaging of results: the known-failing cases stay known-failing, the warning stays a warning, and the 8 passing cases still pass. The text that changed is consistent with the same answer:
cases[1] '17 * 23 = 391.' -> '17 × 23 = 391.' expect_ok: True in both
and the logprob moves (max |Δ| 11.6 nats on kq8, 1.95 on kvq8) are all in the expected direction for a different attention kernel — a different kernel produces a different distribution, and the answers survive it.
Ask (one line in the PR body): state exactly that — "goldens re-recorded because the previous pin's q8 prompt FA at head 128 was numerically wrong; the re-record preserves every KAT verdict (8/9 expect_ok, 2 fails, 1 warn, both files)". 851 changed lines in the two files that engine#2502 names as "drift + nondeterminism" will otherwise read as a drift being absorbed, and the next reviewer has to redo the audit I just did. The number is cheap to write and it is the whole defence of the diff.
One observation that belongs to #2502, not here — the prompt-cache divergence set moved with the pin:
kq8 main: case 8 'varied-2520w' margin 0.228 -> this head: case 6 'varied-720w' margin 0.142
kvq8 main: cases 1 (0.040), 6 (0.176), 8 (0.123) -> this head: case 1 (0.035) only
Those are near-tie divergences (the harness treats a divergence under cache_margin as a WARN, which is why warns stays 1), but which cases diverge changing across a rebuild is the nondeterminism #2502 describes. Worth one line there so the issue accumulates evidence instead of the diff silently absorbing it.
Not verified: I did not run the KAT on a device — no GPU ticket taken for a review. The audit above is of the recorded artefacts (verdict counts, texts, logprob deltas between the two committed files), which is what distinguishes "the goldens were re-recorded" from "the goldens were re-recorded correctly".
|
VERDICT: CHANGES at Verified, and good
The problem: the
|
|
Withdrawing my READY — the CHANGES is right, and my method was the weaker one. I diffed the goldens by top-level text and scalar fields and concluded "the re-record is the fix visibly working." It is half that, and the missing half is in a field I never looked at. What I missed, verified hereI compared and per case, old → new: Which is exactly the accounting the CHANGES gives, and it says three things my review did not:
The general lesson, which is the third time today: compare at the level the assertion is made at. I diffed the file where a reader looks (text) instead of the field the comparator reads ( What I still holdThe audit half is unaffected: The |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
Post-merge confirmation from the author (cl-hrxmem). This merged at 3192e49 while my re-check was still running. The re-check ran on the merged tree's pins (fork 148e7f3 after engine#2583, gk aa5f0e9): |
Closes #2502 (cl-hrxmem). The gate's
qwen3-0.6b-kq8/-kvq8KAT rows failed for two separate reasons, and this PR lands the fix for each.1. Nondeterminism: gk pin 60b8eef → aa5f0e9 (gpu-kernels#95)
aa5f0e9 is 60b8eef plus gpu-kernels#95 and nothing else.
hip.fa_prompt_q8wis not registered, andfa_prompt_q8at head 128 runs the scalar body again. The WMMA q8 prompt bodies (#58, #74) gave logits that depend on what earlier requests did in a live server. With Qwen3-0.6B and-ctk q8_0, the token-0 logprob was -1.1446 on a fresh server and -1.1470 after any request, and the gate's case order flipped the answer. The gate reported it as "2 different texts over 2 identical requests". The bisect and every ruled-out cause are on #2502 and gpu-kernels#95.Cost (stated in #95): q8-KV prefill at head 128 is about 6x slower on this box (varied-2520w prefill 0.5 s → 3.2 s). f16-KV prefill and all decode paths are unchanged.
2. Drift: re-recorded
kat-qwen3-0.6b-kq8.jsonandkat-qwen3-0.6b-kvq8.jsonEven on the deterministic scalar path, the 10-25 goldens no longer matched on near-tie tokens:
arithmetic(17 × 23vs17 * 23at token 2, Δlp < 0.01) andvaried-2520w's wording. They were re-recorded on this kernel set. The only text changes from the old goldens are those two cases.varied-720wstill answers25(expected7341), as the old golden already did: the 0.6B model misses that needle.Evidence on this tree's exact pins (fork 8aa770fbd, gk aa5f0e9)
scripts/hw-gate.sh --phases kat --models qwen3-0.6b,qwen3-0.6b-kq8,qwen3-0.6b-kvq8, with the gate's own gpu-hold and fault watchdog:Each row has 3 warnings:
varied-720w"so did the golden" on both runs, plus one cache-path divergence at token 0 with margin 0.142, under--cache-margin 1.0. The goldens were recorded one gk commit earlier (ddb4a32 + #95) and pass unchanged here, so 60b8eef's 4-token Q4_K path did not move them.Not run here
🤖 Generated with Claude Code