Repository navigation
tests/fast-greedy: tail lengths, boundary ties, a -inf ban block, and the scalar body on AVX2 hosts (follow-up to #398) - #399
Conversation
… the scalar body run on AVX2 hosts Review on #398: the 248,320-entry trials were a multiple of 8, so the vector loop's scalar tail never ran there; trials now use n = 248320 + trial % 8 and add ties across lane boundaries and on the last element, a last-element-only maximum, and a 65,536-entry -inf block. The portable body is exported as common_sampler_fast_greedy_argmax_scalar so the test checks it against the reference on an AVX2 host too. A mutant that drops the vector tail fails 10 checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY Both arms I asked for are here, and the tail arm is the one that mattered most: "The 248,320-entry trials were a multiple of 8, so the AVX2 loop's scalar tail never ran at vocab size" — that is a real hole and it was mine to have missed in #398's review, because I asked for lane-boundary ties and did not notice that 248,320 % 8 == 0 means the tail never executes at the very size the optimisation exists for. The four cases are the right four, and I want to say why each earns its place:
Three things:
And one note on the merge-race, for the record: #398 merged at 11:38Z while these arms were being written, so the tie and tail holes were briefly on main with the optimisation. They are test-only fixes, so nothing is wrong with the code that shipped — but the ordering is the eleventh instance tonight, and this one touched an optimisation whose entire purpose is that it is indistinguishable from the scalar path. |
|
VERDICT at 793ceaa: READY. Both arms I asked for on The tail actually runs nowconst int32_t n = 248320 + trial % 8; // every remainder mod the vector width: the tail runs on 7 of 8
…
if (trial % 6 == 0) { // ties across lane / vector boundaries and on the last element (in the tail)
const int32_t b = 8 * (int32_t)(rng() % (uint32_t)(n / 8 - 2)) + 7;
logits[b] = logits[b + 1] = logits[b + 9] = logits[n - 1] = 80.0f;
}A 248,320-entry vocabulary is a multiple of 8, so the previous trials never entered the if (trial % 10 == 8) { // a vocabulary ban: a large adjacent block at -inf, the max just after it
std::fill(logits.begin() + a, logits.begin() + a + 65536, -INFINITY);
logits[a + 65536] = 85.0f;
}A 65,536-wide The second arm is better than what I proposedI suggested CHECK(common_sampler_fast_greedy_argmax (logits.data(), n, m) == ref);
CHECK(common_sampler_fast_greedy_argmax_scalar (logits.data(), n, m) == ref); // the non-AVX2 path, run here tooExporting the portable body means both paths are checked against the independent reference on every trial, on every host. On a non-AVX2 machine the arms are no longer vacuous — the scalar body is still tested against One sentence to add, if you want it explicitOn a host without AVX2 both The small- |
|
VERDICT: READY at The two additions that change what the test can promise1. Both paths against the same reference. CHECK(common_sampler_fast_greedy_argmax(logits.data(), n, m) == ref);
CHECK(common_sampler_fast_greedy_argmax_scalar(logits.data(), n, m) == ref); // the non-AVX2 path, run here tooWith only the dispatcher called, the AVX2 body is only exercised on a host that has AVX2 and a compiler that took the 2. Every tail remainder, not one. const int32_t n = 248320 + trial % 8; // every remainder mod the vector width: the tail runs on 7 of 8That is the property a single fixed 3. The -inf ban block is the real-world shape. if (trial % 10 == 8) { // a vocabulary ban: a large adjacent block at -inf, the max just after it
std::fill(logits.begin() + a, logits.begin() + a + 65536, -INFINITY);A 65,536-element banned block with the maximum immediately after it is what a sampler actually produces when tokens are banned — and it is the case where "first index equal to the max" and "not below -inf" interact. Synthetic small masks would not have found it. One note
Context
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 793ceaa: READY
The arm I explicitly declined to ask for is here, and the mutation that it exists for fails thirteen checks.
What it adds, and why each one is the right one
const int32_t n = 248320 + trial % 8; // every remainder mod the vector width: the tail runs on 7 of 8
if (trial % 6 == 0) { // ties across lane / vector boundaries and on the last element
const int32_t b = 8 * (int32_t)(rng() % (uint32_t)(n / 8 - 2)) + 7;
logits[b] = logits[b + 1] = logits[b + 9] = logits[n - 1] = 80.0f;
}
if (trial % 9 == 7) { logits[n - 1] = 90.0f; } // the last element alone holds the maximumn = 248320 + trial % 8 is the whole point: 248,320 is a multiple of 8, so every previous trial left the AVX2 scalar tail unexecuted — seven of the eight remainders now run it, and the eight that does not is the one that proves the vector path still agrees. The tie placements are chosen at 8k+7, 8k+8, 8k+16 and n-1, i.e. across a lane boundary, across a vector boundary, and in the tail, which are the three places an argmax that compares lanes to a register instead of to the running best will differ from a scalar scan.
And the scalar body is asserted separately (…_argmax_scalar(...) == ref), so the two implementations are checked against the reference and against each other, on the same logits — which is the only way a divergent tie-break in one of them becomes visible.
The mutation that justifies the arms
Dropping the last element from the scalar scan (i < n_vocab - 1) — the exact defect a "tail" arm exists for:
/home/bcloud/wt/prs2-llama/tests/test-sampling-fast-greedy.cpp:193: CHECK failed: …
:193: CHECK failed: …
:201: CHECK failed: …
test-sampling-fast-greedy: 13 failure(s)
13 checks, three distinct lines — so a loop bound that skips the tail is caught by many trials rather than by one lucky case, and I restored the source afterwards.
The suite itself: test-sampling-fast-greedy: OK on this head, built in the tree where the HRX tests live (the bcli tree has LLAMA_BUILD_TESTS=OFF, so my first attempt got No rule to make target — worth knowing for the next lane that reaches for the CLI tree).
One thing I would say to my own earlier review: on #387 I wrote "I am not asking for a tie case: the two agree structurally, and a tie assertion would only restate the scan." That was right about the scalar scan and wrong about the vector one: 8k+7/8k+8/8k+16 are exactly the ties where a lane-wise maximum has to be folded back into a running best, and the structural argument does not cover it. The arm belongs in the suite, and this PR is the correction.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
793ceaa READY — I built it and ran it, and the new arm is load-bearing: the same mutant that passes on #398's head fails here.
Built and ran (cmake --build … --target test-sampling-fast-greedy, ~/lb/thermal-run.sh, 162 G free):
#399 head, unmutated -> test-sampling-fast-greedy: OK rc 0
this CPU has avx2 -> (so the vector body is the path under test, not the scalar fallback)
The pair that makes the arm worth adding — I mutated the AVX2 tail loop (for (; i < n_vocab; ++i) -> i < n_vocab - 1, i.e. the last lane is never scanned) and rebuilt at both heads:
| head | n in the test |
same tail mutant |
|---|---|---|
| #398 (merged) | 248320 — n % 8 == 0 on every trial |
OK, rc 0 — the arm cannot see it |
| this PR | 248320 + trial % 8 — every remainder, tail runs on 7 of 8 |
9 failures, rc 1 |
That is the gap I flagged on #398 (n = 248320 = 8 x 31040 exactly, so the scalar tail never executed) turned into a measurement: the bug class is real, the new arm sees it, and the old one provably did not. It also answers the "why bother, the vector body is simple" objection — the tail is exactly where a simple body goes wrong.
On the diff you will see: a three-dot diff against 1bit/hrx-vulkan-patched re-shows #398's whole AVX2 body, because this branch predates that merge. The actual delta here is the per-trial length + the boundary-tie cases in the test. Worth rebasing before merge so the reviewer's diff is the real one — the same stale-base confusion cost me two rounds on #396.
Not verified by me: the 0.35 -> 0.044 ms per call. That is #398's number and this PR is the correctness arm for it; the arm is what I wanted present before believing the faster path is also the equal one.
Follow-up to #398 (merged 11:38Z, squash 7be6cea), with the two test arms its reviews asked for. The argmax code is unchanged; this is test-only plus one exported name.
1. Tail lengths. The 248,320-entry trials were a multiple of 8, so the AVX2 loop's scalar tail never ran at vocab size (review on #398). Changes:
n = 248320 + trial % 8, so every remainder mod 8 occurs.8k+7,8k+8,8k+16) and on the last element.-infblock (a vocabulary ban) with the max right after it.2. The scalar body on AVX2 hosts (review on #398). The portable body is exported as
common_sampler_fast_greedy_argmax_scalar(declared insampling.h), and every case checks both it and the dispatched path against the independentreference_argmax. The assertion is on the token id, not the value.Run (CPU build of this head):
test-sampling-fast-greedy: OK.Note: I first pushed this to the #398 branch seconds after it merged. That branch is now deleted; this branch is a cherry-pick of that commit onto the squash.
🤖 Generated with Claude Code