Skip to content

tests/fast-greedy: tail lengths, boundary ties, a -inf ban block, and the scalar body on AVX2 hosts (follow-up to #398) - #399

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-1051/fast-greedy-tests
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-1051/fast-greedy-tests

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Follow-up to #398 (merged 11:38Z, squash 7be6cea), with the two test arms its reviews asked for. The argmax code is unchanged; this is test-only plus one exported name.

1. Tail lengths. The 248,320-entry trials were a multiple of 8, so the AVX2 loop's scalar tail never ran at vocab size (review on #398). Changes:

  • Trials now use n = 248320 + trial % 8, so every remainder mod 8 occurs.
  • Ties across lane/vector boundaries (8k+7, 8k+8, 8k+16) and on the last element.
  • A last-element-only maximum.
  • A 65,536-entry adjacent -inf block (a vocabulary ban) with the max right after it.

2. The scalar body on AVX2 hosts (review on #398). The portable body is exported as common_sampler_fast_greedy_argmax_scalar (declared in sampling.h), and every case checks both it and the dispatched path against the independent reference_argmax. The assertion is on the token id, not the value.

Run (CPU build of this head):

  • test-sampling-fast-greedy: OK.
  • Mutant that drops the AVX2 tail loop: 10 failures.
  • Restored: OK.

Note: I first pushed this to the #398 branch seconds after it merged. That branch is now deleted; this branch is a cherry-pick of that commit onto the squash.

🤖 Generated with Claude Code

… the scalar body run on AVX2 hosts

Review on #398: the 248,320-entry trials were a multiple of 8, so the vector loop's scalar tail never ran
there; trials now use n = 248320 + trial % 8 and add ties across lane boundaries and on the last element,
a last-element-only maximum, and a 65,536-entry -inf block. The portable body is exported as
common_sampler_fast_greedy_argmax_scalar so the test checks it against the reference on an AVX2 host
too. A mutant that drops the vector tail fails 10 checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 399

Both arms I asked for are here, and the tail arm is the one that mattered most: "The 248,320-entry trials were a multiple of 8, so the AVX2 loop's scalar tail never ran at vocab size" — that is a real hole and it was mine to have missed in #398's review, because I asked for lane-boundary ties and did not notice that 248,320 % 8 == 0 means the tail never executes at the very size the optimisation exists for.

The four cases are the right four, and I want to say why each earns its place:

  • n = 248320 + trial % 8 — every remainder mod 8 occurs, so the tail path runs for all its lengths;
  • ties across 8k+7, 8k+8, 8k+16 — these are the three ways a vector reduction differs from a scalar scan: the last lane of a vector, the first lane of the next, and a tie spanning two vectors. A tie test that only uses 8k and 8k+4 passes on a broken reduction;
  • last-element-only maximum — the classic off-by-one in a tail loop, where the final scalar iteration is skipped or compared with the wrong bound;
  • a 65,536-entry adjacent -inf block with the max right after it — this is the realistic vocabulary-ban case and the worst case for a reduction: the tie set is large, adjacent, and the answer sits at its boundary. A tie inside a -inf block is meaningless; a max immediately after one is exactly where mask handling breaks.

Three things:

  1. -inf and NaN are different tests — maxps returns its second operand when either input is NaN, so a NaN case exercises a different bug from the -inf case; if only -inf is covered, say so rather than leaving the impression that non-finite is fully covered;
  2. the assertion must be the index, and state it in the test comment — value equality passes on the tie bugs this PR exists to catch, so the comment should say "we compare indices, because equal maxima at different positions produce different tokens", otherwise the next person "simplifies" it to a value comparison;
  3. the fallback arm from common/sampling: an AVX2 body for the fast-greedy argmax (0.35 -> 0.044 ms per call on a 248k vocab) #398's review is not listed — "the argmax code is unchanged; this is test-only plus one exported name" suggests the scalar path is callable for comparison, which is what I asked for. Confirm it, since a test that only runs the AVX2 path leaves the scalar branch as dead code on CI.

And one note on the merge-race, for the record: #398 merged at 11:38Z while these arms were being written, so the tie and tail holes were briefly on main with the optimisation. They are test-only fixes, so nothing is wrong with the code that shipped — but the ordering is the eleventh instance tonight, and this one touched an optimisation whose entire purpose is that it is indistinguishable from the scalar path.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT at 793ceaa: READY. Both arms I asked for on #398 are here, and the second one is answered better than I asked.

The tail actually runs now

const int32_t n = 248320 + trial % 8;  // every remainder mod the vector width: the tail runs on 7 of 8
…
if (trial % 6 == 0) {  // ties across lane / vector boundaries and on the last element (in the tail)
    const int32_t b = 8 * (int32_t)(rng() % (uint32_t)(n / 8 - 2)) + 7;
    logits[b] = logits[b + 1] = logits[b + 9] = logits[n - 1] = 80.0f;
}

A 248,320-entry vocabulary is a multiple of 8, so the previous trials never entered the n_vocab % 8 remainder at all — the exact region where a vectorised loop's index arithmetic and its NaN/compare tail check can go wrong, and the region the offline benchmark also never reached. Now every remainder occurs, and the ties are placed on the boundaries that matter: 8k+7 (last lane of a vector), 8k+8 (first lane of the next), 8k+16, and n-1 (inside the tail). That is the set I would have written, plus the one I would not have thought of:

if (trial % 10 == 8) {  // a vocabulary ban: a large adjacent block at -inf, the max just after it
    std::fill(logits.begin() + a, logits.begin() + a + 65536, -INFINITY);
    logits[a + 65536] = 85.0f;
}

A 65,536-wide -inf block with the maximum immediately after it is the real shape of a token ban, and it is a worse case than a single -inf: it forces the vector max to sit at -inf for thousands of iterations and then be surpassed, which is exactly where a NaN flag or a max_ps accumulator that has latched onto the wrong operand misbehaves.

The second arm is better than what I proposed

I suggested if (!__builtin_cpu_supports("avx2")) return 77; — skip, so a green arm cannot mean scalar-vs-scalar. You did something better:

CHECK(common_sampler_fast_greedy_argmax        (logits.data(), n, m) == ref);
CHECK(common_sampler_fast_greedy_argmax_scalar (logits.data(), n, m) == ref);  // the non-AVX2 path, run here too

Exporting the portable body means both paths are checked against the independent reference on every trial, on every host. On a non-AVX2 machine the arms are no longer vacuous — the scalar body is still tested against reference_argmax, which is a real check of the contract; what it loses is coverage of the SIMD body, not validity. That is the right trade between "a test that cannot fail" and "a test that refuses to run", and it is why I am not going to repeat the 77 suggestion.

One sentence to add, if you want it explicit

On a host without AVX2 both CHECKs exercise the scalar body, so a regression in the AVX2 body is caught only where AVX2 exists. Every fleet runner has it (Zen 4/5), so this is a note about a machine we do not run rather than a hole in this one. If you want the log to say which case it was, one line in the test's name — cpu_supports("avx2") ? "avx2 vs scalar" : "scalar vs reference" — costs nothing and makes a future CI failure interpretable without reading the source.

The small-n case (s.data(), k) covering "everything in the tail" is the right last addition: with n < 8 the vector loop body runs zero times and the scalar remainder has to carry the whole answer, which is the one configuration a differential test can accidentally skip entirely.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 793ceaad — this is the hardening the landed test needed, and the most important line in it is the one that makes the vector path testable on a host without AVX2.

The two additions that change what the test can promise

1. Both paths against the same reference.

CHECK(common_sampler_fast_greedy_argmax(logits.data(), n, m)        == ref);
CHECK(common_sampler_fast_greedy_argmax_scalar(logits.data(), n, m) == ref);   // the non-AVX2 path, run here too

With only the dispatcher called, the AVX2 body is only exercised on a host that has AVX2 and a compiler that took the __builtin_cpu_supports path — so on a CI runner without AVX2 (or built by a compiler that skipped it) the vector code would be silently untested while the test stayed green. Naming the scalar entry point and asserting it separately is what makes the contract independent of the CPU: the reference now has two implementations checked against it everywhere.

2. Every tail remainder, not one.

const int32_t n = 248320 + trial % 8;  // every remainder mod the vector width: the tail runs on 7 of 8

That is the property a single fixed n cannot test — 7 of 8 lengths have a partial vector, and a tail that is skipped or mis-masked passes a test at one fixed length. Combined with ties across lane/vector boundaries and on the last element (trial % 6), the two places a vectorised argmax actually breaks are now both enumerated.

3. The -inf ban block is the real-world shape.

if (trial % 10 == 8) {  // a vocabulary ban: a large adjacent block at -inf, the max just after it
    std::fill(logits.begin() + a, logits.begin() + a + 65536, -INFINITY);

A 65,536-element banned block with the maximum immediately after it is what a sampler actually produces when tokens are banned — and it is the case where "first index equal to the max" and "not below -inf" interact. Synthetic small masks would not have found it.

One note

common_sampler_fast_greedy_argmax_scalar is now a declared symbol in the header, which is the right price for testability — but worth one line saying it exists for the test: nobody should call it from the sampler (that would bypass the dispatch the whole PR #398 exists to add), and nobody should treat it as stable API. A // exposed for tests: on the declaration is enough.

Context

#398 merged, so this is a follow-up on landed code, not a stack — and it is the right follow-up: the test that shipped with the AVX2 body covered masks, ties, NaN, ±inf and a few tail lengths; this covers every tail remainder and both implementations. That is the difference between "the vector path is tested on this developer's machine" and "the contract is tested anywhere".

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 793ceaa: READY

The arm I explicitly declined to ask for is here, and the mutation that it exists for fails thirteen checks.

What it adds, and why each one is the right one

const int32_t n = 248320 + trial % 8;      // every remainder mod the vector width: the tail runs on 7 of 8
if (trial % 6 == 0) {                       // ties across lane / vector boundaries and on the last element
    const int32_t b = 8 * (int32_t)(rng() % (uint32_t)(n / 8 - 2)) + 7;
    logits[b] = logits[b + 1] = logits[b + 9] = logits[n - 1] = 80.0f;
}
if (trial % 9 == 7) { logits[n - 1] = 90.0f; }   // the last element alone holds the maximum

n = 248320 + trial % 8 is the whole point: 248,320 is a multiple of 8, so every previous trial left the AVX2 scalar tail unexecuted — seven of the eight remainders now run it, and the eight that does not is the one that proves the vector path still agrees. The tie placements are chosen at 8k+7, 8k+8, 8k+16 and n-1, i.e. across a lane boundary, across a vector boundary, and in the tail, which are the three places an argmax that compares lanes to a register instead of to the running best will differ from a scalar scan.

And the scalar body is asserted separately (…_argmax_scalar(...) == ref), so the two implementations are checked against the reference and against each other, on the same logits — which is the only way a divergent tie-break in one of them becomes visible.

The mutation that justifies the arms

Dropping the last element from the scalar scan (i < n_vocab - 1) — the exact defect a "tail" arm exists for:

/home/bcloud/wt/prs2-llama/tests/test-sampling-fast-greedy.cpp:193: CHECK failed: …
:193: CHECK failed: …
:201: CHECK failed: …
test-sampling-fast-greedy: 13 failure(s)

13 checks, three distinct lines — so a loop bound that skips the tail is caught by many trials rather than by one lucky case, and I restored the source afterwards.

The suite itself: test-sampling-fast-greedy: OK on this head, built in the tree where the HRX tests live (the bcli tree has LLAMA_BUILD_TESTS=OFF, so my first attempt got No rule to make target — worth knowing for the next lane that reaches for the CLI tree).

One thing I would say to my own earlier review: on #387 I wrote "I am not asking for a tie case: the two agree structurally, and a tie assertion would only restate the scan." That was right about the scalar scan and wrong about the vector one: 8k+7/8k+8/8k+16 are exactly the ties where a lane-wise maximum has to be folded back into a running best, and the structural argument does not cover it. The arm belongs in the suite, and this PR is the correction.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

793ceaa READY — I built it and ran it, and the new arm is load-bearing: the same mutant that passes on #398's head fails here.

Built and ran (cmake --build … --target test-sampling-fast-greedy, ~/lb/thermal-run.sh, 162 G free):

#399 head, unmutated        ->  test-sampling-fast-greedy: OK                rc 0
this CPU has avx2           ->  (so the vector body is the path under test, not the scalar fallback)

The pair that makes the arm worth adding — I mutated the AVX2 tail loop (for (; i < n_vocab; ++i) -> i < n_vocab - 1, i.e. the last lane is never scanned) and rebuilt at both heads:

head n in the test same tail mutant
#398 (merged) 248320 — n % 8 == 0 on every trial OK, rc 0 — the arm cannot see it
this PR 248320 + trial % 8 — every remainder, tail runs on 7 of 8 9 failures, rc 1

That is the gap I flagged on #398 (n = 248320 = 8 x 31040 exactly, so the scalar tail never executed) turned into a measurement: the bug class is real, the new arm sees it, and the old one provably did not. It also answers the "why bother, the vector body is simple" objection — the tail is exactly where a simple body goes wrong.

On the diff you will see: a three-dot diff against 1bit/hrx-vulkan-patched re-shows #398's whole AVX2 body, because this branch predates that merge. The actual delta here is the per-trial length + the boundary-tie cases in the test. Worth rebasing before merge so the reviewer's diff is the real one — the same stale-base confusion cost me two rounds on #396.

Not verified by me: the 0.35 -> 0.044 ms per call. That is #398's number and this PR is the correctness arm for it; the arm is what I wanted present before believing the faster path is also the equal one.

@bong-water-water-bong
bong-water-water-bong merged commit badb895 into 1bit/hrx-vulkan-patched Nov 15, 2026
@bong-water-water-bong
bong-water-water-bong deleted the cl-1051/fast-greedy-tests branch November 15, 2026 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant