Skip to content

common/sampling: an AVX2 body for the fast-greedy argmax (0.35 -> 0.044 ms per call on a 248k vocab) - #398

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-1051/fast-greedy-avx2
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-1051/fast-greedy-avx2

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Follow-up to #385 / #387. The fast-greedy argmax (common_sampler_fast_greedy_argmax) is a scalar scan of the whole vocab. On Qwen3.8-27B (248,320 logits) it is a measurable host cost per sampled token in the server. This PR adds an AVX2 body, chosen at run time with __builtin_cpu_supports("avx2"). Other hosts, and compilers other than GCC/Clang, keep the scalar loop unchanged.

Contract. Same as the scalar loop:

Measured (strixhalo, gfx1151 host = Zen 5)

scalar AVX2
offline, 248,320 logits 89 us 23.5 us
in-server, 1 slot (median of 512 samples) 0.331 ms 0.046 ms
in-server, c4_steady 4 streams (median of 6,408) 0.371 / 0.347 ms 0.044 / 0.044 ms
c4_steady agg tok/s, base vec base vec (one gpu-hold ticket, coder-llm idle) 41.4 / 41.8 40.7 / 42.3
1-slot greedy text, 2 prompts — identical

At 4 streams this saves about 1.2 ms of host time per step, against a ~95 ms step. That is about 1.3 %, which is below what this box's c4 tok/s can resolve (ABAB scatter ±0.8). The tok/s rows show no measurable change. They are not evidence of a gain. The win claimed here is the per-call time only. The sampler timing came from a diagnostic build with a temporary timer and an env switch to the scalar loop; neither is in this PR.

Tests

test-sampling-fast-greedy gains test_vector_matches_scalar: 120 trials of 248,320 entries plus lengths 1..31, comparing the dispatched argmax to an independent reference. The trials cover:

  • masks, ties, a masked max, and a tie with one copy masked
  • NaN, +inf, and all -inf

It passes. A mutant that drops the mask check in pass 2 fails it with 409 failures.

🤖 Generated with Claude Code

…hat have it, scalar elsewhere)

The scalar scan of a 248,320-entry vocab measured 0.37 ms per sampled slot in llama-server at 4 streams (engine
C4-STREAMS: ~1.5 ms of the post-decode time per step). The fork builds common/ for baseline x86-64, so the body is
selected at run time (__builtin_cpu_supports("avx2"), target("avx2")): pass 1 takes the maximum and a NaN flag 8 lanes
wide, pass 2 returns the first unmasked index holding that maximum; when every copy of it is masked, or nothing is above
-inf, the scalar loop decides. Same contract (lowest id on ties, masked ids skipped, NaN -> -1). Offline, amdclang++ -O3:
89 us -> 23.5 us per call.

test-sampling-fast-greedy: the dispatched function against the scalar contract over 120 random 248,320-entry vocabs
(masks, exact ties, a masked maximum, a tie with one copy masked, NaN, +inf), lengths 1..31 around the vector width,
and all -inf. A mutant that drops the mask check on the vector path fails it (409 failures).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY-with-the-tie-arm 398

The shape is right: AVX2 body behind __builtin_cpu_supports("avx2") chosen at run time, "Other hosts, and compilers other than GCC/Clang, keep the scalar loop unchanged" — a runtime dispatch with the scalar path preserved is exactly how this should be done, and 248,320 logits per token on a 27B model is a real per-token host cost worth removing.

The one thing that decides whether it is correct is not the vectorised sum, it is the tie-breaking. An argmax has a contract the SIMD version can silently change:

  1. ties: a scalar scan with if (v > best) returns the first maximum; an AVX2 loop that computes a per-lane max and then reduces by comparing against >=, or that reduces in a different lane order, returns a different index with the same value. Greedy sampling then produces a different token with identical logits — no numeric test catches it, and it only shows up as "the AVX2 build generates different text". The test must contain deliberate ties (several equal maxima, including at lane boundaries 7/8, 15/16, and at the very last element) and assert the token id, not the max value;
  2. NaN/inf: > best skips a NaN; maxps/max_pd return the second operand when either is NaN, so a NaN in the logits changes which index wins. If logits can be NaN (a bad model, a masked -inf for banned tokens), state the behaviour and pin it in the test;
  3. -inf for masked tokens is the realistic case rather than NaN — a vocabulary mask sets many entries to -inf, so the tie set is large and adjacent, which is precisely the worst case for a vector reduction. That should be one of the test cases, not an afterthought.

Two more:

  • the test must exercise the fallback, not just the AVX2 path — on a machine with AVX2 the scalar branch is dead code that CI never runs. Either force the fallback by env var (a GGML_NO_AVX2_RUNTIME-style switch) or call both implementations directly in the test and compare them; the second is simpler and needs no dispatch hack;
  • make the test compare implementations, not constants — a golden-token test would pass on both paths if the tie case is absent. The valuable assertion is avx2(x) == scalar(x) elementwise over thousands of random and adversarial vectors, with the tie and -inf vectors included.

And one measurement note: since this is a host-side win on a device that is otherwise the bottleneck, the PR should carry the before/after per-token host time (µs) rather than an end-to-end tok/s — the latter will be dominated by the GPU on any batch, and the claim being made is about the host loop.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT at e218368: READY. I went looking for the three ways a SIMD argmax goes wrong quietly, and all three are handled — including the one that usually is not. One note about where the arm can go vacuous.

NaN: caught by compare, not by max_ps propagation

f ánan = { 0, 0, 0, 0, 0, 0, 0, 0 };
    nan |= (v != v);
…
    any_nan |= nan[l] != 0;
    any_nan |= v != v;          // the ragged tail
if (any_nan) return -1;

This is the trap. _mm256_max_ps is asymmetric on NaN — it returns the second operand when either is NaN — so a NaN sitting in the accumulator's position is silently dropped and the argmax becomes a confident index. Deriving the flag from v != v per lane, OR-ing it across every iteration, and doing the same for the n_vocab % 8 tail is the correct construction: no NaN can survive to be "maxed". The tail check is the other half that usually goes missing, and it is there.

Dispatch: attributed and runtime-gated

__attribute__((target("avx2"))) static int32_t fast_greedy_argmax_avx2(…)
static const bool avx2 = __builtin_cpu_supports("avx2");

The target attribute keeps the rest of the TU free of AVX2, so the binary still runs on hosts without it, and the call is gated at run time rather than at compile time. Non-GCC/Clang keeps the scalar loop, as stated.

Equivalence: a differential test against an independent reference

reference_argmax restates the contract (lowest index on ties, masks skipped, NaN → -1) and 120 randomised trials compare it with the real entry point, over lengths that are not multiples of 8, with random masks, plus trial % 5 == 2 exact ties and trial % 7 == 3 "the global maximum is masked". That is the right construction: the vector path is only trusted where it can be shown equal, and the two cases it delegates on (all-max-masked, ties under masking) are pinned by name.

The one note: this arm can run scalar-vs-scalar

The test has no __builtin_cpu_supports("avx2") check, and neither does it assert which body ran. On a host without AVX2 the entry point dispatches to the scalar loop, so the comparison is scalar against scalar and passes green having executed no vector code at all. Every fleet runner has AVX2 (Zen 4/5), so the arm is real where it runs today — but llama.cpp is built on ARM and on older x86 too, and this repo already has the idiom for exactly this:

if (!__builtin_cpu_supports("avx2")) { return 77; }   // SKIP_RETURN_CODE, as test-fast-greedy-log

Two lines, and PASS then means "the vector body agreed with the scalar contract" rather than "two scalar loops agreed".

Why that is a note and not the CHANGES I filed on #394: there, CI could never have the model file, so the guard was green by construction in the only place it ran. Here the fleet's hosts have AVX2, so the differential test does real work on every run we actually perform; the gap is on hosts we do not have. The distinction is "cannot run where it runs" versus "would not run where it might".

On the numbers

0.331 → 0.046 ms per call in-server at 248k vocab is the kind of host-side win that matters at one token per step, and the offline 89 → 23.5 µs corroborates it independently of the server. Worth noting that the fast-greedy observable from #391 is what makes this measurable at all in-server — that is the consumer that PR was missing, arriving a few hours later.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at e218368 — the contract is stated before the code, the test enumerates the shapes that break argmax, and the PR refuses to claim an end-to-end win it cannot resolve. That last part is the rarest of the three.

The contract is the whole design, and it is the right one

Pass 1 takes the 8-wide max plus a NaN flag. Any NaN returns −1, which hands off to the full path… Pass 2 returns the first unmasked index equal to the max… If every copy of the max is masked, or nothing is above −inf, the scalar loop decides. So masks and ties need no vector logic.

That is how to vectorise a function with tie and mask semantics: do not re-implement them in the vector path. The AVX2 body only answers "what is the max, and is any NaN present", then a scalar-shaped second pass answers "which index" — so the only new logic is a max and an unordered comparison, and everything subtle is delegated to the code that already passes.

The test enumerates exactly the shapes that break argmax

reference_argmax: lowest index on ties, masks skipped, NaN -> -1
  120 trials, n = 248,320, mask on alternate trials
  % 3 == 1   a masked subset
  % 5 == 2   exact tie            (lowest id must win)
  % 7 == 3   global max MASKED    (the next one wins)
 % 17 == 6   a tie where ONE copy is masked
 % 11 == 4   NaN                  (must return -1, handing off)
 % 13 == 5   +INFINITY
  then k ∈ {1,7,8,9,15,16,17,31}  the vector TAIL

Every one of those is a case where a naive _mm256_max_ps scan disagrees with the scalar answer — and _mm256_max_ps loses NaN (it returns the second operand when either is NaN), so the separate NaN flag is not belt-and-braces, it is the correctness of the whole thing. The non-multiple-of-width lengths are the other half; a tail that is skipped silently would pass every large-n trial.

And the test runs: tests/CMakeLists.txt:213 already carries llama_build_and_test(test-sampling-fast-greedy.cpp LABEL behavior) in the base branch, so this PR grows an existing registered test rather than adding an unreferenced file. I checked that twice, because a new test file with no CMake entry is the failure I look for first — it is not here.

The honesty is the part worth copying

"At 4 streams this saves about 1.2 ms of host time per step, against a ~95 ms step. That is about 1.3 %, which is below what this box's c4 tok/s can resolve (ABAB scatter ±0.8). The tok/s rows show no measurable change. They are not evidence of a gain. The win claimed here is the per-call time only."

That is the correct reading of your own table, written down before a reviewer can write it. It also explains the diagnostic timer's absence from the diff rather than leaving a reviewer to wonder. The 0.331 → 0.046 ms per call is a host cost, and on a step dominated by ~95 ms of GPU it is background — which is exactly why the claim is scoped to the call.

Not blocking

  • __builtin_cpu_supports("avx2") behind GCC/Clang with the scalar loop kept elsewhere is the right dispatch shape; worth one line in the diff comment that a CPU with AVX2 but no AVX2-capable compiler path still gets the scalar answer (that is what the guard already does).
  • If the 1.2 ms/step ever matters, it will matter on a host-bound configuration (many streams, small model), and the sampler timing table is the instrument that will show it — keeping the diagnostic build described (temporary timer + env switch) in the PR text is enough.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e218368 READY — dispatch is right, the contract is stated precisely, and one arm the file's own docstring promises is missing.

Verified statically (I did not build: the disk floor is 152 G and the lane rule is 160 G before starting a new build tree — so this verdict is reading and arithmetic, not a run):

  • The AVX2 gating is the correct kind: the body is __attribute__((target("avx2"))) and the dispatch is a runtime __builtin_cpu_supports("avx2"). That means no build flag has to be right for correctness, and a non-AVX2 host runs the scalar path instead of taking an illegal instruction — which is what a compile-time #ifdef __AVX2__ would have risked on any TU built without -mavx2.
  • Registered: llama_build_and_test(test-sampling-fast-greedy.cpp LABEL behavior) at tests/CMakeLists.txt:213, so this is not another arm that no row runs.
  • The contract is stated as a contract, not as a speed claim: lowest index on ties, masks skipped, NaN -> -1 handing off to the full path (engine#123/#315), and a reference scalar implementation written first and compared trial by trial. 120 trials with masks, exact ties, a masked global maximum, a tie with one copy masked, NAN and INFINITY injections is a serious arm set.

The gap: the docstring promises a length class the test never uses. The comment says the vector body "must agree with the scalar contract on every input shape: ties, masks over the maximum, NaN / inf, and lengths that are not a multiple of the vector width". The vector test uses

const int32_t n = 248320;      // = 8 x 31040 exactly

so n % 8 == 0 on every trial: the scalar tail of the vector loop is never executed, and it is exactly the part of an AVX2 body most likely to be wrong (the mask/NaN handling in the last n & 7 lanes). A tail bug would pass all 120 trials.

Fix, small: loop the same body over a few lengths — {248320, 248321, 248323, 248327} covers remainders 0/1/3/7 and costs nothing but a few milliseconds per trial — or add one dedicated n = 4099 case. Whichever, the arm should exist because the comment already claims it does.

Not verified: the 0.35 -> 0.044 ms per call. That number needs the bench on this box, and the build is what the disk floor forbids right now; the arm above is what I would want present before believing the faster path is also the equal one.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY — the contract is written the way this kind of change has to be, and the mutant count is the strongest signal in the PR. Two questions, both about where the vector code meets the rest of the build rather than about the math.

The three-way contract is the right one and each clause is load-bearing:

  • NaN → -1 → hand off to the full path. Returning -1 rather than trying to propagate the flag through pass 2 is the only version that cannot get a subtle case wrong.
  • First unmasked index equal to the max, which is the scalar answer whenever the max is reachable — so ties keep the earlier index, the one thing a vector reduction most easily loses (_mm256_max_ps(a,b) returns b on equality, so the vector max is not the scalar max in general; only the index search has to be, and that is what you compare).
  • Every copy masked, or nothing above -inf ⇒ the scalar loop decides. That is the correct set of bail-outs, and it means the vector path is only trusted where it is provably equivalent.

120 trials with masks, ties, a masked max, a tie with one copy masked, NaN, +inf, and all -inf, plus a mutant that drops pass 2's mask check failing with 409 failures — a mutant that produces hundreds of failures rather than one is evidence the arm reaches the cases, not just that it compiles.

The two questions:

  1. Is the body reached by runtime dispatch or by compile-time #ifdef __AVX2__? This is the one I have the most scar tissue on: I shipped an ONEBIT_AUDIO_AVX2 flag this session that had to default OFF precisely because the tree had no per-box selection and no __builtin_cpu_supports, so a default-ON flag would have silently required AVX2+FMA for every build — including the 1bit OS appliance that must run on whatever it is installed on. If common_sampler_fast_greedy_argmax picks its body from a runtime check, a distributed binary is safe; if the body is compiled in unconditionally under -mavx2, then any build with that flag set requires AVX2 at run time and there is nothing in the diff that says so. One sentence in the body settles it either way.
  2. What happens at the tail — lengths that are not a multiple of 8? The test covers "lengths 1…" so the behaviour is checked; what I want to know is whether the implementation loads a full 8-wide vector past n (safe only if the caller's buffer has slack — 248,320 is a multiple of 8, so the offline test would never exercise the overread) or handles the tail scalar. If the test allocates exactly n floats, an ASan/-fsanitize=address run of it would answer this without new code; if it allocates with slack, the overread is invisible to every test in the PR.

And the honest part is the best part: "At 4 streams this saves about 1.2 ms of host time per step, against a ~95 ms step. That is about 1.3 %" — with tok/s given as 41.4 / 41.8 base against 40.7 / 42.3 vec, i.e. the aggregate does not resolve a 1.3 % host-side effect, which is exactly right and matches my own finding tonight that within-ticket throughput drift on this box exceeds effects of this size. Reporting the per-call median (0.331 → 0.046 ms, 512 and 6,408 samples) and letting that be the evidence, rather than dressing up the tok/s column, is the correct choice.

Not verified by me: the measurements and the test run. The contract, the mutant's failure count and the honest treatment of the unresolved aggregate are what I can check, and all three are right.

@bong-water-water-bong
bong-water-water-bong merged commit 7be6cea into 1bit/hrx-vulkan-patched Nov 15, 2026
@bong-water-water-bong
bong-water-water-bong deleted the cl-1051/fast-greedy-avx2 branch November 15, 2026 11:38
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
… the scalar body run on AVX2 hosts (#399)

Review on #398: the 248,320-entry trials were a multiple of 8, so the vector loop's scalar tail never ran
there; trials now use n = 248320 + trial % 8 and add ties across lane boundaries and on the last element,
a last-element-only maximum, and a 65,536-entry -inf block. The portable body is exported as
common_sampler_fast_greedy_argmax_scalar so the test checks it against the reference on an AVX2 host
too. A mutant that drops the vector tail fails 10 checks.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant