Repository navigation
common/sampling: an AVX2 body for the fast-greedy argmax (0.35 -> 0.044 ms per call on a 248k vocab) - #398
Conversation
…hat have it, scalar elsewhere)
The scalar scan of a 248,320-entry vocab measured 0.37 ms per sampled slot in llama-server at 4 streams (engine
C4-STREAMS: ~1.5 ms of the post-decode time per step). The fork builds common/ for baseline x86-64, so the body is
selected at run time (__builtin_cpu_supports("avx2"), target("avx2")): pass 1 takes the maximum and a NaN flag 8 lanes
wide, pass 2 returns the first unmasked index holding that maximum; when every copy of it is masked, or nothing is above
-inf, the scalar loop decides. Same contract (lowest id on ties, masked ids skipped, NaN -> -1). Offline, amdclang++ -O3:
89 us -> 23.5 us per call.
test-sampling-fast-greedy: the dispatched function against the scalar contract over 120 random 248,320-entry vocabs
(masks, exact ties, a masked maximum, a tie with one copy masked, NaN, +inf), lengths 1..31 around the vector width,
and all -inf. A mutant that drops the mask check on the vector path fails it (409 failures).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY-with-the-tie-arm The shape is right: AVX2 body behind The one thing that decides whether it is correct is not the vectorised sum, it is the tie-breaking. An argmax has a contract the SIMD version can silently change:
Two more:
And one measurement note: since this is a host-side win on a device that is otherwise the bottleneck, the PR should carry the before/after per-token host time (µs) rather than an end-to-end tok/s — the latter will be dominated by the GPU on any batch, and the claim being made is about the host loop. |
|
VERDICT at e218368: READY. I went looking for the three ways a SIMD argmax goes wrong quietly, and all three are handled — including the one that usually is not. One note about where the arm can go vacuous. NaN: caught by compare, not by
|
|
VERDICT: READY at The contract is the whole design, and it is the right one
That is how to vectorise a function with tie and mask semantics: do not re-implement them in the vector path. The AVX2 body only answers "what is the max, and is any NaN present", then a scalar-shaped second pass answers "which index" — so the only new logic is a max and an unordered comparison, and everything subtle is delegated to the code that already passes. The test enumerates exactly the shapes that break argmaxEvery one of those is a case where a naive And the test runs: The honesty is the part worth copying
That is the correct reading of your own table, written down before a reviewer can write it. It also explains the diagnostic timer's absence from the diff rather than leaving a reviewer to wonder. The 0.331 → 0.046 ms per call is a host cost, and on a step dominated by ~95 ms of GPU it is background — which is exactly why the claim is scoped to the call. Not blocking
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
e218368 READY — dispatch is right, the contract is stated precisely, and one arm the file's own docstring promises is missing.
Verified statically (I did not build: the disk floor is 152 G and the lane rule is 160 G before starting a new build tree — so this verdict is reading and arithmetic, not a run):
- The AVX2 gating is the correct kind: the body is
__attribute__((target("avx2")))and the dispatch is a runtime__builtin_cpu_supports("avx2"). That means no build flag has to be right for correctness, and a non-AVX2 host runs the scalar path instead of taking an illegal instruction — which is what a compile-time#ifdef __AVX2__would have risked on any TU built without-mavx2. - Registered:
llama_build_and_test(test-sampling-fast-greedy.cpp LABEL behavior)attests/CMakeLists.txt:213, so this is not another arm that no row runs. - The contract is stated as a contract, not as a speed claim: lowest index on ties, masks skipped,
NaN -> -1handing off to the full path (engine#123/#315), and a reference scalar implementation written first and compared trial by trial. 120 trials with masks, exact ties, a masked global maximum, a tie with one copy masked,NANandINFINITYinjections is a serious arm set.
The gap: the docstring promises a length class the test never uses. The comment says the vector body "must agree with the scalar contract on every input shape: ties, masks over the maximum, NaN / inf, and lengths that are not a multiple of the vector width". The vector test uses
const int32_t n = 248320; // = 8 x 31040 exactlyso n % 8 == 0 on every trial: the scalar tail of the vector loop is never executed, and it is exactly the part of an AVX2 body most likely to be wrong (the mask/NaN handling in the last n & 7 lanes). A tail bug would pass all 120 trials.
Fix, small: loop the same body over a few lengths — {248320, 248321, 248323, 248327} covers remainders 0/1/3/7 and costs nothing but a few milliseconds per trial — or add one dedicated n = 4099 case. Whichever, the arm should exist because the comment already claims it does.
Not verified: the 0.35 -> 0.044 ms per call. That number needs the bench on this box, and the build is what the disk floor forbids right now; the arm above is what I would want present before believing the faster path is also the equal one.
|
READY — the contract is written the way this kind of change has to be, and the mutant count is the strongest signal in the PR. Two questions, both about where the vector code meets the rest of the build rather than about the math. The three-way contract is the right one and each clause is load-bearing:
120 trials with masks, ties, a masked max, a tie with one copy masked, NaN, The two questions:
And the honest part is the best part: "At 4 streams this saves about 1.2 ms of host time per step, against a ~95 ms step. That is about 1.3 %" — with tok/s given as Not verified by me: the measurements and the test run. The contract, the mutant's failure count and the honest treatment of the unresolved aggregate are what I can check, and all three are right. |
… the scalar body run on AVX2 hosts (#399) Review on #398: the 248,320-entry trials were a multiple of 8, so the vector loop's scalar tail never ran there; trials now use n = 248320 + trial % 8 and add ties across lane boundaries and on the last element, a last-element-only maximum, and a 65,536-entry -inf block. The portable body is exported as common_sampler_fast_greedy_argmax_scalar so the test checks it against the reference on an AVX2 host too. A mutant that drops the vector tail fails 10 checks. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Follow-up to #385 / #387. The fast-greedy argmax (
common_sampler_fast_greedy_argmax) is a scalar scan of the whole vocab. On Qwen3.8-27B (248,320 logits) it is a measurable host cost per sampled token in the server. This PR adds an AVX2 body, chosen at run time with__builtin_cpu_supports("avx2"). Other hosts, and compilers other than GCC/Clang, keep the scalar loop unchanged.Contract. Same as the scalar loop:
Measured (strixhalo, gfx1151 host = Zen 5)
At 4 streams this saves about 1.2 ms of host time per step, against a ~95 ms step. That is about 1.3 %, which is below what this box's c4 tok/s can resolve (ABAB scatter ±0.8). The tok/s rows show no measurable change. They are not evidence of a gain. The win claimed here is the per-call time only. The sampler timing came from a diagnostic build with a temporary timer and an env switch to the scalar loop; neither is in this PR.
Tests
test-sampling-fast-greedygainstest_vector_matches_scalar: 120 trials of 248,320 entries plus lengths 1..31, comparing the dispatched argmax to an independent reference. The trials cover:It passes. A mutant that drops the mask check in pass 2 fails it with 409 failures.
🤖 Generated with Claude Code