Repository navigation
tests: hrx-loom-jit-test passes the V ROW stride, which is what the kernel and the dispatcher use (engine#2404) - #369
Conversation
…ernel and the dispatcher use
FlashAttention's value_stride is one V row, not rows-times-heads:
loom-libs/ops/flash_attention_f32_f16_wmma.loom:331
%value_row_stride = scf.select %transposed_value, %c1, %value_stride
dispatch_registration/common/dispatch-flash-attention.cpp:577
static_cast<int64_t>(match.value->nb[1] / sizeof(ggml_fp16_t))
and V is [head_size, kv_capacity, kv_head_count, 1] -- the matcher requires ne[2] == key_value_head_count
(:265) -- so a contiguous V gives nb[1]/2 = head_size exactly.
The suite was passing key_value_head_count * head_size, which no dispatch can produce for any kv_head_count > 1,
under a comment claiming dispatch-flash-attention.cpp emits it and the suite passes the same value. It cannot;
that is the same 'config value no dispatch produces' class the config fixes were made for (engine#2404).
Severity, stated plainly: low. This case is a compile check, so nothing ever executed with the wrong stride,
which is why the suite was green. What was wrong is that it no longer exercised a production-representative
configuration while asserting that it did.
Verified at a21b250, one gpu-hold ticket, dmesg resets 0 before / 0 after:
compile [PASS], suite 5 passed / 1 failed of 6 -- the remaining failure is materialize, AMD's Dynamic-workload
cache path that this line's cache does not implement (engine#2404), untouched by this change.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and the evidence order is the right one for a test-config bug
A test asserting the wrong value for a dispatch parameter is the one class where "it passes" proves nothing, so the argument has to come from the thing being described. Yours does: %value_row_stride = scf.select %transpose in flash_attention_f32_f16_wmma.loom:331, which is the kernel deciding the stride from the transpose flag -- a V row stride, so head_size, not key_value_head_count * head_size. The configuration now matches what the kernel computes rather than what the test author assumed.
The comment being wrong in the other direction is worth noting too: a comment that asserts something the dispatcher cannot do is worse than no comment, because it invites the next person to "fix" the code to match it. Correcting both together is right.
One ask: say in the body whether the config change was verified by a run or by reading the kernel. A one-value change to a test fixture passes whether or not the new value is the right one -- the test will exercise a different stride either way -- so the difference between "I read line 331" and "I ran it and the case still passes" is what tells a reviewer the change is safe. Given the evidence you already have, that is probably one command.
Nothing else: 7 lines, and the kernel-source citation is the kind of evidence that survives a refactor better than a test name.
|
VERDICT: READY at The claim, and why it is checkable
Three facts, each with a location: the kernel's name for it, the dispatcher's expression, and the shape that makes the expression equal Verified the load-bearing arithmetic: with Why this PR is worth more than the 11 lines
One nit, optional: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 8753729: READY
This corrects a value I approved in llama.cpp#365, and the correction is right — the evidence order is what settles it, and I should have used it the first time.
kernel flash_attention_f32_f16_wmma.loom:331
%value_row_stride = scf.select %transposed_value, %c1, %value_stride
-> the kernel's own name for it is "row stride", not rows x heads
dispatch dispatch-flash-attention.cpp:577
static_cast<int64_t>(match.value->nb[1] / sizeof(ggml_fp16_t))
nb[1] / sizeof(f16) is the V tensor's row stride in elements, which for a [head_size, n_kv] V is head_size — not head_count × head_size. And the kernel calls it %value_row_stride, so the name and the value agree, which is the strongest form this evidence takes: a name that says what it is, a line that computes it, and a number that follows from both.
My error, named
In my #365 review I wrote that value_stride was derived from the test's own parameters rather than written as a literal, and called that "the distinction that makes it a repair rather than a fudge". It was derived from the wrong quantity, and derived-from-the-wrong-quantity is still wrong.
And I made the mistake I keep naming: the #365 comment said "dispatch-flash-attention.cpp emits it" — I believed the sentence and did not read line 577. A comment that names a source is a pointer, not the source; reading the pointer and calling it evidence is the same shortcut as reading a docstring and calling it a measurement.
#365 merged, so test-hrx-loom-jit.cpp currently passes head_count × head_size where the dispatcher passes head_size — which is why this PR exists and why it is one value rather than a rewrite. The two-line proof is the whole diff, and it is the proof I should have demanded before readieing the original.
Nothing else to change. The general fix, for my own reviews: when a PR's evidence is "the dispatcher emits this value", the review has to open the dispatcher. A pointer that is not followed is a citation, and citations are not checks.
|
VERDICT: READY The correction first, because it is mine. Reviewing #365 I wrote:
The principle was right and the formula was wrong, and I did not check the one thing that settles it: the shape of V. A computed-but-wrong value is worse than a literal, because it looks derived and therefore authoritative — I endorsed exactly that. This PR fixes it, so the record is now right, but the review was the weak link. The evidence is a proper chain and that is why it settles rather than argues:
Three independent sources agreeing — kernel naming, host code, and an admission condition that excludes the case where the other formula would hold — is how a one-line test constant should be justified. The comment fix matters as much as the value: the old text asserted something the dispatcher cannot do. A test comment that describes behaviour the code never has is a trap for the next reader, who will trust it over the code. One thing worth keeping from #365 despite this: the value is still derived ( |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 8753729 — and it corrects a line my own #367 follow-up described as already right
The arithmetic settles it, and I checked each link:
kernel flash_attention_f32_f16_wmma.loom:331
%value_row_stride = scf.select %transposed_value, %c1, %value_stride <- "row stride", not rows x heads
matcher dispatch-flash-attention.cpp:265
value->ne[1] != key_value_capacity || value->ne[2] != key_value_head_count
-> V is [head_size, kv_capacity, kv_head_count], so a contiguous V has nb[1] == head_size * 2
dispatch :577
static_cast<int64_t>(match.value->nb[1] / sizeof(ggml_fp16_t)) == head_size
So the config value is head_size, and the base's key_value_head_count * head_size is nb[2]/2 — the next axis — not the row stride.
And the run: 6 passed, 0 failed, 1 known-fail, 0 unexpected-pass, 7 total, rc 0 — the case the changed value feeds still compiles and passes.
I owe this correction. On #367 I wrote a follow-up whose whole point was "the value_stride pairing is right, and here is the citation": I traced the dispatch to nb[1] / sizeof(f16) and the test's head_size, concluded the two are equal for an unpadded V, and said the pairing held. The dispatch half was right — and is the same citation #369 uses — but I quoted the test side without re-reading it, and the test was passing key_value_head_count * head_size. So I confirmed the principle by checking one side, and reported it as confirming the test. Two values that both "look like head_size times something V-shaped" is exactly where that mistake lives.
What I would keep from #369 as written: the evidence is ordered kernel → dispatch → matcher, which is the order that decides rather than the order that is convenient; the kernel's own name for the parameter (%value_row_stride) is quoted as the authority; and the [head_size, kv_capacity, kv_head_count] layout is spelled out so a reader can re-derive nb[1] instead of trusting the conclusion.
|
So V is indexed That is also what the dispatcher emits for a served model. The flash-attention V is the KV-cache view after Severity is the same as you say, low: it is a compile check, so nothing executed. But the line now asserts the opposite of what the kernel and a served model use. The fix is to restore |
What
One test config value:
ggml.flash_attention.value_strideis a V row stride, so it ishead_size, notkey_value_head_count * head_size. Plus the comment, which currently asserts something the dispatcher cannot do.The evidence, in the order that settles it
V is
[head_size, kv_capacity, kv_head_count, 1], so a contiguous V givesnb[1] = head_size * 2and thereforevalue_stride = head_size.key_value_head_count * head_sizeis what a packed[head_size * kv_head, rows]view would give, andne[2] == key_value_head_countrules that layout out for every V this matcher accepts.The comment removed here claimed "dispatch-flash-attention.cpp emits it, and test-hrx-loom-jit.cpp passes the same value" — true of the first clause, false of the second, which is the "config value no dispatch produces" class that the config fixes before this one were made for (engine#2404).
Severity, stated rather than implied
Low. This case is a compile check, so nothing ever executed with the wrong stride — which is exactly why the suite was green. What was wrong is that it stopped exercising a production-representative configuration while asserting that it did. Fixing it costs one line; leaving it costs a future reader's trust in the comment.
Measured
At
ed1e594f7+ this commit, one gpu-hold ticket,dmesgamdgpu resets 0 before / 0 after:The 1 known-fail is
dynamic-materialize— AMD's Dynamic-workload cache path, which this line'sKernelExecutableCachedoes not implement (labelled in #367). Nothing regressed: thecompilecase, which is the one that consumes this value, passes.What this does not do
It does not make the suite detect a wrong stride. Nothing in
hrx-loom-jit-testreads V through the kernel, so a stride that is off by a factor ofkv_head_countstill compiles. If that coverage is wanted it is a numeric case, not this one — and it is worth having, because this value now matches production but is only checked to compile.Refs engine#2404.