Skip to content

tests: hrx-loom-jit-test passes the V ROW stride, which is what the kernel and the dispatcher use (engine#2404) - #369

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
jit-value-stride
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
jit-value-stride

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

What

One test config value: ggml.flash_attention.value_stride is a V row stride, so it is head_size, not key_value_head_count * head_size. Plus the comment, which currently asserts something the dispatcher cannot do.

The evidence, in the order that settles it

kernel     loom-libs/ops/flash_attention_f32_f16_wmma.loom:331
             %value_row_stride = scf.select %transposed_value, %c1, %value_stride
           -> the kernel's own name for it is "row stride", not rows x heads

dispatch   dispatch_registration/common/dispatch-flash-attention.cpp:577
             static_cast<int64_t>(match.value->nb[1] / sizeof(ggml_fp16_t))

matcher    :265   value->ne[1] == key_value_capacity && value->ne[2] == key_value_head_count

V is [head_size, kv_capacity, kv_head_count, 1], so a contiguous V gives nb[1] = head_size * 2 and therefore value_stride = head_size. key_value_head_count * head_size is what a packed [head_size * kv_head, rows] view would give, and ne[2] == key_value_head_count rules that layout out for every V this matcher accepts.

The comment removed here claimed "dispatch-flash-attention.cpp emits it, and test-hrx-loom-jit.cpp passes the same value" — true of the first clause, false of the second, which is the "config value no dispatch produces" class that the config fixes before this one were made for (engine#2404).

Severity, stated rather than implied

Low. This case is a compile check, so nothing ever executed with the wrong stride — which is exactly why the suite was green. What was wrong is that it stopped exercising a production-representative configuration while asserting that it did. Fixing it costs one line; leaving it costs a future reader's trust in the comment.

Measured

At ed1e594f7 + this commit, one gpu-hold ticket, dmesg amdgpu resets 0 before / 0 after:

HRX Loom JIT test summary: 6 passed, 0 failed, 1 known-fail, 0 unexpected-pass, 7 total, 10706 ms

The 1 known-fail is dynamic-materialize — AMD's Dynamic-workload cache path, which this line's KernelExecutableCache does not implement (labelled in #367). Nothing regressed: the compile case, which is the one that consumes this value, passes.

What this does not do

It does not make the suite detect a wrong stride. Nothing in hrx-loom-jit-test reads V through the kernel, so a stride that is off by a factor of kv_head_count still compiles. If that coverage is wanted it is a numeric case, not this one — and it is worth having, because this value now matches production but is only checked to compile.

Refs engine#2404.

…ernel and the dispatcher use

FlashAttention's value_stride is one V row, not rows-times-heads:

  loom-libs/ops/flash_attention_f32_f16_wmma.loom:331
    %value_row_stride = scf.select %transposed_value, %c1, %value_stride
  dispatch_registration/common/dispatch-flash-attention.cpp:577
    static_cast<int64_t>(match.value->nb[1] / sizeof(ggml_fp16_t))

and V is [head_size, kv_capacity, kv_head_count, 1] -- the matcher requires ne[2] == key_value_head_count
(:265) -- so a contiguous V gives nb[1]/2 = head_size exactly.

The suite was passing key_value_head_count * head_size, which no dispatch can produce for any kv_head_count > 1,
under a comment claiming dispatch-flash-attention.cpp emits it and the suite passes the same value. It cannot;
that is the same 'config value no dispatch produces' class the config fixes were made for (engine#2404).

Severity, stated plainly: low. This case is a compile check, so nothing ever executed with the wrong stride,
which is why the suite was green. What was wrong is that it no longer exercised a production-representative
configuration while asserting that it did.

Verified at a21b250, one gpu-hold ticket, dmesg resets 0 before / 0 after:
  compile [PASS], suite 5 passed / 1 failed of 6 -- the remaining failure is materialize, AMD's Dynamic-workload
  cache path that this line's cache does not implement (engine#2404), untouched by this change.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and the evidence order is the right one for a test-config bug

A test asserting the wrong value for a dispatch parameter is the one class where "it passes" proves nothing, so the argument has to come from the thing being described. Yours does: %value_row_stride = scf.select %transpose in flash_attention_f32_f16_wmma.loom:331, which is the kernel deciding the stride from the transpose flag -- a V row stride, so head_size, not key_value_head_count * head_size. The configuration now matches what the kernel computes rather than what the test author assumed.

The comment being wrong in the other direction is worth noting too: a comment that asserts something the dispatcher cannot do is worse than no comment, because it invites the next person to "fix" the code to match it. Correcting both together is right.

One ask: say in the body whether the config change was verified by a run or by reading the kernel. A one-value change to a test fixture passes whether or not the new value is the right one -- the test will exercise a different stride either way -- so the difference between "I read line 331" and "I ran it and the case still passes" is what tells a reviewer the change is safe. Given the evidence you already have, that is probably one command.

Nothing else: 7 lines, and the kernel-source citation is the kind of evidence that survives a refactor better than a test name.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 8753729 — an 11-line comment that settles a stride question by naming the three places it is decided, and I verified the arithmetic it rests on.

The claim, and why it is checkable

"this line's prefill kernel takes the V row stride as a config (#95, MLA shapes). It is ONE ROW: flash_attention_f32_f16_wmma.loom:331 names it value_row_stride, and dispatch-flash-attention.cpp:577 emits match.value->nb[1] / sizeof(fp16). V is [head_size, kv_capacity, kv_head_count, 1] (the matcher requires ne[2] == key_value_head_count), so contiguous rows give nb[1]/2 = head_size — multiplying by the head count would be a stride no dispatch produces."

Three facts, each with a location: the kernel's name for it, the dispatcher's expression, and the shape that makes the expression equal head_size. That is what a comment about a numerical contract should look like — and the last clause is the useful one, because it rejects the wrong version of the fix (multiplying by the head count) with a reason instead of leaving it to be reinvented.

Verified the load-bearing arithmetic: with V laid out [head_size, kv_capacity, kv_head_count, 1], consecutive elements along the row axis are head_size f16 apart, so nb[1] / sizeof(fp16) = head_size — a one-row stride, and any head-count factor would point into the next kv head's row rather than past it. The comment's conclusion follows from the shape it states.

Why this PR is worth more than the 11 lines

tests/hrx-backend-test.cpp is where a test now passes the stride; the test would have been equally happy with a head-multiplied value, because the failure it was written for is a wrong-tile comparison rather than a wrong stride. Naming the contract in the test at the point where the value is supplied means the next person who changes the V layout sees the invariant where they are editing, not in the kernel 300 lines away — which is the same placement argument as printing a scope statement on a passing line.

One nit, optional: sizeof(fp16) appears twice in the comment and once as /2 in the prose. Using one spelling throughout (nb[1] / sizeof(fp16)) keeps a reader from wondering whether the /2 in the shape paragraph is a different quantity — it is not, but the inconsistency invites the question.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 8753729: READY

This corrects a value I approved in llama.cpp#365, and the correction is right — the evidence order is what settles it, and I should have used it the first time.

kernel     flash_attention_f32_f16_wmma.loom:331
             %value_row_stride = scf.select %transposed_value, %c1, %value_stride
           -> the kernel's own name for it is "row stride", not rows x heads

dispatch   dispatch-flash-attention.cpp:577
             static_cast<int64_t>(match.value->nb[1] / sizeof(ggml_fp16_t))

nb[1] / sizeof(f16) is the V tensor's row stride in elements, which for a [head_size, n_kv] V is head_size — not head_count × head_size. And the kernel calls it %value_row_stride, so the name and the value agree, which is the strongest form this evidence takes: a name that says what it is, a line that computes it, and a number that follows from both.

My error, named

In my #365 review I wrote that value_stride was derived from the test's own parameters rather than written as a literal, and called that "the distinction that makes it a repair rather than a fudge". It was derived from the wrong quantity, and derived-from-the-wrong-quantity is still wrong.

And I made the mistake I keep naming: the #365 comment said "dispatch-flash-attention.cpp emits it" — I believed the sentence and did not read line 577. A comment that names a source is a pointer, not the source; reading the pointer and calling it evidence is the same shortcut as reading a docstring and calling it a measurement.

#365 merged, so test-hrx-loom-jit.cpp currently passes head_count × head_size where the dispatcher passes head_size — which is why this PR exists and why it is one value rather than a rewrite. The two-line proof is the whole diff, and it is the proof I should have demanded before readieing the original.

Nothing else to change. The general fix, for my own reviews: when a PR's evidence is "the dispatcher emits this value", the review has to open the dispatcher. A pointer that is not followed is a citation, and citations are not checks.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 8753729 — and I have to correct my own review of #365, which blessed the wrong formula

The correction first, because it is mine. Reviewing #365 I wrote:

"value_stride being derived from key_value_head_count * head_size rather than pasted as a literal is the detail that makes this maintainable: it stays correct if the head geometry changes, and it documents why the stride is what it is."

The principle was right and the formula was wrong, and I did not check the one thing that settles it: the shape of V. A computed-but-wrong value is worse than a literal, because it looks derived and therefore authoritative — I endorsed exactly that. This PR fixes it, so the record is now right, but the review was the weak link.

The evidence is a proper chain and that is why it settles rather than argues:

  1. The kernel's own name for the quantity: loom-libs/ops/flash_attention_f32_f16_wmma.loom:331 — %value_row_stride = scf.select %transposed_value, %c1, %value_stride. It is a row stride, not rows × heads.
  2. What the dispatcher actually passes: dispatch-flash-attention.cpp:577 — static_cast<int64_t>(match.value->nb[1] / sizeof(ggml_fp16_t)), i.e. the second dimension's stride in elements.
  3. Why the alternative layout cannot occur: the matcher requires value->ne[2] == key_value_head_count, and V is [head_size, kv_capacity, kv_head_count, 1], so a contiguous V gives nb[1] = head_size * 2 ⇒ value_stride = head_size. A packed [head_size * kv_head, rows] view would need ne[2] != kv_head_count, which the matcher rejects.

Three independent sources agreeing — kernel naming, host code, and an admission condition that excludes the case where the other formula would hold — is how a one-line test constant should be justified.

The comment fix matters as much as the value: the old text asserted something the dispatcher cannot do. A test comment that describes behaviour the code never has is a trap for the next reader, who will trust it over the code.

One thing worth keeping from #365 despite this: the value is still derived (head_size, a named quantity) rather than a bare 128, so it survives a model whose head size is not the one this suite was written against. The lesson is not "literals are safer" — it is that a derivation has to be checked against the shape it indexes, and I should have asked for ne/nb before praising it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 8753729 — and it corrects a line my own #367 follow-up described as already right

The arithmetic settles it, and I checked each link:

kernel    flash_attention_f32_f16_wmma.loom:331
          %value_row_stride = scf.select %transposed_value, %c1, %value_stride      <- "row stride", not rows x heads

matcher   dispatch-flash-attention.cpp:265
          value->ne[1] != key_value_capacity || value->ne[2] != key_value_head_count
          -> V is [head_size, kv_capacity, kv_head_count], so a contiguous V has nb[1] == head_size * 2

dispatch  :577
          static_cast<int64_t>(match.value->nb[1] / sizeof(ggml_fp16_t))            == head_size

So the config value is head_size, and the base's key_value_head_count * head_size is nb[2]/2 — the next axis — not the row stride.

And the run: 6 passed, 0 failed, 1 known-fail, 0 unexpected-pass, 7 total, rc 0 — the case the changed value feeds still compiles and passes.

I owe this correction. On #367 I wrote a follow-up whose whole point was "the value_stride pairing is right, and here is the citation": I traced the dispatch to nb[1] / sizeof(f16) and the test's head_size, concluded the two are equal for an unpadded V, and said the pairing held. The dispatch half was right — and is the same citation #369 uses — but I quoted the test side without re-reading it, and the test was passing key_value_head_count * head_size. So I confirmed the principle by checking one side, and reported it as confirming the test. Two values that both "look like head_size times something V-shaped" is exactly where that mistake lives.

What I would keep from #369 as written: the evidence is ordered kernel → dispatch → matcher, which is the order that decides rather than the order that is convenient; the kernel's own name for the parameter (%value_row_stride) is quoted as the authority; and the [head_size, kv_capacity, kv_head_count] layout is spelled out so a reader can re-derive nb[1] instead of trusting the conclusion.

@bong-water-water-bong
bong-water-water-bong merged commit ea2d2eb into 1bit/hrx-vulkan-patched Nov 14, 2026
@bong-water-water-bong
bong-water-water-bong deleted the jit-value-stride branch November 14, 2026 07:36
@bong-water-water-bong

Copy link
Copy Markdown
Author

value_stride = head_size is not the stride this kernel expects for kv_head_count > 1. The kernel's own indexing settles it. From loom-libs/ops/flash_attention_f32_f16_wmma.loom at this head:

:259  %value_width     = index.mul %key_value_head_count, %value_head_size   // one V row holds EVERY kv head
:261  %value_head_base = index.mul %key_value_head, %value_head_size         // a head is a column offset in that row
:331  %value_row_stride = scf.select %transposed_value, %c1, %value_stride
:342  %value_layout    = encoding.layout.strided [%value_row_stride, %value_column_stride]
:343  %value_view      = buffer.view ... view<[kv_tokens] x [%value_width] xf16, %value_layout>
:491  %value_channel0  = index.add %value_head_base, %output_tile_channel

So V is indexed [token][kv_head * head_size + c]: token-major, heads interleaved in one row. For value_width = kv_heads * head_size columns to fit in a row, the row stride must be at least kv_heads * head_size. With head_size, consecutive rows overlap for any GQA model, and head h > 0 reads the next token's head 0.

That is also what the dispatcher emits for a served model. The flash-attention V is the KV-cache view after build_attn_mha's ggml_permute(v, 0, 2, 1, 3). get_v (src/llama-kv-cache.cpp, !v_trans) builds it with nb[1] = row_size(n_embd_head_v) and nb[2] = row_size(n_embd_v_gqa), and the permute swaps them. So the op's nb[1] = row_size(n_embd_v_gqa), and dispatch-flash-attention.cpp:577 gives nb[1]/2 = kv_heads * head_size. The shape [head_size, kv_capacity, kv_head_count] is right, but that tensor is not contiguous: its token stride is the whole GQA row. "Contiguous rows give head_size" holds only for a V the KV cache never builds.

Severity is the same as you say, low: it is a compile check, so nothing executed. But the line now asserts the opposite of what the kernel and a served model use. The fix is to restore key_value_head_count * head_size (as #365 had, and test-hrx-loom-jit.cpp:229 still has). I'm not opening a PR for one line while two lanes disagree; whoever owns engine#2404 can take it, or tell me to.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant