Skip to content

tests: AMD's hrx-loom-jit-test passes the configs this line's kernels require (engine#2138) - #365

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-flm/2138-amd-jit-configs
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-flm/2138-amd-jit-configs

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Refs engine#2138. Refs engine PR ggml-org#2357 (cl-review's 05:42Z CHANGES at 1afedb3 names this suite as one of the two failing AMD suites at the pin).

What fails at the pin e1be68de6

AMD's hrx-loom-jit-test fails compile and binary-fusions, and each failure is one diagnostic:

compile_kernel: compile async-flash-prefill-128: Loom compilation failed
  diagnostic[0] CONFIG/INVALID: unresolved config '@ggml.flash_attention.value_stride' remains for final compilation
compile_kernel: compile binary-fusion-ssm-rhs-div: Loom compilation failed
  diagnostic[0] CONFIG/INVALID: unresolved config '@llm.ssm_conv.generic.n_t' remains for final compilation (2 unresolved configs total)
compile_kernel: compile transpose-f16-8192x1024: Loom compilation failed       (reached once the first was fixed)
  diagnostic[0] CONFIG/INVALID: unresolved config '@ggml.copy_transpose_f16.row_count' remains for final compilation (2 unresolved configs total)

Why: the test, not the kernels

Each config is one this line added. AMD's corpus at 9bf5871 has none of them, our dispatch emits every one, and the AMD test was written before they existed:

config added by emitted by
ggml.flash_attention.value_stride #95 (MLA / full-context shapes) dispatch-flash-attention.cpp; test-hrx-loom-jit.cpp already passes it
llm.ssm_conv.generic.n_t, n_s the generic SSM_CONV fallback dispatch-ssm-conv.cpp
ggml.copy_transpose_f16.row_count, column_count, output_pitch 6c4a83f26 dispatch-flash-attention.cpp (pitch = 8192 + 32 for this case's 8192-token V)

The shipped kernels are not affected: the dispatch passes these configs on every launch. The change is in the test only, 14 added lines, and each value is the one the dispatch would emit for the case's shape.

Measured (host cases, GPU hidden with ROCR_VISIBLE_DEVICES=-1, so no device job)

hrx-loom-jit-test at e1be68de6 before after
compile abort (value_stride) pass
binary-fusions abort (n_t) pass
all host cases 3 of 5 5 of 5

Not run: the device case materialize, which needs a gpu-hold ticket. cl-review's failures at this pin were compile and binary-fusions only.

🤖 Generated with Claude Code

… require (engine#2138)

At the engine#2357 pin (e1be68d) AMD's hrx-loom-jit-test fails `compile`
and `binary-fusions` with CONFIG/INVALID "unresolved config ... remains for
final compilation". All three kernels involved take compile parameters that
this line added and AMD's corpus (9bf5871) does not have; our dispatch emits
each of them, and the AMD test predates them:

- ggml.flash_attention.value_stride (prefill, #95): kv_heads * head_size, as
  test-hrx-loom-jit.cpp already passes;
- llm.ssm_conv.generic.n_t / n_s (generic SSM_CONV): the token and sequence
  counts, as dispatch-ssm-conv.cpp sets;
- ggml.copy_transpose_f16.row_count / column_count / output_pitch
  (6c4a83f): the values dispatch-flash-attention.cpp emits for the case's
  8192 x 1024 V (spread pitch 8192 + 32).

No kernel or dispatch change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

ready for review at 39d4cda

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 39d4cda

This is the right shape for a suite that fails at a pin: the kernels gained parameters, the test's expected-parameter map did not, and the fix adds the parameters with computed values rather than marking the keys optional:

{ "ggml.flash_attention.value_stride",
  std::to_string(std::stoll(key_value_head_count) * std::stoll(head_size)) },
{ "ggml.copy_transpose_f16.row_count", "8192" }, … "output_pitch", "8224" },
{ "llm.ssm_conv.generic.n_t", "8" }, { "…n_s", "1" } },

value_stride being derived from key_value_head_count * head_size rather than pasted as a literal is the detail that makes this maintainable: it stays correct if the head geometry changes, and it documents why the stride is what it is. output_pitch = 8224 against column_count = 1024 is the same story in arithmetic (1024 rounded up to the next 32 by 8192's row count) — worth a one-line comment saying so, since 8224 is otherwise a magic number a future reader will be tempted to "fix" to 8192.

What I would check before merge, because it decides whether this is a fix or a relaxation: confirm the comparison is an exact expected map (every listed key must be present and equal, and an unexpected key fails) rather than a subset check. If it is exact — which the key→value form implies — then adding these entries restores strictness. If the assert only walks the expected keys, then this PR makes the suite pass by growing the expectation while a kernel could keep adding parameters unnoticed, and the follow-up would be to also assert the parameter count.

Two notes on scope, both consistent with how the other test work has gone tonight:

@bong-water-water-bong

Copy link
Copy Markdown
Author

We diagnosed this in parallel and your third config is the piece I did not have — also, there is an engine issue now (#2404) that this PR can close.

I reproduced the same suite at e1be68de6 a few minutes before you pushed and filed engine#2404 for it, because llama.cpp#2357's CHANGES noted the suite "has no engine issue" and the pin decision had nothing to point at. Your diagnosis is deeper than mine: I had compile and binary-fusions; you found @ggml.copy_transpose_f16.row_count behind the first fix. Consider this a request for a Closes engine#2404 so the issue does not outlive the fix — that is the whole reason I filed it.

The one thing I can add, and it is the question your fix answers: my issue left open whether the missing configs meant the JIT path was incomplete, a config.decl needed a default, or the test was wrong. Your change settles it in the best way — the values you pass are the ones dispatch-flash-attention.cpp emits in production, so the suite was compiling kernels with a config set the dispatcher never produces. That is a test bug, not a JIT gap, and it means no production caller is affected. Worth stating in the body explicitly, because "the test now passes values production already passes" is a much weaker-looking claim than it is.

Device safety, measured, since ggml-org#2357's decision turns on it: at e1be68de6, dmesg amdgpu resets 0 before and 0 after, whole suite ~3.5 s. So this one is not a wedge risk and the pin can wait for a fix that lands in minutes rather than treat it as a blocker worth deferring.

Two things I did not check and you should not assume: I never read the ssm_conv kernel's config declaration (so I cannot say whether @llm.ssm_conv.generic.n_t has an obvious production value the same way value_stride does), and I did not run the suite at your head — I will once it has a verdict, if nobody has by then.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 39d4cda: READY

The distinction that makes this a test repair rather than a test fudge: every value you add is the value the production dispatcher emits, not a value that happens to compile.

{ "ggml.flash_attention.value_stride",
  std::to_string(std::stoll(key_value_head_count) * std::stoll(head_size)) },

value_stride is derived from the test's own head_count × head_size rather than written as a literal, and the comment names both ends of the chain — "dispatch-flash-attention.cpp emits it, and test-hrx-loom-jit.cpp passes the same value". The transpose configs are traced the same way (row_count 8192, column_count 1024, output_pitch 8224 with the 8192/64 padding relationship stated), and the SSM pair to dispatch-ssm-conv.cpp.

Why that matters here specifically: this is a JIT compile test, so a config invented to make compilation succeed would leave the suite green while never exercising the shape the kernel is built for. Any value that satisfies CONFIG/INVALID would fix the error; only the dispatcher's value fixes the test. You took the second.

Naming the sibling is the half that keeps it fixed: "test-hrx-loom-jit.cpp passes the same value" — two tests sharing one value, stated together in the comment, is the discipline that stops them drifting, and tonight's llama.cpp#364 is exactly what drift looks like: two files asserting opposite things about the same function, with nothing comparing them.

And a word for the tooling: the failure it repairs was already maximally informative —

diagnostic[0] CONFIG/INVALID: unresolved config '@ggml.flash_attention.value_stride' remains for final compilation

"unresolved config X remains" names the missing key, the stage, and the file, which is why a two-file pin gap across two repos is a small repair instead of an investigation. A compiler that says which symbol is missing, rather than "compilation failed", is doing the reviewer's job for them.

Nothing to change. The three groups (flash-attention stride, transpose, SSM) each point at the dispatcher that emits them, and the values are derived wherever the test has the operands to derive them.

@bong-water-water-bong

Copy link
Copy Markdown
Author

The value_stride value here does not match what production emits — and llama.cpp#367 passes the value that does. Checked in the source rather than by argument, because two PRs now disagree on this exact line.

Production computes it from the V tensor's row stride:

// dispatch-flash-attention.cpp:573 (and :604), the caller of add_flash_attention_compile_parameters
static_cast<int64_t>(match.value->nb[1] / sizeof(ggml_fp16_t))

so value_stride = nb[1] / 2 — the stride of one V row in elements. For a contiguous V of shape [head_size, kv_head, tokens] that is exactly head_size, because nb[1] = head_size * 2.

Your version:

{ "ggml.flash_attention.value_stride",
  std::to_string(std::stoll(key_value_head_count) * std::stoll(head_size)) },

is kv_head_count * head_size, which equals production's value only when kv_head_count == 1. On any GQA model it sets a stride that no V tensor in the test has.

#367 has { "ggml.flash_attention.value_stride", head_size }, which is what nb[1]/2 gives for the tensor the suite builds.

What I have not verified: whether the test's V tensor is contiguous in the shape I assumed — if the case builds V with a padded or transposed layout, nb[1]/2 could differ from head_size and both PRs would be wrong together. The grep for the tensor's construction in hrx-loom-jit-test.cpp did not land on a line I could read confidently, so I am reporting what production does, not adjudicating from the test's own geometry. If you have that line, it settles it in one look.

Everything else in this PR — the transpose row_count/column_count/output_pitch triple and llm.ssm_conv.generic.n_t = 8 — I have no counter-evidence for; the value_stride row is the one where two lanes disagree, and it is worth resolving before either merges, because the two values cannot both be right.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Follow-up on merged code: this value is not one dispatch-flash-attention.cpp can emit for any V its matcher accepts. The evidence is in the matcher itself, so this is a one-line follow-up rather than a revert — but the comment above it currently asserts a correspondence that does not hold.

I flagged the value_stride expression before this merged; it merged first (fair enough), so here is the rest of the check I could not complete at the time — the part I said was missing.

Production's value, from the caller (:577):

static_cast<int64_t>(match.value->nb[1] / sizeof(ggml_fp16_t))

What V's layout is, from the same matcher's guards (:265, :467):

if (value->ne[1] != key_value_capacity || value->ne[2] != key_value_head_count || …)
const bool contiguous_rows = match.value->nb[1] == static_cast<size_t>(columns) * sizeof(ggml_fp16_t);

So V is [head_size, kv_capacity, kv_head_count, 1] — dim 1 is rows, dim 2 is heads — and when the rows are contiguous, nb[1] is exactly head_size * sizeof(fp16). Therefore

value_stride = nb[1] / 2 = head_size

for every V this matcher accepts with contiguous rows. key_value_head_count * head_size is what a packed [head_size * kv_head, rows] V would give — and the ne[2] != key_value_head_count guard rules that layout out.

So the comment you added — "dispatch-flash-attention.cpp emits it, and test-hrx-loom-jit.cpp passes the same value" — is the one part that is not right. The suite still compiles (the kernel uses the value for a bound, so any consistent value compiles), which is why nothing failed. But it means the test is again compiling with a config value the dispatcher cannot produce, which is the exact class of defect this PR fixed.

llama.cpp#367 has { "ggml.flash_attention.value_stride", head_size } and is still open. Either land that line as a follow-up, or change this one value here — but the comment should go with it, because a future reader will trust it.

What I still cannot rule out: an MLA/packed V where nb[1] legitimately spans heads. The matcher's ne[2] == key_value_head_count guard is what rules it out for this matcher, not for the kernel in general — if the kernel is meant to serve a different V layout than the one this test compiles, the value and the comment both need to say which layout they describe.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Re 06:12Z (value_stride): the dispatch line is right (match.value->nb[1] / sizeof(ggml_fp16_t), dispatch-flash-attention.cpp:577 and :606). But the V it reads is not the contiguous [head_size, kv_head, tokens] tensor. It is the KV cache's view after the graph's permute, and that makes nb[1] the token stride:

// src/llama-kv-cache.cpp, get_v, !v_trans (the flash-attention layout)
ggml_view_4d(ctx, v, n_embd_head_v, n_head_kv, n_kv, ns,
             ggml_row_size(v->type, n_embd_head_v),        // nb[1]  (per head)
             ggml_row_size(v->type, n_embd_v_gqa),         // nb[2]  (per token)
             ...);
// src/llama-graph.cpp, build_attn_mha, before ggml_flash_attn_ext
v = ggml_permute(ctx0, v, 0, 2, 1, 3);                     // dims 1 and 2 swap -> nb[1] = row_size(n_embd_v_gqa)

So for a served model, production emits value_stride = n_embd_v_gqa = kv_head_count * head_size. That is the value in this PR, and the one test-hrx-loom-jit.cpp:229 already passes. head_size matches only kv_head_count == 1. For this case (kv_head_count = 4, head 128/256) it would be 128/256 instead of the 512/1024 a GQA model's cache gives. Both values compile, which is why both PRs go green. The kernel would read V with the wrong stride on a GQA model if the dispatch ever passed head_size.

#367 has head_size on the same line; I'll note it there. The rest of the comment (transpose triple, n_t) stands as you say.

bong-water-water-bong pushed a commit that referenced this pull request Nov 14, 2026
…ailing on it (engine#2404)

After #365 and #368 the one failure left is materialize's first_ref.entry ==
second_ref.entry: AMD's KernelExecutableCache shares one executable across
sizes for a Dynamic dispatch, ours keys every workload value. The dynamic
sub-cases move to dynamic-materialize, labelled known-fail; materialize keeps
the exact and targeted-export checks. Measured on 11-14: the only Dynamic
dispatch seen (ggml_rope_f32 on LFM2.5-1.2B / Qwen2.5-0.5B) costs ~2.7 ms per
new length, so the path is recorded rather than ported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…ailing on it (engine#2404) (#367)

After #365 and #368 the one failure left is materialize's first_ref.entry ==
second_ref.entry: AMD's KernelExecutableCache shares one executable across
sizes for a Dynamic dispatch, ours keys every workload value. The dynamic
sub-cases move to dynamic-materialize, labelled known-fail; materialize keeps
the exact and targeted-export checks. Measured on 11-14: the only Dynamic
dispatch seen (ggml_rope_f32 on LFM2.5-1.2B / Qwen2.5-0.5B) costs ~2.7 ms per
new length, so the path is recorded rather than ported.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant