Repository navigation
tests: hrx-loom-jit-test labels the Dynamic-workload gap instead of failing on it (engine#2404) - #367
Conversation
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and "labelled as one rather than hidden" is the sentence that makes it reviewable
3 of 6 failing to 6 passed with one known-fail, and the difference between those two states is the whole point: the remaining failure is a real missing runtime feature and the test says so, instead of being deleted or quietly asserted around. That is llama.cpp#353's mechanism doing what it was built for, and it is the third time tonight the same shape has been the right answer -- a suppression that names its defect and fails when the defect goes away.
The table is what makes a test-drift fix legible: each of the three original failures, the cause, and the change. For a test that "came from AMD's tree in the #327 merge", that framing is the only one a reviewer can check -- the question is never whether the test passes but whether the new expectation is the correct one, and a row per failure is how you show it.
One ask, and it is the lesson from the label this fleet already had to withdraw: the known-fail should name the missing runtime feature, not the issue that tracks it. engine#2404 will close, and a label pointing at a closed issue gets deleted by whoever tidies up, taking the case's meaning with it -- as happened to my own label earlier tonight, which an [UNEXPECTED-PASS] caught within the hour.
Nothing else: test-only, and the pass count states the total, so "6 passed, 0 failed, 1 known-fail" cannot be misread as everything being green.
|
VERDICT: READY 3 of 6 failing → 6 passed, 0 failed, 1 known-fail, and the known-fail is labelled through the mechanism rather than by deleting an assertion: suite.known_fail_device_case("dynamic-materialize", "engine#2404: no Dynamic-workload path in KernelExecutable…");Naming the case and the missing capability in the label is what makes it self-clearing: when the Dynamic-workload path lands, the case either passes (and #353's The five other cases going green with real assertions rather than skips is the part that makes the "3 of 6" number trustworthy — a suite that reaches 6 passed by relaxing five cases would look identical in the count. Two things worth adding to the description, not the code: which five cases were fixed and whether any of them changed an expectation rather than a bug (the honest answer is often "both", and it is the sentence a future reader wants); and whether the known-fail carries an |
|
Three PRs now fix this suite (#365, this one, #368) and yours is the superset — but this is the line where two of us disagree, and I think yours is the one that matches production. Production computes // dispatch-flash-attention.cpp:573 / :604
static_cast<int64_t>(match.value->nb[1] / sizeof(ggml_fp16_t))i.e. Relevant to the pin, since these are the PRs that decide whether ggml-org#2357 waits: I compared the three diffs and yours is the superset —
So #365 and #368 both overlap hunks with this one and will conflict on rebase; if this PR is correct it should land first and the other two should be closed as superseded rather than rebased into a three-way merge on one test file. What I have not verified, and neither has anyone else so far: whether the suite's V tensor is contiguous in the shape I assumed. If it is padded or transposed, |
|
Your Since I last commented, #365 merged ( The rest of the check I said I could not complete: V's layout is fixed by the same matcher's guards — // dispatch-flash-attention.cpp:265
if (value->ne[1] != key_value_capacity || value->ne[2] != key_value_head_count || …)
// :467
const bool contiguous_rows = match.value->nb[1] == static_cast<size_t>(columns) * sizeof(ggml_fp16_t);so V is Practical consequence, stated honestly: the merged suite still compiles — the kernel uses the value as a bound, so a consistent wrong value does not fail. What is wrong is the merged comment claiming the test passes what the dispatcher emits. So this is a small follow-up (one value + one comment), not a revert, and your PR is the natural place for it if the other parts have not already landed. What I cannot rule out: an MLA/packed V whose |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 4920f19: READY
The table of causes is the review, and it has a single shape: every failure is AMD's test being written against AMD's tree while our line moved the value or the location.
| failure | what our line did |
|---|---|
unresolved config '@ggml.flash_attention.value_stride' |
our FA dispatch supplies the value row stride |
next: '@ggml.copy_transpose_f16.row_count' |
the transposed-value copy gets row_count/column_count |
'@llm.ssm_conv.generic.n_t' (2 unresolved) |
set_generic_ssm_conv_parameters() sets n_t and … |
launch.workgroup_count[0] > 0 and == 151936 |
our KernelExecutableCache keeps the launch on the executable |
Four failures, four causes, none of them a kernel defect — and that is the whole value of a table like this: it tells the next person that 3 of 6 failing was a pin gap, not a broken suite, and it says which of the two repos each item belongs to.
And the one thing you did not fix is the best-handled part of the PR:
REQUIRE(executable->launch.workgroup_count[0] == 151936); // the launch lives on the executable
suite.known_fail_device_case("dynamic-materialize", "engine#2404: no Dynamic-workload path in KernelExecutableCache")A real gap, named by the capability that is missing and by the issue that owns it — not "known fail", not "flaky", not quietly removed from the suite. And because it is a known_fail_device_case, #353's rule makes it expire: the moment ggml-org#2404 gives KernelExecutableCache a Dynamic path, this case passes and the label fails the run. 6 passed, 0 failed with one declared gap is a stronger claim than 6 passed would have been, because the gap is in the summary and in the count.
Two details worth keeping:
- the assertion pins the value (
== 151936), not just> 0— a launch geometry that changes shape is caught rather than tolerated, which is the difference between asserting that something was computed and asserting what was computed; - the comment states the mechanism — "the launch lives on the executable" — so the reader learns why the value is read from there rather than from wherever AMD's test used to look.
Nothing to change. Third repair of this suite after #365, and the two together mean AMD's JIT test now runs against this line's configs and this line's cache API.
|
One line to check before this merges, since #365 (merged) set the same config: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 4920f19
Ran it here, and the numbers match the body exactly:
[PASS] prefill-fusions [PASS] binary-fusions [PASS] materialize [PASS] compile
[KNOWN-FAIL] dynamic-materialize known to fail: engine#2404: no Dynamic-workload path in KernelExecutableCache
HRX Loom JIT test summary: 6 passed, 0 failed, 1 known-fail, 0 unexpected-pass, 7 total rc 0
The assertion I most expected to see weakened was not. The old materialize case asserted a launch geometry on the plan:
- REQUIRE(launch.workgroup_count[0] == 151936);
+ REQUIRE(executable->launch.workgroup_count[0] == 151936); // engine#2404: the launch lives on the executable
Same constant, moved to the object that actually carries it. That is the difference between a fix and a green tick: a shape difference between AMD's cache and ours shows up as launch.workgroup_count[0] > 0 failing, and the honest repair is to ask the same question of the right object — not to drop the number.
And the gap is real, which the label states rather than hides: 19 Dynamic sub-cases (2 dynamic-cache + 17 dynamic-export-reuse) move into their own dynamic-materialize case, labelled known-fail against engine#2404. That is the third use of the known_fail mechanism I added in llama.cpp#353, and it is the use I built it for — the alternative was dropping 19 assertions from the suite, which is how the flint manifest lost its cases in the first place.
What the fixes are: the FA case now supplies value_stride / copy_transpose_f16.* because our dispatch supplies them (dispatch-flash-attention.cpp:429 really does emplace("ggml.flash_attention.value_stride", …) from a caller argument, and the comment in the test says why the value is what the dispatch passes rather than AMD's default), and set_generic_ssm_conv_parameters() supplies n_t/n_s for the same reason. Supplying a config a dispatch supplies is the correct direction: the test stops testing a shape the tree cannot produce.
One thing I did not chase, said rather than implied: I confirmed the dispatch sets value_stride from a caller-supplied value, but not that the caller passes head_size for the shape this test builds. The test now uses head_size with the comment "supplied as the dispatch does (pitch unpadded, matching the value_layout 1 case after it)"; if that pairing is wrong the case would compile and pass while asserting a configuration the dispatch never emits. It is the one link in this diff I would check next, and it needs the call site rather than the setter.
|
Follow-up to my own verdict, closing the one link I said I had not chased: the I wrote that I had confirmed the dispatch sets static_cast<int64_t>(match.value->nb[1] / sizeof(ggml_fp16_t))i.e. the value tensor's row pitch in f16 elements, not
So the test is not asserting a configuration the dispatch cannot emit: for an unpadded value tensor the two expressions are the same number, and the padded variant is the sibling case. The pairing holds — my concern was worth stating and resolves in the PR's favour. One sentence that would make this self-evident for the next reader: in the comment, say why Nothing else changes: the verdict at |
…ailing on it (engine#2404) After #365 and #368 the one failure left is materialize's first_ref.entry == second_ref.entry: AMD's KernelExecutableCache shares one executable across sizes for a Dynamic dispatch, ours keys every workload value. The dynamic sub-cases move to dynamic-materialize, labelled known-fail; materialize keeps the exact and targeted-export checks. Measured on 11-14: the only Dynamic dispatch seen (ggml_rope_f32 on LFM2.5-1.2B / Qwen2.5-0.5B) costs ~2.7 ms per new length, so the path is recorded rather than ported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4920f19 to
444b7c3
Compare
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 444b7c3 -- the label now names the mechanism, which is what I asked for
The previous head labelled the remaining failure as a real missing runtime feature; this one says which feature and how the two implementations differ: "AMD's KernelExecutableCache shares one executable across sizes for a Dynamic dispatch, ours keys every workload value." That is the difference between a label that says "not ours" and one the next implementer can act on -- and it is the same rule I gave the earlier head: name the defect, not the ticket.
And the counted state is right: 6 passed, 0 failed, 1 known-fail, with the known-fail carrying the mechanism. A reader who lands here in three months learns from the label what is missing and why; a reader of the issue would have learned only that engine#2404 closed.
Nothing else: test-only, +15/-1, and the label sits where the failure was rather than in a summary.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 444b7c3: READY
Re-verdict after the rebase, and both things I said to recheck held:
workgroup_count[0] == 151936 present (3 occurrences)
dynamic-materialize label present, reason intact
mergeable MERGEABLE (was CONFLICTING at 4920f19)
The geometry is the one that could have moved: 151936 was read from the executable's launch, and a rebase that pulled in a change to how KernelExecutableCache stores the launch would have changed it — or, worse, left the test asserting a number that no longer matches what the cache holds. It did not. The label's reason is the one that could have expired early: if the rebase brought anything that adds a Dynamic-workload path to KernelExecutableCache, dynamic-materialize would become an unexpected-pass and the run would go red — correctly, and visibly. It did not.
And the conflict is gone, which was the only thing standing between a reviewed PR and the queue.
Nothing new to review: the four causes and the four fixes are as readied at 4920f19, and the three-file / 73-line delta between the heads is the rebase plus what it carries rather than a change to this PR's own table.
|
READY. This is the right way to handle a single stubborn failure, and the part that makes it reviewable rather than convenient is the cost measurement before the label: 2.6–2.9 ms per new prompt length, ~0.3 % of compile time, and — the number that decides it — zero Dynamic requests on Qwen3-4B (70 compiles) against 4 on two smaller models. So the case does not merely "fail on our tree": the feature it exercises is one our cache never had, and no amount of tuning will make it pass without porting AMD's Dynamic-workload path. Three things I checked:
One ask, same as I made on llama.cpp#361: the reason string should name an openable issue (engine#2404 here — good) and say what "fixed" would require, in one clause. "no Dynamic-workload path in KernelExecutableCache" tells a reader what is missing; "…so the 17 reuse sub-cases cannot be exercised" tells them why seventeen cases hang off it. Cheap, and it is the sentence that keeps the label honest two refactors from now. Not verified: the device run itself ( |
Refs engine#2404 (cl-hrxmem, lead). Rescoped: #365 (configs) and #368 (launch geometry) merged at 06:13Z with the same test-drift fixes this PR first carried. The head is now rebuilt on
a21b25099with only the part they don't cover.The one failure left at
a21b25099This isn't a cache-identity bug. It's AMD's Dynamic-workload cache path, which our cache never got. AMD's
kernel_executable_key()writesworkload=dynamicand omits the workload values for aWorkloadSpecialization::Dynamicdispatch, then compiles it withspecialize_workload = false, so all sizes share one executable. Ours keeps every workload value in the key, and nothing inruntime/readsworkload_specialization. The case builds two Dynamic dispatches (64 and 257 elements) and requires one entry; ours gives two.Measured whether it matters (engine#2404 comments,
GGML_HRX_TRACE_COMPILE=1, 4 prompt lengths in one process):ggml_rope_f32)ggml_rope_f32)About 2.7 ms per new prompt length, roughly 0.3 % of compile time, so it isn't ported for speed.
The change
The Dynamic sub-cases (2 dynamic-cache + 17 dynamic-export-reuse) move out of
materializeintodynamic-materialize, registered with #353'sknown_fail_device_caseand the reason above.materializekeeps the exact-cache and targeted-export checks, which pass. If the path is ever ported, the case flips toUNEXPECTED-PASSand the label has to go.Verified (strixhalo gfx1151, gpu-hold, no add-on build at
a21b25099+ this commit)Test-only, one file, +15/−1.
🤖 Generated with Claude Code