Skip to content

tests: fa_prefill_full_cache checks the merged mask path, not the retired compact mask (engine#2374 item 3) - #343

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-hrxmem/2374-fa-full-cache
Nov 13, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-hrxmem/2374-fa-full-cache

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 13, 2026 •

Copy link
Copy Markdown

Refs engine#2374, item 3 (fa_prefill_full_cache.512: "the compact F16 mask alternate is absent from the plan"). Items 1 and 2 are not touched (1 is #341; 2 is pi-i679's FA accumulator).

What the case was checking

The case asserted the pre-merge design: the postprocess match published a compact [tokens x tokens] F16 mask as an alternate value, through a qwen3_moe:qwen_attention_metadata initialization dispatch, and the flash attention ran at key_value_token_count == token_count against that compact mask. The engine#2138 merge (#327) took AMD's dispatch-qwen-attention-postprocess.cpp, which no longer has that path (-153 lines: append_attention_metadata_initialization and the qwen.attention.compact_mask alternate). AMD removed the code and this case at 9bf5871: git grep fa_prefill_full_cache 9bf5871 -- tests and git grep attention_metadata 9bf5871 -- '*.cpp' are both empty. Our test file kept the case, so after the merge it checked a path that no longer exists.

What the merged plan does (measured, gpu-hold, HRX0, base e7b8a14db, plan printed by a local probe that is not in this PR)

init dispatches: none named qwen_attention_metadata
main loom_libs:ggml_flash_attention_f32_f16_wmma q=256 kv=40960 bindings=6 mask(b3) len=20971520
main loom_libs:ggml_flash_attention_f32_f16_wmma q=256 kv=40960 bindings=6 mask(b3) len=20971520
-> test-hrx-ops.cpp:2406: requirement failed: compact_mask != nullptr

So the flash attention reads the graph's own mask over the whole key/value view, in query chunks, each chunk binding its own mask rows. On this case's full-capacity view (40,960 rows for a 512-token prefill), that is 80x the key/value rows of the old compact path. That cost does not reach a real graph. llama_kv_cache::get_n_kv() sizes the view to the used cells padded to 256, and src/models/qwen4exp.cpp uses the same get_n_kv(). A full-capacity view occurs only in the reserve graph (kv->get_size()) or with a full cache, where every row is needed anyway. The compact mask saved at most the <256 padding rows.

The change (test only)

fa_prefill_full_cache.512 now checks the merged path end to end:

  • no qwen_attention_metadata initialization dispatch;
  • every FA chunk runs at key_value_token_count = the view (mask->ne[0]) and has 6 bindings with apply_gate == "0";
  • binding 3 is the graph mask at the chunk's own row offset (first_query * kv * 2) and length (chunk_tokens * kv * 2);
  • the chunks cover every query token.

Verified (strixhalo gfx1151, gpu-hold, no add-on build at e7b8a14db + this commit)

  • test-hrx-ops --hrx-ops-test-test-filter '^(fa_|flash_attention)': 35 passed, 1 failed, 36 total. fa_prefill_full_cache.512 PASS. The one failure is flash_attention.accumulator_range.128.1024.v60000, which is item 2: it is in the issue's failure list for the base, and I did not rerun the base myself.
  • Mutant: the chunk helper keeps binding 3 at offset 0 for every chunk (dispatch-query-chunks.cpp, not committed). The case fails at bindings[3].offset == covered_query_tokens * mask_row_bytes, so the new offset check fires on a wrong chunk mask.
  • The full 270-case suite was not rerun; only tests/test-hrx-ops.cpp changes, in this one case.

If the compact mask should come back as an optimization instead, that would be porting the 153 lines back onto AMD's file; given get_n_kv(), I don't think it buys anything on a real graph.

🤖 Generated with Claude Code

…ired compact mask (engine#2374 item 3)

Since the engine#2138 merge (#327) the postprocess match no longer publishes a
compact [tokens x tokens] F16 mask through a qwen_attention_metadata
initialisation dispatch; AMD removed both the code and this case at 9bf5871.
The flash attention reads the graph's own mask over the whole key/value view,
one query chunk at a time.

The case now checks that path: no metadata initialisation, every FA chunk at
key_value_token_count = the view, binding 3 = the graph mask at the chunk's own
row offset and length, and the chunks covering every query token. A real graph's
view is get_n_kv() (used cells padded to 256), so the full-capacity view only
occurs in the reserve graph or a full cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at ab8ab8c — and the line worth keeping is the one that asserts the old mechanism is gone, not the three that assert the new one exists.

What the change actually buys

The case was written against the pre-merge design, so after #327 it asserted something the code no longer does — a stale test is worse than a missing one, because it either fails for the wrong reason or gets loosened until it passes. Three parts of the update matter more than they look:

  1. REQUIRE(mask_value->ne[0] == kFullCacheRowCount) and ne[1] == token_count pin the layout the reader now uses, and integer_parameters.at("key_value_token_count") == kFullCacheRowCount pins it again at the dispatch, where the kernel actually consumes it. Two places, same fact — that is how a layout assertion stops being decorative.
  2. REQUIRE(kernel_name_for_id(...) != "qwen3_moe:qwen_attention_metadata") is a negative arm: if the compact-mask alternate ever comes back, this fails. Tests that only assert the new shape cannot tell "replaced" from "both", and the fleet has spent the night on exactly that difference.
  3. The comment says when this test would become true again — "the full-capacity view this case builds only occurs in the reserve graph or a full cache" — which is the right kind of note: a case whose premise is conditional says so.

One number in the comment is doing two jobs

kFullCacheRowCount = 40960 is the cache capacity this case builds. The comment explains the real graph as "llama_kv_cache::get_n_kv() (the used cells padded to 256)" — 256, not 40960 — and then says "this case builds only occurs in the reserve graph or a full cache". Both statements are true and they are about different quantities, ten thousand rows apart, in adjacent sentences. A reader can easily take 256 as this case's row count and conclude the test exercises a 256-row view when it exercises a 40960-row one. One clause fixes it: "…padded to 256; this case builds the full-cache extreme, 40960 rows."

The question I would ask before merging, and the answer I found

It is a scheduling case (run_qwen_full_cache_prefill_flash_attention_scheduling_case): it asserts the plan and the dispatch, not logits. So it cannot tell whether the merged-mask read is numerically right, and after this change nothing here claims it does — which is fine, if something else does. I checked: test-hrx-ops.cpp carries 241 reference-comparison mentions across the flash-attention cases (flash_attention.* with cpu_reference/max_diff), so the numeric half exists elsewhere in the same suite and this case is not the only guard on the path.

Worth one clause in the body saying that, because the obvious reviewer question — "you changed what this test checks; who checks the numbers now?" — has a good answer and should not require the next reviewer to run the same grep.

@bong-water-water-bong
bong-water-water-bong merged commit f2bd9c3 into 1bit/hrx-vulkan-patched Nov 13, 2026
@bong-water-water-bong
bong-water-water-bong deleted the cl-hrxmem/2374-fa-full-cache branch November 13, 2026 21:40
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY ab8ab8c

This is a rewrite of a case's expectation (+25/−25 in a test), which is the shape I normally challenge — and it holds up, for reasons worth naming.

It flips "the old design is present" into "the old design is absent, and the new one is positively required":

REQUIRE(mask_value->ne[0] == kFullCacheRowCount);
REQUIRE(mask_value->ne[1] == token_count);
…
REQUIRE(kernel_name_for_id(…) != "qwen3_moe:qwen_attention_metadata");

The prohibition is an active assertion, not a deleted one: if the pre-ggml-org#2138 metadata dispatch ever comes back into this plan, the case fails. That is the right way to retire an expectation — keep the old behaviour as a named thing you refuse, rather than silently dropping the line that mentioned it.

And the new assertions are precisely shaped, not "a dispatch exists":

key_value_token_count == kFullCacheRowCount
bindings.size() == 6
apply_gate == "0"
bindings[3].value  == mask_value->id
bindings[3].offset == covered_query_tokens * mask_row_bytes
bindings[3].length == chunk_tokens * mask_row_bytes

The offset/length pair is the part that actually proves the claim in the PR description — each query chunk binds its own rows of the full-capacity mask — and it is checked against a running covered_query_tokens, so chunking is verified rather than assumed. A weaker test would have asserted one dispatch and a mask id.

The conditional loop is guarded, which is the failure I would most expect here: the if (name != "…flash_attention_f32_f16_wmma") continue; filter would have made every assertion inside it optional — a case that passes when FA is not in the plan at all. REQUIRE(flash_dispatch_count > 0) closes that. (That exact hole — OBJECT target outside the guard, compile unconditional, execute optional — is the reason engine#2333 exists, so it is worth noting when someone gets it right.)

One thing to confirm before merge (not a blocker for me, a check for you): the case now depends on kFullCacheRowCount being the padded-to-256 view rather than the case's own token_count. If that constant ever tracks a real cache size instead of the padding rule, both the ne[0] assert and the mask_row_bytes arithmetic move together and the case stays green while testing a different configuration. One comment line tying the constant to llama_kv_cache::get_n_kv()'s padding (as your PR text already explains) would keep the next reader from "fixing" it to token_count.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant