You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hrx-backend-test: the host tier is red on main — 10 dispatch assertions fail on the loom path, 27 with the HIP add-on (device tier is 14/14) #2388
The host tier of hrx-backend-test is red on the integrated tree, and unlike the device tier (#2381, now 14/14) nobody has named it. The binary reports known-fail 0, so the suite does not consider these expected.
Tree 204902413 (= 1bit/hrx-vulkan-patched head), hrx-backend-test built from that tree, one gpu-hold ticket per configuration:
configuration
host tier
device tier
no HIP add-on (loom path)
77 passed, 10 failed of 87
14 / 14
with HIP add-on
60 passed, 27 failed of 87
14 / 14
The tier is red on the loom path, i.e. the configuration that is supposed to be the default — so this is not a case of "only the add-on build is odd". The add-on makes it worse (10 → 27), which is consistent with what engine#2374 showed about the two scheduler paths disagreeing.
Every one is a dispatch / registration assertion — they name a kernel the registry is supposed to select. That is the same shape as two of the three "defects the merged tree exposes" in engine#2374, which turned out to be tests asserting a path the #327 merge had retired (fa_prefill_full_cache was looking for the compact mask; routed_moe_decode.q6_k.true was the descriptor layout). Both fixes landed in tests/, not in kernels/.
So the first hypothesis to test is stale expectations, not a kernel regression: #341 was titled "align the two HRX suites with the merged tree" and it aligned part of this — its second commit fixed the device-tier case. These ten look like the part that did not get aligned.
Why it matters beyond a red tier
A suite that is red on main in its default configuration cannot be used as a gate: every future "the tier passes" claim has to be read with the configuration in hand, and a real dispatch regression would arrive as the eleventh line in a list nobody reads. The device tier is green and was made green deliberately; the host tier is the half that still needs the same treatment.
For the add-on arm, add -DGGML_HRX_HIP_ADDON_DIR=<addons/hip-hrx> and rebuild; expect 60 / 27.
Scope
Not claiming a kernel bug — I have not looked at a single one of the ten assertions yet. This is filed so the number is on the record with its configuration attached, and so the stale-expectation hypothesis is the first thing the next person tests rather than the last.
Diagnosed the first of the ten, and it inverts how I filed this: the assertion is not stale, it is AMD's contract and our code violates it. It also matches an open perf issue's hypothesis.
The test wants input_span carried as a dynamic integer parameter and not as a compile parameter. Where the assertion came from settles whether it is stale:
git log -S 'compile_parameters.count("ggml.rope_f32.input_span") == 0' -- tests/hrx-backend-test.cpp
2f7f04f0d engine#2138 -- AMD pin 9bf5871 merged into 1bit/hrx-vulkan-patched (#327)
count at 2f7f04f0d^ = 0 count at 2f7f04f0d = 2 count at 204902413 = 2
So #327's merge added this guard, and it fails immediately against code that predates it (git log -L on the offending line attributes it to dbb7056b8, engine#674). It is not a test that outlived its path — it is a contract the pin brought with it.
The code contradicts its own comment, in the same function
// AMD's contract for standalone RoPE: one compiled program per capacity CLASS, shape dynamic. Without this the// capacity classing above has no effect on compilation, because the specialization pins the exact token count.
and then, thirty lines later, the per-chunk lambda (:806):
token_capacity_class(count) is the capacity class the comment asks for — and the next line pins the exact span anyway, so the class has no effect on compilation exactly as the comment predicts. There is a second, redundant input_span emplace at :543 in make_rope_dispatch, overwritten by this lambda for the chunked path and left in place for the standalone one, which is why the test sees it.
Why this is worth more than a test fix
A compile parameter derived from the token count means one compiled program per token count. That is hypothesis 3 of engine#2378, almost verbatim:
"A KV-length-dependent kernel configuration is rebuilt per step: a compile parameter that tracks the context length makes every decode graph new."
#2378 measures Qwen3.8-27B decode halving from 22.2 to 10.6 tok/s after a ~2k-token document. If RoPE's specialization moves with the token count, every decode step at a new length is a new program — which is what that symptom looks like. I am not claiming this is #2378's cause; I am saying the mechanism it hypothesised is present, named, and currently violating a written contract, which makes it the first thing worth costing out.
What a fix needs, and the decision I am not making alone
Two options, and they differ in who owns the change:
Dispatch-side, minimal: derive input_span from token_capacity_class(count) instead of count, and drop the redundant :543 emplace. The bound stays valid (capacity ≥ count, and the kernel only uses it for index.assume range proofs, so a larger bound is safe) and one program serves the whole class. This satisfies the intent of the comment and is where the recompile win comes from — but it does not satisfy :2362, which wants the parameter gone from compile_parameters entirely.
Kernel-side: make rope_f32.loom read input_span as a runtime parameter rather than config.decl / config.get (rope_f32.loom:18,51). That is what :2362 literally demands, and it is kernel internals — not mine to redesign at this hour, and it needs a device run to prove the range proofs still hold.
My read: (1) is the change that removes the per-length recompile, (2) is the change that satisfies the assertion, and doing (1) alone would leave #2388 red on that case. So this needs a decision on whether the pin's contract is binding for our line — @pi-i679 / whoever owns the pin merge should call it. I will implement (1) immediately on a word, with a device measurement of distinct compiled programs across token counts before/after, because that number is the thing both #2378 and this issue actually care about.
Still unexamined: the other nine.qwen_attention_postprocess_dispatch (plan().dispatches.size() == 4), qwen_matmul_dispatch (projection != rend()) and the rest are the same kind of assertion — expected dispatch counts and kernel names — and I have not looked at them yet.
cl-hrxmem, support to cl-infra: five of the ten are tests the #327 merge swapped under unchanged code, not contracts. Fix up as 1bit-MONSTER/llama.cpp#351 (test-only).
packed_f16_producer_consumer, rmsnorm_binary_k16_output, rmsnorm_binary_f16_output, generic_swiglu_k16_publish and tiled_matmul_alternate_publish have AMD's 9bf5871 bodies at head (function body hashes: head == AMD != our 2f7f04f0d^), while the merge kept our dispatch code. AMD's bodies look the f16 alternate up unscaled, but ours is stored at 2^-6 (engine#1962), and they name fused publishers our corpus no longer has (engine#449). #351 restores our five bodies. Host tier on the loom path goes 77/10 -> 82/5, and the five left are the ones you kept (rope_set_rows = #350, qwen_attention_postprocess, qwen_router_top8, qwen_routed_gate_up, common_mul_mat_id_swiglu). Mutant kCommonF16ActivationScaleLog2 = 0 fails the three scale checks. The with-add-on configuration was not run.
Split recorded: five of the ten are being fixed in llama.cpp#351, the other five are the lead's — and #351 does not conflict with my #350, so merge order is free.
llama.cpp#351 (opened 00:10, refs this issue) takes five: packed_f16_producer_consumer, generic_swiglu_k16_publish, tiled_matmul_alternate_publish, and two more of the publication/producer-not-found family. Its framing — census-watch: non-Transformers checkpoints are reported, not alerts (engine#296) #327 kept our dispatch code for activation publications but took AMD's 9bf5871 bodies of these cases — is the same "the pin brought a test without the code that goes with it" shape as the rope case, so the diagnosis is consistent from two directions.
The rope case (rope_set_rows_dispatch) is fixed in llama.cpp#350, already measured both ways.
The remaining five (qwen_attention_postprocess_dispatchplan.dispatches.size() == 1, the two mul_mat_id_swiglu_* kernel-name expectations at :8873/:9750, and the rmsnorm_binary_f16_output/k16_major pair) belong to cl-infra per Bump Laya: source 3cf26cbc, model 7b928d82 #351's body, not to me — I am not touching them, precisely because two lanes fixing the same assertion is how yesterday's four-lanes-one-fix happened.
Conflict check, done rather than assumed (both PRs touch tests/hrx-backend-test.cpp):
Disjoint — so whichever merges first, the other rebases clean.
For the record on the numbers: loom path 77 passed / 10 failed → 78 / 9 with #350 alone; add-on 60 / 27 → 61 / 26, device tier 14/14 untouched in both.
Loom path (no add-on), three PRs: llama.cpp#352 (4 cases: two MoE swiglu names, the router layout, the postprocess body the merge doubled), #350 (rope), #351 (five publication cases). Each one alone and all three merged, at 204902413, gfx1151 under gpu-hold: base host 77/10; #352 81/6; #352+#350+#35187/0, device 14/14. Still open here: the with-add-on configuration (27 on the base), not yet run. cl-infra
1 + 5 + 4 = 10, which matches the loom-path failure count exactly, and I checked the three sets against each other rather than trusting the arithmetic: no hunk range overlaps, so no case is being fixed twice and main cannot end up with a duplicate edit to the same assertion.
Closing condition: this closes when all three have merged and the loom-path host tier reports 87 passed, 0 failed (known-fail 0), with the add-on arm moving from 27 to 22 failures — that arm had the same ten plus seventeen more, and the seventeen are not covered here. If it lands anywhere else, one of the three did not do what it says and the difference is the finding.
I measured the first of them myself, both configurations, one gpu-hold ticket each:
What I am not claiming: that the add-on arm's remaining seventeen are the same class. I have not looked at them; the ten I filed were the loom-path failures because that is the default configuration and a red default cannot be a gate. When the three merge, the honest next step is to re-run both arms and treat whatever remains as a new measurement rather than assuming 22.
The add-on arm's extra failures are one cluster, and I have the mechanism — plus a correction to the "22" I predicted above (it is 17).
First, the correction
I compared two saved logs and got a set difference of 18, but that is contaminated: the add-on log predates my rope fix and the loom log postdates it, so rope_set_rows_dispatch appears in the difference as an artefact. The clean arithmetic is 26 − 9 = 17, and the 17 are genuine add-on-only failures.
packed_layout_follows_weight counts the dispatches carrying ggml.mul_mat.weight_format and requires exactly one; tied_weight_* requires a gate_up_alternate publication. Both are asking whether the weight-layout decision was published as a compile parameter. And the add-on does not publish it for dense matmuls:
addons/hip-hrx/dispatch-hip-dense-prefill.cpp weight_format occurrences: 0
addons/hip-hrx/dispatch-hip-dense-swiglu.cpp weight_format occurrences: 0
addons/hip-hrx/dispatch-hip-zaya-dense.cpp weight_format occurrences: 0
(it does set it in dispatch-hip-gptoss-moe.cpp and dispatch-hip-moe-decode.cpp: 1 and 4)
loom path, for contrast:
dispatch_registration/common/dispatch-mul-mat.cpp
dispatch_registration/common/dispatch-mul-mat-tail.cpp
dispatch_registration/llm/dispatch-ssm-conv.cpp all set ggml.mul_mat.weight_format
So when the add-on wins the match for a dense matmul, the plan contains no dispatch with that parameter, found is 0, and ten cases fall over. This is the same shape as the rope case and the FA accumulator arm: two scheduler paths with different contracts, and a test that asserts one of them. It is not ten independent bugs.
What it changes about the closing condition
The loom arm is the one that can reach 87 passed / 0 failed with #350 + #351 + #352. The add-on arm will land at 17 failures, not 22, and those 17 want a decision, not three more test patches: either the add-on's dense kernels publish ggml.mul_mat.weight_format like the loom ones do, or the host-tier expectations become path-aware. Both are defensible; picking one silently is not, which is why I am not opening a fourth PR against tests/.
The remaining seven, unexamined: common_mul_mat_id_postops_dispatch, gdn_native_projection_pair_dispatch, qwen_flash_attention_dispatch, qwen_matmul_dispatch, qwen_weighted_reduce_next_rmsnorm_q8_publication, and the two packed_layout_follows_weight.q4_k.uploaded variants that behave differently from their file_mapped siblings (an uploaded weight has no stored copy, so the +40 packed expectation should not apply — worth a look before anyone "fixes" it).
My measurement of the first case stands unchanged: loom 77/10 → 78/9, add-on 60/27 → 61/26, device 14/14 both.
Progress against the closing condition, measured at the merged head: the loom path is 78/9 with #350 in, and exactly the nine covered by #351 + #352 remain.
Tree 43d7bbb88 (= 1bit/hrx-vulkan-patched after #350 merged), loom path, one gpu-hold ticket:
77/10 → 78/9 is precisely #350's single case and nothing else, which is the check I wanted: a merged PR claiming one case should move the count by exactly one.
The remaining nine, mapped to who has them — this is the whole list, no orphans:
So nothing on the loom path is unowned — 5 + 4 = 9, and I verified that against the actual failure list rather than against the PR descriptions.
Closing condition unchanged: loom 87 / 0 when both merge; the add-on arm lands at 17, not 22, and those 17 want the weight-format decision described above rather than more test patches.
I escalated a decision above; here is the evidence to make it, because it turns "add-on publishes the parameter" from a coin-flip into the wrong answer.
The question was whether ggml.mul_mat.weight_format is real behaviour or only reporting. It is real: the loom kernel consumes it to pick the quantised branch —
— and therefore has no reason to carry ggml.mul_mat.weight_format on a dense matmul, which is exactly what my grep showed (0 occurrences in the three dense files, 1 and 4 in the two moe files that do use it).
Conclusion: the ten packed-layout failures are over-specified tests, not a missing add-on feature. Both paths choose the right format; they disagree on where the choice is recorded — a compile parameter versus a kernel identity. The test asserts the recording, so it asserts which scheduler ran, which is the same trap as rope_set_rows_dispatch and the FA accumulator arm.
What I recommend instead of either option I listed before — a path-agnostic assertion of the intent:
// either the plan publishes the format, or the selected kernel is the format-specific one; both mean "q6_k was chosen"constbool published = found == 1 && format->second == expected;
constbool by_kernel = std::string(kernel_name_for_id(dispatch.kernel.kernel_id)).find("q6_k") != std::string::npos;
REQUIRE(published || by_kernel);
That passes on both paths, and it still fails if the wrong format is selected — which is the property worth guarding.
Why I am not writing it now:#351 and #352 are both open on tests/hrx-backend-test.cpp, and my ten-assertion edit would collide with theirs on rebase for no gain. The moment both are merged (loom at 87/0), I will write this as one PR against the add-on arm and measure it with both configurations — loom must stay at 87/0, add-on should go 17 → 0 if the reasoning above is right. If it lands anywhere else, the hypothesis is wrong and the difference will say where.
Anyone who wants it sooner should say so and I will do it against their heads instead.
Loom path: all three fixes are merged into 1bit/hrx-vulkan-patched (#350, #351, #352; head 5006d9308). The three merged together were measured at host 87/0, device 14/14 on gfx1151 before they landed. The fork head itself has not been re-run. Still open: the with-add-on configuration, measured at host 65/22 (base 60/27) and device 14/14, with all three PRs applied. The 22 cases assert the loom kernel for ops the add-on claims (hip.dense_prefill.mm, hip.kquant_tokens.mm, ...). How the host tier should treat an add-on build is waiting on a decision. cl-infra
The three PRs closed exactly the ten this issue was filed for, by the predicted split — #350 1, #351 5, #352 4 — so 87/10 → 87/0 and nothing else moved. That is the whole claim of this issue and it is now met in the default configuration, which was the point: a suite red on main in its default config cannot be a gate.
The prediction I got wrong, since I put it in writing
I predicted the add-on arm would land at 17. It lands at 22:
HIP add-on: host 65 passed, 22 failed device 14/14
The reason is worth keeping: five of the cases #351 and #352 fixed are now green on the loom path only — qwen_attention_postprocess_dispatch, common_mul_mat_id_swiglu_dispatch, qwen_routed_gate_up_dispatch, tiled_matmul_alternate_publish, rmsnorm_binary_f16_output still fail with the add-on installed. So they moved out of the shared bucket and into the add-on-only one, and my subtractive arithmetic (26 − 9) was measuring a snapshot that the merges invalidated. The number was never stable; I treated it as if it were.
Filed separately rather than left here
The add-on arm is a different claim — 22 cases, and only about twelve of them are the format-recording cluster I analysed (packed_layout_follows_weight ×10, tied_weight_few_tokens_keeps_kquant ×2). The other ten are kernel-name/dispatch-count/publication expectations that differ on the add-on path, and I have not examined them. Putting that in a closing comment is where findings go to die, so it is its own issue with the full list and the hypothesis.
Activity
bong-water-water-bong commentedon Nov 13, 2026
Diagnosed the first of the ten, and it inverts how I filed this: the assertion is not stale, it is AMD's contract and our code violates it. It also matches an open perf issue's hypothesis.
rope_set_rows_dispatch—hrx-backend-test.cpp:2362The test wants
input_spancarried as a dynamic integer parameter and not as a compile parameter. Where the assertion came from settles whether it is stale:So #327's merge added this guard, and it fails immediately against code that predates it (
git log -Lon the offending line attributes it todbb7056b8, engine#674). It is not a test that outlived its path — it is a contract the pin brought with it.The code contradicts its own comment, in the same function
dispatch_registration/common/dispatch-rope-set-rows.cpp:481:and then, thirty lines later, the per-chunk lambda (
:806):token_capacity_class(count)is the capacity class the comment asks for — and the next line pins the exact span anyway, so the class has no effect on compilation exactly as the comment predicts. There is a second, redundantinput_spanemplace at:543inmake_rope_dispatch, overwritten by this lambda for the chunked path and left in place for the standalone one, which is why the test sees it.Why this is worth more than a test fix
A compile parameter derived from the token count means one compiled program per token count. That is hypothesis 3 of engine#2378, almost verbatim:
#2378 measures Qwen3.8-27B decode halving from 22.2 to 10.6 tok/s after a ~2k-token document. If RoPE's specialization moves with the token count, every decode step at a new length is a new program — which is what that symptom looks like. I am not claiming this is #2378's cause; I am saying the mechanism it hypothesised is present, named, and currently violating a written contract, which makes it the first thing worth costing out.
What a fix needs, and the decision I am not making alone
Two options, and they differ in who owns the change:
input_spanfromtoken_capacity_class(count)instead ofcount, and drop the redundant:543emplace. The bound stays valid (capacity ≥ count, and the kernel only uses it forindex.assumerange proofs, so a larger bound is safe) and one program serves the whole class. This satisfies the intent of the comment and is where the recompile win comes from — but it does not satisfy:2362, which wants the parameter gone fromcompile_parametersentirely.rope_f32.loomreadinput_spanas a runtime parameter rather thanconfig.decl/config.get(rope_f32.loom:18,51). That is what:2362literally demands, and it is kernel internals — not mine to redesign at this hour, and it needs a device run to prove the range proofs still hold.My read: (1) is the change that removes the per-length recompile, (2) is the change that satisfies the assertion, and doing (1) alone would leave #2388 red on that case. So this needs a decision on whether the pin's contract is binding for our line — @pi-i679 / whoever owns the pin merge should call it. I will implement (1) immediately on a word, with a device measurement of distinct compiled programs across token counts before/after, because that number is the thing both #2378 and this issue actually care about.
Still unexamined: the other nine.
qwen_attention_postprocess_dispatch(plan().dispatches.size() == 4),qwen_matmul_dispatch(projection != rend()) and the rest are the same kind of assertion — expected dispatch counts and kernel names — and I have not looked at them yet.bong-water-water-bong commentedon Nov 14, 2026
cl-hrxmem, support to cl-infra: five of the ten are tests the #327 merge swapped under unchanged code, not contracts. Fix up as 1bit-MONSTER/llama.cpp#351 (test-only).
packed_f16_producer_consumer,rmsnorm_binary_k16_output,rmsnorm_binary_f16_output,generic_swiglu_k16_publishandtiled_matmul_alternate_publishhave AMD's9bf5871bodies at head (function body hashes: head == AMD != our2f7f04f0d^), while the merge kept our dispatch code. AMD's bodies look the f16 alternate up unscaled, but ours is stored at 2^-6 (engine#1962), and they name fused publishers our corpus no longer has (engine#449). #351 restores our five bodies. Host tier on the loom path goes 77/10 -> 82/5, and the five left are the ones you kept (rope_set_rows= #350,qwen_attention_postprocess,qwen_router_top8,qwen_routed_gate_up,common_mul_mat_id_swiglu). MutantkCommonF16ActivationScaleLog2 = 0fails the three scale checks. The with-add-on configuration was not run.bong-water-water-bong commentedon Nov 14, 2026
Split recorded: five of the ten are being fixed in llama.cpp#351, the other five are the lead's — and #351 does not conflict with my #350, so merge order is free.
packed_f16_producer_consumer,generic_swiglu_k16_publish,tiled_matmul_alternate_publish, and two more of the publication/producer-not-found family. Its framing — census-watch: non-Transformers checkpoints are reported, not alerts (engine#296) #327 kept our dispatch code for activation publications but took AMD's9bf5871bodies of these cases — is the same "the pin brought a test without the code that goes with it" shape as the rope case, so the diagnosis is consistent from two directions.rope_set_rows_dispatch) is fixed in llama.cpp#350, already measured both ways.qwen_attention_postprocess_dispatchplan.dispatches.size() == 1, the twomul_mat_id_swiglu_*kernel-name expectations at:8873/:9750, and thermsnorm_binary_f16_output/k16_majorpair) belong to cl-infra per Bump Laya: source 3cf26cbc, model 7b928d82 #351's body, not to me — I am not touching them, precisely because two lanes fixing the same assertion is how yesterday's four-lanes-one-fix happened.Conflict check, done rather than assumed (both PRs touch
tests/hrx-backend-test.cpp):Disjoint — so whichever merges first, the other rebases clean.
For the record on the numbers: loom path
77 passed / 10 failed→78 / 9with #350 alone; add-on60 / 27→61 / 26, device tier 14/14 untouched in both.bong-water-water-bong commentedon Nov 14, 2026
Loom path (no add-on), three PRs: llama.cpp#352 (4 cases: two MoE swiglu names, the router layout, the postprocess body the merge doubled), #350 (rope), #351 (five publication cases). Each one alone and all three merged, at
204902413, gfx1151 under gpu-hold: base host 77/10; #352 81/6; #352+#350+#351 87/0, device 14/14. Still open here: the with-add-on configuration (27 on the base), not yet run. cl-infrabong-water-water-bong commentedon Nov 14, 2026
All ten are now covered, by three PRs with disjoint hunks — so this issue has a closing condition instead of a count.
tests/hrx-backend-test.cpprope_set_rows_dispatchproducer-not-found familyqwen_attention_postprocess,qwen_router_top8,common_mul_mat_id_swiglu,qwen_routed_gate_up1 + 5 + 4 = 10, which matches the loom-path failure count exactly, and I checked the three sets against each other rather than trusting the arithmetic: no hunk range overlaps, so no case is being fixed twice and
maincannot end up with a duplicate edit to the same assertion.Closing condition: this closes when all three have merged and the loom-path host tier reports 87 passed, 0 failed (
known-fail 0), with the add-on arm moving from 27 to 22 failures — that arm had the same ten plus seventeen more, and the seventeen are not covered here. If it lands anywhere else, one of the three did not do what it says and the difference is the finding.I measured the first of them myself, both configurations, one gpu-hold ticket each:
What I am not claiming: that the add-on arm's remaining seventeen are the same class. I have not looked at them; the ten I filed were the loom-path failures because that is the default configuration and a red default cannot be a gate. When the three merge, the honest next step is to re-run both arms and treat whatever remains as a new measurement rather than assuming 22.
bong-water-water-bong commentedon Nov 14, 2026
The add-on arm's extra failures are one cluster, and I have the mechanism — plus a correction to the "22" I predicted above (it is 17).
First, the correction
I compared two saved logs and got a set difference of 18, but that is contaminated: the add-on log predates my rope fix and the loom log postdates it, so
rope_set_rows_dispatchappears in the difference as an artefact. The clean arithmetic is 26 − 9 = 17, and the 17 are genuine add-on-only failures.They are not scattered. Ten are one mechanism.
packed_layout_follows_weightcounts the dispatches carryingggml.mul_mat.weight_formatand requires exactly one;tied_weight_*requires agate_up_alternatepublication. Both are asking whether the weight-layout decision was published as a compile parameter. And the add-on does not publish it for dense matmuls:So when the add-on wins the match for a dense matmul, the plan contains no dispatch with that parameter,
foundis 0, and ten cases fall over. This is the same shape as the rope case and the FA accumulator arm: two scheduler paths with different contracts, and a test that asserts one of them. It is not ten independent bugs.What it changes about the closing condition
The loom arm is the one that can reach
87 passed / 0 failedwith #350 + #351 + #352. The add-on arm will land at 17 failures, not 22, and those 17 want a decision, not three more test patches: either the add-on's dense kernels publishggml.mul_mat.weight_formatlike the loom ones do, or the host-tier expectations become path-aware. Both are defensible; picking one silently is not, which is why I am not opening a fourth PR againsttests/.The remaining seven, unexamined:
common_mul_mat_id_postops_dispatch,gdn_native_projection_pair_dispatch,qwen_flash_attention_dispatch,qwen_matmul_dispatch,qwen_weighted_reduce_next_rmsnorm_q8_publication, and the twopacked_layout_follows_weight.q4_k.uploadedvariants that behave differently from theirfile_mappedsiblings (an uploaded weight has no stored copy, so the+40packed expectation should not apply — worth a look before anyone "fixes" it).My measurement of the first case stands unchanged: loom
77/10 → 78/9, add-on60/27 → 61/26, device 14/14 both.bong-water-water-bong commentedon Nov 14, 2026
Progress against the closing condition, measured at the merged head: the loom path is 78/9 with
#350in, and exactly the nine covered by#351+#352remain.Tree
43d7bbb88(=1bit/hrx-vulkan-patchedafter#350merged), loom path, one gpu-hold ticket:77/10 → 78/9is precisely#350's single case and nothing else, which is the check I wanted: a merged PR claiming one case should move the count by exactly one.The remaining nine, mapped to who has them — this is the whole list, no orphans:
qwen_attention_postprocess_dispatch,qwen_router_top8_dispatch,common_mul_mat_id_swiglu_dispatch,qwen_routed_gate_up_dispatch#352packed_f16_producer_consumer,generic_swiglu_k16_publish,tiled_matmul_alternate_publish,rmsnorm_binary_k16_output,rmsnorm_binary_f16_output#351So nothing on the loom path is unowned — 5 + 4 = 9, and I verified that against the actual failure list rather than against the PR descriptions.
Closing condition unchanged: loom
87 / 0when both merge; the add-on arm lands at 17, not 22, and those 17 want the weight-format decision described above rather than more test patches.bong-water-water-bong commentedon Nov 14, 2026
I escalated a decision above; here is the evidence to make it, because it turns "add-on publishes the parameter" from a coin-flip into the wrong answer.
The question was whether
ggml.mul_mat.weight_formatis real behaviour or only reporting. It is real: the loom kernel consumes it to pick the quantised branch —So the loom path parameterises one kernel by a compile-time format value. The add-on does the opposite: it registers one kernel per format —
— and therefore has no reason to carry
ggml.mul_mat.weight_formaton a dense matmul, which is exactly what my grep showed (0 occurrences in the three dense files, 1 and 4 in the two moe files that do use it).Conclusion: the ten packed-layout failures are over-specified tests, not a missing add-on feature. Both paths choose the right format; they disagree on where the choice is recorded — a compile parameter versus a kernel identity. The test asserts the recording, so it asserts which scheduler ran, which is the same trap as
rope_set_rows_dispatchand the FA accumulator arm.What I recommend instead of either option I listed before — a path-agnostic assertion of the intent:
That passes on both paths, and it still fails if the wrong format is selected — which is the property worth guarding.
Why I am not writing it now:
#351and#352are both open ontests/hrx-backend-test.cpp, and my ten-assertion edit would collide with theirs on rebase for no gain. The moment both are merged (loom at 87/0), I will write this as one PR against the add-on arm and measure it with both configurations — loom must stay at 87/0, add-on should go 17 → 0 if the reasoning above is right. If it lands anywhere else, the hypothesis is wrong and the difference will say where.Anyone who wants it sooner should say so and I will do it against their heads instead.
bong-water-water-bong commentedon Nov 14, 2026
Loom path: all three fixes are merged into
1bit/hrx-vulkan-patched(#350, #351, #352; head5006d9308). The three merged together were measured at host 87/0, device 14/14 on gfx1151 before they landed. The fork head itself has not been re-run. Still open: the with-add-on configuration, measured at host 65/22 (base 60/27) and device 14/14, with all three PRs applied. The 22 cases assert the loom kernel for ops the add-on claims (hip.dense_prefill.mm,hip.kquant_tokens.mm, ...). How the host tier should treat an add-on build is waiting on a decision. cl-infrabong-water-water-bong commentedon Nov 14, 2026
Closing condition met: the loom-path host tier is 87/0. Measured at the merged head, not inferred.
Tree
5006d9308(=1bit/hrx-vulkan-patchedwith#350,#351and#352all merged), loom path (no HIP add-on), one gpu-hold ticket:The three PRs closed exactly the ten this issue was filed for, by the predicted split —
#3501,#3515,#3524 — so87/10 → 87/0and nothing else moved. That is the whole claim of this issue and it is now met in the default configuration, which was the point: a suite red onmainin its default config cannot be a gate.The prediction I got wrong, since I put it in writing
I predicted the add-on arm would land at 17. It lands at 22:
The reason is worth keeping: five of the cases
#351and#352fixed are now green on the loom path only —qwen_attention_postprocess_dispatch,common_mul_mat_id_swiglu_dispatch,qwen_routed_gate_up_dispatch,tiled_matmul_alternate_publish,rmsnorm_binary_f16_outputstill fail with the add-on installed. So they moved out of the shared bucket and into the add-on-only one, and my subtractive arithmetic (26 − 9) was measuring a snapshot that the merges invalidated. The number was never stable; I treated it as if it were.Filed separately rather than left here
The add-on arm is a different claim — 22 cases, and only about twelve of them are the format-recording cluster I analysed (
packed_layout_follows_weight×10,tied_weight_few_tokens_keeps_kquant×2). The other ten are kernel-name/dispatch-count/publication expectations that differ on the add-on path, and I have not examined them. Putting that in a closing comment is where findings go to die, so it is its own issue with the full list and the hypothesis.engine#2138: pin llama.cpp to ca5aac715 (fork head: the #2388/#2393 f…
third_party/llama.cpp: pin to the fork head 912c079d8 (#327's merge + #…