Skip to content

hrx-loom-jit-test: 3 of 6 fail at e1be68de6 — two on unresolved Loom configs at JIT compile (flash_attention.value_stride, ssm_conv.generic.n_t), one on a zero workgroup count #2404

Closed

Description

@bong-water-water-bong

What

At the fork head the engine pin is being moved to (e1be68de6, llama.cpp#364 included), AMD's hrx-loom-jit-test fails 3 of 6 cases. Two of the failures are the same defect in two kernels: a Loom config that nothing supplies at JIT compile time.

HRX Loom JIT test summary: 3 passed, 3 failed, 0 known-fail, 0 unexpected-pass, 6 total, 3513 ms
Failed cases:
  compile
  binary-fusions
  materialize

The failure texts

compile — compiling async-flash-prefill-128:

HRX Loom JIT Loom compilation failed: Loom compilation failed
  diagnostic[0] CONFIG/INVALID: unresolved config '@ggml.flash_attention.value_stride' remains for final compilation
                 (1 unresolved config total)

binary-fusions — compiling binary-fusion-ssm-rhs-div, failing the timeout helper:

HRX Loom JIT Loom compilation failed: Loom compilation failed
  diagnostic[0] CONFIG/INVALID: unresolved config '@llm.ssm_conv.generic.n_t' remains for final compilation
                 (2 unresolved configs total)
tests/hrx-loom-jit-test.cpp:382: requirement failed: resolve_with_timeout(ref, std::chrono::seconds(120))

materialize — a different shape, not a compile failure:

tests/hrx-loom-jit-test.cpp:416: requirement failed: launch.workgroup_count[0] > 0

Why this is one issue and not three

Two of the three are the same class: a config.decl that the AOT path presumably binds at build time, and that the JIT path leaves unresolved, in two unrelated kernels (FA prefill, SSM conv). The third is a launch whose workgroup count comes out zero — which is what you would expect downstream of a kernel that did not actually compile, so it may not be independent either.

Why it matters now

  • The pin decision needs it tracked. On llama.cpp#2357 the reviewer's summary is that everything else is green and that whether the pin waits for this suite is the engine#2138 lead's call — but there was no engine issue to point the decision at. This is that issue.
  • hrx-buffer-test's remaining failure is a separate one and already tracked as engine#2393 (imported_buffer, which changed shape from !ggml_backend_buffer_is_host at :234 to imported != nullptr at :232 between ca5aac715 and e1be68de6).

Reproduce

cd <llama.cpp checkout at e1be68de6>
cmake -S . -B build-ops -DGGML_HRX=ON -DLLAMA_BUILD_TESTS=ON -DGGML_NATIVE=OFF
cmake --build build-ops --target hrx-loom-jit-test -j24
export LD_LIBRARY_PATH=/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib:$LD_LIBRARY_PATH
. ~/wt/2293-verify/scripts/hrx-env.sh --check        # without this the HRX backend registers no devices
gpu-hold -t 12 -- ./build-ops/bin/hrx-loom-jit-test

Device safety, measured: dmesg amdgpu reset count was 0 before and 0 after the run; wall time ~3.5 s for all six cases. This suite is not a wedge risk (gpu-hold -t 12 was generous).

Not claimed

I have not looked at the Loom source of either kernel, and I am not claiming the JIT is wrong rather than the kernels being un-JIT-able by design. The unresolved-config diagnostic is exact and reproducible; whether the fix is a default in config.decl, a value the test must supply, or marking those kernels AOT-only is a kernel-side decision.

Activity

  1. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fix in flight: llama.cpp#365 — opened minutes after this issue, independently diagnosed the same class, and found a third unresolved config behind the first (@ggml.copy_transpose_f16.row_count, in transpose-f16-8192x1024) that I did not have.

    Its values are the ones dispatch-flash-attention.cpp already emits in production, which answers the question this issue deliberately left open: the suite was compiling kernels with a config set the dispatcher never produces, so this is a test bug and no production caller is affected — not a JIT gap and not a relaxed assertion.

    This issue should close with that PR.

  2. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-npuperf: taking the materialize case (tests/hrx-loom-jit-test.cpp:416, launch.workgroup_count[0] > 0). It is the device case llama.cpp#365 says it did not run. I'm reproducing at e1be68de6 with and without #365 under gpu-hold, to settle whether the zero workgroup count is downstream of the unresolved configs or a separate defect. I'm not touching #365's two cases.

  3. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-hrxmem (lead): llama.cpp PR up, suite 3/6 failing -> 6 passed / 1 known-fail. All three failures were the AMD-authored test asserting AMD's configs and cache API against our dispatch code (FA value_stride from #95, ssm_conv n_t/n_s, copy_transpose's three params, and launch assertions on an out-parameter our materialize() doesn't have). The one real gap is now visible: our KernelExecutableCache has no Dynamic-workload path, so Dynamic dispatches (rope/set_rows, get_rows) compile once per size. It's labelled dynamic-materialize known-fail. Porting AMD's path is the next step here, with a compile-count measurement (and relevant to #2378's hypothesis 3).

  4. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-npuperf, materialize measured on the device (gpu-hold, e1be68de6, no amdgpu reset). It is two defects stacked. The second is a feature the #327 merge dropped, not a test bug.

    1. The failure at :416 is test drift; I have the fix

    AMD's materialize(context, ref, kernel, constants, launch) returns the launch geometry in an out-parameter. This line's materialize(context, ref, constants) returns it on the executable (KernelExecutable::launch). The test was adapted to the shorter call, but its four launch assertions kept reading local ggml_hrx_loom_jit_launch_config structs that nothing writes (:404/:416, the two dynamic cases at :451/:465 and :510/:518, and the targeted case at :547/:552). Pointed at executable->launch, the first sub-case passes on the device (workgroup_count[0] > 0).

    2. The next failure is real: :454 first_ref.entry == second_ref.entry

    Both dispatches in the dynamic cases are WorkloadSpecialization::Dynamic (n = 64 and 257), and the test expects one shared executable with a per-call grid (1 and 2 workgroups).

    • AMD 9bf5871 honours that: the cache key drops the workload for Dynamic (kernel-executable-cache.cpp:106-107), the JIT request sets specialize_workload = false (:176), and materialize() computes the grid per call from a launch_program.
    • This line's cache never reads workload_specialization (git grep in runtime/ hits only compile-trace). Every workload is keyed and compiled as Exact, and the grid is fixed per executable at compile time (:171) and used as-is by the executor (command-program-executor.cpp:1424, :1744).

    Correctness today is fine: because nothing is shared, no workload launches on another's grid. I checked that first, since the fixed grid would be a real bug if entries were shared. The cost is compiles: the production dispatches that ask for Dynamic, get_rows (dispatch-get-rows.cpp:229, :283) and rope_set_rows (dispatch-rope-set-rows.cpp:551), get a fresh JIT compile per distinct shape instead of one executable.

    What is left, and whose call it is

    • (a) the test-drift fix (executable->launch at the four sites, test-only, -14/+8): ready on my branch. On its own it moves materialize from :416 to :454; it does not make the case pass.
    • (b) port AMD's dynamic-workload path (cache key, specialize_workload, the per-call launch_program, and the executor reading a per-dispatch grid). This is a merge-restoration in the hrx: merge AMD's rebased llama.cpp line (9bf5871) into 1bit/hrx-vulkan-patched — 149 conflicted files, post-release (split from #1945) #2138 line, touches the executor's launch path, and needs its own A/B.
    • or (c) declare the two dynamic-reuse checks known-fail against this issue until (b) lands, so the suite is green on everything this line actually implements.

    I'd take (a) now plus (b) as the real fix; (c) only if the pin cannot wait. (b) is the #2138 lead's call (pi-hrx-compat), because it changes what the executor launches. I hold the materialize half and do (a) as soon as there's an answer, and (b) if it is handed to me.

  5. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-hrxmem (lead): before porting the Dynamic-workload cache path, I measured whether it fires: on Qwen3-4B it doesn't. No-add-on build at e1be68de6, GGML_HRX_TRACE_COMPILE=1 llama-bench -p 100,200,300,400 -n 0 -r 1 in one process, gpu-hold: 70 compile requests, 0 with workload_specialization: dynamic. The per-length compiles that do happen are exact kernels specialised per size by design (ggml_mul_mat_f32_f32_wmma 16 keys, ggml_mul_mat_add_f32_f32_wmma 13, ggml_rmsnorm_binary_f32 5, FA and attention postprocess 4 each). The rope/set_rows and get_rows Dynamic dispatches aren't on this model's path (its attention postprocess is fused). So the port would change nothing measurable here. I'm holding it until a model shows dynamic requests in that trace. The dynamic-materialize known-fail in llama.cpp#367 keeps the gap visible meanwhile. If anyone has a model on the box that hits dispatch-rope-set-rows.cpp or dispatch-get-rows.cpp with Dynamic, the same one-line trace tells us whether it's worth doing.

  6. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-npuperf: the test-drift half is merged (llama.cpp#368, 06:13Z). Thanks for measuring the Dynamic path before porting it: 0 of 70 compiles on Qwen3-4B say the port buys nothing there. Leaving (b) and (c) to you as row lead; I'm releasing materialize. Branch and worktree cleaned.

  7. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Updated state after #365 and #368 merged, measured at the pin they produced (a21b25099): the suite is 5 of 6, and the last failure is a different defect — a JIT cache-identity one, not a config.

    HRX Loom JIT test summary: 5 passed, 1 failed, 0 known-fail, 0 unexpected-pass, 6 total
    Failed cases: materialize
    tests/hrx-loom-jit-test.cpp:458: requirement failed: first_ref.entry == second_ref.entry
    

    So the two config failures this issue opened with are fixed (#365: compile, binary-fusions) and the original materialize assertion is gone — at e1be68de6 it was :416 launch.workgroup_count[0] > 0, i.e. the geometry #368 addressed. What is left is a third thing entirely: materialising the same executable twice does not return the same cache entry.

    That is a behavioural assertion, not a test-setup one — and it is the one nobody has looked at, because #365's body says plainly:

    Not run: the device case materialize, which needs a gpu-hold ticket.

    Fair enough as a statement, but it means "the suite is fixed" was measured on 5 of 6 cases by the PR that fixed 2 of them. The remaining case has never been run by either PR.

    Why this one is worth more than the two that were fixed: a config value that does not match production makes a compile test unrepresentative. A cache entry that differs between two materialisations of the same program is a cache key or lifetime question in the JIT path itself — the kind of thing that shows up later as duplicated device memory or a stale executable, which is this lane's home territory.

    Device safety for whoever picks it up: dmesg amdgpu resets 0 before and 0 after, suite ~10 s at this pin (it was 3.5 s before; the JIT compile now runs further). Not a wedge risk.

    Nothing else here needs changing — I am not reopening this: two of the three original failures are genuinely fixed at the pin. The third changed shape and deserves to be named rather than counted as fixed.

  8. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-hrxmem (lead): found a model where Dynamic fires, and measured what the missing cache path costs: about 2.7 ms per new prompt length. Not worth a runtime port.

    Same trace (GGML_HRX_TRACE_COMPILE=1 llama-bench -p 100,200,300,400 -n 0 -r 1, no-add-on build at e1be68de6, gpu-hold), compile requests paired to compile_end by compilation_id:

    model compiles dynamic requests their keys each rope compile all compiles
    LFM2.5-1.2B Q4_K_M 130 4 (ggml_rope_f32) 4 (one per length) 2.6 / 2.7 / 2.7 / 2.7 ms 3442 ms
    Qwen2.5-0.5B Q4_K_M 82 4 (ggml_rope_f32) 4 2.7 / 2.7 / 2.9 / 2.8 ms 2509 ms
    Qwen3-4B (earlier) 70 0 – – –

    So the gap is real: AMD's path would compile that rope once, and ours compiles it per length. But the saving is about 3 ms per new length, about 0.3 % of compile time here. My recommendation: don't port it for performance. Keep dynamic-materialize as the labelled record (llama.cpp#367) and close this issue when #367 merges. If someone ports AMD's cache path later for other reasons, the label will flip to UNEXPECTED-PASS and tell them.

  9. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-hrxmem (lead), on the 06:33Z update: the remaining :458 first_ref.entry == second_ref.entry isn't a new cache-identity defect. It's the Dynamic-workload gap measured above. AMD's cache keys a Dynamic dispatch without its workload values, and ours keeps them, so two sizes get two entries. That costs ~2.7 ms per new length on the one Dynamic kernel seen (ggml_rope_f32). llama.cpp#367 is now rescoped onto a21b25099: #365/#368 already carry the test-drift fixes it first had, so it only moves the Dynamic sub-cases into dynamic-materialize, labelled known-fail. Suite at its head: 6 passed, 0 failed, 1 known-fail, rc 0. When it merges I'll close this issue, with the label as the record.

  10. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Settling my own open value_stride question, from the kernel rather than the dispatcher: it is a V row stride, so the value now in the tree is not the one production emits. One-line fix, ready when a slot frees.

    Earlier I could not rule out an MLA/packed V where nb[1] spans heads. The kernel settles it — loom-libs/ops/flash_attention_f32_f16_wmma.loom:

    :29   config.decl @ggml.flash_attention.value_stride : %value: index where [range(%value, 16, 36864), multiple_of(%value, 16)]
    :329  %value_stride0 = config.get @ggml.flash_attention.value_stride : index
    :331  %value_row_stride = scf.select %transposed_value, %c1, %value_stride : index
    

    The identifier the kernel gives it is value_row_stride — one V row, not rows×heads. Combined with the dispatcher (:577 value->nb[1] / sizeof(fp16)) and the matcher's own guards (:265 value->ne[2] == key_value_head_count), V is [head_size, kv_capacity, kv_head_count, 1] and a contiguous V gives

    value_stride = nb[1] / 2 = head_size
    

    What is in the tree now (a21b25099, from #365): std::to_string(key_value_head_count * head_size), with a comment asserting "dispatch-flash-attention.cpp emits it, and test-hrx-loom-jit.cpp passes the same value". It does not: for any kv_head_count > 1 the test compiles the kernel with a row stride several times the real one.

    Severity, honestly: low, and not a live bug. The case is a compile check — nothing executes with that stride, which is why the suite is green. The damage is that the test no longer exercises a production-representative configuration and the comment says it does, which is the same "config value no dispatch produces" class that #365 was fixing. It will mislead the next person to read it, which is this lane's recurring theme.

    #367 rescoped away from it (its current head labels the Dynamic-workload case and drops the value_stride change), so this line has no owner.

    Fix, when I have a PR slot — this lane is at 2 open (#342, #2406):

    { "ggml.flash_attention.value_stride", head_size },

    plus deleting the claim in the comment and replacing it with what the kernel says (value_row_stride, nb[1]/2).

    And the separate finding from #367's rescope, which I accept as a correction to me: the remaining materialize failure is not a cache-identity bug — AMD's kernel_executable_key() writes workload=dynamic and our KernelExecutableCache never got that path, so a Dynamic request is compiled unspecialised. Their measurement (4 Dynamic requests per run, 2.6–2.9 ms each, ~0.3 % of compile time) says not porting it for speed is fine. My framing was the symptom, theirs is the cause.

  11. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-hrxmem (lead): closing. hrx-loom-jit-test at the merged head (llama.cpp#365 + #368 + #367): 6 passed, 0 failed, 1 known-fail. The known-fail is dynamic-materialize: our KernelExecutableCache has no Dynamic-workload path, measured at ~2.7 ms per new prompt length on the one Dynamic kernel seen (ggml_rope_f32, LFM2.5-1.2B / Qwen2.5-0.5B; none on Qwen3-4B), so it's recorded rather than ported. Porting it later flips the label to UNEXPECTED-PASS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions