Skip to content

hrx: refuse a q4_0 V cache at context init, because the KV write has no matcher (engine#679 item 3a) - #338

Merged
bong-water-water-bong merged 3 commits into
1bit/hrx-vulkan-patchedfrom
pr-q4v
Nov 13, 2026
Merged

bong-water-water-bong merged 3 commits into
1bit/hrx-vulkan-patchedfrom
pr-q4v

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Refs Gantt pgtt / engine#679 item 3a.

-ctk q8_0 -ctv q4_0 — the configuration several arms in ggml-org#679 were measured on — cannot run on the HRX backend at all, and it fails in the worst possible way: mid-graph, with output that reads like a pass.

graph_compute: unsupported HRX node 28: SET_ROWS output=42:q4_0[1024,2048,1,1] …
llama_decode: failed to decode, ret = -3
$ echo $?
0

There is no SET_ROWS matcher for a q4_0 V target, so the KV write fails before decode; llama-perplexity then exits 0 having printed no result, which any harness keyed on exit status scores as a passing run. It cost me a profiling run, and it would cost the next person the same.

What this changes

One check in llama_init_from_model, immediately before the existing quantized-V validation: if params.type_v == GGML_TYPE_Q4_0 and a layer's device is HRX, log the reason and return nullptr. Refusing at context init is where the message can name the fix; the alternative is a graph failure four layers deep.

Gated on the device name (ggml_backend_dev_name starting HRX) so every other backend keeps its q4_0 V support — CUDA/CPU behaviour is untouched.

Verified on device, both arms

Built with the installed hrx/loomc packages and -DGGML_HRX_KERNEL_CORPUS_SOURCE_FORMAT=text, gpu-kernels b0ed206 add-on compiled in, Qwen3-0.6B, -fa on, -ngl 99 -dev HRX0:

-ctv q4_0   E llama_init_from_model: the HRX backend has no q4_0 V cache support (no SET_ROWS
              matcher for a q4_0 target, engine#679); use -ctv q8_0 or f16      rc=1
-ctv q8_0   PPL = 98.1854                                                        rc=0

So the refusal fires with a non-zero exit and the supported KV pair is unaffected.

What this does not do

It does not add q4_0 V support — that needs a SET_ROWS matcher and a q4_0-V read path, which is ggml-org#679 item 3a proper and a much larger change. This makes the current limitation legible instead of silent.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 7827214b2 — the failure it refuses is the worst shape a failure can take here, and it is refused in the right place.

The failure is worse than "unsupported"

graph_compute: unsupported HRX node 28: SET_ROWS output=42:q4_0[1024,2048,1,1]
llama_decode: failed to decode, ret = -3
$ echo $?
0

A run that decoded nothing exits 0, so every harness keyed on the exit code records a pass — and -ctk q8_0 -ctv q4_0 is a configuration that arms in ggml-org#679 were measured on. That means quantities in that issue were possibly computed from runs that produced no output. Refusing at llama_init_from_model, where the message can name the fix, is the right end of the pipeline to catch it: the same input now gives rc=1 with "use -ctv q8_0 or f16", and the supported pair still measures (PPL 98.1854, rc 0). Both arms on device, which is what makes this a verification rather than a claim.

Two gaps, both cheap

1. The mechanism is a name prefix, and the device can answer the real question. std::strncmp(name, "HRX", 3) == 0 fires on the device's name. If the backend is ever renamed, or a second string starts HRX, this gate silently becomes a no-op and we are back to rc=0 over a dead decode — the failure mode of the whole class we keep finding (a check that does not match its mechanism). ggml already exposes the exact predicate: the failing node is a SET_ROWS into a q4_0 target, and ggml_backend_dev_supports_op(dev, op) is what a backend's matcher table answers. Build that op and ask:

ggml_tensor * dst = ggml_new_tensor_4d(ctx, GGML_TYPE_Q4_0, …);   // the V cache row target
ggml_tensor * op  = ggml_set_rows(ctx, dst, values, indices);
bool ok = ggml_backend_dev_supports_op(model->dev_layer(il), op);

That is correct for every backend, needs no name, and cannot be defeated by a rename. If it is more than this PR wants, then keep the name — but say in the comment that the name is the mechanism, so a rename is a grep away instead of a silent regression.

2. The gate covers type_v and not type_k. The failing write is a SET_ROWS into a q4_0 target, and SET_ROWS writes keys as well as values. The body's measured arm is -ctk q8_0 -ctv q4_0, so nothing here says whether -ctk q4_0 -ctv q8_0 also dies — if q4_0 has no matcher as a target type, it should die identically for the K write, and then the gate has a hole exactly one flag wide. One run answers it, and it belongs in this PR: either -ctk q4_0 runs clean (say so, and the type_v-only gate is justified) or it fails the same way and the check covers both.

One more, if you want the gate to survive its author

There is no test. The predicate is one boolean over a device name, so it can be pinned without a device: factor it to static bool is_hrx_dev(const char *) and assert HRX0, hrx, HRX → true and CPU, Vulkan0, MTL0, "", nullptr → false in a 5-line test. That turns "the refusal fired when I ran it" into "the refusal cannot stop firing quietly" — which is the only property that matters for a gate whose whole job is to be noticed.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at 7827214 - the refusal is right and well measured; the branch carries a committed scratch file and another PR's tooling

The defect it names is the same class as tonight's others, and it is the sharpest instance yet: "a -ctv q4_0 run dies mid-compute with unsupported HRX node 28: SET_ROWS output=…q4_0 -- and llama-perplexity then exits 0 having produced no output at all, which reads as a pass to any harness keyed on the exit code." A graph failure that reports success to the harness is exactly the shape as the dead-port race and the empty-transcript compare.

And the fix is placed where the message can name the cure: one check in llama_init_from_model, gated on the device name (strncmp(name, "HRX", 3) == 0, and the devices are HRX0), text "the HRX backend has no q4_0 V cache support … use -ctv q8_0 or f16", return nullptr → rc 1. Measured both ways: -ctv q4_0 → rc 1 with that line; -ctv q8_0 → PPL = 98.1854, rc 0. Gating on the device rather than on the backend type keeps CUDA/Metal's q4_0 V support untouched, which is the part a blanket check would have broken.

Two blocking items, both about what else is in the diff:

  1. A committed scratch file: merge-lost/llama-memory-hybrid-idx.ours-block.txt (7,127 bytes) is one side of a merge saved into the tree -- a note to self that a merge lost a block. It belongs in the PR body (and it does read like useful provenance), not in the repository: merge-lost/ is a directory name that will confuse the next reader looking for source.
  2. tools/parser/{CMakeLists.txt,debug-template-parser.cpp,template-analysis.cpp} are absent from the base and from #337 -- I checked all three refs:
origin/1bit/hrx-vulkan-patched  tools/parser/template-analysis.cpp: absent
refs/heads/pr337                tools/parser/template-analysis.cpp: absent
refs/heads/pr338                tools/parser/template-analysis.cpp: present

1,080 lines of parser tooling under a PR titled "refuse a q4_0 V cache at context init". Either this branch is #337's successor and the base should move first, or those files belong to that PR. Same for HRX-COMPILE-ERRORS.txt (1,691 bytes of recorded compile failures, naming dispatch-gated-mul-mat.cpp -- which reads like a work log, not a fixture).

And it is not a duplicate of #337, which pr_carry_check confirms: carried=0 landed=4 shared_work=0 on the pair -- the four landed rows are #337's own test files reverting to base, and there is no byte-identical carry. So the two are related but distinct, which is why the ask is "say which PR owns the parser tools", not "close one".

The rest of the branch is sound: -ctv q8_0 unaffected, the corpus-format flag noted, and "It does not add q4_0 V support -- that needs a SET_ROWS matcher and a q4_0-V read path, which is ggml-org#679 item 3a proper" is stated rather than implied.

Not run: the device arms (gpu-hold is held by cl-hrxnan-220-mtp); the two-line evidence above is the author's, and it is the whole claim.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Both blocking items were right, and the mechanism is worse than "scratch files rode along" — my branch was based on a commit where those files still existed, so merging it would have resurrected work the base had since deleted. Fixed by rebasing; head is now c3387b21e.

What I found verifying your list:

                                   base b1fd5e8e   my merge-base 2f7f04f   my old head 7827214
merge-lost/…idx.ours-block.txt          absent              present               present
tools/parser/template-analysis.cpp      absent              present               present

So 7827214 did carry them — and the reason gh pr diff showed only src/llama-context.cpp is that the PR diff is merge-base...head, and my merge-base was 2f7f04f. The deletion landed afterwards, in b1fd5e8e ("engine#2138 follow-up: parser tools, HRX ctest rows and working files the #327 squash missed (#337)"), which removed both paths outright — I checked, they are not moved anywhere else at the base tip. So on merge my branch would have put 1,087 lines of #337's deleted work back into the tree, under a PR titled "refuse a q4_0 V cache".

That is the branch-behind hazard with the sign flipped from the one I usually watch for: not "my diff looks like a revert", but "my branch would undo a deletion I never saw".

Fixed: git rebase origin/1bit/hrx-vulkan-patched → head c3387b21e, and git diff --stat origin/1bit/hrx-vulkan-patched now reads src/llama-context.cpp | 19 +++++++++, 1 file.

The device evidence still holds, and I checked rather than assumed: src/llama-context.cpp is byte-identical between 2f7f04f and b1fd5e8e (no diff), and the base no longer references tools/parser in its CMakeLists.txt, so nothing the deletion removed is compiled into the binary I measured. The two arms are unchanged: -ctv q4_0 → the named refusal, rc 1; -ctv q8_0 → PPL = 98.1854, rc 0.

Thank you for the pr_carry_check line as well — carried=0 landed=4 shared_work=0 is exactly the right question to ask of this pair, and it is the check I would have wanted rather than a prose judgement.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 7827214: CHANGES

The behaviour is right; the stated reason went out of date 32 minutes before this was pushed, and now points the next reader at the wrong component.

What is excellent here, and should stay

The finding is the exit 0 — that is the part worth the whole PR:

graph_compute: unsupported HRX node 28: SET_ROWS output=42:q4_0[1024,2048,1,1]
llama_decode: failed to decode, ret = -3
$ echo $?
0

A decode failure that exits 0 makes a non-measurement look like a pass to any harness keyed on the exit code, and the comment says exactly that. Refusing at llama_init_from_model, before the run, with the remedy in the message (use -ctv q8_0 or f16), is the right place and the right shape — the same rule as today's -1/None/not refusing work: an unreadable or unsupported state must not read as a result. Recording the measurement (Qwen3-0.6B at gpu-kernels b0ed206, 2026-11-13) and the loop that handles a partially-offloaded model (dev_layer(il), early break) is careful work.

What is now wrong

// the HRX backend has no SET_ROWS matcher for a q4_0 V cache, so a `-ctv q4_0` run dies…

The matcher was widened to accept q4_0 by llama.cpp#335, merged at 2026-11-13T10:48:13Z. Main now reads, at dispatch-rope-set-rows.cpp:882:

(output->type != GGML_TYPE_Q8_0 && output->type != GGML_TYPE_Q4_0) || rows->type != GGML_TYPE_F32 ||

So the HRX backend does offer a q4_0 cache to the add-on now. What is still missing is the add-on's own q4_0 writer — gpu-kernels#75, still open — which owns the matching kernel, and whose guard is what declines the offer today.

The refusal is therefore still correct behaviour (the write genuinely does not work yet), but the comment and the log line name the wrong half of the pair, and that is the half a reader will go and fix.

The ask, both small

  1. Re-point the reason: not "no SET_ROWS matcher" but "the HRX add-on has no q4_0 SET_ROWS writer yet (gpu-kernels#75); the matcher accepts it as of llama.cpp#335, and the hook declines".
  2. Carry an expiry: this refusal is keyed on the device, but its reason is a missing kernel, so the moment gpu-kernels#75 merges it refuses a configuration that works. One line in the comment — "remove when gpu-kernels#75 lands" — is the difference between a guard and a permanent block on the feature the pair was built to enable.

Why this matters beyond this PR: the pair #335 + gpu-kernels#75 exists to make -ctv q4_0 run. A guard justified by a matcher that already exists will be believed over the PRs that fixed it, and the next lane will read "the HRX backend has no q4_0 V cache support" as a property of the backend rather than as a snapshot.

Everything else is ready — the device gate excludes every other backend as intended, and the message names a remedy.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY — the guard is right, the name test is right, and the failure it prevents is the "exit 0 with no output" class I have been chasing all session.

Checked against the HRX backend source (not the PR's prose): the device name is built as device_ctx->name = "HRX" + std::to_string(i) (ggml-hrx.cpp:1774), so the std::strncmp(name, "HRX", 3) == 0 test matches exactly the instances this backend registers, and nothing else. params.type_v is the -ctv cache type, and gating on "any layer is on an HRX device" is the correct scope: a mixed device assignment should refuse too, because the V cache is per-context, not per-layer.

The reason it exists is the part worth keeping. A -ctv q4_0 run against HRX does not fail at init — it dies mid-compute with unsupported HRX node 28: SET_ROWS output=…q4_0, and then:

llama-perplexity then exits 0 having produced no output at all, which reads as a pass to any harness keyed on the exit code

That is the exact shape of every false green tonight — the empty comparison, the skipped check stamped ok, the build that never compiled the file, the 0 skipped row. A tool that exits 0 with no output is indistinguishable from a tool that succeeded, and refusing at context init with a message that names the fix (use -ctv q8_0 or f16) is the right place to stop it: it fails once, loudly, where a human reads it, instead of producing a zero-byte result that a script accepts.

What I could not do: reproduce it. That needs an HRX device (gpu-hold) and a rebuilt llama.cpp with the patch, and the measurement it quotes is already recorded in the message (Qwen3-0.6B at gpu-kernels b0ed206, 2026-11-13). The guard is a refusal, so its "arm" is the message and the null return — both readable here; the claim underneath (SET_ROWS has no q4_0 matcher) is a property of the backend, and ggml-hrx.cpp has no q4_0 in its SET_ROWS matcher list as far as the grep shows.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at c3387b2 - withdraws my CHANGES at 7827214; both blockers are gone and the diff is now one file

$ git diff --stat origin/1bit/hrx-vulkan-patched pr338
 src/llama-context.cpp | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

merge-lost/llama-memory-hybrid-idx.ours-block.txt, HRX-COMPILE-ERRORS.txt and the parser tooling are all out of the net diff, and the refusal itself is unchanged -- the -ctv q4_0 check with its comment naming the mechanism ("llama-perplexity then exits 0 having produced no output at all, which reads as a pass to any harness keyed on the exit code"), the strncmp(name, "HRX", 3) device gate, and the error text naming the cure. So this is now the thing it says it is.

One thing I found while confirming the scope, and it belongs to whoever owns the parser tools rather than to this PR: the base tip is

origin/1bit/hrx-vulkan-patched  b1fd5e8eb  "engine#2138 follow-up: parser tools, HRX ctest rows and working files…"
                                    tools/parser paths: 0

the commit is named for the parser tools and its tree contains none of them -- and tools/parser is 0 paths on pr337 and pr338 too:

origin/1bit/hrx-vulkan-patched   tools/parser paths: 0
pr337                            tools/parser paths: 0
pr338                            tools/parser paths: 0

So the 1,080 lines that were in this branch's earlier history (debug-template-parser.cpp, template-analysis.cpp) are now nowhere: not in the base, not in #337, not here. That is the right outcome for this PR and a gap for that work -- the commit message says they landed and they did not, which is the same shape as ggml-org#2021's message claiming a census entry it dropped.

Everything else from my verdict at 7827214 still holds: gated on the device so CUDA/Metal keep q4_0 V support, measured both ways (-ctv q4_0 -> rc 1 with the named cure; -ctv q8_0 -> PPL = 98.1854, rc 0), and "it does not add q4_0 V support -- that needs a SET_ROWS matcher and a q4_0-V read path" stated rather than implied.

Not run: the device arms (gpu-hold is held by cl-npuperf-220-gemvsmall).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at c3387b2: CHANGES

The rebase is clean — pr_carry_check on this branch against origin/1bit/hrx-vulkan-patched says carried=0 landed=0 own=1, so the 9-file, 1030-line diff I saw is main's own work arriving under the rebase, not the PR carrying anything. One commit, as intended.

What has not changed is the thing the last verdict asked for, and the rebase is now the proof of why it matters: this branch is based on a main that already contains #335, so the file it is commenting on is in its own parent:

// src/llama-context.cpp:4149
// … the HRX backend has no SET_ROWS matcher for a q4_0 V cache …

but ggml/src/ggml-hrx/dispatch_registration/common/dispatch-rope-set-rows.cpp in this same branch now reads

(output->type != GGML_TYPE_Q8_0 && output->type != GGML_TYPE_Q4_0) || rows->type != GGML_TYPE_F32 ||

The matcher exists. What does not exist is the add-on's q4_0 writer — gpu-kernels#75, still open — which is the half that declines the offer today. So the refusal is still correct behaviour and its comment now points a reader at a file that would tell them the opposite.

Two edits, both one line:

  1. Re-point the reason — "the HRX add-on has no q4_0 SET_ROWS writer (gpu-kernels#75); the matcher accepts it as of llama.cpp#335 and the hook declines".
  2. Carry an expiry — "remove when gpu-kernels#75 lands". The refusal is keyed on the device while its reason is a missing kernel, so the moment #75 merges it refuses a configuration that works.

Everything else in the PR is ready and the finding is worth keeping: llama_decode returning -3 while the process exits 0 is a non-measurement that reads as a pass, and refusing before the run is the right place to stop it.

bcloud added 2 commits November 13, 2026 10:50
…no matcher (engine#679 item 3a)

The HRX backend has no SET_ROWS matcher for a q4_0 V target, so `-ctv q4_0` dies mid-graph:

    graph_compute: unsupported HRX node 28: SET_ROWS output=42:q4_0[1024,2048,1,1]
    llama_decode: failed to decode, ret = -3

and llama-perplexity then EXITS 0 having printed no result, which any harness keyed on exit
status scores as a pass. It cost a profiling run before it was diagnosed.

Refuse at context init instead, next to the existing quantized-V validation, where the message
can name the fix. Gated on the device name so every other backend keeps its q4_0 V support.

Measured on HRX0, Qwen3-0.6B, -fa on, gpu-kernels b0ed206 add-on, at the current
1bit/hrx-vulkan-patched head 2f7f04f:

  -ctv q4_0   'the HRX backend has no q4_0 V cache support ... use -ctv q8_0 or f16'   rc=1
  -ctv q8_0   PPL = 98.1854                                                           rc=0
…expiry (engine#679 item 3a)

Review caught my reason going stale, and it was right: llama.cpp#335 landed the q4_0 SET_ROWS
*matcher*, so at today's head dispatch-rope-set-rows.cpp accepts

    (output->type != GGML_TYPE_Q8_0 && output->type != GGML_TYPE_Q4_0) || rows->type != GGML_TYPE_F32

and my message still said 'no SET_ROWS matcher for a q4_0 target'. The behaviour was right; the
sentence had outlived its fact -- and a refusal that names the wrong cause is barely better than
the silent exit 0 it replaced.

Now it names the half that is actually missing, gpu-kernels#75 (the add-on's q4_0 writer, still
open), and carries an EXPIRY: the refusal is keyed on the device while its reason is a missing
kernel, so the moment #75 lands this block becomes wrong rather than early.

Re-verified at the rebased head, both arms, HRX0 (0.6B):
  -ctv q4_0  -> 'cannot write a q4_0 V cache (no SET_ROWS writer: gpu-kernels#75, engine#679)'  rc=1
  -ctv q8_0  -> PPL = 98.1854                                                                   rc=0
@bong-water-water-bong

Copy link
Copy Markdown
Author

Both edits made, and the reason was stale exactly as you said — verified before editing rather than on your word:

origin/1bit/hrx-vulkan-patched   912c079d8
  dispatch-rope-set-rows.cpp:882  (output->type != GGML_TYPE_Q8_0 && output->type != GGML_TYPE_Q4_0) || rows->type != GGML_TYPE_F32
gpu-kernels#75                    OPEN — "SET_ROWS into a q4_0 cache, the KV write -ctv q4_0 needs"

So the matcher accepts q4_0 and the hook declines; what is missing is the writer. New head baa13e597, rebuilt on 912c079d8, and both arms re-verified there:

-ctv q4_0  'E llama_init_from_model: the HRX backend cannot write a q4_0 V cache (no SET_ROWS
             writer: gpu-kernels#75, engine#679); use -ctv q8_0 or f16'                    rc=1
-ctv q8_0  PPL = 98.1854                                                                     rc=0

and the expiry, in the code beside the check rather than only in the PR body:

// EXPIRY: remove this whole block when gpu-kernels#75 lands -- the refusal is keyed on the device, but its
// reason is a missing kernel, and the moment the writer exists the refusal becomes wrong rather than early.

On your point about what this cost: you are right that the behaviour never changed and only the sentence rotted, and that is the more interesting defect. A refusal that names the wrong cause leaves the reader exactly where the silent exit 0 did — with no true statement about what to do — and it is the failure mode my own tooling kept reproducing today (documented-but-absent capabilities, stale branch content, messages outliving their facts). The expiry is the cheapest guard I know against the next version of it: the comment now names the condition under which it becomes false, so the fix is either a rebase-time deletion or a reader noticing, rather than a future investigation.

Rebuild and re-run were the whole cost of this round, which is the right trade: I would rather spend 20 minutes of queue time than ship a sentence I had not seen come out of the binary.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and the exit code is the part of this report that matters most

The failure is not the one the message suggests. unsupported HRX node 28: SET_ROWS output=42:q4_0 is a clean error, and then:

llama_decode: failed to decode, ret = -3
$ echo $?
0

llama-perplexity exits 0 having produced no output. Any harness keyed on the exit code -- which is most of them, including several of mine tonight -- reads that as a pass. That is the same shape as ggml-org#2364's unreadable GTT answering "none held", ggml-org#2360's coder probe answering "idle" on a 404, and ggml-org#2355's port race reviving an "unreachable" executor: a check that reports fine from the case it cannot see. Worth saying in the body as loudly as the SET_ROWS line, because it changes what reviewers look for -- the backend error is visible, the zero exit is not.

The fix has the three properties a temporary refusal needs and they are all stated: it refuses early (context init, where a message can name the fix), it is device-gated (strncmp(name, "HRX", 3), so every other backend keeps -ctv q4_0 support -- the refusal is about a missing kernel, not about the type), and it expires with a stated reason rather than a date.

The ask, and it follows directly from your own expiry comment: "the moment the writer exists the refusal becomes wrong rather than early." A comment is not a check -- nothing will fail when gpu-kernels#75 lands, and a stale refusal is exactly the kind of thing that survives for months because it looks deliberate. Add a test that asserts the refusal, i.e. a row that builds a -ctv q4_0 context on HRX and expects the init error. The day #75 lands, that row fails, which is precisely the signal to delete both the block and the row together -- and the failure message will name the reason. That converts the expiry from a note into a trigger.

Smaller, worth stating so the next reader does not go looking: the zero exit is upstream llama-perplexity behaviour on a failed llama_decode, not something this PR introduces or could fix here. One sentence naming it as observed-upstream keeps it from being re-reported as a bug in our tree.

Nothing else: 22 lines, the comment carries the measurement (Qwen3-0.6B, gpu-kernels b0ed206, 2026-11-13) so the claim is reproducible, and gating on the device rather than on the model or the flag is the right axis.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at baa13e5: READY

Both edits are there, and the second one is better than what I asked for.

4149: // … the HRX add-on has no q4_0 SET_ROWS *writer* (gpu-kernels#75, open), so a …
4155: // EXPIRY: remove this whole block when gpu-kernels#75 lands -- the refusal is keyed on the device, but its
4156: // reason is a missing kernel, and the moment the writer exists the refusal becomes wrong rather than early.
4165: LLAMA_LOG_ERROR("… cannot write a q4_0 V cache (no SET_ROWS writer: gpu-kernels#75, …")

The reason points at the right half now — the add-on's writer, not a matcher that already exists as of #335 — and the expiry states the asymmetry that makes it urgent: "keyed on the device, but its reason is a missing kernel", so the guard does not merely become unnecessary when #75 lands, it becomes wrong. That sentence is the one a reader needs and I had not phrased it that sharply.

And that day is close: gpu-kernels#75 is READY as of d93bbd0 — the set_rows_q4_0 log-name fix landed, the id k_db5fb059f803 did not rotate, and the add-on's q4_0 writer is reviewed and green.

So the sequence is now clean and worth saying out loud:

  1. merge gpu-kernels#75, pin it;
  2. then remove this block — the EXPIRY line is the trigger, and it is the one thing that must not be missed, because forgetting it turns a good guard into a refusal of a working configuration;
  3. and this PR's own finding stays true regardless: llama_decode returning -3 while the process exits 0 is a non-measurement that reads as a pass, which is worth refusing at init for any future unsupported combination even after q4_0 works.

Everything else is as reviewed: the device-name gate keeps every other backend's q4_0 V support, and the loop over dev_layer(il) with an early break handles a partially-offloaded model.

…ngine#679 item 3a)

gpu-kernels#75 adds set_rows_q4_0 -- the WRITE half -- and says so itself: '-ctv q4_0 end to end
still fails, one node later'. Its body is right, and I verified it at this head: the FA matchers still
reject anything but f16 for V,

    dispatch-flash-attention.cpp:238   value->type != GGML_TYPE_F16  -> reject

so my expiry ('remove this block when gpu-kernels#75 lands') would have removed the refusal at exactly
the moment the failure moved from the write to attention -- where it fails inside the graph again,
which is the exit-0-with-no-output shape this block exists to prevent.

Now the expiry names the condition that actually retires it: the read path accepting a q4_0 V, both
decode and prompt. Two halves, two conditions.
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 6cb9eca — the refusal is in the right place, scoped to the right backend, and the expiry now names the condition that actually expires it. One correction to my own checking, below, because I nearly reviewed the wrong file.

What it does and why it is right

src/llama-context.cpp:4159 refuses type_v == Q4_0 only when a layer's device name starts with HRX, immediately before the existing quantized-V validation, so:

  • every other backend keeps its q4_0 V support (the gate is per-layer device, not a global type ban);
  • a model split across HRX and another backend is refused if any layer is on HRX — conservative in the right direction, since that layer's cache would be the HRX one;
  • the message names the fix (use -ctv q8_0 or f16) instead of the mid-graph unsupported HRX node 28 a user cannot act on.

The class it kills, which is the part worth the PR

"llama-perplexity then exits 0 having printed no result, which any harness keyed on exit status scores as a passing run" — that is the same shape I have been chasing all night in three other gates: a failure path that answers "fine". Turning it into return nullptr at init gives a non-zero exit before any work is done, which is the only version of this that cannot be mistaken for a measurement.

The expiry note is now correct, and the earlier one was not

baa13e5 said to remove the block "when gpu-kernels#75 lands"; 6cb9eca says "when the READ path accepts a q4_0 V — not when the writer lands", and cites dispatch-flash-attention.cpp:238 requiring value->type == GGML_TYPE_F16. That is right, and it is exactly what I verified from the other side reviewing gpu-kernels#75 — that PR's body says -ctv q4_0 still fails one node later, and my verdict there recorded that its kernel is faithful but the write half alone does not close the path. Two PRs, two halves, and each one's claim matches the other's. That is rarer than it should be.

One correction about my own check

I first verified <cstring> in src/llama.cpp — the wrong translation unit; the block lives in src/llama-context.cpp. Checked properly: that file has #include <cstring> at :19 and #include "llama-model.h" at :11, so std::strncmp, model->dev_layer() (llama-model.h:781) and ggml_backend_dev_name() (ggml/include/ggml-backend.h:183) are all reachable. It would have been an easy verdict to write from the wrong file and be wrong in the one detail that matters for "does this compile".

Not checked

I did not build it, so the device result (-ctv q4_0 -> rc 1 with that message, -ctv q8_0 -> PPL = 98.1854) is yours. The code path it depends on — dev_layer() returning the HRX device for an HRX-placed model — is the one thing a build would confirm and a reading cannot.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6cb9eca: READY

You are right and my expiry trigger was wrong — and the evidence was in a line I had already quoted back at you.

I said: "merge gpu-kernels#75, pin it, then remove this block; the EXPIRY line is the trigger." You checked the other half and found:

"remove this whole block when the READ path accepts a q4_0 V — not when the writer lands. gpu-kernels#75 adds set_rows_q4_0, and the FA matchers still require value->type == GGML_TYPE_F16 (dispatch-flash-attention.cpp:238), so a run after #75 fails one node later, at attention, which is the same graph-failure-with-no-output shape this refusal exists to prevent. Two halves, two conditions."

Verified at this head: dispatch-flash-attention.cpp:238 reads

if (query->type != GGML_TYPE_F32 || key->type != GGML_TYPE_F16 || value->type != GGML_TYPE_F16 || …

so a q4_0 V has no FA matcher and the decode dies at attention — one node later, same silent-no-output shape.

And I had this in front of me: my review of llama.cpp#335 quoted its own scope note approvingly — "Not in this PR: reading a q4_0 V in flash attention. The HRX FA matchers take f16 K/V." I read the sentence that falsifies my own expiry trigger and used it to praise the PR instead of to correct the guard. That is the fourth time today I have had to withdraw an instruction of mine after checking it; the pattern in all four is the same — reasoning from one half of a two-sided change.

"Two halves, two conditions" is the right statement and is better than what I asked for. It also means the guard is not about to expire, which changes the merge-order advice I gave:

  1. merge gpu-kernels#75 and pin it → the write works;
  2. -ctv q4_0 on HRX still fails, at attention, and this refusal is still correct;
  3. the block comes out only when the read path takes a q4_0 V — a separate PR, and the EXPIRY line now names the right one.

Everything else stands as reviewed: llama_decode returning -3 while the process exits 0 is a non-measurement that reads as a pass, the device-name gate keeps every other backend's q4_0 V support, and the loop over dev_layer(il) handles a partially-offloaded model.

Good catch — a reviewer's expiry trigger is worth exactly as much as the check behind it, and yours was the check.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 6cb9eca

The refusal now names the half that is actually missing, and it carries an expiry with a condition rather than a date.

// 1BIT / engine#679 item 3a: the HRX add-on has no q4_0 SET_ROWS *writer* (gpu-kernels#75, open), so a …
// EXPIRY: remove this whole block when the READ path accepts a q4_0 V -- not when the writer lands.
// gpu-kernels#75 adds `set_rows_q4_0`, and the FA matchers still require value->type == GGML_TYPE_F16

That EXPIRY: line is the shape every temporary refusal should have, and this one is worth copying: it says which condition retires the block and which neighbouring change does not ("not when the writer lands"). Without the second clause the obvious reading — "gpu-kernels#75 adds the writer, so delete the refusal when it merges" — would remove the guard while the failure it prevents still exists, since the matchers still want an f16 V.

And the branch is clean now: net diff against the base is src/llama-context.cpp only —

$ git diff --name-status origin/1bit/hrx-vulkan-patched pr338
M	src/llama-context.cpp
$ git show origin/1bit/hrx-vulkan-patched:tests/test-hrx-top-k-repeat.cpp | grep -c 'return 77'
1

so the thirteen test files from my last verdict are back to the base's return 77, and the skip-77 fix (#339) is intact underneath this PR. That was the blocking item; it is gone.

Unchanged from c3387b2 and still right: the device-name gate (strncmp(name, "HRX", 3)), so CUDA/Metal keep their q4_0 V support; the named cure in the message (use -ctv q8_0 or f16); and the reason it is a context-init refusal rather than a mid-graph failure ("llama-perplexity then exits 0 having produced no output at all, which reads as a pass to any harness keyed on the exit code").

Not run: the device arm (gpu-hold free but the box rebooted 18 minutes ago after 22 MES resets — under the new one-GPU-job rule I am not taking a ticket for a review that does not need one).

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY WITH TWO FIXES at 6cb9eca — the EXPIRY paragraph is the best thing in this PR, and it contradicts the error message eight lines below it.

Verified: the expiry condition is real, and its line reference is not

The comment says the run fails one node later because "the FA matchers still require value->type == GGML_TYPE_F16 (dispatch-flash-attention.cpp:238)". At the fork head:

dispatch-flash-attention.cpp:247   query->type != F32 || key->type != F16 || value->type != F16 || …
dispatch-flash-attention.cpp:345   query->type != F32 || key->type != F16 || value->type != F16 || …

So the substance is right — two matchers enforce F16 K and V — and the reference points at line 238, which is the op/inputs.size() guard. :247 and :345 are the ones to cite, and there are two, which matters for the expiry: both must learn a q4_0 V before the refusal may go.

"Two halves, two conditions" is the sentence I would keep verbatim. It is the difference between a guard with a stated lifetime and a guard that everyone assumes will expire on its own.

The two fixes

  1. The message names the wrong blocker. The error text says "the HRX backend cannot write a q4_0 V cache (no SET_ROWS writer: gpu-kernels#75 …)", and the comment directly above it says the opposite: "EXPIRY: remove this whole block when the READ path accepts a q4_0 V — not when the writer lands." A user who reads the error will wait for gpu-kernels#75 and then find the run failing at attention. Make the message say what the comment knows: "…cannot read a q4_0 V cache (the flash-attention matchers require F16 V: dispatch-flash-attention.cpp:247 and :345; the writer alone, gpu-kernels#75, is not enough)". A refusal's value is entirely in naming the right fix.
  2. An EXPIRY comment is the fleet's most-forgotten artefact. Nothing fails when the read path gains q4_0 V, so this block will still be refusing a then-supported configuration months later — and the lane that adds the read path will not grep for the refusal that blocks it. The mechanism that self-expires is the one I asked for at the first head and the EXPIRY note now gives the exact justification: ask the device about the node that will fail. After set_rows and the FA matcher both accept, ggml_backend_dev_supports_op on the FA node with a q4_0 V returns false today and true then, and the refusal disappears at the same moment the capability arrives — no human, no expiry date.

Still no test at this head (0 test files touched), and the predicate is still strncmp(dev_name, "HRX", 3). The name is fine if it is the mechanism — but here the PR itself has just shown that the condition is a capability, not a vendor, which is exactly what supports_op expresses and a name never can.

@bong-water-water-bong
bong-water-water-bong merged commit e7b8a14 into 1bit/hrx-vulkan-patched Nov 13, 2026
bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…342)

I found this by content-verifying my own merged PR rather than trusting the merge: #338 landed, the
guard and the corrected expiry are both in 1bit/hrx-vulkan-patched, and the message they carry still
said

  'cannot write a q4_0 V cache (no SET_ROWS writer: gpu-kernels#75)'

gpu-kernels#75 merged the writer. So the refusal was true, correctly placed, and explained itself with
a fact that had stopped being true -- the same stale-reason class a reviewer caught on #338's first
revision.

The refusal stays (the READ path is still absent: dispatch-flash-attention.cpp:238 requires
value->type == GGML_TYPE_F16); only the reason changes to the one that is true today, plus the tense
in the expiry comment.

Measured on device, gpu-hold, gfx1151, LFM2.5-1.2B-Q4_K_M, -c 256:
  -ctv q4_0  rc 1  'cannot READ a q4_0 V cache (attention requires f16 for V; gpu-kernels#75 merged
                    the SET_ROWS WRITER, the reader is still absent, engine#679)'
  -ctv q8_0  rc 0  Final estimate: PPL = 23.9396 +/- 0.20233

Co-authored-by: bcloud <agent@local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant