Skip to content

tests: pin the MUL_MAT_ID activation range the f16 narrowing handles (Refs engine#2127) - #307

Closed
bong-water-water-bong wants to merge 11 commits into
1bit/hrx-vulkan-patchedfrom
1bit/2127-mul-mat-id-f16-scale
Closed

bong-water-water-bong wants to merge 11 commits into
1bit/hrx-vulkan-patchedfrom
1bit/2127-mul-mat-id-f16-scale

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Why

The routed gate/up motifs narrow the f32 activation to f16 with no prescale (engine#2127), and every MUL_MAT_ID fixture filled its activation uniform in [-1, 1] — where that is essentially lossless (~11 mantissa bits, so ≤ 1 loses ~1e-4 relative) and the CPU arm narrows identically. So the range that works was never pinned, and the range that breaks was not reachable by any row.

What

init_mul_mat_id_tensors takes an activation scale now — default 1.0f, which keeps every pre-existing case byte-identical — and one new row reuses a shape the same sweep already reports OK (bf16 × f32, n_mats=4, n_used=2, m=512, n=32, k=256), with activations uniform in [-10000, 10000]:

activation_scale=10000 : OK        (measured on HRX0 at pin 7a1f2b81ce17aa)

Same shape, same weights, same everything except magnitude — so a failure here would be about the range and nothing else. Pinning it is what stops a future motif fix from buying the large case by breaking the ordinary one.

What is deliberately NOT a row

The large case is measured bad today, and stated in the comment instead:

activation_scale=100000 (beyond f16's 65504):  [MUL_MAT_ID] NaN at index 0
                                              HRX0=-nan   CPU=-165957.6875
857/858 tests passed ; Backend HRX0: FAIL

That is engine#2127's failure reproduced in one command with a positive control beside it — posted on the issue with the numbers. It is not in this PR because the harness has no xfail hook (test_case's virtuals are op_desc, vars, build_graph, max_nmse_err, max_maa_err), so adding it now would only red the suite. It belongs with the kernel fix in mul_mat_id_*_f16_wmma_projection.loom, or as an expected-fail row once such a hook exists.

Verification, and the one inference

  • The 1e4 row's result is measured (above).
  • The suite total in this exact tree (with the failing row absent) is inferred, not run: a harness processes rows independently, so 858/858 with Backend HRX0: OK follows from the row measured above — but my confirmation attempt never left the GPU queue, and I am not going to present an inference as a run.

Four earlier attempts at this gate each produced something that looked like a pass and proved nothing (no HRX device; a rejected systemd property; rows inside #if 0; an unsupported shape). The reason the numbers above are trustworthy is that each of those was ruled out first.

Refs engine#2127

bcloud added 11 commits November 9, 2026 04:39
…meters

`fused_context_claim` accepted a per-head ROPE on SHAPE alone (`is_head_row(op) && fed_by_op(op)`), while the
two registrations that execute rotations accept different PARAMETER sets:

  * llm/dispatch-attention-qkv.cpp: `n_dims == head_size && mode == NORMAL`, ext_factor 0, attn_factor 1;
  * common/dispatch-rope-set-rows.cpp: `4 <= n_dims <= head_size`, multiple of 4, mode NORMAL / NEOX /
    collapsible IMROPE.

A vision projector's rotation (`n_dims = 32` on a 64-wide head, `mode = GGML_ROPE_TYPE_VISION`) satisfies
neither, but it is a head row fed by an op, so it was claimed and then refused when the dispatch was built:

  E graph_compute: unsupported HRX node 11: ROPE output=f32[64,16,784,1] <- ROPE inputs=[f32[64,16,784,1] <- VIEW]
  E graph_compute: also: ROPE matcher rejected node: ROPE params or positions shape is unsupported
                   params={n_dims=32, mode=24, …}
  E clip_image_batch_encode: ggml_backend_sched_graph_compute failed with error -1

i.e. the whole CLIP graph failed instead of that one node running on the CPU, which is why engine ggml-org#1467's
projector cannot run with HRX0 in the graph at all.

The condition now lives in one place, `ggml/src/ggml-hrx/rope-claim-params.h`: the two matcher predicates plus
their union. Both matchers forward to it (no behaviour change) and the claim requires the union, so the claim
cannot drift away from the matchers again - which is exactly how two earlier attempts at this fix broke text
decode: each restated the condition and used only the SET_ROWS predicate, which text rotations do not satisfy
because they run inside the attention fusion.

Verified without a device (the GPU was held by another lane's bench for this whole window):
`tests/test-hrx-rope-claim-params.cpp` (ctest `test-hrx-rope-claim-params`) asserts the predicates directly and
drives `fused_context_claim` on a graph shaped like the failing node (3-D, VIEW over a produced tensor, so the
head-row and fed-by-an-op rules both hold):
  * text rotation (n_dims == head width, NORMAL)  -> claimed  (unchanged by this commit)
  * vision rotation (n_dims 32, mode VISION)      -> NOT claimed  <- the fix
and, with ONLY this claim change reverted and everything else in place, the test aborts with
`claim for vision rotation: got 1, wanted 0` - so it fails on the defect and passes with the fix.

Still blocking the rest of ggml-org#1467: with the projector on HRX the next refused node is a pure-alias VIEW
(`unsupported HRX node 20: VIEW f32[2048,2,1,1] <- VIEW`), which is claimed by the standalone path rather than
by a fused claim. That is a separate change and is not in this commit.

Refs ggml-org#1467
…s elision in a host test

engine ggml-org#1467's projector is refused at a node the log names only by shape:

  E graph_compute: unsupported HRX node 20: VIEW output=f32[2048,2,1,1] <- VIEW inputs=[f32[2048,2,1,1]] consumers=[21:ADD]

A layout-alias node is refused there for one of two completely different reasons and the shape cannot tell them
apart: it IS an alias (one input, same storage as its input) whose input was not available when the node was
visited, or it is not aliasing its input at all and therefore needs a dispatch that no registration provides. The
message now says which:

  ... consumers=[…] alias=same_storage inputs=1
  ... consumers=[…] alias=not_aliasing_input inputs=2

`alias=` is `is_layout_alias_node(graph, node)` — the helper the elision path itself calls twelve lines below, not a
re-derivation of it. That is deliberate (review point on this PR): a diagnostic that restates the scheduler's rule
will eventually describe a scheduler that no longer exists, which is the same argument as sharing
rope-claim-params.h between the claim and its matchers in #257. The wording says `not_aliasing_input` rather than
"not an alias" because the op here IS a layout-alias op; what is false is the storage relation.

Alongside it, `tests/test-hrx-view-alias-elision.cpp` (ctest `test-hrx-view-alias-elision`, links ggml-hrx) builds
the graph by hand and pins four storage relations, host-only, no backend:

  no-op alias, external input : plans        <- what can_execute_standalone_op_as_graph builds
  no-op alias, produced input : plans        <- the real graph
  offset alias, produced input: plans
  strided view, produced input: REFUSED      <- and the refusal lands on the CONSUMER, not the view

The first two lines are the useful ones: a lone node's input provenance does NOT decide elision, which refutes the
reading that the projector's refusal comes from value_is_available() returning true for a producer-less value
(dispatch-scheduler.cpp:102-106). That reading cost this lane a wrong comment on the issue, so it is now an
assertion. The last line records that a strided view is elided as well — same_storage() compares the storage ROOT,
not the layout — and that the plan is then refused at the consumer; if a consumer ever accepts a strided alias,
that assertion is where it becomes visible.

Verified: ggml-hrx links clean with -Werror and the test runs green (its status line is the new field through the
scheduler's own plan().status.errors()). The device run this diagnostic exists for is on the projector; it printed
`alias=not_aliasing_input` for node 20, which is what moved the investigation from "which of two causes" to
"ValueMap gave the view a storage of its own".

Refs ggml-org#1467
`alias=not_aliasing_input` (previous commit) says a layout-alias node's storage is not its input's. On
engine ggml-org#1467's text tower that is where the investigation stops, because two very different causes print the
same word and they need different repairs:

* `view_src_unregistered` — the `ggml_tensor` the node's `view_src` points at is not a value in this graph, so
  `find_alias_source`'s pointer lookup cannot hit and the storage-root fallback has nothing to match. Repair:
  register the value, or resolve the alias by storage *range* rather than by tensor-pointer identity.
* `view_src_registered_but_storage_differs` — it WAS registered and the map still gave the view a storage of
  its own. Repair: that is a bug in the assignment itself, not in what the graph contains.

Both now print distinctly:

  ... alias=not_aliasing_input(view_src_unregistered) inputs=1
  ... alias=not_aliasing_input(view_src_registered_but_storage_differs) inputs=1

`find_tensor()` is public and `Value::tensor` is public (`graph/value-map.h:100-101`, `:84`), so this asks the
same map the assignment asked, rather than re-deriving anything.

Why it matters: the difference decides whether the next change is "register the missing value" or "match by
storage range". Guessing wrong costs a device run each time, and this lane has already spent several.

Coverage, stated plainly: the new branch is NOT exercised by `test-hrx-view-alias-elision` — every case there
fails *before* the message is built (the strided case is refused at its consumer), and `unsupported_node_message`
is file-local so the test cannot call it. What the test does show is that the change does not alter the four
relations it pins. Exercising the new branch needs the projector on a device.

Stacked on the previous commit (`1bit/1467-view-alias-diag`); merge that first.

Refs ggml-org#1467
… node

Six hypotheses about engine ggml-org#1467's refused `VIEW` have died under their own tests (input provenance, offsets,
strides, an unregistered view_src resolved by storage range, producer coverage, and the claim). Each one was tested
by building the case it predicted and watching it not reproduce. That is the signal to stop predicting and print the
two storages.

For a layout-alias node whose storage is not its input's, the message now also carries:

  storages=out7/in3 nb=[4,8192,16384,32768] offs=0

* `storages=outN/inM` - the two `ValueStorageId`s. Equal ids with `alias=not_aliasing_input` would be a contradiction
  (the map agreed and the message says it did not); different ids are the fact every hypothesis has been guessing at;
* `nb=[...]` and `offs=` - what layout the view describes. A view whose strides differ from its input's same-shape
  tensor is a strided alias, which is a different thing from "the map gave it a storage".

With the `alias=` and `why=` fields from the two commits below, one run now answers: which storage the map gave the
view, whether it is its input's, whether the view's own `view_src` was registered, and what layout it describes.

Coverage, stated plainly: like the previous commit, this field is not exercised by
`test-hrx-view-alias-elision` - the host cases fail before a message is built, and `unsupported_node_message` is
file-local. What the test does show is that the four pinned relations and the added case still behave.

Stacked on `1bit/1467-alias-reason` (#262), itself stacked on #258 and #257. Merge in that order.

Refs ggml-org#1467
…h the run's own numbers

The projector run printed the facts this case encodes:

  VIEW output=f32[2048,2,1,1]  nb=[4,32768,65536,65536]  offs=8192
  alias=not_aliasing_input(view_src_unregistered)  storages=out24/in11

Rows of 2048 f32 taken from a buffer whose rows are 8192 f32 apart (`nb[1]` = 32768 bytes = 4x the width), starting one
row in (`offs` = 8192 bytes = 2048 f32), and the wide source is never registered as a value in the graph - which is
why `find_alias_source`'s two tensor-identity lookups both miss and the view is given a storage of its own.

`ViewKind::WideWindow` builds exactly that shape, so `test-hrx-view-alias-elision` now reproduces the refusal on a
machine with no device:

  no-op alias, external input : plans
  no-op alias, produced input : plans
  offset alias, produced input: plans
  strided view, produced input: REFUSED    <- refused at its CONSUMER
  wide window view (node 20)  : REFUSED    <- refused AT THE VIEW, as the projector is

The assertion is `REQUIRE(!wide)`: that is today's behaviour and it is the defect. The fix is a densify dispatch for a
layout-alias node that cannot be elided (copy the strided window into a contiguous transient - `dispatch-copy.cpp`'s
append_f32_densify_dispatch already does this for permuted activations), and when it lands this line flips to
`REQUIRE(wide)` - which makes the change visible in the test rather than inferred from a log line.

Two traps for whoever writes it, recorded here because both produce a WRONG NUMBER rather than a failure: the copy
must derive its source offsets from nb[]/offs (a densify that reuses the pre-densify plane offsets copies the wrong
rows), and the transient must hold ne[0]*ne[1] contiguous elements of the same type.

Refs ggml-org#1467
…STER/llama.cpp into 1bit/1467-alias-storages

# Conflicts:
#	ggml/src/ggml-hrx/dispatch/dispatch-scheduler.cpp
#	tests/test-hrx-view-alias-elision.cpp
… branch's superset block

The pin has this message code as #258 landed it; this branch has it with the why=/facts additions, so the superset is the
resolution and nothing from the pin is lost. The previous merge commit carried the markers themselves, which is why the
tree did not compile - caught by rebuilding before pushing, not by review.
…STER/llama.cpp into 1bit/1467-alias-storages

# Conflicts:
#	ggml/src/ggml-hrx/dispatch/dispatch-scheduler.cpp
… on top of it

The previous merge resolved this hunk to *this branch's* older version, which silently removed what the base added
deliberately: the `AliasMiss` switch, i.e. the miss cause read from **assignment-time state**. That is the half an
earlier review called blocking, and my resolution brought the forbidden re-ask of the map back with it. The reviewer
caught it; this commit fixes it.

Resolution now, per file:
- `dispatch-scheduler.cpp`, `value-map.{h,cpp}`, `graph-diagnostics.cpp`: **the pin's versions** (AliasMiss +
  NoAliasSource/ExactViewSrc/RangeFallback, recorded when the storage is chosen);
- and on top of the pin's block, only this branch's own addition: `storages=outN/inM nb=[…] offs=…`, which is the
  fact the six dead hypotheses needed. `why=` still comes from the pin's `AliasMiss`, never from a re-ask.
- `<cinttypes>`/`<cstdio>` added for PRId64/snprintf, which the pin's file did not need.

Rebuilt before committing: `ggml-hrx` links clean with -Werror, and the host test still pins the four relations plus
`wide window view (node 20) : REFUSED`.

Refs ggml-org#1467
The prefill matcher required the q/k/v dim-2 stride to be exactly packed (`nb[2] == hidden*4`), so a q/k norm fed by a
view whose token step is wider than `nb[1]*ne[1]` fell through and the node was left uncovered while its consumer was
claimed by the fused dispatch:

  E graph_compute: unsupported HRX node 160: L2_NORM output=f32[128,16,512,1]
  hip.l2_norm_f32: declined (consumer is GATED_DELTA_NET (left to the fused matcher))
    x ne=[128,16,512,1] nb=[4,512,40960,20971520] off=0     <- nb[2] = 40960 against 8192, 5x

The token stride is now bounded below and required to stay float-aligned instead of being required to be packed. That
is not a new rule: the DECODE matcher in this same file already states it for v --

  v->nb[2] < static_cast<size_t>(width * head_count) * sizeof(float) || v->nb[2] % sizeof(float) != 0

-- so the prefill guard is brought into line with the proof the file already relies on, rather than inventing one.

Nothing else relaxes: `nb[0]`, `nb[1]`, the shared storage, the three storage offsets and `raw_k->nb == raw_q->nb`
still hold exactly as before, which is what keeps the relaxation to "the token dim may be strided" and not "the layout
may differ".

Verified: `libggml-hrx.so` builds clean with -Werror. NOT verified by me, and stated in the PR: the placement census
(engine#1925 asks for 2 nodes off HRX / 2 splits) and the pin's 12.14 tok/s decode / 371.6 prefill at 18,782 tokens
row -- a matcher relaxation that completes but computes wrongly passes the census and shows up only there, so the
device row is the reviewer's gate, not something I am claiming.

Refs: engine#1925, engine#679
…dles (engine#2127)

The routed gate/up motifs narrow the f32 activation to f16 with no prescale, and every MUL_MAT_ID fixture filled its
activation uniform in [-1, 1] -- where that is essentially lossless (~11 mantissa bits) and the CPU arm narrows
identically. So the working range was never pinned and the broken one was never reachable.

`init_mul_mat_id_tensors` takes an activation scale now (default 1.0f, keeping every pre-existing case byte-identical)
and one row reuses a shape the same sweep reports OK -- bf16 x f32, n_mats=4, n_used=2, m=512, n=32, k=256 -- with
activations uniform in [-10000, 10000]:

  activation_scale=10000 : OK        (measured on HRX0 at the pin, 7a1f2b8)

That is the range the narrowing does handle, and pinning it is what keeps a future motif fix from buying the large
case by breaking the ordinary one.

The LARGE case is measured BAD today and is deliberately not a row: with the same shape and scale 1e5 (beyond f16's
65504) HRX0 returns -nan where the CPU arm returns -165957.6875, `857/858`, `Backend HRX0: FAIL` -- engine#2127,
posted with the numbers. It lands with the kernel fix in mul_mat_id_*_f16_wmma_projection.loom, or as expected-fail;
the harness has no xfail hook, so adding it now would only red the suite.

Not verified in this exact tree: the suite TOTAL with the failing row absent (a harness processes rows
independently, so 858/858 with `Backend HRX0: OK` follows mechanically from the row measured above, but it is an
inference and not a run -- the confirmation attempt never left the GPU queue).

Refs engine#2127

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 33f72d6: READY

Pinning the activation range that the f16 narrowing has to handle is the test the 2⁻⁶ prescale change (#285/#298) needed: the fix keeps activations inside f16's range before the truncation, and a test that asserts the range is what stops a future change from quietly moving the boundary back out. A test on the resulting values would pass on this data and fail on the first larger activation; a test on the range states the invariant.

Adds <cinttypes>/<cstdio> for the diagnostic, which is the pattern this fork has settled into — report the value that decided it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 33f72d6: CHANGES

Reviewed head 33f72d6 (refs engine#2127). This is the first verdict. The change itself is small and the reasoning is right: a scale argument defaulting to 1.0f, so every existing case stays byte-identical, plus one positive-control row at [-10000, 10000]. The branch it sits on is the problem.

It conflicts with its base, because it carries another PR. GitHub shows mergeable_state: dirty. git merge-tree of this head onto 1bit/hrx-vulkan-patched (204493d40) gives CONFLICT (content): Merge conflict in tests/test-hrx-view-alias-elision.cpp. The branch is 11 commits past its merge base, and only the last one, 33f72d6 (tests/test-backend-ops.cpp +31/−5), is this PR's change. The other ten are #301's: a1b74a0cf "let the fused GATED_DELTA_NET take a strided q/k norm" is in #301 (open, not merged), and 636150d4a/5cd2181a3 are in #257/#258, merged, and in #301. That is why the file list shows dispatch-scheduler.cpp, dispatch-gated-delta-net.cpp and test-hrx-view-alias-elision.cpp, which a test-only PR does not touch.

The fix is mechanical. 33f72d6 alone applies cleanly onto the current base: git merge-tree --merge-base 33f72d6^ 204493d40 33f72d6 gives rc 0. So either:

  • rebase onto 1bit/hrx-vulkan-patched with just that commit (git rebase --onto origin/1bit/hrx-vulkan-patched 33f72d6^), or
  • if it is meant to land after #301, set the base to #301's branch and say "stacked on #301" in the body, so the diff a reviewer sees is the 31 lines.

Not run by me: the HRX0 row (activation_scale=10000 : OK, and the excluded 100000 NaN). That needs the device and a fork build.

@bong-water-water-bong

Copy link
Copy Markdown
Author

The suite total is measured now, and it corrects my own inference: 857/857, not 858/858.

Run on HRX0 at the pin, against exactly this branch's tree:

MUL_MAT_ID(type_a=bf16,type_b=f32,n_mats=4,n_used=2,b=0,m=512,n=32,k=256,activation_scale=10000.000000): OK
857/857 tests passed
Backend HRX0: OK

The count I predicted in the body was wrong by one, and the reason is worth stating because it is the same mistake in miniature that this PR is about: I wrote "858/858
follows mechanically from the row measured above" while reasoning about run 5, which had 858 rows — including the 1e5 row I then removed. 856 pre-existing + 1 new = 857.
The arithmetic was trivially checkable and I inferred it instead, which is exactly the habit the body's own "inferred, not run" paragraph was apologising for. Now it is a number
from a run: 857/857, Backend HRX0: OK.

What that changes about this PR: nothing in the diff — the row is the same row — but the claim in the body is now a measurement rather than an inference, and the correction
is visible here rather than quietly edited, since a reviewer comparing the body to the run output would otherwise see 858 vs 857 and have to work out which one lied.

Full evidence trail, for anyone re-running: the gate took four invalid attempts before it meant anything — no HRX device (the harness refused to report a pass), a rejected
-p Environment= on the user scope, the rows inside an #if 0 block, and an unsupported shape (m=256,n=2,k=32 → not supported [HRX0]). Each of those is a way a sweep can
print "pass" and prove nothing, which is why the number above is quoted with its row rather than as a summary line.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 33f72d6d1 — the analysis in the body is exactly right ("every MUL_MAT_ID fixture filled its activation uniform in [-1, 1] … and the CPU arm narrows identically, so the range that works was never pinned"), and a scale parameter with a 1.0f default is the right shape: existing rows keep their behaviour and the new rows are the ones that name a range.

One thing that decides whether the new arm discriminates, and it is a rounding fact I got wrong myself today: 65504 is the largest finite f16, but the first value that becomes infinity is 65521, not 65505. The next representable half above 65504 is 65536, so the midpoint is 65520, and round-to-nearest sends everything ≤ 65520 down to 65504. I added a pair at 65504/65505 earlier expecting the second to fail; both passed, because both round to the same finite value. The discriminating pair is 65519 (finite) vs 65521 (inf).

So: if the new arm picks a scale "just above 65504", check where it actually lands — 65505…65520 are all inside the interval that rounds to the largest finite f16, and a test there passes with or without the narrowing being fixed. Worth a comment in the fixture saying which side of 65520 each scale sits on, since the next person will reach for 65505.

Two more, both from testing the sibling fix today:

  1. State that the comparison is against the CPU arm. That is what makes this row different from the test 2b7960bae added for the same defect, which compares hrx_spike.size() == hrx_plain.size() and non_finite == 0 — i.e. HRX vs HRX, so a uniform scale error (say a missing ×64 unscale) passes it while a value comparison against CPU would catch it immediately. If MUL_MAT_ID rows compare to CPU (they do in test-backend-ops), say so in the body — it is the property that makes this PR worth more than the one already landed.
  2. Check the premise against the base. 2b7960bae on 1bit/hrx-vulkan-patched — "the routed gate/up SwiGLU narrows its input at 2^-6" — already adds a 2^-6 prescale to the three projection motifs and the f16 accumulate (verified: prescale count 1 in each; 0 in ops/mul_mat_id_swiglu_f32_f32_wmma; no 64.0 anywhere in that commit or in routed_down_weighted_reduce_next_rmsnorm_f32). If this PR's base contains that commit, "no prescale" is stale and the honest framing is "pin the range the prescale now handles, and assert the pairing"; if it does not, the row documents the gap. Either way it is a good row — but the body's premise should match the base it targets.

And the follow-through this row enables: a scale at 65521 on MUL_MAT_ID is the one case that would have (a) failed before 2b7960bae and must pass after it, and (b) failed loudly if the prescale were added without the ×64 (a 1/64 mismatch against the CPU arm rather than a range error). That is the before/after the routed chain has never had.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 33f72d6 — and the reasoning for not widening the default is the part that makes it a fix rather than a bigger net.

+static void init_mul_mat_id_tensors(ggml_context *ctx, int n_mats, float activation_scale = 1.0f) {
+            // engine#2127: the routed gate/up motifs narrow the f32 activation to f16 with NO prescale, so an
+            // activation outside f16's range is where that shows. The default fill is uniform in [-1, 1], which
+            // cannot reach it (f16 carries ~11 mantissa bits, so values <= 1 lose ~1e-4 relative and both arms
+            // narrow identically) -- hence a per-case scale rather than a wider default

Why per-case beats a wider default

The default fill cannot fail on this defect: inside [-1, 1] an f16 narrowing loses ~1e-4 relative, and both arms lose the same amount, so the comparison agrees whether or not the prescale exists. Widening every fixture would have made that one case fail and perturbed the exactness of every other case — a bigger net with a coarser mesh. A per-case activation_scale puts the stimulus exactly where the defect lives.

Two things to pin while you are in the fixture

  1. the scale has to cross the prescale window, not just f16's limit. hrx: keep dense matmul activations inside the f16 range in the f32 core (engine #1962; stacked on #284) #285's fix is a 2^-6 prescale with a stated valid window [3.9e-3, 4.19e6]; a case at ~1e5 exercises the top (where inf→NaN occurred), while a case near 1e6 exercises the boundary where the prescaled value is the limiting factor. If this PR picks one, say which and why — and if it picks the largest, note that the window's upper edge is what the number measures;
  2. the assertion should distinguish the two arms, not merely pass. The failure to catch is "no prescale": an arm with the prescale and one without should differ outside the window and agree inside it. A single arm asserting "the result is right" passes with a prescale and without if the values stay in range — which is the state Update README.md to add more docs indexes ggml-org/llama.cpp#2127 was in.

On the title

"pin the MUL_MAT_ID activation range the f16 narrowing invalidates" is the correct description of a fixture change: it names the range as the subject. Worth one line in the body saying what the old default measured (nothing, by construction) — that is the sentence a future reader needs before "simplifying" the scale back to 1.0.

bong-water-water-bong added a commit that referenced this pull request Nov 11, 2026
…gine ggml-org#2167) (#315)

test_mul_mat_id gains an activation scale (default 1, so every earlier case is unchanged; only the activation `b`
is scaled). Rows at 1e4 and 1e5 for bf16, f32 and Q4_K weights at a shape the sweep already runs, and bf16 at one
token. The routed kernels narrow the activation at 2^-6 since llama.cpp #285, so the rows pass; they pin that.

Supersedes the test rows of llama.cpp #307, whose parameter also scaled f32 weights.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 33f72d6: READY

This withdraws my CHANGES of 2026-11-11T09:52:50Z, and explains why it stood for two days: my two attempts to withdraw it were posted as comments beginning VERDICT: instead of VERDICT at, which the merge gate does not read. A withdrawal is a verdict and has to be posted as one — that is my error, not the author's, and the gate was right the whole time.

The reason in that CHANGES is disproved by the repo's own checker. It said the PR carries another PR. Against the current base 29616ef4f:

pr_carry_check: 1 ref(s) against origin/1bit/hrx-vulkan-patched: carried=0 landed=0 own=4 shared_work=0

All four commits are its own. The substance of the code review was never in doubt — the body of that verdict says the change is right, and the f16 note below is the only caveat I still hold.

What is still true, and it is a fact rather than a verdict: it conflicts. merge-tree returns rc 1 with conflicted stages in exactly two files:

tests/test-backend-ops.cpp             (stages 1, 2, 3)
tests/test-hrx-view-alias-elision.cpp  (stages 1, 2, 3)

So the conflict is not a carry — the base moved under four of the PR's own commits, and two test files were edited on both sides. The fix is a rebase, and the resolution is new code, so I will re-verdict on the new head rather than treat this READY as covering it. Please rebase rather than close: nothing here is wrong that a rebase does not fix.

Carrying the one caveat forward, because it is the only thing that could still bite in the resolution: the new arm must actually land its scale inside the f16 range it is testing — 65505…65520 are inside the 65504 boundary and round back, so an arm that picks "just above 65504" can end up testing the ordinary path. Check where the value lands, not where it was intended to land.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Rebase attempt report, because this PR is CONFLICTING and the conflict itself is informative.

What I tried. Cherry-pick 33f72d6d1 (the ggml-org#2127 commit, dropping the ggml-org#1925 one, which had its own PR #301 closed as backlog) onto 1bit/hrx-vulkan-patched. All six conflicts land in tests/test-backend-ops.cpp, in the region main rewrote while this PR sat: main added the NVFP4 / W4A8 path (struct test_mul_mat_w4a8, GGML_PREC_Q8 on src1 disallows 4-bit activations) and a mul_mat_id activation-scale branch.

The part worth your attention: base may already carry this fix. The conflicting hunk's HEAD side already reads:

if (activation_scale != 1.0f && strcmp(t->name, "b") == 0) {
    // engine #2127 / #2167: the default fill is uniform in [-1, 1], which no half-precision narrowing of the
    // activation can get wrong, so a case can ask for a wider activation. Only the activation `b` is scaled ...

and origin/1bit/hrx-vulkan-patched has three mentions of #2127/#2167 plus both init_mul_mat_id_tensors and activation_scale. That is the same shape as ggml-org#2384, which I checked a day ago and found merged with a byte-identical file — so before anyone spends an hour resolving these conflicts, the question to answer is: did ggml-org#2167 land this, and is #307 therefore superseded?

I could not settle it cheaply: the file-level diff is useless because base is 41 commits ahead, and a line-presence test over the PR's added lines gives 22 of 39 present in base, which is exactly what you would expect both from "already merged" and from "lots of shared boilerplate". Settling it needs a hunk-by-hunk compare of the PR's own diff against base, which is the rebase itself.

My recommendation, in order:

  1. compare the PR's diff hunks against base and close with --delete-branch if they are all present (the Default for RMS epsilon ggml-org/llama.cpp#2384 outcome);
  2. otherwise rebase keeping only the Update README.md to add more docs indexes ggml-org/llama.cpp#2127 commit — the Fixed incorrectly applying RMS norm twice ggml-org/llama.cpp#1925 commit (a1b74a0cf, "let the fused GATED_DELTA_NET take a strided q/k norm") travels in this PR but its own vehicle was hrx: let the fused GATED_DELTA_NET take a strided q/k norm (Refs engine#1925) #301, which is closed as backlog, so shipping it inside a Update README.md to add more docs indexes ggml-org/llama.cpp#2127 fix makes the PR two things at once;
  3. if the hunk resolution is needed, prefer main's side in the NVFP4/W4A8 region: that content is newer and this PR never touched those tests, so anything of the PR's in that region is an older copy of code main has since changed.

I have not pushed a resolution: 6 conflicts in the large backend test file with no way to run the full backend suite right now is exactly the kind of edit I would rather hand over than half-verify. 1bit/2127-mulmatid-pin is untouched.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Superseded — closing with the evidence, in the same shape as ggml-org#2384.

The PR predicates on init_mul_mat_id_tensors(..., float activation_scale = 1.0f), and base already has a superset of that signature:

origin/1bit/hrx-vulkan-patched:  init_mul_mat_id_tensors(ggml_context *, int n_mats, bool skip = false, float activation_scale = 1.0f)
this PR:                         init_mul_mat_id_tensors(ggml_context *, int n_mats,                    float activation_scale = 1.0f)

So the parameter this PR exists to add is in base, with a further skip argument added since, and base carries three mentions of #2127/#2167 including the comment explaining why the default is not widened. The fix landed and evolved; a rebase would have produced a conflict over code that is already correct.

That also explains the six conflicts I hit: they are all in the region where base added the NVFP4/W4A8 path, and the conflicting hunks' HEAD side already contains this PR's reasoning.

Branch 1bit/2127-mulmatid-pin deleted with the PR. If any part of this change did not make it — the ggml-org#1925 commit a1b74a0cf ('let the fused GATED_DELTA_NET take a strided q/k norm') travelled in the same PR and its own vehicle #301 was closed as backlog — that work now needs its own branch and PR, because it was never this PR's subject.

@bong-water-water-bong
bong-water-water-bong deleted the 1bit/2127-mul-mat-id-f16-scale branch November 14, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants