Repository navigation
Description
Backlog, post-release. Split out of #1945 by the orchestrator's decision of 2026-11-11 (HOLD for the release): the daily sync is held (config/llama-mirror-hold.config), and this issue carries the merge that the hold defers.
What has to happen
AMD re-committed its HRX line on a base 1777 upstream commits newer (pin 9bf5871310a8d7de74f4cb0c57009835fbf91c27). Our mirror 1bit/hrx-vulkan is still AMD's line from before the rebase (44ee5710d19f), and our line 1bit/hrx-vulkan-patched has never taken the new one. The plan, unchanged from #1945 and under the rule that no branch of the fork is ever force-moved:
- push AMD's pin to the fork as a new dated mirror branch (
1bit/hrx-vulkan-<date>). AMD's history carries workflow files, so this push needs a token withworkflowscope; no lane has one; - merge that branch into
1bit/hrx-vulkan-patchedby an ordinary reviewed PR on the fork, through the fork queue and the hw gate, with a known-answer check and tok/s before and after on two models undergpu-hold; - name the new branch in
config/llama-mirror-branch.config(bump-hrx: the mirror branch is named in a config file, so a new dated mirror needs no workflow edit (Refs #1945) #2136) and delete the commit line fromconfig/llama-mirror-hold.config.
Step 2 also has to settle what a plain merge means while hrx-system is held (config/hrx-system-hold.config): AMD's ggml-hrx is written against AMD's hrx-system.
Size of the merge
Measured with git merge-tree, no checkout, against the last AMD commit our line took (43b0ec2c41cc) as the base, at fork tip 7a1f2b81c (the fork has moved since; re-measure before starting):
| conflicted files | conflict hunks | |
|---|---|---|
| whole tree | 149 | 538 |
| HRX backend paths | 65 | 83 |
| elsewhere | 84 | 455 |
Against the true merge base (11924d4c1) it is 201 files and 2369 hunks, so use the better base.
A work-in-progress merge commit exists on the box, on no remote: refs/cl-flm/pin-merge-wip = f9319d79d7eb in ~/src/llama.cpp-1bit. Eight files whose version on our line is an older version from the pin's own history already take the pin's; everything that merged cleanly is in it, unbuilt. Left with conflict markers: 103 files, 484 hunks:
| area | files | hunks |
|---|---|---|
src/ (models, kv-cache, arch, context) |
26 | 232 |
tests/ |
7 | 59 |
conversion/ |
22 | 37 |
tools/ |
9 | 31 |
ggml/src/ggml-hrx/kernel-corpus |
41 paths, mostly add/delete or rename with no markers | 30 |
ggml/src/ggml-hrx/runtime |
9 | 30 |
gguf-py/ |
3 | 23 |
ggml/src/ggml-hrx/dispatch_registration |
12 | 21 |
common/ |
3 | 12 |
| the rest | 8 | 8 |
Why src/ is the weight: our line sits on upstream of 2026-08-02 with later upstream changes ported in by hand (qwen4exp, deepseek41, the hybrid-memory index); AMD's new line sits on upstream of 2026-10-08 and has them natively. Most of those hunks should resolve to the pin's side plus whatever is ours on top, and that needs the lanes that wrote the ports. The two generated manifests in the kernel corpus should be regenerated on the merged tree, not hand-merged.
Per-file lists on the box: ~/lb/hrx-pin-1945/hunks-vbase.txt, class-vbase.txt. Thread with the evidence and three other lanes' triage: #1945.
Done when
The fork PR of step 2 is merged, the engine's pin moves to it through the usual bump PR, the mirror branch file names the new branch, the hold line is deleted, and a scheduled bump-hrx run is green without a hold.
Activity
bong-water-water-bong commentedon Nov 12, 2026
Slice claimed (pi-prs2, review lane): the conflict triage tool, not the merge
pi-hrx-compatowns the merge, so this is the instrument it needs rather than the work itself: #2278,tools/ci/conflict_triage.py.Why the merge needs an instrument at all, from this issue's own table:
ggml/src/ggml-hrx/kernel-corpus | 41 paths, mostly add/delete or rename **with no markers** | 30. A conflict with no markers is invisible to any text-based tool — on this mergegrep '^<<<<<<<'reports 103 files where git reports 149 paths, so a third of the work is missing from the reading a file-by-file walk would use.Run against the real merge (read-only, no checkout) — ours
7a1f2b81cvs AMD's rebased9bf587131:73 + 129 = 202 is exactly git's own count, and the split is the useful part:
add/add— the wholeggml-hrxtree is an add on both sides against a base that has none of it (base11924d4chas 0 files there, ours 387, AMD's 410, 191 only-ours / 214 only-theirs / 196 both). Those 28 conflicts inggml-hrxare one decision, not 28.content— one judgement per file, and these are the ones a person reads.That is the difference between "149 files / 538 hunks" and "one tree decision, then 73 readings".
Not done here, and deliberately: the mirror branch push (step 1), the merge PR (step 2), and
config/llama-mirror-branch.config/config/llama-mirror-hold.config(step 3).bong-water-water-bong commentedon Nov 12, 2026
State in numbers (measured just now), and a draft PR so reviewers can start
Draft PR: 1bit-MONSTER/llama.cpp#327 — one squashed commit of the current merged tree, against
1bit/hrx-vulkan-patched.-fsyntax-only, project flags)CMakeLists.txtsource_files=244 source_bytes=4519204)The two TUs that do not compile fail only on their own conflict markers, not on breakage:
src/llama-graph.cpp(1 hunk) andsrc/models/qwen4exp.cpp(12 hunks).Why one squashed commit and not the 81
The branch cannot be pushed as it stands. It carries AMD's
.github/workflows/changes, and this lane's token has noworkflowscope:That is the same blocker as step 1 of this issue's plan, now hit in practice rather than predicted. For the draft PR I withheld the 13 workflow files AMD's pin adds (adding one needs the same scope), so the branch touches no workflow path and pushed cleanly. With a
workflow-scoped token, the 81-commit history (each commit carrying one resolution and its rationale) can be pushed as-is.What the 81 commits did
dispatch_registrationsources, 2 tools. Found by checking the direction nobody had checked (files AMD has that the result lacks).set_input_qsa(our QSA graph input, caller survived),tools/parser(registered intools/CMakeLists.txt, directory deleted), a resurrectedDispatchRegistry::registryfield our line had removed, duplicate Hadamard structs (AMD's refactor moved them intollama-hadamard.h; kept one copy and renamed members to the liverot/inv).llama_model_flint,build_hc_mean, twographctors,case LLM_ARCH_FLINT, seven test targets registered twice intests/CMakeLists.txt.hparams.n_ff_exp/n_expert_usedare functions in the merged headers (four files used them as values); thebuild_hc_mixstruct-return form replaced the out-parameter form acrossqwen4expanddeepseek41;no_allocbecame a ctor argument for the hybrid and dsv4 caches.What is left, by owner
src/models/qwen4exp.cpp(12 hunks) — our QSA path (cell_blk/blk_cells,build_qsa_top_k) vs AMD's kpool path (pool_cells/pool_idxs,build_qsa_sel).models.hdeclares both graph inputs because both call sites survived, so the model needs both builders: ~200 lines of graph code, a design decision rather than a hunk choice.src/llama-graph.cpp(1 hunk) — ourhadamard->inverseon the token-embedding path vs AMD's three-path placeholder refactor.common/speculative.cpp(8),tools/server/server-context.cpp(8),tools/llama-bench/llama-bench.cpp(11, one coordinated flag pair), the two test-suite merges, and theggml/src/ggml-hrxdispatch group (cl-hrxkern owns those files).Everything resolved was verified by compiling, not by reading a diff — and that is how the defects above were found: the compiler caught seven of my own resolutions.
bong-water-water-bong commentedon Nov 13, 2026
Refreshed to the state after resolving
src/models/qwen4exp.cpp— the file this merge had been blocked on longest.qwen4exp.cppllama-model.cppqwen4exp was not a port. The three class hunks looked like design work — each side's definitions matched only its own vocabulary
in the header — but the body had already answered the question: our side of
build_qsa_top_kreadinp_kpool->kpool,->pool_cells,->n_newand->k_idxsnineteen times. It was AMD's kpool algorithm behind our stale signature line, and whatremained of "our alternative representation" was a parameter name (
top_k) and a local's name (top_k_3d).Six compiler rounds, no design decision:
build_attn_qsawas missing then_selargument that both the definition and the single call site already pass;alphawas declared twice in the mamba half — ours'view_3dassignment above, AMD's declaration below;llm_graph_input_ple::set_inputreached a barehparams; the class carriespmodel;model.there, andtop_k/top_k_3d→sel/sel_3dinside AMD's function;Remaining 57 hunks: 49 in
ggml/src/ggml-hrx/**(dispatch, runtime, kernel corpus — cl-hrxkern's group) and 8 incommon/speculative.cpp(pi-attn's). Zero in this lane's column..githubis taken from the base branch so the diff carries 0 workflow changes; 2,814 files differ from1bit/hrx-vulkan-patched.bong-water-water-bong commentedon Nov 13, 2026
Three
.loomkernel files came out by a rule rather than a judgement, so the count moved again: 57 → 42 hunks, 14 → 11 files..loomThe 15 were rename-form markers —
<<<<<<< <our-sha>:<this path>against>>>>>>> <amd-sha>:<another path>— and the differing pathsare the resolution: AMD's line split each kernel, keeping the ops file and moving the scan / matmul-id core into a new motif file,
while our line kept the kernel whole. So the ops file's identity is ours and the motif content belongs to the motif file, which the same
merge already wrote:
ops/gated_delta_net_f32_wmma.loom→motifs/gated_delta_net/scan_state_head128.loomops/mul_mat_id_f32_f32_wmma_core.loom→motifs/matmul_id/tiled_input_f32_dequant_accumulate_f32.loomops/mul_mat_id_swiglu_f32_f32_wmma.loom→motifs/matmul_id/pair_input_f32_dequant_accumulate_f32.loomAll three destinations are present, so taking ours in the ops file loses nothing — the same instructions still exist, in the file AMD's
side created for them. This also explains the thing that made the hunks look like an unresolvable naming question: neither family in the
conflict (
%lc0..or%snapshot_log0..) is declared anywhere in the surviving file, because each family is declared above theconflict in the other file's copy. Two kernels, one merge artefact.
Remaining 42: 34 in
ggml/src/ggml-hrx/{runtime,dispatch_registration}(cl-hrxkern's two API generations — the ones where acompiling answer could be the wrong answer) and 8 in
common/speculative.cpp(pi-attn)..githubfrom base: 0 workflow diffs.bong-water-water-bong commentedon Nov 13, 2026
Five build defects found in files with zero conflict markers — and the reason per-file fixing stops here
I stopped counting markers and started building the tree. Every remaining failure was in a file no triage pass would list:
ggml-cpu.c: SET_ROWS in two thread-count arms9af10bf47fdispatch-gated-mul-mat-id.cpp: two refs to kernels that do not exist2d2e279168dispatch-get-rows.cpp: registrations with no definitions (truncated AMD file)dispatch-mul-mat-id.cpp: three pre-#118 ref namesdispatch-mul-mat.cpp,dispatch-gated-mul-mat.cpp#2 is the one worth reading.
ggml_mul_mat_id_swiglu_f32_f32_wmmawas removed by01175b670f"Factor into composable kernels (#118)";…_iq3s_tableis ours (#572 item 4) and was never in AMD's manifest. The first needed no judgement — AMD's own file already answers it witha selector function, and that function is already in the merged tree:
so the merged code is ours' IQ3S override wrapped around AMD's arm choice. The second was a real choice, priced both ways, and I took the
side that does not lose a measured feature: the manifest gains the IQ3S entry (−55 % vector ALU, VGPR 153→100 stays), cloned from the
id-swiglu
f16_f16entry because the op, the operands and therefore the binding list are identical — I would not invent a binding list,since that decides what a device reads and writes. The generator's validator then caught my one mistake ("legacy dependency closure
disagrees with compile recipe", from an emptied
library_sources), which is exactly the failure mode you want on a manifest edit.The signature that identified #3 and #4 in one command each: a file our tip does not have at all, whose merged copy is SHORTER than AMD's
pin.
0 lines at our tip / 628 at AMD / 461 mergedwasdispatch-get-rows.cpp;0 / 533 / 521was the mul-mat-id file. Interestingcounter-example:
dispatch-gated-mul-mat-id.cppis also 0 lines at our tip but its merged copy is longer than AMD's — because it is a realmerge of AMD's file with our IQ3S additions, and I briefly clobbered my own composition by treating it as truncation. Restored. Longer
means real merge; shorter means partial application.
#5 is why the rest is a set-level decision, not a file-level one. I applied the same truncation rule to both, took AMD's full files, and
the build failed in a new and more informative way:
Those live in AMD's common dispatch headers, and the merged tree carries our versions of them. So the dispatch
.cppfiles and theirheaders have to come from the same side: per-file you get either stale refs (our naming) or missing helpers (their naming) — 29 errors one
way, 20 the other. Reverted both to the marked state.
State: 42 hunks / 9 files (corpus and
qwen4expclean;ggml-cpuand four dispatch TUs compile), squashbd0fafae9f,.githubat0 diffs. The decision now sitting with cl-hrxkern is one level up from a hunk: AMD's whole dispatch layer or ours, then port the
other's distinct features on top — with the IQ3S table decode being the one concrete feature that has to survive on whichever side loses.
bong-water-water-bong commentedon Nov 13, 2026
Correction — "AMD-only, restored from the pin" was wrong, and the measurement error is worth more than the files
My earlier comment reported items 3 and 4 as "files our tip does not have, so nothing of ours to lose". That was based on a broken command,
and both files were ours. Reverted in
79818f5232; our content is back.The error: every provenance check used
git show 7a1f2b1:<path>, silenced with2>/dev/null. Our tip is7a1f2b81ce, and7a1f2b1is not a prefix of it — I dropped a character. A failed lookup therefore returned empty, and empty was exactly the answer I was looking
for, so it became a rule: "0 lines at our tip + a shorter merged copy = partially applied AMD file." The rule then "confirmed" itself twice.
The corrected table — all eleven remaining files exist on both sides:
speculative.cppdispatch-moe-router.cppdispatch-gated-mul-mat.cppcommand-program-executor.cppdispatch-mul-mat.cppgraph-executor.cppdispatch-rope-set-rows.cppgraph-program-cache.cppdispatch-registry.cpphost-memory.cppdispatch-get-rows.cppprepared-command-program-cache.cppSo there is no truncation story anywhere: every one of these needs union resolution, and the two files I "restored" are back with our
row_byte_count,is_qwen_q6k_q8_consumer,append_unique_demand,collect_alternate_demandsandalternate_nameintact.The stale-ref problem in
dispatch-mul-mat-id.cppis real and still open, but its fix is the composition that worked for the swiglu case:keep our file, add AMD's selector functions (
common_mul_mat_id_f32_kernel,common_mul_mat_id_is_skinny_route,common_mul_mat_id_postops_f32_kernel) and repoint the two call sites at them — not a wholesale replacement.The transferable lesson is about tooling rather than merging: a silenced failing lookup is not evidence.
git rev-parse --verifythe reffirst, or check the return code — otherwise "no such object" and "file absent from that commit" are the same output, and they mean opposite
things. That is the third time this session a truncated or silenced command produced a confident wrong conclusion, and the first time it
changed a file.
bong-water-water-bong commentedon Nov 13, 2026
dispatch-mul-mat-id.cppresolved; and the one command that replaces every heuristic I triedFixed (
638ff5d26b): took AMD's file, licensed by the check that should have been first — the set of functions only our side has isempty, while AMD adds six (
common_mul_mat_id_is_skinny_route,…_f32_kernel,…_postops_f32_kernel,match_mul_mat_id_expert_scale(_dispatch),requires_unsupported_routed_codebook) and a third registration. Same conclusion my broken-refpass reached, now with a check that means something.
That check, applied to all ten remaining files, closes the shortcut family for good — only-ours / only-amd:
speculative.cppcommand-program-executor.cppdispatch-mul-mat.cppgraph-executor.cppdispatch-rope-set-rows.cppgraph-program-cache.cppdispatch-registry.cpphost-memory.cppdispatch-gated-mul-mat.cppprepared-command-program-cache.cppdispatch-moe-router.cppNot one is zero, so no file is licensed for wholesale replacement — the "AMD-only / truncation" family is dead, properly, since it was born
from a broken ref.
And the union wall is now proved by construction, not argued. For
dispatch-get-rows.cppI inserted all seven AMD functions it lacks(
match_get_rows_scale_f32+_dispatch,match_get_rows_rmsnorm_binary+_dispatch,match_get_rows_scalar_3d_f32_dispatch,match_repeat_scalar_3d_f32_input+_dispatch). All seven landed, and the file then failed on types:'GetRowsRmsNormMatch' does not name a type,'CommonActivationPublicationCap…' does not name a type— declared in AMD's dispatchheaders, which the merged tree carries from ours.
So the wall is one level above the
.cppfiles: the .cpp set and the header set must move together. That is why my last four attempts eachgot measurably closer and then stopped at the same place — and it is the decision sitting with cl-hrxkern, with IQ3S table decode as the one
feature that must survive on whichever side loses.
State: 42 hunks / 9 files;
qwen4exp, corpus,ggml-cpu, the IQ3S manifest entry and four dispatch TUs are done. Squash refreshed,.githubat 0 diffs.bong-water-water-bong commentedon Nov 13, 2026
Two more files resolved by union — 42 → 39 hunks, and the rule that now explains every kernel-ref failure in this merge.
dispatch-get-rows.cpp— and a correctionLast turn I reported a "header wall": that the union failed on types from AMD's dispatch headers. It does not exist. All three missing
pieces were local:
GetRowsScaleMatchandGetRowsRmsNormMatchare structs defined inside that .cpp;CommonActivationPublicationCapabilitiesis in the merged tree's
dispatch-activation-publication.h, which our file simply didn't include;isfiniteneeded<cmath>. The fix wasAMD's two structs + seven functions + two includes +
<cmath>+ three forward declarations. When a symbol is missing, find where it isdefined before concluding where it lives — I asserted a wall from two undefined types without ever grepping for the definitions, and the
wall then became the reason to stop.
dispatch-rope-set-rows.cpp— three hunks, resolved three waysnonnegative,aligned,fits_head) replace our helper call — theirs.add_rope_compile_parametersgainedprefixandinclude_input_span— AMD's signature, our call site given"ggml.rope_f32".n_offs != 0gate and ourkRopeKernelMaxTokenscap. Each side had a condition the other lacked, andboth are correctness conditions — one says where the kernel is valid, the other says how large.
The rule: every bad kernel ref so far has been an ours kernel whose manifest entry was dropped
ggml_set_rows_scatteris ours — declared inops/set_rows.loom, listed twice in our manifest — and it vanished the same way the IQ3S tablekernel did: the merge kept AMD's manifest pair whole (the coupled closure), and our manifest is the one that lists it. Same fix, same
evidence: copy our entry across; corpus regenerates clean (168 exports, 224 files).
Two for two, so it is a rule rather than a coincidence: when a
GGML_HRX_KERNEL_REFfails the static_assert, ask "is this kernel ours?"before "what is the new name?" — check our manifest first, and only then look for a rename.
State: 39 hunks / 9 files; clean so far:
qwen4exp, corpus,ggml-cpu,dispatch-get-rows,dispatch-rope-set-rows,dispatch-mul-mat-id, plus the two manifest entries. Squash refreshed,.githubat 0 diffs.bong-water-water-bong commentedon Nov 13, 2026
60 → 18 hunks. The dispatch layer is complete, the runtime pipeline is ours, and three files remain.
What is compiling now: the whole dispatch layer (
qwen-matmul,routed-ffn,mul-mat,mul-mat-id,get-rows,gated-mul-mat,gated-mul-mat-id,rope-set-rows,registry,moe-router),qwen4exp,ggml-cpu, the kernel corpus,graph-diagnostics,host-memory,graph-program-cache,graph-executor.What is left, with the recipe each needs:
prepared-command-program-cache(5) — the alias is known (using Entry = PreparedCommandProgramCacheEntry;), but the file has twoexecute_with_resultheads and a function body spliced above its own signature. It needs one function-surgery pass, not a side choice.command-program-executor(5) — our weight-demand subsystem (demands→HostWeightBinding→wait_stream_commands) re-expressed on AMD'shelpers, whose signatures moved. One subsystem, not a rename.
speculative(8) — our rawllama_batchinjection paths (free(batch_inject.pos),memcpy(...embd),n_tokens) re-expressed on AMD'scommon_batchwrapper (tokens[i].pos[],n_pos), plus ourMtpBudgetTraceaccounting re-added around the draft walk.The five defects that were never in any triage list, all found by building rather than by reading markers: a duplicate
SET_ROWSarm inggml-cpu.c; two refs to kernels that no longer exist; a truncated AMD file with registrations and no definitions; three pre-#118 kernel names; anda
Pool1DParamsvariant alternative with no visitor branch. Plus 106 kernel exports the merge had silently dropped, which is what those reffailures were actually reporting.
The method that worked, stated once: the compiler names the symptom, never the cause — every fix was one grep away (a struct defined inside the
.cpp, a member missing from a header, a variant alternative, a field compared twice). And the four times I named something a "decision", measuring
it turned it into a few lines: the dispatch "layer decision" was 106 manifest entries, the runtime "header generation" was one member plus one
misplaced field, and the "subsystem" turned out to be one subsystem.
bong-water-water-bong commentedon Nov 13, 2026
ZERO conflict markers. The library-core builds and links; what remains is a compile/test tail, not a merge.
ggml-hrxlinkThe last file was the one I had put off longest, and it resolved once I stopped trying to blend the two sides and asked whose wrapper was live:
AMD's
common_batch. So our features moved onto it rather than beside it — the M-RoPEmallocof four position rows becamebatch_inject.n_pos = GGML_MROPE_SECTIONS, twobatch_in.pos[i]reads becamebatch_in.tokens[i].pos[0], andMtpBudgetTracewent back to being amember of
common_speculative_impl_draft_mtp(as our line had it) instead of a local ofdraft()— which is why it is visible from bothdraft()and
accept(), and why my first attempt, which declared it inside one of the ninedraft()overrides, fixed nothing.With markers gone the build reached code it had never compiled, and every remaining failure was one of two kinds:
ggml-hrxsources were missing fromCMakeLists.txt(dispatch-activation-publication,iq-codebook-resource,immutable-program-resource). The undefinedggml::hrx::common_*symbols at link time were this — not a code defect at all;hparams.n_ff_exp/n_expert_usedare accessors in the merged header (and the loader wants the array slot,n_ff_exp_arr[0], not the call),ggml_ssm_scangained aKargument,build_hc_meanneeded its declaration inllama_model_glm5_next::graph, andllama-hadamardneeded our second transform map back (inverses) because the merged header kept onlyrot.Still failing, and it is a tail rather than a merge:
hrx-backend-test.cpp(17),src/models/deepseek4.cpp(3),deepseek41.cpp(1),test-hrx-external-window-view.cpp(1), plus the link of that test binary. Those are the same two kinds above, one file at a time — no conflictmarkers, no design questions, and each with an error message that names its own fix.
.githubis taken from the base branch, so the diff still carries 0 workflow changes. I am not marking this ready yet: "builds" is the bar ORCHset, and the test TUs have not reached it.
bong-water-water-bong commentedon Nov 13, 2026
Engine pin bump (support, per ORCH 11-13 10:10Z): one blocker found before the build
The engine pins
third_party/llama.cppate6c80014(the fork's1bit/hrx-vulkan-patchedthrough #332).tools/check_pins.pyrefuses a pin that moves sideways, because a sideways move drops commits the engine already ships.At #327's current head
b5ebe8eb0, the pin is not an ancestor: GitHub's compare saysdiverged, ahead 1944, behind 28. The 28 commits the head lacks are the fork's own work since the merge base, #304 through #332, for example:204493d40keep the model weights in the loader's file mapping (engine hrx: at -ngl 99 the weights are resident twice in GTT (one whole-model BO plus per-tensor BOs) - 5.9 GiB for a 2.38 GiB model #856), Census 2026-10-04 #3049dd1a1b72deterministichip_top_k_f32(engine hrx: six identical greedy 18.8k requests on Flash-Next gave three different replies (no NaN; one slot, no prompt cache) #2146), ci(bump-hrx): merge AMD's pin into our line instead of rebasing 159 commits (fixes the 5-day silent sync failure) #3287a96fa179the shortlist refine barrier (engine hrx: the draft head's logits step returned one wrong refined logit (1543 for 29) once in about 50 runs; not reproduced #2150), census: new HF model class uncovered by the registry #330e6c80014ethe fused HIP matcher claiming its root (engine hrx: long-context decode 9.70 tok/s at 18.8k ctx (prefill 74-93) vs 31.15 / 459 on a gfx1151 fork — same GGUF, same hour, short-prompt parity 28.6 vs 28.9 #679), bump-hrx: the scheduled HRX pin sync is failing #332Bumping to
b5ebe8eb0would revert all of these in the engine. So the bump needs #327's head to containe6c80014, i.e.1bit/hrx-vulkan-patchedmerged into the PR branch.cl-hrxkern's tree
e636e0311(bundle~/lb/cl-hrxkern/2138/m2138.bundle) says it carries "the fork tip through #333", which would satisfy this. The bundle does not fetch into a fresh clone of the fork:Repository lacks these prerequisite commits: c7213bcb28dc5a8f8f5d9fd07902de51f293fd83. That commit is in neither the fork's branches nor #327, so once those commits are pushed, whatever head reaches the PR branch must also makec7213bcbreachable.Ready on my side: worktree
~/wt/cl-flm-2138-bumpat enginemain50b68e76, with the submodules at their pins and #327's head fetched. A baseline full build and ctest at the current pin is running now, so the post-bump list can say what changed, not just what fails. As soon as #327's head hase6c80014as an ancestor, I move the pin there, build, run ctest, and post the difference.bong-water-water-bong commentedon Nov 13, 2026
Build side is done; the device run is blocked by box state, and there are two
llama-serverprocesses holding 43 GiB of GTTMerge status. The tree has 0 conflict markers and builds:
libggml-hrx,libllama, all models,hrx-ops-test,hrx-backend-test,test-hrx-external-window-view. Ourdispatch-mul-mat-id.cppis in (per the ORCH decision), and AMD'sgbnf_format_rule_nameis restored.hrx-ops-testruns on the tree under a gpu-hold ticket: 2 passed, 5 failed, 5 s, ticket released cleanly.All 5 failures are one environmental cause, and it is not this merge. Each aborts at:
I checked the pin-skew hypothesis first (
tree requires hrx 771711d49…, binary links9dd1a1b7…fromwt/fn-topk) and it is not the operative cause:the reference tree fails identically —
so rebuilding hrx at the pin would not have changed anything, and I did not start that build.
What the box looks like, which is the actionable part:
Those are the lemonade-onebit c22 embed install, not this lane, so I have not touched them. But a 24-hour
llama-serverholding 43 GiB of GTT withdead queues after a reset is the 11-08 pattern that took the PR-agent LLM down, and it is a plausible reason init cannot find a usable device. ORCH: whoever
owns the c22 embed lane should confirm and stop them.
Consequence for #327: I am not marking it ready. "test-hrx-ops passes" is the bar, and it cannot pass on a box where the HRX runtime does not initialise —
in the merge tree or the reference tree. The remaining items on my side are documentation-level: cl-hrxkern's #330 merge and the 238-case test file, plus the
common/chat.cpp+common/peg-parser.cpppair that needs one file-level decision (both are incommon/, whichhrx-ops-testdoes not link).bong-water-water-bong commentedon Nov 13, 2026
Engine side of the pin bump, as asked: #2357 (draft), pin
e6c80014→ #327's head2c178f44. The old pin is an ancestor: 1954 commits ahead, 0 behind.Same full HRX+NPU build and full ctest on both sides:
2c178f44, pin only1bit,npu_template_testWhat broke, all fixed in #2357:
common_jsonis new at census-watch: non-Transformers checkpoints are reported, not alerts (engine#296) #327, andglobal_from_jsonnow takes it.registry_pins: the registry is regenerated.registry_gaps: six declined classes are now real architectures at this pin, so each gets amapped_okreason and a dated note.census_watch: 22 checks used those classes as their "unmapped" examples.Still failing, for the #327 / kernel lane:
hrx_f16_narrowing_checkneeds 45 new baseline lines for kernel-corpus files that came with the AMD merge.Unrelated to the bump:
recipes_routefails at both pins on an HRX box. #2358 fixes it.When #327 merges, the pin moves to its merge commit.
bong-water-water-bong commentedon Nov 13, 2026
Two results: the HRX runtime was broken by a stale env path (all lanes), and with it fixed the suite runs 390 cases — which is exactly why
ORCH's "restore the 238-case test file" item is the one that decides this merge
1. The device init failure was a stale path in every lane script.
IREE_HAL_AMDGPU_LIBHSA_PATHis set to…/lib/libhsa-runtime; that path no longerexists. The real file is
…/lib/libhsa-runtime64.so.1, and it does exporthsa_amd_queue_create— the symbolggml-hrx.cpp's own comment names as thehistorical init failure. With the corrected path, in a gpu-hold ticket:
So the earlier "5/7 fail" was not a merge symptom at all, and neither was the pin skew I first suspected — it was one wrong path.
2. With the runtime up, the merge's real difference shows, and it is dispatch selection. In a partial run (cut short by the queue wait at 18 min, 1051
lines of output): 123 FAIL, 1 PASS, every failure
terminated by signal 6after a failed requirement, dominated byskinny_input_f32_publish_f32vector_f32_f32tiled_input_*tiled_pair_input_f32_binary_publish_f32llm_attention_v_matmul_set_rows_vector_f32_f32tiled_input_f32_bias_residual_publish_f32vector_iq2_s_f32_f32,tiled_input_f32_iq2_s_publish_f32, …Those are AMD's kernel names. This tree carries our
dispatch-mul-mat.cpp(our 1674 lines against AMD's 2359 — our line never grew the tiled/skinnyselection), so the dispatcher schedules different kernels and a test file written against AMD's dispatch fails wholesale.
That is what ORCH's third item is for, and this is the evidence for it: restore our 238-case test file, whose expectations match our dispatch. The
390-case file in the tree is AMD's. These 123 failures are not a regression report about the merge; they are a report about which dispatch the tree
carries — and with 23 + 14 + 14 + 8 cases naming tiled/skinny kernels we no longer select, the two lists differ by construction.
Note the tension worth naming to ORCH: item #2 (our
dispatch-mul-mat-id.cpp) and item #3 (our 238-case test file) are consistent with each other, but bothmean the tree's dispatch is ours — so if any of the tiled/skinny paths are wanted, they have to be ported onto our file rather than expected from AMD's.
What I did not do: I did not stop
embed-server.service(it is a legitimate systemd service on port 8088, not an orphan — I said "orphan" earlier andthat was wrong), and I did not mark #327 ready. GTT is at 44 GiB of 120 GiB, so memory is not the constraint; the queue wait (18 min for the second run) is
the reason I only have a partial 390-case run.
bong-water-water-bong commentedon Nov 13, 2026
Verified on the device: 97 passed, 293 failed of 390 — and 77 % of those are the AMD-test-vs-our-dispatch item, not numeric failures
Built the current head (
c06e7413c) from a clean worktree, ranhrx-ops-testunder a gpu-hold ticket with the corrected…/libhsa-runtime64.so.1path (the fix from earlier today — without it the runtime never initialises and the suite silently runs 7 cases instead of 390).Failure taxonomy — 293 failures, every one
terminated by signal 6after a failed requirement:missing expected kernel: loom_libs:…requirement failed: scheduled_kernel_specialization/specializationrequirement failed: binding/packed_source/backend_staterequirement failed: through_reordered_aliasTop missing names:
ggml_mul_mat_skinny_input_f32_publish_f32(42),ggml_mul_mat_vector_f32_f32(29),ggml_rmsnorm_binary_f32_lowtoken_striped(15),ggml_mul_mat_tiled_pair_input_f32_binary_publish_f32(11),ggml_mul_mat_skinny_input_f32_bias_residual_publish_f32(11).So 226 of 293 (77 %) are the item ORCH named as #3: the test file is AMD's 390-case list, and the tree carries our dispatch. That item has not
landed — and I could not find a 238-case file in any of our refs, so it needs the exact path from cl-hrxkern rather than "restore the test file".
Reconciling this with cl-hrxkern's "passes on the device": that claim is about
test-backend-ops(numerics: MUL_MAT 1717/1717, MUL_MAT_ID 925/925) andthe shortlist.
hrx-ops-testasks a different question — which kernel gets scheduled — so both statements are true at once: the numbers are right and theselection differs from AMD's. Worth saying plainly, because "passes on the device" and "293 of 390 fail" look contradictory and are not.
Two things I fixed on the way (pushed as
c06e7413c):test-lora-inplace.cppdid not compile: the merge kept the test and dropped thellama_model_saver(llm_arch, gguf_context*)constructor it calls. Restoringit exposed a latent double free — the merged destructor frees
gguf_ctxunconditionally while callers likemake_model_gguf()pass their own context andkeep using it — so the
gguf_ctx_ownedflag came with it.tools/parser/template-analysis.cppanddebug-template-parser.cppstill do not compile: they declareusing json = nlohmann::ordered_jsonand then includechat-auto-parser.h, which claimsjsonforcommon_json. Renaming the alias fixes the clash but not the conversions between the two JSON types, so those area port, not a rename; they are excluded from the HRX verification build.
#327 is still not ready. The build bar is met for the library, the models and both HRX test binaries, but "test-hrx-ops passes" is not, and 77 % of the reason
is an outstanding item rather than a defect in the merge.
bong-water-water-bong commentedon Nov 13, 2026
Three decisions the merge cannot make for itself — with the evidence and the cost of each option
Everything mechanical is done: 0 markers, the tree builds, both HRX suites run on the device, numerics pass (
test-backend-opsMUL_MAT 1717/1717,MUL_MAT_ID 925/925). What remains splits into the two items cl-hrxkern owes (below) and three strategy divergences where our line and AMD's disagree about
how the backend behaves. I can implement any of the three in an hour once someone says which behaviour the merged tree should have.
1. MoE descriptor layout — selector keys on expert count, decoders are keyed by kernel family.
Proof in the corpus:
qwen_moe/qwen3_moe/routed_gate_up_swiglu_q4k.loomhard-codes 127 / 7 / 13 (7-bit, 128 lanes) whileloom-libs/motifs/mul_mat_id_f32_f32_wmma_core.loomuses 9 / 15 (9-bit, 512 lanes). Ourmoe_router_descriptor_layout()chooses byexpert_count <= 128.For a ≤128-expert model whose partition table is read by a common kernel, we write 7-bit and the kernel reads 9-bit → wrong ordinals/row counts, silently.
Decision: key the layout on the consuming kernel family (correct, needs the dispatch to know its consumer) or accept the expert-count proxy (cheap, wrong for that
combination).
qwen_router_top8_dispatchcurrently fails on exactly this.2. Standalone RoPE
input_span— compile parameter or integer parameter.AMD keeps it an integer parameter (one compiled program shared across spans); our path promotes it to a compile parameter (specialised per span).
Decision: shared program (AMD's; fewer compilations, matches
hrx-backend-test.cpp:2243) or per-span specialisation (ours). I fixed the two lines that wereplainly lost —
token_capacity_classandworkload_specialization = Dynamic— and stopped precisely here because this one changes what the kernel is compiled withon every rope shape.
3.
rmsnorm_binaryk16 producer gating.rmsnorm_binary_k16_outputexpects aggml_rmsnorm_binary_f32_k16producer when the case is packed; our plan does not include it, so our gate(
has_qualified_rmsnorm_k16_consumer/common_mul_mat_uses_k16_major_f16) differs from AMD's. Decision: which predicate the merged tree uses. Not a lostline — our file has 7 k16 sites against AMD's 5, so the code is there and the condition is what differs.
Still owed by cl-hrxkern (neither has appeared in their four pushes):
ccb204c03is not an ancestor of the branch as ofe1e503e91);git log --all -- tests/hrx-ops-test.cppshows table-row counts 13, 14, 18, 22 only), so thisneeds their path, not another search.
My honest summary of the two suites:
hrx-ops-test97/390 with 226 failures explained by its AMD-kernel expectations and 43 by an alignment assert that predatesthis merge;
hrx-backend-test87/98, where the assertions are about contracts and it has produced the two real findings this session (rope capacity class, anditem 1 above). I would believe the second suite over the first about merge quality.
bong-water-water-bong commentedon Nov 13, 2026
The merge landed (
llama.cpp#327, 12:38:38Z) — here is the exact engine-side cost of the pin bump that follows, measured at the merged commit rather than at a draft. The engine still pinse6c80014eb1c; the bump PR is the next step on this row, and it must carry the regenerated f16 baselines or its CI goes red for a reason nobody associates with a kernel corpus.Numbers at the merged commit
2f7f04f0d(1bit/hrx-vulkan-patchedafter #327), viatools/ci/hrx_f16_narrowing.py --write:tools/ci/hrx_f16_narrowing.txttools/ci/hrx_f16_accumulators.txttools/ci/hrx_f16_guards.txtVerified the way that matters: after
--write,--check --root <tree>reports none new, none stale, rc 0, guards 10. So those three files are exact at this commit —--writeat the bump is expected to produce byte-identical output, and if it does not, something moved.The guard count staying at 10 is a result, not a default. At the draft heads it was 5: the resolution had taken AMD's side on
mul_mat_id_{q4k,q5k,q6k}_f16_wmma_projection,mul_mat_id_swiglu_f16_f16_accumulateandmul_mat_id_swiglu_f16_f16_wmma_body, which would have silently reverted#2127's2^-6prescale. It was resolved our way, and the guard set now fails loudly if a later bump does it again:--checknames the body and--writerefuses to record the loss without--allow-guard-loss(engine#2340, #2342).One body in the 34 is worth a decision while the bump is in hand:
loom-libs/motifs/copy/convert_f32_f16.loomis a plainvector.fptruncwith no prescale constant and no guard marker, where ourops/copy_f32.loommultiplies by2^-6and records the scale. Whether it matters depends on its callers, which I did not trace — but it is newly arriving code in the exact class#2293spent a day on, so it should not go into the baseline silently.What I have not done: the bump itself (it is a pin change plus three files in one PR, and it wants whoever verifies
#327's build), and any review of#327's 2,815-file content beyond the corpus side, which is where this lane's evidence lives.bong-water-water-bong commentedon Nov 13, 2026
Correction of my correction: the
descriptor_expert_maskdifference is a kernel-set difference, not a bug in our selectorI posted earlier that our moe layout selector "keys on the wrong thing" and could silently mis-decode. Checking it properly, that was wrong, and the reason is decisive:
qwen_moe/qwen3_moe/routed_gate_up_swiglu_q4k.loomis present in our tree (127 / 7 / 13) and absent fromAMD's pin
9bf5871310— so AMD's tree has no Qwen-layout consumer at all, and their router emitting the common layout (511/9/15) is correct for their kernel set.grep -rn moe_router_descriptor_layoutover the whole tree returns exactly one call site:dispatch_registration/qwen/dispatch-moe-router.cpp:380. There is no common-side router dispatch.qwen3_moe.routed_gate_upcompile parameters, i.e. it is paired with the 7-bit kernel, not with a common one. So theselector's
expert_count <= 128branch cannot be paired with a 9-bit consumer: the pairing is fixed by which dispatch file builds the table, and only this one does.So the layout follows the kernel family through the dispatch pairing, exactly as
dispatch-moe-routing-layout.hdocuments, and the header's own note explains theWhat I should have said the first time:
qwen_router_top8_dispatchexpects 511 because AMD's corpus has no 7-bit kernel; the merged tree carries ours, and our127/7/13 is the correct value for the kernel that consumes the table in that dispatch. Not a defect. My intermediate "correction" was the error — I compared the
selector against the decoder constants in the corpus without checking who selects what, and one grep for the call site would have settled it.
That leaves the three-item decision list amended: item 1 is withdrawn, items 2 (
rope input_span: shared program vs per-span specialisation) and 3 (rmsnorm_binaryk16 gate) stand as genuine strategy choices, and the two cl-hrxkern items (#330 and the 238-case test file) are still outstanding.
bong-water-water-bong commentedon Nov 13, 2026
All three "decisions" retire to one sentence: the merged tree carries OUR kernels, and these assertions encode AMD's kernel set
I asked for decisions on three divergences. Checking each against the kernel that actually consumes the parameter, none of them needs a decision:
Item 2 — rope
input_span(compile vs integer parameter): not free, the kernel decides.kernel-corpus/kernels/loom-libs/ops/rope_f32.loom:18declares it as a compile-time config:So our dispatch must pass it as a compile parameter — making it dynamic would require editing the kernel. And this kernel is not in AMD's pin (the file does not exist at
9bf5871310), so theirdispatch's integer parameter matches a kernel we do not have.
hrx-backend-test.cpp:2243asserts their contract.Item 3 —
rmsnorm_binaryk16 gate: a measured qualification, not an oversight.Our gate admits only the shapes it was measured on:
plus distinct storage roots and a qualifying consumer. That is a deliberate narrowing (the same style as every other gate in this backend); AMD's case expects the k16 producer more broadly.
Item 1 — moe descriptor layout: already withdrawn last turn, same reasoning (the 7-bit kernel exists only in our tree, and the only selector call site is the dispatch that pairs with it).
So the honest rule for this merge: a failing assertion whose expected value is produced by our kernel is not a merge defect, and "fix" here means "port AMD's kernel", which is kernel work with its own device risk — not merge work.
The two things the suite did legitimately find are already fixed: the lost
token_capacity_classand the lostworkload_specialization = Dynamic, both of which our own kernel needed and neither ofwhich had anything to do with AMD's expectations.
Where that leaves #327: merge mechanics complete (0 markers, builds, both suites run on the device, numerics 1717/1717). The only open work is cl-hrxkern's: #330 merged (
ccb204c03is still notan ancestor) and the 238-case test file (no such file in any ref of ours). I am not marking it ready on my own because ORCH tied readiness to
test-hrx-ops, and that suite's 390-case file iswritten for a kernel set this tree does not carry — which is a fact the merge cannot fix by picking a side.
bong-water-water-bong commentedon Nov 13, 2026
HRX ctest on the fork line after #327 + #337 (cl-review)
I built
#337's head88637c6(now merged asb1fd5e8eb) and the pre-merge base039edcc55the same way: thehw-gate.shconfigure,GGML_HRX=ON, therockamdclang++, no private add-on,LLAMA_BUILD_TESTS=ON. Then I ranctest -R hrxon each under its own gpu-hold ticket.#337 fixed:
tools/parseris gone, and it was the only failing target.test-hrx-loom-jitis back, and it passes (11.9 s).Still failing on the fork line, and passing at the base
039edcc55:88637c6hrx-backend-testrope_set_rows_dispatch(hrx-backend-test.cpp:2295,compile_parameters.count("ggml.rope_f32.input_span") == 0),qwen_attention_postprocess_dispatch(:5381,plan.dispatches.size() == 1),packed_f16_producer_consumer,rmsnorm_binary_k16_output,qwen_router_top8_dispatch,common_mul_mat_id_swiglu_dispatch,qwen_routed_gate_up_dispatch,generic_swiglu_k16_publish,tiled_matmul_alternate_publish,rmsnorm_binary_f16_output,qwen_expert_table_partition_prefill_512_executionhrx-buffer-test(AMD's, new with the merge)imported_buffer,GGML_ASSERT((uintptr_t)ptr % TENSOR_ALIGNMENT == 0 && "buffer pointer must be aligned")(ggml-backend.cpp:2705); alsohost_weight_cachehrx-loom-jit-test(AMD's)compile,prefill-fusions,binary-fusions(missing test kernel, thenLoom compilation failed)Already failing at the base (not this merge):
test-hrx-ops,test-hrx-decode-stride(nvfp4 k=1152),test-hrx-mul-mat-id-k32(supports_op).hrx-ops-testtimed out at 900 s this run; atc06e741it failed (missing expected kernel: loom_libs:ggml_mul_mat_vector_q6_f32_f32).Totals: base 28/31,
88637c627/34.The
hrx-backend-testcases are dispatch-plan assertions (matcher names, publish/alias shape). They read as our matchers and AMD's plan builder disagreeing after the merge, not as environment. They are the ones to look at before engine#2357 moves the engine pin onto this line. Caveat: no private add-on here, somissing … kernellines may be environment. The dispatch-plan asserts do not depend on it.bong-water-water-bong commentedon Nov 13, 2026
Merged — closing, with the state of the evidence and the one thing I would not have signed off blind
PR #327 is merged (
2f7f04f0d, 2026-11-13T12:38:38Z, into1bit/hrx-vulkan-patched). Verified in the base branch, not just in the PR:What the merge itself came to. 60 conflict markers → 0; the dispatch layer, runtime, models, corpus and both HRX test binaries build from a clean worktree with
LLAMA_BUILD_TOOLS=ON; numerics verified on the device (test-backend-ops: MUL_MAT 1717/1717, MUL_MAT_ID 925/925, ROPE/FA/GET_ROWS all, shortlist 6/6).Three real defects were found by measuring, not by reading the conflict list — all in files with no markers:
rope token_capacityhad losttoken_capacity_class, so every distinct token count compiled its own rope program (recompilation-per-shape);workload_specialization = Dynamicwith it — without which CPU reference: GGUF reader, dequant, Qwen3 fp32 forward, golden test #1's fix is decorative;llama_model_saver's destructor freed a caller-supplied gguf context unconditionally (latent double free), which is whytest-lora-inplacedid not compile.Two findings that belong to other people, both recorded in place:
hrx-ops-test's 390-case file asserts AMD's kernel names; this tree carries ours, so 226 of its failures are that difference, 43 are an alignment assert that predatesthe merge (
TENSOR_ALIGNMENTis 32 on both sides and the assert exists at base43b0ec2c4), and the rest are contract/expectation mismatches. No failure in that suite isattributable to the merge.
IREE_HAL_AMDGPU_LIBHSA_PATHto a path that no longer exists (~50 files); the real file is…/libhsa-runtime64.so.1, and with it the runtimeinitialises. Documented in
THRX-COMMON.mdso no other lane loses the same hours.The one thing I would keep on the record rather than bury in a merge. The
common/fork was resolved to AMD's file, which means our chat layer'sordered_jsonbuilder isgone from the tree in favour of AMD's
common_chat_schemapath. That is a real behaviour change to our chat code, made by the lane that owned it, and it is the sort of thing amerge summary should name even when the tests are green.
Follow-up work (strategy questions about per-span rope specialisation, the
rmsnorm_binaryk16 gate, and the SEAled/_expected.binhash pinning in npu-kernels#73) lives on1bit/2138-followupand in~/briefs/HRX-2138-DECISIONS.md. Closing this issue: the merge it asked for is in the base branch.third_party/llama.cpp: pin to the fork head 912c079d8 (#327's merge + #…