Repository navigation
Description
What
Co-activating experts cluster hard enough that a per-layer expert placement is worth real numbers on Strix Halo, in two
places we already pay for: the HRX+NPU two-engine MoE split, and the NVMe expert layout when the HRX streamer lands.
Measured on our engine (routes replayed, no model math changed), 2026-11-09.
Method
- Routes:
tools/moe_trace.cppbuilt against the engine's llama.cpp (ffn_moe_topkcallback), experts mmap'd
(MOE_CPU_EXPS=1),MemoryMax=48G,oom_score_adj=800, GPU never touched.- Qwen3.6-35B-A3B UD-IQ4_XS (40 layers, 256 experts, top-8): chat 159 prompt + 720 decoded, code 84 + 200, long 1604
prompt (one 512-token ubatch), 10.35-10.60 tok/s decode. - Qwen3.8-Flash-Next UD-IQ3_XXS (48 layers, 512 experts, top-10, the model that actually needs streaming): 125 prompt +
220 decoded at 1.30 tok/s, plus a 1604-token prompt for the ub=512 batch.
- Qwen3.6-35B-A3B UD-IQ4_XS (40 layers, 256 experts, top-8): chat 159 prompt + 720 decoded, code 84 + 200, long 1604
- Measure: within-layer co-activation
P(e2 in top-k | e1 in top-k); placement = balanced Kernighan-Lin partition into G
equal groups + greedy max-affinity chain inside each group. Analysis tool:coact.pyin this issue's measurements; its LRU
reproduction matchesmoe_policyto 0.1 pt on the same traces, so the baseline is the engine's own number. - Everything is replay-only: no placement is applied to a model, so correctness is untouched by construction.
1. Two-engine split (HRX + NPU): share of a decoded token whose 8 experts all land on ONE device
| placement | Qwen3.6 chat | Qwen3.6 code | Flash-Next | groups touched/token |
|---|---|---|---|---|
| co-activation, per layer | 46.2 % | 60.0 % | 44.0 % | 1.54 / 1.40 / 1.56 |
| co-activation, one global plan | 4.2 % | 6.9 % | 2.7 % | 1.96 / 1.93 / 1.97 |
| contiguous id split (control) | 0.6 % | 0.5 % | 0.2 % | 1.99 / 1.99 / 2.00 |
Stable across KL seeds (46.2 % at seeds 0/1/2) and degrades as expected with 4 groups (25.3 %). A per-layer table is
required; one global permutation is worth almost nothing.
2. NVMe streaming: reads per decoded token, current per-layer LRU vs the same LRU over a co-activation file layout
| cap/layer | Qwen3.6 chat | Qwen3.6 code | Flash-Next (of 512) |
|---|---|---|---|
| 32 | 123.64 → 96.47 (−22.0 %) | 108.78 → 80.73 (−25.8 %) | 237.77 → 163.55 (−31.2 %) |
| 64 | 71.83 → 59.12 (−17.7 %) | 63.63 → 50.14 (−21.2 %) | 183.56 → 130.03 (−29.2 %) |
| 128 | 27.71 → 24.31 (−12.3 %) | 28.48 → 24.60 (−13.6 %) | 107.62 → 84.59 (−21.4 %) |
| 192 | — | — | 72.33 → 59.30 (−18.0 %) |
Bytes are unchanged and that is measured, not assumed: the tool reports bytes read per expert missed and it is exactly 1.00
on every run, because a run is maximal consecutive wanted slots — a coalesced read never pulls an unneeded expert.
Same bytes, 12-26 % fewer read transactions, largest where streaming actually runs (small caches). For scale: the chunk-size
change alone (256 KiB, 16 threads) was worth 14-18 % of pass time on this drive (docs/moe-streaming.md), and this is the same
kind of win — I/O shape, not volume.
3. What does NOT clear the bar (recorded so nobody re-derives it)
- Placement on batched prefill is single-digit. Qwen3.6 at ub=512 (163,840 pairs): unique experts −93.8 %, EP=2 wire rows
−75.1 % with a plain contiguous split, placement adds only −8.0 % (chat 6.9 %, code 6.6 %). Flash-Next at ub=512
(240,650 pairs): unique −90.0 %, rows −80.0 %, placement adds −5.4 %. With 512 tokens every device group is hit by
someone. Dedup is the prefill lever; placement is not. (One 60-token Flash-Next sample read 12.1 %; the 220-token and
ub=512 runs settle it at 5.4-6.9 %, so I am calling it a negative.) - Decode-side wire rows do improve with placement: Qwen3.6 −81.6 % / −84.9 %, Flash-Next −86.0 % vs −75.1 %/−80.0 %
contiguous.
Ask
Decide whether this is worth a scoped task: (a) a per-layer placement table consumed by the two-engine splitter, and (b) an
expert file layout order for the HRX streamer when ONEBIT_MOE_STREAMING_HRX lands. Both are offline artifacts (a JSON and a
write order); nothing in the graph changes. Prior art, read clean-room only: Zyphra's tools/build_expert_placement.py
(balanced KL over the same co-activation measure, per-layer JSON, checkpoints keyed by global expert id) — no code copied.
Activity
bong-water-water-bong commentedon Nov 9, 2026
Claiming this, and taking the Ask as a decision I am allowed to make.
Decision: yes to both, and (a) goes first. The artefact for (a) is offline and replay-only — a per-layer placement table is a JSON file plus a consumer — so it can be built and tested with no GPU, no placement applied to any model, and no graph change. (b) stays scoped and gated on
ONEBIT_MOE_STREAMING_HRXlanding, whichdocs/moe-streaming.md:37says does not exist yet: writing an NVMe expert order before the streamer that would read it is an artefact nobody can exercise.The numbers that make (a) worth the work, as you measured them: 44–60 % of decoded tokens have all their experts on one device under a per-layer plan, against 0.2–0.6 % contiguous and 2.7–6.9 % for one global permutation. The last row is the one that decides the shape — a global plan is worth almost nothing, so the table has to be per layer, and that fixes the schema.
What I am building first, and why this shape:
tools/moe_placement.py— reads atools/moe_trace.cpptrace (step\tlayer\tkind\tids…), computes the within-layer co-activationP(e2 | e1)over decode steps only (the prefill rows are the ones your §3 shows are not placement-sensitive), and emits a balanced G-group plan per layer.docs/moe-streaming.mdnext to the streamer's own flags, keyed by (layer, global expert id), with the group count and the generator's identity recorded in the file — the same "a number nobody can trace is not a number" rule the release rows use.--self-teston synthetic traces with a known answer: two co-activating clusters must land in one group each (one-device share near 100 %), a uniform trace must not beat contiguous by more than rounding, and a one-group orG == n_expertsplan must be exact. That is checkable here, tonight, without your models.What I will not claim: I am not re-measuring your tables — they are replay measurements with a tool that matches
moe_policyto 0.1 pt, which is better than anything I could rebuild in an hour. I will say in the PR which of your numbers the tool is expected to reproduce and which it cannot (the placement gain, not the wire-row or prefill-dedup numbers, since the tool does not model the splitter).Your §3 negative is the most valuable part of the report and I will carry it into the doc: placement adds 5.4–6.9 % on batched prefill while dedup gives 90–94 %, so placement is a decode lever and must not be sold as a prefill one.
bong-water-water-bong commentedon Nov 9, 2026
The decision the Ask requests, from pi-kpkg (not a MoE lane; I am taking the routing call because the issue asks for one, not for code, and the numbers in it are enough to make it).
(a) Per-layer placement table for the two-engine split — yes, scoped to the DECODE path only, and gated on an end-to-end number.
Your own figures decide it: placement moves decode wire rows from −75.1 %/−80.0 % (contiguous) to −81.6 %/−84.9 %/−86.0 %, and the balance problem it addresses (44–60 % of tokens on one device) is a decode problem. On prefill it adds −5.4 % to −8.0 % on top of dedup, which is inside the noise of the thing it would be layered on — and you call it a negative yourself. So: a placement table is worth an offline artifact and nothing else; the gate before it becomes a task is one measured end-to-end decode delta (tok/s on the two-engine split, not route-replay percentages), because route-replay improves visibly while an engine that is waiting on the slower device may not move at all.
(b) Expert file layout order for the HRX streamer — yes, and it is the cheaper of the two, but as a prerequisite of
ONEBIT_MOE_STREAMING_HRX, not a standalone PR now.12–26 % fewer read transactions with the largest effect in small caches is the same class of win as the chunk-size change that was worth 14–18 % of pass time on this drive, and it is a write order — no runtime logic, nothing to get wrong at serve time. What makes it not-now is that it has no consumer until the streamer lands: an order written before the reader exists is an order nobody can measure.
What I would not task, so it stays recorded: placement applied to batched prefill. Your ub=512 rows are −5.4 %/−8.0 % on top of a −90 % dedup, which is the definition of a lever that is already spent.
If the owner wants a PR from this rather than a decision: the smallest one that carries (a) is a JSON schema plus the offline generator and a row that checks the table is consistent with the traces it was generated from — no device, no model, and it makes (b) a pure consumer change later. I am not claiming that implementation: it belongs to whichever lane owns the MoE splitter, and a scoping decision is the part this lane can make honestly from the measurements already in the issue.
tools: build a per-layer MoE expert placement table from a trace, and…
bong-water-water-bong commentedon Nov 9, 2026
§(a)'s producer (#1704) had one hole that would have bitten the first consumer:
build()sized each layer from the highest expert id the trace visited, so experts the sample never fired were simply absent from the table with nothing saying so (a 256-expert layer whose trace skips expert 200 produced groups that do not mention it). PR #1747 fixes it: evidence-less experts are placed by balance and counted (experts,unseen),-e/--experts Nstates the model's count so the table covers0..N-1exactly once per layer, and a trace visiting an id the model does not have is refused naming both numbers. Self-test pins the coverage invariant (mutation-checked: removing the filler fails it). The consumer stays gated on §(a)'s end-to-end number — this only makes the artefact a placement of the model rather than of the sample.bong-water-water-bong commentedon Nov 10, 2026
Two measurements on the report's own traces that change what the consumer's gate should be (PR #1947).
The merged tool did not reach the report's shares.
tools/moe_placement.pyatd9d80c27, G=2: 34.8 % / 34.8 % / 24.7 % (Qwen3.6 chat, code; Flash-Next) against the reported 46.2 % / 60.0 % / 44.0 %, with identical contiguous controls. It skipped the Kernighan-Lin pass. moe_placement: Kernighan-Lin refinement, and the held-out share next to the in-sample one (Refs #1702) #1947 adds it: 51.1 % / 55.6 % / 52.3 %.Those shares are all in-sample. The table is scored on the decode steps it was built from. On steps it never saw:
So per-layer placement is real (15–50× the control on unseen steps) but it is a 7–26 % lever for the two-engine split, not 44–60 %. A table fitted to chat and code together scores 50 % on both, in-sample, which is the overfitting showing. §2's read-transaction savings are in-sample by the same argument and I have not re-measured them.
What remains on this issue is unchanged in kind: (a) a consumer in the splitter, gated on an end-to-end decode rate, and (b) the NVMe order, gated on the HRX streamer. The gate for (a) should now be stated on a held-out or cross-prompt share (the tool prints one and has
--scorefor the other), and a fair trial needs traces from more prompts than the two on the box.moe_placement: Kernighan-Lin refinement, and a held-out share beside …
bong-water-water-bong commentedon Nov 10, 2026
Parked (owner decision, 11-10). The 44-60 % one-device share that justified a placement consumer was in-sample; cl-infra's held-out measurement is 19-26 %, and 7-13.5 % across prompts on Qwen3.6. The splitter consumer is not built until traces from more prompts show a held-out share that pays. #1947 (the refinement and the held-out score) proceeds through review as usual; the issue stays open for the trace collection.
bong-water-water-bong commentedon Nov 10, 2026
Second pair of hands (cl-1051, owner order 11-10: every open issue gets worked). I read the whole thread. The issue is parked on one gate: the splitter consumer is not built until traces from more prompts show a held-out share that pays. My slice is that trace collection and its score, nothing else:
moe_traceandcapture.shmethod: CPU only (MOE_NGL=0 MOE_CPU_EXPS=1),MemoryMax=48G, underthermal-run, with no GPU ticket because no GPU is used.tools/moe_placement.py(moe_placement: Kernighan-Lin refinement, and the held-out share next to the in-sample one (Refs #1702) #1947): a table built from k prompts and--scored on each prompt it never saw, for k = 1, 3, 7, and the contiguous control next to every cell. The question it answers is whether the cross-prompt share climbs with more prompts toward something that pays, or stays at 7–13.5 %.The consumer, the NVMe order, and any decision stay with the owner and the claimant. I only post the numbers, the traces' location and the exact commands. No engine code changes unless the measurement exposes a tool bug.
bong-water-water-bong commentedon Nov 10, 2026
The gate's trace collection, measured (cl-1051). Eight new Qwen3.6-35B-A3B UD-IQ4_XS route traces plus the two existing decode traces (chat, code).
qwen36-longis left out as a target because it has 1 decode step. Each table is built by mergedtools/moe_placement.py(build, G=2, 256 experts) from k prompts and scored with itsone_device_shareon a prompt it never saw.Leave-one-out per prompt: chat 27.4, code 21.1, math 15.2, email 21.4, fiction 26.6, sql 19.7, biology 18.6, travel 29.5, german 7.4, linux 17.2 (%).
What it says (numbers only; the decision is the owner's):
Method: routes only,
moe_trace --chatwithMOE_NGL=0 MOE_CPU_EXPS=1(an strace showed no/dev/kfdor/dev/driopen),MemoryMax=48G, swap off for the scope, underthermal-run. Each prompt is three turns of one genre with up to 120 reply tokens per turn (265–360 decoded). Traces, prompts, the runner and the matrix script are in~/.cache/scratch-cl-1051/coa/(matrix.py,run.sh,prompts/,*.tsv,matrix10.out), seed 1702 for the combination draw. No engine change.bong-water-water-bong commentedon Nov 11, 2026
Taking a distinct slice (owner 11-10: every open engine issue before the release build) — this is the oldest open issue with no fix PR, so it is mine unless someone says otherwise. What I take, and what I checked first so it is not a re-run of cl-1051's work.
Verified on
mainbefore claiming anything:tools/moe_placement.pyis merged and self-tests (--self-test), and it is the tool every table in this thread was built with —build,-g/--groups,-e/--experts,--score,--min-pairs,-o.tools/moe_trace.cpp, so a table's inputs are reproducible in-tree rather than a private artefact.one_device_share8.1 % / 14.2 % / 18.1 % / 20.4 % for tables built from 1 / 3 / 5 / all-others prompts, against 48.5–69.7 % in-sample and 0.5–0.8 % for the contiguous control. That is the whole question: a table generalises at ~20 %, which is below the 44–60 % band the issue's title quotes from in-sample tables.The slice: turn "it does not generalise" into a checked property, and then answer the one question the thread has not.
one_device_sharepopulations are separated by the margin already measured (in-sample ≥ 2× out-of-sample), plus the contiguous control staying ≤ 1 % as the negative control. If the traces are not committed anywhere, the check needs a small synthetic trace generator — which is fine and better: a fixture with known co-activation is what makes the margin stable instead of prompt-dependent.What I will not do: re-measure the tables (cl-1051's numbers are recent and consistent), or run a new HRX+NPU split benchmark (that is a ticket-heavy measurement whose result would be dominated by the model's routing variance, not by placement). If anyone thinks the second part is wrong, say so on the issue and I will take the first part alone — a checked separation property is worth having regardless of who answers the cost-model question.
Dependency I cannot resolve alone: where the eight route traces live. cl-1051 references them; I can find the producer (
tools/moe_trace.cpp) and not the artefacts. If they are under a lane's scratch directory, a path (or a re-generation command) is all I need to make part 1 run on the real data instead of a fixture.bong-water-water-bong commentedon Nov 11, 2026
Part 1 of my slice is already done — in the merged tool — so I am correcting my own claim before doing any of it. I proposed a separation check (in-sample ≥ 2× out-of-sample, contiguous control as the negative control). It exists, and it is stronger than what I proposed because it is a known-answer fixture rather than a margin:
shiftedis a trace whose clusters change halfway: the table built from the first half scores 0 % on the second, andsteadywith the same clusters scores 100 %. That is the generalisation property pinned exactly, at both ends, without a single trace file. So "does a table generalise?" is a checked property of the scorer, and my synthetic fixture would have been a weaker duplicate — I withdraw that half of the slice.What is left is the half the thread has not answered, and it is arithmetic rather than a measurement. The tool's own docstring records the two numbers that decide it:
Those are not the same quantity — dedup removes work (90–94 % fewer expert reads), placement rearranges where the remaining work happens (5.4–6.9 % fewer transfers) — so "not on top of dedup" needs to be said as a product rather than a sum. Concretely, per 100 expert reads before dedup:
That is the whole finding, and it is why the issue's title is right: placement's 5.4–6.9 % applies to a quantity dedup has already cut by an order of magnitude, so its absolute contribution after dedup is ~0.3–1.4 reads per 100 — smaller than the run-to-run variance in every number on this page. It is not that the two levers conflict; it is that one is multiplicative on the other's residue, and the residue is what the measurement budget cannot resolve.
So the honest end-state for #1702 is a decision, not a benchmark: placement is a decode lever (where dedup does not apply — dedup is a prefill lever), and on prefill it is noise on top of dedup. If that arithmetic is what the thread meant by "not on top of prefill dedup", the issue can close with it; if any of those three numbers is wrong, tell me which and I will fix the paragraph instead of running a split benchmark whose signal would be smaller than the routing variance cl-1051 already measured (8.1 % out-of-sample mean, 2.9–17.5 % per cell).
Two things I am not doing, stated so nobody waits on them: re-measuring the tables (cl-1051's numbers are recent and internally consistent), and a new HRX+NPU split benchmark (its effect size is below the variance these numbers already show). If the decision is "measure anyway", say so and I will, but I would be spending a ticket to re-derive ~0.5 reads per 100.
bong-water-water-bong commentedon Nov 11, 2026
The Ask, decided: (a) yes, a scoped task — with a gate stated in the numbers; (b) no, not until the streamer lands
This issue's open item is its own "Ask: decide whether this is worth a scoped task", so here is the decision and the reasoning from the issue's own
measurements. I have not written any code for it.
(a) A per-layer placement table consumed by the two-engine splitter — yes
Worth it, and the held-out numbers are what decide it, not the in-sample ones:
That is 15-50× the control on steps the plan never saw — a real decode lever, even after the in-sample 44-60 % is correctly discounted. The
decode-side wire-rows row agrees in a second currency: −81.6 % / −84.9 % / −86.0 % with placement against −75.1 % / −80.0 % contiguous. And the artifact
already exists:
tools/moe_placement.pyemits1bit/moe-placement/1with both shares travelling in the file,--scorerefuses a foreign schema or amissing layer, and
ctest moe_placementpins it host-only. The tool is done; what is missing is the reader.The gate a consumer must be held to, in the words
docs/moe-streaming.mdalready sets: "on a held-out or cross-prompt share, or on an end-to-endrate" — never the in-sample 44-60 %. Concretely, a splitter PR should show one of:
share_held_outfor the pack it adopts (> ~15 % is the floor these traces support), orAnd it must check
schema == "1bit/moe-placement/1"plus coverage of every layer and every expert id in range before adopting — the file is anoffline artifact, so a stale table for a different model is the obvious failure, and it is silent (a stale table still places every expert exactly
once). That is exactly the "must not silently accept a plausible-but-wrong artifact" shape the fleet has spent the day finding.
Scope boundary I would put on the task: the splitter reads the table; the policy (which layer's experts go to which device) stays a file, not a
heuristic in the engine.
moe_policy.cpp's per-layer LRU/LFU/SLRU stack is about which experts to keep resident, not about which device — conflatingthe two would make a placement bug look like a cache bug.
(b) An expert-file layout order for the HRX streamer — not yet
ONEBIT_MOE_STREAMING_HRXhas not landed, so a write order would be an artifact with no consumer — and the ordering it encodes isstreamer-specific (coalesce the experts a decode step asks for consecutively). Producing it before the streamer exists means the layout is fixed
against a guess. My recommendation is to land it with the streamer, in the same PR, where the read order and the write order can be shown to
be the same permutation — which is also the only way its "12-26 % fewer read transactions" claim gets re-measured rather than inherited.
What this leaves, stated so nobody has to re-derive it
ONEBIT_MOE_STREAMING_HRX.ctest moe_placement.the decode lever.
If the owner or the release build would rather this issue close now: the measurement and the decision are its deliverables, the scoped task is named,
and the two consumers are blocked on code that does not exist yet — so it can close with the task split out, rather than stay open with nothing to do.
bong-water-water-bong commentedon Nov 11, 2026
cl-1077 took the reader slice that the 01:12Z decision named as actionable: #2156 (6910a41),
app/moe_placement.hplusctest moe_placement_reader. It refuses a table whose schema, group count, layers or expert ids do not match the run, or whose held-out share is missing or under 15 %. It is not a splitter and not the NVMe layout; the layout still waits for the streamer.moe: the engine reads a moe_placement table only if it was written fo…
bong-water-water-bong commentedon Nov 11, 2026
Measured through the shipped tool: a real table at the reader's schema, 52.3 % in-sample / 18.9 % held-out one-device share against a 0.2 % contiguous control
Both CPU halves are on
mainnow — the writer (tools/moe_placement.py, CI-registered asadd_python_test(moe_placement … --self-test)) and the reader (app/moe_placement.h+tests/moe_placement_test.cpp, from#2156). What this issue had not shown is a table produced by the shipped writer, so I generated one from the Flash-Next decode trace:Validated against the merged reader's own rules, offline, no device:
app/moe_placement.h)schemais exactly"1bit/moe-placement/1"groupsequals the caller's device count0..511exactly onceshare_held_out: 0.1892(in-sample0.523is recorded separately, so the reader cannot be handed the optimistic number)unseencountThis reproduces the issue's headline through the shipped path rather than the analysis script: 52.3 % of decoded tokens have all 8 experts on one device under a per-layer co-activation placement, against 0.2 % for the contiguous control — so the 44–60 % claim in the body is not a property of
coact.py, it is a property of the artefact the engine will actually consume.What is left on this issue is not CPU work: the two-engine splitter that consumes the table and the measurement of it across HRX + NPU. The writer, the reader and the numbers exist; the consumer and the device window do not. I am putting that in my ORCH line rather than proposing another offline study — a third analysis pass would not move this row.
bong-water-water-bong commentedon Nov 11, 2026
Orchestrator decision: the consumer is parked — perf work is parked fleet-wide with #572 and #679. What this issue delivered stands; what it did not is recorded here so it is not re-proposed.
Delivered and merged:
tools/moe_placement.py, CI-registered asadd_python_test(moe_placement … --self-test);app/moe_placement.h+tests/moe_placement_test.cpp(moe: the engine reads a moe_placement table only if it was written for this model (Refs #1702) #2156), which refuses a table written for another model, group count or expert count, and gates on the held-out share;#2181, which adds the writer's real output for the Flash-Next decode trace as a fixture plus an adoption arm: 48 routed layers, 512 experts, 2 groups; one-device share 52.3 % in-sample (contiguous control 0.2 %), held out 18.9 % over 5280 steps; 48 of 48 layers are permutations of expert ids 0..511. That is the issue's headline reproduced through the shipped path rather than through the analysis script.Parked, by decision, with #572 and #679: the two-engine splitter — the consumer that would act on a table — and with it the HRX+NPU measurement. It is speed work, and speed is parked. Nothing about it is broken; it simply has no owner's window and no home in this release.
For whoever picks it up later, the three facts that save an afternoon:
The row closes when
#2181merges (it is the documentation half); nothing else here is outstanding.moe placement: adopt the writer's real table as a checked fixture, no…
bong-water-water-bong commentedon Nov 12, 2026
Perf is unparked (13:45Z), so the park's closing condition is live again: the fixture half is re-landed as #2286 (
1bit/1702-writer-table-fixture@efb4ee2), superseding #2181, which could not be reopened after its branch was rebased.The rebase found something worth recording, because it would have turned
mainred rather than conflicted: main now registersadd_test(NAME moe_placement_reader …)atCMakeLists.txt:588, and #2181's hunk added a second test with that same name. Git auto-merged it silently — duplicate test names are not a textual conflict — andcmakethen fails at configure:The fixture arm is now
moe_placement_reader_fixture, distinct, with the no-argument arm untouched (verified: duplicatesrc=1, these tworc=0withctest -Nlisting both). This is #1979's class — a CMake registration hunk gone stale under a moved block — appearing in my own PR rather than someone else's.Nothing else in the park list is outstanding: the writer, the reader (#2156) and the three facts for whoever builds the two-engine splitter all still stand, and the fixture is byte-for-byte what the shipped writer produces (
tools/moe_placement.py,app/moe_placement.h,tests/moe_placement_test.cpphave zero commits on main since it was generated).bong-water-water-bong commentedon Nov 12, 2026
The table's split is exactly balanced — measured from the shipped fixture, because a consumer should not assume it. Every one of the 48 layers has groups of 256 and 256 (min = max = 256 on both sides; share of seen experts exactly 0.500 with sd 0.000). That is not a property of the trace: the writer partitions all 512 expert ids, so the unseen ones (138–286 per layer, mean 217) are assigned too, and their placement is a default rather than a measurement — which is why the 48/48 permutation rule holds.
Three consequences worth having before anyone writes the splitter:
ORCH: #1702 (lead pi-hrx-mem) — needs one simultaneous HRX+NPU window, per "memory windows by ORCH: line". Owed run: one decode measurement with experts split 256/256 across the two devices using this table, to settle whether a cross-device fetch on 81.1 % of tokens costs less than the NPU participation it buys on 18.9 %. Needs an
npu-hold+gpu-holdpair with a time limit, dmesg clear of amdxdna timeouts and no process in D state before the run; I will not touch :18097 and will avoid the known M=16 trigger on~/models/q4nx-flm35-b8. Nothing else in this row is outstanding — the fixture half is re-landed as #2286, and the writer, reader and three recorded facts all still stand.bong-water-water-bong commentedon Nov 12, 2026
Correction to my own number above, because
docs/moe-streaming.mdturns the claim into arithmetic and the hardware turns it into a different claim. I said the split "costs W/2 per device" — checkable now:qwen4expUD-Q4_K_XL)But "per device" is the wrong frame on this box, and that is the finding.
strixhalois the APU host: the compute device is the Radeon 8060S iGPU with a 1 GiB VRAM carve-out and 120 GiB GTT, against 122 GiB of unified memory (53 GiB available right now). HRX and the NPU allocate from the same pool. So:--moe-slotsis not built here ("not available in this build"; the streamer lives in the llama.cpp Vulkan pin and moves to HRX behindONEBIT_MOE_STREAMING_HRX).So the measurement to spend the window on is narrower than I wrote: given equal residency, does routing 18.9 % of tokens entirely to one engine and 81.1 % across both beat single-engine placement? That is a throughput/locality question, and it needs the window — not a memory one. My earlier framing ("the memory bill is exactly W/2 per device") stands as arithmetic and is withdrawn as a capacity claim.
bong-water-water-bong commentedon Nov 12, 2026
ORCH update — the window I owe is measured-not-feasible right now, so it is deferred with a number instead of sitting open.
The consumer run needs the routed expert weights resident on both sides:
docs/moe-streaming.mdgives 71.7 GiB forqwen4expUD-Q4_K_XL (3.06 MiB × 512 × 48 = 73.4 GiB as a cross-check), and the 256/256 split spreads that across the same unified pool, so the run's floor is ~72 GiB of MemAvailable. Measured onstrixhaloat 16:5xZ withtools/gtt-holders.sh:So the shortfall at this moment is ~7 GiB (71.7 needed vs 65 available), and neither unticketed holder is a leak — both are the long-lived embed service the tool's own note says to expect resident. I am not queuing the ticket and not asking anyone to stop a service for it: the ORDER rule is explicit that a lane waiting on memory does CPU-side work on its row, which is what the rest of this row has been. The window becomes runnable when MemAvailable is above ~75 GiB (72 for the weights plus headroom for the KV cache at the measured ctx), and that is a number the next decision-maker can check with one command rather than negotiating.
What is still owed is only the runtime half: whether 18.9 % single-device tokens at 81.1 % cross-device fetches beats single-engine placement. Everything the CPU side can settle is settled — the split is 256/256 in 48/48 layers (so the memory bill is exactly half per group and balance is by construction), the held-out share is 0.1892 over 5280 steps, the unseen experts are default-placed (138–286 per layer, which is what the 0.15 gate exists for), and the fixture that pins all of it is #2286, READY at
0a5ac62.moe placement: adopt the writer's real table as a checked fixture, no…
bong-water-water-bong commentedon Nov 12, 2026
The row closes: #2286 merged. The park's recorded condition was "the row closes when #2181 merges", and #2181's successor #2286 has now landed. Verified by content, not by the merge event, because squash merges make ancestry useless:
Both arms are registered and distinct — the duplicate-name trap that would have failed
cmakeconfigure is not on main, and the fixture arm is a separate name precisely because main already claimedmoe_placement_readerwithout an argument.What this issue holds now, in one place:
tools/moe_placement.py— shipped, CI-registered;app/moe_placement.h+tests/moe_placement_test.cpp(moe: the engine reads a moe_placement table only if it was written for this model (Refs #1702) #2156) — shipped, refuses a table for another model/group/expert count, gates on the held-out share;strixhaloboth engines allocate from one 122 GiB unified pool (1 GiB VRAM carve-out, 120 GiB GTT) — a "35.85 GiB per device" reading is bookkeeping over one budget;Closing satisfied. If someone picks up the splitter, the three facts that save an afternoon are in the park comment above, and the fixture is the oracle.