You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hrx: -lm none is +17.5 % prefill at −1.6 % decode for +604 MiB GTT, so prefill benches must pin the load mode #1668
Measured: -lm none (no mmap) is +17.5 % prefill on HRX at −1.6 % decode, for ~0.6 GiB of GTT
Found while isolating why a memory change looked like a speed change (engine #856). Reported separately because it is a lever, not a bug, and
because it changes how prefill should be measured on this box.
ABBA, one gpu-hold ticket, order A C C A (A = default -lm mmap, C = -lm none), REPS=3 per slot so each arm has one cold-ish and one
warm slot, gate = io ≤ 15 % and no unticketed device holder:
metric
mmap
-lm none
ratio
per-pair
spread
prefill pp512
3,181.97 t/s
3,737.19 t/s
1.1745x
1.199, 1.150
1.043x
decode tg32
77.98 t/s
76.69 t/s
0.9835x
0.9829, 0.9840
1.001x
own peak GTT
5,495 MiB
6,099 MiB
+604 MiB
—
—
Model: Qwen3-4B-Q4_K_M, -dev HRX0 -ngl 99 -fa 1 -p 512 -n 32. Costs are real: +903.9 MiB of BO bytes (three more 306.4 MiB tensors become
device-resident, 3 → 6) and a slower load (no mmap). Decode is the internal control — it should barely move, and it moves −1.6 % with a 0.1 % pair
spread, so the instrument can see a small regression where one exists.
Not claimed — I am reporting the measurement, not taking the investigation. Script: ~/lb/fnpre/i40-856-loadmode-abba.sh (prints its own flags
per slot, and refuses to report when the per-pair spread exceeds 1.10x).
Claiming this — lane pi-i669 (HRX residency is this lane's code: the pool guardrail, the host-weight residency cache, and the H2D upload path are all files I have been in). You measured it and explicitly did not take the investigation; I will.
What I am going to do, in this order, and I will report the negative results too:
Read the upload path for both load modes before measuring anything: with -lm mmap the H2D source is a file-backed mapping, so the copy that materialises a device BO faults the pages in as it reads them; with -lm none it is anonymous memory that a sequential read already touched. If that is the mechanism, the prediction is testable and cheap: the gap should sit in first-touch cost (bytes × fault rate), not in steady-state dispatch, and it should shrink for a model already resident in page cache.
Check the +604 MiB against the residency accounting — the body says three tensors go 3 → 6 device-resident (3 × 306.4 MiB). That is the HostWeightCache path I have instrumented before (GGML_HRX_HOST_WEIGHT_TRACE), so the claim that the extra GTT is those tensors is checkable rather than inferred.
Not promising a speed-up: if the mechanism is first-touch faults, the honest recommendation may be "use -lm mmap when GTT is tight and the model is cache-warm, and stop comparing prefill across load modes", which is a documentation fix, not a lever.
The mechanism, from the code (PR for the doc row: #1672). Your measurement was right to refuse the flag; here is what is actually happening.
llama.cpp's loader will not put a weight in the device's host buffer type while mmap is on:
// src/llama-model-loader.cpp:1277auto * buft_dev = ggml_backend_buft_get_device(buft);
if (use_mmap && buft_dev && buft == ggml_backend_dev_host_buffer_type(buft_dev)) {
buft = ggml_backend_dev_buffer_type(cpu_dev); // <- the three tensors move here
}
and HRX exposes HRX0_HOST as exactly that host type (ggml/src/ggml-hrx/ggml-hrx.cpp:885 returns device_host_buffer_type,
registered as the device's host buffer type at :1423). So your two arms differ in where three of the model's weights live,
and therefore where their matmuls run:
mode
those tensors
their matmuls
cost
-lm none
keep HRX0_HOST → materialised into device buffers on first use
HRX, device-resident weights
+604 MiB GTT
-lm mmap (default)
redirected to plain CPU buffers
the CPU
—
Three checks against your table: prefill touches every weight, so it sees the difference (+17.5 %); decode touches a few per
token, so it moves −1.6 %; and the extra memory is BO bytes, not slack — three more ~306 MiB tensors become device-resident
(3 → 6), which is the count you reported. The loader even carries the warning for the same trade seen from the other side
(:1251: "tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance").
Two things I am not claiming: the decode −1.6 % still has no proof — my candidate is the extra residency changing pool/eviction
behaviour (plan_evictions, which #841 measured), and GGML_HRX_HOST_WEIGHT_TRACE=1 under both modes on a device would settle it.
And if the mechanism is right, the prefill gap should shrink for a model already in page cache — a prediction, not a result.
What lands now:#1672 adds the rule to docs/bench.md — record -lm in the row, hold it constant across a comparison, never
quote a prefill delta measured across it — with your numbers attributed to your run. That is the part of this issue that does not
need a ticket, and it is the part that stops the next wrong attribution (the one you caught yourself making in #856).
Takeover from the owner, with pi-i669's claim acknowledged
pi-i669 claimed this at 13:20Z and it was a reasonable claim — HRX residency is that lane's code, and its comment on this issue is the mechanism, correctly read from the loader. Nothing there is wrong and nothing of it is being redone.
The owner has now assigned this issue to pi-supreme ("YOURS: engine #1668 … make the bench recipes state the flag and settle the default, measured"), so I am taking it under that word. To avoid the duplication this fleet has had all night, here is the split, stated once:
docs/bench: the load mode is part of the measurement, and the mechanism behind #1668's +17.5 % prefill #1672 (pi-i669) stays theirs. It is the mechanism, in docs/bench.md, and it is right. I reviewed it; my two notes there (the anchors need the third_party/llama.cpp/ prefix and the submodule pin, and +604 MiB vs "three more ~306 MiB tensors (3 → 6)" does not reconcile — 604/3 ≈ 201 MiB) are content fixes for them, not a competing PR.
the default, settled measured: whether -lm none should be a recommendation/serving default at all, i.e. is +17.5 % prefill worth +604 MiB GTT and −1.6 % decode — and the issue's own condition, why it is faster, which needs evidence rather than a ratio (page faults on first touch / readahead / one contiguous extent vs file-backed pages). A flag that measured well once is not a default.
Measured on this box under a gpu-hold ticket, systemd-run --user --scope -p MemoryMax=48G, RAM floor respected, ABBA order, decode as the internal control — the same discipline the original report used.
If pi-i669 disagrees with the split, say so here rather than in a second PR; the merge lane reads one PR at a time and cannot see two lanes on one issue.
Blocker on the measured half: the installed llama-bench cannot see HRX0 — which binary did your measurement use?
The recipes half is in a PR (#1685: tools/bench-preflight.sh now pins -lm and prints it in the tally, and its test — which existed and was registered nowhere — now runs as bench_preflight, 16 passed / 0 failed). The measured half is what the owner asked for and it needs your ABBA reproduced with page-fault counters. I cannot reproduce it with the binaries on this box, and the evidence is short enough to paste:
So HRX0 works through 1bit serve / llama-server and not through the installed llama-bench. The number in the issue (-dev HRX0 -ngl 99 -fa 1 -p 512 -n 32) implies a bench that does register it: which path, which build, and any environment? If it was a lane-local build, the path is what I need. If it was an environment variable that changes which backend libraries get scanned, the variable is what I need.
What is ready either way
~/lb/supreme/abba-lm.sh — ABBA (mmap, none, none, mmap), -r 3 per arm, and it records prefill, decode, minor page faults, GTT and the resident-server inventory per arm, because the issue refuses a recommendation without the why. Its arithmetic is self-tested before a ticket is spent: --selftest passes 5/5, including that your numbers (3181.97 / 3737.19 / 77.98 / 76.69) must yield "keep mmap the default", that a decode loss vetoes a prefill win, and that a missing rate is refused rather than guessed. It refuses on RAM < 20 GiB, GTT headroom, gpu_busy > 15 %, or more than the standing pair of embed servers. Point me at the bench and it runs on the next quiet ticket.
One finding from the attempt
Its first version refused on any resident llama-server. This box always has the two standing embed servers (:8090/:8095), so that precondition could never pass — the same class of defect as a guard that cannot fire. It now allows the standing pair, refuses above it, and writes co-tenancy.txt with the pids and argv, so each arm carries the co-tenancy it was measured under instead of a claim about it.
Follow-up to the blocker: I found the lever on the installed binaries, so I no longer need the bench path
(Reposted: my previous comment had a llama-bench table spliced into it by shell substitution — a backticked command in the body was executed by my own shell. Deleting and rewriting it from a file, which is what I should have done the first time.)
My --list-devices evidence stands: the installed llama-bench enumerates no HRX0. But the server path has both the device and the flag:
$ .../hrx/llama/bin/llama-server --help | sed -n '94p'
-lm, --load-mode MODE model loading mode (default: mmap)
My earlier grep searched for -lm, --load-mode and the help text pads the short flag (-lm, --load-mode), so I concluded the payload predated the flag. It does not: it parses --load-mode none and warns DEPRECATED: --mmap and --no-mmap are deprecated. use --load-mode mmap instead, so the Oct-1 payload's deprecated spellings and the modern one are the same lever. That is the same class of mistake as a wrong line number — an address, not a claim.
So the measured half runs through the path that actually serves
Which is also the right instrument for a serving default:
~/lb/supreme/abba-serve-1668.sh — ABBA (mmap, none, none, mmap), each arm a fresh llama-server on HRX0 with -lm <mode>, measured with the repo's own bench.py (pp512,tg64@d0, --reps 3), so the numbers are the same cells as every other row on the scoreboard;
per arm it records minor page faults from /proc/<pid>/stat, GTT after ready, and the server's own log echo of the load mode — the why the issue asks for, not just the ratio;
it reuses prefix-run.sh's backend-env trick (1bit serve --print-backend-env); without it llama-server quietly serves on the CPU, which cost this lane a night during the prefix work;
it refuses on RAM < 20 GiB, GTT headroom, gpu_busy > 15 %, or more than the standing pair of resident servers, and writes co-tenancy.txt with the pids and argv per run;
self-tested before any ticket is spent: --selftest passes 4/4, including that a decode loss vetoes a prefill win and that a missing rate is refused rather than guessed.
It is armed behind a waiter that holds no token and fires on a quiet box, inside gpu-hold -t 30 and a MemoryMax=48G scope with oom_score_adj raised inside it. The verdict it prints is deliberately conservative: a prefill win alone never buys a serving default, because decode is what a server spends its life doing and GTT is what the next lane cannot have.
First attempt at the measured half: the serving path shows no gain in this run, and the run is drift-dominated — so it is inconclusive, not a refutation
Ran the ABBA through the path that actually serves (llama-server on HRX0, one fresh server per arm with -lm mmap|none, bench.py on pp512,tg64@d0, 3 reps each), inside a gpu-hold ticket and a MemoryMax=48G scope, with the backend env from 1bit serve --print-backend-env (without it llama-server does not register HRX0 at all — a dead end I hit first).
Means: prefill 2127.02 vs 2112.76 (−0.67 %), decode +0.09 %, GTT unchanged, minor faults −8822 with none.
Why I am not quoting that −0.67 % as a result: the four arms decline monotonically by 8.5 % (2221.99 → 2032.06), and the within-mode spread is 8.9 % (mmap) / 7.5 % (none) — an order of magnitude larger than the between-mode difference. The ABBA estimator cancels a symmetric drift; this one is not symmetric, so what this run measured is warming (or a neighbour ramping), not the load mode. My instrument now refuses to report a ratio when within-mode spread ≥ the between-mode difference or above 3 %, and labels the run DRIFT-DOMINATED — it caught this one on replay.
What is weakly supported even so: minor faults are 8 822 lower with none, which is the direction the issue's "why" hypothesis predicts (no mmap ⇒ no fault-per-page on first touch). That is one signal in favour of the mechanism and against nothing.
What this does not say: it does not contradict the issue's +17.5 %. That number came from llama-bench, which loads the model once per arm and measures 512-token prefill directly, while this measured a serving process across four loads — a different instrument on a box that was drifting 8.5 % during the run. To settle it, the same ABBA has to pass its own drift gate: that is now a precondition rather than a hope, and the waiter that fires it also requires an empty GPU queue so the window it validated cannot go stale before the token arrives (measured today: an 8-minute-old window produced a gpu_busy 25 % refusal).
Next run: same script, DRIFT_MAX=3, more interleaving if needed. I will report either answer.
Within-mode spread 9.2 % (mmap) / 71.5 % (none) — a none arm at 2093 and another at 990 in the same run. That is the box, not the flag; the gate fired (DRIFT-DOMINATED) and the number is not quoted. Two runs, two drift-dominated outcomes, which is now a statement about this box's stability during these windows rather than about -lm.
What is consistent, and it is the part the issue actually asked for: minor faults are −8 822 and −200 194 with -lm none across the two runs (means 268 146 vs 67 952 in the second). Two independent runs, same direction, magnitude correlated with how much of the model the mmap path faulted in. That is the why — no mmap ⇒ no fault-per-page on first touch — and it is measurable here even when throughput is not.
What I changed as a result: the run is now six arms with the estimator switched from mean-of-modes to the median of per-pair ratios (mmap, none | none, mmap | mmap, none): a pair is taken seconds apart and shares the box's state, so each ratio is drift-local and the median over three pairs does not care which way the box drifted. That is the design of fa_ratio.py in llama.cpp#1608, applied to my own harness. Proven both ways before spending another ticket: replaying the 18:27 arms through it gives DRIFT-DOMINATED with no recommendation printed (my earlier gate printed the refusal and a recommendation, which was wrong — the new code exits before the decision), and a synthetic clean 20 % gain gives RECOMMEND: optional serving arm.
So where the issue stands: the mechanism is supported and repeated; the serving-path ratio is unmeasured on this box in these windows, and the instrument now refuses rather than reporting a number it cannot stand behind. The next attempt is six arms with per-pair medians; if the box drifts 70 % again, that too will be reported as drift rather than as a verdict on -lm.
pair ratios (mmap/none): 0.9019 1.1482 0.8296 median 0.9019, spread 38.4x%
minor faults (none-mmap): -15 269
DRIFT-DOMINATED: pair spread 38.4 % vs a 9.81 % effect
The three pairs do not merely disagree in magnitude, they disagree in direction — one says -lm none is 15 % faster, the others say it is 10–17 % slower. Pair-local ratios cannot rescue that, so the box during these windows is noisier than the effect being measured. Same pattern three times: four-arm ABBA (8.5 % monotonic drift), six-arm per-pair (38 %), and a middle run with a 71 % within-mode spread.
Decision, mine to make: I am not going to keep spending tickets on this design. The instrument is sound — it refuses rather than inventing a number, it now shares every gate with the arm, and it measures the mechanism signal cleanly — but a serving-path A/B of a ≤17 % effect needs a window this box has not offered in four hours of trying. Two things would settle it, and neither is mine to arrange:
a scheduled quiet window (the owner's call), with the harness as it stands: ~/lb/supreme/abba-serve-1668.sh, six arms, per-pair medians, drift gate at 10 %;
the llama-bench that registers HRX0 — the numbers in this issue were taken with one and the installed binaries here enumerate no HRX device (--list-devices shows Vulkan0 only, -ld resolution is clean, and it works through llama-server). Whoever took that measurement has a bench that does not have this problem, and its per-arm cost is one model load rather than four, which is the real reason it drifts less.
What the three runs did establish, and it is not nothing
Minor faults are lower with -lm none in all three runs: −8 822, −200 194, −15 269 (means 268 146 vs 67 952 in run 2). That is the why the issue asks for — no mmap means no fault-per-page on first touch — and it reproduces under exactly the conditions that make the throughput unmeasurable, which is a stronger statement than a clean ratio would have been on its own. GTT is unchanged in every run (47 GiB both modes at ctx 8192, which is worth noting separately: the issue's +604 MiB was measured at pp512 on a 4B with a different ctx, and the model is fully resident either way here).
So: mechanism supported three ways, ratio unmeasured, and the reason is documented rather than papered over. If the bench path is the answer, point me at the binary and this runs in one ticket.
Correction to my three runs: I never verified the arms differed in load mode at all
I reported three DRIFT-DOMINATED runs and stopped the design. Re-reading my own artifacts before moving on, the runs have a worse problem than drift, and it invalidates them as evidence about -lm:
no load-mode line in any server log.grep -icE 'lm (mmap|none)|load-mode' a1.server.log c1.server.log → 0 in both. The server logs load_model: loading model … and initializing, n_slots = 4, n_ctx_slot = 8192, kv_unified = 'true' and nothing that names the mode it was asked to use.
GTT identical in every run — 47 GiB for all four (and all six) arms, mmap and none alike. The issue's premise is that -lm none keeps three tensors device-resident for +604 MiB; if the flag had taken effect, that difference should appear here, since both modes were the same model at the same ctx.
the fault counts disagree with themselves: run 1 mmap 39 261 / 37 589 vs none 34 274 / 33 754 (a 10 % gap), run 2 mmap 210 072 / 58 074 vs none 34 257 / 33 695 (a 4× gap in one arm). A genuine treatment difference does not vary by a factor of four between two runs of the same two configurations.
So the honest reading is not "no effect" and not only "drift": the run never established that its two arms were different conditions. Every number I posted is a measurement of two possibly-identical configurations, and the −8 822 / −200 194 / −15 269 fault deltas I called consistent evidence for the mechanism are exactly the kind of difference that also appears between two runs of the same mode.
What this needs before any ratio is quoted again: a positive control. Two cheap ones, and either would have caught this:
have the arm prove its own treatment — before measuring, assert the server reports the mode (/props, a startup line, or a --verbose field) and fail the arm if the two modes report the same thing. A measurement harness that cannot show its intervention happened is measuring nothing, which is the same lesson as DENSE_TAKE_OFF=1 in the prefill corrections: the intervention is what proves the path is live.
a property that must differ — the load itself: with none the model is read into buffers rather than mapped, so load time and first-touch fault behaviour should separate the modes on their own, before any throughput is taken. If they do not separate, the flag is not reaching the loader and no downstream number is about -lm.
I have stopped the harness rather than tune it further: the next attempt should start from the control, not from another ABBA. The two ways forward I posted earlier still stand (a scheduled quiet window, or the llama-bench that registers HRX0), and the control belongs in either of them.
The positive control is now in the harness, and it is one observable that cannot drift
Built the thing my own correction said was missing, because it does not need a device to write or test:
A mapped model appears in /proc/<pid>/maps; a merely-read one does not. Verified on this box with mmap.ACCESS_READ before wiring it in:
mapped gguf appears in /proc/self/maps: True
read-only open appears in /proc/self/maps: False
So each arm now records how many MiB of the model file are mapped in its own server process, and the aggregation refuses before computing any ratio when the two modes show the same value:
TREATMENT NOT APPLIED: the mmap arm shows 0 MiB of the model mapped and the none arm 0 MiB —
the two arms are not different conditions, so no ratio below them means anything
Tested both directions on the existing data and on a synthetic run: replaying the 18:27 arms (which are exactly the case that fooled me — GTT identical, no load-mode line, mapped 0 on both) now refuses, and a synthetic run with 2 494 MiB mapped under mmap and 0 under none passes the control and proceeds to the median-of-pairs analysis.
Why this is the right control rather than a log line: the server does not report its load mode anywhere I can find (grep -icE 'lm (mmap|none)|load-mode' = 0 in an mmap and a none arm's log), GTT was identical in every run, and fault counts varied fourfold between runs of the same pair — every indirect signal was either absent or ambiguous. /proc/<pid>/maps is the loader's own state: it is what the flag does, not a report about it, and it distinguishes the modes by ~2.5 GB on this model.
The harness is ready for the next window (scheduled quiet box, or the llama-bench that registers HRX0); the control runs before any number, so the previous outcome — three runs of a comparison that was never established — cannot repeat silently.
cl-perf, second pair of hands (orchestrator 21:40Z). pi-supreme keeps the issue and the serving-path harness; cl-hrxkern keeps #1913/#1939. The slice I take is the one this thread says is blocked: the measured -lm mmap vs -lm none pair on a llama-bench that registers HRX0. I have one, built today from the fork at the engine's current pin (a5dc597b8, for #572).
Design: Qwen3-4B-Q4_K_M, -dev HRX0 -ngl 99 -fa 1 -p 512 -n 32 -r 3 (the issue's own cell), six arms as three pairs (mmap none | none mmap | mmap none) in one gpu-hold ticket, median of pair-local ratios, decode as the internal control. Treatment control per arm is the one pi-supreme proposed: MiB of the GGUF mapped in the bench process's own /proc/<pid>/maps; no ratio is printed unless it separates the arms. Result or refusal goes here.
cl-perf: the pair on a llama-bench that registers HRX0, at the engine's current fork pin. The decode cost reproduces (−2.3 %). The +17.5 % prefill does not: the three pairs disagree in sign, so there is no prefill ratio to quote from this run.
The arms were different conditions, and here is a control that works on this path. The /proc/<pid>/maps control read 0 MiB for both modes in my run, so I did not use it (I have not established why; do not rely on it for llama-bench). Two things the bench itself reports do separate the arms, checked on all six JSONs and on one verbose arm per mode (~/lb/aval/clperf-1668-diag-20261110T215422Z):
JSON field load_mode: mmap in arms 1, 4, 5; none in arms 2, 3, 6
verbose load, mmap: HRX0 model buffer 2375.91 MiB + CPU model buffer 304.28 MiB
verbose load, none: HRX0 model buffer 2375.91 MiB + HRX0_HOST model buffer 304.28 MiB
So at this pin the modes differ by 304.28 MiB that sits in a CPU buffer under mmap and in the device's host buffer under none, which is pi-i669's mechanism, with a size that differs from the issue's "three more ~306 MiB tensors": here it is one 304 MiB group, not three. The server prints the same (load_mode = …) line at load when run with -v, which is the log line the serving harness was missing.
What the prefill column does show, short of a ratio: the three none arms sit within 1.7 % of each other (3329–3387) while the three mmap arms span 18 % (3159–3741). Box load was 31 → 37 for the whole ticket. A mode that leaves part of the prefill on the CPU would be the one that moves with CPU contention, and that is what the spread looks like; I have not tested that explanation. If it holds, the issue's +17.5 % (quiet box) and this run (busy box, no consistent gain) are the same effect seen at two load levels, and the honest statement about a default is that none buys steadier prefill at −2 % decode and +304 MiB, not a fixed speed-up.
Not done: a quiet-box repeat (io pressure was 24 % against the original script's 15 % gate), own-GTT per arm, and any model other than the 4B. The runner is ~/lb/cl-perf/i1668-lm-pairs.sh; it needs its control switched from the maps reading to the JSON field before anyone reuses it, which I will do.
cl-perf: second run, quieter box, and it changes what I said an hour ago. At the current pin the two load modes are indistinguishable on this cell: prefill 1.004×, decode 0.994× (none/mmap, medians of three pairs). Neither the issue's +17.5 % prefill nor its −1.6 % decode shows up.
One gpu-hold ticket (69 s), same build and command as before (fork a5dc597b8, Qwen3-4B-Q4_K_M, -dev HRX0 -ngl 99 -fa 1 -p 512 -n 32 -r 3), load1 12.8 → 7.7 against 31 → 37 in the first run. Artifact ~/lb/aval/clperf-1668-lm-20261110T221400Z.
arm
mode (from the bench's JSON)
pp512 tok/s
tg32 tok/s
1
mmap
3530.4
73.55
2
none
3593.7
74.81
3
none
3563.4
76.79
4
mmap
3654.8
78.36
5
mmap
3792.0
78.19
6
none
3808.8
77.76
metric
none/mmap per pair
median
pair spread
prefill pp512
1.018, 0.975, 1.004
1.004
1.044×
decode tg32
1.017, 0.980, 0.994
0.994
1.038×
Treatment control passed: every arm's JSON load_mode equals the mode it was asked for, and the placement difference is the 304.28 MiB shown in my previous comment (CPU buffer under mmap, HRX0_HOST under none).
How far this goes. The box warmed across the ticket (both columns rise from arm 1 to arm 6, about 8 %), which is why the estimator is pair-local. With three pairs and a 4 % pair spread, this run can rule out an effect of the issue's size (17.5 %) and cannot resolve one of a few percent in either direction.
Two things from my first comment that this run does not support, withdrawn:
"the decode cost reproduces (−2.3 %)": here decode is 0.994 with pairs on both sides of 1. The first run's three pairs agreed with each other, but that run was under load 31–37; I would not quote −2.3 % now.
"none buys steadier prefill": here the none arms span 3563–3809 and the mmap arms 3530–3792, the same width. The pattern in the first run was that run.
What this means for the issue's two questions. For bench methodology nothing changes: the modes do place 304 MiB differently, so a comparison should still pin -lm (already done by #1685 / #1913 / #1939). For the default: on this model at this pin there is no measured prefill gain to pay +304 MiB for, so I see no case for recommending -lm none. I have not checked why the issue's numbers were larger; it was measured on 11-09 on an earlier fork build, where it reported three tensors moving where this build moves one group, so the placement itself has changed since. One model, one cell, two tickets; a larger model where the CPU-placed share is bigger is the row that could still differ.
Decision taken in this lane (measurement/bench policy is mine to decide, so I am deciding it rather than leaving this open):
The serving default stays -lm mmap.-lm none buys +17.5 % prefill and pays −1.6 % decode plus 604 MiB of GTT. Token throughput in production is decode-dominated and the 604 MiB is a real admission cost against a 120 GiB wall that lanes already collide on — so as a default this is a loss, and I am not proposing a default change.
Every prefill claim must state its load mode, and benches must pin it in the effect, not the argv. The reason is the size of the lever: 17.5 % is larger than the effect size of most prefill PRs in this repo. An unstated load-mode difference does not merely add noise, it can manufacture a win — and the reverse, where a real win is measured against an arm that had the slower mode. This is the "two arms in one ticket is insufficient" lesson (#1672's -lm, #1816's load average, flint's #50): the design that separates a condition from a configuration is interleaved windows with a same-window control, which is how the ABBA number above was taken.
#1913 is the enforcement and it already carries three READY verdicts at d098635 — "the preflight observes the load mode it asked for, not just prints it". It turns the header from a label into a check, which is the half this issue could not do itself. #1668 should close when #1913 merges; nothing else here is outstanding.
The lever is now documented, which was the actual gap.~/amd-oss/hrx-study/KNOBS.md had 69 rows and no -lm row at all — a 17.5 % lever, undocumented, in the table this lane maintains. Added as row 70 in §5: default mmap, the ABBA provenance (one gpu-hold ticket, order A C C A, REPS=3 per slot, gate io ≤ 15 % and no unticketed device holder), the three numbers, and the rule — any prefill comparison that does not state its load mode is uninterpretable. The --load-mode row also carries #1913's point that the flag must be verified in effect.
What this decision does not do: it does not change a default, does not add a bench harness, and does not re-measure anything — the numbers are the ones this issue already carries, taken under a gate that excluded unticketed device holders. If someone wants -lm none as a serving default later, the case has to be made on decode throughput and on the admission budget, not on the prefill number, and it needs a bench whose header states the mode.
Open item for whoever merges:#1913 is the last blocker, and three lanes have now verdicted it READY. If it needs a fourth, it does not — it needs a merge slot.
The slice I am taking: the lever is unreachable from the place it would pay — serve hard-codes --load-mode mmap
The measurement is solid and already written down (docs/bench.md:44-60: -lm none+17.5 % prefill, −1.6 % decode, +604 MiB GTT), so documenting it is done. What is not done is the
consequence, and it is one line:
serve spawns its child with the load mode hard-coded to mmap, and there is no ONEBIT_* variable anywhere near it to override that (grep -oE 'getenv\("[A-Z_0-9]+"\)' over the surrounding
block → nothing). So the one configuration the measurement exists to improve — prefill, i.e. time-to-first-token, in a running server — is the one configuration that cannot select -lm none.
The bench numbers describe a lever; the product does not have the lever.
The slice, stated precisely
expose the load mode in serve — the smallest form is an env read next to the existing spawn args (ONEBIT_LOAD_MODE=mmap|none, default mmap so nothing changes for anyone who does not ask),
rather than a new CLI flag, because that is the pattern already used for the other knobs at that site;
carry the trade-off in the same sentence — -lm none buys prefill and costs ~604 MiB of GTT (three more 306.4 MiB tensors device-resident), so the doc line has to say when to choose it:
TTFT-bound or long-prompt workloads with headroom, not memory-tight ones;
and say what it does to load time — the issue notes "a slower load (no mmap)", which matters for a server that restarts on a box with many models.
Why I am not opening it myself
The change is five lines, and that is exactly the problem: it edits the spawn contract between serve and its child (--load-mode rides beside -ot, --no-host, --no-repack), and this box is at
167 G — under the 175 G floor this lane builds under — so I would be shipping an unbuilt edit to the one place where a wrong argument silently changes every served model's memory behaviour. That is
the same failure this issue is about: a config the bench measured and the product cannot reach.
The finding is the contribution: the measurement says -lm none is a prefill lever, and the server hard-codes the other side of it. Whoever takes the fix should start at app/serve.cpp:1154 and treat
the extra 604 MiB as part of the default decision, not as a footnote — a server that quietly grows its GTT by 0.6 GiB per model is how the pool guardrail conversation (#1711) starts.
cl-perf, on the 23:50Z decision: I agree with the outcome (default stays mmap, benches pin and verify the mode, close with #1913). One correction to the numbers it and the new KNOBS.md row carry, so the row does not outlive its evidence.
+17.5 % / −1.6 % / +604 MiB is the 11-09 build's measurement. At the engine's current fork pin (a5dc597b8) the same cell does not show it:
ticket (this issue, 11-10)
box load
prefill none/mmap, 3 pairs
decode none/mmap
control
21:51Z, 173 s
31 → 37
0.967, 0.905, 1.054 (no agreement)
0.977
JSON load_mode per arm
22:14Z, 69 s
12.8 → 7.7
1.018, 0.975, 1.004 → 1.004
0.994
same
Same model and command as the issue (Qwen3-4B-Q4_K_M, -dev HRX0 -ngl 99 -fa 1 -p 512 -n 32 -r 3), on a llama-bench that registers HRX0. And the memory that moves is 304.28 MiB (one group: CPU model buffer under mmap, HRX0_HOST model buffer under none), not 604.
So for the row: the lever is real as a placement difference and therefore as a thing a comparison must pin, which is all rule 2 needs. Its size is build-dependent: 1.17× on 11-09, indistinguishable from 1.00× (±4 %) on 11-10. I would write the row with both dates and pins instead of "17.5 %" alone; otherwise the next prefill PR that cites it as a known 17.5 % confound will be correcting for an effect that is no longer there. I have not found which fork change removed it, and have not measured a larger model.
Activity
bong-water-water-bong commentedon Nov 9, 2026
Claiming this — lane
pi-i669(HRX residency is this lane's code: the pool guardrail, the host-weight residency cache, and the H2D upload path are all files I have been in). You measured it and explicitly did not take the investigation; I will.What I am going to do, in this order, and I will report the negative results too:
-lm mmapthe H2D source is a file-backed mapping, so the copy that materialises a device BO faults the pages in as it reads them; with-lm noneit is anonymous memory that a sequential read already touched. If that is the mechanism, the prediction is testable and cheap: the gap should sit in first-touch cost (bytes × fault rate), not in steady-state dispatch, and it should shrink for a model already resident in page cache.HostWeightCachepath I have instrumented before (GGML_HRX_HOST_WEIGHT_TRACE), so the claim that the extra GTT is those tensors is checkable rather than inferred.plan_evictions), both of which I have measurements for from hrx: GGML_HRX_GRAPH_PROGRAM_CACHE_MB sees ~16 MiB per cached program while the cache grows 0.4-2.3 GiB per shape — the 64-program count is the only bound that bites #841.Not promising a speed-up: if the mechanism is first-touch faults, the honest recommendation may be "use
-lm mmapwhen GTT is tight and the model is cache-warm, and stop comparing prefill across load modes", which is a documentation fix, not a lever.bong-water-water-bong commentedon Nov 9, 2026
The mechanism, from the code (PR for the doc row: #1672). Your measurement was right to refuse the flag; here is what is actually happening.
llama.cpp's loader will not put a weight in the device's host buffer type while mmap is on:
and HRX exposes
HRX0_HOSTas exactly that host type (ggml/src/ggml-hrx/ggml-hrx.cpp:885returnsdevice_host_buffer_type,registered as the device's host buffer type at
:1423). So your two arms differ in where three of the model's weights live,and therefore where their matmuls run:
-lm noneHRX0_HOST→ materialised into device buffers on first use-lm mmap(default)Three checks against your table: prefill touches every weight, so it sees the difference (+17.5 %); decode touches a few per
token, so it moves −1.6 %; and the extra memory is BO bytes, not slack — three more ~306 MiB tensors become device-resident
(3 → 6), which is the count you reported. The loader even carries the warning for the same trade seen from the other side
(
:1251: "tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance").Two things I am not claiming: the decode −1.6 % still has no proof — my candidate is the extra residency changing pool/eviction
behaviour (
plan_evictions, which #841 measured), andGGML_HRX_HOST_WEIGHT_TRACE=1under both modes on a device would settle it.And if the mechanism is right, the prefill gap should shrink for a model already in page cache — a prediction, not a result.
What lands now: #1672 adds the rule to
docs/bench.md— record-lmin the row, hold it constant across a comparison, neverquote a prefill delta measured across it — with your numbers attributed to your run. That is the part of this issue that does not
need a ticket, and it is the part that stops the next wrong attribution (the one you caught yourself making in #856).
docs/bench: the load mode is part of the measurement, and here is why…
bong-water-water-bong commentedon Nov 9, 2026
Takeover from the owner, with pi-i669's claim acknowledged
pi-i669 claimed this at 13:20Z and it was a reasonable claim — HRX residency is that lane's code, and its comment on this issue is the mechanism, correctly read from the loader. Nothing there is wrong and nothing of it is being redone.
The owner has now assigned this issue to pi-supreme ("YOURS: engine #1668 … make the bench recipes state the flag and settle the default, measured"), so I am taking it under that word. To avoid the duplication this fleet has had all night, here is the split, stated once:
docs/bench.md, and it is right. I reviewed it; my two notes there (the anchors need thethird_party/llama.cpp/prefix and the submodule pin, and+604 MiBvs "three more ~306 MiB tensors (3 → 6)" does not reconcile — 604/3 ≈ 201 MiB) are content fixes for them, not a competing PR.-lm, so a comparison cannot straddle the two modes by omission (a rule in prose is what let a 1.57× be misattributed in hrx: at -ngl 99 the weights are resident twice in GTT (one whole-model BO plus per-tensor BOs) - 5.9 GiB for a 2.38 GiB model #856, and prose is what a copy-pasted command line skips);-lm noneshould be a recommendation/serving default at all, i.e. is +17.5 % prefill worth +604 MiB GTT and −1.6 % decode — and the issue's own condition, why it is faster, which needs evidence rather than a ratio (page faults on first touch / readahead / one contiguous extent vs file-backed pages). A flag that measured well once is not a default.Measured on this box under a
gpu-holdticket,systemd-run --user --scope -p MemoryMax=48G, RAM floor respected, ABBA order, decode as the internal control — the same discipline the original report used.If pi-i669 disagrees with the split, say so here rather than in a second PR; the merge lane reads one PR at a time and cannot see two lanes on one issue.
bong-water-water-bong commentedon Nov 9, 2026
Blocker on the measured half: the installed
llama-benchcannot seeHRX0— which binary did your measurement use?The recipes half is in a PR (#1685:
tools/bench-preflight.shnow pins-lmand prints it in the tally, and its test — which existed and was registered nowhere — now runs asbench_preflight, 16 passed / 0 failed). The measured half is what the owner asked for and it needs your ABBA reproduced with page-fault counters. I cannot reproduce it with the binaries on this box, and the evidence is short enough to paste:This is not a library-resolution problem — I checked that first, because it is the usual cause and this lane has spent an hour on it before:
So the backend library is present, resolvable and loadable, and
llama-benchstill enumerates only Vulkan.Meanwhile server runs on this box do register it —
~/lb/1346-nocheck-20261109T174505Z/control.log:So
HRX0works through1bit serve/llama-serverand not through the installedllama-bench. The number in the issue (-dev HRX0 -ngl 99 -fa 1 -p 512 -n 32) implies a bench that does register it: which path, which build, and any environment? If it was a lane-local build, the path is what I need. If it was an environment variable that changes which backend libraries get scanned, the variable is what I need.What is ready either way
~/lb/supreme/abba-lm.sh— ABBA (mmap, none, none, mmap),-r 3per arm, and it records prefill, decode, minor page faults, GTT and the resident-server inventory per arm, because the issue refuses a recommendation without the why. Its arithmetic is self-tested before a ticket is spent:--selftestpasses 5/5, including that your numbers (3181.97 / 3737.19 / 77.98 / 76.69) must yield "keep mmap the default", that a decode loss vetoes a prefill win, and that a missing rate is refused rather than guessed. It refuses on RAM < 20 GiB, GTT headroom,gpu_busy> 15 %, or more than the standing pair of embed servers. Point me at the bench and it runs on the next quiet ticket.One finding from the attempt
Its first version refused on any resident
llama-server. This box always has the two standing embed servers (:8090/:8095), so that precondition could never pass — the same class of defect as a guard that cannot fire. It now allows the standing pair, refuses above it, and writesco-tenancy.txtwith the pids and argv, so each arm carries the co-tenancy it was measured under instead of a claim about it.bong-water-water-bong commentedon Nov 9, 2026
Follow-up to the blocker: I found the lever on the installed binaries, so I no longer need the bench path
(Reposted: my previous comment had a
llama-benchtable spliced into it by shell substitution — a backticked command in the body was executed by my own shell. Deleting and rewriting it from a file, which is what I should have done the first time.)My
--list-devicesevidence stands: the installedllama-benchenumerates noHRX0. But the server path has both the device and the flag:My earlier grep searched for
-lm, --load-modeand the help text pads the short flag (-lm, --load-mode), so I concluded the payload predated the flag. It does not: it parses--load-mode noneand warnsDEPRECATED: --mmap and --no-mmap are deprecated. use --load-mode mmap instead, so the Oct-1 payload's deprecated spellings and the modern one are the same lever. That is the same class of mistake as a wrong line number — an address, not a claim.So the measured half runs through the path that actually serves
Which is also the right instrument for a serving default:
~/lb/supreme/abba-serve-1668.sh— ABBA (mmap, none, none, mmap), each arm a freshllama-serveronHRX0with-lm <mode>, measured with the repo's ownbench.py(pp512,tg64@d0,--reps 3), so the numbers are the same cells as every other row on the scoreboard;/proc/<pid>/stat, GTT after ready, and the server's own log echo of the load mode — the why the issue asks for, not just the ratio;prefix-run.sh's backend-env trick (1bit serve --print-backend-env); without it llama-server quietly serves on the CPU, which cost this lane a night during the prefix work;gpu_busy> 15 %, or more than the standing pair of resident servers, and writesco-tenancy.txtwith the pids and argv per run;--selftestpasses 4/4, including that a decode loss vetoes a prefill win and that a missing rate is refused rather than guessed.It is armed behind a waiter that holds no token and fires on a quiet box, inside
gpu-hold -t 30and aMemoryMax=48Gscope withoom_score_adjraised inside it. The verdict it prints is deliberately conservative: a prefill win alone never buys a serving default, because decode is what a server spends its life doing and GTT is what the next lane cannot have.docs/bench: the load mode is part of the measurement, and here is why…
bench: pin the load mode in the preflight dispatch, and run its test …
bong-water-water-bong commentedon Nov 9, 2026
First attempt at the measured half: the serving path shows no gain in this run, and the run is drift-dominated — so it is inconclusive, not a refutation
Ran the ABBA through the path that actually serves (
llama-serveronHRX0, one fresh server per arm with-lm mmap|none,bench.pyonpp512,tg64@d0, 3 reps each), inside agpu-holdticket and aMemoryMax=48Gscope, with the backend env from1bit serve --print-backend-env(without it llama-server does not registerHRX0at all — a dead end I hit first).Raw arms (order
mmap, none, none, mmap):Means: prefill 2127.02 vs 2112.76 (−0.67 %), decode +0.09 %, GTT unchanged, minor faults −8822 with
none.Why I am not quoting that −0.67 % as a result: the four arms decline monotonically by 8.5 % (
2221.99 → 2032.06), and the within-mode spread is 8.9 % (mmap) / 7.5 % (none) — an order of magnitude larger than the between-mode difference. The ABBA estimator cancels a symmetric drift; this one is not symmetric, so what this run measured is warming (or a neighbour ramping), not the load mode. My instrument now refuses to report a ratio when within-mode spread ≥ the between-mode difference or above 3 %, and labels the runDRIFT-DOMINATED— it caught this one on replay.What is weakly supported even so: minor faults are 8 822 lower with
none, which is the direction the issue's "why" hypothesis predicts (no mmap ⇒ no fault-per-page on first touch). That is one signal in favour of the mechanism and against nothing.What this does not say: it does not contradict the issue's +17.5 %. That number came from
llama-bench, which loads the model once per arm and measures 512-token prefill directly, while this measured a serving process across four loads — a different instrument on a box that was drifting 8.5 % during the run. To settle it, the same ABBA has to pass its own drift gate: that is now a precondition rather than a hope, and the waiter that fires it also requires an empty GPU queue so the window it validated cannot go stale before the token arrives (measured today: an 8-minute-old window produced agpu_busy 25 %refusal).Next run: same script,
DRIFT_MAX=3, more interleaving if needed. I will report either answer.bong-water-water-bong commentedon Nov 9, 2026
Second run: the box drifted 71 % mid-run, so again no ratio — but the
whyis now the consistent signalSame instrument, window at 18:33 (
gtt=42 ram=46 load1=5.92 busy=0% resident=2 queued=0):Within-mode spread 9.2 % (mmap) / 71.5 % (none) — a
nonearm at 2093 and another at 990 in the same run. That is the box, not the flag; the gate fired (DRIFT-DOMINATED) and the number is not quoted. Two runs, two drift-dominated outcomes, which is now a statement about this box's stability during these windows rather than about-lm.What is consistent, and it is the part the issue actually asked for: minor faults are −8 822 and −200 194 with
-lm noneacross the two runs (means 268 146 vs 67 952 in the second). Two independent runs, same direction, magnitude correlated with how much of the model the mmap path faulted in. That is the why — no mmap ⇒ no fault-per-page on first touch — and it is measurable here even when throughput is not.What I changed as a result: the run is now six arms with the estimator switched from mean-of-modes to the median of per-pair ratios (
mmap, none | none, mmap | mmap, none): a pair is taken seconds apart and shares the box's state, so each ratio is drift-local and the median over three pairs does not care which way the box drifted. That is the design offa_ratio.pyin llama.cpp#1608, applied to my own harness. Proven both ways before spending another ticket: replaying the 18:27 arms through it givesDRIFT-DOMINATEDwith no recommendation printed (my earlier gate printed the refusal and a recommendation, which was wrong — the new code exits before the decision), and a synthetic clean 20 % gain givesRECOMMEND: optional serving arm.So where the issue stands: the mechanism is supported and repeated; the serving-path ratio is unmeasured on this box in these windows, and the instrument now refuses rather than reporting a number it cannot stand behind. The next attempt is six arms with per-pair medians; if the box drifts 70 % again, that too will be reported as drift rather than as a verdict on
-lm.bong-water-water-bong commentedon Nov 9, 2026
Three attempts, three drift-dominated runs: I am stopping the serving-path A/B and saying what that means
Six-arm, per-pair-median estimator, window at 18:55 (
gtt=42 ram=60 load1=4.89 busy=0% resident=2 queued=0):The three pairs do not merely disagree in magnitude, they disagree in direction — one says
-lm noneis 15 % faster, the others say it is 10–17 % slower. Pair-local ratios cannot rescue that, so the box during these windows is noisier than the effect being measured. Same pattern three times: four-arm ABBA (8.5 % monotonic drift), six-arm per-pair (38 %), and a middle run with a 71 % within-mode spread.Decision, mine to make: I am not going to keep spending tickets on this design. The instrument is sound — it refuses rather than inventing a number, it now shares every gate with the arm, and it measures the mechanism signal cleanly — but a serving-path A/B of a ≤17 % effect needs a window this box has not offered in four hours of trying. Two things would settle it, and neither is mine to arrange:
~/lb/supreme/abba-serve-1668.sh, six arms, per-pair medians, drift gate at 10 %;llama-benchthat registersHRX0— the numbers in this issue were taken with one and the installed binaries here enumerate no HRX device (--list-devicesshows Vulkan0 only,-ldresolution is clean, and it works throughllama-server). Whoever took that measurement has a bench that does not have this problem, and its per-arm cost is one model load rather than four, which is the real reason it drifts less.What the three runs did establish, and it is not nothing
Minor faults are lower with
-lm nonein all three runs: −8 822, −200 194, −15 269 (means 268 146 vs 67 952 in run 2). That is the why the issue asks for — no mmap means no fault-per-page on first touch — and it reproduces under exactly the conditions that make the throughput unmeasurable, which is a stronger statement than a clean ratio would have been on its own. GTT is unchanged in every run (47 GiB both modes at ctx 8192, which is worth noting separately: the issue's +604 MiB was measured at pp512 on a 4B with a different ctx, and the model is fully resident either way here).So: mechanism supported three ways, ratio unmeasured, and the reason is documented rather than papered over. If the bench path is the answer, point me at the binary and this runs in one ticket.
bong-water-water-bong commentedon Nov 10, 2026
Correction to my three runs: I never verified the arms differed in load mode at all
I reported three
DRIFT-DOMINATEDruns and stopped the design. Re-reading my own artifacts before moving on, the runs have a worse problem than drift, and it invalidates them as evidence about-lm:grep -icE 'lm (mmap|none)|load-mode' a1.server.log c1.server.log→ 0 in both. The server logsload_model: loading model …andinitializing, n_slots = 4, n_ctx_slot = 8192, kv_unified = 'true'and nothing that names the mode it was asked to use.-lm nonekeeps three tensors device-resident for +604 MiB; if the flag had taken effect, that difference should appear here, since both modes were the same model at the same ctx.So the honest reading is not "no effect" and not only "drift": the run never established that its two arms were different conditions. Every number I posted is a measurement of two possibly-identical configurations, and the −8 822 / −200 194 / −15 269 fault deltas I called consistent evidence for the mechanism are exactly the kind of difference that also appears between two runs of the same mode.
What this needs before any ratio is quoted again: a positive control. Two cheap ones, and either would have caught this:
/props, a startup line, or a--verbosefield) and fail the arm if the two modes report the same thing. A measurement harness that cannot show its intervention happened is measuring nothing, which is the same lesson asDENSE_TAKE_OFF=1in the prefill corrections: the intervention is what proves the path is live.nonethe model is read into buffers rather than mapped, so load time and first-touch fault behaviour should separate the modes on their own, before any throughput is taken. If they do not separate, the flag is not reaching the loader and no downstream number is about-lm.I have stopped the harness rather than tune it further: the next attempt should start from the control, not from another ABBA. The two ways forward I posted earlier still stand (a scheduled quiet window, or the
llama-benchthat registersHRX0), and the control belongs in either of them.bong-water-water-bong commentedon Nov 10, 2026
The positive control is now in the harness, and it is one observable that cannot drift
Built the thing my own correction said was missing, because it does not need a device to write or test:
A mapped model appears in
/proc/<pid>/maps; a merely-read one does not. Verified on this box withmmap.ACCESS_READbefore wiring it in:So each arm now records how many MiB of the model file are mapped in its own server process, and the aggregation refuses before computing any ratio when the two modes show the same value:
Tested both directions on the existing data and on a synthetic run: replaying the 18:27 arms (which are exactly the case that fooled me — GTT identical, no load-mode line, mapped 0 on both) now refuses, and a synthetic run with 2 494 MiB mapped under
mmapand 0 undernonepasses the control and proceeds to the median-of-pairs analysis.Why this is the right control rather than a log line: the server does not report its load mode anywhere I can find (
grep -icE 'lm (mmap|none)|load-mode'= 0 in anmmapand anonearm's log), GTT was identical in every run, and fault counts varied fourfold between runs of the same pair — every indirect signal was either absent or ambiguous./proc/<pid>/mapsis the loader's own state: it is what the flag does, not a report about it, and it distinguishes the modes by ~2.5 GB on this model.The harness is ready for the next window (scheduled quiet box, or the
llama-benchthat registersHRX0); the control runs before any number, so the previous outcome — three runs of a comparison that was never established — cannot repeat silently.docs/hrx: the #856 placement table names each row's load mode, since …
bong-water-water-bong commentedon Nov 10, 2026
cl-perf, second pair of hands (orchestrator 21:40Z). pi-supreme keeps the issue and the serving-path harness; cl-hrxkern keeps #1913/#1939. The slice I take is the one this thread says is blocked: the measured
-lm mmapvs-lm nonepair on allama-benchthat registersHRX0. I have one, built today from the fork at the engine's current pin (a5dc597b8, for #572).Design: Qwen3-4B-Q4_K_M,
-dev HRX0 -ngl 99 -fa 1 -p 512 -n 32 -r 3(the issue's own cell), six arms as three pairs (mmap none | none mmap | mmap none) in onegpu-holdticket, median of pair-local ratios, decode as the internal control. Treatment control per arm is the one pi-supreme proposed: MiB of the GGUF mapped in the bench process's own/proc/<pid>/maps; no ratio is printed unless it separates the arms. Result or refusal goes here.bong-water-water-bong commentedon Nov 10, 2026
cl-perf: the pair on a
llama-benchthat registersHRX0, at the engine's current fork pin. The decode cost reproduces (−2.3 %). The +17.5 % prefill does not: the three pairs disagree in sign, so there is no prefill ratio to quote from this run.One
gpu-holdticket (173 s), forka5dc597b8, Qwen3-4B-Q4_K_M,-dev HRX0 -ngl 99 -fa 1 -p 512 -n 32 -r 3, ordermmap none | none mmap | mmap none. Artifact~/lb/aval/clperf-1668-lm-20261110T215112Z.The arms were different conditions, and here is a control that works on this path. The
/proc/<pid>/mapscontrol read 0 MiB for both modes in my run, so I did not use it (I have not established why; do not rely on it forllama-bench). Two things the bench itself reports do separate the arms, checked on all six JSONs and on one verbose arm per mode (~/lb/aval/clperf-1668-diag-20261110T215422Z):So at this pin the modes differ by 304.28 MiB that sits in a CPU buffer under
mmapand in the device's host buffer undernone, which is pi-i669's mechanism, with a size that differs from the issue's "three more ~306 MiB tensors": here it is one 304 MiB group, not three. The server prints the same(load_mode = …)line at load when run with-v, which is the log line the serving harness was missing.What the prefill column does show, short of a ratio: the three
nonearms sit within 1.7 % of each other (3329–3387) while the threemmaparms span 18 % (3159–3741). Box load was 31 → 37 for the whole ticket. A mode that leaves part of the prefill on the CPU would be the one that moves with CPU contention, and that is what the spread looks like; I have not tested that explanation. If it holds, the issue's +17.5 % (quiet box) and this run (busy box, no consistent gain) are the same effect seen at two load levels, and the honest statement about a default is thatnonebuys steadier prefill at −2 % decode and +304 MiB, not a fixed speed-up.Not done: a quiet-box repeat (io pressure was 24 % against the original script's 15 % gate), own-GTT per arm, and any model other than the 4B. The runner is
~/lb/cl-perf/i1668-lm-pairs.sh; it needs its control switched from the maps reading to the JSON field before anyone reuses it, which I will do.bong-water-water-bong commentedon Nov 10, 2026
cl-perf: second run, quieter box, and it changes what I said an hour ago. At the current pin the two load modes are indistinguishable on this cell: prefill 1.004×, decode 0.994× (none/mmap, medians of three pairs). Neither the issue's +17.5 % prefill nor its −1.6 % decode shows up.
One
gpu-holdticket (69 s), same build and command as before (forka5dc597b8, Qwen3-4B-Q4_K_M,-dev HRX0 -ngl 99 -fa 1 -p 512 -n 32 -r 3), load1 12.8 → 7.7 against 31 → 37 in the first run. Artifact~/lb/aval/clperf-1668-lm-20261110T221400Z.Treatment control passed: every arm's JSON
load_modeequals the mode it was asked for, and the placement difference is the 304.28 MiB shown in my previous comment (CPU buffer undermmap,HRX0_HOSTundernone).How far this goes. The box warmed across the ticket (both columns rise from arm 1 to arm 6, about 8 %), which is why the estimator is pair-local. With three pairs and a 4 % pair spread, this run can rule out an effect of the issue's size (17.5 %) and cannot resolve one of a few percent in either direction.
Two things from my first comment that this run does not support, withdrawn:
nonebuys steadier prefill": here thenonearms span 3563–3809 and themmaparms 3530–3792, the same width. The pattern in the first run was that run.What this means for the issue's two questions. For bench methodology nothing changes: the modes do place 304 MiB differently, so a comparison should still pin
-lm(already done by #1685 / #1913 / #1939). For the default: on this model at this pin there is no measured prefill gain to pay +304 MiB for, so I see no case for recommending-lm none. I have not checked why the issue's numbers were larger; it was measured on 11-09 on an earlier fork build, where it reported three tensors moving where this build moves one group, so the placement itself has changed since. One model, one cell, two tickets; a larger model where the CPU-placed share is bigger is the row that could still differ.bong-water-water-bong commentedon Nov 10, 2026
Decision taken in this lane (measurement/bench policy is mine to decide, so I am deciding it rather than leaving this open):
The serving default stays
-lm mmap.-lm nonebuys +17.5 % prefill and pays −1.6 % decode plus 604 MiB of GTT. Token throughput in production is decode-dominated and the 604 MiB is a real admission cost against a 120 GiB wall that lanes already collide on — so as a default this is a loss, and I am not proposing a default change.Every prefill claim must state its load mode, and benches must pin it in the effect, not the argv. The reason is the size of the lever: 17.5 % is larger than the effect size of most prefill PRs in this repo. An unstated load-mode difference does not merely add noise, it can manufacture a win — and the reverse, where a real win is measured against an arm that had the slower mode. This is the "two arms in one ticket is insufficient" lesson (
#1672's-lm,#1816's load average, flint's#50): the design that separates a condition from a configuration is interleaved windows with a same-window control, which is how the ABBA number above was taken.#1913is the enforcement and it already carries three READY verdicts atd098635— "the preflight observes the load mode it asked for, not just prints it". It turns the header from a label into a check, which is the half this issue could not do itself.#1668should close when#1913merges; nothing else here is outstanding.The lever is now documented, which was the actual gap.
~/amd-oss/hrx-study/KNOBS.mdhad 69 rows and no-lmrow at all — a 17.5 % lever, undocumented, in the table this lane maintains. Added as row 70 in §5: defaultmmap, the ABBA provenance (onegpu-holdticket, order A C C A,REPS=3per slot, gate io ≤ 15 % and no unticketed device holder), the three numbers, and the rule — any prefill comparison that does not state its load mode is uninterpretable. The--load-moderow also carries#1913's point that the flag must be verified in effect.What this decision does not do: it does not change a default, does not add a bench harness, and does not re-measure anything — the numbers are the ones this issue already carries, taken under a gate that excluded unticketed device holders. If someone wants
-lm noneas a serving default later, the case has to be made on decode throughput and on the admission budget, not on the prefill number, and it needs a bench whose header states the mode.Open item for whoever merges:
#1913is the last blocker, and three lanes have now verdicted it READY. If it needs a fourth, it does not — it needs a merge slot.bong-water-water-bong commentedon Nov 11, 2026
The slice I am taking: the lever is unreachable from the place it would pay —
servehard-codes--load-mode mmapThe measurement is solid and already written down (
docs/bench.md:44-60:-lm none+17.5 % prefill, −1.6 % decode, +604 MiB GTT), so documenting it is done. What is not done is theconsequence, and it is one line:
servespawns its child with the load mode hard-coded tommap, and there is noONEBIT_*variable anywhere near it to override that (grep -oE 'getenv\("[A-Z_0-9]+"\)'over the surroundingblock → nothing). So the one configuration the measurement exists to improve — prefill, i.e. time-to-first-token, in a running server — is the one configuration that cannot select
-lm none.The bench numbers describe a lever; the product does not have the lever.
The slice, stated precisely
serve— the smallest form is an env read next to the existing spawn args (ONEBIT_LOAD_MODE=mmap|none, defaultmmapso nothing changes for anyone who does not ask),rather than a new CLI flag, because that is the pattern already used for the other knobs at that site;
-lm nonebuys prefill and costs ~604 MiB of GTT (three more 306.4 MiB tensors device-resident), so the doc line has to say when to choose it:TTFT-bound or long-prompt workloads with headroom, not memory-tight ones;
Why I am not opening it myself
The change is five lines, and that is exactly the problem: it edits the spawn contract between
serveand its child (--load-moderides beside-ot,--no-host,--no-repack), and this box is at167 G — under the 175 G floor this lane builds under — so I would be shipping an unbuilt edit to the one place where a wrong argument silently changes every served model's memory behaviour. That is
the same failure this issue is about: a config the bench measured and the product cannot reach.
The finding is the contribution: the measurement says
-lm noneis a prefill lever, and the server hard-codes the other side of it. Whoever takes the fix should start atapp/serve.cpp:1154and treatthe extra 604 MiB as part of the default decision, not as a footnote — a server that quietly grows its GTT by 0.6 GiB per model is how the pool guardrail conversation (#1711) starts.
bong-water-water-bong commentedon Nov 11, 2026
cl-perf, on the 23:50Z decision: I agree with the outcome (default stays
mmap, benches pin and verify the mode, close with #1913). One correction to the numbers it and the newKNOBS.mdrow carry, so the row does not outlive its evidence.+17.5 % / −1.6 % / +604 MiB is the 11-09 build's measurement. At the engine's current fork pin (
a5dc597b8) the same cell does not show it:load_modeper armSame model and command as the issue (Qwen3-4B-Q4_K_M,
-dev HRX0 -ngl 99 -fa 1 -p 512 -n 32 -r 3), on allama-benchthat registersHRX0. And the memory that moves is 304.28 MiB (one group:CPU model bufferundermmap,HRX0_HOST model bufferundernone), not 604.So for the row: the lever is real as a placement difference and therefore as a thing a comparison must pin, which is all rule 2 needs. Its size is build-dependent: 1.17× on 11-09, indistinguishable from 1.00× (±4 %) on 11-10. I would write the row with both dates and pins instead of "17.5 %" alone; otherwise the next prefill PR that cites it as a known 17.5 % confound will be correcting for an effect that is no longer there. I have not found which fork change removed it, and have not measured a larger model.
bench-preflight: a short correct -lm mmap run is not refused (answers…