You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hrx: since llama.cpp#304 the first long prompt after load pays a one-time copy (about +28 s on a 27B dense model); later prompts are faster #2163
Corrected 11-11 13:3xZ (cl-856). The first version of this issue said the model load went from 14.9 s to 42.6 s. That was wrong. llama.cpp's "load time" runs until the first decode finishes (src/llama-context.cpp:747), and in those runs the first decode was a 1,282-word prompt. Measured properly, the model load is unchanged, and the cost is a one-time copy at the first long prompt of a process. After it, #304 is faster.
Measured (Qwen3.8-27B UD-Q4_K_XL, llama.cpp 204493d40, hrx-system 7f4236924aad; GGML_HRX_MODEL_BUFFER_COPY=1 is the upload)
Load with a 5-token prompt (llama-completion -c 512 / -c 4096 -n 16 -v, one ticket): upload 14.6 / 10.2 s; #304 9.8 / 15.7 / 10.8 / 2.7 s. The same band on both paths; text identical.
First long prompt (fresh process, 1,282-word prompt, -c 4096): "load time" 14.9 s (upload) vs 42.6 s (#304, warm cache) and 49.3 s (cold). That is load plus the first prefill.
Steady state (llama-bench -p 1700 -n 16 -r 3 after its warmup, arms alternated):
Peak GTT after a long prompt: 20,970.5 MiB (upload) vs 20,386.5 MiB (#304).
What that means
A server on Census 2026-10-04 #304 pays roughly 28 s extra once, on its first long request. After that every long prompt is faster than on the upload path, and it holds less GTT.
cl-856, a pointer for option 2, from source reading only (not profiled yet): HRX's host-weight copy path (runtime/host-memory.cpp at the pin) builds the device layout on the CPU, with per-field and per-group copies, then uploads it, and synchronises the stream after each copy. Under #304 that runs for 850 weights; under the upload it ran for 89, and the rest stayed where the loader's read() put them. The first thing to measure is where the 28 s goes: CPU repack or synchronous upload. One perf record of a 27B load under gpu-hold would split it.
cl-856, one observation, not a measurement: a plain #304 27B run at 11:0xZ, with other processes holding 40.5 GiB of GTT at start and MemAvailable falling to about 12 GiB, loaded in 181.6 s (42.6 s an hour earlier on a quieter box). Under #304 the weights are read through the file mapping, so when the page cache cannot keep the 17 GB file resident the pages fault in repeatedly. The upload reads the file once, sequentially. There is no paired upload arm under the same pressure, so I am not quoting a ratio. It is a second reason to keep the upload for large models (option 1). The profile is waiting for MemAvailable >= 45 GiB (perf -e cpu-clock:u; the hardware event needs perf_event_paranoid <= 1, which I will not change).
cl-856: option (1) is ready as a draft, llama.cpp#311 (4733b339e). It waits on the orchestrator's choice. Measured: Qwen3-4B stays mapped (3,225.3 MiB peak GTT, load 0.7 s). Qwen3.8-27B goes back to the upload (19,075.3 MiB, load 9.0 s); GGML_HRX_MODEL_MAP_MAX_MIB=0 gives the same. Default 4096 MiB.
changed the title [-]hrx: since llama.cpp#304 a 27B dense model loads in 42.6 s instead of 14.9 s and its first long prompt runs at half speed (weights copied through the host staging path)[/-][+]hrx: since llama.cpp#304 the first long prompt after load pays a one-time copy (about +28 s on a 27B dense model); later prompts are faster[/+]on Nov 11, 2026
cl-856: llama.cpp#313 removes this cost. Measured at the fork head caa2d4dde (#304 + #313), hrx-system 7f4236924aad. Qwen3.8-27B UD-Q4_K_XL, fresh process per arm, 1,282-word prompt, -c 4096 -n 16, arms alternated in one gpu-hold ticket:
At 204493d40 (#304 without #313) the same shape was 42.6 s. #313 reads mapped q4_k/q6_k weights in the packed layout, so the first long prompt no longer builds stored-layout copies. That one-time copy was this issue's whole cost. The steady state was already faster on #304 (above).
This closes when engine#2198 merges (the pin bump to caa2d4dde). Logs: ~/wt/cl-856-logs/u-313/.
Closing: the fix is on main. #2198 merged at 22:31Z (5a9a90bad) and pins llama.cpp 4107043c1, which contains llama.cpp#313 (caa2d4dde, the packed layout is the default for mapped weights). #2198's body did not carry a closing keyword, so I'm closing this by hand.
Measured at the merged pin 4107043c1. Run v-4107:
Qwen3.8-27B UD-Q4_K_XL, -ngl 99 -c 4096 -n 16, the same 1,282-word prompt as the report.
Two fresh processes per arm, alternating upload and mapping.
One 6-minute gpu-hold ticket; the command ran 116 s.
upload means GGML_HRX_MODEL_BUFFER_COPY=1; mapping is the default.
arm
load time (includes the first long prefill)
prompt eval
peak GTT
upload 1
12.9 s
19.4 s
20,970.5 MiB
mapping 1
21.4 s
9.5 s
18,160.5 MiB
upload 2
9.0 s
9.1 s
20,970.5 MiB
mapping 2
13.8 s
9.3 s
18,160.5 MiB
Load time: at most 21.4 s on every arm, against the 42.6 s this issue reported at 204493d40. The one-time second copy on the first long prompt is gone. Between the two paths the load times do not order consistently: upload is 12.9/9.0 s and mapping 21.4/13.8 s. The box was shared during the run (MemAvailable 42 to 51 GiB, against 73 to 76 GiB for u-313), and the two upload runs alone differ by 3.9 s.
Activity
bong-water-water-bong commentedon Nov 11, 2026
cl-856, a pointer for option 2, from source reading only (not profiled yet): HRX's host-weight copy path (
runtime/host-memory.cppat the pin) builds the device layout on the CPU, with per-field and per-group copies, then uploads it, and synchronises the stream after each copy. Under #304 that runs for 850 weights; under the upload it ran for 89, and the rest stayed where the loader'sread()put them. The first thing to measure is where the 28 s goes: CPU repack or synchronous upload. Oneperf recordof a 27B load undergpu-holdwould split it.bong-water-water-bong commentedon Nov 11, 2026
cl-856, one observation, not a measurement: a plain #304 27B run at 11:0xZ, with other processes holding 40.5 GiB of GTT at start and MemAvailable falling to about 12 GiB, loaded in 181.6 s (42.6 s an hour earlier on a quieter box). Under #304 the weights are read through the file mapping, so when the page cache cannot keep the 17 GB file resident the pages fault in repeatedly. The upload reads the file once, sequentially. There is no paired upload arm under the same pressure, so I am not quoting a ratio. It is a second reason to keep the upload for large models (option 1). The profile is waiting for MemAvailable >= 45 GiB (
perf -e cpu-clock:u; the hardware event needsperf_event_paranoid<= 1, which I will not change).bong-water-water-bong commentedon Nov 11, 2026
cl-856: option (1) is ready as a draft, llama.cpp#311 (
4733b339e). It waits on the orchestrator's choice. Measured: Qwen3-4B stays mapped (3,225.3 MiB peak GTT, load 0.7 s). Qwen3.8-27B goes back to the upload (19,075.3 MiB, load 9.0 s);GGML_HRX_MODEL_MAP_MAX_MIB=0gives the same. Default 4096 MiB.[-]hrx: since llama.cpp#304 a 27B dense model loads in 42.6 s instead of 14.9 s and its first long prompt runs at half speed (weights copied through the host staging path)[/-][+]hrx: since llama.cpp#304 the first long prompt after load pays a one-time copy (about +28 s on a 27B dense model); later prompts are faster[/+]bong-water-water-bong commentedon Nov 11, 2026
cl-856: llama.cpp#313 removes this cost. Measured at the fork head
caa2d4dde(#304 + #313), hrx-system7f4236924aad. Qwen3.8-27B UD-Q4_K_XL, fresh process per arm, 1,282-word prompt,-c 4096 -n 16, arms alternated in onegpu-holdticket:GGML_HRX_MODEL_BUFFER_COPY=1)a167d083a167d083At
204493d40(#304 without #313) the same shape was 42.6 s. #313 reads mapped q4_k/q6_k weights in the packed layout, so the first long prompt no longer builds stored-layout copies. That one-time copy was this issue's whole cost. The steady state was already faster on #304 (above).This closes when engine#2198 merges (the pin bump to
caa2d4dde). Logs:~/wt/cl-856-logs/u-313/.bong-water-water-bong commentedon Nov 11, 2026
Closing: the fix is on
main. #2198 merged at 22:31Z (5a9a90bad) and pins llama.cpp4107043c1, which contains llama.cpp#313 (caa2d4dde, the packed layout is the default for mapped weights). #2198's body did not carry a closing keyword, so I'm closing this by hand.Measured at the merged pin
4107043c1. Runv-4107:-ngl 99 -c 4096 -n 16, the same 1,282-word prompt as the report.gpu-holdticket; the command ran 116 s.uploadmeansGGML_HRX_MODEL_BUFFER_COPY=1;mappingis the default.204493d40. The one-time second copy on the first long prompt is gone. Between the two paths the load times do not order consistently: upload is 12.9/9.0 s and mapping 21.4/13.8 s. The box was shared during the run (MemAvailable 42 to 51 GiB, against 73 to 76 GiB foru-313), and the two upload runs alone differ by 3.9 s.caa2d4ddeit was 744 MiB lower (20,226.5 vs 20,970.5); hrx: after a long prompt each weight still has two device copies, because prefill reads q4_k/q6_k in the stored layout (remainder of #856) #2137's later commits widened the gap.u-313's atcaa2d4dde.HRX: the weights stay in the file mapping and are packed from it (engine #856); the upload arms do not.As established earlier on this issue, llama.cpp's "load time" runs to the end of the first eval (
llama-context.cpp:747).