Skip to content

hrx: since llama.cpp#304 the first long prompt after load pays a one-time copy (about +28 s on a 27B dense model); later prompts are faster #2163

Closed

Description

@bong-water-water-bong

Corrected 11-11 13:3xZ (cl-856). The first version of this issue said the model load went from 14.9 s to 42.6 s. That was wrong. llama.cpp's "load time" runs until the first decode finishes (src/llama-context.cpp:747), and in those runs the first decode was a 1,282-word prompt. Measured properly, the model load is unchanged, and the cost is a one-time copy at the first long prompt of a process. After it, #304 is faster.

Measured (Qwen3.8-27B UD-Q4_K_XL, llama.cpp 204493d40, hrx-system 7f4236924aad; GGML_HRX_MODEL_BUFFER_COPY=1 is the upload)

Load with a 5-token prompt (llama-completion -c 512 / -c 4096 -n 16 -v, one ticket): upload 14.6 / 10.2 s; #304 9.8 / 15.7 / 10.8 / 2.7 s. The same band on both paths; text identical.

First long prompt (fresh process, 1,282-word prompt, -c 4096): "load time" 14.9 s (upload) vs 42.6 s (#304, warm cache) and 49.3 s (cold). That is load plus the first prefill.

Steady state (llama-bench -p 1700 -n 16 -r 3 after its warmup, arms alternated):

pp1700 tok/s tg16 tok/s
upload 189.1, 231.0 5.8, 7.6
#304 308.4, 315.1 10.8, 11.1

Peak GTT after a long prompt: 20,970.5 MiB (upload) vs 20,386.5 MiB (#304).

What that means

  • A server on Census 2026-10-04 #304 pays roughly 28 s extra once, on its first long request. After that every long prompt is faster than on the upload path, and it holds less GTT.
  • Keeping the upload for large models (draft llama.cpp#311) would give up that speedup and the GTT saving to avoid a one-time cost. Bump HRX: llama.cpp e57beb9721af (HIP plumbing hook, HRX fixes, Flash-Next) #311 is closed for that reason.
  • Not the cause: copying from pageable memory. Staging the uploads through a pinned buffer measured no difference (3.6 / 3.9 s vs 4.1 / 6.0 s).

Remaining choice

  1. Accept the one-time cost and close this as measured.
  2. Move it to load time: warm up with a prompt-sized batch, so the first request does not pay. The upload path paid its copy at load too.

Activity

  1. added
    hrxThe HRX (HIP/IREE) backend route
    backlogReal, but does not hold the release build
    on Nov 11, 2026
  2. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-856, a pointer for option 2, from source reading only (not profiled yet): HRX's host-weight copy path (runtime/host-memory.cpp at the pin) builds the device layout on the CPU, with per-field and per-group copies, then uploads it, and synchronises the stream after each copy. Under #304 that runs for 850 weights; under the upload it ran for 89, and the rest stayed where the loader's read() put them. The first thing to measure is where the 28 s goes: CPU repack or synchronous upload. One perf record of a 27B load under gpu-hold would split it.

  3. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-856, one observation, not a measurement: a plain #304 27B run at 11:0xZ, with other processes holding 40.5 GiB of GTT at start and MemAvailable falling to about 12 GiB, loaded in 181.6 s (42.6 s an hour earlier on a quieter box). Under #304 the weights are read through the file mapping, so when the page cache cannot keep the 17 GB file resident the pages fault in repeatedly. The upload reads the file once, sequentially. There is no paired upload arm under the same pressure, so I am not quoting a ratio. It is a second reason to keep the upload for large models (option 1). The profile is waiting for MemAvailable >= 45 GiB (perf -e cpu-clock:u; the hardware event needs perf_event_paranoid <= 1, which I will not change).

  4. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-856: option (1) is ready as a draft, llama.cpp#311 (4733b339e). It waits on the orchestrator's choice. Measured: Qwen3-4B stays mapped (3,225.3 MiB peak GTT, load 0.7 s). Qwen3.8-27B goes back to the upload (19,075.3 MiB, load 9.0 s); GGML_HRX_MODEL_MAP_MAX_MIB=0 gives the same. Default 4096 MiB.

  5. changed the title [-]hrx: since llama.cpp#304 a 27B dense model loads in 42.6 s instead of 14.9 s and its first long prompt runs at half speed (weights copied through the host staging path)[/-] [+]hrx: since llama.cpp#304 the first long prompt after load pays a one-time copy (about +28 s on a 27B dense model); later prompts are faster[/+] on Nov 11, 2026
  6. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-856: llama.cpp#313 removes this cost. Measured at the fork head caa2d4dde (#304 + #313), hrx-system 7f4236924aad. Qwen3.8-27B UD-Q4_K_XL, fresh process per arm, 1,282-word prompt, -c 4096 -n 16, arms alternated in one gpu-hold ticket:

    "load time" (load + first prefill) peak GTT text
    upload (GGML_HRX_MODEL_BUFFER_COPY=1) 18.7 s, 20.0 s 20,970.5 MiB a167d083
    file mapping, #304 + #313 17.5 s, 20.0 s 20,226.5 MiB a167d083

    At 204493d40 (#304 without #313) the same shape was 42.6 s. #313 reads mapped q4_k/q6_k weights in the packed layout, so the first long prompt no longer builds stored-layout copies. That one-time copy was this issue's whole cost. The steady state was already faster on #304 (above).

    This closes when engine#2198 merges (the pin bump to caa2d4dde). Logs: ~/wt/cl-856-logs/u-313/.

  7. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing: the fix is on main. #2198 merged at 22:31Z (5a9a90bad) and pins llama.cpp 4107043c1, which contains llama.cpp#313 (caa2d4dde, the packed layout is the default for mapped weights). #2198's body did not carry a closing keyword, so I'm closing this by hand.

    Measured at the merged pin 4107043c1. Run v-4107:

    • Qwen3.8-27B UD-Q4_K_XL, -ngl 99 -c 4096 -n 16, the same 1,282-word prompt as the report.
    • Two fresh processes per arm, alternating upload and mapping.
    • One 6-minute gpu-hold ticket; the command ran 116 s.
    • upload means GGML_HRX_MODEL_BUFFER_COPY=1; mapping is the default.
    arm load time (includes the first long prefill) prompt eval peak GTT
    upload 1 12.9 s 19.4 s 20,970.5 MiB
    mapping 1 21.4 s 9.5 s 18,160.5 MiB
    upload 2 9.0 s 9.1 s 20,970.5 MiB
    mapping 2 13.8 s 9.3 s 18,160.5 MiB
    • Load time: at most 21.4 s on every arm, against the 42.6 s this issue reported at 204493d40. The one-time second copy on the first long prompt is gone. Between the two paths the load times do not order consistently: upload is 12.9/9.0 s and mapping 21.4/13.8 s. The box was shared during the run (MemAvailable 42 to 51 GiB, against 73 to 76 GiB for u-313), and the two upload runs alone differ by 3.9 s.
    • Peak GTT: the mapping is 2,810 MiB lower. At caa2d4dde it was 744 MiB lower (20,226.5 vs 20,970.5); hrx: after a long prompt each weight still has two device copies, because prefill reads q4_k/q6_k in the stored layout (remainder of #856) #2137's later commits widened the gap.
    • Text: identical across all four arms, and identical to u-313's at caa2d4dde.
    • Mapped path engaged: the mapping arms log HRX: the weights stay in the file mapping and are packed from it (engine #856); the upload arms do not.

    As established earlier on this issue, llama.cpp's "load time" runs to the end of the first eval (llama-context.cpp:747).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release buildhrxThe HRX (HIP/IREE) backend route

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions