diff --git a/docs/hrx.md b/docs/hrx.md index 9a37038d2..0d6eefa8f 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -650,17 +650,19 @@ length peak at 8.7 GiB instead of growing; greedy answers are identical, and lla is unchanged. A repeated prompt length right after a flush costs 1.75 s instead of 1.12 s, since its program is rebuilt. -### Server memory: the weight-mapping import is opt-in ([engine#856](https://github.com/1bit-MONSTER/engine/issues/856), [engine#1339](https://github.com/1bit-MONSTER/engine/issues/1339)) - -At `-ngl 99` the loader copies every offloaded weight into one device buffer, and HRX then serves each dense -matmul from its own repacked copy, so the weights were resident twice in GTT. Since -[llama.cpp #226](https://github.com/1bit-MONSTER/llama.cpp/pull/226) HRX can import the loader's file mapping -instead of taking that copy (Qwen3-4B-Q4_K_M: 3925 -> 1243 MiB of the process's GTT, wikitext perplexity -16.3000 in both arms). On larger files the import never finishes: a dense Qwen3.8-27B (16.4 GiB) pages the -whole file in, then waits with no error and no progress. Qwen3.8-Flash-Next (76.3 GiB) does the same. -Since [llama.cpp #231](https://github.com/1bit-MONSTER/llama.cpp/pull/231) the import is **off by default**. -`GGML_HRX_WEIGHT_HOST_PLACEMENT=1` turns it on for models that load with it (zaya1-8b and smaller, -measured); unset or `0` keeps the device copy and a read-only mapping. The 27B loads in 24 s with it off. +### Server memory: HRX reads the weights instead of mapping them ([engine#856](https://github.com/1bit-MONSTER/engine/issues/856), [engine#1339](https://github.com/1bit-MONSTER/engine/issues/1339)) + +With a model on HRX (`-ngl` > 0), the loader reads the weights into their device buffers with `read()` and drops each weight's +page-cache pages as soon as it is read ([llama.cpp #234](https://github.com/1bit-MONSTER/llama.cpp/pull/234)). A mapped load +left the whole file in the page cache next to the device copy. On Qwen3.8-27B UD-Q4_K_XL (16.4 GiB), from a cold cache: +peak RSS 2.9 GiB instead of 16.4 GiB, nothing of the file left cached instead of up to 16.5 GiB, and a load of 18.6 s instead +of 14.3 s. Output and wikitext perplexity are unchanged. `--mlock` with mmap keeps the mapping. + +The earlier weight-mapping import ([llama.cpp #226](https://github.com/1bit-MONSTER/llama.cpp/pull/226), opt-in since +[#231](https://github.com/1bit-MONSTER/llama.cpp/pull/231)) is gone, and `GGML_HRX_WEIGHT_HOST_PLACEMENT` does nothing. +It registered the file mapping as a KFD userptr, which the kernel always write-faults, so it dirtied the file, changed +its mtime, and never finished on large models while writeback kept invalidating it (engine#1339). HRX's repacked copy of the +dense weights still sits next to the device buffer in GTT. ### Known issues on `HRX0` @@ -958,10 +960,9 @@ in [hrx.md at 0baf286](https://github.com/1bit-MONSTER/engine/blob/0baf286/docs/ P = 0 x V depend on that V's sign, so identical requests gave different logits (up to 0.13 on Qwen3-0.6B, 13.5 on GLM-4.7-Flash). Masked rows now reach P x V as exactly +0; Qwen3-0.6B and Qwen3.8-27B repeat bit-identically. Cost: pp512 -4.7% on Qwen3-0.6B, the rest within noise. -- **Weight-mapping import, opt-in** ([llama.cpp #226](https://github.com/1bit-MONSTER/llama.cpp/pull/226), - [#231](https://github.com/1bit-MONSTER/llama.cpp/pull/231)): `GGML_HRX_WEIGHT_HOST_PLACEMENT=1` imports the loader's - mapping instead of a second device copy; off by default because it hangs large loads - ([section above](#server-memory-the-weight-mapping-import-is-opt-in-engine856-engine1339)). +- **Weights read, page cache dropped** ([llama.cpp #234](https://github.com/1bit-MONSTER/llama.cpp/pull/234)): with a model + on HRX the loader reads the weights instead of mapping the file and drops their page cache; replaces the weight-mapping + import of #226/#231 ([section above](#server-memory-hrx-reads-the-weights-instead-of-mapping-them-engine856-engine1339)). - **A cap on the graph program cache** ([llama.cpp #63](https://github.com/1bit-MONSTER/llama.cpp/pull/63)): `GGML_HRX_GRAPH_PROGRAM_CACHE`, default 64 ([section above](#server-memory-the-graph-program-cache-cap-llamacpp-63)). - **Hadamard-rotated Q4_0 files** ([llama.cpp #58](https://github.com/1bit-MONSTER/llama.cpp/pull/58)): diff --git a/registry/architectures.json b/registry/architectures.json index e5c237102..24392b3f4 100644 --- a/registry/architectures.json +++ b/registry/architectures.json @@ -1,7 +1,7 @@ { "about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.", "sources": { - "llama.cpp (hrx)": "67d8c42ec65ff6ec963b0d55d166ac888da2b438" + "llama.cpp (hrx)": "ed43da0b4d0fe91fa31497560ea28c7b49e492d8" }, "counts": { "hrx": 298, diff --git a/third_party/llama.cpp b/third_party/llama.cpp index 67d8c42ec..ed43da0b4 160000 --- a/third_party/llama.cpp +++ b/third_party/llama.cpp @@ -1 +1 @@ -Subproject commit 67d8c42ec65ff6ec963b0d55d166ac888da2b438 +Subproject commit ed43da0b4d0fe91fa31497560ea28c7b49e492d8