From b2bed00e6992492c20418ebac96b4b06ff43ec73 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sat, 7 Nov 2026 11:26:00 -0400 Subject: [PATCH 1/2] hrx: pin the fork with the weight-mapping import opt-in, so large models load again (Fixes #1339) Co-Authored-By: Claude Opus 5.5 --- docs/hrx.md | 16 ++++++++++++++++ registry/architectures.json | 2 +- third_party/llama.cpp | 2 +- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/docs/hrx.md b/docs/hrx.md index 4379ab09d..b0cb9927d 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -623,6 +623,18 @@ length peak at 8.7 GiB instead of growing; greedy answers are identical, and lla is unchanged. A repeated prompt length right after a flush costs 1.75 s instead of 1.12 s, since its program is rebuilt. +### Server memory: the weight-mapping import is opt-in ([engine#856](https://github.com/1bit-MONSTER/engine/issues/856), [engine#1339](https://github.com/1bit-MONSTER/engine/issues/1339)) + +At `-ngl 99` the loader copies every offloaded weight into one device buffer, and HRX then serves each dense +matmul from its own repacked copy, so the weights were resident twice in GTT. Since +[llama.cpp #226](https://github.com/1bit-MONSTER/llama.cpp/pull/226) HRX can import the loader's file mapping +instead of taking that copy (Qwen3-4B-Q4_K_M: 3925 -> 1243 MiB of the process's GTT, wikitext perplexity +16.3000 in both arms). On larger files the import never finishes: a dense Qwen3.8-27B (16.4 GiB) pages the +whole file in, then waits with no error and no progress. Qwen3.8-Flash-Next (76.3 GiB) does the same. +Since [llama.cpp #231](https://github.com/1bit-MONSTER/llama.cpp/pull/231) the import is **off by default**. +`GGML_HRX_WEIGHT_HOST_PLACEMENT=1` turns it on for models that load with it (zaya1-8b and smaller, +measured); unset or `0` keeps the device copy and a read-only mapping. The 27B loads in 24 s with it off. + ### Known issues on `HRX0` - **Several sequences per batch fail.** `llama-perplexity` with `n_seq` > 1 stops on an @@ -919,6 +931,10 @@ in [hrx.md at 0baf286](https://github.com/1bit-MONSTER/engine/blob/0baf286/docs/ P = 0 x V depend on that V's sign, so identical requests gave different logits (up to 0.13 on Qwen3-0.6B, 13.5 on GLM-4.7-Flash). Masked rows now reach P x V as exactly +0; Qwen3-0.6B and Qwen3.8-27B repeat bit-identically. Cost: pp512 -4.7% on Qwen3-0.6B, the rest within noise. +- **Weight-mapping import, opt-in** ([llama.cpp #226](https://github.com/1bit-MONSTER/llama.cpp/pull/226), + [#231](https://github.com/1bit-MONSTER/llama.cpp/pull/231)): `GGML_HRX_WEIGHT_HOST_PLACEMENT=1` imports the loader's + mapping instead of a second device copy; off by default because it hangs large loads + ([section above](#server-memory-the-weight-mapping-import-is-opt-in-engine856-engine1339)). - **A cap on the graph program cache** ([llama.cpp #63](https://github.com/1bit-MONSTER/llama.cpp/pull/63)): `GGML_HRX_GRAPH_PROGRAM_CACHE`, default 64 ([section above](#server-memory-the-graph-program-cache-cap-llamacpp-63)). - **Hadamard-rotated Q4_0 files** ([llama.cpp #58](https://github.com/1bit-MONSTER/llama.cpp/pull/58)): diff --git a/registry/architectures.json b/registry/architectures.json index eba3e24a3..110469cb4 100644 --- a/registry/architectures.json +++ b/registry/architectures.json @@ -1,7 +1,7 @@ { "about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.", "sources": { - "llama.cpp (hrx)": "a3e8d42c9984470e88114ed0195759679ffbeb71" + "llama.cpp (hrx)": "da22b63bac63645eed07b54718ad206793c9c69f" }, "counts": { "hrx": 298, diff --git a/third_party/llama.cpp b/third_party/llama.cpp index a3e8d42c9..da22b63ba 160000 --- a/third_party/llama.cpp +++ b/third_party/llama.cpp @@ -1 +1 @@ -Subproject commit a3e8d42c9984470e88114ed0195759679ffbeb71 +Subproject commit da22b63bac63645eed07b54718ad206793c9c69f From f842ead701d4759feb9cf33fff1fe1420cda2b61 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sat, 7 Nov 2026 12:27:46 -0400 Subject: [PATCH 2/2] hrx: pin the merged fork commit 092eb9dbc (#231) on 1bit/hrx-vulkan-patched Co-Authored-By: Claude Opus 5.5 --- registry/architectures.json | 2 +- third_party/llama.cpp | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/registry/architectures.json b/registry/architectures.json index 110469cb4..4c1aefa67 100644 --- a/registry/architectures.json +++ b/registry/architectures.json @@ -1,7 +1,7 @@ { "about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.", "sources": { - "llama.cpp (hrx)": "da22b63bac63645eed07b54718ad206793c9c69f" + "llama.cpp (hrx)": "092eb9dbcffc43405bc9d5652dcc2cf4ae87ceea" }, "counts": { "hrx": 298, diff --git a/third_party/llama.cpp b/third_party/llama.cpp index da22b63ba..092eb9dbc 160000 --- a/third_party/llama.cpp +++ b/third_party/llama.cpp @@ -1 +1 @@ -Subproject commit da22b63bac63645eed07b54718ad206793c9c69f +Subproject commit 092eb9dbcffc43405bc9d5652dcc2cf4ae87ceea