From 64eac099f451064baa4631129c945c4f0cf8f6bf Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 8 Nov 2026 09:26:41 -0400 Subject: [PATCH 1/2] package: the payload is checked for build-machine paths on every packed ELF (Fixes #1461) 146 build-machine paths shipped in c32's payload - lemond 126, lemonade 13, libloomc.so.0 7 - while the row that exists for them printed "no build-machine paths", because it looked at one file of 195: the packed `1bit` (which engine#1460 had just fixed). The row was right about the defect and too narrow about where it looked, and the pattern was too weak as well: it counted matching LINES, and a strings run glues paths to their neighbours, so one of lemonade's 13 does not begin a line and `grep -c` reports 12 for a file carrying 13. Three changes, one per mechanism, and the row that holds the line: 1. scripts/weekly.sh gains `build_path_strings `, and BOTH packages call it: package_checks for the engine tarball (where the 7 Loom strings are) and lemonade_package_checks for the Lemonade package that carries lemond and lemonade (the 139). Every packed ELF is walked, matches are counted with `grep -o` (occurrences, not lines), and the patterns are /home/, the source tree, the workdir and the build tree - a build that ran somewhere else on this machine leaks the same way. 2. cmake/hrx.cmake's ONEBIT_HRX_PATH_MAP_FLAGS also maps ${CMAKE_BINARY_DIR}=build. ggml-hrx fetches Loom into the BUILD tree (ggml-hrx-deps-build/loom), so mapping only the engine source tree and the add-on left every Loom __FILE__ absolute. 3. docs/releases.md's "146 strings" paragraph becomes the claim it promised: the three mechanisms are fixed (engine#1464 for Lemonade, this one for the build tree, llama.cpp#243 for the nested deps build) and the release checks them, instead of describing them. What stays irreproducible from the tree is the Lemonade dev version's ~N. Measured 2026-11-08: the new row on c33's shipped lemonade-onebit payload (the re-roll the issue measured): FAIL libloomc.so.0 names a build-machine path (7 occurrences): /home/bcloud/wt/c33-payload/build ... FAIL lemond names a build-machine path (126 occurrences): /home/bcloud/wt/c33-payload ... FAIL lemonade names a build-machine path (13 occurrences): /home/bcloud/wt/c33-payload ... rc=1 (146 occurrences, exactly the issue's number, and `1bit` is not among them - #1460 holds) the map the Loom half needs, on the object the issue names, compiled with one flag added: loom_ops_low_low/builders.c.o as shipped: 1 x /home/.../ggml-hrx-deps-build/loom/src/loom/ops/low/builders.c the same compile + -ffile-prefix-map==build: 0 x /home/, and it reads loom/loom/ops/low/builders.c cmake/hrx.cmake's variable, evaluated (cmake -P) with and without the add-on: -ffile-prefix-map==. -ffile-prefix-map==addon -ffile-prefix-map==build -ffile-prefix-map==. -ffile-prefix-map==build The Loom 7 go green only once llama.cpp#243 is merged and this repository pins it; until then the row is deliberately RED on a leaking payload, which is what it is for. The pin bump follows in this PR. checks: bash -n clean; shellcheck -S warning reports only the three pre-existing findings (97, 280 and the sha256sum $(ls ...) line); tools/copyright.py --check rc 0. --- cmake/hrx.cmake | 9 +++++++-- docs/releases.md | 15 +++++++++++---- scripts/weekly.sh | 40 +++++++++++++++++++++++++++++----------- 3 files changed, 47 insertions(+), 17 deletions(-) diff --git a/cmake/hrx.cmake b/cmake/hrx.cmake index 2b64394cc..6d3bd2690 100644 --- a/cmake/hrx.cmake +++ b/cmake/hrx.cmake @@ -68,13 +68,18 @@ if(NOT ONEBIT_HRX_TOOLCHAIN) endif() message(STATUS "ONEBIT_HRX: toolchain ${ONEBIT_HRX_TOOLCHAIN}") -# The HRX binaries carry strings from this tree and, with -DONEBIT_GPU_PRIVATE, from the add-on -# checkout; both are rewritten so the shipped libraries name no build-machine path. +# The HRX binaries carry strings from this tree, from the add-on checkout (-DONEBIT_GPU_PRIVATE), and from +# the BUILD tree: ggml-hrx builds its deps as a nested ExternalProject that fetches Loom into +# /hrx/llama/ggml/src/ggml-hrx/hrx/src/ggml-hrx-deps-build/loom, which is where libloomc.so.0's +# __FILE__ strings come from (engine #1461: 7 of the payload's 146 build-machine paths, and mapping only +# the source tree left every one of them absolute). All three are rewritten so the shipped libraries name +# no build-machine path. -ffile-prefix-map covers debug info as well, which the Release -g needs. set(ONEBIT_HRX_PATH_MAP_FLAGS "-ffile-prefix-map=${CMAKE_SOURCE_DIR}=." CACHE STRING "Path mapping for the HRX llama.cpp build") if(ONEBIT_GPU_PRIVATE) string(APPEND ONEBIT_HRX_PATH_MAP_FLAGS " -ffile-prefix-map=${ONEBIT_GPU_PRIVATE}=addon") endif() +string(APPEND ONEBIT_HRX_PATH_MAP_FLAGS " -ffile-prefix-map=${CMAKE_BINARY_DIR}=build") # Flags the HRX llama.cpp build needs from this toolchain, kept apart from the path maps above. set(ONEBIT_HRX_EXTRA_FLAGS "") diff --git a/docs/releases.md b/docs/releases.md index 863bf5e53..ec61c5f7a 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -140,10 +140,17 @@ shipped c32 payload: `bin/lemond` carries 126, `bin/lemonade` 13, `1bit/hrx/llam is 0 since engine#1460), every one of them naming the rolling machine's home directory, e.g. `…/wt/release-rc/work/src/third_party/lemonade/src/cpp/server/server.cpp`. They are `__FILE__` values: the engine's `-ffile-prefix-map` (`cmake/hrx.cmake`) rewrites its own objects but does not reach the nested -`ggml-hrx-deps-build` that compiles Loom, and `scripts/build-lemonade.sh` sets no prefix map at all. So **two rolls of -the same commit produce different payload hashes**, and a sha you are given cannot be reproduced from the source -alone — rebuild, compare behaviour, and compare the engine's own files. engine#1461 carries the mechanism and the fix; -when it lands this paragraph becomes the claim it should have been: the payload is reproducible from the tree. +`ggml-hrx-deps-build` that compiles Loom, and `scripts/build-lemonade.sh` set no prefix map at all — so **two rolls of +the same commit produced different payload hashes**, and a sha you are given could not be reproduced from the source +alone. + +The three mechanisms are fixed (engine#1461), and the payload is now **checked** rather than described: +`scripts/build-lemonade.sh` maps its source tree and its build directory (engine#1464); `cmake/hrx.cmake` also maps +the build tree, which is where Loom's sources are fetched; llama.cpp passes the same flags into its nested +`GGML_HRX_CMAKE_ARGS` (llama.cpp#243), which is where the 7 Loom strings came from; and `scripts/weekly.sh` +(`build_path_strings`) walks **every** packed ELF of both packages — not just the engine binary — counting occurrences +rather than lines, so a roll that leaks one fails instead of shipping. What stays irreproducible from the tree is the +Lemonade dev version's `~N` above, which is a count of commits in the checkout that built it. Artifacts of the gated c19 candidate — sizes and sha256 exactly as its own `SHA256SUMS` records them: diff --git a/scripts/weekly.sh b/scripts/weekly.sh index 20f008417..e814bd49c 100755 --- a/scripts/weekly.sh +++ b/scripts/weekly.sh @@ -323,6 +323,31 @@ stage_test() { run_tests; } # plaintext ELF; every ELF in the tarball is for the host (x86-64); the packed 1bit has no # symbol table and its strings name no private source file and no build-machine home # Checks the build's 1bit and every file in the tarball. +# engine #1461: no packed ELF may name a build-machine path. One row, used for BOTH packages - the engine +# tarball and the Lemonade package that carries it - over every ELF rather than the packed `1bit` alone: that +# narrow scope is why 146 build paths shipped in c32's payload (lemond 126, lemonade 13, libloomc.so.0 7) +# while the row printed "no build-machine paths", exactly as #1460's `/home/`-only predicate had been too +# narrow a pattern. OCCURRENCES are counted, not lines: a strings run glues paths to their neighbours, so one +# of lemonade's 13 does not begin a line and `grep -c` reports 12 for a file carrying 13 (measured +# 2026-11-08 on c32's own bytes). /home/ covers any builder's home; the source tree, the workdir and the +# build tree cover a build that ran elsewhere on this machine. +build_path_strings() { # build_path_strings -> one FAIL line per offending file, rc 1 + local t=$1 f n pats rc=0 + pats=$(printf '%s\n' /home/ "${SRC:-/nonexistent-build-src}" "${W:-/nonexistent-build-w}" \ + "${BUILD:-/nonexistent-build-build}") + while IFS= read -r -d '' f; do + pack_is_elf "$f" || continue + n=$(strings -a "$f" | grep -o -F -f <(printf '%s\n' "$pats") | wc -l || true) + [ "$n" = 0 ] || { + printf 'FAIL %s names a build-machine path (%s occurrences): %s\n' "$(basename "$f")" "$n" \ + "$(strings -a "$f" | grep -o -F -f <(printf '%s\n' "$pats") | head -2 | tr '\n' ' ')" + rc=1 + } + done < <(find "$t" -type f -print0) + [ "$rc" = 0 ] && echo "ok no packed ELF names a build-machine path (/home/, the source tree, the workdir, the build tree)" + return $rc +} + package_checks() { # package_checks # (no `cmd | grep -q` here: under pipefail grep's early exit fails the pipeline on a match) # (lists go to stderr through `tee >(cat >&2)`, not `tee /dev/stderr`: reopening the log by @@ -439,6 +464,7 @@ package_checks() { # package_checks [ "$m" = 62 ] || { echo "FAIL $f is an ELF for machine $m, not the host (x86-64)"; nk=$((nk + 1)); } done < <(find "$t" -type f -print0) [ "$nk" = 0 ] && echo "ok every ELF in $(basename "$1") is a host (x86-64) program or library" || bad=1 + build_path_strings "$t" || bad=1 f=$(find "$t" -maxdepth 2 -type f -name 1bit) f=${f%%$'\n'*} if [ -n "$f" ]; then @@ -451,17 +477,7 @@ package_checks() { # package_checks done | grep -E '\.(c|cc|cpp|cxx|h|hpp|py)$' | xargs -r -n1 basename | LC_ALL=C sort -u) n=0; [ -z "$privsrc" ] || n=$(strings -a "$f" | grep -c -F -f <(printf '%s\n' "$privsrc") || true) [ "$n" = 0 ] || { echo "FAIL the packed 1bit names a private source file ($n strings)"; b1=1; } - # Any path that names THIS machine's tree, not just /home/: a build under /build or /opt/ci leaks the builder - # the same way, and a /home/-only row let c32 ship a builder path under /opt. Measured 2026-11-08: the c32 and - # c33 tarballs baked /opt/rocm-therock/... and /home//wt/.../libhsa-runtime64.so.1 respectively, and - # only the second tripped this row - the row was right about the leak and too narrow about where it lives. - _badpats=$(printf '%s\n' /home/ "${SRC:-/nonexistent-build-src}" "${W:-/nonexistent-build-w}") - n=$(strings -a "$f" | grep -c -F -f <(printf '%s\n' "$_badpats") || true) - [ "$n" = 0 ] || { - echo "FAIL the packed 1bit names a build-machine path ($n strings): $(strings -a "$f" | grep -F -f <(printf '%s\n' "$_badpats") | head -2 | tr '\n' ' ')" - b1=1 - } - if [ $b1 = 0 ]; then echo "ok the packed 1bit: no symbol table, no private source names, no build-machine paths"; else bad=1; fi + if [ $b1 = 0 ]; then echo "ok the packed 1bit: no symbol table, no private source names"; else bad=1; fi fi for f in LICENSE NOTICE LICENSE-BINARY NOTICE-BINARY; do n=$(find "$t" -maxdepth 2 -type f -name "$f" | wc -l) @@ -497,6 +513,8 @@ lemonade_package_checks() { # lemonade_package_checks n=$(find "$t" -path '*/bin/resources/web-app/index.html' -type f | wc -l) if [ "$n" != 0 ]; then echo "ok Lemonade's web app: bin/resources/web-app/index.html and $(find "$t" -path '*/bin/resources/web-app/*' -type f | wc -l) files in $(basename "$1")" else echo "FAIL no Lemonade web app (bin/resources/web-app/index.html) in $(basename "$1")"; rc=1; fi + build_path_strings "$t" || rc=1 # engine#1461: lemond 126 and lemonade 13 shipped from here + # the engine's own binaries are checked in package_checks; this is the other half of the same payload n=$(find "$t" -maxdepth 2 -name lemond-onebit -type f -exec grep -c '^export ONEBIT_KCRYPT_REQUIRE_SEALED=1$' {} + 2>/dev/null) n=${n%%$'\n'*} if [ "${n:-0}" != 0 ]; then echo "ok lemond-onebit loads sealed NPU kernels only (ONEBIT_KCRYPT_REQUIRE_SEALED=1)" From 5b87274fa97dc73296637058a718c1dca9d088cf Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 8 Nov 2026 10:38:06 -0400 Subject: [PATCH 2/2] third_party: pin llama.cpp 6ea38d0 (main's pin + the path-map forwarding) and follow it in the registry (Fixes #1461) The Loom 7 need llama.cpp to pass CMAKE_C_FLAGS/CMAKE_CXX_FLAGS into its nested deps ExternalProject (llama.cpp#243, merged into 1bit/hrx-vulkan-patched as 6ea38d0). Main's own pin moved to fb10c19caf01 in #1476, i.e. exactly the parent of that merge, so the pin moves by ONE commit and drops nothing: compare fb10c19...6ea38d0: status=ahead ahead_by=1 behind_by=0 (My first attempt pinned a branch cut from the then-current pin 0ce01daf; the compare API answered status=diverged behind_by=423 against the new pin, i.e. it would have dropped the whole llama.cpp sync #1476 had just brought in. Rebased instead of forcing it.) registry/architectures.json regenerated with tools/registry_build.py: one line, the recorded sha, and the architecture set unchanged (hrx 298, npu 9, mapped 298), which is the expected shape for a cmake-only commit. ctest registry_pins is what caught the omission on the previous head, and tools/check_pins.py origin/main is rc 0 here. --- registry/architectures.json | 2 +- third_party/llama.cpp | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/registry/architectures.json b/registry/architectures.json index 0d1fd43a4..70c3616c1 100644 --- a/registry/architectures.json +++ b/registry/architectures.json @@ -1,7 +1,7 @@ { "about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.", "sources": { - "llama.cpp (hrx)": "fb10c19caf0142d4029421d487de596900e6f73a" + "llama.cpp (hrx)": "6ea38d0457d32c30583ed11d634e9afd146f9103" }, "counts": { "hrx": 298, diff --git a/third_party/llama.cpp b/third_party/llama.cpp index fb10c19ca..6ea38d045 160000 --- a/third_party/llama.cpp +++ b/third_party/llama.cpp @@ -1 +1 @@ -Subproject commit fb10c19caf0142d4029421d487de596900e6f73a +Subproject commit 6ea38d0457d32c30583ed11d634e9afd146f9103