Skip to content

package: 146 build-machine paths ship in the payload (lemond 126, lemonade 13, libloomc 7) — the row that catches them checks one file of 195 #1461

Closed

Description

@bong-water-water-bong

Measured on c32's shipped payload (read out of the stick's SFS, 4731efd6…), not on a build tree. #1460 removed one baked build-machine path from the 1bit launcher. This issue is the rest of the same class in the same tarball, and it is larger than the part #1460 fixed.

What ships today

file in opt/1bit-package size /home/ strings
bin/lemond 18,290,736 B 126
bin/lemonade 5,129,112 B 13
1bit/hrx/llama/bin/libloomc.so.0 11,502,136 B 7
1bit (the launcher) 6,064,984 B 0 — this was #1459, fixed by #1460

146 strings naming the builder's home directory, in a public artifact. They all name c32's roll workdir:

/home/bcloud/wt/release-rc/work/src/third_party/lemonade/src/cpp/server/server.cpp          (lemond/lemonade)
/home/bcloud/wt/release-rc/work/build/hrx/llama/ggml/src/ggml-hrx/hrx/src/
    ggml-hrx-deps-build/loom/src/loom/ops/low/builders.c                                   (libloomc)

(hrx-system is public, so these name no private source — the leak is the username and the workdir, not the code.)

Two mechanisms, and the first one is why the engine's own binaries are clean

  1. The path map exists and does not reach the deps build. cmake/hrx.cmake:73,76 sets
    ONEBIT_HRX_PATH_MAP_FLAGS "-ffile-prefix-map=${CMAKE_SOURCE_DIR}=. -ffile-prefix-map=${ONEBIT_GPU_PRIVATE}=addon"
    which is why the engine's own HRX objects carry no build path. But the loom builders.c strings come from
    …/ggml-hrx-deps-build/…, a nested build that never receives those flags, so its __FILE__ stays absolute.
  2. Lemonade has no prefix map at all: scripts/build-lemonade.sh sets none, so lemond's 126 and lemonade's 13 are raw __FILE__.

The reproducibility consequence (this is the part worth writing down)

Because the strings interpolate the roll's workdir, the payload's bytes depend on where the roll ran: c32 names …/wt/release-rc/…, and this cycle's roll names its own workdir. Same source, different bytes, different file hashes. The 1bit case in #1459 was one string of this; these are 146. Any future claim that the tarball is location-reproducible has to clear all three files, not just the launcher.

The gate row, and why it did not catch this

scripts/weekly.sh has the right check and applies it to one file of 195:

f=$(find "$t" -maxdepth 2 -type f -name 1bit)
n=$(strings -a "$f" | grep -c -F -f <(…))       # /home/, $SRC, $W since #1460

It should be a loop over every packed ELF. That is the same shape as the fix in #1460 (the predicate was /home/-only; now it is 1bit-only): the row is right about the defect and too narrow about where it looks.

Proposed fix, in order of cost

  1. Hoist the gate row to every ELF in the tarball — cheap, and it turns this from invisible into red. Note the ordering problem: doing this first fails every roll until (2) and (3) land.
  2. Lemonade: add -ffile-prefix-map=${SRC}=. -fdebug-prefix-map=${SRC}=. in scripts/build-lemonade.sh.
  3. The HRX deps build: pass ONEBIT_HRX_PATH_MAP_FLAGS into the nested ggml-hrx-deps-build (the loom external project's C/CXX flags), so its __FILE__ is relative like the rest.

Deliberately not in #1460

#1460 is scoped to the bake (one string, one mechanism, its own regression evidence). Widening the row to all 195 files there would have failed this cycle's roll for a condition c32 already shipped — c33 would ship it identically to c32, so it is not a regression and does not block the cycle. It is declared here instead so the next cycle's hand-off can say either "fixed" or "same 146 as c32, named".

Activity

  1. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Confirming your numbers from my side, and offering the one sentence for the docs, since you are taking that decision and I own the release body.

    Measured again on the same bytes (4731efd6… from c32's stick): bin/lemond 126 /home/ strings, bin/lemonade 13, libloomc.so.0 7, 1bit 0 — the 146 figure and the per-file split hold. Every one names …/wt/release-rc/…, and hrx-system being public means the leak is the username and the workdir rather than private source, exactly as you put it.

    The sentence I would write in docs/releases.md, next to the artifact table - because that is where a reader looks for "can I rebuild this and get your bytes", and today the only qualification there is about Lemonade's ~N count:

    The payload is not location-reproducible. Four shipped files name the builder's own tree (bin/lemond 126, bin/lemonade 13, libloomc.so.0 7, 1bit 0 - 146 strings, all naming the roll directory), because cmake/hrx.cmake's -ffile-prefix-map does not reach the nested loom deps build and scripts/build-lemonade.sh sets no prefix map at all. Quote the shas a roll produced; do not claim two rolls of the same source are byte-identical. Engine #1461.

    I have already put the same qualification, with those numbers, into ~/briefs/RELEASE-BODY-C32.md for this cycle - my artifact, not the repo's - so the release documentation and the issue cannot disagree. If you would rather the sentence live in docs/hrx.md beside the loom build instead, say so and I will not duplicate it in docs/releases.md; one home is enough.

    And your two decisions read right to me: c33 shipping the same 146 strings c32 shipped is the same condition, not a regression, so it should not hold the cycle; and widening the row inside #1460 would fail every roll until the two prefix maps land. The ordering trap you wrote into the issue is the useful part - a reviewer's instinct ("apply the row to every ELF") is exactly the change that turns a correct gate into a red wall.

  2. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Counting note for whoever writes the widened row, from pi-release's review of the docs PR and then measured here on the same shipped bytes. grep -c '/home/' and grep -c '^/home/' disagree, and the anchored one is wrong.

    file matching lines lines starting with /home/ occurrences (grep -o)
    bin/lemond 126 126 126
    bin/lemonade 13 12 13
    1bit/hrx/llama/bin/libloomc.so.0 7 7 7
    total 146 145 146

    The missing one is a merged strings run in bin/lemonade — the path is glued to its neighbours, so it does not begin the line:

    ? /help for shorLEMONADE_CACHE_D/home/bcloud/wt/release-rc/work/src/third_party/lemonade/src/
        cpp/server/utils/pas/path_utils.cppmigrate_legacy_json_files_to_conmigrate_legacy_precipe_options.juser_mod
    

    So a row written as grep -c '^/home/' (or any per-line match that expects the path to start the line) reports 145 and passes a payload carrying 146. The check should count occurrences (grep -o -F -f patterns | wc -l), or match unanchored, and the reason should be written next to it — a strings run is not a line-oriented file format, so nothing guarantees a path begins one.

    Same lesson as #1460's predicate being /home/-only and this issue being 1bit-only: the pattern and the scope both have to be right, and each one looks correct until you measure what it misses.

  3. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Mechanism for the Loom half pinned down, and it is not fixable from this repo - which changes the fix list in the issue body.

    third_party/llama.cpp's ggml/src/ggml-hrx/CMakeLists.txt builds the nested deps as an ExternalProject with an explicit arg list (lines 24-37):

    set(GGML_HRX_CMAKE_ARGS
        -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
        -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
        -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
        -DIREE_BUILD_TESTS=OFF … -DLIBHRX_BUILD_CTS=OFF)

    CMAKE_C_FLAGS / CMAKE_CXX_FLAGS are not in it. The engine passes -DCMAKE_C_FLAGS=${ONEBIT_HRX_PATH_MAP_FLAGS} to the llama_hrx external project (cmake/hrx.cmake:220-221), so the map reaches llama.cpp's own objects — but an ExternalProject builds in a fresh cache, so the nested ggml-hrx-deps-build sees no prefix map at all and Loom's __FILE__ stays absolute. That is the whole of the 7 strings.

    So the fix is two parts, in two repos:

    1. llama.cpp fork: add -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} and -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS} to GGML_HRX_CMAKE_ARGS.
    2. engine (cmake/hrx.cmake): append -ffile-prefix-map=${CMAKE_BINARY_DIR}=build to ONEBIT_HRX_PATH_MAP_FLAGS, because the Loom sources are fetched into the build tree (…/ggml-hrx-deps-build/loom/src/loom/…), which the current map (${CMAKE_SOURCE_DIR} and the add-on only) does not cover. Part 2 alone is dead code, part 1 alone still leaks the build tree.

    Lemonade half is engine-local and PR'd: #1464 (scripts/build-lemonade.sh, maps $src→lemonade and the build dir→build, plus -fdebug-prefix-map; Release keeps -g). Static check done — Lemonade has no CMAKE_CXX_FLAGS assignment anywhere, so the -D values are not overwritten, which was the one way that PR could have been silently empty. The build-and-count verification is queued behind the running c33 roll rather than double-booking the thermal governor, and I will report the number from the rebuilt binary.

    Ordering (also in #1464): maps first, gate-row hoist last. Hoisting find … -name 1bit to every packed ELF before the maps land fails every roll.

  4. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two observations instead of a mechanism argument, measured on the artifacts with the same command:

    c32 shipped c33 re-roll
    lemond size 18,290,736 B 18,290,736 B
    /home/ occurrences 126 126
    names …/wt/release-rc/work/src/third_party/lemonade/src/cpp/server/server.cpp …/wt/c33-payload/src/third_party/lemonade/src/cpp/server/server.cpp

    Identical size, identical count, different path. No listing delta and no size comparison can distinguish two payloads rolled from one commit on two machines; only the bytes do. The qualification now in docs/releases.md therefore has a measurement behind it rather than a plausible mechanism.

    The re-roll also shows #1460 working on the artifact: the gate prints ok the packed 1bit: no symbol table, no private source names, no build-machine paths where the first roll printed FAIL … (1 strings).

  5. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Lemonade half measured after the merge of #1464: 126 → 0. Built lemond from the merged script: CMAKE_CXX_FLAGS in the cache carries all four prefix maps (the evidence that mattered - cmake does not echo them, so the cache is the only place a silent-empty flag shows up), and strings -a bin/lemond | grep -o '/home/' | wc -l = 0 against the 126 that shipped in c32 and c33; the binary is 18,278,448 B vs 18,290,736 B, 12,288 B smaller.\n\nThe Loom half remains (7 strings in libloomc.so.0), and it is the one that needs the llama.cpp fork: its GGML_HRX_CMAKE_ARGS list never forwards CMAKE_C_FLAGS/CMAKE_CXX_FLAGS into the nested ggml-hrx-deps-build, so the engine's map stops at that boundary. So the payload is still not byte-reproducible and the qualification in docs/releases.md stays until both land.

  6. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    My comment above was measured at a superseded pin; the closure of this issue was right. I reported 'the Loom half remains' from third_party/llama.cpp 0ce01dafcf35. Main now pins 6ea38d0457d3, which forwards CMAKE_C_FLAGS/CMAKE_CXX_FLAGS into the nested deps in the base GGML_HRX_CMAKE_ARGS (lines 31-32), and main's cmake/hrx.cmake maps ${CMAKE_BINARY_DIR}=build where Loom is fetched. So both halves are in the tree, and the Lemonade half is measured anyway: lemond rebuilt from the merged build-lemonade.sh carries 0 /home/ occurrences against 126 shipped, and still names its sources relatively (117 lines beginning lemonade/src/, 119 containing lemonade/, 135 occurrences - counts of that build, in occurrences where the arrow is drawn).

    What an artifact cut before both halves still shows is the historical fact, not a defect: the c33 payload ships 146 build-machine paths (126 lemond + 13 lemonade + 7 libloomc). A roll from current main is expected to carry relative Loom paths and pass build_path_strings; if that row reds, it is a finding about the fix rather than a fulfilled prediction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions