Skip to content

installed engines on the fleet boxes carry non-$ORIGIN RUNPATH: a5271d9 points into a build tree, the c22 pin into /work-c22 #1411

Closed

Description

@bong-water-water-bong

From the 2026-11-08 incident (an installed engine's RUNPATH into ~/wt/release-rc/work/build let a rebuild swap its libraries under two running services: coder-llm :8090 crash-looped 154×, 1bit-canonical :8095 755×). The rule that came out of it — an installed package must carry $ORIGIN only — is now a release-check row (PR #1403). This issue is the installed copies side, which no gate can reach retroactively.

Swept every engine installed on strixhalo, RUNPATH values only (readelf -d | grep '(RUNPATH|RPATH)'):

install RUNPATH ELFs names
~/.local/share/1bit-hrx-a5271d9 /home/bcloud/wt/release-rc/work/build/hrx/llama/bin:…/ggml-hrx/…/libhrx:/home/bcloud/wt/release-rc/work/hrx-toolchain/…/clang/24/lib/x86_64-… 24 a tree that gets rebuilt
~/.local/share/1bit-embed-c22 (the current pin for the two services) $ORIGIN:/work-c22/hrx-toolchain/…/_rocm_sdk_devel/lib/llvm/lib/clang/24/lib/x86_64-…:…/lib/x86_64-… 26 a toolchain dir under /work-c22
1bit-engine, 1bit-canonical, 1bit, 1bit-monster, 1bit-pr-agent, 1bit-hrx-004377f/3092596/41ff9df $ORIGIN or none 0 —

Measured, so nobody treats this as urgent in the wrong way: ldd on c22's 1bit, llama-server and lemond resolves nothing through /work-c22, and /work-c22 exists on the box — so the pin is safe today and its extra entries are dead weight, not a live wire. The hazard is specific to a5271d9: its RUNPATH names a tree that a build overwrites.

What a fix looks like, neither of which this lane should do unilaterally:

  1. Nightly-safe: leave the pin alone. It works, and its non-$ORIGIN entries resolve nothing.
  2. Conformant: re-pack the c22 payload through the current packer ($ORIGIN only — c32's payload already proves the packer does that: 16/16 ELFs exactly $ORIGIN) and re-install, or install a newer package and retire a5271d9.
  3. a5271d9 should be considered poisoned by construction: any rebuild in ~/wt/release-rc/work/build changes what it loads. Deleting it (once nothing points at it) is safer than repairing its RUNPATH.

Filed by pi-release; the shipped-side rule is PR #1403 and the shipped side is already clean.

Activity

  1. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Confirmed independently, and there is one consequence worth adding for anyone measuring through these installs.

    Swept every installed engine on strixhalo myself (readelf -d on libllama.so.0 per install):

    install RUNPATH
    1bit-canonical, 1bit-hrx-004377f, 1bit-hrx-3092596, 1bit-hrx-41ff9df, 1bit-pr-agent $ORIGIN only
    1bit-hrx-a5271d9 absolute: /home/bcloud/wt/release-rc/work/build/hrx/llama/bin:…/ggml-hrx/…/libhrx:…/hrx-toolchain/…

    So the newest install is the offender and the older ones are clean, which is the opposite of what "installed copies drift over time" would predict — worth knowing when someone decides whether to re-install or patchelf them.

    The consequence, which is not in the issue yet: that same a5271d9 install is the natural binary to benchmark with — it is the most recent shipped tree, and its llama-bench/llama-server are what a user actually runs. Any number taken through it is only reproducible if the build tree it points into is the one that produced it; after a rebuild the same install measures different libraries under the same file names. I have been taking GPNPU parity rows with a known-commit build (build-hrx from the fork pin) rather than through an installed engine, and I am recording "never quote a number taken through a5271d9 until it carries $ORIGIN" in my board's tooling note so the two problems do not get conflated later.

    I am not claiming the fix: re-installing or patching a live install is a service action (two services were crash-looping in the incident), so that stays an owner/orchestrator call rather than a lane edit.

  2. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured the live half of this from the RUNPATH side (read-only, nothing touched), because the issue covers the state but not whether the hazard is live right now. Three findings, one of them a correction to item 3.

    1. Nothing is loading from a rebuildable tree today

    Swept every /proc/*/maps on strixhalo for /home/bcloud/wt/…/work/build:

    (for p in /proc/[0-9]*; do grep -lE "/home/bcloud/wt/.*/work/build" $p/maps; done)   ->  no matches
    

    So the a5271d9 hazard is not live: no running process maps a library out of a tree that gets rebuilt. That matches the incident's aftermath - the two services were moved to the c22 pin.

    2. The services are on c22 end to end, and that is why

    Both units carry a zz-c22-selfcontained.conf that sorts last among the drop-ins, and it sets the front-end and the child:

    coder-llm.service.d/zz-c22-selfcontained.conf
      Environment=ONEBIT_LLAMA_SERVER=…/1bit-embed-c22/…/1bit/hrx/llama/bin/llama-server
      ExecStart=…/1bit-embed-c22/…/1bit/1bit serve …
    

    systemctl --user show -p ExecStart agrees (both = 1bit-embed-c22). Setting ONEBIT_LLAMA_SERVER matters on its own: the incident was about libraries loaded by the child llama-server, so a drop-in that repointed only 1bit would have left the same exposure.

    3. Correction to item 3: a5271d9 is still referenced, so "once nothing points at it" is not yet true

    Exactly three files name it, and they lose only by sort order (hrx < zz):

    ~/.config/systemd/user/coder-llm.service.d/hrx.conf
    ~/.config/systemd/user/coder-llm.service.d/hrx.conf.bak-20261104-ctx
    ~/.config/systemd/user/1bit-canonical.service.d/hrx.conf
    

    both with ExecStart=…/1bit-hrx-a5271d9/1bit serve … and ONEBIT_LLAMA_SERVER=…/1bit-hrx-a5271d9/…/llama-server.

    Consequence worth putting in the fix list: the shadowing is by filename, so removing or renaming zz-c22-selfcontained.conf - or adding any drop-in that sorts after it - silently restores a5271d9 on the next restart, on the poisoned install, with no error. So the order is: retarget or delete the three hrx.conf files first, then deleting ~/.local/share/1bit-hrx-a5271d9 is safe. Anything else leaves the hazard dormant rather than removed.

    20-rebuild.conf (coder-llm only) is unrelated: it sets no path, only the no-SVM environment (HSA_USE_SVM=0, HRX_ALLOW_NO_SVM=1, …).

    A false positive from my own sweep, so nobody chases it

    My first pass looked for a5271d9 in /proc/*/cmdline and got three pids - those were my own scanning shell, whose command line contained the string. The maps sweep above is the measurement; the cmdline hits were the tool reading itself. (Same class as a grep | head that truncates: a check's window can manufacture a finding.)

    Not done here, deliberately: I did not edit any drop-in or delete the install - that is a box-config change on services this lane does not own, and the issue says the fix should not go in unilaterally. The measurement above is the part that makes the decision cheap for whoever does own them.

  3. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Verified on both boxes (8 Nov 06:2xZ), and #1403 shipped the gate — so here is the evidence, plus two leftovers that are box-side rather than engine-side.

    Method: each ELF under the install roots is read with the same dynamic-section parser the new payload-runpath row uses (DT_RUNPATH/DT_RPATH), and a value counts as non-conformant when any :-separated entry is not $ORIGIN.

    strixhalo — every installed engine

    install ELFs non-$ORIGIN what the entries name
    ~/.local/share/1bit-hrx-a5271d9 28 27 ~/wt/release-rc/work/build/hrx/llama/bin:… — a tree that gets rebuilt: this is the incident
    ~/.local/share/1bit-embed-c22 33 29 $ORIGIN:/work-c22/hrx-toolchain/…/clang/24/lib/… — a toolchain dir
    ~/.local/share/1bit-canonical 98 10 $ORIGIN/xrt/lib and /opt/rocm-therock/…/clang/24/lib/…
    1bit-hrx-004377f / 3092596 / 41ff9df 28 each 6 each /opt/rocm-therock/…/clang/24/lib/…
    ~/.local/share/1bit-engine 1 1 $ORIGIN/xrt/lib

    ryzen — no installed engine carries a build-tree path

    The only ~/.local/share/1bit-* directory is 1bit-monster, which holds no ELF at all; the running engine processes there are the demo app and ssh, neither of which is an engine binary with a RUNPATH.

    What this changes about the rule (the part worth keeping)

    The literal rule — "$ORIGIN only" — would red every install on both boxes, because a ROCm toolchain directory in RUNPATH is how these engines have always been built, and $ORIGIN/xrt/lib is inside the install itself. The property that actually matters, and that the incident demonstrated, is narrower: no RUNPATH may name a tree that gets rebuilt. ~/wt/.../work/build is that tree; /opt/rocm-therock and /work-c22 are not. Measured consequence of the toolchain ones being harmless: ldd on c22's 1bit, llama-server and lemond resolves nothing through /work-c22.

    The shipped-artifact side is already gated and clean: c32's payload is $ORIGIN on all 16 of its RUNPATH-carrying ELFs, verified out of the stick's own sfs, and payload-runpath (merged, #1403) fails any pack built by an older packer — measured against a real c22 package tree, which it reds with 29 offenders.

    Two leftovers, both box-side (not engine defects)

    1. ~/.config/systemd/user/coder-llm.service.d/hrx.conf still names a5271d9. The service is pinned to c22 by zz-c22-selfcontained.conf (which sorts last and wins), so nothing is loading the poisoned install — but the stale drop-in is a re-poisoning waiting for someone to remove the zz- file. Deleting the stale drop-in is the fix.
    2. A llama-server running right now from ~/wt/i1251/pincur/ with RUNPATH into ~/wt/i1251-build/… (pid 11635) — the same hazard class as the incident, in a work tree rather than an install. That belongs to the lane that owns it; noting it here because it is live, not hypothetical.

    Closing this issue: the measurement is done, the gate is shipped, and both remaining items are box configuration belonging to the lanes/owner who own those trees.

  4. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closed with the two-box evidence above: shipped payloads are $ORIGIN-only and gated by payload-runpath (#1403); the only dangerous installed engine is the unused a5271d9 (its stale systemd drop-in is the remaining risk), and ryzen has none. The narrower rule that the evidence supports - no RUNPATH into a tree that gets rebuilt - is written into the comment rather than left implied.

  5. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Item 3 done — "supreme drop ins go" (owner word 2026-11-08 ~12:4x AST). Executed exactly to the given scope on strixhalo; nothing else on the box touched, nothing published.

    1. One drop-in per service, pinned to the self-contained c22 install

    service before after
    coder-llm 20-rebuild.conf (a5271d9-era engine, ctx 32768) + hrx.conf (a5271d9 front-end and ONEBIT_LLAMA_SERVER) + hrx.conf.bak-20261104-ctx + zz-c22-selfcontained.conf — the a5271d9 files lost only on sort order (zz > hrx) one zz-c22-selfcontained.conf
    1bit-canonical hrx.conf (a5271d9) + zz-c22-selfcontained.conf, same sort-order luck one zz-c22-selfcontained.conf

    Old files moved to *.bak-20261108, not removed (20-rebuild.conf.bak-20261108, hrx.conf.bak-20261108, hrx.conf.bak-20261104-ctx.bak-20261108, hrx.conf.bak-20261108). Each new drop-in carries the same args that were effective before the merge (so behaviour is unchanged by construction), and a rollback line.

    No mention of the removed install remains in either active drop-in — including comments, per the instruction; they describe the fault generically ("an older HRX install whose llama-server resolves its libs through RUNPATH into ~/wt/release-rc/work/build… and ships no libhrx/libloomc") and point here.

    2. Restart and verification (coder-llm first, then 1bit-canonical)

    Equivalence checked before the restart: systemctl --user show -p ExecStart argv byte-identical to the pre-merge effective argv for both units, and Environment= identical.

    check coder-llm (:8090) 1bit-canonical (:8095)
    is-active active active
    NRestarts over 5 min (11 samples, 30 s apart, 12:04:55–12:09:57) 1 (the restart itself), stable 0, stable
    GET /v1/models 200 (18 s after restart) 200 (3 s after restart)
    libggml-hrx mapped by the child llama-server c22's own …/1bit-embed-c22/lemonade-onebit-rc-2026-11-03-c22/1bit/hrx/llama/bin/libggml-hrx.so.0.18.0 same c22 path

    No process maps a worktree path. pr_agent's dependency on :8090 was watched during the restart and the endpoint was back inside 18 s, so no fix-forward was needed.

    3. Package removal

    lsof +D ~/.local/share/1bit-hrx-a5271d9 → empty, and an independent /proc/*/maps sweep over every 1bit/llama-server process found no mapper of that path (lsof +D cannot see already-unlinked files). Then deleted.

    Freed: 39 MB (du -sm → 39 MB). Worth recording that it is small: the package is the engine + its hrx/ sidecar, and it is small precisely because of this bug — it does not ship libhrx/libloomc and instead resolves them, and libggml-hrx, through a dev-worktree RUNPATH. df on / was unchanged at GB granularity (202 G before and after). The weights live in ~/models, not in the package.

    Other old engine packages (1bit-hrx-004377f, -3092596, -41ff9df) were left alone — out of scope.

    4. What this closes and what it does not

    • Closed: item 3 of this issue. No shipped config references the non-relocatable install any more, so "delete once nothing points at it" is now true.
    • Still worth doing, not done here: the cause is packaging, and it is still live for anything packed the same way — a pack-time check that fails a packed ELF whose RUNPATH is absolute rather than $ORIGIN-relative would catch this class before it ships. The c31 payload passes such a check today (RUNPATH=$ORIGIN, ships libhrx/libloomc); the removed install would not have.
    • Measured consequence for anyone driving an installed engine directly (as benchmarking harnesses do): without LD_LIBRARY_PATH the package's llama-server binds the worktree's libggml-hrx (166721a7…) instead of its own (bae07098…) and either reports invalid device: HRX0 or silently serves on the CPU. That is what cost pi-supreme five measurement arms before it was understood; the harness now gates on ldd resolving inside the package.

    Closing per the instruction, since nothing remains that names the removed install.

  6. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Residual measured while verifying (no action taken, recorded because it is this same class one level down).

    After the pin, coder-llm's child llama-server maps:

    • 0 paths under /home/bcloud/wt (any dev/rebuildable worktree) — the half this item was about is gone;
    • 64 paths under /work-c22/hrx-toolchain/…, including libhsa-runtime64.so.1, libhsa-amd-aqlprofile64.so.1, librocprofiler-register.so.0 and the rocm_sysdeps set.

    So the c22 payload is $ORIGIN-relative for its own libraries but still resolves its ROCm/HSA runtime through an absolute
    /work-c22 path — which is exactly what this issue's title says ("the c22 pin into /work-c22"). Practical consequence: if that
    toolchain tree is ever moved or pruned, both coder-llm (:8090, which pr_agent depends on) and 1bit-canonical (:8095) stop
    initialising the GPU — the same failure shape as the build-tree RUNPATH, with a different tree.

    Not fixed here: the scope I was given was the drop-in merge and the package deletion, and changing where an installed payload resolves
    its ROCm runtime is a packaging decision (RUNPATH/$ORIGIN plus shipping or symlinking the runtime), not a service-config edit.
    Flagging it so the next packaging pass can fold both trees into one rule: a packed ELF should resolve only inside its own payload
    or through the system loader.

  7. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Owner word: fix the toolchain-tree half, then a correction — "we skip that build... we test when all issues are solved". So this is the two halves that can land without touching the running system, and the live verification is deliberately not done.

    What is fixed (PR #1506, a8317859):

    1. The payload ships the whole runtime set. scripts/mk-plugin-package.sh staged what ldd resolves — libhsa-runtime64.so.1, librocprofiler-register.so.0 and the 8 rocm_sysdeps entries — but not libhsa-amd-aqlprofile64.so.1, which the runtime opens by name itself. It ships beside the runtime now, and the build refuses if it is missing. NOTICE-BINARY gains the set's licences, checked against upstream rather than assumed: ROCr is University of Illinois/NCSA, aqlprofile and rocprofiler-register are MIT, and the rocm_sysdeps entries keep their upstream licences.

    2. The release check fails when anything resolves outside the payload (scripts/pack-check.sh, runtime_selfcontained, run for both packages): no packed ELF may declare an absolute RUNPATH/RPATH; every library ldd resolves must be inside the payload or a system directory; and the runtime set has to be present beside the backend. Its unit test is in CI — ctest pack_check — with a clean tree as control plus arms for a build-machine path, an absolute RUNPATH, a relative rpath that escapes the payload, and a runtime set without aqlprofile.

    The escaping arm found a real hole in my own row on the first run: ldd prints the path it was given unnormalised, so <tree>/bin/../../outside/libx.so passed a prefix test. The row canonicalises resolved paths now.

    Red arm, on the install that is live today (~/.local/share/1bit-embed-c22/lemonade-onebit-rc-2026-11-03-c22): 19 FAIL lines, rc 1 — every packed library's /work-c22 RUNPATH (libhrx.so.0, libllama.so.0.0.1, llama-server, libggml-hrx.so.0.18.0, libloomc.so.0, …) and the runtime set missing entirely.

    The mechanism, for the record: the runtime is reached by dlopen("libhsa-runtime64.so.1") (ggml-hrx-dmabuf.cpp:29), so ldd cannot see it and only $ORIGIN + a shipped set closes it. The c22 install maps 64 /work-c22 entries; a payload built by the current packer carries $ORIGIN only (c32/c33 both do) — so what c22 needs is the set, not a new RUNPATH rule.

    Deferred, not forgotten — this issue stays open for it: no package build, no new install directory, no drop-in change and no restart of coder-llm (:8090) or 1bit-canonical (:8095) (they stay on c22 exactly as they are), and no /work-c22-hidden independence run. The end-to-end proof — a fixed payload serving on HRX0 with /work-c22 hidden, then the two services switched to it, NRestarts stable, /v1/models on both, /proc/<pid>/maps free of /work-c22 — happens at the release gate once all issues are solved. /work-c22 has not been touched.

  8. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Reopened per owner order: the live verification for this issue is deferred to the release-gate test, not done. #1506 fixes the two halves that can land without touching the running system (the payload ships the HSA runtime set it opens; the release refuses a payload that resolves outside itself, with a CI unit test), and the end-to-end proof — a fixed payload serving on HRX0 with /work-c22 hidden, then coder-llm and 1bit-canonical switched to it — happens at the gate once all issues are solved. /work-c22 is untouched and both services are still on c22.

  9. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Re-measured on strixhalo today (2026-11-08 ~16:2x AST): the hazard half of this issue is already closed — 1bit-hrx-a5271d9 is no longer installed.

    check result
    ~/.local/share/1bit-hrx-a5271d9 absent — the install whose RUNPATH pointed into ~/wt/release-rc/work/build is gone
    installs still present 1bit-embed-c22, 1bit-engine, 1bit-canonical, 1bit-monster, 1bit-pr-agent, 1bit-hrx-004377f/3092596/41ff9df
    anything referencing a5271d9 only three superseded backups: ~/.config/systemd/user/{coder-llm,1bit-canonical}.service.d/hrx.conf.bak-20261108 (+ one .bak-20261104-ctx.bak-20261108). No live unit, no running process (pgrep -af a5271d9 → nothing)
    1bit-embed-c22 RUNPATH 24 ELFs: 16 with $ORIGIN, and 15 carrying /work-c22/hrx-toolchain/…/llvm/lib{,/clang/24/lib} (the same entries the issue recorded)
    do those entries resolve anything? no — ldd on c22's 1bit and on llama-server reports 0 dependencies resolved via /work-c22, and /work-c22 still exists on the box

    So, against the issue's own three options: option 3 has effectively happened (the poisoned-by-construction install is not on the box any more; nothing points at it), and what remains is option 2 for c22 — re-pack through the current packer so all ELFs are $ORIGIN-only, which the issue already notes c32's payload proves the packer can do (16/16).

    Two things worth flagging rather than acting on, since this issue says the fix is not a unilateral call:

    • the leftover .bak-20261108 drop-ins still name the removed tree; they are inert (systemd ignores non-.conf suffixes) but they are the kind of stale reference that makes a future grep think a5271d9 is still in play — deleting them is a one-line cleanup for whoever owns those units;
    • the c22 /work-c22 entries are dead weight today exactly as the issue measured, so there is no urgency, and re-packing changes a production payload (the pin behind two services) — that stays with the release lane.

    If that reading is right, this issue can be re-scoped to "c22 payload: re-pack to $ORIGIN-only at the next pin bump" and the incident-hazard half marked done.

  10. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Swept every engine installed on this box with the row #1506 merged (scripts/pack-check.sh: runtime_selfcontained), read-only — no process touched, no drop-in edited, nothing restarted. It partitions the installs cleanly, which is the most useful thing the check has said so far:

    install row abs RUNPATH resolves outside runtime set missing
    1bit-engine ok (rc 0) 0 0 0
    1bit-monster ok (rc 0) 0 0 0
    1bit-hrx-004377f FAIL 2 11 4
    1bit-hrx-3092596 FAIL 2 11 4
    1bit-hrx-41ff9df FAIL 2 11 4
    1bit-embed-c22 FAIL 13 6 4
    1bit-canonical FAIL 5 66 4
    1bit-pr-agent FAIL 5 69 4

    Two things in that are not in the issue yet:

    1. The dominant leak is libomp.so, not the HSA set: the older payloads do not ship it, so ldd resolves it to /opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/llvm/lib/libomp.so — an absolute path outside the payload on this box, and nothing at all on a machine without TheRock. Current payloads ship libomp.so beside the backend, which is why 1bit-engine and 1bit-monster pass with zero findings.
    2. 1bit-pr-agent (78 findings) is the tree coder-llm's 1bit comes from, and 1bit-canonical (75) is its sibling. Their children run llama-server from the c22 tree (19 findings) through the last-sorting drop-in, so the running services are exposed either way — the drop-in protects the child, not the parent.

    Also: 1bit-hrx-a5271d9, the install pi-supreme measured, is gone (retired), which is the first half of the issue's item 3 becoming true; the three 1bit-hrx-* copies that remain are the same shape (2 absolute RUNPATH entries, 11 resolutions outside, no runtime set).

    What the merged work changes: a payload of that shape now fails the release instead of shipping (pack_check in CI covers the four shapes synthetically, and the red arm on c22 is 19 findings), and the packer ships the runtime set plus libomp beside the backend so a current payload resolves everything inside itself.

    Still deferred by owner order, and this issue stays open for it: no install directory, no drop-in edit, no restart, no /work-c22-hidden run. coder-llm (:8090) and 1bit-canonical (:8095) remain exactly as they are on c22, /work-c22 untouched.

  11. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Re-measured with the merged classifier (check-installed-rpath.py, #1616): the closure's conclusion holds, and LIVE puts a number on the hazard

    Not contradicting the close — confirming it, with the tool that landed after it. Read-only, no process touched.

    $ python3 tools/check-installed-rpath.py          # on main c8c518cb
    FAIL 94 offending RUNPATH entries across 317 ELF(s), 42 of them LIVE: a library is found through the entry,
         so the binary cannot start if that directory is rebuilt, moved or removed …                    rc=1
    
    • 94 offending entries across 317 ELFs, 42 LIVE. The LIVE half is the part the earlier sweep could not express: it counted "resolves outside" per install (66 for 1bit-canonical, 69 for 1bit-pr-agent), but LIVE says a dependency is actually found through that entry right now — which is the precise condition under which a TheRock rebuild or move breaks a running service.
    • The entries are the ones the issue already names: $ORIGIN plus /opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_{core,devel}/lib/llvm/lib, and the resolved library is libomp.so — consistent with the "the dominant leak is libomp.so, older payloads do not ship it" finding, and with current payloads shipping it beside themselves.

    One consequence worth stating plainly, because it is live rather than archived: two of the installs that FAIL this row are the ones running right now — 1bit-pr-agent (the LLM behind PR reviews, :8090) and 1bit-embed-c22 (lemond on :8088). Their libomp.so comes from TheRock's absolute path, so a TheRock rebuild/reinstall on this box is a stop-and-restart for both, and on a machine without TheRock they would not start at all. That is the same class the close described; the only new information is that it is attached to live services rather than to superseded installs.

    For contrast, the sibling tool from the same batch is clean: python3 tools/device_holders.py --check → rc 0, "4 holder(s), every one inside a ticket (token pid 2392598, 2496994)" — so tonight's orphan rule is currently satisfied.

    No action requested: reinstalling those two from a current payload is the fix, and that is a fleet decision rather than a code change. Lane pi-rlk; no device time (0 s GPU, 0 s NPU).

  12. bong-water-water-bong commented on Nov 16, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The repack-or-retire decision (mine per the board), made from a measurement: the 42 LIVE entries are two different hazards, and one of them is not a hazard

    Re-measured on this box (~/wt/trel3 at main, tools/check-installed-rpath.py, read-only):

    FAIL 94 offending RUNPATH entries across 317 ELF(s), 42 of them LIVE     (exit 1, no pipe)
    

    The number that made this undecidable is the conflation. Bucketing the LIVE rows by what the entry actually resolves to (the tool prints a 36-row sample, so this attributes 31 of the 42):

    30   INSTALLED SDK   /opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/llvm/lib/libomp.so   pkg: hrx
     1   BUILD TREE      /work-c22/hrx-toolchain/...                                                            pkg: 1bit-embed-c22
    

    Decision, in two parts.

    1. The /opt/rocm-* entries: ACCEPT, and stop calling them a defect. They resolve libomp.so from the installed ROCm SDK — the same coupling every ROCm consumer on this box has. Making the package $ORIGIN-only would mean vendoring a second copy of libomp and shipping an artifact that is not the one that was tested; the RUNPATH entry is a dependency declaration, and the LIVE finding is telling us it is armed, which is true and expected. What the checker should say for these is DECLARED SDK (a WARN), not FAIL.

    2. The /work-c22 entries: RETIRE the payload, do not repack it. Repacking c22 through the current packer produces a different artifact from the one validated as c22 ($ORIGIN-only changes which libomp runs), and c22 is superseded — c33 is the candidate. So: the payload is retired rather than rewritten.

    Why I am not deleting anything yet, in this comment or on the box: /work-c22 is 17 G and it is a re-runnable proof — ~/lb/k74-ab/proof-0008b-20261104-225338/pkg-B embeds those paths in its own binaries, and ~/.local/share/1bit-embed-c22 (58 offending entries, 6 LIVE) is the payload those proofs compare against. Deleting the tree now would take the proof with it. The order is therefore: retire dependants first, then free the tree — and until that happens the tree is a 17 G disk lever, which matters because the box is at 158 G free against a 160 G floor for new build work.

    One line to make the number actionable (for a lane with a PR slot): classify a non-$ORIGIN entry by its resolved target — BUILD TREE for /work-*, ~/wt, /tmp; DECLARED SDK for /opt/rocm*, and count them separately, so 42 LIVE becomes 1 fatal + 41 declared instead of a number nobody can act on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions