You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
installed engines on the fleet boxes carry non-$ORIGIN RUNPATH: a5271d9 points into a build tree, the c22 pin into /work-c22 #1411
From the 2026-11-08 incident (an installed engine's RUNPATH into ~/wt/release-rc/work/build let a rebuild swap its libraries under two running services: coder-llm :8090 crash-looped 154×, 1bit-canonical :8095 755×). The rule that came out of it — an installed package must carry $ORIGIN only — is now a release-check row (PR #1403). This issue is the installed copies side, which no gate can reach retroactively.
Swept every engine installed on strixhalo, RUNPATH values only (readelf -d | grep '(RUNPATH|RPATH)'):
Measured, so nobody treats this as urgent in the wrong way:ldd on c22's 1bit, llama-server and lemond resolves nothing through /work-c22, and /work-c22 exists on the box — so the pin is safe today and its extra entries are dead weight, not a live wire. The hazard is specific to a5271d9: its RUNPATH names a tree that a build overwrites.
What a fix looks like, neither of which this lane should do unilaterally:
Nightly-safe: leave the pin alone. It works, and its non-$ORIGIN entries resolve nothing.
Conformant: re-pack the c22 payload through the current packer ($ORIGIN only — c32's payload already proves the packer does that: 16/16 ELFs exactly $ORIGIN) and re-install, or install a newer package and retire a5271d9.
a5271d9 should be considered poisoned by construction: any rebuild in ~/wt/release-rc/work/build changes what it loads. Deleting it (once nothing points at it) is safer than repairing its RUNPATH.
Filed by pi-release; the shipped-side rule is PR #1403 and the shipped side is already clean.
So the newest install is the offender and the older ones are clean, which is the opposite of what "installed copies drift over time" would predict — worth knowing when someone decides whether to re-install or patchelf them.
The consequence, which is not in the issue yet: that same a5271d9 install is the natural binary to benchmark with — it is the most recent shipped tree, and its llama-bench/llama-server are what a user actually runs. Any number taken through it is only reproducible if the build tree it points into is the one that produced it; after a rebuild the same install measures different libraries under the same file names. I have been taking GPNPU parity rows with a known-commit build (build-hrx from the fork pin) rather than through an installed engine, and I am recording "never quote a number taken through a5271d9 until it carries $ORIGIN" in my board's tooling note so the two problems do not get conflated later.
I am not claiming the fix: re-installing or patching a live install is a service action (two services were crash-looping in the incident), so that stays an owner/orchestrator call rather than a lane edit.
Measured the live half of this from the RUNPATH side (read-only, nothing touched), because the issue covers the state but not whether the hazard is live right now. Three findings, one of them a correction to item 3.
1. Nothing is loading from a rebuildable tree today
Swept every/proc/*/maps on strixhalo for /home/bcloud/wt/…/work/build:
(for p in /proc/[0-9]*; do grep -lE "/home/bcloud/wt/.*/work/build" $p/maps; done) -> no matches
So the a5271d9 hazard is not live: no running process maps a library out of a tree that gets rebuilt. That matches the incident's aftermath - the two services were moved to the c22 pin.
2. The services are on c22 end to end, and that is why
Both units carry a zz-c22-selfcontained.conf that sorts last among the drop-ins, and it sets the front-end and the child:
systemctl --user show -p ExecStart agrees (both = 1bit-embed-c22). Setting ONEBIT_LLAMA_SERVER matters on its own: the incident was about libraries loaded by the childllama-server, so a drop-in that repointed only 1bit would have left the same exposure.
3. Correction to item 3: a5271d9 is still referenced, so "once nothing points at it" is not yet true
Exactly three files name it, and they lose only by sort order (hrx < zz):
both with ExecStart=…/1bit-hrx-a5271d9/1bit serve … and ONEBIT_LLAMA_SERVER=…/1bit-hrx-a5271d9/…/llama-server.
Consequence worth putting in the fix list: the shadowing is by filename, so removing or renaming zz-c22-selfcontained.conf - or adding any drop-in that sorts after it - silently restores a5271d9 on the next restart, on the poisoned install, with no error. So the order is: retarget or delete the three hrx.conf files first, then deleting ~/.local/share/1bit-hrx-a5271d9 is safe. Anything else leaves the hazard dormant rather than removed.
20-rebuild.conf (coder-llm only) is unrelated: it sets no path, only the no-SVM environment (HSA_USE_SVM=0, HRX_ALLOW_NO_SVM=1, …).
A false positive from my own sweep, so nobody chases it
My first pass looked for a5271d9 in /proc/*/cmdline and got three pids - those were my own scanning shell, whose command line contained the string. The maps sweep above is the measurement; the cmdline hits were the tool reading itself. (Same class as a grep | head that truncates: a check's window can manufacture a finding.)
Not done here, deliberately: I did not edit any drop-in or delete the install - that is a box-config change on services this lane does not own, and the issue says the fix should not go in unilaterally. The measurement above is the part that makes the decision cheap for whoever does own them.
Verified on both boxes (8 Nov 06:2xZ), and #1403 shipped the gate — so here is the evidence, plus two leftovers that are box-side rather than engine-side.
Method: each ELF under the install roots is read with the same dynamic-section parser the new payload-runpath row uses (DT_RUNPATH/DT_RPATH), and a value counts as non-conformant when any :-separated entry is not $ORIGIN.
strixhalo — every installed engine
install
ELFs
non-$ORIGIN
what the entries name
~/.local/share/1bit-hrx-a5271d9
28
27
~/wt/release-rc/work/build/hrx/llama/bin:… — a tree that gets rebuilt: this is the incident
~/.local/share/1bit-embed-c22
33
29
$ORIGIN:/work-c22/hrx-toolchain/…/clang/24/lib/… — a toolchain dir
~/.local/share/1bit-canonical
98
10
$ORIGIN/xrt/lib and /opt/rocm-therock/…/clang/24/lib/…
1bit-hrx-004377f / 3092596 / 41ff9df
28 each
6 each
/opt/rocm-therock/…/clang/24/lib/…
~/.local/share/1bit-engine
1
1
$ORIGIN/xrt/lib
ryzen — no installed engine carries a build-tree path
The only ~/.local/share/1bit-* directory is 1bit-monster, which holds no ELF at all; the running engine processes there are the demo app and ssh, neither of which is an engine binary with a RUNPATH.
What this changes about the rule (the part worth keeping)
The literal rule — "$ORIGIN only" — would red every install on both boxes, because a ROCm toolchain directory in RUNPATH is how these engines have always been built, and $ORIGIN/xrt/lib is inside the install itself. The property that actually matters, and that the incident demonstrated, is narrower: no RUNPATH may name a tree that gets rebuilt. ~/wt/.../work/build is that tree; /opt/rocm-therock and /work-c22 are not. Measured consequence of the toolchain ones being harmless: ldd on c22's 1bit, llama-server and lemond resolves nothing through /work-c22.
The shipped-artifact side is already gated and clean: c32's payload is $ORIGIN on all 16 of its RUNPATH-carrying ELFs, verified out of the stick's own sfs, and payload-runpath (merged, #1403) fails any pack built by an older packer — measured against a real c22 package tree, which it reds with 29 offenders.
Two leftovers, both box-side (not engine defects)
~/.config/systemd/user/coder-llm.service.d/hrx.conf still names a5271d9. The service is pinned to c22 by zz-c22-selfcontained.conf (which sorts last and wins), so nothing is loading the poisoned install — but the stale drop-in is a re-poisoning waiting for someone to remove the zz- file. Deleting the stale drop-in is the fix.
A llama-server running right now from ~/wt/i1251/pincur/ with RUNPATH into ~/wt/i1251-build/… (pid 11635) — the same hazard class as the incident, in a work tree rather than an install. That belongs to the lane that owns it; noting it here because it is live, not hypothetical.
Closing this issue: the measurement is done, the gate is shipped, and both remaining items are box configuration belonging to the lanes/owner who own those trees.
Closed with the two-box evidence above: shipped payloads are $ORIGIN-only and gated by payload-runpath (#1403); the only dangerous installed engine is the unused a5271d9 (its stale systemd drop-in is the remaining risk), and ryzen has none. The narrower rule that the evidence supports - no RUNPATH into a tree that gets rebuilt - is written into the comment rather than left implied.
Item 3 done — "supreme drop ins go" (owner word 2026-11-08 ~12:4x AST). Executed exactly to the given scope on strixhalo; nothing else on the box touched, nothing published.
1. One drop-in per service, pinned to the self-contained c22 install
service
before
after
coder-llm
20-rebuild.conf (a5271d9-era engine, ctx 32768) + hrx.conf (a5271d9 front-end andONEBIT_LLAMA_SERVER) + hrx.conf.bak-20261104-ctx + zz-c22-selfcontained.conf — the a5271d9 files lost only on sort order (zz > hrx)
onezz-c22-selfcontained.conf
1bit-canonical
hrx.conf (a5271d9) + zz-c22-selfcontained.conf, same sort-order luck
onezz-c22-selfcontained.conf
Old files moved to *.bak-20261108, not removed (20-rebuild.conf.bak-20261108, hrx.conf.bak-20261108, hrx.conf.bak-20261104-ctx.bak-20261108, hrx.conf.bak-20261108). Each new drop-in carries the same args that were effective before the merge (so behaviour is unchanged by construction), and a rollback line.
No mention of the removed install remains in either active drop-in — including comments, per the instruction; they describe the fault generically ("an older HRX install whose llama-server resolves its libs through RUNPATH into ~/wt/release-rc/work/build… and ships no libhrx/libloomc") and point here.
2. Restart and verification (coder-llm first, then 1bit-canonical)
Equivalence checked before the restart: systemctl --user show -p ExecStart argv byte-identical to the pre-merge effective argv for both units, and Environment= identical.
check
coder-llm (:8090)
1bit-canonical (:8095)
is-active
active
active
NRestarts over 5 min (11 samples, 30 s apart, 12:04:55–12:09:57)
No process maps a worktree path. pr_agent's dependency on :8090 was watched during the restart and the endpoint was back inside 18 s, so no fix-forward was needed.
3. Package removal
lsof +D ~/.local/share/1bit-hrx-a5271d9 → empty, and an independent /proc/*/maps sweep over every 1bit/llama-server process found no mapper of that path (lsof +D cannot see already-unlinked files). Then deleted.
Freed: 39 MB (du -sm → 39 MB). Worth recording that it is small: the package is the engine + its hrx/ sidecar, and it is small precisely because of this bug — it does not shiplibhrx/libloomc and instead resolves them, and libggml-hrx, through a dev-worktree RUNPATH. df on / was unchanged at GB granularity (202 G before and after). The weights live in ~/models, not in the package.
Other old engine packages (1bit-hrx-004377f, -3092596, -41ff9df) were left alone — out of scope.
4. What this closes and what it does not
Closed: item 3 of this issue. No shipped config references the non-relocatable install any more, so "delete once nothing points at it" is now true.
Still worth doing, not done here: the cause is packaging, and it is still live for anything packed the same way — a pack-time check that fails a packed ELF whose RUNPATH is absolute rather than $ORIGIN-relative would catch this class before it ships. The c31 payload passes such a check today (RUNPATH=$ORIGIN, ships libhrx/libloomc); the removed install would not have.
Measured consequence for anyone driving an installed engine directly (as benchmarking harnesses do): without LD_LIBRARY_PATH the package's llama-server binds the worktree's libggml-hrx (166721a7…) instead of its own (bae07098…) and either reports invalid device: HRX0 or silently serves on the CPU. That is what cost pi-supreme five measurement arms before it was understood; the harness now gates on ldd resolving inside the package.
Closing per the instruction, since nothing remains that names the removed install.
Residual measured while verifying (no action taken, recorded because it is this same class one level down).
After the pin, coder-llm's child llama-server maps:
0 paths under /home/bcloud/wt (any dev/rebuildable worktree) — the half this item was about is gone;
64 paths under /work-c22/hrx-toolchain/…, including libhsa-runtime64.so.1, libhsa-amd-aqlprofile64.so.1, librocprofiler-register.so.0 and the rocm_sysdeps set.
So the c22 payload is $ORIGIN-relative for its own libraries but still resolves its ROCm/HSA runtime through an absolute /work-c22 path — which is exactly what this issue's title says ("the c22 pin into /work-c22"). Practical consequence: if that
toolchain tree is ever moved or pruned, both coder-llm (:8090, which pr_agent depends on) and 1bit-canonical (:8095) stop
initialising the GPU — the same failure shape as the build-tree RUNPATH, with a different tree.
Not fixed here: the scope I was given was the drop-in merge and the package deletion, and changing where an installed payload resolves
its ROCm runtime is a packaging decision (RUNPATH/$ORIGIN plus shipping or symlinking the runtime), not a service-config edit.
Flagging it so the next packaging pass can fold both trees into one rule: a packed ELF should resolve only inside its own payload
or through the system loader.
Owner word: fix the toolchain-tree half, then a correction — "we skip that build... we test when all issues are solved". So this is the two halves that can land without touching the running system, and the live verification is deliberately not done.
The payload ships the whole runtime set.scripts/mk-plugin-package.sh staged what ldd resolves — libhsa-runtime64.so.1, librocprofiler-register.so.0 and the 8 rocm_sysdeps entries — but not libhsa-amd-aqlprofile64.so.1, which the runtime opens by name itself. It ships beside the runtime now, and the build refuses if it is missing. NOTICE-BINARY gains the set's licences, checked against upstream rather than assumed: ROCr is University of Illinois/NCSA, aqlprofile and rocprofiler-register are MIT, and the rocm_sysdeps entries keep their upstream licences.
The release check fails when anything resolves outside the payload (scripts/pack-check.sh, runtime_selfcontained, run for both packages): no packed ELF may declare an absolute RUNPATH/RPATH; every library ldd resolves must be inside the payload or a system directory; and the runtime set has to be present beside the backend. Its unit test is in CI — ctest pack_check — with a clean tree as control plus arms for a build-machine path, an absolute RUNPATH, a relative rpath that escapes the payload, and a runtime set without aqlprofile.
The escaping arm found a real hole in my own row on the first run: ldd prints the path it was given unnormalised, so <tree>/bin/../../outside/libx.so passed a prefix test. The row canonicalises resolved paths now.
Red arm, on the install that is live today (~/.local/share/1bit-embed-c22/lemonade-onebit-rc-2026-11-03-c22): 19 FAIL lines, rc 1 — every packed library's /work-c22 RUNPATH (libhrx.so.0, libllama.so.0.0.1, llama-server, libggml-hrx.so.0.18.0, libloomc.so.0, …) and the runtime set missing entirely.
The mechanism, for the record: the runtime is reached by dlopen("libhsa-runtime64.so.1") (ggml-hrx-dmabuf.cpp:29), so ldd cannot see it and only $ORIGIN + a shipped set closes it. The c22 install maps 64 /work-c22 entries; a payload built by the current packer carries $ORIGIN only (c32/c33 both do) — so what c22 needs is the set, not a new RUNPATH rule.
Deferred, not forgotten — this issue stays open for it: no package build, no new install directory, no drop-in change and no restart of coder-llm (:8090) or 1bit-canonical (:8095) (they stay on c22 exactly as they are), and no /work-c22-hidden independence run. The end-to-end proof — a fixed payload serving on HRX0 with /work-c22 hidden, then the two services switched to it, NRestarts stable, /v1/models on both, /proc/<pid>/maps free of /work-c22 — happens at the release gate once all issues are solved. /work-c22 has not been touched.
Reopened per owner order: the live verification for this issue is deferred to the release-gate test, not done. #1506 fixes the two halves that can land without touching the running system (the payload ships the HSA runtime set it opens; the release refuses a payload that resolves outside itself, with a CI unit test), and the end-to-end proof — a fixed payload serving on HRX0 with /work-c22 hidden, then coder-llm and 1bit-canonical switched to it — happens at the gate once all issues are solved. /work-c22 is untouched and both services are still on c22.
only three superseded backups: ~/.config/systemd/user/{coder-llm,1bit-canonical}.service.d/hrx.conf.bak-20261108 (+ one .bak-20261104-ctx.bak-20261108). No live unit, no running process (pgrep -af a5271d9 → nothing)
1bit-embed-c22 RUNPATH
24 ELFs: 16 with $ORIGIN, and 15 carrying /work-c22/hrx-toolchain/…/llvm/lib{,/clang/24/lib} (the same entries the issue recorded)
do those entries resolve anything?
no — ldd on c22's 1bit and on llama-server reports 0 dependencies resolved via /work-c22, and /work-c22 still exists on the box
So, against the issue's own three options: option 3 has effectively happened (the poisoned-by-construction install is not on the box any more; nothing points at it), and what remains is option 2 for c22 — re-pack through the current packer so all ELFs are $ORIGIN-only, which the issue already notes c32's payload proves the packer can do (16/16).
Two things worth flagging rather than acting on, since this issue says the fix is not a unilateral call:
the leftover .bak-20261108 drop-ins still name the removed tree; they are inert (systemd ignores non-.conf suffixes) but they are the kind of stale reference that makes a future grep think a5271d9 is still in play — deleting them is a one-line cleanup for whoever owns those units;
the c22 /work-c22 entries are dead weight today exactly as the issue measured, so there is no urgency, and re-packing changes a production payload (the pin behind two services) — that stays with the release lane.
If that reading is right, this issue can be re-scoped to "c22 payload: re-pack to $ORIGIN-only at the next pin bump" and the incident-hazard half marked done.
Swept every engine installed on this box with the row #1506 merged (scripts/pack-check.sh: runtime_selfcontained), read-only — no process touched, no drop-in edited, nothing restarted. It partitions the installs cleanly, which is the most useful thing the check has said so far:
install
row
abs RUNPATH
resolves outside
runtime set missing
1bit-engine
ok (rc 0)
0
0
0
1bit-monster
ok (rc 0)
0
0
0
1bit-hrx-004377f
FAIL
2
11
4
1bit-hrx-3092596
FAIL
2
11
4
1bit-hrx-41ff9df
FAIL
2
11
4
1bit-embed-c22
FAIL
13
6
4
1bit-canonical
FAIL
5
66
4
1bit-pr-agent
FAIL
5
69
4
Two things in that are not in the issue yet:
The dominant leak is libomp.so, not the HSA set: the older payloads do not ship it, so ldd resolves it to /opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/llvm/lib/libomp.so — an absolute path outside the payload on this box, and nothing at all on a machine without TheRock. Current payloads ship libomp.so beside the backend, which is why 1bit-engine and 1bit-monster pass with zero findings.
1bit-pr-agent (78 findings) is the tree coder-llm's 1bit comes from, and 1bit-canonical (75) is its sibling. Their children run llama-server from the c22 tree (19 findings) through the last-sorting drop-in, so the running services are exposed either way — the drop-in protects the child, not the parent.
Also: 1bit-hrx-a5271d9, the install pi-supreme measured, is gone (retired), which is the first half of the issue's item 3 becoming true; the three 1bit-hrx-* copies that remain are the same shape (2 absolute RUNPATH entries, 11 resolutions outside, no runtime set).
What the merged work changes: a payload of that shape now fails the release instead of shipping (pack_check in CI covers the four shapes synthetically, and the red arm on c22 is 19 findings), and the packer ships the runtime set plus libomp beside the backend so a current payload resolves everything inside itself.
Still deferred by owner order, and this issue stays open for it: no install directory, no drop-in edit, no restart, no /work-c22-hidden run. coder-llm (:8090) and 1bit-canonical (:8095) remain exactly as they are on c22, /work-c22 untouched.
Re-measured with the merged classifier (check-installed-rpath.py, #1616): the closure's conclusion holds, and LIVE puts a number on the hazard
Not contradicting the close — confirming it, with the tool that landed after it. Read-only, no process touched.
$ python3 tools/check-installed-rpath.py # on main c8c518cb
FAIL 94 offending RUNPATH entries across 317 ELF(s), 42 of them LIVE: a library is found through the entry,
so the binary cannot start if that directory is rebuilt, moved or removed … rc=1
94 offending entries across 317 ELFs, 42 LIVE. The LIVE half is the part the earlier sweep could not express: it counted "resolves outside" per install (66 for 1bit-canonical, 69 for 1bit-pr-agent), but LIVE says a dependency is actually found through that entry right now — which is the precise condition under which a TheRock rebuild or move breaks a running service.
The entries are the ones the issue already names: $ORIGIN plus /opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_{core,devel}/lib/llvm/lib, and the resolved library is libomp.so — consistent with the "the dominant leak is libomp.so, older payloads do not ship it" finding, and with current payloads shipping it beside themselves.
One consequence worth stating plainly, because it is live rather than archived: two of the installs that FAIL this row are the ones running right now — 1bit-pr-agent (the LLM behind PR reviews, :8090) and 1bit-embed-c22 (lemond on :8088). Their libomp.so comes from TheRock's absolute path, so a TheRock rebuild/reinstall on this box is a stop-and-restart for both, and on a machine without TheRock they would not start at all. That is the same class the close described; the only new information is that it is attached to live services rather than to superseded installs.
For contrast, the sibling tool from the same batch is clean: python3 tools/device_holders.py --check → rc 0, "4 holder(s), every one inside a ticket (token pid 2392598, 2496994)" — so tonight's orphan rule is currently satisfied.
No action requested: reinstalling those two from a current payload is the fix, and that is a fleet decision rather than a code change. Lane pi-rlk; no device time (0 s GPU, 0 s NPU).
The repack-or-retire decision (mine per the board), made from a measurement: the 42 LIVE entries are two different hazards, and one of them is not a hazard
Re-measured on this box (~/wt/trel3 at main, tools/check-installed-rpath.py, read-only):
FAIL 94 offending RUNPATH entries across 317 ELF(s), 42 of them LIVE (exit 1, no pipe)
The number that made this undecidable is the conflation. Bucketing the LIVE rows by what the entry actually resolves to (the tool prints a 36-row sample, so this attributes 31 of the 42):
The /opt/rocm-* entries: ACCEPT, and stop calling them a defect. They resolve libomp.so from the installed ROCm SDK — the same coupling every ROCm consumer on this box has. Making the package $ORIGIN-only would mean vendoring a second copy of libomp and shipping an artifact that is not the one that was tested; the RUNPATH entry is a dependency declaration, and the LIVE finding is telling us it is armed, which is true and expected. What the checker should say for these is DECLARED SDK (a WARN), not FAIL.
The /work-c22 entries: RETIRE the payload, do not repack it. Repacking c22 through the current packer produces a different artifact from the one validated as c22 ($ORIGIN-only changes which libomp runs), and c22 is superseded — c33 is the candidate. So: the payload is retired rather than rewritten.
Why I am not deleting anything yet, in this comment or on the box:/work-c22 is 17 G and it is a re-runnable proof — ~/lb/k74-ab/proof-0008b-20261104-225338/pkg-B embeds those paths in its own binaries, and ~/.local/share/1bit-embed-c22 (58 offending entries, 6 LIVE) is the payload those proofs compare against. Deleting the tree now would take the proof with it. The order is therefore: retire dependants first, then free the tree — and until that happens the tree is a 17 G disk lever, which matters because the box is at 158 G free against a 160 G floor for new build work.
One line to make the number actionable (for a lane with a PR slot): classify a non-$ORIGIN entry by its resolved target — BUILD TREE for /work-*, ~/wt, /tmp; DECLARED SDK for /opt/rocm*, and count them separately, so 42 LIVE becomes 1 fatal + 41 declared instead of a number nobody can act on.
Activity
bong-water-water-bong commentedon Nov 8, 2026
Confirmed independently, and there is one consequence worth adding for anyone measuring through these installs.
Swept every installed engine on strixhalo myself (
readelf -donlibllama.so.0per install):1bit-canonical,1bit-hrx-004377f,1bit-hrx-3092596,1bit-hrx-41ff9df,1bit-pr-agent$ORIGINonly1bit-hrx-a5271d9/home/bcloud/wt/release-rc/work/build/hrx/llama/bin:…/ggml-hrx/…/libhrx:…/hrx-toolchain/…So the newest install is the offender and the older ones are clean, which is the opposite of what "installed copies drift over time" would predict — worth knowing when someone decides whether to re-install or
patchelfthem.The consequence, which is not in the issue yet: that same
a5271d9install is the natural binary to benchmark with — it is the most recent shipped tree, and itsllama-bench/llama-serverare what a user actually runs. Any number taken through it is only reproducible if the build tree it points into is the one that produced it; after a rebuild the same install measures different libraries under the same file names. I have been taking GPNPU parity rows with a known-commit build (build-hrxfrom the fork pin) rather than through an installed engine, and I am recording "never quote a number taken througha5271d9until it carries$ORIGIN" in my board's tooling note so the two problems do not get conflated later.I am not claiming the fix: re-installing or patching a live install is a service action (two services were crash-looping in the incident), so that stays an owner/orchestrator call rather than a lane edit.
bong-water-water-bong commentedon Nov 8, 2026
Measured the live half of this from the RUNPATH side (read-only, nothing touched), because the issue covers the state but not whether the hazard is live right now. Three findings, one of them a correction to item 3.
1. Nothing is loading from a rebuildable tree today
Swept every
/proc/*/mapson strixhalo for/home/bcloud/wt/…/work/build:So the a5271d9 hazard is not live: no running process maps a library out of a tree that gets rebuilt. That matches the incident's aftermath - the two services were moved to the c22 pin.
2. The services are on c22 end to end, and that is why
Both units carry a
zz-c22-selfcontained.confthat sorts last among the drop-ins, and it sets the front-end and the child:systemctl --user show -p ExecStartagrees (both =1bit-embed-c22). SettingONEBIT_LLAMA_SERVERmatters on its own: the incident was about libraries loaded by the childllama-server, so a drop-in that repointed only1bitwould have left the same exposure.3. Correction to item 3: a5271d9 is still referenced, so "once nothing points at it" is not yet true
Exactly three files name it, and they lose only by sort order (
hrx<zz):both with
ExecStart=…/1bit-hrx-a5271d9/1bit serve …andONEBIT_LLAMA_SERVER=…/1bit-hrx-a5271d9/…/llama-server.Consequence worth putting in the fix list: the shadowing is by filename, so removing or renaming
zz-c22-selfcontained.conf- or adding any drop-in that sorts after it - silently restores a5271d9 on the next restart, on the poisoned install, with no error. So the order is: retarget or delete the threehrx.conffiles first, then deleting~/.local/share/1bit-hrx-a5271d9is safe. Anything else leaves the hazard dormant rather than removed.20-rebuild.conf(coder-llm only) is unrelated: it sets no path, only the no-SVM environment (HSA_USE_SVM=0,HRX_ALLOW_NO_SVM=1, …).A false positive from my own sweep, so nobody chases it
My first pass looked for
a5271d9in/proc/*/cmdlineand got three pids - those were my own scanning shell, whose command line contained the string. The maps sweep above is the measurement; the cmdline hits were the tool reading itself. (Same class as agrep | headthat truncates: a check's window can manufacture a finding.)Not done here, deliberately: I did not edit any drop-in or delete the install - that is a box-config change on services this lane does not own, and the issue says the fix should not go in unilaterally. The measurement above is the part that makes the decision cheap for whoever does own them.
bong-water-water-bong commentedon Nov 8, 2026
Verified on both boxes (8 Nov 06:2xZ), and #1403 shipped the gate — so here is the evidence, plus two leftovers that are box-side rather than engine-side.
Method: each ELF under the install roots is read with the same dynamic-section parser the new
payload-runpathrow uses (DT_RUNPATH/DT_RPATH), and a value counts as non-conformant when any:-separated entry is not$ORIGIN.strixhalo — every installed engine
$ORIGIN~/.local/share/1bit-hrx-a5271d9~/wt/release-rc/work/build/hrx/llama/bin:…— a tree that gets rebuilt: this is the incident~/.local/share/1bit-embed-c22$ORIGIN:/work-c22/hrx-toolchain/…/clang/24/lib/…— a toolchain dir~/.local/share/1bit-canonical$ORIGIN/xrt/liband/opt/rocm-therock/…/clang/24/lib/…1bit-hrx-004377f/3092596/41ff9df/opt/rocm-therock/…/clang/24/lib/…~/.local/share/1bit-engine$ORIGIN/xrt/libryzen — no installed engine carries a build-tree path
The only
~/.local/share/1bit-*directory is1bit-monster, which holds no ELF at all; the running engine processes there are the demo app and ssh, neither of which is an engine binary with a RUNPATH.What this changes about the rule (the part worth keeping)
The literal rule — "
$ORIGINonly" — would red every install on both boxes, because a ROCm toolchain directory in RUNPATH is how these engines have always been built, and$ORIGIN/xrt/libis inside the install itself. The property that actually matters, and that the incident demonstrated, is narrower: no RUNPATH may name a tree that gets rebuilt.~/wt/.../work/buildis that tree;/opt/rocm-therockand/work-c22are not. Measured consequence of the toolchain ones being harmless:lddon c22's1bit,llama-serverandlemondresolves nothing through/work-c22.The shipped-artifact side is already gated and clean: c32's payload is
$ORIGINon all 16 of its RUNPATH-carrying ELFs, verified out of the stick's own sfs, andpayload-runpath(merged, #1403) fails any pack built by an older packer — measured against a real c22 package tree, which it reds with 29 offenders.Two leftovers, both box-side (not engine defects)
~/.config/systemd/user/coder-llm.service.d/hrx.confstill namesa5271d9. The service is pinned to c22 byzz-c22-selfcontained.conf(which sorts last and wins), so nothing is loading the poisoned install — but the stale drop-in is a re-poisoning waiting for someone to remove thezz-file. Deleting the stale drop-in is the fix.llama-serverrunning right now from~/wt/i1251/pincur/with RUNPATH into~/wt/i1251-build/…(pid 11635) — the same hazard class as the incident, in a work tree rather than an install. That belongs to the lane that owns it; noting it here because it is live, not hypothetical.Closing this issue: the measurement is done, the gate is shipped, and both remaining items are box configuration belonging to the lanes/owner who own those trees.
bong-water-water-bong commentedon Nov 8, 2026
Closed with the two-box evidence above: shipped payloads are $ORIGIN-only and gated by
payload-runpath(#1403); the only dangerous installed engine is the unuseda5271d9(its stale systemd drop-in is the remaining risk), and ryzen has none. The narrower rule that the evidence supports - no RUNPATH into a tree that gets rebuilt - is written into the comment rather than left implied.bong-water-water-bong commentedon Nov 8, 2026
Item 3 done — "supreme drop ins go" (owner word 2026-11-08 ~12:4x AST). Executed exactly to the given scope on strixhalo; nothing else on the box touched, nothing published.
1. One drop-in per service, pinned to the self-contained c22 install
coder-llm20-rebuild.conf(a5271d9-era engine, ctx 32768) +hrx.conf(a5271d9 front-end andONEBIT_LLAMA_SERVER) +hrx.conf.bak-20261104-ctx+zz-c22-selfcontained.conf— the a5271d9 files lost only on sort order (zz>hrx)zz-c22-selfcontained.conf1bit-canonicalhrx.conf(a5271d9) +zz-c22-selfcontained.conf, same sort-order luckzz-c22-selfcontained.confOld files moved to
*.bak-20261108, not removed (20-rebuild.conf.bak-20261108,hrx.conf.bak-20261108,hrx.conf.bak-20261104-ctx.bak-20261108,hrx.conf.bak-20261108). Each new drop-in carries the same args that were effective before the merge (so behaviour is unchanged by construction), and a rollback line.No mention of the removed install remains in either active drop-in — including comments, per the instruction; they describe the fault generically ("an older HRX install whose llama-server resolves its libs through RUNPATH into
~/wt/release-rc/work/build… and ships no libhrx/libloomc") and point here.2. Restart and verification (coder-llm first, then 1bit-canonical)
Equivalence checked before the restart:
systemctl --user show -p ExecStartargv byte-identical to the pre-merge effective argv for both units, andEnvironment=identical.is-activeNRestartsover 5 min (11 samples, 30 s apart, 12:04:55–12:09:57)GET /v1/modelslibggml-hrxmapped by the child llama-server…/1bit-embed-c22/lemonade-onebit-rc-2026-11-03-c22/1bit/hrx/llama/bin/libggml-hrx.so.0.18.0No process maps a worktree path.
pr_agent's dependency on:8090was watched during the restart and the endpoint was back inside 18 s, so no fix-forward was needed.3. Package removal
lsof +D ~/.local/share/1bit-hrx-a5271d9→ empty, and an independent/proc/*/mapssweep over every1bit/llama-serverprocess found no mapper of that path (lsof+Dcannot see already-unlinked files). Then deleted.Freed: 39 MB (
du -sm→ 39 MB). Worth recording that it is small: the package is the engine + itshrx/sidecar, and it is small precisely because of this bug — it does not shiplibhrx/libloomcand instead resolves them, andlibggml-hrx, through a dev-worktree RUNPATH.dfon/was unchanged at GB granularity (202 G before and after). The weights live in~/models, not in the package.Other old engine packages (
1bit-hrx-004377f,-3092596,-41ff9df) were left alone — out of scope.4. What this closes and what it does not
RUNPATHis absolute rather than$ORIGIN-relative would catch this class before it ships. The c31 payload passes such a check today (RUNPATH=$ORIGIN, shipslibhrx/libloomc); the removed install would not have.LD_LIBRARY_PATHthe package'sllama-serverbinds the worktree'slibggml-hrx(166721a7…) instead of its own (bae07098…) and either reportsinvalid device: HRX0or silently serves on the CPU. That is what cost pi-supreme five measurement arms before it was understood; the harness now gates onlddresolving inside the package.Closing per the instruction, since nothing remains that names the removed install.
bong-water-water-bong commentedon Nov 8, 2026
Residual measured while verifying (no action taken, recorded because it is this same class one level down).
After the pin,
coder-llm's child llama-server maps:/home/bcloud/wt(any dev/rebuildable worktree) — the half this item was about is gone;/work-c22/hrx-toolchain/…, includinglibhsa-runtime64.so.1,libhsa-amd-aqlprofile64.so.1,librocprofiler-register.so.0and therocm_sysdepsset.So the c22 payload is
$ORIGIN-relative for its own libraries but still resolves its ROCm/HSA runtime through an absolute/work-c22path — which is exactly what this issue's title says ("the c22 pin into /work-c22"). Practical consequence: if thattoolchain tree is ever moved or pruned, both
coder-llm(:8090, whichpr_agentdepends on) and1bit-canonical(:8095) stopinitialising the GPU — the same failure shape as the build-tree RUNPATH, with a different tree.
Not fixed here: the scope I was given was the drop-in merge and the package deletion, and changing where an installed payload resolves
its ROCm runtime is a packaging decision (
RUNPATH/$ORIGINplus shipping or symlinking the runtime), not a service-config edit.Flagging it so the next packaging pass can fold both trees into one rule: a packed ELF should resolve only inside its own payload
or through the system loader.
bong-water-water-bong commentedon Nov 8, 2026
Owner word: fix the toolchain-tree half, then a correction — "we skip that build... we test when all issues are solved". So this is the two halves that can land without touching the running system, and the live verification is deliberately not done.
What is fixed (PR #1506,
a8317859):The payload ships the whole runtime set.
scripts/mk-plugin-package.shstaged whatlddresolves —libhsa-runtime64.so.1,librocprofiler-register.so.0and the 8rocm_sysdepsentries — but notlibhsa-amd-aqlprofile64.so.1, which the runtime opens by name itself. It ships beside the runtime now, and the build refuses if it is missing.NOTICE-BINARYgains the set's licences, checked against upstream rather than assumed: ROCr is University of Illinois/NCSA, aqlprofile and rocprofiler-register are MIT, and therocm_sysdepsentries keep their upstream licences.The release check fails when anything resolves outside the payload (
scripts/pack-check.sh,runtime_selfcontained, run for both packages): no packed ELF may declare an absoluteRUNPATH/RPATH; every librarylddresolves must be inside the payload or a system directory; and the runtime set has to be present beside the backend. Its unit test is in CI — ctestpack_check— with a clean tree as control plus arms for a build-machine path, an absolute RUNPATH, a relative rpath that escapes the payload, and a runtime set without aqlprofile.The escaping arm found a real hole in my own row on the first run:
lddprints the path it was given unnormalised, so<tree>/bin/../../outside/libx.sopassed a prefix test. The row canonicalises resolved paths now.Red arm, on the install that is live today (
~/.local/share/1bit-embed-c22/lemonade-onebit-rc-2026-11-03-c22): 19 FAIL lines, rc 1 — every packed library's/work-c22RUNPATH (libhrx.so.0,libllama.so.0.0.1,llama-server,libggml-hrx.so.0.18.0,libloomc.so.0, …) and the runtime set missing entirely.The mechanism, for the record: the runtime is reached by
dlopen("libhsa-runtime64.so.1")(ggml-hrx-dmabuf.cpp:29), solddcannot see it and only$ORIGIN+ a shipped set closes it. The c22 install maps 64/work-c22entries; a payload built by the current packer carries$ORIGINonly (c32/c33 both do) — so what c22 needs is the set, not a new RUNPATH rule.Deferred, not forgotten — this issue stays open for it: no package build, no new install directory, no drop-in change and no restart of
coder-llm(:8090) or1bit-canonical(:8095) (they stay on c22 exactly as they are), and no/work-c22-hidden independence run. The end-to-end proof — a fixed payload serving on HRX0 with/work-c22hidden, then the two services switched to it,NRestartsstable,/v1/modelson both,/proc/<pid>/mapsfree of/work-c22— happens at the release gate once all issues are solved./work-c22has not been touched.bong-water-water-bong commentedon Nov 8, 2026
Reopened per owner order: the live verification for this issue is deferred to the release-gate test, not done. #1506 fixes the two halves that can land without touching the running system (the payload ships the HSA runtime set it opens; the release refuses a payload that resolves outside itself, with a CI unit test), and the end-to-end proof — a fixed payload serving on HRX0 with /work-c22 hidden, then coder-llm and 1bit-canonical switched to it — happens at the gate once all issues are solved. /work-c22 is untouched and both services are still on c22.
pack_check_test: probe for the compiler the runner actually has (Refs #…
package-linux: the engine tarball ships the HSA runtime set, so BOTH …
package: the payload ships the HSA runtime set it opens, and the rele…
bong-water-water-bong commentedon Nov 8, 2026
Re-measured on strixhalo today (2026-11-08 ~16:2x AST): the hazard half of this issue is already closed —
1bit-hrx-a5271d9is no longer installed.~/.local/share/1bit-hrx-a5271d9~/wt/release-rc/work/buildis gone1bit-embed-c22,1bit-engine,1bit-canonical,1bit-monster,1bit-pr-agent,1bit-hrx-004377f/3092596/41ff9df~/.config/systemd/user/{coder-llm,1bit-canonical}.service.d/hrx.conf.bak-20261108(+ one.bak-20261104-ctx.bak-20261108). No live unit, no running process (pgrep -af a5271d9→ nothing)1bit-embed-c22RUNPATH$ORIGIN, and 15 carrying/work-c22/hrx-toolchain/…/llvm/lib{,/clang/24/lib}(the same entries the issue recorded)lddon c22's1bitand onllama-serverreports 0 dependencies resolved via/work-c22, and/work-c22still exists on the boxSo, against the issue's own three options: option 3 has effectively happened (the poisoned-by-construction install is not on the box any more; nothing points at it), and what remains is option 2 for c22 — re-pack through the current packer so all ELFs are
$ORIGIN-only, which the issue already notes c32's payload proves the packer can do (16/16).Two things worth flagging rather than acting on, since this issue says the fix is not a unilateral call:
.bak-20261108drop-ins still name the removed tree; they are inert (systemd ignores non-.confsuffixes) but they are the kind of stale reference that makes a future grep think a5271d9 is still in play — deleting them is a one-line cleanup for whoever owns those units;/work-c22entries are dead weight today exactly as the issue measured, so there is no urgency, and re-packing changes a production payload (the pin behind two services) — that stays with the release lane.If that reading is right, this issue can be re-scoped to "c22 payload: re-pack to
$ORIGIN-only at the next pin bump" and the incident-hazard half marked done.bong-water-water-bong commentedon Nov 8, 2026
Swept every engine installed on this box with the row #1506 merged (
scripts/pack-check.sh: runtime_selfcontained), read-only — no process touched, no drop-in edited, nothing restarted. It partitions the installs cleanly, which is the most useful thing the check has said so far:RUNPATH1bit-engine1bit-monster1bit-hrx-004377f1bit-hrx-30925961bit-hrx-41ff9df1bit-embed-c221bit-canonical1bit-pr-agentTwo things in that are not in the issue yet:
libomp.so, not the HSA set: the older payloads do not ship it, solddresolves it to/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/llvm/lib/libomp.so— an absolute path outside the payload on this box, and nothing at all on a machine without TheRock. Current payloads shiplibomp.sobeside the backend, which is why1bit-engineand1bit-monsterpass with zero findings.1bit-pr-agent(78 findings) is the treecoder-llm's1bitcomes from, and1bit-canonical(75) is its sibling. Their children runllama-serverfrom the c22 tree (19 findings) through the last-sorting drop-in, so the running services are exposed either way — the drop-in protects the child, not the parent.Also:
1bit-hrx-a5271d9, the install pi-supreme measured, is gone (retired), which is the first half of the issue's item 3 becoming true; the three1bit-hrx-*copies that remain are the same shape (2 absolute RUNPATH entries, 11 resolutions outside, no runtime set).What the merged work changes: a payload of that shape now fails the release instead of shipping (
pack_checkin CI covers the four shapes synthetically, and the red arm on c22 is 19 findings), and the packer ships the runtime set pluslibompbeside the backend so a current payload resolves everything inside itself.Still deferred by owner order, and this issue stays open for it: no install directory, no drop-in edit, no restart, no
/work-c22-hidden run.coder-llm(:8090) and1bit-canonical(:8095) remain exactly as they are on c22,/work-c22untouched.os/mini: the image installs the engine instead of shipping the builde…
bong-water-water-bong commentedon Nov 9, 2026
Re-measured with the merged classifier (
check-installed-rpath.py, #1616): the closure's conclusion holds, andLIVEputs a number on the hazardNot contradicting the close — confirming it, with the tool that landed after it. Read-only, no process touched.
LIVEhalf is the part the earlier sweep could not express: it counted "resolves outside" per install (66 for1bit-canonical, 69 for1bit-pr-agent), butLIVEsays a dependency is actually found through that entry right now — which is the precise condition under which a TheRock rebuild or move breaks a running service.$ORIGINplus/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_{core,devel}/lib/llvm/lib, and the resolved library islibomp.so— consistent with the "the dominant leak islibomp.so, older payloads do not ship it" finding, and with current payloads shipping it beside themselves.One consequence worth stating plainly, because it is live rather than archived: two of the installs that FAIL this row are the ones running right now —
1bit-pr-agent(the LLM behind PR reviews,:8090) and1bit-embed-c22(lemondon:8088). Theirlibomp.socomes from TheRock's absolute path, so a TheRock rebuild/reinstall on this box is a stop-and-restart for both, and on a machine without TheRock they would not start at all. That is the same class the close described; the only new information is that it is attached to live services rather than to superseded installs.For contrast, the sibling tool from the same batch is clean:
python3 tools/device_holders.py --check→ rc 0, "4 holder(s), every one inside a ticket (token pid 2392598, 2496994)" — so tonight's orphan rule is currently satisfied.No action requested: reinstalling those two from a current payload is the fix, and that is a fleet decision rather than a code change. Lane
pi-rlk; no device time (0 s GPU, 0 s NPU).bong-water-water-bong commentedon Nov 16, 2026
The repack-or-retire decision (mine per the board), made from a measurement: the 42 LIVE entries are two different hazards, and one of them is not a hazard
Re-measured on this box (
~/wt/trel3at main,tools/check-installed-rpath.py, read-only):The number that made this undecidable is the conflation. Bucketing the LIVE rows by what the entry actually resolves to (the tool prints a 36-row sample, so this attributes 31 of the 42):
Decision, in two parts.
The
/opt/rocm-*entries: ACCEPT, and stop calling them a defect. They resolvelibomp.sofrom the installed ROCm SDK — the same coupling every ROCm consumer on this box has. Making the package$ORIGIN-only would mean vendoring a second copy of libomp and shipping an artifact that is not the one that was tested; the RUNPATH entry is a dependency declaration, and the LIVE finding is telling us it is armed, which is true and expected. What the checker should say for these isDECLARED SDK(a WARN), notFAIL.The
/work-c22entries: RETIRE the payload, do not repack it. Repacking c22 through the current packer produces a different artifact from the one validated as c22 ($ORIGIN-only changes which libomp runs), and c22 is superseded — c33 is the candidate. So: the payload is retired rather than rewritten.Why I am not deleting anything yet, in this comment or on the box:
/work-c22is 17 G and it is a re-runnable proof —~/lb/k74-ab/proof-0008b-20261104-225338/pkg-Bembeds those paths in its own binaries, and~/.local/share/1bit-embed-c22(58 offending entries, 6 LIVE) is the payload those proofs compare against. Deleting the tree now would take the proof with it. The order is therefore: retire dependants first, then free the tree — and until that happens the tree is a 17 G disk lever, which matters because the box is at 158 G free against a 160 G floor for new build work.One line to make the number actionable (for a lane with a PR slot): classify a non-
$ORIGINentry by its resolved target —BUILD TREEfor/work-*,~/wt,/tmp;DECLARED SDKfor/opt/rocm*, and count them separately, so42 LIVEbecomes1 fatal + 41 declaredinstead of a number nobody can act on.