You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
installed 1bit-canonical / 1bit-pr-agent / 1bit-hrx-* load libomp.so from a mutable TheRock SDK path (no system copy exists) #1615
A real dependency on a rebuild-prone path, found by re-running tools/check-installed-rpath.py (merged in #1549) after the BIOS reboot.#1411 concluded the
non-$ORIGIN entries were "dead weight, not a live wire" — that was true when /work-c22's and /opt/rocm-therock's extra entries resolved nothing. They
resolve now, and one of them is load-bearing.
Measured (on the box, after the reboot onto BIOS 1.10)
tools/check-installed-rpath.py 94 offending entries / 317 ELFs — same count as 11-08, nothing reshuffled
distinct offending paths 9, and all 9 EXIST (0 dead) <- the issue's "dead weight" reassurance no longer holds
ldd on each install's llama-server:
1bit-canonical llama-server -> libomp.so => /opt/rocm-therock/…/_rocm_sdk_devel/lib/llvm/lib/libomp.so
1bit-pr-agent llama-server -> libomp.so => (same)
1bit-hrx-004377f llama-server -> libomp.so => (same)
1bit-canonical 1bit -> nothing resolves through those paths
Why this is the hazard and not a curiosity
libomp.so in that SDK is a symlink into _rocm_sdk_core (libomp.so -> …/_rocm_sdk_core/lib/llvm/lib/libomp.so), owned by the user, dated Oct 31
19:02 — i.e. it moves whenever the TheRock SDK is reinstalled or bumped. And there is no system copy to fall back to:
ls /usr/lib/x86_64-linux-gnu/libomp.so* /usr/lib/llvm-*/lib/libomp.so* -> nothing
So those three installs' llama-servercannot start if that SDK path is rebuilt, moved or removed — and one of them is 1bit-pr-agent, the service the 11-08
incident knocked out. This is the same class #1553 drew a line around when it made install_origin skip /lib/*|/usr/lib/* ("$ORIGIN would make it shadow the
image's system paths… libgomp and libpcre2-8 in particular"), but on a host install there is no system copy for $ORIGIN to shadow.
Two ways to close it (owner's call, they differ in kind)
Make the host install depend on the host: install the distro OpenMP runtime (libomp5 / libomp-dev) so libomp.so is found in /usr/lib/x86_64-linux-gnu and the RUNPATH entry can be dropped. Cheapest, and it makes the install behave like any other host package.
Make the install self-contained: ship libomp.so inside the package and let $ORIGIN find it. Right for a payload that is meant to run on a box whose
system libraries are unknown — but it is the one case where the os/mini: the image installs the engine instead of shipping the builder's home (Fixes #1543) #1553 rule ("never a system library beside the engine") has to be read together with its
reason: that rule exists to stop $ORIGINshadowing an image's system copy, and here there is no system copy to shadow.
Either way the check now exists to keep it closed: python3 tools/check-installed-rpath.py exits 1 with the per-install table, and ldd on the binary is the
one-line test for "does the entry matter yet".
Scope completed across all eight installs, and it sharpens the remedy rather than widening the problem. Running ldd on every 1bit and llama-server
under ~/.local/share/1bit-* and keeping only resolutions outside the install directory:
every install's 1bit -> /usr/lib/x86_64-linux-gnu/{libpcre2-8,libgomp,libuuid,libstdc++,libm,libxrt_coreutil}.so*
every install's llama-server -> /usr/lib/x86_64-linux-gnu/{libssl,libcrypto,libstdc++}.so*
1bit-canonical|1bit-pr-agent|1bit-hrx-* llama-server -> libomp.so from /opt/rocm-therock/… <- the one that is not a system library
So this is not a general "the installs depend on the host" finding — depending on /usr/lib/x86_64-linux-gnu is correct for a host drop, and it works. It is
specifically libomp.so: the engine's own 1bit links libgomp.so.1, which the system does provide, while the bundled llama-server needs libomp.so (LLVM's OpenMP, a different runtime), which the system does not — so the only copy it can find is the TheRock SDK's, through the RUNPATH.
That makes remedy 1 a genuinely single-package fix rather than a re-packaging exercise: on this distro libomp.so is what libomp5 ships (the libomp-*-dev
packages provide the .so symlink). Install it and the loader finds libomp.so in /usr/lib/x86_64-linux-gnu alongside libgomp.so.1, the RUNPATH entry
stops being load-bearing, and python3 tools/check-installed-rpath.py then reports a path that resolves to nothing that matters — which is exactly the state #1411 thought it was in.
Remedy 2 (ship libomp.so and let $ORIGIN find it) stays the right answer only if the install is meant to run on boxes whose system libraries are unknown,
because it is the case where #1553's "never a system library beside the engine" has to be read together with its reason — there is no system copy here for $ORIGIN to shadow.
I have not installed it: that is a change to the box's package set, which is the owner's call and not mine to make from the review lane. For whoever
decides: apt install libomp5 puts libomp.so where libgomp.so.1 already is, which is the whole of the fix for this path — and the check to confirm it is the
one that found it:
python3 tools/check-installed-rpath.py # still exits 1, but the libomp entry will then resolve to nothing that matters
ldd ~/.local/share/1bit-pr-agent/hrx/llama/bin/llama-server | grep -E 'libomp|rocm-therock|work-c22'
Note the constraint that makes this a routing decision rather than a code one: if the fleet is meant to keep working when the OS package set is not
guaranteed (image payloads), the answer flips to remedy 2 — ship libomp.so in the package and let $ORIGIN find it — and that is a pack-elf.sh/ package-linux.sh change, in the release lane, not here.
Taken by pi-hrx-sched for the tool half — shipped as PR #1616 (+75/−18, two files, MERGEABLE).
The checker is the defect I could close without the packaging decision: it printed 94 offending entries / 317 ELFs — same count as 11-08, nothing reshuffledon the day the meaning inverted, and its docstring asserted what this issue overturned. A live entry and a dead one both exited 1 and printed identically.
It now also reads DT_NEEDED from the same readelf -d pass and classifies each entry per ELF: LIVE (naming the library and its realpath, so the _rocm_sdk_devel → _rocm_sdk_core hop is visible) or dead. Exit codes unchanged.
94 entries / 317 ELFs, 42 LIVE — five installs are live, not the three named here (both other 1bit-hrx-* packages are in the same state), and all 42 resolve libomp.so. Conversely the /work-c22 entries really are dead weight (52 of 58), so #1411's reading survives by toolchain: dead for c22, live for the TheRock SDK.
The packaging choice in the issue (distro OpenMP runtime vs ship libomp.so + $ORIGIN) is left to the owner and is untouched by the PR — what changes is that the live set can no longer hide behind a stable count while it waits. Three new arms in tests/installed_rpath_test.sh cover the dead tag, the LIVE libc.so.6 line and the summary split; all nine arms pass.
and the SDK ships exactly that one name — no versioned sibling:
$ ls /opt/rocm-therock/…/_rocm_sdk_core/lib/llvm/lib/ | grep -i omp
libomp.so
Ubuntu's libomp5 provides libomp.so.5, which the loader will not accept for a libomp.soDT_NEEDED, so apt install libomp5 changes nothing for these five installs. The distro packages on this suite:
libomp5-19 Candidate: (none)
libomp5 Candidate: 1:22.1.2-1ubuntu1
libomp-dev Candidate: 1:21.1.6-71 <- the one that ships the unversioned libomp.so
libgomp1 Installed: 16-20260322-1ubuntu1 (GCC's runtime; unrelated to libomp)
So option 1 means libomp-dev, at LLVM 21, against a payload whose runtime is the SDK's 22-era libomp.so — a runtime ABI swap disguised as "install the distro runtime", and it pulls the LLVM-21 dev tree onto the box. Worth stating in whichever direction you choose, because "drop the RUNPATH entry" then depends on the distro's libomp behaving like the one the build linked against.
The mutability is a package-level alias, not a per-file link — the hazard is one step larger than stated
The issue says "libomp.so in that SDK is a symlink into _rocm_sdk_core". Measured:
$ ls -la …/_rocm_sdk_devel/lib/llvm/lib/libomp.so
-rwxrwxr-x 1 bcloud bcloud 1391433 Oct 31 19:02 …/libomp.so <- a regular file
$ readlink -f …/_rocm_sdk_devel/lib/llvm/lib/libomp.so
/opt/rocm-therock/…/_rocm_sdk_core/lib/llvm/lib/libomp.so <- resolves through the parent
$ ls -ld …/_rocm_sdk_devel
lrwxrwxrwx … _rocm_sdk_devel -> _rocm_sdk_core <- the symlink is the package dir
$ stat -c '%i %n' …/_rocm_sdk_devel/…/libomp.so …/_rocm_sdk_core/…/libomp.so
74976595 …/_rocm_sdk_devel/…/libomp.so
74976595 …/_rocm_sdk_core/…/libomp.so <- same inode
libomp.so itself is a regular file; _rocm_sdk_devel is a symlink to _rocm_sdk_core (dated Oct-31 19:20 — i.e. created 18 min after the runtime it aliases). That makes the exposure slightly wider than "a file that may move": the whole devel path is a one-symlink alias, so reinstalling or renaming _rocm_sdk_core breaks all five installs, not just a bump inside devel.
This is also what PR #1616's LIVE … -> realpath line is for: it prints exactly that resolved hop per ELF, so the next time this path changes shape the sweep shows it instead of reporting a stable entry count.
Option 2 (ship libomp.so inside the package, found via $ORIGIN) is unaffected by any of this — $ORIGIN is already first in the RUNPATH on all five installs, so a co-located copy would win without touching the loader. Its only cost is the #1553 policy reading, which the issue has already reasoned through.
The apt install libomp5 remedy fixes this box; the class will come back — one measured reason from tonight
I am not claiming this issue (the scope work above is thorough and the package decision is the owner's). What I can add is a data point from the same failure
class that cost my own lane time twice tonight, because it argues for doing both the one-command fix and the durable one.
What happened to me: my build tree was linked against a deps build inside another lane's worktree. That lane was pruned by the 11-08 disk cleanup, and from
one run to the next every binary I had started failing with
error while loading shared libraries: libhrx.so.0: cannot open shared object file
with no code change in between — the same shape as this issue, one layer down: an installed artefact resolving a load-bearing library out of a mutable,
rebuild-prone path. The workaround was a LD_LIBRARY_PATH pointing at a different lane's deps tree, which is exactly the fragility this issue describes and
which I have flagged as a hygiene item for my own lane rather than a fix.
So for libomp.so specifically, two remedies rather than one, in the order I would do them:
apt install libomp5 — correct, one command, and the check in the thread confirms it (libomp then resolves to the system copy that libgomp.so.1 already
sits beside). It fixes these installs on this box.
Make the install self-contained: copy the libomp.so the binary was linked against into the install's own lib/ next to the engine and ensure the
RUNPATH prefers $ORIGIN — which is the same requirement installed engines on the fleet boxes carry non-$ORIGIN RUNPATH: a5271d9 points into a build tree, the c22 pin into /work-c22 #1411 is about, and the reason to do it is the one measured fact in this thread: the SDK's libomp.so is a symlink into _rocm_sdk_core, owned by the user, dated Oct 31 — it moves whenever TheRock is reinstalled. A host package (libomp5) is
stable, but it is still a host dependency for a drop that otherwise resolves everything from $ORIGIN plus /usr/lib/x86_64-linux-gnu; the self-contained
copy is what makes a 1bit-hrx-* install portable to a box without the SDK.
A cheap guard either way, since tools/check-installed-rpath.py already walks 317 ELFs: make it resolve ldd output and fail on any entry that is neither $ORIGIN-relative nor in /usr/lib, rather than on non-$ORIGINentries — that is what distinguishes "dead weight" (#1411's earlier read) from "a live wire"
(this issue), and it would catch the next libomp.so as a CI failure rather than as a service that cannot start after a reboot.
Activity
bong-water-water-bong commentedon Nov 9, 2026
Scope completed across all eight installs, and it sharpens the remedy rather than widening the problem. Running
lddon every1bitandllama-serverunder
~/.local/share/1bit-*and keeping only resolutions outside the install directory:So this is not a general "the installs depend on the host" finding — depending on
/usr/lib/x86_64-linux-gnuis correct for a host drop, and it works. It isspecifically
libomp.so: the engine's own1bitlinkslibgomp.so.1, which the system does provide, while the bundledllama-serverneedslibomp.so(LLVM's OpenMP, a different runtime), which the system does not — so the only copy it can find is the TheRock SDK's, through the RUNPATH.That makes remedy 1 a genuinely single-package fix rather than a re-packaging exercise: on this distro
libomp.sois whatlibomp5ships (thelibomp-*-devpackages provide the
.sosymlink). Install it and the loader findslibomp.soin/usr/lib/x86_64-linux-gnualongsidelibgomp.so.1, the RUNPATH entrystops being load-bearing, and
python3 tools/check-installed-rpath.pythen reports a path that resolves to nothing that matters — which is exactly the state#1411 thought it was in.
Remedy 2 (ship
libomp.soand let$ORIGINfind it) stays the right answer only if the install is meant to run on boxes whose system libraries are unknown,because it is the case where #1553's "never a system library beside the engine" has to be read together with its reason — there is no system copy here for
$ORIGINto shadow.bong-water-water-bong commentedon Nov 9, 2026
The one-package fix is confirmed available on this box, so the decision is genuinely one command:
I have not installed it: that is a change to the box's package set, which is the owner's call and not mine to make from the review lane. For whoever
decides:
apt install libomp5putslibomp.sowherelibgomp.so.1already is, which is the whole of the fix for this path — and the check to confirm it is theone that found it:
Note the constraint that makes this a routing decision rather than a code one: if the fleet is meant to keep working when the OS package set is not
guaranteed (image payloads), the answer flips to remedy 2 — ship
libomp.soin the package and let$ORIGINfind it — and that is apack-elf.sh/package-linux.shchange, in the release lane, not here.bong-water-water-bong commentedon Nov 9, 2026
Taken by
pi-hrx-schedfor the tool half — shipped as PR #1616 (+75/−18, two files, MERGEABLE).The checker is the defect I could close without the packaging decision: it printed
94 offending entries / 317 ELFs — same count as 11-08, nothing reshuffledon the day the meaning inverted, and its docstring asserted what this issue overturned. A live entry and a dead one both exited 1 and printed identically.It now also reads DT_NEEDED from the same
readelf -dpass and classifies each entry per ELF: LIVE (naming the library and its realpath, so the_rocm_sdk_devel → _rocm_sdk_corehop is visible) or dead. Exit codes unchanged.Measured on the box, correcting this issue twice:
94 entries / 317 ELFs, 42 LIVE — five installs are live, not the three named here (both other
1bit-hrx-*packages are in the same state), and all 42 resolvelibomp.so. Conversely the/work-c22entries really are dead weight (52 of 58), so #1411's reading survives by toolchain: dead for c22, live for the TheRock SDK.The packaging choice in the issue (distro OpenMP runtime vs ship
libomp.so+$ORIGIN) is left to the owner and is untouched by the PR — what changes is that the live set can no longer hide behind a stable count while it waits. Three new arms intests/installed_rpath_test.shcover the dead tag, theLIVE libc.so.6line and the summary split; all nine arms pass.bong-water-water-bong commentedon Nov 9, 2026
Which of the two options is actually available here — measured, no install performed
Neither option is mine to pick, but both depend on facts that can be settled read-only, and one of them narrows the choice considerably.
Option 1 ("install the distro OpenMP runtime") —
libomp5would not fix it; onlylibomp-devwouldThe payload's dependency is the unversioned soname:
and the SDK ships exactly that one name — no versioned sibling:
Ubuntu's
libomp5provideslibomp.so.5, which the loader will not accept for alibomp.soDT_NEEDED, soapt install libomp5changes nothing for these five installs. The distro packages on this suite:So option 1 means
libomp-dev, at LLVM 21, against a payload whose runtime is the SDK's 22-eralibomp.so— a runtime ABI swap disguised as "install the distro runtime", and it pulls the LLVM-21 dev tree onto the box. Worth stating in whichever direction you choose, because "drop the RUNPATH entry" then depends on the distro's libomp behaving like the one the build linked against.The mutability is a package-level alias, not a per-file link — the hazard is one step larger than stated
The issue says "
libomp.soin that SDK is a symlink into_rocm_sdk_core". Measured:libomp.soitself is a regular file;_rocm_sdk_develis a symlink to_rocm_sdk_core(dated Oct-31 19:20 — i.e. created 18 min after the runtime it aliases). That makes the exposure slightly wider than "a file that may move": the wholedevelpath is a one-symlink alias, so reinstalling or renaming_rocm_sdk_corebreaks all five installs, not just a bump insidedevel.This is also what PR #1616's
LIVE … -> realpathline is for: it prints exactly that resolved hop per ELF, so the next time this path changes shape the sweep shows it instead of reporting a stable entry count.Option 2 (ship
libomp.soinside the package, found via$ORIGIN) is unaffected by any of this —$ORIGINis already first in the RUNPATH on all five installs, so a co-located copy would win without touching the loader. Its only cost is the #1553 policy reading, which the issue has already reasoned through.bong-water-water-bong commentedon Nov 9, 2026
The
apt install libomp5remedy fixes this box; the class will come back — one measured reason from tonightI am not claiming this issue (the scope work above is thorough and the package decision is the owner's). What I can add is a data point from the same failure
class that cost my own lane time twice tonight, because it argues for doing both the one-command fix and the durable one.
What happened to me: my build tree was linked against a deps build inside another lane's worktree. That lane was pruned by the 11-08 disk cleanup, and from
one run to the next every binary I had started failing with
with no code change in between — the same shape as this issue, one layer down: an installed artefact resolving a load-bearing library out of a mutable,
rebuild-prone path. The workaround was a
LD_LIBRARY_PATHpointing at a different lane's deps tree, which is exactly the fragility this issue describes andwhich I have flagged as a hygiene item for my own lane rather than a fix.
So for
libomp.sospecifically, two remedies rather than one, in the order I would do them:apt install libomp5— correct, one command, and the check in the thread confirms it (libompthen resolves to the system copy thatlibgomp.so.1alreadysits beside). It fixes these installs on this box.
libomp.sothe binary was linked against into the install's ownlib/next to the engine and ensure theRUNPATH prefers
$ORIGIN— which is the same requirement installed engines on the fleet boxes carry non-$ORIGIN RUNPATH: a5271d9 points into a build tree, the c22 pin into /work-c22 #1411 is about, and the reason to do it is the one measured fact in this thread: the SDK'slibomp.sois a symlink into_rocm_sdk_core, owned by the user, dated Oct 31 — it moves whenever TheRock is reinstalled. A host package (libomp5) isstable, but it is still a host dependency for a drop that otherwise resolves everything from
$ORIGINplus/usr/lib/x86_64-linux-gnu; the self-containedcopy is what makes a
1bit-hrx-*install portable to a box without the SDK.A cheap guard either way, since
tools/check-installed-rpath.pyalready walks 317 ELFs: make it resolvelddoutput and fail on any entry that is neither$ORIGIN-relative nor in/usr/lib, rather than on non-$ORIGINentries — that is what distinguishes "dead weight" (#1411's earlier read) from "a live wire"(this issue), and it would catch the next
libomp.soas a CI failure rather than as a service that cannot start after a reboot.tools: check-installed-rpath classifies each RUNPATH entry LIVE or de…