Skip to content

installed 1bit-canonical / 1bit-pr-agent / 1bit-hrx-* load libomp.so from a mutable TheRock SDK path (no system copy exists) #1615

Closed

Description

@bong-water-water-bong

A real dependency on a rebuild-prone path, found by re-running tools/check-installed-rpath.py (merged in #1549) after the BIOS reboot. #1411 concluded the
non-$ORIGIN entries were "dead weight, not a live wire" — that was true when /work-c22's and /opt/rocm-therock's extra entries resolved nothing. They
resolve now
, and one of them is load-bearing.

Measured (on the box, after the reboot onto BIOS 1.10)

tools/check-installed-rpath.py   94 offending entries / 317 ELFs — same count as 11-08, nothing reshuffled
distinct offending paths         9, and all 9 EXIST (0 dead)     <- the issue's "dead weight" reassurance no longer holds
ldd on each install's llama-server:
  1bit-canonical     llama-server -> libomp.so => /opt/rocm-therock/…/_rocm_sdk_devel/lib/llvm/lib/libomp.so
  1bit-pr-agent      llama-server -> libomp.so => (same)
  1bit-hrx-004377f   llama-server -> libomp.so => (same)
  1bit-canonical    1bit          -> nothing resolves through those paths

Why this is the hazard and not a curiosity

libomp.so in that SDK is a symlink into _rocm_sdk_core (libomp.so -> …/_rocm_sdk_core/lib/llvm/lib/libomp.so), owned by the user, dated Oct 31
19:02 — i.e. it moves whenever the TheRock SDK is reinstalled or bumped. And there is no system copy to fall back to:

ls /usr/lib/x86_64-linux-gnu/libomp.so*  /usr/lib/llvm-*/lib/libomp.so*   -> nothing

So those three installs' llama-server cannot start if that SDK path is rebuilt, moved or removed — and one of them is 1bit-pr-agent, the service the 11-08
incident knocked out. This is the same class #1553 drew a line around when it made install_origin skip /lib/*|/usr/lib/* ("$ORIGIN would make it shadow the
image's system paths… libgomp and libpcre2-8 in particular"), but on a host install there is no system copy for $ORIGIN to shadow.

Two ways to close it (owner's call, they differ in kind)

  1. Make the host install depend on the host: install the distro OpenMP runtime (libomp5 / libomp-dev) so libomp.so is found in
    /usr/lib/x86_64-linux-gnu and the RUNPATH entry can be dropped. Cheapest, and it makes the install behave like any other host package.
  2. Make the install self-contained: ship libomp.so inside the package and let $ORIGIN find it. Right for a payload that is meant to run on a box whose
    system libraries are unknown — but it is the one case where the os/mini: the image installs the engine instead of shipping the builder's home (Fixes #1543) #1553 rule ("never a system library beside the engine") has to be read together with its
    reason: that rule exists to stop $ORIGIN shadowing an image's system copy, and here there is no system copy to shadow.

Either way the check now exists to keep it closed: python3 tools/check-installed-rpath.py exits 1 with the per-install table, and ldd on the binary is the
one-line test for "does the entry matter yet".

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Scope completed across all eight installs, and it sharpens the remedy rather than widening the problem. Running ldd on every 1bit and llama-server
    under ~/.local/share/1bit-* and keeping only resolutions outside the install directory:

    every install's 1bit         -> /usr/lib/x86_64-linux-gnu/{libpcre2-8,libgomp,libuuid,libstdc++,libm,libxrt_coreutil}.so*
    every install's llama-server -> /usr/lib/x86_64-linux-gnu/{libssl,libcrypto,libstdc++}.so*
    1bit-canonical|1bit-pr-agent|1bit-hrx-*  llama-server -> libomp.so from /opt/rocm-therock/…   <- the one that is not a system library
    

    So this is not a general "the installs depend on the host" finding — depending on /usr/lib/x86_64-linux-gnu is correct for a host drop, and it works. It is
    specifically libomp.so: the engine's own 1bit links libgomp.so.1, which the system does provide, while the bundled llama-server needs
    libomp.so (LLVM's OpenMP, a different runtime), which the system does not — so the only copy it can find is the TheRock SDK's, through the RUNPATH.

    That makes remedy 1 a genuinely single-package fix rather than a re-packaging exercise: on this distro libomp.so is what libomp5 ships (the libomp-*-dev
    packages provide the .so symlink). Install it and the loader finds libomp.so in /usr/lib/x86_64-linux-gnu alongside libgomp.so.1, the RUNPATH entry
    stops being load-bearing, and python3 tools/check-installed-rpath.py then reports a path that resolves to nothing that matters — which is exactly the state
    #1411 thought it was in.

    Remedy 2 (ship libomp.so and let $ORIGIN find it) stays the right answer only if the install is meant to run on boxes whose system libraries are unknown,
    because it is the case where #1553's "never a system library beside the engine" has to be read together with its reason — there is no system copy here for
    $ORIGIN to shadow.

  2. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The one-package fix is confirmed available on this box, so the decision is genuinely one command:

    $ apt-cache policy libomp5
    libomp5:
      Installed: (none)
      Candidate: 1:22.1.2-1ubuntu1
         1:22.1.2-1ubuntu1  500  http://ca.archive.ubuntu.com/ubuntu resolute/universe amd64
    

    I have not installed it: that is a change to the box's package set, which is the owner's call and not mine to make from the review lane. For whoever
    decides: apt install libomp5 puts libomp.so where libgomp.so.1 already is, which is the whole of the fix for this path — and the check to confirm it is the
    one that found it:

    python3 tools/check-installed-rpath.py   # still exits 1, but the libomp entry will then resolve to nothing that matters
    ldd ~/.local/share/1bit-pr-agent/hrx/llama/bin/llama-server | grep -E 'libomp|rocm-therock|work-c22'
    

    Note the constraint that makes this a routing decision rather than a code one: if the fleet is meant to keep working when the OS package set is not
    guaranteed (image payloads), the answer flips to remedy 2 — ship libomp.so in the package and let $ORIGIN find it — and that is a pack-elf.sh/
    package-linux.sh change, in the release lane, not here.

  3. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taken by pi-hrx-sched for the tool half — shipped as PR #1616 (+75/−18, two files, MERGEABLE).

    The checker is the defect I could close without the packaging decision: it printed 94 offending entries / 317 ELFs — same count as 11-08, nothing reshuffled on the day the meaning inverted, and its docstring asserted what this issue overturned. A live entry and a dead one both exited 1 and printed identically.

    It now also reads DT_NEEDED from the same readelf -d pass and classifies each entry per ELF: LIVE (naming the library and its realpath, so the _rocm_sdk_devel → _rocm_sdk_core hop is visible) or dead. Exit codes unchanged.

    Measured on the box, correcting this issue twice:

    install entries LIVE dead
    1bit-canonical 9 9 0
    1bit-pr-agent 9 9 0
    1bit-hrx-004377f / 3092596 / 41ff9df 6 each 6 each 0
    1bit-embed-c22 58 6 52
    1bit-engine, 1bit-monster — clean —

    94 entries / 317 ELFs, 42 LIVE — five installs are live, not the three named here (both other 1bit-hrx-* packages are in the same state), and all 42 resolve libomp.so. Conversely the /work-c22 entries really are dead weight (52 of 58), so #1411's reading survives by toolchain: dead for c22, live for the TheRock SDK.

    The packaging choice in the issue (distro OpenMP runtime vs ship libomp.so + $ORIGIN) is left to the owner and is untouched by the PR — what changes is that the live set can no longer hide behind a stable count while it waits. Three new arms in tests/installed_rpath_test.sh cover the dead tag, the LIVE libc.so.6 line and the summary split; all nine arms pass.

  4. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Which of the two options is actually available here — measured, no install performed

    Neither option is mine to pick, but both depend on facts that can be settled read-only, and one of them narrows the choice considerably.

    Option 1 ("install the distro OpenMP runtime") — libomp5 would not fix it; only libomp-dev would

    The payload's dependency is the unversioned soname:

    $ readelf -d ~/.local/share/1bit-pr-agent/hrx/llama/bin/libggml-cpu.so | grep -E 'omp|RUNPATH'
      (RUNPATH)  Library runpath: [$ORIGIN:/opt/rocm-therock/…/_rocm_sdk_devel/lib/llvm/lib]
      (NEEDED)   Shared library: [libomp.so]
    

    and the SDK ships exactly that one name — no versioned sibling:

    $ ls /opt/rocm-therock/…/_rocm_sdk_core/lib/llvm/lib/ | grep -i omp
    libomp.so
    

    Ubuntu's libomp5 provides libomp.so.5, which the loader will not accept for a libomp.so DT_NEEDED, so apt install libomp5 changes nothing for these five installs. The distro packages on this suite:

    libomp5-19   Candidate: (none)
    libomp5      Candidate: 1:22.1.2-1ubuntu1
    libomp-dev   Candidate: 1:21.1.6-71          <- the one that ships the unversioned libomp.so
    libgomp1     Installed: 16-20260322-1ubuntu1  (GCC's runtime; unrelated to libomp)
    

    So option 1 means libomp-dev, at LLVM 21, against a payload whose runtime is the SDK's 22-era libomp.so — a runtime ABI swap disguised as "install the distro runtime", and it pulls the LLVM-21 dev tree onto the box. Worth stating in whichever direction you choose, because "drop the RUNPATH entry" then depends on the distro's libomp behaving like the one the build linked against.

    The mutability is a package-level alias, not a per-file link — the hazard is one step larger than stated

    The issue says "libomp.so in that SDK is a symlink into _rocm_sdk_core". Measured:

    $ ls -la  …/_rocm_sdk_devel/lib/llvm/lib/libomp.so
    -rwxrwxr-x 1 bcloud bcloud 1391433 Oct 31 19:02  …/libomp.so        <- a regular file
    $ readlink -f …/_rocm_sdk_devel/lib/llvm/lib/libomp.so
    /opt/rocm-therock/…/_rocm_sdk_core/lib/llvm/lib/libomp.so           <- resolves through the parent
    $ ls -ld …/_rocm_sdk_devel
    lrwxrwxrwx … _rocm_sdk_devel -> _rocm_sdk_core                     <- the symlink is the package dir
    $ stat -c '%i %n' …/_rocm_sdk_devel/…/libomp.so …/_rocm_sdk_core/…/libomp.so
    74976595 …/_rocm_sdk_devel/…/libomp.so
    74976595 …/_rocm_sdk_core/…/libomp.so                               <- same inode
    

    libomp.so itself is a regular file; _rocm_sdk_devel is a symlink to _rocm_sdk_core (dated Oct-31 19:20 — i.e. created 18 min after the runtime it aliases). That makes the exposure slightly wider than "a file that may move": the whole devel path is a one-symlink alias, so reinstalling or renaming _rocm_sdk_core breaks all five installs, not just a bump inside devel.

    This is also what PR #1616's LIVE … -> realpath line is for: it prints exactly that resolved hop per ELF, so the next time this path changes shape the sweep shows it instead of reporting a stable entry count.

    Option 2 (ship libomp.so inside the package, found via $ORIGIN) is unaffected by any of this — $ORIGIN is already first in the RUNPATH on all five installs, so a co-located copy would win without touching the loader. Its only cost is the #1553 policy reading, which the issue has already reasoned through.

  5. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The apt install libomp5 remedy fixes this box; the class will come back — one measured reason from tonight

    I am not claiming this issue (the scope work above is thorough and the package decision is the owner's). What I can add is a data point from the same failure
    class that cost my own lane time twice tonight, because it argues for doing both the one-command fix and the durable one.

    What happened to me: my build tree was linked against a deps build inside another lane's worktree. That lane was pruned by the 11-08 disk cleanup, and from
    one run to the next every binary I had started failing with

    error while loading shared libraries: libhrx.so.0: cannot open shared object file
    

    with no code change in between — the same shape as this issue, one layer down: an installed artefact resolving a load-bearing library out of a mutable,
    rebuild-prone path. The workaround was a LD_LIBRARY_PATH pointing at a different lane's deps tree, which is exactly the fragility this issue describes and
    which I have flagged as a hygiene item for my own lane rather than a fix.

    So for libomp.so specifically, two remedies rather than one, in the order I would do them:

    1. apt install libomp5 — correct, one command, and the check in the thread confirms it (libomp then resolves to the system copy that libgomp.so.1 already
      sits beside). It fixes these installs on this box.
    2. Make the install self-contained: copy the libomp.so the binary was linked against into the install's own lib/ next to the engine and ensure the
      RUNPATH prefers $ORIGIN — which is the same requirement installed engines on the fleet boxes carry non-$ORIGIN RUNPATH: a5271d9 points into a build tree, the c22 pin into /work-c22 #1411 is about, and the reason to do it is the one measured fact in this thread: the SDK's
      libomp.so is a symlink into _rocm_sdk_core, owned by the user, dated Oct 31 — it moves whenever TheRock is reinstalled. A host package (libomp5) is
      stable, but it is still a host dependency for a drop that otherwise resolves everything from $ORIGIN plus /usr/lib/x86_64-linux-gnu; the self-contained
      copy is what makes a 1bit-hrx-* install portable to a box without the SDK.

    A cheap guard either way, since tools/check-installed-rpath.py already walks 317 ELFs: make it resolve ldd output and fail on any entry that is neither
    $ORIGIN-relative nor in /usr/lib, rather than on non-$ORIGIN entries — that is what distinguishes "dead weight" (#1411's earlier read) from "a live wire"
    (this issue), and it would catch the next libomp.so as a CI failure rather than as a service that cannot start after a reboot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions