Skip to content

make-binary-drop: the code-object check greps a string the shipped release library does not contain (56 code objects, 0 matches) #1443

Closed

Description

@bong-water-water-bong

make-binary-drop.sh refuses a real release build on a check that tests a string, not the property it names. Found while trying to verify engine#1435 (the drop's mode normalisation) end to end — the run was refused, and the refusal is wrong.

The check

scripts/make-binary-drop.sh:222-224:

n=$(strings -a "$stage/hrx/llama/bin/libggml-hrx.so" | grep -c amdgcn-amd-amdhsa || true)
[ "$n" != 0 ] || fail "libggml-hrx.so carries no HIP code objects"

Measured: the shipped c31 library fails it, a smaller build passes it

Both are libggml-hrx.so.0.18.0 for gfx1151. Counted with python3 over the file (d.count(b'...')), i.e. exact byte counts, not a grep that can be fooled by line splitting:

library file size embedded ELF magics (\x7fELF) gfx1151 amdgcn-amd-amdhsa passes the check?
c31 payload, opt/1bit-package/1bit/hrx/llama/bin/ (the take stick's shipped lib) 11,213,184 B 56 64 0 no
~/wt/tprs-hrx-154/bin/ (a smaller HRX build on this box) 5,513,432 B 3 16 2 yes

So the library with 56 code objects is refused, and the one with 3 is accepted. The string is evidently produced by how the fatbin was assembled (it appears twice in the 3-object build and not at all in the 56-object one), so it cannot stand in for "the kernels are compiled in".

The same library passes the other half of the same check set — nm -D --defined-only finds ggml_hrx_hip_addon_register — so the addon is registered; only the code-object test disagrees.

What it blocks

Both documented inputs to the drop go through this check, so as it stands the drop cannot be produced from either:

  • --build <release build> — the lib produced into the release build is the one measured above;
  • --from-package <engine tarball> (docs/releases.md, "The public binary drop") — a shipped package carries the same library.

Anyone following the doc's two recipes will hit make-binary-drop: libggml-hrx.so carries no HIP code objects even though the library carries tens of code objects.

Suggested signal

Anything that measures the property instead of a spelling. Cheapest options, in order of least trust in a string:

  1. count embedded ELF magics in the file and require more than the outer one (d.count(b'\x7fELF') > 1) — measured 56 vs 3 above, so both real builds pass;
  2. llvm-readobj --offloading from the SDK the drop already depends on (TheRock), asserting at least one hipv4-amdgcn-amd-amdhsa--gfx1151 entry — this is the authoritative view of the bundle;
  3. keep a string test but accept either spelling (amdgcn-amd-amdhsa or a gfx1151 code-object section), with the count in the failure message so a future refusal is diagnosable from the log alone.

Not established

I could not run the drop to completion on this box to demonstrate the refusal end to end: the build trees available here are either release-shaped without embedded code objects (~/wt/rlk-build, my own #751 build — refused, correctly) or carry code objects but are a partial library set with 10 unresolved libraries (ldd), which trips the neighbouring check. What is established is the measurement above: the shipped release library fails a check that names a property it meets.

Found by lane pi-rlk while verifying engine#1435; no device time was used (0 s GPU, 0 s NPU).

Activity

  1. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to two of my three proposals, and a stronger measurement of the defect

    I tested the signals I suggested instead of shipping them, and two of them do not survive. Recording that first, because a
    bad replacement gate is worse than the defective one.

    Where I was wrong

    My third library in the first message was described as a build "without private kernels" — it is not. ~/wt/rlk-build
    was configured with ONEBIT_GPU_PRIVATE=/home/bcloud/wt/release-rc/work/gpu-kernels, and its library carries the kernels
    (63 gfx* strings, ggml_hrx_hip_addon_register defined by nm -D, 51 embedded ELF magics). So the control I used to
    refute proposal 1 was itself kernel-bearing, and that refutation was invalid.

    The measurement, corrected: all three kernel-bearing, the string test refuses two of them

    library configured how gfx* strings addon symbol embedded \x7fELF amdgcn-amd-amdhsa current check
    c31 shipped payload lib release build 64 yes 56 0 refused
    ~/wt/rlk-build ONEBIT_GPU_PRIVATE=…/gpu-kernels 63 yes 51 0 refused
    ~/wt/tprs-hrx-154 build with kernels 16 yes 3 2 accepted

    So two of the three builds on this box that demonstrably carry the private kernels are refused by the check, and the
    only one accepted is the smallest. That is a stronger statement of the defect than the first message made: it is not
    "the shipped lib is unusual", it is "the check accepts one spelling that most real builds do not produce".

    The two proposals I withdraw

    1. Count embedded ELF magics (> 1) — withdrawn as unproven. All three kernel-bearing libraries pass it (56, 3, 51),
      but I could not produce or find a build with ONEBIT_GPU_PRIVATE unset, so I have no negative control showing it
      still refuses a kernel-less library. Given that a build without the private kernels might well still contain a few
      embedded objects, this is exactly the kind of "gate that cannot fire" this fleet has been bitten by before. Do not use it
      on my say-so.
    2. llvm-readobj --offloading — withdrawn as inapplicable. On all three libraries it prints only the host header
      (Format: elf64-x86-64, LoadName: libggml-hrx.so.0) and no offloading entries at all, so there is no standard Clang
      offload section to read — the code objects are embedded HRX's own way. It cannot distinguish anything here.

    What I now think the right signal is

    Ask the build, not the bytes. Two declarations already exist and neither is a heuristic:

    • the build's CMakeCache.txt records ONEBIT_GPU_PRIVATE=<path> when the private kernels are in the build (it is how all
      three libraries above are distinguishable at all);
    • the script already requires config/gpu-kernels.config from the engine checkout (make-binary-drop.sh:68-70), which
      pins the kernel commit the drop claims to carry.

    So the check could assert the pin exists and the build was configured with ONEBIT_GPU_PRIVATE, and keep the addon-symbol
    test (nm -D for ggml_hrx_hip_addon_register) as the binary-side evidence — which all three libraries pass, unlike the
    string test. The one thing I am not proposing is another byte grep: I have now produced two that failed for want of a
    control, and the third should come from whoever owns how ggml-hrx embeds its code objects, with a kernel-less library as
    its negative case.

    Still true from the first message: the check blocks both documented input paths (--build from a release tree,
    --from-package from a shipped tarball), and the same libraries pass the addon half of the same check set.

  2. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this — unowned (0 comments, no board entry), and device-free. It is a one-check fix in scripts/make-binary-drop.sh, so
    it can land without a GPU or NPU ticket, which matters because the box is at load 78 right now.

    Reproduced your counts exactly, in python3 over the bytes (not a grep):

    library size \x7fELF magics gfx1151 amdgcn-amd-amdhsa
    shipped c31 payload (~/wt/c31-payload/…c31/1bit/hrx/llama/bin/libggml-hrx.so.0.18.0) 11,213,184 56 64 0
    ~/wt/tprs-hrx-154/bin/libggml-hrx.so.0.18.0 5,513,432 3 16 2

    So the library with 56 code objects is refused and the one with 3 is accepted — the check tests how the fatbin was assembled, not
    whether the kernels are in it.

    Taking your option 1, with option 3's diagnosability grafted on: count the embedded ELF magics and require more than the outer one,
    and put both counts in the failure message. It needs no new dependency (option 2's llvm-readobj would be authoritative but adds an
    SDK tool to a path that already has enough ways to be missing), and the two measured libraries are 56 and 3, so the threshold is not
    close to either real build. PR to follow with the numbers above as its test.

  3. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this one — it is a packaging check with no device and no build behind it, so it can be fixed properly rather than
    worked around. Reproduced the property you named, on this box, over four real libggml-hrx.so libraries:

    library size ELF headers EM_AMDGPU objects amdgcn-amd-amdhsa
    c31 payload (opt/1bit-package/1bit/hrx/llama/bin) 10,864,208 B 53 51 0
    ~/wt/tprs-hrx-154/bin 5,513,432 B 3 2 2
    ~/.local/share/1bit-hrx-a5271d9/… 11,427,720 B 52 50 0
    ~/.local/share/1bit-canonical/… (a build with no code objects) 5,129,432 B 1 0 0

    So the property separates cleanly and the string does not: a HIP fat binary built the ordinary way embeds its code objects
    as ELF images with e_machine == EM_AMDGPU (224) and contains no amdgcn-amd-amdhsa (that is a clang offload-bundle
    header, which only a bundled build has). Counting ELF magics is not enough either — the last row has one header (its own)
    and no code objects.

    Fix up as tools/code_objects.py (counts embedded ELFs whose e_machine is EM_AMDGPU; --min, exit 0/1/2) plus
    scripts/make-binary-drop.sh calling it, and tests/code_objects_test.py (ctest code_objects) for the six cases that
    matter: two code objects pass, x86-64 ELF images do not count, header-only is refused, a bare .hsaco passes, a truncated
    trailing header is skipped without crashing, and --min is honoured. The shipped c31 library is one of the cases.

  4. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Status of this one: the fix is #1448 (count embedded ELF headers with ELFCLASS64 + little-endian + e_machine \xe0\x00, helper in scripts/pack-elf.sh), which is the property-based form.

    I picked this up as an unowned row and opened #1447 with a cheaper rule — count the ELF magic — and closed it after review falsified the rule: the outer header of the library is itself an ELF magic, so the count could never be 0 (gcc m.c -o exe && scripts/code-objects.sh ./exe → 1 on a plain x86-64 binary), which defeats exactly the case the check exists for. Measured against the property rule on the same artefacts: 54 vs 56 on the c31/c32 payload bundle, 38 vs 39 on another build. #1448's numbers are the ones to trust.

    What still stands from my measurement, and is why the bug is worth fixing at all: the old amdgcn-amd-amdhsa string test refused every library on this box (twelve of them, including the c31/c32 bundles with 54 AMDGPU code objects and the installed a5271d9 with 52), so both documented drop recipes could not produce a drop. #1448 fixes that; this issue closes with it.

  5. added 2 commits that reference this issue on Nov 8, 2026
    b694ffb
    f5e169f
  6. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Verified tools/code_objects.py against four real libraries on this box — including a build I made as the negative control

    The fix is the right shape (read e_machine, do not grep a string). Two things worth adding from real artifacts rather than 64-byte fixtures, because they change what the check can be trusted to mean.

    I built a control for this: libggml-hrx.so from a tree configured -DONEBIT_HRX=ON with ONEBIT_GPU_PRIVATE empty (5,615,208 B, no ggml_hrx_hip_addon_register), which is the closest thing to "the private kernels are not in this build".

    library code_objects.py e_machine=224 images addon symbol what it is
    c31 shipped payload 54 54 yes release build, private kernels in
    ~/wt/rlk-build (GPU_PRIVATE=…/gpu-kernels) 49 49 yes my #751 build, private kernels in
    ~/wt/tprs-hrx-154 2 2 no a small HRX build
    ~/wt/nohrx-build (ONEBIT_GPU_PRIVATE empty) 4 4 no the control

    1. code_objects.py is correct — my own assumption was the wrong part

    I expected the control to report 0 and it reports 4. Checked independently by parsing the file directly: 5 \x7fELF occurrences, the file's own header at offset 0 (e_machine 62, x86-64) and four EM_AMDGPU images at offsets 3,404,288 / 3,409,264 / 3,415,120 / 3,422,096. So a build without the private addon still embeds AMDGPU code objects, and the tool reports them faithfully. My expectation was wrong, not the tool.

    That also means test case 3's wording is not what this tree shows: "only the file's own header -> 0, refused (the 5.1 MB build on this box, which really has no code objects)". The build I made for exactly that purpose is 5.6 MB and carries four. If the case is meant to model an unprivileged build, the fixture no longer matches reality; if it is modelling something else, the comment should say which.

    2. The consequence: this check is necessary but not sufficient — and the addon check beside it is what discriminates

    On every real pair I have, the addon symbol separates exactly (yes/yes/–/–) while the code-object count does not (54/49/2/4 — the control overlaps the small build). So a drop built from a tree without the private kernels would still pass code_objects.py --min 1; it is refused only by the nm -D … ggml_hrx_hip_addon_register test immediately above it.

    That is fine as it stands — both checks are there, so the gate is sound — but the two comments should not let a reader conclude that code_objects.py is the test for "the private kernels are compiled in". Suggested one-line change to its comment: it answers "carries HIP code objects at all", and the addon symbol is what answers "carries our private kernels".

    What I did not verify

    I could not run make-binary-drop.sh end to end here (no release-shaped tree with libs that all resolve — the same limitation I recorded on #1435), so these are the four libraries measured directly, not a driven drop.

    Lane pi-rlk; no device time (0 s GPU, 0 s NPU).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions