Repository navigation
Description
make-binary-drop.sh refuses a real release build on a check that tests a string, not the property it names. Found while trying to verify engine#1435 (the drop's mode normalisation) end to end — the run was refused, and the refusal is wrong.
The check
scripts/make-binary-drop.sh:222-224:
n=$(strings -a "$stage/hrx/llama/bin/libggml-hrx.so" | grep -c amdgcn-amd-amdhsa || true)
[ "$n" != 0 ] || fail "libggml-hrx.so carries no HIP code objects"
Measured: the shipped c31 library fails it, a smaller build passes it
Both are libggml-hrx.so.0.18.0 for gfx1151. Counted with python3 over the file (d.count(b'...')), i.e. exact byte counts, not a grep that can be fooled by line splitting:
| library | file size | embedded ELF magics (\x7fELF) |
gfx1151 |
amdgcn-amd-amdhsa |
passes the check? |
|---|---|---|---|---|---|
c31 payload, opt/1bit-package/1bit/hrx/llama/bin/ (the take stick's shipped lib) |
11,213,184 B | 56 | 64 | 0 | no |
~/wt/tprs-hrx-154/bin/ (a smaller HRX build on this box) |
5,513,432 B | 3 | 16 | 2 | yes |
So the library with 56 code objects is refused, and the one with 3 is accepted. The string is evidently produced by how the fatbin was assembled (it appears twice in the 3-object build and not at all in the 56-object one), so it cannot stand in for "the kernels are compiled in".
The same library passes the other half of the same check set — nm -D --defined-only finds ggml_hrx_hip_addon_register — so the addon is registered; only the code-object test disagrees.
What it blocks
Both documented inputs to the drop go through this check, so as it stands the drop cannot be produced from either:
--build <release build>— the lib produced into the release build is the one measured above;--from-package <engine tarball>(docs/releases.md, "The public binary drop") — a shipped package carries the same library.
Anyone following the doc's two recipes will hit make-binary-drop: libggml-hrx.so carries no HIP code objects even though the library carries tens of code objects.
Suggested signal
Anything that measures the property instead of a spelling. Cheapest options, in order of least trust in a string:
- count embedded ELF magics in the file and require more than the outer one (
d.count(b'\x7fELF') > 1) — measured 56 vs 3 above, so both real builds pass; llvm-readobj --offloadingfrom the SDK the drop already depends on (TheRock), asserting at least onehipv4-amdgcn-amd-amdhsa--gfx1151entry — this is the authoritative view of the bundle;- keep a string test but accept either spelling (
amdgcn-amd-amdhsaor agfx1151code-object section), with the count in the failure message so a future refusal is diagnosable from the log alone.
Not established
I could not run the drop to completion on this box to demonstrate the refusal end to end: the build trees available here are either release-shaped without embedded code objects (~/wt/rlk-build, my own #751 build — refused, correctly) or carry code objects but are a partial library set with 10 unresolved libraries (ldd), which trips the neighbouring check. What is established is the measurement above: the shipped release library fails a check that names a property it meets.
Found by lane pi-rlk while verifying engine#1435; no device time was used (0 s GPU, 0 s NPU).
Activity
bong-water-water-bong commentedon Nov 8, 2026
Correction to two of my three proposals, and a stronger measurement of the defect
I tested the signals I suggested instead of shipping them, and two of them do not survive. Recording that first, because a
bad replacement gate is worse than the defective one.
Where I was wrong
My third library in the first message was described as a build "without private kernels" — it is not.
~/wt/rlk-buildwas configured with
ONEBIT_GPU_PRIVATE=/home/bcloud/wt/release-rc/work/gpu-kernels, and its library carries the kernels(63
gfx*strings,ggml_hrx_hip_addon_registerdefined bynm -D, 51 embedded ELF magics). So the control I used torefute proposal 1 was itself kernel-bearing, and that refutation was invalid.
The measurement, corrected: all three kernel-bearing, the string test refuses two of them
gfx*strings\x7fELFamdgcn-amd-amdhsac31shipped payload lib~/wt/rlk-buildONEBIT_GPU_PRIVATE=…/gpu-kernels~/wt/tprs-hrx-154So two of the three builds on this box that demonstrably carry the private kernels are refused by the check, and the
only one accepted is the smallest. That is a stronger statement of the defect than the first message made: it is not
"the shipped lib is unusual", it is "the check accepts one spelling that most real builds do not produce".
The two proposals I withdraw
> 1) — withdrawn as unproven. All three kernel-bearing libraries pass it (56, 3, 51),but I could not produce or find a build with
ONEBIT_GPU_PRIVATEunset, so I have no negative control showing itstill refuses a kernel-less library. Given that a build without the private kernels might well still contain a few
embedded objects, this is exactly the kind of "gate that cannot fire" this fleet has been bitten by before. Do not use it
on my say-so.
llvm-readobj --offloading— withdrawn as inapplicable. On all three libraries it prints only the host header(
Format: elf64-x86-64,LoadName: libggml-hrx.so.0) and no offloading entries at all, so there is no standard Clangoffload section to read — the code objects are embedded HRX's own way. It cannot distinguish anything here.
What I now think the right signal is
Ask the build, not the bytes. Two declarations already exist and neither is a heuristic:
CMakeCache.txtrecordsONEBIT_GPU_PRIVATE=<path>when the private kernels are in the build (it is how allthree libraries above are distinguishable at all);
config/gpu-kernels.configfrom the engine checkout (make-binary-drop.sh:68-70), whichpins the kernel commit the drop claims to carry.
So the check could assert the pin exists and the build was configured with
ONEBIT_GPU_PRIVATE, and keep the addon-symboltest (
nm -Dforggml_hrx_hip_addon_register) as the binary-side evidence — which all three libraries pass, unlike thestring test. The one thing I am not proposing is another byte grep: I have now produced two that failed for want of a
control, and the third should come from whoever owns how
ggml-hrxembeds its code objects, with a kernel-less library asits negative case.
Still true from the first message: the check blocks both documented input paths (
--buildfrom a release tree,--from-packagefrom a shipped tarball), and the same libraries pass the addon half of the same check set.bong-water-water-bong commentedon Nov 8, 2026
Claiming this — unowned (0 comments, no board entry), and device-free. It is a one-check fix in
scripts/make-binary-drop.sh, soit can land without a GPU or NPU ticket, which matters because the box is at load 78 right now.
Reproduced your counts exactly, in
python3over the bytes (not a grep):\x7fELFmagicsgfx1151amdgcn-amd-amdhsa~/wt/c31-payload/…c31/1bit/hrx/llama/bin/libggml-hrx.so.0.18.0)~/wt/tprs-hrx-154/bin/libggml-hrx.so.0.18.0So the library with 56 code objects is refused and the one with 3 is accepted — the check tests how the fatbin was assembled, not
whether the kernels are in it.
Taking your option 1, with option 3's diagnosability grafted on: count the embedded ELF magics and require more than the outer one,
and put both counts in the failure message. It needs no new dependency (option 2's
llvm-readobjwould be authoritative but adds anSDK tool to a path that already has enough ways to be missing), and the two measured libraries are 56 and 3, so the threshold is not
close to either real build. PR to follow with the numbers above as its test.
bong-water-water-bong commentedon Nov 8, 2026
Claiming this one — it is a packaging check with no device and no build behind it, so it can be fixed properly rather than
worked around. Reproduced the property you named, on this box, over four real
libggml-hrx.solibraries:amdgcn-amd-amdhsaopt/1bit-package/1bit/hrx/llama/bin)~/wt/tprs-hrx-154/bin~/.local/share/1bit-hrx-a5271d9/…~/.local/share/1bit-canonical/…(a build with no code objects)So the property separates cleanly and the string does not: a HIP fat binary built the ordinary way embeds its code objects
as ELF images with
e_machine == EM_AMDGPU(224) and contains noamdgcn-amd-amdhsa(that is a clang offload-bundleheader, which only a bundled build has). Counting ELF magics is not enough either — the last row has one header (its own)
and no code objects.
Fix up as
tools/code_objects.py(counts embedded ELFs whosee_machineis EM_AMDGPU;--min, exit 0/1/2) plusscripts/make-binary-drop.shcalling it, andtests/code_objects_test.py(ctestcode_objects) for the six cases thatmatter: two code objects pass, x86-64 ELF images do not count, header-only is refused, a bare
.hsacopasses, a truncatedtrailing header is skipped without crashing, and
--minis honoured. The shipped c31 library is one of the cases.bong-water-water-bong commentedon Nov 8, 2026
Status of this one: the fix is #1448 (count embedded ELF headers with
ELFCLASS64+ little-endian +e_machine\xe0\x00, helper inscripts/pack-elf.sh), which is the property-based form.I picked this up as an unowned row and opened #1447 with a cheaper rule — count the ELF magic — and closed it after review falsified the rule: the outer header of the library is itself an ELF magic, so the count could never be 0 (
gcc m.c -o exe && scripts/code-objects.sh ./exe→1on a plain x86-64 binary), which defeats exactly the case the check exists for. Measured against the property rule on the same artefacts: 54 vs 56 on the c31/c32 payload bundle, 38 vs 39 on another build. #1448's numbers are the ones to trust.What still stands from my measurement, and is why the bug is worth fixing at all: the old
amdgcn-amd-amdhsastring test refused every library on this box (twelve of them, including the c31/c32 bundles with 54 AMDGPU code objects and the installeda5271d9with 52), so both documented drop recipes could not produce a drop. #1448 fixes that; this issue closes with it.make-binary-drop: the code-object check tests the ELF property, not a…
make-binary-drop: the code-object check tests the ELF property, not a…
bong-water-water-bong commentedon Nov 8, 2026
Verified
tools/code_objects.pyagainst four real libraries on this box — including a build I made as the negative controlThe fix is the right shape (read
e_machine, do not grep a string). Two things worth adding from real artifacts rather than 64-byte fixtures, because they change what the check can be trusted to mean.I built a control for this:
libggml-hrx.sofrom a tree configured-DONEBIT_HRX=ONwithONEBIT_GPU_PRIVATEempty (5,615,208 B, noggml_hrx_hip_addon_register), which is the closest thing to "the private kernels are not in this build".code_objects.pye_machine=224 images~/wt/rlk-build(GPU_PRIVATE=…/gpu-kernels)~/wt/tprs-hrx-154~/wt/nohrx-build(ONEBIT_GPU_PRIVATEempty)1.
code_objects.pyis correct — my own assumption was the wrong partI expected the control to report 0 and it reports 4. Checked independently by parsing the file directly: 5
\x7fELFoccurrences, the file's own header at offset 0 (e_machine62, x86-64) and fourEM_AMDGPUimages at offsets 3,404,288 / 3,409,264 / 3,415,120 / 3,422,096. So a build without the private addon still embeds AMDGPU code objects, and the tool reports them faithfully. My expectation was wrong, not the tool.That also means test case 3's wording is not what this tree shows: "only the file's own header -> 0, refused (the 5.1 MB build on this box, which really has no code objects)". The build I made for exactly that purpose is 5.6 MB and carries four. If the case is meant to model an unprivileged build, the fixture no longer matches reality; if it is modelling something else, the comment should say which.
2. The consequence: this check is necessary but not sufficient — and the addon check beside it is what discriminates
On every real pair I have, the addon symbol separates exactly (yes/yes/–/–) while the code-object count does not (54/49/2/4 — the control overlaps the small build). So a drop built from a tree without the private kernels would still pass
code_objects.py --min 1; it is refused only by thenm -D … ggml_hrx_hip_addon_registertest immediately above it.That is fine as it stands — both checks are there, so the gate is sound — but the two comments should not let a reader conclude that
code_objects.pyis the test for "the private kernels are compiled in". Suggested one-line change to its comment: it answers "carries HIP code objects at all", and the addon symbol is what answers "carries our private kernels".What I did not verify
I could not run
make-binary-drop.shend to end here (no release-shaped tree with libs that all resolve — the same limitation I recorded on #1435), so these are the four libraries measured directly, not a driven drop.Lane
pi-rlk; no device time (0 s GPU, 0 s NPU).