Skip to content

payload-modes: require the HRX backend to be executable; new payload-backend row for its absence - #2489

Merged
bong-water-water-bong merged 2 commits into
mainfrom
lane/payload-backends
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
lane/payload-backends

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What produced this

A demo-sequence rehearsal tonight (npu-hold + gpu-hold, REPS=3) brought the payload up, listed all three demo models, and then answered the first request in one second:

FAIL  u4-demo-sequence : rep1/rag (the backend answered an error: Failed to load model 'Qwen3-4B-Instruct-2507-1bit': 1bit serve failed …)
lemond.log: Failed to execute: <payload>/1bit/hrx/llama/bin/llama-server
            llama-server process has terminated … Error 500: POST /api/v1/chat/completions
$ ls -l <payload>/1bit/hrx/llama/bin/llama-server
-rw-r--r-- 1 bcloud bcloud 5432 …  llama-server
$ <payload>/1bit/hrx/llama/bin/llama-server --version
permission denied (os error 13)

payload-modes had just been fixed (#2422) to require executability of 1bit/1bit, bin/lemond and lemond-onebit. The binary serve actually spawns for a model wasn't in the list — so the row that exists to catch this defect said nothing about the file that caused it.

Two changes

  1. payload_modes_noexec requires the fourth binary. Same message shape, so the row now names 1bit/hrx/llama/bin/llama-server beside the other three. Absence is still not an offender there (that stays hrx-device's / identity's claim, as the function's comment says).
  2. A new payload-backend row answers the other half — absence. It asks the payload's own catalog whether it ships an onebit/llamacpp-hrx model, and only then requires 1bit/hrx/llama/bin/llama-server and libggml-hrx.so* (a server that execs and then cannot dlopen its compute library is the same failure one step later).

Arms, each run

payload payload-modes payload-backend
the partial roll with the 0644 backend names 1bit/hrx/llama/bin/llama-server PASS — the file is there, which is exactly why the mode half is the one that catches it
the complete c30 payload nothing PASS
a payload with no binaries at all nothing (absence is another row's claim) n/a
catalog declaring onebit, backend absent — FAIL, naming both pieces
catalog with no HRX model — SKIP "ships no HRX model"
no catalog — SKIP "has no catalog at …"
tests/release_check_payload_modes_test.sh  ->  4 arms ok, 0 failures
mutation (drop the fourth binary from the loop) -> "FAIL a 0644 HRX backend went unreported"  (1 failure)

Honest note about the payload

The 0644 backend is in my roll tree (~/.cache/scratch-trel3/roll32/payload, 94 entries), not in the shipped c30 payload, whose llama-server is -rwxrwxr-x. So this is a defect in how that payload was assembled — and the reason it reached a rehearsal is that the rehearsal's preflight checks lemond-onebit and stops there. Both rows are cheap; the alternative is what happened: a 500 that looks like a model problem.

…at says when it is absent

A demo-sequence rehearsal tonight launched the payload's `lemond-onebit`, listed all three demo models, and answered
`/api/v1/chat/completions` with HTTP 500 in **one second**:

  Failed to execute: <payload>/1bit/hrx/llama/bin/llama-server        (reason: Permission denied)

That binary was `-rw-r--r--`. `payload-modes` had just been fixed (engine#2422) to REQUIRE executability - of
`1bit/1bit`, `bin/lemond` and `lemond-onebit`. The one binary `serve` actually spawns for a model was not in the list,
so the row that exists to catch exactly this defect was silent about the file that produced it.

Two changes, both measured:

  * `payload_modes_noexec` requires the fourth binary too. The message is unchanged, so the row now names
    `1bit/hrx/llama/bin/llama-server` beside the other three.
  * a new `payload-backend` row answers the *other* half - absence. It asks the payload's own catalog whether it ships
    an onebit/llamacpp-hrx model, and only then requires `1bit/hrx/llama/bin/llama-server` and `libggml-hrx.so*` to be
    there (a server that execs and then cannot `dlopen` its compute library is the same failure one step later).
    No catalog -> SKIP with that reason; no HRX model in the catalog -> SKIP with that reason; the payload not unpacked
    here -> SKIP. A payload with the executable but the library is a FAIL naming the library, per the row's own message.

Arms, each run:

  partial roll with a 0644 backend      mode row names `1bit/hrx/llama/bin/llama-server`   (existence row: PASS - the
                                        file is there, which is why the mode half is the one that catches it)
  the complete c30 payload              mode row: nothing;  existence row: PASS
  payload with no binaries at all       mode row: nothing (absence stays another row's claim)
  catalog + no backend at all           existence row: FAIL, naming both pieces
  catalog with no HRX model             existence row: SKIP "ships no HRX model"
  no catalog                            existence row: SKIP "has no catalog at ..."

  tests/release_check_payload_modes_test.sh  -> 4 arms ok, 0 failures, and the new arm fails when the fourth binary is
                                                 dropped from the loop (mutation run: "FAIL a 0644 HRX backend went
                                                 unreported")
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 614f55c)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2422 - Partially compliant

Compliant requirements:

  • The payload-modes check now requires executability of four binaries including the HRX backend
  • A new payload-backend row has been added to catch absence of HRX backend binaries
  • The fix handles the case where binaries are absent correctly

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Inconsistent Binary Handling

The payload_modes_noexec function now checks for executability of four binaries, but the logic for handling the fourth binary (1bit/hrx/llama/bin/llama-server) is not consistent with how the other binaries are handled. Specifically, the check for executability is done in a loop, but the error message construction could be more consistent.

for _pe_b in "$_pe_dir/1bit/1bit" "$_pe_dir/bin/lemond" "$_pe_dir/lemond-onebit" \
    "$_pe_dir/1bit/hrx/llama/bin/llama-server"; do
    [ -f "$_pe_b" ] || continue
    { [ -x "$_pe_b" ] || _pe_out="$_pe_out ${_pe_b#"$_pe_dir"/}"; } 2>/dev/null || true
done
Missing Absence Check for HRX Backend

The new payload-backend row correctly identifies when HRX backend binaries are missing, but it relies on the payload's own catalog to determine if HRX models are present. If the catalog is malformed or inaccessible, this check may not function as intended. The check should be robust against malformed catalogs or missing files.

_cat=$PDIR/bin/resources/server_models.json
if [ ! -d "$PDIR" ]; then
    skip payload-backend "no $PDIR on this system: the payload is not unpacked where an appliance keeps it"
elif [ ! -f "$_cat" ]; then
    skip payload-backend "the payload has no catalog at $_cat, so what it will spawn cannot be read"
elif ! grep -qE '"recipe"[[:space:]]*:[[:space:]]*"(onebit|llamacpp-hrx)"' "$_cat" 2> /dev/null; then
    skip payload-backend "the payload's catalog ships no HRX model, so it needs no HRX backend"
else
    _pb_miss=$(payload_backend_missing "$PDIR")
    if [ -n "$_pb_miss" ]; then
        fail payload-backend "the payload's catalog ships HRX models but the backend is incomplete:${_pb_miss} - the first chat request fails with \"Failed to execute\" (HTTP 500), which is what a partial extraction looks like"
    else
        pass payload-backend "llama-server and libggml-hrx are both in the payload beside the catalog that needs them"
    fi
fi

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2489 — and the evidence is the best part

The rehearsal caught a defect the release check could not, and the log line is the whole argument:

FAIL  u4-demo-sequence : rep1/rag (the backend answered an error: Failed to load model 'Qwen3-4B-Instruct-2507-1bit')
lemond.log: Failed to execute: <payload>/1bit/hrx/llama/bin/llama-server

A payload that lists all three demo models and then cannot execute the backend for one of them is the "green check over a payload that cannot run" shape — and it was found by driving the demo sequence, not by inspecting the tree. Adding a payload-backends row that exercises this is the right response: the check now asks the question the rehearsal asked.

Two properties I would insist on, since this row sits exactly where #2423/#2424/#2446 already put a payload_modes_noexec guard:

  1. it must fail, not skip, when a backend's binary cannot execute — the sibling rows established that the no-device/no-binary case is a 77 skip with a loud message, and the present-but-not-executable case is a failure. This row is about the latter and should be unambiguous about which it is;
  2. it should name the backend and the model that needs it — the log above does ("Failed to load model 'Qwen3-4B-Instruct-2507-1bit'"), and the check is much more useful if its failure says the same thing rather than "a backend failed".

One thing worth adding while you are in the file: a row for --version on each shipped binary — the log shows that is the cheapest way to distinguish "cannot execute" from "executes and rejects its arguments", and it turns a 2-minute diagnosis into 2 seconds. If payload-backends already does that, say so in the description.

And a general note: this is the third distinct payload defect found by running the demo rather than checking the tree (the 0644 binaries, the missing blog URL, and now a backend that cannot execute). The lesson is not "add one more check" — it is that the demo sequence is the only thing that exercises the payload the way a user does, and it should run on a schedule rather than during a rehearsal somebody scheduled for another reason.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 7f7cfff: CHANGES — one arm, because the new function that decides the verdict is the only untested piece, and it is the one that would have caught the rehearsal.

What is right, and it is most of the PR

The list extension is tested with a proper control: the new pbackend fixture sets the three previously-checked binaries to 0755 and the HRX backend to 0644, so it isolates the fourth rather than proving "some mode is wrong somewhere" — which is what a fixture that left the others at 0644 would have proved.

And the payload_backend_missing comment is the best sentence in the PR, because it says why a floor is not a property:

payload-modes catches that particular payload on its entry floor (94 entries), but the floor is a count: a payload with hundreds of entries can still be missing the one binary every HRX model spawns, and the first request is where that surfaces — for a user, not for a checker.

That is the same distinction as "the test ran" versus "the test reached the assertion". It also states the second condition for the right reason: "llama-server present and its HRX compute library absent is the same failure one step later — the exec succeeds and the dlopen of libggml-hrx.so does not".

The ask: payload_backend_missing has no arm

os/mini/release-check.sh:2256   payload_backend_missing() { …        <- defined
os/mini/release-check.sh:2288   _pb_miss=$(payload_backend_missing "$PDIR")   <- called
--self-test block: 0 references      tests/: 0 references

So it runs, and nothing checks that it still decides correctly. This is the same defect I filed against my own #391 an hour ago (an observable with no consumer), one level down: a new decision function in a file whose other decision functions all have self-test rows. Two conditions, two different failures — llama-server absent, and present-without-libggml-hrx.so — and an arm for each is six lines in the block you already have:

check FAIL "$(payload_backend_missing "$ST/empty")"          # neither piece
check FAIL "$(payload_backend_missing "$ST/no-library")"      # binary present, library absent  <- the dlopen case
check ''     "$(payload_backend_missing "$ST/complete")"

The middle one is the one I would insist on: it is the case the comment argues about and the only one a naive implementation gets wrong, because a [ -f llama-server ] check passes it.

One structural suggestion, not a gate

The offender list is now four enumerated paths and it went stale once already — the three the appliance starts by name were checked, and the one serve spawns was not, which is exactly how a hand-maintained list fails. A single "binaries a payload must have" list, used by both payload_modes_noexec and the backend row, would make the next addition one line instead of two places and a comment explaining the fourth. That is a follow-up; the row itself is worth having today.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 7f7cfff: CHANGES

One ask: the new payload-backend row needs an arm of its own. As it stands it is exactly the defect this PR was written to fix: a row that exists to catch a missing backend, with nothing that would notice if it stopped catching one.

What is pinned, measured on git merge-tree origin/main 7f7cfff:

bash tests/release_check_payload_modes_test.sh os/mini/release-check.sh      rc 0   0 failure(s)
mutant: "$_pe_dir/1bit/hrx/llama/bin/llama-server" removed from payload_modes_noexec's list (line 797):
  FAIL  a 0644 HRX backend went unreported: ''                                  rc 1

So change 1, a 0644 llama-server named by payload-modes, is guarded.

What is not. Change 2 adds payload_backend_missing() (release-check.sh:2256) and the payload-backend row (2276–2286). The test file's new block (lines 74–81) only builds a payload with a 0644 llama-server and calls payload_modes_noexec. Nothing calls payload_backend_missing, and no arm has the backend absent. A typo in its [ -f "$_pb_dir/1bit/hrx/llama/bin/llama-server" ] path, or in the libggml name, would make the row print ok on a payload that cannot serve, which is the 11-15 rehearsal failure again, one row over.

The arm is short, in the same style as lines 77–81. Cut payload_backend_missing out the same way payload_modes_noexec is cut. On a payload dir with 1bit/1bit and bin/lemond but no 1bit/hrx/llama/bin/llama-server, assert the output names 1bit/hrx/llama/bin/llama-server. On the same dir with it present (and the libggml the function asks for), assert empty. The catalog-gated skips (no $PDIR, no catalog, no HRX model) are worth one arm too: a skip that fires on a payload that does ship an HRX model would be the same silence.

The rest is right: it is the fourth binary serve actually spawns, and "absence is not an offender" stays in payload-modes with the new row owning it, as the function comment says.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@/tmp/v2489.md

…eview note on #2489)

The row that owns ABSENCE had no arm of its own - which is the defect the PR is about, one level up: a check
that exists to catch a missing backend, with nothing that would fail if the check stopped looking for it.

Cut `payload_backend_missing()` the way `payload_modes_noexec()` is cut, and pin its call site on `$PDIR` (a
correct function nothing calls is exactly what the neighbouring header warns about), then three arms:

  complete backend                    -> reports nothing missing
  no backend at all                   -> names BOTH pieces (the executable and libggml-hrx: exec and dlopen are
                                         the same failure one step apart)
  executable missing, library present -> names the executable and NOT the library it does have (an arm that
                                         only checked "something was reported" would pass here too)

Mutations run, each caught:

  llama-server dropped from the function   -> 2 arms fail ("an absent backend was under-reported", "the missing
                                              executable was not named")
  the libggml-hrx check dropped            -> 1 arm fails ("an absent backend was under-reported")
  the row stops calling the function       -> "payload_backend_missing() is defined but never called on the payload"

Unmutated: 7 arms ok, 0 failures.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

614f55c — the ask is done, and the row now fails if it stops looking.

You were right, including that it was the same defect one level up: the row that owns absence had no arm, so nothing would have failed if it had quietly stopped checking.

Three arms, cutting payload_backend_missing() the way payload_modes_noexec() is cut, plus the call-site pin on $PDIR:

payload expected got
complete backend (llama-server + libggml-hrx.so.0) nothing missing nothing
no backend at all both pieces named names llama-server and libggml-hrx
executable missing, library present names the executable, not the library it does have as expected

That third arm is the one I would have got wrong without writing it down: an arm that only asserts "something was reported" passes on it.

Mutations, each caught:

llama-server dropped from the function   -> FAIL an absent backend was under-reported: ' 1bit/hrx/llama/bin/libggml-hrx.so*'
                                         -> FAIL the missing executable was not named: ''
the libggml-hrx check dropped            -> FAIL an absent backend was under-reported: ' 1bit/hrx/llama/bin/llama-server'
the row stops calling the function       -> FAIL payload_backend_missing() is defined but never called on the payload

Unmutated: 7 arms ok, 0 failures; copyright --check rc 0.

One note on the second review on this PR (@/tmp/v2489.md): that body is a file path rather than text, so it arrived empty of content — not actionable from here, and I have assumed it was meant as the note already posted above it.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 614f55c

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 614f55c: READY — my CHANGES is answered, and the arms are better than the three I asked for.

I asked for behavioural arms on payload_backend_missing because the row decides a release verdict with no arm of its own. What landed covers that and one layer more:

:53  FN_BK=$(awk '/payload_backend_missing() \{/…')   [ -n "$FN_BK" ]  -> the decision is a function
:55  grep -q 'payload_backend_missing "$PDIR"'         -> the row actually CALLS it
:97  a complete payload            -> names nothing
:102 payload with no backend       -> "names the executable and the library"
:108 binary absent, library present -> names only the executable     <- the arm I insisted on
     …and its mirror: library absent, binary present -> names only the library

The structural pair first (defined, called) is the one I did not think to ask for and should have: a function with no caller passes any behavioural arm you write about the function. And the two asymmetry arms are exactly the case the comment argues about — "the exec succeeds and the dlopen does not" — split in both directions, so an implementation that collapses the two conditions into one && fails.

Correction on my side, twice in five minutes. My "0 arms" came from grepping the --self-test block and tests/ at the previous head; the arms are in tests/release_check_payload_modes_test.sh and arrived with this head. And twenty minutes ago on #2490 I wrote "nothing implements the form limit" from a grep over app common tools while the limit is a CMakeLists.txt compile definition — both times the claim was true of the slice I looked at and false of the system. That is the same defect as grep -c 'error:' counting system_error: the instrument answered the question I typed.

Nothing else from me: the extended offender list is tested with its control, the absence row now has both halves, and the count ≠ property argument in the comment is the reason to keep the row even after payload-modes gained its floor.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 614f55c: READY

Both halves verified on fixtures I built by extracting the functions the way the test does — the mode row names a 0644 HRX backend, and the new backend row names its absence, and each stays silent on the other's case.

0644 1bit/hrx/llama/bin/llama-server, library present
    payload_modes_noexec  -> [ 1bit/hrx/llama/bin/llama-server]      named
    chmod 0755            -> [ ]                                     silent
payload with no backend at all
    payload_backend_missing -> " 1bit/hrx/llama/bin/llama-server 1bit/hrx/llama/bin/libggml-hrx.so*"

bash tests/release_check_payload_modes_test.sh os/mini/release-check.sh → rc 0, seven arms, and they are the seven I would have asked for: a 0644 engine and lemond are both named; a 0755 payload reports nothing; a 0644 HRX backend is named even when the three named binaries are 0755 (the case that produced this PR); a complete HRX backend reports nothing missing; a payload with no backend names the executable and the library; an executable-only gap names the executable and not the library it does have; and absent binaries report nothing, because absence is another row's claim.

That last one is the discipline that makes the pair work: mode owns can it execute, backend owns is it here, and neither answers the other's question. Two rows that both fired on absence would make every fresh payload look like two failures.

The provenance is the strongest part of the PR: the row had just been fixed to require the other three binaries and was silent about the fourth — the one serve spawns per model — while a real demo failed with Failed to execute: … Permission denied and an HTTP 500 in one second. A check that was just widened, missing the one member that mattered, is the most useful kind of bug report: it is evidence about how the widening was done, not just about the payload.

And the test's own note is worth keeping: the first version ran release-check.sh --check three times (34 s here) and was replaced by extracting the functions and testing them, the same shape release_check_hrx_init_reason_test.sh uses — plus one grep that pins the wiring (payload_backend_missing "$PDIR" must be called). Function extracted for the arms, grep for the call site: that is exactly the split that catches both "the logic is wrong" and "the logic is right and unused".

One note: payload_backend_missing reports libggml-hrx.so* as a glob while the executable is reported by exact path — a reader sees two different kinds of name in one message. Either is fine; saying which in the row's output ("executable … / library …") would make the next operator's ls obvious.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 614f55c — and I ran its two predicates against real artifacts rather than reading them, because a row that greps the wrong key would skip forever and still look green.

The catalogue predicate is right, measured on a real payload

$ grep -oE '"(recipe|backend|engine|kind)": *"[^"]*"' …/payload-c19/…/bin/resources/server_models.json | sort | uniq -c
  28 "recipe": "llamacpp-hrx"
  27 "recipe": "onebit"
  93 "recipe": "llamacpp"        (and 79 ryzenai-llm, 13 thenoise, 12 sd-cpp, …)
$ grep -qE '"recipe"[[:space:]]*:[[:space:]]*"(onebit|llamacpp-hrx)"' → MATCH

So recipe is the key the real catalog uses, and 55 of its entries put this row in play. If it had been backend/engine, the row would have been dead code on every appliance payload and the test — built on a synthetic catalog — would still have been green. That was the check worth making.

And a real payload passes all four pieces, so the row is not a permanent red

1bit/hrx/llama/bin/llama-server    -rwxrwxr-x  5304 B   (exec)
1bit/hrx/llama/bin/libggml-hrx.so, .so.0            (present)
1bit/1bit / bin/lemond / lemond-onebit              (exec)

The failing rehearsal artifact was the same 5 KB launcher class with mode -rw-r--r--, so the row now names the file that actually failed. Two-step check (llama-server and libggml-hrx.so*) is the right shape for the reason given: "the exec succeeds and the dlopen of libggml-hrx.so does not" — same failure one step later.

One ask: apply that same argument one more step, to the exec-time libraries

llama-server here is an ELF launcher, and its DT_NEEDED includes payload-shipped libraries that are not system libs:

libllama-server-impl.so     libllama-common.so.0     libmtmd.so.0
libllama.so.0               libggml.so.0             (+ libggml-hrx.so, loaded later)

The bin dir ships 27 .so* files. If libllama-server-impl.so (or libllama.so.0, libmtmd.so.0) is absent, ld.so fails before main, and lemond prints the same Failed to execute: <payload>/1bit/hrx/llama/bin/llama-server it printed for permission denied (os error 13). So the user-visible symptom is identical, and the row would still pass, because it checks the dlopen-later library and not the exec-time ones.

That is the PR's own argument applied one step earlier, and it is cheap: ldd "$B" on a payload, restricted to lines resolving inside the payload dir, gives the exact set — which also avoids hardcoding a library list that drifts with the llama.cpp version. (A hardcoded list would be the "reader narrower than the artifact" failure: 27 shipped .sos today, more or fewer tomorrow.)

One prose note

llama-server at 5304 B is a launcher, and a reader who opens the row expecting "the backend binary" will wonder what a 5 KB file can be. One clause — "llama-server in a payload is a small launcher that resolves its implementation through libllama-server-impl.so and its compute library through libggml-hrx.so" — makes the two-piece check obviously right to someone who has never unpacked a payload.

The payload-modes message shape being reused, and absence staying with the identity/hrx-device rows, both keep the row honest. The best sentence in the body is the diagnosis: the row that existed to catch this defect said nothing about the file that caused it — a count floor (94 entries) cannot see one wrong mode bit inside it.

@bong-water-water-bong
bong-water-water-bong merged commit b319f87 into main Nov 15, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the lane/payload-backends branch November 15, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant