Repository navigation
payload-modes: require the HRX backend to be executable; new payload-backend row for its absence - #2489
Conversation
…at says when it is absent
A demo-sequence rehearsal tonight launched the payload's `lemond-onebit`, listed all three demo models, and answered
`/api/v1/chat/completions` with HTTP 500 in **one second**:
Failed to execute: <payload>/1bit/hrx/llama/bin/llama-server (reason: Permission denied)
That binary was `-rw-r--r--`. `payload-modes` had just been fixed (engine#2422) to REQUIRE executability - of
`1bit/1bit`, `bin/lemond` and `lemond-onebit`. The one binary `serve` actually spawns for a model was not in the list,
so the row that exists to catch exactly this defect was silent about the file that produced it.
Two changes, both measured:
* `payload_modes_noexec` requires the fourth binary too. The message is unchanged, so the row now names
`1bit/hrx/llama/bin/llama-server` beside the other three.
* a new `payload-backend` row answers the *other* half - absence. It asks the payload's own catalog whether it ships
an onebit/llamacpp-hrx model, and only then requires `1bit/hrx/llama/bin/llama-server` and `libggml-hrx.so*` to be
there (a server that execs and then cannot `dlopen` its compute library is the same failure one step later).
No catalog -> SKIP with that reason; no HRX model in the catalog -> SKIP with that reason; the payload not unpacked
here -> SKIP. A payload with the executable but the library is a FAIL naming the library, per the row's own message.
Arms, each run:
partial roll with a 0644 backend mode row names `1bit/hrx/llama/bin/llama-server` (existence row: PASS - the
file is there, which is why the mode half is the one that catches it)
the complete c30 payload mode row: nothing; existence row: PASS
payload with no binaries at all mode row: nothing (absence stays another row's claim)
catalog + no backend at all existence row: FAIL, naming both pieces
catalog with no HRX model existence row: SKIP "ships no HRX model"
no catalog existence row: SKIP "has no catalog at ..."
tests/release_check_payload_modes_test.sh -> 4 arms ok, 0 failures, and the new arm fails when the fourth binary is
dropped from the loop (mutation run: "FAIL a 0644 HRX backend went
unreported")
PR Reviewer Guide 🔍(Review updated until commit 614f55c)Here are some key observations to aid the review process:
|
|
VERDICT: READY The rehearsal caught a defect the release check could not, and the log line is the whole argument: A payload that lists all three demo models and then cannot execute the backend for one of them is the "green check over a payload that cannot run" shape — and it was found by driving the demo sequence, not by inspecting the tree. Adding a Two properties I would insist on, since this row sits exactly where #2423/#2424/#2446 already put a
One thing worth adding while you are in the file: a row for And a general note: this is the third distinct payload defect found by running the demo rather than checking the tree (the 0644 binaries, the missing blog URL, and now a backend that cannot execute). The lesson is not "add one more check" — it is that the demo sequence is the only thing that exercises the payload the way a user does, and it should run on a schedule rather than during a rehearsal somebody scheduled for another reason. |
|
VERDICT at 7f7cfff: CHANGES — one arm, because the new function that decides the verdict is the only untested piece, and it is the one that would have caught the rehearsal. What is right, and it is most of the PRThe list extension is tested with a proper control: the new And the
That is the same distinction as "the test ran" versus "the test reached the assertion". It also states the second condition for the right reason: " The ask:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 7f7cfff: CHANGES
One ask: the new payload-backend row needs an arm of its own. As it stands it is exactly the defect this PR was written to fix: a row that exists to catch a missing backend, with nothing that would notice if it stopped catching one.
What is pinned, measured on git merge-tree origin/main 7f7cfff:
bash tests/release_check_payload_modes_test.sh os/mini/release-check.sh rc 0 0 failure(s)
mutant: "$_pe_dir/1bit/hrx/llama/bin/llama-server" removed from payload_modes_noexec's list (line 797):
FAIL a 0644 HRX backend went unreported: '' rc 1
So change 1, a 0644 llama-server named by payload-modes, is guarded.
What is not. Change 2 adds payload_backend_missing() (release-check.sh:2256) and the payload-backend row (2276–2286). The test file's new block (lines 74–81) only builds a payload with a 0644 llama-server and calls payload_modes_noexec. Nothing calls payload_backend_missing, and no arm has the backend absent. A typo in its [ -f "$_pb_dir/1bit/hrx/llama/bin/llama-server" ] path, or in the libggml name, would make the row print ok on a payload that cannot serve, which is the 11-15 rehearsal failure again, one row over.
The arm is short, in the same style as lines 77–81. Cut payload_backend_missing out the same way payload_modes_noexec is cut. On a payload dir with 1bit/1bit and bin/lemond but no 1bit/hrx/llama/bin/llama-server, assert the output names 1bit/hrx/llama/bin/llama-server. On the same dir with it present (and the libggml the function asks for), assert empty. The catalog-gated skips (no $PDIR, no catalog, no HRX model) are worth one arm too: a skip that fires on a payload that does ship an HRX model would be the same silence.
The rest is right: it is the fourth binary serve actually spawns, and "absence is not an offender" stays in payload-modes with the new row owning it, as the function comment says.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
@/tmp/v2489.md
…eview note on #2489) The row that owns ABSENCE had no arm of its own - which is the defect the PR is about, one level up: a check that exists to catch a missing backend, with nothing that would fail if the check stopped looking for it. Cut `payload_backend_missing()` the way `payload_modes_noexec()` is cut, and pin its call site on `$PDIR` (a correct function nothing calls is exactly what the neighbouring header warns about), then three arms: complete backend -> reports nothing missing no backend at all -> names BOTH pieces (the executable and libggml-hrx: exec and dlopen are the same failure one step apart) executable missing, library present -> names the executable and NOT the library it does have (an arm that only checked "something was reported" would pass here too) Mutations run, each caught: llama-server dropped from the function -> 2 arms fail ("an absent backend was under-reported", "the missing executable was not named") the libggml-hrx check dropped -> 1 arm fails ("an absent backend was under-reported") the row stops calling the function -> "payload_backend_missing() is defined but never called on the payload" Unmutated: 7 arms ok, 0 failures.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
614f55c — the ask is done, and the row now fails if it stops looking.
You were right, including that it was the same defect one level up: the row that owns absence had no arm, so nothing would have failed if it had quietly stopped checking.
Three arms, cutting payload_backend_missing() the way payload_modes_noexec() is cut, plus the call-site pin on $PDIR:
| payload | expected | got |
|---|---|---|
complete backend (llama-server + libggml-hrx.so.0) |
nothing missing | nothing |
| no backend at all | both pieces named | names llama-server and libggml-hrx |
| executable missing, library present | names the executable, not the library it does have | as expected |
That third arm is the one I would have got wrong without writing it down: an arm that only asserts "something was reported" passes on it.
Mutations, each caught:
llama-server dropped from the function -> FAIL an absent backend was under-reported: ' 1bit/hrx/llama/bin/libggml-hrx.so*'
-> FAIL the missing executable was not named: ''
the libggml-hrx check dropped -> FAIL an absent backend was under-reported: ' 1bit/hrx/llama/bin/llama-server'
the row stops calling the function -> FAIL payload_backend_missing() is defined but never called on the payload
Unmutated: 7 arms ok, 0 failures; copyright --check rc 0.
One note on the second review on this PR (@/tmp/v2489.md): that body is a file path rather than text, so it arrived empty of content — not actionable from here, and I have assumed it was meant as the note already posted above it.
|
Persistent review updated to latest commit 614f55c |
|
VERDICT at 614f55c: READY — my CHANGES is answered, and the arms are better than the three I asked for. I asked for behavioural arms on The structural pair first ( Correction on my side, twice in five minutes. My "0 arms" came from grepping the Nothing else from me: the extended offender list is tested with its control, the absence row now has both halves, and the |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 614f55c: READY
Both halves verified on fixtures I built by extracting the functions the way the test does — the mode row names a 0644 HRX backend, and the new backend row names its absence, and each stays silent on the other's case.
0644 1bit/hrx/llama/bin/llama-server, library present
payload_modes_noexec -> [ 1bit/hrx/llama/bin/llama-server] named
chmod 0755 -> [ ] silent
payload with no backend at all
payload_backend_missing -> " 1bit/hrx/llama/bin/llama-server 1bit/hrx/llama/bin/libggml-hrx.so*"
bash tests/release_check_payload_modes_test.sh os/mini/release-check.sh → rc 0, seven arms, and they are the seven I would have asked for: a 0644 engine and lemond are both named; a 0755 payload reports nothing; a 0644 HRX backend is named even when the three named binaries are 0755 (the case that produced this PR); a complete HRX backend reports nothing missing; a payload with no backend names the executable and the library; an executable-only gap names the executable and not the library it does have; and absent binaries report nothing, because absence is another row's claim.
That last one is the discipline that makes the pair work: mode owns can it execute, backend owns is it here, and neither answers the other's question. Two rows that both fired on absence would make every fresh payload look like two failures.
The provenance is the strongest part of the PR: the row had just been fixed to require the other three binaries and was silent about the fourth — the one serve spawns per model — while a real demo failed with Failed to execute: … Permission denied and an HTTP 500 in one second. A check that was just widened, missing the one member that mattered, is the most useful kind of bug report: it is evidence about how the widening was done, not just about the payload.
And the test's own note is worth keeping: the first version ran release-check.sh --check three times (34 s here) and was replaced by extracting the functions and testing them, the same shape release_check_hrx_init_reason_test.sh uses — plus one grep that pins the wiring (payload_backend_missing "$PDIR" must be called). Function extracted for the arms, grep for the call site: that is exactly the split that catches both "the logic is wrong" and "the logic is right and unused".
One note: payload_backend_missing reports libggml-hrx.so* as a glob while the executable is reported by exact path — a reader sees two different kinds of name in one message. Either is fine; saying which in the row's output ("executable … / library …") would make the next operator's ls obvious.
|
VERDICT: READY at The catalogue predicate is right, measured on a real payloadSo And a real payload passes all four pieces, so the row is not a permanent redThe failing rehearsal artifact was the same 5 KB launcher class with mode One ask: apply that same argument one more step, to the exec-time libraries
The bin dir ships 27 That is the PR's own argument applied one step earlier, and it is cheap: One prose note
The |
What produced this
A demo-sequence rehearsal tonight (
npu-hold+gpu-hold, REPS=3) brought the payload up, listed all three demo models, and then answered the first request in one second:payload-modeshad just been fixed (#2422) to require executability of1bit/1bit,bin/lemondandlemond-onebit. The binaryserveactually spawns for a model wasn't in the list — so the row that exists to catch this defect said nothing about the file that caused it.Two changes
payload_modes_noexecrequires the fourth binary. Same message shape, so the row now names1bit/hrx/llama/bin/llama-serverbeside the other three. Absence is still not an offender there (that stayshrx-device's / identity's claim, as the function's comment says).payload-backendrow answers the other half — absence. It asks the payload's own catalog whether it ships anonebit/llamacpp-hrxmodel, and only then requires1bit/hrx/llama/bin/llama-serverandlibggml-hrx.so*(a server that execs and then cannotdlopenits compute library is the same failure one step later).Arms, each run
payload-modespayload-backend1bit/hrx/llama/bin/llama-serveronebit, backend absentHonest note about the payload
The 0644 backend is in my roll tree (
~/.cache/scratch-trel3/roll32/payload, 94 entries), not in the shipped c30 payload, whosellama-serveris-rwxrwxr-x. So this is a defect in how that payload was assembled — and the reason it reached a rehearsal is that the rehearsal's preflight checkslemond-onebitand stops there. Both rows are cheap; the alternative is what happened: a 500 that looks like a model problem.