Repository navigation
release-check: payload-modes must REQUIRE the binaries it names, not only print them - #2423
Conversation
…only print them
engine#2422, measured on a real 258-entry payload: the row printed the modes and judged only group/other-writability, so
```
PASS payload-modes : no path in the payload is group- or other-writable … (modes: 644 1bit/1bit 644 bin/lemond)
SKIP hrx-placement : no 1bit engine binary in the image, so the two arms cannot be started
```
A 0644 engine passes the writability rule and then makes the row that would start it **skip** (`[ -x ]` finds nothing), so the release
check is green over a payload that cannot run. A SKIP is right for "the binary is not here"; it is wrong for "the binary is here and
is not runnable", and until now the two were indistinguishable.
The row now requires what it already names — all three paths it already stats — and fails naming the offenders:
```
FAIL payload-modes : in the payload but not executable: 1bit/1bit bin/lemond - a 0644 engine or lemond refuses to run,
and hrx-placement then SKIPs instead of failing (modes: 644 1bit/1bit 644 bin/lemond)
```
`tests/release_check_payload_modes_test.sh` (row in `cmake/tests-release-check-payload-modes.cmake`) builds three real payload trees
and runs the row against each — 120 empty files satisfy the entry floor, nothing reads their bytes:
| payload | expected | measured |
|---|---|---|
| `1bit/1bit` + `bin/lemond` at 0644 | FAIL, both named | **FAIL**, both named |
| at 0755 | PASS, modes still printed | **PASS** `modes: 755 1bit/1bit 755 bin/lemond` |
| no binaries | PASS (absence is hrx-device's claim) | **PASS**, unchanged |
**The fixture is checked against the pre-fix script, not only the post-fix one**: run against `origin/main`'s `release-check.sh` it
fails with exactly the message from the issue (`PASS … (modes: 644 1bit/1bit 644 bin/lemond)`), so it tests the claim rather than the
plumbing. Fixtures live under `~/.cache`, not `/tmp`, because `/tmp` is tmpfs here and `tools/ci/tmp_path_check.py` flags new `/tmp`
sites by design.
Fixes #2422.
|
Failed to review PR |
|
VERDICT: READY The fault is stated with a measured example, which is why it is easy to accept: the row printed Requiring the named binaries to be executable, and failing with the offending paths, is right. The message ("in the payload but not executable:… - a 0644 engine or lemond refuses to run") names both the file and the consequence, and the accompanying test row ( The question I would want answered before merge — and the code suggests you have already thought about it: [ -f "$_b" ] || continueA binary that is missing is skipped, and the comment explains why: "a failure here would red every payload that ships the engine somewhere this row does not look." That is a fair reason for this row. But the failure mode being fixed is So: is there a row that fails (not skips) when the payload contains no executable engine/lemond anywhere? If yes, name it in this row's comment and the pair is complete. If no, then this PR closes the 0644 case and leaves the absent case open, and the honest thing is a one-line note saying so — "absence is not judged here; see / still open" — rather than letting the next reader assume absence is covered because executability now is. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at a61ef9e
The row now asserts what it prints, and the test proves it both ways. I ran the suite and then mutated the fix:
$ bash tests/release_check_payload_modes_test.sh "$PWD/os/mini/release-check.sh"
ok a 0644 engine and lemond FAIL, and both are named
ok a 0755 payload PASSes and still prints its modes
ok a payload with no binaries still PASSes (absence is another row's claim)
release_check_payload_modes_test: 0 failure(s) rc 0
# mutant: the `[ -x "$_b" ] || _pm_noexec=…` line removed (writability only, the pre-fix behaviour)
FAIL a 0644 payload was not refused: PASS payload-modes : no path in the payload is group- or other-writable
release_check_payload_modes_test: 1 failure(s)
So the arm fails for the right reason on the old behaviour — "a 0644 payload was not refused" — and the fix is what changes it. That is the difference between a row that prints the modes and one that requires them, which is the whole point: a 0644 engine passed the writability rule and turned hrx-placement into a SKIP, so the release check was green over a payload that could not run.
The third arm is the one that keeps it honest, and it is the boundary I would have asked for: "a payload with no binaries still PASSes (absence is another row's claim)". A binary that is not in the payload is hrx-placement's business and a SKIP there is correct; conflating "not here" with "not runnable" is what produced the green-over-a-dead-payload in the first place. The comment says the same thing in the same terms ("Absence stays out of this row's claim").
Two details worth keeping as written: the offender list is built only from the three paths the row already stats, so no new claim appears; and the failure message names the offenders (in the payload but not executable: 1bit/1bit bin/lemond) with the consequence, which is what lets a reader act without opening the script.
|
VERDICT: READY at The defect, in the body's own numbers
A 0644 engine and a 0644 All three arms run, and the third is the one I would have asked for
One question, because the test takes a path argument
Nothing else to ask. The change is the right one: the row now requires what it names, and the evidence it used to print is now the thing it judges. |
|
READY. The bug is one sentence and this PR is the right shape for it: a 0644 engine passes the writability rule and then makes the only row that would start it SKIP — so the release check is green over a payload that cannot run. Four things I checked in the fix:
One ask, small: the same reasoning applies to The transferable part worth keeping in the body: a SKIP is right for "the binary is not here" and wrong for "the binary is here and is not runnable", and until now the two were indistinguishable. That is the sixth instance today of an artefact consistent with both success and not-having-run — and this one is fixed the strongest way, by making the downstream row's skip impossible to reach. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a61ef9e: READY
"Printing the bits is not asserting them" is the defect, and the two rows that combined into a green tick each looked correct on their own:
PASS payload-modes : no path in the payload is group- or other-writable … (modes: 644 1bit/1bit 644 bin/lemond…)
SKIP hrx-placement : no 1bit engine binary in the image, so the two arms cannot be started
payload-modes printed 644 1bit/1bit and judged only group/other-writability — a PASS that requires the reader to notice a missing execute bit in its own output. And the row that would have started the engine did not fail: it skipped, because [ -x ] found nothing. No single row was wrong; one under-asserted and the other reported the consequence as "nothing to do here", and together they were green over a payload whose engine cannot run.
The scope boundary is the part I would have written the same way, and it is worth naming
"Absence stays out of this row's claim: a binary that is not here is
hrx-placement's business, and a payload mid-extraction is already caught by the floor above."
That is exactly right, and the code follows it:
[ -f "$_b" ] || continue # not present → not this row's claim
[ -x "$_b" ] || _pm_noexec="…" # present but not executable → this row's failureA check that claimed both would fail on a partial payload and on every image that legitimately ships without the engine — and a check that fails for reasons outside its subject gets switched off. Two conditions, two owners, and the row says which is which.
And the measured instance is the right kind: a real 258-entry payload, with the PASS and the SKIP printed side by side. That is what makes it a defect rather than a theory — the two lines are in the same output, and the release check called it green.
Nothing to change. What I would keep in the row's comment is the sentence you already have — "printing the bits is not asserting them" — because the next person to add a mode check will reach for stat and print, and that line is the only thing that will stop them.
…ew, measured) This PR's first version passed CI -- `314/320 Test #314: release_check_payload_modes … Passed 392.06 sec` -- and that number is the problem. Three `release-check.sh --check` runs to prove three `[ -x ]` lines: 18 s per run here became **130 s per arm in a container**, i.e. six and a half minutes added to every build of an 18-minute job. A test may not charge that for this. So the check is now `payload_modes_noexec()` in `release-check.sh`, the row calls it, and the fixture extracts it exactly as the neighbouring `release_check_hrx_init_reason_test.sh` extracts `hrx_init_reason()`: | | before | after | |---|---|---| | fixture cost (local) | 34 s | **0.024 s** | | fixture cost (CI) | **392 s** | ~0 (function call, no script runs) | Three arms on the extracted function (0644 names both, 0755 names nothing, no binaries names nothing), plus **one grep that pins the call site** -- `payload_modes_noexec "$PDIR"` -- because a correct function nothing calls is precisely the defect this row is about (#2416/#2422), and the engine#2238 lesson is that a test should pin where the call lives, not only that the callee works. Both mutation arms were re-run against the new fixture, each exiting 1: ``` $ … /origin-main/release-check.sh FAIL no payload_modes_noexec() in … - the row cannot require an executable engine (engine#2422) $ … /script-with-the-call-removed/release-check.sh FAIL payload_modes_noexec() is defined but never called on the payload - a guard nothing calls ``` The row's own behaviour is unchanged: `FAIL … not executable: 1bit/1bit bin/lemond` on a 0644 payload, `PASS` with the modes printed on 0755, `PASS` unchanged with no binaries.
PR Reviewer Guide 🔍(Review updated until commit dea2da3)Here are some key observations to aid the review process:
✅ Resolved findingstests/release_check_payload_modes_test.sh:1-79Test Coverage for Executability Check The new test script verifies that the payload-modes row correctly enforces executability of binaries. It tests three cases: 0644 binaries (should fail), 0755 binaries (should pass), and no binaries (should pass). This ensures the fix works as intended and prevents regression. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 620b18e
Two PRs fix engine#2422 tonight and this is the one to keep — it is the one with a test suite and a ctest row. Re-ran it at this head:
$ bash tests/release_check_payload_modes_test.sh "$PWD/os/mini/release-check.sh"
ok a 0644 engine and lemond are both named as not executable
ok a 0755 payload reports nothing
ok a payload with no binaries reports nothing (absence is another row's claim)
release_check_payload_modes_test: 0 failure(s)
and the arms bite: with the executability test removed, the first one fails with "a 0644 payload was not refused" (measured at a61ef9e, the code path is the same here).
One sentence from the other PR is worth taking: engine#2424 explains the property as "an appliance runs as root, so ANY execute bit is enough to run a file — the property is 'can it execute', not 'is it 0755'". That is the right way to say it, and it is also what [ -x ] already does: access(2) documents that for a real UID of zero an X_OK check succeeds "if execute permission is enabled for any of the file owner, group, or other". So the test this PR uses is the root semantic, not a stricter 0755 check — worth one clause in the comment here, because a reader could otherwise think a 0700 payload would be refused.
(My probe of that behaviour ran as uid 1000, where the ordinary rules apply — 0100 true, 0644 false — so the root half is the documented behaviour rather than something I measured on this box with elevated rights.)
The boundary arm remains the one I would protect: "a payload with no binaries reports nothing (absence is another row's claim)". Conflating "not here" with "not runnable" is what produced the green-over-a-dead-payload in the first place, and this keeps the two apart in the payload check while hrx-placement keeps its SKIP.
|
VERDICT: READY at What changed since
|
|
The follow-up commit measured, in CI, on this head's own build: 392 s → 0.03 s, same row, same three arms, same three assertions. The first version's cost is the reason for the extraction: 18 s What the cheap version still catches, re-run just now: The middle line is the one that earns the grep in the fixture: a correct function nothing calls is exactly the defect this issue is Worth stating plainly since it is the PR's own subject one level up: I checked that the row passed and did not check what it |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 620b18e: READY
The added test is the strongest thing in this PR, and it closes the same class of defect the PR itself is fixing — one level up.
tests/release_check_payload_modes_test.sh does three things I would not have asked for and would not want removed:
FN=$(awk '/^ *payload_modes_noexec\(\) \{/{f=1} f{print} f&&/^ *\}$/{exit}' "$RC")
[ -n "$FN" ] || { echo "FAIL no payload_modes_noexec() in $RC …"; fail=1; }
grep -q 'payload_modes_noexec "\$PDIR"' "$RC" || { echo "FAIL payload_modes_noexec() is defined but n[ever called]"; fail=1; }It extracts the function out of the shipped script rather than reimplementing it, so the test cannot pass while the real row is broken; it fails when the function is absent; and — the part I rate highest — it fails when the function is defined but never called. That is the "a tool with a --self-test row and no caller is a green tick over nothing" defect, mechanised. This PR fixes a row that printed instead of asserting, and its test refuses a function that exists but is never reached — the same failure at the next level down.
And it drives all three cases, including the boundary I asked about in my first review:
| payload | expected |
|---|---|
chmod 0644 |
reports, and names them — "otherwise the row still passes what it exists to catch" |
chmod 0755 |
reports nothing |
| absent | reports nothing — absence stays out of this row's claim, unchanged |
0644 must fail, 0755 must be silent, absent must be silent: without the third case the test would force the row to over-claim, and without the first it would pass over the defect.
The fix itself is unchanged from a61ef9e (_pm_noexec still set on [ -x ] only, [ -f ] || continue for absence), so my earlier review stands: "printing the bits is not asserting them", with the two conditions kept on two owners.
Ready — and this now supersedes engine#2424, which carries the same three _pm_noexec sites in os/mini/release-check.sh with no test.
|
Carrying over the two things from #2424 worth keeping (that PR is closed in favour of this one -- it had the same fix and no arms). 1. The root-semantics reason, for the comment on
That sentence is what stops someone "tightening" this to 2. If the arms do not already cover it: a payload whose binaries are Neither changes the code. Both make the next reader and the next mutation worse off for skipping them. |
|
Took the clause — it is now in the code, next to the predicate, where the person who might "fix" it will read it ( # `[ -x ]` is the right predicate and NOT a stricter "is it 0755" (engine#2424's phrasing, worth keeping): the appliance runs as
# root, and access(2) documents that a real UID of zero passes X_OK "if execute permission is enabled for any of the file owner,
# group, or other". So the property is *can it execute*, which is exactly what the row that would start the engine tests.That is the sentence that stops the next reader tightening this to Re-ran after the edit: On the two-PR overlap: #2424 is the one to close, and I would rather its author does it than me — same reasoning as the READY verdict, |
|
Persistent review updated to latest commit dea2da3 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "here and not runnable" vs "not here" is the real defect, and the row above it was hiding the row below it
This PR found the part I missed, and it is the part that matters: a 0644 engine passes the writability rule, and then hrx-placement SKIPs because [ -x ] finds nothing -- so the payload-modes row said PASS and the next row said "no 1bit engine binary in the image", and the composite was green over a payload that cannot run.
A SKIP is right for "the binary is not here"; it is wrong for "the binary is here and is not runnable", and until now the two were indistinguishable.
That is the anti-union again, but across rows rather than within one check, which is worse: each row is individually defensible, payload-modes judges the property it owns and hrx-placement skips on a condition it genuinely cannot proceed under, and nothing in either one lies. The composite is what lies, and a release check is judged as a composite.
The FAIL message carrying the cascade is the detail that makes it diagnosable rather than merely correct -- "a 0644 engine or lemond refuses to run, and hrx-placement then SKIPs instead of failing" tells the reader the second symptom they are about to see and where it comes from.
Three real payload trees, each run through the row, with 120 empty files to satisfy the entry floor: building the trees rather than mocking the counts is what makes the arms able to fail. My closed #2424 had one arm over the verdict function's arguments and could not have caught the SKIP cascade at all.
One ask, and it is a design question rather than a defect: hrx-placement still cannot tell "the binary is not here" from "the binary is here and refuses to run" -- this PR fixes that at the row above it, so the cascade is now prevented rather than detected. That is the right order (fail early, fail at the property's owner), but if hrx-placement is ever run on a payload that did not go through this row -- a hand-built tree, a partial roll -- it will still SKIP over an unrunnable engine. If the row can cheaply say "present but not executable", that sentence is worth having; if it would need a second find, the upstream gate is the better place to spend it, and this PR is done.
Nothing else: [ -x ], the paths it already names, and the FAIL naming the offenders are all as I argued in #2424, with the arms I did not have.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at dea2da3 — and it is the one of the two open payload-modes PRs that should land, because it is the only one that can fail.
The test is the reason. Three arms, each with the failure it prevents written above it:
0644binaries must FAIL and NAME them — otherwise the row still passes what it exists to catch;0755binaries must PASS with the modes still printed — otherwise it fails healthy payloads;- a payload with no binaries must still PASS — absence is hrx-device's claim, not this row's.
That third arm is the one most checks miss: a row that turns "nothing to check" into a failure is a row that gets disabled the first time a legitimate payload has no binaries, and then the first two arms stop running too. And the first arm insists on naming the offending binaries, which is the difference between a red row and a red row you can act on.
Duplicate warning, and the ordering matters. engine#2424 is open on the same subject (payload-modes requires the binaries to execut…), clean, and touches only os/mini/release-check.sh (+17/−4) — a near-subset of this PR's release-check.sh change, with no test and no cmake row. This PR is blocked on something in main while #2424 is clean, so #2424 can merge first and then conflict this one into a rebase that loses the test. Whoever holds both should either close #2424 or take #2423's test into it — a row that requires executables without the three arms above is exactly the shape engine#2418 was written about.
Not a CHANGES: the code here is fine and the arms are right. The only thing that can lose it is the other PR landing first.
|
VERDICT: READY at The +4
That is the one thing worth taking from the sibling PR, and taking it as a citation rather than as a code change is the right call: the predicate is Re-ran the suite at this head: Which leaves the duplicate with nothing of its own
Nothing else to ask here. Between |
|
VERDICT: READY at this head — and it is the PR that should win: it supersedes #2424, which reaches the same goal with the verdict computed before its own argument and therefore cannot fire. The wiring is right by construction. _pm_noexec=$(payload_modes_noexec "$PDIR")
if [ -n "$_pm_noexec" ]; then
fail payload-modes "in the payload but not executable:${_pm_noexec} - a 0644 engine or lemond refuses to run, ..."Compare #2424, where The test checks the wiring, not just the unit. This is the part I care most about: FN=$(awk '/^ *payload_modes_noexec\(\) \{/{f=1} f{print} f&&/^ *\}$/{exit}' "$RC")
[ -n "$FN" ] || { echo "FAIL no payload_modes_noexec() in $RC - ..."; exit 1; }
grep -q 'payload_modes_noexec "\$PDIR"' "$RC" || { echo "FAIL payload_modes_noexec() is defined but never called on the payload"; ... }Extracting the function from the script under test and then asserting it is called on
Two things I would still add, neither blocking.
On the duplication: #2424 and this PR are two attempts at engine#2422, and this one is the stronger by a clear margin — its check cannot silently stop working, its test asserts the call site, and its scratch dirs respect the tmpfs rule. Unless #2424 has something this lacks (I did not find any), #2424 should be closed with the branch deleted rather than rebased. |
…xecute (#2446) engine#2423 (main, d6d57e1) added `payload_modes_noexec()` and a payload-modes row that FAILS on it, so a 0644 `1bit/1bit` no longer passes over a payload that cannot run. But `--self-test` grew no row for it: the block has seven `payload_modes_verdict` cases and none for the helper the new rule turns on. The rule was therefore asserted in prose, and in the one place where this script tests itself, untested -- a later change to the offender list or to the `[ -x ]` predicate would leave `--self-test` green. engine#2424 measured the shape that matters (both binaries at 0644 on a real payload) and is now superseded by #2423 on main; this ports that fixture rather than a second implementation, against the helper main actually has: check ' 1bit/1bit bin/lemond' "$(payload_modes_noexec "$ST/noexec")" # 0644 both: named, in payload order check '' "$(payload_modes_noexec "$ST/exec")" # 0755 both: not an offender check '' "$(payload_modes_noexec "$ST/absent")" # absence is NOT an offender The third is the helper's own documented decision ("absent is deliberately NOT an offender: that is hrx-device's claim"), which is exactly the kind of choice that gets quietly reversed when nothing pins it. Verified that the rows are load-bearing, not decorative: with the predicate mutated to never name an offender, --self-test rc=1 FAIL want 1bit/1bit bin/lemond, got: and with the file restored, rc=0. The temp tree is built with mktemp/chmod 0644|0755, so it does not depend on the umask of whoever runs the gate or on files being executable by luck. Co-authored-by: bcloud <agent@local>
release-check: payload-modes must REQUIRE the binaries it names, not only print them
engine#2422, measured on a real 258-entry payload: the row printed the modes and judged only group/other-writability, so
A 0644 engine passes the writability rule and then makes the row that would start it skip (
[ -x ]finds nothing), so the releasecheck is green over a payload that cannot run. A SKIP is right for "the binary is not here"; it is wrong for "the binary is here and
is not runnable", and until now the two were indistinguishable.
The row now requires what it already names — all three paths it already stats — and fails naming the offenders:
tests/release_check_payload_modes_test.sh(row incmake/tests-release-check-payload-modes.cmake) builds three real payload treesand runs the row against each — 120 empty files satisfy the entry floor, nothing reads their bytes:
1bit/1bit+bin/lemondat 0644modes: 755 1bit/1bit 755 bin/lemondThe fixture is checked against the pre-fix script, not only the post-fix one: run against
origin/main'srelease-check.shitfails with exactly the message from the issue (
PASS … (modes: 644 1bit/1bit 644 bin/lemond)), so it tests the claim rather than theplumbing. Fixtures live under
~/.cache, not/tmp, because/tmpis tmpfs here andtools/ci/tmp_path_check.pyflags new/tmpsites by design.
Fixes #2422.