Skip to content

release-check: test the helper that decides a payload binary cannot execute (#2423 added the rule and no self-test) - #2446

Merged
bong-water-water-bong merged 1 commit into
mainfrom
noexec-selftest
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
noexec-selftest

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

Three --self-test rows for payload_modes_noexec() — the helper that decides whether a payload's own binaries can execute.

Why

#2423 (main d6d57e11) did the real work: it added payload_modes_noexec() and made the payload-modes row fail on it, so a 0644 1bit/1bit no longer passes over a payload that cannot run. But the tool's own self-test grew no row for it:

$ git show origin/main:os/mini/release-check.sh | sed -n '/--self-test/,/^fi/p' | grep -c payload_modes_noexec
0
$ … | grep -c payload_modes_verdict
7   # every case is about the writability helper, none about the new rule

So the new rule was asserted in prose and — in the one place where this script tests itself — untested. A later change to the offender list, or to the [ -x ] predicate, would leave --self-test green while the gate went back to passing unrunnable payloads. That is the same hole as #2424 reported, one layer down.

What it ports

#2424 measured the shape that matters (both binaries at 0644 on a real 258-entry payload) and is superseded by #2423 on main. This ports that fixture, against the helper main actually has — not the PR's four-parameter variant — so there is one implementation and its test:

check ' 1bit/1bit bin/lemond' "$(payload_modes_noexec "$ST/noexec")"   # 0644 both: named, in payload order
check ''                      "$(payload_modes_noexec "$ST/exec")"     # 0755 both: not an offender
check ''                      "$(payload_modes_noexec "$ST/absent")"   # absence is NOT an offender

The third pins the helper's own documented decision — "absent is deliberately NOT an offender: a binary that is not here is hrx-device's or identity's claim" — which is exactly the kind of choice that gets quietly reversed when nothing holds it.

Load-bearing, measured

With the predicate mutated to never name an offender:

--self-test rc=1    FAIL want  1bit/1bit bin/lemond, got:

With the file restored: --self-test rc=0. The temp tree is mktemp + explicit chmod 0644|0755, so the result does not depend on the runner's umask or on a file being executable by luck.

Refs #2423 (the rule), #2424 (the measured fixture, superseded), engine#2422.

…xecute

engine#2423 (main, d6d57e1) added `payload_modes_noexec()` and a payload-modes row that FAILS on it, so a 0644
`1bit/1bit` no longer passes over a payload that cannot run. But `--self-test` grew no row for it: the block has
seven `payload_modes_verdict` cases and none for the helper the new rule turns on. The rule was therefore asserted
in prose, and in the one place where this script tests itself, untested -- a later change to the offender list or to
the `[ -x ]` predicate would leave `--self-test` green.

engine#2424 measured the shape that matters (both binaries at 0644 on a real payload) and is now superseded by
#2423 on main; this ports that fixture rather than a second implementation, against the helper main actually has:

  check ' 1bit/1bit bin/lemond' "$(payload_modes_noexec "$ST/noexec")"   # 0644 both: named, in payload order
  check ''                      "$(payload_modes_noexec "$ST/exec")"     # 0755 both: not an offender
  check ''                      "$(payload_modes_noexec "$ST/absent")"   # absence is NOT an offender

The third is the helper's own documented decision ("absent is deliberately NOT an offender: that is hrx-device's
claim"), which is exactly the kind of choice that gets quietly reversed when nothing pins it.

Verified that the rows are load-bearing, not decorative: with the predicate mutated to never name an offender,

  --self-test rc=1   FAIL want  1bit/1bit bin/lemond, got:

and with the file restored, rc=0. The temp tree is built with mktemp/chmod 0644|0755, so it does not depend on the
umask of whoever runs the gate or on files being executable by luck.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Test Coverage for Noexec Rule

The PR adds self-test cases for the payload_modes_noexec() helper function, which determines whether a payload's own binaries can execute. These tests ensure that payloads with binaries lacking execute permissions (0644) are correctly flagged as offenders, while executable payloads and absent binaries are not. This addresses a gap where the rule was previously untested, potentially leading to regressions if the logic changes.

# engine#2422 (the helper main kept from #2423): the row now FAILS on a payload that carries its own binaries
# without an execute bit, and nothing above exercised the helper that decides it -- so the rule was asserted in
# prose and untested where the tool tests itself. engine#2424 measured the shape: both binaries at 0644.
ST=$(mktemp -d)
mkdir -p "$ST/noexec/1bit" "$ST/noexec/bin" "$ST/exec/1bit" "$ST/exec/bin"
for _f in "$ST/noexec/1bit/1bit" "$ST/noexec/bin/lemond" "$ST/exec/1bit/1bit" "$ST/exec/bin/lemond"; do : > "$_f"; done
chmod 0644 "$ST/noexec/1bit/1bit" "$ST/noexec/bin/lemond"
chmod 0755 "$ST/exec/1bit/1bit" "$ST/exec/bin/lemond"
check ' 1bit/1bit bin/lemond' "$(payload_modes_noexec "$ST/noexec")"   # the measured 0644 shape: named, in payload order
check '' "$(payload_modes_noexec "$ST/exec")"                        # an executable payload is not an offender
check '' "$(payload_modes_noexec "$ST/absent")"                      # absence is deliberately NOT an offender (the helper's own note)
rm -rf "$ST"

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2446

This is exactly the layer I asked for when reviewing #2424: "the inline check FAIL/check PASS pair exercises the branch without needing a payload tree". The three arms here do it properly with real files and real modes (mktemp -d, chmod 0644 / 0755, then payload_modes_noexec on each tree) rather than mocking stat, which is the difference between testing the decision and testing a stub.

The comment on the first arm is the part that makes it a regression test: "the measured 0644 shape: named, …" — it encodes the case engine#2422 was filed about, so if someone "simplifies" the helper back to a writability check, this arm fails on the shape that actually shipped broken.

Two asks:

  1. assert the empty case too — a payload whose binaries are all executable must give an empty string, and a tree with no binaries must not be reported as "noexec" (they are different from "none found"). If payload_modes_noexec conflates them, that is a defect the third arm should catch.
  2. clean up the mktemp -d (trap or explicit rm -rf) — a self-test that leaves directories behind in /tmp will eventually trip the tmp_path_check this same repo runs, and it is cheaper to add the trap now than to debug that later.

Together with #2423's fixture (which builds whole payload trees and asserts the row's verdict) and this (which unit-tests the helper), the payload-modes logic now has both layers — the pairing I asked for when #2423/#2424 were competing.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY. This is the arm I asked for on #2423 and again on #2424 — and it tests the helper with three cases where I asked for two, because the third is the one that decides whether the rule can be trusted.

check ' 1bit/1bit bin/lemond' "$(payload_modes_noexec "$ST/noexec")"   # the measured 0644 shape
check '' "$(payload_modes_noexec "$ST/exec")"                          # an executable payload is not one
check '' "$(payload_modes_noexec "$ST/absent")"                        # absence is deliberately NOT one

Why the third matters most: the rule this helper implements is "present and not executable", and its nearest wrong implementations are "any binary missing" and "any binary not 0755". The exec arm rules out the second; the absent arm rules out the first — without it, a helper that returned every path it could not stat would pass the other two and then fail a legitimate partial payload. That is precisely the distinction the original sentence in the row was making in prose, and the comment says so: "the rule was asserted in prose and untested where the tool tests itself."

And the fixtures are built rather than assumed: real files at chmod 0644 and chmod 0755 in a mktemp -d that is removed afterwards — so this exercises the actual [ -x ] behaviour on the running filesystem, not a stubbed predicate. It also means the test is honest about the one thing that could differ between hosts (a filesystem mounted noexec, or root semantics), because it measures the same predicate the row uses.

One small thing I would add if it is cheap: the noexec arm asserts the exact string ' 1bit/1bit bin/lemond', which pins order and leading space as well as content. That is good — a helper that returned them in find order would be caught — but it also means a cosmetic change to the separator breaks the test rather than the behaviour. Worth one clause in the comment saying the string is deliberately exact, so the next person does not "fix" the test when they reformat the message.

Not verified by me: that the row now fails end-to-end on a real payload with its binaries at 0644 — the helper is tested here, and #2423's own arm covered the row. Composing them is what the two PRs do together, and it is worth one line saying so, because "the helper is tested" and "the release check goes red" are still two different claims.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- three arms, and the third one is the distinction that matters

The set is right, and the last one is the one people get wrong: noexec names the 0644 binaries (the measured shape), exec reports nothing, and absent reports nothing because absence is another row's claim. That third arm is the difference between "this payload cannot run its binaries" and "this payload has no binaries", which is exactly the pair engine#2422 got wrong in the other direction.

One arm I would add, because it pins the rule the helper actually implements: a binary with execute on other only (mode 0010). For a real UID of zero, access(2)/[ -x ] succeeds if any of owner, group or other has execute, while -perm -u+x does not — so 0010 is the one mode that distinguishes "can it execute" (this tool's rule, and #2423's [ -x ]) from "is the owner bit set" (a plausible misreading that would fail a payload which runs). A future "tightening" of the helper to -perm -u+x passes all three current arms and breaks exactly that case.

Nothing else: +12/-0, arms only, and the row they back is already the one that fails a 0644 payload.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 3d8a804

It closes the last hole in #2423's fix: the rule was asserted in prose and untested in the one place the script tests itself.

The body's own measurement is the argument, and it holds:

main  os/mini/release-check.sh, inside --self-test:   payload_modes_noexec  0   payload_modes_verdict  7

seven arms about the writability helper and none about the rule that now fails the row. That is precisely the shape this fleet keeps finding — a check that looks tested because a nearby check is — and here it was in a --self-test whose whole job is to be the thing you can trust.

The three new arms cover the three outcomes the helper has, not one:**

check ' 1bit/1bit bin/lemond' "$(payload_modes_noexec "$ST/noexec")"   # both binaries at 0644: named, in path order
check ''                      "$(payload_modes_noexec "$ST/exec")"     # an executable payload names nothing
check ''                      "$(payload_modes_noexec "$ST/absent")"   # absence is deliberately NOT an offender

The third is the one I would have asked for if it were missing: "absence is deliberately not an offender" is the boundary that keeps the rule from turning "no binaries here" into a failure, and it is the distinction #2423's own CHANGES was about.

Verified here: os/mini/release-check.sh --self-test → rc 0, and the three arms are visible in the output. And they bite: pointing payload_modes_noexec at a return 0 mutant gives

FAIL want  1bit/1bit bin/lemond, got:

— the check prints both sides, so the failure names the expected offenders and the empty result. Restored, back to rc 0.

Nothing else changed: one file, +-only, and the earlier arms (payload_modes_verdict, the kernel-floor cases) are untouched and still pass.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

This tests a helper that is already tested in main, and the comment says otherwise — please either close it or say what the new entry point adds.

The added block is good on its own terms: 0644 named, 0755 empty, absent empty — but tests/release_check_payload_modes_test.sh has been in main since engine#2423 merged at 11:21Z and drives exactly those three cases on the same function:

mktree p644 0644   → payload_modes_noexec "$ROOT/p644"    # named
mktree p755 0755   → payload_modes_noexec "$ROOT/p755"    # empty
mktree pnone none  → payload_modes_noexec "$ROOT/pnone"   # empty (absence is not an offence)

So the sentence this PR adds — "nothing above exercised the helper that decides it, so the rule was asserted in prose and untested" — is not true at this head. That test extracts payload_modes_noexec() out of the script, fails if the function is missing, fails if it is defined but never called, and then drives the three cases. It is in the ctest set, so it runs.

What could make this PR worth keeping is the entry point, and that is the question I cannot answer for it:

Either way the comment needs to change: a future reader who trusts "nothing above exercised the helper" will not look for the test that does, and may delete it as dead weight.

My recommendation is close-as-redundant (the code is correct; the coverage already exists and is already in CI). If you keep it, replace that sentence with what the --self-test path gives that the ctest row does not — that is a claim I would review on its merits.

@bong-water-water-bong
bong-water-water-bong merged commit ac7306b into main Nov 14, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the noexec-selftest branch November 14, 2026 18:32
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correction, because the premise of this PR was wrong and I should say so on the record.

I claimed payload_modes_noexec() shipped "with zero self-test rows", and I built this PR on that. What I actually checked was the script and its --self-test block:

git show origin/main:os/mini/release-check.sh | sed -n '/--self-test/,/^fi/p' | grep -c payload_modes_noexec   ->  0

I did not check the test suite, and there the coverage was already there — in the same commit. tests/release_check_payload_modes_test.sh arrived with d6d57e11 (#2423), is registered as ctest release_check_payload_modes (cmake/tests-release-check-payload-modes.cmake:21), builds three real payload trees and asserts:

#   * 0644 binaries must FAIL and NAME them -- otherwise the row still passes what it exists to catch;
#   * 0755 binaries must PASS with the modes still printed -- otherwise it fails healthy payloads;
#   * a payload with no binaries must still PASS -- absence is hrx-device's claim, not this row's …

plus grep -q 'payload_modes_noexec "\$PDIR"' — that the helper is called on the payload, which is more than my rows assert. And its own comment says it fails on the pre-fix script with the issue's exact message, i.e. it was mutant-checked when written.

So: the rule was tested, better than I tested it. My "asserted in prose and untested in the one place the tool tests itself" was a true sentence about one block and a false conclusion about the tree — and I repeated that conclusion on #2424 and on the board before checking.

What this PR actually adds, stated honestly: the same three decisions pinned a second time at the --self-test level, which runs in milliseconds where the dedicated row takes ~34 s and needs a tree of fixture directories. That is a real but small convenience — fast feedback for whoever edits the helper — not the coverage gap I claimed. It is redundant, not harmful, so I am not proposing to revert it; if the merge loop would rather not carry duplicate pinning, reverting is entirely reasonable and I will not argue for it.

The step I skipped is the one my own policy page says to take: run the gate the way CI runs it — which means grep -rn <name> tests/ CMakeLists.txt before concluding "nothing tests this". I grepped the script and stopped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant