Test how the pruner reads a build that compiled nothing

[MarkusPaulsen]

Summary

Before a sandbox policy is written, the prune phase decides from each test run whether a hidden directory was needed. Two rules keep it from misreading a build that compiled nothing, or one whose tests simply fail. Nothing tested either rule. Both are now tested in both directions, and the test suite for the real pruner runs in CI.

Linked issues

Closes #16. Item 4 was done in #27; this finishes item 3.

1. Problem

The prune phase hides a directory, runs the exercise's build, and reads a failed run as "this directory was needed". Gradle reports a build that compiled nothing (NO-SOURCE) with exit code zero. Without a rule reading that line as a failure, the pruner would conclude that the source directory it just hid was unnecessary and write a policy that keeps it hidden, so a correct submission would fail. A second rule stops ordinary failing tests from being read as a missing directory. The two patterns in var/tmp/pruning/detect_minimal_fs.sh were the whole defence and nothing exercised them.

tests/prune_sandbox.sh, the suite that runs the real pruner, also ran in no workflow.

2. Improvement from the user's perspective

No Improvement from the user's perspective

3. Improvement from the maintainer's perspective

Changing either rule, or how the pruner reads a build's outcome, now fails a check instead of quietly producing a policy that is too tight or cannot be generated. The checks need no Bubblewrap: a stand-in bwrap builds no sandbox, records which paths each run would hide and runs the build directly, so they test how outcomes are classified, not what a sandbox contains. Each check reads the pruner's build logs, so a stand-in that breaks fails the checks rather than passing them. They run in the Test workflow on every pull request; the suite's Bubblewrap checks skip there, visibly, because the runner has no bubblewrap.

4. Testing manual

Prerequisites

1. Docker, and a checkout of this branch.

Steps

1. As CI runs it, in an ordinary container:
   docker run --rm -v "$PWD:/repo:ro" ubuntu:26.04 bash -c 'apt-get update -qq && apt-get install -y -qq python3 >/dev/null && cp -a /repo /w && bash /w/tests/prune_sandbox.sh'
2. Show that the checks catch a broken rule, first without the failing-tests pattern:
   docker run --rm -v "$PWD:/repo:ro" ubuntu:26.04 bash -c 'apt-get update -qq && apt-get install -y -qq python3 >/dev/null && cp -a /repo /w && sed -i "s/There were failing tests|//" /w/var/tmp/pruning/detect_minimal_fs.sh && bash /w/tests/prune_sandbox.sh'
3. Then without the NO-SOURCE rule:
   docker run --rm -v "$PWD:/repo:ro" ubuntu:26.04 bash -c 'apt-get update -qq && apt-get install -y -qq python3 >/dev/null && cp -a /repo /w && sed -i "s/^UNIGNORABLE_SUCCESS_PATTERNS=.*/UNIGNORABLE_SUCCESS_PATTERNS=a^/" /w/var/tmp/pruning/detect_minimal_fs.sh && bash /w/tests/prune_sandbox.sh'
4. Optionally, the whole suite with Bubblewrap, which needs a non-root user and, on Docker Desktop, --security-opt seccomp=unconfined --security-opt systempaths=unconfined, as tests/prune_sandbox.sh and Make the pruner safe to run against code it did not write #22 describe. Neither is --privileged.

Expected result

1. Five new ok lines between the two --target checks and the Bubblewrap skip, and 7 passed, 0 failed, 1 skipped.
2. FAIL failing tests alone do not make a directory needed, and 6 passed, 1 failed, 1 skipped.
3. FAIL on both checks about a build that compiles nothing, and 5 passed, 2 failed, 1 skipped.
4. 15 passed, 0 failed, 0 skipped.

Negative case (what must still be rejected)

A build that compiles nothing, whether it exits zero or not, keeps the directory it reads as r rather than hidden. Two control runs show each rule is load-bearing: with the NO-SOURCE pattern disabled the same fixture records n, and with the failing-tests pattern disabled the pruner stops after its first attempt, whose log shows the build ran and printed its failing tests.

Layers exercised

No layer-specific behaviour changed

• Filesystem layer
• Network layer
• Timeout layer
• All three together, as a run uses them by default

5. Test case coverage regarding this PR

Suite	Passed	Failed	Skipped	What it covers regarding this PR
tests/prune_sandbox.sh, Ubuntu 26.04 container, root, no bubblewrap, as in CI	7	0	1	Five new checks. Kept: NO-SOURCE with exit 0 and with exit 1 keeps target/needed as r. Hidden: failing tests alone leave target/unneeded as n. Controls: the NO-SOURCE rule disabled records n; the failing-tests rule disabled stops the prune after one attempt in which the build ran. The Bubblewrap group skips because nothing provides it.
tests/prune_sandbox.sh, Ubuntu 26.04 container, non-root, bubblewrap, the two --security-opt flags above	15	0	0	The five new checks and the ten existing sandbox checks, unchanged.
tests/prune_sandbox.sh with the NO-SOURCE pattern removed	5	2	1	Both NO-SOURCE checks fail, as they must.
tests/prune_sandbox.sh with the failing-tests pattern removed	6	1	1	The failing-tests check fails, as it must.
tests/prune_sandbox.sh with the stand-in bwrap broken so that it refuses every run	2	5	1	All five new checks fail, the controls included, so a broken stand-in cannot pass for a working rule.

All runs used kernel 7.0 on arm64 under Docker Desktop.

Breaking changes and migration

No breaking changes or migration

Checklist

• The title of this pull request describes the change, not the implementation.
• I have self-reviewed the diff of this pull request.
• Tests were added or updated for the behaviour changed here, in both directions: the forbidden case stays denied and the permitted case still works.

• The change was exercised in a container started without --privileged, --cap-add or --security-opt, or the manual says why that was not possible.

• CI is green, or every remaining failure is explained above.
• No secrets, tokens or absolute local paths are contained in the diff.

Review progress

• Code review
• Manual test