Stop the prune phase producing a policy that is quietly incomplete

[MarkusPaulsen]

Summary

Eight defects in the prune phase and the helpers that turn its output into an allow-list. Three of them each meant on their own that the prune phase could never have run at all. Four more let a run finish successfully while producing less than it should, which matters because what it produces is a policy: everything it does not name is denied. The eighth writes every tail flag twice. Adds a Test workflow that runs the suites this repository already had.

Linked issues

No linked issues.

1. Problem

The prune phase discovers which paths a language environment needs, and its output becomes an allow-list. Nothing ran it in CI, nothing tested it, and it had stopped working.

It could not start. It asked bubblewrap for --unshare-utc, which has never existed. Both prune images name run_minimal_fs_all.sh as their entry point, and it was committed non-executable, so the container stopped at exec. And log() returned 1 whenever logging was off, ending the run at its first call under set -e.

It could also finish while producing too little. A failing language was merged around. A language that exited zero producing nothing, or naming no path, dropped out in silence. An emitter failure was only a warning, so a language kept whichever exercises happened to work. A policy built from part of what was asked for is indistinguishable from one built from all of it.

And merge_tail appended every token rather than only the unseen ones, so the tail gained a duplicate on each run.

2. Improvement from the user's perspective

A submission is no longer at risk of running under an allow-list that is quietly narrower than the one the instructor asked for. That is the failure that is hardest to recognise from the outside: the sandbox denies a path the exercise legitimately needs, the submission fails, and nothing in the output says the policy was built from half a pruning run.

For an instructor generating a policy, a pruning run that goes wrong now says so and stops, instead of writing files that look complete. The prune phase can also be run at all, which it could not before.

3. Improvement from the maintainer's perspective

The two entry points and the orchestrator gained the seams they needed to be tested: run_minimal_fs_all.sh takes TESTING_DIR, PRUNE_SCRIPT and OUTPUT_DIR the same way it already took HELPER_DIR, and the orchestrator takes --prune-script and --core-dir. With stubs behind them the new suites need no bubblewrap, no container and no exercises, so they run in seconds on an ordinary runner.

CI now executes the suites rather than only linting the files. Three of the seven defects were the kind that a single test run would have caught the day they were introduced, and the executable modes are asserted, because every suite invokes its script through bash and so none of them would ever notice a lost mode.

4. Testing manual

Prerequisites

1. A checkout of this branch. Nothing else: no container, no bubblewrap, no exercise repository, and no host preparation.
2. gcc, python3 and bash, which the suites use. On Ubuntu, build-essential and python3.

Steps

1. Run bash tests/prune_producer.sh.
2. Run bash tests/timeout_units.sh.
3. Run bash tests/network_cache_ports.sh.
4. Run python3 -m pip install pytest==8.4.2 and then python3 -m pytest tests/python -q. On a PEP 668 distribution, which Ubuntu 26.04 is, pip refuses to write into the system installation, so install it into a virtual environment or add --break-system-packages.
5. Confirm the prune entry point can start at all:
   docker run --rm -v "$PWD/var/tmp:/var/tmp" ubuntu:26.04 /var/tmp/pruning/run_minimal_fs_all.sh java

Expected result

1. 7 passed, 0 failed.
2. 58 passed, 0 failed, 0 skipped.
3. 20 passed, 0 failed, 0 skipped.
4. 13 passed.
5. [FAIL] Language folder not found: /var/tmp/testing-dir/java. That is the correct next error, because this branch adds no exercises. Before this branch the same command printed exec /var/tmp/pruning/run_minimal_fs_all.sh: permission denied and never reached the script.

Negative case (what must still be rejected)

Nothing here widens the sandbox; the point of the change is that a policy can no longer come out narrower than intended without saying so. Each fix is therefore checked against its own broken state, and each check fails when its fix is removed:

1. Remove return 0 from log() in run_minimal_fs_all.sh: a run without --verbose completes fails.
2. Remove the rm -f -- "${stale_artefacts[@]}" line: an earlier run's artefacts are removed fails, while another language's artefacts are left alone keeps passing.
3. Turn the emitter failure back into a warning: an emitter failure fails the run fails.
4. Replace missing_languages with an empty list in orchestrate.py: three tests fail, one for a language that produces nothing, one for a stale artefact standing in for it, and one for a result naming no path. The third belongs here because an empty union leaves the language out of collect_language_data, so this guard is what reports it, and removing the guard removes both protections at once.
5. Move merged.append(t) back outside the if in emit_artifacts.py: two of the tail-merge tests fail.

Confirm also that a language failing does not cancel the others: test_pruning_still_runs_for_the_other_languages asserts on a line the stub prints itself.

Layers exercised

No layer-specific behaviour changed. The prune phase sits upstream of all three layers and produces the allow-list the filesystem layer later reads, so the suites here assert on that output rather than on a layer at runtime.

• Filesystem layer
• Network layer
• Timeout layer
• All three together, as a run uses them by default

5. Test case coverage regarding this PR

Suite	Passed	Failed	Skipped	What it covers regarding this PR
tests/prune_producer.sh	7	0	0	New. A quiet run completing; this run's artefacts replacing the last run's; another language's artefacts left alone; an emitter failure failing the run and leaving nothing behind
tests/python/test_orchestrate.py	8	0	0	New. Both directions: every language contributing to the policy, against a failure, an empty result, a silent result and a stale artefact each stopping the merge
tests/python/test_emit_artifacts.py	5	0	0	New. The tail merge writing each flag once, keeping order, dropping the per-exercise --chdir and dropping a flag outside the allow-list
tests/timeout_units.sh	58	0	0	Unchanged. The timeout contract, confirming nothing here disturbed it
tests/network_cache_ports.sh	20	0	0	Unchanged. The network layer, confirming the same

Run on ubuntu:26.04, kernel 7.0.12-linuxkit, arm64, in a container started with a plain docker run --rm: no --privileged, no --cap-add, no --security-opt. None of these suites needs any of those, which is why they are the ones that can run on every pull request.

Breaking changes and migration

Exit codes change, deliberately. A pruning run that previously reported success while producing an incomplete result now fails: a language whose emitter failed, a language that produced no artefacts, and a language whose result names no path all stop the run instead of being merged around. A pipeline that was silently generating partial policies will therefore start failing, and that failure is the point rather than a regression. The fix is to look at why that language produces nothing, not to reinstate the old behaviour.

run_minimal_fs_all.sh now honours TESTING_DIR, PRUNE_SCRIPT and OUTPUT_DIR if they are set in the environment, and orchestrate.py gained --prune-script and --core-dir. All five keep their previous values as defaults, so an existing invocation behaves as before. No configuration file, policy key or path requirement changes.

Checklist

• The title of this pull request describes the change, not the implementation.
• I have self-reviewed the diff of this pull request.
• Tests were added or updated for the behaviour changed here, in both directions: the forbidden case stays denied and the permitted case still works.
• Any weakening of the sandbox boundary is stated explicitly above, including what it now permits that it did not permit before.
• The change was exercised in a container started without --privileged, --cap-add or --security-opt, or the manual says why that was not possible.

• CI is green, or every remaining failure is explained above.
• No secrets, tokens or absolute local paths are contained in the diff.

Review progress

• Code review
• Manual test