[MaxGhenis]

Main has no required checks today, so a PR can merge with no CI run on its head. For example, a stacked PR that retargets to main gets no run, because GitHub starts none for a base change. I decided to require passing CI on a PR's head before it merges into main. This PR adds a ci job that gives the main ruleset one check covering the country jobs. It also raises four job timeouts, since a job that times out will block a merge once the ruleset requires CI.

What changes

• .github/workflows/test.yml gains a ci job. It needs every other job, runs with if: always(), and passes GitHub's needs context to python3 tools/ci_test_plan.py require-results.
• require-results fails unless each job in the test plan reported success, or skipped because select-countries succeeded and reported false for the job's country. The plan names the jobs the selection may leave out: engine-us, engine-uk and integration-uk, whose test groups all need the same country's engine.
• The lint job runs a new command, tools/ci_test_plan.py verify-workflow, after it syncs the workspace. The command parses the workflow with PyYAML and refuses:
  • two keys that PyYAML reads as one, a merge key (<<), and U+0085, U+2028 or U+2029, which YAML 1.1 reads as line breaks and YAML 1.2 doesn't;
  • a job outside the plan, a plan job the workflow lacks, and a job whose if differs from the one the plan gives it;
  • a job the selection may leave out that doesn't need select-countries, and a job that needs one the selection may leave out, unless both follow the same country's selection;
  • a ci job that lacks if: always(), omits a job from needs, or does anything but check out the repository and run require-results on its needs context;
  • a select-countries job that doesn't publish the outputs its plan step writes;
  • continue-on-error on any job or step, a step if other than always(), success() or !cancelled(), and a job-level name;
  • a workflow defaults other than run: shell: bash, a job-level defaults and a step shell.
• The workflow now sets defaults: run: shell: bash. GitHub runs an explicit shell: bash as bash -eo pipefail, and a step with no shell as bash -e, where a pipe hides the exit status of the command that feeds it.
• tools/ci_test_plan.py verify, which lint already runs, now refuses a plan whose own tests engine-free would not run.
• select-countries now selects both countries for a PR whose file listing reaches GitHub's 3,000-file limit, since the listing stops there.
• Four timeouts rise to at least 1.5 times each job's slowest successful leg on 2026-11-09: engine-free from 120 to 165 minutes (its slowest leg took 109), engine-us from 90 to 110 (73), engine-uk from 60 to 90 (45) and integration-uk from 15 to 20 (8).
• docs/agent-guide.md describes ci and verify-workflow, and says that retargeting a stacked PR to main starts no run.

Why ci, and what checks it

A ruleset can't require engine-us or engine-uk by name. A matrix job that runs reports as engine-uk (3.13) and engine-uk (3.14), and one that skips reports as engine-uk, so a US-only PR never produces the two names a ruleset would wait for. ci reports one result for all the jobs, so a ruleset that requires ci covers the country jobs.

GitHub counts a skipped required check as passing ("A job that is skipped will report its status as 'Success'", per its docs on job conditions), so ci runs with if: always(). An edit that lets ci skip would leave it passing, and so would an edit that makes require-results pass a failed job. The plan's tests catch both. They run in engine-free, and the ruleset keeps naming both engine-free legs, so their failure blocks the merge without passing through ci. The tests also run the tool as a script, the way ci does, so a dead entry point fails them.

History

María added a ci-ok job for the same reason on 2026-08-25 (404e457). It accepted a lane's skip only when classification succeeded. #1048 removed it on 2026-09-29 when it unified test orchestration, and #888 added and removed one again on its branch. Neither PR says why it removed the job. This PR keeps María's rule, declares the job in the test plan's registry of infrastructure jobs, and puts the logic in tools/ci_test_plan.py, where pytest covers it.

Invariants

• ci reports one error for each job that neither passed nor skipped because the selection succeeded and reported false for it. Two property tests and an exhaustive test of 38,880 combinations on a five-job plan check that against an independent statement of the rule.
• A failed, cancelled or timed-out job always fails ci (a property test).
• verify-workflow passes a workflow only when it breaks none of the rules listed above. One property test checks the rules on jobs and steps against an independent statement of them, and another checks that a workflow keeps passing through changes no rule reads.
• The loader reads the repository's workflow as PyYAML does (a differential test).

Limits

• A PR's run takes the workflow and the plan from the PR's own merge with main, so a PR that rewrites them can pass itself. Review has to catch that.
• verify-workflow reads the workflow as PyYAML does. GitHub's own parser could read a file differently.
• ci checks each job's result, and a job decides its own result. A test step that swallows a failure, for example with || true, still passes, and so does a runner script that exits 0 after a failed test.
• The ruleset lets a PR merge without first updating it from main, so a PR that passed before main changed can still break main when it merges.
• The ruleset names both engine-free legs, so a change to that job's matrix needs a ruleset change with it. A test pins the names.

Rollout

A ruleset on main (id 26905965) logs what it would block and blocks nothing yet. Once active it requires six checks that run on every PR: lint, wheels, select-countries, installed-emitter and both engine-free legs. I'll add ci to that list after this lands and open PRs have run with the job. GitHub runs a PR's workflow on its merge with main, so an open PR gains ci on its next push, or when someone closes and reopens it. Re-running a run from before this lands reuses that run's commit and adds no ci, while re-running failed jobs in a later run re-runs ci too. A PR that adds a job must add it to the plan and to ci's needs, or lint fails.

Tests

• test_ci_test_plan.py and test_ci_integration_job.py: 225 tests pass locally.
• I changed one check in tools/ci_test_plan.py at a time, 81 variants in all, and a test failed for each.
• ruff check and ruff format --check pass on the touched files. python3 tools/ci_test_plan.py verify prints verification=ok, and verify-workflow prints workflow=ok.

axiom: n/a: CI infrastructure

🤖 Generated with Claude Code

[MaxGhenis]

I merged this at 2782c2c. All 12 checks passed on that head, including ci on its first run, where it read eight success results and printed results=ok. An independent Opus 5.5 review approved the same head with no findings.