Repository navigation
Reduce GitHub Actions jobs and runner queue time across the organization #128
Description
Waiting for runners, not money, is the real cost of GitHub Actions across the organization. Private repositories bill about 42 hosted minutes a week, which the Free plan's included minutes cover, so the bill is effectively $0. What hurts is queue time, and this issue tracks the set of pull requests that cut the number of jobs and runaway runs behind it.
Measured drivers
Measured over 2026-09-24 to 2026-10-01.
cratis-arcsaturation. The scale set allows 3 runners at a time (arcMaxRunners: "3",arcMinRunners: "0"), and every job starts a fresh pod that takes about 1.5-2 minutes to begin. Across the organization's private repositories that is about 4,470 jobs a week, and about 66% of them do less than a minute of work. Demand (about 10,670 job-minutes plus about 5,800 minutes of pod start-up) is roughly 16,500 slot-minutes a week against about 30,000 in theory, mostly in working hours, so the pool is saturated during the day.- "Determine runner" jobs. The shared runner-selection job decided nothing: it always returned
cratis-arc. It still took a pod, about 660 times a week, and added 1.5-2 minutes serially to every run that used it. - Work-record guard runs. "Verify No Work Records" only inspects
*.mdfiles and.ai-work/, yet it ran on every pull request and every push to main: about 570 pods a week oncratis-arc. In public repositories the same guard takes about 10 minutes of wall-clock time for a roughly 10-second job, which points to queueing for hosted runners, and the Free plan caps the organization at 20 hosted jobs running at once. - Other shared-pool load. Scheduled dependency updates that fail on every run, push-to-main builds that repeat a pull request build, and workflows with no timeout (a hung job holds one of the 3 slots for up to 6 hours).
Pull requests
- Add timeouts to the work-record guard and the deployment auto-approval #126
- Skip CodeQL on documentation-only pull requests Arc#2943
- Skip CodeQL on documentation-only pull requests Chronicle#4450
- Skip CodeQL on documentation-only pull requests Fundamentals#1139
- Collapse bursts of dependency updates into one run Samples#617
- Cratis/Chronicle.Wolverine#127
- Cratis/Chronicle.Dapr#95
- Cratis/Direct#1452
- Cratis/Ensemble#123
- Cratis/Experiments#22
- Cratis/Infrastructure#96
- Cratis/Strategy#311
- Cratis/Studio#1489
Together they remove the runner-selection job, run the work-record guard only for markdown and .ai-work/ changes, skip CodeQL for documentation-only pull requests, drop duplicate push-to-main builds, add job timeouts, collapse bursts of dependency updates, and pause or stagger failing scheduled updates.
Follow-up decisions
Raising arcMaxRunners, or adding a second small scale set for gate and guard jobs, is the largest remaining lever for queue time and is not part of these pull requests.
Activity
woksin commentedon Oct 1, 2026
Bootstrap verification: merged #127 triggered https://github.com/Cratis/Workflows/actions/runs/36861584784 at 2026-10-01 12:24:53 UTC; it succeeded. Spot checks confirm release-intent and release-notes callers on Arc, Chronicle and Fundamentals, with recent passing runs for both gates. Public repository rulesets (including inherited rulesets) and main branch protections do not require verify / verify. Direct organization ruleset listing is unavailable to the current token (admin:org missing), so that limitation remains explicit. Package cleanup remains blocked by the fine-grained PAT_WORKFLOWS credential; the existing focused issue #103 now has the exact failure and admin action, avoiding a duplicate issue.
woksin commentedon Oct 1, 2026
Remaining implementations are prepared as local commits: Q3 public caller paths/concurrency (the shared five-minute timeout already exists), current SHA pins, ignored-repository release callers, SDK-pin installation, and Q11 job-level usage reporting with explicit missing-private-coverage warnings. No pushes or PRs yet: the complete package-updater offline suite exceeded its first 120-second execution limit, and subsequent bounded phase admission could not obtain a build/test slot; other repositories have not passed their full local gates. YAML validation, reporting fixtures, release gate/script tests, cleanup tests and the Dapr workflow-boundary check passed. This is incomplete, not ready-to-merge work. No D1–D10 change or schedule restoration was made. Cleanup permission remediation is tracked in #103.
woksin commentedon Oct 1, 2026
Decisions on the remaining CI-cost items (made 2026-10-01, delegated by the owner):
cratis-arc-smallrunner scale set for gate and guard jobs (release notes, release intent, work-record checks, PR gate checks) instead of doublingcratis-arc. About two thirds of jobs do under a minute of work, so they stop competing with builds for the 3 large slots.verify-pr-checksalready enforces the same rule at merge time.auditrun on the merged tree.run-integrationlabel, on changes underSource/Integrations/**, and nightly, following Chronicle's pattern.cratis-arc-small(D1) rather than to hosted runners.woksin commentedon Oct 1, 2026
The remaining public-repository CI follow-ups are pushed with one
no-releasePR per branch, all assigned to woksin:The complete updater suite passed once via pi-phase (
--timeout 300 --queue-timeout 600): 64 tests in 131.204 seconds, exit 0; no retries or direct fallback were needed. SDK-pin passing tests and other unchanged passing preparation checks were reused. YAML-only caller branches use the requested actionlint/diff-check gate. Main was fetched for every branch; Ante merged origin/main and its resulting YAML-only diff passed both checks again.Verification details and limitations are in PR comments, not release-note bodies. The first post-push CI snapshot has no failing checks, but several builds remain pending; this is not an all-green or merge-ready claim. No PR was merged, and no schedules, secrets, publication settings, or D1–D10 work were changed.
woksin commentedon Oct 5, 2026
Completed: all 13 listed PRs and 10 of 11 follow-ups are merged. Decisions D2-D9 delivered: Studio pr-gate removed; Studio#1495 publish job merge plus integration-spec timing; chronicle-health runs hourly at 17 * * * *; push-to-main builds removed in Strategy, Wolverine, Dapr and Ensemble; #131 covers D8; #2943 covers D9. Remaining items live in their own repos: D1/D10 runner scale set in Cratis/Infrastructure#100 and Cratis/Infrastructure#102 (Infrastructure#117 raised arcMaxRunners 3 to 6, partly covering D1), and Cratis/Chronicle.Python#35.