From f268df8e2dac33c1b7894f2b26a4ff1c0debaf8e Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Tue, 17 Nov 2026 23:09:37 +0000 Subject: [PATCH 1/2] fork_checks_ran: base-commit config, pull_request_target runs only, require success (#430) - Read fork_checks from the pull request's base commit (git show, fetched when missing); missing or undecodable fails closed. A rename pull request passes. - Count only pull_request_target runs of fork-checks.yml; the newest run decides, latest attempt only. Contract: pull_request_target is its only trigger. - Require the job to have concluded success: failed, cancelled, skipped or a finished run without the job fail at once; the wait cap stays below the job timeout (contract margin 300 s). - Direct non-PR runs of fork-tests.yml pass only on the default branch; runs called by another workflow log a skip. - Contract pins the job's if, env, steps and no continue-on-error; the job checks out with the shared token-free step; failure messages say how to recover; a test kills mutant M11. Fixes #430 Part of #401 Co-Authored-By: Claude Opus 5.5 --- .github/fork-actions-policies.json | 2 +- .github/scripts/fork-checks-ran.py | 202 +++++++++---- .github/scripts/fork-checks-ran.test.py | 361 ++++++++++++++++++------ .github/scripts/fork-contracts.test.py | 85 +++++- .github/workflows/fork-follow.yml | 8 +- .github/workflows/fork-release.yml | 8 +- .github/workflows/fork-tests.yml | 42 ++- fork-ci/actions/head-check/README.md | 3 +- 8 files changed, 541 insertions(+), 170 deletions(-) diff --git a/.github/fork-actions-policies.json b/.github/fork-actions-policies.json index 79bf4e967cd3e..4f8f31337b05c 100644 --- a/.github/fork-actions-policies.json +++ b/.github/fork-actions-policies.json @@ -1,5 +1,5 @@ { - "about": "The one place the repository's Actions policy for pull_request_target is defined (issue #401). From 2026-12-02 GitHub blocks pull_request_target unless a repository Actions policy (Settings > Actions > Policies, rule 'Restrict events') allows it for the workflow file. pull_request_target.allowed: the fork workflows the policy must allow, exactly these. pull_request_target.upstream_blocked: upstream's workflows with that trigger; the fork keeps them disabled and the policy does not name them, so they stay blocked. fork-contracts.test.py (section policies) checks every workflow with the trigger is in one list. fork_checks: the check run fork-tests.yml's fork_checks_ran job must find on a pull request's head (fork-checks-ran.py), so a blocked or renamed fork-checks.yml fails the required fork-ci-result instead of passing silently.", + "about": "The one place the repository's Actions policy for pull_request_target is defined (issue #401). From 2026-12-02 GitHub blocks pull_request_target unless a repository Actions policy (Settings > Actions > Policies, rule 'Restrict events') allows it for the workflow file. pull_request_target.allowed: the fork workflows the policy must allow, exactly these. pull_request_target.upstream_blocked: upstream's workflows with that trigger; the fork keeps them disabled and the policy does not name them, so they stay blocked. fork-contracts.test.py (section policies) checks every workflow with the trigger is in one list. fork_checks: the job fork-tests.yml's fork_checks_ran job requires to have passed in the newest pull_request_target run of that workflow on a pull request's head (fork-checks-ran.py), so a blocked, failed or renamed fork-checks.yml fails the required fork-ci-result instead of passing silently. fork-checks-ran.py reads this section from the pull request's base commit (#430), so a pull request that renames the workflow or the job is judged by the names that are live; outside_check names the job that refuses outside pull requests, pinned by the contract test.", "pull_request_target": { "allowed": [".github/workflows/fork-checks.yml"], "upstream_blocked": [ diff --git a/.github/scripts/fork-checks-ran.py b/.github/scripts/fork-checks-ran.py index 8b2117443e039..bb8fbfa5fea7d 100644 --- a/.github/scripts/fork-checks-ran.py +++ b/.github/scripts/fork-checks-ran.py @@ -1,26 +1,43 @@ #!/usr/bin/env python3 -"""Did fork-checks.yml run on this pull request's head? (issue #401, run by fork-tests.yml's fork_checks_ran job) +"""Did fork-checks.yml pass on this pull request's head? (issues #401 and #430, run by fork-tests.yml's fork_checks_ran job) fork-checks.yml runs on pull_request_target, which GitHub blocks from 2026-12-02 unless a repository Actions policy -allows it for that file (.github/fork-actions-policies.json). A blocked run shows only as a failed run that no ruleset -requires, so the schema and upstream-file checks would stop without anyone noticing. This script makes the required -fork-ci-result notice: on the head commit, a check run of the configured name (`check`, or `outside_check` for a pull -request from another fork) must exist as a job of a run of the configured workflow file (a job is a check run; the -Actions API needs only actions: read, which every caller of fork-tests.yml grants). Any state counts (queued, running, failed: exists = ran; failing is fork-checks' own job). Only skipped check runs fail at once. - -It polls every poll_seconds for at most wait_seconds, then fails with "fork-checks did not run: check the Actions -policy (#401)". An API error fails closed: a rate limit or a server error is retried within the same cap (and named in -the failure if it lasts), any other error (403, 404 ...) fails at once. Not a pull_request event: nothing to check. - -Environment: EVENT_NAME, HEAD_SHA, OUTSIDE_PR (true/false), GITHUB_TOKEN (actions: read), -GITHUB_REPOSITORY, GITHUB_API_URL. Test: fork-checks-ran.test.py. +allows it for that file (.github/fork-actions-policies.json). Its checks are not required, so a blocked, failed or +cancelled run would merge unseen. This script makes the required fork-ci-result notice: on the head commit, the newest +pull_request_target run of the configured workflow file must have a job (= check run) of the configured name (`check`) +whose latest attempt concluded success. Runs of other events do not count: they run the pull request's own copy of the +file. It waits while that job is queued or running; failed, cancelled or skipped fails at once, and so does a finished +run without the job (a blocked event leaves one). The Actions API needs only actions: read, which every caller of +fork-tests.yml grants. + +The config is read from the pull request's base commit (BASE_SHA, fetched when the checkout lacks it), not from the +checkout: pull_request_target runs main's copy of fork-checks.yml, so a pull request that renames its job or file is +judged by the names main's copy produces, and cannot edit the config its own check reads. Missing or undecodable +there: fail. + +It polls every poll_seconds for at most wait_seconds (below the job's timeout, so the message shows), then fails. An +API error fails closed: a rate limit or a server error is retried within the same cap (and named in the failure if it +lasts), any other error (403, 404 ...) fails at once. A pull request from another fork fails at once (the job's if: +skips those; fork-checks' only job for them always fails). + +Not a pull_request event: a run that another workflow called (GITHUB_WORKFLOW_REF names the caller) has nothing to +check, since its check runs are named " / ... / fork-ci-result", never the required "fork-ci-result". A +direct run (push, workflow_dispatch) passes only on the default branch: on any other branch its passing fork-ci-result +would land on that branch's tip, which may be an open pull request's head that fork-checks never passed (#430). + +Environment: EVENT_NAME, HEAD_SHA, BASE_SHA, OUTSIDE_PR (true/false), DEFAULT_BRANCH, GITHUB_REF, GITHUB_WORKFLOW_REF, +GITHUB_TOKEN (actions: read), GITHUB_REPOSITORY, GITHUB_API_URL. Test: fork-checks-ran.test.py. """ -import argparse, http.client, json, os, re, sys, time, urllib.error, urllib.parse, urllib.request +import http.client, json, os, re, subprocess, sys, time, urllib.error, urllib.parse, urllib.request -HERE = os.path.dirname(os.path.abspath(__file__)) -CONFIG = os.path.join(HERE, '..', 'fork-actions-policies.json') +CONFIG_PATH = '.github/fork-actions-policies.json' +TESTS_WORKFLOW = '.github/workflows/fork-tests.yml' # the workflow whose fork-ci-result a ruleset requires SHA = re.compile(r'[0-9a-f]{40}') NOT_RAN = 'fork-checks did not run: check the Actions policy (#401)' +WAY_OUT = ('Once that is fixed, push again or close and reopen the pull request (a blocked run cannot be re-run), ' + 'then this job runs again.') +RERUN = ('Fix what it reports and push again; if it was cancelled or timed out, re-run it (or push again, or close and ' + 'reopen the pull request), then re-run the failed jobs of this run.') class Transient(Exception): @@ -37,6 +54,10 @@ class Fatal(Exception): """Retrying cannot help (no permission, not found, bad request): fail now.""" +class ConfigError(Exception): + """The base commit's config cannot be read.""" + + class GitHub: def __init__(self, token, repo, api): self.token, self.repo, self.api = token, repo, api.rstrip('/') @@ -71,8 +92,32 @@ def get(self, path): raise Transient(f'{type(e).__name__} on {path.split("?")[0]}: {e}') from None +def base_config(sha, cwd=None, env=None): + """fork_checks of CONFIG_PATH on commit sha, fetched from origin when the checkout lacks it.""" + def git(*args): + try: + return subprocess.run(['git', *args], cwd=cwd, env=env, capture_output=True, timeout=180) + except (OSError, subprocess.TimeoutExpired) as e: + raise ConfigError(f'git {args[0]} failed: {e}') from None + + if git('cat-file', '-e', f'{sha}^{{commit}}').returncode != 0: + r = git('fetch', '-q', '--no-tags', '--no-recurse-submodules', '--depth=1', '--filter=blob:none', 'origin', sha) + if r.returncode != 0: + raise ConfigError(f'cannot fetch the base commit {sha}: {r.stderr.decode(errors="replace").strip()}') + r = git('show', f'{sha}:{CONFIG_PATH}') + if r.returncode != 0: + raise ConfigError(f'no {CONFIG_PATH} on the base commit {sha} ({r.stderr.decode(errors="replace").strip()})') + try: + cfg = json.loads(r.stdout.decode())['fork_checks'] + if not isinstance(cfg, dict): + raise TypeError('fork_checks is not an object') + except (ValueError, KeyError, TypeError) as e: + raise ConfigError(f'cannot read {CONFIG_PATH} on the base commit {sha}: {e!r}') from None + return cfg + + def config_problems(cfg): - problems = [f'fork_checks.{k} is not a non-empty string' for k in ('workflow', 'check', 'outside_check') + problems = [f'fork_checks.{k} is not a non-empty string' for k in ('workflow', 'check') if not isinstance(cfg.get(k), str) or not cfg.get(k)] for k in ('wait_seconds', 'poll_seconds'): if not isinstance(cfg.get(k), int) or isinstance(cfg.get(k), bool) or cfg.get(k) <= 0: @@ -81,38 +126,76 @@ def config_problems(cfg): def look(api, sha, name, workflow): - """'ran', 'skipped' (only skipped jobs of that name) or None: the jobs (= check runs) of the workflow's runs on sha.""" + """What the newest pull_request_target run of the workflow on sha says about its job `name`, latest attempt: + None (no such run yet), ('going', url), ('blocked', run url: finished without the job) or (conclusion, url).""" runs = api.get(f'actions/workflows/{urllib.parse.quote(os.path.basename(workflow))}/runs?' + urllib.parse.urlencode({'head_sha': sha, 'per_page': 100}))['workflow_runs'] - mine = [] - for run in runs: - if run['path'].split('@')[0] != workflow: - continue - jobs = api.get(f'actions/runs/{int(run["id"])}/jobs?filter=all&per_page=100')['jobs'] - mine += [j for j in jobs if j['name'] == name] - if any(not (j['status'] == 'completed' and j['conclusion'] == 'skipped') for j in mine): - return 'ran' - return 'skipped' if mine else None - - -def check(env, cfg, api, now=time.time, sleep=time.sleep, say=print): + mine = [r for r in runs if r['path'].split('@')[0] == workflow and r.get('event') == 'pull_request_target'] + for run in sorted(mine, key=lambda r: int(r['id']), reverse=True): + jobs = api.get(f'actions/runs/{int(run["id"])}/jobs?filter=latest&per_page=100')['jobs'] + named = [j for j in jobs if j['name'] == name] + if not named: + return ('blocked', run.get('html_url', f'run {run["id"]}')) if run['status'] == 'completed' \ + else ('going', run.get('html_url', '')) + j = named[0] + if j['status'] != 'completed': + return 'going', j.get('html_url', '') + return j['conclusion'] or 'without a conclusion', j.get('html_url', '') + return None + + +def not_a_pull_request(env, say): + event, ref = env.get('EVENT_NAME', '') or 'no event', env.get('GITHUB_REF', '') + repo, wf_ref = env.get('GITHUB_REPOSITORY', ''), env.get('GITHUB_WORKFLOW_REF', '') + if not repo or not wf_ref.startswith(repo + '/') or '@' not in wf_ref: + say(f'::error::cannot tell which workflow runs this ({event}; GITHUB_WORKFLOW_REF {wf_ref!r}): failing closed') + return 1 + workflow = wf_ref[len(repo) + 1:].split('@')[0] + if workflow != TESTS_WORKFLOW: + say(f'not a pull request ({event}), called by {workflow}: its check runs are named " / ... / ' + 'fork-ci-result", not the required fork-ci-result; nothing to check') + return 0 + default = env.get('DEFAULT_BRANCH', '') + if default and ref == f'refs/heads/{default}': + say(f'not a pull request ({event}) on {default}: fork-checks runs only on pull requests; nothing to check') + return 0 + say(f'::error::a {event} run of {TESTS_WORKFLOW} on {ref or "no ref"} puts its fork-ci-result on that commit, ' + f'which may be an open pull request\'s head that fork-checks never passed (#430), so it fails. Run it from ' + f'the default branch ({default or "unknown"}) and name the commit or branch to test: ' + f'dispatch fork-tests.yml on {default or "the default branch"} with ref=') + return 1 + + +def check(env, api, config, now=time.time, sleep=time.sleep, say=print): + """config(base_sha) -> the fork_checks dict (base_config in the workflow); raises ConfigError.""" event = env.get('EVENT_NAME', '') if event != 'pull_request': - say(f'not a pull request ({event or "no event"}): fork-checks runs only on pull requests; nothing to check') - return 0 - problems = config_problems(cfg) - sha, outside = env.get('HEAD_SHA', ''), env.get('OUTSIDE_PR', '') - if not SHA.fullmatch(sha): - problems.append(f'HEAD_SHA {sha!r} is not a commit id') + return not_a_pull_request(env, say) + sha, base, outside = env.get('HEAD_SHA', ''), env.get('BASE_SHA', ''), env.get('OUTSIDE_PR', '') + problems = [f'{k} {v!r} is not a commit id' for k, v in (('HEAD_SHA', sha), ('BASE_SHA', base)) + if not SHA.fullmatch(v)] if outside not in ('true', 'false'): problems.append(f'OUTSIDE_PR {outside!r} is not true or false') if problems: say('::error::' + '; '.join(problems)) return 1 - name = cfg['outside_check'] if outside == 'true' else cfg['check'] - workflow, poll = cfg['workflow'], cfg['poll_seconds'] + if outside == 'true': + say('::error::outside pull request: fork-checks does not check it (its Outside pull request job fails); ' + 'a maintainer must review') + return 1 + try: + cfg = config(base) + except ConfigError as e: + say(f'::error::fork-checks could not be confirmed (failing closed): {e}') + return 1 + problems = config_problems(cfg) + if problems: + say(f'::error::{CONFIG_PATH} on the base commit {base}: ' + '; '.join(problems)) + return 1 + name, workflow, poll = cfg['check'], cfg['workflow'], cfg['poll_seconds'] deadline = now() + cfg['wait_seconds'] - say(f'Looking for check run "{name}" of {workflow} on {sha} (at most {cfg["wait_seconds"]} s).') + say(f'Waiting for check run "{name}" of {workflow} on {sha} to pass (at most {cfg["wait_seconds"]} s; ' + f'config from the base commit {base}).') last_error = None while True: wait = poll @@ -121,47 +204,50 @@ def check(env, cfg, api, now=time.time, sleep=time.sleep, say=print): last_error = None except Fatal as e: say(f'::error::fork-checks could not be confirmed (failing closed): GitHub API error: {e}. A 404: ' - f'{workflow} renamed or gone? Then update .github/fork-actions-policies.json and the Actions policy (#401)') + f'{workflow} renamed or gone? Then update {CONFIG_PATH} and the Actions policy (#401)') return 1 except Transient as e: found, last_error = None, str(e) say(f'::warning::{e}; retrying') if isinstance(e, RateLimited): wait = max(poll, e.reset - now()) - except (KeyError, TypeError, IndexError, AttributeError) as e: + except (KeyError, TypeError, IndexError, AttributeError, ValueError) as e: say(f'::error::fork-checks could not be confirmed (failing closed): unexpected GitHub reply ({e!r})') return 1 - if found == 'ran': - say(f'ok: fork-checks ran on {sha} (check run "{name}" of {workflow})') + state, url = found or (None, '') + if state == 'success': + say(f'ok: fork-checks passed on {sha} (check run "{name}" of {workflow}: {url})') return 0 - if found == 'skipped': - say(f'::error::{NOT_RAN}: its check run "{name}" on {sha} was skipped, so nothing was checked') + if state == 'blocked': + say(f'::error::{NOT_RAN}: its newest run on {sha} finished without a "{name}" job ({url}). {WAY_OUT}') + return 1 + if state == 'skipped': + say(f'::error::fork-checks did not check {sha}: its check run "{name}" was skipped ({url}). {RERUN}') + return 1 + if state not in (None, 'going'): + say(f'::error::fork-checks did not pass on {sha}: check run "{name}" of {workflow} ended {state} ' + f'({url}). {RERUN}') return 1 left = deadline - now() if wait > left: if last_error: say(f'::error::fork-checks could not be confirmed (failing closed) within {cfg["wait_seconds"]} s: ' - f'the last GitHub API call failed: {last_error}') + f'the last GitHub API call failed: {last_error}. Re-run the failed jobs of this run.') + elif state == 'going': + say(f'::error::fork-checks did not finish on {sha} within {cfg["wait_seconds"]} s (check run "{name}" ' + f'of {workflow}: {url}). Once it has passed, re-run the failed jobs of this run.') else: - say(f'::error::{NOT_RAN}: no check run "{name}" of {workflow} on {sha} after {cfg["wait_seconds"]} s') + say(f'::error::{NOT_RAN}: no pull_request_target run of {workflow} with a check run "{name}" on {sha} ' + f'after {cfg["wait_seconds"]} s. {WAY_OUT}') return 1 sleep(wait) -def main(argv=None): - ap = argparse.ArgumentParser(description=__doc__.split('\n', 1)[0]) - ap.add_argument('--config', default=CONFIG) - args = ap.parse_args(argv) +def main(): env = os.environ - try: - with open(args.config) as f: - cfg = json.load(f).get('fork_checks') or {} - except (OSError, ValueError, AttributeError) as e: - print(f'::error::cannot read {args.config}: {e}') - return 1 api = GitHub(env.get('GITHUB_TOKEN', ''), env.get('GITHUB_REPOSITORY', ''), env.get('GITHUB_API_URL') or 'https://api.github.com') - return check(env, cfg, api, say=lambda m: print(m, flush=True)) + return check(env, api, base_config, say=lambda m: print(m, flush=True)) if __name__ == '__main__': diff --git a/.github/scripts/fork-checks-ran.test.py b/.github/scripts/fork-checks-ran.test.py index 510e0ae6ac2e3..571c30c79f4e9 100644 --- a/.github/scripts/fork-checks-ran.test.py +++ b/.github/scripts/fork-checks-ran.test.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 -"""Tests of fork-checks-ran.py: a fake GitHub (in process, and a local HTTP server for the real client), a fake clock. -Nothing reaches GitHub. Run: python3 .""" -import http.server, importlib.util, json, os, re, subprocess, sys, tempfile, threading, unittest, urllib.parse +"""Tests of fork-checks-ran.py: a fake GitHub (in process, and a local HTTP server for the real client), a fake clock, +throwaway git repositories for the base commit's config. Nothing reaches GitHub. Run: python3 .""" +import http.server, importlib.util, json, os, re, shutil, subprocess, sys, tempfile, threading, unittest, urllib.parse sys.dont_write_bytecode = True HERE = os.path.dirname(os.path.abspath(__file__)) @@ -11,18 +11,25 @@ fcr = importlib.util.module_from_spec(_spec) _spec.loader.exec_module(fcr) -SHA = 'a' * 40 +SHA, BASE = 'a' * 40, 'b' * 40 WF = '.github/workflows/fork-checks.yml' CFG = {'workflow': WF, 'check': 'No database schema changes', 'outside_check': 'Outside pull request', 'wait_seconds': 600, 'poll_seconds': 20} +PR_ENV = {'EVENT_NAME': 'pull_request', 'HEAD_SHA': SHA, 'BASE_SHA': BASE, 'OUTSIDE_PR': 'false', + 'DEFAULT_BRANCH': 'main', 'GITHUB_REPOSITORY': 'o/r', 'GITHUB_REF': 'refs/pull/7/merge', + 'GITHUB_WORKFLOW_REF': 'o/r/.github/workflows/fork-tests.yml@refs/pull/7/merge'} +GIT_ENV = {'GIT_CONFIG_NOSYSTEM': '1', 'GIT_CONFIG_GLOBAL': os.devnull, 'GIT_TERMINAL_PROMPT': '0', + 'GIT_AUTHOR_NAME': 't', 'GIT_AUTHOR_EMAIL': 't@example.invalid', + 'GIT_COMMITTER_NAME': 't', 'GIT_COMMITTER_EMAIL': 't@example.invalid'} def job(name, status='completed', conclusion='success'): - return {'name': name, 'status': status, 'conclusion': conclusion} + return {'name': name, 'status': status, 'conclusion': conclusion, + 'html_url': f'https://github.invalid/job/{abs(hash(name)) % 1000}'} -def wr(rid, jobs, path=WF): - return {'id': rid, 'path': path, 'jobs': jobs} +def wr(rid, jobs, path=WF, event='pull_request_target', status='completed'): + return {'id': rid, 'path': path, 'event': event, 'status': status, 'jobs': jobs} class Clock: @@ -39,7 +46,8 @@ def sleep(self, s): class Fake: - """GitHub's runs of fork-checks.yml on SHA (one list per poll, the last one repeats) and each run's jobs.""" + """GitHub's runs of fork-checks.yml on SHA (one list per poll, the last one repeats) and each run's jobs (the + latest attempt only: the script must ask for filter=latest).""" def __init__(self, polls, errors=()): self.polls, self.errors, self.paths = list(polls), list(errors), [] @@ -56,99 +64,209 @@ def get(self, path): if url.path == 'actions/workflows/fork-checks.yml/runs': assert q['head_sha'] == [SHA], path self.current = self.polls.pop(0) if len(self.polls) > 1 else self.polls[0] - return {'workflow_runs': [{'id': r['id'], 'path': r['path']} for r in self.current]} + return {'workflow_runs': [{k: v for k, v in r.items() if k != 'jobs'} for r in self.current]} m = re.fullmatch(r'actions/runs/(\d+)/jobs', url.path) - if m and q['filter'] == ['all']: + if m and q['filter'] == ['latest']: return {'jobs': next(r['jobs'] for r in self.current if r['id'] == int(m.group(1)))} raise AssertionError(path) -def check(fake, event='pull_request', outside='false', sha=SHA, cfg=CFG): - clock, out = Clock(), [] - code = fcr.check({'EVENT_NAME': event, 'HEAD_SHA': sha, 'OUTSIDE_PR': outside}, dict(cfg), fake, - clock.now, clock.sleep, out.append) +def check(fake, cfg=CFG, **env): + clock, out, asked = Clock(), [], [] + + def config(base): + asked.append(base) + return dict(cfg) + code = fcr.check({**PR_ENV, **env}, fake, config, clock.now, clock.sleep, out.append) return code, '\n'.join(out), clock +def git(cwd, *args): + return subprocess.run(['git', *args], cwd=cwd, env={**os.environ, **GIT_ENV}, capture_output=True, text=True, + check=True).stdout.strip() + + +def make_repo(path, commits): + """A repository at path with one commit per dict of {path: text or None (deleted)}; returns the commit ids.""" + os.makedirs(path) + git(path, 'init', '-q') + shas = [] + for files in commits: + for name, text in files.items(): + full = os.path.join(path, name) + if text is None: + git(path, 'rm', '-q', name) + continue + os.makedirs(os.path.dirname(full), exist_ok=True) + with open(full, 'wb') as f: + f.write(text if isinstance(text, bytes) else text.encode()) + git(path, 'add', name) + git(path, 'commit', '-q', '--allow-empty', '-m', f'commit {len(shas)}') + shas.append(git(path, 'rev-parse', 'HEAD')) + return shas + + +def policies(**fork_checks): + return json.dumps({'fork_checks': {**CFG, **fork_checks}}) + + class Check(unittest.TestCase): - def test_not_a_pull_request_skips_with_a_log_line(self): - for event in ('push', 'merge_group', 'schedule', 'workflow_dispatch', 'workflow_call'): + # --- not a pull request (#430 item 3) --------------------------------------------------------------------------- + def test_direct_run_on_the_default_branch_needs_no_check(self): + for event in ('push', 'workflow_dispatch'): fake = Fake([[]]) - code, out, _ = check(fake, event=event) + code, out, _ = check(fake, EVENT_NAME=event, GITHUB_REF='refs/heads/main', + GITHUB_WORKFLOW_REF='o/r/.github/workflows/fork-tests.yml@refs/heads/main') self.assertEqual(code, 0, out) self.assertIn(f'not a pull request ({event})', out) self.assertEqual(fake.paths, []) - def test_check_there_at_once_passes_without_sleeping(self): - code, out, clock = check(Fake([[wr(1, [job(CFG['check'], 'in_progress', None)])]])) + def test_direct_run_on_another_ref_fails(self): + # Its fork-ci-result lands on that ref's tip, which may be an open pull request's head. + for event, ref in (('workflow_dispatch', 'refs/heads/feature'), ('push', 'refs/heads/feature'), + ('workflow_dispatch', 'refs/tags/v1'), ('schedule', 'refs/heads/main2')): + fake = Fake([[]]) + code, out, _ = check(fake, EVENT_NAME=event, GITHUB_REF=ref, + GITHUB_WORKFLOW_REF=f'o/r/.github/workflows/fork-tests.yml@{ref}') + self.assertEqual(code, 1, f'{event} {ref}: {out}') + self.assertIn('#430', out) + self.assertIn('dispatch fork-tests.yml on main with ref=', out) + self.assertEqual(fake.paths, []) + + def test_called_run_needs_no_check(self): + # fork-release.yml calls fork-tests.yml: its check runs are named " / ... / fork-ci-result". + for event, ref in (('workflow_dispatch', 'refs/heads/feature'), ('schedule', 'refs/heads/main')): + fake = Fake([[]]) + code, out, _ = check(fake, EVENT_NAME=event, GITHUB_REF=ref, DEFAULT_BRANCH='', + GITHUB_WORKFLOW_REF=f'o/r/.github/workflows/fork-follow.yml@{ref}') + self.assertEqual(code, 0, out) + self.assertIn('called by .github/workflows/fork-follow.yml', out) + self.assertEqual(fake.paths, []) + + def test_direct_run_that_cannot_be_placed_fails(self): + for env in ({'DEFAULT_BRANCH': ''}, {'GITHUB_WORKFLOW_REF': ''}, {'GITHUB_WORKFLOW_REF': 'x/y/fork-tests.yml@a'}, + {'GITHUB_WORKFLOW_REF': 'o/r/.github/workflows/fork-tests.yml'}): + code, out, _ = check(Fake([[]]), EVENT_NAME='workflow_dispatch', GITHUB_REF='refs/heads/main', **env) + self.assertEqual(code, 1, f'{env}: {out}') + + # --- the pull request's fork-checks job must succeed (#430 item 4) ------------------------------------------------ + def test_success_at_once_passes_without_sleeping(self): + code, out, clock = check(Fake([[wr(1, [job(CFG['check'])])]])) + self.assertEqual(code, 0, out) + self.assertEqual(clock.slept, []) + self.assertIn('passed', out) + + def test_still_running_waits_for_it(self): + fake = Fake([[wr(1, [job(CFG['check'], 'queued', None)], status='queued')], + [wr(1, [job(CFG['check'], 'in_progress', None)], status='in_progress')], + [wr(1, [job(CFG['check'])])]]) + code, out, clock = check(fake) self.assertEqual(code, 0, out) + self.assertEqual(clock.slept, [20, 20]) + + def test_not_finished_at_the_cap_fails_saying_so(self): + code, out, clock = check(Fake([[wr(1, [job(CFG['check'], 'in_progress', None)], status='in_progress')]])) + self.assertEqual(code, 1, out) + self.assertIn('did not finish', out) + self.assertIn('re-run', out) + self.assertLessEqual(sum(clock.slept), CFG['wait_seconds']) + + def test_failed_or_cancelled_fails_at_once_with_a_way_out(self): + for conclusion in ('failure', 'cancelled', 'timed_out', 'neutral', 'action_required', 'stale', None): + code, out, clock = check(Fake([[wr(1, [job(CFG['check'], 'completed', conclusion)])]])) + self.assertEqual(code, 1, f'{conclusion}: {out}') + self.assertEqual(clock.slept, [], conclusion) + self.assertIn('https://github.invalid/job/', out) + self.assertIn('push again', out) + + def test_only_skipped_fails_at_once(self): + code, out, clock = check(Fake([[wr(1, [job(CFG['check'], 'completed', 'skipped')])]])) + self.assertEqual(code, 1, out) + self.assertIn('skipped', out) self.assertEqual(clock.slept, []) - self.assertIn('fork-checks ran', out) - def test_queued_or_failed_counts_as_ran(self): - for status, conclusion in (('queued', None), ('completed', 'failure'), ('completed', 'cancelled')): - code, out, _ = check(Fake([[wr(1, [job(CFG['check'], status, conclusion)])]])) - self.assertEqual(code, 0, f'{status} {conclusion}: {out}') + def test_newest_run_decides(self): + # An `edited` event runs fork-checks again on the same head: the newest run is the one that counts. + for runs, want in (([wr(9, [job(CFG['check'], 'completed', 'failure')]), wr(3, [job(CFG['check'])])], 1), + ([wr(3, [job(CFG['check'], 'completed', 'failure')]), wr(9, [job(CFG['check'])])], 0), + ([wr(3, [job(CFG['check'])]), wr(9, [job(CFG['check'], 'completed', 'cancelled')])], 1)): + code, out, _ = check(Fake([runs])) + self.assertEqual(code, want, f'{[r["id"] for r in runs]}: {out}') + + def test_reads_the_latest_attempt_only(self): + fake = Fake([[wr(1, [job(CFG['check'])])]]) + self.assertEqual(check(fake)[0], 0) + self.assertTrue(any('filter=latest' in p for p in fake.paths), fake.paths) + + # --- which runs count ------------------------------------------------------------------------------------------- + def test_only_pull_request_target_runs_count(self): + # A pull request that adds push: or pull_request: to its own fork-checks.yml gets a same-named job from its + # own copy (#430 item 2). Only pull_request_target runs main's copy. + for event in ('pull_request', 'push', 'workflow_dispatch', 'schedule', None): + fake = Fake([[wr(4, [job(CFG['check'])], event=event)]]) + code, out, _ = check(fake) + self.assertEqual(code, 1, f'{event}: {out}') + self.assertNotIn('actions/runs/4/jobs', ' '.join(fake.paths), event) + + def test_a_pull_request_target_run_next_to_a_no_op_one_decides(self): + fake = Fake([[wr(8, [job(CFG['check'])], event='push'), + wr(5, [job(CFG['check'], 'completed', 'failure')])]]) + self.assertEqual(check(fake)[0], 1) def test_run_path_with_a_ref_suffix_matches(self): self.assertEqual(check(Fake([[wr(1, [job(CFG['check'])], WF + '@refs/heads/main')]]))[0], 0) + def test_a_run_of_another_workflow_does_not_count(self): + fake = Fake([[wr(2, [job(CFG['check'])], '.github/workflows/fork-tests.yml')]]) + code, out, _ = check(fake) + self.assertEqual(code, 1, out) + self.assertNotIn('actions/runs/2/jobs', ' '.join(fake.paths)) + def test_appears_later_passes_after_bounded_polls(self): - code, out, clock = check(Fake([[], [wr(1, [])], [wr(1, [job(CFG['check'])])]])) + code, out, clock = check(Fake([[], [wr(1, [], status='queued')], [wr(1, [job(CFG['check'])])]])) self.assertEqual(code, 0, out) self.assertEqual(clock.slept, [20, 20]) - def test_never_appears_fails_after_the_cap_with_the_policy_message(self): + def test_never_appears_fails_after_the_cap_with_the_policy_message_and_a_way_out(self): code, out, clock = check(Fake([[]])) self.assertEqual(code, 1) self.assertIn('fork-checks did not run: check the Actions policy (#401)', out) + self.assertIn('push again', out) + self.assertIn('close and reopen', out) self.assertLessEqual(sum(clock.slept), CFG['wait_seconds']) self.assertGreaterEqual(sum(clock.slept), CFG['wait_seconds'] - CFG['poll_seconds']) - def test_blocked_run_without_jobs_fails_after_the_cap(self): - # What a blocked event leaves: a failed run of fork-checks.yml with no job in it. - code, out, _ = check(Fake([[{'id': 5, 'path': WF, 'jobs': []}]])) + def test_blocked_run_without_jobs_fails_at_once(self): + # What a blocked event leaves: a finished run of fork-checks.yml with no job in it. + code, out, clock = check(Fake([[wr(5, [], status='completed')]])) self.assertEqual(code, 1) self.assertIn('fork-checks did not run', out) - - def test_a_run_of_another_workflow_does_not_count(self): - fake = Fake([[wr(2, [job(CFG['outside_check'])], '.github/workflows/fork-tests.yml')]]) - code, out, _ = check(fake, outside='true') - self.assertEqual(code, 1, out) - self.assertNotIn('actions/runs/2/jobs', ' '.join(fake.paths)) - - def test_outside_pull_request_looks_for_the_outside_check(self): - fake = Fake([[wr(1, [job(CFG['check'], 'completed', 'skipped'), job(CFG['outside_check'], 'completed', 'failure')])]]) - code, out, _ = check(fake, outside='true') - self.assertEqual(code, 0, out) - self.assertIn('"Outside pull request"', out) - code, out, _ = check(Fake([[wr(1, [job(CFG['check'])])]]), outside='true') - self.assertEqual(code, 1, out) - - def test_own_pull_request_ignores_the_outside_check(self): - self.assertEqual(check(Fake([[wr(1, [job(CFG['outside_check'])])]]))[0], 1) - - def test_only_skipped_fails_at_once(self): - code, out, clock = check(Fake([[wr(1, [job(CFG['check'], 'completed', 'skipped')])]])) - self.assertEqual(code, 1, out) - self.assertIn('skipped', out) + self.assertIn('close and reopen', out) self.assertEqual(clock.slept, []) - def test_skipped_in_one_run_and_ran_in_another_passes(self): - fake = Fake([[wr(1, [job(CFG['check'], 'completed', 'skipped')]), wr(3, [job(CFG['check'])])]]) - self.assertEqual(check(fake)[0], 0) - - def test_stops_reading_runs_at_the_first_that_ran(self): - fake = Fake([[wr(1, [job(CFG['check'])]), wr(2, [job(CFG['check'])])]]) - self.assertEqual(check(fake)[0], 0) - self.assertNotIn('actions/runs/2/jobs', ' '.join(fake.paths)) + def test_outside_pull_request_is_refused_at_once(self): + # fork_checks_ran is skipped for them (its if:); its only fork-checks job, Outside pull request, always fails. + fake = Fake([[wr(1, [job(CFG['outside_check'])])]]) + code, out, _ = check(fake, OUTSIDE_PR='true') + self.assertEqual(code, 1, out) + self.assertIn('outside', out) + self.assertEqual(fake.paths, []) + # --- the API ------------------------------------------------------------------------------------------------------ def test_transient_error_is_retried(self): fake = Fake([[wr(1, [job(CFG['check'])])]], errors=[fcr.Transient('HTTP 502'), None, None]) code, out, _ = check(fake) self.assertEqual(code, 0, out) self.assertIn('HTTP 502', out) + def test_a_good_poll_after_an_error_clears_it(self): + # Mutant M11 of #420: the cap message must name what the last poll saw, not an error two polls back. + fake = Fake([[]], errors=[fcr.Transient('HTTP 502')]) + code, out, _ = check(fake) + self.assertEqual(code, 1) + self.assertIn('fork-checks did not run: check the Actions policy (#401)', out) + self.assertNotIn('last GitHub API call failed', out) + def test_persistent_transient_error_fails_closed_naming_it(self): fake = Fake([[]], errors=[fcr.Transient('HTTP 503')] * 1000) code, out, clock = check(fake) @@ -186,16 +304,90 @@ def get(self, path): self.assertIn('could not be confirmed', out) def test_bad_input_fails(self): - self.assertEqual(check(Fake([[]]), sha='main')[0], 1) - self.assertEqual(check(Fake([[]]), outside='maybe')[0], 1) - self.assertEqual(check(Fake([[]]), cfg={**CFG, 'wait_seconds': 0})[0], 1) - self.assertEqual(check(Fake([[]]), cfg={k: v for k, v in CFG.items() if k != 'check'})[0], 1) + # Each against a passing fork-checks run, so only the bad input can fail it. + def passing(): + return Fake([[wr(1, [job(CFG['check'])])]]) + self.assertEqual(check(passing())[0], 0) + for env, what in (({'HEAD_SHA': 'main'}, 'HEAD_SHA'), ({'BASE_SHA': ''}, 'BASE_SHA'), + ({'BASE_SHA': 'b' * 39}, 'BASE_SHA'), ({'OUTSIDE_PR': 'maybe'}, 'OUTSIDE_PR')): + code, out, _ = check(passing(), **env) + self.assertEqual((code, what in out), (1, True), (env, out)) + self.assertEqual(check(passing(), cfg={**CFG, 'wait_seconds': 0})[0], 1) + self.assertEqual(check(passing(), cfg={k: v for k, v in CFG.items() if k != 'check'})[0], 1) def test_real_config_has_the_fields(self): with open(CONFIG) as f: cfg = json.load(f)['fork_checks'] self.assertEqual(fcr.config_problems(cfg), []) - self.assertLessEqual(cfg['wait_seconds'], 900, 'the cap must fit the job timeout') + + +class BaseConfig(unittest.TestCase): + """The config comes from the pull request's base commit (#430 item 1): fork-checks runs main's copy, so a pull + request that renames its job or file must still be judged by the names main's copy produces.""" + + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix='fork-checks-ran-') + + def tearDown(self): + shutil.rmtree(self.tmp, ignore_errors=True) + + def load(self, repo, sha): + return fcr.base_config(sha, cwd=repo, env={**os.environ, **GIT_ENV}) + + def test_rename_pull_request_is_judged_by_the_base_config(self): + repo = os.path.join(self.tmp, 'r') + base, head = make_repo(repo, [ + {fcr.CONFIG_PATH: policies(), WF: 'on: pull_request_target\n'}, + {fcr.CONFIG_PATH: policies(workflow='.github/workflows/fork-checks2.yml', check='Schema'), + WF: None, '.github/workflows/fork-checks2.yml': 'on: pull_request_target\n'}]) + self.assertEqual(git(repo, 'rev-parse', 'HEAD'), head) # the checkout holds the renamed config + cfg = self.load(repo, base) + self.assertEqual((cfg['workflow'], cfg['check']), (WF, CFG['check'])) + clock, out = Clock(), [] + code = fcr.check({**PR_ENV, 'BASE_SHA': base}, Fake([[wr(1, [job(CFG['check'])])]]), + lambda sha: self.load(repo, sha), clock.now, clock.sleep, out.append) + self.assertEqual(code, 0, out) + + def test_missing_config_on_the_base_fails_clearly(self): + repo = os.path.join(self.tmp, 'r') + base, _ = make_repo(repo, [{'README': 'x'}, {fcr.CONFIG_PATH: policies()}]) + with self.assertRaises(fcr.ConfigError) as e: + self.load(repo, base) + self.assertIn(f'no {fcr.CONFIG_PATH} on the base commit {base}', str(e.exception)) + out = [] + code = fcr.check({**PR_ENV, 'BASE_SHA': base}, Fake([[]]), lambda sha: self.load(repo, sha), + Clock().now, Clock().sleep, out.append) + self.assertEqual(code, 1) + self.assertIn('on the base commit', '\n'.join(out)) + + def test_undecodable_config_on_the_base_fails_clearly(self): + for i, text in enumerate(('{', '[]', '{"fork_checks": 3}', '{"other": {}}', b'{"fork_checks": "\xff"}')): + repo = os.path.join(self.tmp, f'r{i}') + base, = make_repo(repo, [{fcr.CONFIG_PATH: text}]) + with self.assertRaises(fcr.ConfigError, msg=text) as e: + self.load(repo, base) + self.assertIn('cannot read', str(e.exception)) + + def test_base_commit_is_fetched_when_the_checkout_lacks_it(self): + origin = os.path.join(self.tmp, 'origin') + base, head = make_repo(origin, [{fcr.CONFIG_PATH: policies()}, {fcr.CONFIG_PATH: policies(check='Schema')}]) + git(origin, 'config', 'uploadpack.allowAnySHA1InWant', 'true') + git(origin, 'config', 'uploadpack.allowFilter', 'true') + work = os.path.join(self.tmp, 'work') + os.makedirs(work) + git(work, 'init', '-q') + git(work, 'remote', 'add', 'origin', 'file://' + origin) + git(work, 'fetch', '-q', '--depth=1', 'origin', head) + self.assertNotEqual(subprocess.run(['git', 'cat-file', '-e', base + '^{commit}'], cwd=work, + capture_output=True).returncode, 0) + self.assertEqual(self.load(work, base)['check'], CFG['check']) + + def test_base_commit_that_cannot_be_fetched_fails(self): + repo = os.path.join(self.tmp, 'r') + make_repo(repo, [{fcr.CONFIG_PATH: policies()}]) + with self.assertRaises(fcr.ConfigError) as e: + self.load(repo, 'c' * 40) + self.assertIn('cannot fetch the base commit', str(e.exception)) class Handler(http.server.BaseHTTPRequestHandler): @@ -216,30 +408,34 @@ def log_message(self, *a): class Real(unittest.TestCase): - """The script as the workflow runs it, against a local HTTP server: the real client and its error classes.""" + """The script as the workflow runs it (its config from the base commit of a real repository), against a local + HTTP server: the real client and its error classes.""" @classmethod def setUpClass(cls): cls.server = http.server.ThreadingHTTPServer(('127.0.0.1', 0), Handler) threading.Thread(target=cls.server.serve_forever, daemon=True).start() + cls.tmp = tempfile.mkdtemp(prefix='fork-checks-ran-real-') + cls.repo = os.path.join(cls.tmp, 'r') + cls.base, = make_repo(cls.repo, [{fcr.CONFIG_PATH: policies()}]) + cls.quick = os.path.join(cls.tmp, 'q') # a base config with a 2 s cap and 1 s polls + cls.quick_base, = make_repo(cls.quick, [{fcr.CONFIG_PATH: policies(wait_seconds=2, poll_seconds=1)}]) @classmethod def tearDownClass(cls): cls.server.shutdown() cls.server.server_close() + shutil.rmtree(cls.tmp, ignore_errors=True) - def script(self, replies, event='pull_request', cfg=None): + def script(self, replies, event='pull_request', fast=False): Handler.replies, Handler.seen = list(replies), [] - with tempfile.NamedTemporaryFile('w', suffix='.json', delete=False) as f: - json.dump({'fork_checks': cfg or CFG}, f) - try: - return subprocess.run( - [sys.executable, '-I', SCRIPT, '--config', f.name], capture_output=True, text=True, timeout=60, - env={'PATH': os.environ['PATH'], 'EVENT_NAME': event, 'HEAD_SHA': SHA, 'OUTSIDE_PR': 'false', - 'GITHUB_TOKEN': 'tok', 'GITHUB_REPOSITORY': 'o/r', - 'GITHUB_API_URL': f'http://127.0.0.1:{self.server.server_port}'}) - finally: - os.unlink(f.name) + return subprocess.run( + [sys.executable, '-I', SCRIPT], capture_output=True, text=True, timeout=60, + cwd=self.quick if fast else self.repo, + env={'PATH': os.environ['PATH'], 'HOME': self.tmp, **GIT_ENV, **PR_ENV, 'EVENT_NAME': event, + 'BASE_SHA': self.quick_base if fast else self.base, + 'GITHUB_REF': 'refs/heads/main' if event != 'pull_request' else PR_ENV['GITHUB_REF'], + 'GITHUB_TOKEN': 'tok', 'GITHUB_API_URL': f'http://127.0.0.1:{self.server.server_port}'}) def test_push_needs_no_token_or_call(self): r = self.script([], event='push') @@ -248,14 +444,13 @@ def test_push_needs_no_token_or_call(self): self.assertEqual(Handler.seen, []) def test_found(self): - r = self.script([(200, {}, {'workflow_runs': [{'id': 7, 'path': WF}]}), - (200, {}, {'jobs': [job(CFG['check'])]})]) + r = self.script([(200, {}, {'workflow_runs': [wr(7, [])]}), (200, {}, {'jobs': [job(CFG['check'])]})]) self.assertEqual(r.returncode, 0, r.stdout + r.stderr) self.assertEqual(Handler.seen[0][1], 'Bearer tok') self.assertTrue(Handler.seen[0][0].startswith('/repos/o/r/actions/workflows/fork-checks.yml/runs?head_sha=' + SHA), Handler.seen) def test_not_found_fails(self): - r = self.script([(200, {}, {'workflow_runs': []})] * 3, cfg={**CFG, 'wait_seconds': 2, 'poll_seconds': 1}) + r = self.script([(200, {}, {'workflow_runs': []})] * 3, fast=True) self.assertEqual(r.returncode, 1, r.stdout + r.stderr) self.assertIn('::error::fork-checks did not run: check the Actions policy (#401)', r.stdout) @@ -267,15 +462,15 @@ def test_forbidden_fails_at_once(self): def test_rate_limited_then_found(self): r = self.script([(403, {'X-RateLimit-Remaining': '0', 'X-RateLimit-Reset': '0'}, {'message': 'API rate limit'}), - (200, {}, {'workflow_runs': [{'id': 7, 'path': WF}]}), - (200, {}, {'jobs': [job(CFG['check'])]})], cfg={**CFG, 'poll_seconds': 1}) + (200, {}, {'workflow_runs': [wr(7, [])]}), + (200, {}, {'jobs': [job(CFG['check'])]})], fast=True) self.assertEqual(r.returncode, 0, r.stdout + r.stderr) self.assertIn('rate limit', r.stdout) def test_server_error_retried(self): r = self.script([(502, {}, {'message': 'bad gateway'}), - (200, {}, {'workflow_runs': [{'id': 7, 'path': WF}]}), - (200, {}, {'jobs': [job(CFG['check'])]})], cfg={**CFG, 'poll_seconds': 1}) + (200, {}, {'workflow_runs': [wr(7, [])]}), + (200, {}, {'jobs': [job(CFG['check'])]})], fast=True) self.assertEqual(r.returncode, 0, r.stdout + r.stderr) diff --git a/.github/scripts/fork-contracts.test.py b/.github/scripts/fork-contracts.test.py index 3afc665d6aba3..5f423bf7ec724 100644 --- a/.github/scripts/fork-contracts.test.py +++ b/.github/scripts/fork-contracts.test.py @@ -45,9 +45,11 @@ can merge. policies every workflow with a pull_request_target trigger is listed in .github/fork-actions-policies.json (#401): allowed (the fork workflows the repository's - Actions policy allows; each there, with the trigger) or upstream_blocked; fork-checks' - check names are its jobs' names, and fork-tests.yml's fork_checks_ran runs - fork-checks-ran.py with actions: read. + Actions policy allows; each there, with that trigger and no other) or upstream_blocked; + fork-checks' check names are its jobs' names, and fork-tests.yml's fork_checks_ran is + exactly the pinned job (#430): its if:, permissions, the shared checkout and the step + running fork-checks-ran.py with the pinned env, no continue-on-error, and a wait cap + that leaves 5 minutes of its timeout. python3 .github/scripts/fork-contracts.test.py @@ -1620,6 +1622,47 @@ def result_mutant(mutate): # (fork-tests.yml's fork_checks_ran) proves the effect on each pull request. policies_path = root / ".github/fork-actions-policies.json" POLICY_SCRIPT = "python3 -I .github/scripts/fork-checks-ran.py" +# fork_checks_ran as it must be (#430): any other env value could turn it into a silent pass (EVENT_NAME +# from another expression, OUTSIDE_PR hard-coded, the base or head from elsewhere). +POLICY_ENV = { + "EVENT_NAME": "${{ github.event_name }}", + "HEAD_SHA": "${{ github.event.pull_request.head.sha }}", + "BASE_SHA": "${{ github.event.pull_request.base.sha }}", + "OUTSIDE_PR": "${{ github.event.pull_request && github.event.pull_request.head.repo.full_name != github.repository }}", + "DEFAULT_BRANCH": "${{ github.event.repository.default_branch }}", + "GITHUB_TOKEN": "${{ github.token }}", +} +POLICY_CHECKOUT_ENV = {"REF": "${{ github.sha }}", "HISTORY": "tip", "SPARSE": "/.github/scripts/"} +POLICY_JOB_KEYS = {"name", "if", "runs-on", "timeout-minutes", "permissions", "steps"} +POLICY_MARGIN = 300 # seconds of the job's timeout left after the wait cap: checkout, fetch, the last API calls + + +def policy_job_problems(job, fc): + """fork-tests.yml's fork_checks_ran: exactly the pinned shape (the shared checkout step itself is held by the + checkout section).""" + problems = [] + where = "fork-tests.yml: job fork_checks_ran" + if set(job) != POLICY_JOB_KEYS: + problems.append(f"{where} has keys {sorted(set(job))}, not {sorted(POLICY_JOB_KEYS)} (continue-on-error, needs, env ...)") + if expression(job.get("if", "")) != PR_GUARD: + problems.append(f"{where}: if: is not the pull request guard alone") + if job.get("permissions") != {"actions": "read", "contents": "read"}: + problems.append(f"{where}: permissions are not exactly actions: read, contents: read") + steps_ = job.get("steps") or [] + if len(steps_) != 2: + problems.append(f"{where}: not exactly two steps (checkout, script)") + else: + checkout, script = steps_ + if "git init" not in str(checkout.get("run", "")) or checkout.get("env") != POLICY_CHECKOUT_ENV: + problems.append(f"{where}: the first step is not the shared checkout of github.sha with only the script") + if set(script) != {"name", "env", "run"} or str(script.get("run", "")).strip() != POLICY_SCRIPT \ + or script.get("env") != POLICY_ENV: + problems.append(f"{where}: the second step is not `{POLICY_SCRIPT}` with the pinned env and nothing else") + timeout, cap = job.get("timeout-minutes"), fc.get("wait_seconds") + if not isinstance(timeout, int) or not isinstance(cap, int) or cap > timeout * 60 - POLICY_MARGIN: + problems.append(f"{where}: fork_checks.wait_seconds {cap!r} is not at least {POLICY_MARGIN} s below the " + f"timeout ({timeout!r} min), so a timeout would cancel the job before its message") + return problems def policy_problems(policies, wfs, tests_wf): @@ -1633,8 +1676,9 @@ def policy_problems(policies, wfs, tests_wf): for path in allowed: if path not in wfs: problems.append(f"allowed {path} is not in the tree (renamed? the Actions policy names the old path)") - elif "pull_request_target" not in triggers(wfs[path]): - problems.append(f"allowed {path} has no pull_request_target trigger") + elif triggers(wfs[path]) != {"pull_request_target"}: + # Another trigger runs the pull request's own copy of the file: a same-named job of no worth (#430). + problems.append(f"allowed {path} has triggers {sorted(triggers(wfs[path]))}, not pull_request_target alone") if not path.startswith(".github/workflows/fork-"): problems.append(f"allowed {path} is not a fork workflow") for path in blocked: @@ -1647,10 +1691,7 @@ def policy_problems(policies, wfs, tests_wf): names = {str(j.get("name")) for j in (wfs.get(fc["workflow"], {}).get("jobs") or {}).values()} problems += [f"{fc['workflow']} has no job named {fc.get(k)!r} (fork_checks.{k})" for k in ("check", "outside_check") if fc.get(k) not in names] - job = (tests_wf.get("jobs") or {}).get("fork_checks_ran") or {} - if not any(str(s.get("run", "")).strip() == POLICY_SCRIPT for s in job.get("steps") or []) \ - or (job.get("permissions") or {}).get("actions") != "read": - problems.append(f"fork-tests.yml: job fork_checks_ran does not run `{POLICY_SCRIPT}` with actions: read") + problems += policy_job_problems((tests_wf.get("jobs") or {}).get("fork_checks_ran") or {}, fc) return problems @@ -1686,6 +1727,32 @@ def policy_mutant(mutate): "fork_checks_ran without actions: read": lambda p, w, t: t["jobs"]["fork_checks_ran"]["permissions"].pop("actions"), "fork_checks_ran not running the script": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1].__setitem__( "run", POLICY_SCRIPT + " || true"), + "fork-checks.yml also on push": lambda p, w, t: w[FC][True].__setitem__("push", None), + "fork-checks.yml also on pull_request": lambda p, w, t: w[FC][True].__setitem__("pull_request", None), + "fork_checks_ran job may fail quietly": lambda p, w, t: t["jobs"]["fork_checks_ran"].__setitem__( + "continue-on-error", True), + "fork_checks_ran step may fail quietly": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1].__setitem__( + "continue-on-error", True), + "fork_checks_ran with another if": lambda p, w, t: t["jobs"]["fork_checks_ran"].__setitem__( + "if", "${{ " + PR_GUARD + " && github.actor != 'x' }}"), + "fork_checks_ran with OUTSIDE_PR hard-coded": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"] + .__setitem__("OUTSIDE_PR", "false"), + "fork_checks_ran with EVENT_NAME from elsewhere": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"] + .__setitem__("EVENT_NAME", "push"), + "fork_checks_ran with the base from the head": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"] + .__setitem__("BASE_SHA", "${{ github.event.pull_request.head.sha }}"), + "fork_checks_ran with an extra env": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"] + .__setitem__("BASH_ENV", "x"), + "fork_checks_ran with a job env": lambda p, w, t: t["jobs"]["fork_checks_ran"].__setitem__("env", {"X": "1"}), + "fork_checks_ran with a step between": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"].insert( + 1, {"run": "echo 'EVENT_NAME=push' >> $GITHUB_ENV"}), + "fork_checks_ran checking out with actions/checkout": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"] + .__setitem__(0, {"uses": "actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803"}), + "fork_checks_ran with write permissions": lambda p, w, t: t["jobs"]["fork_checks_ran"]["permissions"].__setitem__( + "actions", "write"), + "a wait cap equal to the job timeout": lambda p, w, t: p["fork_checks"].__setitem__( + "wait_seconds", t["jobs"]["fork_checks_ran"]["timeout-minutes"] * 60), + "a job timeout cut below the cap": lambda p, w, t: t["jobs"]["fork_checks_ran"].__setitem__("timeout-minutes", 10), }.items(): check(f"policies: {label} fails", policy_mutant(mutate)) diff --git a/.github/workflows/fork-follow.yml b/.github/workflows/fork-follow.yml index d05c33aee90b9..ed4fdb41c387d 100644 --- a/.github/workflows/fork-follow.yml +++ b/.github/workflows/fork-follow.yml @@ -100,7 +100,7 @@ jobs: # .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git # submodule foreach`, which fails on them, sparse checkout or not. The repository is public, # so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this - # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout. + # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran. - name: Checkout shell: bash env: @@ -176,7 +176,7 @@ jobs: # .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git # submodule foreach`, which fails on them, sparse checkout or not. The repository is public, # so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this - # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout. + # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran. - name: Checkout shell: bash env: @@ -337,7 +337,7 @@ jobs: # .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git # submodule foreach`, which fails on them, sparse checkout or not. The repository is public, # so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this - # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout. + # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran. - name: Checkout shell: bash env: @@ -443,7 +443,7 @@ jobs: # .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git # submodule foreach`, which fails on them, sparse checkout or not. The repository is public, # so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this - # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout. + # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran. - name: Checkout shell: bash env: diff --git a/.github/workflows/fork-release.yml b/.github/workflows/fork-release.yml index 4163540a18ec5..0287ebde326e7 100644 --- a/.github/workflows/fork-release.yml +++ b/.github/workflows/fork-release.yml @@ -101,7 +101,7 @@ jobs: # .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git # submodule foreach`, which fails on them, sparse checkout or not. The repository is public, # so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this - # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout. + # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran. - name: Checkout shell: bash env: @@ -171,7 +171,7 @@ jobs: # .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git # submodule foreach`, which fails on them, sparse checkout or not. The repository is public, # so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this - # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout. + # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran. - name: Checkout shell: bash env: @@ -405,7 +405,7 @@ jobs: # .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git # submodule foreach`, which fails on them, sparse checkout or not. The repository is public, # so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this - # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout. + # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran. - name: Checkout shell: bash env: @@ -545,7 +545,7 @@ jobs: # .gitmodules, and checkout's credential cleanup (persist-credentials: false) runs `git # submodule foreach`, which fails on them, sparse checkout or not. The repository is public, # so no token is used. Every job of fork-follow.yml and fork-release.yml checks out with this - # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout. + # same step (fork-contracts.test.py); fork-tests.yml keeps actions/checkout but in fork_checks_ran. - name: Checkout shell: bash env: diff --git a/.github/workflows/fork-tests.yml b/.github/workflows/fork-tests.yml index 1c2ec6af25161..4bed166df9f5f 100644 --- a/.github/workflows/fork-tests.yml +++ b/.github/workflows/fork-tests.yml @@ -245,8 +245,12 @@ jobs: # fork-checks.yml runs on pull_request_target, which GitHub blocks unless the repository's Actions # policy allows it (#401, .github/fork-actions-policies.json); its checks are not required, so a - # blocked run would pass unseen. This job waits (capped) for its check run on the pull request's - # head, and fork_ci_result needs it. Other events: the script logs a skip and passes. + # blocked, failed or cancelled run would pass unseen. This job waits (capped) for its check run on + # the pull request's head to pass, judged by the config on the pull request's base commit (#430), + # and fork_ci_result needs it. Other events: a called run (release) logs a skip; a direct run passes + # only on the default branch. Checks out without a token (the shared plain-git step: actions/checkout + # with persist-credentials: false fails on upstream's gitlinks); the script fetches the base commit. + # fork-contracts.test.py pins this job (if, env, steps, no continue-on-error, cap below the timeout). fork_checks_ran: name: Fork checks ran if: ${{ (!github.event.pull_request || github.event.pull_request.head.repo.full_name == github.repository) }} @@ -258,18 +262,36 @@ jobs: contents: read steps: - name: Checkout - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 - with: - sparse-checkout: | - /.github/fork-actions-policies.json - /.github/scripts/fork-checks-ran.py - sparse-checkout-cone-mode: false - - - name: Require a fork-checks check run on the head commit + shell: bash + env: + REF: ${{ github.sha }} + HISTORY: tip + SPARSE: /.github/scripts/ + run: | + set -euo pipefail + [[ -n "$REF" ]] || { echo "::error::No commit or branch to check out." >&2; exit 1; } + case "$HISTORY" in + full) history=(--filter=blob:none) ;; + tip) history=(--depth=1 --filter=blob:none) ;; + *) echo "::error::HISTORY is full or tip, not '$HISTORY'." >&2; exit 1 ;; + esac + read -r -a sparse <<< "$SPARSE" + git init -q + git remote add origin "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY" + # A name is a branch, fetched as refs/heads/: a bare name would take a tag of the same + # name first. + if [[ "$REF" =~ ^[0-9a-f]{40}$ ]]; then src="$REF"; else src="refs/heads/$REF"; fi + git fetch -q --no-tags --no-recurse-submodules "${history[@]}" origin "$src" + if (( ${#sparse[@]} )); then git sparse-checkout set --no-cone "${sparse[@]}"; fi + git checkout -q --detach FETCH_HEAD + + - name: Require fork-checks to pass on the head commit env: EVENT_NAME: ${{ github.event_name }} HEAD_SHA: ${{ github.event.pull_request.head.sha }} + BASE_SHA: ${{ github.event.pull_request.base.sha }} OUTSIDE_PR: ${{ github.event.pull_request && github.event.pull_request.head.repo.full_name != github.repository }} + DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} GITHUB_TOKEN: ${{ github.token }} run: python3 -I .github/scripts/fork-checks-ran.py diff --git a/fork-ci/actions/head-check/README.md b/fork-ci/actions/head-check/README.md index 41c7e9c5dfa48..5efe1cb31d8c1 100644 --- a/fork-ci/actions/head-check/README.md +++ b/fork-ci/actions/head-check/README.md @@ -298,7 +298,8 @@ Setup (the repository owner's steps; nothing here creates them): blocked again, and a blocked run is only a failed run that nothing requires; this fork lists its allowed files in `.github/fork-actions-policies.json` and fails its required check when - `fork-checks.yml` did not run (`fork-checks-ran.py`, issue 401). + `fork-checks.yml` did not run and pass (`fork-checks-ran.py`, issues 401 + and 430). 5. Dispatch it once without a number (the backfill), so open pull requests get the check under its name. 6. Bind the required check to the App: in a ruleset, the required status From 343a03b8f5d93c76a60e3466599c080477284df8 Mon Sep 17 00:00:00 2001 From: scuttletoby <123843277+scuttletoby@users.noreply.github.com> Date: Tue, 17 Nov 2026 23:51:17 +0000 Subject: [PATCH 2/2] fork_checks_ran: count only this pull request's runs; cap covers git (#433 fix round 1) - Only fork-checks runs from this repository whose pull_requests list holds PR_NUMBER count (pinned env). A newer run of another pull request on the same head, or of an outside fork's pull request (empty list), no longer decides; an empty or missing list never passes. - The wait cap starts before the git calls that read the base config; git 60 s per call. The contract checks the worst case (checkout, cap or git, one last poll) against the job timeout, using the script's own limits. - The skipped message no longer offers a re-run that cannot help. - Tests for the bool guard in config_problems and for the checkout env pin. - Docstrings: pull_request_target runs the base branch's copy. Co-Authored-By: Claude Opus 5.5 --- .github/scripts/fork-checks-ran.py | 85 +++++++++++----- .github/scripts/fork-checks-ran.test.py | 127 ++++++++++++++++++++++-- .github/scripts/fork-contracts.test.py | 61 ++++++++++-- .github/workflows/fork-tests.yml | 8 +- 4 files changed, 233 insertions(+), 48 deletions(-) diff --git a/.github/scripts/fork-checks-ran.py b/.github/scripts/fork-checks-ran.py index bb8fbfa5fea7d..abdd9dec015f6 100644 --- a/.github/scripts/fork-checks-ran.py +++ b/.github/scripts/fork-checks-ran.py @@ -4,40 +4,54 @@ fork-checks.yml runs on pull_request_target, which GitHub blocks from 2026-12-02 unless a repository Actions policy allows it for that file (.github/fork-actions-policies.json). Its checks are not required, so a blocked, failed or cancelled run would merge unseen. This script makes the required fork-ci-result notice: on the head commit, the newest -pull_request_target run of the configured workflow file must have a job (= check run) of the configured name (`check`) -whose latest attempt concluded success. Runs of other events do not count: they run the pull request's own copy of the -file. It waits while that job is queued or running; failed, cancelled or skipped fails at once, and so does a finished -run without the job (a blocked event leaves one). The Actions API needs only actions: read, which every caller of -fork-tests.yml grants. +pull_request_target run of the configured workflow file that belongs to THIS pull request must have a job (= check run) +of the configured name (`check`) whose latest attempt concluded success. Runs of other events do not count: they run +the pull request's own copy of the file. A run belongs to this pull request when its head repository is this +repository and its pull_requests list holds PR_NUMBER (#433): anyone can open a pull request from their own fork at +the same public commit (its run skips the job), and a second pull request at the same commit may target a base branch +whose fork-checks.yml is a no-op. GitHub leaves pull_requests empty on runs of a pull request from another +repository, so such a run is never counted (fail closed). Two pull requests from the same head branch may both be +listed on each other's runs; both come from someone who can push to this repository. It waits while that job is queued or running; failed, +cancelled or skipped fails at once, and so does a finished run without the job (a blocked event leaves one). The +Actions API needs only actions: read, which every caller of fork-tests.yml grants. The config is read from the pull request's base commit (BASE_SHA, fetched when the checkout lacks it), not from the -checkout: pull_request_target runs main's copy of fork-checks.yml, so a pull request that renames its job or file is -judged by the names main's copy produces, and cannot edit the config its own check reads. Missing or undecodable -there: fail. +checkout: pull_request_target runs the base branch's copy of fork-checks.yml, so a pull request that renames its job +or file is judged by the names the base branch's copy produces, and cannot edit the config its own check reads. +Missing or undecodable there: fail. -It polls every poll_seconds for at most wait_seconds (below the job's timeout, so the message shows), then fails. An -API error fails closed: a rate limit or a server error is retried within the same cap (and named in the failure if it -lasts), any other error (403, 404 ...) fails at once. A pull request from another fork fails at once (the job's if: -skips those; fork-checks' only job for them always fails). +It polls every poll_seconds for at most wait_seconds, counted from before the config is read (its git calls count), +then fails. The job's timeout leaves room for the checkout, the cap and one last poll: fork-contracts.test.py checks +it with GIT_TIMEOUT, GIT_CALLS, API_TIMEOUT and API_CALLS_PER_POLL below, so the message shows. An API error fails +closed: a rate limit or a server error is retried within the same cap (and named in the failure if it lasts), any +other error (403, 404 ...) fails at once. A pull request from another fork fails at once (the job's if: skips those; +fork-checks' only job for them always fails). Not a pull_request event: a run that another workflow called (GITHUB_WORKFLOW_REF names the caller) has nothing to check, since its check runs are named " / ... / fork-ci-result", never the required "fork-ci-result". A direct run (push, workflow_dispatch) passes only on the default branch: on any other branch its passing fork-ci-result would land on that branch's tip, which may be an open pull request's head that fork-checks never passed (#430). -Environment: EVENT_NAME, HEAD_SHA, BASE_SHA, OUTSIDE_PR (true/false), DEFAULT_BRANCH, GITHUB_REF, GITHUB_WORKFLOW_REF, -GITHUB_TOKEN (actions: read), GITHUB_REPOSITORY, GITHUB_API_URL. Test: fork-checks-ran.test.py. +Environment: EVENT_NAME, HEAD_SHA, BASE_SHA, PR_NUMBER, OUTSIDE_PR (true/false), DEFAULT_BRANCH, GITHUB_REF, +GITHUB_WORKFLOW_REF, GITHUB_TOKEN (actions: read), GITHUB_REPOSITORY, GITHUB_API_URL. Test: fork-checks-ran.test.py. """ import http.client, json, os, re, subprocess, sys, time, urllib.error, urllib.parse, urllib.request CONFIG_PATH = '.github/fork-actions-policies.json' TESTS_WORKFLOW = '.github/workflows/fork-tests.yml' # the workflow whose fork-ci-result a ruleset requires SHA = re.compile(r'[0-9a-f]{40}') +NUMBER = re.compile(r'[1-9][0-9]*') +GIT_TIMEOUT, GIT_CALLS = 60, 3 # seconds per git call; at most cat-file, fetch, show +API_TIMEOUT, API_CALLS_PER_POLL = 30, 2 # seconds per API call; one poll: the run list, then one run's jobs NOT_RAN = 'fork-checks did not run: check the Actions policy (#401)' WAY_OUT = ('Once that is fixed, push again or close and reopen the pull request (a blocked run cannot be re-run), ' 'then this job runs again.') RERUN = ('Fix what it reports and push again; if it was cancelled or timed out, re-run it (or push again, or close and ' 'reopen the pull request), then re-run the failed jobs of this run.') +SKIPPED = ('fork-checks skips that job only when it sees the pull request as coming from another repository (or the ' + 'base branch\'s fork-checks.yml changed its if:), and a re-run skips it again (it replays the same event). ' + 'Check that the head branch is still in this repository and look at the base branch\'s fork-checks.yml, then ' + 'push again or close and reopen the pull request. A pull request from another repository needs a maintainer.') class Transient(Exception): @@ -67,7 +81,7 @@ def get(self, path): 'Accept': 'application/vnd.github+json', 'User-Agent': 'fork-checks-ran', 'Authorization': f'Bearer {self.token}'}) try: - with urllib.request.urlopen(req, timeout=30) as r: + with urllib.request.urlopen(req, timeout=API_TIMEOUT) as r: return json.loads(r.read()) except urllib.error.HTTPError as e: try: @@ -96,7 +110,7 @@ def base_config(sha, cwd=None, env=None): """fork_checks of CONFIG_PATH on commit sha, fetched from origin when the checkout lacks it.""" def git(*args): try: - return subprocess.run(['git', *args], cwd=cwd, env=env, capture_output=True, timeout=180) + return subprocess.run(['git', *args], cwd=cwd, env=env, capture_output=True, timeout=GIT_TIMEOUT) except (OSError, subprocess.TimeoutExpired) as e: raise ConfigError(f'git {args[0]} failed: {e}') from None @@ -125,12 +139,22 @@ def config_problems(cfg): return problems -def look(api, sha, name, workflow): - """What the newest pull_request_target run of the workflow on sha says about its job `name`, latest attempt: - None (no such run yet), ('going', url), ('blocked', run url: finished without the job) or (conclusion, url).""" +def of_this_pull_request(run, repo, number): + """A run from this repository whose pull_requests list holds this pull request. An empty or missing list (what + GitHub gives a pull request from another repository) is not this pull request's run: never counted.""" + head, prs = run.get('head_repository'), run.get('pull_requests') + return (isinstance(head, dict) and head.get('full_name') == repo and isinstance(prs, list) + and any(isinstance(p, dict) and p.get('number') == number for p in prs)) + + +def look(api, sha, name, workflow, repo, number): + """What the newest pull_request_target run of the workflow on sha for pull request `number` from `repo` says + about its job `name`, latest attempt: None (no such run yet), ('going', url), ('blocked', run url: finished + without the job) or (conclusion, url). At most API_CALLS_PER_POLL calls.""" runs = api.get(f'actions/workflows/{urllib.parse.quote(os.path.basename(workflow))}/runs?' + urllib.parse.urlencode({'head_sha': sha, 'per_page': 100}))['workflow_runs'] - mine = [r for r in runs if r['path'].split('@')[0] == workflow and r.get('event') == 'pull_request_target'] + mine = [r for r in runs if r['path'].split('@')[0] == workflow and r.get('event') == 'pull_request_target' + and of_this_pull_request(r, repo, number)] for run in sorted(mine, key=lambda r: int(r['id']), reverse=True): jobs = api.get(f'actions/runs/{int(run["id"])}/jobs?filter=latest&per_page=100')['jobs'] named = [j for j in jobs if j['name'] == name] @@ -172,8 +196,13 @@ def check(env, api, config, now=time.time, sleep=time.sleep, say=print): if event != 'pull_request': return not_a_pull_request(env, say) sha, base, outside = env.get('HEAD_SHA', ''), env.get('BASE_SHA', ''), env.get('OUTSIDE_PR', '') + number, repo = env.get('PR_NUMBER', ''), env.get('GITHUB_REPOSITORY', '') problems = [f'{k} {v!r} is not a commit id' for k, v in (('HEAD_SHA', sha), ('BASE_SHA', base)) if not SHA.fullmatch(v)] + if not NUMBER.fullmatch(number): + problems.append(f'PR_NUMBER {number!r} is not a pull request number') + if not repo: + problems.append('GITHUB_REPOSITORY is empty') if outside not in ('true', 'false'): problems.append(f'OUTSIDE_PR {outside!r} is not true or false') if problems: @@ -183,6 +212,7 @@ def check(env, api, config, now=time.time, sleep=time.sleep, say=print): say('::error::outside pull request: fork-checks does not check it (its Outside pull request job fails); ' 'a maintainer must review') return 1 + start = now() # the cap counts the git calls that read the config try: cfg = config(base) except ConfigError as e: @@ -193,14 +223,14 @@ def check(env, api, config, now=time.time, sleep=time.sleep, say=print): say(f'::error::{CONFIG_PATH} on the base commit {base}: ' + '; '.join(problems)) return 1 name, workflow, poll = cfg['check'], cfg['workflow'], cfg['poll_seconds'] - deadline = now() + cfg['wait_seconds'] - say(f'Waiting for check run "{name}" of {workflow} on {sha} to pass (at most {cfg["wait_seconds"]} s; ' - f'config from the base commit {base}).') + deadline = start + cfg['wait_seconds'] + say(f'Waiting for check run "{name}" of {workflow} on {sha} for pull request #{number} to pass (at most ' + f'{cfg["wait_seconds"]} s; config from the base commit {base}).') last_error = None while True: wait = poll try: - found = look(api, sha, name, workflow) + found = look(api, sha, name, workflow, repo, int(number)) last_error = None except Fatal as e: say(f'::error::fork-checks could not be confirmed (failing closed): GitHub API error: {e}. A 404: ' @@ -222,7 +252,7 @@ def check(env, api, config, now=time.time, sleep=time.sleep, say=print): say(f'::error::{NOT_RAN}: its newest run on {sha} finished without a "{name}" job ({url}). {WAY_OUT}') return 1 if state == 'skipped': - say(f'::error::fork-checks did not check {sha}: its check run "{name}" was skipped ({url}). {RERUN}') + say(f'::error::fork-checks did not check {sha}: its check run "{name}" was skipped ({url}). {SKIPPED}') return 1 if state not in (None, 'going'): say(f'::error::fork-checks did not pass on {sha}: check run "{name}" of {workflow} ended {state} ' @@ -237,8 +267,9 @@ def check(env, api, config, now=time.time, sleep=time.sleep, say=print): say(f'::error::fork-checks did not finish on {sha} within {cfg["wait_seconds"]} s (check run "{name}" ' f'of {workflow}: {url}). Once it has passed, re-run the failed jobs of this run.') else: - say(f'::error::{NOT_RAN}: no pull_request_target run of {workflow} with a check run "{name}" on {sha} ' - f'after {cfg["wait_seconds"]} s. {WAY_OUT}') + say(f'::error::{NOT_RAN}: no pull_request_target run of {workflow} for pull request #{number} from ' + f'this repository with a check run "{name}" on {sha} after {cfg["wait_seconds"]} s (runs of other ' + f'pull requests on the same commit do not count). {WAY_OUT}') return 1 sleep(wait) diff --git a/.github/scripts/fork-checks-ran.test.py b/.github/scripts/fork-checks-ran.test.py index 571c30c79f4e9..4951242af78c3 100644 --- a/.github/scripts/fork-checks-ran.test.py +++ b/.github/scripts/fork-checks-ran.test.py @@ -15,7 +15,7 @@ WF = '.github/workflows/fork-checks.yml' CFG = {'workflow': WF, 'check': 'No database schema changes', 'outside_check': 'Outside pull request', 'wait_seconds': 600, 'poll_seconds': 20} -PR_ENV = {'EVENT_NAME': 'pull_request', 'HEAD_SHA': SHA, 'BASE_SHA': BASE, 'OUTSIDE_PR': 'false', +PR_ENV = {'EVENT_NAME': 'pull_request', 'HEAD_SHA': SHA, 'BASE_SHA': BASE, 'OUTSIDE_PR': 'false', 'PR_NUMBER': '7', 'DEFAULT_BRANCH': 'main', 'GITHUB_REPOSITORY': 'o/r', 'GITHUB_REF': 'refs/pull/7/merge', 'GITHUB_WORKFLOW_REF': 'o/r/.github/workflows/fork-tests.yml@refs/pull/7/merge'} GIT_ENV = {'GIT_CONFIG_NOSYSTEM': '1', 'GIT_CONFIG_GLOBAL': os.devnull, 'GIT_TERMINAL_PROMPT': '0', @@ -28,8 +28,11 @@ def job(name, status='completed', conclusion='success'): 'html_url': f'https://github.invalid/job/{abs(hash(name)) % 1000}'} -def wr(rid, jobs, path=WF, event='pull_request_target', status='completed'): - return {'id': rid, 'path': path, 'event': event, 'status': status, 'jobs': jobs} +def wr(rid, jobs, path=WF, event='pull_request_target', status='completed', prs=(7,), head_repo='o/r'): + """A run of fork-checks.yml on SHA; by default one of pull request #7 (PR_ENV) from this repository.""" + return {'id': rid, 'path': path, 'event': event, 'status': status, 'jobs': jobs, + 'pull_requests': [{'number': n, 'head': {'sha': SHA}} for n in prs], + 'head_repository': {'full_name': head_repo}} class Clock: @@ -184,6 +187,10 @@ def test_only_skipped_fails_at_once(self): self.assertEqual(code, 1, out) self.assertIn('skipped', out) self.assertEqual(clock.slept, []) + # A re-run replays the same event, so it skips again: the message must not offer one (#433 fix round 1). + self.assertNotIn('re-run it', out) + self.assertIn('a re-run skips it again', out) + self.assertIn('push again', out) def test_newest_run_decides(self): # An `edited` event runs fork-checks again on the same head: the newest run is the one that counts. @@ -201,13 +208,60 @@ def test_reads_the_latest_attempt_only(self): # --- which runs count ------------------------------------------------------------------------------------------- def test_only_pull_request_target_runs_count(self): # A pull request that adds push: or pull_request: to its own fork-checks.yml gets a same-named job from its - # own copy (#430 item 2). Only pull_request_target runs main's copy. + # own copy (#430 item 2). Only pull_request_target runs the base branch's copy. for event in ('pull_request', 'push', 'workflow_dispatch', 'schedule', None): fake = Fake([[wr(4, [job(CFG['check'])], event=event)]]) code, out, _ = check(fake) self.assertEqual(code, 1, f'{event}: {out}') self.assertNotIn('actions/runs/4/jobs', ' '.join(fake.paths), event) + # --- only this pull request's runs count (#433 fix round 1, finding 1 of both lenses) -------------------------- + def test_a_newer_passing_run_of_another_pull_request_on_the_same_head_does_not_pass(self): + # A second pull request at the same commit, against a base branch whose fork-checks.yml is a no-op. + for runs in ([wr(3, [job(CFG['check'], 'completed', 'failure')]), wr(9, [job(CFG['check'])], prs=(11,))], + [wr(9, [job(CFG['check'])], prs=(11,))]): + fake = Fake([runs]) + code, out, _ = check(fake) + self.assertEqual(code, 1, out) + self.assertNotIn('actions/runs/9/jobs', ' '.join(fake.paths)) + + def test_a_newer_skipped_run_of_an_outside_pull_request_on_the_same_head_does_not_turn_it_red(self): + # Anyone can open a pull request from their own fork at a public commit; its run skips the job. + for outsider in (wr(9, [job(CFG['check'], 'completed', 'skipped')], prs=(), head_repo='x/r'), + wr(9, [job(CFG['check'], 'completed', 'skipped')], prs=(12,), head_repo='x/r'), + wr(9, [job(CFG['check'], 'completed', 'skipped')], prs=(7,), head_repo='x/r')): + fake = Fake([[outsider, wr(3, [job(CFG['check'])])]]) + code, out, _ = check(fake) + self.assertEqual(code, 0, out) + self.assertNotIn('actions/runs/9/jobs', ' '.join(fake.paths)) + + def test_a_run_with_an_empty_or_missing_pull_request_list_never_passes(self): + # GitHub leaves pull_requests empty on runs of a pull request from another repository: such a run cannot be + # tied to this pull request, so it does not count, and with no other run the job fails at the cap. + bare = wr(5, [job(CFG['check'])], prs=()) + for run in (bare, {**bare, 'pull_requests': None}, {k: v for k, v in bare.items() if k != 'pull_requests'}, + {**bare, 'head_repository': None}, {k: v for k, v in bare.items() if k != 'head_repository'}, + {**bare, 'pull_requests': [{'number': '7'}]}, {**bare, 'pull_requests': [{}]}): + fake = Fake([[run]]) + code, out, clock = check(fake) + self.assertEqual(code, 1, (run, out)) + self.assertIn('fork-checks did not run', out) + self.assertNotIn('actions/runs/5/jobs', ' '.join(fake.paths)) + + def test_no_run_of_this_pull_request_fails_with_a_clear_message(self): + code, out, _ = check(Fake([[wr(9, [job(CFG['check'])], prs=(11,))]])) + self.assertEqual(code, 1, out) + self.assertIn('for pull request #7 from this repository', out) + self.assertIn('runs of other pull requests on the same commit do not count', out) + self.assertIn('close and reopen', out) + + def test_one_poll_makes_at_most_the_budgeted_api_calls(self): + # The contract's timeout budget counts API_CALLS_PER_POLL calls after the cap (fork-contracts.test.py). + runs = [wr(20 - i, [job(CFG['check'])], prs=(11,)) for i in range(5)] + [wr(3, [job(CFG['check'])])] + fake = Fake([runs]) + self.assertEqual(check(fake)[0], 0) + self.assertLessEqual(len(fake.paths), fcr.API_CALLS_PER_POLL) + def test_a_pull_request_target_run_next_to_a_no_op_one_decides(self): fake = Fake([[wr(8, [job(CFG['check'])], event='push'), wr(5, [job(CFG['check'], 'completed', 'failure')])]]) @@ -309,12 +363,48 @@ def passing(): return Fake([[wr(1, [job(CFG['check'])])]]) self.assertEqual(check(passing())[0], 0) for env, what in (({'HEAD_SHA': 'main'}, 'HEAD_SHA'), ({'BASE_SHA': ''}, 'BASE_SHA'), - ({'BASE_SHA': 'b' * 39}, 'BASE_SHA'), ({'OUTSIDE_PR': 'maybe'}, 'OUTSIDE_PR')): + ({'BASE_SHA': 'b' * 39}, 'BASE_SHA'), ({'OUTSIDE_PR': 'maybe'}, 'OUTSIDE_PR'), + ({'PR_NUMBER': ''}, 'PR_NUMBER'), ({'PR_NUMBER': '7a'}, 'PR_NUMBER'), + ({'PR_NUMBER': '0'}, 'PR_NUMBER'), ({'PR_NUMBER': '07'}, 'PR_NUMBER'), + ({'GITHUB_REPOSITORY': ''}, 'GITHUB_REPOSITORY')): code, out, _ = check(passing(), **env) self.assertEqual((code, what in out), (1, True), (env, out)) self.assertEqual(check(passing(), cfg={**CFG, 'wait_seconds': 0})[0], 1) self.assertEqual(check(passing(), cfg={k: v for k, v in CFG.items() if k != 'check'})[0], 1) + def test_a_true_or_false_wait_is_not_a_number(self): + # bool is an int in Python: `wait_seconds: true` would be a 1 s cap (#433 fix round 1, state finding 5). + for k in ('wait_seconds', 'poll_seconds'): + for v in (True, False): + self.assertEqual(fcr.config_problems({**CFG, k: v}), [f'fork_checks.{k} is not a positive whole number']) + code, out, _ = check(Fake([[wr(1, [job(CFG['check'])])]]), cfg={**CFG, k: v}) + self.assertEqual(code, 1, (k, v, out)) + + # --- time: the cap covers reading the config (#433 fix round 1, nit of both lenses) -------------------------------- + def test_the_cap_counts_from_before_the_config_is_read(self): + clock, out = Clock(), [] + + def slow_config(base): + clock.t += 590 # a slow git fetch of the base commit + return dict(CFG) + code = fcr.check(PR_ENV, Fake([[]]), slow_config, clock.now, clock.sleep, out.append) + self.assertEqual(code, 1, out) + self.assertLessEqual(clock.t - 1000.0, CFG['wait_seconds'], out) + + def test_api_calls_use_the_budgeted_timeout(self): + seen = [] + + def urlopen(req, timeout=None): + seen.append(timeout) + raise fcr.urllib.error.URLError('down') + real, fcr.urllib.request.urlopen = fcr.urllib.request.urlopen, urlopen + try: + with self.assertRaises(fcr.Transient): + fcr.GitHub('t', 'o/r', 'http://127.0.0.1:9').get('x') + finally: + fcr.urllib.request.urlopen = real + self.assertEqual(seen, [fcr.API_TIMEOUT]) + def test_real_config_has_the_fields(self): with open(CONFIG) as f: cfg = json.load(f)['fork_checks'] @@ -322,8 +412,8 @@ def test_real_config_has_the_fields(self): class BaseConfig(unittest.TestCase): - """The config comes from the pull request's base commit (#430 item 1): fork-checks runs main's copy, so a pull - request that renames its job or file must still be judged by the names main's copy produces.""" + """The config comes from the pull request's base commit (#430 item 1): fork-checks runs the base branch's copy, + so a pull request that renames its job or file must still be judged by the names that copy produces.""" def setUp(self): self.tmp = tempfile.mkdtemp(prefix='fork-checks-ran-') @@ -382,6 +472,29 @@ def test_base_commit_is_fetched_when_the_checkout_lacks_it(self): capture_output=True).returncode, 0) self.assertEqual(self.load(work, base)['check'], CFG['check']) + def test_git_calls_stay_within_the_budget(self): + # The contract's timeout budget counts GIT_CALLS calls of GIT_TIMEOUT s each (fork-contracts.test.py). + origin = os.path.join(self.tmp, 'origin') + base, head = make_repo(origin, [{fcr.CONFIG_PATH: policies()}, {fcr.CONFIG_PATH: policies(check='Schema')}]) + git(origin, 'config', 'uploadpack.allowAnySHA1InWant', 'true') + git(origin, 'config', 'uploadpack.allowFilter', 'true') + work = os.path.join(self.tmp, 'work') + os.makedirs(work) + git(work, 'init', '-q') + git(work, 'remote', 'add', 'origin', 'file://' + origin) + git(work, 'fetch', '-q', '--depth=1', 'origin', head) + timeouts, real = [], fcr.subprocess.run + + def run(*a, **kw): + timeouts.append(kw.get('timeout')) + return real(*a, **kw) + fcr.subprocess.run = run + try: + self.assertEqual(self.load(work, base)['check'], CFG['check']) + finally: + fcr.subprocess.run = real + self.assertEqual(timeouts, [fcr.GIT_TIMEOUT] * fcr.GIT_CALLS) + def test_base_commit_that_cannot_be_fetched_fails(self): repo = os.path.join(self.tmp, 'r') make_repo(repo, [{fcr.CONFIG_PATH: policies()}]) diff --git a/.github/scripts/fork-contracts.test.py b/.github/scripts/fork-contracts.test.py index 47125765ac078..ccf48cc42454a 100644 --- a/.github/scripts/fork-contracts.test.py +++ b/.github/scripts/fork-contracts.test.py @@ -48,8 +48,9 @@ Actions policy allows; each there, with that trigger and no other) or upstream_blocked; fork-checks' check names are its jobs' names, and fork-tests.yml's fork_checks_ran is exactly the pinned job (#430): its if:, permissions, the shared checkout and the step - running fork-checks-ran.py with the pinned env, no continue-on-error, and a wait cap - that leaves 5 minutes of its timeout. + running fork-checks-ran.py with the pinned env (PR_NUMBER too, #433), no + continue-on-error, and a timeout that holds the worst case: the checkout, the wait cap (or + the script's git calls, if longer) and one last poll, from the script's own limits. python3 .github/scripts/fork-contracts.test.py @@ -57,6 +58,7 @@ """ import copy +import importlib.util import json import os import pathlib @@ -1637,16 +1639,30 @@ def result_mutant(mutate): "EVENT_NAME": "${{ github.event_name }}", "HEAD_SHA": "${{ github.event.pull_request.head.sha }}", "BASE_SHA": "${{ github.event.pull_request.base.sha }}", + "PR_NUMBER": "${{ github.event.pull_request.number }}", "OUTSIDE_PR": "${{ github.event.pull_request && github.event.pull_request.head.repo.full_name != github.repository }}", "DEFAULT_BRANCH": "${{ github.event.repository.default_branch }}", "GITHUB_TOKEN": "${{ github.token }}", } POLICY_CHECKOUT_ENV = {"REF": "${{ github.sha }}", "HISTORY": "tip", "SPARSE": "/.github/scripts/"} POLICY_JOB_KEYS = {"name", "if", "runs-on", "timeout-minutes", "permissions", "steps"} -POLICY_MARGIN = 300 # seconds of the job's timeout left after the wait cap: checkout, fetch, the last API calls +POLICY_CHECKOUT_BUDGET = 180 # seconds for the token-free sparse checkout of one commit (a few seconds in CI) +_fcr_spec = importlib.util.spec_from_file_location("fork_checks_ran", root / ".github/scripts/fork-checks-ran.py") +_fcr = importlib.util.module_from_spec(_fcr_spec) +sys.dont_write_bytecode = True # no __pycache__ in the tree +_fcr_spec.loader.exec_module(_fcr) +# The script's own limits (fork-checks-ran.test.py checks it keeps to them): its cap counts its git calls. +POLICY_LIMITS = {k: getattr(_fcr, k) for k in ("GIT_TIMEOUT", "GIT_CALLS", "API_TIMEOUT", "API_CALLS_PER_POLL")} -def policy_job_problems(job, fc): +def policy_worst_case(cap, limits): + """Seconds fork_checks_ran can take: the checkout, then the cap (the git calls run inside it, or past it if they + take longer), then one last poll after the deadline.""" + return (POLICY_CHECKOUT_BUDGET + max(cap, limits["GIT_TIMEOUT"] * limits["GIT_CALLS"]) + + limits["API_TIMEOUT"] * limits["API_CALLS_PER_POLL"]) + + +def policy_job_problems(job, fc, limits=POLICY_LIMITS): """fork-tests.yml's fork_checks_ran: exactly the pinned shape (the shared checkout step itself is held by the checkout section).""" problems = [] @@ -1668,13 +1684,14 @@ def policy_job_problems(job, fc): or script.get("env") != POLICY_ENV: problems.append(f"{where}: the second step is not `{POLICY_SCRIPT}` with the pinned env and nothing else") timeout, cap = job.get("timeout-minutes"), fc.get("wait_seconds") - if not isinstance(timeout, int) or not isinstance(cap, int) or cap > timeout * 60 - POLICY_MARGIN: - problems.append(f"{where}: fork_checks.wait_seconds {cap!r} is not at least {POLICY_MARGIN} s below the " - f"timeout ({timeout!r} min), so a timeout would cancel the job before its message") + if not isinstance(timeout, int) or not isinstance(cap, int) or policy_worst_case(cap, limits) > timeout * 60: + problems.append(f"{where}: its worst case (checkout {POLICY_CHECKOUT_BUDGET} s, fork_checks.wait_seconds " + f"{cap!r} or the git calls {limits}, one last poll) does not fit the timeout ({timeout!r} min), " + "so a timeout would cancel the job before its message") return problems -def policy_problems(policies, wfs, tests_wf): +def policy_problems(policies, wfs, tests_wf, limits=POLICY_LIMITS): """wfs: tree path -> parsed workflow (every workflow file, upstream's too).""" prt = policies.get("pull_request_target") or {} allowed, blocked = prt.get("allowed") or [], prt.get("upstream_blocked") or [] @@ -1700,7 +1717,7 @@ def policy_problems(policies, wfs, tests_wf): names = {str(j.get("name")) for j in (wfs.get(fc["workflow"], {}).get("jobs") or {}).values()} problems += [f"{fc['workflow']} has no job named {fc.get(k)!r} (fork_checks.{k})" for k in ("check", "outside_check") if fc.get(k) not in names] - problems += policy_job_problems((tests_wf.get("jobs") or {}).get("fork_checks_ran") or {}, fc) + problems += policy_job_problems((tests_wf.get("jobs") or {}).get("fork_checks_ran") or {}, fc, limits) return problems @@ -1712,10 +1729,10 @@ def policy_problems(policies, wfs, tests_wf): "names are its jobs'", not problems, "; ".join(problems)) -def policy_mutant(mutate): +def policy_mutant(mutate, limits=POLICY_LIMITS): p, w, t = copy.deepcopy(policies), copy.deepcopy(all_wfs), copy.deepcopy(fork_tests) mutate(p, w, t) - return bool(policy_problems(p, w, t)) + return bool(policy_problems(p, w, t, limits)) FC = ".github/workflows/fork-checks.yml" @@ -1762,8 +1779,30 @@ def policy_mutant(mutate): "a wait cap equal to the job timeout": lambda p, w, t: p["fork_checks"].__setitem__( "wait_seconds", t["jobs"]["fork_checks_ran"]["timeout-minutes"] * 60), "a job timeout cut below the cap": lambda p, w, t: t["jobs"]["fork_checks_ran"].__setitem__("timeout-minutes", 10), + "a cap that leaves no room for the last poll": lambda p, w, t: p["fork_checks"].__setitem__( + "wait_seconds", t["jobs"]["fork_checks_ran"]["timeout-minutes"] * 60 - POLICY_CHECKOUT_BUDGET - 1), + "fork_checks_ran without PR_NUMBER": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"] + .pop("PR_NUMBER"), + "fork_checks_ran with PR_NUMBER from elsewhere": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][-1]["env"] + .__setitem__("PR_NUMBER", "${{ github.run_number }}"), + # The checkout's env (#433 fix round 1, state finding 5): only github.sha, depth 1, only the scripts. + "fork_checks_ran checking out the pull request's head": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][0] + ["env"].__setitem__("REF", "${{ github.event.pull_request.head.sha }}"), + "fork_checks_ran checking out the whole tree": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][0]["env"] + .__setitem__("SPARSE", ""), + "fork_checks_ran checking out the full history": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][0]["env"] + .__setitem__("HISTORY", "full"), + "fork_checks_ran's checkout with an extra env": lambda p, w, t: t["jobs"]["fork_checks_ran"]["steps"][0]["env"] + .__setitem__("BASH_ENV", "x"), }.items(): check(f"policies: {label} fails", policy_mutant(mutate)) +# The script's limits, as if it changed them: the budget must still fit (#433 fix round 1, nit of both lenses). +for label, change in { + "git calls that outlast the job": {"GIT_TIMEOUT": 300}, + "more git calls than the budget holds": {"GIT_CALLS": 20}, + "a slower last poll than the budget holds": {"API_TIMEOUT": 200}, +}.items(): + check(f"policies: {label} fails", policy_mutant(lambda p, w, t: None, {**POLICY_LIMITS, **change})) shutil.rmtree(work, ignore_errors=True) if failures: diff --git a/.github/workflows/fork-tests.yml b/.github/workflows/fork-tests.yml index 6a83376024a41..29a797b78605b 100644 --- a/.github/workflows/fork-tests.yml +++ b/.github/workflows/fork-tests.yml @@ -77,7 +77,7 @@ concurrency: # its own, and its code runs as soon as a maintainer approves the run. The real protection here is # the repository's approval setting (approval for every outside contributor): keep it on, and never # approve an outside pull request's run; push its branch to this repository after review instead. -# (fork-checks.yml is different: pull_request_target always runs main's copy.) +# (fork-checks.yml is different: pull_request_target always runs the base branch's copy.) jobs: # Fails on purpose: an outside pull request is not tested here, see above. outside_pr: @@ -247,11 +247,12 @@ jobs: # fork-checks.yml runs on pull_request_target, which GitHub blocks unless the repository's Actions # policy allows it (#401, .github/fork-actions-policies.json); its checks are not required, so a # blocked, failed or cancelled run would pass unseen. This job waits (capped) for its check run on - # the pull request's head to pass, judged by the config on the pull request's base commit (#430), - # and fork_ci_result needs it. Other events: a called run (release) logs a skip; a direct run passes + # the pull request's head to pass, in a run of this pull request (PR_NUMBER, #433), judged by the + # config on the pull request's base commit (#430), and fork_ci_result needs it. Other events: a called run (release) logs a skip; a direct run passes # only on the default branch. Checks out without a token (the shared plain-git step: actions/checkout # with persist-credentials: false fails on upstream's gitlinks); the script fetches the base commit. # fork-contracts.test.py pins this job (if, env, steps, no continue-on-error, cap below the timeout). + # The checkout and the base fetch use no token, so this job works only in a public repository. fork_checks_ran: name: Fork checks ran if: ${{ (!github.event.pull_request || github.event.pull_request.head.repo.full_name == github.repository) }} @@ -291,6 +292,7 @@ jobs: EVENT_NAME: ${{ github.event_name }} HEAD_SHA: ${{ github.event.pull_request.head.sha }} BASE_SHA: ${{ github.event.pull_request.base.sha }} + PR_NUMBER: ${{ github.event.pull_request.number }} OUTSIDE_PR: ${{ github.event.pull_request && github.event.pull_request.head.repo.full_name != github.repository }} DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} GITHUB_TOKEN: ${{ github.token }}