From c9402c3763358aac0249a9ddf9fde55458afe4c3 Mon Sep 17 00:00:00 2001 From: Piotr Zaniewski Date: Fri, 30 Oct 2026 11:11:44 +0100 Subject: [PATCH] feat(ci): add bounded retries for approved pull requests Approved pull requests can leave authors repeating failed CI by hand. Existing approval automation waits for green checks but cannot request a rerun. Add a disabled, dry-run-first retry action with an explicit workflow allowlist, current-head human approval, freshness checks and a three-rerun budget that includes manual attempts. Leave an immutable diagnosis handoff when the budget is exhausted. Keep merges and AI dispatch outside this pilot until their policies and input contract are verified. --- .../actions/retry-approved-pr-ci/README.md | 170 +++++++++ .../actions/retry-approved-pr-ci/action.yml | 64 ++++ .../retry-approved-pr-ci/src/retry_ci.py | 216 +++++++++++ .../test/test_retry_ci.py | 342 ++++++++++++++++++ .../workflows/test-retry-approved-pr-ci.yaml | 25 ++ Makefile | 7 +- README.md | 7 + 7 files changed, 829 insertions(+), 2 deletions(-) create mode 100644 .github/actions/retry-approved-pr-ci/README.md create mode 100644 .github/actions/retry-approved-pr-ci/action.yml create mode 100644 .github/actions/retry-approved-pr-ci/src/retry_ci.py create mode 100644 .github/actions/retry-approved-pr-ci/test/test_retry_ci.py create mode 100644 .github/workflows/test-retry-approved-pr-ci.yaml diff --git a/.github/actions/retry-approved-pr-ci/README.md b/.github/actions/retry-approved-pr-ci/README.md new file mode 100644 index 00000000..c8c923a1 --- /dev/null +++ b/.github/actions/retry-approved-pr-ci/README.md @@ -0,0 +1,170 @@ +# Retry approved PR CI + +Retry failed jobs in an explicitly allowed PR workflow, at most three times. The +action is disabled by default and dry run is on by default. It never approves, +merges, changes code, or starts an AI session. No repository is enrolled by this PR. + +The pilot leaves review timing with the team. It only helps an already approved +commit recover from a failed run. A red check stays red until its own jobs pass. +The PR author owns diagnosis when retries fail. + +## Policy + +All of these must hold before a retry: + +- The completed run failed on a `pull_request` event in this repository. Forks, + drafts, closed PRs, push runs and ambiguous run/PR associations are skipped. +- The numeric workflow ID is in the caller's explicit allowlist. Enroll test + workflows only, after reviewing their side effects and common failure modes. + Do not enroll deployment, credential, or permission-failure workflows. +- The PR head equals the failed run's SHA. GitHub's review decision is `APPROVED`, + and a human writer other than the author approved that exact commit. Outstanding + change requests, dismissed approvals and incomplete review reads block retries. +- The run ID is the latest for that PR, workflow and SHA. All pages are checked, + up to 1,000 runs; larger histories and missing metadata are skipped. +- The attempt equals the completed event's attempt. Manual reruns consume the + same budget: attempt 1 can request retry 1; attempt 4 has used all three retries. + +The action reads this policy again before the write. Dry run performs the same +reads. It calls only `rerun-failed-jobs`, once; it never retries an uncertain POST. +API errors leave the decision `error`, with a link and remedy in the job summary. +This advisory step exits green; it does not replace or hide the failed CI checks. + +## Caller + +Requires `gh` and Python 3 on the runner. Use a trusted default-branch +`workflow_run` handler without checking out or executing PR code. Never consume +artifacts or scripts supplied by the failed workflow in this privileged job. + +Example for a repository owner to adapt after the pilot is approved and +`retry-approved-pr-ci/v1` is released. The tag is not published by this PR. +Replace the workflow name and numeric ID with a reviewed test workflow. + +```yaml +name: Retry approved PR CI +on: + workflow_run: + workflows: [PR tests] + types: [completed] + +permissions: {} + +concurrency: + group: approved-ci-retry-${{ github.event.workflow_run.id }} + cancel-in-progress: false + +jobs: + retry: + if: github.event.workflow_run.event == 'pull_request' + runs-on: ubuntu-latest + permissions: + actions: write + contents: read + pull-requests: read + steps: + - uses: loft-sh/github-actions/.github/actions/retry-approved-pr-ci@retry-approved-pr-ci/v1 + id: retry + with: + enabled: 'true' + dry-run: 'true' + workflow-ids: '123456' + run-id: ${{ github.event.workflow_run.id }} + run-attempt: ${{ github.event.workflow_run.run_attempt }} + github-token: ${{ github.token }} +``` + +Keep exactly one handler and the concurrency group above per repository/run. +This serializes duplicate events. The attempt check then skips an older event +after a rerun starts. GitHub does not offer an atomic head/approval condition on +the rerun endpoint: a change after the final read can still start work on the old +SHA. This action cannot merge that work and never transfers approval to a new SHA. + +The default token may suppress follow-on workflow events from its writes. Before +live enrollment, verify that reruns produce the next completed `workflow_run` +event with the chosen token. Without that evidence, only a single automatic +retry is proven. If an App token is needed, review its scope and credential path +separately; do not reuse a merge-bypass identity. + +## Diagnosis handoff + +`outcome=exhausted` stops all retry requests and emits `diagnosis`, a JSON string: + +```json +{"version":1,"repository":"owner/repo","pr_number":42,"head_sha":"<40 hex characters>","workflow_id":55,"run_id":123,"run_attempt":4,"run_url":"https://github.com/owner/repo/actions/runs/123","reason":"retry_budget_exhausted"} +``` + +This is failure context for a human or a future diagnosis adapter, not the QA +bot's existing input schema. It contains no logs, credentials or PR prose. +Consumers must fetch evidence for this exact SHA, preserve the author's ownership, +and recheck current state before acting. There is no automatic dispatch, issue +creation or code-fix permission in this action. + +The QA diagnosis path consumes groups of failed nightly specs and history. +Arbitrary PR workflows need an adapter to that input contract. Verify its input +producer, deployment and supported failure types before connecting it. Any fix +must get its own human review. + +## Pilot acceptance + +1. Offline tests cover retries, budget exhaustion, duplicate events, stale heads + and approvals, forks, incomplete API data, pagination, and dry-run parity. +2. A repository owner selects test-only workflow IDs and inspects dry-run job + summaries before enabling writes. `enabled=false` turns the action off. +3. A controlled failing PR proves one retry, follow-on event delivery, and the + full three-retry stop with the selected token. No live proof is claimed here. +4. A new commit or withdrawn approval stops retries. Exhaustion leaves the CI red + and produces the diagnosis handoff. The author diagnoses or asks QA for help. +5. Automatic merging stays disabled. Required-check and merge policy need a + separate explicit decision, including the current head and branch protections. + +## Workflow and failure table + +| Stage | Trigger and reads | Mutation | Emits / skip remedy | Coverage | +|---|---|---|---|---| +| Select | Completed event, run metadata, workflow allowlist | None | Skip unsupported run; select a test workflow | New policy; existing bot approval stays unchanged | +| Review | Current PR SHA, effective writer reviews, review decision | None | Skip changed head or review; author obtains review of current commit | Existing GitHub review state; new retry gate | +| Budget | Event attempt, current attempt, paginated runs | None | Exhausted handoff or skip old event/run; author diagnoses | New bounded retry policy; GitHub persists attempts | +| Revalidate | Repeat run, review, and latest-run reads | None in dry run | `would_retry` or skip changed state | New guard; API race above remains | +| Retry | All gates passed | New: one failed-jobs rerun request | `retried`; uncertain response is `error`, inspect run before any retry | GitHub executes original workflow; no local PR execution | +| Escalate | Budget exhausted | None | JSON diagnosis and run link; human owns next action | New output; QA adapter and deployment are unverified | + +Each stage emits a fixed `outcome`/`reason` vocabulary to stdout and action outputs; +the job summary links the failed run. Exhaustion and errors name the diagnosis +remedy. No service metrics, Slack alert or separate dashboard is introduced. + +## Inputs + + + +| INPUT | TYPE | REQUIRED | DEFAULT | DESCRIPTION | +|--------------|--------|----------|-----------|--------------------------------------------------------------------------------------------| +| dry-run | string | false | `"true"` | Perform the same policy reads but
suppress the retry request | +| enabled | string | false | `"false"` | Explicit opt-in to policy reads and
retries | +| github-token | string | true | | Token with Actions write, Contents read
and Pull requests read in this
repository | +| max-retries | string | false | `"3"` | Maximum reruns per workflow run, including
manual reruns; integer from 0 to
3 | +| run-attempt | string | true | | Attempt from that event; stale or
duplicate events are skipped | +| run-id | string | true | | Workflow run ID from a completed
workflow_run event | +| workflow-ids | string | true | | Comma-separated numeric workflow IDs approved for
retries in this repository | + + + +## Outputs + + + +| OUTPUT | TYPE | DESCRIPTION | +|--------------|--------|--------------------------------------------------------------------------------------------------------| +| diagnosis | string | JSON handoff with verified repository, PR,
SHA and run identifiers when retries
are exhausted | +| head_sha | string | Verified PR head SHA, empty when
eligibility is not established | +| outcome | string | disabled, skipped, would_retry, retried, exhausted or
error | +| reason | string | Fixed policy reason for the decision | +| retries_used | string | Observed reruns before this invocation, including
manual reruns | +| run_url | string | Workflow run for human diagnosis | + + + +## Tests + +`make test-retry-approved-pr-ci` runs offline pytest with mocked GitHub responses. +The test workflow also runs on changes to its own definition. `make lint` checks +workflow syntax and security, and `make generate-docs` updates the tables above. diff --git a/.github/actions/retry-approved-pr-ci/action.yml b/.github/actions/retry-approved-pr-ci/action.yml new file mode 100644 index 00000000..c0f98dd2 --- /dev/null +++ b/.github/actions/retry-approved-pr-ci/action.yml @@ -0,0 +1,64 @@ +name: Retry approved PR CI +description: Retry allowlisted failed PR workflows with current human approval and a hard retry cap +branding: + icon: repeat + color: blue +inputs: + run-id: + description: Workflow run ID from a completed workflow_run event + required: true + run-attempt: + description: Attempt from that event; stale or duplicate events are skipped + required: true + workflow-ids: + description: Comma-separated numeric workflow IDs approved for retries in this repository + required: true + enabled: + description: Explicit opt-in to policy reads and retries + required: false + default: 'false' + dry-run: + description: Perform the same policy reads but suppress the retry request + required: false + default: 'true' + max-retries: + description: Maximum reruns per workflow run, including manual reruns; integer from 0 to 3 + required: false + default: '3' + github-token: + description: Token with Actions write, Contents read and Pull requests read in this repository + required: true +outputs: + outcome: + description: disabled, skipped, would_retry, retried, exhausted or error + value: ${{ steps.retry.outputs.outcome }} + reason: + description: Fixed policy reason for the decision + value: ${{ steps.retry.outputs.reason }} + run_url: + description: Workflow run for human diagnosis + value: ${{ steps.retry.outputs.run_url }} + head_sha: + description: Verified PR head SHA, empty when eligibility is not established + value: ${{ steps.retry.outputs.head_sha }} + retries_used: + description: Observed reruns before this invocation, including manual reruns + value: ${{ steps.retry.outputs.retries_used }} + diagnosis: + description: JSON handoff with verified repository, PR, SHA and run identifiers when retries are exhausted + value: ${{ steps.retry.outputs.diagnosis }} +runs: + using: composite + steps: + - name: Check policy and request failed-job retry + id: retry + shell: bash + run: python3 "$GITHUB_ACTION_PATH/src/retry_ci.py" + env: + GH_TOKEN: ${{ inputs.github-token }} + INPUT_RUN_ID: ${{ inputs.run-id }} + INPUT_RUN_ATTEMPT: ${{ inputs.run-attempt }} + INPUT_WORKFLOW_IDS: ${{ inputs.workflow-ids }} + INPUT_ENABLED: ${{ inputs.enabled }} + INPUT_DRY_RUN: ${{ inputs.dry-run }} + INPUT_MAX_RETRIES: ${{ inputs.max-retries }} diff --git a/.github/actions/retry-approved-pr-ci/src/retry_ci.py b/.github/actions/retry-approved-pr-ci/src/retry_ci.py new file mode 100644 index 00000000..5102f9d6 --- /dev/null +++ b/.github/actions/retry-approved-pr-ci/src/retry_ci.py @@ -0,0 +1,216 @@ +"""Bounded, opt-in retries. All policy reads run again before the single write.""" + +import json +import os +import re +import subprocess +from dataclasses import dataclass +from pathlib import Path + + +class PolicySkip(Exception): + pass + + +@dataclass(frozen=True) +class Config: + repository: str + run_id: int + run_attempt: int + workflow_ids: frozenset[int] + max_retries: int = 3 + enabled: bool = False + dry_run: bool = True + + def validate(self): + if (not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9-]*/[A-Za-z0-9_.-]+", self.repository) + or self.repository.split("/")[-1] in (".", "..")): + raise ValueError("repository must be owner/name") + ids = [self.run_id, self.run_attempt, *self.workflow_ids] + if not self.workflow_ids or any(type(x) is not int or x <= 0 for x in ids): + raise ValueError("run, attempt and workflow IDs must be positive integers") + if type(self.max_retries) is not int or not 0 <= self.max_retries <= 3: + raise ValueError("max-retries must be in 0..3") + if type(self.enabled) is not bool or type(self.dry_run) is not bool: + raise ValueError("enabled and dry-run must be true or false") + + +REVIEW_QUERY = """ +query($owner: String!, $name: String!, $number: Int!) { + repository(owner: $owner, name: $name) { + pullRequest(number: $number) { + state isDraft headRefOid reviewDecision + headRepository { nameWithOwner } + baseRepository { nameWithOwner } + author { login } + latestOpinionatedReviews(first: 100, writersOnly: true) { + nodes { state commit { oid } author { login __typename } } + pageInfo { hasNextPage } + } + } + } +} +""" + + +class GitHub: + """Use the runner's gh auth. Never log API errors, which can carry private data.""" + + @staticmethod + def _call(args, payload=None, parse=True): + result = subprocess.run( + ["gh", "api", *args], + input=json.dumps(payload) if payload is not None else None, + text=True, capture_output=True, timeout=30, check=False, + ) + if result.returncode: + raise RuntimeError("GitHub API request failed; check token permissions and run logs") + return json.loads(result.stdout) if parse else None + + def get(self, path): + return self._call([path]) + + def post(self, path): + # A lost response is ambiguous. Do not retry this request automatically. + return self._call(["--method", "POST", path], parse=False) + + def graphql(self, query, variables): + data = self._call(["graphql", "--input", "-"], {"query": query, "variables": variables}) + if data.get("errors"): + raise RuntimeError("GitHub GraphQL query failed") + return data + + +def require(condition, reason): + if not condition: + raise PolicySkip(reason) + + +def snapshot(config, api): + repo = config.repository + run = api.get(f"repos/{repo}/actions/runs/{config.run_id}") + require(type(run["id"]) is int and run["id"] == config.run_id, "wrong_run") + require(run["repository"]["full_name"] == repo, "wrong_repository") + require(run["head_repository"]["full_name"] == repo, "fork") + require(run["event"] == "pull_request", "unsupported_event") + require(type(run["workflow_id"]) is int and run["workflow_id"] in config.workflow_ids, + "workflow_not_allowed") + require(type(run["run_attempt"]) is int and run["run_attempt"] == config.run_attempt, + "stale_attempt") + require(run["status"] == "completed" and run["conclusion"] == "failure", "not_failed") + sha = run["head_sha"] + require(isinstance(sha, str) and re.fullmatch(r"[a-f0-9]{40}", sha), "invalid_head") + require(len(run["pull_requests"]) == 1, "ambiguous_pr") + number = run["pull_requests"][0]["number"] + require(type(number) is int and number > 0, "invalid_pr") + owner, name = repo.split("/") + data = api.graphql(REVIEW_QUERY, {"owner": owner, "name": name, "number": number}) + require(not data.get("errors"), "review_query_failed") + pr = data["data"]["repository"]["pullRequest"] + require(pr["state"] == "OPEN" and pr["isDraft"] is False, "pr_not_ready") + require(pr["headRepository"]["nameWithOwner"] == repo + and pr["baseRepository"]["nameWithOwner"] == repo, "fork") + require(pr["headRefOid"] == sha, "head_changed") + require(pr["reviewDecision"] == "APPROVED", "approval_required") + reviews = pr["latestOpinionatedReviews"] + require(reviews["pageInfo"]["hasNextPage"] is False, "too_many_reviews") + require(all(r["state"] != "CHANGES_REQUESTED" for r in reviews["nodes"]), + "changes_requested") + require(any(r["state"] == "APPROVED" and r["commit"]["oid"] == sha + and r["author"]["__typename"] == "User" + and r["author"]["login"] != pr["author"]["login"] + for r in reviews["nodes"]), "current_human_approval_required") + + # Do not revive superseded runs. Pagination is bounded and fails closed. + found = False + for page in range(1, 11): + path = (f"repos/{repo}/actions/workflows/{run['workflow_id']}/runs" + f"?event=pull_request&head_sha={sha}&per_page=100&page={page}") + runs = api.get(path)["workflow_runs"] + require(isinstance(runs, list), "invalid_runs") + for other in runs: + require(type(other["id"]) is int, "invalid_runs") + # A missing association cannot prove a different PR. Stop conservatively. + require(other["head_sha"] == sha and other["workflow_id"] == run["workflow_id"], + "invalid_runs") + require(isinstance(other["pull_requests"], list) and other["pull_requests"], + "unassociated_run") + same_pr = any(p["number"] == number for p in other["pull_requests"]) + require(not same_pr or other["id"] <= config.run_id, "superseded_run") + found |= same_pr and other["id"] == config.run_id + if len(runs) < 100: + break + else: + raise PolicySkip("too_many_runs") + require(found, "run_not_listed") + return {"head_sha": sha, "pr_number": number, "workflow_id": run["workflow_id"]} + + +def evaluate(config, api): + config.validate() + result = {"outcome": "disabled", "reason": "enable_explicitly", "head_sha": "", + "run_url": f"https://github.com/{config.repository}/actions/runs/{config.run_id}", + "retries_used": config.run_attempt - 1} + if not config.enabled: + return result + try: + first = snapshot(config, api) + result.update(first) + if config.run_attempt - 1 >= config.max_retries: + diagnosis = dict(first, version=1, repository=config.repository, + run_id=config.run_id, run_attempt=config.run_attempt, + run_url=result["run_url"], reason="retry_budget_exhausted") + return dict(result, outcome="exhausted", reason="author_diagnosis_required", + diagnosis=json.dumps(diagnosis, sort_keys=True)) + second = snapshot(config, api) + require(first == second, "state_changed") + if config.dry_run: + return dict(result, outcome="would_retry", reason="dry_run") + api.post(f"repos/{config.repository}/actions/runs/{config.run_id}/rerun-failed-jobs") + return dict(result, outcome="retried", reason="failed_jobs_requested") + except PolicySkip as exc: + return dict(result, outcome="skipped", reason=str(exc)) + except (KeyError, TypeError, ValueError, RuntimeError, subprocess.SubprocessError): + return dict(result, outcome="error", reason="verify_api_access_and_state") + + +def boolean(name, default): + value = os.environ.get(name, default) + if value not in ("true", "false"): + raise ValueError(f"{name} must be true or false") + return value == "true" + + +def main(): + try: + config = Config( + repository=os.environ["GITHUB_REPOSITORY"], + run_id=int(os.environ["INPUT_RUN_ID"]), + run_attempt=int(os.environ["INPUT_RUN_ATTEMPT"]), + workflow_ids=frozenset(int(x.strip()) for x in os.environ["INPUT_WORKFLOW_IDS"].split(",")), + max_retries=int(os.environ.get("INPUT_MAX_RETRIES", "3")), + enabled=boolean("INPUT_ENABLED", "false"), + dry_run=boolean("INPUT_DRY_RUN", "true"), + ) + result = evaluate(config, GitHub()) + except (KeyError, ValueError): + result = {"outcome": "error", "reason": "invalid_configuration"} + # Only fixed vocabulary, validated IDs/SHA and constructed URLs reach outputs. + print(json.dumps(result, sort_keys=True)) + if os.environ.get("GITHUB_OUTPUT"): + with Path(os.environ["GITHUB_OUTPUT"]).open("a") as output: + for key, value in result.items(): + output.write(f"{key}={value}\n") + if os.environ.get("GITHUB_STEP_SUMMARY"): + with Path(os.environ["GITHUB_STEP_SUMMARY"]).open("a") as summary: + summary.write(f"CI retry: {result['outcome']} ({result['reason']}).\n\n") + if result.get("run_url"): + summary.write(f"[Workflow run]({result['run_url']})\n\n") + if result["outcome"] in ("exhausted", "error"): + summary.write("PR author: inspect the failed jobs and diagnose the cause before another run.\n") + # Advisory automation does not turn an unrelated PR check red. + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.github/actions/retry-approved-pr-ci/test/test_retry_ci.py b/.github/actions/retry-approved-pr-ci/test/test_retry_ci.py new file mode 100644 index 00000000..891f31ae --- /dev/null +++ b/.github/actions/retry-approved-pr-ci/test/test_retry_ci.py @@ -0,0 +1,342 @@ +import copy +import importlib.util +from pathlib import Path +import sys + +import pytest + +MODULE_PATH = Path(__file__).parents[1] / 'src' / 'retry_ci.py' +spec = importlib.util.spec_from_file_location('retry_ci', MODULE_PATH) +retry_ci = importlib.util.module_from_spec(spec) +sys.modules[spec.name] = retry_ci +spec.loader.exec_module(retry_ci) + +REPO = 'acme/project' +SHA = 'a' * 40 +RUN_PATH = f'repos/{REPO}/actions/runs/123' +POST_PATH = RUN_PATH + '/rerun-failed-jobs' + + +def run(**changes): + result = { + 'id': 123, 'workflow_id': 55, 'event': 'pull_request', + 'status': 'completed', 'conclusion': 'failure', 'head_sha': SHA, + 'run_attempt': 1, 'repository': {'full_name': REPO}, + 'head_repository': {'full_name': REPO}, 'pull_requests': [{'number': 42}], + 'html_url': 'https://github.com/acme/project/actions/runs/123', + 'created_at': '2026-10-30T10:00:00Z', + } + result.update(changes) + return result + + +def pr(**changes): + result = { + 'number': 42, 'state': 'OPEN', 'isDraft': False, 'headRefOid': SHA, + 'headRepository': {'nameWithOwner': REPO}, + 'baseRepository': {'nameWithOwner': REPO}, + 'author': {'login': 'author'}, 'reviewDecision': 'APPROVED', + 'latestOpinionatedReviews': { + 'nodes': [{'state': 'APPROVED', 'commit': {'oid': SHA}, + 'author': {'login': 'reviewer', '__typename': 'User'}}], + 'pageInfo': {'hasNextPage': False}, + }, + } + result.update(changes) + return result + + +class API: + def __init__(self): + self.runs = [run()] + self.prs = [pr()] + self.pages = {1: [run()]} + self.calls = [] + self.posts = [] + self.post_error = None + + @staticmethod + def next_state(items): + return copy.deepcopy(items.pop(0) if len(items) > 1 else items[0]) + + def get(self, path): + self.calls.append(('get', path)) + if path == RUN_PATH: + return self.next_state(self.runs) + if '/actions/workflows/55/runs?' in path: + from urllib.parse import parse_qs, urlsplit + params = parse_qs(urlsplit(path).query) + page = int(params.get('page', ['1'])[0]) + return {'workflow_runs': copy.deepcopy(self.pages.get(page, [])), + 'total_count': sum(map(len, self.pages.values()))} + raise AssertionError(f'unexpected GET {path}') + + def graphql(self, query, variables): + self.calls.append(('graphql', variables)) + return {'data': {'repository': {'pullRequest': self.next_state(self.prs)}}} + + def post(self, path): + self.calls.append(('post', path)) + self.posts.append(path) + if self.post_error: + raise self.post_error + + +def config(**changes): + values = dict(repository=REPO, run_id=123, run_attempt=1, + workflow_ids=frozenset({55}), max_retries=3, + enabled=True, dry_run=False) + values.update(changes) + return retry_ci.Config(**values) + + +def evaluate(api, **changes): + return retry_ci.evaluate(config(**changes), api) + + +def assert_safe(result, api): + assert result['outcome'] in {'skipped', 'exhausted', 'error'} + assert not api.posts + + +def test_happy_path_revalidates_and_posts_once(): + api = API() + result = evaluate(api) + assert result['outcome'] == 'retried' + assert api.posts == [POST_PATH] + assert sum(call == ('get', RUN_PATH) for call in api.calls) == 2 + assert sum(call[0] == 'graphql' for call in api.calls) == 2 + assert result['head_sha'] == SHA + assert result['retries_used'] == 0 + + +def test_dry_run_checks_but_does_not_post(): + api = API() + result = evaluate(api, dry_run=True) + assert result['outcome'] == 'would_retry' + assert any(call[0] == 'graphql' for call in api.calls) + assert not api.posts + + +def test_disabled_has_no_api_calls(): + api = API() + assert evaluate(api, enabled=False)['outcome'] == 'disabled' + assert not api.calls + + +def test_attempt_four_is_exhausted_even_if_manually_rerun(): + api = API() + api.runs = [run(run_attempt=4)] + api.pages = {1: [run(run_attempt=4)]} + result = evaluate(api, run_attempt=4) + assert result['outcome'] == 'exhausted' + assert result['retries_used'] == 3 + assert not api.posts + + +def test_attempt_mismatch_rejects_duplicate_event(): + api = API() + api.runs = [run(run_attempt=2)] + assert_safe(evaluate(api, run_attempt=1), api) + + +@pytest.mark.parametrize('change', [ + {'headRefOid': 'b' * 40}, {'reviewDecision': 'REVIEW_REQUIRED'}, + {'isDraft': True}, {'headRepository': {'nameWithOwner': 'other/fork'}}, + {'state': 'CLOSED'}, +]) +def test_pr_state_must_remain_eligible_before_write(change): + api = API() + api.prs = [pr(), pr(**change)] + assert_safe(evaluate(api), api) + + +def test_newer_run_on_second_page_blocks_retry(): + api = API() + # A full first page forces pagination even when the matching newer run is later. + api.pages = {1: [run(id=123 - i, pull_requests=[{'number': 1000 + i}]) for i in range(100)], + 2: [run(id=124, created_at='2026-10-30T10:01:00Z')]} + assert_safe(evaluate(api), api) + assert any('page=2' in call[1] for call in api.calls if call[0] == 'get') + + +@pytest.mark.parametrize('changed_run', [ + {'head_repository': {'full_name': 'other/fork'}}, + {'pull_requests': []}, {'pull_requests': [{'number': 42}, {'number': 43}]}, + {'event': 'push'}, {'status': 'in_progress'}, {'conclusion': 'success'}, + {'workflow_id': 99}, {'head_sha': None}, {'repository': None}, +]) +def test_ineligible_or_missing_run_metadata_fails_closed(changed_run): + api = API() + api.runs = [run(**changed_run)] + assert_safe(evaluate(api), api) + + +@pytest.mark.parametrize('reviews', [ + {'nodes': [{'state': 'APPROVED', 'commit': {'oid': 'b' * 40}, + 'author': {'login': 'reviewer', '__typename': 'User'}}], 'pageInfo': {'hasNextPage': False}}, + {'nodes': [{'state': 'CHANGES_REQUESTED', 'commit': {'oid': SHA}, + 'author': {'login': 'reviewer', '__typename': 'User'}}], 'pageInfo': {'hasNextPage': False}}, + {'nodes': [], 'pageInfo': {'hasNextPage': False}}, + {'nodes': [], 'pageInfo': {'hasNextPage': True}}, +]) +def test_reviews_fail_closed(reviews): + api = API() + api.prs = [pr(latestOpinionatedReviews=reviews)] + assert_safe(evaluate(api), api) + + +def test_malformed_response_fails_closed(): + api = API() + api.prs = [None] + assert_safe(evaluate(api), api) + + +def test_failed_post_is_not_retried(): + api = API() + api.post_error = RuntimeError('network error') + result = evaluate(api) + assert result['outcome'] == 'error' + assert api.posts == [POST_PATH] + + +@pytest.mark.parametrize('changes', [ + {'repository': 'bad'}, {'repository': '../bad'}, {'run_id': 0}, + {'run_attempt': 0}, {'workflow_ids': frozenset()}, {'workflow_ids': frozenset({0})}, + {'max_retries': -1}, {'max_retries': 4}, {'enabled': 'true'}, {'dry_run': 1}, +]) +def test_invalid_configuration_is_rejected_even_in_dry_run(changes): + cfg = config(**changes) + with pytest.raises(ValueError): + cfg.validate() + + +def test_run_attempt_changes_before_write(): + api = API() + api.runs = [run(), run(run_attempt=2)] + assert_safe(evaluate(api), api) + + +def test_newer_run_appearing_before_write_blocks_post(): + class NewRunAPI(API): + def __init__(self): + super().__init__() + self.list_reads = 0 + + def get(self, path): + if '/actions/workflows/55/runs?' in path: + self.list_reads += 1 + if self.list_reads == 2: + self.pages = {1: [run(id=124, created_at='2026-10-30T10:01:00Z'), run()]} + return super().get(path) + + api = NewRunAPI() + assert_safe(evaluate(api), api) + assert api.list_reads >= 2 + + +def test_exhausted_handoff_contains_only_verified_identifiers(): + import json + api = API() + api.runs = [run(run_attempt=4, name='untrusted\ninjected=1')] + api.pages = {1: [run(run_attempt=4)]} + result = evaluate(api, run_attempt=4) + assert result['outcome'] == 'exhausted' + assert json.loads(result['diagnosis']) == { + 'version': 1, 'repository': REPO, 'pr_number': 42, 'head_sha': SHA, + 'workflow_id': 55, 'run_id': 123, 'run_attempt': 4, + 'run_url': 'https://github.com/acme/project/actions/runs/123', + 'reason': 'retry_budget_exhausted', + } + assert not api.posts + assert all(call[0] in {'get', 'graphql'} for call in api.calls) + + +@pytest.mark.parametrize('author', [ + {'login': 'reviewer', '__typename': 'Bot'}, + {'login': 'author', '__typename': 'User'}, +]) +def test_bot_or_pr_author_cannot_supply_approval(author): + api = API() + api.prs = [pr(latestOpinionatedReviews={ + 'nodes': [{'state': 'APPROVED', 'commit': {'oid': SHA}, 'author': author}], + 'pageInfo': {'hasNextPage': False}, + })] + assert_safe(evaluate(api), api) + + +@pytest.mark.parametrize('budget,attempt,expected', [ + (0, 1, 'exhausted'), (0, 2, 'exhausted'), (0, 3, 'exhausted'), (0, 4, 'exhausted'), + (3, 1, 'retried'), (3, 2, 'retried'), (3, 3, 'retried'), (3, 4, 'exhausted'), +]) +def test_retry_budget_boundaries(budget, attempt, expected): + api = API() + api.runs = [run(run_attempt=attempt)] + api.pages = {1: [run(run_attempt=attempt)]} + result = evaluate(api, max_retries=budget, run_attempt=attempt) + assert result['outcome'] == expected + assert result['retries_used'] == attempt - 1 + assert api.posts == ([POST_PATH] if expected == 'retried' else []) + assert all('/merge' not in str(call) for call in api.calls) + + +@pytest.mark.parametrize('failure_point', ['run', 'graphql', 'listing']) +def test_api_read_errors_fail_closed_and_do_not_echo_error(failure_point): + class BrokenAPI(API): + def get(self, path): + if (failure_point == 'run' and path == RUN_PATH or + failure_point == 'listing' and '/actions/workflows/' in path): + raise RuntimeError('secret-token-and-untrusted-message') + return super().get(path) + + def graphql(self, query, variables): + if failure_point == 'graphql': + raise RuntimeError('secret-token-and-untrusted-message') + return super().graphql(query, variables) + + api = BrokenAPI() + result = evaluate(api) + assert result['outcome'] == 'error' + assert 'secret-token' not in str(result) + assert not api.posts + + +def test_main_writes_safe_output_and_summary(monkeypatch, tmp_path, capsys): + import json + api = API() + api.runs = [run(run_attempt=4, name='untrusted\ninjected=1')] + api.pages = {1: [run(run_attempt=4)]} + output = tmp_path / 'output' + summary = tmp_path / 'summary' + for key, value in { + 'GITHUB_REPOSITORY': REPO, 'INPUT_RUN_ID': '123', 'INPUT_RUN_ATTEMPT': '4', + 'INPUT_WORKFLOW_IDS': '55', 'INPUT_MAX_RETRIES': '3', 'INPUT_ENABLED': 'true', + 'INPUT_DRY_RUN': 'false', 'GITHUB_OUTPUT': str(output), + 'GITHUB_STEP_SUMMARY': str(summary), + }.items(): + monkeypatch.setenv(key, value) + monkeypatch.setattr(retry_ci, 'GitHub', lambda: api) + assert retry_ci.main() == 0 + result = json.loads(capsys.readouterr().out) + outputs = dict(line.split('=', 1) for line in output.read_text().splitlines()) + assert json.loads(outputs['diagnosis']) == json.loads(result['diagnosis']) + assert outputs['outcome'] == 'exhausted' + assert 'inspect the failed jobs' in summary.read_text() + assert 'untrusted' not in output.read_text() + summary.read_text() + assert not api.posts + + +def test_main_invalid_input_cannot_inject_output(monkeypatch, tmp_path, capsys): + import json + output = tmp_path / 'output' + summary = tmp_path / 'summary' + monkeypatch.setenv('GITHUB_REPOSITORY', 'acme/project\ninjected=token') + monkeypatch.setenv('INPUT_RUN_ID', '123') + monkeypatch.setenv('INPUT_RUN_ATTEMPT', '1') + monkeypatch.setenv('INPUT_WORKFLOW_IDS', '55') + monkeypatch.setenv('GITHUB_OUTPUT', str(output)) + monkeypatch.setenv('GITHUB_STEP_SUMMARY', str(summary)) + assert retry_ci.main() == 0 + assert json.loads(capsys.readouterr().out) == {'outcome': 'error', 'reason': 'invalid_configuration'} + assert output.read_text() == 'outcome=error\nreason=invalid_configuration\n' + assert 'injected' not in summary.read_text() diff --git a/.github/workflows/test-retry-approved-pr-ci.yaml b/.github/workflows/test-retry-approved-pr-ci.yaml new file mode 100644 index 00000000..f37c2060 --- /dev/null +++ b/.github/workflows/test-retry-approved-pr-ci.yaml @@ -0,0 +1,25 @@ +name: Test retry-approved-pr-ci + +on: + pull_request: + paths: + - '.github/actions/retry-approved-pr-ci/**' + - '.github/workflows/test-retry-approved-pr-ci.yaml' + +permissions: + contents: read + +jobs: + pytest: + name: Run approved PR CI retry tests + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Install uv + uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 + - name: Run retry policy tests + run: uv run --with pytest==8.3.3 pytest .github/actions/retry-approved-pr-ci/test/ -q diff --git a/Makefile b/Makefile index 7c716638..9a3cdba0 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: test test-release-on-merge test-secret-broker-request test-secret-broker-response test-semver-validation test-setup-semstat test-linear-pr-commenter test-link-backport-prs test-fork-review-check test-select-backport-source-prs test-release-notification test-linear-release-sync test-linear-cve-summary test-aws-test-infra test-cleanup-head-charts test-ci-test-notify test-auto-approve-bot-prs test-ai-pr-review test-ai-step test-release-summary test-publish-helm-chart test-govulncheck test-go-licenses test-run-ginkgo test-sticky-pr-comment test-repository-dispatch test-parse-label-filter test-release-branch-freeze test-prerelease-setup test-vcluster-release test-platform-release test-release-lib test-subtree-mirror test-oss-commit-sync test-oss-mirror-staleness test-backport-legacy-allowlist test-backport-legacy-split test-backport-fix-imports test-promote-release test-wait-for-release test-cve-scan test-commitlint test-resolve-github-release test-comment-triggered-check test-conflict-check build-linear-release-sync lint check-bats-jobs install-auto-doc generate-docs check-docs help +.PHONY: test test-release-on-merge test-retry-approved-pr-ci test-secret-broker-request test-secret-broker-response test-semver-validation test-setup-semstat test-linear-pr-commenter test-link-backport-prs test-fork-review-check test-select-backport-source-prs test-release-notification test-linear-release-sync test-linear-cve-summary test-aws-test-infra test-cleanup-head-charts test-ci-test-notify test-auto-approve-bot-prs test-ai-pr-review test-ai-step test-release-summary test-publish-helm-chart test-govulncheck test-go-licenses test-run-ginkgo test-sticky-pr-comment test-repository-dispatch test-parse-label-filter test-release-branch-freeze test-prerelease-setup test-vcluster-release test-platform-release test-release-lib test-subtree-mirror test-oss-commit-sync test-oss-mirror-staleness test-backport-legacy-allowlist test-backport-legacy-split test-backport-fix-imports test-promote-release test-wait-for-release test-cve-scan test-commitlint test-resolve-github-release test-comment-triggered-check test-conflict-check build-linear-release-sync lint check-bats-jobs install-auto-doc generate-docs check-docs help ACTIONS_DIR := .github/actions .PHONY: test-check-action-pins @@ -94,7 +94,10 @@ check-docs: generate-docs ## verify docs are up to date (fails if drift detected help: ## show this help @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf " %-30s %s\n", $$1, $$2}' -test: test-release-on-merge test-secret-broker-request test-secret-broker-response test-semver-validation test-setup-semstat test-linear-pr-commenter test-link-backport-prs test-fork-review-check test-select-backport-source-prs test-release-notification test-linear-release-sync test-linear-cve-summary test-aws-test-infra test-cleanup-head-charts test-auto-approve-bot-prs test-ai-pr-review test-ai-step test-release-summary test-ci-test-notify test-go-licenses test-publish-helm-chart test-govulncheck test-run-ginkgo test-sticky-pr-comment test-repository-dispatch test-parse-label-filter test-release-branch-freeze test-prerelease-setup test-vcluster-release test-platform-release test-release-lib test-subtree-mirror test-oss-commit-sync test-oss-mirror-staleness test-backport-legacy-allowlist test-backport-legacy-split test-backport-fix-imports test-promote-release test-wait-for-release test-cve-scan test-commitlint test-resolve-github-release test-comment-triggered-check test-conflict-check ## run all action tests +test: test-release-on-merge test-retry-approved-pr-ci test-secret-broker-request test-secret-broker-response test-semver-validation test-setup-semstat test-linear-pr-commenter test-link-backport-prs test-fork-review-check test-select-backport-source-prs test-release-notification test-linear-release-sync test-linear-cve-summary test-aws-test-infra test-cleanup-head-charts test-auto-approve-bot-prs test-ai-pr-review test-ai-step test-release-summary test-ci-test-notify test-go-licenses test-publish-helm-chart test-govulncheck test-run-ginkgo test-sticky-pr-comment test-repository-dispatch test-parse-label-filter test-release-branch-freeze test-prerelease-setup test-vcluster-release test-platform-release test-release-lib test-subtree-mirror test-oss-commit-sync test-oss-mirror-staleness test-backport-legacy-allowlist test-backport-legacy-split test-backport-fix-imports test-promote-release test-wait-for-release test-cve-scan test-commitlint test-resolve-github-release test-comment-triggered-check test-conflict-check ## run all action tests + +test-retry-approved-pr-ci: ## run approved PR CI retry pytest tests + uv run --with pytest==8.3.3 pytest $(ACTIONS_DIR)/retry-approved-pr-ci/test/ -q test: test-check-action-pins diff --git a/README.md b/README.md index 3ed69b24..0035af75 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,13 @@ Location: `.github/actions/check-action-pins` See [check-action-pins README](./.github/actions/check-action-pins/README.md) for usage, runner requirements, and the release and required-check followup. +### Retry approved PR CI + +Opt-in, dry-run-first retries for allowlisted failed PR workflows with current human +approval. At most three reruns; exhaustion emits a diagnosis handoff. Never merges. + +See [retry-approved-pr-ci README](./.github/actions/retry-approved-pr-ci/README.md). + ### Secret Broker Request Action Validates a short-lived secret request from an immutable commit, verifies the