diff --git a/.github/actions/retry-approved-pr-ci/README.md b/.github/actions/retry-approved-pr-ci/README.md new file mode 100644 index 00000000..c8c923a1 --- /dev/null +++ b/.github/actions/retry-approved-pr-ci/README.md @@ -0,0 +1,170 @@ +# Retry approved PR CI + +Retry failed jobs in an explicitly allowed PR workflow, at most three times. The +action is disabled by default and dry run is on by default. It never approves, +merges, changes code, or starts an AI session. No repository is enrolled by this PR. + +The pilot leaves review timing with the team. It only helps an already approved +commit recover from a failed run. A red check stays red until its own jobs pass. +The PR author owns diagnosis when retries fail. + +## Policy + +All of these must hold before a retry: + +- The completed run failed on a `pull_request` event in this repository. Forks, + drafts, closed PRs, push runs and ambiguous run/PR associations are skipped. +- The numeric workflow ID is in the caller's explicit allowlist. Enroll test + workflows only, after reviewing their side effects and common failure modes. + Do not enroll deployment, credential, or permission-failure workflows. +- The PR head equals the failed run's SHA. GitHub's review decision is `APPROVED`, + and a human writer other than the author approved that exact commit. Outstanding + change requests, dismissed approvals and incomplete review reads block retries. +- The run ID is the latest for that PR, workflow and SHA. All pages are checked, + up to 1,000 runs; larger histories and missing metadata are skipped. +- The attempt equals the completed event's attempt. Manual reruns consume the + same budget: attempt 1 can request retry 1; attempt 4 has used all three retries. + +The action reads this policy again before the write. Dry run performs the same +reads. It calls only `rerun-failed-jobs`, once; it never retries an uncertain POST. +API errors leave the decision `error`, with a link and remedy in the job summary. +This advisory step exits green; it does not replace or hide the failed CI checks. + +## Caller + +Requires `gh` and Python 3 on the runner. Use a trusted default-branch +`workflow_run` handler without checking out or executing PR code. Never consume +artifacts or scripts supplied by the failed workflow in this privileged job. + +Example for a repository owner to adapt after the pilot is approved and +`retry-approved-pr-ci/v1` is released. The tag is not published by this PR. +Replace the workflow name and numeric ID with a reviewed test workflow. + +```yaml +name: Retry approved PR CI +on: + workflow_run: + workflows: [PR tests] + types: [completed] + +permissions: {} + +concurrency: + group: approved-ci-retry-${{ github.event.workflow_run.id }} + cancel-in-progress: false + +jobs: + retry: + if: github.event.workflow_run.event == 'pull_request' + runs-on: ubuntu-latest + permissions: + actions: write + contents: read + pull-requests: read + steps: + - uses: loft-sh/github-actions/.github/actions/retry-approved-pr-ci@retry-approved-pr-ci/v1 + id: retry + with: + enabled: 'true' + dry-run: 'true' + workflow-ids: '123456' + run-id: ${{ github.event.workflow_run.id }} + run-attempt: ${{ github.event.workflow_run.run_attempt }} + github-token: ${{ github.token }} +``` + +Keep exactly one handler and the concurrency group above per repository/run. +This serializes duplicate events. The attempt check then skips an older event +after a rerun starts. GitHub does not offer an atomic head/approval condition on +the rerun endpoint: a change after the final read can still start work on the old +SHA. This action cannot merge that work and never transfers approval to a new SHA. + +The default token may suppress follow-on workflow events from its writes. Before +live enrollment, verify that reruns produce the next completed `workflow_run` +event with the chosen token. Without that evidence, only a single automatic +retry is proven. If an App token is needed, review its scope and credential path +separately; do not reuse a merge-bypass identity. + +## Diagnosis handoff + +`outcome=exhausted` stops all retry requests and emits `diagnosis`, a JSON string: + +```json +{"version":1,"repository":"owner/repo","pr_number":42,"head_sha":"<40 hex characters>","workflow_id":55,"run_id":123,"run_attempt":4,"run_url":"https://github.com/owner/repo/actions/runs/123","reason":"retry_budget_exhausted"} +``` + +This is failure context for a human or a future diagnosis adapter, not the QA +bot's existing input schema. It contains no logs, credentials or PR prose. +Consumers must fetch evidence for this exact SHA, preserve the author's ownership, +and recheck current state before acting. There is no automatic dispatch, issue +creation or code-fix permission in this action. + +The QA diagnosis path consumes groups of failed nightly specs and history. +Arbitrary PR workflows need an adapter to that input contract. Verify its input +producer, deployment and supported failure types before connecting it. Any fix +must get its own human review. + +## Pilot acceptance + +1. Offline tests cover retries, budget exhaustion, duplicate events, stale heads + and approvals, forks, incomplete API data, pagination, and dry-run parity. +2. A repository owner selects test-only workflow IDs and inspects dry-run job + summaries before enabling writes. `enabled=false` turns the action off. +3. A controlled failing PR proves one retry, follow-on event delivery, and the + full three-retry stop with the selected token. No live proof is claimed here. +4. A new commit or withdrawn approval stops retries. Exhaustion leaves the CI red + and produces the diagnosis handoff. The author diagnoses or asks QA for help. +5. Automatic merging stays disabled. Required-check and merge policy need a + separate explicit decision, including the current head and branch protections. + +## Workflow and failure table + +| Stage | Trigger and reads | Mutation | Emits / skip remedy | Coverage | +|---|---|---|---|---| +| Select | Completed event, run metadata, workflow allowlist | None | Skip unsupported run; select a test workflow | New policy; existing bot approval stays unchanged | +| Review | Current PR SHA, effective writer reviews, review decision | None | Skip changed head or review; author obtains review of current commit | Existing GitHub review state; new retry gate | +| Budget | Event attempt, current attempt, paginated runs | None | Exhausted handoff or skip old event/run; author diagnoses | New bounded retry policy; GitHub persists attempts | +| Revalidate | Repeat run, review, and latest-run reads | None in dry run | `would_retry` or skip changed state | New guard; API race above remains | +| Retry | All gates passed | New: one failed-jobs rerun request | `retried`; uncertain response is `error`, inspect run before any retry | GitHub executes original workflow; no local PR execution | +| Escalate | Budget exhausted | None | JSON diagnosis and run link; human owns next action | New output; QA adapter and deployment are unverified | + +Each stage emits a fixed `outcome`/`reason` vocabulary to stdout and action outputs; +the job summary links the failed run. Exhaustion and errors name the diagnosis +remedy. No service metrics, Slack alert or separate dashboard is introduced. + +## Inputs + + + +| INPUT | TYPE | REQUIRED | DEFAULT | DESCRIPTION | +|--------------|--------|----------|-----------|--------------------------------------------------------------------------------------------| +| dry-run | string | false | `"true"` | Perform the same policy reads but
suppress the retry request | +| enabled | string | false | `"false"` | Explicit opt-in to policy reads and
retries | +| github-token | string | true | | Token with Actions write, Contents read
and Pull requests read in this
repository | +| max-retries | string | false | `"3"` | Maximum reruns per workflow run, including
manual reruns; integer from 0 to
3 | +| run-attempt | string | true | | Attempt from that event; stale or
duplicate events are skipped | +| run-id | string | true | | Workflow run ID from a completed
workflow_run event | +| workflow-ids | string | true | | Comma-separated numeric workflow IDs approved for
retries in this repository | + + + +## Outputs + + + +| OUTPUT | TYPE | DESCRIPTION | +|--------------|--------|--------------------------------------------------------------------------------------------------------| +| diagnosis | string | JSON handoff with verified repository, PR,
SHA and run identifiers when retries
are exhausted | +| head_sha | string | Verified PR head SHA, empty when
eligibility is not established | +| outcome | string | disabled, skipped, would_retry, retried, exhausted or
error | +| reason | string | Fixed policy reason for the decision | +| retries_used | string | Observed reruns before this invocation, including
manual reruns | +| run_url | string | Workflow run for human diagnosis | + + + +## Tests + +`make test-retry-approved-pr-ci` runs offline pytest with mocked GitHub responses. +The test workflow also runs on changes to its own definition. `make lint` checks +workflow syntax and security, and `make generate-docs` updates the tables above. diff --git a/.github/actions/retry-approved-pr-ci/action.yml b/.github/actions/retry-approved-pr-ci/action.yml new file mode 100644 index 00000000..c0f98dd2 --- /dev/null +++ b/.github/actions/retry-approved-pr-ci/action.yml @@ -0,0 +1,64 @@ +name: Retry approved PR CI +description: Retry allowlisted failed PR workflows with current human approval and a hard retry cap +branding: + icon: repeat + color: blue +inputs: + run-id: + description: Workflow run ID from a completed workflow_run event + required: true + run-attempt: + description: Attempt from that event; stale or duplicate events are skipped + required: true + workflow-ids: + description: Comma-separated numeric workflow IDs approved for retries in this repository + required: true + enabled: + description: Explicit opt-in to policy reads and retries + required: false + default: 'false' + dry-run: + description: Perform the same policy reads but suppress the retry request + required: false + default: 'true' + max-retries: + description: Maximum reruns per workflow run, including manual reruns; integer from 0 to 3 + required: false + default: '3' + github-token: + description: Token with Actions write, Contents read and Pull requests read in this repository + required: true +outputs: + outcome: + description: disabled, skipped, would_retry, retried, exhausted or error + value: ${{ steps.retry.outputs.outcome }} + reason: + description: Fixed policy reason for the decision + value: ${{ steps.retry.outputs.reason }} + run_url: + description: Workflow run for human diagnosis + value: ${{ steps.retry.outputs.run_url }} + head_sha: + description: Verified PR head SHA, empty when eligibility is not established + value: ${{ steps.retry.outputs.head_sha }} + retries_used: + description: Observed reruns before this invocation, including manual reruns + value: ${{ steps.retry.outputs.retries_used }} + diagnosis: + description: JSON handoff with verified repository, PR, SHA and run identifiers when retries are exhausted + value: ${{ steps.retry.outputs.diagnosis }} +runs: + using: composite + steps: + - name: Check policy and request failed-job retry + id: retry + shell: bash + run: python3 "$GITHUB_ACTION_PATH/src/retry_ci.py" + env: + GH_TOKEN: ${{ inputs.github-token }} + INPUT_RUN_ID: ${{ inputs.run-id }} + INPUT_RUN_ATTEMPT: ${{ inputs.run-attempt }} + INPUT_WORKFLOW_IDS: ${{ inputs.workflow-ids }} + INPUT_ENABLED: ${{ inputs.enabled }} + INPUT_DRY_RUN: ${{ inputs.dry-run }} + INPUT_MAX_RETRIES: ${{ inputs.max-retries }} diff --git a/.github/actions/retry-approved-pr-ci/src/retry_ci.py b/.github/actions/retry-approved-pr-ci/src/retry_ci.py new file mode 100644 index 00000000..5102f9d6 --- /dev/null +++ b/.github/actions/retry-approved-pr-ci/src/retry_ci.py @@ -0,0 +1,216 @@ +"""Bounded, opt-in retries. All policy reads run again before the single write.""" + +import json +import os +import re +import subprocess +from dataclasses import dataclass +from pathlib import Path + + +class PolicySkip(Exception): + pass + + +@dataclass(frozen=True) +class Config: + repository: str + run_id: int + run_attempt: int + workflow_ids: frozenset[int] + max_retries: int = 3 + enabled: bool = False + dry_run: bool = True + + def validate(self): + if (not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9-]*/[A-Za-z0-9_.-]+", self.repository) + or self.repository.split("/")[-1] in (".", "..")): + raise ValueError("repository must be owner/name") + ids = [self.run_id, self.run_attempt, *self.workflow_ids] + if not self.workflow_ids or any(type(x) is not int or x <= 0 for x in ids): + raise ValueError("run, attempt and workflow IDs must be positive integers") + if type(self.max_retries) is not int or not 0 <= self.max_retries <= 3: + raise ValueError("max-retries must be in 0..3") + if type(self.enabled) is not bool or type(self.dry_run) is not bool: + raise ValueError("enabled and dry-run must be true or false") + + +REVIEW_QUERY = """ +query($owner: String!, $name: String!, $number: Int!) { + repository(owner: $owner, name: $name) { + pullRequest(number: $number) { + state isDraft headRefOid reviewDecision + headRepository { nameWithOwner } + baseRepository { nameWithOwner } + author { login } + latestOpinionatedReviews(first: 100, writersOnly: true) { + nodes { state commit { oid } author { login __typename } } + pageInfo { hasNextPage } + } + } + } +} +""" + + +class GitHub: + """Use the runner's gh auth. Never log API errors, which can carry private data.""" + + @staticmethod + def _call(args, payload=None, parse=True): + result = subprocess.run( + ["gh", "api", *args], + input=json.dumps(payload) if payload is not None else None, + text=True, capture_output=True, timeout=30, check=False, + ) + if result.returncode: + raise RuntimeError("GitHub API request failed; check token permissions and run logs") + return json.loads(result.stdout) if parse else None + + def get(self, path): + return self._call([path]) + + def post(self, path): + # A lost response is ambiguous. Do not retry this request automatically. + return self._call(["--method", "POST", path], parse=False) + + def graphql(self, query, variables): + data = self._call(["graphql", "--input", "-"], {"query": query, "variables": variables}) + if data.get("errors"): + raise RuntimeError("GitHub GraphQL query failed") + return data + + +def require(condition, reason): + if not condition: + raise PolicySkip(reason) + + +def snapshot(config, api): + repo = config.repository + run = api.get(f"repos/{repo}/actions/runs/{config.run_id}") + require(type(run["id"]) is int and run["id"] == config.run_id, "wrong_run") + require(run["repository"]["full_name"] == repo, "wrong_repository") + require(run["head_repository"]["full_name"] == repo, "fork") + require(run["event"] == "pull_request", "unsupported_event") + require(type(run["workflow_id"]) is int and run["workflow_id"] in config.workflow_ids, + "workflow_not_allowed") + require(type(run["run_attempt"]) is int and run["run_attempt"] == config.run_attempt, + "stale_attempt") + require(run["status"] == "completed" and run["conclusion"] == "failure", "not_failed") + sha = run["head_sha"] + require(isinstance(sha, str) and re.fullmatch(r"[a-f0-9]{40}", sha), "invalid_head") + require(len(run["pull_requests"]) == 1, "ambiguous_pr") + number = run["pull_requests"][0]["number"] + require(type(number) is int and number > 0, "invalid_pr") + owner, name = repo.split("/") + data = api.graphql(REVIEW_QUERY, {"owner": owner, "name": name, "number": number}) + require(not data.get("errors"), "review_query_failed") + pr = data["data"]["repository"]["pullRequest"] + require(pr["state"] == "OPEN" and pr["isDraft"] is False, "pr_not_ready") + require(pr["headRepository"]["nameWithOwner"] == repo + and pr["baseRepository"]["nameWithOwner"] == repo, "fork") + require(pr["headRefOid"] == sha, "head_changed") + require(pr["reviewDecision"] == "APPROVED", "approval_required") + reviews = pr["latestOpinionatedReviews"] + require(reviews["pageInfo"]["hasNextPage"] is False, "too_many_reviews") + require(all(r["state"] != "CHANGES_REQUESTED" for r in reviews["nodes"]), + "changes_requested") + require(any(r["state"] == "APPROVED" and r["commit"]["oid"] == sha + and r["author"]["__typename"] == "User" + and r["author"]["login"] != pr["author"]["login"] + for r in reviews["nodes"]), "current_human_approval_required") + + # Do not revive superseded runs. Pagination is bounded and fails closed. + found = False + for page in range(1, 11): + path = (f"repos/{repo}/actions/workflows/{run['workflow_id']}/runs" + f"?event=pull_request&head_sha={sha}&per_page=100&page={page}") + runs = api.get(path)["workflow_runs"] + require(isinstance(runs, list), "invalid_runs") + for other in runs: + require(type(other["id"]) is int, "invalid_runs") + # A missing association cannot prove a different PR. Stop conservatively. + require(other["head_sha"] == sha and other["workflow_id"] == run["workflow_id"], + "invalid_runs") + require(isinstance(other["pull_requests"], list) and other["pull_requests"], + "unassociated_run") + same_pr = any(p["number"] == number for p in other["pull_requests"]) + require(not same_pr or other["id"] <= config.run_id, "superseded_run") + found |= same_pr and other["id"] == config.run_id + if len(runs) < 100: + break + else: + raise PolicySkip("too_many_runs") + require(found, "run_not_listed") + return {"head_sha": sha, "pr_number": number, "workflow_id": run["workflow_id"]} + + +def evaluate(config, api): + config.validate() + result = {"outcome": "disabled", "reason": "enable_explicitly", "head_sha": "", + "run_url": f"https://github.com/{config.repository}/actions/runs/{config.run_id}", + "retries_used": config.run_attempt - 1} + if not config.enabled: + return result + try: + first = snapshot(config, api) + result.update(first) + if config.run_attempt - 1 >= config.max_retries: + diagnosis = dict(first, version=1, repository=config.repository, + run_id=config.run_id, run_attempt=config.run_attempt, + run_url=result["run_url"], reason="retry_budget_exhausted") + return dict(result, outcome="exhausted", reason="author_diagnosis_required", + diagnosis=json.dumps(diagnosis, sort_keys=True)) + second = snapshot(config, api) + require(first == second, "state_changed") + if config.dry_run: + return dict(result, outcome="would_retry", reason="dry_run") + api.post(f"repos/{config.repository}/actions/runs/{config.run_id}/rerun-failed-jobs") + return dict(result, outcome="retried", reason="failed_jobs_requested") + except PolicySkip as exc: + return dict(result, outcome="skipped", reason=str(exc)) + except (KeyError, TypeError, ValueError, RuntimeError, subprocess.SubprocessError): + return dict(result, outcome="error", reason="verify_api_access_and_state") + + +def boolean(name, default): + value = os.environ.get(name, default) + if value not in ("true", "false"): + raise ValueError(f"{name} must be true or false") + return value == "true" + + +def main(): + try: + config = Config( + repository=os.environ["GITHUB_REPOSITORY"], + run_id=int(os.environ["INPUT_RUN_ID"]), + run_attempt=int(os.environ["INPUT_RUN_ATTEMPT"]), + workflow_ids=frozenset(int(x.strip()) for x in os.environ["INPUT_WORKFLOW_IDS"].split(",")), + max_retries=int(os.environ.get("INPUT_MAX_RETRIES", "3")), + enabled=boolean("INPUT_ENABLED", "false"), + dry_run=boolean("INPUT_DRY_RUN", "true"), + ) + result = evaluate(config, GitHub()) + except (KeyError, ValueError): + result = {"outcome": "error", "reason": "invalid_configuration"} + # Only fixed vocabulary, validated IDs/SHA and constructed URLs reach outputs. + print(json.dumps(result, sort_keys=True)) + if os.environ.get("GITHUB_OUTPUT"): + with Path(os.environ["GITHUB_OUTPUT"]).open("a") as output: + for key, value in result.items(): + output.write(f"{key}={value}\n") + if os.environ.get("GITHUB_STEP_SUMMARY"): + with Path(os.environ["GITHUB_STEP_SUMMARY"]).open("a") as summary: + summary.write(f"CI retry: {result['outcome']} ({result['reason']}).\n\n") + if result.get("run_url"): + summary.write(f"[Workflow run]({result['run_url']})\n\n") + if result["outcome"] in ("exhausted", "error"): + summary.write("PR author: inspect the failed jobs and diagnose the cause before another run.\n") + # Advisory automation does not turn an unrelated PR check red. + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.github/actions/retry-approved-pr-ci/test/test_retry_ci.py b/.github/actions/retry-approved-pr-ci/test/test_retry_ci.py new file mode 100644 index 00000000..891f31ae --- /dev/null +++ b/.github/actions/retry-approved-pr-ci/test/test_retry_ci.py @@ -0,0 +1,342 @@ +import copy +import importlib.util +from pathlib import Path +import sys + +import pytest + +MODULE_PATH = Path(__file__).parents[1] / 'src' / 'retry_ci.py' +spec = importlib.util.spec_from_file_location('retry_ci', MODULE_PATH) +retry_ci = importlib.util.module_from_spec(spec) +sys.modules[spec.name] = retry_ci +spec.loader.exec_module(retry_ci) + +REPO = 'acme/project' +SHA = 'a' * 40 +RUN_PATH = f'repos/{REPO}/actions/runs/123' +POST_PATH = RUN_PATH + '/rerun-failed-jobs' + + +def run(**changes): + result = { + 'id': 123, 'workflow_id': 55, 'event': 'pull_request', + 'status': 'completed', 'conclusion': 'failure', 'head_sha': SHA, + 'run_attempt': 1, 'repository': {'full_name': REPO}, + 'head_repository': {'full_name': REPO}, 'pull_requests': [{'number': 42}], + 'html_url': 'https://github.com/acme/project/actions/runs/123', + 'created_at': '2026-10-30T10:00:00Z', + } + result.update(changes) + return result + + +def pr(**changes): + result = { + 'number': 42, 'state': 'OPEN', 'isDraft': False, 'headRefOid': SHA, + 'headRepository': {'nameWithOwner': REPO}, + 'baseRepository': {'nameWithOwner': REPO}, + 'author': {'login': 'author'}, 'reviewDecision': 'APPROVED', + 'latestOpinionatedReviews': { + 'nodes': [{'state': 'APPROVED', 'commit': {'oid': SHA}, + 'author': {'login': 'reviewer', '__typename': 'User'}}], + 'pageInfo': {'hasNextPage': False}, + }, + } + result.update(changes) + return result + + +class API: + def __init__(self): + self.runs = [run()] + self.prs = [pr()] + self.pages = {1: [run()]} + self.calls = [] + self.posts = [] + self.post_error = None + + @staticmethod + def next_state(items): + return copy.deepcopy(items.pop(0) if len(items) > 1 else items[0]) + + def get(self, path): + self.calls.append(('get', path)) + if path == RUN_PATH: + return self.next_state(self.runs) + if '/actions/workflows/55/runs?' in path: + from urllib.parse import parse_qs, urlsplit + params = parse_qs(urlsplit(path).query) + page = int(params.get('page', ['1'])[0]) + return {'workflow_runs': copy.deepcopy(self.pages.get(page, [])), + 'total_count': sum(map(len, self.pages.values()))} + raise AssertionError(f'unexpected GET {path}') + + def graphql(self, query, variables): + self.calls.append(('graphql', variables)) + return {'data': {'repository': {'pullRequest': self.next_state(self.prs)}}} + + def post(self, path): + self.calls.append(('post', path)) + self.posts.append(path) + if self.post_error: + raise self.post_error + + +def config(**changes): + values = dict(repository=REPO, run_id=123, run_attempt=1, + workflow_ids=frozenset({55}), max_retries=3, + enabled=True, dry_run=False) + values.update(changes) + return retry_ci.Config(**values) + + +def evaluate(api, **changes): + return retry_ci.evaluate(config(**changes), api) + + +def assert_safe(result, api): + assert result['outcome'] in {'skipped', 'exhausted', 'error'} + assert not api.posts + + +def test_happy_path_revalidates_and_posts_once(): + api = API() + result = evaluate(api) + assert result['outcome'] == 'retried' + assert api.posts == [POST_PATH] + assert sum(call == ('get', RUN_PATH) for call in api.calls) == 2 + assert sum(call[0] == 'graphql' for call in api.calls) == 2 + assert result['head_sha'] == SHA + assert result['retries_used'] == 0 + + +def test_dry_run_checks_but_does_not_post(): + api = API() + result = evaluate(api, dry_run=True) + assert result['outcome'] == 'would_retry' + assert any(call[0] == 'graphql' for call in api.calls) + assert not api.posts + + +def test_disabled_has_no_api_calls(): + api = API() + assert evaluate(api, enabled=False)['outcome'] == 'disabled' + assert not api.calls + + +def test_attempt_four_is_exhausted_even_if_manually_rerun(): + api = API() + api.runs = [run(run_attempt=4)] + api.pages = {1: [run(run_attempt=4)]} + result = evaluate(api, run_attempt=4) + assert result['outcome'] == 'exhausted' + assert result['retries_used'] == 3 + assert not api.posts + + +def test_attempt_mismatch_rejects_duplicate_event(): + api = API() + api.runs = [run(run_attempt=2)] + assert_safe(evaluate(api, run_attempt=1), api) + + +@pytest.mark.parametrize('change', [ + {'headRefOid': 'b' * 40}, {'reviewDecision': 'REVIEW_REQUIRED'}, + {'isDraft': True}, {'headRepository': {'nameWithOwner': 'other/fork'}}, + {'state': 'CLOSED'}, +]) +def test_pr_state_must_remain_eligible_before_write(change): + api = API() + api.prs = [pr(), pr(**change)] + assert_safe(evaluate(api), api) + + +def test_newer_run_on_second_page_blocks_retry(): + api = API() + # A full first page forces pagination even when the matching newer run is later. + api.pages = {1: [run(id=123 - i, pull_requests=[{'number': 1000 + i}]) for i in range(100)], + 2: [run(id=124, created_at='2026-10-30T10:01:00Z')]} + assert_safe(evaluate(api), api) + assert any('page=2' in call[1] for call in api.calls if call[0] == 'get') + + +@pytest.mark.parametrize('changed_run', [ + {'head_repository': {'full_name': 'other/fork'}}, + {'pull_requests': []}, {'pull_requests': [{'number': 42}, {'number': 43}]}, + {'event': 'push'}, {'status': 'in_progress'}, {'conclusion': 'success'}, + {'workflow_id': 99}, {'head_sha': None}, {'repository': None}, +]) +def test_ineligible_or_missing_run_metadata_fails_closed(changed_run): + api = API() + api.runs = [run(**changed_run)] + assert_safe(evaluate(api), api) + + +@pytest.mark.parametrize('reviews', [ + {'nodes': [{'state': 'APPROVED', 'commit': {'oid': 'b' * 40}, + 'author': {'login': 'reviewer', '__typename': 'User'}}], 'pageInfo': {'hasNextPage': False}}, + {'nodes': [{'state': 'CHANGES_REQUESTED', 'commit': {'oid': SHA}, + 'author': {'login': 'reviewer', '__typename': 'User'}}], 'pageInfo': {'hasNextPage': False}}, + {'nodes': [], 'pageInfo': {'hasNextPage': False}}, + {'nodes': [], 'pageInfo': {'hasNextPage': True}}, +]) +def test_reviews_fail_closed(reviews): + api = API() + api.prs = [pr(latestOpinionatedReviews=reviews)] + assert_safe(evaluate(api), api) + + +def test_malformed_response_fails_closed(): + api = API() + api.prs = [None] + assert_safe(evaluate(api), api) + + +def test_failed_post_is_not_retried(): + api = API() + api.post_error = RuntimeError('network error') + result = evaluate(api) + assert result['outcome'] == 'error' + assert api.posts == [POST_PATH] + + +@pytest.mark.parametrize('changes', [ + {'repository': 'bad'}, {'repository': '../bad'}, {'run_id': 0}, + {'run_attempt': 0}, {'workflow_ids': frozenset()}, {'workflow_ids': frozenset({0})}, + {'max_retries': -1}, {'max_retries': 4}, {'enabled': 'true'}, {'dry_run': 1}, +]) +def test_invalid_configuration_is_rejected_even_in_dry_run(changes): + cfg = config(**changes) + with pytest.raises(ValueError): + cfg.validate() + + +def test_run_attempt_changes_before_write(): + api = API() + api.runs = [run(), run(run_attempt=2)] + assert_safe(evaluate(api), api) + + +def test_newer_run_appearing_before_write_blocks_post(): + class NewRunAPI(API): + def __init__(self): + super().__init__() + self.list_reads = 0 + + def get(self, path): + if '/actions/workflows/55/runs?' in path: + self.list_reads += 1 + if self.list_reads == 2: + self.pages = {1: [run(id=124, created_at='2026-10-30T10:01:00Z'), run()]} + return super().get(path) + + api = NewRunAPI() + assert_safe(evaluate(api), api) + assert api.list_reads >= 2 + + +def test_exhausted_handoff_contains_only_verified_identifiers(): + import json + api = API() + api.runs = [run(run_attempt=4, name='untrusted\ninjected=1')] + api.pages = {1: [run(run_attempt=4)]} + result = evaluate(api, run_attempt=4) + assert result['outcome'] == 'exhausted' + assert json.loads(result['diagnosis']) == { + 'version': 1, 'repository': REPO, 'pr_number': 42, 'head_sha': SHA, + 'workflow_id': 55, 'run_id': 123, 'run_attempt': 4, + 'run_url': 'https://github.com/acme/project/actions/runs/123', + 'reason': 'retry_budget_exhausted', + } + assert not api.posts + assert all(call[0] in {'get', 'graphql'} for call in api.calls) + + +@pytest.mark.parametrize('author', [ + {'login': 'reviewer', '__typename': 'Bot'}, + {'login': 'author', '__typename': 'User'}, +]) +def test_bot_or_pr_author_cannot_supply_approval(author): + api = API() + api.prs = [pr(latestOpinionatedReviews={ + 'nodes': [{'state': 'APPROVED', 'commit': {'oid': SHA}, 'author': author}], + 'pageInfo': {'hasNextPage': False}, + })] + assert_safe(evaluate(api), api) + + +@pytest.mark.parametrize('budget,attempt,expected', [ + (0, 1, 'exhausted'), (0, 2, 'exhausted'), (0, 3, 'exhausted'), (0, 4, 'exhausted'), + (3, 1, 'retried'), (3, 2, 'retried'), (3, 3, 'retried'), (3, 4, 'exhausted'), +]) +def test_retry_budget_boundaries(budget, attempt, expected): + api = API() + api.runs = [run(run_attempt=attempt)] + api.pages = {1: [run(run_attempt=attempt)]} + result = evaluate(api, max_retries=budget, run_attempt=attempt) + assert result['outcome'] == expected + assert result['retries_used'] == attempt - 1 + assert api.posts == ([POST_PATH] if expected == 'retried' else []) + assert all('/merge' not in str(call) for call in api.calls) + + +@pytest.mark.parametrize('failure_point', ['run', 'graphql', 'listing']) +def test_api_read_errors_fail_closed_and_do_not_echo_error(failure_point): + class BrokenAPI(API): + def get(self, path): + if (failure_point == 'run' and path == RUN_PATH or + failure_point == 'listing' and '/actions/workflows/' in path): + raise RuntimeError('secret-token-and-untrusted-message') + return super().get(path) + + def graphql(self, query, variables): + if failure_point == 'graphql': + raise RuntimeError('secret-token-and-untrusted-message') + return super().graphql(query, variables) + + api = BrokenAPI() + result = evaluate(api) + assert result['outcome'] == 'error' + assert 'secret-token' not in str(result) + assert not api.posts + + +def test_main_writes_safe_output_and_summary(monkeypatch, tmp_path, capsys): + import json + api = API() + api.runs = [run(run_attempt=4, name='untrusted\ninjected=1')] + api.pages = {1: [run(run_attempt=4)]} + output = tmp_path / 'output' + summary = tmp_path / 'summary' + for key, value in { + 'GITHUB_REPOSITORY': REPO, 'INPUT_RUN_ID': '123', 'INPUT_RUN_ATTEMPT': '4', + 'INPUT_WORKFLOW_IDS': '55', 'INPUT_MAX_RETRIES': '3', 'INPUT_ENABLED': 'true', + 'INPUT_DRY_RUN': 'false', 'GITHUB_OUTPUT': str(output), + 'GITHUB_STEP_SUMMARY': str(summary), + }.items(): + monkeypatch.setenv(key, value) + monkeypatch.setattr(retry_ci, 'GitHub', lambda: api) + assert retry_ci.main() == 0 + result = json.loads(capsys.readouterr().out) + outputs = dict(line.split('=', 1) for line in output.read_text().splitlines()) + assert json.loads(outputs['diagnosis']) == json.loads(result['diagnosis']) + assert outputs['outcome'] == 'exhausted' + assert 'inspect the failed jobs' in summary.read_text() + assert 'untrusted' not in output.read_text() + summary.read_text() + assert not api.posts + + +def test_main_invalid_input_cannot_inject_output(monkeypatch, tmp_path, capsys): + import json + output = tmp_path / 'output' + summary = tmp_path / 'summary' + monkeypatch.setenv('GITHUB_REPOSITORY', 'acme/project\ninjected=token') + monkeypatch.setenv('INPUT_RUN_ID', '123') + monkeypatch.setenv('INPUT_RUN_ATTEMPT', '1') + monkeypatch.setenv('INPUT_WORKFLOW_IDS', '55') + monkeypatch.setenv('GITHUB_OUTPUT', str(output)) + monkeypatch.setenv('GITHUB_STEP_SUMMARY', str(summary)) + assert retry_ci.main() == 0 + assert json.loads(capsys.readouterr().out) == {'outcome': 'error', 'reason': 'invalid_configuration'} + assert output.read_text() == 'outcome=error\nreason=invalid_configuration\n' + assert 'injected' not in summary.read_text() diff --git a/.github/workflows/test-retry-approved-pr-ci.yaml b/.github/workflows/test-retry-approved-pr-ci.yaml new file mode 100644 index 00000000..f37c2060 --- /dev/null +++ b/.github/workflows/test-retry-approved-pr-ci.yaml @@ -0,0 +1,25 @@ +name: Test retry-approved-pr-ci + +on: + pull_request: + paths: + - '.github/actions/retry-approved-pr-ci/**' + - '.github/workflows/test-retry-approved-pr-ci.yaml' + +permissions: + contents: read + +jobs: + pytest: + name: Run approved PR CI retry tests + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Install uv + uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 + - name: Run retry policy tests + run: uv run --with pytest==8.3.3 pytest .github/actions/retry-approved-pr-ci/test/ -q diff --git a/Makefile b/Makefile index 15fee152..7dc93019 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: test test-ghcr-tag-guard test-release-on-merge test-secret-broker-request test-secret-broker-response test-semver-validation test-setup-semstat test-linear-pr-commenter test-link-backport-prs test-fork-review-check test-select-backport-source-prs test-release-notification test-linear-release-sync test-linear-cve-summary test-delete-previous-head-image test-aws-test-infra test-cleanup-head-charts test-ci-test-notify test-auto-approve-bot-prs test-ai-pr-review test-ai-step test-release-summary test-publish-helm-chart test-govulncheck test-go-licenses test-run-ginkgo test-sticky-pr-comment test-repository-dispatch test-parse-label-filter test-release-branch-freeze test-prerelease-setup test-vcluster-release test-platform-release test-release-lib test-subtree-mirror test-oss-commit-sync test-oss-mirror-staleness test-backport-legacy-allowlist test-backport-legacy-split test-backport-fix-imports test-promote-release test-wait-for-release test-cve-scan test-commitlint test-resolve-github-release test-comment-triggered-check test-conflict-check build-linear-release-sync lint check-bats-jobs install-auto-doc generate-docs check-docs help +.PHONY: test test-ghcr-tag-guard test-release-on-merge test-retry-approved-pr-ci test-secret-broker-request test-secret-broker-response test-semver-validation test-setup-semstat test-linear-pr-commenter test-link-backport-prs test-fork-review-check test-select-backport-source-prs test-release-notification test-linear-release-sync test-linear-cve-summary test-delete-previous-head-image test-aws-test-infra test-cleanup-head-charts test-ci-test-notify test-auto-approve-bot-prs test-ai-pr-review test-ai-step test-release-summary test-publish-helm-chart test-govulncheck test-go-licenses test-run-ginkgo test-sticky-pr-comment test-repository-dispatch test-parse-label-filter test-release-branch-freeze test-prerelease-setup test-vcluster-release test-platform-release test-release-lib test-subtree-mirror test-oss-commit-sync test-oss-mirror-staleness test-backport-legacy-allowlist test-backport-legacy-split test-backport-fix-imports test-promote-release test-wait-for-release test-cve-scan test-commitlint test-resolve-github-release test-comment-triggered-check test-conflict-check build-linear-release-sync lint check-bats-jobs install-auto-doc generate-docs check-docs help ACTIONS_DIR := .github/actions .PHONY: test-check-action-pins @@ -94,7 +94,10 @@ check-docs: generate-docs ## verify docs are up to date (fails if drift detected help: ## show this help @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf " %-30s %s\n", $$1, $$2}' -test: test-ghcr-tag-guard test-release-on-merge test-secret-broker-request test-secret-broker-response test-semver-validation test-setup-semstat test-linear-pr-commenter test-link-backport-prs test-fork-review-check test-select-backport-source-prs test-release-notification test-linear-release-sync test-linear-cve-summary test-delete-previous-head-image test-aws-test-infra test-cleanup-head-charts test-auto-approve-bot-prs test-ai-pr-review test-ai-step test-release-summary test-ci-test-notify test-go-licenses test-publish-helm-chart test-govulncheck test-run-ginkgo test-sticky-pr-comment test-repository-dispatch test-parse-label-filter test-release-branch-freeze test-prerelease-setup test-vcluster-release test-platform-release test-release-lib test-subtree-mirror test-oss-commit-sync test-oss-mirror-staleness test-backport-legacy-allowlist test-backport-legacy-split test-backport-fix-imports test-promote-release test-wait-for-release test-cve-scan test-commitlint test-resolve-github-release test-comment-triggered-check test-conflict-check ## run all action tests +test: test-ghcr-tag-guard test-release-on-merge test-retry-approved-pr-ci test-secret-broker-request test-secret-broker-response test-semver-validation test-setup-semstat test-linear-pr-commenter test-link-backport-prs test-fork-review-check test-select-backport-source-prs test-release-notification test-linear-release-sync test-linear-cve-summary test-delete-previous-head-image test-aws-test-infra test-cleanup-head-charts test-auto-approve-bot-prs test-ai-pr-review test-ai-step test-release-summary test-ci-test-notify test-go-licenses test-publish-helm-chart test-govulncheck test-run-ginkgo test-sticky-pr-comment test-repository-dispatch test-parse-label-filter test-release-branch-freeze test-prerelease-setup test-vcluster-release test-platform-release test-release-lib test-subtree-mirror test-oss-commit-sync test-oss-mirror-staleness test-backport-legacy-allowlist test-backport-legacy-split test-backport-fix-imports test-promote-release test-wait-for-release test-cve-scan test-commitlint test-resolve-github-release test-comment-triggered-check test-conflict-check ## run all action tests + +test-retry-approved-pr-ci: ## run approved PR CI retry pytest tests + uv run --with pytest==8.3.3 pytest $(ACTIONS_DIR)/retry-approved-pr-ci/test/ -q test: test-check-action-pins diff --git a/README.md b/README.md index 1de9d2ec..2af2af6b 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,13 @@ Location: `.github/actions/check-action-pins` See [check-action-pins README](./.github/actions/check-action-pins/README.md) for usage, runner requirements, and the release and required-check followup. +### Retry approved PR CI + +Opt-in, dry-run-first retries for allowlisted failed PR workflows with current human +approval. At most three reruns; exhaustion emits a diagnosis handoff. Never merges. + +See [retry-approved-pr-ci README](./.github/actions/retry-approved-pr-ci/README.md). + ### Secret Broker Request Action Validates a short-lived secret request from an immutable commit, verifies the