Skip to content

ci: check that every pull request references an issue #1676

Description

Goal

Check that every pull request references an issue of this repository, and flag the ones that do not. A new workflow reads each pull request's description for a closing keyword (Closes #N, Fixes #N, Resolves #N) or for Part of #N / Refs #N. A pull request with none gets a needs-issue label, one comment that says how to fix it, and a failed check; editing the description to hold a reference removes the label, updates the comment and passes the check.

The contribution guide and the pull request template already ask for the reference on the first line of the description. Nothing checks it, and 41 of the 300 most recently merged pull requests (14%) hold no such reference. opencode runs such a check; this issue adapts it to NVX.

Background

opencode (anomalyco/opencode, formerly sst/opencode, at 7b5b7a80) has three relevant files:

  • triage-pr-standards.yml runs on pull_request_target (opened, edited, synchronize) with contents: read and pull-requests: write. It skips bots, the logins in .github/TEAM_MEMBERS, and titles that start with docs, refactor or feat. For the rest it asks GraphQL for closingIssuesReferences, then falls back to the description, because GitHub fills that field only for pull requests that target the default branch. The fallback accepts closes|fixes|resolves #N, or any #N, in the template's "Issue for this PR" section. With no link it adds needs:issue and posts one comment, found again by a hidden marker, and it removes the label once the link exists.
  • triage-compliance-close.yml runs every 30 minutes and closes the issues and pull requests that stay labelled needs:compliance, opencode's template check, 72 hours after its comment.
  • CONTRIBUTING.md states the policy: bug fixes, chores and tests must reference an existing issue.

NVX takes the trigger and permissions, the two sources of truth (GitHub's link and the description), and a label with one marker comment that follows the verdict. It leaves the bare #N fallback, which any #N satisfies, a pull request's number included (24 of the 300 pull requests below hold only such a mention); the team list and the title exemptions, because the convention has no exception for maintainers and NVX titles name an area, not a type; and the automatic closing.

Current state

At dev b6e0a7be:

  • No workflow or script checks the reference.
  • ci.yml does not listen for edited (Stacked pull requests), so the check needs a workflow of its own.
  • test-changes.yml is the closest precedent: pull_request_target for pull requests from forks too, permissions: {} with contents: read and pull-requests: write on the job, a checkout of the default branch with persist-credentials: false, a default-branch script that reads the pull request as data, one marker comment, and concurrency per pull request. No workflow uses actions/github-script; scripts such as scripts/label_issues.py and scripts/check_test_changes.py do the work, with unit tests.
  • Remote actions are pinned to full commit SHAs with a release comment, and a test enforces it (Action references, ci: pin GitHub Actions references to full commit SHAs #1664).

The 300 most recently merged pull requests (merged 2026-10-30 to 2026-11-17), with the rule of the first task applied to each description:

The description holds Pull requests
a closing keyword and an issue 101
Part of or Refs and an issue 158
only a bare #N, such as "stacked on #1195" 24
no reference 17

Tasks

  1. Script. Add scripts/check_issue_link.py, with its tests in scripts/test_issue_link.py (as check_test_changes.py has test_test_changes.py). It reads the pull request through the API, not from the event payload, so that a run that starts late never acts on stale text.
    • Linked means GitHub links the pull request to an issue of this repository (the GraphQL field closingIssuesReferences holds one), or the description holds KEYWORD TARGET:
      • KEYWORD is close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved, Part of or Refs, in any case, with an optional colon.
      • TARGET is #N, microsoft/nvx#N or https://github.com/microsoft/nvx/issues/N, where N is an issue of this repository, open or closed. A pull request number (the issues API returns pull requests too, with a pull_request key), a missing number and a bare #N do not count.
      • HTML comments, fenced code and inline code are ignored: the template's guidance is an HTML comment.
    • Exempt are pull requests by dependabot[bot] or github-actions[bot], and those labelled no-issue, which only a maintainer can apply. A closed or merged pull request is left alone.
    • Linked or exempt: remove needs-issue, rewrite the marker comment, if there is one, to say so, and exit 0. Unlinked: add needs-issue (creating it and no-issue when they are missing), create or update the one marker comment, and exit 1. A GitHub API error: change nothing and exit 2.
    • The comment names the four forms, says that the reference goes on the first line of the description, says to open an issue first when none exists, says that editing the description runs the check again, and names no-issue.
    • Find the comment as check_test_changes.py does: only a comment of github-actions[bot] that starts with a hidden marker, so that a comment which quotes the marker is never edited (Track nightly CI reporter bugs: an issue that quotes the report marker is closed or overwritten #741). Write a job summary with the verdict and the numbers found, and never echo the description.
  2. Workflow. Add .github/workflows/issue-link.yml, named "Issue link", with one job, check, named "Check that the pull request references an issue":
    • pull_request_target on opened, reopened, synchronize, edited, labeled and unlabeled: pull requests from forks, with any base branch, and drafts are checked, and no-issue takes effect as soon as a maintainer applies it.
    • permissions: {} for the workflow; the job holds contents: read and pull-requests: write, never issues: write.
    • The job checks out the default branch with persist-credentials: false and runs that branch's script. The pull request is data: nothing of it runs, and only its number reaches the job through env.
    • Job-level concurrency per pull request (issue-link-<number>, cancel-in-progress: true) and timeout-minutes: 5.
    • No job-level if. A run whose job is skipped replaces the result of the run before it in the checks list: on ci: pin GitHub Actions references to full commit SHAs #1672 test-changes ran success and then skipped, and gh pr checks reports SKIPPED. Every event evaluates the pull request again instead.
    • actions/checkout pinned as in test-changes.yml.
  3. Wiring. Add scripts/test_issue_link.py to both unittest commands of .github/actions/validate-nvx/action.yml, for Linux and Windows; test_cli_validation_runs_every_script_test_module requires it.
  4. Tests. In scripts/test_issue_link.py:
    • the parser: each keyword, case, the colon, the three target forms, another repository, a bare #N, an HTML comment, code, a pull request number, a missing number;
    • the verdict: GitHub's link alone, the description alone on a non-dev base, an exempt author, no-issue, flagged and then fixed, an API error, a comment that quotes the marker;
    • the workflow's structure, modeled on WorkflowTests in scripts/test_label_issues.py: triggers, permissions, checkout, concurrency, no github.event.pull_request.* text in run:, no job-level if.
  5. Docs. doc/ci.md gets an "Issue link" section after "Test changes": what the workflow checks, why it is pull_request_target, its permissions, the exemptions, and that it runs the default branch's copy of the workflow and the script. The "Issue references" row of doc/contribute.md says that every pull request needs one, and names the workflow and no-issue. doc/project-structure.md lists the workflow and the script.
  6. Release pins. Section 6 of the nvx-rebase-openvmm-chain skill, which builds the pull request of a release pin, tells the agent to apply no-issue, since a release re-pin completes no NVX issue. The pin of an OpenVMM fix keeps referencing its issue.
  7. Verify after merge. pull_request_target runs the default branch's workflow, so the pull request that adds this one is not checked by it (Test changes). Verify with throwaway pull requests, closed afterwards: (a) no reference, then an edit to Part of #N; (b) Fixes #N against a base other than dev; (c) one from a fork, if one is at hand.

Acceptance criteria

  • A pull request whose description references no issue gets needs-issue, one comment and a failed Issue link check. Editing the description to hold a reference removes the label, rewrites the comment and passes the check, with no push.
  • That holds for pull requests from forks, for any base branch, and for drafts.
  • Closes, Fixes and Resolves (in each form), Part of and Refs count, with #N, microsoft/nvx#N and the issue URL. A bare #N, a pull request number, a missing number, another repository, and a reference inside a comment or code do not.
  • A pull request that GitHub links to an issue, from the Development sidebar too, is never flagged.
  • Pull requests by Dependabot or github-actions[bot], and those labelled no-issue, pass without a label or a comment. A closed pull request is left alone.
  • However many runs there are, one comment holds the verdict, and no comment of another author is edited.
  • The token cannot write issues (test_only_the_report_job_writes_and_only_issues still passes), no code of the pull request runs, and no github.event.pull_request.* text reaches a shell command. Once CodeQL's Actions analysis (ci: enable CodeQL code scanning for Actions, Python, and Rust #1655) is enabled, it reports no alert for the workflow.
  • Required status check, the merge queue and ci.yml are unchanged.
  • The tests, the wiring, the docs and the skill change above are in, and Ruff, Pyright, check_test_layout.py and the validate-nvx suites pass.
  • The verification of task 7 is done and recorded in the pull request.

Out of scope

  • Closing pull requests that stay unlinked, as opencode's triage-compliance-close.yml does.
  • Making the check required. It would also have to report on merge_group, or the queue would wait for a required check that never reports (Merge queue).
  • opencode's title-format and template-compliance checks.
  • Keywords in commit messages, which do not link a pull request, and checks of issues.
  • Changes to ci.yml, and a backfill: an open pull request is checked at its next event.

Notes

  • The job fails although test-changes always passes. The label and the comment tell the author what to do, and the failed check shows it in the checks list; the check is not required, so it never blocks a merge. If a failed check proves too loud, exit 0 after publishing: nothing else changes.
  • The pull_request webhook has no action for linking an issue from the Development sidebar, so a flagged pull request clears at its next event, or when a maintainer re-runs the job, which reads the pull request afresh.
  • The labels and the comment that the job writes use GITHUB_TOKEN, whose events start no workflow run, so labeled cannot make the job trigger itself.
  • Dependabot pull requests, which ci: enable Dependabot version updates for Actions, Python, and Cargo #1654 would start, carry no issue, which is why that author is exempt.
  • Adding the module to validate-nvx changes a line of CI test wiring, so test-changes labels the pull request tests-modified; the pull request explains the change.

Metadata

Metadata

Labels

area: ciInitial title-keyword triage suggestion

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions