Repository navigation
Description
Goal
Check that every pull request references an issue of this repository, and flag the ones that do not. A new workflow reads each pull request's description for a closing keyword (Closes #N, Fixes #N, Resolves #N) or for Part of #N / Refs #N. A pull request with none gets a needs-issue label, one comment that says how to fix it, and a failed check; editing the description to hold a reference removes the label, updates the comment and passes the check.
The contribution guide and the pull request template already ask for the reference on the first line of the description. Nothing checks it, and 41 of the 300 most recently merged pull requests (14%) hold no such reference. opencode runs such a check; this issue adapts it to NVX.
Background
opencode (anomalyco/opencode, formerly sst/opencode, at 7b5b7a80) has three relevant files:
triage-pr-standards.ymlruns onpull_request_target(opened,edited,synchronize) withcontents: readandpull-requests: write. It skips bots, the logins in.github/TEAM_MEMBERS, and titles that start withdocs,refactororfeat. For the rest it asks GraphQL forclosingIssuesReferences, then falls back to the description, because GitHub fills that field only for pull requests that target the default branch. The fallback acceptscloses|fixes|resolves #N, or any#N, in the template's "Issue for this PR" section. With no link it addsneeds:issueand posts one comment, found again by a hidden marker, and it removes the label once the link exists.triage-compliance-close.ymlruns every 30 minutes and closes the issues and pull requests that stay labelledneeds:compliance, opencode's template check, 72 hours after its comment.CONTRIBUTING.mdstates the policy: bug fixes, chores and tests must reference an existing issue.
NVX takes the trigger and permissions, the two sources of truth (GitHub's link and the description), and a label with one marker comment that follows the verdict. It leaves the bare #N fallback, which any #N satisfies, a pull request's number included (24 of the 300 pull requests below hold only such a mention); the team list and the title exemptions, because the convention has no exception for maintainers and NVX titles name an area, not a type; and the automatic closing.
Current state
At dev b6e0a7be:
- No workflow or script checks the reference.
ci.ymldoes not listen foredited(Stacked pull requests), so the check needs a workflow of its own.test-changes.ymlis the closest precedent:pull_request_targetfor pull requests from forks too,permissions: {}withcontents: readandpull-requests: writeon the job, a checkout of the default branch withpersist-credentials: false, a default-branch script that reads the pull request as data, one marker comment, and concurrency per pull request. No workflow usesactions/github-script; scripts such asscripts/label_issues.pyandscripts/check_test_changes.pydo the work, with unit tests.- Remote actions are pinned to full commit SHAs with a release comment, and a test enforces it (Action references, ci: pin GitHub Actions references to full commit SHAs #1664).
The 300 most recently merged pull requests (merged 2026-10-30 to 2026-11-17), with the rule of the first task applied to each description:
| The description holds | Pull requests |
|---|---|
| a closing keyword and an issue | 101 |
Part of or Refs and an issue |
158 |
only a bare #N, such as "stacked on #1195" |
24 |
| no reference | 17 |
- The last two rows, 41 pull requests, would have been flagged. 8 are release pins of the OpenVMM chain (
openvmm: pin microVM v7.0, reorganized onto upstream mainand its kin: openvmm: pin microVM v7.0, reorganized onto upstream main #1668, openvmm: pin microVM v6.1, rebased onto upstream main #1567, openvmm: pin microVM v6.0, reorganized onto upstream main #1398, openvmm: pin microVM v5.1, rebased onto upstream main #1338, openvmm: pin microVM v5.0, reorganized onto upstream main #1313, openvmm: pin microVM v4.0, reorganized onto upstream main #1238, openvmm: pin microVM v3.0, reorganized onto upstream main #1133, openvmm: pin microVM v2.4, rebased onto upstream main #1075), which complete no NVX issue; the pin of an OpenVMM fix, such as openvmm: pin the XSAVE restore's bounds check #1642, opens withFixes #N. The rest are chores, runner updates, docs and features. - GitHub linked 93 of the 300: a closing keyword links only a pull request that targets
dev, andPart ofandRefslink none. It linked none of the 158Part oforRefspull requests, nor the 8 closing-keyword ones with another base; 136 of the 300 target another base (feature/nvx-refactor, stacked pull requests). A check on GitHub's link alone would flag 207 (69%), so the check reads the description too.
Tasks
- Script. Add
scripts/check_issue_link.py, with its tests inscripts/test_issue_link.py(ascheck_test_changes.pyhastest_test_changes.py). It reads the pull request through the API, not from the event payload, so that a run that starts late never acts on stale text.- Linked means GitHub links the pull request to an issue of this repository (the GraphQL field
closingIssuesReferencesholds one), or the description holdsKEYWORD TARGET:KEYWORDisclose,closes,closed,fix,fixes,fixed,resolve,resolves,resolved,Part oforRefs, in any case, with an optional colon.TARGETis#N,microsoft/nvx#Norhttps://github.com/microsoft/nvx/issues/N, whereNis an issue of this repository, open or closed. A pull request number (the issues API returns pull requests too, with apull_requestkey), a missing number and a bare#Ndo not count.- HTML comments, fenced code and inline code are ignored: the template's guidance is an HTML comment.
- Exempt are pull requests by
dependabot[bot]orgithub-actions[bot], and those labelledno-issue, which only a maintainer can apply. A closed or merged pull request is left alone. - Linked or exempt: remove
needs-issue, rewrite the marker comment, if there is one, to say so, and exit 0. Unlinked: addneeds-issue(creating it andno-issuewhen they are missing), create or update the one marker comment, and exit 1. A GitHub API error: change nothing and exit 2. - The comment names the four forms, says that the reference goes on the first line of the description, says to open an issue first when none exists, says that editing the description runs the check again, and names
no-issue. - Find the comment as
check_test_changes.pydoes: only a comment ofgithub-actions[bot]that starts with a hidden marker, so that a comment which quotes the marker is never edited (Track nightly CI reporter bugs: an issue that quotes the report marker is closed or overwritten #741). Write a job summary with the verdict and the numbers found, and never echo the description.
- Linked means GitHub links the pull request to an issue of this repository (the GraphQL field
- Workflow. Add
.github/workflows/issue-link.yml, named "Issue link", with one job,check, named "Check that the pull request references an issue":pull_request_targetonopened,reopened,synchronize,edited,labeledandunlabeled: pull requests from forks, with any base branch, and drafts are checked, andno-issuetakes effect as soon as a maintainer applies it.permissions: {}for the workflow; the job holdscontents: readandpull-requests: write, neverissues: write.- The job checks out the default branch with
persist-credentials: falseand runs that branch's script. The pull request is data: nothing of it runs, and only its number reaches the job throughenv. - Job-level concurrency per pull request (
issue-link-<number>,cancel-in-progress: true) andtimeout-minutes: 5. - No job-level
if. A run whose job is skipped replaces the result of the run before it in the checks list: on ci: pin GitHub Actions references to full commit SHAs #1672test-changesransuccessand thenskipped, andgh pr checksreportsSKIPPED. Every event evaluates the pull request again instead. actions/checkoutpinned as intest-changes.yml.
- Wiring. Add
scripts/test_issue_link.pyto both unittest commands of.github/actions/validate-nvx/action.yml, for Linux and Windows;test_cli_validation_runs_every_script_test_modulerequires it. - Tests. In
scripts/test_issue_link.py:- the parser: each keyword, case, the colon, the three target forms, another repository, a bare
#N, an HTML comment, code, a pull request number, a missing number; - the verdict: GitHub's link alone, the description alone on a non-
devbase, an exempt author,no-issue, flagged and then fixed, an API error, a comment that quotes the marker; - the workflow's structure, modeled on
WorkflowTestsinscripts/test_label_issues.py: triggers, permissions, checkout, concurrency, nogithub.event.pull_request.*text inrun:, no job-levelif.
- the parser: each keyword, case, the colon, the three target forms, another repository, a bare
- Docs.
doc/ci.mdgets an "Issue link" section after "Test changes": what the workflow checks, why it ispull_request_target, its permissions, the exemptions, and that it runs the default branch's copy of the workflow and the script. The "Issue references" row ofdoc/contribute.mdsays that every pull request needs one, and names the workflow andno-issue.doc/project-structure.mdlists the workflow and the script. - Release pins. Section 6 of the
nvx-rebase-openvmm-chainskill, which builds the pull request of a release pin, tells the agent to applyno-issue, since a release re-pin completes no NVX issue. The pin of an OpenVMM fix keeps referencing its issue. - Verify after merge.
pull_request_targetruns the default branch's workflow, so the pull request that adds this one is not checked by it (Test changes). Verify with throwaway pull requests, closed afterwards: (a) no reference, then an edit toPart of #N; (b)Fixes #Nagainst a base other thandev; (c) one from a fork, if one is at hand.
Acceptance criteria
- A pull request whose description references no issue gets
needs-issue, one comment and a failedIssue linkcheck. Editing the description to hold a reference removes the label, rewrites the comment and passes the check, with no push. - That holds for pull requests from forks, for any base branch, and for drafts.
-
Closes,FixesandResolves(in each form),Part ofandRefscount, with#N,microsoft/nvx#Nand the issue URL. A bare#N, a pull request number, a missing number, another repository, and a reference inside a comment or code do not. - A pull request that GitHub links to an issue, from the Development sidebar too, is never flagged.
- Pull requests by Dependabot or
github-actions[bot], and those labelledno-issue, pass without a label or a comment. A closed pull request is left alone. - However many runs there are, one comment holds the verdict, and no comment of another author is edited.
- The token cannot write issues (
test_only_the_report_job_writes_and_only_issuesstill passes), no code of the pull request runs, and nogithub.event.pull_request.*text reaches a shell command. Once CodeQL's Actions analysis (ci: enable CodeQL code scanning for Actions, Python, and Rust #1655) is enabled, it reports no alert for the workflow. -
Required status check, the merge queue andci.ymlare unchanged. - The tests, the wiring, the docs and the skill change above are in, and Ruff, Pyright,
check_test_layout.pyand thevalidate-nvxsuites pass. - The verification of task 7 is done and recorded in the pull request.
Out of scope
- Closing pull requests that stay unlinked, as opencode's
triage-compliance-close.ymldoes. - Making the check required. It would also have to report on
merge_group, or the queue would wait for a required check that never reports (Merge queue). - opencode's title-format and template-compliance checks.
- Keywords in commit messages, which do not link a pull request, and checks of issues.
- Changes to
ci.yml, and a backfill: an open pull request is checked at its next event.
Notes
- The job fails although
test-changesalways passes. The label and the comment tell the author what to do, and the failed check shows it in the checks list; the check is not required, so it never blocks a merge. If a failed check proves too loud, exit 0 after publishing: nothing else changes. - The
pull_requestwebhook has no action for linking an issue from the Development sidebar, so a flagged pull request clears at its next event, or when a maintainer re-runs the job, which reads the pull request afresh. - The labels and the comment that the job writes use
GITHUB_TOKEN, whose events start no workflow run, solabeledcannot make the job trigger itself. - Dependabot pull requests, which ci: enable Dependabot version updates for Actions, Python, and Cargo #1654 would start, carry no issue, which is why that author is exempt.
- Adding the module to
validate-nvxchanges a line of CI test wiring, sotest-changeslabels the pull requesttests-modified; the pull request explains the change.
Activity
ci: check that every pull request references an issue
ci: check that every pull request references an issue
ci: check that every pull request references an issue
ci: check that every pull request references an issue
ci: check that every pull request references an issue