Skip to content

Accept only changelog fragments the release reads, from one shared definition - #4115

Merged
anth-volk merged 17 commits into
masterfrom
changelog-fragment-check
Oct 29, 2026
Merged

anth-volk merged 17 commits into
masterfrom
changelog-fragment-check

Conversation

@anth-volk

@anth-volk anth-volk commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4114

Summary

The Check changelog fragment PR job now accepts only the fragments the release reads, and the job and the release's version bump share one definition of a fragment.

  • One definition in .github/changelog_fragments.py. It reads the fragment directory and the types from [tool.towncrier] in pyproject.toml. A fragment is a regular file directly in changelog.d/, named <name>.<type>.md, where <type> is breaking, added, changed, fixed or removed and <name> is letters and digits in groups joined by single dots, hyphens or underscores, holding UTF-8 text with at least one visible character (not empty, only whitespace, or only invisible characters such as a zero-width space). .gitkeep is the only other file allowed. Every other file gets an error that says what is wrong and how to fix it; Report when the deployed policyengine.py version went live #4084's changelog.d/added/4083.md gets "Move it to changelog.d/4083.added.md."
  • The PR check (.github/scripts/check_changelog_fragment.sh) compares HEAD with its merge base with BASE_SHA, the event's github.event.pull_request.base.sha, and passes git diff --find-renames --name-status -z for changelog.d/ to changelog_fragments.py check-pull-request. It fails unless the PR adds a fragment of its own, and fails on every file in changelog.d/ the release can't read, including one already on the base branch (which would stop the next release). It also fails on a fragment the base branch has already released that the PR adds back with the same text, which happens when a stacked PR wasn't rebased after the PR below it was squash-merged. The job now checks out with fetch-depth: 0, sets up Python 3.12, and has contents: read; its stack condition and its place in CI complete's needs are unchanged.
  • The version bump (.github/bump_version.py) reads changelog.d/ through the same definition. It prints each file it can't read and exits with status 1 before changing pyproject.toml, and refuses to run when BUMP_BY_TYPE and the configured types differ. The bump rule is unchanged: breaking gives a major bump, added or removed a minor one, changed or fixed a patch.
  • Towncrier gets ignore = [".gitkeep"]. With the locked Towncrier (25.8.0) and no ignore, a draft build skipped changelog.d/added/4083.md and 4083.feature.md and exited 0; with it, the build fails with Invalid news fragment name. Towncrier renders an empty fragment as an empty bullet either way, which the bump, run first, now refuses.
  • The rule api.process.changelog-fragment states the accepted form, adds the versioning job to its CI wiring, and has a proof: the definition rejects Report when the deployed policyengine.py version went live #4084's file and accepts Move the live-since changelog fragment to the flat name the version bump reads #4106's rename of it. The PR skill, AGENTS.md, the PR template and the README (whose changelog section still described the retired changelog_entry.yaml) state the exact form.

How renames count

A file git reports as added or copied counts. A renamed file counts only when its old path was not a fragment name. Renaming a fragment already on master adds no release note: the release would have published it anyway, so counting the rename would let a PR pass on another PR's fragment, the gap this PR closes. Moving a file the release couldn't read into place (what #4106 did) adds a note the release would otherwise have stopped at, so it counts. Rename detection is limited to changelog.d/, so a large diff elsewhere can't exhaust git's rename limit.

GitHub's test merge commit

For a pull_request event, actions/checkout checks out refs/pull/<n>/merge, whose first parent is the event's base commit (checked on #4104: its merge ref's first parent is the API's base.sha). The merge base of BASE_SHA and that commit is BASE_SHA itself, so the comparison is exactly what merging the PR adds; fragments merged to master after the PR branched are in both sides and don't count. A re-run reuses the same event and merge commit, so the two stay consistent. A shallow clone fails with an instruction to use fetch-depth: 0. The job log of this PR's first run shows HEAD is now at aae704f1 Merge 0e798824… into cc25843f… and BASE_SHA: cc25843f….

To find fragments a release already removed, the script takes the PR's own head (the merge commit's second parent, or HEAD when HEAD is the head itself), lists with git log --diff-filter=D --raw the fragments the base branch deleted since the branch left it, and passes them with their blob ids. A fragment the PR adds counts as released only when its blob id (computed as git hash-object does) matches; a new fragment that reuses a released name counts as the PR's own.

Automated PRs

open_relock_pull_request.sh writes changelog.d/relock-dependencies.changed.md and update-policyengine-package.sh writes changelog.d/update-policyengine-bundle-${LATEST}.changed.md; their tests now check both pass the definition (the bundle for 4.1.0, 4.1.0rc1 and 4.1.0.post1). They are the only scripts that open PRs: no workflow uses create-pull-request, and gh pr create appears only in those two scripts.

Red-green commits

Red (test first) Green
1153dae75 Test what a changelog fragment is 6c5189cab Define a changelog fragment once
3567c22f1 Test that the PR check needs the PR's own readable fragment; 7a54bf5a0 empty folders and submodules dd1d2f776 Check the PR's own fragments against the shared definition
e63b7c03e Test that the bump reads fragments through the definition a37f0c1ce Bump the version through the definition
e9b27ff44 Test that the locked Towncrier stops at unreadable files 2c252186d Make Towncrier fail on unreadable files
f3b34ed70 Test that a fragment of only invisible characters is refused (review round 1) c8995b0b8 Refuse a fragment with no visible character
a95087afb Test that a fragment the base branch already released doesn't count (review round 1) 1770bb9f3 Refuse a fragment the base branch has already released
34c9a40df Test that only a released fragment's own text counts as released (review round 2) 8919c3282 Count a fragment as released only when it holds the released text

0e798824c updates the rule, the docs and adds this PR's fragment; 9aeb9d393 rewords two test docstrings. Each red commit was run at its own commit and failed for the intended reason (the module missing, the old script exiting 0, the old bump returning patch or not exiting, Towncrier exiting 0), and each green partner passed.

Environment variables and secrets

Name Read by Provider How it reaches the step
BASE_SHA check_changelog_fragment.sh the pull_request event payload, github.event.pull_request.base.sha step env: in check-changelog; the script reads ${BASE_SHA:?} and exits if it is unset
PYTHON check_changelog_fragment.sh (optional) not set in CI; defaults to python3 from actions/setup-python@v5 (3.12) tests set it to their interpreter
github.token actions/checkout@v4 the Actions runner (the job's GITHUB_TOKEN) the action's default token input; the job grants contents: read
TMPDIR mktemp in check_changelog_fragment.sh (implicitly) the runner's environment, if set mktemp falls back to /tmp when it is unset; the file is removed on exit
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION, OLD_SIMULATION_GATEWAY_URL not read by this job workflow-level env: (a literal, and the OLD_SIMULATION_GATEWAY_URL repository variable) inherited by every job; unchanged

push.yml is unchanged: the versioning job still runs python .github/bump_version.py, which now imports changelog_fragments.py from beside it, using only the standard library on the job's Python 3.12. Its APP_ID and APP_PRIVATE_KEY secrets are untouched.

Testing

  • ruff format --check . and ruff check . with ruff 0.15.12 (the version uv.lock pins): pass.
  • python -m scripts.rules guard --base-ref origin/master, python scripts/run_quality_guards.py and python -m unittest discover -s scripts/rules/tests -t .: pass; python -m scripts.rules render leaves no diff.
  • uv lock --check with uv 0.13.4: pass.
  • shellcheck 0.11.0 on check_changelog_fragment.sh: pass.
  • pytest on the new and changed tests and the workflow tests (211 tests): pass.
  • towncrier build --draft with Towncrier 25.8.0, on this branch merged into the current master: builds this PR's fragment under "Fixed"; .github/bump_version.py on the same tree infers a patch bump.
  • BASE_SHA=origin/master bash .github/scripts/check_changelog_fragment.sh on this branch: "This pull request adds 1 changelog fragment: changelog.d/changelog-fragment-check.fixed.md (fixed)."
  • No inline suppressions added (noqa count 95 before and after; no type: ignore or pyright: ignore).

Open pull requests

Each of the 29 other open PRs was merged locally into its current base, as GitHub's test merge does (the 7 that conflict were checked at their own head), and checked with this PR's script and module: all 29 pass. That includes the relock PR (#4101) and the bundle-update PR (#4047).

Self-review

This is a self-review by the authoring agent, not an independent review, in two rounds on the full diff against master. Each round tried to build a fragment or PR that the check accepts but the release can't read, or that it refuses but should accept: ./ and .. paths, symbolic links to files and folders, uppercase types, extra dots, trailing spaces, invisible text, GitHub's test merge commit, a shallow clone and stacked PRs.

Round 1

  • Substantiated: a stacked PR that wasn't rebased after the PR below it was squash-merged and released added the released fragment back, and the check counted it as the PR's own; the next release would have published that entry twice. Fixed in a95087afb/1770bb9f3.
  • Substantiated: a fragment of only a zero-width space or other format characters passed str.strip(), and Towncrier would publish an empty-looking bullet. Fixed in f3b34ed70/c8995b0b8.
  • Not a defect: ./, .., symbolic links, uppercase types, extra dots and trailing spaces are each refused with a fix; git never reports ./ or .. paths.
  • Code quality: the docs didn't say that git's rename detection decides what is a rename. The PR skill now says so.

Round 2

  • Substantiated: the round 1 fix compared paths only, so a new fragment that reused the name of another PR's fragment, released after this branch was created, was refused, and the error listed a released fragment as still on the base branch. Fixed in 34c9a40df/8919c3282 by comparing blob ids.
  • Remaining, not fixed: check_pull_request now gathers problems from four sources (the scan, files already on the base branch, changed paths with no file such as submodules, and released fragments) and could be split up. The script path that echoes the module's error when [tool.towncrier] can't be read is untested.
  • Remaining design limits: an empty folder the PR didn't change is ignored by the check, because git can't commit one, though the bump and Towncrier stop at it. In a fresh checkout that only happens with a submodule already on master. Whether a non-UTF-8 file name prints a clean annotation on the Linux runner is unverified.

🤖 Generated with Claude Code

anth-volk and others added 10 commits October 29, 2026 19:13
Cover the shared definition the PR check and the version bump will both use:
#4084's changelog.d/added/4083.md, unknown and uppercase types, other name
shapes, empty and whitespace-only files, symbolic links and a valid fragment,
with the types read from [tool.towncrier] in pyproject.toml. Check that the
fragments the relock and bundle-update pull requests write are accepted, and
add the rule proof fixtures: #4084's file and #4106's rename of it.

These fail because .github/changelog_fragments.py does not exist yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add .github/changelog_fragments.py, which the PR check and the version bump
will both read fragments through. It reads the fragment directory and types
from [tool.towncrier] in pyproject.toml. A fragment is a regular file directly
in changelog.d/ named <name>.<type>.md with a configured type and text that
isn't empty or only whitespace; .gitkeep is the only other file allowed.
Every other file, including files in folders such as #4084's
changelog.d/added/4083.md, gets a message that says what is wrong and how to
fix it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run .github/scripts/check_changelog_fragment.sh in temporary repositories
whose master holds a pending fragment. A pull request that adds no fragment,
or only edits, deletes or renames a pending one, must fail; so must #4084's
changelog.d/added/4083.md, other unreadable files, and an unreadable file
already on master. Moving a file the release couldn't read into place counts
as adding a fragment. On GitHub's test merge commit, only the pull request's
own changes count, and a shallow clone or an unknown base fails with what to
fix. The job must fetch the history and pass the event's base commit.

These fail because the check still counts every file under changelog.d/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… out

Git can't commit an empty folder, so one in a checkout is either left behind
by a local git mv or is a submodule. List empty folders apart from the other
problems, so the PR check can ignore a leftover one, and fail a pull request
that adds a submodule to changelog.d/. Also write the byte-order mark in a
test case as an escape.

These fail because the definition still reports every empty folder as a
problem and the check still counts every file under changelog.d/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check_changelog_fragment.sh now compares HEAD with its merge base with
BASE_SHA, the event's base commit, and passes git's name-status output for
changelog.d/ to changelog_fragments.py check-pull-request. The check fails
unless the pull request adds a fragment of its own and every file in
changelog.d/ is a fragment, with one error annotation per problem. A rename of
a fragment already on the base branch adds no release note, so it doesn't
count; moving a file the release couldn't read into place does. A shallow
clone or a base commit the clone lacks fails with how to fetch the history.

On GitHub's test merge commit, the first parent is BASE_SHA, so the
comparison is exactly what merging the pull request adds. The job now fetches
the full history, sets up Python 3.12, passes
github.event.pull_request.base.sha as BASE_SHA and reads with contents: read
only. Empty folders are listed apart from the other problems, so a folder a
local git mv left behind doesn't fail the check, while a submodule does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
infer_bump now takes the repository root and must read changelog.d/ through
changelog_fragments.py, keep its rule (breaking gives a major bump, added or
removed a minor one, changed or fixed a patch), refuse to run when its bump
table and the types in [tool.towncrier] differ, and stop with status 1 at any
file it can't read, such as #4084's changelog.d/added/4083.md or an empty
fragment, instead of skipping it. Run the script as the versioning job does,
from a copy of .github/.

These fail because bump_version.py still reads every top-level file, takes a
type from any suffix and skips the rest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bump_version.py imports changelog_fragments.py from beside it, as
python .github/bump_version.py puts .github/ first on sys.path, and reads
changelog.d/ through FragmentDefinition. It prints each file it can't read and
exits with status 1 before changing pyproject.toml, instead of skipping the
file, and refuses to run when BUMP_BY_TYPE and the types in [tool.towncrier]
differ. The bump rule is unchanged: breaking gives a major bump, added or
removed a minor one, and changed or fixed a patch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run the Towncrier version uv.lock pins on the repository's [tool.towncrier]
configuration, as the release's changelog build does. With #4084's
changelog.d/added/4083.md, an unknown type or an uppercase type, the build must
fail instead of leaving the file out; the fragments the shared definition
accepts must still build.

These fail because [tool.towncrier] has no ignore setting, so Towncrier 25.8.0
skips such files and builds without them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Set ignore = [".gitkeep"] in [tool.towncrier]. With an ignore list, the
locked Towncrier (25.8.0) fails on a folder or a file whose name has no
configured type, instead of leaving it out of CHANGELOG.md. It still renders an
empty fragment as an empty bullet, which the version bump, run first, now
refuses. Note beside the configuration which files read it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Update api.process.changelog-fragment: its statement names the accepted form
and the shared definition, its rationale points to the new "Changelog
fragments" section of the PR skill, its scope adds the definition, the bump
and the check script, its CI wiring adds the versioning job's bump step, and
its proof runs the definition on #4084's fragment and #4106's rename of it.
Regenerate docs/generated/rules.md and the AGENTS.md rules block.

State the accepted form, changelog.d/<name>.<type>.md, in the PR skill,
AGENTS.md, the PR template and the README, whose changelog section still
described the retired changelog_entry.yaml. Add this pull request's fragment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.84%. Comparing base (2a03173) to head (9aeb9d3).
⚠️ Report is 22 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #4115       +/-   ##
===========================================
+ Coverage   50.79%   87.84%   +37.04%     
===========================================
  Files         255      271       +16     
  Lines       14658    16081     +1423     
  Branches     2342     2629      +287     
===========================================
+ Hits         7446    14126     +6680     
+ Misses       6585     1285     -5300     
- Partials      627      670       +43     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

anth-volk and others added 4 commits October 29, 2026 19:34
Review finding: a fragment holding only a zero-width space, or other Unicode
format characters such as a word joiner or a soft hyphen, passes the
str.strip() test, and Towncrier publishes it as a bullet that looks empty.
Require at least one visible character, and keep accepting text that has one.

These fail because the definition only checks that stripping whitespace
leaves something.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The definition now accepts a fragment's text only when it has a character in
a visible Unicode category: a letter, mark, number, punctuation mark or
symbol. A file of only whitespace, a byte-order mark or format characters such
as a zero-width space gets "holds no visible text" or "holds only whitespace",
in the PR check and in the version bump alike.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review finding: when a lower pull request in a stack is squash-merged and the
release removes its fragment, an upper branch that wasn't rebased still holds
the lower pull request's commits. GitHub's test merge commit then adds that
fragment back, the check counted it as the upper pull request's own, and the
next release would publish the entry a second time. Cover the merge commit,
the pull request's own head, and a pull request that also adds its own
fragment.

These fail because the check doesn't look at what the base branch removed
after the pull request's branch left it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check_changelog_fragment.sh now finds the pull request's own head (the
second parent of GitHub's test merge commit, or HEAD), and lists the
fragments the base branch removed between the point the branch left it and
BASE_SHA, with git log --diff-filter=D. changelog_fragments.py
check-pull-request reads them with --released-paths: a fragment the pull
request adds back is a problem that says to rebase, and it doesn't count as
the pull request's own. A fragment the branch inherited from the base and the
release removed stays removed in the merge, so it isn't affected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 2 commits October 29, 2026 19:41
Second review finding: the released-fragment check compared paths only, so a
new fragment that reused the name of another pull request's fragment,
released after this branch was cut, was refused as if merging would publish
the old entry again. The "no fragment of its own" message also listed a
released fragment as already on the base branch. A fragment counts as
released only when its text is the blob the base branch removed; read those
blobs from git log --raw, and compute a file's blob id as git hash-object
does.

These fail because the check still matches released fragments by path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check_changelog_fragment.sh now passes git log --raw deletions, which carry
the blob each removed fragment held, as --released-fragments.
changelog_fragments.py computes the blob id of each fragment the pull request
adds, as git hash-object does (SHA-1, or SHA-256 when the ids are that long),
and refuses it as already released only when it matches a blob the base
branch removed at that path. A new fragment that reuses a released name now
counts, and a released fragment is no longer listed as already on the base
branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two test docstrings described where a pull request branch starts with a
word that also means slicing; name the operation instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review October 29, 2026 17:39
@anth-volk
anth-volk requested a review from a team as a code owner October 29, 2026 17:39
@anth-volk
anth-volk merged commit 95c8da1 into master Oct 29, 2026
15 checks passed
@anth-volk
anth-volk deleted the changelog-fragment-check branch October 29, 2026 17:39

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make the changelog fragment check accept only fragments the release reads

1 participant