Skip to content

feat: compare who may bypass each repository-level ruleset - #110

Merged
zoeyrose merged 1 commit into
mainfrom
feat/ruleset-bypass-check
Nov 15, 2026
Merged

zoeyrose merged 1 commit into
mainfrom
feat/ruleset-bypass-check

Conversation

@zoeyrose

Copy link
Copy Markdown
Member

Summary

The settings check now compares who may bypass each repository-level ruleset. Until now it could not see that: a branch's effective rules do not show bypass, and only organization rulesets were compared in full.

One thing to decide by merging or not: this takes the script to 563 lines, over the 550-line stop that its own size test enforced. The pull request raises that stop to 575 and changes "about 500 lines" to "about 550" in the texts. See "Intent brief" for why I did not make room another way.

Intent brief

  • Goal: if the bypass of a repository-level ruleset goes back to "always", or gains another actor, the next run of the check says so. The owner asked for this on 2026-11-15.
  • Decisions:
    • A new optional profile key, ruleset_bypass, lists the bypass actors a repository-level ruleset may have. It is an allow-list: every actor of such a ruleset must match one entry. An empty list allows none. A ruleset with fewer actors than allowed passes, because it is stricter.
    • Entries are compared as subsets, like everything else in the baseline, so {"actor_type": "OrganizationAdmin", "bypass_mode": "pull_request"} matches whatever actor_id GitHub reports.
    • The check reads each repository-level ruleset from the listing it already fetches for the tag rules, plus one read per such ruleset. A ruleset whose bypass list is hidden from the login, or cannot be read, is a "could not verify" line.
    • The fix text names the ruleset's path and carries no command, because changing a bypass list means sending the whole ruleset back.
    • The line limit. The addition is 18 lines. The only real simplification I found is to drop the per-repository tag-rule check, which the full comparison of the organization tag ruleset now mostly covers. That is a larger change that touches both organizations' baselines, so it is not done here. Squeezing other code to save lines would make the script harder to read, which is what the limit is meant to prevent.
  • Constraints: only gh api -X GET; exit codes unchanged; standard library only; works without admin:org.
  • Out of scope: the other organization's baseline, which gets the same key in its own private repository after this merges; comparing anything else inside a repository-level ruleset.

Implementation / behavior

  • check_settings.py: check_ruleset_bypass, called for a profile that has the key; the key's shape in --validate.
  • baselines/atrinik.json: standard allows the organization owner through a pull request only; archived allows none. Atrinik has one repository-level ruleset today, on the archived nawerhals, with no bypass.
  • README.md and AGENTS.md: the new comparison, the new key, and the size limit.
  • tests/test_check_settings.py: one test with six cases (allowed, none, "always", an extra team, hidden list, unreadable ruleset); the size test's stop.

Validation

  • python3 -m unittest discover -s tests: 38 tests pass, offline.
  • python3 .github/scripts/check_repository.py: no problems. python3 check_settings.py --validate baselines/atrinik.json: valid. git diff --check: clean.
  • Live read-only run against atrinik, exit 2, with only the expected line:
could not verify: orgs/atrinik webhooks: Not Found (HTTP 404) (the login lacks the admin:org_hook scope)
  • Against a temporary copy of the other organization's baseline with the key added, the run gives the same single line. With a deliberately stricter copy that allows no bypass, it reports every one of that organization's repository-level rulesets and exits 1, which shows that it reads them all.

Limitations / follow-up

  • A profile without the key is not checked for bypass, so the other organization is covered only once its baseline has the key.
  • An override of ruleset_bypass replaces the list.

🤖 Generated with Claude Code

The effective rules of a branch do not show who may bypass them, and only
organization rulesets were compared in full. A repository-level ruleset
whose bypass went back to "always" was therefore invisible to the check.

A profile can now list the bypass actors a repository-level ruleset may
have (`ruleset_bypass`). Every actor of such a ruleset must be one of
them, so an empty list allows none and a ruleset with fewer actors than
allowed passes. A ruleset whose bypass list the login cannot see is a
could-not-verify line.

Atrinik's baseline records the owner's rule: bypass only through a pull
request, and none on the archived repository's ruleset.

The script is 563 lines, over the 550 at which its size test stopped it.
The test now stops it at 575 and the texts say about 550 lines.

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

None

@zoeyrose

Copy link
Copy Markdown
Member Author

Self-review of ba46883665dc003205256bce0de12cdef2101936.

What was reviewed

A fresh-context reviewer reviewed the whole diff and all of check_settings.py at this head. It ran the 38 tests, the repository check, --validate and the live read-only run (exit 2, the one expected line). It read the live API shape with GET calls: the ruleset listing carries source_type, organization rulesets say "Organization", and a repository ruleset's body has bypass_actors. It confirmed that Atrinik has exactly one repository-level ruleset. It checked that all six cases of the new test fail on main, and that the test would also catch a dropped source_type filter, a dropped could-not-verify line and swapped matches arguments.

It also ran these cases offline against the fixtures, and each behaved as intended: a clean organization; a failed ruleset listing (two could-not-verify lines, one for each check that could not run); an archived repository whose ruleset has an owner bypass; an override that replaces the list; a repository-level tag ruleset with an "always" bypass for an integration; a bypass_actors of null (exit 3); an unreadable repository.

Findings: none.

Noted, not changed: the fixture baseline has no ruleset_bypass, so only the new test exercises the comparison. The reviewer ran the two untested cases (failed listing, archived empty list) by hand and both were correct.

State

38 offline tests pass. The script is 563 lines; the size test now stops it at 575, and README and AGENTS.md say about 550. The description states this as the owner's decision.

@zoeyrose
zoeyrose marked this pull request as ready for review November 15, 2026 03:19
@zoeyrose zoeyrose self-assigned this Nov 15, 2026
@zoeyrose
zoeyrose requested a review from a team November 15, 2026 03:19
@zoeyrose
zoeyrose merged commit a6eefc9 into main Nov 15, 2026
6 checks passed
@zoeyrose
zoeyrose deleted the feat/ruleset-bypass-check branch November 15, 2026 03:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant