Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,9 @@ jobs:

- name: Check repository files
run: python3 .github/scripts/check_repository.py

- name: Test the settings check (offline fixtures)
run: python3 -m unittest discover -s tests -v

- name: Validate Atrinik's baseline file
run: python3 check_settings.py --validate baselines/atrinik.json
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
/build/
__pycache__/
59 changes: 39 additions & 20 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,36 +1,55 @@
# Atrinik GitHub settings repository guide

- This repository is the home of Atrinik's GitHub organization settings
record. Read `README.md` first. Today it holds `config/planning.json` and a
small CI check; a read-only settings check and Atrinik's baseline file are
coming in a later pull request.
- This repository holds Atrinik's settings baseline
(`baselines/atrinik.json`) and the read-only check that compares it with
the live organization (`check_settings.py`). Read `README.md` first.
- Run the check before and after any settings change, and report any
difference you did not cause first:
`python3 check_settings.py baselines/atrinik.json`. Exit 0 is a match, 1
differences, 2 only "could not verify", 3 could not run. The organization
webhooks line is expected: the everyday login lacks `admin:org_hook`.
- How a change is made: change the setting in GitHub under the owner's login,
only when the current task authorizes that change; run the check; update
the baseline until the check is clean; open a pull request here that quotes
the clean output. Live state wins and the record catches up in the same
task. Never record a change first and apply it later.
- The check only reads. Never run its fix commands unless the current task
authorizes that exact change. Never add code to it that writes, stores a
token or runs on a schedule.
- Keep within its limits: nothing that writes, no dates that fail it, at
most three repository profiles, about ten per-repository exceptions, about
500 lines, standard library only. If a change would cross one, simplify
instead.
- Another organization's baseline can be checked by path. Keep a private
organization's baseline, its check output and its names out of this public
repository, its pull requests and its CI logs. Test fixtures are invented.
- The settings automation that used to live here (publisher, Project
synchronizer, drift audit, Project health check, community-health generator,
semantic release) was retired on 2026-11-14 by owner decision: there is no
standing organization-owner credential any more, and the rules now live in
organization rulesets that target all repositories. Do not restore, run or
imitate those tools, including from an old checkout or release. They are in
the Git history and in release `v2.26.0` for reference only.
- Settings are changed by hand under the owner's login, only when the current
task authorizes that change. Record each change in this repository in the
same task, through a pull request. Until the baseline file lands, record it
in the description of the pull request or task that made it; do not invent
a record file here.
synchronizer, drift audit, Project health check, community-health
generator, semantic release) was retired on 2026-11-14 by owner decision.
Do not restore, run or imitate it, including from an old checkout or
release. It is in the Git history and in release `v2.26.0` for reference
only.
- The **Atrinik work** Project is no longer filled automatically. Keep
`config/planning.json` and its Status names as they are: agent skills read
them.
- `atrinik/.github` is edited by hand, through pull requests in that
repository. Nothing here generates it.
- Settled rollouts: required full-SHA action pinning is on; CodeQL as a merge
requirement and the `playtester` merge queue are dropped.
- Keep the two required check names exactly: the `Policy validation` job in
`.github/workflows/check.yml` and `Conventional PR title` in
`.github/workflows/pr-title.yml`. The ruleset `03 - Required CI -
github-settings` requires both.
- Pin every action to a full commit SHA with a version comment. The
organization requires it.
- Before you push, run `python3 .github/scripts/check_repository.py` and
`git diff --check`. Run actionlint when workflows change, if you have it.
- Before you push, run:

```sh
python3 .github/scripts/check_repository.py
python3 -m unittest discover -s tests
python3 check_settings.py --validate baselines/atrinik.json
git diff --check
```

Run actionlint when workflows change, if you have it.
- Commits and pull-request titles use Conventional Commits.
- Never print or commit tokens, secrets or API responses that contain
credentials.
credentials. The baseline records names only.
14 changes: 11 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,23 @@ Before you push, run:

```sh
python3 .github/scripts/check_repository.py
python3 -m unittest discover -s tests
python3 check_settings.py --validate baselines/atrinik.json
git diff --check
```

The check confirms that every JSON file parses, that every relative link in
the Markdown files points at a file that exists, and that every action in the
workflows is pinned to a full commit SHA. Run
The first command confirms that every JSON file parses, that every relative
link in the Markdown files points at a file that exists, and that every action
in the workflows is pinned to a full commit SHA. The tests run the settings
check offline against invented fixtures in `tests/fixtures/`. Run
[actionlint](https://github.com/rhysd/actionlint) as well when you change a
workflow.

A pull request that changes `baselines/atrinik.json` follows a settings change
that was already made. Quote the output of
`python3 check_settings.py baselines/atrinik.json` in its description: empty
when everything matches, otherwise each remaining line and why it remains.

Keep `config/planning.json` unchanged unless the **Atrinik work** Project
itself changes: agent skills read its Status names.

Expand Down
228 changes: 185 additions & 43 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,59 +1,201 @@
# Atrinik GitHub settings

This repository is the home of Atrinik's GitHub organization settings record.
Today it holds very little:
This repository is the home of Atrinik's GitHub organization settings record:

- [`check_settings.py`](check_settings.py): a read-only check that compares
an organization's live settings with a baseline file.
- [`baselines/atrinik.json`](baselines/atrinik.json): Atrinik's baseline,
written from the live organization on 2026-11-14.
- [`config/planning.json`](config/planning.json): the **Atrinik work** Project
and its status names. Agent skills read the status names from here.
- A small CI check ([`.github/scripts/check_repository.py`](.github/scripts/check_repository.py))
that keeps the files in this repository valid.
- [`tests/`](tests): offline tests with invented fixtures, and a small check
of this repository's own files
([`.github/scripts/check_repository.py`](.github/scripts/check_repository.py)).

A read-only settings check and Atrinik's settings baseline will be added here
by a later pull request.
GitHub holds the rules. Organization rulesets aim at all repositories instead
of lists of names, so a new repository is covered without being registered
anywhere. The check and the baseline only say whether the live organization
still looks the way it was last recorded.

## The old automation is retired
## Run the check

On 2026-11-14 the owner retired the automation that used to live here: the
settings publisher, the Project synchronizer, the drift audit, the Project
health check, the community-health generator and semantic release. They were
removed because:

- They needed a standing organization-owner credential stored in GitHub. That
credential is gone and will not be replaced.
- The rules they managed now live in organization rulesets that target all
repositories, so a new repository is covered without being registered
anywhere.
- The replacement is a read-only check and a short baseline file, both coming
to this repository.

## How settings change now

- The owner, or an agent session she authorizes, changes settings by hand
under her own login, in GitHub's web pages or with `gh`.
- The change is recorded here in the same task, through a pull request. Until
the baseline file lands, it is recorded in the description of the pull
request or task that made it.
- The **Atrinik work** Project board is no longer filled automatically. Add
issues and pull requests to it by hand when they belong there.
- [`atrinik/.github`](https://github.com/atrinik/.github) (issue forms,
pull-request template, contribution guidance, public profile) is no longer
generated from here. Edit it by hand through pull requests in that
repository.
You need Python 3.10 or newer and `gh`, logged in as someone who can read the
organization's settings (for Atrinik, the owner).

```sh
python3 check_settings.py baselines/atrinik.json
```

It prints nothing and exits 0 when everything matches. Otherwise it prints one
line per difference, each ending with `| fix:` and what would make the live
organization match the baseline, usually a command, then a blank line and the
things it could not verify:

```text
atrinik/pathfinder: allow_merge_commit is true, baseline false | fix: gh api --method PATCH repos/atrinik/pathfinder -F allow_merge_commit=false
could not verify: orgs/atrinik webhooks: Not Found (HTTP 404) (the login lacks the admin:org_hook scope)
```

| Exit | Meaning |
| --- | --- |
| 0 | Everything matches. |
| 1 | At least one difference. |
| 2 | No difference, but something could not be verified. |
| 3 | The check could not run: bad arguments, an invalid baseline, no `gh`, `gh` not logged in or its token rejected, an unreadable repository list, or any unexpected error, such as an API response or a baseline shape it does not know (the traceback goes to standard error). Exit 1 always means differences, never a crash. The run stopped and printed nothing on standard output, so any difference it had found is not shown: fix the cause and rerun it. A single read that fails or times out (for example one network blip) is a "could not verify" line instead, and the run goes on. |

The check never runs a fix itself. A fix command is a suggestion: when live
state is right and the record is out of date, update the baseline instead.

Other uses:

```sh
python3 check_settings.py --validate baselines/atrinik.json # the file only (keys, shapes, limits), offline
python3 check_settings.py baselines/atrinik.json --ruleset '02 - Changes through pull requests'
```

`--validate` catches the usual mistakes in a baseline file. A shape it does
not know still ends in exit 3 at the live run. `--ruleset` prints a ruleset's
body from the baseline. The fix commands for rulesets pipe it into
`gh api --method PUT|POST ... --input -`.

### Another organization

The script holds no organization's data. Pass any organization's baseline by
path, for example one kept in a private repository:

```sh
python3 check_settings.py /path/to/private/baseline.json
```

Keep a private organization's baseline and its check output out of this
public repository, its pull requests and its CI logs.

### Without `admin:org`

If the login lacks `admin:org`, the per-repository checks still run: merge
settings, effective rules on each default branch and the tag rules, security
configuration attachment, team grants, environments and repository webhooks.
Organization-level reads (Actions policy, organization rulesets, security
configurations, teams, installed Apps) and settings GitHub hides from that
login become "could not verify" lines. When the login does see the whole
response (an owner with `admin:org` for the organization, admin on the
repository for merge settings, and always for Actions policy and security
configurations), a baseline field the response lacks is a difference
("is not in the API response"): usually a misspelled field name. Organization webhooks need
`admin:org_hook`, which the everyday login does not have, so that line is
expected.

## Rollouts that were never switched on
## What it checks

The old configuration declared three rollouts that were never switched on.
They are now settled:
**Policy**, compared field by field:

- Required full-SHA pinning of actions is on for the organization.
- CodeQL as a merge requirement is dropped. CodeQL alerts stay advisory.
- The merge queue pilot on `playtester` is dropped.
- organization settings (`organization_settings`);
- Actions policy, allowed actions and default workflow permissions
(`actions`);
- organization rulesets (`rulesets`, plus one required-check ruleset per
repository generated from `required_check_ruleset`); a ruleset missing
from either side is a difference;
- each repository's effective rules on its default branch, its required
checks, and the rules on its release tags (`branch_rules`, `checks`,
`tag_rules`). Branch rules come from GitHub's own view of what applies to
the default branch, so they also catch a repository-level ruleset or an
organization ruleset that stopped targeting the repository. Every rule of
a type must match the profile's parameters, and a rule type the profile
does not name is a difference. So list every parameter of a rule in the
profile, as `baselines/atrinik.json` does for `pull_request`, or a
repository ruleset could tighten or loosen the unlisted ones unseen.
These lines name the ruleset a rule comes from and always print, even
when that ruleset has a line of its own. They carry a command only for a
repository ruleset (`repos/O/R/rulesets/ID`). For an organization ruleset
the command is on that ruleset's own line, which exists when the ruleset
is missing, unrecorded or different from the baseline (a "could not
verify" line if it could not be read). When there is neither, the
baseline's rulesets and the profile disagree, and the fix is in the
baseline. Tag rules are checked for presence only, and a tag
ruleset with ref excludes does not count; if a tag ruleset's body cannot
be read, the tag check is one "could not verify" line;
- security configurations, which one is the default for new repositories,
and which one each repository has attached (`security_configurations`,
`security_configuration`);
- each repository's merge settings (`repository`).

## History
**Changed since last recorded**, names only (`recorded`): teams and their
repository grants, installed Apps and their repository selection mode
(`all` or `selected`), environment names, and webhook hosts. A line here
means something changed since the baseline was written, not that it is
wrong.

The retired tooling, its configuration and its documentation are in the Git
history. The last release that contains them is
[`v2.26.0`](https://github.com/atrinik/github-settings/releases/tag/v2.26.0).
**Repositories it cannot classify fail it.** Every repository in the
organization must be listed under `repositories` with a profile, and every
listed one must exist.

## The baseline file

One JSON file per organization:

| Key | What it holds |
| --- | --- |
| `organization` | The organization's login. |
| `notes` | Free text for people. Not checked. |
| `organization_settings`, `actions` | Values compared with the live ones. |
| `rulesets` | Organization rulesets by name; each is the body a `PUT` would send. |
| `required_check_ruleset`, `check_integration_id` | The template for the one-per-repository required-check ruleset (`{repository}` is replaced) and the App that must report the checks (15368 is GitHub Actions). |
| `security_configurations` | Configurations by name, with `default_for_new_repos`. |
| `profiles` | At most three repository profiles: `repository` (merge settings), `branch_rules`, `tag_rules`, `security_configuration`. |
| `repositories` | Every repository: its `profile`, its required `checks`, and rarely `overrides` of profile keys with a `why`. |
| `recorded` | Names only: `teams`, `apps`, `environments`, `webhooks`, `organization_webhooks`. |
| `reminders` | Plain notes about credentials and Apps, such as rotation dates. Never checked. |

Values are compared as subsets: a field GitHub adds later does not make a
difference, and lists match in any order.

## How a change is made

Live state wins and the record catches up in the same task:

1. Run the check. Report any difference you did not cause first, before you
change anything.
2. Make the change in GitHub, under the owner's login, only when the current
task authorizes it.
3. Run the check again. It shows exactly what you changed.
4. Update the baseline until the check is clean, and open a pull request
here that quotes the clean output (or explains each remaining line).

Changes made by hand in GitHub's web pages show up at the next run. GitHub
keeps its own history too: the organization audit log and each ruleset's
history.

## Limits

These keep the check small enough to read in one sitting. Simplify instead of
growing past them.

- Nothing that writes. The script only runs `gh api -X GET`; the tests check
that.
- No dates that fail it. Rotation and review dates are reminders only.
- At most three repository profiles and about ten per-repository
exceptions (`overrides`). `--validate` enforces three and ten.
- About 500 lines for the script. A test stops it at 550.
- Standard library only, and no stored token: it runs as whoever runs it,
by hand. Nothing runs it on a schedule.

## The old automation is retired

On 2026-11-14 the owner retired the automation that used to live here: the
settings publisher, the Project synchronizer, the drift audit, the Project
health check, the community-health generator and semantic release. It needed
a standing organization-owner credential stored in GitHub; that credential is
gone and will not be replaced. The tooling is in the Git history and in
release
[`v2.26.0`](https://github.com/atrinik/github-settings/releases/tag/v2.26.0)
for reference only.

- The **Atrinik work** Project board is no longer filled automatically. Add
issues and pull requests to it by hand when they belong there.
- [`atrinik/.github`](https://github.com/atrinik/.github) is edited by hand
through pull requests in that repository.
- Required full-SHA action pinning is on. CodeQL is not a merge requirement;
its alerts stay advisory. The `playtester` merge queue pilot is dropped.

## Contributing

Expand Down
Loading
Loading