bench.py follow-up; ruff format --check as a CI gate - #100
Merged
Merged
Conversation
…son (Ola) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two follow-ups from @Reviewer's round on tools/bench.py, approved by Ola (2026-09-27), pinned in bench-py.md's "Pinned by the red suite": - Threshold boundary: exactly 5 % (20.0 -> 21.0) is ACCEPTED, 5.5 % is a REGRESSION; same at 7.5. Red today: 21.0 / 20.0 - 1 rounds above 0.05, so the ratio test calls exactly 5 % a regression. - Clean exits: a missing --dem or --domain, and a malformed run.json the caller named (compare NEW_DIR, compare/run --baseline), exit 3 with the file named and no evidence; run refuses before building. A malformed run.json met in the baseline search is skipped with a UserWarning naming it (settled here, reasons in bench-py.md). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- verdict: a time regression is `new * 100 > base * (100 + threshold)`, exact products instead of the rounded ratio, so exactly 5 % is ACCEPTED. - _load raises BenchError naming the run.json it could not read or validate; run refuses a missing --dem/--domain and a bad --baseline before the build and any child, compare refuses a bad NEW_DIR or --baseline; both exit 3 through typer.Exit. - find_baseline skips a malformed run.json with a UserWarning naming it. - bench-py.md "Size": the 700-line ceiling is per PR (lines the PR adds or changes), replacing the "about 10 lines of room" misreading. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed quotations Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Layout only, produced by `ruff format .` (0.16.8). Listed in .git-blame-ignore-revs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…format Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…inned or corrected; bench-py.md points at §2 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…VER fixture Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…26-09-27) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nwards The top-level `# fmt: off` before run() was paired with a `# fmt: on` indented inside the function body. ruff honours `fmt: on` only at the indentation of its `fmt: off`, so formatting stayed off to end of file and `ruff format --check` skipped run(), compare() and child_main(). The run() signature is already in ruff's own format (one parameter per line, none over 100 columns), so no exemption is needed: both lines are removed and ruff format changes nothing. AST unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y it went Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
# fmt: skip/# fmt: offis allowed with a reason.tools/bench.pyfixes (red48cb90fbefore greenc854edd):--dem/--domainor a malformed/missing namedrun.jsonexits 3 naming the file, before any build;run.jsonwith aUserWarning.ruff format --check .as a gate (Ola): ruff pinned~=0.16.8;docs/(cited code quotations) and.claude/excluded from formatting; one layout-only reformat41db2ab(ASTs and comments identical), listed in.git-blame-ignore-revs; the step in CI, in the after-commit hook (Ola's yes) and in CLAUDE.md's local gates.# fmt: offhad its# fmt: onindented, so formatting never switched back on and the gate skippedrunto EOF; removed in91d5b76(AST identical).Review
@Reviewer: several rounds, APPROVED at
d6d6beb. Local: pytest 1719 passed / 6 skipped; mypy, ruff check, ruff format --check, governance gates green.Merge with a merge commit, not a squash:
.git-blame-ignore-revsnames41db2ab.🤖 Generated with Claude Code