Skip to content

test(mutation): take auth manager.py off the mutation threshold line - #257

Merged
brownjuly2003-code merged 9 commits into
mainfrom
test/auth-manager-mutation-threshold
Sep 11, 2026
Merged

brownjuly2003-code merged 9 commits into
mainfrom
test/auth-manager-mutation-threshold

Conversation

@brownjuly2003-code

@brownjuly2003-code brownjuly2003-code commented Sep 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

Builds on #256 (fix/auth-rate-windows-and-key-identity). The base is main, so this PR also carries #256's six commits; its own three start after 6231b73. Merge #256 first, and this diff shrinks to those three.

Takes serving/api/auth/manager.py off the mutation threshold line and raises the threshold to match.

  • Before: on 6231b73, CI run 34464697021 scored it 82.1% (430 killed of 524), against a 0.80 threshold.

  • After: on this branch it scores 98.4% and gates at 0.90.

  • test(mutation) — the survivors. tests/unit/test_auth_manager_mutation.py is the only file the mutation gate runs against the module. It now pins what the 94 survivors changed:

    • the key-store writability probe, including a read-only file;
    • the construction wiring: the injected store and audit publisher, usage rows through the real UsageWriter, the default store following db_path, the security config path, the grace-period fallback and its warning, and the Redis server the limiter targets;
    • the load() log events;
    • what load() writes back, and the key file it leaves byte-for-byte alone;
    • that a match's slot comes from the material that matched;
    • that rate-limit windows carry over by bucket;
    • authenticate() scan order;
    • when check_rate_limit() trusts Redis;
    • AGENTFLOW_API_KEYS parsing.
  • Two simplifications remove mutants no test could kill:

    • the probe opens the key file in binary append mode;
    • _legacy_env_keys() no longer passes allowed_entity_types=None.
  • The residue is named. Nine equivalents are listed with their reasons in the test file's docstring. A line-level pragma would also silence killable mutants on the same lines, so none is used. Two survivors lower-case an os.getenv name, so they survive only on Windows.

  • Stale comments. Two test comments that still described the pre-fix(auth): keep rate-limit windows across reloads, and give id-less keys a stable id #256 behaviour are corrected. Only the words change.

  • fix(mutation) — local verdicts mean something again. scripts/mutation_local.py gave each mutant a --basetemp inside a scratch directory it never created. pytest creates --basetemp without its parents, so the first tmp_path fixture errored at setup and the mutant was scored killed, equivalents included. run_mutant() now creates the parent first. Two tests run it on a real one-test file with the parent missing: a passing tmp_path test is scored survived, and a failing one is still killed.

  • test(mutation) — the threshold follows the measurement. manager.py now gates at 0.90, up from 0.80.

    • The comment above its ModuleTarget in scripts/mutation_report.py no longer calls the logging and Redis-URL survivors equivalents. It now gives the measured score and names the nine equivalents that are left, grouped by function.
    • The [tool.mutmut] comment in pyproject.toml no longer repeats "an honest 0.80"; it points to that file instead.

Verification

  • The full unit gate, lint, format, mypy and contracts pass on all three committed trees.
  • The first full gate was red, and that was a real catch. Dropping rate_limit_rpm=DEFAULT_RATE_LIMIT_RPM from _legacy_env_keys() was not equivalent. TenantKey's field default is fixed at import, while the call reads the module global at load time. test_auth_key_identity.py depends on that load-time read. The argument is restored, and a test in the gate file now pins the read.
  • Local mutant verdicts for the survivors: 77 of 94 killed and 6 removed by the simplifications. They came from a scratch runner, because scripts/mutation_local.py had the false-kill bug the fix(mutation) commit removes. With the fix, a run of three named equivalents and three killable mutants through the script agrees with CI: the three equivalents survive and the other three are killed.
  • Mutation workflow on this branch: run 34542418689, on 77825f6. manager.py scores 98.4% (553 killed of 562), up from 82.1% (430 of 524). The 9 survivors are exactly the 9 equivalents named in the docstring. The three survivors that stayed alive locally only because of Windows (case-insensitive environment names) were killed on the Linux runner. Every module meets its threshold.
  • Mutation workflow on the threshold commit: run 34550745164, on dc911d3: every module meets its threshold. manager.py holds 98.4% (553 killed of 562) against the new 0.90 threshold, and every other module is unchanged.
  • PR checks on dc911d3: 28 pass, 4 skip.

Review

  • test(mutation): written by opus-high, because grok's account had run out of its free usage; grok, on its second account, wrote the repair for the red gate. Reviewed by codex, with no findings. The controller skipped the review of the repair delta, so the orchestrator reviewed that delta and recorded the skip as a contour defect.
  • fix(mutation): written by grok on its second account. Reviewed by codex, with no findings and no comments, so the orchestrator also read the diff.
  • test(mutation), the threshold: written by grok on its second account. codex hung for 38 minutes with no output and was cancelled, so opus-cli reviewed it.
    • Pass 1 found four problems, and one repair fixed them:
      • a CHANGELOG sentence contradicted the comment it described;
      • the pyproject.toml comment was stale;
      • two named survivors were listed out of source order;
      • one phrase was garbled.
    • Pass 2 confirmed the fixes and flagged a hard-to-read sentence. A second repair split it.
    • Pass 3 found the rewrite correct. It left one advisory note: the entry names T-48, an internal task id that no other CHANGELOG entry carries. The id stays, because removing it would have needed a pass beyond the review limit.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UUTW71FTTNxbF8JAKwv3Am

JuliaEdom and others added 7 commits September 10, 2026 03:32
…ments

Two auth defects surfaced while classifying the 96 mutants that keep
serving/api/auth/manager.py two points above its mutation threshold:

- load() carries in-memory rate-limit windows over by plaintext key, but
  since audit S-6 every window is named kid:/kh:, so every reload -- SIGHUP
  or the one that ends each key create, rotate and revoke -- empties them.
- AGENTFLOW_API_KEYS entries get a random key_id on every load, so their
  Redis bucket, usage rows and admin views change per reload, restart and
  replica.

The requirements land first, on their own, so the fixes that follow are
reviewed against a statement that predates them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHUBaNNPWihtsgeFWnRmaS
7cd4a37 added docs/specs/ with two capability files and no inbound link, and
the docs-orphan gate (tests/unit/test_docs_orphans.py) failed the full unit
run on them. The security section of architecture.md now points its API
authentication and rate-limiting bullets at their requirements, and the docs
index names specs/ as the home of required behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHUBaNNPWihtsgeFWnRmaS
…d too

The key-identity requirement covered only AGENTFLOW_API_KEYS. A key-file
entry without a key_id gets a random id from ensure_key_ids on every load,
and load() writes it back only when the file is writable. The shipped
production compose mounts ./config read-only and config/api_keys.yaml
carries no key_id, so both of its keys draw a new id -- and with it a new
rate-limit bucket -- on every load, restart and replica. The requirement now
covers every key without a persisted id, derives the id from the stored
key_lookup when there is one, and leaves a legacy hash-only entry random.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHUBaNNPWihtsgeFWnRmaS
…indows

AuthManager.load() carried the in-memory rate-limit windows over by the
plaintext key index (keys_by_value). Since audit S-6 every window is named
by its bucket (kid:<key_id>), so no window ever matched: SIGHUP, and the
reload that ends each key create, rotate and revoke, emptied them all.
is_rate_limited() and the in-memory secondary check in check_rate_limit()
then handed every tenant a fresh budget -- during a Redis outage, the
whole limit.

load() now keeps the windows whose bucket belongs to a key this reload
still configures, and drops the rest. The new tests take each scenario of
docs/specs/api-key-rate-limiting.md against both readers of the window;
four of the five failed before the fix.

Environment keys still draw a random key_id on every load, so their
windows keep resetting; the next commit derives that id from the key.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHUBaNNPWihtsgeFWnRmaS
test_an_aggregate_sums_only_the_readers_rows compared the revenue metric
with round(amount, 2). But orders_v2.total_amount is DECIMAL(10,2): DuckDB
casts a double into it half away from zero, while Python's round() rounds
the exact binary value half to even. Hypothesis drew amount=8.125. The
store held 8.13, round() said 8.12, and the 0.01 tolerance missed by float
noise. Once the example database saved that case, the failure replayed on
every run and turned a full local gate red.

The test now asserts that the metric is the reader's own amount to the
cent (abs 0.006). That holds under either rounding mode and stays far
below the 1.0 minimum a leaked row would add. amount=8.125 is pinned with
@example.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHUBaNNPWihtsgeFWnRmaS
…y load

A key whose configuration carries no key_id -- every key from
AGENTFLOW_API_KEYS, and a key-file entry without one -- drew a random id
from generate_key_id on every load. A key-file entry kept its id only when
load() could write it back, and docker-compose.prod.yml mounts the key file
read-only, over a config/api_keys.yaml whose two entries carry no key_id.
The id names the key's Redis bucket, its api_usage rows and the admin views
keyed by id: replicas sharing Redis each kept their own bucket for the same
key (N replicas, N times its rpm), and every restart and reload started a
fresh bucket and split its usage history.

ensure_key_ids now derives the id -- <tenant>-<name>-<8 hex of the entry's
peppered lookup digest>, from its stored key_lookup or compute_key_lookup
over its plaintext -- and _legacy_env_keys runs environment keys through
it. A clash lengthens the digest prefix, so ids stay unique and stable. A
new pepper changes only an id derived from plaintext and never written back
-- every environment key, and a plaintext entry of a key file the process
cannot write; an entry with a stored key_lookup, and a derived id already
written to a writable key file, keep theirs. A legacy hash-only entry has
nothing to derive from and keeps a random id; generate_key_id and
create_key are unchanged.

tests/unit/test_auth_key_identity.py takes each scenario of
docs/specs/api-key-identity.md (nine of its twelve tests fail against the
code before this change), with false-reject controls that load the shipped
key file and a persisted id beside a hash-only entry. The new key_rotation
helpers are killed from test_key_rotation_mutation.py, the file the
mutation gate runs; its one test that pinned a random id for entries with a
plaintext key now builds hash-only entries, the case that keeps one. The
CHANGELOG's ten LF-only lines are now CRLF like the rest of the file.

docs/architecture.md and the reload requirement of
docs/specs/api-key-rate-limiting.md now name the one key a reload still
cannot keep: a legacy hash-only entry in a key file the process cannot
write, whose random id -- and so its bucket and window -- changes on every
load. docs/specs/api-key-identity.md already specified that id.

Reviewed by opus-cli, the writer's own engine: glm-5.3 was rate-limited
until 2026-09-14 and codex until 2026-10-05.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHUBaNNPWihtsgeFWnRmaS
CI run 34464697021 scored serving/api/auth/manager.py 82.1% (430 killed
of 524) on 6231b73, two points above its 0.80 threshold. Most of the 94
survivors changed something a behaviour test can see. That includes the
log events and the Redis URL, which scripts/mutation_report.py still
calls equivalents: a log event is the operator's interface, and the Redis
URL decides which server holds the rate-limit budget.

tests/unit/test_auth_manager_mutation.py is the only file the gate runs
against the module. It now pins:
- the key-store writability probe, read-only file included;
- the construction wiring: the injected store and audit publisher, usage
  rows through the real UsageWriter, the default store following
  db_path, the security config path at construction and on load(), the
  grace-period fallback and its warning, and the Redis server the
  limiter targets;
- the load() log events, what load() writes back, and the key file it
  leaves byte-for-byte alone;
- that a match's slot comes from the material that matched, never from
  the stored entry;
- that load() carries rate-limit windows over by bucket;
- authenticate()'s scan order, and when check_rate_limit() trusts Redis;
- AGENTFLOW_API_KEYS parsing.

Two simplifications in manager.py remove mutants no test could kill.
The writability probe opens the file in binary append mode: the text
layer and its encoding did nothing for an open-and-close probe.
_legacy_env_keys() stops passing allowed_entity_types=None, which is
TenantKey's own default. rate_limit_rpm stays. TenantKey's default froze
at import, while the call reads DEFAULT_RATE_LIMIT_RPM at load time.
Dropping it too turned the full gate red on test_auth_key_identity, and a
gate-file test now pins the load-time read.

Nine equivalent mutants stay alive. Each sits on a line that also carries
killable mutants, so the test file's docstring names them with their
reasons instead of a line-level pragma. Two more lower-case an os.getenv
name, so they survive only on Windows, whose environment names are
case-insensitive. The local verdicts (77 of 94 killed, 6 removed by the
simplifications) came from a scratch runner. scripts/mutation_local.py
scores a mutant killed as soon as a tmp_path test runs, because it never
creates the parent of its --basetemp; that gets its own fix. The
threshold is unchanged here. It is set from the CI measurement of this
commit.

Two test comments still described the old window carry-over and the
random ids of environment keys. Their words are corrected; the
assertions are unchanged.

Written by opus-high after grok's account ran out of its free usage.
grok, on its second account, wrote the repair for the red gate.
Reviewed by codex, which returned no findings. The controller skipped
review of the repair delta, so the orchestrator reviewed that one-line
restore and its test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UUTW71FTTNxbF8JAKwv3Am
@github-actions

Copy link
Copy Markdown

DORA Metrics

  • Window: last 30 days
  • Branch: main
  • Deployment frequency: 19 total / 4.43 per week
  • Lead time for changes: avg 0.71h / median 0.0h
  • Change failure rate: 68.42% (13/19)
  • MTTR: 79.52h across 11 incident(s)

JuliaEdom and others added 2 commits September 10, 2026 19:56
scripts/mutation_local.py scored every mutant that reached a tmp_path
test as killed. measure_module() gives each mutant a --basetemp inside
a per-run scratch directory that nothing created, and pytest creates
--basetemp with a non-recursive mkdir. The first tmp_path fixture then
errored at setup, pytest exited 1 under -x, and the runner counted a
kill. A mutant that survived every test before that fixture came back
killed, equivalents included, so a local "all killed" said nothing
about a gate file that uses tmp_path.

run_mutant() now creates that parent before it starts pytest. Two tests
call run_mutant() on a real one-test file with the parent missing. A
passing tmp_path test is scored survived. A failing one is still
killed, and the test checks that its body ran, so the kill comes from
the test and not from setup. Both tests fail on HEAD.

grok's run of six manager.py mutants through the script now agrees
with CI run 34542418689: the three named equivalents survive and the
three killable mutants are killed.

Written by grok on its second account. Reviewed by codex, which
returned no findings and no comments, so the orchestrator also read
the diff.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UUTW71FTTNxbF8JAKwv3Am
CI run 34542418689 scored serving/api/auth/manager.py 98.4% (553 killed
of 562) on 77825f6, the commit that pinned its survivors. The nine left
are the equivalents named in tests/unit/test_auth_manager_mutation.py.
The threshold goes from 0.80 to 0.90, the bar the other serving modules
hold. That is 8.4 points under the measured score, so the named
equivalents cannot fail the gate, while a real loss of killed mutants
still does.

The comment above the module's ModuleTarget said that equivalents no
behaviour test could kill dominated the survivors, and it named the
structured-logging and Redis-URL mutants among them. T-48's tests
killed those. The comment now gives the measured score, the run and the
commit, and lists the nine equivalents by function with a short reason
each; the test file's docstring keeps the full reasons. The
[tool.mutmut] comment in pyproject.toml repeated the old claim ("an
honest 0.80") and now points to scripts/mutation_report.py instead.

Written by grok on its second account. codex, the first reviewer in the
chain, hung without output for 38 minutes and was cancelled, so opus-cli
reviewed the slice. Its first pass found four problems:
- the CHANGELOG entry contradicted the comment it described;
- pyproject.toml still said "an honest 0.80";
- the two __init__ survivors were listed out of source order;
- one phrase was garbled.
grok fixed all four in one repair pass. The second pass found them fixed
and flagged one hard-to-read CHANGELOG sentence, which a second repair
pass split. The third pass found the rewrite correct. It left one
advisory note (sev 2): the entry names T-48, an internal task id that no
other CHANGELOG entry carries. The id stays, because removing it would
have taken a closing pass beyond the review limit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UUTW71FTTNxbF8JAKwv3Am
@brownjuly2003-code
brownjuly2003-code merged commit 14f93d4 into main Sep 11, 2026
33 checks passed
@brownjuly2003-code
brownjuly2003-code deleted the test/auth-manager-mutation-threshold branch September 11, 2026 02:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants