test(mutation): take auth manager.py off the mutation threshold line - #257
Merged
Merged
Conversation
…ments Two auth defects surfaced while classifying the 96 mutants that keep serving/api/auth/manager.py two points above its mutation threshold: - load() carries in-memory rate-limit windows over by plaintext key, but since audit S-6 every window is named kid:/kh:, so every reload -- SIGHUP or the one that ends each key create, rotate and revoke -- empties them. - AGENTFLOW_API_KEYS entries get a random key_id on every load, so their Redis bucket, usage rows and admin views change per reload, restart and replica. The requirements land first, on their own, so the fixes that follow are reviewed against a statement that predates them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHUBaNNPWihtsgeFWnRmaS
7cd4a37 added docs/specs/ with two capability files and no inbound link, and the docs-orphan gate (tests/unit/test_docs_orphans.py) failed the full unit run on them. The security section of architecture.md now points its API authentication and rate-limiting bullets at their requirements, and the docs index names specs/ as the home of required behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHUBaNNPWihtsgeFWnRmaS
…d too The key-identity requirement covered only AGENTFLOW_API_KEYS. A key-file entry without a key_id gets a random id from ensure_key_ids on every load, and load() writes it back only when the file is writable. The shipped production compose mounts ./config read-only and config/api_keys.yaml carries no key_id, so both of its keys draw a new id -- and with it a new rate-limit bucket -- on every load, restart and replica. The requirement now covers every key without a persisted id, derives the id from the stored key_lookup when there is one, and leaves a legacy hash-only entry random. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHUBaNNPWihtsgeFWnRmaS
…indows AuthManager.load() carried the in-memory rate-limit windows over by the plaintext key index (keys_by_value). Since audit S-6 every window is named by its bucket (kid:<key_id>), so no window ever matched: SIGHUP, and the reload that ends each key create, rotate and revoke, emptied them all. is_rate_limited() and the in-memory secondary check in check_rate_limit() then handed every tenant a fresh budget -- during a Redis outage, the whole limit. load() now keeps the windows whose bucket belongs to a key this reload still configures, and drops the rest. The new tests take each scenario of docs/specs/api-key-rate-limiting.md against both readers of the window; four of the five failed before the fix. Environment keys still draw a random key_id on every load, so their windows keep resetting; the next commit derives that id from the key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHUBaNNPWihtsgeFWnRmaS
test_an_aggregate_sums_only_the_readers_rows compared the revenue metric with round(amount, 2). But orders_v2.total_amount is DECIMAL(10,2): DuckDB casts a double into it half away from zero, while Python's round() rounds the exact binary value half to even. Hypothesis drew amount=8.125. The store held 8.13, round() said 8.12, and the 0.01 tolerance missed by float noise. Once the example database saved that case, the failure replayed on every run and turned a full local gate red. The test now asserts that the metric is the reader's own amount to the cent (abs 0.006). That holds under either rounding mode and stays far below the 1.0 minimum a leaked row would add. amount=8.125 is pinned with @example. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHUBaNNPWihtsgeFWnRmaS
…y load A key whose configuration carries no key_id -- every key from AGENTFLOW_API_KEYS, and a key-file entry without one -- drew a random id from generate_key_id on every load. A key-file entry kept its id only when load() could write it back, and docker-compose.prod.yml mounts the key file read-only, over a config/api_keys.yaml whose two entries carry no key_id. The id names the key's Redis bucket, its api_usage rows and the admin views keyed by id: replicas sharing Redis each kept their own bucket for the same key (N replicas, N times its rpm), and every restart and reload started a fresh bucket and split its usage history. ensure_key_ids now derives the id -- <tenant>-<name>-<8 hex of the entry's peppered lookup digest>, from its stored key_lookup or compute_key_lookup over its plaintext -- and _legacy_env_keys runs environment keys through it. A clash lengthens the digest prefix, so ids stay unique and stable. A new pepper changes only an id derived from plaintext and never written back -- every environment key, and a plaintext entry of a key file the process cannot write; an entry with a stored key_lookup, and a derived id already written to a writable key file, keep theirs. A legacy hash-only entry has nothing to derive from and keeps a random id; generate_key_id and create_key are unchanged. tests/unit/test_auth_key_identity.py takes each scenario of docs/specs/api-key-identity.md (nine of its twelve tests fail against the code before this change), with false-reject controls that load the shipped key file and a persisted id beside a hash-only entry. The new key_rotation helpers are killed from test_key_rotation_mutation.py, the file the mutation gate runs; its one test that pinned a random id for entries with a plaintext key now builds hash-only entries, the case that keeps one. The CHANGELOG's ten LF-only lines are now CRLF like the rest of the file. docs/architecture.md and the reload requirement of docs/specs/api-key-rate-limiting.md now name the one key a reload still cannot keep: a legacy hash-only entry in a key file the process cannot write, whose random id -- and so its bucket and window -- changes on every load. docs/specs/api-key-identity.md already specified that id. Reviewed by opus-cli, the writer's own engine: glm-5.3 was rate-limited until 2026-09-14 and codex until 2026-10-05. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HHUBaNNPWihtsgeFWnRmaS
CI run 34464697021 scored serving/api/auth/manager.py 82.1% (430 killed of 524) on 6231b73, two points above its 0.80 threshold. Most of the 94 survivors changed something a behaviour test can see. That includes the log events and the Redis URL, which scripts/mutation_report.py still calls equivalents: a log event is the operator's interface, and the Redis URL decides which server holds the rate-limit budget. tests/unit/test_auth_manager_mutation.py is the only file the gate runs against the module. It now pins: - the key-store writability probe, read-only file included; - the construction wiring: the injected store and audit publisher, usage rows through the real UsageWriter, the default store following db_path, the security config path at construction and on load(), the grace-period fallback and its warning, and the Redis server the limiter targets; - the load() log events, what load() writes back, and the key file it leaves byte-for-byte alone; - that a match's slot comes from the material that matched, never from the stored entry; - that load() carries rate-limit windows over by bucket; - authenticate()'s scan order, and when check_rate_limit() trusts Redis; - AGENTFLOW_API_KEYS parsing. Two simplifications in manager.py remove mutants no test could kill. The writability probe opens the file in binary append mode: the text layer and its encoding did nothing for an open-and-close probe. _legacy_env_keys() stops passing allowed_entity_types=None, which is TenantKey's own default. rate_limit_rpm stays. TenantKey's default froze at import, while the call reads DEFAULT_RATE_LIMIT_RPM at load time. Dropping it too turned the full gate red on test_auth_key_identity, and a gate-file test now pins the load-time read. Nine equivalent mutants stay alive. Each sits on a line that also carries killable mutants, so the test file's docstring names them with their reasons instead of a line-level pragma. Two more lower-case an os.getenv name, so they survive only on Windows, whose environment names are case-insensitive. The local verdicts (77 of 94 killed, 6 removed by the simplifications) came from a scratch runner. scripts/mutation_local.py scores a mutant killed as soon as a tmp_path test runs, because it never creates the parent of its --basetemp; that gets its own fix. The threshold is unchanged here. It is set from the CI measurement of this commit. Two test comments still described the old window carry-over and the random ids of environment keys. Their words are corrected; the assertions are unchanged. Written by opus-high after grok's account ran out of its free usage. grok, on its second account, wrote the repair for the red gate. Reviewed by codex, which returned no findings. The controller skipped review of the repair delta, so the orchestrator reviewed that one-line restore and its test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UUTW71FTTNxbF8JAKwv3Am
DORA Metrics
|
scripts/mutation_local.py scored every mutant that reached a tmp_path test as killed. measure_module() gives each mutant a --basetemp inside a per-run scratch directory that nothing created, and pytest creates --basetemp with a non-recursive mkdir. The first tmp_path fixture then errored at setup, pytest exited 1 under -x, and the runner counted a kill. A mutant that survived every test before that fixture came back killed, equivalents included, so a local "all killed" said nothing about a gate file that uses tmp_path. run_mutant() now creates that parent before it starts pytest. Two tests call run_mutant() on a real one-test file with the parent missing. A passing tmp_path test is scored survived. A failing one is still killed, and the test checks that its body ran, so the kill comes from the test and not from setup. Both tests fail on HEAD. grok's run of six manager.py mutants through the script now agrees with CI run 34542418689: the three named equivalents survive and the three killable mutants are killed. Written by grok on its second account. Reviewed by codex, which returned no findings and no comments, so the orchestrator also read the diff. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UUTW71FTTNxbF8JAKwv3Am
CI run 34542418689 scored serving/api/auth/manager.py 98.4% (553 killed of 562) on 77825f6, the commit that pinned its survivors. The nine left are the equivalents named in tests/unit/test_auth_manager_mutation.py. The threshold goes from 0.80 to 0.90, the bar the other serving modules hold. That is 8.4 points under the measured score, so the named equivalents cannot fail the gate, while a real loss of killed mutants still does. The comment above the module's ModuleTarget said that equivalents no behaviour test could kill dominated the survivors, and it named the structured-logging and Redis-URL mutants among them. T-48's tests killed those. The comment now gives the measured score, the run and the commit, and lists the nine equivalents by function with a short reason each; the test file's docstring keeps the full reasons. The [tool.mutmut] comment in pyproject.toml repeated the old claim ("an honest 0.80") and now points to scripts/mutation_report.py instead. Written by grok on its second account. codex, the first reviewer in the chain, hung without output for 38 minutes and was cancelled, so opus-cli reviewed the slice. Its first pass found four problems: - the CHANGELOG entry contradicted the comment it described; - pyproject.toml still said "an honest 0.80"; - the two __init__ survivors were listed out of source order; - one phrase was garbled. grok fixed all four in one repair pass. The second pass found them fixed and flagged one hard-to-read CHANGELOG sentence, which a second repair pass split. The third pass found the rewrite correct. It left one advisory note (sev 2): the entry names T-48, an internal task id that no other CHANGELOG entry carries. The id stays, because removing it would have taken a closing pass beyond the review limit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UUTW71FTTNxbF8JAKwv3Am
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on #256 (
fix/auth-rate-windows-and-key-identity). The base ismain, so this PR also carries #256's six commits; its own three start after6231b73. Merge #256 first, and this diff shrinks to those three.Takes
serving/api/auth/manager.pyoff the mutation threshold line and raises the threshold to match.Before: on
6231b73, CI run 34464697021 scored it 82.1% (430 killed of 524), against a 0.80 threshold.After: on this branch it scores 98.4% and gates at 0.90.
test(mutation)— the survivors.tests/unit/test_auth_manager_mutation.pyis the only file the mutation gate runs against the module. It now pins what the 94 survivors changed:UsageWriter, the default store followingdb_path, the security config path, the grace-period fallback and its warning, and the Redis server the limiter targets;load()log events;load()writes back, and the key file it leaves byte-for-byte alone;authenticate()scan order;check_rate_limit()trusts Redis;AGENTFLOW_API_KEYSparsing.Two simplifications remove mutants no test could kill:
_legacy_env_keys()no longer passesallowed_entity_types=None.The residue is named. Nine equivalents are listed with their reasons in the test file's docstring. A line-level pragma would also silence killable mutants on the same lines, so none is used. Two survivors lower-case an
os.getenvname, so they survive only on Windows.Stale comments. Two test comments that still described the pre-fix(auth): keep rate-limit windows across reloads, and give id-less keys a stable id #256 behaviour are corrected. Only the words change.
fix(mutation)— local verdicts mean something again.scripts/mutation_local.pygave each mutant a--basetempinside a scratch directory it never created. pytest creates--basetempwithout its parents, so the firsttmp_pathfixture errored at setup and the mutant was scored killed, equivalents included.run_mutant()now creates the parent first. Two tests run it on a real one-test file with the parent missing: a passingtmp_pathtest is scored survived, and a failing one is still killed.test(mutation)— the threshold follows the measurement.manager.pynow gates at 0.90, up from 0.80.ModuleTargetinscripts/mutation_report.pyno longer calls the logging and Redis-URL survivors equivalents. It now gives the measured score and names the nine equivalents that are left, grouped by function.[tool.mutmut]comment inpyproject.tomlno longer repeats "an honest 0.80"; it points to that file instead.Verification
rate_limit_rpm=DEFAULT_RATE_LIMIT_RPMfrom_legacy_env_keys()was not equivalent.TenantKey's field default is fixed at import, while the call reads the module global at load time.test_auth_key_identity.pydepends on that load-time read. The argument is restored, and a test in the gate file now pins the read.scripts/mutation_local.pyhad the false-kill bug thefix(mutation)commit removes. With the fix, a run of three named equivalents and three killable mutants through the script agrees with CI: the three equivalents survive and the other three are killed.77825f6.manager.pyscores 98.4% (553 killed of 562), up from 82.1% (430 of 524). The 9 survivors are exactly the 9 equivalents named in the docstring. The three survivors that stayed alive locally only because of Windows (case-insensitive environment names) were killed on the Linux runner. Every module meets its threshold.dc911d3: every module meets its threshold.manager.pyholds 98.4% (553 killed of 562) against the new 0.90 threshold, and every other module is unchanged.dc911d3: 28 pass, 4 skip.Review
test(mutation): written by opus-high, because grok's account had run out of its free usage; grok, on its second account, wrote the repair for the red gate. Reviewed by codex, with no findings. The controller skipped the review of the repair delta, so the orchestrator reviewed that delta and recorded the skip as a contour defect.fix(mutation): written by grok on its second account. Reviewed by codex, with no findings and no comments, so the orchestrator also read the diff.test(mutation), the threshold: written by grok on its second account. codex hung for 38 minutes with no output and was cancelled, so opus-cli reviewed it.pyproject.tomlcomment was stale;🤖 Generated with Claude Code
https://claude.ai/code/session_01UUTW71FTTNxbF8JAKwv3Am