test(mutation): take auth manager.py off the mutation threshold line

CHANGELOG.md

@@ -4,6 +4,105 @@ All notable changes to AgentFlow are documented in this file.
 
 ## [Unreleased]
 
+### Quality — the auth-manager mutation threshold ratchets to 0.90
+
+* **`serving/api/auth/manager.py` now gates at 0.90.** CI run 34542418689 on
+  77825f6 scored it 98.4% (553 killed of 562). The nine survivors are the
+  named residue in `tests/unit/test_auth_manager_mutation.py`. The old
+  comment above its `ModuleTarget` in `scripts/mutation_report.py` called
+  the structured-logging and Redis-URL survivors equivalents. T-48's
+  behaviour tests killed all of them except the upper-cased Redis URL
+  default, which is one of the nine. The comment now gives the measured
+  score and names the nine equivalent survivors, pointing to the test
+  file's docstring for their reasons. 0.90 is the bar the other serving
+  modules hold, and it sits 8.4 points under the measured 98.4%, so the
+  nine named equivalents cannot fail the gate while a real loss of killed
+  mutants still does.
+
+### Fixed — mutation_local no longer scores a missing pytest temp directory as a kill
+
+* **`scripts/mutation_local.py` recorded every mutant that reached a
+  `tmp_path` test as killed.** `measure_module()` handed pytest a `--basetemp`
+  under a scratch directory it never created, and pytest `mkdir`s that path
+  without parents, so the first `tmp_path` fixture failed at setup and pytest
+  exited 1. Under `-x` that false kill covered every mutant that survived the
+  tests before it. `run_mutant()` now creates the `--basetemp` parent before it
+  starts pytest, so a mutant that reaches a passing `tmp_path` test is scored
+  survived, and a real failure is still a kill.
+
+### Quality — the auth-manager mutation lane pins what its survivors changed
+
+* **`serving/api/auth/manager.py` sat two points above its 0.80 threshold.**
+  CI run 34464697021 on 6231b73 scored it 82.1% (430 killed of 524). Most of
+  the 94 survivors changed something a behaviour test can see, including the
+  logging and Redis-URL mutants that `scripts/mutation_report.py` calls
+  equivalents: a log event is the operator's interface, and the Redis URL
+  decides which server holds the rate-limit budget.
+  `tests/unit/test_auth_manager_mutation.py` now pins the key-store
+  writability probe, read-only file included; the construction wiring (the
+  injected store and audit publisher, usage rows flowing through the real
+  `UsageWriter`, the default store following `db_path`, the security config
+  path at construction and on `load()`, the grace-period fallback and its
+  warning, the Redis server the limiter targets); the `load()` log events
+  `api_keys_loaded`, `api_key_store_write_skipped_readonly` and
+  `hashed_key_count_exceeds_guidance` (strictly above the soft limit,
+  unindexed entries only); what `load()` writes back, and the key file it
+  leaves byte-for-byte alone; that a match's slot comes from the material that
+  matched, never from the stored entry; that `load()` carries rate-limit
+  windows over by bucket; scan order in `authenticate()`; when
+  `check_rate_limit()` trusts Redis; and `AGENTFLOW_API_KEYS` parsing.
+* **Two simplifications remove mutants no test could kill.** The writability
+  probe opens the key file in binary append mode: the text layer and its
+  `encoding="utf-8"` did nothing for an open-and-close probe, and three
+  mutants changed only them. `_legacy_env_keys()` stops passing
+  `allowed_entity_types` at `TenantKey`'s own default (`None`);
+  `rate_limit_rpm=DEFAULT_RATE_LIMIT_RPM` stays because `TenantKey`'s Field
+  default is evaluated at import, while a monkeypatch of the module global
+  is a load-time read.
+* **The residue is named, not suppressed.** Nine equivalent mutants stay
+  alive, each on a line that also carries killable mutants, so a line-level
+  `# pragma: no mutate` would silence those too; the test file's docstring
+  lists them with their reasons. The threshold is unchanged here: it is set
+  from the CI measurement of this tree.
+
+### Fixed — a key configured without a key_id keeps the same id on every load
+
+A key whose configuration carries no `key_id` — every key from
+`AGENTFLOW_API_KEYS`, and a key-file entry without one — drew a random id on
+every `AuthManager.load()`. A key-file entry kept its id only when the id could
+be written back, and `docker-compose.prod.yml` mounts the key file read-only.
+The id names the key's Redis bucket (`kid:<key_id>`), its `api_usage` rows and
+the admin views keyed by id, so replicas sharing Redis each kept their own
+bucket for the same key — N replicas, N times its rpm — and every restart and
+reload started a fresh bucket and split the key's usage history. Such a key now
+gets `<tenant>-<name>-<first 8 hex of its key-lookup digest>`: the stored
+`key_lookup`, else the peppered `compute_key_lookup` of its plaintext, never
+the plaintext or an unpeppered hash of it. The same key gets the same id on
+every load, restart and replica that shares the pepper. Changing the pepper
+changes only an id derived from a plaintext key and never written back: every
+environment key, and a plaintext key-file entry in a file the process cannot
+write. A writable key file keeps the id written to it on the first load, and an
+entry with a stored `key_lookup` keeps the id derived from that digest. An id
+already taken lengthens the digest prefix. A legacy hash-only entry, with
+neither a plaintext key nor a `key_lookup`, still gets a random id.
+
+### Fixed — reloading the key store no longer resets rate-limit windows
+
+`AuthManager.load()` carried the in-memory rate-limit windows over by the
+plaintext key index (`keys_by_value`), but since audit S-6 every window is
+named by its bucket (`kid:<key_id>`), so no window ever matched: every reload
+— SIGHUP, and the reload that ends every key create, rotate and revoke —
+emptied them all. `is_rate_limited()` and the in-memory secondary check in
+`check_rate_limit()` then handed every tenant a fresh budget, which during a
+Redis outage is the whole limit. Windows are now carried over by bucket name:
+a still-configured key keeps its window, a removed key loses it, and no
+plaintext key names a window at any point of the reload. A window survives
+only while its key keeps its `key_id`, which a key configured without one —
+from `AGENTFLOW_API_KEYS`, or in a key file the process cannot write — now does
+too (entry above), with one exception: a legacy hash-only entry, with neither a
+plaintext key nor a `key_lookup`, in a key file the process cannot write still
+gets a new id, and so an emptied window, on every reload.
+
 ### Docs — the pre-push hedges outlived the push
 
 * **Several notes described work the owner "still has to do" that has since
@@ -591,16 +690,16 @@ applies `--ignore` only for `safety_id` values whose waiver scope matches that
 bucket. Expired waivers stop suppressing findings. A waiver whose scope is not
 scanned, or a duplicate `safety_id`, fails closed.
 
-### Documentation — root records archived (2026-09-02–2026-09-04)
-
-Twenty-one immutable tracked Markdown records moved from the repository root
-to `docs/evidence/records/` with unchanged filenames and SHA-256 digests.
-Living citations now use those paths. After all eight documentation-cleanup
-items closed, `plan_26_08_2026.md` moved to `docs/archive/plans/` with its
-closure evidence intact. The retired local 2026-04-17 benchmark baseline moved
-from the standalone `docs/benchmark-baseline-archive/` directory into the
-performance archive.
-
+### Documentation — root records archived (2026-09-02–2026-09-04)
+
+Twenty-one immutable tracked Markdown records moved from the repository root
+to `docs/evidence/records/` with unchanged filenames and SHA-256 digests.
+Living citations now use those paths. After all eight documentation-cleanup
+items closed, `plan_26_08_2026.md` moved to `docs/archive/plans/` with its
+closure evidence intact. The retired local 2026-04-17 benchmark baseline moved
+from the standalone `docs/benchmark-baseline-archive/` directory into the
+performance archive.
+
 ### Security — nltk 3.10.0 -> 3.10.3 in uv.lock (Dependabot GHSA-m4rf-3fr8-xwx3, GHSA-6hwm-xvph-95vm)
 
 - `uv lock --upgrade-package nltk` only; nltk is a transitive dependency of `llama-index-core` and is not part of the `cloud`/`postgres` export, so `requirements-docker.lock` is unchanged. Closes the critical (JVM argument injection in the Stanford wrappers) and high (uncontrolled `dot` search path) advisories GitHub reported on the default branch on 2026-09-01.

docs/README.md

@@ -154,6 +154,7 @@ the former mutable pre-Q1.2 report is preserved as the
 | Current engineering gates | [Engineering status](STATUS.md) | Dated acceptance/evidence records |
 | Lifecycle and non-goals | [Project closure](PROJECT_CLOSURE.md) | Audit and planning records |
 | Runtime design | [Architecture reference](architecture.md) | [Walkthrough](architecture/index.md) and ADRs |
+| Required behaviour | One file per capability in `specs/`: [API key rate limiting](specs/api-key-rate-limiting.md), [API key identity](specs/api-key-identity.md) | The tests that exercise each scenario |
 | API contract | [`openapi.json`](openapi.json) and running FastAPI schema | API guide/reference and SDKs |
 | Security policy | [`SECURITY.md`](../SECURITY.md) | Security audit and dated remediation evidence |
 | Release history | [Changelog](../CHANGELOG.md) | [Archived narrative](archive/release-history-v1-v2.md) |
@@ -166,7 +167,9 @@ the former mutable pre-Q1.2 report is preserved as the
   `scripts/check_docs_root_placement.py`; update it only for an intentional
   stable entrypoint or current reference.
 - Put immutable measurements in `perf/` or `evidence/`, operational procedures
-  in `operations/` or `runbooks/`, and decisions in `decisions/`.
+  in `operations/` or `runbooks/`, decisions in `decisions/`, and the required
+  behaviour of a capability (requirements with scenarios) in `specs/`, one
+  file per capability.
 - Do not delete documentation. Move superseded or duplicate narrative to
   `archive/` with its original path, archive date, reason, and replacement.
 - Update every inbound link in the same commit as a move. Use `git mv` so file

docs/architecture.md

@@ -134,9 +134,9 @@ See [Architecture Decision Records](decisions/) for detailed trade-off analysis.
 ## Security
 
 ### Implemented
-- **API authentication**: API key via `X-API-Key` header (set `AGENTFLOW_API_KEYS` env var)
+- **API authentication**: API key via `X-API-Key` header (set `AGENTFLOW_API_KEYS` env var). A key configured without a `key_id` (every environment key, and a key-file entry without one) gets an id derived from its key-lookup digest, stable across reloads, restarts and replicas (except a legacy hash-only entry, with neither a plaintext key nor a `key_lookup`: it has nothing to derive from and keeps a random id, which in a key file the process cannot write changes on every load); changing the pepper changes only an id derived from a plaintext key and never written back (every environment key, and a plaintext key-file entry in a file the process cannot write), while a writable key file keeps the id written to it on the first load and an entry with a stored `key_lookup` keeps the id derived from that digest. Requirements for a key's id: [API key identity](specs/api-key-identity.md)
 - **Transport gate (audit P2-3)**: `AGENTFLOW_PROFILE=production` refuses to boot over plaintext transport to an external ClickHouse/Redis/PostgreSQL (loopback exempt; deliberate exceptions named in `AGENTFLOW_INSECURE_TRANSPORT_OK`), and refuses a wildcard CORS origin outside demo mode. The ClickHouse client supports HTTPS with hostname verification and a private-CA bundle (`CLICKHOUSE_SECURE`, `CLICKHOUSE_CA_CERT`)
-- **Rate limiting**: Per-key sliding window with Redis backing when available and in-memory fallback for local/test, configurable via `AGENTFLOW_RATE_LIMIT_RPM` (default: 120/min)
+- **Rate limiting**: Per-key sliding window with Redis backing when available and in-memory fallback for local/test, configurable via `AGENTFLOW_RATE_LIMIT_RPM` (default: 120/min). Requirements: [API key rate limiting](specs/api-key-rate-limiting.md)
 - **Health/docs exempt**: `/v1/health`, `/docs`, `/metrics` don't require auth
 - **No secrets in code**: All credentials via environment variables
 - **Terraform state**: Encrypted S3 backend with DynamoDB locking

docs/specs/api-key-identity.md

@@ -0,0 +1,62 @@
+# Capability: api-key-identity
+
+A key's `key_id` names its rate-limit bucket (`kid:<key_id>`), its usage rows
+(`api_usage.key_id`) and the admin views addressed by id. An id that changes
+while the key stays the same splits all three.
+
+## Requirement: a key without a persisted id has a stable id
+A key whose configuration carries no `key_id` SHALL get a `key_id` derived from
+the key's identity, so the same key has the same id on every load, every
+restart and every replica that runs with the same key-lookup pepper. This
+covers every key from `AGENTFLOW_API_KEYS`, and every key-file entry without a
+`key_id` — including one in a key file the process cannot write the id back
+to, as when the file is mounted read-only.
+
+### Scenario: same environment key, two managers
+- **GIVEN** `AGENTFLOW_API_KEYS="k1:Support Agent"` and a fixed key-lookup pepper
+- **WHEN** two separate `AuthManager` instances load their keys
+- **THEN** both give the key the same `key_id`, of the form `default-support-agent-<8 lowercase hex>`
+
+### Scenario: a reload keeps the id and the bucket
+- **GIVEN** a loaded environment-configured key
+- **WHEN** the manager reloads
+- **THEN** the key's `key_id` is unchanged, and so is its rate-limit bucket
+
+### Scenario: different keys under one name get different ids
+- **GIVEN** `AGENTFLOW_API_KEYS="k1:bot,k2:bot"`
+- **WHEN** the manager loads its keys
+- **THEN** the two keys have different `key_id` values
+
+### Scenario: an id-less entry in a read-only key file
+- **GIVEN** a key file the process cannot write, holding an entry that has a `key_lookup` and no `key_id`
+- **WHEN** two separate `AuthManager` instances load it, and one of them reloads
+- **THEN** the entry has the same `key_id` in both managers and after the reload
+
+### Scenario: a writable key file persists the derived id
+- **GIVEN** a writable key file holding an entry that has a plaintext `key` and no `key_id`
+- **WHEN** the manager loads it
+- **THEN** the file on disk now carries the `key_id` the manager uses, and it is the id derived from that key
+
+## Requirement: the id never exposes the key
+The derived id SHALL come from the peppered key-lookup digest — the entry's
+stored `key_lookup`, or `compute_key_lookup` over its plaintext key — never
+from the plaintext key or an unpeppered hash of it: the id is written to logs,
+Redis key names, usage rows and admin responses, and must not let a guessed
+key be confirmed offline. An entry that has neither a plaintext key nor a
+`key_lookup` (a legacy hash-only entry) has nothing to derive from and keeps a
+random id.
+
+### Scenario: the suffix is the lookup digest's prefix
+- **GIVEN** an environment key `k1` and key-lookup pepper `P`
+- **WHEN** the manager loads it
+- **THEN** the `key_id` suffix equals the first 8 characters of `compute_key_lookup("k1", P)`
+
+### Scenario: a stored key_lookup is used as it is
+- **GIVEN** a key-file entry with `key_lookup: L` and no `key_id`
+- **WHEN** the manager loads it
+- **THEN** the `key_id` suffix equals the first 8 characters of `L`
+
+### Scenario: a different pepper gives a different id
+- **GIVEN** the same environment key and two different key-lookup peppers
+- **WHEN** one manager loads it under each pepper
+- **THEN** the two `key_id` values differ

docs/specs/api-key-rate-limiting.md

@@ -0,0 +1,46 @@
+# Capability: api-key-rate-limiting
+
+Per-key request budgets enforced by `AuthManager`
+(`src/agentflow_runtime/serving/api/auth/manager.py`). The budget shared across
+replicas lives in Redis behind `RateLimiter`. The manager also keeps an
+in-memory window per bucket: it is the whole of `is_rate_limited()`, and it is
+the secondary check in `check_rate_limit()` when the Redis limiter answers
+"allowed, full quota" while a Redis handle is live.
+
+## Requirement: a bucket is named by non-secret key identity
+The rate-limit bucket of a key SHALL be named `kid:<key_id>` when the key has an
+id, and no bucket name SHALL contain a plaintext API key.
+
+### Scenario: key with an id
+- **GIVEN** a key whose `key_id` is `acme-support-1a2b3c4d`
+- **WHEN** a request authenticated by that key is rate-limited
+- **THEN** its bucket is `kid:acme-support-1a2b3c4d`
+
+## Requirement: reloading the key store keeps live windows
+Reloading the key configuration — `load()`, a SIGHUP reload, or the reload that
+ends every key create, rotate and revoke — SHALL keep the in-memory rate-limit
+window of every key that is still configured, and SHALL drop the window of a
+key the reload removed. A key whose id changes on every load, a legacy
+hash-only entry in a key file the process cannot write (see
+[API key identity](api-key-identity.md)), gets a new bucket on every reload, so
+its window starts empty.
+
+### Scenario: a full window survives a reload
+- **GIVEN** a key with `rate_limit_rpm: 1` that has already made its one request in the current window
+- **WHEN** the key store is reloaded and the same key makes another request inside that window
+- **THEN** that request is rate-limited
+
+### Scenario: the secondary window survives a reload
+- **GIVEN** a Redis limiter that answers "allowed, full quota" with a live handle, and a key with `rate_limit_rpm: 1` that has made its one request
+- **WHEN** the key store is reloaded and `check_rate_limit()` is called for the same key inside the window
+- **THEN** the answer is not allowed, with zero remaining
+
+### Scenario: a removed key's window is dropped
+- **GIVEN** two configured keys that both made a request in the current window
+- **WHEN** one of them is removed from the key file and the store is reloaded
+- **THEN** the in-memory windows hold the remaining key's bucket and not the removed key's
+
+### Scenario: no plaintext key becomes a window name
+- **GIVEN** a plaintext key configured in the key file, which has made a request
+- **WHEN** the store is reloaded
+- **THEN** no in-memory window is named by that plaintext key, at any point of the reload

pyproject.toml

@@ -338,13 +338,9 @@ asyncio_mode = "auto"
 # starting with `src.`, which (not duckdb) was the real blocker. (The gate runner
 # keeps mutmut workspaces free of relative pytest --basetemp overrides because
 # under py3.11 they break coverage->mutant mapping for file-I/O targets like
-# key_rotation; see scripts/mutation_report.py.) manager.py runs at an honest 0.80
-# threshold (not 0.90): it is a
-# large stateful auth class whose residual survivors are equivalent mutants
-# (structured-logging args, model_copy updates equal to their defaults, redis-url
-# strings masked by the `_redis = None` override, the env-only-dead write path) --
-# every behaviour-reachable mutant, including every auth-bypass and throttle
-# off-by-one, is killed. See scripts/mutation_report.py.
+# key_rotation; see scripts/mutation_report.py.) manager.py gates at 0.90 like
+# the other serving modules; scripts/mutation_report.py gives its measured
+# score and names its nine equivalent survivors.
 paths_to_mutate = [
     "src/agentflow_runtime/serving/api/auth/manager.py",
     "src/agentflow_runtime/serving/api/auth/key_rotation.py",

scripts/mutation_local.py

@@ -495,8 +495,12 @@ def run_mutant(
 
     Each mutant gets its own `--basetemp`: pytest wipes and recreates that
     directory at startup, so concurrent runs sharing one abort each other and
-    exit 2.
+    exit 2. pytest creates `--basetemp` itself with a non-recursive mkdir, so
+    the parent has to exist before pytest starts -- otherwise the first
+    `tmp_path` fixture errors at setup, pytest exits 1, and the mutant is
+    scored killed.
     """
+    basetemp.parent.mkdir(parents=True, exist_ok=True)
     command = [
         python,
         "-m",