fix(auth): keep rate-limit windows across reloads, and give id-less keys a stable id

CHANGELOG.md

@@ -4,6 +4,44 @@ All notable changes to AgentFlow are documented in this file.
 
 ## [Unreleased]
 
+### Fixed — a key configured without a key_id keeps the same id on every load
+
+A key whose configuration carries no `key_id` — every key from
+`AGENTFLOW_API_KEYS`, and a key-file entry without one — drew a random id on
+every `AuthManager.load()`. A key-file entry kept its id only when the id could
+be written back, and `docker-compose.prod.yml` mounts the key file read-only.
+The id names the key's Redis bucket (`kid:<key_id>`), its `api_usage` rows and
+the admin views keyed by id, so replicas sharing Redis each kept their own
+bucket for the same key — N replicas, N times its rpm — and every restart and
+reload started a fresh bucket and split the key's usage history. Such a key now
+gets `<tenant>-<name>-<first 8 hex of its key-lookup digest>`: the stored
+`key_lookup`, else the peppered `compute_key_lookup` of its plaintext, never
+the plaintext or an unpeppered hash of it. The same key gets the same id on
+every load, restart and replica that shares the pepper. Changing the pepper
+changes only an id derived from a plaintext key and never written back: every
+environment key, and a plaintext key-file entry in a file the process cannot
+write. A writable key file keeps the id written to it on the first load, and an
+entry with a stored `key_lookup` keeps the id derived from that digest. An id
+already taken lengthens the digest prefix. A legacy hash-only entry, with
+neither a plaintext key nor a `key_lookup`, still gets a random id.
+
+### Fixed — reloading the key store no longer resets rate-limit windows
+
+`AuthManager.load()` carried the in-memory rate-limit windows over by the
+plaintext key index (`keys_by_value`), but since audit S-6 every window is
+named by its bucket (`kid:<key_id>`), so no window ever matched: every reload
+— SIGHUP, and the reload that ends every key create, rotate and revoke —
+emptied them all. `is_rate_limited()` and the in-memory secondary check in
+`check_rate_limit()` then handed every tenant a fresh budget, which during a
+Redis outage is the whole limit. Windows are now carried over by bucket name:
+a still-configured key keeps its window, a removed key loses it, and no
+plaintext key names a window at any point of the reload. A window survives
+only while its key keeps its `key_id`, which a key configured without one —
+from `AGENTFLOW_API_KEYS`, or in a key file the process cannot write — now does
+too (entry above), with one exception: a legacy hash-only entry, with neither a
+plaintext key nor a `key_lookup`, in a key file the process cannot write still
+gets a new id, and so an emptied window, on every reload.
+
 ### Docs — the pre-push hedges outlived the push
 
 * **Several notes described work the owner "still has to do" that has since
@@ -591,16 +629,16 @@ applies `--ignore` only for `safety_id` values whose waiver scope matches that
 bucket. Expired waivers stop suppressing findings. A waiver whose scope is not
 scanned, or a duplicate `safety_id`, fails closed.
 
-### Documentation — root records archived (2026-09-02–2026-09-04)
-
-Twenty-one immutable tracked Markdown records moved from the repository root
-to `docs/evidence/records/` with unchanged filenames and SHA-256 digests.
-Living citations now use those paths. After all eight documentation-cleanup
-items closed, `plan_26_08_2026.md` moved to `docs/archive/plans/` with its
-closure evidence intact. The retired local 2026-04-17 benchmark baseline moved
-from the standalone `docs/benchmark-baseline-archive/` directory into the
-performance archive.
-
+### Documentation — root records archived (2026-09-02–2026-09-04)
+
+Twenty-one immutable tracked Markdown records moved from the repository root
+to `docs/evidence/records/` with unchanged filenames and SHA-256 digests.
+Living citations now use those paths. After all eight documentation-cleanup
+items closed, `plan_26_08_2026.md` moved to `docs/archive/plans/` with its
+closure evidence intact. The retired local 2026-04-17 benchmark baseline moved
+from the standalone `docs/benchmark-baseline-archive/` directory into the
+performance archive.
+
 ### Security — nltk 3.10.0 -> 3.10.3 in uv.lock (Dependabot GHSA-m4rf-3fr8-xwx3, GHSA-6hwm-xvph-95vm)
 
 - `uv lock --upgrade-package nltk` only; nltk is a transitive dependency of `llama-index-core` and is not part of the `cloud`/`postgres` export, so `requirements-docker.lock` is unchanged. Closes the critical (JVM argument injection in the Stanford wrappers) and high (uncontrolled `dot` search path) advisories GitHub reported on the default branch on 2026-09-01.

docs/README.md

@@ -154,6 +154,7 @@ the former mutable pre-Q1.2 report is preserved as the
 | Current engineering gates | [Engineering status](STATUS.md) | Dated acceptance/evidence records |
 | Lifecycle and non-goals | [Project closure](PROJECT_CLOSURE.md) | Audit and planning records |
 | Runtime design | [Architecture reference](architecture.md) | [Walkthrough](architecture/index.md) and ADRs |
+| Required behaviour | One file per capability in `specs/`: [API key rate limiting](specs/api-key-rate-limiting.md), [API key identity](specs/api-key-identity.md) | The tests that exercise each scenario |
 | API contract | [`openapi.json`](openapi.json) and running FastAPI schema | API guide/reference and SDKs |
 | Security policy | [`SECURITY.md`](../SECURITY.md) | Security audit and dated remediation evidence |
 | Release history | [Changelog](../CHANGELOG.md) | [Archived narrative](archive/release-history-v1-v2.md) |
@@ -166,7 +167,9 @@ the former mutable pre-Q1.2 report is preserved as the
   `scripts/check_docs_root_placement.py`; update it only for an intentional
   stable entrypoint or current reference.
 - Put immutable measurements in `perf/` or `evidence/`, operational procedures
-  in `operations/` or `runbooks/`, and decisions in `decisions/`.
+  in `operations/` or `runbooks/`, decisions in `decisions/`, and the required
+  behaviour of a capability (requirements with scenarios) in `specs/`, one
+  file per capability.
 - Do not delete documentation. Move superseded or duplicate narrative to
   `archive/` with its original path, archive date, reason, and replacement.
 - Update every inbound link in the same commit as a move. Use `git mv` so file

docs/architecture.md

@@ -134,9 +134,9 @@ See [Architecture Decision Records](decisions/) for detailed trade-off analysis.
 ## Security
 
 ### Implemented
-- **API authentication**: API key via `X-API-Key` header (set `AGENTFLOW_API_KEYS` env var)
+- **API authentication**: API key via `X-API-Key` header (set `AGENTFLOW_API_KEYS` env var). A key configured without a `key_id` (every environment key, and a key-file entry without one) gets an id derived from its key-lookup digest, stable across reloads, restarts and replicas (except a legacy hash-only entry, with neither a plaintext key nor a `key_lookup`: it has nothing to derive from and keeps a random id, which in a key file the process cannot write changes on every load); changing the pepper changes only an id derived from a plaintext key and never written back (every environment key, and a plaintext key-file entry in a file the process cannot write), while a writable key file keeps the id written to it on the first load and an entry with a stored `key_lookup` keeps the id derived from that digest. Requirements for a key's id: [API key identity](specs/api-key-identity.md)
 - **Transport gate (audit P2-3)**: `AGENTFLOW_PROFILE=production` refuses to boot over plaintext transport to an external ClickHouse/Redis/PostgreSQL (loopback exempt; deliberate exceptions named in `AGENTFLOW_INSECURE_TRANSPORT_OK`), and refuses a wildcard CORS origin outside demo mode. The ClickHouse client supports HTTPS with hostname verification and a private-CA bundle (`CLICKHOUSE_SECURE`, `CLICKHOUSE_CA_CERT`)
-- **Rate limiting**: Per-key sliding window with Redis backing when available and in-memory fallback for local/test, configurable via `AGENTFLOW_RATE_LIMIT_RPM` (default: 120/min)
+- **Rate limiting**: Per-key sliding window with Redis backing when available and in-memory fallback for local/test, configurable via `AGENTFLOW_RATE_LIMIT_RPM` (default: 120/min). Requirements: [API key rate limiting](specs/api-key-rate-limiting.md)
 - **Health/docs exempt**: `/v1/health`, `/docs`, `/metrics` don't require auth
 - **No secrets in code**: All credentials via environment variables
 - **Terraform state**: Encrypted S3 backend with DynamoDB locking

docs/specs/api-key-identity.md

@@ -0,0 +1,62 @@
+# Capability: api-key-identity
+
+A key's `key_id` names its rate-limit bucket (`kid:<key_id>`), its usage rows
+(`api_usage.key_id`) and the admin views addressed by id. An id that changes
+while the key stays the same splits all three.
+
+## Requirement: a key without a persisted id has a stable id
+A key whose configuration carries no `key_id` SHALL get a `key_id` derived from
+the key's identity, so the same key has the same id on every load, every
+restart and every replica that runs with the same key-lookup pepper. This
+covers every key from `AGENTFLOW_API_KEYS`, and every key-file entry without a
+`key_id` — including one in a key file the process cannot write the id back
+to, as when the file is mounted read-only.
+
+### Scenario: same environment key, two managers
+- **GIVEN** `AGENTFLOW_API_KEYS="k1:Support Agent"` and a fixed key-lookup pepper
+- **WHEN** two separate `AuthManager` instances load their keys
+- **THEN** both give the key the same `key_id`, of the form `default-support-agent-<8 lowercase hex>`
+
+### Scenario: a reload keeps the id and the bucket
+- **GIVEN** a loaded environment-configured key
+- **WHEN** the manager reloads
+- **THEN** the key's `key_id` is unchanged, and so is its rate-limit bucket
+
+### Scenario: different keys under one name get different ids
+- **GIVEN** `AGENTFLOW_API_KEYS="k1:bot,k2:bot"`
+- **WHEN** the manager loads its keys
+- **THEN** the two keys have different `key_id` values
+
+### Scenario: an id-less entry in a read-only key file
+- **GIVEN** a key file the process cannot write, holding an entry that has a `key_lookup` and no `key_id`
+- **WHEN** two separate `AuthManager` instances load it, and one of them reloads
+- **THEN** the entry has the same `key_id` in both managers and after the reload
+
+### Scenario: a writable key file persists the derived id
+- **GIVEN** a writable key file holding an entry that has a plaintext `key` and no `key_id`
+- **WHEN** the manager loads it
+- **THEN** the file on disk now carries the `key_id` the manager uses, and it is the id derived from that key
+
+## Requirement: the id never exposes the key
+The derived id SHALL come from the peppered key-lookup digest — the entry's
+stored `key_lookup`, or `compute_key_lookup` over its plaintext key — never
+from the plaintext key or an unpeppered hash of it: the id is written to logs,
+Redis key names, usage rows and admin responses, and must not let a guessed
+key be confirmed offline. An entry that has neither a plaintext key nor a
+`key_lookup` (a legacy hash-only entry) has nothing to derive from and keeps a
+random id.
+
+### Scenario: the suffix is the lookup digest's prefix
+- **GIVEN** an environment key `k1` and key-lookup pepper `P`
+- **WHEN** the manager loads it
+- **THEN** the `key_id` suffix equals the first 8 characters of `compute_key_lookup("k1", P)`
+
+### Scenario: a stored key_lookup is used as it is
+- **GIVEN** a key-file entry with `key_lookup: L` and no `key_id`
+- **WHEN** the manager loads it
+- **THEN** the `key_id` suffix equals the first 8 characters of `L`
+
+### Scenario: a different pepper gives a different id
+- **GIVEN** the same environment key and two different key-lookup peppers
+- **WHEN** one manager loads it under each pepper
+- **THEN** the two `key_id` values differ

docs/specs/api-key-rate-limiting.md

@@ -0,0 +1,46 @@
+# Capability: api-key-rate-limiting
+
+Per-key request budgets enforced by `AuthManager`
+(`src/agentflow_runtime/serving/api/auth/manager.py`). The budget shared across
+replicas lives in Redis behind `RateLimiter`. The manager also keeps an
+in-memory window per bucket: it is the whole of `is_rate_limited()`, and it is
+the secondary check in `check_rate_limit()` when the Redis limiter answers
+"allowed, full quota" while a Redis handle is live.
+
+## Requirement: a bucket is named by non-secret key identity
+The rate-limit bucket of a key SHALL be named `kid:<key_id>` when the key has an
+id, and no bucket name SHALL contain a plaintext API key.
+
+### Scenario: key with an id
+- **GIVEN** a key whose `key_id` is `acme-support-1a2b3c4d`
+- **WHEN** a request authenticated by that key is rate-limited
+- **THEN** its bucket is `kid:acme-support-1a2b3c4d`
+
+## Requirement: reloading the key store keeps live windows
+Reloading the key configuration — `load()`, a SIGHUP reload, or the reload that
+ends every key create, rotate and revoke — SHALL keep the in-memory rate-limit
+window of every key that is still configured, and SHALL drop the window of a
+key the reload removed. A key whose id changes on every load, a legacy
+hash-only entry in a key file the process cannot write (see
+[API key identity](api-key-identity.md)), gets a new bucket on every reload, so
+its window starts empty.
+
+### Scenario: a full window survives a reload
+- **GIVEN** a key with `rate_limit_rpm: 1` that has already made its one request in the current window
+- **WHEN** the key store is reloaded and the same key makes another request inside that window
+- **THEN** that request is rate-limited
+
+### Scenario: the secondary window survives a reload
+- **GIVEN** a Redis limiter that answers "allowed, full quota" with a live handle, and a key with `rate_limit_rpm: 1` that has made its one request
+- **WHEN** the key store is reloaded and `check_rate_limit()` is called for the same key inside the window
+- **THEN** the answer is not allowed, with zero remaining
+
+### Scenario: a removed key's window is dropped
+- **GIVEN** two configured keys that both made a request in the current window
+- **WHEN** one of them is removed from the key file and the store is reloaded
+- **THEN** the in-memory windows hold the remaining key's bucket and not the removed key's
+
+### Scenario: no plaintext key becomes a window name
+- **GIVEN** a plaintext key configured in the key file, which has made a request
+- **WHEN** the store is reloaded
+- **THEN** no in-memory window is named by that plaintext key, at any point of the reload

src/agentflow_runtime/serving/api/auth/key_rotation.py

@@ -4,6 +4,7 @@
 import re
 import secrets
 import threading
+from collections.abc import Collection
 from datetime import UTC, datetime, timedelta
 
 try:
@@ -22,6 +23,42 @@
     is_permission_denied,
 )
 
+# A derived key_id ends in this many characters of the key's lookup digest: the
+# same shape as the 8 hex characters `generate_key_id` draws at random.
+KEY_ID_DIGEST_CHARS = 8
+
+
+def key_id_slug(value: str, fallback: str) -> str:
+    return "-".join(re.findall(r"[a-z0-9]+", value.lower())) or fallback
+
+
+def derived_key_id(item: TenantKey, existing_ids: Collection[str]) -> str | None:
+    """A stable key_id for an entry configured without one, or None.
+
+    The suffix is a prefix of the entry's peppered lookup digest -- its stored
+    `key_lookup`, else `compute_key_lookup` over its plaintext key -- so the
+    same key gets the same id on every load, restart and replica that shares
+    the pepper. Never the plaintext or an unpeppered hash of it: the id lands in
+    logs, Redis key names, usage rows and admin responses, and must not let a
+    guessed key be confirmed offline (audit FB-07).
+
+    An id already taken lengthens the prefix, so the later entry of a clashing
+    pair still gets the same id on every load. None when there is nothing to
+    derive from (a legacy hash-only entry) or the whole digest is taken.
+    """
+    if item.key_lookup is not None:
+        lookup = item.key_lookup
+    elif item.key is not None:
+        lookup = compute_key_lookup(item.key)
+    else:
+        return None
+    stem = f"{key_id_slug(item.tenant, 'tenant')}-{key_id_slug(item.name, 'agent')}-"
+    for size in range(KEY_ID_DIGEST_CHARS, len(lookup) + 1):
+        candidate = stem + lookup[:size]
+        if candidate not in existing_ids:
+            return candidate
+    return None
+
 
 class KeyRotator:
     def __init__(self, manager: AuthManager) -> None:
@@ -274,8 +311,8 @@ def generate_key_id(
         name: str,
         existing_ids: set[str] | None = None,
     ) -> str:
-        tenant_slug = re.sub(r"[^a-z0-9]+", "-", tenant.lower()).strip("-") or "tenant"
-        name_slug = re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-") or "agent"
+        tenant_slug = key_id_slug(tenant, "tenant")
+        name_slug = key_id_slug(name, "agent")
         seen_ids = set(existing_ids or ())
         while True:
             candidate = f"{tenant_slug}-{name_slug}-{secrets.token_hex(4)}"
@@ -288,7 +325,13 @@ def ensure_key_ids(self, config: ApiKeysConfig) -> bool:
         for index, item in enumerate(config.keys):
             if item.key_id is not None:
                 continue
-            key_id = self.generate_key_id(item.tenant, item.name, existing_ids)
+            # Derived, not drawn: an id that never reaches the file (a read-only
+            # key store, or AGENTFLOW_API_KEYS) must come out the same on the next
+            # load and in every replica. Only a legacy hash-only entry, with
+            # nothing to derive from, still gets a random one.
+            key_id = derived_key_id(item, existing_ids) or self.generate_key_id(
+                item.tenant, item.name, existing_ids
+            )
             existing_ids.add(key_id)
             config.keys[index] = item.model_copy(update={"key_id": key_id})
             changed = True

src/agentflow_runtime/serving/api/auth/manager.py

@@ -328,9 +328,17 @@ def load(self) -> None:
                     if item.previous_key_lookup is not None:
                         self._previous_keys_by_lookup[item.previous_key_lookup] = item
                     self._key_rotator.schedule_rotation_cleanup(item)
+            # Carry windows over by bucket name (`_rate_limit_key`), never by the
+            # plaintext `keys_by_value` index: a still-configured key keeps its
+            # window across the reload, a removed key's window is dropped.
+            live_buckets = {self._rate_limit_key(item) for item in config.keys}
             self._rate_windows = defaultdict(
                 list,
-                {key: self._rate_windows.get(key, []) for key in self.keys_by_value},
+                {
+                    bucket: window
+                    for bucket, window in self._rate_windows.items()
+                    if bucket in live_buckets
+                },
             )
             # H-C4: drop cached plaintext entries for hashes that no longer
             # exist after this reload (revoked/rotated keys). Without this the
@@ -696,7 +704,6 @@ def _legacy_env_keys(self) -> list[TenantKey]:
                 key, name = pair, "unnamed"
             items.append(
                 TenantKey(
-                    key_id=self._key_rotator.generate_key_id("default", name.strip(), set()),
                     key=key.strip(),
                     name=name.strip(),
                     tenant="default",
@@ -705,7 +712,12 @@ def _legacy_env_keys(self) -> list[TenantKey]:
                     created_at=datetime.now(UTC).date(),
                 )
             )
-        return items
+        # The same derivation as an id-less key-file entry: from the key's
+        # lookup digest, so the key keeps its id -- and its rate-limit bucket --
+        # across reloads, restarts and replicas.
+        config = ApiKeysConfig(keys=items)
+        self._key_rotator.ensure_key_ids(config)
+        return config.keys
 
     def _rate_limit_key(self, tenant_key: TenantKey) -> str:
         # This string becomes a Redis sorted-set key NAME (``rate_limiter`` calls