diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 83bef1c4d..31ff6a3d6 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -6588,6 +6588,18 @@ contention; recorded in the #1095 handoff note rather than lost. ## 1100. The master test plan asserts document contradictions that were resolved before it was written +> **PROGRESS 2026-08-26 (lander), NOT A CLOSURE -- this item stays OPEN.** Landed the documentation +> half of stranded PR #433 on a fresh branch (the original ref carried an owner-armed auto-merge that +> would have fused it to six unrelated engine commits; disarmed and split per the owner's 2026-08-26 +> ruling, ADR 0165 authorship note below). Measured directly against this landed diff, not carried +> forward from the branch's own claim: 18 chapter files, +90/-75, matching the branch's own count +> exactly. **WHAT IS NOT ESTABLISHED, and it is why this stays open: nobody has verified that the NINE +> SITES THIS ITEM NAMES are among those revised.** Closing needs a pass that walks those nine sites +> and checks each against the now-landed chapters. **Author's note, ADR 0165:** this progress note is +> Lander-authored on the builder's behalf -- the 64 underlying commits are the builder's own verified +> work, cherry-picked unmodified; only this ledger paragraph is mine, because ledger authorship on an +> open item is reserved to the Dispatcher or Lander. +> > πŸ”’ **Re-scored 2026-08-20 -> P2.** Value **5/10** Β· Difficulty **3/10** Β· _fill-in_. The stale claims are not confined to narrative: :593 is item 4 of the chapter's numbered release sign-off list (the three contradicting documents agree with the code) and :606 declares Blocks: ALERT-08, 09, 58, 67 as the chapter's only P0 with a real security consequence, so a plan executor is held on a question the code answered on 2026-08-02. Difficulty is nine content edits, each requiring the claim re-checked against code first, at least one possibly warranting deletion, with the item forbidding a scripted pass. _(was 6/10 Β· 3/10.)_ > > **Filed 2026-08-07 - not started.** Nine sites in the master test diff --git a/docs/testing/master-test-plan/01-environments-data-and-tooling.md b/docs/testing/master-test-plan/01-environments-data-and-tooling.md index df81684c8..93adb2c6e 100644 --- a/docs/testing/master-test-plan/01-environments-data-and-tooling.md +++ b/docs/testing/master-test-plan/01-environments-data-and-tooling.md @@ -126,7 +126,7 @@ is never redirected to a committed file, a ticket, or a CI log. message families the engine handles: `adt`, `oru`, `orm`, `oml`, `orl`, `mdm`, `mfn`, `dft`, `bar`, `ras`, `rde`, `siu`, `vxu`, `documents`, with `all_types.py` as the registry. Two entry points: -- `messagefoundry generate` β€” the CLI subcommand ([`__main__.py:349`](../../../messagefoundry/__main__.py)). +- `messagefoundry generate` β€” the CLI subcommand ([`__main__.py:350`](../../../messagefoundry/__main__.py) β€” **RE-POINTED 2026-08-15 (BACKLOG #1100), was `:349`, which is now blank**; the subparser is `generate = sub.add_parser(` at `:350` with the name `"generate"` on `:351`, and its flags follow at `:353-361`). - `python -m messagefoundry.generators.adt [--triggers A01,A04] [--count N] [--out DIR]` β€” the ADT corpus builder: **57 triggers across 25 message structures** (A01–A62 excluding the A19 query event and reserved A56–A59), with segment order and the allowed segment set driven by **hl7apy's own diff --git a/docs/testing/master-test-plan/02-pipeline-reliability.md b/docs/testing/master-test-plan/02-pipeline-reliability.md index bbc5f0e9f..39caa19ca 100644 --- a/docs/testing/master-test-plan/02-pipeline-reliability.md +++ b/docs/testing/master-test-plan/02-pipeline-reliability.md @@ -117,8 +117,13 @@ chapter or any downstream one: run `test_load_failover_{sqlserver,postgres}` with the harness setting **no** `claim_mode`, i.e. under the pooled default, and hard-gate zero acknowledged loss + `lane_inversions == 0`. 4. **Postgres 2-engine crash-and-restart recovery.** `FCP:STORE-10`'s "not built" is stale β€” - `tests/test_shard_recovery_postgres.py` exists (4 tests). Its problem is that it **runs nowhere** - (PIPE-01), not that it is missing. + `tests/test_shard_recovery_postgres.py` exists (4 tests). **CORRECTED (BACKLOG #1100): it does not + "run nowhere".** It runs in `ci.yml`'s `postgres-store` job, step *"Run the failover + + engine-shard recovery suites on real Postgres"* β€” but that job is gated on + `schedule || workflow_dispatch || changes.outputs.serverdb == 'true'`, so it does **not** run on a + PR that touches no server-DB path. The accurate statement is *runs only on the server-DB + path-gated leg*, which is what PIPE-01 is actually for. "Runs nowhere" understated the coverage and + "runs in CI" would overstate it; only the gated form supports a decision about PIPE-01. 5. **`accepts=` static fail-closed validation** β€” `FCP:PIPE-9` is closed by `test_accepts_seam.py` (the three static-validation negatives). 6. **Purity replay-equality harness** β€” `FCP:PIPE-14`'s replay half is closed by @@ -137,8 +142,8 @@ chapter or any downstream one: | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| Six `MEFOR_TEST_*`-gated pipeline suites (~36 tests) are named in **no** workflow step | A live regression in `batch_handoff_statements` (DEFAULT-ON, SQL-Server-only, restructures the route/transform handoff DML) or in ownership-scoped **engine-shard** recovery ships green | Silent data loss, wrong disposition, or **duplicate PHI deliveries across engine shards** on the production-scale backend | **No.** `ci.yml:424-428`'s own comment says the path gate "MUST list every file the sqlserver/postgres steps run"; these are in neither the steps nor the gate regex | **P0** | -| No poison-crash attempts ceiling on the **default split** ingress/routed path | A hard abort with no Python exception (C-extension segfault, OOM kill) inside `route_only`/`transform_one`/handoff is caught by neither the internal-error policy nor the ADR 0070 T17 handler; `reset_stale_inflight` re-pends the head, the lane re-runs, the process dies again | Lane head-of-line blocked **forever** across NSSM/supervisor restarts; nothing dead-letters; every message behind it stops flowing. The G6 ceiling exists only inside `if inline:` (`wiring_runner.py:4475-4496`), and its own comment at `:4479-4480` states no ingress/routed path enforces `max_attempts` on the split path. `supervisor.py:22` lists "restart backoff / crash-loop breaker" as deferred | **No** β€” the ADR 0087 sandbox that would contain it is default OFF, and no test drives a hard abort on the split path | **P0** | +| Gated suites the path gate PULLS but no step RUNS β€” **the inverse of the invariant `ci.yml:983` states** | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): the original evidence was false in both parts.** It said `batch_handoff_statements` and engine-shard recovery "are in neither the steps nor the gate regex". **Both are in both**: `tests/test_adr0075_batch_sqlserver.py` (`ci.yml:1375`) and `tests/test_shard_recovery_sqlserver.py` (`:1371`) run in the `sqlserver-store` step *"Run the engine-shard + statement-dispatch suites on real SQL Server"*, and `adr0075`/`shard_recovery` are both in the gate alternation at `:993`. The cited anchor `:424-428` had drifted onto a comment about CI run timings; the real comment is `:983-988`. **THE RESIDUAL GAP IS REAL AND SHARPER:** of **54** `MEFOR_TEST_*`-gated suites, **19 are named in no workflow step**, and every server-DB step invokes **explicit files** (never a directory), so an unnamed suite runs nowhere on a real backend. **Three of the 19 are MATCHED BY THE GATE REGEX** β€” `test_adr0157_fence_scope` (8 tests), `test_sqlserver_sync_handoff_offline` (6), `test_adr0071_fusion_wiring` β€” each named **0** times anywhere in `.github/` (positive control: `shard_recovery_sqlserver` = 1) | A change to those three **pulls the expensive server-DB legs and then never executes them** β€” the legs go green having not run the suite the gate fired for. `test_adr0157_fence_scope` is the sharpest: ADR 0157 fence scope, 8 tests, no real-Postgres execution anywhere | **Partly.** `ci.yml:983-988` states only ONE direction β€” the alternation must list every file the steps run. **The inverse is unstated and violated three times**: every file the alternation lists should be run by some step | **P0** | +| No poison-crash attempts ceiling on the **default split** ingress/routed path | A hard abort with no Python exception (C-extension segfault, OOM kill) inside `route_only`/`transform_one`/handoff is caught by neither the internal-error policy nor the ADR 0070 T17 handler; `reset_stale_inflight` re-pends the head, the lane re-runs, the process dies again. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100), and the mechanism is sharper than "no ceiling exists":** the `queue` table DOES carry an `attempts` column (`store.py:1319`) for **every** stage, and a `max_attempts` ceiling IS enforced β€” but **only inside `mark_failed()` (`:5756`) and `mark_batch_failed()` (`:5816`)**, which are the *caught-Python-exception* paths. A hard abort calls neither, and recovery instead runs `reset_stale_inflight` (`:5948`, 69 lines) which references **neither `attempts` nor `dead_letter`** β€” verified by reading the whole function, not a window. So the row is right, and the fix has a precise location: **the increment must happen at CLAIM time or in `reset_stale_inflight`, not on the failure path**, because the failure path is exactly what a hard abort skips. Note also that G6 is an **outbox-stage** mechanism throughout (every reference reads `OutboxItem.attempts`), which is why it does not cover this | Lane head-of-line blocked **forever** across NSSM/supervisor restarts; nothing dead-letters; every message behind it stops flowing. The G6 ceiling exists only inside `if inline:` (`wiring_runner.py:4475-4496`), and its own comment at `:4479-4480` states no ingress/routed path enforces `max_attempts` on the split path. `supervisor.py:22` lists "restart backoff / crash-loop breaker" as deferred | **No** β€” the ADR 0087 sandbox that would contain it is default OFF, and no test drives a hard abort on the split path | **P0** | | `W25:S3.4` / `W25:S2.7` and `harness/config/coverage.py:17` assert an **AE NAK** for a post-ACK Handler raise | Under ACK-on-receipt the AA fires at the ingress commit (`wiring_runner.py:3726-3745`) before the Router or Handler runs; a Handler raise **cannot** NAK | A human running `W25:S3.4` either fails a correct system or records a NAK that never happened. `harness/scenarios.py:63` already expects only disposition `error`, so the docs contradict both the code and the harness they instruct the tester to run β€” on the single most partner-visible behaviour change in ADR 0001 | **No** β€” the docs *are* the detector, and they are wrong | **P0** | | No live end-to-end committed-transactions-per-message ceiling in CI | An accidental extra handoff commit doubles `committed_txns/msg` | Passes every test; surfaces only as a production capacity shortfall. ADR 0051 sizes capacity on `3 + 2H + 2N`; the counters already exist (`store/base.py:220-234`, surfaced at `api/app.py:4142-4143`) | Partly β€” `test_txn_per_message_cost_model.py` pins the **model** over a recording connection, not the **live** counter through a real runner | P1 | | A refactor hoists the lookup-runner `ExitStack` to wrap `route_only` | Routers silently gain live `db_lookup` access; the at-least-once re-run invariant breaks and an unbudgeted live DB read lands on the routing hot path for every message | Non-pure Routers β‡’ duplicate/divergent downstream side effects on every crash re-run | **No** for `db_lookup` β€” the FHIR twin exists (`test_fhir_lookup.py:545`); the db_lookup side has only "no active runner" (`test_db_lookup.py:104`) and "dry-run raises" (`:310`). The guarantee is *positional* (`wiring_runner.py:5027-5031`), not structural | P1 | diff --git a/docs/testing/master-test-plan/03-store-and-data-lifecycle.md b/docs/testing/master-test-plan/03-store-and-data-lifecycle.md index 87d33db03..42edc8fb8 100644 --- a/docs/testing/master-test-plan/03-store-and-data-lifecycle.md +++ b/docs/testing/master-test-plan/03-store-and-data-lifecycle.md @@ -103,9 +103,9 @@ the scheduling rows below it were never corrected). | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| **R1** ADR 0027 per-connection purge `CASE` has never run on PG/SQL Server (`connection_cutoffs` appears **0** times in `tests/test_postgres_store.py` and `tests/test_sqlserver_store.py`; the AC-8 parity case lives only in `tests/test_per_connection_retention.py:262`, which no CI step names and which the `serverdb` regex at `ci.yml:434` does not match) | A wrong `_pg_cutoff_case` / `_qmark_cutoff_case` predicate purges the **wrong feed's** PHI bodies (irreversible; the count-and-log row survives so nothing looks broken) or silently purges nothing (unbounded PHI at rest) | Every PHI feed on the production backend; irreversible either way | **No.** SQLite stays green in both directions | **P0** | -| **R2** The shipped at-rest **writer** is barely exercised. `[store].aad_bind` defaults `True` β†’ `open_store` builds a `write_v2` / `mfenc:v2` cell-AAD cipher ([base.py:1736](../../../messagefoundry/store/base.py)), but `make_cipher`'s library default is `write_v2=False` and **no** workflow sets `MEFOR_TEST_FORCE_AAD_BIND` (only `tests/conftest.py:126,138` and ADR 0019:516 mention it) | An unbound or mis-threaded `cell_aad` on any write path β€” purge re-encrypt, document strip write-back, attachment re-seal, restore β€” yields rows the shipped cipher cannot decrypt (unreadable PHI) or silently drops the binding (ASVS 11.3.3 regression) | Whole store; discovered only in production | **No.** 13 SQLite-only targeted tests | **P0** | -| **R3** The PHI serve retention gate reads **only global** windows ([__main__.py:1968-1975](../../../messagefoundry/__main__.py) builds `unbounded_windows` from `settings.retention.messages_days` / `dead_letter_days` alone) and never consults the registry's per-connection overrides | A PHI instance with a global 30-day window and every inbound at `messages_days=0` passes the fail-closed gate and retains PHI forever | An audited, security-labelled fail-closed control (ASVS 14.2.4) is bypassable by ordinary Connection config | **No.** No warning, no audit entry, no test | **P0** | +| **R1** ADR 0027 per-connection purge `CASE` has never run on PG/SQL Server. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100) β€” every claim holds, with a positive control, and the gap is DOUBLE.** `connection_cutoffs` appears **0** times in `tests/test_postgres_store.py` and `tests/test_sqlserver_store.py` (control: it appears in exactly **2** files repo-wide, so the zero is real and not a broken grep). **Those two files are `test_per_connection_retention.py` AND `test_embedded_document_pruning.py`** β€” the row named only the first. Neither is named by any workflow step, and the `serverdb` path-gate regex (**`ci.yml:993`**, not the cited `:434`) matches `per_connection_retention` **0** times. **THAT IS A STRICTLY WORSE SHAPE THAN THE ONE AT `02-pipeline-reliability.md:145`:** those three suites are *matched by the gate but run by no step*, so the expensive leg at least fires; **these two are matched by nothing and named by nothing**, so editing the per-connection purge predicate triggers no server-DB leg at all and runs no test that mentions `connection_cutoffs` | A wrong `_pg_cutoff_case` / `_qmark_cutoff_case` predicate purges the **wrong feed's** PHI bodies (irreversible; the count-and-log row survives so nothing looks broken) or silently purges nothing (unbounded PHI at rest) | Every PHI feed on the production backend; irreversible either way | **No.** SQLite stays green in both directions | **P0** | +| **R2** The **exhaustive** at-rest AAD sweep never runs in CI (was: "the shipped writer is barely exercised" β€” **narrowed 2026-08-15, BACKLOG #1100**). **CONFIRMED, and the framing corrected.** Still true: `[store].aad_bind` defaults `True` (`settings.py:388`), `make_cipher`'s library default is `write_v2=False` (`crypto.py:812`), and **no workflow sets `MEFOR_TEST_FORCE_AAD_BIND`** β€” **0** occurrences across `.github/workflows/` against a positive control of **14** for `MEFOR_TEST_SQLSERVER`. **But "barely exercised" now overstates it**, and the flag's own docstring (`tests/conftest.py:133-138`) says why: *"The flag is OFF by default and stays meaningful even though `[store].aad_bind` now DEFAULTS TRUE (ADR 0148 GIVEN 1)... The settings default governs what `open_store` builds; **this flag governs every cipher in the process, which is what makes the sweep exhaustive rather than merely representative**."* So the shipped `mfenc:v2` writer **is** exercised by every ordinary store test via the default; what is missing is the process-wide forcing that also catches ciphers built with an **explicit** `write_v2=False`. **The parenthetical was also wrong: EIGHT test files reference the flag**, not `conftest.py` alone β€” `test_store_aad_binding`, `test_store_encryption`, `test_transform_state`, `test_ack_sent_store`, `test_alert_state`, `test_connection_event_store`, `test_ed_documents_e2e`, `test_sqlserver_store`. **The residual risk is unchanged and still P0:** a half-threaded `cell_aad` on purge re-encrypt, document strip write-back, attachment re-seal or restore would surface only under the forced sweep, and that sweep runs nowhere | An unbound or mis-threaded `cell_aad` on any write path β€” purge re-encrypt, document strip write-back, attachment re-seal, restore β€” yields rows the shipped cipher cannot decrypt (unreadable PHI) or silently drops the binding (ASVS 11.3.3 regression) | Whole store; discovered only in production | **No.** 13 SQLite-only targeted tests | **P0** | +| **R3** The PHI serve retention gate reads **only global** windows and never consults the registry's per-connection overrides. **CONFIRMED 2026-08-15 (BACKLOG #1100); anchors re-pointed β€” the gate MOVED and is no longer built inline.** `unbounded_windows` is now defined at [`config/retention_classification.py:186`](../../../messagefoundry/config/retention_classification.py) (20 lines) and called from [`__main__.py:2240`](../../../messagefoundry/__main__.py) as `_unbounded_windows(settings)` β€” **`settings` alone, no registry**. Read whole via AST, the function contains **no** `registry`, `connection`, `per_connection`, `overrides` or `inbound` token, and `__main__.py:2240` is its **only** caller in the package. **The bypass is documented in the code that creates it** (`wiring.py:3125-3126`): *"Per-connection retention override (#34, ADR 0027): None = inherit the global `[retention].messages_days` window; **0 = keep this connection's bodies forever**; >0 = days"* β€” and that override **is** honoured at purge time (`pipeline/retention.py:150`, "inbound name -> messages_days"). So the two halves are individually correct and jointly permissive: **a deploying PHI instance would pass the gate on its global window while retaining every body forever per-connection** | A PHI instance with a global 30-day window and every inbound at `messages_days=0` passes the fail-closed gate and retains PHI forever | An audited, security-labelled fail-closed control (ASVS 14.2.4) is bypassable by ordinary Connection config | **No.** No warning, no audit entry, no test | **P0** | | **R4** ADR 0042 `strip_embedded_documents` has never run on PG/SQL Server (**0** occurrences in both server suites) β€” and it is a select β†’ decrypt β†’ codec-transform β†’ **re-encrypt write-back** over stored PHI bodies ([postgres.py:6395](../../../messagefoundry/store/postgres.py), [sqlserver.py:5626](../../../messagefoundry/store/sqlserver.py)) | A dialect or write-back bug corrupts stored bodies (unparseable HL7) or leaves bulky base64 PHI in place forever | Every document feed on the production backend | **No** | **P1** | | **R5** `strip_embedded_documents` materializes and **decrypts every eligible row with no LIMIT/TOP/batch** on all three backends ([store.py:8465](../../../messagefoundry/store/store.py), [postgres.py:6420](../../../messagefoundry/store/postgres.py), [sqlserver.py:5650](../../../messagefoundry/store/sqlserver.py)); the ADR 0137 deadline is checked only **before** the phase ([retention.py:443](../../../messagefoundry/pipeline/retention.py)), never inside the per-threshold loop | The first pass after enabling `prune_documents_after` pulls the entire un-stripped backlog of full PHI bodies into engine heap: OOM / engine crash mid-purge, unbounded PHI plaintext in memory, and a pass that blows straight through `max_pass_seconds` | Engine availability + PHI-in-heap exposure at exactly the moment PHI is being reduced | **No.** No test bounds the candidate set | **P1** | | **R6** Purge under sustained load is untested on every backend. SQLite holds the single writer lock across the whole multi-statement purge transaction (`async with self._lock`, [store.py:8345](../../../messagefoundry/store/store.py)); SQL Server builds an unbatched `#eligible` temp table ([sqlserver.py:5566](../../../messagefoundry/store/sqlserver.py)) and **`messages` has no `LOCK_ESCALATION=DISABLE`** (only `queue` does, [sqlserver.py:1057-1063](../../../messagefoundry/store/sqlserver.py)) | A first large purge escalates to a table X lock on `messages` and blocks ingress inserts, stalling ACKs past the MLLP receive timeout | Availability incident on the production backend, during a scheduled maintenance window | **No.** No concurrent-purge test anywhere; no purge profile in `docs/LOAD-TESTING.md`; no metric to see it coming | **P1** | diff --git a/docs/testing/master-test-plan/04-high-availability-and-dr.md b/docs/testing/master-test-plan/04-high-availability-and-dr.md index cceceefaf..f46e478c0 100644 --- a/docs/testing/master-test-plan/04-high-availability-and-dr.md +++ b/docs/testing/master-test-plan/04-high-availability-and-dr.md @@ -77,11 +77,11 @@ bare `HA-nn` in this chapter always means **this chapter's own Β§3.4 row** β€” t | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| The live PostgreSQL election suite runs nowhere | A changed predicate, a dropped `ON CONFLICT`/`WHERE` clause, or an epoch that bumps on renew ships green. `tests/test_cluster_failover_postgres.py` (9 live scenarios) is referenced by **no** workflow β€” grep of `.github/` finds only the SQL Server twin at `ci.yml:614`. | Two leaders, duplicate delivery, or a cluster that never fails over β€” on the **default** server backend | No | P0 | -| Every live server-DB DR suite runs nowhere | `test_dr_server_seed_gate_{postgres,sqlserver}.py`, `test_dr7_server_config_only_backup_{postgres,sqlserver}.py`, `test_backup_runner_server_db_{postgres,sqlserver}.py` are `MEFOR_TEST_*`-gated and invoked by no leg. | DR activation promotes onto a **fresh/unrestored** server DB β€” precisely the data-loss case BACKLOG #102 exists to close | No | P0 | -| A leaderless cluster pages nobody | `leadership_lost` is auto-resolve-only and deliberately emits no notification (`pipeline/alerts.py:206-213`); the allowed alert-type set `_ALERT_EVENT_TYPES` (`config/settings.py:2499-2526`, **18** members) has no "no leader" member. All nodes self-fenced (DB outage), all nodes `promotable=false`, or a lease nobody can take β†’ total stop. | Full silent intake stop with a filling staged queue; only surface is a console badge someone must be looking at | No | P0 | -| Engine shards and `[cluster]` collide on one lease | `_lease_key = f"{db_schema or 'public'}:mefor_cluster_leader"` (`cluster.py:414`) is **not** engine-shard-namespaced, and no validator relates `serve --shard` to `[cluster].enabled`. N engine-shard processes over one unified store would contend for the same row; N-1 bind nothing. | Silent full stop for every engine shard but one β€” looks like a healthy cluster on `/cluster/status` | No | P0 | -| Cross-store divergence after a DR fail-back | `pipeline/dr.py:296-301` states the engine gives no cross-store loss/duplicate guarantee and defers to a runbook. `harness/reconcile/` exists (capture / compare / normalize / report) but nothing binds it to the DR release path. | Messages processed only at DR, or reprocessed at both β€” silent clinical-data divergence with no failing test and no operator artifact | No | P0 | +| The live PostgreSQL election suite runs only on the server-DB gated leg | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): "runs nowhere" and "referenced by no workflow" were both false.** `tests/test_cluster_failover_postgres.py` is invoked at `ci.yml:1570`, in the `postgres-store` job's step *"Run the failover + engine-shard recovery suites on real Postgres"* (`pytest -v tests/test_cluster_failover_postgres.py tests/test_shard_recovery_postgres.py`). The cited anchor `ci.yml:614` has drifted and now lands on a comment. **The residual risk is real and stays P0, but it is a GATING risk, not an absence:** the job runs only on `schedule \|\| workflow_dispatch \|\| needs.changes.outputs.serverdb == 'true'`, so a PR that changes election logic without touching a server-DB path still ships unverified. A changed predicate, a dropped `ON CONFLICT`/`WHERE`, or an epoch that bumps on renew is caught nightly, not at review time | Two leaders, duplicate delivery, or a cluster that never fails over β€” on the **default** server backend | Partly β€” on the gated server-DB leg only, never on an ordinary PR | P0 | +| Every live server-DB DR suite runs only on the server-DB gated leg | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): "invoked by no leg" was false for all six.** `test_dr_server_seed_gate_{postgres,sqlserver}.py`, `test_dr7_server_config_only_backup_{postgres,sqlserver}.py` and `test_backup_runner_server_db_{postgres,sqlserver}.py` all exist and are all invoked β€” the Postgres trio by the `postgres-store` job's step *"Run the DR seed-gate + backup suites on real Postgres"*, the SQL Server trio by the `sqlserver-store` job's *"...on real SQL Server"*. **Same correction as the row above, and it stays P0 for the same reason:** both jobs carry the `schedule \|\| workflow_dispatch \|\| serverdb == 'true'` gate, so DR regressions are caught nightly rather than at review time. The `MEFOR_TEST_*` env gating is a second, independent condition and was *not* re-measured here β€” it is the part of this row that may still bite | DR activation promotes onto a **fresh/unrestored** server DB β€” precisely the data-loss case BACKLOG #102 exists to close | Partly β€” on the gated server-DB leg only, never on an ordinary PR | P0 | +| A leaderless cluster pages nobody | `leadership_lost` is auto-resolve-only and deliberately emits no notification (`pipeline/alerts.py:206-213`); the allowed alert-type set `_ALERT_EVENT_TYPES` (`config/settings.py:2599`, a `frozenset`, **18** members) has no "no leader" member. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100): both counts and both claims are EXACTLY right** β€” the set has 18 members and `leadership_acquired` is the only leader-related one, so nothing fires when leadership is *absent*; the `alerts.py:206-213` anchor still lands on `leadership_lost` ("Emits **no** notification"). Only the settings anchor had drifted (cited `:2499-2526`, declared at `:2599`) and is corrected; it is cited by **name and declaration line** now, because a range around a growing `frozenset` re-drifts on the next member added. All nodes self-fenced (DB outage), all nodes `promotable=false`, or a lease nobody can take β†’ total stop. | Full silent intake stop with a filling staged queue; only surface is a console badge someone must be looking at | No | P0 | +| Engine shards and `[cluster]` cannot be combined β€” **the collision is refused, not latent** | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): the load-bearing half of this row is false.** "No validator relates `serve --shard` to `[cluster].enabled`" β€” **there is one, it fails closed, and it is tested.** Per **ADR 0073** the two are mutually exclusive; `__main__.py:2649-2659` refuses the combination with *"error: `--shard` cannot be combined with `[cluster].enabled`"* and **exit 2, before the store or app is built**, and `tests/test_shard_recovery_engine.py:321::test_serve_refuses_shard_with_cluster_enabled` asserts `rc == 2`. The lease-key half is still accurate β€” `_lease_key = f"{db_schema or 'public'}:mefor_cluster_leader"` (**`cluster.py:484`**, not `:414`; anchor drifted) is **not** engine-shard-namespaced β€” **but that is now MOOT for this failure mode**, because N engine-shard processes can never come up against an enabled `[cluster]` to contend at all | **The stated blast radius cannot occur.** It was "silent full stop for every engine shard but one, looking healthy on `/cluster/status`"; a fail-closed startup refusal is the opposite of silent. What remains is only the residual design note that the lease key is un-namespaced, which matters solely if ADR 0073's mutual exclusion is ever lifted β€” so it is a **constraint to preserve**, not a gap to close. This also satisfies the HA-03 exit criterion at `:399` by its **first** alternative (a config-load rejection rather than an engine-shard-namespaced lease key), though the assertion lives in `test_shard_recovery_engine.py`, not the `tests/test_cluster_shard_interaction.py` that criterion names | Yes β€” fail-closed at startup, with a test | **Closed** | +| Cross-store divergence after a DR fail-back | `pipeline/dr.py:296-301` states the engine gives no cross-store loss/duplicate guarantee and defers to a runbook. `harness/reconcile/` exists (capture / compare / normalize / report) but nothing binds it to the DR release path. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100) β€” all three claims hold, and the anchor is exact:** `dr.py:296-301` still reads *"the engine gives no cross-store loss/duplicate guarantee β€” documented, not an engine AC"*; all four reconcile modules are present; and the binding is still absent β€” 4 unit-test files exercise `harness.reconcile`, but **no workflow and no runbook step invokes it** (the two `reconcile` hits in `.github/workflows/` are an unrelated ingress probe and an English verb). **SHARPER THAN "NOTHING BINDS IT":** `AOAG-DEPLOYMENT.md:708` *does* instruct the operator β€” "reconcile before you discard; do not rubber-stamp the loss" β€” **and never names `harness/reconcile/`**. So the runbook asks for the activity and points at no tool, which is worse than silence: the operator is told to do it and left to invent how | Messages processed only at DR, or reprocessed at both β€” silent clinical-data divergence with no failing test and no operator artifact. The fix is cheap and specific: have `:708` name the tool | No | P0 | | RTO is unbounded in absolute terms | `harness/load/failover.py:640-646` bounds `functional_recovery_seconds` only as `recovery_ttl_multiple Γ— leader_lease_ttl_seconds` (2.0 in `failover.toml`, 3.0 in `_failover_load_support.py`); `max_promotion_seconds` is optional (`profile.py:129-131`) and set in **neither** profile. No RPO figure is asserted anywhere. | Promotion can regress from ~7s to ~60s with every SLO green, because the bound scales with whatever TTL the profile sets | Partially (reported, never gated) | P1 | | A DB blip wedges the pool | `docs/AOAG-DEPLOYMENT.md:513-520`: "the engine has no reconnect logic of its own … treat automatic pool recovery as driver-dependent behavior to **drill**". Automated coverage simulates partition only via a fake pool raising `RuntimeError` (`test_cluster_lease.py`). | Engine wedged after a DB restart β€” listeners bound or unbound, no forward progress | Manual only (`W25:S4.10`) | P1 | | Always On AG failover is entirely documentation | 919 lines of `AOAG-DEPLOYMENT.md` specify replica modes, quorum, the phase-dependent 10s-vs-60s fence boundary, and the stranded-in-flight remedy. No lab, no leg, no drill script. `MultiSubnetFailover` is tested only as a DSN substring (`store/sqlserver.py:1414`). | Every enterprise SQL Server adopter hits this; wrong guidance means stuck in-flight messages nobody finds | No | P1 | @@ -94,7 +94,7 @@ bare `HA-nn` in this chapter always means **this chapter's own Β§3.4 row** β€” t | The PG store-side epoch guard is live-proven nowhere | `test_resumed_ex_leader_is_fenced_after_real_handover` exists in both live suites but only the SS one runs. The PG claim guard appears in **three** separate statements (`postgres.py:2686-2688, 2816-2820, 2934-2938`). | One missed append across three claim paths is invisible to fake-pool tests and shows up only as duplicate/out-of-order delivery under a real handover | No | P1 | | DR readiness is asserted nowhere on the box that needs it | `messagefoundry verify`'s sections are exactly `("host", "store", "smoke", "manual", "federation")` (`messagefoundry/verify/runner.py:22`) β€” no cluster, HA or DR group. `harness/acceptance/matrix.py:429-470` has G1-G5 with no DR row; `WIN2025-TEST-MATRIX.md` Β§G likewise. | An operator gets zero signal about DEK availability at the DR site, backup-destination writability, or cluster posture | No | P1 | | Windows port-rebind lag is a one-off manual number | Captured only as prose in `W25:S4.9`; not in the report JSON, no trend, no threshold. | On the intended production OS this dominates observed recovery; a regression is indistinguishable from normal | Manual only | P2 | -| Catalog drift understates the area | `FEATURE-MAP.md` Β§6's nine rows omit ADR 0047/0048/0049/0056/0096, `/cluster/nodes`, the H1 fence, on-promotion recovery and the DR run-profile. BACKLOG #100's ranked-backlog Tier cell still reads "P1" though it shipped 2026-07-10 per its own SHIPPED banner; BACKLOG #101's ranked-backlog Tier cell reads "DEMAND-GATE" though ADR 0096 is Accepted and the code is in `settings.py:2896-2910`. | Readers conclude DR standby, turnkey backup and cloud HA packaging do not exist, so they are never scoped for testing or release notes | No | P2 | +| Catalog drift understates the area | `FEATURE-MAP.md` Β§6's nine rows omit ADR 0047/0048/0049/0056/0096, `/cluster/nodes`, the H1 fence, on-promotion recovery and the DR run-profile. BACKLOG #100's ranked-backlog Tier cell still reads "P1" though it shipped 2026-07-10 per its own SHIPPED banner; BACKLOG #101's ranked-backlog Tier cell reads "DEMAND-GATE" though ADR 0096 is Accepted and the code is in `settings.py:3016-3026` (`promotable: bool = True`). **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100) β€” this row is CORRECT and its gap is still open**, which is why it is annotated rather than rewritten: FEATURE-MAP still returns **0 hits** for each of ADR 0047/0048/0049/0056/0096; #100's ranked Tier cell still reads "P1" (`BACKLOG.md:373`, rank 14) though the item is archived as shipped; #101's still reads "DEMAND-GATE" (`:380`). **Only the code anchor had drifted** β€” it cited `settings.py:2896-2910`, which is now ADR 0014 alert-rule code, and is corrected above. **The two Tier cells live in `docs/BACKLOG.md`, which a Builder may not edit (owner ruling), so fixing them is the owner's or the Dispatcher's** | Readers conclude DR standby, turnkey backup and cloud HA packaging do not exist, so they are never scoped for testing or release notes | No | P2 | ### 3.4 Test matrix diff --git a/docs/testing/master-test-plan/05-connections-and-transports.md b/docs/testing/master-test-plan/05-connections-and-transports.md index 050dd95b3..d54bf1c32 100644 --- a/docs/testing/master-test-plan/05-connections-and-transports.md +++ b/docs/testing/master-test-plan/05-connections-and-transports.md @@ -95,20 +95,20 @@ This chapter covers **every registered inbound/outbound connector and the connec | Eight post-plan connector ADRs have no coverage-plan row | Verified zero grep hits each for 0122, 0123, 0124, 0126, 0128, 0129, 0132, 0153 in `FEATURE-COVERAGE-PLAN.md` | Each has unit tests, but nothing owns their cross-backend, live-partner, on-box or PHI dimensions and no plan section notices a regression. The audit is point-in-time with no drift guard against the ADR ledger, so the hole widens with every demand-gate wave | No | P1 | | ADR 0124 `no_ack` finalizes on a TCP write | `_send_once_no_ack:897` / `_send_persistent_no_ack:1046` complete a delivery on a successful write with no ACK read and no MSA-1 validation | A `no_ack` outbound reports `PROCESSED` for a message the partner never processed. Operators read disposition as delivery confirmation, so the at-most-once-confirmation trade-off needs an explicit tested disposition and an operator-visible marker | No β€” `test_mllp_no_ack.py` and `test_no_ack_wiring.py` stop at the connector and wiring | P1 | | Five shipped connectors absent from the on-box Β§D matrix | Email/SMTP, Direct S/MIME, DICOMweb STOW-RS, the ADR 0023 HTTP listener and PassThrough have no `WIN2025-TEST-MATRIX.md` Β§D row (lines 47-59) | The box gate cannot certify connectors it does not list. Direct and Email are PHI **egress** paths and the HTTP listener is a PHI **intake** path, so `W25:S6.3` would declare the box accepted with three PHI paths untested on the host | No | P1 | -| Inbound HTTP listener has no application-layer authentication | `Http()` (`wiring.py:1063`) exposes caps + TLS/mTLS only; grep for auth/Authorization/bearer/hmac in `http_listener.py` returns **0** hits; ADR 0023 defers auth | A partner-facing HTTP PHI intake accepting unauthenticated POSTs is a security decision that is currently implicit. An operator exposing it off-loopback with `tls=true` but no `tls_ca_file` lets any host reaching the port inject messages into the pipeline, and nothing fails | No test states the accepted control set (mTLS + `source_ip_allowlist` + bind guard) as a deliberate posture | P1 | +| Inbound HTTP listener authenticates, but **defaults to `none`** | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15). The evidence in this row was false in every part.** "grep ... returns **0** hits" measures **52** matching lines today; `Http()` does *not* expose "caps + TLS/mTLS only"; and ADR 0023 no longer defers auth β€” **ADR 0154 (Accepted 2026-07-31) is literally titled its "deferred tail"** and delivered it. Built: `intake_auth: Literal["none","api_key","bearer","mtls_subject"]`, enforced by `_authorize_head` (`http_listener.py:618`) with a 401 + `WWW-Authenticate`, credentials `env()`-only, rotation via `intake_api_key_next`, health probes inside the gate by default (`intake_auth_health="require"`), and failed-attempt rate limits (10/min/peer, 60/min global) | **The mechanism is no longer the gap; the DEFAULT is.** `intake_auth` defaults to `"none"`, and `_authorize_head` returns early when it is unset β€” so a site that deploys the listener off-loopback without setting it **would** accept unauthenticated POSTs into the pipeline. That is now an explicit, documented, tested opt-in rather than an implicit omission, which is a different and much narrower decision: whether `"none"` is the right default for a PHI intake path | **Yes** β€” `tests/test_inbound_http_intake_auth.py` plus `test_auth_entry_hardening.py`, `test_credential_parameter_mapping.py`, `test_connection_factory_redaction_domain.py` and `test_inbound_http_sync_reply.py` (**`test_inbound_http_intake_auth.py` alone holds 22 test functions**, counted by AST; the five files together hold 76, of which 6 name `intake` directly. An earlier revision said "18 test functions naming the surface" β€” that figure counted `def test_.*intake` LINES across all of `tests/` and was then attributed to these five files, conflating search scope with cited scope. The correction strengthens the point rather than weakening it.). What remains unasserted is only the **default-posture** question above | P1 | | No negative mTLS handshake on any accept path | `test_mllp_tls.py:83` asserts `verify_mode` on the constructed context, not a live refusal. Nothing drives a client with no cert or a wrong-CA cert at an MLLP / HTTP-listener / DICOM-SCP mTLS listener | mTLS is the only peer authentication the raw listeners and the HTTP listener have. A regression leaving `CERT_OPTIONAL` in place admits unauthenticated PHI intake with no failing test | Owned as `FCP:MLLP-9`'s open gap; unclosed | P1 | | `FEATURE-MAP.md` Β§1 understates shipped capability | `:45` marks REST-IN/SOAP-IN/FHIR-IN deferred (ADR 0023 built); `:46` marks MLLP persistent deferred (ADR 0067 built). No row at all for Email, Direct, Timer, PassThrough, Loopback, the HTTP listener, compression, `no_ack`, or the forward proxy | The public capability catalogue is what a test author, an evaluator and an adopter scope from. Understated connectors go untested and unaudited; `tests/test_feature_map_claims.py` guards only the ASVS score and private links, so it rots further | No | P1 | | Listener `SSLContext` built once at construction | `mllp.py:1324`, `http_listener.py:292`, `dicom.py:178` all build the context in `__init__`, so a renewed cert on disk is not presented until a connection restart or config reload | An operator rotating a listener cert in place believes it took effect; when the old cert expires the listener stops accepting partners β€” a full intake outage β€” and the only prior signal is an expiry alert they may have silenced after "rotating" | No test or doc states it; `CONNECTIONS.md:714` carries rotation guidance for the DB CA only | P2 | | `peer_reset` and `framing_error` emitted but never asserted | `mllp.py:1471`/`:1476`, `tcp.py:547`/`:552`, `http_listener.py:391`/`:397`; `test_connection_event_emit.py` (8 tests) covers only the other six kinds | Peer resets and framing errors are the two kinds an operator most needs when diagnosing a flapping partner. A refactor that swallows them leaves the event log quietly incomplete with nothing failing | No | P2 | -| `docs/CONNECTIONS.md` contradicts the code | SFTP-IN/OUT "⏳ planned" (`:35-36`) vs `remotefile.py:1016-1017`; SOAP-IN/REST-IN/FHIR-IN "⏳ planned" (`:37`, `:39`, `:43`) vs the built listener; SMTP-OUT "⏳ planned" (`:50`) vs `email.py:265`; DB-IN "(SQL Server, exp.)" (`:41`) vs FEATURE-MAP "production". No settings sections for `Sftp()`/`Ftp()`, `Timer()`, `PassThrough()`, `Http()` or `Direct()` (heading list `:195-1372`) | Adopters and test authors read this file as the connector contract. A partner integration gets scoped away because the doc says "planned", and five shipped connectors have no documented settings surface to validate against β€” against CLAUDE.md Β§11 | No β€” `test_communications_inventory.py` requires only a resource-table row, not a settings heading | P2 | +| `docs/CONNECTIONS.md` contradicts the code β€” **CLOSED, re-measured 2026-08-15 (BACKLOG #1100)** | **Every citation in this row is stale, and the one doc line that still reads "planned" is right.** Measured: `SFTP-IN` `:42` is `βœ… (Sftp(), [sftp] extra)`; `REST-IN` `:48` is `βœ… (Http(), ADR 0023)`; `SMTP-OUT` `:57` is `βœ… (Email()/SMTP(), ADR 0029)`; `DB-IN` `:50` is `βœ… (SQL Server + generic ODBC)`, **agreeing with** `FEATURE-MAP.md:41` "Production"; `SOAP-IN` `:46` is `~ receive-only`, not "planned". All six connectors called undocumented now have settings headings: `Sftp()`/`Ftp()` `:845`, `Http()` `:533`, `Direct()` `:1385`, `Timer()` `:1794`, `PassThrough()` `:1867`. **`FHIR-IN` `:52` does still read `⏳ planned (BACKLOG #20)` β€” and that is accurate, not a contradiction:** `:2503` records the FHIR-IN facade as unshipped, "a consumer of the shipped `Http()` listener, not new substrate". This row inferred *the listener is built, therefore FHIR-IN is built*; shipping the substrate does not ship the facade | None β€” no work is scoped by this row. Kept rather than deleted so the re-measurement is auditable, and as the sixth recorded instance of the anchor rule: **all seven** line citations here (`:35-36`, `:37`, `:39`, `:41`, `:43`, `:50`, `:195-1372`) had drifted off their subjects, so each had to be re-found by content before it could be judged | n/a β€” closed | β€” | | SFTP key loading is RSA-only | `remotefile.py:411` calls `paramiko.RSAKey.from_private_key`; an Ed25519 or ECDSA partner key fails | Modern SFTP partners default to Ed25519. The failure is loud but undiagnosable from the docs, and nothing pins the constraint so it cannot silently narrow further | No | P2 | -| Five ADR status lines lag their code | 0011 "Proposed"; 0015 and 0016 "No code written yet"; 0021 "No code yet"; 0022 "Design-only (no code yet)" β€” all verifiably built (`timer.py:341`; `soap.py:128-147`/`:406`; `test_x12_rte.py` + `ci.yml:702`/`:855`; `test_ack_capture_runner.py`; `fhir.py:647`) | Anyone scoping test work off ADR status either skips a built feature or plans to build one that exists. ADR 0020 shows status is used to record culls, so a wrong status is a governance defect | No | P2 | +| **Four** ADR status lines lag their code (**was five** β€” re-measured 2026-08-15, BACKLOG #1100) | Still lagging, each re-verified on **both** halves β€” the status line AND the code it contradicts: **0011** "Proposed" (`transports/timer.py`, 341 lines); **0015** "**No code written yet**" (`transports/soap.py`, 844 lines); **0016** "**No code written yet**" (`tests/test_x12_rte.py`, 625 lines, referenced twice in `ci.yml`); **0022** "Design-only (no code yet)" (`tests/test_ack_capture_runner.py`, `fhir.py`). **0021 HAS SINCE BEEN CORRECTED** β€” its status now reads "Accepted (2026-06-19, owner go). Built and shipped (`eventlog` #16, Lane S)" β€” so it is struck from the count rather than left inflating it | Anyone scoping test work off ADR status either skips a built feature or plans to build one that exists. ADR 0020 shows status is used to record culls, so a wrong status is a governance defect. **0021 is itself the evidence that these get fixed one at a time**, which is why the count must be re-measured rather than carried forward | No | P2 | ### 4.4 Test matrix **Row class.** Every row carries a **Cls**: **T** = *Test* (a falsifiable assertion with an observable pass criterion β€” **only T rows count toward the release gate**); **C** = *Characterisation* (produces a recorded measurement, finding or dated decision, with no threshold yet β€” legitimate work that **cannot fail**, so it never gates a release, and becomes a T row the day its threshold is recorded); **A** = *Assurance* (an external engagement β€” pen test, third-party review, DAST β€” blocking only for an off-loopback / production-exposure release, and excluded from the ordinary P0 count). -This chapter has **67 rows: 63 T, 4 C (CONN-34, CONN-35, CONN-38, CONN-49 β€” all editorial doc/ledger reconciliations), 0 A.** Nine of the 63 T rows are **P0**: CONN-01 through CONN-08, plus the CONN-37 pointer row (P0 because its owner SEC-01 is P0). **Foreign IDs carry a prefix** β€” `FCP:` for a `docs/testing/FEATURE-COVERAGE-PLAN.md` gap ID and `W25:` for a WIN2025 test/matrix ID; a bare `CONN-nn`, `MIG-nn`, `SEC-nn`, `CFG-nn` etc. is always a row of **this** plan. Four rows are **pointers** (Method `β€”`, no work scoped here, kept so the deliverable is still counted): CONN-32 and CONN-33 β†’ **MIG-74**, CONN-37 β†’ **SEC-01**, CONN-54 β†’ `FCP:CFG-3`/`FCP:CFG-4` + this plan's CFG-19/CFG-20. +This chapter has **67 rows: 63 T, 4 C (CONN-34, CONN-35, CONN-38, CONN-49 β€” all editorial doc/ledger reconciliations), 0 A.** Nine of the 63 T rows are **P0**: CONN-01 through CONN-08, plus the CONN-37 pointer row (P0 because its owner SEC-01 is P0). **EVERY FIGURE IN THIS PARAGRAPH RE-VERIFIED EXACT 2026-08-15 (BACKLOG #1100)** by parsing the tables rather than reading them: 67 `CONN-nn` rows, the 63/4/0 split holds, CONN-34/35/38/49 are all present as the C rows, and the nine claimed P0 T rows are all present and all P0 with **nothing claimed that is missing**. **A caution for anyone re-checking it, because it caught me:** a naive scan for a `P0` priority cell across this chapter returns **11**, not 9 β€” the two extras are *risk-table* rows (first cell "FrameDecoder treats SB inside an open frame as payload" and "WIN2025 acceptance rows bound to tests that do not exercise the named connector"), because the risk table carries its own priority column. **The chapter has TWO tables with P0 cells and only ONE of them is the T-row matrix this sentence counts**, so a discrepancy against 9 is the counter's error, not the doc's. **Foreign IDs carry a prefix** β€” `FCP:` for a `docs/testing/FEATURE-COVERAGE-PLAN.md` gap ID and `W25:` for a WIN2025 test/matrix ID; a bare `CONN-nn`, `MIG-nn`, `SEC-nn`, `CFG-nn` etc. is always a row of **this** plan. Four rows are **pointers** (Method `β€”`, no work scoped here, kept so the deliverable is still counted): CONN-32 and CONN-33 β†’ **MIG-74**, CONN-37 β†’ **SEC-01**, CONN-54 β†’ `FCP:CFG-3`/`FCP:CFG-4` + this plan's CFG-19/CFG-20. | ID | Test | Type | Method | Env | Backend | Cls | Pri | Pass criteria | |---|---|---|---|---|---|---|---|---| diff --git a/docs/testing/master-test-plan/06-parsing-and-codecs.md b/docs/testing/master-test-plan/06-parsing-and-codecs.md index f7ac61846..d35cb1dc9 100644 --- a/docs/testing/master-test-plan/06-parsing-and-codecs.md +++ b/docs/testing/master-test-plan/06-parsing-and-codecs.md @@ -71,8 +71,8 @@ | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| Blank segment in any feed (i.e. any partner writing `\r\n\r\n`) | `Peek.field`, every routing property and `summarize()` raise a bare `IndexError` from `_builtin_hl7.raise_if_blank_segment_scan`. The reads at `wiring_runner.py:3721` (`peek.control_id`) and `:3732` (`summarize(peek)`) sit **outside every catch**, so it unwinds out of `_handle_inbound`; `transports/mllp.py` logs, emits `framing_error` and drops the TCP connection | Every message on that connection: 0 `messages` rows, 0 `queue` rows, no ACK/NAK. Direct count-and-log break **plus** a one-packet, infinitely repeatable per-connection DoS | **No β€” worse than no.** `tests/test_builtin_hl7_parity.py:183-189` (`adv:empty-fields`) *contains* a blank segment, and `_eq()` at `:74` compares exceptions **by type**, so both backends raising `IndexError` scores a PASS. `samples/messages/hapi-hl7v2/oml_o21.hl7` (26 interior blank lines) is in the globbed parity corpus and is certified this way | **P0** | -| `\X00\` in any summarized field | Unescapes to a real U+0000 **inside `summarize()`** β€” after the `FCP:INGEST-4` post-decode NUL guard (`wiring_runner.py:3523`), which inspects only the raw decoded text (the raw carries the 5-char escape, no NUL). The NUL rides `summary=summarize(peek)` into `enqueue_ingress` at `:3732`, a **pre-ACK** commit. Reproduced: `'MRN AB\x00CD Β· DOE, JANE'` | Postgres rejects a NUL at bind (`DataError`/SQLSTATE 22021) β†’ the raise unwinds out of `_handle_inbound` exactly as the ADR 0028 `FCP:INGEST-4` amendment describes, dropping the connection with **no ERROR row**. SQLite/SQL Server truncate the summary β†’ operator blind spot in list/search | No. `grep -r 'X00' tests/` returns nothing. The whole `FCP:INGEST-4` retrofit was built for this class and missed the derived-value path | **P0** | +| Blank segment in any feed (i.e. any partner writing `\r\n\r\n`) | `Peek.field`, every routing property and `summarize()` raise a bare `IndexError` from `_builtin_hl7.raise_if_blank_segment_scan`. The reads sit **outside every catch**, so it unwinds out of `_handle_inbound`. **RE-VERIFIED STRUCTURALLY 2026-08-15 (BACKLOG #1100) β€” the citations had drifted ~230 lines, and the claim is re-established by AST rather than by line number so it survives the next move.** `_handle_inbound` is now **`wiring_runner.py:3950-4248`**; walking every `ast.Try` inside it and testing membership: **`control_id=peek.control_id` (`:4232`) and `summary=summarize(peek)` (`:4235`) are both OUTSIDE every `try` body**, as are all three `enqueue_ingress` calls (`:3977`, `:4077`, `:4229`). The raising mechanism is real β€” `_builtin_hl7.raise_if_blank_segment_scan`, called from `parsing/peek.py:265`. **The cited `:3721`/`:3732` now land on a blank line and on `record_received` respectively**, so a spot-check of this row would have found nothing wrong and nothing right; `transports/mllp.py` logs, emits `framing_error` and drops the TCP connection | Every message on that connection: 0 `messages` rows, 0 `queue` rows, no ACK/NAK. Direct count-and-log break **plus** a one-packet, infinitely repeatable per-connection DoS | **No β€” worse than no.** `tests/test_builtin_hl7_parity.py:183-189` (`adv:empty-fields`) *contains* a blank segment, and `_eq()` at `:74` compares exceptions **by type**, so both backends raising `IndexError` scores a PASS. `samples/messages/hapi-hl7v2/oml_o21.hl7` (26 interior blank lines) is in the globbed parity corpus and is certified this way. **RE-VERIFIED VERBATIM 2026-08-15 (BACKLOG #1100) as part of an audit of the claims I had carried forward WITHOUT checking while correcting this row's anchors** β€” this one holds exactly: `_eq()` at `:74` is `return type(expected) is type(got)`, its docstring reading "exceptions compare by **type** (message text differs)", and `_EMPTY_FIELDS` at `:183-189` contains a literal `"\r" # blank segment` line whose own comment names it. **So the parity suite does certify the defect as a PASS**, and it does so by a deliberate design choice (type-only exception comparison) that is correct for its purpose and wrong for this case | **P0** | +| `\X00\` in any summarized field | Unescapes to a real U+0000 **inside `summarize()`** β€” after the `FCP:INGEST-4` post-decode NUL guard which inspects only the raw decoded text (the raw carries the 5-char escape, no NUL). **RE-VERIFIED AND RE-POINTED 2026-08-15 (BACKLOG #1100) β€” anchors had drifted ~500 lines; substance CONFIRMED BY LIVE REPRODUCTION, not by reading.** The guard is now `wiring_runner.py:4026` (`if "\x00" in text:`); the NUL rides `summary=summarize(peek)` (`:4235`) into `enqueue_ingress` (`:4229`), a **pre-ACK** commit that the same AST pass used for the row above confirms is **outside every `try`**. **The guard's OWN COMMENT is the false premise (SDS-3.7): `:4031-4033` states it dead-letters "BEFORE Peek.parse and any store write, so text (and control_id/summary/strict-fail errors derived from it) is NUL-free for the rest of this handler" β€” `summary` is precisely a derived value for which that is UNTRUE.** Re-run 2026-08-15 against `parsing/summary.py:27` (note: `summarize` lives there, NOT in `parsing/peek.py`): body containing `DOE^JANE\X00\SMITH` gives `"\x00" in text` = **False** at the guard, and `summarize()` then returns `'MRN MRN123 \xb7 DOE, JANE\x00SMITH'` β€” **a literal U+0000 manufactured after the control that promised there was none** | Postgres rejects a NUL at bind (`DataError`/SQLSTATE 22021) β€” **attributed, NOT independently tested here: this is the engine's own statement at `wiring_runner.py:410`, "Postgres REJECTS it at bind (the raise is uncaught)". A live Postgres reproduction is still owed** β€” the raise unwinds out of `_handle_inbound` exactly as the ADR 0028 `FCP:INGEST-4` amendment describes, dropping the connection with **no ERROR row**. SQLite/SQL Server truncate the summary β†’ operator blind spot in list/search | No. `grep -r 'X00' tests/` returns nothing. The whole `FCP:INGEST-4` retrofit was built for this class and missed the derived-value path | **P0** | | ADR 0054 Phase 2 drops python-hl7 (`pyproject.toml:48` still pins `hl7>=0.4.5`) | The dual-backend oracle structurally requires python-hl7 installed. On removal `test_builtin_hl7_parity.py` becomes vacuous or uncollectable | Every subsequent edit to a 991-line from-scratch reimplementation of python-hl7's tolerant semantics ships unguarded | No golden vectors exist. Already flagged as `FCP:PARSE-12` (row at `FEATURE-COVERAGE-PLAN.md:857`, recommendation at `:874`) | P1 | | HL7 version breadth is 2.5.1-only | Generators are hl7apy `v2_5_1`-driven (`generators/_core.py:24,142,495`); `hl7schema.py:22` and `hl7structures.py:31` are pinned to `SUPPORTED_VERSION = "2.5.1"`; no test drives `validate()` over a 2.3/2.3.1/2.4/2.6/2.7 body | Real hospital feeds are dominated by 2.3/2.4. An hl7apy upgrade could start NAKing an entire production feed synchronously with no failing test | No | P1 | | Z-segment tolerance under strict validation is unpinned | Verified today: appending `ZPD\|1\|custom^data` to a conformant ADT^A01 still yields `ok=True` β€” but nothing asserts it | If an hl7apy upgrade tightens this, **every** message on **every** strict inbound NAKs AE synchronously β€” total feed outage β€” and the suite stays green | No | P1 | diff --git a/docs/testing/master-test-plan/07-config-wiring-and-cli.md b/docs/testing/master-test-plan/07-config-wiring-and-cli.md index 51723f093..277fc8964 100644 --- a/docs/testing/master-test-plan/07-config-wiring-and-cli.md +++ b/docs/testing/master-test-plan/07-config-wiring-and-cli.md @@ -15,12 +15,12 @@ This chapter covers the **configuration bundle and the command-line surface that - **Config-source trust** β€” POSIX group/world-writable + foreign-owner refusal (`wiring.py:4147`, `:4124`) and the Windows NTFS-DACL guard (`_evaluate_config_dacl:3889`, `_assert_safe_config_source_windows:3917`, ADR 0036), plus the `MEFOR_ALLOW_INSECURE_CONFIG_SOURCE` dev escape (`settings.py:244/247`). - **`connections.toml` config-as-data (ADR 0007)** β€” the read path (`connections_file.py:156/273`, `_TRANSPORTS:70`), the comment-preserving write path (`connections_edit.py:161/202/349/363`), the `connection list|upsert|remove|schema` CLI (`__main__.py:3788`), the `connection schema` form contract (`connection_schema.py:56`), and the consoleβ†’TOML `flagged` write seam (ADR 0007 amendment). - **Service settings** β€” precedence CLI > env(`MEFOR_*`) > toml > default (`settings.py:4128`), the `MEFOR_
_` parser (`:3777`) against `_SECTIONS` (`:109`), the ADR 0118 `[security]` desugar + relocated-key rejection (`:3861`, `:3848`), secrets-in-file warning (`:3789`), and `extra="ignore"` tolerance (`ServiceSettings:3623`). -- **Environments + deferred `env()`** β€” `environments/.toml` + `MEFOR_VALUE_*` overlay (`environments.py`), `EnvRef` inline-table decode and bounded casts (`wiring.py:195/226`), fail-loud-at-build resolution (`wiring.py:563`). +- **Environments + deferred `env()`** β€” `environments/.toml` + `MEFOR_VALUE_*` overlay (`environments.py`), `EnvRef` inline-table decode and bounded casts (`wiring.py:197/226` β€” **`:195` RE-POINTED to `:197` 2026-08-15 (BACKLOG #1100): it had drifted onto a BLANK LINE between the end of the `EnvRef` dataclass and the factory.** `:197` is `def env(key, *, default=_UNSET, cast=...) -> EnvRef:`, which carries the `cast` this row names; `EnvRef`'s own `cast` field is `:194`. **`:226` NOT CHECKED and deliberately left alone** β€” it lands on code and I have no stated intent to test it against, so re-pointing it would manufacture precision I do not have), fail-loud-at-build resolution (`wiring.py:563`). - **Single project-root anchoring (ADR 0050)** β€” `anchor.py` (`resolve_project_root`, `anchor_under_root`, `graph_references_env`), the offline anchor trio on `validate`/`graph`/`dryrun`/`check` (`__main__.py:754/783`), the AC-3 scoped fail-loud (`:846`), and the drive-relative member guard. - **The `messagefoundry check` gate** β€” 5 **required** checks (`validate`, `dryrun`, `posture`, `build-check`, `reference-backend`) and 9 **advisory** checks (`cleartext-accepted`, `ruff`, `mypy`, `ruff-security`, `raise-fstring`, `accepts-candidate`, `dead-config`, `send-target`, `handler-security`), assembled in `run_checks` (`checks.py:109`, roster at `:132-188`); the ADR 0144 handler-security lint (`:863`) with `--strict-handler-security` / `--handler-security-allow`. - **`messagefoundry init` scaffold** (`scaffold.py:353/370`) including the generated adopter CI workflow. - **The full CLI surface** β€” the 30 subcommands in `_DISPATCH` (`__main__.py:4302`) and their flags. -- **The ADR 0076 typed action vocabulary** (`actions.py`, 15 pure `Message` helpers) and the **lens Steps backend** (`lens.py:270/285/1406/1423`) that the VS Code Steps view consumes. +- **The ADR 0076 typed action vocabulary** (`actions.py`, 15 pure `Message` helpers) and the **lens Steps backend** (`lens.py:271/285/1406/1423` β€” **`:270` RE-POINTED to `:271` 2026-08-15 (BACKLOG #1100): it had drifted onto a BLANK LINE, and `:271` is `def _native_action_row(`.** The other three land on real code (`:285` an assignment inside a params loop, `:1406` a `return (`, `:1423` an `if field is not None:`) but all three are MID-FUNCTION rather than at a definition, and this row states no intent to check them against β€” **so they are NOT VERIFIED, only NOT OBVIOUSLY WRONG**, which is the limit of a blank-line scan) that the VS Code Steps view consumes. **Explicitly NOT in scope here.** *Promotion, publishing, config reload and environment targeting belong to the **PUB** chapter* β€” `POST /config/reload`, the IDE Stageβ†’Promote flow, the promote pre-flight and env-aware targeting are cross-referenced, never re-planned here. Also out of scope and owned elsewhere: @@ -75,11 +75,11 @@ This chapter covers the **configuration bundle and the command-line surface that | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| The shipped gate runs in no engine CI leg | A change demotes `build-check` from required, or breaks a sub-check, and every unit slice still passes | Every adopter's whole config-quality gate exits 0 on a config `serve` will refuse; discovered at deploy or in production | No β€” `.pre-commit-config.yaml` owns `.git/hooks/pre-commit`; `.mefor-hooks/pre-commit` is generated but not installed; no workflow invokes `check` | P0 | -| No pin on the check roster | A check is dropped or its `required` flag flipped; exit-code contract changes silently | `build-check` is the only gate catching an ADR 0092 production-PHI cleartext hop pre-`serve`; `reference-backend` the only one catching a `Reference()` against SQL Server that raises post-ACK forever | No test enumerates `run_checks`' result names or flags | P0 | -| Unknown settings sections/keys silently dropped | `[secrity]`, a misspelled `require_mfa`, or `allow_unencrypted_phi` in the wrong section loads clean and applies a default | An operator believes a posture switch is set; in the mis-sectioned case the **insecure** default applies with zero signal | No β€” `ServiceSettings` and each section are `extra="ignore"` (`settings.py:3623`) and `tests/test_settings.py:118` pins the silence as intended. Only `[security]` warns (`settings.py:3880`), and that warning itself is untested | P0 | -| IDE Steps contract drift | `lens.py` changes `kind` / `params` / `literal_params` / `line_start` / `line_end`; both suites stay green against committed static fixtures | Steps view mis-renders or, worse, `lens rewrite` splices an edit into the wrong statement of a clinical transform β€” wrong data downstream | No β€” `ide/src/test/suite/steps.test.ts:28` reads `ide/src/test/fixtures/lens/*.json`; the `ide` CI job has no Python and nothing regenerates or diffs them | P0 | -| 5 of 28 settings sections unreachable by `MEFOR_*` | `sandbox`, `service`, `cert_monitor`, `secret_rotation`, `update_check` are absent from `_SECTIONS`; the three multi-word names are additionally unreachable through `.partition("_")` | `MEFOR_SANDBOX_MODE=subprocess` in an NSSM environment block yields `off` β€” the ADR 0087 Router/Handler isolation boundary believed on and silently off | No test asserts `_SECTIONS` equals `ServiceSettings.model_fields`; an unmatched `MEFOR_*` var is dropped without a warning | P0 | +| The shipped gate runs in no engine CI leg | A change demotes `build-check` from required, or breaks a sub-check, and every unit slice still passes | Every adopter's whole config-quality gate exits 0 on a config `serve` will refuse; discovered at deploy or in production | **No β€” CONFIRMED ON ALL THREE CLAIMS 2026-08-15 (BACKLOG #1100).** `.git/hooks/pre-commit` begins *"File generated by pre-commit"*, so `.pre-commit-config.yaml` owns it. `.mefor-hooks/pre-commit` exists **and differs byte-for-byte from the installed hook** β€” generated, not installed, verified by `cmp` rather than by inference. And **no workflow invokes `messagefoundry check`**: 0 files, against a positive control of 10 workflows invoking `pytest`. **BEWARE THE COLLISION A RE-CHECK WILL HIT:** grepping for lines containing both `messagefoundry` and `check` returns **7**, and every one is `ruff check … messagefoundry` β€” ruff's subcommand beside the package name β€” plus one comment. The engine's own gate CLI appears nowhere. So the shipped config-quality gate runs in no CI leg, and an adopter's gate can exit 0 on a config `serve` will refuse | P0 | +| No pin on the check roster | A check is dropped or its `required` flag flipped; exit-code contract changes silently | `build-check` is the only gate catching an ADR 0092 production-PHI cleartext hop pre-`serve`; `reference-backend` the only one catching a `Reference()` against SQL Server that raises post-ACK forever | **CONFIRMED IN SUBSTANCE, SHARPENED 2026-08-15 (BACKLOG #1100): THE SCHEMA IS PINNED, THE ROSTER IS NOT β€” and the row's wording understates the first half.** `tests/test_checks.py:383-385` pins the result SHAPE tightly: `set(report.keys()) == {"ok","checks"}` and `set(c.keys()) == {"name","ok","required","skipped","detail"}`. Individual flags are asserted too (`validate["required"] is True` at `:39`/`:50`, `dryrun` at `:64`). **So a dropped FIELD is caught immediately.** What nothing asserts is WHICH CHECKS EXIST: `run_checks` (`checks.py:114`, 103 lines) assembles its results by calling a series of `_check_*` functions plus tool runs, and **no test enumerates that set**. Dropping a check, or flipping its `required`, leaves every assertion above still passing. **DELIBERATELY NOT STATING A COUNT HERE:** three separate structural extractions of the roster returned 0 or partial results, and an unverified count is exactly what this sweep keeps finding rotted. The schema-vs-roster distinction is the finding and needs no number | P0 | +| Unknown settings sections/keys silently dropped | `[secrity]`, a misspelled `require_mfa`, or `allow_unencrypted_phi` in the wrong section loads clean and applies a default | An operator believes a posture switch is set; in the mis-sectioned case the **insecure** default applies with zero signal | **Partly β€” RE-MEASURED 2026-08-15 (BACKLOG #1100); one clause is FALSE and the surviving gap is SHARPER than written.** Confirmed: `extra="ignore"` holds via the `_Section` base (**`settings.py:173`**) and `ServiceSettings` (**`:3811`**, cited `:3623`) β€” the only `extra="forbid"` models are `EscalationTier` and `AlertRule`, nested rule objects, not sections. `tests/test_settings.py:118` (`test_unknown_sections_and_keys_ignored`) does pin the silence as intended; that anchor lands. **FALSE: "that warning itself is untested."** `tests/test_security_config.py` covers it with a positive AND a negative control β€” `test_unknown_security_key_warns_and_is_ignored` (`:386`) and `test_known_security_keys_warn_about_nothing` (`:401`, asserting "unrecognized key" is absent for valid keys). The warning is at **`:4060-4074`** (cited `:3880`, which is now `_cluster_requires_server_db`). **THE REAL GAP, WHICH THIS ROW'S OWN FIRST EXAMPLE FALLS INTO:** that warning diffs `set(raw) - set(SecuritySettings.model_fields)` β€” it catches an unknown **key inside a present `[security]` section**. A misspelled **section header** like `[secrity]` is dropped by `ServiceSettings`' `extra="ignore"` and **never reaches the check**, so it warns nothing. Mis-sectioning is the uncovered half, not mis-keying | P0 | +| IDE Steps contract drift | `lens.py` changes `kind` / `params` / `literal_params` / `line_start` / `line_end`; both suites stay green against committed static fixtures | Steps view mis-renders or, worse, `lens rewrite` splices an edit into the wrong statement of a clinical transform β€” wrong data downstream | **No β€” CONFIRMED 2026-08-15 (BACKLOG #1100), and a DRIFT GUARD NEXT DOOR IS WHAT MAKES THE GAP INVISIBLE.** `steps.test.ts` sets `FIXTURE_DIR` to `src/test/fixtures/lens` (7 fixtures present), and the `ide` CI job has **7 steps, none mentioning Python**. **The trap:** `tests/test_lens_schema.py::test_ide_fixture_in_sync` (`:101`) *does* regenerate-and-diff a fixture β€” and it guards **`fixtures/lens-SCHEMA/op-schema.json`**, a **different directory holding ONE file**, with the exact regeneration command in its failure text. `fixtures/lens/` (the **seven** parse-result fixtures this row is about) is referenced by **TypeScript only**; no Python touches it and nothing regenerates or diffs it. A grep for `fixtures/lens` matches `fixtures/lens-schema` as a prefix and reports the gap as covered. **So the schema is drift-guarded and the CONTENT is not β€” the same shape as row 79's "schema pinned, roster not", one row apart in the same chapter** | P0 | +| 5 of 28 settings sections unreachable by `MEFOR_*` | `sandbox`, `service`, `cert_monitor`, `secret_rotation`, `update_check` are absent from `_SECTIONS`; the three multi-word names are additionally unreachable through `.partition("_")` | `MEFOR_SANDBOX_MODE=subprocess` in an NSSM environment block yields `off` β€” the ADR 0087 Router/Handler isolation boundary believed on and silently off | **No β€” CONFIRMED ON EVERY CLAIM 2026-08-15 (BACKLOG #1100), measured by AST.** `_SECTIONS` (**`settings.py:109`**) holds **23** entries; `ServiceSettings` has **28** annotated fields; the difference is **exactly 5** and exactly the five named β€” `sandbox`, `cert_monitor`, `secret_rotation`, `update_check`, `service`. The `.partition("_")` mechanism is at **`:3971`** (`section, _, key = name[len(_ENV_PREFIX):].lower().partition("_")`), so `MEFOR_CERT_MONITOR_*` parses as section `cert`, key `monitor_*` β€” the three multi-word names are unreachable a second way. **`MEFOR_SANDBOX` appears in ZERO tests.** And no test pins `_SECTIONS` against `ServiceSettings.model_fields`: five test files match the token, but **none imports `settings._SECTIONS`** β€” `test_verify.py`'s hit is `ALL_SECTIONS` from `verify.runner`, an unrelated constant (`host, store, smoke, manual, federation`), i.e. a substring collision rather than coverage. **So ADR 0087's Router/Handler isolation boundary can be set in an NSSM environment block, read back as `off`, and say nothing** | P0 | | Shift-left checks disable themselves on the likeliest-wrong input | `posture` / `build-check` / `reference-backend` return `ok=True, skipped=True` when `messagefoundry.toml` will not load (`checks.py:1189`, `:1266`, `:1415`) | An adopter with an ADR 0118 relocated key (a hard `ValueError`) gets fully green CI then a `serve` refusal at deploy | No test covers the settings-did-not-load skip path | P1 | | `_TRANSPORTS` vs the factory set drifts | A transport factory added to `wiring.py` without a `_TRANSPORTS` entry is invisible to config-as-data and the GUI form; a mis-mapped name produces a wrong `ConnectionSpec` | A silently unauthorable connector, or a byte-wrong spec that only shows up at runtime | No β€” `test_connection_schema.py` asserts the schema mirrors `_TRANSPORTS`, not that `_TRANSPORTS` mirrors the 19 factories | P1 | | Write CLIs carry no `--project-root` | `connection`/`codeset`/`alert`/`security`/`impact` resolve `env()` from `Path.cwd()` only (`__main__.py:3836`) | Under a CLI-only anchor or an IDE cwd off the repo root, validate-before-persist resolves a different `env()` view than `serve` β€” an upsert persists a connection `serve` will refuse at reload | No test exercises the asymmetry | P1 | diff --git a/docs/testing/master-test-plan/08-publishing-and-promotion.md b/docs/testing/master-test-plan/08-publishing-and-promotion.md index 562650f26..09ad6abb9 100644 --- a/docs/testing/master-test-plan/08-publishing-and-promotion.md +++ b/docs/testing/master-test-plan/08-publishing-and-promotion.md @@ -104,12 +104,12 @@ post-quiesce failure. Those are this chapter's centre of gravity. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| Restart-based publish has zero attribution | `Engine.start()` does `load_config` + `add_registry` (`api/app.py:5455-5462`); only `Engine.reload()` sets `loaded_config_fingerprint` (`engine.py:1488-1500`). Copy-files + NSSM restart (and the tray **Restart Service**) leave `loaded=false` and write no fingerprint-bearing audit row. | Every engine that was ever restarted rather than reloaded β€” i.e. the primary CI/CD path and every service restart. `/ui` badge blank, IDE can never show DRIFTED, no `who activated these bytes` row. | **No.** `GET /config/provenance` returns `loaded=false`, which reads as "nothing loaded", not "unattested". | P0 | -| Fingerprint is blind to the real `environments/` | `_FINGERPRINT_GLOBS` includes `environments/*.toml` **relative to the config dir** (`fingerprint.py:39-45`), but every shipped/documented layout keeps `environments/` as a project-root **sibling** of `--config` (repo root here; `samples/config/` has no `environments/` subdir; ADR 0017 layout; ADR 0050 anchoring). | Repointing `acme_adt_host` in `environments/prod.toml` redirects PHI to a new peer with an **identical** fingerprint, no drift flag, and an audit row indistinguishable from a clean reload. | **No** β€” and ADR 0041 explicitly claims the opposite ("spans … `environments/`"). The existing test passes only because it writes `environments/` *inside* the tmp config dir. | P0 | -| "Promoted to PROD" can be false | A remote promote sends `config_dir: null` (`promote.ts:135-149`) so the engine reloads its **own** on-disk dir; `ReloadResult` (`api/models.py:351-360`) carries no fingerprint. If CI/CD never delivered the commit, PROD reloads stale bytes and the toast still says "promoted". | Silent whenever element counts are unchanged β€” i.e. almost every Handler-logic change. Highest-frequency wrong belief in the publish path. | **No.** Nothing recomputes or compares a fingerprint; there is no `messagefoundry fingerprint` CLI. | P0 | -| A held (dual-control) promote renders as success | `engineClient.postJson` resolves any 2xx and casts the body (`engineClient.ts:92-102`); a 202 `PendingApprovalResponse` becomes a `ReloadResult` with `undefined` counts, and `promote.ts:191` prints "promoted to PROD β€” live graph: undefined inbound…". | Exactly the deployments that enabled dual-control **for production**. Operator believes the graph swapped; it is sitting in a queue. | **No.** No test drives `promote.ts` at all. | P0 | -| Split-config cluster / split engine-shard estate | Convergence coordinates *when* nodes reload; each reloads its **own** dir (`config_convergence.py:16-20`). `ClusterStatus` exposes only the integer `config_version` (`api/models.py:737-748`), never a content fingerprint. Promote picks **one** engine-shard URL (`promoteTarget.ts:53-55`). | Node A runs the new graph, node B re-applies its old one; both write clean `config_reload` rows and report the same `config_version`. After a failover the wrong graph serves production. Engine shards over one unified store diverge per lane. | **No.** ADR 0041 lists this as unresolved β€” its open item to coordinate with the engine-shard owner. | P0 | -| Composite pipeline never exercised | Every unit exists; the assembly (author β†’ check β†’ non-prod β†’ traffic β†’ identical artifact to prod β†’ per-environment substitution β†’ rollback) has no test, no harness rig, and `FEATURE-COVERAGE-PLAN.md` excludes the IDE from every subsystem (`:946, :1216, :1379, :1420, :1471, :1521`). | The interfaces between the pieces β€” env substitution, artifact identity, pre-flight-vs-apply ordering, rollback β€” are where the real defects live. | **No.** | P0 | +| Restart-based publish has zero attribution | **CONFIRMED 2026-08-15 (BACKLOG #1100); both anchors had drifted and are re-pointed.** The startup path loads at **`api/app.py:5725`** (`loaded = load_config(config_dir)`) and **`:5730`** (`engine.add_registry(loaded)`) β€” the cited `:5455-5462` now lands on a function signature's keyword parameters, ~270 lines away. And **only `Engine.reload()` sets `loaded_config_fingerprint`**: it is initialised `None` at **`engine.py:450`** and assigned at **`:1610`/`:1615`, both inside `reload()`** (def `:1479`; the cited `:1488-1500` is the right method, wrong lines). Verified from the other side too β€” **`Engine.start()` (`engine.py:875`, 301 lines) mentions none of `loaded_config_fingerprint`, `config_fingerprint`, `add_registry` or `load_config`.** The fingerprint-bearing `config_reload` audit row is written only from the reload path (`api/app.py:554-573`, registered as a gate action at `:547`). Copy-files + NSSM restart (and the tray **Restart Service**) leave `loaded=false` and write no fingerprint-bearing audit row. | Every engine that was ever restarted rather than reloaded β€” i.e. the primary CI/CD path and every service restart. `/ui` badge blank, IDE can never show DRIFTED, no `who activated these bytes` row. | **No.** `GET /config/provenance` returns `loaded=false`, which reads as "nothing loaded", not "unattested". | P0 | +| Fingerprint is blind to the real `environments/` | `_FINGERPRINT_GLOBS` includes `environments/*.toml` **relative to the config dir** (`fingerprint.py:44`, globbed at `:59` as `base.glob(pattern)` where `base` is the directory passed in β€” i.e. `engine.last_reload_dir` / `engine.config_dir`), but every shipped/documented layout keeps `environments/` as a project-root **sibling** of `--config` (ADR 0017 layout; ADR 0050 anchoring). **CONFIRMED 2026-08-15 (BACKLOG #1100), measured on this checkout:** repo-root `environments/` holds **2** `.toml` files, and **`samples/config/environments/` does not exist** β€” so the glob resolves to nothing for the documented layout. **The redirect surface is exactly the peer endpoints:** those files declare `acme_adt_host`, `acme_adt_port`, `demo_oru_host`, `demo_oru_port`, `fhir_base_url`, `payer_rte_host`, `payer_rte_port` (14 keys each). **And ADR 0041 states the opposite as a REQUIREMENT, not a description** β€” `:100` *"It spans `connections.toml` **and** `environments/` so a transport/env-value redirect cannot change [the graph without changing the fingerprint]"*, and `:153` *"...or `environments/*.toml` β€” **THE SYSTEM SHALL** produce a different fingerprint."* **The shipped code does not meet that SHALL in the documented layout.** (ADR 0041 `:238` separately leaves "Fingerprint env-value scope" open, but that question is about hashing *resolved* values β€” it does not cover this, which is the declared scope failing to resolve at all.) | Repointing `acme_adt_host` in `environments/prod.toml` redirects PHI to a new peer with an **identical** fingerprint, no drift flag, and an audit row indistinguishable from a clean reload. | **No** β€” and ADR 0041 explicitly claims the opposite ("spans … `environments/`"). The existing test passes only because it writes `environments/` *inside* the tmp config dir. | P0 | +| "Promoted to PROD" can be false | A remote promote sends `config_dir: null` (`promote.ts:135-149`) so the engine reloads its **own** on-disk dir; `ReloadResult` (`api/models.py:351-360`) carries no fingerprint. If CI/CD never delivered the commit, PROD reloads stale bytes and the toast still says "promoted". | Silent whenever element counts are unchanged β€” i.e. almost every Handler-logic change. Highest-frequency wrong belief in the publish path. | **Partly β€” and the partial coverage is the trap. RE-MEASURED 2026-08-15 (BACKLOG #1100).** "Nothing recomputes or compares a fingerprint" is **FALSE**: `ConfigProvenance` (`api/app.py:4570-4590`) reads `engine.loaded_config_fingerprint`, **recomputes** `config_fingerprint_detail(target)` off the loop, and **compares** them β€” `drift = current.get("fingerprint") != fp` β€” returning a `drift` flag. ADR 0041 D1 also writes a fingerprint-bearing `config_reload` audit row on every reload (`:554-581`). **BUT IT COMPARES THE WRONG TWO THINGS FOR THIS RISK.** `drift` is *loaded-vs-disk-now*; the failure here is *disk-vs-the-commit-CI-was-meant-to-deliver*. **If CI/CD never delivered, disk still equals what was loaded, so `drift` is `False` and provenance reports clean.** So the row's concern survives its evidence completely β€” and is now *harder* to see, because a reviewer who finds `ConfigProvenance` will reasonably conclude it is covered. Still true: `ReloadResult` (**`api/models.py:371`**, cited `:351-360`) carries no fingerprint, so the promoting caller never receives one; and there is no `messagefoundry fingerprint` CLI β€” **verified by AST over every `add_parser` call: 37 distinct subcommands, none named `fingerprint`.** (An earlier revision of this note said the subcommands were "`graph`, `serve`, `validate`"; that came from a line-oriented grep that matched only 3 of 38 call sites and was wrong. The conclusion was unaffected β€” `fingerprint` is absent either way β€” but the evidence was not, which is the exact defect this sweep exists to fix.). **The fix is smaller than this row implied** β€” the fingerprint is already computed on every reload; it needs *surfacing in `ReloadResult`* and comparing against an expected value, not building | P0 | +| A held (dual-control) promote renders as success | `engineClient.postJson` resolves any 2xx and casts the body (`engineClient.ts:92-102`); a 202 `PendingApprovalResponse` becomes a `ReloadResult` with `undefined` counts, and `promote.ts:191` prints "promoted to PROD β€” live graph: undefined inbound…". | Exactly the deployments that enabled dual-control **for production**. Operator believes the graph swapped; it is sitting in a queue. | **No β€” CONFIRMED 2026-08-15 (BACKLOG #1100), and both anchors still land, which is rare in this sweep.** `engineClient.ts:94-96` is exactly `if (status >= 200 && status < 300) { resolve((text ? JSON.parse(text) : {}) as T)` β€” **any** 2xx resolves and blind-casts, so a 202 body becomes a `ReloadResult` of `undefined`s; `promote.ts:191-195` then prints the success toast with **no status check**. Measured: **41 IDE test files exist and NOT ONE drives `promote.ts`** (`promote-target.test.ts` tests `promoteTarget.ts`, a different module). **Stronger than the row states: `PendingApproval` appears nowhere in `ide/`, and there is no 202 handling anywhere in `ide/src`** β€” the client does not model the pending-approval response at all, so this is not a missed branch but an absent concept. **The fix is one status check** at the `postJson` call site, not new machinery | P0 | +| Split-config cluster / split engine-shard estate | Convergence coordinates *when* nodes reload; each reloads its **own** dir (`config_convergence.py:16-20`). `ClusterStatus` exposes only the integer `config_version` (**`api/models.py:786`**, cited `:737-748`), never a content fingerprint. **CONFIRMED 2026-08-15 (BACKLOG #1100):** `ClusterStatus` fields are exactly `node_id, clustered, is_leader, role, config_version`, and `ClusterNode` (`:800`) adds none either β€” neither mentions `fingerprint`. The convergence anchor lands exactly: `config_convergence.py:16-18` states the assumption in its own words β€” *"The version token coordinates when nodes reload; each node reloads its OWN config dir. Skewed config dirs would diverge."* **THIS IS THE THIRD FACET OF ONE GAP, NOT THREE GAPS.** ADR 0041 D1 computes a content fingerprint on every reload, and it is surfaced in **none** of the three places a consumer could compare it: not in `ReloadResult` (row 109), not in `ClusterStatus` here, and not at all on the restart path (row 107, which never sets it). **One change β€” surfacing the already-computed fingerprint on those responses β€” addresses all three rows**, which is worth knowing before any of them is scoped separately. Promote picks **one** engine-shard URL (`promoteTarget.ts:53-55`). | Node A runs the new graph, node B re-applies its old one; both write clean `config_reload` rows and report the same `config_version`. After a failover the wrong graph serves production. Engine shards over one unified store diverge per lane. | **No.** ADR 0041 lists this as unresolved β€” its open item to coordinate with the engine-shard owner. | P0 | +| Composite pipeline never exercised | Every unit exists; the assembly (author β†’ check β†’ non-prod β†’ traffic β†’ identical artifact to prod β†’ per-environment substitution β†’ rollback) has no test, no harness rig, and `FEATURE-COVERAGE-PLAN.md` excludes the IDE from every subsystem (`:946, :1216, :1379, :1420, :1471, :1521`). **CONFIRMED 2026-08-15 (BACKLOG #1100) β€” all six anchors LAND, which is unusual in this sweep, and every one is an explicit exclusion**, not a co-occurrence: *"Excludes the VS Code IDE (owner's lane)"* (`:946`), *"OUT of scope: the VS Code IDE extension"* (`:1216`), *"...and the VS Code IDE extension are out of scope"* (`:1379`), *"Excludes the VS Code IDE extension (owner's parallel work)"* (`:1420`, `:1521`), and *"...the VS Code IDE Steps view / element-centric connections graph ... are out of scope"* (`:1471`). **The reason matters for whoever picks this up: the exclusions are ATTRIBUTED β€” "owner's lane", "owner's parallel work" β€” so this is a deliberate scoping decision, not an oversight.** The composite pipeline is therefore unowned **by construction**: each half sits inside a scope that explicitly disclaims the other. That is why it has stayed open, and it means the fix requires crossing an ownership boundary rather than writing a missing test. | The interfaces between the pieces β€” env substitution, artifact identity, pre-flight-vs-apply ordering, rollback β€” are where the real defects live. | **No.** | P0 | | Post-quiesce failure leaves partial state | Rollback restores **only** `self.registry` + inbound intake (`wiring_runner.py:3151-3164`). By then the live-lookup executor has been rebuilt and the old one `aclose`d (`:3096-3104`), sandbox sessions dropped (`:3086-3095`), and `_reconcile_outbounds` may have partially applied (`:3149`). | Old Routers/Handlers run against **new-graph** `db_lookup`/`fhir_lookup` pools. If the new graph dropped a `DbLookup`, every old-graph lookup raises post-ACK β€” a silent per-message ERROR/dead-letter storm behind an audit row the operator reads as a clean no-op. | **No.** No test drives a failure at the inbound-bind or `_reconcile_outbounds` step. | P1 | | Same bytes, two different outcomes | `reload()` has **no** ADR-0031 per-inbound fault isolation (`wiring_runner.py:3108-3134`) while `start()` does (`:2236-2242` `_record_failed`). One unbindable inbound aborts the whole publish; the same bundle on a restart comes up with that connection isolated and everything else running. | An operator who "retries via a service restart" gets a partially-live graph they believed the engine had refused. | **No**, and undocumented. | P1 | | Duplicate ingest on publish | Quiescing closes established MLLP client connections (`transports/mllp.py:1352-1372`); the body is committed before the ACK, so nothing is lost, but a **not-yet-sent ACK** is β€” the sender retries and the engine ingests a duplicate. | Clinically material for non-idempotent downstreams (duplicate orders/results). The scope requirement is "must not lose **or duplicate**". | Partially β€” the consequence is documented in the code comment, but no test drives a reload under concurrent inbound traffic and no duplicate rate is published. | P1 | diff --git a/docs/testing/master-test-plan/09-engine-api.md b/docs/testing/master-test-plan/09-engine-api.md index 6ba181c3b..f6bc1f5ac 100644 --- a/docs/testing/master-test-plan/09-engine-api.md +++ b/docs/testing/master-test-plan/09-engine-api.md @@ -110,8 +110,8 @@ never run on, or a measurement that has never been taken. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| Wire contract unpinned for ~72 of 121 models | A field rename/retype/optionality flip on `UploadedFileInfo`, `SearchPresetList`, `LogTailPage`, `ResendResult`, `EditResendResult`, `ConnectionMetadata`, `AiPolicy`, `Health`, `StatsResponse`, `ApprovalList`, `MessageResponses`, `OutboundPayloads`… ships green | Every non-console consumer: `apiclient` (harness load/scenario/failover runners, service CLI), the IDE extension, the tray. Runtime `ApiError`/silent wrong render at the customer, not in CI | **No** β€” the seam snapshot pins field *names* for the 49 console-rendered DTOs only, and no types, optionality, path, method or status code | **P0** | -| No route ever runs against SQL Server or PostgreSQL | `create_app` appears in **zero** test that sets `MEFOR_TEST_SQLSERVER`/`MEFOR_TEST_POSTGRES` (verified by cross-grep). A backend divergence in `list_messages` + `count_messages` composition, `search_messages` truncation, `record_audit` with the ADR 0150 `client` column on `NVARCHAR(256)`, preset listing, dead-letter paging or audit filters surfaces first in production | Wrong/missing clinical rows on an operator read; a 500 on the console; an audit write that silently fails on the one backend a hospital runs | **No** | **P0** | +| Wire contract unpinned for most models. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100).** Re-counted by AST walk over all of `api/`: **117** `BaseModel`/`RootModel` classes (`models.py` 84, `auth_models.py` 32, `phi_gate.py` 1) against the "~121" written here β€” **inside the row's own hedge, so the figure is left as an approximation rather than re-pinned to a number that will drift again.** The load-bearing half is CONFIRMED: `tests/test_webconsole_seam_snapshot.py` (6 tests) is a **digest over rendered-DTO field names**, and contains **no** `status_code`, `methods`, `__annotations__`, `type_` or `optional` reference at all. **Two tokens in that file look like counter-evidence and are not** β€” `annotation` is `from __future__ import annotations` (`:14`) and `required` is the English word (`:76`); a grep for either would wrongly suggest types are pinned | A field rename/retype/optionality flip on `UploadedFileInfo`, `SearchPresetList`, `LogTailPage`, `ResendResult`, `EditResendResult`, `ConnectionMetadata`, `AiPolicy`, `Health`, `StatsResponse`, `ApprovalList`, `MessageResponses`, `OutboundPayloads`… ships green | Every non-console consumer: `apiclient` (harness load/scenario/failover runners, service CLI), the IDE extension, the tray. Runtime `ApiError`/silent wrong render at the customer, not in CI | **No** β€” the seam snapshot pins field *names* for the 49 console-rendered DTOs only, and no types, optionality, path, method or status code | **P0** | +| No route ever runs against SQL Server or PostgreSQL | `create_app` appears in **zero** test that sets `MEFOR_TEST_SQLSERVER`/`MEFOR_TEST_POSTGRES`. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100), and the measurement is stronger than "cross-grep" conveys: 46 tests set a server-DB env gate, 63 tests construct `create_app`, and the intersection is EXACTLY 0.** Both populations are large, so this is a real disjointness between two well-populated sets, not the empty-set artifact a bare "zero hits" can hide. **The two halves of the test estate do not meet:** everything exercising a route runs on SQLite, and everything running on a real server backend addresses the store directly. A backend divergence in `list_messages` + `count_messages` composition, `search_messages` truncation, `record_audit` with the ADR 0150 `client` column on `NVARCHAR(256)`, preset listing, dead-letter paging or audit filters surfaces first in production | Wrong/missing clinical rows on an operator read; a 500 on the console; an audit write that silently fails on the one backend a hospital runs | **No** | **P0** | | Framing/DoS defences never tested on a real socket | Every API test uses `httpx.ASGITransport`/`TestClient` with headers httpx synthesized (`tests/test_api.py:88-105`). The CL.TE 400, chunked-411, invalid-CL 400 and 1 MiB/`max_upload_bytes` 413 (`api/app.py:1279-1333`) are asserted against fabricated scopes; h11's own limits, keep-alive, real chunked decoding and slow-loris behaviour are unexercised | A pre-auth memory/connection DoS, or a desync between a front proxy and h11, that CI cannot see. ADR 0092 already records "a full uvicorn-on-a-real-socket handshake through the live serve bind" as a deferred residual | **No** | **P1** | | Deny-by-default proven structurally, not by execution | `test_security_doc_drift.py` reads the dependency graph. A route whose handler answers before its dependency is reached, a middleware short-circuit, a hand-registered Starlette route, or a closure shape that makes `_gate_of` return `None` while the allowlist is updated to match, all pass | Unauthenticated read of PHI or an admin write. Silent | Partially β€” spot checks only (`test_api_auth.py:97`, `:301`) | **P1** | | `/ws/stats` cap and `ws_count` hygiene untested behaviourally | The 64-cap is asserted only as a **constant** (`test_threat_model_doc_drift.py:573`). Nothing proves the 65th handshake is refused, nor that `state.ws_count` returns to 0 on every exit path (`api/app.py:4846`, `:4854`, `:4918-4919`). Note the cap is checked *before* `accept()` and incremented *after* β€” a TOCTOU window and a Starlette pre-accept-close that surfaces as a rejected handshake, not a 1013 frame | A leaked counter permanently refuses every future console socket estate-wide; the live monitor dies silently and the 5 s poll fallback masks it | **No** | **P1** | diff --git a/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md b/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md index 81785d364..471d3c9d7 100644 --- a/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md +++ b/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md @@ -87,10 +87,10 @@ This chapter covers the whole authentication/authorization core and, specificall | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| The real-directory acceptor path is mock-seam only. The `ldap3` `LDAPException` arms (`auth/ldap.py:252`, `:274`) and every SPNEGO acceptor line (`:298`, `:306`, `:358`, `:363`) are `# pragma: no cover`. No CI leg, no self-hosted runner, no containerised LDAP exists in `.github/workflows/` | A change to the `ldap3` / `pyspnego` call shape, or a dependency bump, ships green while AD login and Windows SSO are broken β€” or silently permissive | Every enterprise deployment's primary authentication path | **No.** `HANDOFF-AD-LAB-aws.md` states it outright: "The entire AD acceptor path is mock-seam only" | **P0** | -| Suspected live SPN defect. Both acceptor sites pass the *whole* SPN to `pyspnego`'s `service=`: `spnego.server(service=settings.kerberos_spn)` at `auth/ldap.py:300` and `:360`, while `config/settings.py:1846` documents `kerberos_spn` as `HTTP/host.example.com` β€” so the acceptor principal reads `HTTP/host.example.com/` | Kerberos SSO can never authenticate against a real KDC, yet `kerberos_acceptor_preflight` reports the acceptor healthy and `GET /auth/providers` advertises `kerberos=true` | Windows SSO estate-wide; a working-looking feature that always fails, every failure an audited generic reject | **No test can see it.** Both sites are `pragma: no cover` | **P0** | -| ADR 0142 has never met a real IdP. Status line: "Proposed β€” **code COMPLETE, awaiting lab validation**"; cells L6a, L9, L18 have not run; no AD-lab run record exists under `docs/testing/` | L18 is the real-IdP proof of the AC-11 username UPN-suffix binding β€” a review found the unchecked-suffix path was a **live** privilege-escalation route (a guest presenting `Administrator@attacker.example` resolving to the on-prem Domain Admin). L9 can invalidate the architecture: a passwordless/smartcard AD account cannot complete the password step-up, so every sensitive route permanently 403s | Federated sign-in as a whole; potentially a domain-admin takeover | **No** | **P0** | -| AD account states beyond `ACCOUNTDISABLE` are never consulted. `_find_user` (`auth/ldap.py:182-187`) rejects only `userAccountControl & 0x2`; `accountExpires`, `lockoutTime` and `UF_LOCKOUT` are not even requested in the attribute list (`:170-177`). `resolve_principal` (`:262`) β€” the password-free path used by Kerberos SSO, OIDC **and** the ADR-0079 reconciler, via `_probe_principal` at `service.py:1207` β€” therefore accepts them | An AD account that is expired or locked but not explicitly disabled keeps a live engine session (the reconciler probes it `PRESENT` every pass) and can federate in. Offboarding-by-expiry β€” the common HR pattern β€” does not propagate | Every offboarded-by-expiry operator retains console access | **No.** `tests/test_auth_hardening.py:415-455` pins the disabled bit only | **P0** | +| The real-directory acceptor path is mock-seam only. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100); anchors re-pointed and ONE clause corrected.** The `ldap3` `LDAPException` arms are at **`auth/ldap.py:267` and `:289`** (cited `:252`/`:274`) and the SPNEGO acceptor lines at **`:313`, `:321`, `:373`, `:378`** (cited `:298`/`:306`/`:358`/`:363`) β€” all six still `# pragma: no cover`, annotated *"needs real AD"* and *"requires a domain-joined server + keytab"*. **Zero** workflows mention `ldap`, and there is **no** OpenLDAP/Samba/389ds service container anywhere. **CORRECTION: "no self-hosted runner … exists" is FALSE.** One does β€” `selfhosted-win2025-sql.yml :: sqlserver-real` ("SQL Server 2025 suites, real hardware") β€” it simply mentions no `ldap`/`kerberos`/`spnego`/`ad_domain`/`directory`. **That is worth stating in BOTH directions:** as written, a reader can refute this P0 by pointing at that file and dismiss the whole row; and constructively, **the expensive prerequisite for an AD-lab leg is already built and proven** β€” a self-hosted Windows runner with a label scheme β€” so the remedy is "point an existing pattern at a directory", not "stand up runner infrastructure" | A change to the `ldap3` / `pyspnego` call shape, or a dependency bump, ships green while AD login and Windows SSO are broken β€” or silently permissive | Every enterprise deployment's primary authentication path | **No.** `HANDOFF-AD-LAB-aws.md` states it outright: "The entire AD acceptor path is mock-seam only" | **P0** | +| **CONFIRMED live SPN defect** (was "suspected"; verified against the library itself 2026-08-15, BACKLOG #1100). Both acceptor sites pass the *whole* SPN to `pyspnego`'s `service=`: `spnego.server(service=settings.kerberos_spn)` at **`auth/ldap.py:315` and `:375`** (anchors drifted from `:300`/`:360`), while **`config/settings.py:1905`** (drifted from `:1846`) documents `kerberos_spn` as `HTTP/host.example.com`. **The library's own signature settles it:** `spnego.server(hostname: str = 'unspecified', service: str = 'host', ...)`, documented as *"hostname: The principal part of the SPN"* and *"service: The service part of the SPN"* β€” pyspnego **builds** the SPN from the two. **AND IT IS WORSE THAN THIS ROW ASSUMED: `hostname` is never passed at either site**, so it takes its default and the acceptor principal is `HTTP/host.example.com/unspecified`, not `.../`. **Scope, stated honestly:** `kerberos_enabled` defaults to `False` and is marked experimental (`settings.py:1904`), so this **would** bite the first site that enables Kerberos SSO rather than affecting anything today | Kerberos SSO can never authenticate against a real KDC, yet `kerberos_acceptor_preflight` reports the acceptor healthy and `GET /auth/providers` advertises `kerberos=true` | Windows SSO estate-wide; a working-looking feature that always fails, every failure an audited generic reject | **No test can see it.** Both sites are `pragma: no cover` | **P0** | +| ADR 0142 has never met a real IdP. Status line: "Proposed β€” **code COMPLETE, awaiting lab validation**"; cells L6a, L9, L18 have not run; no AD-lab run record exists under `docs/testing/`. **PARTIALLY RE-VERIFIED 2026-08-15 (BACKLOG #1100), and the unverifiable part is named rather than assumed.** CONFIRMED: the ADR's status line still reads exactly that, and its inline note adds *"flips to Accepted only when runbook cells L6a, L9 and L18 report"*; and `docs/testing/` holds 26 files, all plan documents, with no run record among them. **NOT CHECKED, AND NOT CHECKABLE FROM THIS REPOSITORY:** whether L6a/L9/L18 have since run. The runbook is `docs/security/AD-FEDERATION-LAB-RUNBOOK.md`, and **`docs/security/` does not exist in this checkout at all** β€” it is the private companion repo. So this row's middle clause can only be graded by someone with the vault, and a reader here should treat it as *unknown*, not as *confirmed open*. The ADR status is the honest public proxy: it has not flipped | L18 is the real-IdP proof of the AC-11 username UPN-suffix binding β€” a review found the unchecked-suffix path was a **live** privilege-escalation route (a guest presenting `Administrator@attacker.example` resolving to the on-prem Domain Admin). L9 can invalidate the architecture: a passwordless/smartcard AD account cannot complete the password step-up, so every sensitive route permanently 403s | Federated sign-in as a whole; potentially a domain-admin takeover | **No** | **P0** | +| AD account states beyond `ACCOUNTDISABLE` are never consulted. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100) β€” every claim holds; only the anchors had drifted, and all four are re-pointed here.** `_find_user` (**`auth/ldap.py:174`**) rejects only `userAccountControl & 0x2` at **`:197-201`**; the attribute list at **`:185-192`** requests `userAccountControl` but **not** `accountExpires`, `lockoutTime` or `UF_LOCKOUT` β€” verified by reading the whole function via AST, which contains none of those three tokens. `resolve_principal` (**`:277`**) calls `_find_user` and adds no state check of its own, and it is the password-free path used by Kerberos SSO, OIDC **and** the ADR-0079 reconciler via `_probe_principal` (**`service.py:1244`**, calling it at `:1253`) β€” so all three accept such an account. `service.py:959` states the shared path in as many words: *"Roles come from `resolve_principal` β€” the same password-free LDAP lookup Kerberos uses"* | An AD account that is expired or locked but not explicitly disabled keeps a live engine session (the reconciler probes it `PRESENT` every pass) and can federate in. Offboarding-by-expiry β€” the common HR pattern β€” does not propagate | Every offboarded-by-expiry operator retains console access | **No.** `tests/test_auth_hardening.py:415-455` pins the disabled bit only | **P0** | | SQL Server never exercises the AD group β†’ per-connection scope map (`tests/test_sqlserver_store.py:423-424` covers `ad_group_role_map` only); PostgreSQL has one line (`test_postgres_store.py:769-770`); the store-side `allowed_channels` message filter is SQLite-only (`tests/test_channel_rbac.py:99`) | A backend-specific bug returns an empty set; `_sync_ad_channel_scope` (`auth/service.py:1147-1148`) then leaves the scope untouched **by design** and the AD operator silently keeps the all-connections default | Cross-connection PHI exposure on both production backends | **No** | **P1** | | No meta-guard that a request-scoped audit call threads `client=`. 66 `record_audit(` call sites in `messagefoundry/` (51 in `api/`); the chain hashes a missing client as the legacy 6-element payload (`tests/test_audit_integrity.py:391`), so an omission **verifies perfectly clean** | A new PHI-read or admin-write route that forgets `client=client_ip(request)` silently loses the ADR-0150 "from where" β€” the exact incident question the ADR was written to answer | Post-incident forensics on any new route | **No** | **P1** | | Two **verified** NULL-client audit paths on the unauthenticated attack surface: `audit_kerberos_reject` and `audit_oidc_reject` funnel into `_directory_reject_audit(actor, mech, reason)` (`auth/service.py:1046-1053`) which never takes or passes a `client`, so every `/ui/sso` and `/ui/oidc` route-level reject row has `client = NULL`. Separately, `auth.ad_scope_resynced` (`:1157-1161`) omits `client` while its sibling `auth.ad_roles_resynced` (`:1085-1090`) passes it at `:1089` | A Kerberos/OIDC probing campaign produces audit rows with no source address at all β€” the one place an address is most needed | Directory-SSO abuse is un-attributable | **No** | **P1** | diff --git a/docs/testing/master-test-plan/11-web-console.md b/docs/testing/master-test-plan/11-web-console.md index a936edd5b..e56a6b55e 100644 --- a/docs/testing/master-test-plan/11-web-console.md +++ b/docs/testing/master-test-plan/11-web-console.md @@ -94,9 +94,9 @@ that audit left open (FCP:UI-8, FCP:UI-23, FCP:UI-32) or that this pass newly fo | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| `app.js` is never executed by any test | A refactor preserving the grep-matched strings but breaking execution (a thrown init, a changed selector, a bad `await`) silently disables the 14.3.1 watchdog | Rendered PHI stays on an abandoned or terminated tab indefinitely; both WebAuthn ceremonies dead; live fragment stops updating | **No.** `test_ui_session_watchdog.py:396-486` reads `app.js` as text and compares `index()` positions. CodeQL SAST only. | **P0** | +| `app.js` is never executed by any test | A refactor preserving the grep-matched strings but breaking execution (a thrown init, a changed selector, a bad `await`) silently disables the 14.3.1 watchdog. **CONFIRMED 2026-08-15 (BACKLOG #1100), and the confirmation is sharper than the claim.** There is **no JavaScript-execution tooling anywhere in the repo** β€” `playwright`, `jsdom`, `selenium`, `puppeteer` and `node --test` return zero across `tests/`, `packaging/` and `.github/workflows/` (positive control: 173 test files match "console", so the probe was sound). **But EIGHT test files DO reference `app.js`** β€” `test_ui_csp_canary`, `test_ui_hardening`, `test_ui_logout_affordance`, `test_ui_mfa_gate`, `test_ui_session_watchdog`, `test_ui_static_allowlist`, `test_webui`, `test_webconsole_mount` β€” all asserting on the SERVED TEXT. **Those eight are exactly the "grep-matched strings" this row warns about, including one named for the watchdog itself**, so a coverage review that counts files touching `app.js` finds eight and concludes the surface is tested. The distinction that matters is asserted-as-text versus executed, and no count distinguishes them | Rendered PHI stays on an abandoned or terminated tab indefinitely; both WebAuthn ceremonies dead; live fragment stops updating | **No.** `test_ui_session_watchdog.py:396-486` reads `app.js` as text and compares `index()` positions. CodeQL SAST only. | **P0** | | Never served through real uvicorn + TLS + a browser cookie jar | Browser silently **rejects** the `__Host-`/`Secure` cookie, or the nonce CSP is not actually enforced | Login broken, or a script-injection control believed present is absent β€” first discovered by the first customer to expose `/ui` off-box | **No.** Everything is `httpx.ASGITransport` with a simulated scheme (`test_ui_hardening.py:34-37`). | **P0** | -| Dual-control operations initiated from `/ui` cannot be approved from `/ui` | Config reload / purge / bulk dead-letter replay held for approval renders "Approval id: X" and dead-ends (`pages/config.py:96-105`, `pages/connections.py:207-212`, `pages/messages.py:676-685`) | Under incident pressure the second approver must drop to the JSON API β€” or the org disables dual-control, defeating the control via its own UX | **No.** No `/ui/approvals` route in the golden table; no approvals field in the seam snapshot. Engine has `GET /approvals` + approve/reject at `api/app.py:2692-2721`. | **P0** | +| Dual-control operations initiated from `/ui` cannot be approved from `/ui` | Config reload / purge / bulk dead-letter replay held for approval renders "Approval id: X" and dead-ends (`pages/config.py:96-105`, `pages/connections.py:207-212`, `pages/messages.py:676-685`) β€” **ALL THREE RE-VERIFIED EXACT 2026-08-15 (BACKLOG #1100)**, each range ending precisely on its `el("p", text(f"Approval id: {approval_id}"), class_="muted")` line (`:105`, `:212`, `:685`), under headings "Reload held for approval", "Purge held for approval" and "Replay held for approval". **Three independently-written dead-ends with identical shape, which is what makes this a design gap rather than three oversights.** *Navigation note for the next reader, since it cost time here: the module tree is `messagefoundry_webconsole/pages/` at the REPO ROOT β€” `packaging/messagefoundry-webconsole/` holds only the packaging metadata and its tests* | Under incident pressure the second approver must drop to the JSON API β€” or the org disables dual-control, defeating the control via its own UX | **No.** No `/ui/approvals` route in the golden table; no approvals field in the seam snapshot. Engine has `GET /approvals` + approve/reject at `api/app.py:2692-2721`. | **P0** | | Three `app.js` features fetch `/ui` routes that do not exist | `/ui/logging/level` (`app.js:1259`), `/ui/logs/tail` (`:1308`), `/ui/messages/export` (`:1385-1386`, fetch at `:1412`) β€” no route, and no page builder emits `data-mf-log-level` / `data-mf-log-viewer` / `data-mf-msg-export` | ADR 0130/0131 and FEATURE-MAP present shipped capabilities that are unreachable; a PHI-egress feature is documented shipped with its audit/step-up path never exercised | **No.** Nothing in CI resolves a `fetch("/ui/...")` literal against the route table. | **P1** | | No pagination controls anywhere | Message log and dead-letters accept `limit`/`offset` (`routes/core.py:410-411`, `:519-520`) but render a text readout only (`pages/messages.py:113-117`, `:601-605`); audit is hard-coded `limit=200` with no offset or filter (`routes/audit.py:30,38`); search takes `limit` with no `offset` at all (`routes/search.py:66`) | An incident investigator sees the newest 50 messages / 200 audit rows and cannot page back **in the console at all** β€” a blind spot on the tamper-evident trail a HIPAA investigation depends on | **No.** FEATURE-COVERAGE-PLAN rows FCP:UI-8 and FCP:UI-23 are open. | **P1** | | No render-at-scale test, no performance budget | The connections fragment is re-rendered **server-side** and pushed on every `/ws/stats` tick to every connected operator (`api/app.py:4850-4852`, `pages.connections_fragment`) | A linear render over hundreds of Connections Γ— N operators contends with the same event loop that runs the pipeline β€” presents as a *delivery* problem | **No.** `docs/LOAD-TESTING.md` covers pipeline throughput only. | **P1** | diff --git a/docs/testing/master-test-plan/12-vs-code-ide-extension.md b/docs/testing/master-test-plan/12-vs-code-ide-extension.md index daa0734d0..69d9750a0 100644 --- a/docs/testing/master-test-plan/12-vs-code-ide-extension.md +++ b/docs/testing/master-test-plan/12-vs-code-ide-extension.md @@ -119,16 +119,16 @@ TypeScript and Python, and the **delivery vehicle**. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| `ide` leg is advisory only (`ci.yml:265-274`, not in `ci-gate` `needs:` at `ci.yml:1386-1392`) | A red IDE leg merges; a pure-Python PR renaming a CLI JSON field never runs the leg at all | The whole authoring surface stops working with a fully green CI | **No** β€” by design today | P0 | -| `auth.ts` has zero tests | `signIn`'s pre-prompt `assertTargetAllowed` (`auth.ts:110-115`) is dropped in a refactor; credentials + bearer go in clear to a non-loopback host | Credential exfil (the exact `ADR0035:SEC-005` hole ADR 0035 closed), on a shared clinical workstation | No | P0 | -| `signOut` revoke is unpinned (`auth.ts:73-86`; ADR 0110 AC-9 links to `auth.ts` itself, not a test) | Only the local token is dropped; the engine session lives until the 30-min idle cap | "Signed out" is a lie on a walked-away-from workstation | No | P0 | -| The `@messagefoundry` prompt is assembled inline in the handler (`chat.ts:117-146`), not behind a testable function | A future `parts.push()` of a dry-run row, a Test Bench payload or a message body ships green | **PHI egress to a third-party model provider** β€” the extension's headline safety claim | No β€” `chat.test.ts` covers only `capCode` + command wiring | P0 | +| `ide` leg is advisory only β€” **CONFIRMED 2026-08-15 (BACKLOG #1100), both anchors RE-POINTED.** The `ide` job is `ci.yml:818` (cited `:265-274`, now a `uv pip install` block) and `ci-gate` is `:2132` with its `needs:` at **`:2135-2141`** (cited `:1386-1392`, now a DR seed-gate step). Parsed rather than read, that `needs:` list is exactly **`changes`, `sqlserver-store`, `postgres-store`, `load-test`, `load-test-sqlserver`, `windows-service-smoke`** β€” **`ide` is absent**, so the leg is genuinely not rolled up and a red IDE leg does not block a merge. **The distinction is easy to get backwards and I had it backwards in my own notes**: `ci-gate` DOES roll up the server-DB and load legs, which are also not individually required checks, so "not individually required" and "cannot block a merge" are different properties and only `ide` has both | A red IDE leg merges; a pure-Python PR renaming a CLI JSON field never runs the leg at all | The whole authoring surface stops working with a fully green CI | **No** β€” by design today | P0 | +| ~~`auth.ts` has zero tests~~ **`auth.ts`'s SIGN-IN PATH is unexercised β€” corrected 2026-08-15 (BACKLOG #1100); "zero tests" is literally false, `ai-policy.test.ts:14` imports `peekToken` from it** | `signIn`'s pre-prompt `assertTargetAllowed` (`auth.ts:110-115`) is dropped in a refactor; credentials + bearer go in clear to a non-loopback host. **RE-VERIFIED 2026-08-15 (BACKLOG #1100) β€” anchor EXACT and the risk is REAL, but the row's framing understates the trap and its title is wrong.** `signIn` is at `auth.ts:108` and the guard is `const gate = assertTargetAllowed(url);` at **`:111`**, inside the cited range. **`assertTargetAllowed` is NOT untested β€” it is thoroughly tested**, in `ide/src/test/suite/engine-target.test.ts`, which imports it from `../../engineTarget` (not from `auth`) and asserts at least six cases including "non-loopback over plain http is REFUSED (the core SEC-005 case)" and "an unparseable URL fails safe". **WHAT IS UNTESTED IS THE INVOCATION, NOT THE GATE: no test exercises `signIn`, so nothing asserts that `:111` is ever reached. Deleting that one line leaves every one of those six tests GREEN** β€” the control is proven correct and its call site is unpinned, which is the same shape as the `planSave` row below and the reason "write tests for `auth.ts`" is the wrong instruction. **The test to write asserts that `signIn` consults the gate BEFORE prompting** β€” an ordering property, which no test of the gate alone can express | Credential exfil (the exact `ADR0035:SEC-005` hole ADR 0035 closed), on a shared clinical workstation | No | P0 | +| `signOut` revoke is unpinned (`auth.ts:73-86`; ADR 0110 AC-9 links to `auth.ts` itself, not a test) β€” **CONFIRMED 2026-08-15 (BACKLOG #1100), anchor EXACT, and the code says it in its own comments**: `:76` is `return true; // nothing to revoke` and `:82` is `return false; // could not reach the engine to revoke β€” the session lives until it times out`. **Both non-revoking paths return a value the caller cannot distinguish from success**, which is what makes the missing test load-bearing rather than merely absent | Only the local token is dropped; the engine session lives until the 30-min idle cap | "Signed out" is a lie on a walked-away-from workstation | No | P0 | +| The `@messagefoundry` prompt is assembled inline in the handler (`chat.ts:117-146`), not behind a testable function β€” **CONFIRMED 2026-08-15 (BACKLOG #1100), anchor EXACT, and the shape is precisely the hazard**: the cited span is four bare `parts.push(...)` calls in the handler body β€” `summary`, `"Active editor code:\n\`\`\`python\n" + capped.text`, `"Task: " + task`, `"User request: " + request.prompt`. **Adding a fifth is a one-line edit with no seam to assert against**, which is why this is a PHI risk and not a style complaint: CLAUDE.md Β§9 confines the MVP assistant to `code_only`, and that confinement currently exists only as the absence of a `parts.push` of a message body | A future `parts.push()` of a dry-run row, a Test Bench payload or a message body ships green | **PHI egress to a third-party model provider** β€” the extension's headline safety claim | No β€” `chat.test.ts` covers only `capCode` + command wiring | P0 | | Test Bench collections persist message bodies (`testBench.ts:23` `COLLECTIONS_KEY`, `:136-142`); ADR 0121 AC-4 is explicitly *design/review-enforced* | One word changed from `workspaceState` to `globalState` pushes saved case bodies into VS Code Settings Sync | PHI off-box to the user's cloud profile | No | P0 | | Transient PHI temp dir (`testBench.ts:233-236` `mkdtempSync("mefor-testbench-")`, `:271-279` `rmSync` in `finally`) | A cleanup regression leaves plaintext message bodies in `os.tmpdir()` | PHI at rest outside the store, unaudited | No | P0 | -| Nothing asserts `run()`/`runWithStdin()`/`runJson()` actually short-circuit (`cli.ts:147-152`, `:183-189`) or that activation execs nothing (`extension.ts:433-461`) | One dropped `isExecGated()` call β‡’ opening a cloned "starter config" repo runs a trojaned `.venv` interpreter on first open | Arbitrary code execution (CWE-426) from merely opening a folder | No β€” only the pure `resolvePythonPath` is pinned | P0 | -| The packaged VSIX is never built, installed or smoke-tested; every test runs from `extensionDevelopmentPath` (`runTest.ts:13`) | A `.vscodeignore` or asset regression drops `media/hl7schema.json`, `media/hl7structures.json`, `media/stepsWebview.js`, `snippets/`, `media/walkthrough/` or the copied `LICENSE` | An extension that installs and then silently has no autocomplete / no field picker / no walkthrough | No | P0 | -| The IDE↔CLI JSON contract is hand-mirrored in TypeScript; only the two HL7 media artifacts are pinned (`tests/test_ide_artifacts.py`) | A field rename in `graph --json`, `connection schema --json`, `dryrun --json`/`--trace json`, `codeset list\|show`, `alert list`, `security show` or `ai-policy --json` breaks the consumer | Tree, form, Test Bench, code-set grid, alert/security editors β€” with both suites green | No | P0 | -| The webview β†’ CLI β†’ file β†’ tree write path is never exercised end to end | `connection upsert` is a **FULL REPLACE** (`connectionForm.ts:12-17`); a key the *webview* fails to post is a key the save DELETES, even though `planSave` is unit-tested | Silent loss of `schedule` / `shard` (the engine-shard partition tag, `connections_edit.py:86-88`) / allowlist / retry keys from a live `connections.toml` | No β€” `planSave` is tested only against synthetic objects | P0 | +| Nothing asserts `run()`/`runWithStdin()`/`runJson()` actually short-circuit (`cli.ts:147-152`, `:183-189` β€” **RE-VERIFIED 2026-08-15 (BACKLOG #1100): both cited RANGES are correct**, the guards being `if (isExecGated()) {` at `:152` and `:186`, with the function defined at `:67`) β€” **and the "nothing asserts" half is CONFIRMED WITH A POSITIVE CONTROL: `isExecGated` is referenced by NO file under the IDE test directories, while 8 files there mention `exec`, so the directories exist and the search works. The zero is a fact about coverage, not about the probe** or that activation execs nothing (`extension.ts:433-461`) | One dropped `isExecGated()` call β‡’ opening a cloned "starter config" repo runs a trojaned `.venv` interpreter on first open | Arbitrary code execution (CWE-426) from merely opening a folder | No β€” only the pure `resolvePythonPath` is pinned | P0 | +| The packaged VSIX is never built, installed or smoke-tested; every test runs from `extensionDevelopmentPath` (`runTest.ts:13` β€” **CONFIRMED EXACT 2026-08-15 (BACKLOG #1100): that line is `const extensionDevelopmentPath = path.resolve(__dirname, "../../");`**, so the suite loads the source tree and never the packaged artefact, and no `.vscodeignore` or asset regression is observable from it) | A `.vscodeignore` or asset regression drops `media/hl7schema.json`, `media/hl7structures.json`, `media/stepsWebview.js`, `snippets/`, `media/walkthrough/` or the copied `LICENSE` | An extension that installs and then silently has no autocomplete / no field picker / no walkthrough | No | P0 | +| The IDE↔CLI JSON contract is hand-mirrored in TypeScript; only the two HL7 media artifacts are pinned (`tests/test_ide_artifacts.py`) β€” **CONFIRMED EXACT 2026-08-15 (BACKLOG #1100): that file contains exactly TWO tests, `test_hl7structures_artifact_in_sync` and `test_hl7schema_artifact_in_sync`, so both the count and the subject are right.** Worth flagging as a clean enumeration, because several counts elsewhere in this plan were not | A field rename in `graph --json`, `connection schema --json`, `dryrun --json`/`--trace json`, `codeset list\|show`, `alert list`, `security show` or `ai-policy --json` breaks the consumer | Tree, form, Test Bench, code-set grid, alert/security editors β€” with both suites green | No | P0 | +| The webview β†’ CLI β†’ file β†’ tree write path is never exercised end to end | `connection upsert` is a **FULL REPLACE** (`connectionForm.ts:12-17`); a key the *webview* fails to post is a key the save DELETES, even though `planSave` is unit-tested. **CONFIRMED VERBATIM 2026-08-15 (BACKLOG #1100) β€” the module states BOTH halves itself**, in one comment: invariant 1 opens "NOTHING ON THE RECORD IS EVER DROPPED" and then explains why that has to be engineered β€” "`connection upsert` is a FULL REPLACE of the table, so a key the form fails to render is a key the form DELETES." **The invariant is the MITIGATION; the full-replace is the HAZARD that makes it load-bearing, and this row's point is that the tested part is the mitigation (`planSave`) while the untested part is what the mitigation depends on β€” the webview actually posting every key.** *Caution for a re-checker, because it nearly caught me: reading only the cited first line makes the module look like it CONTRADICTS this row. It does not β€” the qualifying sentence is two lines below, and the invariant is a claim about what the module achieves, not about what `upsert` does* | Silent loss of `schedule` / `shard` (the engine-shard partition tag, `connections_edit.py:86-88`) / allowlist / retry keys from a live `connections.toml` | No β€” `planSave` is tested only against synthetic objects | P0 | | No plan artifact owns the IDE | Every gap above is unassigned | The largest single UI surface has no accountable owner | N/A | P0 | | `runTest.ts:15` passes no `version` to `runTests()` while `engines.vscode` is `^1.95.0` | A runtime-only newer API breaks every user on the declared minimum | Install-quality gate for the pending Marketplace publish | No | P1 | | `newRoute.ts` generates the Python the engine executes; `q()` (`:76`) escapes only `\` and `"` | A newline or control char in a name emits a syntactically broken `.py`; a wrong `ibSpec`/`obSpec`/router binding emits a valid-but-miswired graph | Mis-routed clinical messages from an authored artifact the user then promotes | No β€” zero tests | P1 | diff --git a/docs/testing/master-test-plan/13-steps-editor.md b/docs/testing/master-test-plan/13-steps-editor.md index 2e52b2d54..6ebd534f4 100644 --- a/docs/testing/master-test-plan/13-steps-editor.md +++ b/docs/testing/master-test-plan/13-steps-editor.md @@ -62,9 +62,9 @@ | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| Row-contract drift Pythonβ†’TypeScript | `lens.py` renames a field or changes the partition; the IDE keeps parsing the *frozen* fixture snapshot, so every `ide` test stays green while the live view mis-projects rows. A mis-projected line range means a byte-stable edit splices into the **wrong statement** | Silent wrong transform β†’ wrong clinical data on every message through that Handler | **No.** Verified on this checkout: all 7 committed fixtures are stale β€” `suite` is missing from all of them, `label`/`operand` from `adt.json` and `IB_RADIOLOGY_SR.json`. `suite` is load-bearing for drag/drop scoping (`stepsModel.ts:29-32`) | P0 | -| Webview mirror divergence | `ide/media/stepsWebview.js` re-implements 10 pure model functions (`blockExtent:68`, `captureBlock:83`, `buildDropSlots:100`, `walkMove:126`, `clipLabel:309` β€” the `blockLabel` mirror, `canDrop:384`, `scopeLabel:397`, `resolveDrop:404`, `barAnchor:433`, menu enablement ~`:590`) and, until BACKLOG #233, **was** explicitly not unit-tested (`steps-edit.test.ts` said so in as many words: "verified manually"). A diverged mirror computes a wrong move/drop target; the engine then applies it byte-stably and it re-parses clean | Moving a `msg.set` out of an `if` guard, or into the wrong branch, is a semantic change the byte-stability gates structurally cannot see | **Yes, as of BACKLOG #233** β€” `ide/src/test/suite/steps-mirror.test.ts` loads the webview script under jsdom and asserts every mirror against its model counterpart on every `ide` leg. Both of this row's original grounds are now spent: `buildDropSlots` **is** exported from `stepsModel.ts`, and the drop/clipboard mirrors are **no longer** "verified manually" (`steps-edit.test.ts` now points at the parity suite; what genuinely stays manual is the menu's positioning/dismissal/keyboard wiring, STEPS-76). The suite found exactly **one** live divergence β€” the model's `canDropRow` accepted a read-only `code` row as a drop target while the webview refused it, contradicting the model's own stated contract β€” and it is now closed. ADR 0108's "model and mirror in agreement" acceptance line is a gate rather than a claim | P0 | -| Engine change never triggers the IDE tests | The `ide` job's PR path filter is `^(ide/\|\.github/workflows/ci\.yml)` (`ci.yml:448`). A PR touching `messagefoundry/lens.py` β€” the exact contract the Steps view consumes β€” does not run it at all. And `ci-gate` deliberately does **not** `needs: ide` (`ci.yml:265`), so even a red `ide` leg cannot block a merge | The whole analyst-facing surface can regress green. (It does re-run on push-to-main, `ci.yml:410` β€” after the merge, when it can no longer block anything) | **No** | P0 | +| Row-contract drift Pythonβ†’TypeScript | `lens.py` renames a field or changes the partition; the IDE keeps parsing the *frozen* fixture snapshot, so every `ide` test stays green while the live view mis-projects rows. A mis-projected line range means a byte-stable edit splices into the **wrong statement** | Silent wrong transform β†’ wrong clinical data on every message through that Handler | **No.** Verified on this checkout: all 7 committed fixtures are stale β€” `suite` is missing from all of them, `label`/`operand` from `adt.json` and `IB_RADIOLOGY_SR.json`. `suite` is load-bearing for drag/drop scoping (**`stepsModel.ts:42-44`, RE-POINTED 2026-08-15 (BACKLOG #1100) β€” the cited `:29-32` is now a row-KIND union ending in `"route"`**; the real declaration is `suite?: string` at `:44` under the comment at `:42` "greys an up/down at a suite edge (a reorder never crosses into/out of an if/for body)", and it is consumed at `:552-553`). **THE STALENESS IS WIDER THAN THIS ROW STATES β€” measured, not read.** All **7** committed fixtures under `ide/src/test/fixtures/lens/` lack `suite`, and they also **all** lack `label` and `operand`, not merely `adt.json` and `IB_RADIOLOGY_SR.json`. `adt.json`'s complete key set is 13 keys β€” `control, def_line, handler, handlers, kind, line_end, line_start, module, nesting, outbounds, recognized, rows, test_src` β€” with none of the three present at any depth, while `messagefoundry/lens.py` emits all three. *(Positive control: the same walker that reports the three absent returns those 13 keys, so the absence is a fact about the fixtures and not about the probe.)* | P0 | +| Webview mirror divergence | `ide/media/stepsWebview.js` re-implements 10 pure model functions (`blockExtent:68`, `captureBlock:83`, `buildDropSlots:101` (was `:100`), `walkMove:127` (was `:126`), `clipLabel:326` (was `:309`) β€” the `blockLabel` mirror, `canDrop:401` (was `:384`), `scopeLabel:415` (was `:397`), `resolveDrop:422` (was `:404`), `barAnchor:451` (was `:433`), menu enablement ~`:612` (was ~`:590`) β€” **ALL RE-POINTED 2026-08-15 (BACKLOG #1100) against the 33 function declarations in that file; `blockExtent:68` and `captureBlock:83` were already exact.** The drift is **monotonic in line number** (+0, +0, +1, +1, +17, +17, +18, +18, +18, ~+22), which is the signature of insertions above rather than of edits within, so **a single uniform offset would NOT have re-pointed these correctly** and each had to be resolved individually) and, until BACKLOG #233, **was** explicitly not unit-tested (`steps-edit.test.ts` said so in as many words: "verified manually"). A diverged mirror computes a wrong move/drop target; the engine then applies it byte-stably and it re-parses clean | Moving a `msg.set` out of an `if` guard, or into the wrong branch, is a semantic change the byte-stability gates structurally cannot see | **Yes, as of BACKLOG #233** β€” `ide/src/test/suite/steps-mirror.test.ts` loads the webview script under jsdom and asserts every mirror against its model counterpart on every `ide` leg. Both of this row's original grounds are now spent: `buildDropSlots` **is** exported from `stepsModel.ts`, and the drop/clipboard mirrors are **no longer** "verified manually" (`steps-edit.test.ts` now points at the parity suite; what genuinely stays manual is the menu's positioning/dismissal/keyboard wiring, STEPS-76). The suite found exactly **one** live divergence β€” the model's `canDropRow` accepted a read-only `code` row as a drop target while the webview refused it, contradicting the model's own stated contract β€” and it is now closed. ADR 0108's "model and mirror in agreement" acceptance line is a gate rather than a claim | P0 | +| Engine change never triggers the IDE tests | The `ide` job's PR path filter is `^(ide/\|\.github/workflows/ci\.yml)` (**`ci.yml:1007`, RE-POINTED 2026-08-15 (BACKLOG #1100)** β€” the filter matches paths starting `ide/` or the workflow file itself; the cited `:448` is now an unrelated benchmark comment). A PR touching `messagefoundry/lens.py` β€” the exact contract the Steps view consumes β€” does not run it at all. And `ci-gate` deliberately does **not** `needs: ide` (**RE-POINTED: the `ide` job is `ci.yml:818` and `ci-gate`'s `needs:` list is `:2135-2141`, containing `changes`, `sqlserver-store`, `postgres-store`, `load-test`, `load-test-sqlserver`, `windows-service-smoke` and NOT `ide`; the cited `:265` is a `uv pip install` comment. CONFIRMED, and `ci.yml:820` states it outright in its own words β€” "`ide` is NOT a required check and ci-gate does NOT `needs: ide`"**), so even a red `ide` leg cannot block a merge | The whole analyst-facing surface can regress green. (It does re-run on push-to-main, `ci.yml:410` β€” after the merge, when it can no longer block anything) | **No** | P0 | | Zero action rows in the tested corpus | Census on this checkout: `lens parse` over all 12 `samples/config` handlers yields **12 code rows, 12 send rows, 4 control rows and 0 action/lookup/diagnostic rows**. ADR 0076 gate 1's named corpus therefore proves nothing about the action-row, param-edit or Add-palette surface β€” the part an analyst actually uses. The IDE fixtures inherit the same hole | Every projection/edit path for the editable surface is only ever tested against ad-hoc inline strings written by whoever wrote the test β€” no shared, reviewed adversarial corpus | Partially (inline strings in `test_lens_native/palette/fanout`) β€” but no corpus-level partition/byte-stability/ruff/`check` sweep over action rows | P1 | | `{"expr": …}` splice writes arbitrary, unnormalized Python | `_validated_expr` (`lens.py:1869`) checks only "parses as one expression" and "is exactly one call argument". **Verified on this checkout:** `set_field(msg, "PID-3", __import__("os").popen("whoami").read())` is accepted and written into the Handler body. **Also newly verified:** an expr is spliced **verbatim**, so `foo( 1,2 )` produces output that **fails `ruff format --check`** and **fails `ruff check --select F` (F821 undefined name)** β€” a direct breach of ADR 0076 gate 3 ("emitted code is first-class") that no existing test covers | Handlers execute in the engine process. ADR 0144's lint runs only inside `messagefoundry check`, never on the rewrite path, and the Steps view gives no in-editor signal β€” while pitching a form field at an analyst who does not know Python. It also silently breaks the purity invariant the at-least-once contract depends on | **No** on all three counts | P1 | | False completeness: helper-body writes invisible and unmarked | ADR 0089 Phase D (helper descent) is unbuilt β€” `_msh(msg)` renders as an opaque `code` row. ADR 0104 AC-10's **"unmodeled code present"** marker does not exist: grep for `unmodeled` across `ide/`, `messagefoundry/`, `tests/` returns only the ADR and `docs/research/message-model-eval.md`; the named test is absent | An analyst edits a PID mapping in the Steps view, sees no other write to that field, saves β€” and a helper's later write silently overrides it. Wrong clinical data, no failing test, no operator signal | **No** | P1 | diff --git a/docs/testing/master-test-plan/14-tray-service-and-distribution.md b/docs/testing/master-test-plan/14-tray-service-and-distribution.md index d6ac918d3..7a0d65f73 100644 --- a/docs/testing/master-test-plan/14-tray-service-and-distribution.md +++ b/docs/testing/master-test-plan/14-tray-service-and-distribution.md @@ -103,16 +103,16 @@ icon state rendering and light/dark iconset (`theme.py`, `iconset.py`, 18 `.ico` | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| `install-service.ps1` privilege/ACL regression merges green | A `.ps1` edit drops the virtual-account default (`:472-477`), mis-orders the S4 ACL grants (`:516-548`), breaks `Resolve-Nssm`, or removes `AppExit Restart` | Every new install on every customer box; discovered at go-live | **No** β€” only `windows-service-smoke`, `(schedule \|\| workflow_dispatch) && github.repository == 'MEFORORG/MessageFoundry'` (`ci.yml:1087`); never on a PR, never in a fork/mirror. Static regexes only | **P0** | +| `install-service.ps1` privilege/ACL regression merges green | A `.ps1` edit drops the virtual-account default (`:472-477`), mis-orders the S4 ACL grants (`:516-548`), breaks `Resolve-Nssm`, or removes `AppExit Restart`. **ALL ANCHORS RE-VERIFIED EXACT 2026-08-15 (BACKLOG #1100)** against `install-service.ps1` (572 lines): `:472` is the `if (-not $ServiceAccount -and -not $AllowLocalSystem)` default and `:477` closes it; `:516` opens the S4 ACL block whose own comment reads "applied AFTER the service exists + ObjectName is set (S4 order)"; `Resolve-Nssm` is at `:81`; and for the row below, `:450` is `Start SERVICE_AUTO_START` with `:463-464` `AppExit Default Restart` + `AppThrottle 5000`. **Worth recording that this chapter did NOT drift** β€” every `wiring_runner.py` anchor checked in chapters 02-10 had moved, so drift tracks the CHURN OF THE CITED FILE, not the age of the plan, and a re-sweep should re-point by file rather than uniformly | Every new install on every customer box; discovered at go-live | **No** β€” only `windows-service-smoke`, `(schedule \|\| workflow_dispatch) && github.repository == 'MEFORORG/MessageFoundry'` (`ci.yml:1087`); never on a PR, never in a fork/mirror. Static regexes only | **P0** | | Silent regression to LocalSystem, or a world-readable log dir | Nothing queries `nssm get MessageFoundry ObjectName`, `Start`, or `icacls` on DataDir/logs/config after install | NSSM captures engine stdout to `C:\ProgramData\MessageFoundry\logs` β€” a PHI sink (`install-service.ps1:119-137` exists precisely for this). A lost `/inheritance:r` re-exposes it; a lost ObjectName widens compromise blast radius | **No** β€” `FCP:DEPLOY-8` / `FCP:DEPLOY-11` rated high-risk, partial/none in `docs/testing/FEATURE-COVERAGE-PLAN.md` Β§24 (rows at `:332`, `:334`) | **P0** | | Engine does not come back after host reboot or engine crash | `Start SERVICE_AUTO_START` (`:450`) or `AppExit Default Restart` + `AppThrottle` (`:463-464`) silently ineffective | A silent clinical outage: feeds stop, senders queue or drop, nothing alerts because the engine simply is not running | **No** β€” static regex (`test_service_control.py:118`) + a MANUAL row (`ACC:G1`, `harness/acceptance/matrix.py:430-437`; `W25:S2.5`) | **P0** | | The tray's shipped launch path is unverified end to end | A hatchling change drops `tray/assets`, or the `gui-scripts` entry (`pyproject.toml:202-203`) typos | `docs/TRAY.md:22-37` promises "present on every `pip install messagefoundry`". A tray that starts with no icon, or does not start, on every fresh install | **No** β€” `release.yml:127-180` installs the wheel on **Linux** and asserts only import/version/`py.typed`; `test_tray_iconset.py` reads `ASSETS_DIR` from `__file__` (the repo); `windows-service-smoke` installs `-e .` | **P1** | | ADR 0113 Β§1's import contract unenforced | A future edit imports `messagefoundry.config.settings` into the tray to "read TLS properly" | Pulls the engine + pydantic into an unprivileged, always-running process; breaches the boundary the whole ADR rests on | **No** β€” `tests/test_dependency_boundaries.py:14` lists only `pipeline/transports/parsing/store/config`; nothing scans `messagefoundry/tray/` | **P1** | | `tray.log` destroyed by httpx per-tick noise | `_setup_logging` attaches the handler to the **root** logger at INFO (`__main__.py:29-31`); nothing raises the `httpx` logger, which logs one INFO line per request β€” two requests every 5 s | ~34k lines/day churn a 1 MB Γ— 3 rotation, so transitions, user actions and elevation outcomes rotate out within hours β€” destroying the only forensic record of who stopped a clinical interface. Contradicts `docs/TRAY.md:162` ("state transitions, never per-tick") | **No** | **P1** | | `poll_seconds` is inert | Parsed and clamped (`config.py:257-259`), documented (`config.py:359-360`, `docs/TRAY.md:89`), but `StatusPoller._run` (`poller.py:169`) always calls `next_poll_seconds()`, which returns the hardcoded `POLL_BASE_S = 5.0` (`state.py:24,166-172`) | An operator throttling a busy interface box sees no change and gets no signal the setting was ignored | **No** β€” tests assert only that the value lands in `TrayConfig` (`test_tray_config.py:126,236`) | **P1** | -| The tray's Win32 layer is behaviourally untested | `TrayApp` (154 lines) and the pump (`winshell.py:299-451`) decide which icon paints, whether a toast fires, whether Exit tears down, whether the icon is removed on `WM_DESTROY`/`WM_ENDSESSION` | A ghost icon after logoff, a frozen icon on a state change, an action routed to the wrong handler β€” the exact failure class ADR 0113 named as the design's main risk | **No** β€” `test_tray_shell.py` only asserts imports + non-zero struct sizes | **P0** β€” the design's *named* main risk cannot be priced below the install rows it sits beside; closed by TRAY-19, TRAY-20 and TRAY-22 | +| The tray's Win32 layer is behaviourally untested | `TrayApp` (**`tray/app.py:42-154`, i.e. 113 lines β€” CORRECTED 2026-08-15 (BACKLOG #1100); the row said "154 lines", which is the END LINE NUMBER relabelled as a LENGTH, overstating the surface by 36%**) and the pump (`winshell.py:299-451` β€” **anchor VERIFIED EXACT: that range is precisely the ten dispatch functions `_add_icon` through `_message_loop`, which ends at :451**) decide which icon paints, whether a toast fires, whether Exit tears down, whether the icon is removed on `WM_DESTROY`/`WM_ENDSESSION` | A ghost icon after logoff, a frozen icon on a state change, an action routed to the wrong handler β€” the exact failure class ADR 0113 named as the design's main risk | ~~**No** β€” `test_tray_shell.py` only asserts imports + non-zero struct sizes~~ **PARTLY. CORRECTED 2026-08-15 (BACKLOG #1100) β€” the coverage claim was FALSE and I carried it forward once before checking it.** `test_tray_shell.py` has **six** test functions, of which only `test_tray_windows_modules_import_and_build_structs` is the imports-and-structs test this cell describes. The other five are behavioural, and **two of them cover this row's own stated risk**: `test_disabled_action_is_not_dispatchable` (an action routed to the wrong handler) and `test_assign_command_ids_maps_actionable_items_only` (menu command-id mapping, i.e. TRAY-20's subject). Also present: `test_launcher_command_quotes_absolute_pythonw`, `test_pythonw_executable_falls_back_to_given_path`, `test_single_instance_second_acquire_detects_running`. **The gap is real but MUCH narrower than "behaviourally untested"**: what remains uncovered is the Win32 message-pump surface β€” which icon paints, whether a toast fires, whether the icon is removed on `WM_DESTROY`/`WM_ENDSESSION` β€” none of which any of the six exercises. Re-scope TRAY-19/20/22 to the pump, not to action routing | **P0** β€” the design's *named* main risk cannot be priced below the install rows it sits beside; closed by TRAY-19, TRAY-20 and TRAY-22 | | Autostart writes are untested | `set_autostart` / `is_autostart_enabled` (`autostart.py:34-64`) write and read the real `HKCU\…\Run` value; only the pure `launcher_command` helper is covered | A stale or wrong-interpreter command silently breaks Start-at-Login (no tray after reboot) or strands a value pointing at a deleted venv; the menu checkbox reads the same untested getter | **No** | **P1** | -| Hardened box + non-admin operator = a lying tray | Under `-LockConfigDir`, a standard user cannot read the engine's settings TOML, so `service_toml_uses_tls` sees nothing, `build_engine_url` yields `http://`, the https probe fails, and a healthy engine renders `WEDGED`/`STOPPED` | The 2026-07-22 amendment's bug, resurfacing for the least-privileged user on the most-hardened box, with no on-screen path to the fix β€” a healthy clinical interface rendered as down, on the exact configuration the security guidance prescribes | **No** β€” the fail-soft-to-http behaviour is asserted as *correct* (`test_tray_config.py:274-287`); the operational consequence is untested and undocumented | **P0** β€” a tray that lies about a healthy engine is worse than no tray; closed by TRAY-27 (+ TRAY-43 on the box) | +| Hardened box + non-admin operator = a lying tray | Under `-LockConfigDir`, a standard user cannot read the engine's settings TOML, so `service_toml_uses_tls` sees nothing, `build_engine_url` yields `http://`, the https probe fails, and a healthy engine renders `WEDGED`/`STOPPED`. **CONFIRMED END-TO-END IN CODE 2026-08-15 (BACKLOG #1100)**, every hop: `_read_toml` (`tray/config.py:279-286`) is documented "``None`` on any read/parse failure. Never raises" and catches `(OSError, ValueError)` at `:284` β€” **`PermissionError` IS an `OSError`**, so the lock yields `None`; `service_toml_uses_tls` then hits `if not service_toml: return False` (`:212-213`); `build_engine_url` renders `f"{'https' if tls else 'http'}://..."` (`:188`) as **`http://`**. **The root defect is a THREE-STATE COLLAPSE: parsed-and-no-TLS, file-absent, and file-present-but-UNREADABLE all become the same `None`, then the same `False`. The distinguishing information is destroyed at `:284` and cannot be recovered by any caller**, so no downstream fix is possible and the test must drive the unreadable case specifically. Instructively, the fleet's own `seat-tick.ps1` handles the identical distinction correctly and says so in comment β€” a roster that could not be read is not evidence that nobody is live β€” carrying a separate roster-blind flag rather than folding it into the empty case | The 2026-07-22 amendment's bug, resurfacing for the least-privileged user on the most-hardened box, with no on-screen path to the fix β€” a healthy clinical interface rendered as down, on the exact configuration the security guidance prescribes | **No** β€” the fail-soft-to-http behaviour is asserted as *correct* (`test_tray_config.py:274-287`); the operational consequence is untested and undocumented | **P0** β€” a tray that lies about a healthy engine is worse than no tray; closed by TRAY-27 (+ TRAY-43 on the box) | | Windows-only lock resolution failure | `pip install --require-hashes -r requirements.lock` β€” prescribed for Windows production at `docs/SERVICE.md:26-33` β€” runs only on ubuntu (`security.yml:71-77`) | A platform-marker gap, an sdist-only transitive, or a missing `win_amd64` wheel breaks the documented production install on the primary supported platform | **No** | **P1** | | Uninstall leaves user-scope tray artifacts | `uninstall-service.ps1` touches only the service. Nothing removes the `HKCU\…\Run` value `MessageFoundryTray` (`autostart.py:19`), `%LOCALAPPDATA%\MessageFoundry` (tray.toml + tray.log), or `Scripts\MessageFoundryTray.exe` + the ~7 MB staged DLLs (`branding.py:280-290`) | After `pip uninstall messagefoundry` the Run key fires at every login and fails silently; a branded exe lingers indefinitely | **No** β€” no doc, no test, no mention in `docs/SERVICE.md` or `docs/TRAY.md` | **P1** | | AV/EDR quarantines the branded launcher | `branding.ensure_branded_launcher` copies an interpreter, rewrites its `RT_VERSION` and stages DLLs into `Scripts\` β€” textbook EDR heuristic bait | Design is fail-soft (tray runs unbranded), but an EDR that quarantines the *source* `pythonw.exe` can break the tray **and** the engine. `docs/ANTIVIRUS-FIREWALL.md` process-exclusion table lists `python.exe`, `messagefoundry.exe`, `nssm.exe` β€” no `pythonw.exe`, no `MessageFoundryTray.exe` | **No** | **P1** | diff --git a/docs/testing/master-test-plan/15-alerting-and-observability.md b/docs/testing/master-test-plan/15-alerting-and-observability.md index 76f21c5c7..696c2049c 100644 --- a/docs/testing/master-test-plan/15-alerting-and-observability.md +++ b/docs/testing/master-test-plan/15-alerting-and-observability.md @@ -132,11 +132,11 @@ ALERT-13 and ALERT-11. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| **G1 β€” three alert vocabularies bound only by convention** | An emit site's `type` literal, `_ALERT_EVENT_TYPES`, and the `AlertSink` Protocol / `LoggingAlertSink` / `NotifierAlertSink` method sets drift. A type absent from the routable set can never be escalated, routed, suppressed or muted by any operator rule; a method present only on `NotifierAlertSink` raises `AttributeError` on the logging fallback | Every deployment. Drift has fired **twice** already: `lane_stuck` + `rcsi_off_degraded` once shipped unroutable β€” **both are members today** ([`settings.py:2515-2516`](../../../messagefoundry/config/settings.py)), rule-targetable end to end at `test_alert_rules.py:357/370/395`, so the only live drift is that no guard stops the next one; `content_match` is **still** divergent (only on `NotifierAlertSink` at [`alert_sinks.py:676`](../../../messagefoundry/pipeline/alert_sinks.py), absent from the Protocol and the fallback). Note the non-obvious third form: `AlertSink.saturation_rising()` emits type `"saturation"` β€” method name β‰  event type, so a naive guard would false-fail | **No.** No guard exists in either direction | **P0** | -| **G2 β€” `content_match` has no Handler-reachable surface** | ADR 0133 AC-3 says "WHEN a Handler emits a `content_match`" β€” but there is no export in `messagefoundry/__init__.py` (unlike `db_lookup`/`fhir_lookup` at lines 32-33/150/153), no injected sink on a Handler context, and no dry-run path. Every test calls `sink.content_match(...)` directly | The differentiating Corepoint "Action Point" parity capability is unusable in practice; the PHI-free-by-contract guarantee has never been exercised through real Handler code | **No** β€” AC-3/AC-4 pass at the sink so nothing fails | **P0** | -| **G3 β€” `connection_started` is mapped but emitted nowhere** | `_AUTO_RESOLVE["connection_started"] = "connection_stopped"` ([`alert_sinks.py:100`](../../../messagefoundry/pipeline/alert_sinks.py)) but a repo-wide search finds **no emit site**. A lane that STOPs on an internal error and is later restarted (by hand or by a #144 `control_action`) leaves its `alert_instance` permanently `open` | `alerts_active` on the connections dashboard stays non-zero forever; `/alerts/active` accumulates; the nav bell's `list_active_alerts(limit=200)` ([`webconsole/routes/status.py:166`](../../../messagefoundry_webconsole/routes/status.py)) saturates. Textbook alert fatigue β€” the operator learns to ignore the list and the next real stop is missed | **Partially** β€” `tests/test_phi_logging_inventory.py:439` *documents* the dead key; nothing asserts the operator consequence | **P0** | -| **G4 β€” the alert/security-notify SMTP hop is encrypted but unauthenticated** | `send_plain_email` calls `smtp.starttls()` with **no** `SSLContext` ([`alert_sinks.py:384`](../../../messagefoundry/pipeline/alert_sinks.py)), so the stdlib default applies (`check_hostname=False`, `verify_mode=CERT_NONE`). The same transport carries the ASVS 6.3.5/6.3.7 per-user security notifications | A MITM on the SMTP hop reads every alert body (connection names, queue shapes, `safe_exc` detail strings) and every account-security notification (username, source IP, failed-attempt count) β€” and can suppress them | **No, and the docs disagree with themselves.** `docs/PHI.md` row 11 states the caveat plainly; `docs/BACKLOG.md:5152` claims the sink "uses STARTTLS with a verifying context by design". No test pins either | **P0** | -| **G5 β€” `[alerts]` is startup-only** | `app.state.alerts_settings` is assigned only at app construction ([`api/app.py:1120`](../../../messagefoundry/api/app.py)) and lifespan startup (`:5485`). `POST /config/reload` (`:2741`) re-runs the `--config` graph, never the service-settings TOML | An operator adds a suppression rule mid-incident via the IDE or `messagefoundry alert add`; the IDE re-lists from the **file** and shows it; `/alerts/rules` still shows the **startup** set; the running notifier keeps paging until a restart. The requirement is documented only in `alerts_edit.py:19-21` and the CLI docstring β€” nowhere an operator looks | **No** | **P0** | +| **G1 β€” three alert vocabularies bound only by convention** | An emit site's `type` literal, `_ALERT_EVENT_TYPES`, and the `AlertSink` Protocol / `LoggingAlertSink` / `NotifierAlertSink` method sets drift. A type absent from the routable set can never be escalated, routed, suppressed or muted by any operator rule; a method present only on `NotifierAlertSink` raises `AttributeError` on the logging fallback | Every deployment. Drift has fired **twice** already: `lane_stuck` + `rcsi_off_degraded` once shipped unroutable β€” **both are members today** ([`settings.py:2515-2516`](../../../messagefoundry/config/settings.py)), rule-targetable end to end at `test_alert_rules.py:357/370/395`, so the only live drift is that no guard stops the next one; `content_match` is **still** divergent (only on `NotifierAlertSink` at [`alert_sinks.py:676`](../../../messagefoundry/pipeline/alert_sinks.py), absent from the Protocol and the fallback). Note the non-obvious third form: `AlertSink.saturation_rising()` emits type `"saturation"` β€” method name β‰  event type, so a naive guard would false-fail. **STRUCTURALLY RE-VERIFIED 2026-08-15 (BACKLOG #1100): the row's own "no LIVE type-level drift" assessment is CONFIRMED.** An AST census of `_ALERT_EVENT_TYPES` (**18** types) against every `{"type": "..."}` literal in `messagefoundry/` (**22** distinct) yields **4** emitted-but-not-routable and **0** routable-but-never-emitted β€” and all 4 are deliberate: `connection_restored`, `leadership_lost` and `dr_released` are three of the four `_AUTO_RESOLVE` **keys** (`alert_sinks.py:99-106`), i.e. resolution events rather than raisable alerts, and `test_email` is the API's diagnostic transport probe (`app.py:2579`, documented "NO message body, NO PHI"). **A SECOND NAIVE-GUARD TRAP, MEASURED THE HARD WAY HERE AND MORE DANGEROUS THAN THE method-name-vs-event-type ONE THIS ROW ALREADY NAMES:** `_ALERT_EVENT_TYPES` is `frozenset({...})`, **not** a bare set literal, so a regex expecting `{` after the `=` matches nothing and yields an **EMPTY** routable set β€” whereupon **every** emitted type reads as unroutable. That produced a confident **22-of-22 "unroutable"** result on the first pass here, which is indistinguishable from a catastrophic real finding and is refuted only by noticing that 0 routable-but-never-emitted is impossible alongside it. **Parse this constant with AST, and assert the routable set is non-empty before comparing** β€” the guard's own failure mode is to fail LOUD and WRONG. *Independent corroboration of G3 fell out of the same census: exactly 3 of the 4 `_AUTO_RESOLVE` keys are emitted, and the missing one is `connection_started`* | **No.** No guard exists in either direction | **P0** | +| **G2 β€” `content_match` has no Handler-reachable surface** | ADR 0133 AC-3 says "WHEN a Handler emits a `content_match`" β€” but there is no export in `messagefoundry/__init__.py` (unlike `db_lookup`/`fhir_lookup` at lines 32-33/150/153 β€” **all four anchors RE-VERIFIED EXACT 2026-08-15, BACKLOG #1100**: the imports are at `:32`/`:33` and the `__all__` entries at `:150`/`:153`, while **`content_match` appears NOWHERE in the module**. **This row carries its own positive control and that is why it is trustworthy**: the same search that finds zero for `content_match` finds both comparators, so the zero is a fact about the code and not about the probe), no injected sink on a Handler context, and no dry-run path. Every test calls `sink.content_match(...)` directly | The differentiating Corepoint "Action Point" parity capability is unusable in practice; the PHI-free-by-contract guarantee has never been exercised through real Handler code | **No** β€” AC-3/AC-4 pass at the sink so nothing fails | **P0** | +| **G3 β€” `connection_started` is mapped but emitted nowhere** | `_AUTO_RESOLVE["connection_started"] = "connection_stopped"` ([`alert_sinks.py`](../../../messagefoundry/pipeline/alert_sinks.py) β€” **RE-POINTED 2026-08-15 (BACKLOG #1100): the mapping is at `:101`, not the cited `:100`, which holds the sibling `"connection_restored": "connection_error"` entry**) but a repo-wide search finds **no emit site**. **CONFIRMED WITH A POSITIVE CONTROL**: `"connection_started"` occurs in exactly ONE place in `messagefoundry/` β€” the `_AUTO_RESOLVE` mapping itself β€” whereas the same search finds `"connection_stopped"` genuinely emitted at `alert_sinks.py:682` (`self._emit({"type": "connection_stopped", ...})`) plus a `settings.py:2601` reference. **So the search does find emit sites when they exist, and the asymmetry IS the defect: one half of an auto-resolve PAIR can never fire.** `connection_stopped` raises the alert and `connection_started` is only ever a resolution TARGET, so a lane that stops on an internal error and is later restarted never clears β€” the alert is raisable but not resolvable, which is worse than an alert that was never wired at all because the operator sees a live alert for a healthy lane. A lane that STOPs on an internal error and is later restarted (by hand or by a #144 `control_action`) leaves its `alert_instance` permanently `open` | `alerts_active` on the connections dashboard stays non-zero forever; `/alerts/active` accumulates; the nav bell's `list_active_alerts(limit=200)` ([`webconsole/routes/status.py:166`](../../../messagefoundry_webconsole/routes/status.py)) saturates. Textbook alert fatigue β€” the operator learns to ignore the list and the next real stop is missed | **Partially** β€” `tests/test_phi_logging_inventory.py:439` *documents* the dead key; nothing asserts the operator consequence | **P0** | +| **G4 β€” CLOSED by #323 (2026-08-02). The alert/security-notify SMTP hop is VERIFIED.** This row asserted the hop was encrypted but unauthenticated and that `docs/PHI.md` and `docs/BACKLOG.md` contradicted each other about it. **Both halves are false at HEAD** and were already false when this plan was written (BACKLOG #1100). | `send_plain_email` builds an explicit verifying context via `tls_policy.build_smtp_tls_context()` and passes it β€” `smtp.starttls(context=tls_context)` ([`alert_sinks.py:430-431`](../../../messagefoundry/pipeline/alert_sinks.py)), whose comment reads *"context= is REQUIRED (#323): starttls()'s own default verifies NOTHING"*. The `CERT_NONE`/`check_hostname=False` text at `:387-388` is a **historical note about the fixed defect**, not the current posture β€” reading it as current is the mistake this row made. | β€” | **Yes.** `docs/PHI.md` row 11 states the verifying posture and records the pre-#323 state explicitly as history; `tests/test_alert_smtp_tls.py` exists with 21 tests. The documents agree with the code and with each other. | **CLOSED** | +| **G5 β€” `[alerts]` is startup-only** | `app.state.alerts_settings` is assigned only at app construction. **RE-VERIFIED 2026-08-15 (BACKLOG #1100) β€” substance CONFIRMED AND STRONGER THAN WRITTEN, all three anchors drifted.** Corrected: the assignment is [`api/app.py:1134`](../../../messagefoundry/api/app.py) (cited `:1120`), and `POST /config/reload` is at `:2791` (cited `:2741`, now a blank line). The cited `:5485` lands on an unrelated `phi_read_hop_secure` parameter. **The row claims TWO write sites; there is exactly ONE.** `app.state.alerts_settings =` occurs once in the whole module (`:1134`) β€” every other occurrence is either a constructor parameter (`:1038`, `:5453`), a call argument (`:1546`, `:5580`), or a READ via `getattr(request.app.state, "alerts_settings", None) or AlertsSettings()` (`:1516`, `:2517`, `:4517`). So the "lifespan startup" site is not a second write, it is the plumbing that feeds the one write. **That makes the gap cleaner to test than the row implies: a single assignment means a single point to prove unreachable from reload**, and the `getattr(..., None) or AlertsSettings()` read pattern means a reload-time change does not merely fail to apply, it silently falls back to defaults at every read site | An operator adds a suppression rule mid-incident via the IDE or `messagefoundry alert add`; the IDE re-lists from the **file** and shows it; `/alerts/rules` still shows the **startup** set; the running notifier keeps paging until a restart. The requirement is documented only in `alerts_edit.py:19-21` and the CLI docstring β€” nowhere an operator looks | **No** | **P0** | | **G6 β€” IDE alert editor offers 4 of the 18 event types and 7 of the 15 fields** | [`ide/src/alertEditor.ts:13-19`](../../../ide/src/alertEditor.ts) offers a 5-entry dropdown β€” `any` plus only `connection_stopped`/`queue_buildup`/`storage_threshold`/`cert_expiry`; `:25-32` supports only `event_type`/`connection`/`min_depth`/`min_oldest_seconds`/`severity`/`transports`/`cooldown_seconds` (7 of the 15 `AlertRule` fields). `ide/src/test/suite/` has **no** alert test file (35 suites, none for the alert editor) | An operator on the supported GUI authoring path cannot rule on **14 of the 18** signals, nor set `id`, `recipients`, `mute`, `escalate`, `schedule`, `content_label`, `control_action`, `control_target` | **No** β€” the `ide` CI leg runs and tests nothing here | **P1** | | **G7 β€” no end-to-end drive of the runner's buildup / stall / saturation emit sites** | `_maybe_alert_buildup` (`:5401`), `_maybe_alert_saturation` (`:5444`), `_maybe_alert_stall` (`:5511`) in `wiring_runner.py`. Only the pure `SaturationDetector` and the **engine-shard** non-owned-lane watchdog are tested | These are the three alerts an operator relies on to notice a stalled or drowning feed. Threshold resolution, the `_outbound_paused` suppression guard, the per-`(stage,lane)` `_BUILDUP_REALERT_SECONDS` throttle and the `pending_depth` read could all break silently | **No** β€” FEATURE-COVERAGE-PLAN `FCP:ALERT-10` flagged it and it is still open | **P1** | | **G8 β€” the alert-storm bound is untested** | `_MAX_QUEUE = 1000` with drop-with-warning ([`alert_sinks.py:126-131`](../../../messagefoundry/pipeline/alert_sinks.py)). With a wedged webhook (a hung POST inside the 10 s timeout) and a large estate, a burst silently exceeds the bound and the excess is dropped with only a `WARNING` β€” there is **no dropped-alert counter or metric** | The operator sees neither the alerts nor a countable drop signal. A regression that lowers the bound, blocks the drain loop, or turns the drop into a *stall* on the emitting delivery worker is invisible | **No** β€” `tests/test_communications_inventory.py:372` pins `_MAX_QUEUE` as documentation only | **P1** | @@ -238,7 +238,7 @@ observations whose falsifiable halves are carried by T rows (ALERT-18/40/41 and | ALERT-64 | DST-boundary correctness of schedule-aware rules | Functional | manual | W2025-box | n/a | T | P2 | A rule with an IANA-tz window spanning a DST transition activates and deactivates at the correct wall-clock local times on a real host clock across the boundary, in both the spring-forward and fall-back directions | | ALERT-65 | `docs/FEATURE-MAP.md` Β§9/Β§10 currency guard | Compat | β€” | dev-PC | n/a | T | P2 | **Pointer.** Covered by MIG's consolidated FEATURE-MAP drift-guard row (`MIG-28`, one extension of `tests/test_feature_map_claims.py`); no separate work scoped here. The alerting-specific claims this chapter hands to MIG as that row's inputs: Β§9 names **every** member of `_ALERT_EVENT_TYPES` (all **18** β€” `messagefoundry/config/settings.py`); Β§9 has a row for each of alert state, escalation, templates, per-rule recipients, control actions, test-email, support bundle, crashdump suppression, `connection_event`, `/logs/tail`, `/metrics/history`, host metrics; and **no** Β§10 row nor any line references a module path absent from disk (catches `messagefoundry/console` and the `:131` "PySide6 desktop console stays" claim) | | ALERT-66 | `docs/BACKLOG.md` #171 build-state correction | Functional | pytest | dev-PC | n/a | T | P2 | #171's banner no longer claims DEMAND-GATE / "no runtime control and no viewer"; the existing `tests/test_backlog_status_check.py` + `.github/workflows/backlog-hygiene.yml` pass on the edited entry, and the entry cites `api/app.py:4541`, `:4570` and ADR 0130 | -| ALERT-67 | `docs/PHI.md` row 11 vs `docs/BACKLOG.md:5152` β€” a CI guard binding both docs to the code's SMTP posture | Functional | pytest | dev-PC | n/a | T | P0 | The pass criterion is the **guard**, not the decision (OQ-3 decides *which* posture; the row can fail under either). `tests/test_phi_logging_inventory.py` gains an assertion that reads the observed `starttls` posture the way ALERT-08 pins it β€” verifying context vs `check_hostname=False`/`verify_mode=CERT_NONE` β€” and asserts (a) `docs/PHI.md` row 11's STARTTLS wording describes **that** posture, and (b) no other doc sentence contradicts it, so `docs/BACKLOG.md:5152`'s "uses STARTTLS with a verifying context by design" must either match the code or be gone. Fails today, because the two documents disagree; fails again on any future doc↔code divergence | +| ALERT-67 | `docs/PHI.md` row 11 vs `docs/BACKLOG.md:5152` β€” a CI guard binding both docs to the code's SMTP posture | Functional | pytest | dev-PC | n/a | T | P0 | The pass criterion is the **guard**, not the decision (OQ-3 decides *which* posture; the row can fail under either). `tests/test_phi_logging_inventory.py` gains an assertion that reads the observed `starttls` posture the way ALERT-08 pins it β€” verifying context vs `check_hostname=False`/`verify_mode=CERT_NONE` β€” and asserts (a) `docs/PHI.md` row 11's STARTTLS wording describes **that** posture, and (b) no other doc sentence contradicts it, so any doc sentence about this posture must either match the code or be gone. **THE GUARD IS STILL WORTH BUILDING; ITS STATED STARTING CONDITION IS NOT.** "Fails today, because the two documents disagree" was already false when written (BACKLOG #1100): #323 closed the divergence on 2026-08-02, and `docs/PHI.md` row 11 now states the verifying posture with the old one recorded as history. So this test is expected to PASS on arrival and earns its keep by failing on a FUTURE doc-code divergence β€” which is the only thing it was ever able to catch. Note also that the cited `docs/BACKLOG.md:5152` anchor has drifted; re-derive it by content rather than by line number | | ALERT-68 | The notifier itself is the thing that is down: an unreachable SMTP host and a 500-looping webhook block no pipeline stage, and the failure is itself observable | HA/Resilience | pytest | dev-PC | SQLite | T | P1 | Two fault modes driven through a live `RegistryRunner` with a real `NotifierAlertSink` and a live store: (a) `[alerts].email_smtp_host` pointed at a closed port so `send_plain_email` raises `ConnectionRefusedError`/`socket.gaierror` at (and at the far end of) the connect timeout; (b) a webhook target returning HTTP 500 to every POST β€” 50 consecutive events each. Assert for both: **no stage blocks** β€” every `_emit` returns sub-millisecond, the ingressβ†’routedβ†’outbound handoffs keep committing, and end-to-end message throughput stays within 5% of a no-alert control run (a transport that awaits inside `_emit`, or a retry loop on the emitting worker, fails here); the durable `alert_instance` row is still upserted `open`, so `/ui/alerts` shows the condition nobody was paged about; a healthy sibling transport still delivers every event (distinct from ALERT-21's *wedged* transport β€” here the transport **fails fast** rather than hanging); and the failure is observable β€” exactly one WARNING per event naming the transport and the event type, carrying neither recipient addresses nor webhook credentials, plus the ALERT-20 counter if OQ-9 adds one. A silent swallow with no record fails the row | ### 14.5 Detailed scenarios @@ -337,8 +337,16 @@ a synthetic mailbox; `email_use_tls = true`; `smtp_allowed_hosts` containing the 6. Cross-check every observation against ALERT-08's asserted posture. **Expected result.** For (a): all three succeed. For (b) and (c): the observed behaviour is -**identical to what ALERT-08 asserts** β€” today that means all three still succeed (unauthenticated -STARTTLS). Any divergence between this recorded matrix and the unit test's assertion is a finding. +**identical to what ALERT-08 asserts**. Any divergence between this recorded matrix and the unit +test's assertion is a finding. + +**THE PARENTHETICAL THAT USED TO STAND HERE WAS STALE WHEN WRITTEN (BACKLOG #1100):** it read "today +that means all three still succeed (unauthenticated STARTTLS)". #323 closed that on 2026-08-02 β€” +`send_plain_email` passes a verifying context β€” so a self-signed (b) and a hostname-mismatched (c) +cert are now expected to be **REJECTED**, not to succeed. Running this procedure against the old +sentence would have recorded three passes as the expected matrix and read a working control as a +finding, which is the inversion this row exists to prevent. Derive the expectation from ALERT-08's +assertion at run time rather than from any prose here. **Cleanup.** Restore cert (a); unlock the synthetic account; purge the synthetic mailbox. @@ -590,7 +598,8 @@ This area is signed off for release when **all** of the following hold: 1. **All 12 P0 (class-`T`) rows pass or are explicitly waived by the owner with a dated ADR entry:** ALERT-01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 58, 67. (ALERT-02 and ALERT-06 may exit as `xfail` **only** if OQ-1/OQ-2 resolve to "designed-not-built", and only with the ADR amendment merged.) 2. **The G1 mirror invariant is enforced in CI.** A planted emit type outside `_ALERT_EVENT_TYPES`, and a planted method on one of the three classes only, each fail the `test` leg. Falsifiability is itself asserted (mirror `tests/test_alerts_edit.py:160`). 3. **Zero PHI leaks.** ALERT-16, 17, 18, 45, 46, 48, 49 all pass; the ALERT-45 residual count is at or under its committed ceiling; the leak gate is fail-closed. -4. **The three contradicting documents agree with the code.** `docs/PHI.md` row 11, `docs/BACKLOG.md:5152` and the ALERT-08 assertion state one posture (ALERT-67); `docs/BACKLOG.md` #171 no longer claims unbuilt (ALERT-66); `docs/FEATURE-MAP.md` Β§9 names every routable event type and Β§10 references no non-existent module (ALERT-65). +4. **The documents agree with the code, and the SMTP-posture half is ALREADY SATISFIED.** `docs/PHI.md` row 11 and the ALERT-08 assertion state one posture (ALERT-67); `docs/BACKLOG.md` #171 no longer claims unbuilt (ALERT-66); `docs/FEATURE-MAP.md` Β§9 names every routable event type and Β§10 references no non-existent module (ALERT-65). + **This criterion previously read "the three CONTRADICTING documents agree with the code" and named a contradiction that #323 had already closed on 2026-08-02 (BACKLOG #1100).** An exit criterion demanding that a resolved contradiction be resolved is a gate that can never inform anyone: it cannot fail, so passing it says nothing, and a reader who trusts it believes a check ran. What remains genuinely gating is the *guard* (ALERT-67), which earns its keep on a future divergence rather than on this one. 5. **The four superseded FEATURE-COVERAGE-PLAN rows are recorded as closed** with the dated re-verification in 14.2, and the remaining open ones (`FCP:ALERT-3`, `FCP:ALERT-5`, `FCP:ALERT-9`, `FCP:ALERT-10`) map 1:1 onto ALERT-19, ALERT-22, ALERT-13, ALERT-11 here. 6. **Every P1 class-`T` row passes or carries a dated, owner-accepted waiver.** In particular the three that are pure *pins of known non-coverage* (ALERT-15 outbound saturation blind spot, ALERT-26 restart asymmetry, ALERT-47 support-bundle no-RBAC) must be **passing pins**, not waivers β€” a pin that is waived is worthless. 7. **The detectability sweep (ALERT-55) reports zero unnamed holes.** Any hole is either closed or listed by name in the release notes with an operator workaround. @@ -603,7 +612,8 @@ This area is signed off for release when **all** of the following hold: 1. **`content_match` reachability (ADR 0133 D3).** Should a Handler get a first-class exported way to raise it β€” an injected sink, or a `messagefoundry.alert_content(...)` export alongside `db_lookup`/`fhir_lookup` β€” or should ADR 0133 D3 be re-scoped as *designed, not built*? Today no Handler can reach it and `LoggingAlertSink` has no fallback method, so a fallback-path caller would `AttributeError`. **Blocks:** ALERT-02, 04, 05; whether `content_match` stays in `_ALERT_EVENT_TYPES`; the ADR 0133 AC-3 wording. 2. **`connection_started`.** Should a lane restart auto-resolve an open `connection_stopped`, or is manual operator resolve the intended workflow β€” and if the latter, should the dead `_AUTO_RESOLVE` key be removed? **Blocks:** ALERT-06, 07; and, downstream, ALERT-35's growth expectation (G16 is largely a consequence of this). -3. **Alert / security-notification SMTP posture.** Is the unauthenticated STARTTLS (`check_hostname=False`, `verify_mode=CERT_NONE`) an accepted residual, or must this path adopt the ADR 0092/0153 hop gradient like the EMAIL *destination* connector? `docs/PHI.md` row 11 and `docs/BACKLOG.md:5152` currently contradict each other and **one of them must be corrected either way**. **Blocks:** ALERT-08, 09, 58, 67 β€” and this is the chapter's only P0 with a real security consequence. +3. **Alert / security-notification SMTP posture β€” ANSWERED, AND IT WAS ANSWERED BEFORE THIS QUESTION WAS WRITTEN (BACKLOG #1100).** This asked whether the unauthenticated STARTTLS was an accepted residual or had to adopt the ADR 0092/0153 hop gradient, and asserted that `docs/PHI.md` row 11 and `docs/BACKLOG.md` "currently contradict each other". **Neither premise holds.** #323 closed it on 2026-08-02: `send_plain_email` passes an explicit verifying context, so the hop is authenticated. And the gradient half has a recorded answer too β€” per `docs/PHI.md` row 11 this path did **not** adopt the connector hop gradient; its deviations (`email_use_tls = false`, `email_tls_verify = false`) are gated by a `[security].allow_unverified_alert_smtp_tls` **acknowledgment switch at the serve gate**, which refuses to start on an enforcing PHI instance without it and `AUDIT`-logs the start with it. + **It therefore BLOCKS NOTHING.** ALERT-08, 09, 58 and 67 are unblocked and should be built against the posture as shipped. Leaving this open was the more expensive error of the two: an open P0 "with a real security consequence" reads as a live exposure, and this chapter's only such row was describing a fix. 4. **Should `[alerts]` become reloadable** (via `POST /config/reload` or a dedicated route), or is restart-only the intended contract β€” and if so, must the IDE editor **and** the console surface a `pending restart` state? **Blocks:** ALERT-10, 32, 33 and scenario S5. 5. **Leader-gating for `cert_expiry` / `secret_rotation` / `update_check` / `gcm_invocations`.** They report **cluster-wide** facts, unlike the per-node lane events ADR 0014 Β§4 reasons about, so an N-node cluster pages N times per condition. Gate them, dedupe at the store, or accept? Note `gcm_invocations.py:28-29` already argues against gating the *refill* β€” the alert is a separable question. **Blocks:** ALERT-27, 28, 63. 6. **Is the outbound saturation blind spot acceptable for release** (a healthy-but-behind lane is never sampled), or must BACKLOG #93's periodic owned-outbound depth sweep land first? **Blocks:** whether ALERT-15 is a pin or a bug; and one entry in the ALERT-55 sweep. diff --git a/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md b/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md index 3e4e452ea..9944d91b3 100644 --- a/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md +++ b/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md @@ -196,9 +196,9 @@ its nine rows are settled evidence, not open work. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| The post-2026-07-14 security wave has no coverage-plan owner | `FEATURE-COVERAGE-PLAN.md` is dated 2026-07-13/14 and contains **zero** references to ADRs 0135, 0138–0153 (verified by grep). Every "is it covered?" answer drawn from it is wrong by omission for exactly the newest controls. | The enforcement dial, both ADR 0140 carve-outs, `allowed_client_networks`, in-use protection, the cleartext-hop collapse, the Transit cipher, the handler-security lint, hardened runtime isolation, SBOM/VEX | **No** β€” the gap-audit instrument itself is the blind spot | **P0** | -| A one-line YAML edit disarms the whole blocking CI gate set | Adding `continue-on-error: true` or an `if:` to any of the seven blocking `security.yml` jobs converts it to advisory with a green tick. The guard pattern exists for `freethread-smoke.yml` (`tests/test_freethread_smoke_liveness.py:55`) and `quality-advisory.yml` (`test_quality_advisory_invariants.py:132`) but was **never extended to `security.yml`** | pip-audit, npm-audit, bandit, gitleaks, semgrep, crypto-inventory, forbidden-content β€” every SAST/SCA/secret/leak control the Secure Build Scorecard grades "Strong" | **No** | **P0** | -| The customer/PHI leak gate's detector floor silently falls behind the token list | `MEFOR_MIN_DETECTORS=names=7,estate=13,site_prefixes=1` exists only as a shell literal (`security.yml:403`). Nothing asserts it still matches the real list, that the spec parses, or that a partially-mangled secret is caught. The real list arrives only from `MEFOR_FORBIDDEN_TOKENS`; `scan-tokens.local.txt` is gitignored (`.gitignore:126`) and never committed, so no checkout can hold it | Real partner/site names and estate identifiers reaching a **public** repository | Partially β€” `test_scan_tokens_source.py` covers the floor **parser**, nothing covers the **value** | **P0** | +| The post-2026-07-14 security wave has no coverage-plan owner | `FEATURE-COVERAGE-PLAN.md` is dated 2026-07-13/14 and contains **zero** references to ADRs 0135, 0138–0153 (verified by grep). Every "is it covered?" answer drawn from it is wrong by omission for exactly the newest controls. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100): still zero for all 17 ADRs**, re-measured with a positive control (ADRs 0010/0023/0030/0066 *do* resolve in that file, so the zero is a real absence and not a broken grep), and its status blocks still date to 2026-07-13. **This row is the one whose staleness would be invisible from inside**, because the instrument it indicts is the same one a reader would use to check it | The enforcement dial, both ADR 0140 carve-outs, `allowed_client_networks`, in-use protection, the cleartext-hop collapse, the Transit cipher, the handler-security lint, hardened runtime isolation, SBOM/VEX | **No** β€” the gap-audit instrument itself is the blind spot | **P0** | +| A one-line YAML edit disarms the blocking CI gate set β€” **CLOSED, the guard was extended (re-measured 2026-08-15, BACKLOG #1100)** | **`tests/test_security_posture.py` IS that guard** (12 tests, all passing; last touched `72e67095`, 2026-08-10, BACKLOG #1079). It covers **both** vectors this row names: `test_required_jobs_carry_no_continue_on_error` and `test_required_jobs_declare_no_skippable_job_level_if` (*"a job-level `if:` is the other way a required context silently never reports"*, allowlisted per job). `_BLOCKING_SECURITY_JOBS` is **exactly the seven listed here**, each asserted present in `.github/required-contexts.txt`, so "blocking" is a checked claim rather than a label. It also runs the **reverse** direction (`sbom`/`trivy` must KEEP `continue-on-error`, so a silent promotion is caught too) and is **not vacuous by construction**: `test_every_security_job_is_classified` fails when a new job is added to `security.yml` without a classification. **BEWARE THE TRAP THAT KEPT THIS ROW ALIVE:** that module's own docstring restates this row's claim *verbatim* β€” "nothing guarded it: the three tests in this repo that read a workflow's `continue-on-error` cover `quality-advisory.yml` and `freethread-smoke.yml`" β€” **as the history it was written to close**. Skimming it CONFIRMS the row | pip-audit, npm-audit, bandit, gitleaks, semgrep, crypto-inventory, forbidden-content β€” every SAST/SCA/secret/leak control the Secure Build Scorecard grades "Strong" | **Yes** β€” 12 tests, both vectors, plus an anti-vacuity check | **Closed** | +| The leak gate's detector floor is unasserted **in its production VALUE** β€” narrowed 2026-08-15 | **RE-MEASURED (BACKLOG #1100). Two of the three "nothing asserts" clauses are now FALSE**, and the anchor drifted (`security.yml:403` cited; `MEFOR_MIN_DETECTORS` is at **`:674`**, commented at `:655`). `test_scan_tokens_source.py` now carries **56 tests**: the spec **is** parse-asserted (`test_min_spec_parsing_rejects_nonsense`, `test_unrecognised_require_value_refuses`), and a **partially-mangled secret IS covered** (`test_present_but_unusable_token_source_fails_closed`, parametrized over `mangled` β€” its docstring records that a mangled secret once "yielded ZERO detectors"). **Materially, the estate half no longer depends on the token list at all:** BACKLOG #321's structural estate-identifier *shape* detector fires with **no token source present** (`test_estate_identifier_shape_is_flagged_without_any_token_source`, `test_the_estate_identifier_shape_detector_is_live`, and `test_allowlist_rejects_an_entry_broad_enough_to_disable_the_estate_shape`). **DEPENDENCY, STATED SO A PARTIAL LAND IS OBVIOUS: that detector is commit `c3959449`, which is on the builder branch and NOT on `main` β€” if this row lands without it, this cell is wrong** | Real partner/site names reaching a **public** repository. The **estate** half of that radius is now structurally covered; the **names** half still rests entirely on the token list | **Still the gap, and it is the one that cannot be closed the easy way:** every floor test uses a SYNTHETIC floor (`names=2 + estate=2 + site_prefixes=1`), so nothing asserts the real `names=7,estate=13,site_prefixes=1`. It **cannot** be asserted from a checkout β€” `scan-tokens.local.txt` is gitignored and the real list arrives only via `MEFOR_FORBIDDEN_TOKENS`. Any fix has to compare the floor to the list **inside the gate run**, not in pytest | **P0** | | Nothing external has adversarially challenged the posture | No third-party ASVS review, no penetration test, no DAST, no fuzzing. No `hypothesis`/`atheris`/`schemathesis`/ZAP in `pyproject.toml` or any workflow. Held by a dated signed risk acceptance that **voids on any off-loopback or production exposure** (`Secure_Build_Scorecard_MEFOR.md:63`; `BACKLOG.md:390` β€” external review + pentest are the GA/v1.0 gate) | The entire security claim set is self-assessed and AI-assisted | **No, by definition** | **P0** | | A correctly-configured Transit PHI instance is refused at startup | `__main__.py:1161` keys the keyless-PHI refusal on `settings.store.encryption_key or …encryption_key_file` β€” **cipher_provider-blind**. A `cipher_provider=vault_transit` deployment sets `MEFOR_STORE_TRANSIT_*` instead, so the *strongest* at-rest posture trips "no key β†’ refuse to start" (exit 2). The documented workaround makes the operator assert `allow_unencrypted_phi` about the most-protected configuration | ADR 0138 unusable on the PHI instances it exists for; `GET /security/posture` and the loosening register report a falsehood | **No** β€” no test drives `serve` with `cipher_provider=vault_transit` | **P1** | | The `/ai/chat` scope control is an honour-system label | `api/app.py:1417-1423` compares `body.data_scope` only; nothing inspects prompt **content**. Any `AI_ASSIST` holder can paste an HL7 body into a `code_only` prompt and have it brokered off-box. The audit records `prompt_chars`, not content (`:1449`) | The only sanctioned PHI-egress-shaped path in the product; a single pasted body is an unlogged PHI disclosure to a third party | **No** β€” no test, no detector, no written disposition | **P1** | diff --git a/docs/testing/master-test-plan/17-performance-and-scale.md b/docs/testing/master-test-plan/17-performance-and-scale.md index f8343c8a7..72aa208e9 100644 --- a/docs/testing/master-test-plan/17-performance-and-scale.md +++ b/docs/testing/master-test-plan/17-performance-and-scale.md @@ -108,9 +108,9 @@ Nothing below re-tests any of that; the rows below attack the **verdict**, the * |---|---|---|---|---| | The `--load` sustainable-rate verdict has no filling / backlog-slope term | A phase whose in-flight backlog grows through the whole hold passes `zero_loss` + eventual drain and reads as sustained. `report.py:503 _run_slos` checks only zero-loss, drain seconds, dead letters, dup rate β€” no slope, no latency divergence | Every published `--load` ceiling; every adopter sizing run; a cutover sized 3–5.5x over capacity drops clinical messages | **No.** The gate exists only on the co-located shardcert path (`shardcert.py:1758`); `shardcert_ladder.py:1092` documents its own absence on the two-box path in capitals | **P0** | | `/stats` poller-zero contamination | `enginepoll.py:556 await_drain` declares drained on `backlog == 0 and queue_depth == 0 and in_pipeline == 0` plus two equal `read`/`written` samples. A zeroed or frozen `/stats` under exactly the overload the gate exists for satisfies all four | A saturating, lossy run reports PASS and drained | No staleness precondition anywhere | **P0** | -| Engine sharding has zero CI execution | No workflow file contains the string `shard`. `tests/test_supervisor.py` drives `_FakeProcess`, never a real subprocess. `tests/test_shard_recovery_sqlserver.py`, `test_shard_recovery_postgres.py`, `test_shard_cert_sqlserver.py` are `MEFOR_TEST_*`-gated **and** named by no CI step, so they execute nowhere | ADR 0073 ownership-scoped recovery and single-delivery-consumer-per-lane are what keep per-lane FIFO and at-least-once correct on the only built multicore topology. A regression yields duplicate delivery, cross-engine-shard FIFO inversion, or permanently stranded rows | No | **P0** | -| Published sizing contradicts the measured record and itself | `docs/SYSTEM-REQUIREMENTS.md:195-198` offers tiers to "~500 – low-thousands msg/s / ~40M+/day"; the publishable measured figure is ~72 ev/s (`FCP:SCALE-19`, FEATURE-COVERAGE-PLAN:286, 7.23x short). The same file says multi-process scale-out "**is built**" (:163) and "a **future direction, not built**" (:213, :234) | The document an adopter sizes hardware from. Order-of-magnitude over-claim + a self-contradiction about whether the scale-out path exists | No doc-vs-measurement guard exists; `tests/test_feature_map_claims.py` checks only the ASVS score, private-path links and superseded docs | **P0** | -| The one published multi-process scale-out number was measured on a now-forbidden topology | `docs/benchmarks/TUNING-BASELINE.md:150` records the Ξ· β‰ˆ 0.85 / E_core β‰ˆ 42 msg/s **engine-sharding** result on the store line "**per-shard SQLite** (one store file per shard β€” **no shared DB**, no shared-DB commit contention by design)" (quoted verbatim from the source table), measured 2026-06-27. ADR 0063 (2026-07-01) and `sharding.py:81 require_unified_store` now **refuse** >1 engine shard on any non-server backend | The only published **engine-sharding** speedup β€” the shape adopters are told to multiply by their own `E_core` β€” is not reproducible with the shipped code, and it was measured with the shared-store contention deliberately absent | No | **P0** | +| ~~Engine sharding has zero CI execution~~ **WITHDRAWN β€” FALSE. Corrected 2026-08-15 (BACKLOG #1100)** | **This row asserted a total coverage gap over work that has DEDICATED CI STEPS BUILT TO COVER IT, and it would have driven someone to build tests that already exist and already run.** Two of its three clauses are refuted: (1) "no workflow file contains the string `shard`" is **FALSE** β€” `ci.yml` carries it at least seven times, including the named steps "Run the engine-shard + statement-dispatch suites on real SQL Server" (`:1836`) and "Run the failover + engine-shard recovery suites on real Postgres" (`:2039`), plus `shard_recovery`/`shard_cert` in the changed-path gate at `:1407` (positive control: `test` matched 24 of 24 workflow files, so the search was sound); (2) "named by no CI step, so they execute nowhere" is **FALSE** β€” all three are named at `ci.yml:1859`, `:1860`, `:2059`, and the steps set **exactly the variable the modules gate on** (`MEFOR_TEST_SQLSERVER: "1"` against `not os.getenv("MEFOR_TEST_SQLSERVER")`), so the gating makes them run rather than skip. The ci.yml comments record that these steps were ADDED to close this very gap ("before this step they were" dark), and `tests/test_serverdb_ci_coverage.py` now fails if a module-gated suite is added without being named β€” a meta-gate against this exact regression. **What SURVIVES, and it is real but far narrower than P0:** clause (3) holds β€” `tests/test_supervisor.py` uses `_FakeProcess` (11 references, docstring "without launching real engine subprocesses (Windows-safe)") with no `Popen`/`create_subprocess` anywhere, so the supervisor's process management is exercised only against a stand-in. **Re-scope to that: engine-shard RECOVERY is covered on both server backends; SUPERVISOR PROCESS HANDLING against a real subprocess is not** | ADR 0073 ownership-scoped recovery and single-delivery-consumer-per-lane are what keep per-lane FIFO and at-least-once correct on the only built multicore topology. A regression yields duplicate delivery, cross-engine-shard FIFO inversion, or permanently stranded rows. **THESE LAST THREE COLUMNS WERE WRITTEN FOR THE WITHDRAWN CLAIM AND I LEFT THEM STANDING WHEN I WITHDREW IT β€” corrected 2026-08-15 (BACKLOG #1100).** As published between those two edits the row CONTRADICTED ITSELF: the evidence column said the suites run in CI on both server backends, while the coverage column beside it still said "No" and the priority still said P0. The consequence above remains a true statement about what ADR 0073 protects; it is simply **no longer a consequence of THIS row**, because the recovery paths it names are the ones now shown to be exercised | **Was "No" β€” actually COVERED for engine-shard recovery** on real SQL Server and real Postgres (`ci.yml:1859`, `:1860`, `:2059`, each under a step that sets the gating `MEFOR_TEST_*` variable). **NOT covered: supervisor process handling against a real subprocess** (`test_supervisor.py` drives `_FakeProcess` only) | **NEEDS RE-GRADING, deliberately not re-graded here.** The P0 priced a total absence of engine-shard CI. The residual gap is narrower and I am not the right party to assign it a number β€” but **leaving P0 on a withdrawn finding is the worse error**, so it is flagged rather than silently carried | +| ~~Published sizing contradicts the measured record and itself~~ **WITHDRAWN β€” STALE. Corrected 2026-08-15 (BACKLOG #1100)** | **CORRECTED TWICE β€” my first withdrawal reason ("the document was fixed") was itself unverified, and the true verdict is MIS-CITATION rather than staleness.** Measured 2026-08-15 with `git log -S`: the phrase "low-thousands" **never appears in `docs/SYSTEM-REQUIREMENTS.md` at any point in that path's recorded history**, so nothing was removed from it. The phrase **did** exist elsewhere in the repo before this plan was written (`5fa6db9f`, `abedf865`, and `39990f8c` "bring the documentation set back in line with the code"), and today it survives in `docs/` **only in this row** β€” while the file's current `~165 msg/s` tier table arrived via `959821a7` (#79, 2026-07-30). **So the over-claim was real in SOME document and this row attributes it to a file that did not contain it; whether it was ever live in THIS file is UNDETERMINED and my path-scoped query cannot settle it.** A mis-citation is worse than staleness: staleness is blameless drift, whereas a quotation that cannot be sourced to the file it names discredits the row that carries it. *Recorded at length because the correction took three passes β€” the first two were "stale" and "never existed", and both were reached faster and on thinner evidence than I would have accepted for an assertion.* The "~500 – low-thousands msg/s / ~40M+/day" tier **no longer exists in `SYSTEM-REQUIREMENTS.md`**. The current tier table (`:216-219`) tops out at **~165 msg/s at 4 engine shards**, and `:219` now carries the caveat inline β€” "on per-shard SQLite β€” a topology that is not supported above one shard". The cited anchors have also drifted: `:163` is **blank**, and `:195-198` is bench/active-passive guidance, not tiers. The alleged self-contradiction cannot be reproduced either: **"future direction, not built" appears NOWHERE in `docs/`**, so there is no counterpart to the `is built` statements (which are real, at `:184`, `:187`, `:193`, `:297`). **The "7.23x short" arithmetic is therefore moot β€” but flagging it anyway, because ~500/~72 is 6.94x, not 7.23x, so the multiplier did not derive from the two numbers the row put beside it.** What is still worth keeping from this row is its LAST column β€” **but its SUPPORT is wrong, and I carried that forward unchecked before auditing it.** The **conclusion** stands: no doc-vs-measurement guard exists, so nothing would have caught the over-claim while it was live and nothing will catch the next one. The **enumeration** behind it does not: `tests/test_feature_map_claims.py` does not check "only the ASVS score, private-path links and superseded docs" β€” it has **six** tests, the three named plus `test_link_check_catches_a_missing_target_without_a_denylist`, `test_feature_map_is_actually_published` and `test_asvs_row_keeps_its_framing`. **None of the six compares a PUBLISHED number against a MEASURED one, which is why the conclusion survives the correction** β€” but a reader who quotes the enumeration to argue the file is nearly empty would be wrong, and the enumeration is the part that gets quoted | The document an adopter sizes hardware from. Order-of-magnitude over-claim + a self-contradiction about whether the scale-out path exists | No doc-vs-measurement guard exists β€” **TRUE, and the reason it survives is that none of `test_feature_map_claims.py`'s tests compares a PUBLISHED figure against a MEASURED one.** But the enumeration originally here was wrong and is corrected: the file does **not** check "only the ASVS score, private-path links and superseded docs" β€” it has **six** tests, adding `test_link_check_catches_a_missing_target_without_a_denylist`, `test_feature_map_is_actually_published` and `test_asvs_row_keeps_its_framing` | **NEEDS RE-GRADING β€” the finding above is WITHDRAWN as stale, so this P0 no longer prices anything live.** Flagged rather than re-graded, for the same reason as the row above: I am not the party to assign the number, but a P0 left standing on a withdrawn row reads as an open defect forever | +| The one published multi-process scale-out number was measured on a now-forbidden topology | `docs/benchmarks/TUNING-BASELINE.md:150` records the Ξ· β‰ˆ 0.85 / E_core β‰ˆ 42 msg/s **engine-sharding** result on the store line "**per-shard SQLite** (one store file per shard β€” **no shared DB**, no shared-DB commit contention by design)" (quoted verbatim from the source table), measured 2026-06-27. **RE-VERIFIED 2026-08-15 (BACKLOG #1100) β€” CONFIRMED, and the anchor is EXACT**, which is worth stating because the two rows above it in this same chapter were withdrawn as false and stale: `TUNING-BASELINE.md:150` is verbatim that store line, and `E_core β‰ˆ 42 msg/s/shard` / `Ξ· β‰ˆ 0.85` sit at `:161-162`. **One qualification to the row's framing: the topology is now DISCLOSED in both published files, so this is a reproducibility problem rather than a hidden-caveat one.** `TUNING-BASELINE.md:165-171` already separates the non-portable absolute from the portable "speedup shape", and `SYSTEM-REQUIREMENTS.md:219` names the unsupported topology inline. ADR 0063 (2026-07-01) and `sharding.py:81 require_unified_store` now **refuse** >1 engine shard on any non-server backend | The only published **engine-sharding** speedup β€” the shape adopters are told to multiply by their own `E_core` β€” is not reproducible with the shipped code, and it was measured with the shared-store contention deliberately absent | No | **P0** | | No throughput/latency regression gate anywhere | `.github/workflows/benchmark.yml` is `workflow_dispatch`-only, and each run wraps the harness in `set +e` … `set -e` so **even the harness's own SLO exit code is discarded**. Numbers are hand-transcribed into TUNING-BASELINE. `--baseline`/`--tolerance` exist (`report.py:753`) but are invoked by no workflow. `docs/CI-QUALITY.md` has no perf section | A 5x throughput or p99 regression from a claim-path, store or connector change merges green and is found on the next manual rig run | No | **P1** | | The published "reference performance floor" is enforced nowhere | TUNING-BASELINE:29 states the floor "β‰₯ 200 msg/s sustained Β· ACK p99 ≀ 50 ms Β· e2e p99 ≀ 5 s". `harness/load/profiles/reference.toml` `[load.slo]` carries only `max_error_rate`, `max_dead_letters`, `zero_loss` β€” deliberately, but nothing else asserts the floor either | A release can clear the two-tier gate's performance tier by assertion rather than measurement | No | **P1** | | No "harness-was-the-limit" boolean in the report | `deferred_backpressure` / `deferred_schedule` exist (`metrics.py:199-202`, written at `governor.py:51,58,86,123`) but `PhaseReport` and `_counters_dict` (`report.py:726`) emit only the total. Only shardcert has the attribution (`RungFidelity.DRIVE_SHORTFALL`) | `docs/LOAD-TESTING.md:296` and `W25:S4.2` (:755) / `W25:S4.7` (:907) / the validity gate (:1042) all require this flag FALSE before a ceiling counts β€” as written those acceptance steps are **unexecutable**, and a sender-bound number can be published as an engine ceiling | No | **P1** | diff --git a/docs/testing/master-test-plan/18-interop-migration-and-uat.md b/docs/testing/master-test-plan/18-interop-migration-and-uat.md index fdae2560f..982107522 100644 --- a/docs/testing/master-test-plan/18-interop-migration-and-uat.md +++ b/docs/testing/master-test-plan/18-interop-migration-and-uat.md @@ -88,10 +88,10 @@ This chapter covers the paths an organisation actually walks to *adopt*, *cut ov | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | Greenfield-only rule enforced nowhere | An operator flips `[store].type` to `postgres`/`sqlserver`; `open_store` ([`store/base.py:1739`](../../../messagefoundry/store/base.py)) selects the backend with **no prior-backend detection, no drain check, no warning**. The engine starts clean on an empty schema | The populated SQLite store β€” in-flight ingress/routed/outbound rows plus all retained PHI history and audit β€” is silently orphaned. Named as "Greenfield-only data loss" at `docs/releases/v0.1-PLAN.md:457`; the only shipped mitigation is prose in 6 documents and a manual checkbox at `docs/DEPLOY-SERVER-DB.md:274` | **No** | **P0** | -| Engine upgrade past a seam bump **bricks the whole engine**, not just `/ui` | `[api].serve_ui` defaults **True** ([`settings.py:672`](../../../messagefoundry/config/settings.py)). `create_app`'s `serve_ui` tail calls `assert_engine_seam(ENGINE_UI_SEAM)` ([`api/app.py:4951`](../../../messagefoundry/api/app.py)) and lets `UiSeamMismatch` propagate out of `create_app`. `SUPPORTED_ENGINE_SEAMS == frozenset({15})` β€” strict lockstep, no N-1 | `messagefoundry serve` **fails to start** on a PHI box mid-upgrade. Intake stops. The runbooks operators follow (EARLY-ADOPTER-GUIDE Β§13 step 4 :679, INSTALL-GUIDE Β§10 :306-313) bump only `messagefoundry==` and never mention the console wheel | Fails loud at startup β€” but nothing tests the *sequence*, and nothing warns beforehand | **P0** | -| The documented install-time seam defence does not exist | WEBCONSOLE-PACKAGE.md Β§2 claims a three-layer handshake whose layer 1 is "the PEP 508 range on the engine dependency fails an out-of-range pair at `pip`/`uv` resolve". `packaging/messagefoundry-webconsole/pyproject.toml:38-42` declares a **bare `"messagefoundry"`** with a comment conceding "a bare dep is fine for now" | `pip install -U messagefoundry` resolves happily across a seam bump. The first symptom is the P0 above. A documented control that is absent is worse than a known-absent one: the runbook author trusted it | **No** β€” nothing binds the doc claim to the metadata | **P0** | -| Reconcile diff output is PHI and **ungated** | `harness/reconcile/report.py` `render_text` prints up to 20 mismatched messages' field-level `left`/`right` **values** by default (:63-66); `render_json` **always** emits them (:31-42). `_run_compare` ([`__main__.py:80-93`](../../../harness/reconcile/__main__.py)) prints unconditionally and writes `--report-json` verbatim | This is the tool an operator runs during a live cutover, and its output is what gets pasted into a ticket. `tee` gates the same class of output behind `--show-diffs` with an explicit warning (`tee/__main__.py:269`, :451-453); the reconcile harness has **no equivalent** | **No** | **P0** | -| Reconcile capture writes PHI bodies by default | `CaptureSink` accepts `anonymizer=` ([`capture.py:46`](../../../harness/reconcile/capture.py)) but `_run_capture` never passes it and the `capture` subparser exposes **no** `--anonymize`/`--salt-env` flag | During a shadow phase the JSONL holds real production bodies on disk, on a non-production box, with no purge story | **No** | **P0** | +| Engine upgrade past a seam bump **bricks the whole engine**, not just `/ui` | `[api].serve_ui` defaults **True** (**RE-POINTED 2026-08-15, BACKLOG #1100: `settings.py:710`, was cited as `:672`** β€” [`settings.py`](../../../messagefoundry/config/settings.py)). `create_app`'s `serve_ui` tail calls `assert_engine_seam(ENGINE_UI_SEAM)` (**RE-POINTED: `api/app.py:5166`, was cited as `:4951` β€” a ~215-line drift, and `:4951` now lands on an unrelated `DR_OPERATE` dependency, so the stale anchor reads as plausible code rather than as an obvious miss**; the import sits at `:5149` β€” [`api/app.py`](../../../messagefoundry/api/app.py)) and lets `UiSeamMismatch` propagate out of `create_app`. `SUPPORTED_ENGINE_SEAMS == frozenset({15})` β€” strict lockstep, no N-1 | `messagefoundry serve` **fails to start** on a PHI box mid-upgrade. Intake stops. The runbooks operators follow (EARLY-ADOPTER-GUIDE Β§13 step 4 :679, INSTALL-GUIDE Β§10 :306-313) bump only `messagefoundry==` and never mention the console wheel | Fails loud at startup β€” but nothing tests the *sequence*, and nothing warns beforehand | **P0** | +| The documented install-time seam defence does not exist | WEBCONSOLE-PACKAGE.md Β§2 claims a three-layer handshake whose layer 1 is "the PEP 508 range on the engine dependency fails an out-of-range pair at `pip`/`uv` resolve". `packaging/messagefoundry-webconsole/pyproject.toml:38-42` declares a **bare `"messagefoundry"`** with a comment conceding "a bare dep is fine for now". **CONFIRMED VERBATIM 2026-08-15 (BACKLOG #1100), anchor exact**: the dependency list is exactly `"messagefoundry"`, and the comment reads in full "A PEP 508 compat RANGE is ideal (`messagefoundry>=X,