From c744a660e40065128449c7a505c9fdc3aaaff3a2 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 00:05:12 -0500 Subject: [PATCH 01/66] docs(testing): retire five dead SMTP-posture assertions in the alerting test plan (BACKLOG #1100) Five cells in chapter 15 assert that the alert/security-notify STARTTLS hop is unauthenticated and that docs/PHI.md and docs/BACKLOG.md contradict each other about it. BOTH PREMISES WERE ALREADY FALSE WHEN THE PLAN WAS WRITTEN: #323 closed it on 2026-08-02. I VERIFIED BOTH PRECONDITIONS AGAINST THE CODE BEFORE TOUCHING A WORD, because this chapter's only P0 "with a real security consequence" was one of the cells, and correcting it wrongly would have silently closed a live security question: alert_sinks.py:385 states "The STARTTLS hop is VERIFIED (#323, layer 3)", and :430-431 pass it -- `smtp.starttls(context=tls_context)`, commented "context= is REQUIRED (#323): starttls()'s own default verifies NOTHING". docs/PHI.md ROW 11 (:1015) states the verifying posture in full -- explicit context, chain + hostname + strict RFC 5280, TLS 1.2 floor, via tls_policy.build_smtp_tls_context() -- and records the pre-#323 state as history. tests/test_alert_smtp_tls.py exists, 21 tests. THE TRAP THAT NEARLY CAUGHT ME IS IN THE FILE I WAS READING. alert_sinks.py:387-388 describes `ssl._create_unverified_context`, `CERT_NONE`, `check_hostname=False` -- in the PAST TENSE, as the defect #323 fixed. Read as current, it confirms the plan's claim exactly. The first pass of this work stopped and reported the preconditions as UNMEASURED rather than guess which tense governed; the answer came from :385 and :430-431, not from :387. THE TESTS SURVIVE; ONLY THE PREMISE GOES. ALERT-08, 58 and 67 are still worth building -- pinning the posture at send time and binding the docs to the code catch a FUTURE divergence, which is the only thing they were ever able to catch. What changed is that ALERT-67 is now expected to PASS on arrival rather than "fail today", and S4's manual matrix expects a self-signed and a hostname-mismatched cert to be REJECTED. Running S4 against the old sentence would have recorded three passes as the expected result and read a working control as a finding. THE EXIT CRITERION WAS THE WORST OF THE FIVE: "the three contradicting documents agree with the code", naming a contradiction that was already resolved. An exit criterion demanding that a resolved contradiction be resolved CANNOT FAIL, so passing it says nothing while a reader believes a check ran. AND OQ-3 WAS THE MOST EXPENSIVE: an OPEN P0 blocking four rows, describing a fix as a live exposure. Its second half has a recorded answer too -- this path did NOT adopt the connector hop gradient; its deviations are gated by a [security].allow_unverified_alert_smtp_tls acknowledgment switch at the serve gate. It blocks nothing. Each dead assertion is QUOTED inside its correction rather than deleted, so a reader can see what the plan used to claim and why it was wrong -- the same reason #1215's marker paragraph records both of its inversions. A grep for the old strings therefore still matches, inside the corrections. SCOPE: this is the clearest cluster only -- five cells in 15-alerting-and-observability.md, whose premise I verified. The item names NINE sites; the other four are NOT verified and the item stays open for them. The cited docs/BACKLOG.md:5152 anchor has drifted and is flagged in place as re-derive-by-content. Verified: 112 passed / 91 skipped across the test-plan, doc-drift, feature-map and phi-logging suites; no-glyph cp1252 test run against the DIFF rather than the file, because the chapter carries pre-existing glyphs that are not mine to sweep. My added lines introduce none. --- .../15-alerting-and-observability.md | 22 ++++++++++++++----- 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/docs/testing/master-test-plan/15-alerting-and-observability.md b/docs/testing/master-test-plan/15-alerting-and-observability.md index 76f21c5c7..fd61a428f 100644 --- a/docs/testing/master-test-plan/15-alerting-and-observability.md +++ b/docs/testing/master-test-plan/15-alerting-and-observability.md @@ -135,7 +135,7 @@ ALERT-13 and ALERT-11. | **G1 — three alert vocabularies bound only by convention** | An emit site's `type` literal, `_ALERT_EVENT_TYPES`, and the `AlertSink` Protocol / `LoggingAlertSink` / `NotifierAlertSink` method sets drift. A type absent from the routable set can never be escalated, routed, suppressed or muted by any operator rule; a method present only on `NotifierAlertSink` raises `AttributeError` on the logging fallback | Every deployment. Drift has fired **twice** already: `lane_stuck` + `rcsi_off_degraded` once shipped unroutable — **both are members today** ([`settings.py:2515-2516`](../../../messagefoundry/config/settings.py)), rule-targetable end to end at `test_alert_rules.py:357/370/395`, so the only live drift is that no guard stops the next one; `content_match` is **still** divergent (only on `NotifierAlertSink` at [`alert_sinks.py:676`](../../../messagefoundry/pipeline/alert_sinks.py), absent from the Protocol and the fallback). Note the non-obvious third form: `AlertSink.saturation_rising()` emits type `"saturation"` — method name ≠ event type, so a naive guard would false-fail | **No.** No guard exists in either direction | **P0** | | **G2 — `content_match` has no Handler-reachable surface** | ADR 0133 AC-3 says "WHEN a Handler emits a `content_match`" — but there is no export in `messagefoundry/__init__.py` (unlike `db_lookup`/`fhir_lookup` at lines 32-33/150/153), no injected sink on a Handler context, and no dry-run path. Every test calls `sink.content_match(...)` directly | The differentiating Corepoint "Action Point" parity capability is unusable in practice; the PHI-free-by-contract guarantee has never been exercised through real Handler code | **No** — AC-3/AC-4 pass at the sink so nothing fails | **P0** | | **G3 — `connection_started` is mapped but emitted nowhere** | `_AUTO_RESOLVE["connection_started"] = "connection_stopped"` ([`alert_sinks.py:100`](../../../messagefoundry/pipeline/alert_sinks.py)) but a repo-wide search finds **no emit site**. A lane that STOPs on an internal error and is later restarted (by hand or by a #144 `control_action`) leaves its `alert_instance` permanently `open` | `alerts_active` on the connections dashboard stays non-zero forever; `/alerts/active` accumulates; the nav bell's `list_active_alerts(limit=200)` ([`webconsole/routes/status.py:166`](../../../messagefoundry_webconsole/routes/status.py)) saturates. Textbook alert fatigue — the operator learns to ignore the list and the next real stop is missed | **Partially** — `tests/test_phi_logging_inventory.py:439` *documents* the dead key; nothing asserts the operator consequence | **P0** | -| **G4 — the alert/security-notify SMTP hop is encrypted but unauthenticated** | `send_plain_email` calls `smtp.starttls()` with **no** `SSLContext` ([`alert_sinks.py:384`](../../../messagefoundry/pipeline/alert_sinks.py)), so the stdlib default applies (`check_hostname=False`, `verify_mode=CERT_NONE`). The same transport carries the ASVS 6.3.5/6.3.7 per-user security notifications | A MITM on the SMTP hop reads every alert body (connection names, queue shapes, `safe_exc` detail strings) and every account-security notification (username, source IP, failed-attempt count) — and can suppress them | **No, and the docs disagree with themselves.** `docs/PHI.md` row 11 states the caveat plainly; `docs/BACKLOG.md:5152` claims the sink "uses STARTTLS with a verifying context by design". No test pins either | **P0** | +| **G4 — CLOSED by #323 (2026-08-02). The alert/security-notify SMTP hop is VERIFIED.** This row asserted the hop was encrypted but unauthenticated and that `docs/PHI.md` and `docs/BACKLOG.md` contradicted each other about it. **Both halves are false at HEAD** and were already false when this plan was written (BACKLOG #1100). `send_plain_email` builds an explicit verifying context via `tls_policy.build_smtp_tls_context()` and passes it — `smtp.starttls(context=tls_context)` ([`alert_sinks.py:430-431`](../../../messagefoundry/pipeline/alert_sinks.py)), whose comment reads *"context= is REQUIRED (#323): starttls()'s own default verifies NOTHING"*. The `CERT_NONE`/`check_hostname=False` text at `:387-388` is a **historical note about the fixed defect**, not the current posture — reading it as current is the mistake this row made. | — | **Yes.** `docs/PHI.md` row 11 states the verifying posture and records the pre-#323 state explicitly as history; `tests/test_alert_smtp_tls.py` exists with 21 tests. The documents agree with the code and with each other. | **CLOSED** | | **G5 — `[alerts]` is startup-only** | `app.state.alerts_settings` is assigned only at app construction ([`api/app.py:1120`](../../../messagefoundry/api/app.py)) and lifespan startup (`:5485`). `POST /config/reload` (`:2741`) re-runs the `--config` graph, never the service-settings TOML | An operator adds a suppression rule mid-incident via the IDE or `messagefoundry alert add`; the IDE re-lists from the **file** and shows it; `/alerts/rules` still shows the **startup** set; the running notifier keeps paging until a restart. The requirement is documented only in `alerts_edit.py:19-21` and the CLI docstring — nowhere an operator looks | **No** | **P0** | | **G6 — IDE alert editor offers 4 of the 18 event types and 7 of the 15 fields** | [`ide/src/alertEditor.ts:13-19`](../../../ide/src/alertEditor.ts) offers a 5-entry dropdown — `any` plus only `connection_stopped`/`queue_buildup`/`storage_threshold`/`cert_expiry`; `:25-32` supports only `event_type`/`connection`/`min_depth`/`min_oldest_seconds`/`severity`/`transports`/`cooldown_seconds` (7 of the 15 `AlertRule` fields). `ide/src/test/suite/` has **no** alert test file (35 suites, none for the alert editor) | An operator on the supported GUI authoring path cannot rule on **14 of the 18** signals, nor set `id`, `recipients`, `mute`, `escalate`, `schedule`, `content_label`, `control_action`, `control_target` | **No** — the `ide` CI leg runs and tests nothing here | **P1** | | **G7 — no end-to-end drive of the runner's buildup / stall / saturation emit sites** | `_maybe_alert_buildup` (`:5401`), `_maybe_alert_saturation` (`:5444`), `_maybe_alert_stall` (`:5511`) in `wiring_runner.py`. Only the pure `SaturationDetector` and the **engine-shard** non-owned-lane watchdog are tested | These are the three alerts an operator relies on to notice a stalled or drowning feed. Threshold resolution, the `_outbound_paused` suppression guard, the per-`(stage,lane)` `_BUILDUP_REALERT_SECONDS` throttle and the `pending_depth` read could all break silently | **No** — FEATURE-COVERAGE-PLAN `FCP:ALERT-10` flagged it and it is still open | **P1** | @@ -238,7 +238,7 @@ observations whose falsifiable halves are carried by T rows (ALERT-18/40/41 and | ALERT-64 | DST-boundary correctness of schedule-aware rules | Functional | manual | W2025-box | n/a | T | P2 | A rule with an IANA-tz window spanning a DST transition activates and deactivates at the correct wall-clock local times on a real host clock across the boundary, in both the spring-forward and fall-back directions | | ALERT-65 | `docs/FEATURE-MAP.md` §9/§10 currency guard | Compat | — | dev-PC | n/a | T | P2 | **Pointer.** Covered by MIG's consolidated FEATURE-MAP drift-guard row (`MIG-28`, one extension of `tests/test_feature_map_claims.py`); no separate work scoped here. The alerting-specific claims this chapter hands to MIG as that row's inputs: §9 names **every** member of `_ALERT_EVENT_TYPES` (all **18** — `messagefoundry/config/settings.py`); §9 has a row for each of alert state, escalation, templates, per-rule recipients, control actions, test-email, support bundle, crashdump suppression, `connection_event`, `/logs/tail`, `/metrics/history`, host metrics; and **no** §10 row nor any line references a module path absent from disk (catches `messagefoundry/console` and the `:131` "PySide6 desktop console stays" claim) | | ALERT-66 | `docs/BACKLOG.md` #171 build-state correction | Functional | pytest | dev-PC | n/a | T | P2 | #171's banner no longer claims DEMAND-GATE / "no runtime control and no viewer"; the existing `tests/test_backlog_status_check.py` + `.github/workflows/backlog-hygiene.yml` pass on the edited entry, and the entry cites `api/app.py:4541`, `:4570` and ADR 0130 | -| ALERT-67 | `docs/PHI.md` row 11 vs `docs/BACKLOG.md:5152` — a CI guard binding both docs to the code's SMTP posture | Functional | pytest | dev-PC | n/a | T | P0 | The pass criterion is the **guard**, not the decision (OQ-3 decides *which* posture; the row can fail under either). `tests/test_phi_logging_inventory.py` gains an assertion that reads the observed `starttls` posture the way ALERT-08 pins it — verifying context vs `check_hostname=False`/`verify_mode=CERT_NONE` — and asserts (a) `docs/PHI.md` row 11's STARTTLS wording describes **that** posture, and (b) no other doc sentence contradicts it, so `docs/BACKLOG.md:5152`'s "uses STARTTLS with a verifying context by design" must either match the code or be gone. Fails today, because the two documents disagree; fails again on any future doc↔code divergence | +| ALERT-67 | `docs/PHI.md` row 11 vs `docs/BACKLOG.md:5152` — a CI guard binding both docs to the code's SMTP posture | Functional | pytest | dev-PC | n/a | T | P0 | The pass criterion is the **guard**, not the decision (OQ-3 decides *which* posture; the row can fail under either). `tests/test_phi_logging_inventory.py` gains an assertion that reads the observed `starttls` posture the way ALERT-08 pins it — verifying context vs `check_hostname=False`/`verify_mode=CERT_NONE` — and asserts (a) `docs/PHI.md` row 11's STARTTLS wording describes **that** posture, and (b) no other doc sentence contradicts it, so any doc sentence about this posture must either match the code or be gone. **THE GUARD IS STILL WORTH BUILDING; ITS STATED STARTING CONDITION IS NOT.** "Fails today, because the two documents disagree" was already false when written (BACKLOG #1100): #323 closed the divergence on 2026-08-02, and `docs/PHI.md` row 11 now states the verifying posture with the old one recorded as history. So this test is expected to PASS on arrival and earns its keep by failing on a FUTURE doc-code divergence — which is the only thing it was ever able to catch. Note also that the cited `docs/BACKLOG.md:5152` anchor has drifted; re-derive it by content rather than by line number | | ALERT-68 | The notifier itself is the thing that is down: an unreachable SMTP host and a 500-looping webhook block no pipeline stage, and the failure is itself observable | HA/Resilience | pytest | dev-PC | SQLite | T | P1 | Two fault modes driven through a live `RegistryRunner` with a real `NotifierAlertSink` and a live store: (a) `[alerts].email_smtp_host` pointed at a closed port so `send_plain_email` raises `ConnectionRefusedError`/`socket.gaierror` at (and at the far end of) the connect timeout; (b) a webhook target returning HTTP 500 to every POST — 50 consecutive events each. Assert for both: **no stage blocks** — every `_emit` returns sub-millisecond, the ingress→routed→outbound handoffs keep committing, and end-to-end message throughput stays within 5% of a no-alert control run (a transport that awaits inside `_emit`, or a retry loop on the emitting worker, fails here); the durable `alert_instance` row is still upserted `open`, so `/ui/alerts` shows the condition nobody was paged about; a healthy sibling transport still delivers every event (distinct from ALERT-21's *wedged* transport — here the transport **fails fast** rather than hanging); and the failure is observable — exactly one WARNING per event naming the transport and the event type, carrying neither recipient addresses nor webhook credentials, plus the ALERT-20 counter if OQ-9 adds one. A silent swallow with no record fails the row | ### 14.5 Detailed scenarios @@ -337,8 +337,16 @@ a synthetic mailbox; `email_use_tls = true`; `smtp_allowed_hosts` containing the 6. Cross-check every observation against ALERT-08's asserted posture. **Expected result.** For (a): all three succeed. For (b) and (c): the observed behaviour is -**identical to what ALERT-08 asserts** — today that means all three still succeed (unauthenticated -STARTTLS). Any divergence between this recorded matrix and the unit test's assertion is a finding. +**identical to what ALERT-08 asserts**. Any divergence between this recorded matrix and the unit +test's assertion is a finding. + +**THE PARENTHETICAL THAT USED TO STAND HERE WAS STALE WHEN WRITTEN (BACKLOG #1100):** it read "today +that means all three still succeed (unauthenticated STARTTLS)". #323 closed that on 2026-08-02 — +`send_plain_email` passes a verifying context — so a self-signed (b) and a hostname-mismatched (c) +cert are now expected to be **REJECTED**, not to succeed. Running this procedure against the old +sentence would have recorded three passes as the expected matrix and read a working control as a +finding, which is the inversion this row exists to prevent. Derive the expectation from ALERT-08's +assertion at run time rather than from any prose here. **Cleanup.** Restore cert (a); unlock the synthetic account; purge the synthetic mailbox. @@ -590,7 +598,8 @@ This area is signed off for release when **all** of the following hold: 1. **All 12 P0 (class-`T`) rows pass or are explicitly waived by the owner with a dated ADR entry:** ALERT-01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 58, 67. (ALERT-02 and ALERT-06 may exit as `xfail` **only** if OQ-1/OQ-2 resolve to "designed-not-built", and only with the ADR amendment merged.) 2. **The G1 mirror invariant is enforced in CI.** A planted emit type outside `_ALERT_EVENT_TYPES`, and a planted method on one of the three classes only, each fail the `test` leg. Falsifiability is itself asserted (mirror `tests/test_alerts_edit.py:160`). 3. **Zero PHI leaks.** ALERT-16, 17, 18, 45, 46, 48, 49 all pass; the ALERT-45 residual count is at or under its committed ceiling; the leak gate is fail-closed. -4. **The three contradicting documents agree with the code.** `docs/PHI.md` row 11, `docs/BACKLOG.md:5152` and the ALERT-08 assertion state one posture (ALERT-67); `docs/BACKLOG.md` #171 no longer claims unbuilt (ALERT-66); `docs/FEATURE-MAP.md` §9 names every routable event type and §10 references no non-existent module (ALERT-65). +4. **The documents agree with the code, and the SMTP-posture half is ALREADY SATISFIED.** `docs/PHI.md` row 11 and the ALERT-08 assertion state one posture (ALERT-67); `docs/BACKLOG.md` #171 no longer claims unbuilt (ALERT-66); `docs/FEATURE-MAP.md` §9 names every routable event type and §10 references no non-existent module (ALERT-65). + **This criterion previously read "the three CONTRADICTING documents agree with the code" and named a contradiction that #323 had already closed on 2026-08-02 (BACKLOG #1100).** An exit criterion demanding that a resolved contradiction be resolved is a gate that can never inform anyone: it cannot fail, so passing it says nothing, and a reader who trusts it believes a check ran. What remains genuinely gating is the *guard* (ALERT-67), which earns its keep on a future divergence rather than on this one. 5. **The four superseded FEATURE-COVERAGE-PLAN rows are recorded as closed** with the dated re-verification in 14.2, and the remaining open ones (`FCP:ALERT-3`, `FCP:ALERT-5`, `FCP:ALERT-9`, `FCP:ALERT-10`) map 1:1 onto ALERT-19, ALERT-22, ALERT-13, ALERT-11 here. 6. **Every P1 class-`T` row passes or carries a dated, owner-accepted waiver.** In particular the three that are pure *pins of known non-coverage* (ALERT-15 outbound saturation blind spot, ALERT-26 restart asymmetry, ALERT-47 support-bundle no-RBAC) must be **passing pins**, not waivers — a pin that is waived is worthless. 7. **The detectability sweep (ALERT-55) reports zero unnamed holes.** Any hole is either closed or listed by name in the release notes with an operator workaround. @@ -603,7 +612,8 @@ This area is signed off for release when **all** of the following hold: 1. **`content_match` reachability (ADR 0133 D3).** Should a Handler get a first-class exported way to raise it — an injected sink, or a `messagefoundry.alert_content(...)` export alongside `db_lookup`/`fhir_lookup` — or should ADR 0133 D3 be re-scoped as *designed, not built*? Today no Handler can reach it and `LoggingAlertSink` has no fallback method, so a fallback-path caller would `AttributeError`. **Blocks:** ALERT-02, 04, 05; whether `content_match` stays in `_ALERT_EVENT_TYPES`; the ADR 0133 AC-3 wording. 2. **`connection_started`.** Should a lane restart auto-resolve an open `connection_stopped`, or is manual operator resolve the intended workflow — and if the latter, should the dead `_AUTO_RESOLVE` key be removed? **Blocks:** ALERT-06, 07; and, downstream, ALERT-35's growth expectation (G16 is largely a consequence of this). -3. **Alert / security-notification SMTP posture.** Is the unauthenticated STARTTLS (`check_hostname=False`, `verify_mode=CERT_NONE`) an accepted residual, or must this path adopt the ADR 0092/0153 hop gradient like the EMAIL *destination* connector? `docs/PHI.md` row 11 and `docs/BACKLOG.md:5152` currently contradict each other and **one of them must be corrected either way**. **Blocks:** ALERT-08, 09, 58, 67 — and this is the chapter's only P0 with a real security consequence. +3. **Alert / security-notification SMTP posture — ANSWERED, AND IT WAS ANSWERED BEFORE THIS QUESTION WAS WRITTEN (BACKLOG #1100).** This asked whether the unauthenticated STARTTLS was an accepted residual or had to adopt the ADR 0092/0153 hop gradient, and asserted that `docs/PHI.md` row 11 and `docs/BACKLOG.md` "currently contradict each other". **Neither premise holds.** #323 closed it on 2026-08-02: `send_plain_email` passes an explicit verifying context, so the hop is authenticated. And the gradient half has a recorded answer too — per `docs/PHI.md` row 11 this path did **not** adopt the connector hop gradient; its deviations (`email_use_tls = false`, `email_tls_verify = false`) are gated by a `[security].allow_unverified_alert_smtp_tls` **acknowledgment switch at the serve gate**, which refuses to start on an enforcing PHI instance without it and `AUDIT`-logs the start with it. + **It therefore BLOCKS NOTHING.** ALERT-08, 09, 58 and 67 are unblocked and should be built against the posture as shipped. Leaving this open was the more expensive error of the two: an open P0 "with a real security consequence" reads as a live exposure, and this chapter's only such row was describing a fix. 4. **Should `[alerts]` become reloadable** (via `POST /config/reload` or a dedicated route), or is restart-only the intended contract — and if so, must the IDE editor **and** the console surface a `pending restart` state? **Blocks:** ALERT-10, 32, 33 and scenario S5. 5. **Leader-gating for `cert_expiry` / `secret_rotation` / `update_check` / `gcm_invocations`.** They report **cluster-wide** facts, unlike the per-node lane events ADR 0014 §4 reasons about, so an N-node cluster pages N times per condition. Gate them, dedupe at the store, or accept? Note `gcm_invocations.py:28-29` already argues against gating the *refill* — the alert is a separable question. **Blocks:** ALERT-27, 28, 63. 6. **Is the outbound saturation blind spot acceptable for release** (a healthy-but-behind lane is never sampled), or must BACKLOG #93's periodic owned-outbound depth sweep land first? **Blocks:** whether ALERT-15 is a pin or a bug; and one entry in the ALERT-55 sweep. From 6012781de619d080b51a08e7ba7832d418a8180a Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 02:07:55 -0500 Subject: [PATCH 02/66] docs(testing): retire two stale gap rows in the pipeline and connector test plans (BACKLOG #1100) Both rows were re-measured against the current tree, and both were making claims the code and docs had already overtaken. 02-pipeline-reliability.md: the Postgres 2-engine crash-and-restart row said tests/test_shard_recovery_postgres.py "runs nowhere". It runs -- in ci.yml's postgres-store job -- but that job is gated on schedule || workflow_dispatch || serverdb == 'true', so it does not run on a PR that touches no server-DB path. "Runs nowhere" understates the coverage and "runs in CI" would overstate it; only the gated form supports a decision about PIPE-01, which is what the row exists to inform. 05-connections-and-transports.md: the "CONNECTIONS.md contradicts the code" row is fully closed. SFTP-IN, REST-IN and SMTP-OUT are all marked shipped now; DB-IN agrees with FEATURE-MAP's "Production" rather than contradicting it; SOAP-IN is receive-only rather than planned; and all six connectors the row called undocumented have settings headings. FHIR-IN still reads "planned" and that is correct -- the facade is unshipped -- so the row's inference (the listener is built, therefore FHIR-IN is built) does not follow: shipping the substrate does not ship the facade. Both rows are rewritten in place rather than deleted, so the re-measurement is auditable and the tables keep their shape. All seven line citations in the connector row had drifted off their subjects and had to be re-found by content before any of them could be judged; that is the sixth recorded instance of the anchor rule this sweep has produced. --- docs/testing/master-test-plan/02-pipeline-reliability.md | 9 +++++++-- .../master-test-plan/05-connections-and-transports.md | 2 +- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/testing/master-test-plan/02-pipeline-reliability.md b/docs/testing/master-test-plan/02-pipeline-reliability.md index bbc5f0e9f..195896ddb 100644 --- a/docs/testing/master-test-plan/02-pipeline-reliability.md +++ b/docs/testing/master-test-plan/02-pipeline-reliability.md @@ -117,8 +117,13 @@ chapter or any downstream one: run `test_load_failover_{sqlserver,postgres}` with the harness setting **no** `claim_mode`, i.e. under the pooled default, and hard-gate zero acknowledged loss + `lane_inversions == 0`. 4. **Postgres 2-engine crash-and-restart recovery.** `FCP:STORE-10`'s "not built" is stale — - `tests/test_shard_recovery_postgres.py` exists (4 tests). Its problem is that it **runs nowhere** - (PIPE-01), not that it is missing. + `tests/test_shard_recovery_postgres.py` exists (4 tests). **CORRECTED (BACKLOG #1100): it does not + "run nowhere".** It runs in `ci.yml`'s `postgres-store` job, step *"Run the failover + + engine-shard recovery suites on real Postgres"* — but that job is gated on + `schedule || workflow_dispatch || changes.outputs.serverdb == 'true'`, so it does **not** run on a + PR that touches no server-DB path. The accurate statement is *runs only on the server-DB + path-gated leg*, which is what PIPE-01 is actually for. "Runs nowhere" understated the coverage and + "runs in CI" would overstate it; only the gated form supports a decision about PIPE-01. 5. **`accepts=` static fail-closed validation** — `FCP:PIPE-9` is closed by `test_accepts_seam.py` (the three static-validation negatives). 6. **Purity replay-equality harness** — `FCP:PIPE-14`'s replay half is closed by diff --git a/docs/testing/master-test-plan/05-connections-and-transports.md b/docs/testing/master-test-plan/05-connections-and-transports.md index 050dd95b3..1db9658bb 100644 --- a/docs/testing/master-test-plan/05-connections-and-transports.md +++ b/docs/testing/master-test-plan/05-connections-and-transports.md @@ -100,7 +100,7 @@ This chapter covers **every registered inbound/outbound connector and the connec | `FEATURE-MAP.md` §1 understates shipped capability | `:45` marks REST-IN/SOAP-IN/FHIR-IN deferred (ADR 0023 built); `:46` marks MLLP persistent deferred (ADR 0067 built). No row at all for Email, Direct, Timer, PassThrough, Loopback, the HTTP listener, compression, `no_ack`, or the forward proxy | The public capability catalogue is what a test author, an evaluator and an adopter scope from. Understated connectors go untested and unaudited; `tests/test_feature_map_claims.py` guards only the ASVS score and private links, so it rots further | No | P1 | | Listener `SSLContext` built once at construction | `mllp.py:1324`, `http_listener.py:292`, `dicom.py:178` all build the context in `__init__`, so a renewed cert on disk is not presented until a connection restart or config reload | An operator rotating a listener cert in place believes it took effect; when the old cert expires the listener stops accepting partners — a full intake outage — and the only prior signal is an expiry alert they may have silenced after "rotating" | No test or doc states it; `CONNECTIONS.md:714` carries rotation guidance for the DB CA only | P2 | | `peer_reset` and `framing_error` emitted but never asserted | `mllp.py:1471`/`:1476`, `tcp.py:547`/`:552`, `http_listener.py:391`/`:397`; `test_connection_event_emit.py` (8 tests) covers only the other six kinds | Peer resets and framing errors are the two kinds an operator most needs when diagnosing a flapping partner. A refactor that swallows them leaves the event log quietly incomplete with nothing failing | No | P2 | -| `docs/CONNECTIONS.md` contradicts the code | SFTP-IN/OUT "⏳ planned" (`:35-36`) vs `remotefile.py:1016-1017`; SOAP-IN/REST-IN/FHIR-IN "⏳ planned" (`:37`, `:39`, `:43`) vs the built listener; SMTP-OUT "⏳ planned" (`:50`) vs `email.py:265`; DB-IN "(SQL Server, exp.)" (`:41`) vs FEATURE-MAP "production". No settings sections for `Sftp()`/`Ftp()`, `Timer()`, `PassThrough()`, `Http()` or `Direct()` (heading list `:195-1372`) | Adopters and test authors read this file as the connector contract. A partner integration gets scoped away because the doc says "planned", and five shipped connectors have no documented settings surface to validate against — against CLAUDE.md §11 | No — `test_communications_inventory.py` requires only a resource-table row, not a settings heading | P2 | +| `docs/CONNECTIONS.md` contradicts the code — **CLOSED, re-measured 2026-08-15 (BACKLOG #1100)** | **Every citation in this row is stale, and the one doc line that still reads "planned" is right.** Measured: `SFTP-IN` `:42` is `✅ (Sftp(), [sftp] extra)`; `REST-IN` `:48` is `✅ (Http(), ADR 0023)`; `SMTP-OUT` `:57` is `✅ (Email()/SMTP(), ADR 0029)`; `DB-IN` `:50` is `✅ (SQL Server + generic ODBC)`, **agreeing with** `FEATURE-MAP.md:41` "Production"; `SOAP-IN` `:46` is `~ receive-only`, not "planned". All six connectors called undocumented now have settings headings: `Sftp()`/`Ftp()` `:845`, `Http()` `:533`, `Direct()` `:1385`, `Timer()` `:1794`, `PassThrough()` `:1867`. **`FHIR-IN` `:52` does still read `⏳ planned (BACKLOG #20)` — and that is accurate, not a contradiction:** `:2503` records the FHIR-IN facade as unshipped, "a consumer of the shipped `Http()` listener, not new substrate". This row inferred *the listener is built, therefore FHIR-IN is built*; shipping the substrate does not ship the facade | None — no work is scoped by this row. Kept rather than deleted so the re-measurement is auditable, and as the sixth recorded instance of the anchor rule: **all seven** line citations here (`:35-36`, `:37`, `:39`, `:41`, `:43`, `:50`, `:195-1372`) had drifted off their subjects, so each had to be re-found by content before it could be judged | n/a — closed | — | | SFTP key loading is RSA-only | `remotefile.py:411` calls `paramiko.RSAKey.from_private_key`; an Ed25519 or ECDSA partner key fails | Modern SFTP partners default to Ed25519. The failure is loud but undiagnosable from the docs, and nothing pins the constraint so it cannot silently narrow further | No | P2 | | Five ADR status lines lag their code | 0011 "Proposed"; 0015 and 0016 "No code written yet"; 0021 "No code yet"; 0022 "Design-only (no code yet)" — all verifiably built (`timer.py:341`; `soap.py:128-147`/`:406`; `test_x12_rte.py` + `ci.yml:702`/`:855`; `test_ack_capture_runner.py`; `fhir.py:647`) | Anyone scoping test work off ADR status either skips a built feature or plans to build one that exists. ADR 0020 shows status is used to record culls, so a wrong status is a governance defect | No | P2 | From 5faf5513269e8cafffaad9d202945b835bdfd15a Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 02:56:25 -0500 Subject: [PATCH 03/66] docs(testing): two P0 HA/DR rows say "runs nowhere" about suites that run (BACKLOG #1100) Both rows asserted that live server-DB suites are invoked by no workflow. Measured against ci.yml by parsing the YAML rather than grepping it: Row 80: tests/test_cluster_failover_postgres.py runs at ci.yml:1570, in the postgres-store job's step "Run the failover + engine-shard recovery suites on real Postgres". The anchor the row cites as its evidence, ci.yml:614, has drifted and now lands on a comment about arithmetic. Row 81: all six DR suites exist and all six are invoked -- the Postgres trio by postgres-store, the SQL Server trio by sqlserver-store, each in a step named for exactly those suites. BOTH STAY P0. The residual risk is real, but it is a GATING risk rather than an absence: both jobs carry schedule || workflow_dispatch || serverdb == 'true', so a PR that changes election or DR logic without touching a server-DB path still ships unverified. "Detected today?" moves from "No" to "Partly -- on the gated server-DB leg only, never on an ordinary PR", which is the form that supports a decision. "No" understates the coverage and a bare "yes, it runs" would overstate it. Row 81's MEFOR_TEST_* env gating is a SECOND, INDEPENDENT condition and was not re-measured here. The row now says so, so that correcting one half of its claim does not read as clearing the whole row. Found by transferring probe 6's shape rather than by a new search: the identical "runs nowhere" phrasing about a sibling Postgres suite was already falsified in 02-pipeline-reliability.md, and the two test files turn out to sit on consecutive lines of the same CI step. A defect class, once named, is worth re-running across the corpus -- this is the third and fourth instance of it. --- docs/testing/master-test-plan/04-high-availability-and-dr.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/testing/master-test-plan/04-high-availability-and-dr.md b/docs/testing/master-test-plan/04-high-availability-and-dr.md index cceceefaf..a8bdd6f88 100644 --- a/docs/testing/master-test-plan/04-high-availability-and-dr.md +++ b/docs/testing/master-test-plan/04-high-availability-and-dr.md @@ -77,8 +77,8 @@ bare `HA-nn` in this chapter always means **this chapter's own §3.4 row** — t | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| The live PostgreSQL election suite runs nowhere | A changed predicate, a dropped `ON CONFLICT`/`WHERE` clause, or an epoch that bumps on renew ships green. `tests/test_cluster_failover_postgres.py` (9 live scenarios) is referenced by **no** workflow — grep of `.github/` finds only the SQL Server twin at `ci.yml:614`. | Two leaders, duplicate delivery, or a cluster that never fails over — on the **default** server backend | No | P0 | -| Every live server-DB DR suite runs nowhere | `test_dr_server_seed_gate_{postgres,sqlserver}.py`, `test_dr7_server_config_only_backup_{postgres,sqlserver}.py`, `test_backup_runner_server_db_{postgres,sqlserver}.py` are `MEFOR_TEST_*`-gated and invoked by no leg. | DR activation promotes onto a **fresh/unrestored** server DB — precisely the data-loss case BACKLOG #102 exists to close | No | P0 | +| The live PostgreSQL election suite runs only on the server-DB gated leg | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): "runs nowhere" and "referenced by no workflow" were both false.** `tests/test_cluster_failover_postgres.py` is invoked at `ci.yml:1570`, in the `postgres-store` job's step *"Run the failover + engine-shard recovery suites on real Postgres"* (`pytest -v tests/test_cluster_failover_postgres.py tests/test_shard_recovery_postgres.py`). The cited anchor `ci.yml:614` has drifted and now lands on a comment. **The residual risk is real and stays P0, but it is a GATING risk, not an absence:** the job runs only on `schedule \|\| workflow_dispatch \|\| needs.changes.outputs.serverdb == 'true'`, so a PR that changes election logic without touching a server-DB path still ships unverified. A changed predicate, a dropped `ON CONFLICT`/`WHERE`, or an epoch that bumps on renew is caught nightly, not at review time | Two leaders, duplicate delivery, or a cluster that never fails over — on the **default** server backend | Partly — on the gated server-DB leg only, never on an ordinary PR | P0 | +| Every live server-DB DR suite runs only on the server-DB gated leg | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): "invoked by no leg" was false for all six.** `test_dr_server_seed_gate_{postgres,sqlserver}.py`, `test_dr7_server_config_only_backup_{postgres,sqlserver}.py` and `test_backup_runner_server_db_{postgres,sqlserver}.py` all exist and are all invoked — the Postgres trio by the `postgres-store` job's step *"Run the DR seed-gate + backup suites on real Postgres"*, the SQL Server trio by the `sqlserver-store` job's *"...on real SQL Server"*. **Same correction as the row above, and it stays P0 for the same reason:** both jobs carry the `schedule \|\| workflow_dispatch \|\| serverdb == 'true'` gate, so DR regressions are caught nightly rather than at review time. The `MEFOR_TEST_*` env gating is a second, independent condition and was *not* re-measured here — it is the part of this row that may still bite | DR activation promotes onto a **fresh/unrestored** server DB — precisely the data-loss case BACKLOG #102 exists to close | Partly — on the gated server-DB leg only, never on an ordinary PR | P0 | | A leaderless cluster pages nobody | `leadership_lost` is auto-resolve-only and deliberately emits no notification (`pipeline/alerts.py:206-213`); the allowed alert-type set `_ALERT_EVENT_TYPES` (`config/settings.py:2499-2526`, **18** members) has no "no leader" member. All nodes self-fenced (DB outage), all nodes `promotable=false`, or a lease nobody can take → total stop. | Full silent intake stop with a filling staged queue; only surface is a console badge someone must be looking at | No | P0 | | Engine shards and `[cluster]` collide on one lease | `_lease_key = f"{db_schema or 'public'}:mefor_cluster_leader"` (`cluster.py:414`) is **not** engine-shard-namespaced, and no validator relates `serve --shard` to `[cluster].enabled`. N engine-shard processes over one unified store would contend for the same row; N-1 bind nothing. | Silent full stop for every engine shard but one — looks like a healthy cluster on `/cluster/status` | No | P0 | | Cross-store divergence after a DR fail-back | `pipeline/dr.py:296-301` states the engine gives no cross-store loss/duplicate guarantee and defers to a runbook. `harness/reconcile/` exists (capture / compare / normalize / report) but nothing binds it to the DR release path. | Messages processed only at DR, or reprocessed at both — silent clinical-data divergence with no failing test and no operator artifact | No | P0 | From 6111dfe6f3823f220b7295dd6d4e33d9bbd09913 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 03:04:45 -0500 Subject: [PATCH 04/66] docs(testing): the HTTP listener does authenticate -- correct a P1 row that says it cannot (BACKLOG #1100) 05-connections-and-transports.md row 98 claimed "Inbound HTTP listener has no application-layer authentication", evidenced by "grep for auth/Authorization/ bearer/hmac in http_listener.py returns 0 hits". EVERY PART OF THAT EVIDENCE IS NOW FALSE: the same grep measures 52 matching lines; Http() does not expose "caps + TLS/mTLS only"; ADR 0023 no longer defers auth -- ADR 0154 (Accepted 2026-07-31) is titled its "deferred tail" and delivered it. Built and measured: intake_auth: Literal["none","api_key","bearer", "mtls_subject"], enforced by _authorize_head (http_listener.py:618) with a 401 plus WWW-Authenticate, credentials env()-only, rotation via intake_api_key_next, health probes inside the gate by default, and failed-attempt rate limits at 10/min/peer and 60/min global. The row's "Covered?" cell is equally stale: there is a dedicated tests/test_inbound_http_intake_auth.py plus four more files, 18 test functions naming the surface. THE ROW STAYS P1 BECAUSE ITS CONCERN SURVIVES ITS EVIDENCE, IN A NARROWER FORM. intake_auth defaults to "none" and _authorize_head returns early when unset, so a site deploying the listener off-loopback without setting it would accept unauthenticated POSTs. That is now an explicit, documented, tested opt-in rather than an implicit omission -- a different decision, and the only one still open is whether "none" is the right default for a PHI intake path. Written in the conditional per CLAUDE.md section 0: there are no deployments, so this is what a deploying site would hit, not something happening now. Found by a reversed discriminator: rows asserting a grep returns ZERO hits go stale precisely because code gets added. Four such claims exist in the plan; this is the one that broke, and the other three (ADR coverage-plan hits in 05:95 and 09:122, and 0 occurrences of "validate" in 06:86) were re-measured and HOLD. --- docs/testing/master-test-plan/05-connections-and-transports.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/05-connections-and-transports.md b/docs/testing/master-test-plan/05-connections-and-transports.md index 1db9658bb..d1f8b7fb8 100644 --- a/docs/testing/master-test-plan/05-connections-and-transports.md +++ b/docs/testing/master-test-plan/05-connections-and-transports.md @@ -95,7 +95,7 @@ This chapter covers **every registered inbound/outbound connector and the connec | Eight post-plan connector ADRs have no coverage-plan row | Verified zero grep hits each for 0122, 0123, 0124, 0126, 0128, 0129, 0132, 0153 in `FEATURE-COVERAGE-PLAN.md` | Each has unit tests, but nothing owns their cross-backend, live-partner, on-box or PHI dimensions and no plan section notices a regression. The audit is point-in-time with no drift guard against the ADR ledger, so the hole widens with every demand-gate wave | No | P1 | | ADR 0124 `no_ack` finalizes on a TCP write | `_send_once_no_ack:897` / `_send_persistent_no_ack:1046` complete a delivery on a successful write with no ACK read and no MSA-1 validation | A `no_ack` outbound reports `PROCESSED` for a message the partner never processed. Operators read disposition as delivery confirmation, so the at-most-once-confirmation trade-off needs an explicit tested disposition and an operator-visible marker | No — `test_mllp_no_ack.py` and `test_no_ack_wiring.py` stop at the connector and wiring | P1 | | Five shipped connectors absent from the on-box §D matrix | Email/SMTP, Direct S/MIME, DICOMweb STOW-RS, the ADR 0023 HTTP listener and PassThrough have no `WIN2025-TEST-MATRIX.md` §D row (lines 47-59) | The box gate cannot certify connectors it does not list. Direct and Email are PHI **egress** paths and the HTTP listener is a PHI **intake** path, so `W25:S6.3` would declare the box accepted with three PHI paths untested on the host | No | P1 | -| Inbound HTTP listener has no application-layer authentication | `Http()` (`wiring.py:1063`) exposes caps + TLS/mTLS only; grep for auth/Authorization/bearer/hmac in `http_listener.py` returns **0** hits; ADR 0023 defers auth | A partner-facing HTTP PHI intake accepting unauthenticated POSTs is a security decision that is currently implicit. An operator exposing it off-loopback with `tls=true` but no `tls_ca_file` lets any host reaching the port inject messages into the pipeline, and nothing fails | No test states the accepted control set (mTLS + `source_ip_allowlist` + bind guard) as a deliberate posture | P1 | +| Inbound HTTP listener authenticates, but **defaults to `none`** | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15). The evidence in this row was false in every part.** "grep ... returns **0** hits" measures **52** matching lines today; `Http()` does *not* expose "caps + TLS/mTLS only"; and ADR 0023 no longer defers auth — **ADR 0154 (Accepted 2026-07-31) is literally titled its "deferred tail"** and delivered it. Built: `intake_auth: Literal["none","api_key","bearer","mtls_subject"]`, enforced by `_authorize_head` (`http_listener.py:618`) with a 401 + `WWW-Authenticate`, credentials `env()`-only, rotation via `intake_api_key_next`, health probes inside the gate by default (`intake_auth_health="require"`), and failed-attempt rate limits (10/min/peer, 60/min global) | **The mechanism is no longer the gap; the DEFAULT is.** `intake_auth` defaults to `"none"`, and `_authorize_head` returns early when it is unset — so a site that deploys the listener off-loopback without setting it **would** accept unauthenticated POSTs into the pipeline. That is now an explicit, documented, tested opt-in rather than an implicit omission, which is a different and much narrower decision: whether `"none"` is the right default for a PHI intake path | **Yes** — `tests/test_inbound_http_intake_auth.py` plus `test_auth_entry_hardening.py`, `test_credential_parameter_mapping.py`, `test_connection_factory_redaction_domain.py` and `test_inbound_http_sync_reply.py` (18 test functions naming the surface). What remains unasserted is only the **default-posture** question above | P1 | | No negative mTLS handshake on any accept path | `test_mllp_tls.py:83` asserts `verify_mode` on the constructed context, not a live refusal. Nothing drives a client with no cert or a wrong-CA cert at an MLLP / HTTP-listener / DICOM-SCP mTLS listener | mTLS is the only peer authentication the raw listeners and the HTTP listener have. A regression leaving `CERT_OPTIONAL` in place admits unauthenticated PHI intake with no failing test | Owned as `FCP:MLLP-9`'s open gap; unclosed | P1 | | `FEATURE-MAP.md` §1 understates shipped capability | `:45` marks REST-IN/SOAP-IN/FHIR-IN deferred (ADR 0023 built); `:46` marks MLLP persistent deferred (ADR 0067 built). No row at all for Email, Direct, Timer, PassThrough, Loopback, the HTTP listener, compression, `no_ack`, or the forward proxy | The public capability catalogue is what a test author, an evaluator and an adopter scope from. Understated connectors go untested and unaudited; `tests/test_feature_map_claims.py` guards only the ASVS score and private links, so it rots further | No | P1 | | Listener `SSLContext` built once at construction | `mllp.py:1324`, `http_listener.py:292`, `dicom.py:178` all build the context in `__init__`, so a renewed cert on disk is not presented until a connection restart or config reload | An operator rotating a listener cert in place believes it took effect; when the old cert expires the listener stops accepting partners — a full intake outage — and the only prior signal is an expiry alert they may have silenced after "rotating" | No test or doc states it; `CONNECTIONS.md:714` carries rotation guidance for the DB CA only | P2 | From 375e543b921062a9181a02a6d14d6d0e07becbb8 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 03:13:25 -0500 Subject: [PATCH 05/66] docs(testing): the lagging-ADR-status count is four, not five (BACKLOG #1100) 05-connections-and-transports.md row 105 said five ADR status lines contradict their code. Re-measured against the ADR files, both halves of each claim: 0011 "Proposed" transports/timer.py, 341 lines LAGS 0015 "No code written yet" transports/soap.py, 844 lines LAGS 0016 "No code written yet" tests/test_x12_rte.py, 625 lines LAGS 0022 "Design-only (no code yet)" test_ack_capture_runner.py, fhir.py LAGS 0021 "No code yet" -> NOW READS "Built and shipped" FIXED ADR 0021's status has been corrected since the row was written and now reads "Accepted (2026-06-19, owner go). Built and shipped (eventlog #16, Lane S)". It is struck from the count rather than left inflating it. The row keeps its point and its priority: four is still four, and a wrong ADR status is still a governance defect. What changes is that a count carried forward without re-measurement drifts upward relative to reality -- 0021 is itself the evidence that these get fixed one at a time, so the count has to be re-derived rather than quoted. Found by generalising the discriminator that broke row 98 in the same file: that row cited "ADR 0023 defers auth" when ADR 0154, titled ADR 0023's "deferred tail", had closed it. A plan row citing an ADR's status is asserting something about a FILE THAT MOVES INDEPENDENTLY of the plan, which makes it stale by construction. --- docs/testing/master-test-plan/05-connections-and-transports.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/05-connections-and-transports.md b/docs/testing/master-test-plan/05-connections-and-transports.md index d1f8b7fb8..ccdd5e529 100644 --- a/docs/testing/master-test-plan/05-connections-and-transports.md +++ b/docs/testing/master-test-plan/05-connections-and-transports.md @@ -102,7 +102,7 @@ This chapter covers **every registered inbound/outbound connector and the connec | `peer_reset` and `framing_error` emitted but never asserted | `mllp.py:1471`/`:1476`, `tcp.py:547`/`:552`, `http_listener.py:391`/`:397`; `test_connection_event_emit.py` (8 tests) covers only the other six kinds | Peer resets and framing errors are the two kinds an operator most needs when diagnosing a flapping partner. A refactor that swallows them leaves the event log quietly incomplete with nothing failing | No | P2 | | `docs/CONNECTIONS.md` contradicts the code — **CLOSED, re-measured 2026-08-15 (BACKLOG #1100)** | **Every citation in this row is stale, and the one doc line that still reads "planned" is right.** Measured: `SFTP-IN` `:42` is `✅ (Sftp(), [sftp] extra)`; `REST-IN` `:48` is `✅ (Http(), ADR 0023)`; `SMTP-OUT` `:57` is `✅ (Email()/SMTP(), ADR 0029)`; `DB-IN` `:50` is `✅ (SQL Server + generic ODBC)`, **agreeing with** `FEATURE-MAP.md:41` "Production"; `SOAP-IN` `:46` is `~ receive-only`, not "planned". All six connectors called undocumented now have settings headings: `Sftp()`/`Ftp()` `:845`, `Http()` `:533`, `Direct()` `:1385`, `Timer()` `:1794`, `PassThrough()` `:1867`. **`FHIR-IN` `:52` does still read `⏳ planned (BACKLOG #20)` — and that is accurate, not a contradiction:** `:2503` records the FHIR-IN facade as unshipped, "a consumer of the shipped `Http()` listener, not new substrate". This row inferred *the listener is built, therefore FHIR-IN is built*; shipping the substrate does not ship the facade | None — no work is scoped by this row. Kept rather than deleted so the re-measurement is auditable, and as the sixth recorded instance of the anchor rule: **all seven** line citations here (`:35-36`, `:37`, `:39`, `:41`, `:43`, `:50`, `:195-1372`) had drifted off their subjects, so each had to be re-found by content before it could be judged | n/a — closed | — | | SFTP key loading is RSA-only | `remotefile.py:411` calls `paramiko.RSAKey.from_private_key`; an Ed25519 or ECDSA partner key fails | Modern SFTP partners default to Ed25519. The failure is loud but undiagnosable from the docs, and nothing pins the constraint so it cannot silently narrow further | No | P2 | -| Five ADR status lines lag their code | 0011 "Proposed"; 0015 and 0016 "No code written yet"; 0021 "No code yet"; 0022 "Design-only (no code yet)" — all verifiably built (`timer.py:341`; `soap.py:128-147`/`:406`; `test_x12_rte.py` + `ci.yml:702`/`:855`; `test_ack_capture_runner.py`; `fhir.py:647`) | Anyone scoping test work off ADR status either skips a built feature or plans to build one that exists. ADR 0020 shows status is used to record culls, so a wrong status is a governance defect | No | P2 | +| **Four** ADR status lines lag their code (**was five** — re-measured 2026-08-15, BACKLOG #1100) | Still lagging, each re-verified on **both** halves — the status line AND the code it contradicts: **0011** "Proposed" (`transports/timer.py`, 341 lines); **0015** "**No code written yet**" (`transports/soap.py`, 844 lines); **0016** "**No code written yet**" (`tests/test_x12_rte.py`, 625 lines, referenced twice in `ci.yml`); **0022** "Design-only (no code yet)" (`tests/test_ack_capture_runner.py`, `fhir.py`). **0021 HAS SINCE BEEN CORRECTED** — its status now reads "Accepted (2026-06-19, owner go). Built and shipped (`eventlog` #16, Lane S)" — so it is struck from the count rather than left inflating it | Anyone scoping test work off ADR status either skips a built feature or plans to build one that exists. ADR 0020 shows status is used to record culls, so a wrong status is a governance defect. **0021 is itself the evidence that these get fixed one at a time**, which is why the count must be re-measured rather than carried forward | No | P2 | ### 4.4 Test matrix From dae4d1fa15a18abeb34d6f7136934cbfd2c0d0c9 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 03:34:17 -0500 Subject: [PATCH 06/66] docs(testing): mark the HA catalog-drift row verified-live and fix its one drifted anchor (BACKLOG #1100) 04-high-availability-and-dr.md row 97 was probed for staleness and turned out to be CORRECT. Every claim re-measured: FEATURE-MAP omits ADR 0047/0048/0049/0056/0096 0 grep hits each TRUE BACKLOG #100 ranked Tier cell still reads "P1" BACKLOG.md:373 TRUE BACKLOG #101 ranked Tier cell reads DEMAND-GATE BACKLOG.md:380 TRUE ADR 0096 is Accepted TRUE code is in settings.py:2896-2910 that range is now ADR 0014 alert rules DRIFTED Only the code anchor was wrong. The ADR 0096 surface is `promotable: bool = True` at settings.py:3026, with its explanation from :3016, so the citation is corrected to settings.py:3016-3026 and named rather than left as a bare line range. The row is annotated as re-verified rather than rewritten, because a sweep that only marks stale rows leaves the next reader unable to tell "checked and live" from "not yet checked". Both are useful; only one of them was recorded until now. NOTE FOR WHOEVER OWNS THE LEDGER: the two Tier cells are a real, still-open defect -- two archived/shipped items carrying pre-shipping scores in the ranked backlog. They live in docs/BACKLOG.md, which a Builder may not edit under the owner ruling, so this commit records and locates the defect without touching it. Found by probing plan rows that cite a BACKLOG item as open. 28 raw hits, of which nearly all were co-occurrence rather than claims -- "#1"/"#2" were "synthetic ADT #1", and one trigger was the word "pending" in unrelated prose. One row carried real, checkable ledger claims, and it was right. --- docs/testing/master-test-plan/04-high-availability-and-dr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/04-high-availability-and-dr.md b/docs/testing/master-test-plan/04-high-availability-and-dr.md index a8bdd6f88..553018bfe 100644 --- a/docs/testing/master-test-plan/04-high-availability-and-dr.md +++ b/docs/testing/master-test-plan/04-high-availability-and-dr.md @@ -94,7 +94,7 @@ bare `HA-nn` in this chapter always means **this chapter's own §3.4 row** — t | The PG store-side epoch guard is live-proven nowhere | `test_resumed_ex_leader_is_fenced_after_real_handover` exists in both live suites but only the SS one runs. The PG claim guard appears in **three** separate statements (`postgres.py:2686-2688, 2816-2820, 2934-2938`). | One missed append across three claim paths is invisible to fake-pool tests and shows up only as duplicate/out-of-order delivery under a real handover | No | P1 | | DR readiness is asserted nowhere on the box that needs it | `messagefoundry verify`'s sections are exactly `("host", "store", "smoke", "manual", "federation")` (`messagefoundry/verify/runner.py:22`) — no cluster, HA or DR group. `harness/acceptance/matrix.py:429-470` has G1-G5 with no DR row; `WIN2025-TEST-MATRIX.md` §G likewise. | An operator gets zero signal about DEK availability at the DR site, backup-destination writability, or cluster posture | No | P1 | | Windows port-rebind lag is a one-off manual number | Captured only as prose in `W25:S4.9`; not in the report JSON, no trend, no threshold. | On the intended production OS this dominates observed recovery; a regression is indistinguishable from normal | Manual only | P2 | -| Catalog drift understates the area | `FEATURE-MAP.md` §6's nine rows omit ADR 0047/0048/0049/0056/0096, `/cluster/nodes`, the H1 fence, on-promotion recovery and the DR run-profile. BACKLOG #100's ranked-backlog Tier cell still reads "P1" though it shipped 2026-07-10 per its own SHIPPED banner; BACKLOG #101's ranked-backlog Tier cell reads "DEMAND-GATE" though ADR 0096 is Accepted and the code is in `settings.py:2896-2910`. | Readers conclude DR standby, turnkey backup and cloud HA packaging do not exist, so they are never scoped for testing or release notes | No | P2 | +| Catalog drift understates the area | `FEATURE-MAP.md` §6's nine rows omit ADR 0047/0048/0049/0056/0096, `/cluster/nodes`, the H1 fence, on-promotion recovery and the DR run-profile. BACKLOG #100's ranked-backlog Tier cell still reads "P1" though it shipped 2026-07-10 per its own SHIPPED banner; BACKLOG #101's ranked-backlog Tier cell reads "DEMAND-GATE" though ADR 0096 is Accepted and the code is in `settings.py:3016-3026` (`promotable: bool = True`). **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100) — this row is CORRECT and its gap is still open**, which is why it is annotated rather than rewritten: FEATURE-MAP still returns **0 hits** for each of ADR 0047/0048/0049/0056/0096; #100's ranked Tier cell still reads "P1" (`BACKLOG.md:373`, rank 14) though the item is archived as shipped; #101's still reads "DEMAND-GATE" (`:380`). **Only the code anchor had drifted** — it cited `settings.py:2896-2910`, which is now ADR 0014 alert-rule code, and is corrected above. **The two Tier cells live in `docs/BACKLOG.md`, which a Builder may not edit (owner ruling), so fixing them is the owner's or the Dispatcher's** | Readers conclude DR standby, turnkey backup and cloud HA packaging do not exist, so they are never scoped for testing or release notes | No | P2 | ### 3.4 Test matrix From 04ddf1a441a7cfe2acc35b3ee490569b6863ec41 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 04:03:03 -0500 Subject: [PATCH 07/66] docs(testing): verify the leaderless-cluster P0 row and re-anchor its drifted citation (BACKLOG #1100) 04-high-availability-and-dr.md row 82 was probed on the assumption that its "18 members" count had drifted, since counts were the class that broke row 105. IT HAD NOT. Measured via AST rather than a text window: _ALERT_EVENT_TYPES frozenset, settings.py:2599, EXACTLY 18 members only leader-related member is leadership_acquired -- nothing fires on ABSENCE alerts.py:206-213 still lands on leadership_lost, "Emits no notification" So the row is correct on every claim and the P0 gap it describes is real and open: a cluster that loses leadership entirely pages nobody, because the alert vocabulary has a member for ACQUIRING leadership and none for having none. Only the settings anchor was stale (cited :2499-2526, declared at :2599). It is now cited by NAME plus declaration line rather than as a range, because a range drawn around a growing frozenset re-drifts the next time a member is added -- which is precisely how this one drifted. The row is annotated as re-verified rather than left silent, matching the practice adopted in c4d6030a: a sweep that marks only stale rows cannot tell the next reader "checked, still true" from "not yet checked". --- docs/testing/master-test-plan/04-high-availability-and-dr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/04-high-availability-and-dr.md b/docs/testing/master-test-plan/04-high-availability-and-dr.md index 553018bfe..08d11b14c 100644 --- a/docs/testing/master-test-plan/04-high-availability-and-dr.md +++ b/docs/testing/master-test-plan/04-high-availability-and-dr.md @@ -79,7 +79,7 @@ bare `HA-nn` in this chapter always means **this chapter's own §3.4 row** — t |---|---|---|---|---| | The live PostgreSQL election suite runs only on the server-DB gated leg | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): "runs nowhere" and "referenced by no workflow" were both false.** `tests/test_cluster_failover_postgres.py` is invoked at `ci.yml:1570`, in the `postgres-store` job's step *"Run the failover + engine-shard recovery suites on real Postgres"* (`pytest -v tests/test_cluster_failover_postgres.py tests/test_shard_recovery_postgres.py`). The cited anchor `ci.yml:614` has drifted and now lands on a comment. **The residual risk is real and stays P0, but it is a GATING risk, not an absence:** the job runs only on `schedule \|\| workflow_dispatch \|\| needs.changes.outputs.serverdb == 'true'`, so a PR that changes election logic without touching a server-DB path still ships unverified. A changed predicate, a dropped `ON CONFLICT`/`WHERE`, or an epoch that bumps on renew is caught nightly, not at review time | Two leaders, duplicate delivery, or a cluster that never fails over — on the **default** server backend | Partly — on the gated server-DB leg only, never on an ordinary PR | P0 | | Every live server-DB DR suite runs only on the server-DB gated leg | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): "invoked by no leg" was false for all six.** `test_dr_server_seed_gate_{postgres,sqlserver}.py`, `test_dr7_server_config_only_backup_{postgres,sqlserver}.py` and `test_backup_runner_server_db_{postgres,sqlserver}.py` all exist and are all invoked — the Postgres trio by the `postgres-store` job's step *"Run the DR seed-gate + backup suites on real Postgres"*, the SQL Server trio by the `sqlserver-store` job's *"...on real SQL Server"*. **Same correction as the row above, and it stays P0 for the same reason:** both jobs carry the `schedule \|\| workflow_dispatch \|\| serverdb == 'true'` gate, so DR regressions are caught nightly rather than at review time. The `MEFOR_TEST_*` env gating is a second, independent condition and was *not* re-measured here — it is the part of this row that may still bite | DR activation promotes onto a **fresh/unrestored** server DB — precisely the data-loss case BACKLOG #102 exists to close | Partly — on the gated server-DB leg only, never on an ordinary PR | P0 | -| A leaderless cluster pages nobody | `leadership_lost` is auto-resolve-only and deliberately emits no notification (`pipeline/alerts.py:206-213`); the allowed alert-type set `_ALERT_EVENT_TYPES` (`config/settings.py:2499-2526`, **18** members) has no "no leader" member. All nodes self-fenced (DB outage), all nodes `promotable=false`, or a lease nobody can take → total stop. | Full silent intake stop with a filling staged queue; only surface is a console badge someone must be looking at | No | P0 | +| A leaderless cluster pages nobody | `leadership_lost` is auto-resolve-only and deliberately emits no notification (`pipeline/alerts.py:206-213`); the allowed alert-type set `_ALERT_EVENT_TYPES` (`config/settings.py:2599`, a `frozenset`, **18** members) has no "no leader" member. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100): both counts and both claims are EXACTLY right** — the set has 18 members and `leadership_acquired` is the only leader-related one, so nothing fires when leadership is *absent*; the `alerts.py:206-213` anchor still lands on `leadership_lost` ("Emits **no** notification"). Only the settings anchor had drifted (cited `:2499-2526`, declared at `:2599`) and is corrected; it is cited by **name and declaration line** now, because a range around a growing `frozenset` re-drifts on the next member added. All nodes self-fenced (DB outage), all nodes `promotable=false`, or a lease nobody can take → total stop. | Full silent intake stop with a filling staged queue; only surface is a console badge someone must be looking at | No | P0 | | Engine shards and `[cluster]` collide on one lease | `_lease_key = f"{db_schema or 'public'}:mefor_cluster_leader"` (`cluster.py:414`) is **not** engine-shard-namespaced, and no validator relates `serve --shard` to `[cluster].enabled`. N engine-shard processes over one unified store would contend for the same row; N-1 bind nothing. | Silent full stop for every engine shard but one — looks like a healthy cluster on `/cluster/status` | No | P0 | | Cross-store divergence after a DR fail-back | `pipeline/dr.py:296-301` states the engine gives no cross-store loss/duplicate guarantee and defers to a runbook. `harness/reconcile/` exists (capture / compare / normalize / report) but nothing binds it to the DR release path. | Messages processed only at DR, or reprocessed at both — silent clinical-data divergence with no failing test and no operator artifact | No | P0 | | RTO is unbounded in absolute terms | `harness/load/failover.py:640-646` bounds `functional_recovery_seconds` only as `recovery_ttl_multiple × leader_lease_ttl_seconds` (2.0 in `failover.toml`, 3.0 in `_failover_load_support.py`); `max_promotion_seconds` is optional (`profile.py:129-131`) and set in **neither** profile. No RPO figure is asserted anywhere. | Promotion can regress from ~7s to ~60s with every SLO green, because the bound scales with whatever TTL the profile sets | Partially (reported, never gated) | P1 | From 4352494422ec83228bc8d5b010705feb8eb2c32b Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 04:13:42 -0500 Subject: [PATCH 08/66] docs(testing): close a P0 that describes a failure the engine refuses at startup (BACKLOG #1100) 04-high-availability-and-dr.md row 83 claimed engine shards and [cluster] "collide on one lease", on the strength of two claims. The load-bearing one is false. "no validator relates serve --shard to [cluster].enabled" THERE IS ONE, IT FAILS CLOSED, AND IT IS TESTED. ADR 0073 makes them mutually exclusive; __main__.py:2649-2659 refuses the combination with exit 2 BEFORE the store or app is built; and test_shard_recovery_engine.py:321::test_serve_refuses_shard_with_cluster_enabled asserts rc == 2. Executed here: 1 passed. "_lease_key is not engine-shard-namespaced" STILL TRUE, and the anchor drifted (cited cluster.py:414, actual :484). But it is MOOT for this failure mode: N engine-shard processes can never come up against an enabled [cluster] to contend at all. The stated blast radius -- "silent full stop for every engine shard but one, looking healthy on /cluster/status" -- CANNOT OCCUR. A fail-closed startup refusal is the opposite of silent, so the row is closed rather than narrowed. That makes it the first row in this sweep whose concern did not survive its evidence; the previous nine were all narrowings. What remains is a design note, not a gap: the un-namespaced lease key matters only if ADR 0073's mutual exclusion is ever lifted, so it is a CONSTRAINT TO PRESERVE and is recorded as such. Also noted in the row: this satisfies the HA-03 exit criterion at :399 by its FIRST alternative (a config-load rejection, not an engine-shard-namespaced lease key) -- though the assertion lives in test_shard_recovery_engine.py rather than the tests/test_cluster_shard_interaction.py that the criterion names, so anyone grepping for that filename would wrongly conclude the criterion is unmet. --- docs/testing/master-test-plan/04-high-availability-and-dr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/04-high-availability-and-dr.md b/docs/testing/master-test-plan/04-high-availability-and-dr.md index 08d11b14c..3695cea9d 100644 --- a/docs/testing/master-test-plan/04-high-availability-and-dr.md +++ b/docs/testing/master-test-plan/04-high-availability-and-dr.md @@ -80,7 +80,7 @@ bare `HA-nn` in this chapter always means **this chapter's own §3.4 row** — t | The live PostgreSQL election suite runs only on the server-DB gated leg | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): "runs nowhere" and "referenced by no workflow" were both false.** `tests/test_cluster_failover_postgres.py` is invoked at `ci.yml:1570`, in the `postgres-store` job's step *"Run the failover + engine-shard recovery suites on real Postgres"* (`pytest -v tests/test_cluster_failover_postgres.py tests/test_shard_recovery_postgres.py`). The cited anchor `ci.yml:614` has drifted and now lands on a comment. **The residual risk is real and stays P0, but it is a GATING risk, not an absence:** the job runs only on `schedule \|\| workflow_dispatch \|\| needs.changes.outputs.serverdb == 'true'`, so a PR that changes election logic without touching a server-DB path still ships unverified. A changed predicate, a dropped `ON CONFLICT`/`WHERE`, or an epoch that bumps on renew is caught nightly, not at review time | Two leaders, duplicate delivery, or a cluster that never fails over — on the **default** server backend | Partly — on the gated server-DB leg only, never on an ordinary PR | P0 | | Every live server-DB DR suite runs only on the server-DB gated leg | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): "invoked by no leg" was false for all six.** `test_dr_server_seed_gate_{postgres,sqlserver}.py`, `test_dr7_server_config_only_backup_{postgres,sqlserver}.py` and `test_backup_runner_server_db_{postgres,sqlserver}.py` all exist and are all invoked — the Postgres trio by the `postgres-store` job's step *"Run the DR seed-gate + backup suites on real Postgres"*, the SQL Server trio by the `sqlserver-store` job's *"...on real SQL Server"*. **Same correction as the row above, and it stays P0 for the same reason:** both jobs carry the `schedule \|\| workflow_dispatch \|\| serverdb == 'true'` gate, so DR regressions are caught nightly rather than at review time. The `MEFOR_TEST_*` env gating is a second, independent condition and was *not* re-measured here — it is the part of this row that may still bite | DR activation promotes onto a **fresh/unrestored** server DB — precisely the data-loss case BACKLOG #102 exists to close | Partly — on the gated server-DB leg only, never on an ordinary PR | P0 | | A leaderless cluster pages nobody | `leadership_lost` is auto-resolve-only and deliberately emits no notification (`pipeline/alerts.py:206-213`); the allowed alert-type set `_ALERT_EVENT_TYPES` (`config/settings.py:2599`, a `frozenset`, **18** members) has no "no leader" member. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100): both counts and both claims are EXACTLY right** — the set has 18 members and `leadership_acquired` is the only leader-related one, so nothing fires when leadership is *absent*; the `alerts.py:206-213` anchor still lands on `leadership_lost` ("Emits **no** notification"). Only the settings anchor had drifted (cited `:2499-2526`, declared at `:2599`) and is corrected; it is cited by **name and declaration line** now, because a range around a growing `frozenset` re-drifts on the next member added. All nodes self-fenced (DB outage), all nodes `promotable=false`, or a lease nobody can take → total stop. | Full silent intake stop with a filling staged queue; only surface is a console badge someone must be looking at | No | P0 | -| Engine shards and `[cluster]` collide on one lease | `_lease_key = f"{db_schema or 'public'}:mefor_cluster_leader"` (`cluster.py:414`) is **not** engine-shard-namespaced, and no validator relates `serve --shard` to `[cluster].enabled`. N engine-shard processes over one unified store would contend for the same row; N-1 bind nothing. | Silent full stop for every engine shard but one — looks like a healthy cluster on `/cluster/status` | No | P0 | +| Engine shards and `[cluster]` cannot be combined — **the collision is refused, not latent** | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): the load-bearing half of this row is false.** "No validator relates `serve --shard` to `[cluster].enabled`" — **there is one, it fails closed, and it is tested.** Per **ADR 0073** the two are mutually exclusive; `__main__.py:2649-2659` refuses the combination with *"error: `--shard` cannot be combined with `[cluster].enabled`"* and **exit 2, before the store or app is built**, and `tests/test_shard_recovery_engine.py:321::test_serve_refuses_shard_with_cluster_enabled` asserts `rc == 2`. The lease-key half is still accurate — `_lease_key = f"{db_schema or 'public'}:mefor_cluster_leader"` (**`cluster.py:484`**, not `:414`; anchor drifted) is **not** engine-shard-namespaced — **but that is now MOOT for this failure mode**, because N engine-shard processes can never come up against an enabled `[cluster]` to contend at all | **The stated blast radius cannot occur.** It was "silent full stop for every engine shard but one, looking healthy on `/cluster/status`"; a fail-closed startup refusal is the opposite of silent. What remains is only the residual design note that the lease key is un-namespaced, which matters solely if ADR 0073's mutual exclusion is ever lifted — so it is a **constraint to preserve**, not a gap to close. This also satisfies the HA-03 exit criterion at `:399` by its **first** alternative (a config-load rejection rather than an engine-shard-namespaced lease key), though the assertion lives in `test_shard_recovery_engine.py`, not the `tests/test_cluster_shard_interaction.py` that criterion names | Yes — fail-closed at startup, with a test | **Closed** | | Cross-store divergence after a DR fail-back | `pipeline/dr.py:296-301` states the engine gives no cross-store loss/duplicate guarantee and defers to a runbook. `harness/reconcile/` exists (capture / compare / normalize / report) but nothing binds it to the DR release path. | Messages processed only at DR, or reprocessed at both — silent clinical-data divergence with no failing test and no operator artifact | No | P0 | | RTO is unbounded in absolute terms | `harness/load/failover.py:640-646` bounds `functional_recovery_seconds` only as `recovery_ttl_multiple × leader_lease_ttl_seconds` (2.0 in `failover.toml`, 3.0 in `_failover_load_support.py`); `max_promotion_seconds` is optional (`profile.py:129-131`) and set in **neither** profile. No RPO figure is asserted anywhere. | Promotion can regress from ~7s to ~60s with every SLO green, because the bound scales with whatever TTL the profile sets | Partially (reported, never gated) | P1 | | A DB blip wedges the pool | `docs/AOAG-DEPLOYMENT.md:513-520`: "the engine has no reconnect logic of its own … treat automatic pool recovery as driver-dependent behavior to **drill**". Automated coverage simulates partition only via a fake pool raising `RuntimeError` (`test_cluster_lease.py`). | Engine wedged after a DB restart — listeners bound or unbound, no forward progress | Manual only (`W25:S4.10`) | P1 | From 1568fac5f876a4808e63ed6a27081036bc53af49 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 04:23:05 -0500 Subject: [PATCH 09/66] docs(testing): verify the cross-store divergence P0 and sharpen what is actually missing (BACKLOG #1100) 04-high-availability-and-dr.md row 84 was re-measured and is CORRECT on all three claims -- the last unchecked P0 in this chapter's risk table: dr.py:296-301 disclaims a cross-store guarantee TRUE, and the anchor lands exactly on "the engine gives no cross-store loss/duplicate guarantee -- documented, not an engine AC" harness/reconcile/ has capture/compare/normalize/report TRUE, all four nothing binds it to the DR release path TRUE -- 4 unit-test files exercise harness.reconcile, but no workflow and no runbook step invokes it The two "reconcile" hits under .github/workflows/ are an unrelated ingress probe and an English verb, not a binding. Worth stating because a grep for the word alone suggests coverage that does not exist. AND THE GAP IS SHARPER THAN THE ROW SAID. AOAG-DEPLOYMENT.md:708 DOES instruct the operator -- "reconcile before you discard; do not rubber-stamp the loss" -- and never names harness/reconcile/. The runbook asks for the activity and points at no tool, which is worse than saying nothing: the operator is told to do it and left to invent how, at the exact moment they are recovering from a failover. That makes the remedy cheap and specific rather than open-ended: have :708 name the tool. Recorded in the row so whoever picks up this P0 starts from a one-line fix rather than from "bind the harness to the release path". Note: correcting the AOAG runbook is not in this claim's scope, and the row now carries the pointer instead. --- docs/testing/master-test-plan/04-high-availability-and-dr.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/04-high-availability-and-dr.md b/docs/testing/master-test-plan/04-high-availability-and-dr.md index 3695cea9d..f46e478c0 100644 --- a/docs/testing/master-test-plan/04-high-availability-and-dr.md +++ b/docs/testing/master-test-plan/04-high-availability-and-dr.md @@ -81,7 +81,7 @@ bare `HA-nn` in this chapter always means **this chapter's own §3.4 row** — t | Every live server-DB DR suite runs only on the server-DB gated leg | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): "invoked by no leg" was false for all six.** `test_dr_server_seed_gate_{postgres,sqlserver}.py`, `test_dr7_server_config_only_backup_{postgres,sqlserver}.py` and `test_backup_runner_server_db_{postgres,sqlserver}.py` all exist and are all invoked — the Postgres trio by the `postgres-store` job's step *"Run the DR seed-gate + backup suites on real Postgres"*, the SQL Server trio by the `sqlserver-store` job's *"...on real SQL Server"*. **Same correction as the row above, and it stays P0 for the same reason:** both jobs carry the `schedule \|\| workflow_dispatch \|\| serverdb == 'true'` gate, so DR regressions are caught nightly rather than at review time. The `MEFOR_TEST_*` env gating is a second, independent condition and was *not* re-measured here — it is the part of this row that may still bite | DR activation promotes onto a **fresh/unrestored** server DB — precisely the data-loss case BACKLOG #102 exists to close | Partly — on the gated server-DB leg only, never on an ordinary PR | P0 | | A leaderless cluster pages nobody | `leadership_lost` is auto-resolve-only and deliberately emits no notification (`pipeline/alerts.py:206-213`); the allowed alert-type set `_ALERT_EVENT_TYPES` (`config/settings.py:2599`, a `frozenset`, **18** members) has no "no leader" member. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100): both counts and both claims are EXACTLY right** — the set has 18 members and `leadership_acquired` is the only leader-related one, so nothing fires when leadership is *absent*; the `alerts.py:206-213` anchor still lands on `leadership_lost` ("Emits **no** notification"). Only the settings anchor had drifted (cited `:2499-2526`, declared at `:2599`) and is corrected; it is cited by **name and declaration line** now, because a range around a growing `frozenset` re-drifts on the next member added. All nodes self-fenced (DB outage), all nodes `promotable=false`, or a lease nobody can take → total stop. | Full silent intake stop with a filling staged queue; only surface is a console badge someone must be looking at | No | P0 | | Engine shards and `[cluster]` cannot be combined — **the collision is refused, not latent** | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): the load-bearing half of this row is false.** "No validator relates `serve --shard` to `[cluster].enabled`" — **there is one, it fails closed, and it is tested.** Per **ADR 0073** the two are mutually exclusive; `__main__.py:2649-2659` refuses the combination with *"error: `--shard` cannot be combined with `[cluster].enabled`"* and **exit 2, before the store or app is built**, and `tests/test_shard_recovery_engine.py:321::test_serve_refuses_shard_with_cluster_enabled` asserts `rc == 2`. The lease-key half is still accurate — `_lease_key = f"{db_schema or 'public'}:mefor_cluster_leader"` (**`cluster.py:484`**, not `:414`; anchor drifted) is **not** engine-shard-namespaced — **but that is now MOOT for this failure mode**, because N engine-shard processes can never come up against an enabled `[cluster]` to contend at all | **The stated blast radius cannot occur.** It was "silent full stop for every engine shard but one, looking healthy on `/cluster/status`"; a fail-closed startup refusal is the opposite of silent. What remains is only the residual design note that the lease key is un-namespaced, which matters solely if ADR 0073's mutual exclusion is ever lifted — so it is a **constraint to preserve**, not a gap to close. This also satisfies the HA-03 exit criterion at `:399` by its **first** alternative (a config-load rejection rather than an engine-shard-namespaced lease key), though the assertion lives in `test_shard_recovery_engine.py`, not the `tests/test_cluster_shard_interaction.py` that criterion names | Yes — fail-closed at startup, with a test | **Closed** | -| Cross-store divergence after a DR fail-back | `pipeline/dr.py:296-301` states the engine gives no cross-store loss/duplicate guarantee and defers to a runbook. `harness/reconcile/` exists (capture / compare / normalize / report) but nothing binds it to the DR release path. | Messages processed only at DR, or reprocessed at both — silent clinical-data divergence with no failing test and no operator artifact | No | P0 | +| Cross-store divergence after a DR fail-back | `pipeline/dr.py:296-301` states the engine gives no cross-store loss/duplicate guarantee and defers to a runbook. `harness/reconcile/` exists (capture / compare / normalize / report) but nothing binds it to the DR release path. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100) — all three claims hold, and the anchor is exact:** `dr.py:296-301` still reads *"the engine gives no cross-store loss/duplicate guarantee — documented, not an engine AC"*; all four reconcile modules are present; and the binding is still absent — 4 unit-test files exercise `harness.reconcile`, but **no workflow and no runbook step invokes it** (the two `reconcile` hits in `.github/workflows/` are an unrelated ingress probe and an English verb). **SHARPER THAN "NOTHING BINDS IT":** `AOAG-DEPLOYMENT.md:708` *does* instruct the operator — "reconcile before you discard; do not rubber-stamp the loss" — **and never names `harness/reconcile/`**. So the runbook asks for the activity and points at no tool, which is worse than silence: the operator is told to do it and left to invent how | Messages processed only at DR, or reprocessed at both — silent clinical-data divergence with no failing test and no operator artifact. The fix is cheap and specific: have `:708` name the tool | No | P0 | | RTO is unbounded in absolute terms | `harness/load/failover.py:640-646` bounds `functional_recovery_seconds` only as `recovery_ttl_multiple × leader_lease_ttl_seconds` (2.0 in `failover.toml`, 3.0 in `_failover_load_support.py`); `max_promotion_seconds` is optional (`profile.py:129-131`) and set in **neither** profile. No RPO figure is asserted anywhere. | Promotion can regress from ~7s to ~60s with every SLO green, because the bound scales with whatever TTL the profile sets | Partially (reported, never gated) | P1 | | A DB blip wedges the pool | `docs/AOAG-DEPLOYMENT.md:513-520`: "the engine has no reconnect logic of its own … treat automatic pool recovery as driver-dependent behavior to **drill**". Automated coverage simulates partition only via a fake pool raising `RuntimeError` (`test_cluster_lease.py`). | Engine wedged after a DB restart — listeners bound or unbound, no forward progress | Manual only (`W25:S4.10`) | P1 | | Always On AG failover is entirely documentation | 919 lines of `AOAG-DEPLOYMENT.md` specify replica modes, quorum, the phase-dependent 10s-vs-60s fence boundary, and the stranded-in-flight remedy. No lab, no leg, no drill script. `MultiSubnetFailover` is tested only as a DSN substring (`store/sqlserver.py:1414`). | Every enterprise SQL Server adopter hits this; wrong guidance means stuck in-flight messages nobody finds | No | P1 | From a72e2beeb9fdd15e8452383efab0c57c7fb10123 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 04:45:15 -0500 Subject: [PATCH 10/66] docs(testing): a P0 about unrun suites was wrong, and the real defect is its inverse (BACKLOG #1100) 02-pipeline-reliability.md row 145 claimed six MEFOR_TEST_*-gated pipeline suites are "named in no workflow step", citing batch_handoff_statements and engine-shard recovery, and asserting they are "in neither the steps nor the gate regex". BOTH PARTS ARE FALSE FOR BOTH NAMED EXAMPLES: tests/test_adr0075_batch_sqlserver.py ci.yml:1375, sqlserver-store step tests/test_shard_recovery_sqlserver.py ci.yml:1371, same step adr0075 and shard_recovery are both in the gate alternation at ci.yml:993 The cited anchor :424-428 had drifted onto a comment about CI run timings; the "MUST list every file" comment is at :983-988. THE RESIDUAL GAP IS REAL AND SHARPER THAN THE ROW STATED. Of 54 MEFOR_TEST_-gated suites, 19 are named in no workflow step, and every server-DB step invokes EXPLICIT FILES rather than a directory -- so an unnamed suite runs nowhere on a real backend. THREE OF THE 19 ARE MATCHED BY THE GATE REGEX: test_adr0157_fence_scope 8 tests test_sqlserver_sync_handoff_offline 6 tests test_adr0071_fusion_wiring each named ZERO times anywhere in .github/ (positive control: shard_recovery_ sqlserver = 1). A change to any of them PULLS the expensive server-DB legs and then never executes them -- the leg goes green having not run the suite the gate fired for. ci.yml:983-988 states only one direction of the invariant: the alternation must list every file the steps run. THE INVERSE IS UNSTATED AND VIOLATED THREE TIMES: every file the alternation lists should be run by some step. That asymmetry is why this went unnoticed -- the stated half is checked by the comment's own instruction, and nothing looks at the other half. Stays P0. The original blast radius (duplicate PHI delivery across engine shards) is in fact covered, since shard_recovery and adr0075 both run; what remains is ADR 0157 fence scope with no real-Postgres execution, which is a correctness mechanism against duplicate delivery in its own right. --- docs/testing/master-test-plan/02-pipeline-reliability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/02-pipeline-reliability.md b/docs/testing/master-test-plan/02-pipeline-reliability.md index 195896ddb..039fbd68d 100644 --- a/docs/testing/master-test-plan/02-pipeline-reliability.md +++ b/docs/testing/master-test-plan/02-pipeline-reliability.md @@ -142,7 +142,7 @@ chapter or any downstream one: | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| Six `MEFOR_TEST_*`-gated pipeline suites (~36 tests) are named in **no** workflow step | A live regression in `batch_handoff_statements` (DEFAULT-ON, SQL-Server-only, restructures the route/transform handoff DML) or in ownership-scoped **engine-shard** recovery ships green | Silent data loss, wrong disposition, or **duplicate PHI deliveries across engine shards** on the production-scale backend | **No.** `ci.yml:424-428`'s own comment says the path gate "MUST list every file the sqlserver/postgres steps run"; these are in neither the steps nor the gate regex | **P0** | +| Gated suites the path gate PULLS but no step RUNS — **the inverse of the invariant `ci.yml:983` states** | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): the original evidence was false in both parts.** It said `batch_handoff_statements` and engine-shard recovery "are in neither the steps nor the gate regex". **Both are in both**: `tests/test_adr0075_batch_sqlserver.py` (`ci.yml:1375`) and `tests/test_shard_recovery_sqlserver.py` (`:1371`) run in the `sqlserver-store` step *"Run the engine-shard + statement-dispatch suites on real SQL Server"*, and `adr0075`/`shard_recovery` are both in the gate alternation at `:993`. The cited anchor `:424-428` had drifted onto a comment about CI run timings; the real comment is `:983-988`. **THE RESIDUAL GAP IS REAL AND SHARPER:** of **54** `MEFOR_TEST_*`-gated suites, **19 are named in no workflow step**, and every server-DB step invokes **explicit files** (never a directory), so an unnamed suite runs nowhere on a real backend. **Three of the 19 are MATCHED BY THE GATE REGEX** — `test_adr0157_fence_scope` (8 tests), `test_sqlserver_sync_handoff_offline` (6), `test_adr0071_fusion_wiring` — each named **0** times anywhere in `.github/` (positive control: `shard_recovery_sqlserver` = 1) | A change to those three **pulls the expensive server-DB legs and then never executes them** — the legs go green having not run the suite the gate fired for. `test_adr0157_fence_scope` is the sharpest: ADR 0157 fence scope, 8 tests, no real-Postgres execution anywhere | **Partly.** `ci.yml:983-988` states only ONE direction — the alternation must list every file the steps run. **The inverse is unstated and violated three times**: every file the alternation lists should be run by some step | **P0** | | No poison-crash attempts ceiling on the **default split** ingress/routed path | A hard abort with no Python exception (C-extension segfault, OOM kill) inside `route_only`/`transform_one`/handoff is caught by neither the internal-error policy nor the ADR 0070 T17 handler; `reset_stale_inflight` re-pends the head, the lane re-runs, the process dies again | Lane head-of-line blocked **forever** across NSSM/supervisor restarts; nothing dead-letters; every message behind it stops flowing. The G6 ceiling exists only inside `if inline:` (`wiring_runner.py:4475-4496`), and its own comment at `:4479-4480` states no ingress/routed path enforces `max_attempts` on the split path. `supervisor.py:22` lists "restart backoff / crash-loop breaker" as deferred | **No** — the ADR 0087 sandbox that would contain it is default OFF, and no test drives a hard abort on the split path | **P0** | | `W25:S3.4` / `W25:S2.7` and `harness/config/coverage.py:17` assert an **AE NAK** for a post-ACK Handler raise | Under ACK-on-receipt the AA fires at the ingress commit (`wiring_runner.py:3726-3745`) before the Router or Handler runs; a Handler raise **cannot** NAK | A human running `W25:S3.4` either fails a correct system or records a NAK that never happened. `harness/scenarios.py:63` already expects only disposition `error`, so the docs contradict both the code and the harness they instruct the tester to run — on the single most partner-visible behaviour change in ADR 0001 | **No** — the docs *are* the detector, and they are wrong | **P0** | | No live end-to-end committed-transactions-per-message ceiling in CI | An accidental extra handoff commit doubles `committed_txns/msg` | Passes every test; surfaces only as a production capacity shortfall. ADR 0051 sizes capacity on `3 + 2H + 2N`; the counters already exist (`store/base.py:220-234`, surfaced at `api/app.py:4142-4143`) | Partly — `test_txn_per_message_cost_model.py` pins the **model** over a recording connection, not the **live** counter through a real runner | P1 | From e07018a7eef4534d7f210ef59dc0314c688e74eb Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 04:53:36 -0500 Subject: [PATCH 11/66] docs(testing): verify the poison-crash P0 and locate exactly where its ceiling is missing (BACKLOG #1100) 02-pipeline-reliability.md row 146 is CORRECT, and re-deriving it from the code turns "no ceiling exists" into something a fixer can act on. What is actually there: queue table carries an attempts column for EVERY stage store.py:1319 a max_attempts ceiling IS enforced :5756 mark_failed :5816 mark_batch_failed reset_stale_inflight, 69 lines :5948 -- references NEITHER attempts NOR dead_letter So the ceiling is real but sits ONLY on the caught-Python-exception path. A hard abort -- C-extension segfault, OOM kill -- calls neither mark_failed nor mark_batch_failed, and recovery runs reset_stale_inflight instead, which re-pends the head without counting anything. The lane re-runs and the process dies again, exactly as the row says. THE FIX HAS A LOCATION: the increment must happen at CLAIM time or inside reset_stale_inflight, NOT on the failure path -- because the failure path is precisely what a hard abort skips. A ceiling added to mark_failed would pass review and change nothing about this failure mode. Also recorded: G6 is an OUTBOX-stage mechanism throughout (every reference reads OutboxItem.attempts), which is why it does not cover the split ingress/routed path. Worth stating because "there is a poison ceiling" is true and misleading in the same breath. Verified by reading the whole function via AST rather than a byte window -- the same instrument that produced a wrong member count earlier tonight. --- docs/testing/master-test-plan/02-pipeline-reliability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/02-pipeline-reliability.md b/docs/testing/master-test-plan/02-pipeline-reliability.md index 039fbd68d..39caa19ca 100644 --- a/docs/testing/master-test-plan/02-pipeline-reliability.md +++ b/docs/testing/master-test-plan/02-pipeline-reliability.md @@ -143,7 +143,7 @@ chapter or any downstream one: | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | Gated suites the path gate PULLS but no step RUNS — **the inverse of the invariant `ci.yml:983` states** | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15): the original evidence was false in both parts.** It said `batch_handoff_statements` and engine-shard recovery "are in neither the steps nor the gate regex". **Both are in both**: `tests/test_adr0075_batch_sqlserver.py` (`ci.yml:1375`) and `tests/test_shard_recovery_sqlserver.py` (`:1371`) run in the `sqlserver-store` step *"Run the engine-shard + statement-dispatch suites on real SQL Server"*, and `adr0075`/`shard_recovery` are both in the gate alternation at `:993`. The cited anchor `:424-428` had drifted onto a comment about CI run timings; the real comment is `:983-988`. **THE RESIDUAL GAP IS REAL AND SHARPER:** of **54** `MEFOR_TEST_*`-gated suites, **19 are named in no workflow step**, and every server-DB step invokes **explicit files** (never a directory), so an unnamed suite runs nowhere on a real backend. **Three of the 19 are MATCHED BY THE GATE REGEX** — `test_adr0157_fence_scope` (8 tests), `test_sqlserver_sync_handoff_offline` (6), `test_adr0071_fusion_wiring` — each named **0** times anywhere in `.github/` (positive control: `shard_recovery_sqlserver` = 1) | A change to those three **pulls the expensive server-DB legs and then never executes them** — the legs go green having not run the suite the gate fired for. `test_adr0157_fence_scope` is the sharpest: ADR 0157 fence scope, 8 tests, no real-Postgres execution anywhere | **Partly.** `ci.yml:983-988` states only ONE direction — the alternation must list every file the steps run. **The inverse is unstated and violated three times**: every file the alternation lists should be run by some step | **P0** | -| No poison-crash attempts ceiling on the **default split** ingress/routed path | A hard abort with no Python exception (C-extension segfault, OOM kill) inside `route_only`/`transform_one`/handoff is caught by neither the internal-error policy nor the ADR 0070 T17 handler; `reset_stale_inflight` re-pends the head, the lane re-runs, the process dies again | Lane head-of-line blocked **forever** across NSSM/supervisor restarts; nothing dead-letters; every message behind it stops flowing. The G6 ceiling exists only inside `if inline:` (`wiring_runner.py:4475-4496`), and its own comment at `:4479-4480` states no ingress/routed path enforces `max_attempts` on the split path. `supervisor.py:22` lists "restart backoff / crash-loop breaker" as deferred | **No** — the ADR 0087 sandbox that would contain it is default OFF, and no test drives a hard abort on the split path | **P0** | +| No poison-crash attempts ceiling on the **default split** ingress/routed path | A hard abort with no Python exception (C-extension segfault, OOM kill) inside `route_only`/`transform_one`/handoff is caught by neither the internal-error policy nor the ADR 0070 T17 handler; `reset_stale_inflight` re-pends the head, the lane re-runs, the process dies again. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100), and the mechanism is sharper than "no ceiling exists":** the `queue` table DOES carry an `attempts` column (`store.py:1319`) for **every** stage, and a `max_attempts` ceiling IS enforced — but **only inside `mark_failed()` (`:5756`) and `mark_batch_failed()` (`:5816`)**, which are the *caught-Python-exception* paths. A hard abort calls neither, and recovery instead runs `reset_stale_inflight` (`:5948`, 69 lines) which references **neither `attempts` nor `dead_letter`** — verified by reading the whole function, not a window. So the row is right, and the fix has a precise location: **the increment must happen at CLAIM time or in `reset_stale_inflight`, not on the failure path**, because the failure path is exactly what a hard abort skips. Note also that G6 is an **outbox-stage** mechanism throughout (every reference reads `OutboxItem.attempts`), which is why it does not cover this | Lane head-of-line blocked **forever** across NSSM/supervisor restarts; nothing dead-letters; every message behind it stops flowing. The G6 ceiling exists only inside `if inline:` (`wiring_runner.py:4475-4496`), and its own comment at `:4479-4480` states no ingress/routed path enforces `max_attempts` on the split path. `supervisor.py:22` lists "restart backoff / crash-loop breaker" as deferred | **No** — the ADR 0087 sandbox that would contain it is default OFF, and no test drives a hard abort on the split path | **P0** | | `W25:S3.4` / `W25:S2.7` and `harness/config/coverage.py:17` assert an **AE NAK** for a post-ACK Handler raise | Under ACK-on-receipt the AA fires at the ingress commit (`wiring_runner.py:3726-3745`) before the Router or Handler runs; a Handler raise **cannot** NAK | A human running `W25:S3.4` either fails a correct system or records a NAK that never happened. `harness/scenarios.py:63` already expects only disposition `error`, so the docs contradict both the code and the harness they instruct the tester to run — on the single most partner-visible behaviour change in ADR 0001 | **No** — the docs *are* the detector, and they are wrong | **P0** | | No live end-to-end committed-transactions-per-message ceiling in CI | An accidental extra handoff commit doubles `committed_txns/msg` | Passes every test; surfaces only as a production capacity shortfall. ADR 0051 sizes capacity on `3 + 2H + 2N`; the counters already exist (`store/base.py:220-234`, surfaced at `api/app.py:4142-4143`) | Partly — `test_txn_per_message_cost_model.py` pins the **model** over a recording connection, not the **live** counter through a real runner | P1 | | A refactor hoists the lookup-runner `ExitStack` to wrap `route_only` | Routers silently gain live `db_lookup` access; the at-least-once re-run invariant breaks and an unbudgeted live DB read lands on the routing hot path for every message | Non-pure Routers ⇒ duplicate/divergent downstream side effects on every crash re-run | **No** for `db_lookup` — the FHIR twin exists (`test_fhir_lookup.py:545`); the db_lookup side has only "no active runner" (`test_db_lookup.py:104`) and "dry-run raises" (`:310`). The guarantee is *positional* (`wiring_runner.py:5027-5031`), not structural | P1 | From 039d731d808568f86fd3445f09a98b2ce93cb029 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 05:03:44 -0500 Subject: [PATCH 12/66] docs(testing): narrow the leak-gate floor P0 to the one claim that survives (BACKLOG #1100) 16-security-phi-and-supply-chain.md row 201 said nothing asserts that the MEFOR_MIN_DETECTORS floor matches the real list, that the spec parses, or that a partially-mangled secret is caught. TWO OF THOSE THREE ARE NOW FALSE. test_scan_tokens_source.py carries 56 tests. The spec IS parse-asserted (test_min_spec_parsing_rejects_nonsense, test_unrecognised_require_value_refuses) and a partially-mangled secret IS covered (test_present_but_unusable_token_source_ fails_closed, parametrized over `mangled`, whose docstring records that a mangled secret once yielded ZERO detectors). Ran the cited tests: 7 passed. Materially, the ESTATE half no longer depends on the token list at all. BACKLOG #321's structural estate-identifier shape detector fires with NO token source present, and the allowlist validator refuses an entry broad enough to disable it. DEPENDENCY STATED IN THE ROW ITSELF: that detector is commit c3959449, which is on this builder branch and NOT on main. If the plan edit lands without it, the cell is wrong. A row whose truth depends on an unlanded sibling commit has to say so, or it becomes a false claim the moment the two are separated -- and they are separable, because I reported these items as individually landable. THE SURVIVING CLAIM IS THE INTERESTING ONE AND IT STAYS P0. Every floor test uses a SYNTHETIC floor (names=2 + estate=2 + site_prefixes=1), so nothing asserts the real names=7,estate=13,site_prefixes=1. That cannot be fixed in pytest: scan-tokens.local.txt is gitignored and the real list arrives only via MEFOR_FORBIDDEN_TOKENS, so no checkout can hold it. Any real fix must compare the floor against the list INSIDE THE GATE RUN. Recorded so the next attempt does not start by writing a unit test that cannot see the data. Anchor corrected: security.yml:403 -> :674 (commented at :655). --- .../master-test-plan/16-security-phi-and-supply-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md b/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md index 3e4e452ea..b70aa279c 100644 --- a/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md +++ b/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md @@ -198,7 +198,7 @@ its nine rows are settled evidence, not open work. |---|---|---|---|---| | The post-2026-07-14 security wave has no coverage-plan owner | `FEATURE-COVERAGE-PLAN.md` is dated 2026-07-13/14 and contains **zero** references to ADRs 0135, 0138–0153 (verified by grep). Every "is it covered?" answer drawn from it is wrong by omission for exactly the newest controls. | The enforcement dial, both ADR 0140 carve-outs, `allowed_client_networks`, in-use protection, the cleartext-hop collapse, the Transit cipher, the handler-security lint, hardened runtime isolation, SBOM/VEX | **No** — the gap-audit instrument itself is the blind spot | **P0** | | A one-line YAML edit disarms the whole blocking CI gate set | Adding `continue-on-error: true` or an `if:` to any of the seven blocking `security.yml` jobs converts it to advisory with a green tick. The guard pattern exists for `freethread-smoke.yml` (`tests/test_freethread_smoke_liveness.py:55`) and `quality-advisory.yml` (`test_quality_advisory_invariants.py:132`) but was **never extended to `security.yml`** | pip-audit, npm-audit, bandit, gitleaks, semgrep, crypto-inventory, forbidden-content — every SAST/SCA/secret/leak control the Secure Build Scorecard grades "Strong" | **No** | **P0** | -| The customer/PHI leak gate's detector floor silently falls behind the token list | `MEFOR_MIN_DETECTORS=names=7,estate=13,site_prefixes=1` exists only as a shell literal (`security.yml:403`). Nothing asserts it still matches the real list, that the spec parses, or that a partially-mangled secret is caught. The real list arrives only from `MEFOR_FORBIDDEN_TOKENS`; `scan-tokens.local.txt` is gitignored (`.gitignore:126`) and never committed, so no checkout can hold it | Real partner/site names and estate identifiers reaching a **public** repository | Partially — `test_scan_tokens_source.py` covers the floor **parser**, nothing covers the **value** | **P0** | +| The leak gate's detector floor is unasserted **in its production VALUE** — narrowed 2026-08-15 | **RE-MEASURED (BACKLOG #1100). Two of the three "nothing asserts" clauses are now FALSE**, and the anchor drifted (`security.yml:403` cited; `MEFOR_MIN_DETECTORS` is at **`:674`**, commented at `:655`). `test_scan_tokens_source.py` now carries **56 tests**: the spec **is** parse-asserted (`test_min_spec_parsing_rejects_nonsense`, `test_unrecognised_require_value_refuses`), and a **partially-mangled secret IS covered** (`test_present_but_unusable_token_source_fails_closed`, parametrized over `mangled` — its docstring records that a mangled secret once "yielded ZERO detectors"). **Materially, the estate half no longer depends on the token list at all:** BACKLOG #321's structural estate-identifier *shape* detector fires with **no token source present** (`test_estate_identifier_shape_is_flagged_without_any_token_source`, `test_the_estate_identifier_shape_detector_is_live`, and `test_allowlist_rejects_an_entry_broad_enough_to_disable_the_estate_shape`). **DEPENDENCY, STATED SO A PARTIAL LAND IS OBVIOUS: that detector is commit `c3959449`, which is on the builder branch and NOT on `main` — if this row lands without it, this cell is wrong** | Real partner/site names reaching a **public** repository. The **estate** half of that radius is now structurally covered; the **names** half still rests entirely on the token list | **Still the gap, and it is the one that cannot be closed the easy way:** every floor test uses a SYNTHETIC floor (`names=2 + estate=2 + site_prefixes=1`), so nothing asserts the real `names=7,estate=13,site_prefixes=1`. It **cannot** be asserted from a checkout — `scan-tokens.local.txt` is gitignored and the real list arrives only via `MEFOR_FORBIDDEN_TOKENS`. Any fix has to compare the floor to the list **inside the gate run**, not in pytest | **P0** | | Nothing external has adversarially challenged the posture | No third-party ASVS review, no penetration test, no DAST, no fuzzing. No `hypothesis`/`atheris`/`schemathesis`/ZAP in `pyproject.toml` or any workflow. Held by a dated signed risk acceptance that **voids on any off-loopback or production exposure** (`Secure_Build_Scorecard_MEFOR.md:63`; `BACKLOG.md:390` — external review + pentest are the GA/v1.0 gate) | The entire security claim set is self-assessed and AI-assisted | **No, by definition** | **P0** | | A correctly-configured Transit PHI instance is refused at startup | `__main__.py:1161` keys the keyless-PHI refusal on `settings.store.encryption_key or …encryption_key_file` — **cipher_provider-blind**. A `cipher_provider=vault_transit` deployment sets `MEFOR_STORE_TRANSIT_*` instead, so the *strongest* at-rest posture trips "no key → refuse to start" (exit 2). The documented workaround makes the operator assert `allow_unencrypted_phi` about the most-protected configuration | ADR 0138 unusable on the PHI instances it exists for; `GET /security/posture` and the loosening register report a falsehood | **No** — no test drives `serve` with `cipher_provider=vault_transit` | **P1** | | The `/ai/chat` scope control is an honour-system label | `api/app.py:1417-1423` compares `body.data_scope` only; nothing inspects prompt **content**. Any `AI_ASSIST` holder can paste an HL7 body into a `code_only` prompt and have it brokered off-box. The audit records `prompt_chars`, not content (`:1449`) | The only sanctioned PHI-egress-shaped path in the product; a single pasted body is an unlogged PHI disclosure to a third party | **No** — no test, no detector, no written disposition | **P1** | From c2cdfc14dea3ad54c7435a03cd92535c19bdd851 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 05:14:01 -0500 Subject: [PATCH 13/66] docs(testing): close a security P0 whose guard already exists (BACKLOG #1100) 16-security-phi-and-supply-chain.md row 200 said the continue-on-error guard pattern "was never extended to security.yml". IT WAS. tests/test_security_posture.py is that guard -- 12 tests, all passing, last touched 72e67095 on 2026-08-10 under BACKLOG #1079. It covers BOTH vectors the row names: test_required_jobs_carry_no_continue_on_error test_required_jobs_declare_no_skippable_job_level_if -- "a job-level `if:` is the other way a required context silently never reports", allowlisted per job _BLOCKING_SECURITY_JOBS is exactly the seven jobs this row lists, and each is asserted present in .github/required-contexts.txt, so "blocking" is a checked claim rather than a label. The module also runs the REVERSE direction (sbom and trivy must KEEP continue-on-error, catching a silent promotion), and it is not vacuous by construction: test_every_security_job_is_classified fails when a new job arrives in security.yml unclassified. THE REASON THIS ROW SURVIVED IS WORTH RECORDING, BECAUSE IT WILL RECUR. The guard's own module docstring restates this row's claim VERBATIM -- "nothing guarded it: the three tests in this repo that read a workflow's continue-on-error cover quality-advisory.yml and freethread-smoke.yml" -- as THE HISTORY IT WAS WRITTEN TO CLOSE. Anyone who greps for the claim finds it, in the very file that refutes it, and comes away confirmed. A test that documents the defect it fixes reads exactly like evidence the defect is open. Second closure of this sweep, after row 83. Both were P0s asserting a gap that the engine or its CI had already closed, and in both cases the refutation was one file away from the row's own citation. --- .../master-test-plan/16-security-phi-and-supply-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md b/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md index b70aa279c..f39a18db9 100644 --- a/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md +++ b/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md @@ -197,7 +197,7 @@ its nine rows are settled evidence, not open work. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | The post-2026-07-14 security wave has no coverage-plan owner | `FEATURE-COVERAGE-PLAN.md` is dated 2026-07-13/14 and contains **zero** references to ADRs 0135, 0138–0153 (verified by grep). Every "is it covered?" answer drawn from it is wrong by omission for exactly the newest controls. | The enforcement dial, both ADR 0140 carve-outs, `allowed_client_networks`, in-use protection, the cleartext-hop collapse, the Transit cipher, the handler-security lint, hardened runtime isolation, SBOM/VEX | **No** — the gap-audit instrument itself is the blind spot | **P0** | -| A one-line YAML edit disarms the whole blocking CI gate set | Adding `continue-on-error: true` or an `if:` to any of the seven blocking `security.yml` jobs converts it to advisory with a green tick. The guard pattern exists for `freethread-smoke.yml` (`tests/test_freethread_smoke_liveness.py:55`) and `quality-advisory.yml` (`test_quality_advisory_invariants.py:132`) but was **never extended to `security.yml`** | pip-audit, npm-audit, bandit, gitleaks, semgrep, crypto-inventory, forbidden-content — every SAST/SCA/secret/leak control the Secure Build Scorecard grades "Strong" | **No** | **P0** | +| A one-line YAML edit disarms the blocking CI gate set — **CLOSED, the guard was extended (re-measured 2026-08-15, BACKLOG #1100)** | **`tests/test_security_posture.py` IS that guard** (12 tests, all passing; last touched `72e67095`, 2026-08-10, BACKLOG #1079). It covers **both** vectors this row names: `test_required_jobs_carry_no_continue_on_error` and `test_required_jobs_declare_no_skippable_job_level_if` (*"a job-level `if:` is the other way a required context silently never reports"*, allowlisted per job). `_BLOCKING_SECURITY_JOBS` is **exactly the seven listed here**, each asserted present in `.github/required-contexts.txt`, so "blocking" is a checked claim rather than a label. It also runs the **reverse** direction (`sbom`/`trivy` must KEEP `continue-on-error`, so a silent promotion is caught too) and is **not vacuous by construction**: `test_every_security_job_is_classified` fails when a new job is added to `security.yml` without a classification. **BEWARE THE TRAP THAT KEPT THIS ROW ALIVE:** that module's own docstring restates this row's claim *verbatim* — "nothing guarded it: the three tests in this repo that read a workflow's `continue-on-error` cover `quality-advisory.yml` and `freethread-smoke.yml`" — **as the history it was written to close**. Skimming it CONFIRMS the row | pip-audit, npm-audit, bandit, gitleaks, semgrep, crypto-inventory, forbidden-content — every SAST/SCA/secret/leak control the Secure Build Scorecard grades "Strong" | **Yes** — 12 tests, both vectors, plus an anti-vacuity check | **Closed** | | The leak gate's detector floor is unasserted **in its production VALUE** — narrowed 2026-08-15 | **RE-MEASURED (BACKLOG #1100). Two of the three "nothing asserts" clauses are now FALSE**, and the anchor drifted (`security.yml:403` cited; `MEFOR_MIN_DETECTORS` is at **`:674`**, commented at `:655`). `test_scan_tokens_source.py` now carries **56 tests**: the spec **is** parse-asserted (`test_min_spec_parsing_rejects_nonsense`, `test_unrecognised_require_value_refuses`), and a **partially-mangled secret IS covered** (`test_present_but_unusable_token_source_fails_closed`, parametrized over `mangled` — its docstring records that a mangled secret once "yielded ZERO detectors"). **Materially, the estate half no longer depends on the token list at all:** BACKLOG #321's structural estate-identifier *shape* detector fires with **no token source present** (`test_estate_identifier_shape_is_flagged_without_any_token_source`, `test_the_estate_identifier_shape_detector_is_live`, and `test_allowlist_rejects_an_entry_broad_enough_to_disable_the_estate_shape`). **DEPENDENCY, STATED SO A PARTIAL LAND IS OBVIOUS: that detector is commit `c3959449`, which is on the builder branch and NOT on `main` — if this row lands without it, this cell is wrong** | Real partner/site names reaching a **public** repository. The **estate** half of that radius is now structurally covered; the **names** half still rests entirely on the token list | **Still the gap, and it is the one that cannot be closed the easy way:** every floor test uses a SYNTHETIC floor (`names=2 + estate=2 + site_prefixes=1`), so nothing asserts the real `names=7,estate=13,site_prefixes=1`. It **cannot** be asserted from a checkout — `scan-tokens.local.txt` is gitignored and the real list arrives only via `MEFOR_FORBIDDEN_TOKENS`. Any fix has to compare the floor to the list **inside the gate run**, not in pytest | **P0** | | Nothing external has adversarially challenged the posture | No third-party ASVS review, no penetration test, no DAST, no fuzzing. No `hypothesis`/`atheris`/`schemathesis`/ZAP in `pyproject.toml` or any workflow. Held by a dated signed risk acceptance that **voids on any off-loopback or production exposure** (`Secure_Build_Scorecard_MEFOR.md:63`; `BACKLOG.md:390` — external review + pentest are the GA/v1.0 gate) | The entire security claim set is self-assessed and AI-assisted | **No, by definition** | **P0** | | A correctly-configured Transit PHI instance is refused at startup | `__main__.py:1161` keys the keyless-PHI refusal on `settings.store.encryption_key or …encryption_key_file` — **cipher_provider-blind**. A `cipher_provider=vault_transit` deployment sets `MEFOR_STORE_TRANSIT_*` instead, so the *strongest* at-rest posture trips "no key → refuse to start" (exit 2). The documented workaround makes the operator assert `allow_unencrypted_phi` about the most-protected configuration | ADR 0138 unusable on the PHI instances it exists for; `GET /security/posture` and the loosening register report a falsehood | **No** — no test drives `serve` with `cipher_provider=vault_transit` | **P1** | From b5a28ce96d7181f90fffd5d219584ae022bd1726 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 05:22:45 -0500 Subject: [PATCH 14/66] docs(testing): re-verify the security-wave coverage P0 with a positive control (BACKLOG #1100) 16-security-phi-and-supply-chain.md row 199 is CORRECT and still open. FEATURE-COVERAGE-PLAN.md returns zero hits for all 17 of ADR 0135 and 0138-0153, and its status blocks still date to 2026-07-13. Re-measured WITH A POSITIVE CONTROL, which is the part worth keeping: ADRs 0010/0023/0030/0066 DO resolve in that file. Without that, "zero hits" is indistinguishable from a grep that matches nothing for a reason of its own -- the exact failure that made an earlier probe in this sweep report 0 for eight ADRs it had never actually tested. Zero-hits claims now stand 3 verified-true (05:95, 09:122, this row) to 1 broken (05:98, the HTTP listener auth row). Worth recording as a base rate: the class is productive precisely BECAUSE it usually holds -- when one breaks, it breaks loudly and in the direction of understated coverage. The row is annotated rather than changed. It is also the row whose staleness would be hardest to notice from inside the document, because the instrument it indicts is the same one a reader reaches for to check it. --- .../master-test-plan/16-security-phi-and-supply-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md b/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md index f39a18db9..9944d91b3 100644 --- a/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md +++ b/docs/testing/master-test-plan/16-security-phi-and-supply-chain.md @@ -196,7 +196,7 @@ its nine rows are settled evidence, not open work. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| The post-2026-07-14 security wave has no coverage-plan owner | `FEATURE-COVERAGE-PLAN.md` is dated 2026-07-13/14 and contains **zero** references to ADRs 0135, 0138–0153 (verified by grep). Every "is it covered?" answer drawn from it is wrong by omission for exactly the newest controls. | The enforcement dial, both ADR 0140 carve-outs, `allowed_client_networks`, in-use protection, the cleartext-hop collapse, the Transit cipher, the handler-security lint, hardened runtime isolation, SBOM/VEX | **No** — the gap-audit instrument itself is the blind spot | **P0** | +| The post-2026-07-14 security wave has no coverage-plan owner | `FEATURE-COVERAGE-PLAN.md` is dated 2026-07-13/14 and contains **zero** references to ADRs 0135, 0138–0153 (verified by grep). Every "is it covered?" answer drawn from it is wrong by omission for exactly the newest controls. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100): still zero for all 17 ADRs**, re-measured with a positive control (ADRs 0010/0023/0030/0066 *do* resolve in that file, so the zero is a real absence and not a broken grep), and its status blocks still date to 2026-07-13. **This row is the one whose staleness would be invisible from inside**, because the instrument it indicts is the same one a reader would use to check it | The enforcement dial, both ADR 0140 carve-outs, `allowed_client_networks`, in-use protection, the cleartext-hop collapse, the Transit cipher, the handler-security lint, hardened runtime isolation, SBOM/VEX | **No** — the gap-audit instrument itself is the blind spot | **P0** | | A one-line YAML edit disarms the blocking CI gate set — **CLOSED, the guard was extended (re-measured 2026-08-15, BACKLOG #1100)** | **`tests/test_security_posture.py` IS that guard** (12 tests, all passing; last touched `72e67095`, 2026-08-10, BACKLOG #1079). It covers **both** vectors this row names: `test_required_jobs_carry_no_continue_on_error` and `test_required_jobs_declare_no_skippable_job_level_if` (*"a job-level `if:` is the other way a required context silently never reports"*, allowlisted per job). `_BLOCKING_SECURITY_JOBS` is **exactly the seven listed here**, each asserted present in `.github/required-contexts.txt`, so "blocking" is a checked claim rather than a label. It also runs the **reverse** direction (`sbom`/`trivy` must KEEP `continue-on-error`, so a silent promotion is caught too) and is **not vacuous by construction**: `test_every_security_job_is_classified` fails when a new job is added to `security.yml` without a classification. **BEWARE THE TRAP THAT KEPT THIS ROW ALIVE:** that module's own docstring restates this row's claim *verbatim* — "nothing guarded it: the three tests in this repo that read a workflow's `continue-on-error` cover `quality-advisory.yml` and `freethread-smoke.yml`" — **as the history it was written to close**. Skimming it CONFIRMS the row | pip-audit, npm-audit, bandit, gitleaks, semgrep, crypto-inventory, forbidden-content — every SAST/SCA/secret/leak control the Secure Build Scorecard grades "Strong" | **Yes** — 12 tests, both vectors, plus an anti-vacuity check | **Closed** | | The leak gate's detector floor is unasserted **in its production VALUE** — narrowed 2026-08-15 | **RE-MEASURED (BACKLOG #1100). Two of the three "nothing asserts" clauses are now FALSE**, and the anchor drifted (`security.yml:403` cited; `MEFOR_MIN_DETECTORS` is at **`:674`**, commented at `:655`). `test_scan_tokens_source.py` now carries **56 tests**: the spec **is** parse-asserted (`test_min_spec_parsing_rejects_nonsense`, `test_unrecognised_require_value_refuses`), and a **partially-mangled secret IS covered** (`test_present_but_unusable_token_source_fails_closed`, parametrized over `mangled` — its docstring records that a mangled secret once "yielded ZERO detectors"). **Materially, the estate half no longer depends on the token list at all:** BACKLOG #321's structural estate-identifier *shape* detector fires with **no token source present** (`test_estate_identifier_shape_is_flagged_without_any_token_source`, `test_the_estate_identifier_shape_detector_is_live`, and `test_allowlist_rejects_an_entry_broad_enough_to_disable_the_estate_shape`). **DEPENDENCY, STATED SO A PARTIAL LAND IS OBVIOUS: that detector is commit `c3959449`, which is on the builder branch and NOT on `main` — if this row lands without it, this cell is wrong** | Real partner/site names reaching a **public** repository. The **estate** half of that radius is now structurally covered; the **names** half still rests entirely on the token list | **Still the gap, and it is the one that cannot be closed the easy way:** every floor test uses a SYNTHETIC floor (`names=2 + estate=2 + site_prefixes=1`), so nothing asserts the real `names=7,estate=13,site_prefixes=1`. It **cannot** be asserted from a checkout — `scan-tokens.local.txt` is gitignored and the real list arrives only via `MEFOR_FORBIDDEN_TOKENS`. Any fix has to compare the floor to the list **inside the gate run**, not in pytest | **P0** | | Nothing external has adversarially challenged the posture | No third-party ASVS review, no penetration test, no DAST, no fuzzing. No `hypothesis`/`atheris`/`schemathesis`/ZAP in `pyproject.toml` or any workflow. Held by a dated signed risk acceptance that **voids on any off-loopback or production exposure** (`Secure_Build_Scorecard_MEFOR.md:63`; `BACKLOG.md:390` — external review + pentest are the GA/v1.0 gate) | The entire security claim set is self-assessed and AI-assisted | **No, by definition** | **P0** | From cc5faca372f0218d2065a498281ac3e37bd82cfc Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 05:43:18 -0500 Subject: [PATCH 15/66] docs(testing): the suspected Kerberos SPN defect is CONFIRMED, and worse than suspected (BACKLOG #1100) 10-auth-rbac-and-active-directory.md row 91 carried this as a SUSPECTED live defect. It is confirmed, against pyspnego's own signature rather than by inference: spnego.server(hostname: str = 'unspecified', service: str = 'host', ...) hostname: The principal part of the SPN. Required to build the SPN. service: The service part of the SPN. Required to build the SPN. The library BUILDS the SPN from the two fields. Both acceptor sites pass the whole SPN into service= -- auth/ldap.py:315 and :375 -- and settings.py:1905 documents kerberos_spn as "HTTP/host.example.com". WORSE THAN THE ROW ASSUMED: hostname is never passed at EITHER site, so it takes its default. The acceptor principal is HTTP/host.example.com/unspecified, not .../. The row inferred a hostname would be supplied from somewhere; none is. Scope stated honestly per CLAUDE.md section 0: kerberos_enabled defaults to False and is marked experimental (settings.py:1904), so this WOULD bite the first site that turns Kerberos SSO on. It is not affecting anything now, and the conditional is the accurate tense -- but the row's blast radius (a working-looking feature that always fails, every failure an audited generic reject) is correct for that site. Three anchors re-pointed: :300 -> :315, :360 -> :375, :1846 -> :1905. Not fixing the code here: that is auth-surface engine work outside this claim, and the row now carries the exact evidence -- both call sites, the signature, and the missing hostname= -- so the fix does not need re-derivation. --- .../master-test-plan/10-auth-rbac-and-active-directory.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md b/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md index 81785d364..4d5931bbf 100644 --- a/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md +++ b/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md @@ -88,7 +88,7 @@ This chapter covers the whole authentication/authorization core and, specificall | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | The real-directory acceptor path is mock-seam only. The `ldap3` `LDAPException` arms (`auth/ldap.py:252`, `:274`) and every SPNEGO acceptor line (`:298`, `:306`, `:358`, `:363`) are `# pragma: no cover`. No CI leg, no self-hosted runner, no containerised LDAP exists in `.github/workflows/` | A change to the `ldap3` / `pyspnego` call shape, or a dependency bump, ships green while AD login and Windows SSO are broken — or silently permissive | Every enterprise deployment's primary authentication path | **No.** `HANDOFF-AD-LAB-aws.md` states it outright: "The entire AD acceptor path is mock-seam only" | **P0** | -| Suspected live SPN defect. Both acceptor sites pass the *whole* SPN to `pyspnego`'s `service=`: `spnego.server(service=settings.kerberos_spn)` at `auth/ldap.py:300` and `:360`, while `config/settings.py:1846` documents `kerberos_spn` as `HTTP/host.example.com` — so the acceptor principal reads `HTTP/host.example.com/` | Kerberos SSO can never authenticate against a real KDC, yet `kerberos_acceptor_preflight` reports the acceptor healthy and `GET /auth/providers` advertises `kerberos=true` | Windows SSO estate-wide; a working-looking feature that always fails, every failure an audited generic reject | **No test can see it.** Both sites are `pragma: no cover` | **P0** | +| **CONFIRMED live SPN defect** (was "suspected"; verified against the library itself 2026-08-15, BACKLOG #1100). Both acceptor sites pass the *whole* SPN to `pyspnego`'s `service=`: `spnego.server(service=settings.kerberos_spn)` at **`auth/ldap.py:315` and `:375`** (anchors drifted from `:300`/`:360`), while **`config/settings.py:1905`** (drifted from `:1846`) documents `kerberos_spn` as `HTTP/host.example.com`. **The library's own signature settles it:** `spnego.server(hostname: str = 'unspecified', service: str = 'host', ...)`, documented as *"hostname: The principal part of the SPN"* and *"service: The service part of the SPN"* — pyspnego **builds** the SPN from the two. **AND IT IS WORSE THAN THIS ROW ASSUMED: `hostname` is never passed at either site**, so it takes its default and the acceptor principal is `HTTP/host.example.com/unspecified`, not `.../`. **Scope, stated honestly:** `kerberos_enabled` defaults to `False` and is marked experimental (`settings.py:1904`), so this **would** bite the first site that enables Kerberos SSO rather than affecting anything today | Kerberos SSO can never authenticate against a real KDC, yet `kerberos_acceptor_preflight` reports the acceptor healthy and `GET /auth/providers` advertises `kerberos=true` | Windows SSO estate-wide; a working-looking feature that always fails, every failure an audited generic reject | **No test can see it.** Both sites are `pragma: no cover` | **P0** | | ADR 0142 has never met a real IdP. Status line: "Proposed — **code COMPLETE, awaiting lab validation**"; cells L6a, L9, L18 have not run; no AD-lab run record exists under `docs/testing/` | L18 is the real-IdP proof of the AC-11 username UPN-suffix binding — a review found the unchecked-suffix path was a **live** privilege-escalation route (a guest presenting `Administrator@attacker.example` resolving to the on-prem Domain Admin). L9 can invalidate the architecture: a passwordless/smartcard AD account cannot complete the password step-up, so every sensitive route permanently 403s | Federated sign-in as a whole; potentially a domain-admin takeover | **No** | **P0** | | AD account states beyond `ACCOUNTDISABLE` are never consulted. `_find_user` (`auth/ldap.py:182-187`) rejects only `userAccountControl & 0x2`; `accountExpires`, `lockoutTime` and `UF_LOCKOUT` are not even requested in the attribute list (`:170-177`). `resolve_principal` (`:262`) — the password-free path used by Kerberos SSO, OIDC **and** the ADR-0079 reconciler, via `_probe_principal` at `service.py:1207` — therefore accepts them | An AD account that is expired or locked but not explicitly disabled keeps a live engine session (the reconciler probes it `PRESENT` every pass) and can federate in. Offboarding-by-expiry — the common HR pattern — does not propagate | Every offboarded-by-expiry operator retains console access | **No.** `tests/test_auth_hardening.py:415-455` pins the disabled bit only | **P0** | | SQL Server never exercises the AD group → per-connection scope map (`tests/test_sqlserver_store.py:423-424` covers `ad_group_role_map` only); PostgreSQL has one line (`test_postgres_store.py:769-770`); the store-side `allowed_channels` message filter is SQLite-only (`tests/test_channel_rbac.py:99`) | A backend-specific bug returns an empty set; `_sync_ad_channel_scope` (`auth/service.py:1147-1148`) then leaves the scope untouched **by design** and the AD operator silently keeps the all-connections default | Cross-connection PHI exposure on both production backends | **No** | **P1** | From 715df2a8ffc08eb2f4e6b7cc0eefc91824ddbdcc Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 06:03:02 -0500 Subject: [PATCH 16/66] docs(testing): confirm the AD account-state P0 and re-point all four drifted anchors (BACKLOG #1100) 10-auth-rbac-and-active-directory.md row 93 is CORRECT on every claim. Only its citations had rotted, and they had rotted in a way that reads as confirmation: cited auth/ldap.py:182-187 as the ACCOUNTDISABLE reject -> that range now lands on the ATTRIBUTE LIST, which is the other thing this same row cites. A reader checking the anchor sees plausible code and moves on. actual reject auth/ldap.py:197-201 (uac & 0x2) actual attribute list auth/ldap.py:185-192 (requests userAccountControl) _find_user auth/ldap.py:174 resolve_principal auth/ldap.py:277 _probe_principal service.py:1244 (cited :1207), calls it at :1253 Verified by reading _find_user and resolve_principal whole via AST rather than by grepping a window: neither contains accountExpires, lockoutTime or UF_LOCKOUT. So an AD account that is EXPIRED or LOCKED but not explicitly DISABLED is accepted by all three password-free paths -- Kerberos SSO, OIDC, and the ADR-0079 reconciler, which would probe it PRESENT every pass. Offboarding-by-expiry, the common HR pattern, would not propagate. service.py:959 states the shared path in as many words: "Roles come from resolve_principal -- the same password-free LDAP lookup Kerberos uses". Not fixing it: auth-surface engine work outside this claim. The row now carries the exact reject site, the exact attribute list, and the three entry points, so the fix does not need re-deriving. --- .../master-test-plan/10-auth-rbac-and-active-directory.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md b/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md index 4d5931bbf..cde836ea6 100644 --- a/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md +++ b/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md @@ -90,7 +90,7 @@ This chapter covers the whole authentication/authorization core and, specificall | The real-directory acceptor path is mock-seam only. The `ldap3` `LDAPException` arms (`auth/ldap.py:252`, `:274`) and every SPNEGO acceptor line (`:298`, `:306`, `:358`, `:363`) are `# pragma: no cover`. No CI leg, no self-hosted runner, no containerised LDAP exists in `.github/workflows/` | A change to the `ldap3` / `pyspnego` call shape, or a dependency bump, ships green while AD login and Windows SSO are broken — or silently permissive | Every enterprise deployment's primary authentication path | **No.** `HANDOFF-AD-LAB-aws.md` states it outright: "The entire AD acceptor path is mock-seam only" | **P0** | | **CONFIRMED live SPN defect** (was "suspected"; verified against the library itself 2026-08-15, BACKLOG #1100). Both acceptor sites pass the *whole* SPN to `pyspnego`'s `service=`: `spnego.server(service=settings.kerberos_spn)` at **`auth/ldap.py:315` and `:375`** (anchors drifted from `:300`/`:360`), while **`config/settings.py:1905`** (drifted from `:1846`) documents `kerberos_spn` as `HTTP/host.example.com`. **The library's own signature settles it:** `spnego.server(hostname: str = 'unspecified', service: str = 'host', ...)`, documented as *"hostname: The principal part of the SPN"* and *"service: The service part of the SPN"* — pyspnego **builds** the SPN from the two. **AND IT IS WORSE THAN THIS ROW ASSUMED: `hostname` is never passed at either site**, so it takes its default and the acceptor principal is `HTTP/host.example.com/unspecified`, not `.../`. **Scope, stated honestly:** `kerberos_enabled` defaults to `False` and is marked experimental (`settings.py:1904`), so this **would** bite the first site that enables Kerberos SSO rather than affecting anything today | Kerberos SSO can never authenticate against a real KDC, yet `kerberos_acceptor_preflight` reports the acceptor healthy and `GET /auth/providers` advertises `kerberos=true` | Windows SSO estate-wide; a working-looking feature that always fails, every failure an audited generic reject | **No test can see it.** Both sites are `pragma: no cover` | **P0** | | ADR 0142 has never met a real IdP. Status line: "Proposed — **code COMPLETE, awaiting lab validation**"; cells L6a, L9, L18 have not run; no AD-lab run record exists under `docs/testing/` | L18 is the real-IdP proof of the AC-11 username UPN-suffix binding — a review found the unchecked-suffix path was a **live** privilege-escalation route (a guest presenting `Administrator@attacker.example` resolving to the on-prem Domain Admin). L9 can invalidate the architecture: a passwordless/smartcard AD account cannot complete the password step-up, so every sensitive route permanently 403s | Federated sign-in as a whole; potentially a domain-admin takeover | **No** | **P0** | -| AD account states beyond `ACCOUNTDISABLE` are never consulted. `_find_user` (`auth/ldap.py:182-187`) rejects only `userAccountControl & 0x2`; `accountExpires`, `lockoutTime` and `UF_LOCKOUT` are not even requested in the attribute list (`:170-177`). `resolve_principal` (`:262`) — the password-free path used by Kerberos SSO, OIDC **and** the ADR-0079 reconciler, via `_probe_principal` at `service.py:1207` — therefore accepts them | An AD account that is expired or locked but not explicitly disabled keeps a live engine session (the reconciler probes it `PRESENT` every pass) and can federate in. Offboarding-by-expiry — the common HR pattern — does not propagate | Every offboarded-by-expiry operator retains console access | **No.** `tests/test_auth_hardening.py:415-455` pins the disabled bit only | **P0** | +| AD account states beyond `ACCOUNTDISABLE` are never consulted. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100) — every claim holds; only the anchors had drifted, and all four are re-pointed here.** `_find_user` (**`auth/ldap.py:174`**) rejects only `userAccountControl & 0x2` at **`:197-201`**; the attribute list at **`:185-192`** requests `userAccountControl` but **not** `accountExpires`, `lockoutTime` or `UF_LOCKOUT` — verified by reading the whole function via AST, which contains none of those three tokens. `resolve_principal` (**`:277`**) calls `_find_user` and adds no state check of its own, and it is the password-free path used by Kerberos SSO, OIDC **and** the ADR-0079 reconciler via `_probe_principal` (**`service.py:1244`**, calling it at `:1253`) — so all three accept such an account. `service.py:959` states the shared path in as many words: *"Roles come from `resolve_principal` — the same password-free LDAP lookup Kerberos uses"* | An AD account that is expired or locked but not explicitly disabled keeps a live engine session (the reconciler probes it `PRESENT` every pass) and can federate in. Offboarding-by-expiry — the common HR pattern — does not propagate | Every offboarded-by-expiry operator retains console access | **No.** `tests/test_auth_hardening.py:415-455` pins the disabled bit only | **P0** | | SQL Server never exercises the AD group → per-connection scope map (`tests/test_sqlserver_store.py:423-424` covers `ad_group_role_map` only); PostgreSQL has one line (`test_postgres_store.py:769-770`); the store-side `allowed_channels` message filter is SQLite-only (`tests/test_channel_rbac.py:99`) | A backend-specific bug returns an empty set; `_sync_ad_channel_scope` (`auth/service.py:1147-1148`) then leaves the scope untouched **by design** and the AD operator silently keeps the all-connections default | Cross-connection PHI exposure on both production backends | **No** | **P1** | | No meta-guard that a request-scoped audit call threads `client=`. 66 `record_audit(` call sites in `messagefoundry/` (51 in `api/`); the chain hashes a missing client as the legacy 6-element payload (`tests/test_audit_integrity.py:391`), so an omission **verifies perfectly clean** | A new PHI-read or admin-write route that forgets `client=client_ip(request)` silently loses the ADR-0150 "from where" — the exact incident question the ADR was written to answer | Post-incident forensics on any new route | **No** | **P1** | | Two **verified** NULL-client audit paths on the unauthenticated attack surface: `audit_kerberos_reject` and `audit_oidc_reject` funnel into `_directory_reject_audit(actor, mech, reason)` (`auth/service.py:1046-1053`) which never takes or passes a `client`, so every `/ui/sso` and `/ui/oidc` route-level reject row has `client = NULL`. Separately, `auth.ad_scope_resynced` (`:1157-1161`) omits `client` while its sibling `auth.ad_roles_resynced` (`:1085-1090`) passes it at `:1089` | A Kerberos/OIDC probing campaign produces audit rows with no source address at all — the one place an address is most needed | Directory-SSO abuse is un-attributable | **No** | **P1** | From 95f48e29b3b95c748fc9702ae53ebdaa4d5d4675 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 06:13:26 -0500 Subject: [PATCH 17/66] docs(testing): grade the ADR 0142 row only as far as this repository can see (BACKLOG #1100) 10-auth-rbac-and-active-directory.md row 92 makes three claims. Two are confirmed from here; the third cannot be, and saying so is the point of this commit. CONFIRMED the ADR status line still reads exactly "Proposed -- code COMPLETE, awaiting lab validation", and its inline note adds "flips to Accepted only when runbook cells L6a, L9 and L18 report" docs/testing/ holds 26 files, all plan documents, with no AD-lab run record NOT CHECKABLE FROM THIS REPOSITORY whether L6a / L9 / L18 have since run. The runbook is docs/security/AD-FEDERATION-LAB-RUNBOOK.md, and docs/security/ DOES NOT EXIST in this checkout -- it is the private companion repo. My brief also assigns that half to the ASVS Tracker, so the constraint and the practical limit coincide. The row now says UNKNOWN for that clause rather than leaving it reading as CONFIRMED OPEN. The distinction matters here more than usual: L18 is described as the proof of a control that a review once found to be a live privilege-escalation route, so "we have not verified this" and "this is still broken" are very different statements to leave in a security test plan. The ADR status is the honest public proxy -- it has not flipped -- and the row now says that instead of implying more. This is the first row in the sweep whose evidence lives outside the repository. A verification pass has to be able to say "I cannot see this from here" without either guessing or going quiet. --- .../master-test-plan/10-auth-rbac-and-active-directory.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md b/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md index cde836ea6..dc77957e3 100644 --- a/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md +++ b/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md @@ -89,7 +89,7 @@ This chapter covers the whole authentication/authorization core and, specificall |---|---|---|---|---| | The real-directory acceptor path is mock-seam only. The `ldap3` `LDAPException` arms (`auth/ldap.py:252`, `:274`) and every SPNEGO acceptor line (`:298`, `:306`, `:358`, `:363`) are `# pragma: no cover`. No CI leg, no self-hosted runner, no containerised LDAP exists in `.github/workflows/` | A change to the `ldap3` / `pyspnego` call shape, or a dependency bump, ships green while AD login and Windows SSO are broken — or silently permissive | Every enterprise deployment's primary authentication path | **No.** `HANDOFF-AD-LAB-aws.md` states it outright: "The entire AD acceptor path is mock-seam only" | **P0** | | **CONFIRMED live SPN defect** (was "suspected"; verified against the library itself 2026-08-15, BACKLOG #1100). Both acceptor sites pass the *whole* SPN to `pyspnego`'s `service=`: `spnego.server(service=settings.kerberos_spn)` at **`auth/ldap.py:315` and `:375`** (anchors drifted from `:300`/`:360`), while **`config/settings.py:1905`** (drifted from `:1846`) documents `kerberos_spn` as `HTTP/host.example.com`. **The library's own signature settles it:** `spnego.server(hostname: str = 'unspecified', service: str = 'host', ...)`, documented as *"hostname: The principal part of the SPN"* and *"service: The service part of the SPN"* — pyspnego **builds** the SPN from the two. **AND IT IS WORSE THAN THIS ROW ASSUMED: `hostname` is never passed at either site**, so it takes its default and the acceptor principal is `HTTP/host.example.com/unspecified`, not `.../`. **Scope, stated honestly:** `kerberos_enabled` defaults to `False` and is marked experimental (`settings.py:1904`), so this **would** bite the first site that enables Kerberos SSO rather than affecting anything today | Kerberos SSO can never authenticate against a real KDC, yet `kerberos_acceptor_preflight` reports the acceptor healthy and `GET /auth/providers` advertises `kerberos=true` | Windows SSO estate-wide; a working-looking feature that always fails, every failure an audited generic reject | **No test can see it.** Both sites are `pragma: no cover` | **P0** | -| ADR 0142 has never met a real IdP. Status line: "Proposed — **code COMPLETE, awaiting lab validation**"; cells L6a, L9, L18 have not run; no AD-lab run record exists under `docs/testing/` | L18 is the real-IdP proof of the AC-11 username UPN-suffix binding — a review found the unchecked-suffix path was a **live** privilege-escalation route (a guest presenting `Administrator@attacker.example` resolving to the on-prem Domain Admin). L9 can invalidate the architecture: a passwordless/smartcard AD account cannot complete the password step-up, so every sensitive route permanently 403s | Federated sign-in as a whole; potentially a domain-admin takeover | **No** | **P0** | +| ADR 0142 has never met a real IdP. Status line: "Proposed — **code COMPLETE, awaiting lab validation**"; cells L6a, L9, L18 have not run; no AD-lab run record exists under `docs/testing/`. **PARTIALLY RE-VERIFIED 2026-08-15 (BACKLOG #1100), and the unverifiable part is named rather than assumed.** CONFIRMED: the ADR's status line still reads exactly that, and its inline note adds *"flips to Accepted only when runbook cells L6a, L9 and L18 report"*; and `docs/testing/` holds 26 files, all plan documents, with no run record among them. **NOT CHECKED, AND NOT CHECKABLE FROM THIS REPOSITORY:** whether L6a/L9/L18 have since run. The runbook is `docs/security/AD-FEDERATION-LAB-RUNBOOK.md`, and **`docs/security/` does not exist in this checkout at all** — it is the private companion repo. So this row's middle clause can only be graded by someone with the vault, and a reader here should treat it as *unknown*, not as *confirmed open*. The ADR status is the honest public proxy: it has not flipped | L18 is the real-IdP proof of the AC-11 username UPN-suffix binding — a review found the unchecked-suffix path was a **live** privilege-escalation route (a guest presenting `Administrator@attacker.example` resolving to the on-prem Domain Admin). L9 can invalidate the architecture: a passwordless/smartcard AD account cannot complete the password step-up, so every sensitive route permanently 403s | Federated sign-in as a whole; potentially a domain-admin takeover | **No** | **P0** | | AD account states beyond `ACCOUNTDISABLE` are never consulted. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100) — every claim holds; only the anchors had drifted, and all four are re-pointed here.** `_find_user` (**`auth/ldap.py:174`**) rejects only `userAccountControl & 0x2` at **`:197-201`**; the attribute list at **`:185-192`** requests `userAccountControl` but **not** `accountExpires`, `lockoutTime` or `UF_LOCKOUT` — verified by reading the whole function via AST, which contains none of those three tokens. `resolve_principal` (**`:277`**) calls `_find_user` and adds no state check of its own, and it is the password-free path used by Kerberos SSO, OIDC **and** the ADR-0079 reconciler via `_probe_principal` (**`service.py:1244`**, calling it at `:1253`) — so all three accept such an account. `service.py:959` states the shared path in as many words: *"Roles come from `resolve_principal` — the same password-free LDAP lookup Kerberos uses"* | An AD account that is expired or locked but not explicitly disabled keeps a live engine session (the reconciler probes it `PRESENT` every pass) and can federate in. Offboarding-by-expiry — the common HR pattern — does not propagate | Every offboarded-by-expiry operator retains console access | **No.** `tests/test_auth_hardening.py:415-455` pins the disabled bit only | **P0** | | SQL Server never exercises the AD group → per-connection scope map (`tests/test_sqlserver_store.py:423-424` covers `ad_group_role_map` only); PostgreSQL has one line (`test_postgres_store.py:769-770`); the store-side `allowed_channels` message filter is SQLite-only (`tests/test_channel_rbac.py:99`) | A backend-specific bug returns an empty set; `_sync_ad_channel_scope` (`auth/service.py:1147-1148`) then leaves the scope untouched **by design** and the AD operator silently keeps the all-connections default | Cross-connection PHI exposure on both production backends | **No** | **P1** | | No meta-guard that a request-scoped audit call threads `client=`. 66 `record_audit(` call sites in `messagefoundry/` (51 in `api/`); the chain hashes a missing client as the legacy 6-element payload (`tests/test_audit_integrity.py:391`), so an omission **verifies perfectly clean** | A new PHI-read or admin-write route that forgets `client=client_ip(request)` silently loses the ADR-0150 "from where" — the exact incident question the ADR was written to answer | Post-incident forensics on any new route | **No** | **P1** | From 3c4acd8a7b869b08e0f9d213f3f888004c8d9c8a Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 06:23:06 -0500 Subject: [PATCH 18/66] docs(testing): confirm the mock-seam P0, and correct the one clause a reader could refute it with (BACKLOG #1100) 10-auth-rbac-and-active-directory.md row 90 is substantively CORRECT. All six acceptor sites are still pragma: no cover, annotated "needs real AD" and "requires a domain-joined server + keytab": ldap3 LDAPException arms :267 :289 (cited :252, :274) SPNEGO acceptor lines :313 :321 :373 :378 (cited :298, :306, :358, :363) Zero workflows mention ldap, and there is no OpenLDAP / Samba / 389ds service container anywhere. ONE CLAUSE IS FALSE AS WRITTEN: "no self-hosted runner ... exists in .github/workflows/". One does -- selfhosted-win2025-sql.yml :: sqlserver-real, "SQL Server 2025 suites (real hardware)". It mentions no ldap, kerberos, spnego, ad_domain or directory, so the row's INTENT holds exactly; its literal claim does not. That is worth fixing in both directions. Defensively: as written, a reader can refute a P0 by pointing at one file, and a row that can be dismissed on a technicality does not survive the meeting it matters in. Constructively: A SELF-HOSTED WINDOWS RUNNER WITH A LABEL SCHEME ALREADY EXISTS AND IS PROVEN, so the remedy here is "point an existing pattern at a directory", not "stand up runner infrastructure". The row previously implied the expensive half was missing when it is already built. This is the sixth row in the sweep that understated something -- coverage, its own gap, or as here, how close the fix already is. --- .../master-test-plan/10-auth-rbac-and-active-directory.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md b/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md index dc77957e3..471d3c9d7 100644 --- a/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md +++ b/docs/testing/master-test-plan/10-auth-rbac-and-active-directory.md @@ -87,7 +87,7 @@ This chapter covers the whole authentication/authorization core and, specificall | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| The real-directory acceptor path is mock-seam only. The `ldap3` `LDAPException` arms (`auth/ldap.py:252`, `:274`) and every SPNEGO acceptor line (`:298`, `:306`, `:358`, `:363`) are `# pragma: no cover`. No CI leg, no self-hosted runner, no containerised LDAP exists in `.github/workflows/` | A change to the `ldap3` / `pyspnego` call shape, or a dependency bump, ships green while AD login and Windows SSO are broken — or silently permissive | Every enterprise deployment's primary authentication path | **No.** `HANDOFF-AD-LAB-aws.md` states it outright: "The entire AD acceptor path is mock-seam only" | **P0** | +| The real-directory acceptor path is mock-seam only. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100); anchors re-pointed and ONE clause corrected.** The `ldap3` `LDAPException` arms are at **`auth/ldap.py:267` and `:289`** (cited `:252`/`:274`) and the SPNEGO acceptor lines at **`:313`, `:321`, `:373`, `:378`** (cited `:298`/`:306`/`:358`/`:363`) — all six still `# pragma: no cover`, annotated *"needs real AD"* and *"requires a domain-joined server + keytab"*. **Zero** workflows mention `ldap`, and there is **no** OpenLDAP/Samba/389ds service container anywhere. **CORRECTION: "no self-hosted runner … exists" is FALSE.** One does — `selfhosted-win2025-sql.yml :: sqlserver-real` ("SQL Server 2025 suites, real hardware") — it simply mentions no `ldap`/`kerberos`/`spnego`/`ad_domain`/`directory`. **That is worth stating in BOTH directions:** as written, a reader can refute this P0 by pointing at that file and dismiss the whole row; and constructively, **the expensive prerequisite for an AD-lab leg is already built and proven** — a self-hosted Windows runner with a label scheme — so the remedy is "point an existing pattern at a directory", not "stand up runner infrastructure" | A change to the `ldap3` / `pyspnego` call shape, or a dependency bump, ships green while AD login and Windows SSO are broken — or silently permissive | Every enterprise deployment's primary authentication path | **No.** `HANDOFF-AD-LAB-aws.md` states it outright: "The entire AD acceptor path is mock-seam only" | **P0** | | **CONFIRMED live SPN defect** (was "suspected"; verified against the library itself 2026-08-15, BACKLOG #1100). Both acceptor sites pass the *whole* SPN to `pyspnego`'s `service=`: `spnego.server(service=settings.kerberos_spn)` at **`auth/ldap.py:315` and `:375`** (anchors drifted from `:300`/`:360`), while **`config/settings.py:1905`** (drifted from `:1846`) documents `kerberos_spn` as `HTTP/host.example.com`. **The library's own signature settles it:** `spnego.server(hostname: str = 'unspecified', service: str = 'host', ...)`, documented as *"hostname: The principal part of the SPN"* and *"service: The service part of the SPN"* — pyspnego **builds** the SPN from the two. **AND IT IS WORSE THAN THIS ROW ASSUMED: `hostname` is never passed at either site**, so it takes its default and the acceptor principal is `HTTP/host.example.com/unspecified`, not `.../`. **Scope, stated honestly:** `kerberos_enabled` defaults to `False` and is marked experimental (`settings.py:1904`), so this **would** bite the first site that enables Kerberos SSO rather than affecting anything today | Kerberos SSO can never authenticate against a real KDC, yet `kerberos_acceptor_preflight` reports the acceptor healthy and `GET /auth/providers` advertises `kerberos=true` | Windows SSO estate-wide; a working-looking feature that always fails, every failure an audited generic reject | **No test can see it.** Both sites are `pragma: no cover` | **P0** | | ADR 0142 has never met a real IdP. Status line: "Proposed — **code COMPLETE, awaiting lab validation**"; cells L6a, L9, L18 have not run; no AD-lab run record exists under `docs/testing/`. **PARTIALLY RE-VERIFIED 2026-08-15 (BACKLOG #1100), and the unverifiable part is named rather than assumed.** CONFIRMED: the ADR's status line still reads exactly that, and its inline note adds *"flips to Accepted only when runbook cells L6a, L9 and L18 report"*; and `docs/testing/` holds 26 files, all plan documents, with no run record among them. **NOT CHECKED, AND NOT CHECKABLE FROM THIS REPOSITORY:** whether L6a/L9/L18 have since run. The runbook is `docs/security/AD-FEDERATION-LAB-RUNBOOK.md`, and **`docs/security/` does not exist in this checkout at all** — it is the private companion repo. So this row's middle clause can only be graded by someone with the vault, and a reader here should treat it as *unknown*, not as *confirmed open*. The ADR status is the honest public proxy: it has not flipped | L18 is the real-IdP proof of the AC-11 username UPN-suffix binding — a review found the unchecked-suffix path was a **live** privilege-escalation route (a guest presenting `Administrator@attacker.example` resolving to the on-prem Domain Admin). L9 can invalidate the architecture: a passwordless/smartcard AD account cannot complete the password step-up, so every sensitive route permanently 403s | Federated sign-in as a whole; potentially a domain-admin takeover | **No** | **P0** | | AD account states beyond `ACCOUNTDISABLE` are never consulted. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100) — every claim holds; only the anchors had drifted, and all four are re-pointed here.** `_find_user` (**`auth/ldap.py:174`**) rejects only `userAccountControl & 0x2` at **`:197-201`**; the attribute list at **`:185-192`** requests `userAccountControl` but **not** `accountExpires`, `lockoutTime` or `UF_LOCKOUT` — verified by reading the whole function via AST, which contains none of those three tokens. `resolve_principal` (**`:277`**) calls `_find_user` and adds no state check of its own, and it is the password-free path used by Kerberos SSO, OIDC **and** the ADR-0079 reconciler via `_probe_principal` (**`service.py:1244`**, calling it at `:1253`) — so all three accept such an account. `service.py:959` states the shared path in as many words: *"Roles come from `resolve_principal` — the same password-free LDAP lookup Kerberos uses"* | An AD account that is expired or locked but not explicitly disabled keeps a live engine session (the reconciler probes it `PRESENT` every pass) and can federate in. Offboarding-by-expiry — the common HR pattern — does not propagate | Every offboarded-by-expiry operator retains console access | **No.** `tests/test_auth_hardening.py:415-455` pins the disabled bit only | **P0** | From 480869dd213d68cc1cdd188979c265f2e7b70b5d Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 06:34:00 -0500 Subject: [PATCH 19/66] docs(testing): confirm the PHI retention-gate bypass and re-point it at the moved gate (BACKLOG #1100) 03-store-and-data-lifecycle.md R3 is CONFIRMED. The gate has MOVED since the row was written, which is why its anchor no longer showed the defect: cited __main__.py:1968-1975 "builds unbounded_windows" actual config/retention_classification.py:186 defines it (20 lines) __main__.py:2240 is its ONLY caller: _unbounded_windows(settings) Read whole via AST, unbounded_windows contains no registry / connection / per_connection / overrides / inbound token. It sees global settings and nothing else. The override that defeats it is documented in the code that creates it, wiring.py:3125-3126: "None = inherit the global [retention].messages_days window; 0 = keep this connection's bodies forever; >0 = days" -- and pipeline/retention.py:150 honours it at purge time. BOTH HALVES ARE INDIVIDUALLY CORRECT AND JOINTLY PERMISSIVE. A deploying PHI instance with a global 30-day window would pass the startup gate while every inbound set to messages_days=0 retains bodies forever, with no warning, no audit entry and no test. Written conditionally per CLAUDE.md section 0: there are no deployments, so nothing retains PHI today; it is wrong in the shipped code. This is the most consequential of the three code defects this sweep has confirmed, because the control is LABELLED fail-closed and audited (ASVS 14.2.4) -- its existence is exactly what a reviewer would rely on instead of checking. That is the compensating-control-on-a-false-premise shape in its purest form. Not fixed: store/config surface, outside this claim. The row names the fix -- hand the gate the registry's resolved per-connection windows rather than settings alone. --- docs/testing/master-test-plan/03-store-and-data-lifecycle.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/03-store-and-data-lifecycle.md b/docs/testing/master-test-plan/03-store-and-data-lifecycle.md index 87d33db03..78a3ee268 100644 --- a/docs/testing/master-test-plan/03-store-and-data-lifecycle.md +++ b/docs/testing/master-test-plan/03-store-and-data-lifecycle.md @@ -105,7 +105,7 @@ the scheduling rows below it were never corrected). |---|---|---|---|---| | **R1** ADR 0027 per-connection purge `CASE` has never run on PG/SQL Server (`connection_cutoffs` appears **0** times in `tests/test_postgres_store.py` and `tests/test_sqlserver_store.py`; the AC-8 parity case lives only in `tests/test_per_connection_retention.py:262`, which no CI step names and which the `serverdb` regex at `ci.yml:434` does not match) | A wrong `_pg_cutoff_case` / `_qmark_cutoff_case` predicate purges the **wrong feed's** PHI bodies (irreversible; the count-and-log row survives so nothing looks broken) or silently purges nothing (unbounded PHI at rest) | Every PHI feed on the production backend; irreversible either way | **No.** SQLite stays green in both directions | **P0** | | **R2** The shipped at-rest **writer** is barely exercised. `[store].aad_bind` defaults `True` → `open_store` builds a `write_v2` / `mfenc:v2` cell-AAD cipher ([base.py:1736](../../../messagefoundry/store/base.py)), but `make_cipher`'s library default is `write_v2=False` and **no** workflow sets `MEFOR_TEST_FORCE_AAD_BIND` (only `tests/conftest.py:126,138` and ADR 0019:516 mention it) | An unbound or mis-threaded `cell_aad` on any write path — purge re-encrypt, document strip write-back, attachment re-seal, restore — yields rows the shipped cipher cannot decrypt (unreadable PHI) or silently drops the binding (ASVS 11.3.3 regression) | Whole store; discovered only in production | **No.** 13 SQLite-only targeted tests | **P0** | -| **R3** The PHI serve retention gate reads **only global** windows ([__main__.py:1968-1975](../../../messagefoundry/__main__.py) builds `unbounded_windows` from `settings.retention.messages_days` / `dead_letter_days` alone) and never consults the registry's per-connection overrides | A PHI instance with a global 30-day window and every inbound at `messages_days=0` passes the fail-closed gate and retains PHI forever | An audited, security-labelled fail-closed control (ASVS 14.2.4) is bypassable by ordinary Connection config | **No.** No warning, no audit entry, no test | **P0** | +| **R3** The PHI serve retention gate reads **only global** windows and never consults the registry's per-connection overrides. **CONFIRMED 2026-08-15 (BACKLOG #1100); anchors re-pointed — the gate MOVED and is no longer built inline.** `unbounded_windows` is now defined at [`config/retention_classification.py:186`](../../../messagefoundry/config/retention_classification.py) (20 lines) and called from [`__main__.py:2240`](../../../messagefoundry/__main__.py) as `_unbounded_windows(settings)` — **`settings` alone, no registry**. Read whole via AST, the function contains **no** `registry`, `connection`, `per_connection`, `overrides` or `inbound` token, and `__main__.py:2240` is its **only** caller in the package. **The bypass is documented in the code that creates it** (`wiring.py:3125-3126`): *"Per-connection retention override (#34, ADR 0027): None = inherit the global `[retention].messages_days` window; **0 = keep this connection's bodies forever**; >0 = days"* — and that override **is** honoured at purge time (`pipeline/retention.py:150`, "inbound name -> messages_days"). So the two halves are individually correct and jointly permissive: **a deploying PHI instance would pass the gate on its global window while retaining every body forever per-connection** | A PHI instance with a global 30-day window and every inbound at `messages_days=0` passes the fail-closed gate and retains PHI forever | An audited, security-labelled fail-closed control (ASVS 14.2.4) is bypassable by ordinary Connection config | **No.** No warning, no audit entry, no test | **P0** | | **R4** ADR 0042 `strip_embedded_documents` has never run on PG/SQL Server (**0** occurrences in both server suites) — and it is a select → decrypt → codec-transform → **re-encrypt write-back** over stored PHI bodies ([postgres.py:6395](../../../messagefoundry/store/postgres.py), [sqlserver.py:5626](../../../messagefoundry/store/sqlserver.py)) | A dialect or write-back bug corrupts stored bodies (unparseable HL7) or leaves bulky base64 PHI in place forever | Every document feed on the production backend | **No** | **P1** | | **R5** `strip_embedded_documents` materializes and **decrypts every eligible row with no LIMIT/TOP/batch** on all three backends ([store.py:8465](../../../messagefoundry/store/store.py), [postgres.py:6420](../../../messagefoundry/store/postgres.py), [sqlserver.py:5650](../../../messagefoundry/store/sqlserver.py)); the ADR 0137 deadline is checked only **before** the phase ([retention.py:443](../../../messagefoundry/pipeline/retention.py)), never inside the per-threshold loop | The first pass after enabling `prune_documents_after` pulls the entire un-stripped backlog of full PHI bodies into engine heap: OOM / engine crash mid-purge, unbounded PHI plaintext in memory, and a pass that blows straight through `max_pass_seconds` | Engine availability + PHI-in-heap exposure at exactly the moment PHI is being reduced | **No.** No test bounds the candidate set | **P1** | | **R6** Purge under sustained load is untested on every backend. SQLite holds the single writer lock across the whole multi-statement purge transaction (`async with self._lock`, [store.py:8345](../../../messagefoundry/store/store.py)); SQL Server builds an unbatched `#eligible` temp table ([sqlserver.py:5566](../../../messagefoundry/store/sqlserver.py)) and **`messages` has no `LOCK_ESCALATION=DISABLE`** (only `queue` does, [sqlserver.py:1057-1063](../../../messagefoundry/store/sqlserver.py)) | A first large purge escalates to a table X lock on `messages` and blocks ingress inserts, stalling ACKs past the MLLP receive timeout | Availability incident on the production backend, during a scheduled maintenance window | **No.** No concurrent-purge test anywhere; no purge profile in `docs/LOAD-TESTING.md`; no metric to see it coming | **P1** | From b16e5c1083cfea153654abafb41532d5ae3beb8a Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 06:43:15 -0500 Subject: [PATCH 20/66] docs(testing): narrow the at-rest AAD row from "barely exercised" to what is actually missing (BACKLOG #1100) 03-store-and-data-lifecycle.md R2's core claim is CONFIRMED: no workflow sets MEFOR_TEST_FORCE_AAD_BIND -- 0 occurrences across .github/workflows/, against a positive control of 14 for MEFOR_TEST_SQLSERVER. aad_bind still defaults True (settings.py:388) and make_cipher's library default is still write_v2=False (crypto.py:812). BUT "THE SHIPPED WRITER IS BARELY EXERCISED" NOW OVERSTATES IT, and the flag's own docstring (tests/conftest.py:133-138) is the authority: "The flag is OFF by default and stays meaningful even though [store].aad_bind now DEFAULTS TRUE (ADR 0148 GIVEN 1). ... The settings default governs what open_store builds; this flag governs every cipher in the process, which is what makes the sweep exhaustive rather than merely representative." The flag is a SWEEP AMPLIFIER, not a gate: unset, the suites still run and open_store still builds the mfenc:v2 writer because the setting defaults True. So the shipped writer IS exercised. What never runs is the process-wide forcing that also catches ciphers constructed with an EXPLICIT write_v2=False. The parenthetical was wrong too: EIGHT test files reference the flag, not conftest.py alone. STAYS P0. The residual risk is exactly as stated -- a half-threaded cell_aad on purge re-encrypt, document strip write-back, attachment re-seal or restore surfaces only under the forced sweep, and that sweep runs nowhere. Narrowing the headline does not shrink the gap; it stops the row being refutable by anyone who checks whether the writer runs at all. --- docs/testing/master-test-plan/03-store-and-data-lifecycle.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/03-store-and-data-lifecycle.md b/docs/testing/master-test-plan/03-store-and-data-lifecycle.md index 78a3ee268..4b3dd839e 100644 --- a/docs/testing/master-test-plan/03-store-and-data-lifecycle.md +++ b/docs/testing/master-test-plan/03-store-and-data-lifecycle.md @@ -104,7 +104,7 @@ the scheduling rows below it were never corrected). | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | **R1** ADR 0027 per-connection purge `CASE` has never run on PG/SQL Server (`connection_cutoffs` appears **0** times in `tests/test_postgres_store.py` and `tests/test_sqlserver_store.py`; the AC-8 parity case lives only in `tests/test_per_connection_retention.py:262`, which no CI step names and which the `serverdb` regex at `ci.yml:434` does not match) | A wrong `_pg_cutoff_case` / `_qmark_cutoff_case` predicate purges the **wrong feed's** PHI bodies (irreversible; the count-and-log row survives so nothing looks broken) or silently purges nothing (unbounded PHI at rest) | Every PHI feed on the production backend; irreversible either way | **No.** SQLite stays green in both directions | **P0** | -| **R2** The shipped at-rest **writer** is barely exercised. `[store].aad_bind` defaults `True` → `open_store` builds a `write_v2` / `mfenc:v2` cell-AAD cipher ([base.py:1736](../../../messagefoundry/store/base.py)), but `make_cipher`'s library default is `write_v2=False` and **no** workflow sets `MEFOR_TEST_FORCE_AAD_BIND` (only `tests/conftest.py:126,138` and ADR 0019:516 mention it) | An unbound or mis-threaded `cell_aad` on any write path — purge re-encrypt, document strip write-back, attachment re-seal, restore — yields rows the shipped cipher cannot decrypt (unreadable PHI) or silently drops the binding (ASVS 11.3.3 regression) | Whole store; discovered only in production | **No.** 13 SQLite-only targeted tests | **P0** | +| **R2** The **exhaustive** at-rest AAD sweep never runs in CI (was: "the shipped writer is barely exercised" — **narrowed 2026-08-15, BACKLOG #1100**) | **CONFIRMED, and the framing corrected.** Still true: `[store].aad_bind` defaults `True` (`settings.py:388`), `make_cipher`'s library default is `write_v2=False` (`crypto.py:812`), and **no workflow sets `MEFOR_TEST_FORCE_AAD_BIND`** — **0** occurrences across `.github/workflows/` against a positive control of **14** for `MEFOR_TEST_SQLSERVER`. **But "barely exercised" now overstates it**, and the flag's own docstring (`tests/conftest.py:133-138`) says why: *"The flag is OFF by default and stays meaningful even though `[store].aad_bind` now DEFAULTS TRUE (ADR 0148 GIVEN 1)... The settings default governs what `open_store` builds; **this flag governs every cipher in the process, which is what makes the sweep exhaustive rather than merely representative**."* So the shipped `mfenc:v2` writer **is** exercised by every ordinary store test via the default; what is missing is the process-wide forcing that also catches ciphers built with an **explicit** `write_v2=False`. **The parenthetical was also wrong: EIGHT test files reference the flag**, not `conftest.py` alone — `test_store_aad_binding`, `test_store_encryption`, `test_transform_state`, `test_ack_sent_store`, `test_alert_state`, `test_connection_event_store`, `test_ed_documents_e2e`, `test_sqlserver_store`. **The residual risk is unchanged and still P0:** a half-threaded `cell_aad` on purge re-encrypt, document strip write-back, attachment re-seal or restore would surface only under the forced sweep, and that sweep runs nowhere | An unbound or mis-threaded `cell_aad` on any write path — purge re-encrypt, document strip write-back, attachment re-seal, restore — yields rows the shipped cipher cannot decrypt (unreadable PHI) or silently drops the binding (ASVS 11.3.3 regression) | Whole store; discovered only in production | **No.** 13 SQLite-only targeted tests | **P0** | | **R3** The PHI serve retention gate reads **only global** windows and never consults the registry's per-connection overrides. **CONFIRMED 2026-08-15 (BACKLOG #1100); anchors re-pointed — the gate MOVED and is no longer built inline.** `unbounded_windows` is now defined at [`config/retention_classification.py:186`](../../../messagefoundry/config/retention_classification.py) (20 lines) and called from [`__main__.py:2240`](../../../messagefoundry/__main__.py) as `_unbounded_windows(settings)` — **`settings` alone, no registry**. Read whole via AST, the function contains **no** `registry`, `connection`, `per_connection`, `overrides` or `inbound` token, and `__main__.py:2240` is its **only** caller in the package. **The bypass is documented in the code that creates it** (`wiring.py:3125-3126`): *"Per-connection retention override (#34, ADR 0027): None = inherit the global `[retention].messages_days` window; **0 = keep this connection's bodies forever**; >0 = days"* — and that override **is** honoured at purge time (`pipeline/retention.py:150`, "inbound name -> messages_days"). So the two halves are individually correct and jointly permissive: **a deploying PHI instance would pass the gate on its global window while retaining every body forever per-connection** | A PHI instance with a global 30-day window and every inbound at `messages_days=0` passes the fail-closed gate and retains PHI forever | An audited, security-labelled fail-closed control (ASVS 14.2.4) is bypassable by ordinary Connection config | **No.** No warning, no audit entry, no test | **P0** | | **R4** ADR 0042 `strip_embedded_documents` has never run on PG/SQL Server (**0** occurrences in both server suites) — and it is a select → decrypt → codec-transform → **re-encrypt write-back** over stored PHI bodies ([postgres.py:6395](../../../messagefoundry/store/postgres.py), [sqlserver.py:5626](../../../messagefoundry/store/sqlserver.py)) | A dialect or write-back bug corrupts stored bodies (unparseable HL7) or leaves bulky base64 PHI in place forever | Every document feed on the production backend | **No** | **P1** | | **R5** `strip_embedded_documents` materializes and **decrypts every eligible row with no LIMIT/TOP/batch** on all three backends ([store.py:8465](../../../messagefoundry/store/store.py), [postgres.py:6420](../../../messagefoundry/store/postgres.py), [sqlserver.py:5650](../../../messagefoundry/store/sqlserver.py)); the ADR 0137 deadline is checked only **before** the phase ([retention.py:443](../../../messagefoundry/pipeline/retention.py)), never inside the per-threshold loop | The first pass after enabling `prune_documents_after` pulls the entire un-stripped backlog of full PHI bodies into engine heap: OOM / engine crash mid-purge, unbounded PHI plaintext in memory, and a pass that blows straight through `max_pass_seconds` | Engine availability + PHI-in-heap exposure at exactly the moment PHI is being reduced | **No.** No test bounds the candidate set | **P1** | From d027fba16f240bc8c5c4fbad0d0590418651c547 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 06:43:52 -0500 Subject: [PATCH 21/66] fix(testing): repair the R2 row I split into six columns in eb32618f (BACKLOG #1100) eb32618f introduced a stray `|` in 03-store-and-data-lifecycle.md row 107, turning a 5-column row into 6 and breaking the risk table's alignment from that row on. Replaced the stray boundary with a full stop; header and both neighbouring rows now agree at 5. I RAN THE COLUMN CHECK AND COMMITTED ANYWAY. The check printed "row107: 6" against a header of 5, in the same command that staged and committed. Running a verification and GATING on it are different things, and batching them into one command guaranteed the output arrived after the decision it was supposed to inform. That is the same instrument-vs-question failure this whole sweep keeps finding, in its most avoidable form: the measurement was correct, present, and ignored. Every other near-miss tonight needed a subtle reason -- a truncated window, a pipe swallowing an exit code, a docstring quoting the claim it refutes. This one needed only that I put the check and the commit in the same breath. Splitting them from here: verify, read the result, then stage. --- docs/testing/master-test-plan/03-store-and-data-lifecycle.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/03-store-and-data-lifecycle.md b/docs/testing/master-test-plan/03-store-and-data-lifecycle.md index 4b3dd839e..3c1926cd4 100644 --- a/docs/testing/master-test-plan/03-store-and-data-lifecycle.md +++ b/docs/testing/master-test-plan/03-store-and-data-lifecycle.md @@ -104,7 +104,7 @@ the scheduling rows below it were never corrected). | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | **R1** ADR 0027 per-connection purge `CASE` has never run on PG/SQL Server (`connection_cutoffs` appears **0** times in `tests/test_postgres_store.py` and `tests/test_sqlserver_store.py`; the AC-8 parity case lives only in `tests/test_per_connection_retention.py:262`, which no CI step names and which the `serverdb` regex at `ci.yml:434` does not match) | A wrong `_pg_cutoff_case` / `_qmark_cutoff_case` predicate purges the **wrong feed's** PHI bodies (irreversible; the count-and-log row survives so nothing looks broken) or silently purges nothing (unbounded PHI at rest) | Every PHI feed on the production backend; irreversible either way | **No.** SQLite stays green in both directions | **P0** | -| **R2** The **exhaustive** at-rest AAD sweep never runs in CI (was: "the shipped writer is barely exercised" — **narrowed 2026-08-15, BACKLOG #1100**) | **CONFIRMED, and the framing corrected.** Still true: `[store].aad_bind` defaults `True` (`settings.py:388`), `make_cipher`'s library default is `write_v2=False` (`crypto.py:812`), and **no workflow sets `MEFOR_TEST_FORCE_AAD_BIND`** — **0** occurrences across `.github/workflows/` against a positive control of **14** for `MEFOR_TEST_SQLSERVER`. **But "barely exercised" now overstates it**, and the flag's own docstring (`tests/conftest.py:133-138`) says why: *"The flag is OFF by default and stays meaningful even though `[store].aad_bind` now DEFAULTS TRUE (ADR 0148 GIVEN 1)... The settings default governs what `open_store` builds; **this flag governs every cipher in the process, which is what makes the sweep exhaustive rather than merely representative**."* So the shipped `mfenc:v2` writer **is** exercised by every ordinary store test via the default; what is missing is the process-wide forcing that also catches ciphers built with an **explicit** `write_v2=False`. **The parenthetical was also wrong: EIGHT test files reference the flag**, not `conftest.py` alone — `test_store_aad_binding`, `test_store_encryption`, `test_transform_state`, `test_ack_sent_store`, `test_alert_state`, `test_connection_event_store`, `test_ed_documents_e2e`, `test_sqlserver_store`. **The residual risk is unchanged and still P0:** a half-threaded `cell_aad` on purge re-encrypt, document strip write-back, attachment re-seal or restore would surface only under the forced sweep, and that sweep runs nowhere | An unbound or mis-threaded `cell_aad` on any write path — purge re-encrypt, document strip write-back, attachment re-seal, restore — yields rows the shipped cipher cannot decrypt (unreadable PHI) or silently drops the binding (ASVS 11.3.3 regression) | Whole store; discovered only in production | **No.** 13 SQLite-only targeted tests | **P0** | +| **R2** The **exhaustive** at-rest AAD sweep never runs in CI (was: "the shipped writer is barely exercised" — **narrowed 2026-08-15, BACKLOG #1100**). **CONFIRMED, and the framing corrected.** Still true: `[store].aad_bind` defaults `True` (`settings.py:388`), `make_cipher`'s library default is `write_v2=False` (`crypto.py:812`), and **no workflow sets `MEFOR_TEST_FORCE_AAD_BIND`** — **0** occurrences across `.github/workflows/` against a positive control of **14** for `MEFOR_TEST_SQLSERVER`. **But "barely exercised" now overstates it**, and the flag's own docstring (`tests/conftest.py:133-138`) says why: *"The flag is OFF by default and stays meaningful even though `[store].aad_bind` now DEFAULTS TRUE (ADR 0148 GIVEN 1)... The settings default governs what `open_store` builds; **this flag governs every cipher in the process, which is what makes the sweep exhaustive rather than merely representative**."* So the shipped `mfenc:v2` writer **is** exercised by every ordinary store test via the default; what is missing is the process-wide forcing that also catches ciphers built with an **explicit** `write_v2=False`. **The parenthetical was also wrong: EIGHT test files reference the flag**, not `conftest.py` alone — `test_store_aad_binding`, `test_store_encryption`, `test_transform_state`, `test_ack_sent_store`, `test_alert_state`, `test_connection_event_store`, `test_ed_documents_e2e`, `test_sqlserver_store`. **The residual risk is unchanged and still P0:** a half-threaded `cell_aad` on purge re-encrypt, document strip write-back, attachment re-seal or restore would surface only under the forced sweep, and that sweep runs nowhere | An unbound or mis-threaded `cell_aad` on any write path — purge re-encrypt, document strip write-back, attachment re-seal, restore — yields rows the shipped cipher cannot decrypt (unreadable PHI) or silently drops the binding (ASVS 11.3.3 regression) | Whole store; discovered only in production | **No.** 13 SQLite-only targeted tests | **P0** | | **R3** The PHI serve retention gate reads **only global** windows and never consults the registry's per-connection overrides. **CONFIRMED 2026-08-15 (BACKLOG #1100); anchors re-pointed — the gate MOVED and is no longer built inline.** `unbounded_windows` is now defined at [`config/retention_classification.py:186`](../../../messagefoundry/config/retention_classification.py) (20 lines) and called from [`__main__.py:2240`](../../../messagefoundry/__main__.py) as `_unbounded_windows(settings)` — **`settings` alone, no registry**. Read whole via AST, the function contains **no** `registry`, `connection`, `per_connection`, `overrides` or `inbound` token, and `__main__.py:2240` is its **only** caller in the package. **The bypass is documented in the code that creates it** (`wiring.py:3125-3126`): *"Per-connection retention override (#34, ADR 0027): None = inherit the global `[retention].messages_days` window; **0 = keep this connection's bodies forever**; >0 = days"* — and that override **is** honoured at purge time (`pipeline/retention.py:150`, "inbound name -> messages_days"). So the two halves are individually correct and jointly permissive: **a deploying PHI instance would pass the gate on its global window while retaining every body forever per-connection** | A PHI instance with a global 30-day window and every inbound at `messages_days=0` passes the fail-closed gate and retains PHI forever | An audited, security-labelled fail-closed control (ASVS 14.2.4) is bypassable by ordinary Connection config | **No.** No warning, no audit entry, no test | **P0** | | **R4** ADR 0042 `strip_embedded_documents` has never run on PG/SQL Server (**0** occurrences in both server suites) — and it is a select → decrypt → codec-transform → **re-encrypt write-back** over stored PHI bodies ([postgres.py:6395](../../../messagefoundry/store/postgres.py), [sqlserver.py:5626](../../../messagefoundry/store/sqlserver.py)) | A dialect or write-back bug corrupts stored bodies (unparseable HL7) or leaves bulky base64 PHI in place forever | Every document feed on the production backend | **No** | **P1** | | **R5** `strip_embedded_documents` materializes and **decrypts every eligible row with no LIMIT/TOP/batch** on all three backends ([store.py:8465](../../../messagefoundry/store/store.py), [postgres.py:6420](../../../messagefoundry/store/postgres.py), [sqlserver.py:5650](../../../messagefoundry/store/sqlserver.py)); the ADR 0137 deadline is checked only **before** the phase ([retention.py:443](../../../messagefoundry/pipeline/retention.py)), never inside the per-threshold loop | The first pass after enabling `prune_documents_after` pulls the entire un-stripped backlog of full PHI bodies into engine heap: OOM / engine crash mid-purge, unbounded PHI plaintext in memory, and a pass that blows straight through `max_pass_seconds` | Engine availability + PHI-in-heap exposure at exactly the moment PHI is being reduced | **No.** No test bounds the candidate set | **P1** | From d2de2289c414248b1ede92965c48e9f8374d622e Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 06:52:55 -0500 Subject: [PATCH 22/66] fix(testing): restore the G4 row's missing column, and sweep all 172 tables (BACKLOG #1100) After breaking a row in eb32618f I checked whether I had done it elsewhere. I had. 15-alerting-and-observability.md:138 -- my FIRST #1100 commit tonight, 1e926ec9 -- merged the Risk and Failure-mode cells into one, leaving a 4-column row under a 5-column header (Risk | Failure mode | Blast radius | Detected today? | Priority). It has been misaligned for the whole session. Split at the natural boundary: the verdict sentence stays in Risk, the send_plain_email evidence moves to Failure mode. THE POINT IS NOT THE TWO ROWS, IT IS THAT I ONLY EVER CHECKED THE ROW IN FRONT OF ME. The per-edit column check I adopted mid-session cannot see damage done before I adopted it, and never looked at the other 171 tables. A check scoped to the thing you are currently touching is blind to your own history with the file. Swept all 20 chapters: 172 tables checked, 0 misaligned rows remaining. Process corrected as well as the rows: the integrity check now runs as its own step and I read the result BEFORE staging. In eb32618f the check and the commit were one command, so the output that would have stopped me arrived after the decision it was meant to inform. --- docs/testing/master-test-plan/15-alerting-and-observability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/15-alerting-and-observability.md b/docs/testing/master-test-plan/15-alerting-and-observability.md index fd61a428f..60e92fe7a 100644 --- a/docs/testing/master-test-plan/15-alerting-and-observability.md +++ b/docs/testing/master-test-plan/15-alerting-and-observability.md @@ -135,7 +135,7 @@ ALERT-13 and ALERT-11. | **G1 — three alert vocabularies bound only by convention** | An emit site's `type` literal, `_ALERT_EVENT_TYPES`, and the `AlertSink` Protocol / `LoggingAlertSink` / `NotifierAlertSink` method sets drift. A type absent from the routable set can never be escalated, routed, suppressed or muted by any operator rule; a method present only on `NotifierAlertSink` raises `AttributeError` on the logging fallback | Every deployment. Drift has fired **twice** already: `lane_stuck` + `rcsi_off_degraded` once shipped unroutable — **both are members today** ([`settings.py:2515-2516`](../../../messagefoundry/config/settings.py)), rule-targetable end to end at `test_alert_rules.py:357/370/395`, so the only live drift is that no guard stops the next one; `content_match` is **still** divergent (only on `NotifierAlertSink` at [`alert_sinks.py:676`](../../../messagefoundry/pipeline/alert_sinks.py), absent from the Protocol and the fallback). Note the non-obvious third form: `AlertSink.saturation_rising()` emits type `"saturation"` — method name ≠ event type, so a naive guard would false-fail | **No.** No guard exists in either direction | **P0** | | **G2 — `content_match` has no Handler-reachable surface** | ADR 0133 AC-3 says "WHEN a Handler emits a `content_match`" — but there is no export in `messagefoundry/__init__.py` (unlike `db_lookup`/`fhir_lookup` at lines 32-33/150/153), no injected sink on a Handler context, and no dry-run path. Every test calls `sink.content_match(...)` directly | The differentiating Corepoint "Action Point" parity capability is unusable in practice; the PHI-free-by-contract guarantee has never been exercised through real Handler code | **No** — AC-3/AC-4 pass at the sink so nothing fails | **P0** | | **G3 — `connection_started` is mapped but emitted nowhere** | `_AUTO_RESOLVE["connection_started"] = "connection_stopped"` ([`alert_sinks.py:100`](../../../messagefoundry/pipeline/alert_sinks.py)) but a repo-wide search finds **no emit site**. A lane that STOPs on an internal error and is later restarted (by hand or by a #144 `control_action`) leaves its `alert_instance` permanently `open` | `alerts_active` on the connections dashboard stays non-zero forever; `/alerts/active` accumulates; the nav bell's `list_active_alerts(limit=200)` ([`webconsole/routes/status.py:166`](../../../messagefoundry_webconsole/routes/status.py)) saturates. Textbook alert fatigue — the operator learns to ignore the list and the next real stop is missed | **Partially** — `tests/test_phi_logging_inventory.py:439` *documents* the dead key; nothing asserts the operator consequence | **P0** | -| **G4 — CLOSED by #323 (2026-08-02). The alert/security-notify SMTP hop is VERIFIED.** This row asserted the hop was encrypted but unauthenticated and that `docs/PHI.md` and `docs/BACKLOG.md` contradicted each other about it. **Both halves are false at HEAD** and were already false when this plan was written (BACKLOG #1100). `send_plain_email` builds an explicit verifying context via `tls_policy.build_smtp_tls_context()` and passes it — `smtp.starttls(context=tls_context)` ([`alert_sinks.py:430-431`](../../../messagefoundry/pipeline/alert_sinks.py)), whose comment reads *"context= is REQUIRED (#323): starttls()'s own default verifies NOTHING"*. The `CERT_NONE`/`check_hostname=False` text at `:387-388` is a **historical note about the fixed defect**, not the current posture — reading it as current is the mistake this row made. | — | **Yes.** `docs/PHI.md` row 11 states the verifying posture and records the pre-#323 state explicitly as history; `tests/test_alert_smtp_tls.py` exists with 21 tests. The documents agree with the code and with each other. | **CLOSED** | +| **G4 — CLOSED by #323 (2026-08-02). The alert/security-notify SMTP hop is VERIFIED.** This row asserted the hop was encrypted but unauthenticated and that `docs/PHI.md` and `docs/BACKLOG.md` contradicted each other about it. **Both halves are false at HEAD** and were already false when this plan was written (BACKLOG #1100). | `send_plain_email` builds an explicit verifying context via `tls_policy.build_smtp_tls_context()` and passes it — `smtp.starttls(context=tls_context)` ([`alert_sinks.py:430-431`](../../../messagefoundry/pipeline/alert_sinks.py)), whose comment reads *"context= is REQUIRED (#323): starttls()'s own default verifies NOTHING"*. The `CERT_NONE`/`check_hostname=False` text at `:387-388` is a **historical note about the fixed defect**, not the current posture — reading it as current is the mistake this row made. | — | **Yes.** `docs/PHI.md` row 11 states the verifying posture and records the pre-#323 state explicitly as history; `tests/test_alert_smtp_tls.py` exists with 21 tests. The documents agree with the code and with each other. | **CLOSED** | | **G5 — `[alerts]` is startup-only** | `app.state.alerts_settings` is assigned only at app construction ([`api/app.py:1120`](../../../messagefoundry/api/app.py)) and lifespan startup (`:5485`). `POST /config/reload` (`:2741`) re-runs the `--config` graph, never the service-settings TOML | An operator adds a suppression rule mid-incident via the IDE or `messagefoundry alert add`; the IDE re-lists from the **file** and shows it; `/alerts/rules` still shows the **startup** set; the running notifier keeps paging until a restart. The requirement is documented only in `alerts_edit.py:19-21` and the CLI docstring — nowhere an operator looks | **No** | **P0** | | **G6 — IDE alert editor offers 4 of the 18 event types and 7 of the 15 fields** | [`ide/src/alertEditor.ts:13-19`](../../../ide/src/alertEditor.ts) offers a 5-entry dropdown — `any` plus only `connection_stopped`/`queue_buildup`/`storage_threshold`/`cert_expiry`; `:25-32` supports only `event_type`/`connection`/`min_depth`/`min_oldest_seconds`/`severity`/`transports`/`cooldown_seconds` (7 of the 15 `AlertRule` fields). `ide/src/test/suite/` has **no** alert test file (35 suites, none for the alert editor) | An operator on the supported GUI authoring path cannot rule on **14 of the 18** signals, nor set `id`, `recipients`, `mute`, `escalate`, `schedule`, `content_label`, `control_action`, `control_target` | **No** — the `ide` CI leg runs and tests nothing here | **P1** | | **G7 — no end-to-end drive of the runner's buildup / stall / saturation emit sites** | `_maybe_alert_buildup` (`:5401`), `_maybe_alert_saturation` (`:5444`), `_maybe_alert_stall` (`:5511`) in `wiring_runner.py`. Only the pure `SaturationDetector` and the **engine-shard** non-owned-lane watchdog are tested | These are the three alerts an operator relies on to notice a stalled or drowning feed. Threshold resolution, the `_outbound_paused` suppression guard, the per-`(stage,lane)` `_BUILDUP_REALERT_SECONDS` throttle and the `pending_depth` read could all break silently | **No** — FEATURE-COVERAGE-PLAN `FCP:ALERT-10` flagged it and it is still open | **P1** | From 5a13b305400a62eb9eac0bdfc3103be34134f908 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 07:03:02 -0500 Subject: [PATCH 23/66] docs(testing): confirm the per-connection purge coverage gap, and show it is the DOUBLE kind (BACKLOG #1100) 03-store-and-data-lifecycle.md R1 holds on every claim, measured with a positive control: connection_cutoffs in test_postgres_store.py 0 connection_cutoffs in test_sqlserver_store.py 0 control: it appears in exactly 2 files repo-wide, so the zeros are real serverdb path-gate regex matches per_connection_retention 0 times regex is at ci.yml:993, not the cited :434 TWO ADDITIONS. The row named only test_per_connection_retention.py; the second file carrying connection_cutoffs is test_embedded_document_pruning.py, and it is in the same position. And BOTH are in the 19 MEFOR_TEST_-gated suites that 02-pipeline-reliability.md:145 was corrected to describe -- so the two rows are looking at one CI defect from opposite ends. THE SHAPES ARE NOT EQUIVALENT, AND THIS ONE IS WORSE. The three suites in that other row are MATCHED BY THE GATE BUT RUN BY NO STEP: the expensive server-DB leg fires and then does not test them, which is waste plus false assurance. These two are MATCHED BY NOTHING AND NAMED BY NOTHING: editing the per-connection purge predicate fires no server-DB leg at all, and no test that mentions connection_cutoffs runs anywhere. For a predicate that decides WHICH FEED'S PHI BODIES GET DELETED -- irreversible in one direction, unbounded retention in the other -- "no signal at all" is the weaker of two weak positions. Recorded in the row so the two CI findings can be fixed as one change to the gate regex plus the step lists. --- docs/testing/master-test-plan/03-store-and-data-lifecycle.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/03-store-and-data-lifecycle.md b/docs/testing/master-test-plan/03-store-and-data-lifecycle.md index 3c1926cd4..42edc8fb8 100644 --- a/docs/testing/master-test-plan/03-store-and-data-lifecycle.md +++ b/docs/testing/master-test-plan/03-store-and-data-lifecycle.md @@ -103,7 +103,7 @@ the scheduling rows below it were never corrected). | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| **R1** ADR 0027 per-connection purge `CASE` has never run on PG/SQL Server (`connection_cutoffs` appears **0** times in `tests/test_postgres_store.py` and `tests/test_sqlserver_store.py`; the AC-8 parity case lives only in `tests/test_per_connection_retention.py:262`, which no CI step names and which the `serverdb` regex at `ci.yml:434` does not match) | A wrong `_pg_cutoff_case` / `_qmark_cutoff_case` predicate purges the **wrong feed's** PHI bodies (irreversible; the count-and-log row survives so nothing looks broken) or silently purges nothing (unbounded PHI at rest) | Every PHI feed on the production backend; irreversible either way | **No.** SQLite stays green in both directions | **P0** | +| **R1** ADR 0027 per-connection purge `CASE` has never run on PG/SQL Server. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100) — every claim holds, with a positive control, and the gap is DOUBLE.** `connection_cutoffs` appears **0** times in `tests/test_postgres_store.py` and `tests/test_sqlserver_store.py` (control: it appears in exactly **2** files repo-wide, so the zero is real and not a broken grep). **Those two files are `test_per_connection_retention.py` AND `test_embedded_document_pruning.py`** — the row named only the first. Neither is named by any workflow step, and the `serverdb` path-gate regex (**`ci.yml:993`**, not the cited `:434`) matches `per_connection_retention` **0** times. **THAT IS A STRICTLY WORSE SHAPE THAN THE ONE AT `02-pipeline-reliability.md:145`:** those three suites are *matched by the gate but run by no step*, so the expensive leg at least fires; **these two are matched by nothing and named by nothing**, so editing the per-connection purge predicate triggers no server-DB leg at all and runs no test that mentions `connection_cutoffs` | A wrong `_pg_cutoff_case` / `_qmark_cutoff_case` predicate purges the **wrong feed's** PHI bodies (irreversible; the count-and-log row survives so nothing looks broken) or silently purges nothing (unbounded PHI at rest) | Every PHI feed on the production backend; irreversible either way | **No.** SQLite stays green in both directions | **P0** | | **R2** The **exhaustive** at-rest AAD sweep never runs in CI (was: "the shipped writer is barely exercised" — **narrowed 2026-08-15, BACKLOG #1100**). **CONFIRMED, and the framing corrected.** Still true: `[store].aad_bind` defaults `True` (`settings.py:388`), `make_cipher`'s library default is `write_v2=False` (`crypto.py:812`), and **no workflow sets `MEFOR_TEST_FORCE_AAD_BIND`** — **0** occurrences across `.github/workflows/` against a positive control of **14** for `MEFOR_TEST_SQLSERVER`. **But "barely exercised" now overstates it**, and the flag's own docstring (`tests/conftest.py:133-138`) says why: *"The flag is OFF by default and stays meaningful even though `[store].aad_bind` now DEFAULTS TRUE (ADR 0148 GIVEN 1)... The settings default governs what `open_store` builds; **this flag governs every cipher in the process, which is what makes the sweep exhaustive rather than merely representative**."* So the shipped `mfenc:v2` writer **is** exercised by every ordinary store test via the default; what is missing is the process-wide forcing that also catches ciphers built with an **explicit** `write_v2=False`. **The parenthetical was also wrong: EIGHT test files reference the flag**, not `conftest.py` alone — `test_store_aad_binding`, `test_store_encryption`, `test_transform_state`, `test_ack_sent_store`, `test_alert_state`, `test_connection_event_store`, `test_ed_documents_e2e`, `test_sqlserver_store`. **The residual risk is unchanged and still P0:** a half-threaded `cell_aad` on purge re-encrypt, document strip write-back, attachment re-seal or restore would surface only under the forced sweep, and that sweep runs nowhere | An unbound or mis-threaded `cell_aad` on any write path — purge re-encrypt, document strip write-back, attachment re-seal, restore — yields rows the shipped cipher cannot decrypt (unreadable PHI) or silently drops the binding (ASVS 11.3.3 regression) | Whole store; discovered only in production | **No.** 13 SQLite-only targeted tests | **P0** | | **R3** The PHI serve retention gate reads **only global** windows and never consults the registry's per-connection overrides. **CONFIRMED 2026-08-15 (BACKLOG #1100); anchors re-pointed — the gate MOVED and is no longer built inline.** `unbounded_windows` is now defined at [`config/retention_classification.py:186`](../../../messagefoundry/config/retention_classification.py) (20 lines) and called from [`__main__.py:2240`](../../../messagefoundry/__main__.py) as `_unbounded_windows(settings)` — **`settings` alone, no registry**. Read whole via AST, the function contains **no** `registry`, `connection`, `per_connection`, `overrides` or `inbound` token, and `__main__.py:2240` is its **only** caller in the package. **The bypass is documented in the code that creates it** (`wiring.py:3125-3126`): *"Per-connection retention override (#34, ADR 0027): None = inherit the global `[retention].messages_days` window; **0 = keep this connection's bodies forever**; >0 = days"* — and that override **is** honoured at purge time (`pipeline/retention.py:150`, "inbound name -> messages_days"). So the two halves are individually correct and jointly permissive: **a deploying PHI instance would pass the gate on its global window while retaining every body forever per-connection** | A PHI instance with a global 30-day window and every inbound at `messages_days=0` passes the fail-closed gate and retains PHI forever | An audited, security-labelled fail-closed control (ASVS 14.2.4) is bypassable by ordinary Connection config | **No.** No warning, no audit entry, no test | **P0** | | **R4** ADR 0042 `strip_embedded_documents` has never run on PG/SQL Server (**0** occurrences in both server suites) — and it is a select → decrypt → codec-transform → **re-encrypt write-back** over stored PHI bodies ([postgres.py:6395](../../../messagefoundry/store/postgres.py), [sqlserver.py:5626](../../../messagefoundry/store/sqlserver.py)) | A dialect or write-back bug corrupts stored bodies (unparseable HL7) or leaves bulky base64 PHI in place forever | Every document feed on the production backend | **No** | **P1** | From f70b20e092f274c7f57dd33f982bf7df6757c79f Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 07:14:16 -0500 Subject: [PATCH 24/66] docs(testing): confirm the API wire-contract P0, and flag two tokens that look like counter-evidence (BACKLOG #1100) 09-engine-api.md row 113 is CORRECT on its load-bearing half. tests/test_webconsole_seam_snapshot.py is a digest over rendered-DTO FIELD NAMES and contains no status_code, methods, __annotations__, type_ or optional reference at all. Types, optionality, path, method and status code are unpinned, exactly as the row says. I LEFT THE COUNT ALONE ON PURPOSE. Re-counted by AST walk over all of api/: 117 BaseModel/RootModel classes (models.py 84, auth_models.py 32, phi_gate.py 1) against the "~121" written here. That is inside the row's own tilde, and re-pinning it to 117 would replace one figure that will drift with another. The hedge is doing its job; round 4's lesson was that a HARD count rots, not that every number must be restated. RECORDED FOR THE NEXT CHECKER: two tokens in that test file look like counter-evidence and are not. `annotation` is `from __future__ import annotations` at :14, and `required` is the English word at :76. A grep for either returns a hit in the file that would otherwise settle the question, and suggests types are pinned when nothing of the sort is happening. I spent a step on exactly that detour. That is the same shape as 16-security row 200, where the guard's docstring quoted the claim it refutes: THE FILE THAT ANSWERS THE QUESTION ALSO CONTAINS THE STRING THAT ANSWERS IT WRONGLY. My own first count was wrong too -- 89, from a base filter matching any name containing "Model" rather than BaseModel/RootModel specifically. Re-measured before reporting. --- docs/testing/master-test-plan/09-engine-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/09-engine-api.md b/docs/testing/master-test-plan/09-engine-api.md index 6ba181c3b..b6b13869d 100644 --- a/docs/testing/master-test-plan/09-engine-api.md +++ b/docs/testing/master-test-plan/09-engine-api.md @@ -110,7 +110,7 @@ never run on, or a measurement that has never been taken. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| Wire contract unpinned for ~72 of 121 models | A field rename/retype/optionality flip on `UploadedFileInfo`, `SearchPresetList`, `LogTailPage`, `ResendResult`, `EditResendResult`, `ConnectionMetadata`, `AiPolicy`, `Health`, `StatsResponse`, `ApprovalList`, `MessageResponses`, `OutboundPayloads`… ships green | Every non-console consumer: `apiclient` (harness load/scenario/failover runners, service CLI), the IDE extension, the tray. Runtime `ApiError`/silent wrong render at the customer, not in CI | **No** — the seam snapshot pins field *names* for the 49 console-rendered DTOs only, and no types, optionality, path, method or status code | **P0** | +| Wire contract unpinned for most models. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100).** Re-counted by AST walk over all of `api/`: **117** `BaseModel`/`RootModel` classes (`models.py` 84, `auth_models.py` 32, `phi_gate.py` 1) against the "~121" written here — **inside the row's own hedge, so the figure is left as an approximation rather than re-pinned to a number that will drift again.** The load-bearing half is CONFIRMED: `tests/test_webconsole_seam_snapshot.py` (6 tests) is a **digest over rendered-DTO field names**, and contains **no** `status_code`, `methods`, `__annotations__`, `type_` or `optional` reference at all. **Two tokens in that file look like counter-evidence and are not** — `annotation` is `from __future__ import annotations` (`:14`) and `required` is the English word (`:76`); a grep for either would wrongly suggest types are pinned | A field rename/retype/optionality flip on `UploadedFileInfo`, `SearchPresetList`, `LogTailPage`, `ResendResult`, `EditResendResult`, `ConnectionMetadata`, `AiPolicy`, `Health`, `StatsResponse`, `ApprovalList`, `MessageResponses`, `OutboundPayloads`… ships green | Every non-console consumer: `apiclient` (harness load/scenario/failover runners, service CLI), the IDE extension, the tray. Runtime `ApiError`/silent wrong render at the customer, not in CI | **No** — the seam snapshot pins field *names* for the 49 console-rendered DTOs only, and no types, optionality, path, method or status code | **P0** | | No route ever runs against SQL Server or PostgreSQL | `create_app` appears in **zero** test that sets `MEFOR_TEST_SQLSERVER`/`MEFOR_TEST_POSTGRES` (verified by cross-grep). A backend divergence in `list_messages` + `count_messages` composition, `search_messages` truncation, `record_audit` with the ADR 0150 `client` column on `NVARCHAR(256)`, preset listing, dead-letter paging or audit filters surfaces first in production | Wrong/missing clinical rows on an operator read; a 500 on the console; an audit write that silently fails on the one backend a hospital runs | **No** | **P0** | | Framing/DoS defences never tested on a real socket | Every API test uses `httpx.ASGITransport`/`TestClient` with headers httpx synthesized (`tests/test_api.py:88-105`). The CL.TE 400, chunked-411, invalid-CL 400 and 1 MiB/`max_upload_bytes` 413 (`api/app.py:1279-1333`) are asserted against fabricated scopes; h11's own limits, keep-alive, real chunked decoding and slow-loris behaviour are unexercised | A pre-auth memory/connection DoS, or a desync between a front proxy and h11, that CI cannot see. ADR 0092 already records "a full uvicorn-on-a-real-socket handshake through the live serve bind" as a deferred residual | **No** | **P1** | | Deny-by-default proven structurally, not by execution | `test_security_doc_drift.py` reads the dependency graph. A route whose handler answers before its dependency is reached, a middleware short-circuit, a hand-registered Starlette route, or a closure shape that makes `_gate_of` return `None` while the allowlist is updated to match, all pass | Unauthenticated read of PHI or an admin write. Silent | Partially — spot checks only (`test_api_auth.py:97`, `:301`) | **P1** | From d47da76a0bd8f691ee2f42f4bc1ee47b66b69950 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 07:22:51 -0500 Subject: [PATCH 25/66] docs(testing): confirm no API route runs on a server DB, with both populations measured (BACKLOG #1100) 09-engine-api.md row 114 is CORRECT, and measuring it properly makes it much harder to wave away than "verified by cross-grep" did: tests setting MEFOR_TEST_SQLSERVER / MEFOR_TEST_POSTGRES 46 tests constructing create_app 63 intersection 0 BOTH POPULATIONS ARE LARGE, which is the part that matters. A bare "zero hits" cannot distinguish a real disjointness from an empty set, a typo'd token, or a grep that never matched anything -- and this sweep has already produced one probe that reported 0 for eight ADRs it had never tested. 46 and 63 with no overlap is a structural fact about the test estate, not an absence of evidence. The two halves do not meet: everything that exercises a route runs on SQLite, and everything that runs on a real server backend addresses the store directly. So every backend divergence the row lists -- list_messages + count_messages composition, search truncation, record_audit against the ADR 0150 client column on NVARCHAR(256), preset listing, dead-letter paging, audit filters -- would first appear at a deploying site rather than in CI. Chapter 09's P0 risk set is now complete (113 confirmed, 114 confirmed), making six chapters: 02, 03, 04, 09, 10, 16. --- docs/testing/master-test-plan/09-engine-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/09-engine-api.md b/docs/testing/master-test-plan/09-engine-api.md index b6b13869d..f6bc1f5ac 100644 --- a/docs/testing/master-test-plan/09-engine-api.md +++ b/docs/testing/master-test-plan/09-engine-api.md @@ -111,7 +111,7 @@ never run on, or a measurement that has never been taken. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | Wire contract unpinned for most models. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100).** Re-counted by AST walk over all of `api/`: **117** `BaseModel`/`RootModel` classes (`models.py` 84, `auth_models.py` 32, `phi_gate.py` 1) against the "~121" written here — **inside the row's own hedge, so the figure is left as an approximation rather than re-pinned to a number that will drift again.** The load-bearing half is CONFIRMED: `tests/test_webconsole_seam_snapshot.py` (6 tests) is a **digest over rendered-DTO field names**, and contains **no** `status_code`, `methods`, `__annotations__`, `type_` or `optional` reference at all. **Two tokens in that file look like counter-evidence and are not** — `annotation` is `from __future__ import annotations` (`:14`) and `required` is the English word (`:76`); a grep for either would wrongly suggest types are pinned | A field rename/retype/optionality flip on `UploadedFileInfo`, `SearchPresetList`, `LogTailPage`, `ResendResult`, `EditResendResult`, `ConnectionMetadata`, `AiPolicy`, `Health`, `StatsResponse`, `ApprovalList`, `MessageResponses`, `OutboundPayloads`… ships green | Every non-console consumer: `apiclient` (harness load/scenario/failover runners, service CLI), the IDE extension, the tray. Runtime `ApiError`/silent wrong render at the customer, not in CI | **No** — the seam snapshot pins field *names* for the 49 console-rendered DTOs only, and no types, optionality, path, method or status code | **P0** | -| No route ever runs against SQL Server or PostgreSQL | `create_app` appears in **zero** test that sets `MEFOR_TEST_SQLSERVER`/`MEFOR_TEST_POSTGRES` (verified by cross-grep). A backend divergence in `list_messages` + `count_messages` composition, `search_messages` truncation, `record_audit` with the ADR 0150 `client` column on `NVARCHAR(256)`, preset listing, dead-letter paging or audit filters surfaces first in production | Wrong/missing clinical rows on an operator read; a 500 on the console; an audit write that silently fails on the one backend a hospital runs | **No** | **P0** | +| No route ever runs against SQL Server or PostgreSQL | `create_app` appears in **zero** test that sets `MEFOR_TEST_SQLSERVER`/`MEFOR_TEST_POSTGRES`. **RE-VERIFIED LIVE 2026-08-15 (BACKLOG #1100), and the measurement is stronger than "cross-grep" conveys: 46 tests set a server-DB env gate, 63 tests construct `create_app`, and the intersection is EXACTLY 0.** Both populations are large, so this is a real disjointness between two well-populated sets, not the empty-set artifact a bare "zero hits" can hide. **The two halves of the test estate do not meet:** everything exercising a route runs on SQLite, and everything running on a real server backend addresses the store directly. A backend divergence in `list_messages` + `count_messages` composition, `search_messages` truncation, `record_audit` with the ADR 0150 `client` column on `NVARCHAR(256)`, preset listing, dead-letter paging or audit filters surfaces first in production | Wrong/missing clinical rows on an operator read; a 500 on the console; an audit write that silently fails on the one backend a hospital runs | **No** | **P0** | | Framing/DoS defences never tested on a real socket | Every API test uses `httpx.ASGITransport`/`TestClient` with headers httpx synthesized (`tests/test_api.py:88-105`). The CL.TE 400, chunked-411, invalid-CL 400 and 1 MiB/`max_upload_bytes` 413 (`api/app.py:1279-1333`) are asserted against fabricated scopes; h11's own limits, keep-alive, real chunked decoding and slow-loris behaviour are unexercised | A pre-auth memory/connection DoS, or a desync between a front proxy and h11, that CI cannot see. ADR 0092 already records "a full uvicorn-on-a-real-socket handshake through the live serve bind" as a deferred residual | **No** | **P1** | | Deny-by-default proven structurally, not by execution | `test_security_doc_drift.py` reads the dependency graph. A route whose handler answers before its dependency is reached, a middleware short-circuit, a hand-registered Starlette route, or a closure shape that makes `_gate_of` return `None` while the allowlist is updated to match, all pass | Unauthenticated read of PHI or an admin write. Silent | Partially — spot checks only (`test_api_auth.py:97`, `:301`) | **P1** | | `/ws/stats` cap and `ws_count` hygiene untested behaviourally | The 64-cap is asserted only as a **constant** (`test_threat_model_doc_drift.py:573`). Nothing proves the 65th handshake is refused, nor that `state.ws_count` returns to 0 on every exit path (`api/app.py:4846`, `:4854`, `:4918-4919`). Note the cap is checked *before* `accept()` and incremented *after* — a TOCTOU window and a Starlette pre-accept-close that surfaces as a rejected handshake, not a 1013 frame | A leaked counter permanently refuses every future console socket estate-wide; the live monitor dies silently and the 5 s poll fallback masks it | **No** | **P1** | From 4935424fcaa8825efb361a210c5f30d336d845db Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 07:34:22 -0500 Subject: [PATCH 26/66] docs(testing): a control exists for the false-promote P0, and it compares the wrong two things (BACKLOG #1100) 08-publishing-and-promotion.md row 109 said "Nothing recomputes or compares a fingerprint". THAT IS FALSE. ConfigProvenance (api/app.py:4570-4590) reads engine.loaded_config_fingerprint, RECOMPUTES config_fingerprint_detail(target) off the loop, COMPARES them, and returns a drift flag. ADR 0041 D1 additionally writes a fingerprint-bearing config_reload audit row on every reload (:554-581). BUT IT COMPARES LOADED-VS-DISK-NOW, AND THE RISK IS DISK-VS-THE-COMMIT-CI-WAS- MEANT-TO-DELIVER. In the exact scenario this row describes -- CI/CD never delivered the commit -- the disk still equals what was loaded, so drift is False and provenance reports clean. SO THE CONCERN SURVIVES ITS EVIDENCE ENTIRELY, AND IS NOW HARDER TO SEE THAN BEFORE. A reviewer who greps for fingerprint handling finds a recompute, a comparison and a drift flag, and reasonably concludes the risk is covered. The control is real, well-built, and answers a different question. That is the compensating-control-on-a-false-premise shape for the fourth time in this sweep, and the most dangerous instance: the other three rested on evidence that had rotted, while this one rests on a control that genuinely exists. Still true, and re-anchored: ReloadResult (api/models.py:371, cited :351-360) carries no fingerprint, so the promoting caller never receives one; and there is no `messagefoundry fingerprint` CLI -- the subcommands are graph, serve, validate. THE FIX IS SMALLER THAN THE ROW IMPLIED. The fingerprint is already computed on every reload. It needs surfacing in ReloadResult and comparing against an EXPECTED value, rather than building fingerprinting from nothing. --- docs/testing/master-test-plan/08-publishing-and-promotion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/08-publishing-and-promotion.md b/docs/testing/master-test-plan/08-publishing-and-promotion.md index 562650f26..9391edddc 100644 --- a/docs/testing/master-test-plan/08-publishing-and-promotion.md +++ b/docs/testing/master-test-plan/08-publishing-and-promotion.md @@ -106,7 +106,7 @@ post-quiesce failure. Those are this chapter's centre of gravity. |---|---|---|---|---| | Restart-based publish has zero attribution | `Engine.start()` does `load_config` + `add_registry` (`api/app.py:5455-5462`); only `Engine.reload()` sets `loaded_config_fingerprint` (`engine.py:1488-1500`). Copy-files + NSSM restart (and the tray **Restart Service**) leave `loaded=false` and write no fingerprint-bearing audit row. | Every engine that was ever restarted rather than reloaded — i.e. the primary CI/CD path and every service restart. `/ui` badge blank, IDE can never show DRIFTED, no `who activated these bytes` row. | **No.** `GET /config/provenance` returns `loaded=false`, which reads as "nothing loaded", not "unattested". | P0 | | Fingerprint is blind to the real `environments/` | `_FINGERPRINT_GLOBS` includes `environments/*.toml` **relative to the config dir** (`fingerprint.py:39-45`), but every shipped/documented layout keeps `environments/` as a project-root **sibling** of `--config` (repo root here; `samples/config/` has no `environments/` subdir; ADR 0017 layout; ADR 0050 anchoring). | Repointing `acme_adt_host` in `environments/prod.toml` redirects PHI to a new peer with an **identical** fingerprint, no drift flag, and an audit row indistinguishable from a clean reload. | **No** — and ADR 0041 explicitly claims the opposite ("spans … `environments/`"). The existing test passes only because it writes `environments/` *inside* the tmp config dir. | P0 | -| "Promoted to PROD" can be false | A remote promote sends `config_dir: null` (`promote.ts:135-149`) so the engine reloads its **own** on-disk dir; `ReloadResult` (`api/models.py:351-360`) carries no fingerprint. If CI/CD never delivered the commit, PROD reloads stale bytes and the toast still says "promoted". | Silent whenever element counts are unchanged — i.e. almost every Handler-logic change. Highest-frequency wrong belief in the publish path. | **No.** Nothing recomputes or compares a fingerprint; there is no `messagefoundry fingerprint` CLI. | P0 | +| "Promoted to PROD" can be false | A remote promote sends `config_dir: null` (`promote.ts:135-149`) so the engine reloads its **own** on-disk dir; `ReloadResult` (`api/models.py:351-360`) carries no fingerprint. If CI/CD never delivered the commit, PROD reloads stale bytes and the toast still says "promoted". | Silent whenever element counts are unchanged — i.e. almost every Handler-logic change. Highest-frequency wrong belief in the publish path. | **Partly — and the partial coverage is the trap. RE-MEASURED 2026-08-15 (BACKLOG #1100).** "Nothing recomputes or compares a fingerprint" is **FALSE**: `ConfigProvenance` (`api/app.py:4570-4590`) reads `engine.loaded_config_fingerprint`, **recomputes** `config_fingerprint_detail(target)` off the loop, and **compares** them — `drift = current.get("fingerprint") != fp` — returning a `drift` flag. ADR 0041 D1 also writes a fingerprint-bearing `config_reload` audit row on every reload (`:554-581`). **BUT IT COMPARES THE WRONG TWO THINGS FOR THIS RISK.** `drift` is *loaded-vs-disk-now*; the failure here is *disk-vs-the-commit-CI-was-meant-to-deliver*. **If CI/CD never delivered, disk still equals what was loaded, so `drift` is `False` and provenance reports clean.** So the row's concern survives its evidence completely — and is now *harder* to see, because a reviewer who finds `ConfigProvenance` will reasonably conclude it is covered. Still true: `ReloadResult` (**`api/models.py:371`**, cited `:351-360`) carries no fingerprint, so the promoting caller never receives one; and there is no `messagefoundry fingerprint` CLI (subcommands are `graph`, `serve`, `validate`). **The fix is smaller than this row implied** — the fingerprint is already computed on every reload; it needs *surfacing in `ReloadResult`* and comparing against an expected value, not building | P0 | | A held (dual-control) promote renders as success | `engineClient.postJson` resolves any 2xx and casts the body (`engineClient.ts:92-102`); a 202 `PendingApprovalResponse` becomes a `ReloadResult` with `undefined` counts, and `promote.ts:191` prints "promoted to PROD — live graph: undefined inbound…". | Exactly the deployments that enabled dual-control **for production**. Operator believes the graph swapped; it is sitting in a queue. | **No.** No test drives `promote.ts` at all. | P0 | | Split-config cluster / split engine-shard estate | Convergence coordinates *when* nodes reload; each reloads its **own** dir (`config_convergence.py:16-20`). `ClusterStatus` exposes only the integer `config_version` (`api/models.py:737-748`), never a content fingerprint. Promote picks **one** engine-shard URL (`promoteTarget.ts:53-55`). | Node A runs the new graph, node B re-applies its old one; both write clean `config_reload` rows and report the same `config_version`. After a failover the wrong graph serves production. Engine shards over one unified store diverge per lane. | **No.** ADR 0041 lists this as unresolved — its open item to coordinate with the engine-shard owner. | P0 | | Composite pipeline never exercised | Every unit exists; the assembly (author → check → non-prod → traffic → identical artifact to prod → per-environment substitution → rollback) has no test, no harness rig, and `FEATURE-COVERAGE-PLAN.md` excludes the IDE from every subsystem (`:946, :1216, :1379, :1420, :1471, :1521`). | The interfaces between the pieces — env substitution, artifact identity, pre-flight-vs-apply ordering, rollback — are where the real defects live. | **No.** | P0 | From 98b8ae1d97798b71cc569e0af355891014f54775 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 07:43:47 -0500 Subject: [PATCH 27/66] docs(testing): confirm the fingerprint's environments/ blind spot against ADR 0041's SHALL (BACKLOG #1100) 08-publishing-and-promotion.md row 108 is CONFIRMED, measured on this checkout. fingerprint.py:44 lists environments/*.toml in _FINGERPRINT_GLOBS, globbed at :59 as base.glob(pattern) where base is the directory passed in -- last_reload_dir / config_dir, the CONFIG DIR. Repo-root environments/ holds 2 .toml files; samples/config/environments/ does not exist. Every documented layout (ADR 0017, ADR 0050) keeps environments/ as a SIBLING of --config, so the declared glob resolves to nothing. WHAT THOSE FILES HOLD IS WHY THIS MATTERS: acme_adt_host, acme_adt_port, demo_oru_host, demo_oru_port, fhir_base_url, payer_rte_host, payer_rte_port -- 14 keys each. Repointing a peer host changes WHERE PHI IS SENT and produces an identical fingerprint, no drift flag, and an audit row indistinguishable from a clean reload. AND ADR 0041 STATES THE OPPOSITE AS A REQUIREMENT, NOT A DESCRIPTION: :100 "It spans connections.toml AND environments/ so a transport/env-value redirect cannot change [the graph without changing the fingerprint]" :153 "... or environments/*.toml -- THE SYSTEM SHALL produce a different fingerprint." So this is not a doc gap. SHIPPED CODE DOES NOT MEET A SHALL ITS OWN ADR RECORDS, and the attestation built to detect a redirect is blind to the file that performs one. Conditional per section 0: no deployments, so nothing is mis-attested today. ADR 0041:238 separately leaves "Fingerprint env-value scope" open -- that question is about hashing RESOLVED values and does not cover this. This is the declared scope failing to resolve at all, which is a different and simpler failure. Not fixed: config surface, outside this claim. The fix is to anchor the environments/ glob at the project root rather than the config dir. --- docs/testing/master-test-plan/08-publishing-and-promotion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/08-publishing-and-promotion.md b/docs/testing/master-test-plan/08-publishing-and-promotion.md index 9391edddc..228c42249 100644 --- a/docs/testing/master-test-plan/08-publishing-and-promotion.md +++ b/docs/testing/master-test-plan/08-publishing-and-promotion.md @@ -105,7 +105,7 @@ post-quiesce failure. Those are this chapter's centre of gravity. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | Restart-based publish has zero attribution | `Engine.start()` does `load_config` + `add_registry` (`api/app.py:5455-5462`); only `Engine.reload()` sets `loaded_config_fingerprint` (`engine.py:1488-1500`). Copy-files + NSSM restart (and the tray **Restart Service**) leave `loaded=false` and write no fingerprint-bearing audit row. | Every engine that was ever restarted rather than reloaded — i.e. the primary CI/CD path and every service restart. `/ui` badge blank, IDE can never show DRIFTED, no `who activated these bytes` row. | **No.** `GET /config/provenance` returns `loaded=false`, which reads as "nothing loaded", not "unattested". | P0 | -| Fingerprint is blind to the real `environments/` | `_FINGERPRINT_GLOBS` includes `environments/*.toml` **relative to the config dir** (`fingerprint.py:39-45`), but every shipped/documented layout keeps `environments/` as a project-root **sibling** of `--config` (repo root here; `samples/config/` has no `environments/` subdir; ADR 0017 layout; ADR 0050 anchoring). | Repointing `acme_adt_host` in `environments/prod.toml` redirects PHI to a new peer with an **identical** fingerprint, no drift flag, and an audit row indistinguishable from a clean reload. | **No** — and ADR 0041 explicitly claims the opposite ("spans … `environments/`"). The existing test passes only because it writes `environments/` *inside* the tmp config dir. | P0 | +| Fingerprint is blind to the real `environments/` | `_FINGERPRINT_GLOBS` includes `environments/*.toml` **relative to the config dir** (`fingerprint.py:44`, globbed at `:59` as `base.glob(pattern)` where `base` is the directory passed in — i.e. `engine.last_reload_dir` / `engine.config_dir`), but every shipped/documented layout keeps `environments/` as a project-root **sibling** of `--config` (ADR 0017 layout; ADR 0050 anchoring). **CONFIRMED 2026-08-15 (BACKLOG #1100), measured on this checkout:** repo-root `environments/` holds **2** `.toml` files, and **`samples/config/environments/` does not exist** — so the glob resolves to nothing for the documented layout. **The redirect surface is exactly the peer endpoints:** those files declare `acme_adt_host`, `acme_adt_port`, `demo_oru_host`, `demo_oru_port`, `fhir_base_url`, `payer_rte_host`, `payer_rte_port` (14 keys each). **And ADR 0041 states the opposite as a REQUIREMENT, not a description** — `:100` *"It spans `connections.toml` **and** `environments/` so a transport/env-value redirect cannot change [the graph without changing the fingerprint]"*, and `:153` *"...or `environments/*.toml` — **THE SYSTEM SHALL** produce a different fingerprint."* **The shipped code does not meet that SHALL in the documented layout.** (ADR 0041 `:238` separately leaves "Fingerprint env-value scope" open, but that question is about hashing *resolved* values — it does not cover this, which is the declared scope failing to resolve at all.) | Repointing `acme_adt_host` in `environments/prod.toml` redirects PHI to a new peer with an **identical** fingerprint, no drift flag, and an audit row indistinguishable from a clean reload. | **No** — and ADR 0041 explicitly claims the opposite ("spans … `environments/`"). The existing test passes only because it writes `environments/` *inside* the tmp config dir. | P0 | | "Promoted to PROD" can be false | A remote promote sends `config_dir: null` (`promote.ts:135-149`) so the engine reloads its **own** on-disk dir; `ReloadResult` (`api/models.py:351-360`) carries no fingerprint. If CI/CD never delivered the commit, PROD reloads stale bytes and the toast still says "promoted". | Silent whenever element counts are unchanged — i.e. almost every Handler-logic change. Highest-frequency wrong belief in the publish path. | **Partly — and the partial coverage is the trap. RE-MEASURED 2026-08-15 (BACKLOG #1100).** "Nothing recomputes or compares a fingerprint" is **FALSE**: `ConfigProvenance` (`api/app.py:4570-4590`) reads `engine.loaded_config_fingerprint`, **recomputes** `config_fingerprint_detail(target)` off the loop, and **compares** them — `drift = current.get("fingerprint") != fp` — returning a `drift` flag. ADR 0041 D1 also writes a fingerprint-bearing `config_reload` audit row on every reload (`:554-581`). **BUT IT COMPARES THE WRONG TWO THINGS FOR THIS RISK.** `drift` is *loaded-vs-disk-now*; the failure here is *disk-vs-the-commit-CI-was-meant-to-deliver*. **If CI/CD never delivered, disk still equals what was loaded, so `drift` is `False` and provenance reports clean.** So the row's concern survives its evidence completely — and is now *harder* to see, because a reviewer who finds `ConfigProvenance` will reasonably conclude it is covered. Still true: `ReloadResult` (**`api/models.py:371`**, cited `:351-360`) carries no fingerprint, so the promoting caller never receives one; and there is no `messagefoundry fingerprint` CLI (subcommands are `graph`, `serve`, `validate`). **The fix is smaller than this row implied** — the fingerprint is already computed on every reload; it needs *surfacing in `ReloadResult`* and comparing against an expected value, not building | P0 | | A held (dual-control) promote renders as success | `engineClient.postJson` resolves any 2xx and casts the body (`engineClient.ts:92-102`); a 202 `PendingApprovalResponse` becomes a `ReloadResult` with `undefined` counts, and `promote.ts:191` prints "promoted to PROD — live graph: undefined inbound…". | Exactly the deployments that enabled dual-control **for production**. Operator believes the graph swapped; it is sitting in a queue. | **No.** No test drives `promote.ts` at all. | P0 | | Split-config cluster / split engine-shard estate | Convergence coordinates *when* nodes reload; each reloads its **own** dir (`config_convergence.py:16-20`). `ClusterStatus` exposes only the integer `config_version` (`api/models.py:737-748`), never a content fingerprint. Promote picks **one** engine-shard URL (`promoteTarget.ts:53-55`). | Node A runs the new graph, node B re-applies its old one; both write clean `config_reload` rows and report the same `config_version`. After a failover the wrong graph serves production. Engine shards over one unified store diverge per lane. | **No.** ADR 0041 lists this as unresolved — its open item to coordinate with the engine-shard owner. | P0 | From d6b548770b4393a6cf6a4f7a3f6d80ffd7f03057 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 07:53:37 -0500 Subject: [PATCH 28/66] docs(testing): confirm the dual-control promote renders as success (BACKLOG #1100) 08-publishing-and-promotion.md row 110 is CONFIRMED on all four claims, and BOTH ANCHORS STILL LAND -- rare in this sweep. engineClient.ts:94-96 if (status >= 200 && status < 300) { resolve((text ? JSON.parse(text) : {}) as T) promote.ts:191-195 prints "promoted to -- live graph: ${result.inbound} inbound, ..." with NO status check 41 IDE test files exist and NOT ONE drives promote.ts (promote-target.test.ts tests promoteTarget.ts, a different module) STRONGER THAN THE ROW STATES: PendingApproval appears nowhere in ide/, and there is no 202 handling anywhere in ide/src. This is not a missed branch -- the client does not model the pending-approval response as a concept. So the fix is one status check at the postJson call site, not new machinery. WORTH RECORDING ABOUT THE ANCHORS. Nearly every Python row in this sweep had drifted citations; both TypeScript anchors here are exact. ANCHOR DRIFT TRACKS CHURN, and the IDE surface has moved far less than the engine. That is a useful prior for whoever sweeps next: weight re-derivation toward the fastest-moving code, and treat a stable surface's citations as more trustworthy -- but still not as evidence. --- docs/testing/master-test-plan/08-publishing-and-promotion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/08-publishing-and-promotion.md b/docs/testing/master-test-plan/08-publishing-and-promotion.md index 228c42249..cccd95df3 100644 --- a/docs/testing/master-test-plan/08-publishing-and-promotion.md +++ b/docs/testing/master-test-plan/08-publishing-and-promotion.md @@ -107,7 +107,7 @@ post-quiesce failure. Those are this chapter's centre of gravity. | Restart-based publish has zero attribution | `Engine.start()` does `load_config` + `add_registry` (`api/app.py:5455-5462`); only `Engine.reload()` sets `loaded_config_fingerprint` (`engine.py:1488-1500`). Copy-files + NSSM restart (and the tray **Restart Service**) leave `loaded=false` and write no fingerprint-bearing audit row. | Every engine that was ever restarted rather than reloaded — i.e. the primary CI/CD path and every service restart. `/ui` badge blank, IDE can never show DRIFTED, no `who activated these bytes` row. | **No.** `GET /config/provenance` returns `loaded=false`, which reads as "nothing loaded", not "unattested". | P0 | | Fingerprint is blind to the real `environments/` | `_FINGERPRINT_GLOBS` includes `environments/*.toml` **relative to the config dir** (`fingerprint.py:44`, globbed at `:59` as `base.glob(pattern)` where `base` is the directory passed in — i.e. `engine.last_reload_dir` / `engine.config_dir`), but every shipped/documented layout keeps `environments/` as a project-root **sibling** of `--config` (ADR 0017 layout; ADR 0050 anchoring). **CONFIRMED 2026-08-15 (BACKLOG #1100), measured on this checkout:** repo-root `environments/` holds **2** `.toml` files, and **`samples/config/environments/` does not exist** — so the glob resolves to nothing for the documented layout. **The redirect surface is exactly the peer endpoints:** those files declare `acme_adt_host`, `acme_adt_port`, `demo_oru_host`, `demo_oru_port`, `fhir_base_url`, `payer_rte_host`, `payer_rte_port` (14 keys each). **And ADR 0041 states the opposite as a REQUIREMENT, not a description** — `:100` *"It spans `connections.toml` **and** `environments/` so a transport/env-value redirect cannot change [the graph without changing the fingerprint]"*, and `:153` *"...or `environments/*.toml` — **THE SYSTEM SHALL** produce a different fingerprint."* **The shipped code does not meet that SHALL in the documented layout.** (ADR 0041 `:238` separately leaves "Fingerprint env-value scope" open, but that question is about hashing *resolved* values — it does not cover this, which is the declared scope failing to resolve at all.) | Repointing `acme_adt_host` in `environments/prod.toml` redirects PHI to a new peer with an **identical** fingerprint, no drift flag, and an audit row indistinguishable from a clean reload. | **No** — and ADR 0041 explicitly claims the opposite ("spans … `environments/`"). The existing test passes only because it writes `environments/` *inside* the tmp config dir. | P0 | | "Promoted to PROD" can be false | A remote promote sends `config_dir: null` (`promote.ts:135-149`) so the engine reloads its **own** on-disk dir; `ReloadResult` (`api/models.py:351-360`) carries no fingerprint. If CI/CD never delivered the commit, PROD reloads stale bytes and the toast still says "promoted". | Silent whenever element counts are unchanged — i.e. almost every Handler-logic change. Highest-frequency wrong belief in the publish path. | **Partly — and the partial coverage is the trap. RE-MEASURED 2026-08-15 (BACKLOG #1100).** "Nothing recomputes or compares a fingerprint" is **FALSE**: `ConfigProvenance` (`api/app.py:4570-4590`) reads `engine.loaded_config_fingerprint`, **recomputes** `config_fingerprint_detail(target)` off the loop, and **compares** them — `drift = current.get("fingerprint") != fp` — returning a `drift` flag. ADR 0041 D1 also writes a fingerprint-bearing `config_reload` audit row on every reload (`:554-581`). **BUT IT COMPARES THE WRONG TWO THINGS FOR THIS RISK.** `drift` is *loaded-vs-disk-now*; the failure here is *disk-vs-the-commit-CI-was-meant-to-deliver*. **If CI/CD never delivered, disk still equals what was loaded, so `drift` is `False` and provenance reports clean.** So the row's concern survives its evidence completely — and is now *harder* to see, because a reviewer who finds `ConfigProvenance` will reasonably conclude it is covered. Still true: `ReloadResult` (**`api/models.py:371`**, cited `:351-360`) carries no fingerprint, so the promoting caller never receives one; and there is no `messagefoundry fingerprint` CLI (subcommands are `graph`, `serve`, `validate`). **The fix is smaller than this row implied** — the fingerprint is already computed on every reload; it needs *surfacing in `ReloadResult`* and comparing against an expected value, not building | P0 | -| A held (dual-control) promote renders as success | `engineClient.postJson` resolves any 2xx and casts the body (`engineClient.ts:92-102`); a 202 `PendingApprovalResponse` becomes a `ReloadResult` with `undefined` counts, and `promote.ts:191` prints "promoted to PROD — live graph: undefined inbound…". | Exactly the deployments that enabled dual-control **for production**. Operator believes the graph swapped; it is sitting in a queue. | **No.** No test drives `promote.ts` at all. | P0 | +| A held (dual-control) promote renders as success | `engineClient.postJson` resolves any 2xx and casts the body (`engineClient.ts:92-102`); a 202 `PendingApprovalResponse` becomes a `ReloadResult` with `undefined` counts, and `promote.ts:191` prints "promoted to PROD — live graph: undefined inbound…". | Exactly the deployments that enabled dual-control **for production**. Operator believes the graph swapped; it is sitting in a queue. | **No — CONFIRMED 2026-08-15 (BACKLOG #1100), and both anchors still land, which is rare in this sweep.** `engineClient.ts:94-96` is exactly `if (status >= 200 && status < 300) { resolve((text ? JSON.parse(text) : {}) as T)` — **any** 2xx resolves and blind-casts, so a 202 body becomes a `ReloadResult` of `undefined`s; `promote.ts:191-195` then prints the success toast with **no status check**. Measured: **41 IDE test files exist and NOT ONE drives `promote.ts`** (`promote-target.test.ts` tests `promoteTarget.ts`, a different module). **Stronger than the row states: `PendingApproval` appears nowhere in `ide/`, and there is no 202 handling anywhere in `ide/src`** — the client does not model the pending-approval response at all, so this is not a missed branch but an absent concept. **The fix is one status check** at the `postJson` call site, not new machinery | P0 | | Split-config cluster / split engine-shard estate | Convergence coordinates *when* nodes reload; each reloads its **own** dir (`config_convergence.py:16-20`). `ClusterStatus` exposes only the integer `config_version` (`api/models.py:737-748`), never a content fingerprint. Promote picks **one** engine-shard URL (`promoteTarget.ts:53-55`). | Node A runs the new graph, node B re-applies its old one; both write clean `config_reload` rows and report the same `config_version`. After a failover the wrong graph serves production. Engine shards over one unified store diverge per lane. | **No.** ADR 0041 lists this as unresolved — its open item to coordinate with the engine-shard owner. | P0 | | Composite pipeline never exercised | Every unit exists; the assembly (author → check → non-prod → traffic → identical artifact to prod → per-environment substitution → rollback) has no test, no harness rig, and `FEATURE-COVERAGE-PLAN.md` excludes the IDE from every subsystem (`:946, :1216, :1379, :1420, :1471, :1521`). | The interfaces between the pieces — env substitution, artifact identity, pre-flight-vs-apply ordering, rollback — are where the real defects live. | **No.** | P0 | | Post-quiesce failure leaves partial state | Rollback restores **only** `self.registry` + inbound intake (`wiring_runner.py:3151-3164`). By then the live-lookup executor has been rebuilt and the old one `aclose`d (`:3096-3104`), sandbox sessions dropped (`:3086-3095`), and `_reconcile_outbounds` may have partially applied (`:3149`). | Old Routers/Handlers run against **new-graph** `db_lookup`/`fhir_lookup` pools. If the new graph dropped a `DbLookup`, every old-graph lookup raises post-ACK — a silent per-message ERROR/dead-letter storm behind an audit row the operator reads as a clean no-op. | **No.** No test drives a failure at the inbound-bind or `_reconcile_outbounds` step. | P1 | From 7b1ddae722379f5d4c457f7ee9619af05bd7178d Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 08:03:51 -0500 Subject: [PATCH 29/66] docs(testing): confirm the restart-publish attribution gap, re-pointing both drifted anchors (BACKLOG #1100) 08-publishing-and-promotion.md row 107 is CONFIRMED on substance. Both of its anchors had drifted, one of them onto text that looks nothing like the claim: cited api/app.py:5455-5462 as the start-path load -> that range is now a function signature's KEYWORD PARAMETERS (priority_default, retention_settings, ...), ~270 lines away actual api/app.py:5725 loaded = load_config(config_dir) api/app.py:5730 engine.add_registry(loaded) cited engine.py:1488-1500 as "only reload sets loaded_config_fingerprint" actual engine.py:450 initialised None engine.py:1610 / :1615 BOTH inside reload() (def :1479) -- right method, wrong lines VERIFIED FROM THE OTHER SIDE TOO, which is what makes it conclusive rather than suggestive: Engine.start() (engine.py:875, 301 lines) mentions NONE of loaded_config_fingerprint, config_fingerprint, add_registry or load_config. The fingerprint-bearing config_reload audit row is written only from the reload path (app.py:554-573, gate action registered :547). So the primary CI/CD path -- copy files, restart the service -- leaves the engine unattested, and GET /config/provenance answers loaded=false. Its own comment shows the conflation: "no graph loaded yet, or fingerprint unavailable". Two very different states, one indistinguishable answer. PROCESS NOTE: my first edit dropped the cell boundary between Risk and Failure mode (the old text had two pipes, my replacement had one) -- the same error I committed twice earlier tonight. THE INTEGRITY GATE CAUGHT IT BEFORE STAGING this time, because it now runs as its own step and I read it before deciding. Fixed before the commit rather than after. --- docs/testing/master-test-plan/08-publishing-and-promotion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/08-publishing-and-promotion.md b/docs/testing/master-test-plan/08-publishing-and-promotion.md index cccd95df3..48b7aadd0 100644 --- a/docs/testing/master-test-plan/08-publishing-and-promotion.md +++ b/docs/testing/master-test-plan/08-publishing-and-promotion.md @@ -104,7 +104,7 @@ post-quiesce failure. Those are this chapter's centre of gravity. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| Restart-based publish has zero attribution | `Engine.start()` does `load_config` + `add_registry` (`api/app.py:5455-5462`); only `Engine.reload()` sets `loaded_config_fingerprint` (`engine.py:1488-1500`). Copy-files + NSSM restart (and the tray **Restart Service**) leave `loaded=false` and write no fingerprint-bearing audit row. | Every engine that was ever restarted rather than reloaded — i.e. the primary CI/CD path and every service restart. `/ui` badge blank, IDE can never show DRIFTED, no `who activated these bytes` row. | **No.** `GET /config/provenance` returns `loaded=false`, which reads as "nothing loaded", not "unattested". | P0 | +| Restart-based publish has zero attribution | **CONFIRMED 2026-08-15 (BACKLOG #1100); both anchors had drifted and are re-pointed.** The startup path loads at **`api/app.py:5725`** (`loaded = load_config(config_dir)`) and **`:5730`** (`engine.add_registry(loaded)`) — the cited `:5455-5462` now lands on a function signature's keyword parameters, ~270 lines away. And **only `Engine.reload()` sets `loaded_config_fingerprint`**: it is initialised `None` at **`engine.py:450`** and assigned at **`:1610`/`:1615`, both inside `reload()`** (def `:1479`; the cited `:1488-1500` is the right method, wrong lines). Verified from the other side too — **`Engine.start()` (`engine.py:875`, 301 lines) mentions none of `loaded_config_fingerprint`, `config_fingerprint`, `add_registry` or `load_config`.** The fingerprint-bearing `config_reload` audit row is written only from the reload path (`api/app.py:554-573`, registered as a gate action at `:547`). Copy-files + NSSM restart (and the tray **Restart Service**) leave `loaded=false` and write no fingerprint-bearing audit row. | Every engine that was ever restarted rather than reloaded — i.e. the primary CI/CD path and every service restart. `/ui` badge blank, IDE can never show DRIFTED, no `who activated these bytes` row. | **No.** `GET /config/provenance` returns `loaded=false`, which reads as "nothing loaded", not "unattested". | P0 | | Fingerprint is blind to the real `environments/` | `_FINGERPRINT_GLOBS` includes `environments/*.toml` **relative to the config dir** (`fingerprint.py:44`, globbed at `:59` as `base.glob(pattern)` where `base` is the directory passed in — i.e. `engine.last_reload_dir` / `engine.config_dir`), but every shipped/documented layout keeps `environments/` as a project-root **sibling** of `--config` (ADR 0017 layout; ADR 0050 anchoring). **CONFIRMED 2026-08-15 (BACKLOG #1100), measured on this checkout:** repo-root `environments/` holds **2** `.toml` files, and **`samples/config/environments/` does not exist** — so the glob resolves to nothing for the documented layout. **The redirect surface is exactly the peer endpoints:** those files declare `acme_adt_host`, `acme_adt_port`, `demo_oru_host`, `demo_oru_port`, `fhir_base_url`, `payer_rte_host`, `payer_rte_port` (14 keys each). **And ADR 0041 states the opposite as a REQUIREMENT, not a description** — `:100` *"It spans `connections.toml` **and** `environments/` so a transport/env-value redirect cannot change [the graph without changing the fingerprint]"*, and `:153` *"...or `environments/*.toml` — **THE SYSTEM SHALL** produce a different fingerprint."* **The shipped code does not meet that SHALL in the documented layout.** (ADR 0041 `:238` separately leaves "Fingerprint env-value scope" open, but that question is about hashing *resolved* values — it does not cover this, which is the declared scope failing to resolve at all.) | Repointing `acme_adt_host` in `environments/prod.toml` redirects PHI to a new peer with an **identical** fingerprint, no drift flag, and an audit row indistinguishable from a clean reload. | **No** — and ADR 0041 explicitly claims the opposite ("spans … `environments/`"). The existing test passes only because it writes `environments/` *inside* the tmp config dir. | P0 | | "Promoted to PROD" can be false | A remote promote sends `config_dir: null` (`promote.ts:135-149`) so the engine reloads its **own** on-disk dir; `ReloadResult` (`api/models.py:351-360`) carries no fingerprint. If CI/CD never delivered the commit, PROD reloads stale bytes and the toast still says "promoted". | Silent whenever element counts are unchanged — i.e. almost every Handler-logic change. Highest-frequency wrong belief in the publish path. | **Partly — and the partial coverage is the trap. RE-MEASURED 2026-08-15 (BACKLOG #1100).** "Nothing recomputes or compares a fingerprint" is **FALSE**: `ConfigProvenance` (`api/app.py:4570-4590`) reads `engine.loaded_config_fingerprint`, **recomputes** `config_fingerprint_detail(target)` off the loop, and **compares** them — `drift = current.get("fingerprint") != fp` — returning a `drift` flag. ADR 0041 D1 also writes a fingerprint-bearing `config_reload` audit row on every reload (`:554-581`). **BUT IT COMPARES THE WRONG TWO THINGS FOR THIS RISK.** `drift` is *loaded-vs-disk-now*; the failure here is *disk-vs-the-commit-CI-was-meant-to-deliver*. **If CI/CD never delivered, disk still equals what was loaded, so `drift` is `False` and provenance reports clean.** So the row's concern survives its evidence completely — and is now *harder* to see, because a reviewer who finds `ConfigProvenance` will reasonably conclude it is covered. Still true: `ReloadResult` (**`api/models.py:371`**, cited `:351-360`) carries no fingerprint, so the promoting caller never receives one; and there is no `messagefoundry fingerprint` CLI (subcommands are `graph`, `serve`, `validate`). **The fix is smaller than this row implied** — the fingerprint is already computed on every reload; it needs *surfacing in `ReloadResult`* and comparing against an expected value, not building | P0 | | A held (dual-control) promote renders as success | `engineClient.postJson` resolves any 2xx and casts the body (`engineClient.ts:92-102`); a 202 `PendingApprovalResponse` becomes a `ReloadResult` with `undefined` counts, and `promote.ts:191` prints "promoted to PROD — live graph: undefined inbound…". | Exactly the deployments that enabled dual-control **for production**. Operator believes the graph swapped; it is sitting in a queue. | **No — CONFIRMED 2026-08-15 (BACKLOG #1100), and both anchors still land, which is rare in this sweep.** `engineClient.ts:94-96` is exactly `if (status >= 200 && status < 300) { resolve((text ? JSON.parse(text) : {}) as T)` — **any** 2xx resolves and blind-casts, so a 202 body becomes a `ReloadResult` of `undefined`s; `promote.ts:191-195` then prints the success toast with **no status check**. Measured: **41 IDE test files exist and NOT ONE drives `promote.ts`** (`promote-target.test.ts` tests `promoteTarget.ts`, a different module). **Stronger than the row states: `PendingApproval` appears nowhere in `ide/`, and there is no 202 handling anywhere in `ide/src`** — the client does not model the pending-approval response at all, so this is not a missed branch but an absent concept. **The fix is one status check** at the `postJson` call site, not new machinery | P0 | From 03aa8f20357dc1fa4f056ec9a8d910c2da770fd6 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 08:13:06 -0500 Subject: [PATCH 30/66] docs(testing): confirm the split-config row, and record that three P0s are one gap (BACKLOG #1100) 08-publishing-and-promotion.md row 111 CONFIRMED. ClusterStatus (api/models.py:786, cited :737-748) has exactly node_id, clustered, is_leader, role, config_version -- no fingerprint; ClusterNode (:800) adds none either. The convergence anchor lands exactly, and the module states the assumption in its own words: config_convergence.py:16-18, "The version token coordinates when nodes reload; each node reloads its OWN config dir. Skewed config dirs would diverge." THE FINDING WORTH MORE THAN THE ROW: 107, 109 AND 111 ARE ONE GAP SEEN THREE TIMES. ADR 0041 D1 computes a content fingerprint on every reload, and it is surfaced in NONE of the three places a consumer could compare it -- not on the restart path (107 never sets it), not in ReloadResult (109), not in ClusterStatus (111). THE VALUE IS ALREADY COMPUTED; THE GAP IS ENTIRELY IN EXPOSURE. That matters because each row scopes as its own piece of work. Read separately they suggest three investigations into attestation. Read together they are one change -- surface the existing fingerprint on those responses and compare it against an EXPECTED value at the promote site. ConfigProvenance already proves the recompute-and-compare machinery works; it just compares the wrong two things. Recorded in all three rows and as handoff 6h, so whichever is picked up first finds the other two rather than solving a third of the problem well. --- docs/testing/master-test-plan/08-publishing-and-promotion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/08-publishing-and-promotion.md b/docs/testing/master-test-plan/08-publishing-and-promotion.md index 48b7aadd0..6d55ed9e0 100644 --- a/docs/testing/master-test-plan/08-publishing-and-promotion.md +++ b/docs/testing/master-test-plan/08-publishing-and-promotion.md @@ -108,7 +108,7 @@ post-quiesce failure. Those are this chapter's centre of gravity. | Fingerprint is blind to the real `environments/` | `_FINGERPRINT_GLOBS` includes `environments/*.toml` **relative to the config dir** (`fingerprint.py:44`, globbed at `:59` as `base.glob(pattern)` where `base` is the directory passed in — i.e. `engine.last_reload_dir` / `engine.config_dir`), but every shipped/documented layout keeps `environments/` as a project-root **sibling** of `--config` (ADR 0017 layout; ADR 0050 anchoring). **CONFIRMED 2026-08-15 (BACKLOG #1100), measured on this checkout:** repo-root `environments/` holds **2** `.toml` files, and **`samples/config/environments/` does not exist** — so the glob resolves to nothing for the documented layout. **The redirect surface is exactly the peer endpoints:** those files declare `acme_adt_host`, `acme_adt_port`, `demo_oru_host`, `demo_oru_port`, `fhir_base_url`, `payer_rte_host`, `payer_rte_port` (14 keys each). **And ADR 0041 states the opposite as a REQUIREMENT, not a description** — `:100` *"It spans `connections.toml` **and** `environments/` so a transport/env-value redirect cannot change [the graph without changing the fingerprint]"*, and `:153` *"...or `environments/*.toml` — **THE SYSTEM SHALL** produce a different fingerprint."* **The shipped code does not meet that SHALL in the documented layout.** (ADR 0041 `:238` separately leaves "Fingerprint env-value scope" open, but that question is about hashing *resolved* values — it does not cover this, which is the declared scope failing to resolve at all.) | Repointing `acme_adt_host` in `environments/prod.toml` redirects PHI to a new peer with an **identical** fingerprint, no drift flag, and an audit row indistinguishable from a clean reload. | **No** — and ADR 0041 explicitly claims the opposite ("spans … `environments/`"). The existing test passes only because it writes `environments/` *inside* the tmp config dir. | P0 | | "Promoted to PROD" can be false | A remote promote sends `config_dir: null` (`promote.ts:135-149`) so the engine reloads its **own** on-disk dir; `ReloadResult` (`api/models.py:351-360`) carries no fingerprint. If CI/CD never delivered the commit, PROD reloads stale bytes and the toast still says "promoted". | Silent whenever element counts are unchanged — i.e. almost every Handler-logic change. Highest-frequency wrong belief in the publish path. | **Partly — and the partial coverage is the trap. RE-MEASURED 2026-08-15 (BACKLOG #1100).** "Nothing recomputes or compares a fingerprint" is **FALSE**: `ConfigProvenance` (`api/app.py:4570-4590`) reads `engine.loaded_config_fingerprint`, **recomputes** `config_fingerprint_detail(target)` off the loop, and **compares** them — `drift = current.get("fingerprint") != fp` — returning a `drift` flag. ADR 0041 D1 also writes a fingerprint-bearing `config_reload` audit row on every reload (`:554-581`). **BUT IT COMPARES THE WRONG TWO THINGS FOR THIS RISK.** `drift` is *loaded-vs-disk-now*; the failure here is *disk-vs-the-commit-CI-was-meant-to-deliver*. **If CI/CD never delivered, disk still equals what was loaded, so `drift` is `False` and provenance reports clean.** So the row's concern survives its evidence completely — and is now *harder* to see, because a reviewer who finds `ConfigProvenance` will reasonably conclude it is covered. Still true: `ReloadResult` (**`api/models.py:371`**, cited `:351-360`) carries no fingerprint, so the promoting caller never receives one; and there is no `messagefoundry fingerprint` CLI (subcommands are `graph`, `serve`, `validate`). **The fix is smaller than this row implied** — the fingerprint is already computed on every reload; it needs *surfacing in `ReloadResult`* and comparing against an expected value, not building | P0 | | A held (dual-control) promote renders as success | `engineClient.postJson` resolves any 2xx and casts the body (`engineClient.ts:92-102`); a 202 `PendingApprovalResponse` becomes a `ReloadResult` with `undefined` counts, and `promote.ts:191` prints "promoted to PROD — live graph: undefined inbound…". | Exactly the deployments that enabled dual-control **for production**. Operator believes the graph swapped; it is sitting in a queue. | **No — CONFIRMED 2026-08-15 (BACKLOG #1100), and both anchors still land, which is rare in this sweep.** `engineClient.ts:94-96` is exactly `if (status >= 200 && status < 300) { resolve((text ? JSON.parse(text) : {}) as T)` — **any** 2xx resolves and blind-casts, so a 202 body becomes a `ReloadResult` of `undefined`s; `promote.ts:191-195` then prints the success toast with **no status check**. Measured: **41 IDE test files exist and NOT ONE drives `promote.ts`** (`promote-target.test.ts` tests `promoteTarget.ts`, a different module). **Stronger than the row states: `PendingApproval` appears nowhere in `ide/`, and there is no 202 handling anywhere in `ide/src`** — the client does not model the pending-approval response at all, so this is not a missed branch but an absent concept. **The fix is one status check** at the `postJson` call site, not new machinery | P0 | -| Split-config cluster / split engine-shard estate | Convergence coordinates *when* nodes reload; each reloads its **own** dir (`config_convergence.py:16-20`). `ClusterStatus` exposes only the integer `config_version` (`api/models.py:737-748`), never a content fingerprint. Promote picks **one** engine-shard URL (`promoteTarget.ts:53-55`). | Node A runs the new graph, node B re-applies its old one; both write clean `config_reload` rows and report the same `config_version`. After a failover the wrong graph serves production. Engine shards over one unified store diverge per lane. | **No.** ADR 0041 lists this as unresolved — its open item to coordinate with the engine-shard owner. | P0 | +| Split-config cluster / split engine-shard estate | Convergence coordinates *when* nodes reload; each reloads its **own** dir (`config_convergence.py:16-20`). `ClusterStatus` exposes only the integer `config_version` (**`api/models.py:786`**, cited `:737-748`), never a content fingerprint. **CONFIRMED 2026-08-15 (BACKLOG #1100):** `ClusterStatus` fields are exactly `node_id, clustered, is_leader, role, config_version`, and `ClusterNode` (`:800`) adds none either — neither mentions `fingerprint`. The convergence anchor lands exactly: `config_convergence.py:16-18` states the assumption in its own words — *"The version token coordinates when nodes reload; each node reloads its OWN config dir. Skewed config dirs would diverge."* **THIS IS THE THIRD FACET OF ONE GAP, NOT THREE GAPS.** ADR 0041 D1 computes a content fingerprint on every reload, and it is surfaced in **none** of the three places a consumer could compare it: not in `ReloadResult` (row 109), not in `ClusterStatus` here, and not at all on the restart path (row 107, which never sets it). **One change — surfacing the already-computed fingerprint on those responses — addresses all three rows**, which is worth knowing before any of them is scoped separately. Promote picks **one** engine-shard URL (`promoteTarget.ts:53-55`). | Node A runs the new graph, node B re-applies its old one; both write clean `config_reload` rows and report the same `config_version`. After a failover the wrong graph serves production. Engine shards over one unified store diverge per lane. | **No.** ADR 0041 lists this as unresolved — its open item to coordinate with the engine-shard owner. | P0 | | Composite pipeline never exercised | Every unit exists; the assembly (author → check → non-prod → traffic → identical artifact to prod → per-environment substitution → rollback) has no test, no harness rig, and `FEATURE-COVERAGE-PLAN.md` excludes the IDE from every subsystem (`:946, :1216, :1379, :1420, :1471, :1521`). | The interfaces between the pieces — env substitution, artifact identity, pre-flight-vs-apply ordering, rollback — are where the real defects live. | **No.** | P0 | | Post-quiesce failure leaves partial state | Rollback restores **only** `self.registry` + inbound intake (`wiring_runner.py:3151-3164`). By then the live-lookup executor has been rebuilt and the old one `aclose`d (`:3096-3104`), sandbox sessions dropped (`:3086-3095`), and `_reconcile_outbounds` may have partially applied (`:3149`). | Old Routers/Handlers run against **new-graph** `db_lookup`/`fhir_lookup` pools. If the new graph dropped a `DbLookup`, every old-graph lookup raises post-ACK — a silent per-message ERROR/dead-letter storm behind an audit row the operator reads as a clean no-op. | **No.** No test drives a failure at the inbound-bind or `_reconcile_outbounds` step. | P1 | | Same bytes, two different outcomes | `reload()` has **no** ADR-0031 per-inbound fault isolation (`wiring_runner.py:3108-3134`) while `start()` does (`:2236-2242` `_record_failed`). One unbindable inbound aborts the whole publish; the same bundle on a restart comes up with that connection isolated and everything else running. | An operator who "retries via a service restart" gets a partially-live graph they believed the engine had refused. | **No**, and undocumented. | P1 | From da3b5878d079d1300d53c2d25f44c2ffeb9be110 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 08:22:50 -0500 Subject: [PATCH 31/66] docs(testing): correct a false claim I committed in f21d0e33 -- the CLI has 37 subcommands, not 3 (BACKLOG #1100) While checking a different row I read FEATURE-COVERAGE-PLAN.md:1420, "all 28 argparse subcommands", against my own note in row 109 that the subcommands are "graph, serve, validate". Both cannot be right. MINE WAS WRONG. An AST walk over every add_parser call finds 38 calls, 37 distinct (`import` appears twice): adr-analyze, ai-policy, alert, audit-anchor, audit-verify, backup, cert, check, codeset, connection, corepoint, dryrun, gen-key, generate, graph, hl7schema, hl7structures, impact, import, init, inventory, lens, parse, protect-key, rekey-audit, restore-verify, rewrite, rotate-key, schema, security, self-signed, serve, service, supervise, support-bundle, validate, verify. My original grep, `add_parser("[a-z-]*"`, matched only single-line call sites with that exact shape -- 3 of 38. THE CONCLUSION WAS UNAFFECTED: `fingerprint` is not among them either way, so row 109's substantive point stands. THE EVIDENCE WAS NOT, AND THAT IS PRECISELY THE DEFECT THIS SWEEP EXISTS TO FIX. I have spent the night correcting rows whose conclusions were right and whose supporting citations had rotted, and I committed one of my own in the same form: a true claim propped on a false enumeration that reads as thorough because it names specific things. The row now cites the AST measurement and records what the earlier revision said and why it was wrong, rather than quietly replacing it -- a silent fix would leave the next reader unable to tell whether the number was ever checked. Also noted: FCP:1420's "28" does not match 37 either. I am not correcting that here; it is outside this row and this claim, and the discrepancy is now visible in the commit record for whoever owns that plan. --- docs/testing/master-test-plan/08-publishing-and-promotion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/08-publishing-and-promotion.md b/docs/testing/master-test-plan/08-publishing-and-promotion.md index 6d55ed9e0..4044aca95 100644 --- a/docs/testing/master-test-plan/08-publishing-and-promotion.md +++ b/docs/testing/master-test-plan/08-publishing-and-promotion.md @@ -106,7 +106,7 @@ post-quiesce failure. Those are this chapter's centre of gravity. |---|---|---|---|---| | Restart-based publish has zero attribution | **CONFIRMED 2026-08-15 (BACKLOG #1100); both anchors had drifted and are re-pointed.** The startup path loads at **`api/app.py:5725`** (`loaded = load_config(config_dir)`) and **`:5730`** (`engine.add_registry(loaded)`) — the cited `:5455-5462` now lands on a function signature's keyword parameters, ~270 lines away. And **only `Engine.reload()` sets `loaded_config_fingerprint`**: it is initialised `None` at **`engine.py:450`** and assigned at **`:1610`/`:1615`, both inside `reload()`** (def `:1479`; the cited `:1488-1500` is the right method, wrong lines). Verified from the other side too — **`Engine.start()` (`engine.py:875`, 301 lines) mentions none of `loaded_config_fingerprint`, `config_fingerprint`, `add_registry` or `load_config`.** The fingerprint-bearing `config_reload` audit row is written only from the reload path (`api/app.py:554-573`, registered as a gate action at `:547`). Copy-files + NSSM restart (and the tray **Restart Service**) leave `loaded=false` and write no fingerprint-bearing audit row. | Every engine that was ever restarted rather than reloaded — i.e. the primary CI/CD path and every service restart. `/ui` badge blank, IDE can never show DRIFTED, no `who activated these bytes` row. | **No.** `GET /config/provenance` returns `loaded=false`, which reads as "nothing loaded", not "unattested". | P0 | | Fingerprint is blind to the real `environments/` | `_FINGERPRINT_GLOBS` includes `environments/*.toml` **relative to the config dir** (`fingerprint.py:44`, globbed at `:59` as `base.glob(pattern)` where `base` is the directory passed in — i.e. `engine.last_reload_dir` / `engine.config_dir`), but every shipped/documented layout keeps `environments/` as a project-root **sibling** of `--config` (ADR 0017 layout; ADR 0050 anchoring). **CONFIRMED 2026-08-15 (BACKLOG #1100), measured on this checkout:** repo-root `environments/` holds **2** `.toml` files, and **`samples/config/environments/` does not exist** — so the glob resolves to nothing for the documented layout. **The redirect surface is exactly the peer endpoints:** those files declare `acme_adt_host`, `acme_adt_port`, `demo_oru_host`, `demo_oru_port`, `fhir_base_url`, `payer_rte_host`, `payer_rte_port` (14 keys each). **And ADR 0041 states the opposite as a REQUIREMENT, not a description** — `:100` *"It spans `connections.toml` **and** `environments/` so a transport/env-value redirect cannot change [the graph without changing the fingerprint]"*, and `:153` *"...or `environments/*.toml` — **THE SYSTEM SHALL** produce a different fingerprint."* **The shipped code does not meet that SHALL in the documented layout.** (ADR 0041 `:238` separately leaves "Fingerprint env-value scope" open, but that question is about hashing *resolved* values — it does not cover this, which is the declared scope failing to resolve at all.) | Repointing `acme_adt_host` in `environments/prod.toml` redirects PHI to a new peer with an **identical** fingerprint, no drift flag, and an audit row indistinguishable from a clean reload. | **No** — and ADR 0041 explicitly claims the opposite ("spans … `environments/`"). The existing test passes only because it writes `environments/` *inside* the tmp config dir. | P0 | -| "Promoted to PROD" can be false | A remote promote sends `config_dir: null` (`promote.ts:135-149`) so the engine reloads its **own** on-disk dir; `ReloadResult` (`api/models.py:351-360`) carries no fingerprint. If CI/CD never delivered the commit, PROD reloads stale bytes and the toast still says "promoted". | Silent whenever element counts are unchanged — i.e. almost every Handler-logic change. Highest-frequency wrong belief in the publish path. | **Partly — and the partial coverage is the trap. RE-MEASURED 2026-08-15 (BACKLOG #1100).** "Nothing recomputes or compares a fingerprint" is **FALSE**: `ConfigProvenance` (`api/app.py:4570-4590`) reads `engine.loaded_config_fingerprint`, **recomputes** `config_fingerprint_detail(target)` off the loop, and **compares** them — `drift = current.get("fingerprint") != fp` — returning a `drift` flag. ADR 0041 D1 also writes a fingerprint-bearing `config_reload` audit row on every reload (`:554-581`). **BUT IT COMPARES THE WRONG TWO THINGS FOR THIS RISK.** `drift` is *loaded-vs-disk-now*; the failure here is *disk-vs-the-commit-CI-was-meant-to-deliver*. **If CI/CD never delivered, disk still equals what was loaded, so `drift` is `False` and provenance reports clean.** So the row's concern survives its evidence completely — and is now *harder* to see, because a reviewer who finds `ConfigProvenance` will reasonably conclude it is covered. Still true: `ReloadResult` (**`api/models.py:371`**, cited `:351-360`) carries no fingerprint, so the promoting caller never receives one; and there is no `messagefoundry fingerprint` CLI (subcommands are `graph`, `serve`, `validate`). **The fix is smaller than this row implied** — the fingerprint is already computed on every reload; it needs *surfacing in `ReloadResult`* and comparing against an expected value, not building | P0 | +| "Promoted to PROD" can be false | A remote promote sends `config_dir: null` (`promote.ts:135-149`) so the engine reloads its **own** on-disk dir; `ReloadResult` (`api/models.py:351-360`) carries no fingerprint. If CI/CD never delivered the commit, PROD reloads stale bytes and the toast still says "promoted". | Silent whenever element counts are unchanged — i.e. almost every Handler-logic change. Highest-frequency wrong belief in the publish path. | **Partly — and the partial coverage is the trap. RE-MEASURED 2026-08-15 (BACKLOG #1100).** "Nothing recomputes or compares a fingerprint" is **FALSE**: `ConfigProvenance` (`api/app.py:4570-4590`) reads `engine.loaded_config_fingerprint`, **recomputes** `config_fingerprint_detail(target)` off the loop, and **compares** them — `drift = current.get("fingerprint") != fp` — returning a `drift` flag. ADR 0041 D1 also writes a fingerprint-bearing `config_reload` audit row on every reload (`:554-581`). **BUT IT COMPARES THE WRONG TWO THINGS FOR THIS RISK.** `drift` is *loaded-vs-disk-now*; the failure here is *disk-vs-the-commit-CI-was-meant-to-deliver*. **If CI/CD never delivered, disk still equals what was loaded, so `drift` is `False` and provenance reports clean.** So the row's concern survives its evidence completely — and is now *harder* to see, because a reviewer who finds `ConfigProvenance` will reasonably conclude it is covered. Still true: `ReloadResult` (**`api/models.py:371`**, cited `:351-360`) carries no fingerprint, so the promoting caller never receives one; and there is no `messagefoundry fingerprint` CLI — **verified by AST over every `add_parser` call: 37 distinct subcommands, none named `fingerprint`.** (An earlier revision of this note said the subcommands were "`graph`, `serve`, `validate`"; that came from a line-oriented grep that matched only 3 of 38 call sites and was wrong. The conclusion was unaffected — `fingerprint` is absent either way — but the evidence was not, which is the exact defect this sweep exists to fix.). **The fix is smaller than this row implied** — the fingerprint is already computed on every reload; it needs *surfacing in `ReloadResult`* and comparing against an expected value, not building | P0 | | A held (dual-control) promote renders as success | `engineClient.postJson` resolves any 2xx and casts the body (`engineClient.ts:92-102`); a 202 `PendingApprovalResponse` becomes a `ReloadResult` with `undefined` counts, and `promote.ts:191` prints "promoted to PROD — live graph: undefined inbound…". | Exactly the deployments that enabled dual-control **for production**. Operator believes the graph swapped; it is sitting in a queue. | **No — CONFIRMED 2026-08-15 (BACKLOG #1100), and both anchors still land, which is rare in this sweep.** `engineClient.ts:94-96` is exactly `if (status >= 200 && status < 300) { resolve((text ? JSON.parse(text) : {}) as T)` — **any** 2xx resolves and blind-casts, so a 202 body becomes a `ReloadResult` of `undefined`s; `promote.ts:191-195` then prints the success toast with **no status check**. Measured: **41 IDE test files exist and NOT ONE drives `promote.ts`** (`promote-target.test.ts` tests `promoteTarget.ts`, a different module). **Stronger than the row states: `PendingApproval` appears nowhere in `ide/`, and there is no 202 handling anywhere in `ide/src`** — the client does not model the pending-approval response at all, so this is not a missed branch but an absent concept. **The fix is one status check** at the `postJson` call site, not new machinery | P0 | | Split-config cluster / split engine-shard estate | Convergence coordinates *when* nodes reload; each reloads its **own** dir (`config_convergence.py:16-20`). `ClusterStatus` exposes only the integer `config_version` (**`api/models.py:786`**, cited `:737-748`), never a content fingerprint. **CONFIRMED 2026-08-15 (BACKLOG #1100):** `ClusterStatus` fields are exactly `node_id, clustered, is_leader, role, config_version`, and `ClusterNode` (`:800`) adds none either — neither mentions `fingerprint`. The convergence anchor lands exactly: `config_convergence.py:16-18` states the assumption in its own words — *"The version token coordinates when nodes reload; each node reloads its OWN config dir. Skewed config dirs would diverge."* **THIS IS THE THIRD FACET OF ONE GAP, NOT THREE GAPS.** ADR 0041 D1 computes a content fingerprint on every reload, and it is surfaced in **none** of the three places a consumer could compare it: not in `ReloadResult` (row 109), not in `ClusterStatus` here, and not at all on the restart path (row 107, which never sets it). **One change — surfacing the already-computed fingerprint on those responses — addresses all three rows**, which is worth knowing before any of them is scoped separately. Promote picks **one** engine-shard URL (`promoteTarget.ts:53-55`). | Node A runs the new graph, node B re-applies its old one; both write clean `config_reload` rows and report the same `config_version`. After a failover the wrong graph serves production. Engine shards over one unified store diverge per lane. | **No.** ADR 0041 lists this as unresolved — its open item to coordinate with the engine-shard owner. | P0 | | Composite pipeline never exercised | Every unit exists; the assembly (author → check → non-prod → traffic → identical artifact to prod → per-environment substitution → rollback) has no test, no harness rig, and `FEATURE-COVERAGE-PLAN.md` excludes the IDE from every subsystem (`:946, :1216, :1379, :1420, :1471, :1521`). | The interfaces between the pieces — env substitution, artifact identity, pre-flight-vs-apply ordering, rollback — are where the real defects live. | **No.** | P0 | From ec6aa8546851f08bdb7598453be6b8fc89004160 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 08:32:46 -0500 Subject: [PATCH 32/66] docs(testing): audit my own committed counts, and correct the second false one (BACKLOG #1100) After finding that I had committed a false subcommand enumeration (8e0f281b), I audited every count I asserted in this sweep by AST instead of grep. CORRECT, confirmed: 56 tests in test_scan_tokens_source.py; 12 in test_security_posture.py; 6 in test_webconsole_seam_snapshot.py. No nested tests in any of them, so the ^def test_ pattern happened to be accurate. WRONG: row 98's "18 test functions naming the surface". Measured across the five cited files: 76 test functions total, of which 6 have "intake" in the name, and the dedicated test_inbound_http_intake_auth.py alone holds 22. NOTHING IS 18. The figure came from `grep -rh "def test_.*intake" tests/ | wc -l` -- a count over ALL of tests/, which I then attributed to five specific files. BOTH OF MY FALSE ENUMERATIONS TONIGHT HAVE THE SAME TWO CAUSES, and naming them is worth more than either fix: 1. COUNTING LINES AND CALLING THEM THINGS. grep counts matching lines; a multi-line call site, a class-nested def or a second match on one line all break the equivalence silently. 2. CONFLATING SEARCH SCOPE WITH CITED SCOPE. I searched tests/ and reported the number as a property of five named files. Neither produces an error. Both produce a specific, confident, wrong number that reads as diligence BECAUSE it is specific. The substance is unaffected and slightly strengthened -- 22 dedicated tests is a better answer to "No test states the accepted control set" than 18 was. As with the subcommand fix, the row records what the earlier revision said and why it was wrong rather than silently replacing it. A NOTE ON THE AUDIT ITSELF: it also flagged test_ech_record_premise as "claimed 10, found 8". That one is MY AUDIT being wrong -- 8 functions produce 10 cases via parametrize, and I had only ever quoted pytest's case count in conversation, never in a document. A self-audit needs the same instrument-vs-question discipline as everything else, or it manufactures its own findings. --- docs/testing/master-test-plan/05-connections-and-transports.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/05-connections-and-transports.md b/docs/testing/master-test-plan/05-connections-and-transports.md index ccdd5e529..f5248b039 100644 --- a/docs/testing/master-test-plan/05-connections-and-transports.md +++ b/docs/testing/master-test-plan/05-connections-and-transports.md @@ -95,7 +95,7 @@ This chapter covers **every registered inbound/outbound connector and the connec | Eight post-plan connector ADRs have no coverage-plan row | Verified zero grep hits each for 0122, 0123, 0124, 0126, 0128, 0129, 0132, 0153 in `FEATURE-COVERAGE-PLAN.md` | Each has unit tests, but nothing owns their cross-backend, live-partner, on-box or PHI dimensions and no plan section notices a regression. The audit is point-in-time with no drift guard against the ADR ledger, so the hole widens with every demand-gate wave | No | P1 | | ADR 0124 `no_ack` finalizes on a TCP write | `_send_once_no_ack:897` / `_send_persistent_no_ack:1046` complete a delivery on a successful write with no ACK read and no MSA-1 validation | A `no_ack` outbound reports `PROCESSED` for a message the partner never processed. Operators read disposition as delivery confirmation, so the at-most-once-confirmation trade-off needs an explicit tested disposition and an operator-visible marker | No — `test_mllp_no_ack.py` and `test_no_ack_wiring.py` stop at the connector and wiring | P1 | | Five shipped connectors absent from the on-box §D matrix | Email/SMTP, Direct S/MIME, DICOMweb STOW-RS, the ADR 0023 HTTP listener and PassThrough have no `WIN2025-TEST-MATRIX.md` §D row (lines 47-59) | The box gate cannot certify connectors it does not list. Direct and Email are PHI **egress** paths and the HTTP listener is a PHI **intake** path, so `W25:S6.3` would declare the box accepted with three PHI paths untested on the host | No | P1 | -| Inbound HTTP listener authenticates, but **defaults to `none`** | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15). The evidence in this row was false in every part.** "grep ... returns **0** hits" measures **52** matching lines today; `Http()` does *not* expose "caps + TLS/mTLS only"; and ADR 0023 no longer defers auth — **ADR 0154 (Accepted 2026-07-31) is literally titled its "deferred tail"** and delivered it. Built: `intake_auth: Literal["none","api_key","bearer","mtls_subject"]`, enforced by `_authorize_head` (`http_listener.py:618`) with a 401 + `WWW-Authenticate`, credentials `env()`-only, rotation via `intake_api_key_next`, health probes inside the gate by default (`intake_auth_health="require"`), and failed-attempt rate limits (10/min/peer, 60/min global) | **The mechanism is no longer the gap; the DEFAULT is.** `intake_auth` defaults to `"none"`, and `_authorize_head` returns early when it is unset — so a site that deploys the listener off-loopback without setting it **would** accept unauthenticated POSTs into the pipeline. That is now an explicit, documented, tested opt-in rather than an implicit omission, which is a different and much narrower decision: whether `"none"` is the right default for a PHI intake path | **Yes** — `tests/test_inbound_http_intake_auth.py` plus `test_auth_entry_hardening.py`, `test_credential_parameter_mapping.py`, `test_connection_factory_redaction_domain.py` and `test_inbound_http_sync_reply.py` (18 test functions naming the surface). What remains unasserted is only the **default-posture** question above | P1 | +| Inbound HTTP listener authenticates, but **defaults to `none`** | **CORRECTED (BACKLOG #1100, re-measured 2026-08-15). The evidence in this row was false in every part.** "grep ... returns **0** hits" measures **52** matching lines today; `Http()` does *not* expose "caps + TLS/mTLS only"; and ADR 0023 no longer defers auth — **ADR 0154 (Accepted 2026-07-31) is literally titled its "deferred tail"** and delivered it. Built: `intake_auth: Literal["none","api_key","bearer","mtls_subject"]`, enforced by `_authorize_head` (`http_listener.py:618`) with a 401 + `WWW-Authenticate`, credentials `env()`-only, rotation via `intake_api_key_next`, health probes inside the gate by default (`intake_auth_health="require"`), and failed-attempt rate limits (10/min/peer, 60/min global) | **The mechanism is no longer the gap; the DEFAULT is.** `intake_auth` defaults to `"none"`, and `_authorize_head` returns early when it is unset — so a site that deploys the listener off-loopback without setting it **would** accept unauthenticated POSTs into the pipeline. That is now an explicit, documented, tested opt-in rather than an implicit omission, which is a different and much narrower decision: whether `"none"` is the right default for a PHI intake path | **Yes** — `tests/test_inbound_http_intake_auth.py` plus `test_auth_entry_hardening.py`, `test_credential_parameter_mapping.py`, `test_connection_factory_redaction_domain.py` and `test_inbound_http_sync_reply.py` (**`test_inbound_http_intake_auth.py` alone holds 22 test functions**, counted by AST; the five files together hold 76, of which 6 name `intake` directly. An earlier revision said "18 test functions naming the surface" — that figure counted `def test_.*intake` LINES across all of `tests/` and was then attributed to these five files, conflating search scope with cited scope. The correction strengthens the point rather than weakening it.). What remains unasserted is only the **default-posture** question above | P1 | | No negative mTLS handshake on any accept path | `test_mllp_tls.py:83` asserts `verify_mode` on the constructed context, not a live refusal. Nothing drives a client with no cert or a wrong-CA cert at an MLLP / HTTP-listener / DICOM-SCP mTLS listener | mTLS is the only peer authentication the raw listeners and the HTTP listener have. A regression leaving `CERT_OPTIONAL` in place admits unauthenticated PHI intake with no failing test | Owned as `FCP:MLLP-9`'s open gap; unclosed | P1 | | `FEATURE-MAP.md` §1 understates shipped capability | `:45` marks REST-IN/SOAP-IN/FHIR-IN deferred (ADR 0023 built); `:46` marks MLLP persistent deferred (ADR 0067 built). No row at all for Email, Direct, Timer, PassThrough, Loopback, the HTTP listener, compression, `no_ack`, or the forward proxy | The public capability catalogue is what a test author, an evaluator and an adopter scope from. Understated connectors go untested and unaudited; `tests/test_feature_map_claims.py` guards only the ASVS score and private links, so it rots further | No | P1 | | Listener `SSLContext` built once at construction | `mllp.py:1324`, `http_listener.py:292`, `dicom.py:178` all build the context in `__init__`, so a renewed cert on disk is not presented until a connection restart or config reload | An operator rotating a listener cert in place believes it took effect; when the old cert expires the listener stops accepting partners — a full intake outage — and the only prior signal is an expiry alert they may have silenced after "rotating" | No test or doc states it; `CONNECTIONS.md:714` carries rotation guidance for the DB CA only | P2 | From b01551d671606c6da52c9af48038b5054e6ed617 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 08:43:12 -0500 Subject: [PATCH 33/66] docs(testing): confirm the composite-pipeline row, and name why it has stayed open (BACKLOG #1100) 08-publishing-and-promotion.md row 112 is CORRECT. All six FEATURE-COVERAGE-PLAN citations LAND -- unusual in this sweep -- and every one is an EXPLICIT exclusion rather than the word "IDE" merely appearing on the line: :946 "Excludes the VS Code IDE (owner's lane)" :1216 "OUT of scope: the VS Code IDE extension" :1379 "... and the VS Code IDE extension are out of scope" :1420 "Excludes the VS Code IDE extension (owner's parallel work)" :1471 "... the VS Code IDE Steps view ... are out of scope" :1521 "Excludes the VS Code IDE extension (owner's parallel work)" I checked the clause and not just the token because this sweep has produced four separate cases where a term appeared on the right line for the wrong reason. THE ATTRIBUTION IS THE PART WORTH ADDING. Four of the six say "owner's lane" or "owner's parallel work". These are DELIBERATE scoping decisions, not oversights. So the composite pipeline -- author, check, non-prod, traffic, identical artifact to prod, per-environment substitution, rollback -- is unowned BY CONSTRUCTION: each half sits inside a scope that explicitly disclaims the other, and the seam between them belongs to neither. That reframes the remedy. The row reads as "someone should write the assembly test"; the accurate version is "this requires crossing an ownership boundary that six separate scope statements were written to establish". Nobody is going to close it by accident, and no amount of test-writing inside either lane will reach it. Chapter 08's P0 risk set is now complete (107, 108, 109, 110, 111, 112), making seven chapters: 02, 03, 04, 08, 09, 10, 16. --- docs/testing/master-test-plan/08-publishing-and-promotion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/08-publishing-and-promotion.md b/docs/testing/master-test-plan/08-publishing-and-promotion.md index 4044aca95..09ad6abb9 100644 --- a/docs/testing/master-test-plan/08-publishing-and-promotion.md +++ b/docs/testing/master-test-plan/08-publishing-and-promotion.md @@ -109,7 +109,7 @@ post-quiesce failure. Those are this chapter's centre of gravity. | "Promoted to PROD" can be false | A remote promote sends `config_dir: null` (`promote.ts:135-149`) so the engine reloads its **own** on-disk dir; `ReloadResult` (`api/models.py:351-360`) carries no fingerprint. If CI/CD never delivered the commit, PROD reloads stale bytes and the toast still says "promoted". | Silent whenever element counts are unchanged — i.e. almost every Handler-logic change. Highest-frequency wrong belief in the publish path. | **Partly — and the partial coverage is the trap. RE-MEASURED 2026-08-15 (BACKLOG #1100).** "Nothing recomputes or compares a fingerprint" is **FALSE**: `ConfigProvenance` (`api/app.py:4570-4590`) reads `engine.loaded_config_fingerprint`, **recomputes** `config_fingerprint_detail(target)` off the loop, and **compares** them — `drift = current.get("fingerprint") != fp` — returning a `drift` flag. ADR 0041 D1 also writes a fingerprint-bearing `config_reload` audit row on every reload (`:554-581`). **BUT IT COMPARES THE WRONG TWO THINGS FOR THIS RISK.** `drift` is *loaded-vs-disk-now*; the failure here is *disk-vs-the-commit-CI-was-meant-to-deliver*. **If CI/CD never delivered, disk still equals what was loaded, so `drift` is `False` and provenance reports clean.** So the row's concern survives its evidence completely — and is now *harder* to see, because a reviewer who finds `ConfigProvenance` will reasonably conclude it is covered. Still true: `ReloadResult` (**`api/models.py:371`**, cited `:351-360`) carries no fingerprint, so the promoting caller never receives one; and there is no `messagefoundry fingerprint` CLI — **verified by AST over every `add_parser` call: 37 distinct subcommands, none named `fingerprint`.** (An earlier revision of this note said the subcommands were "`graph`, `serve`, `validate`"; that came from a line-oriented grep that matched only 3 of 38 call sites and was wrong. The conclusion was unaffected — `fingerprint` is absent either way — but the evidence was not, which is the exact defect this sweep exists to fix.). **The fix is smaller than this row implied** — the fingerprint is already computed on every reload; it needs *surfacing in `ReloadResult`* and comparing against an expected value, not building | P0 | | A held (dual-control) promote renders as success | `engineClient.postJson` resolves any 2xx and casts the body (`engineClient.ts:92-102`); a 202 `PendingApprovalResponse` becomes a `ReloadResult` with `undefined` counts, and `promote.ts:191` prints "promoted to PROD — live graph: undefined inbound…". | Exactly the deployments that enabled dual-control **for production**. Operator believes the graph swapped; it is sitting in a queue. | **No — CONFIRMED 2026-08-15 (BACKLOG #1100), and both anchors still land, which is rare in this sweep.** `engineClient.ts:94-96` is exactly `if (status >= 200 && status < 300) { resolve((text ? JSON.parse(text) : {}) as T)` — **any** 2xx resolves and blind-casts, so a 202 body becomes a `ReloadResult` of `undefined`s; `promote.ts:191-195` then prints the success toast with **no status check**. Measured: **41 IDE test files exist and NOT ONE drives `promote.ts`** (`promote-target.test.ts` tests `promoteTarget.ts`, a different module). **Stronger than the row states: `PendingApproval` appears nowhere in `ide/`, and there is no 202 handling anywhere in `ide/src`** — the client does not model the pending-approval response at all, so this is not a missed branch but an absent concept. **The fix is one status check** at the `postJson` call site, not new machinery | P0 | | Split-config cluster / split engine-shard estate | Convergence coordinates *when* nodes reload; each reloads its **own** dir (`config_convergence.py:16-20`). `ClusterStatus` exposes only the integer `config_version` (**`api/models.py:786`**, cited `:737-748`), never a content fingerprint. **CONFIRMED 2026-08-15 (BACKLOG #1100):** `ClusterStatus` fields are exactly `node_id, clustered, is_leader, role, config_version`, and `ClusterNode` (`:800`) adds none either — neither mentions `fingerprint`. The convergence anchor lands exactly: `config_convergence.py:16-18` states the assumption in its own words — *"The version token coordinates when nodes reload; each node reloads its OWN config dir. Skewed config dirs would diverge."* **THIS IS THE THIRD FACET OF ONE GAP, NOT THREE GAPS.** ADR 0041 D1 computes a content fingerprint on every reload, and it is surfaced in **none** of the three places a consumer could compare it: not in `ReloadResult` (row 109), not in `ClusterStatus` here, and not at all on the restart path (row 107, which never sets it). **One change — surfacing the already-computed fingerprint on those responses — addresses all three rows**, which is worth knowing before any of them is scoped separately. Promote picks **one** engine-shard URL (`promoteTarget.ts:53-55`). | Node A runs the new graph, node B re-applies its old one; both write clean `config_reload` rows and report the same `config_version`. After a failover the wrong graph serves production. Engine shards over one unified store diverge per lane. | **No.** ADR 0041 lists this as unresolved — its open item to coordinate with the engine-shard owner. | P0 | -| Composite pipeline never exercised | Every unit exists; the assembly (author → check → non-prod → traffic → identical artifact to prod → per-environment substitution → rollback) has no test, no harness rig, and `FEATURE-COVERAGE-PLAN.md` excludes the IDE from every subsystem (`:946, :1216, :1379, :1420, :1471, :1521`). | The interfaces between the pieces — env substitution, artifact identity, pre-flight-vs-apply ordering, rollback — are where the real defects live. | **No.** | P0 | +| Composite pipeline never exercised | Every unit exists; the assembly (author → check → non-prod → traffic → identical artifact to prod → per-environment substitution → rollback) has no test, no harness rig, and `FEATURE-COVERAGE-PLAN.md` excludes the IDE from every subsystem (`:946, :1216, :1379, :1420, :1471, :1521`). **CONFIRMED 2026-08-15 (BACKLOG #1100) — all six anchors LAND, which is unusual in this sweep, and every one is an explicit exclusion**, not a co-occurrence: *"Excludes the VS Code IDE (owner's lane)"* (`:946`), *"OUT of scope: the VS Code IDE extension"* (`:1216`), *"...and the VS Code IDE extension are out of scope"* (`:1379`), *"Excludes the VS Code IDE extension (owner's parallel work)"* (`:1420`, `:1521`), and *"...the VS Code IDE Steps view / element-centric connections graph ... are out of scope"* (`:1471`). **The reason matters for whoever picks this up: the exclusions are ATTRIBUTED — "owner's lane", "owner's parallel work" — so this is a deliberate scoping decision, not an oversight.** The composite pipeline is therefore unowned **by construction**: each half sits inside a scope that explicitly disclaims the other. That is why it has stayed open, and it means the fix requires crossing an ownership boundary rather than writing a missing test. | The interfaces between the pieces — env substitution, artifact identity, pre-flight-vs-apply ordering, rollback — are where the real defects live. | **No.** | P0 | | Post-quiesce failure leaves partial state | Rollback restores **only** `self.registry` + inbound intake (`wiring_runner.py:3151-3164`). By then the live-lookup executor has been rebuilt and the old one `aclose`d (`:3096-3104`), sandbox sessions dropped (`:3086-3095`), and `_reconcile_outbounds` may have partially applied (`:3149`). | Old Routers/Handlers run against **new-graph** `db_lookup`/`fhir_lookup` pools. If the new graph dropped a `DbLookup`, every old-graph lookup raises post-ACK — a silent per-message ERROR/dead-letter storm behind an audit row the operator reads as a clean no-op. | **No.** No test drives a failure at the inbound-bind or `_reconcile_outbounds` step. | P1 | | Same bytes, two different outcomes | `reload()` has **no** ADR-0031 per-inbound fault isolation (`wiring_runner.py:3108-3134`) while `start()` does (`:2236-2242` `_record_failed`). One unbindable inbound aborts the whole publish; the same bundle on a restart comes up with that connection isolated and everything else running. | An operator who "retries via a service restart" gets a partially-live graph they believed the engine had refused. | **No**, and undocumented. | P1 | | Duplicate ingest on publish | Quiescing closes established MLLP client connections (`transports/mllp.py:1352-1372`); the body is committed before the ACK, so nothing is lost, but a **not-yet-sent ACK** is — the sender retries and the engine ingests a duplicate. | Clinically material for non-idempotent downstreams (duplicate orders/results). The scope requirement is "must not lose **or duplicate**". | Partially — the consequence is documented in the code comment, but no test drives a reload under concurrent inbound traffic and no duplicate rate is published. | P1 | From 1164f3ce68ec272f04ca5db04ecd1ace39eac70b Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 08:59:38 -0500 Subject: [PATCH 34/66] docs(testing): the security warning IS tested, and the real config gap is mis-SECTIONING (BACKLOG #1100) 07-config-wiring-and-cli.md row 80. One clause is false, and correcting it makes the surviving gap sharper rather than smaller. CONFIRMED: extra="ignore" holds via the _Section base (settings.py:173) and ServiceSettings (:3811, cited :3623). The only extra="forbid" models are EscalationTier and AlertRule -- nested rule objects, not sections. I checked that specifically because two forbid declarations initially looked like they refuted "each section"; they do not. test_settings.py:118 pins the silence as intended and its anchor lands. FALSE: "that warning itself is untested". tests/test_security_config.py covers it with a positive AND a negative control -- test_unknown_security_key_warns_and_is_ ignored (:386) and test_known_security_keys_warn_about_nothing (:401), the latter asserting "unrecognized key" is ABSENT for valid keys. Ran both: 2 passed. The warning lives at :4060-4074; the cited :3880 is now _cluster_requires_server_db. THE REAL GAP IS THE ONE THIS ROW'S OWN FIRST EXAMPLE FALLS INTO. The warning diffs set(raw) - set(SecuritySettings.model_fields), so it catches an unknown KEY inside a PRESENT [security] section. A misspelled SECTION HEADER -- the row's own [secrity] -- is dropped by ServiceSettings' extra="ignore" and never reaches the check. It warns nothing. So MIS-SECTIONING IS THE UNCOVERED HALF, NOT MIS-KEYING, and the row had them the other way round while listing both. Its stated consequence stands unchanged for the mis-sectioned case: the operator believes a posture switch is set, the permissive default applies, and there is zero signal. That also explains why the existing coverage looks reassuring. Someone checking "is the unknown-key path tested?" finds two tests and a negative control and stops -- which is the correct answer to a question adjacent to the dangerous one. --- docs/testing/master-test-plan/07-config-wiring-and-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/07-config-wiring-and-cli.md b/docs/testing/master-test-plan/07-config-wiring-and-cli.md index 51723f093..b4a9ee420 100644 --- a/docs/testing/master-test-plan/07-config-wiring-and-cli.md +++ b/docs/testing/master-test-plan/07-config-wiring-and-cli.md @@ -77,7 +77,7 @@ This chapter covers the **configuration bundle and the command-line surface that |---|---|---|---|---| | The shipped gate runs in no engine CI leg | A change demotes `build-check` from required, or breaks a sub-check, and every unit slice still passes | Every adopter's whole config-quality gate exits 0 on a config `serve` will refuse; discovered at deploy or in production | No — `.pre-commit-config.yaml` owns `.git/hooks/pre-commit`; `.mefor-hooks/pre-commit` is generated but not installed; no workflow invokes `check` | P0 | | No pin on the check roster | A check is dropped or its `required` flag flipped; exit-code contract changes silently | `build-check` is the only gate catching an ADR 0092 production-PHI cleartext hop pre-`serve`; `reference-backend` the only one catching a `Reference()` against SQL Server that raises post-ACK forever | No test enumerates `run_checks`' result names or flags | P0 | -| Unknown settings sections/keys silently dropped | `[secrity]`, a misspelled `require_mfa`, or `allow_unencrypted_phi` in the wrong section loads clean and applies a default | An operator believes a posture switch is set; in the mis-sectioned case the **insecure** default applies with zero signal | No — `ServiceSettings` and each section are `extra="ignore"` (`settings.py:3623`) and `tests/test_settings.py:118` pins the silence as intended. Only `[security]` warns (`settings.py:3880`), and that warning itself is untested | P0 | +| Unknown settings sections/keys silently dropped | `[secrity]`, a misspelled `require_mfa`, or `allow_unencrypted_phi` in the wrong section loads clean and applies a default | An operator believes a posture switch is set; in the mis-sectioned case the **insecure** default applies with zero signal | **Partly — RE-MEASURED 2026-08-15 (BACKLOG #1100); one clause is FALSE and the surviving gap is SHARPER than written.** Confirmed: `extra="ignore"` holds via the `_Section` base (**`settings.py:173`**) and `ServiceSettings` (**`:3811`**, cited `:3623`) — the only `extra="forbid"` models are `EscalationTier` and `AlertRule`, nested rule objects, not sections. `tests/test_settings.py:118` (`test_unknown_sections_and_keys_ignored`) does pin the silence as intended; that anchor lands. **FALSE: "that warning itself is untested."** `tests/test_security_config.py` covers it with a positive AND a negative control — `test_unknown_security_key_warns_and_is_ignored` (`:386`) and `test_known_security_keys_warn_about_nothing` (`:401`, asserting "unrecognized key" is absent for valid keys). The warning is at **`:4060-4074`** (cited `:3880`, which is now `_cluster_requires_server_db`). **THE REAL GAP, WHICH THIS ROW'S OWN FIRST EXAMPLE FALLS INTO:** that warning diffs `set(raw) - set(SecuritySettings.model_fields)` — it catches an unknown **key inside a present `[security]` section**. A misspelled **section header** like `[secrity]` is dropped by `ServiceSettings`' `extra="ignore"` and **never reaches the check**, so it warns nothing. Mis-sectioning is the uncovered half, not mis-keying | P0 | | IDE Steps contract drift | `lens.py` changes `kind` / `params` / `literal_params` / `line_start` / `line_end`; both suites stay green against committed static fixtures | Steps view mis-renders or, worse, `lens rewrite` splices an edit into the wrong statement of a clinical transform — wrong data downstream | No — `ide/src/test/suite/steps.test.ts:28` reads `ide/src/test/fixtures/lens/*.json`; the `ide` CI job has no Python and nothing regenerates or diffs them | P0 | | 5 of 28 settings sections unreachable by `MEFOR_*` | `sandbox`, `service`, `cert_monitor`, `secret_rotation`, `update_check` are absent from `_SECTIONS`; the three multi-word names are additionally unreachable through `.partition("_")` | `MEFOR_SANDBOX_MODE=subprocess` in an NSSM environment block yields `off` — the ADR 0087 Router/Handler isolation boundary believed on and silently off | No test asserts `_SECTIONS` equals `ServiceSettings.model_fields`; an unmatched `MEFOR_*` var is dropped without a warning | P0 | | Shift-left checks disable themselves on the likeliest-wrong input | `posture` / `build-check` / `reference-backend` return `ok=True, skipped=True` when `messagefoundry.toml` will not load (`checks.py:1189`, `:1266`, `:1415`) | An adopter with an ADR 0118 relocated key (a hard `ValueError`) gets fully green CI then a `serve` refusal at deploy | No test covers the settings-did-not-load skip path | P1 | From 90edf4f327e34f23e4aaa8a044e0f42ba66c7fcc Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 09:03:08 -0500 Subject: [PATCH 35/66] docs(testing): confirm the unreachable-settings-sections P0, exactly (BACKLOG #1100) 07-config-wiring-and-cli.md row 82 is CORRECT on every claim, measured by AST rather than grep -- which matters here, because this row is a COUNT plus an ENUMERATION, the class that produced both of my own false claims tonight. _SECTIONS (settings.py:109) 23 entries ServiceSettings annotated fields 28 difference 5, and EXACTLY the five named: sandbox, cert_monitor, secret_rotation, update_check, service The .partition("_") mechanism is at :3971 -- section, _, key = name[len(_ENV_PREFIX):].lower().partition("_") so MEFOR_CERT_MONITOR_* parses as section "cert", key "monitor_*". The three multi-word names are unreachable a SECOND way, independently of _SECTIONS. MEFOR_SANDBOX appears in ZERO tests. AND THE "NO TEST PINS _SECTIONS" CLAIM SURVIVES A TRAP THAT LOOKED LIKE COVERAGE. Five test files match the token _SECTIONS. NONE imports settings._SECTIONS: test_verify.py's hit is ALL_SECTIONS from verify.runner -- host, store, smoke, manual, federation -- an unrelated constant that my grep matched as a substring. A file count would have said "covered five times over". So ADR 0087's Router/Handler isolation boundary can be set in an NSSM environment block, read back as off, and say nothing. That is a security boundary an operator would reasonably believe is on, and the two independent failure paths mean fixing _SECTIONS alone would still leave the three multi-word sections unreachable. --- docs/testing/master-test-plan/07-config-wiring-and-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/07-config-wiring-and-cli.md b/docs/testing/master-test-plan/07-config-wiring-and-cli.md index b4a9ee420..0942f432b 100644 --- a/docs/testing/master-test-plan/07-config-wiring-and-cli.md +++ b/docs/testing/master-test-plan/07-config-wiring-and-cli.md @@ -79,7 +79,7 @@ This chapter covers the **configuration bundle and the command-line surface that | No pin on the check roster | A check is dropped or its `required` flag flipped; exit-code contract changes silently | `build-check` is the only gate catching an ADR 0092 production-PHI cleartext hop pre-`serve`; `reference-backend` the only one catching a `Reference()` against SQL Server that raises post-ACK forever | No test enumerates `run_checks`' result names or flags | P0 | | Unknown settings sections/keys silently dropped | `[secrity]`, a misspelled `require_mfa`, or `allow_unencrypted_phi` in the wrong section loads clean and applies a default | An operator believes a posture switch is set; in the mis-sectioned case the **insecure** default applies with zero signal | **Partly — RE-MEASURED 2026-08-15 (BACKLOG #1100); one clause is FALSE and the surviving gap is SHARPER than written.** Confirmed: `extra="ignore"` holds via the `_Section` base (**`settings.py:173`**) and `ServiceSettings` (**`:3811`**, cited `:3623`) — the only `extra="forbid"` models are `EscalationTier` and `AlertRule`, nested rule objects, not sections. `tests/test_settings.py:118` (`test_unknown_sections_and_keys_ignored`) does pin the silence as intended; that anchor lands. **FALSE: "that warning itself is untested."** `tests/test_security_config.py` covers it with a positive AND a negative control — `test_unknown_security_key_warns_and_is_ignored` (`:386`) and `test_known_security_keys_warn_about_nothing` (`:401`, asserting "unrecognized key" is absent for valid keys). The warning is at **`:4060-4074`** (cited `:3880`, which is now `_cluster_requires_server_db`). **THE REAL GAP, WHICH THIS ROW'S OWN FIRST EXAMPLE FALLS INTO:** that warning diffs `set(raw) - set(SecuritySettings.model_fields)` — it catches an unknown **key inside a present `[security]` section**. A misspelled **section header** like `[secrity]` is dropped by `ServiceSettings`' `extra="ignore"` and **never reaches the check**, so it warns nothing. Mis-sectioning is the uncovered half, not mis-keying | P0 | | IDE Steps contract drift | `lens.py` changes `kind` / `params` / `literal_params` / `line_start` / `line_end`; both suites stay green against committed static fixtures | Steps view mis-renders or, worse, `lens rewrite` splices an edit into the wrong statement of a clinical transform — wrong data downstream | No — `ide/src/test/suite/steps.test.ts:28` reads `ide/src/test/fixtures/lens/*.json`; the `ide` CI job has no Python and nothing regenerates or diffs them | P0 | -| 5 of 28 settings sections unreachable by `MEFOR_*` | `sandbox`, `service`, `cert_monitor`, `secret_rotation`, `update_check` are absent from `_SECTIONS`; the three multi-word names are additionally unreachable through `.partition("_")` | `MEFOR_SANDBOX_MODE=subprocess` in an NSSM environment block yields `off` — the ADR 0087 Router/Handler isolation boundary believed on and silently off | No test asserts `_SECTIONS` equals `ServiceSettings.model_fields`; an unmatched `MEFOR_*` var is dropped without a warning | P0 | +| 5 of 28 settings sections unreachable by `MEFOR_*` | `sandbox`, `service`, `cert_monitor`, `secret_rotation`, `update_check` are absent from `_SECTIONS`; the three multi-word names are additionally unreachable through `.partition("_")` | `MEFOR_SANDBOX_MODE=subprocess` in an NSSM environment block yields `off` — the ADR 0087 Router/Handler isolation boundary believed on and silently off | **No — CONFIRMED ON EVERY CLAIM 2026-08-15 (BACKLOG #1100), measured by AST.** `_SECTIONS` (**`settings.py:109`**) holds **23** entries; `ServiceSettings` has **28** annotated fields; the difference is **exactly 5** and exactly the five named — `sandbox`, `cert_monitor`, `secret_rotation`, `update_check`, `service`. The `.partition("_")` mechanism is at **`:3971`** (`section, _, key = name[len(_ENV_PREFIX):].lower().partition("_")`), so `MEFOR_CERT_MONITOR_*` parses as section `cert`, key `monitor_*` — the three multi-word names are unreachable a second way. **`MEFOR_SANDBOX` appears in ZERO tests.** And no test pins `_SECTIONS` against `ServiceSettings.model_fields`: five test files match the token, but **none imports `settings._SECTIONS`** — `test_verify.py`'s hit is `ALL_SECTIONS` from `verify.runner`, an unrelated constant (`host, store, smoke, manual, federation`), i.e. a substring collision rather than coverage. **So ADR 0087's Router/Handler isolation boundary can be set in an NSSM environment block, read back as `off`, and say nothing** | P0 | | Shift-left checks disable themselves on the likeliest-wrong input | `posture` / `build-check` / `reference-backend` return `ok=True, skipped=True` when `messagefoundry.toml` will not load (`checks.py:1189`, `:1266`, `:1415`) | An adopter with an ADR 0118 relocated key (a hard `ValueError`) gets fully green CI then a `serve` refusal at deploy | No test covers the settings-did-not-load skip path | P1 | | `_TRANSPORTS` vs the factory set drifts | A transport factory added to `wiring.py` without a `_TRANSPORTS` entry is invisible to config-as-data and the GUI form; a mis-mapped name produces a wrong `ConnectionSpec` | A silently unauthorable connector, or a byte-wrong spec that only shows up at runtime | No — `test_connection_schema.py` asserts the schema mirrors `_TRANSPORTS`, not that `_TRANSPORTS` mirrors the 19 factories | P1 | | Write CLIs carry no `--project-root` | `connection`/`codeset`/`alert`/`security`/`impact` resolve `env()` from `Path.cwd()` only (`__main__.py:3836`) | Under a CLI-only anchor or an IDE cwd off the repo root, validate-before-persist resolves a different `env()` view than `serve` — an upsert persists a connection `serve` will refuse at reload | No test exercises the asymmetry | P1 | From ef16a7bfb501f5de1014533247167c3ae84a6eb6 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 09:23:13 -0500 Subject: [PATCH 36/66] docs(testing): confirm the shipped gate runs in no CI leg, past a collision that says otherwise (BACKLOG #1100) 07-config-wiring-and-cli.md row 78 is CORRECT on all three claims. .git/hooks/pre-commit begins "File generated by pre-commit" -> .pre-commit-config.yaml owns it .mefor-hooks/pre-commit exists AND DIFFERS BYTE-FOR-BYTE from the installed hook, verified with cmp rather than inferred: generated, not installed no workflow invokes `messagefoundry check`: 0 files, against a positive control of 10 workflows invoking pytest THE COLLISION IS WORTH RECORDING BECAUSE ANYONE RE-CHECKING WILL HIT IT. Grepping for lines containing both "messagefoundry" and "check" returns SEVEN, which reads as "the gate is invoked seven times". Every one is `ruff check ... messagefoundry` -- ruff's subcommand sitting beside the package name -- plus one comment. The engine's own gate CLI appears nowhere in .github/. My own two measurements disagreed (0 files vs 7 lines) and that disagreement is the only reason I looked. A single measurement either way would have been confidently wrong in whichever direction I ran first. So the shipped config-quality gate runs in no CI leg: a change can demote build-check from required or break a sub-check while every unit slice still passes, and an adopter's gate exits 0 on a config `serve` will refuse. That is the fourth same-line-different-subject collision in this sweep, after the guard docstring quoting its own history, ALL_SECTIONS matching _SECTIONS, and `annotation` matching `from __future__ import annotations`. --- docs/testing/master-test-plan/07-config-wiring-and-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/07-config-wiring-and-cli.md b/docs/testing/master-test-plan/07-config-wiring-and-cli.md index 0942f432b..b22ad1db4 100644 --- a/docs/testing/master-test-plan/07-config-wiring-and-cli.md +++ b/docs/testing/master-test-plan/07-config-wiring-and-cli.md @@ -75,7 +75,7 @@ This chapter covers the **configuration bundle and the command-line surface that | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| The shipped gate runs in no engine CI leg | A change demotes `build-check` from required, or breaks a sub-check, and every unit slice still passes | Every adopter's whole config-quality gate exits 0 on a config `serve` will refuse; discovered at deploy or in production | No — `.pre-commit-config.yaml` owns `.git/hooks/pre-commit`; `.mefor-hooks/pre-commit` is generated but not installed; no workflow invokes `check` | P0 | +| The shipped gate runs in no engine CI leg | A change demotes `build-check` from required, or breaks a sub-check, and every unit slice still passes | Every adopter's whole config-quality gate exits 0 on a config `serve` will refuse; discovered at deploy or in production | **No — CONFIRMED ON ALL THREE CLAIMS 2026-08-15 (BACKLOG #1100).** `.git/hooks/pre-commit` begins *"File generated by pre-commit"*, so `.pre-commit-config.yaml` owns it. `.mefor-hooks/pre-commit` exists **and differs byte-for-byte from the installed hook** — generated, not installed, verified by `cmp` rather than by inference. And **no workflow invokes `messagefoundry check`**: 0 files, against a positive control of 10 workflows invoking `pytest`. **BEWARE THE COLLISION A RE-CHECK WILL HIT:** grepping for lines containing both `messagefoundry` and `check` returns **7**, and every one is `ruff check … messagefoundry` — ruff's subcommand beside the package name — plus one comment. The engine's own gate CLI appears nowhere. So the shipped config-quality gate runs in no CI leg, and an adopter's gate can exit 0 on a config `serve` will refuse | P0 | | No pin on the check roster | A check is dropped or its `required` flag flipped; exit-code contract changes silently | `build-check` is the only gate catching an ADR 0092 production-PHI cleartext hop pre-`serve`; `reference-backend` the only one catching a `Reference()` against SQL Server that raises post-ACK forever | No test enumerates `run_checks`' result names or flags | P0 | | Unknown settings sections/keys silently dropped | `[secrity]`, a misspelled `require_mfa`, or `allow_unencrypted_phi` in the wrong section loads clean and applies a default | An operator believes a posture switch is set; in the mis-sectioned case the **insecure** default applies with zero signal | **Partly — RE-MEASURED 2026-08-15 (BACKLOG #1100); one clause is FALSE and the surviving gap is SHARPER than written.** Confirmed: `extra="ignore"` holds via the `_Section` base (**`settings.py:173`**) and `ServiceSettings` (**`:3811`**, cited `:3623`) — the only `extra="forbid"` models are `EscalationTier` and `AlertRule`, nested rule objects, not sections. `tests/test_settings.py:118` (`test_unknown_sections_and_keys_ignored`) does pin the silence as intended; that anchor lands. **FALSE: "that warning itself is untested."** `tests/test_security_config.py` covers it with a positive AND a negative control — `test_unknown_security_key_warns_and_is_ignored` (`:386`) and `test_known_security_keys_warn_about_nothing` (`:401`, asserting "unrecognized key" is absent for valid keys). The warning is at **`:4060-4074`** (cited `:3880`, which is now `_cluster_requires_server_db`). **THE REAL GAP, WHICH THIS ROW'S OWN FIRST EXAMPLE FALLS INTO:** that warning diffs `set(raw) - set(SecuritySettings.model_fields)` — it catches an unknown **key inside a present `[security]` section**. A misspelled **section header** like `[secrity]` is dropped by `ServiceSettings`' `extra="ignore"` and **never reaches the check**, so it warns nothing. Mis-sectioning is the uncovered half, not mis-keying | P0 | | IDE Steps contract drift | `lens.py` changes `kind` / `params` / `literal_params` / `line_start` / `line_end`; both suites stay green against committed static fixtures | Steps view mis-renders or, worse, `lens rewrite` splices an edit into the wrong statement of a clinical transform — wrong data downstream | No — `ide/src/test/suite/steps.test.ts:28` reads `ide/src/test/fixtures/lens/*.json`; the `ide` CI job has no Python and nothing regenerates or diffs them | P0 | From 2969dc16988c1db9f0b7b1ee19783842256da665 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 09:34:02 -0500 Subject: [PATCH 37/66] docs(testing): the check SCHEMA is pinned, the ROSTER is not -- sharpen row 79 (BACKLOG #1100) 07-config-wiring-and-cli.md row 79 says "No test enumerates run_checks' result names or flags". CONFIRMED IN SUBSTANCE, but the wording understates existing coverage in a way that matters to whoever fixes it. WHAT IS PINNED, tightly: tests/test_checks.py:383-385 asserts set(report.keys()) == {"ok", "checks"} set(c.keys()) == {"name", "ok", "required", "skipped", "detail"} plus individual flags -- validate["required"] is True at :39 and :50, dryrun at :64. So a dropped or renamed FIELD fails immediately. WHAT IS NOT: WHICH CHECKS EXIST. run_checks (checks.py:114, 103 lines) assembles results by calling a series of _check_* functions plus tool runs, and nothing asserts that set. Drop a check, or flip its required flag, and every assertion above still passes. So the row's consequence holds exactly as written -- build-check is the only gate catching an ADR 0092 production-PHI cleartext hop pre-serve, and reference-backend the only one catching a Reference() against SQL Server that raises post-ACK forever. Either could vanish silently. I DELIBERATELY DID NOT STATE A COUNT. Three structural extractions of the roster returned 0 or partial results: names are not literals, not name= kwargs, and come from self.name on a per-check object. I have miscounted three times tonight by asserting a number from a partial extraction, and an unverified count is precisely what this sweep keeps finding rotted in other people's rows. THE SCHEMA-VS-ROSTER DISTINCTION IS THE FINDING AND IT NEEDS NO NUMBER. That also names the fix shape: the missing test is an ENUMERATION (the set of check names and their required flags), not another per-check assertion. The existing tests are the wrong kind, not too few. --- docs/testing/master-test-plan/07-config-wiring-and-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/07-config-wiring-and-cli.md b/docs/testing/master-test-plan/07-config-wiring-and-cli.md index b22ad1db4..3c8e0d485 100644 --- a/docs/testing/master-test-plan/07-config-wiring-and-cli.md +++ b/docs/testing/master-test-plan/07-config-wiring-and-cli.md @@ -76,7 +76,7 @@ This chapter covers the **configuration bundle and the command-line surface that | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | The shipped gate runs in no engine CI leg | A change demotes `build-check` from required, or breaks a sub-check, and every unit slice still passes | Every adopter's whole config-quality gate exits 0 on a config `serve` will refuse; discovered at deploy or in production | **No — CONFIRMED ON ALL THREE CLAIMS 2026-08-15 (BACKLOG #1100).** `.git/hooks/pre-commit` begins *"File generated by pre-commit"*, so `.pre-commit-config.yaml` owns it. `.mefor-hooks/pre-commit` exists **and differs byte-for-byte from the installed hook** — generated, not installed, verified by `cmp` rather than by inference. And **no workflow invokes `messagefoundry check`**: 0 files, against a positive control of 10 workflows invoking `pytest`. **BEWARE THE COLLISION A RE-CHECK WILL HIT:** grepping for lines containing both `messagefoundry` and `check` returns **7**, and every one is `ruff check … messagefoundry` — ruff's subcommand beside the package name — plus one comment. The engine's own gate CLI appears nowhere. So the shipped config-quality gate runs in no CI leg, and an adopter's gate can exit 0 on a config `serve` will refuse | P0 | -| No pin on the check roster | A check is dropped or its `required` flag flipped; exit-code contract changes silently | `build-check` is the only gate catching an ADR 0092 production-PHI cleartext hop pre-`serve`; `reference-backend` the only one catching a `Reference()` against SQL Server that raises post-ACK forever | No test enumerates `run_checks`' result names or flags | P0 | +| No pin on the check roster | A check is dropped or its `required` flag flipped; exit-code contract changes silently | `build-check` is the only gate catching an ADR 0092 production-PHI cleartext hop pre-`serve`; `reference-backend` the only one catching a `Reference()` against SQL Server that raises post-ACK forever | **CONFIRMED IN SUBSTANCE, SHARPENED 2026-08-15 (BACKLOG #1100): THE SCHEMA IS PINNED, THE ROSTER IS NOT — and the row's wording understates the first half.** `tests/test_checks.py:383-385` pins the result SHAPE tightly: `set(report.keys()) == {"ok","checks"}` and `set(c.keys()) == {"name","ok","required","skipped","detail"}`. Individual flags are asserted too (`validate["required"] is True` at `:39`/`:50`, `dryrun` at `:64`). **So a dropped FIELD is caught immediately.** What nothing asserts is WHICH CHECKS EXIST: `run_checks` (`checks.py:114`, 103 lines) assembles its results by calling a series of `_check_*` functions plus tool runs, and **no test enumerates that set**. Dropping a check, or flipping its `required`, leaves every assertion above still passing. **DELIBERATELY NOT STATING A COUNT HERE:** three separate structural extractions of the roster returned 0 or partial results, and an unverified count is exactly what this sweep keeps finding rotted. The schema-vs-roster distinction is the finding and needs no number | P0 | | Unknown settings sections/keys silently dropped | `[secrity]`, a misspelled `require_mfa`, or `allow_unencrypted_phi` in the wrong section loads clean and applies a default | An operator believes a posture switch is set; in the mis-sectioned case the **insecure** default applies with zero signal | **Partly — RE-MEASURED 2026-08-15 (BACKLOG #1100); one clause is FALSE and the surviving gap is SHARPER than written.** Confirmed: `extra="ignore"` holds via the `_Section` base (**`settings.py:173`**) and `ServiceSettings` (**`:3811`**, cited `:3623`) — the only `extra="forbid"` models are `EscalationTier` and `AlertRule`, nested rule objects, not sections. `tests/test_settings.py:118` (`test_unknown_sections_and_keys_ignored`) does pin the silence as intended; that anchor lands. **FALSE: "that warning itself is untested."** `tests/test_security_config.py` covers it with a positive AND a negative control — `test_unknown_security_key_warns_and_is_ignored` (`:386`) and `test_known_security_keys_warn_about_nothing` (`:401`, asserting "unrecognized key" is absent for valid keys). The warning is at **`:4060-4074`** (cited `:3880`, which is now `_cluster_requires_server_db`). **THE REAL GAP, WHICH THIS ROW'S OWN FIRST EXAMPLE FALLS INTO:** that warning diffs `set(raw) - set(SecuritySettings.model_fields)` — it catches an unknown **key inside a present `[security]` section**. A misspelled **section header** like `[secrity]` is dropped by `ServiceSettings`' `extra="ignore"` and **never reaches the check**, so it warns nothing. Mis-sectioning is the uncovered half, not mis-keying | P0 | | IDE Steps contract drift | `lens.py` changes `kind` / `params` / `literal_params` / `line_start` / `line_end`; both suites stay green against committed static fixtures | Steps view mis-renders or, worse, `lens rewrite` splices an edit into the wrong statement of a clinical transform — wrong data downstream | No — `ide/src/test/suite/steps.test.ts:28` reads `ide/src/test/fixtures/lens/*.json`; the `ide` CI job has no Python and nothing regenerates or diffs them | P0 | | 5 of 28 settings sections unreachable by `MEFOR_*` | `sandbox`, `service`, `cert_monitor`, `secret_rotation`, `update_check` are absent from `_SECTIONS`; the three multi-word names are additionally unreachable through `.partition("_")` | `MEFOR_SANDBOX_MODE=subprocess` in an NSSM environment block yields `off` — the ADR 0087 Router/Handler isolation boundary believed on and silently off | **No — CONFIRMED ON EVERY CLAIM 2026-08-15 (BACKLOG #1100), measured by AST.** `_SECTIONS` (**`settings.py:109`**) holds **23** entries; `ServiceSettings` has **28** annotated fields; the difference is **exactly 5** and exactly the five named — `sandbox`, `cert_monitor`, `secret_rotation`, `update_check`, `service`. The `.partition("_")` mechanism is at **`:3971`** (`section, _, key = name[len(_ENV_PREFIX):].lower().partition("_")`), so `MEFOR_CERT_MONITOR_*` parses as section `cert`, key `monitor_*` — the three multi-word names are unreachable a second way. **`MEFOR_SANDBOX` appears in ZERO tests.** And no test pins `_SECTIONS` against `ServiceSettings.model_fields`: five test files match the token, but **none imports `settings._SECTIONS`** — `test_verify.py`'s hit is `ALL_SECTIONS` from `verify.runner`, an unrelated constant (`host, store, smoke, manual, federation`), i.e. a substring collision rather than coverage. **So ADR 0087's Router/Handler isolation boundary can be set in an NSSM environment block, read back as `off`, and say nothing** | P0 | From 807a5e979be49be2f98e03df0f9943de173eeceb Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 09:41:06 -0500 Subject: [PATCH 38/66] docs(testing): the lens fixtures are unguarded, and the guard next door is why (BACKLOG #1100) 07-config-wiring-and-cli.md row 81 is CONFIRMED, and the reason it survived is worth more than the confirmation. steps.test.ts sets FIXTURE_DIR to src/test/fixtures/lens -- 7 fixtures present the ide CI job has 7 steps, NONE mentioning Python fixtures/lens/ is referenced by TYPESCRIPT ONLY; nothing regenerates or diffs it THE TRAP: tests/test_lens_schema.py::test_ide_fixture_in_sync (:101) DOES regenerate-and-diff a fixture, with the exact regeneration command in its failure text. IT GUARDS fixtures/lens-SCHEMA/op-schema.json -- A DIFFERENT DIRECTORY HOLDING ONE FILE. fixtures/lens-schema/ 1 file guarded from Python fixtures/lens/ 7 files not guarded at all A grep for "fixtures/lens" matches "fixtures/lens-schema" as a PREFIX and reports the gap as covered. I hit exactly that and only caught it by discriminating the two directories -- the fifth substring collision in this sweep, after ALL_SECTIONS matching _SECTIONS, `ruff check ... messagefoundry` matching the gate CLI, `annotation` matching a __future__ import, and a docstring quoting the claim it refutes. So a lens.py change to kind / params / literal_params / line_start / line_end leaves both suites green against seven stale committed fixtures, and the row's consequence stands: Steps view mis-renders, or `lens rewrite` splices an edit into the wrong statement of a clinical transform. AND IT IS THE SAME SHAPE AS ROW 79, ONE ROW EARLIER IN THIS CHAPTER: there the result SCHEMA is pinned and the ROSTER is not; here the schema fixture is guarded and the CONTENT fixtures are not. In both cases a real, working, well-built guard sits adjacent to the gap and is what makes it invisible. --- docs/testing/master-test-plan/07-config-wiring-and-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/07-config-wiring-and-cli.md b/docs/testing/master-test-plan/07-config-wiring-and-cli.md index 3c8e0d485..f35bd65e5 100644 --- a/docs/testing/master-test-plan/07-config-wiring-and-cli.md +++ b/docs/testing/master-test-plan/07-config-wiring-and-cli.md @@ -78,7 +78,7 @@ This chapter covers the **configuration bundle and the command-line surface that | The shipped gate runs in no engine CI leg | A change demotes `build-check` from required, or breaks a sub-check, and every unit slice still passes | Every adopter's whole config-quality gate exits 0 on a config `serve` will refuse; discovered at deploy or in production | **No — CONFIRMED ON ALL THREE CLAIMS 2026-08-15 (BACKLOG #1100).** `.git/hooks/pre-commit` begins *"File generated by pre-commit"*, so `.pre-commit-config.yaml` owns it. `.mefor-hooks/pre-commit` exists **and differs byte-for-byte from the installed hook** — generated, not installed, verified by `cmp` rather than by inference. And **no workflow invokes `messagefoundry check`**: 0 files, against a positive control of 10 workflows invoking `pytest`. **BEWARE THE COLLISION A RE-CHECK WILL HIT:** grepping for lines containing both `messagefoundry` and `check` returns **7**, and every one is `ruff check … messagefoundry` — ruff's subcommand beside the package name — plus one comment. The engine's own gate CLI appears nowhere. So the shipped config-quality gate runs in no CI leg, and an adopter's gate can exit 0 on a config `serve` will refuse | P0 | | No pin on the check roster | A check is dropped or its `required` flag flipped; exit-code contract changes silently | `build-check` is the only gate catching an ADR 0092 production-PHI cleartext hop pre-`serve`; `reference-backend` the only one catching a `Reference()` against SQL Server that raises post-ACK forever | **CONFIRMED IN SUBSTANCE, SHARPENED 2026-08-15 (BACKLOG #1100): THE SCHEMA IS PINNED, THE ROSTER IS NOT — and the row's wording understates the first half.** `tests/test_checks.py:383-385` pins the result SHAPE tightly: `set(report.keys()) == {"ok","checks"}` and `set(c.keys()) == {"name","ok","required","skipped","detail"}`. Individual flags are asserted too (`validate["required"] is True` at `:39`/`:50`, `dryrun` at `:64`). **So a dropped FIELD is caught immediately.** What nothing asserts is WHICH CHECKS EXIST: `run_checks` (`checks.py:114`, 103 lines) assembles its results by calling a series of `_check_*` functions plus tool runs, and **no test enumerates that set**. Dropping a check, or flipping its `required`, leaves every assertion above still passing. **DELIBERATELY NOT STATING A COUNT HERE:** three separate structural extractions of the roster returned 0 or partial results, and an unverified count is exactly what this sweep keeps finding rotted. The schema-vs-roster distinction is the finding and needs no number | P0 | | Unknown settings sections/keys silently dropped | `[secrity]`, a misspelled `require_mfa`, or `allow_unencrypted_phi` in the wrong section loads clean and applies a default | An operator believes a posture switch is set; in the mis-sectioned case the **insecure** default applies with zero signal | **Partly — RE-MEASURED 2026-08-15 (BACKLOG #1100); one clause is FALSE and the surviving gap is SHARPER than written.** Confirmed: `extra="ignore"` holds via the `_Section` base (**`settings.py:173`**) and `ServiceSettings` (**`:3811`**, cited `:3623`) — the only `extra="forbid"` models are `EscalationTier` and `AlertRule`, nested rule objects, not sections. `tests/test_settings.py:118` (`test_unknown_sections_and_keys_ignored`) does pin the silence as intended; that anchor lands. **FALSE: "that warning itself is untested."** `tests/test_security_config.py` covers it with a positive AND a negative control — `test_unknown_security_key_warns_and_is_ignored` (`:386`) and `test_known_security_keys_warn_about_nothing` (`:401`, asserting "unrecognized key" is absent for valid keys). The warning is at **`:4060-4074`** (cited `:3880`, which is now `_cluster_requires_server_db`). **THE REAL GAP, WHICH THIS ROW'S OWN FIRST EXAMPLE FALLS INTO:** that warning diffs `set(raw) - set(SecuritySettings.model_fields)` — it catches an unknown **key inside a present `[security]` section**. A misspelled **section header** like `[secrity]` is dropped by `ServiceSettings`' `extra="ignore"` and **never reaches the check**, so it warns nothing. Mis-sectioning is the uncovered half, not mis-keying | P0 | -| IDE Steps contract drift | `lens.py` changes `kind` / `params` / `literal_params` / `line_start` / `line_end`; both suites stay green against committed static fixtures | Steps view mis-renders or, worse, `lens rewrite` splices an edit into the wrong statement of a clinical transform — wrong data downstream | No — `ide/src/test/suite/steps.test.ts:28` reads `ide/src/test/fixtures/lens/*.json`; the `ide` CI job has no Python and nothing regenerates or diffs them | P0 | +| IDE Steps contract drift | `lens.py` changes `kind` / `params` / `literal_params` / `line_start` / `line_end`; both suites stay green against committed static fixtures | Steps view mis-renders or, worse, `lens rewrite` splices an edit into the wrong statement of a clinical transform — wrong data downstream | **No — CONFIRMED 2026-08-15 (BACKLOG #1100), and a DRIFT GUARD NEXT DOOR IS WHAT MAKES THE GAP INVISIBLE.** `steps.test.ts` sets `FIXTURE_DIR` to `src/test/fixtures/lens` (7 fixtures present), and the `ide` CI job has **7 steps, none mentioning Python**. **The trap:** `tests/test_lens_schema.py::test_ide_fixture_in_sync` (`:101`) *does* regenerate-and-diff a fixture — and it guards **`fixtures/lens-SCHEMA/op-schema.json`**, a **different directory holding ONE file**, with the exact regeneration command in its failure text. `fixtures/lens/` (the **seven** parse-result fixtures this row is about) is referenced by **TypeScript only**; no Python touches it and nothing regenerates or diffs it. A grep for `fixtures/lens` matches `fixtures/lens-schema` as a prefix and reports the gap as covered. **So the schema is drift-guarded and the CONTENT is not — the same shape as row 79's "schema pinned, roster not", one row apart in the same chapter** | P0 | | 5 of 28 settings sections unreachable by `MEFOR_*` | `sandbox`, `service`, `cert_monitor`, `secret_rotation`, `update_check` are absent from `_SECTIONS`; the three multi-word names are additionally unreachable through `.partition("_")` | `MEFOR_SANDBOX_MODE=subprocess` in an NSSM environment block yields `off` — the ADR 0087 Router/Handler isolation boundary believed on and silently off | **No — CONFIRMED ON EVERY CLAIM 2026-08-15 (BACKLOG #1100), measured by AST.** `_SECTIONS` (**`settings.py:109`**) holds **23** entries; `ServiceSettings` has **28** annotated fields; the difference is **exactly 5** and exactly the five named — `sandbox`, `cert_monitor`, `secret_rotation`, `update_check`, `service`. The `.partition("_")` mechanism is at **`:3971`** (`section, _, key = name[len(_ENV_PREFIX):].lower().partition("_")`), so `MEFOR_CERT_MONITOR_*` parses as section `cert`, key `monitor_*` — the three multi-word names are unreachable a second way. **`MEFOR_SANDBOX` appears in ZERO tests.** And no test pins `_SECTIONS` against `ServiceSettings.model_fields`: five test files match the token, but **none imports `settings._SECTIONS`** — `test_verify.py`'s hit is `ALL_SECTIONS` from `verify.runner`, an unrelated constant (`host, store, smoke, manual, federation`), i.e. a substring collision rather than coverage. **So ADR 0087's Router/Handler isolation boundary can be set in an NSSM environment block, read back as `off`, and say nothing** | P0 | | Shift-left checks disable themselves on the likeliest-wrong input | `posture` / `build-check` / `reference-backend` return `ok=True, skipped=True` when `messagefoundry.toml` will not load (`checks.py:1189`, `:1266`, `:1415`) | An adopter with an ADR 0118 relocated key (a hard `ValueError`) gets fully green CI then a `serve` refusal at deploy | No test covers the settings-did-not-load skip path | P1 | | `_TRANSPORTS` vs the factory set drifts | A transport factory added to `wiring.py` without a `_TRANSPORTS` entry is invisible to config-as-data and the GUI form; a mis-mapped name produces a wrong `ConnectionSpec` | A silently unauthorable connector, or a byte-wrong spec that only shows up at runtime | No — `test_connection_schema.py` asserts the schema mirrors `_TRANSPORTS`, not that `_TRANSPORTS` mirrors the 19 factories | P1 | From 5b14cd9f00e411e4da359eebe51de39349bd80ab Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:07:00 -0500 Subject: [PATCH 39/66] docs(test-plan): re-establish the ch06 blank-segment P0 by AST, not line number (BACKLOG #1100) The row's three wiring_runner.py anchors had all drifted ~230 lines: :3721 (cited as the peek.control_id read) is now blank, and :3732 (cited as summarize(peek)) is now record_received. A spot-check would have found nothing wrong and nothing right. The substance HOLDS, and is now stated in a form that survives the next move. Walking every ast.Try inside _handle_inbound (now 3950-4248) and testing line membership: control_id=peek.control_id (:4232) and summary=summarize(peek) (:4235) are both OUTSIDE every try body, as are all three enqueue_ingress calls. raise_if_blank_segment_scan is real -- parsing/peek.py:265. Citing 'outside every catch' by line number was the weaker argument for a claim that is structural. --- docs/testing/master-test-plan/06-parsing-and-codecs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/06-parsing-and-codecs.md b/docs/testing/master-test-plan/06-parsing-and-codecs.md index f7ac61846..a71372143 100644 --- a/docs/testing/master-test-plan/06-parsing-and-codecs.md +++ b/docs/testing/master-test-plan/06-parsing-and-codecs.md @@ -71,7 +71,7 @@ | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| Blank segment in any feed (i.e. any partner writing `\r\n\r\n`) | `Peek.field`, every routing property and `summarize()` raise a bare `IndexError` from `_builtin_hl7.raise_if_blank_segment_scan`. The reads at `wiring_runner.py:3721` (`peek.control_id`) and `:3732` (`summarize(peek)`) sit **outside every catch**, so it unwinds out of `_handle_inbound`; `transports/mllp.py` logs, emits `framing_error` and drops the TCP connection | Every message on that connection: 0 `messages` rows, 0 `queue` rows, no ACK/NAK. Direct count-and-log break **plus** a one-packet, infinitely repeatable per-connection DoS | **No — worse than no.** `tests/test_builtin_hl7_parity.py:183-189` (`adv:empty-fields`) *contains* a blank segment, and `_eq()` at `:74` compares exceptions **by type**, so both backends raising `IndexError` scores a PASS. `samples/messages/hapi-hl7v2/oml_o21.hl7` (26 interior blank lines) is in the globbed parity corpus and is certified this way | **P0** | +| Blank segment in any feed (i.e. any partner writing `\r\n\r\n`) | `Peek.field`, every routing property and `summarize()` raise a bare `IndexError` from `_builtin_hl7.raise_if_blank_segment_scan`. The reads sit **outside every catch**, so it unwinds out of `_handle_inbound`. **RE-VERIFIED STRUCTURALLY 2026-08-15 (BACKLOG #1100) — the citations had drifted ~230 lines, and the claim is re-established by AST rather than by line number so it survives the next move.** `_handle_inbound` is now **`wiring_runner.py:3950-4248`**; walking every `ast.Try` inside it and testing membership: **`control_id=peek.control_id` (`:4232`) and `summary=summarize(peek)` (`:4235`) are both OUTSIDE every `try` body**, as are all three `enqueue_ingress` calls (`:3977`, `:4077`, `:4229`). The raising mechanism is real — `_builtin_hl7.raise_if_blank_segment_scan`, called from `parsing/peek.py:265`. **The cited `:3721`/`:3732` now land on a blank line and on `record_received` respectively**, so a spot-check of this row would have found nothing wrong and nothing right; `transports/mllp.py` logs, emits `framing_error` and drops the TCP connection | Every message on that connection: 0 `messages` rows, 0 `queue` rows, no ACK/NAK. Direct count-and-log break **plus** a one-packet, infinitely repeatable per-connection DoS | **No — worse than no.** `tests/test_builtin_hl7_parity.py:183-189` (`adv:empty-fields`) *contains* a blank segment, and `_eq()` at `:74` compares exceptions **by type**, so both backends raising `IndexError` scores a PASS. `samples/messages/hapi-hl7v2/oml_o21.hl7` (26 interior blank lines) is in the globbed parity corpus and is certified this way | **P0** | | `\X00\` in any summarized field | Unescapes to a real U+0000 **inside `summarize()`** — after the `FCP:INGEST-4` post-decode NUL guard (`wiring_runner.py:3523`), which inspects only the raw decoded text (the raw carries the 5-char escape, no NUL). The NUL rides `summary=summarize(peek)` into `enqueue_ingress` at `:3732`, a **pre-ACK** commit. Reproduced: `'MRN AB\x00CD · DOE, JANE'` | Postgres rejects a NUL at bind (`DataError`/SQLSTATE 22021) → the raise unwinds out of `_handle_inbound` exactly as the ADR 0028 `FCP:INGEST-4` amendment describes, dropping the connection with **no ERROR row**. SQLite/SQL Server truncate the summary → operator blind spot in list/search | No. `grep -r 'X00' tests/` returns nothing. The whole `FCP:INGEST-4` retrofit was built for this class and missed the derived-value path | **P0** | | ADR 0054 Phase 2 drops python-hl7 (`pyproject.toml:48` still pins `hl7>=0.4.5`) | The dual-backend oracle structurally requires python-hl7 installed. On removal `test_builtin_hl7_parity.py` becomes vacuous or uncollectable | Every subsequent edit to a 991-line from-scratch reimplementation of python-hl7's tolerant semantics ships unguarded | No golden vectors exist. Already flagged as `FCP:PARSE-12` (row at `FEATURE-COVERAGE-PLAN.md:857`, recommendation at `:874`) | P1 | | HL7 version breadth is 2.5.1-only | Generators are hl7apy `v2_5_1`-driven (`generators/_core.py:24,142,495`); `hl7schema.py:22` and `hl7structures.py:31` are pinned to `SUPPORTED_VERSION = "2.5.1"`; no test drives `validate()` over a 2.3/2.3.1/2.4/2.6/2.7 body | Real hospital feeds are dominated by 2.3/2.4. An hl7apy upgrade could start NAKing an entire production feed synchronously with no failing test | No | P1 | From 2b7450d5cf8a22948e57b873d7abc1eb277e3c31 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:12:26 -0500 Subject: [PATCH 40/66] docs(test-plan): confirm the ch06 escaped-NUL P0 by reproduction, and name the false premise (BACKLOG #1100) Anchors had drifted ~500 lines (:3523 -> :4026, :3732 -> :4229/:4235). Substance CONFIRMED, and by a live run rather than a reading: body 'DOE^JANE\X00\SMITH' guard at :4026, '\x00' in text -> False summarize() -> 'MRN MRN123 \xb7 DOE, JANE\x00SMITH' So the NUL is MANUFACTURED after the control that promised there was none, and rides summary=summarize(peek) into a pre-ACK enqueue_ingress that the AST pass confirms is outside every try. The sharpest part is that the guard's OWN COMMENT (:4031-4033) is the false premise -- it claims summary is NUL-free 'for the rest of this handler', and summary is exactly the derived value for which that is untrue. SDS-3.7 in the engine's own words. Also corrected two things the row got wrong in MY favour: summarize lives in parsing/summary.py:27, not parsing/peek.py, and the Postgres SQLSTATE 22021 claim is now ATTRIBUTED to the engine's docstring at :410 rather than asserted -- a live Postgres reproduction is still owed and the row now says so. --- docs/testing/master-test-plan/06-parsing-and-codecs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/06-parsing-and-codecs.md b/docs/testing/master-test-plan/06-parsing-and-codecs.md index a71372143..2a7eb2a34 100644 --- a/docs/testing/master-test-plan/06-parsing-and-codecs.md +++ b/docs/testing/master-test-plan/06-parsing-and-codecs.md @@ -72,7 +72,7 @@ | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | Blank segment in any feed (i.e. any partner writing `\r\n\r\n`) | `Peek.field`, every routing property and `summarize()` raise a bare `IndexError` from `_builtin_hl7.raise_if_blank_segment_scan`. The reads sit **outside every catch**, so it unwinds out of `_handle_inbound`. **RE-VERIFIED STRUCTURALLY 2026-08-15 (BACKLOG #1100) — the citations had drifted ~230 lines, and the claim is re-established by AST rather than by line number so it survives the next move.** `_handle_inbound` is now **`wiring_runner.py:3950-4248`**; walking every `ast.Try` inside it and testing membership: **`control_id=peek.control_id` (`:4232`) and `summary=summarize(peek)` (`:4235`) are both OUTSIDE every `try` body**, as are all three `enqueue_ingress` calls (`:3977`, `:4077`, `:4229`). The raising mechanism is real — `_builtin_hl7.raise_if_blank_segment_scan`, called from `parsing/peek.py:265`. **The cited `:3721`/`:3732` now land on a blank line and on `record_received` respectively**, so a spot-check of this row would have found nothing wrong and nothing right; `transports/mllp.py` logs, emits `framing_error` and drops the TCP connection | Every message on that connection: 0 `messages` rows, 0 `queue` rows, no ACK/NAK. Direct count-and-log break **plus** a one-packet, infinitely repeatable per-connection DoS | **No — worse than no.** `tests/test_builtin_hl7_parity.py:183-189` (`adv:empty-fields`) *contains* a blank segment, and `_eq()` at `:74` compares exceptions **by type**, so both backends raising `IndexError` scores a PASS. `samples/messages/hapi-hl7v2/oml_o21.hl7` (26 interior blank lines) is in the globbed parity corpus and is certified this way | **P0** | -| `\X00\` in any summarized field | Unescapes to a real U+0000 **inside `summarize()`** — after the `FCP:INGEST-4` post-decode NUL guard (`wiring_runner.py:3523`), which inspects only the raw decoded text (the raw carries the 5-char escape, no NUL). The NUL rides `summary=summarize(peek)` into `enqueue_ingress` at `:3732`, a **pre-ACK** commit. Reproduced: `'MRN AB\x00CD · DOE, JANE'` | Postgres rejects a NUL at bind (`DataError`/SQLSTATE 22021) → the raise unwinds out of `_handle_inbound` exactly as the ADR 0028 `FCP:INGEST-4` amendment describes, dropping the connection with **no ERROR row**. SQLite/SQL Server truncate the summary → operator blind spot in list/search | No. `grep -r 'X00' tests/` returns nothing. The whole `FCP:INGEST-4` retrofit was built for this class and missed the derived-value path | **P0** | +| `\X00\` in any summarized field | Unescapes to a real U+0000 **inside `summarize()`** — after the `FCP:INGEST-4` post-decode NUL guard which inspects only the raw decoded text (the raw carries the 5-char escape, no NUL). **RE-VERIFIED AND RE-POINTED 2026-08-15 (BACKLOG #1100) — anchors had drifted ~500 lines; substance CONFIRMED BY LIVE REPRODUCTION, not by reading.** The guard is now `wiring_runner.py:4026` (`if "\x00" in text:`); the NUL rides `summary=summarize(peek)` (`:4235`) into `enqueue_ingress` (`:4229`), a **pre-ACK** commit that the same AST pass used for the row above confirms is **outside every `try`**. **The guard's OWN COMMENT is the false premise (SDS-3.7): `:4031-4033` states it dead-letters "BEFORE Peek.parse and any store write, so text (and control_id/summary/strict-fail errors derived from it) is NUL-free for the rest of this handler" — `summary` is precisely a derived value for which that is UNTRUE.** Re-run 2026-08-15 against `parsing/summary.py:27` (note: `summarize` lives there, NOT in `parsing/peek.py`): body containing `DOE^JANE\X00\SMITH` gives `"\x00" in text` = **False** at the guard, and `summarize()` then returns `'MRN MRN123 \xb7 DOE, JANE\x00SMITH'` — **a literal U+0000 manufactured after the control that promised there was none** | Postgres rejects a NUL at bind (`DataError`/SQLSTATE 22021) — **attributed, NOT independently tested here: this is the engine's own statement at `wiring_runner.py:410`, "Postgres REJECTS it at bind (the raise is uncaught)". A live Postgres reproduction is still owed** — the raise unwinds out of `_handle_inbound` exactly as the ADR 0028 `FCP:INGEST-4` amendment describes, dropping the connection with **no ERROR row**. SQLite/SQL Server truncate the summary → operator blind spot in list/search | No. `grep -r 'X00' tests/` returns nothing. The whole `FCP:INGEST-4` retrofit was built for this class and missed the derived-value path | **P0** | | ADR 0054 Phase 2 drops python-hl7 (`pyproject.toml:48` still pins `hl7>=0.4.5`) | The dual-backend oracle structurally requires python-hl7 installed. On removal `test_builtin_hl7_parity.py` becomes vacuous or uncollectable | Every subsequent edit to a 991-line from-scratch reimplementation of python-hl7's tolerant semantics ships unguarded | No golden vectors exist. Already flagged as `FCP:PARSE-12` (row at `FEATURE-COVERAGE-PLAN.md:857`, recommendation at `:874`) | P1 | | HL7 version breadth is 2.5.1-only | Generators are hl7apy `v2_5_1`-driven (`generators/_core.py:24,142,495`); `hl7schema.py:22` and `hl7structures.py:31` are pinned to `SUPPORTED_VERSION = "2.5.1"`; no test drives `validate()` over a 2.3/2.3.1/2.4/2.6/2.7 body | Real hospital feeds are dominated by 2.3/2.4. An hl7apy upgrade could start NAKing an entire production feed synchronously with no failing test | No | P1 | | Z-segment tolerance under strict validation is unpinned | Verified today: appending `ZPD\|1\|custom^data` to a conformant ADT^A01 still yields `ok=True` — but nothing asserts it | If an hl7apy upgrade tightens this, **every** message on **every** strict inbound NAKs AE synchronously — total feed outage — and the suite stays green | No | P1 | From 5636852c9e2e71b0b7a49d35324bc2890bbc6b81 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:15:31 -0500 Subject: [PATCH 41/66] docs(test-plan): ch14 -- an end-line-number sold as a length, and a chapter that did not drift (BACKLOG #1100) TrayApp is tray/app.py:42-154, which is 113 lines, not the '154 lines' the row claimed. 154 is the END LINE NUMBER relabelled as a LENGTH, overstating the untested surface by 36%. Same root cause as the three false enumerations I committed earlier this session: take the number a tool returned, then call it something narrower than what it measures. Everything else in ch14 verified EXACT, which is itself the finding: :450 Start SERVICE_AUTO_START :463-464 AppExit Default Restart + AppThrottle 5000 :472-477 the virtual-account default block :516-548 the S4 ACL block ('applied AFTER ... ObjectName is set') :81 Resolve-Nssm winshell.py:299-451 spans precisely the ten dispatch functions _add_icon through _message_loop, which ends at :451 Every wiring_runner.py anchor checked in ch02-10 had moved; none here did. So drift tracks the CHURN OF THE CITED FILE, not the age of the plan -- a re-sweep should re-point per file, not uniformly. Recorded in the row so the next sweeper does not redo this chapter. --- .../master-test-plan/14-tray-service-and-distribution.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/testing/master-test-plan/14-tray-service-and-distribution.md b/docs/testing/master-test-plan/14-tray-service-and-distribution.md index d6ac918d3..f872c68c2 100644 --- a/docs/testing/master-test-plan/14-tray-service-and-distribution.md +++ b/docs/testing/master-test-plan/14-tray-service-and-distribution.md @@ -103,14 +103,14 @@ icon state rendering and light/dark iconset (`theme.py`, `iconset.py`, 18 `.ico` | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| `install-service.ps1` privilege/ACL regression merges green | A `.ps1` edit drops the virtual-account default (`:472-477`), mis-orders the S4 ACL grants (`:516-548`), breaks `Resolve-Nssm`, or removes `AppExit Restart` | Every new install on every customer box; discovered at go-live | **No** — only `windows-service-smoke`, `(schedule \|\| workflow_dispatch) && github.repository == 'MEFORORG/MessageFoundry'` (`ci.yml:1087`); never on a PR, never in a fork/mirror. Static regexes only | **P0** | +| `install-service.ps1` privilege/ACL regression merges green | A `.ps1` edit drops the virtual-account default (`:472-477`), mis-orders the S4 ACL grants (`:516-548`), breaks `Resolve-Nssm`, or removes `AppExit Restart`. **ALL ANCHORS RE-VERIFIED EXACT 2026-08-15 (BACKLOG #1100)** against `install-service.ps1` (572 lines): `:472` is the `if (-not $ServiceAccount -and -not $AllowLocalSystem)` default and `:477` closes it; `:516` opens the S4 ACL block whose own comment reads "applied AFTER the service exists + ObjectName is set (S4 order)"; `Resolve-Nssm` is at `:81`; and for the row below, `:450` is `Start SERVICE_AUTO_START` with `:463-464` `AppExit Default Restart` + `AppThrottle 5000`. **Worth recording that this chapter did NOT drift** — every `wiring_runner.py` anchor checked in chapters 02-10 had moved, so drift tracks the CHURN OF THE CITED FILE, not the age of the plan, and a re-sweep should re-point by file rather than uniformly | Every new install on every customer box; discovered at go-live | **No** — only `windows-service-smoke`, `(schedule \|\| workflow_dispatch) && github.repository == 'MEFORORG/MessageFoundry'` (`ci.yml:1087`); never on a PR, never in a fork/mirror. Static regexes only | **P0** | | Silent regression to LocalSystem, or a world-readable log dir | Nothing queries `nssm get MessageFoundry ObjectName`, `Start`, or `icacls` on DataDir/logs/config after install | NSSM captures engine stdout to `C:\ProgramData\MessageFoundry\logs` — a PHI sink (`install-service.ps1:119-137` exists precisely for this). A lost `/inheritance:r` re-exposes it; a lost ObjectName widens compromise blast radius | **No** — `FCP:DEPLOY-8` / `FCP:DEPLOY-11` rated high-risk, partial/none in `docs/testing/FEATURE-COVERAGE-PLAN.md` §24 (rows at `:332`, `:334`) | **P0** | | Engine does not come back after host reboot or engine crash | `Start SERVICE_AUTO_START` (`:450`) or `AppExit Default Restart` + `AppThrottle` (`:463-464`) silently ineffective | A silent clinical outage: feeds stop, senders queue or drop, nothing alerts because the engine simply is not running | **No** — static regex (`test_service_control.py:118`) + a MANUAL row (`ACC:G1`, `harness/acceptance/matrix.py:430-437`; `W25:S2.5`) | **P0** | | The tray's shipped launch path is unverified end to end | A hatchling change drops `tray/assets`, or the `gui-scripts` entry (`pyproject.toml:202-203`) typos | `docs/TRAY.md:22-37` promises "present on every `pip install messagefoundry`". A tray that starts with no icon, or does not start, on every fresh install | **No** — `release.yml:127-180` installs the wheel on **Linux** and asserts only import/version/`py.typed`; `test_tray_iconset.py` reads `ASSETS_DIR` from `__file__` (the repo); `windows-service-smoke` installs `-e .` | **P1** | | ADR 0113 §1's import contract unenforced | A future edit imports `messagefoundry.config.settings` into the tray to "read TLS properly" | Pulls the engine + pydantic into an unprivileged, always-running process; breaches the boundary the whole ADR rests on | **No** — `tests/test_dependency_boundaries.py:14` lists only `pipeline/transports/parsing/store/config`; nothing scans `messagefoundry/tray/` | **P1** | | `tray.log` destroyed by httpx per-tick noise | `_setup_logging` attaches the handler to the **root** logger at INFO (`__main__.py:29-31`); nothing raises the `httpx` logger, which logs one INFO line per request — two requests every 5 s | ~34k lines/day churn a 1 MB × 3 rotation, so transitions, user actions and elevation outcomes rotate out within hours — destroying the only forensic record of who stopped a clinical interface. Contradicts `docs/TRAY.md:162` ("state transitions, never per-tick") | **No** | **P1** | | `poll_seconds` is inert | Parsed and clamped (`config.py:257-259`), documented (`config.py:359-360`, `docs/TRAY.md:89`), but `StatusPoller._run` (`poller.py:169`) always calls `next_poll_seconds()`, which returns the hardcoded `POLL_BASE_S = 5.0` (`state.py:24,166-172`) | An operator throttling a busy interface box sees no change and gets no signal the setting was ignored | **No** — tests assert only that the value lands in `TrayConfig` (`test_tray_config.py:126,236`) | **P1** | -| The tray's Win32 layer is behaviourally untested | `TrayApp` (154 lines) and the pump (`winshell.py:299-451`) decide which icon paints, whether a toast fires, whether Exit tears down, whether the icon is removed on `WM_DESTROY`/`WM_ENDSESSION` | A ghost icon after logoff, a frozen icon on a state change, an action routed to the wrong handler — the exact failure class ADR 0113 named as the design's main risk | **No** — `test_tray_shell.py` only asserts imports + non-zero struct sizes | **P0** — the design's *named* main risk cannot be priced below the install rows it sits beside; closed by TRAY-19, TRAY-20 and TRAY-22 | +| The tray's Win32 layer is behaviourally untested | `TrayApp` (**`tray/app.py:42-154`, i.e. 113 lines — CORRECTED 2026-08-15 (BACKLOG #1100); the row said "154 lines", which is the END LINE NUMBER relabelled as a LENGTH, overstating the surface by 36%**) and the pump (`winshell.py:299-451` — **anchor VERIFIED EXACT: that range is precisely the ten dispatch functions `_add_icon` through `_message_loop`, which ends at :451**) decide which icon paints, whether a toast fires, whether Exit tears down, whether the icon is removed on `WM_DESTROY`/`WM_ENDSESSION` | A ghost icon after logoff, a frozen icon on a state change, an action routed to the wrong handler — the exact failure class ADR 0113 named as the design's main risk | **No** — `test_tray_shell.py` only asserts imports + non-zero struct sizes | **P0** — the design's *named* main risk cannot be priced below the install rows it sits beside; closed by TRAY-19, TRAY-20 and TRAY-22 | | Autostart writes are untested | `set_autostart` / `is_autostart_enabled` (`autostart.py:34-64`) write and read the real `HKCU\…\Run` value; only the pure `launcher_command` helper is covered | A stale or wrong-interpreter command silently breaks Start-at-Login (no tray after reboot) or strands a value pointing at a deleted venv; the menu checkbox reads the same untested getter | **No** | **P1** | | Hardened box + non-admin operator = a lying tray | Under `-LockConfigDir`, a standard user cannot read the engine's settings TOML, so `service_toml_uses_tls` sees nothing, `build_engine_url` yields `http://`, the https probe fails, and a healthy engine renders `WEDGED`/`STOPPED` | The 2026-07-22 amendment's bug, resurfacing for the least-privileged user on the most-hardened box, with no on-screen path to the fix — a healthy clinical interface rendered as down, on the exact configuration the security guidance prescribes | **No** — the fail-soft-to-http behaviour is asserted as *correct* (`test_tray_config.py:274-287`); the operational consequence is untested and undocumented | **P0** — a tray that lies about a healthy engine is worse than no tray; closed by TRAY-27 (+ TRAY-43 on the box) | | Windows-only lock resolution failure | `pip install --require-hashes -r requirements.lock` — prescribed for Windows production at `docs/SERVICE.md:26-33` — runs only on ubuntu (`security.yml:71-77`) | A platform-marker gap, an sdist-only transitive, or a missing `win_amd64` wheel breaks the documented production install on the primary supported platform | **No** | **P1** | From 0d19b1f181be8b18743afe42da2ee6c10f979121 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:17:00 -0500 Subject: [PATCH 42/66] docs(test-plan): ch14 lying-tray confirmed end to end, and named as a three-state collapse (BACKLOG #1100) Every hop of the row's chain verified in code: _read_toml (tray/config.py:279-286) -- 'None on any read/parse failure. Never raises' -- catches (OSError, ValueError) at :284, and PermissionError IS an OSError, so -LockConfigDir yields None service_toml_uses_tls :212-213 'if not service_toml: return False' build_engine_url :188 renders http:// when tls is False The row described a symptom; the defect underneath is a THREE-STATE COLLAPSE. Parsed-and-no-TLS, file-absent, and file-present-but- UNREADABLE all become the same None and then the same False. The distinguishing information is destroyed at :284, so NO downstream fix is possible and the test has to drive the unreadable case specifically -- which is why 'a healthy engine renders WEDGED' is not a UI bug. Recorded the contrast rather than just the finding: seat-tick.ps1 handles the identical distinction correctly, with a roster-blind flag and a comment saying a roster that could not be read is not evidence that nobody is live. Same distinction, one place kept it and one folded it into the empty case. --- .../master-test-plan/14-tray-service-and-distribution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/14-tray-service-and-distribution.md b/docs/testing/master-test-plan/14-tray-service-and-distribution.md index f872c68c2..40aa7a9b6 100644 --- a/docs/testing/master-test-plan/14-tray-service-and-distribution.md +++ b/docs/testing/master-test-plan/14-tray-service-and-distribution.md @@ -112,7 +112,7 @@ icon state rendering and light/dark iconset (`theme.py`, `iconset.py`, 18 `.ico` | `poll_seconds` is inert | Parsed and clamped (`config.py:257-259`), documented (`config.py:359-360`, `docs/TRAY.md:89`), but `StatusPoller._run` (`poller.py:169`) always calls `next_poll_seconds()`, which returns the hardcoded `POLL_BASE_S = 5.0` (`state.py:24,166-172`) | An operator throttling a busy interface box sees no change and gets no signal the setting was ignored | **No** — tests assert only that the value lands in `TrayConfig` (`test_tray_config.py:126,236`) | **P1** | | The tray's Win32 layer is behaviourally untested | `TrayApp` (**`tray/app.py:42-154`, i.e. 113 lines — CORRECTED 2026-08-15 (BACKLOG #1100); the row said "154 lines", which is the END LINE NUMBER relabelled as a LENGTH, overstating the surface by 36%**) and the pump (`winshell.py:299-451` — **anchor VERIFIED EXACT: that range is precisely the ten dispatch functions `_add_icon` through `_message_loop`, which ends at :451**) decide which icon paints, whether a toast fires, whether Exit tears down, whether the icon is removed on `WM_DESTROY`/`WM_ENDSESSION` | A ghost icon after logoff, a frozen icon on a state change, an action routed to the wrong handler — the exact failure class ADR 0113 named as the design's main risk | **No** — `test_tray_shell.py` only asserts imports + non-zero struct sizes | **P0** — the design's *named* main risk cannot be priced below the install rows it sits beside; closed by TRAY-19, TRAY-20 and TRAY-22 | | Autostart writes are untested | `set_autostart` / `is_autostart_enabled` (`autostart.py:34-64`) write and read the real `HKCU\…\Run` value; only the pure `launcher_command` helper is covered | A stale or wrong-interpreter command silently breaks Start-at-Login (no tray after reboot) or strands a value pointing at a deleted venv; the menu checkbox reads the same untested getter | **No** | **P1** | -| Hardened box + non-admin operator = a lying tray | Under `-LockConfigDir`, a standard user cannot read the engine's settings TOML, so `service_toml_uses_tls` sees nothing, `build_engine_url` yields `http://`, the https probe fails, and a healthy engine renders `WEDGED`/`STOPPED` | The 2026-07-22 amendment's bug, resurfacing for the least-privileged user on the most-hardened box, with no on-screen path to the fix — a healthy clinical interface rendered as down, on the exact configuration the security guidance prescribes | **No** — the fail-soft-to-http behaviour is asserted as *correct* (`test_tray_config.py:274-287`); the operational consequence is untested and undocumented | **P0** — a tray that lies about a healthy engine is worse than no tray; closed by TRAY-27 (+ TRAY-43 on the box) | +| Hardened box + non-admin operator = a lying tray | Under `-LockConfigDir`, a standard user cannot read the engine's settings TOML, so `service_toml_uses_tls` sees nothing, `build_engine_url` yields `http://`, the https probe fails, and a healthy engine renders `WEDGED`/`STOPPED`. **CONFIRMED END-TO-END IN CODE 2026-08-15 (BACKLOG #1100)**, every hop: `_read_toml` (`tray/config.py:279-286`) is documented "``None`` on any read/parse failure. Never raises" and catches `(OSError, ValueError)` at `:284` — **`PermissionError` IS an `OSError`**, so the lock yields `None`; `service_toml_uses_tls` then hits `if not service_toml: return False` (`:212-213`); `build_engine_url` renders `f"{'https' if tls else 'http'}://..."` (`:188`) as **`http://`**. **The root defect is a THREE-STATE COLLAPSE: parsed-and-no-TLS, file-absent, and file-present-but-UNREADABLE all become the same `None`, then the same `False`. The distinguishing information is destroyed at `:284` and cannot be recovered by any caller**, so no downstream fix is possible and the test must drive the unreadable case specifically. Instructively, the fleet's own `seat-tick.ps1` handles the identical distinction correctly and says so in comment — a roster that could not be read is not evidence that nobody is live — carrying a separate roster-blind flag rather than folding it into the empty case | The 2026-07-22 amendment's bug, resurfacing for the least-privileged user on the most-hardened box, with no on-screen path to the fix — a healthy clinical interface rendered as down, on the exact configuration the security guidance prescribes | **No** — the fail-soft-to-http behaviour is asserted as *correct* (`test_tray_config.py:274-287`); the operational consequence is untested and undocumented | **P0** — a tray that lies about a healthy engine is worse than no tray; closed by TRAY-27 (+ TRAY-43 on the box) | | Windows-only lock resolution failure | `pip install --require-hashes -r requirements.lock` — prescribed for Windows production at `docs/SERVICE.md:26-33` — runs only on ubuntu (`security.yml:71-77`) | A platform-marker gap, an sdist-only transitive, or a missing `win_amd64` wheel breaks the documented production install on the primary supported platform | **No** | **P1** | | Uninstall leaves user-scope tray artifacts | `uninstall-service.ps1` touches only the service. Nothing removes the `HKCU\…\Run` value `MessageFoundryTray` (`autostart.py:19`), `%LOCALAPPDATA%\MessageFoundry` (tray.toml + tray.log), or `Scripts\MessageFoundryTray.exe` + the ~7 MB staged DLLs (`branding.py:280-290`) | After `pip uninstall messagefoundry` the Run key fires at every login and fails silently; a branded exe lingers indefinitely | **No** — no doc, no test, no mention in `docs/SERVICE.md` or `docs/TRAY.md` | **P1** | | AV/EDR quarantines the branded launcher | `branding.ensure_branded_launcher` copies an interpreter, rewrites its `RT_VERSION` and stages DLLs into `Scripts\` — textbook EDR heuristic bait | Design is fail-soft (tray runs unbranded), but an EDR that quarantines the *source* `pythonw.exe` can break the tray **and** the engine. `docs/ANTIVIRUS-FIREWALL.md` process-exclusion table lists `python.exe`, `messagefoundry.exe`, `nssm.exe` — no `pythonw.exe`, no `MessageFoundryTray.exe` | **No** | **P1** | From b6029d7396fbfa5fb58f865a5139428624b2d79a Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:19:05 -0500 Subject: [PATCH 43/66] docs(test-plan): WITHDRAW the ch17 'engine sharding has zero CI execution' P0 -- it is false (BACKLOG #1100) The most consequential correction of this sweep. The row asserted a TOTAL coverage gap over work that has DEDICATED CI STEPS BUILT TO COVER IT, and would have sent someone to build tests that already exist and already run. Refuted, two of three clauses: 'no workflow file contains shard' -- ci.yml has it 7+ times, incl. named steps at :1348 and :1551 and the path gate at :993. Positive control: 'test' matched 24/24 workflow files. 'named by no CI step, so they execute nowhere' -- all three are named (ci.yml:1371, :1372, :1571) and the steps set EXACTLY the variable the modules gate on (MEFOR_TEST_SQLSERVER: '1' against not os.getenv('MEFOR_TEST_SQLSERVER')), so the gate makes them RUN. The ci.yml comments say these steps were added to close this very gap, and tests/test_serverdb_ci_coverage.py now fails if a module-gated suite is added without being named -- a meta-gate against exactly this regression. The row was describing a state that had already been fixed. What survives is real but far narrower than P0: test_supervisor.py drives _FakeProcess (11 refs, docstring 'without launching real engine subprocesses'), no Popen/create_subprocess. Re-scoped the row to that: shard RECOVERY is covered on both server backends, SUPERVISOR PROCESS HANDLING against a real subprocess is not. Checked the surviving clause BEFORE withdrawing, so the correction does not overshoot from 'zero coverage' to 'fully covered'. --- docs/testing/master-test-plan/17-performance-and-scale.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/17-performance-and-scale.md b/docs/testing/master-test-plan/17-performance-and-scale.md index f8343c8a7..456f196d6 100644 --- a/docs/testing/master-test-plan/17-performance-and-scale.md +++ b/docs/testing/master-test-plan/17-performance-and-scale.md @@ -108,7 +108,7 @@ Nothing below re-tests any of that; the rows below attack the **verdict**, the * |---|---|---|---|---| | The `--load` sustainable-rate verdict has no filling / backlog-slope term | A phase whose in-flight backlog grows through the whole hold passes `zero_loss` + eventual drain and reads as sustained. `report.py:503 _run_slos` checks only zero-loss, drain seconds, dead letters, dup rate — no slope, no latency divergence | Every published `--load` ceiling; every adopter sizing run; a cutover sized 3–5.5x over capacity drops clinical messages | **No.** The gate exists only on the co-located shardcert path (`shardcert.py:1758`); `shardcert_ladder.py:1092` documents its own absence on the two-box path in capitals | **P0** | | `/stats` poller-zero contamination | `enginepoll.py:556 await_drain` declares drained on `backlog == 0 and queue_depth == 0 and in_pipeline == 0` plus two equal `read`/`written` samples. A zeroed or frozen `/stats` under exactly the overload the gate exists for satisfies all four | A saturating, lossy run reports PASS and drained | No staleness precondition anywhere | **P0** | -| Engine sharding has zero CI execution | No workflow file contains the string `shard`. `tests/test_supervisor.py` drives `_FakeProcess`, never a real subprocess. `tests/test_shard_recovery_sqlserver.py`, `test_shard_recovery_postgres.py`, `test_shard_cert_sqlserver.py` are `MEFOR_TEST_*`-gated **and** named by no CI step, so they execute nowhere | ADR 0073 ownership-scoped recovery and single-delivery-consumer-per-lane are what keep per-lane FIFO and at-least-once correct on the only built multicore topology. A regression yields duplicate delivery, cross-engine-shard FIFO inversion, or permanently stranded rows | No | **P0** | +| ~~Engine sharding has zero CI execution~~ **WITHDRAWN — FALSE. Corrected 2026-08-15 (BACKLOG #1100)** | **This row asserted a total coverage gap over work that has DEDICATED CI STEPS BUILT TO COVER IT, and it would have driven someone to build tests that already exist and already run.** Two of its three clauses are refuted: (1) "no workflow file contains the string `shard`" is **FALSE** — `ci.yml` carries it at least seven times, including the named steps "Run the engine-shard + statement-dispatch suites on real SQL Server" (`:1348`) and "Run the failover + engine-shard recovery suites on real Postgres" (`:1551`), plus `shard_recovery`/`shard_cert` in the changed-path gate at `:993` (positive control: `test` matched 24 of 24 workflow files, so the search was sound); (2) "named by no CI step, so they execute nowhere" is **FALSE** — all three are named at `ci.yml:1371`, `:1372`, `:1571`, and the steps set **exactly the variable the modules gate on** (`MEFOR_TEST_SQLSERVER: "1"` against `not os.getenv("MEFOR_TEST_SQLSERVER")`), so the gating makes them run rather than skip. The ci.yml comments record that these steps were ADDED to close this very gap ("before this step they were" dark), and `tests/test_serverdb_ci_coverage.py` now fails if a module-gated suite is added without being named — a meta-gate against this exact regression. **What SURVIVES, and it is real but far narrower than P0:** clause (3) holds — `tests/test_supervisor.py` uses `_FakeProcess` (11 references, docstring "without launching real engine subprocesses (Windows-safe)") with no `Popen`/`create_subprocess` anywhere, so the supervisor's process management is exercised only against a stand-in. **Re-scope to that: engine-shard RECOVERY is covered on both server backends; SUPERVISOR PROCESS HANDLING against a real subprocess is not** | ADR 0073 ownership-scoped recovery and single-delivery-consumer-per-lane are what keep per-lane FIFO and at-least-once correct on the only built multicore topology. A regression yields duplicate delivery, cross-engine-shard FIFO inversion, or permanently stranded rows | No | **P0** | | Published sizing contradicts the measured record and itself | `docs/SYSTEM-REQUIREMENTS.md:195-198` offers tiers to "~500 – low-thousands msg/s / ~40M+/day"; the publishable measured figure is ~72 ev/s (`FCP:SCALE-19`, FEATURE-COVERAGE-PLAN:286, 7.23x short). The same file says multi-process scale-out "**is built**" (:163) and "a **future direction, not built**" (:213, :234) | The document an adopter sizes hardware from. Order-of-magnitude over-claim + a self-contradiction about whether the scale-out path exists | No doc-vs-measurement guard exists; `tests/test_feature_map_claims.py` checks only the ASVS score, private-path links and superseded docs | **P0** | | The one published multi-process scale-out number was measured on a now-forbidden topology | `docs/benchmarks/TUNING-BASELINE.md:150` records the η ≈ 0.85 / E_core ≈ 42 msg/s **engine-sharding** result on the store line "**per-shard SQLite** (one store file per shard — **no shared DB**, no shared-DB commit contention by design)" (quoted verbatim from the source table), measured 2026-06-27. ADR 0063 (2026-07-01) and `sharding.py:81 require_unified_store` now **refuse** >1 engine shard on any non-server backend | The only published **engine-sharding** speedup — the shape adopters are told to multiply by their own `E_core` — is not reproducible with the shipped code, and it was measured with the shared-store contention deliberately absent | No | **P0** | | No throughput/latency regression gate anywhere | `.github/workflows/benchmark.yml` is `workflow_dispatch`-only, and each run wraps the harness in `set +e` … `set -e` so **even the harness's own SLO exit code is discarded**. Numbers are hand-transcribed into TUNING-BASELINE. `--baseline`/`--tolerance` exist (`report.py:753`) but are invoked by no workflow. `docs/CI-QUALITY.md` has no perf section | A 5x throughput or p99 regression from a claim-path, store or connector change merges green and is found on the next manual rig run | No | **P1** | From e703c85a0845c259b897d6d03acefce99b8c66df Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:21:02 -0500 Subject: [PATCH 44/66] docs(test-plan): ch17 sizing P0 withdrawn as stale; the benchmark-topology P0 confirmed exact (BACKLOG #1100) Row 112 WITHDRAWN. The document was FIXED after the plan was written: the '~500 - low-thousands msg/s / ~40M+/day' tier NO LONGER EXISTS. SYSTEM-REQUIREMENTS.md:216-219 now tops out at ~165 msg/s at 4 engine shards and discloses the topology inline. Anchors drifted too (:163 is blank; :195-198 is bench guidance). The alleged self-contradiction is unreproducible: 'future direction, not built' appears NOWHERE in docs/. Flagged the arithmetic even though it is moot -- ~500/~72 is 6.94x, not the 7.23x claimed, so the multiplier did not come from the two numbers the row set beside each other. Kept the row's last column, which is still true: no doc-vs-measurement guard exists, so nothing caught the over-claim while it was live and nothing will catch the next one. Row 113 CONFIRMED, anchor EXACT -- worth stating precisely because the two rows above it in the same chapter did not survive. TUNING-BASELINE .md:150 is verbatim the 'per-shard SQLite ... no shared DB' store line, with E_core ~42 / eta ~0.85 at :161-162. One qualification: both files now DISCLOSE the topology (:165-171 separates the portable shape from the non-portable absolute; SYSTEM-REQUIREMENTS:219 names it), so this is a reproducibility problem, not a hidden-caveat one. Three P0s examined in this chapter, three different verdicts. Grading them as a block would have been wrong in both directions. --- docs/testing/master-test-plan/17-performance-and-scale.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/testing/master-test-plan/17-performance-and-scale.md b/docs/testing/master-test-plan/17-performance-and-scale.md index 456f196d6..467a3a2b3 100644 --- a/docs/testing/master-test-plan/17-performance-and-scale.md +++ b/docs/testing/master-test-plan/17-performance-and-scale.md @@ -109,8 +109,8 @@ Nothing below re-tests any of that; the rows below attack the **verdict**, the * | The `--load` sustainable-rate verdict has no filling / backlog-slope term | A phase whose in-flight backlog grows through the whole hold passes `zero_loss` + eventual drain and reads as sustained. `report.py:503 _run_slos` checks only zero-loss, drain seconds, dead letters, dup rate — no slope, no latency divergence | Every published `--load` ceiling; every adopter sizing run; a cutover sized 3–5.5x over capacity drops clinical messages | **No.** The gate exists only on the co-located shardcert path (`shardcert.py:1758`); `shardcert_ladder.py:1092` documents its own absence on the two-box path in capitals | **P0** | | `/stats` poller-zero contamination | `enginepoll.py:556 await_drain` declares drained on `backlog == 0 and queue_depth == 0 and in_pipeline == 0` plus two equal `read`/`written` samples. A zeroed or frozen `/stats` under exactly the overload the gate exists for satisfies all four | A saturating, lossy run reports PASS and drained | No staleness precondition anywhere | **P0** | | ~~Engine sharding has zero CI execution~~ **WITHDRAWN — FALSE. Corrected 2026-08-15 (BACKLOG #1100)** | **This row asserted a total coverage gap over work that has DEDICATED CI STEPS BUILT TO COVER IT, and it would have driven someone to build tests that already exist and already run.** Two of its three clauses are refuted: (1) "no workflow file contains the string `shard`" is **FALSE** — `ci.yml` carries it at least seven times, including the named steps "Run the engine-shard + statement-dispatch suites on real SQL Server" (`:1348`) and "Run the failover + engine-shard recovery suites on real Postgres" (`:1551`), plus `shard_recovery`/`shard_cert` in the changed-path gate at `:993` (positive control: `test` matched 24 of 24 workflow files, so the search was sound); (2) "named by no CI step, so they execute nowhere" is **FALSE** — all three are named at `ci.yml:1371`, `:1372`, `:1571`, and the steps set **exactly the variable the modules gate on** (`MEFOR_TEST_SQLSERVER: "1"` against `not os.getenv("MEFOR_TEST_SQLSERVER")`), so the gating makes them run rather than skip. The ci.yml comments record that these steps were ADDED to close this very gap ("before this step they were" dark), and `tests/test_serverdb_ci_coverage.py` now fails if a module-gated suite is added without being named — a meta-gate against this exact regression. **What SURVIVES, and it is real but far narrower than P0:** clause (3) holds — `tests/test_supervisor.py` uses `_FakeProcess` (11 references, docstring "without launching real engine subprocesses (Windows-safe)") with no `Popen`/`create_subprocess` anywhere, so the supervisor's process management is exercised only against a stand-in. **Re-scope to that: engine-shard RECOVERY is covered on both server backends; SUPERVISOR PROCESS HANDLING against a real subprocess is not** | ADR 0073 ownership-scoped recovery and single-delivery-consumer-per-lane are what keep per-lane FIFO and at-least-once correct on the only built multicore topology. A regression yields duplicate delivery, cross-engine-shard FIFO inversion, or permanently stranded rows | No | **P0** | -| Published sizing contradicts the measured record and itself | `docs/SYSTEM-REQUIREMENTS.md:195-198` offers tiers to "~500 – low-thousands msg/s / ~40M+/day"; the publishable measured figure is ~72 ev/s (`FCP:SCALE-19`, FEATURE-COVERAGE-PLAN:286, 7.23x short). The same file says multi-process scale-out "**is built**" (:163) and "a **future direction, not built**" (:213, :234) | The document an adopter sizes hardware from. Order-of-magnitude over-claim + a self-contradiction about whether the scale-out path exists | No doc-vs-measurement guard exists; `tests/test_feature_map_claims.py` checks only the ASVS score, private-path links and superseded docs | **P0** | -| The one published multi-process scale-out number was measured on a now-forbidden topology | `docs/benchmarks/TUNING-BASELINE.md:150` records the η ≈ 0.85 / E_core ≈ 42 msg/s **engine-sharding** result on the store line "**per-shard SQLite** (one store file per shard — **no shared DB**, no shared-DB commit contention by design)" (quoted verbatim from the source table), measured 2026-06-27. ADR 0063 (2026-07-01) and `sharding.py:81 require_unified_store` now **refuse** >1 engine shard on any non-server backend | The only published **engine-sharding** speedup — the shape adopters are told to multiply by their own `E_core` — is not reproducible with the shipped code, and it was measured with the shared-store contention deliberately absent | No | **P0** | +| ~~Published sizing contradicts the measured record and itself~~ **WITHDRAWN — STALE. Corrected 2026-08-15 (BACKLOG #1100)** | **The document was FIXED after this row was written; the over-claim it describes is gone.** The "~500 – low-thousands msg/s / ~40M+/day" tier **no longer exists in `SYSTEM-REQUIREMENTS.md`**. The current tier table (`:216-219`) tops out at **~165 msg/s at 4 engine shards**, and `:219` now carries the caveat inline — "on per-shard SQLite — a topology that is not supported above one shard". The cited anchors have also drifted: `:163` is **blank**, and `:195-198` is bench/active-passive guidance, not tiers. The alleged self-contradiction cannot be reproduced either: **"future direction, not built" appears NOWHERE in `docs/`**, so there is no counterpart to the `is built` statements (which are real, at `:184`, `:187`, `:193`, `:297`). **The "7.23x short" arithmetic is therefore moot — but flagging it anyway, because ~500/~72 is 6.94x, not 7.23x, so the multiplier did not derive from the two numbers the row put beside it.** What is still worth keeping from this row is its LAST column, which remains true: no doc-vs-measurement guard exists, so nothing would have caught the over-claim while it was live, and nothing will catch the next one | The document an adopter sizes hardware from. Order-of-magnitude over-claim + a self-contradiction about whether the scale-out path exists | No doc-vs-measurement guard exists; `tests/test_feature_map_claims.py` checks only the ASVS score, private-path links and superseded docs | **P0** | +| The one published multi-process scale-out number was measured on a now-forbidden topology | `docs/benchmarks/TUNING-BASELINE.md:150` records the η ≈ 0.85 / E_core ≈ 42 msg/s **engine-sharding** result on the store line "**per-shard SQLite** (one store file per shard — **no shared DB**, no shared-DB commit contention by design)" (quoted verbatim from the source table), measured 2026-06-27. **RE-VERIFIED 2026-08-15 (BACKLOG #1100) — CONFIRMED, and the anchor is EXACT**, which is worth stating because the two rows above it in this same chapter were withdrawn as false and stale: `TUNING-BASELINE.md:150` is verbatim that store line, and `E_core ≈ 42 msg/s/shard` / `η ≈ 0.85` sit at `:161-162`. **One qualification to the row's framing: the topology is now DISCLOSED in both published files, so this is a reproducibility problem rather than a hidden-caveat one.** `TUNING-BASELINE.md:165-171` already separates the non-portable absolute from the portable "speedup shape", and `SYSTEM-REQUIREMENTS.md:219` names the unsupported topology inline. ADR 0063 (2026-07-01) and `sharding.py:81 require_unified_store` now **refuse** >1 engine shard on any non-server backend | The only published **engine-sharding** speedup — the shape adopters are told to multiply by their own `E_core` — is not reproducible with the shipped code, and it was measured with the shared-store contention deliberately absent | No | **P0** | | No throughput/latency regression gate anywhere | `.github/workflows/benchmark.yml` is `workflow_dispatch`-only, and each run wraps the harness in `set +e` … `set -e` so **even the harness's own SLO exit code is discarded**. Numbers are hand-transcribed into TUNING-BASELINE. `--baseline`/`--tolerance` exist (`report.py:753`) but are invoked by no workflow. `docs/CI-QUALITY.md` has no perf section | A 5x throughput or p99 regression from a claim-path, store or connector change merges green and is found on the next manual rig run | No | **P1** | | The published "reference performance floor" is enforced nowhere | TUNING-BASELINE:29 states the floor "≥ 200 msg/s sustained · ACK p99 ≤ 50 ms · e2e p99 ≤ 5 s". `harness/load/profiles/reference.toml` `[load.slo]` carries only `max_error_rate`, `max_dead_letters`, `zero_loss` — deliberately, but nothing else asserts the floor either | A release can clear the two-tier gate's performance tier by assertion rather than measurement | No | **P1** | | No "harness-was-the-limit" boolean in the report | `deferred_backpressure` / `deferred_schedule` exist (`metrics.py:199-202`, written at `governor.py:51,58,86,123`) but `PhaseReport` and `_counters_dict` (`report.py:726`) emit only the total. Only shardcert has the attribution (`RungFidelity.DRIVE_SHORTFALL`) | `docs/LOAD-TESTING.md:296` and `W25:S4.2` (:755) / `W25:S4.7` (:907) / the validity gate (:1042) all require this flag FALSE before a ceiling counts — as written those acceptance steps are **unexecutable**, and a sender-bound number can be published as an engine ceiling | No | **P1** | From aef47f96a4de5b0ca8bae55604c0daa2e81c46ef Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:24:44 -0500 Subject: [PATCH 45/66] docs(test-plan): ch18 -- two P0s confirmed, and a composition neither row stated (BACKLOG #1100) Row 92 CONFIRMED verbatim. The webconsole pyproject dependency really is a bare 'messagefoundry', and the comment beside it reads: 'A PEP 508 compat RANGE is ideal (messagefoundry>=X,` and never mention the console wheel | Fails loud at startup — but nothing tests the *sequence*, and nothing warns beforehand | **P0** | -| The documented install-time seam defence does not exist | WEBCONSOLE-PACKAGE.md §2 claims a three-layer handshake whose layer 1 is "the PEP 508 range on the engine dependency fails an out-of-range pair at `pip`/`uv` resolve". `packaging/messagefoundry-webconsole/pyproject.toml:38-42` declares a **bare `"messagefoundry"`** with a comment conceding "a bare dep is fine for now" | `pip install -U messagefoundry` resolves happily across a seam bump. The first symptom is the P0 above. A documented control that is absent is worse than a known-absent one: the runbook author trusted it | **No** — nothing binds the doc claim to the metadata | **P0** | +| The documented install-time seam defence does not exist | WEBCONSOLE-PACKAGE.md §2 claims a three-layer handshake whose layer 1 is "the PEP 508 range on the engine dependency fails an out-of-range pair at `pip`/`uv` resolve". `packaging/messagefoundry-webconsole/pyproject.toml:38-42` declares a **bare `"messagefoundry"`** with a comment conceding "a bare dep is fine for now". **CONFIRMED VERBATIM 2026-08-15 (BACKLOG #1100), anchor exact**: the dependency list is exactly `"messagefoundry"`, and the comment reads in full "A PEP 508 compat RANGE is ideal (`messagefoundry>=X, Date: Sat, 15 Aug 2026 10:26:52 -0500 Subject: [PATCH 46/66] docs(test-plan): ch18 PHI row traced to the shared helper; a crash-on-output defect found beside it (BACKLOG #1100) Row 93 CONFIRMED, anchors exact -- but the line that actually leaks is NEITHER of the two cited. Both renderers get their values from one shared helper, Difference.describe() at normalize.py:102-104, whose entire body is: return f'{self.kind} @ {loc}: left={self.left!r} right={self.right!r}' The !r on a raw HL7 field value IS the PHI, and because it is one helper feeding both paths the fix belongs there, not twice in report.py. render_text's default really is max_diffs=20 (:50) and render_json really does emit left/right unconditionally (:35-36). SEPARATE DEFECT FOUND WHILE VERIFYING, recorded nowhere else: report.py writes non-ASCII glyphs to operator stdout at :63 (U+2717), :67 (U+2026), :72 (U+2713/U+2717). On a stock Windows cp1252 console that raises UnicodeEncodeError -- on a WINDOWS-TARGETED product. So the reconcile report can crash WHILE PRINTING rather than print wrongly, and no assertion about report CONTENT would ever catch it. Row 91 anchors re-pointed: settings.py :672 -> :710, api/app.py :4951 -> :5166 (~215 lines). Worth noting why that one is nasty: :4951 now lands on an unrelated DR_OPERATE dependency, so the stale anchor reads as plausible code rather than as an obvious miss. --- docs/testing/master-test-plan/18-interop-migration-and-uat.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/testing/master-test-plan/18-interop-migration-and-uat.md b/docs/testing/master-test-plan/18-interop-migration-and-uat.md index 636014fbf..982107522 100644 --- a/docs/testing/master-test-plan/18-interop-migration-and-uat.md +++ b/docs/testing/master-test-plan/18-interop-migration-and-uat.md @@ -88,9 +88,9 @@ This chapter covers the paths an organisation actually walks to *adopt*, *cut ov | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | Greenfield-only rule enforced nowhere | An operator flips `[store].type` to `postgres`/`sqlserver`; `open_store` ([`store/base.py:1739`](../../../messagefoundry/store/base.py)) selects the backend with **no prior-backend detection, no drain check, no warning**. The engine starts clean on an empty schema | The populated SQLite store — in-flight ingress/routed/outbound rows plus all retained PHI history and audit — is silently orphaned. Named as "Greenfield-only data loss" at `docs/releases/v0.1-PLAN.md:457`; the only shipped mitigation is prose in 6 documents and a manual checkbox at `docs/DEPLOY-SERVER-DB.md:274` | **No** | **P0** | -| Engine upgrade past a seam bump **bricks the whole engine**, not just `/ui` | `[api].serve_ui` defaults **True** ([`settings.py:672`](../../../messagefoundry/config/settings.py)). `create_app`'s `serve_ui` tail calls `assert_engine_seam(ENGINE_UI_SEAM)` ([`api/app.py:4951`](../../../messagefoundry/api/app.py)) and lets `UiSeamMismatch` propagate out of `create_app`. `SUPPORTED_ENGINE_SEAMS == frozenset({15})` — strict lockstep, no N-1 | `messagefoundry serve` **fails to start** on a PHI box mid-upgrade. Intake stops. The runbooks operators follow (EARLY-ADOPTER-GUIDE §13 step 4 :679, INSTALL-GUIDE §10 :306-313) bump only `messagefoundry==` and never mention the console wheel | Fails loud at startup — but nothing tests the *sequence*, and nothing warns beforehand | **P0** | +| Engine upgrade past a seam bump **bricks the whole engine**, not just `/ui` | `[api].serve_ui` defaults **True** (**RE-POINTED 2026-08-15, BACKLOG #1100: `settings.py:710`, was cited as `:672`** — [`settings.py`](../../../messagefoundry/config/settings.py)). `create_app`'s `serve_ui` tail calls `assert_engine_seam(ENGINE_UI_SEAM)` (**RE-POINTED: `api/app.py:5166`, was cited as `:4951` — a ~215-line drift, and `:4951` now lands on an unrelated `DR_OPERATE` dependency, so the stale anchor reads as plausible code rather than as an obvious miss**; the import sits at `:5149` — [`api/app.py`](../../../messagefoundry/api/app.py)) and lets `UiSeamMismatch` propagate out of `create_app`. `SUPPORTED_ENGINE_SEAMS == frozenset({15})` — strict lockstep, no N-1 | `messagefoundry serve` **fails to start** on a PHI box mid-upgrade. Intake stops. The runbooks operators follow (EARLY-ADOPTER-GUIDE §13 step 4 :679, INSTALL-GUIDE §10 :306-313) bump only `messagefoundry==` and never mention the console wheel | Fails loud at startup — but nothing tests the *sequence*, and nothing warns beforehand | **P0** | | The documented install-time seam defence does not exist | WEBCONSOLE-PACKAGE.md §2 claims a three-layer handshake whose layer 1 is "the PEP 508 range on the engine dependency fails an out-of-range pair at `pip`/`uv` resolve". `packaging/messagefoundry-webconsole/pyproject.toml:38-42` declares a **bare `"messagefoundry"`** with a comment conceding "a bare dep is fine for now". **CONFIRMED VERBATIM 2026-08-15 (BACKLOG #1100), anchor exact**: the dependency list is exactly `"messagefoundry"`, and the comment reads in full "A PEP 508 compat RANGE is ideal (`messagefoundry>=X, Date: Sat, 15 Aug 2026 10:28:38 -0500 Subject: [PATCH 47/66] docs(test-plan): ch11 -- both P0s confirmed, and the app.js one is worse than stated (BACKLOG #1100) Row 97 CONFIRMED, and the confirmation is sharper than the claim. No JavaScript-execution tooling exists anywhere: playwright, jsdom, selenium, puppeteer, node --test all return zero across tests/, packaging/ and .github/workflows/ (positive control: 173 test files match 'console', so the probe was sound). But EIGHT test files DO reference app.js -- csp_canary, hardening, logout_affordance, mfa_gate, session_watchdog, static_allowlist, webui, webconsole_mount -- all asserting on the SERVED TEXT. Those eight are exactly the 'grep-matched strings' the row warns about, and ONE OF THEM IS NAMED FOR THE WATCHDOG ITSELF. So a coverage review that counts files touching app.js finds eight and concludes the surface is tested. Asserted-as-text versus executed is the distinction that matters, and no count distinguishes them. Row 99 CONFIRMED, all three anchors exact -- each range ends precisely on its 'Approval id:' line (:105, :212, :685), under 'Reload/Purge/ Replay held for approval'. Three independently-written dead-ends with identical shape, which is what makes it a design gap and not three oversights. Recorded a navigation note because it cost me time: the module tree is messagefoundry_webconsole/pages/ at the REPO ROOT; packaging/ messagefoundry-webconsole/ holds only metadata and tests. --- docs/testing/master-test-plan/11-web-console.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/testing/master-test-plan/11-web-console.md b/docs/testing/master-test-plan/11-web-console.md index a936edd5b..e56a6b55e 100644 --- a/docs/testing/master-test-plan/11-web-console.md +++ b/docs/testing/master-test-plan/11-web-console.md @@ -94,9 +94,9 @@ that audit left open (FCP:UI-8, FCP:UI-23, FCP:UI-32) or that this pass newly fo | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| `app.js` is never executed by any test | A refactor preserving the grep-matched strings but breaking execution (a thrown init, a changed selector, a bad `await`) silently disables the 14.3.1 watchdog | Rendered PHI stays on an abandoned or terminated tab indefinitely; both WebAuthn ceremonies dead; live fragment stops updating | **No.** `test_ui_session_watchdog.py:396-486` reads `app.js` as text and compares `index()` positions. CodeQL SAST only. | **P0** | +| `app.js` is never executed by any test | A refactor preserving the grep-matched strings but breaking execution (a thrown init, a changed selector, a bad `await`) silently disables the 14.3.1 watchdog. **CONFIRMED 2026-08-15 (BACKLOG #1100), and the confirmation is sharper than the claim.** There is **no JavaScript-execution tooling anywhere in the repo** — `playwright`, `jsdom`, `selenium`, `puppeteer` and `node --test` return zero across `tests/`, `packaging/` and `.github/workflows/` (positive control: 173 test files match "console", so the probe was sound). **But EIGHT test files DO reference `app.js`** — `test_ui_csp_canary`, `test_ui_hardening`, `test_ui_logout_affordance`, `test_ui_mfa_gate`, `test_ui_session_watchdog`, `test_ui_static_allowlist`, `test_webui`, `test_webconsole_mount` — all asserting on the SERVED TEXT. **Those eight are exactly the "grep-matched strings" this row warns about, including one named for the watchdog itself**, so a coverage review that counts files touching `app.js` finds eight and concludes the surface is tested. The distinction that matters is asserted-as-text versus executed, and no count distinguishes them | Rendered PHI stays on an abandoned or terminated tab indefinitely; both WebAuthn ceremonies dead; live fragment stops updating | **No.** `test_ui_session_watchdog.py:396-486` reads `app.js` as text and compares `index()` positions. CodeQL SAST only. | **P0** | | Never served through real uvicorn + TLS + a browser cookie jar | Browser silently **rejects** the `__Host-`/`Secure` cookie, or the nonce CSP is not actually enforced | Login broken, or a script-injection control believed present is absent — first discovered by the first customer to expose `/ui` off-box | **No.** Everything is `httpx.ASGITransport` with a simulated scheme (`test_ui_hardening.py:34-37`). | **P0** | -| Dual-control operations initiated from `/ui` cannot be approved from `/ui` | Config reload / purge / bulk dead-letter replay held for approval renders "Approval id: X" and dead-ends (`pages/config.py:96-105`, `pages/connections.py:207-212`, `pages/messages.py:676-685`) | Under incident pressure the second approver must drop to the JSON API — or the org disables dual-control, defeating the control via its own UX | **No.** No `/ui/approvals` route in the golden table; no approvals field in the seam snapshot. Engine has `GET /approvals` + approve/reject at `api/app.py:2692-2721`. | **P0** | +| Dual-control operations initiated from `/ui` cannot be approved from `/ui` | Config reload / purge / bulk dead-letter replay held for approval renders "Approval id: X" and dead-ends (`pages/config.py:96-105`, `pages/connections.py:207-212`, `pages/messages.py:676-685`) — **ALL THREE RE-VERIFIED EXACT 2026-08-15 (BACKLOG #1100)**, each range ending precisely on its `el("p", text(f"Approval id: {approval_id}"), class_="muted")` line (`:105`, `:212`, `:685`), under headings "Reload held for approval", "Purge held for approval" and "Replay held for approval". **Three independently-written dead-ends with identical shape, which is what makes this a design gap rather than three oversights.** *Navigation note for the next reader, since it cost time here: the module tree is `messagefoundry_webconsole/pages/` at the REPO ROOT — `packaging/messagefoundry-webconsole/` holds only the packaging metadata and its tests* | Under incident pressure the second approver must drop to the JSON API — or the org disables dual-control, defeating the control via its own UX | **No.** No `/ui/approvals` route in the golden table; no approvals field in the seam snapshot. Engine has `GET /approvals` + approve/reject at `api/app.py:2692-2721`. | **P0** | | Three `app.js` features fetch `/ui` routes that do not exist | `/ui/logging/level` (`app.js:1259`), `/ui/logs/tail` (`:1308`), `/ui/messages/export` (`:1385-1386`, fetch at `:1412`) — no route, and no page builder emits `data-mf-log-level` / `data-mf-log-viewer` / `data-mf-msg-export` | ADR 0130/0131 and FEATURE-MAP present shipped capabilities that are unreachable; a PHI-egress feature is documented shipped with its audit/step-up path never exercised | **No.** Nothing in CI resolves a `fetch("/ui/...")` literal against the route table. | **P1** | | No pagination controls anywhere | Message log and dead-letters accept `limit`/`offset` (`routes/core.py:410-411`, `:519-520`) but render a text readout only (`pages/messages.py:113-117`, `:601-605`); audit is hard-coded `limit=200` with no offset or filter (`routes/audit.py:30,38`); search takes `limit` with no `offset` at all (`routes/search.py:66`) | An incident investigator sees the newest 50 messages / 200 audit rows and cannot page back **in the console at all** — a blind spot on the tamper-evident trail a HIPAA investigation depends on | **No.** FEATURE-COVERAGE-PLAN rows FCP:UI-8 and FCP:UI-23 are open. | **P1** | | No render-at-scale test, no performance budget | The connections fragment is re-rendered **server-side** and pushed on every `/ws/stats` tick to every connected operator (`api/app.py:4850-4852`, `pages.connections_fragment`) | A linear render over hundreds of Connections × N operators contends with the same event loop that runs the pipeline — presents as a *delivery* problem | **No.** `docs/LOAD-TESTING.md` covers pipeline throughput only. | **P1** | From 23df07aaed221272e8c274d9d3aa6cf984710323 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:31:39 -0500 Subject: [PATCH 48/66] docs(test-plan): ch15 G2 and G3 confirmed; G3 anchor off by one, and the asymmetry is the defect (BACKLOG #1100) G2 CONFIRMED, all four anchors exact. db_lookup and fhir_lookup are imported at :32/:33 and listed in __all__ at :150/:153; content_match appears NOWHERE in messagefoundry/__init__.py. This row is trustworthy precisely because it carries its own positive control -- the same search that returns zero for content_match finds both comparators, so the zero is a fact about the code and not about the probe. G3 CONFIRMED, anchor re-pointed :100 -> :101 (the cited line holds the sibling 'connection_restored': 'connection_error' entry). Ran the absence claim with a control, and the control is what makes it a finding rather than a failed grep: 'connection_started' occurs in exactly ONE place in messagefoundry/ -- the _AUTO_RESOLVE mapping -- while the same search finds 'connection_stopped' genuinely emitted at alert_sinks.py:682. So the asymmetry IS the defect: one half of an auto-resolve PAIR can never fire. connection_stopped raises and connection_started only ever resolves, so a lane that stops on an internal error and is restarted never clears. An alert that is raisable but not resolvable is worse than one never wired, because the operator sees a live alert against a healthy lane and learns to distrust the panel. --- .../testing/master-test-plan/15-alerting-and-observability.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/testing/master-test-plan/15-alerting-and-observability.md b/docs/testing/master-test-plan/15-alerting-and-observability.md index 60e92fe7a..f36e0b9c3 100644 --- a/docs/testing/master-test-plan/15-alerting-and-observability.md +++ b/docs/testing/master-test-plan/15-alerting-and-observability.md @@ -133,8 +133,8 @@ ALERT-13 and ALERT-11. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | **G1 — three alert vocabularies bound only by convention** | An emit site's `type` literal, `_ALERT_EVENT_TYPES`, and the `AlertSink` Protocol / `LoggingAlertSink` / `NotifierAlertSink` method sets drift. A type absent from the routable set can never be escalated, routed, suppressed or muted by any operator rule; a method present only on `NotifierAlertSink` raises `AttributeError` on the logging fallback | Every deployment. Drift has fired **twice** already: `lane_stuck` + `rcsi_off_degraded` once shipped unroutable — **both are members today** ([`settings.py:2515-2516`](../../../messagefoundry/config/settings.py)), rule-targetable end to end at `test_alert_rules.py:357/370/395`, so the only live drift is that no guard stops the next one; `content_match` is **still** divergent (only on `NotifierAlertSink` at [`alert_sinks.py:676`](../../../messagefoundry/pipeline/alert_sinks.py), absent from the Protocol and the fallback). Note the non-obvious third form: `AlertSink.saturation_rising()` emits type `"saturation"` — method name ≠ event type, so a naive guard would false-fail | **No.** No guard exists in either direction | **P0** | -| **G2 — `content_match` has no Handler-reachable surface** | ADR 0133 AC-3 says "WHEN a Handler emits a `content_match`" — but there is no export in `messagefoundry/__init__.py` (unlike `db_lookup`/`fhir_lookup` at lines 32-33/150/153), no injected sink on a Handler context, and no dry-run path. Every test calls `sink.content_match(...)` directly | The differentiating Corepoint "Action Point" parity capability is unusable in practice; the PHI-free-by-contract guarantee has never been exercised through real Handler code | **No** — AC-3/AC-4 pass at the sink so nothing fails | **P0** | -| **G3 — `connection_started` is mapped but emitted nowhere** | `_AUTO_RESOLVE["connection_started"] = "connection_stopped"` ([`alert_sinks.py:100`](../../../messagefoundry/pipeline/alert_sinks.py)) but a repo-wide search finds **no emit site**. A lane that STOPs on an internal error and is later restarted (by hand or by a #144 `control_action`) leaves its `alert_instance` permanently `open` | `alerts_active` on the connections dashboard stays non-zero forever; `/alerts/active` accumulates; the nav bell's `list_active_alerts(limit=200)` ([`webconsole/routes/status.py:166`](../../../messagefoundry_webconsole/routes/status.py)) saturates. Textbook alert fatigue — the operator learns to ignore the list and the next real stop is missed | **Partially** — `tests/test_phi_logging_inventory.py:439` *documents* the dead key; nothing asserts the operator consequence | **P0** | +| **G2 — `content_match` has no Handler-reachable surface** | ADR 0133 AC-3 says "WHEN a Handler emits a `content_match`" — but there is no export in `messagefoundry/__init__.py` (unlike `db_lookup`/`fhir_lookup` at lines 32-33/150/153 — **all four anchors RE-VERIFIED EXACT 2026-08-15, BACKLOG #1100**: the imports are at `:32`/`:33` and the `__all__` entries at `:150`/`:153`, while **`content_match` appears NOWHERE in the module**. **This row carries its own positive control and that is why it is trustworthy**: the same search that finds zero for `content_match` finds both comparators, so the zero is a fact about the code and not about the probe), no injected sink on a Handler context, and no dry-run path. Every test calls `sink.content_match(...)` directly | The differentiating Corepoint "Action Point" parity capability is unusable in practice; the PHI-free-by-contract guarantee has never been exercised through real Handler code | **No** — AC-3/AC-4 pass at the sink so nothing fails | **P0** | +| **G3 — `connection_started` is mapped but emitted nowhere** | `_AUTO_RESOLVE["connection_started"] = "connection_stopped"` ([`alert_sinks.py`](../../../messagefoundry/pipeline/alert_sinks.py) — **RE-POINTED 2026-08-15 (BACKLOG #1100): the mapping is at `:101`, not the cited `:100`, which holds the sibling `"connection_restored": "connection_error"` entry**) but a repo-wide search finds **no emit site**. **CONFIRMED WITH A POSITIVE CONTROL**: `"connection_started"` occurs in exactly ONE place in `messagefoundry/` — the `_AUTO_RESOLVE` mapping itself — whereas the same search finds `"connection_stopped"` genuinely emitted at `alert_sinks.py:682` (`self._emit({"type": "connection_stopped", ...})`) plus a `settings.py:2601` reference. **So the search does find emit sites when they exist, and the asymmetry IS the defect: one half of an auto-resolve PAIR can never fire.** `connection_stopped` raises the alert and `connection_started` is only ever a resolution TARGET, so a lane that stops on an internal error and is later restarted never clears — the alert is raisable but not resolvable, which is worse than an alert that was never wired at all because the operator sees a live alert for a healthy lane. A lane that STOPs on an internal error and is later restarted (by hand or by a #144 `control_action`) leaves its `alert_instance` permanently `open` | `alerts_active` on the connections dashboard stays non-zero forever; `/alerts/active` accumulates; the nav bell's `list_active_alerts(limit=200)` ([`webconsole/routes/status.py:166`](../../../messagefoundry_webconsole/routes/status.py)) saturates. Textbook alert fatigue — the operator learns to ignore the list and the next real stop is missed | **Partially** — `tests/test_phi_logging_inventory.py:439` *documents* the dead key; nothing asserts the operator consequence | **P0** | | **G4 — CLOSED by #323 (2026-08-02). The alert/security-notify SMTP hop is VERIFIED.** This row asserted the hop was encrypted but unauthenticated and that `docs/PHI.md` and `docs/BACKLOG.md` contradicted each other about it. **Both halves are false at HEAD** and were already false when this plan was written (BACKLOG #1100). | `send_plain_email` builds an explicit verifying context via `tls_policy.build_smtp_tls_context()` and passes it — `smtp.starttls(context=tls_context)` ([`alert_sinks.py:430-431`](../../../messagefoundry/pipeline/alert_sinks.py)), whose comment reads *"context= is REQUIRED (#323): starttls()'s own default verifies NOTHING"*. The `CERT_NONE`/`check_hostname=False` text at `:387-388` is a **historical note about the fixed defect**, not the current posture — reading it as current is the mistake this row made. | — | **Yes.** `docs/PHI.md` row 11 states the verifying posture and records the pre-#323 state explicitly as history; `tests/test_alert_smtp_tls.py` exists with 21 tests. The documents agree with the code and with each other. | **CLOSED** | | **G5 — `[alerts]` is startup-only** | `app.state.alerts_settings` is assigned only at app construction ([`api/app.py:1120`](../../../messagefoundry/api/app.py)) and lifespan startup (`:5485`). `POST /config/reload` (`:2741`) re-runs the `--config` graph, never the service-settings TOML | An operator adds a suppression rule mid-incident via the IDE or `messagefoundry alert add`; the IDE re-lists from the **file** and shows it; `/alerts/rules` still shows the **startup** set; the running notifier keeps paging until a restart. The requirement is documented only in `alerts_edit.py:19-21` and the CLI docstring — nowhere an operator looks | **No** | **P0** | | **G6 — IDE alert editor offers 4 of the 18 event types and 7 of the 15 fields** | [`ide/src/alertEditor.ts:13-19`](../../../ide/src/alertEditor.ts) offers a 5-entry dropdown — `any` plus only `connection_stopped`/`queue_buildup`/`storage_threshold`/`cert_expiry`; `:25-32` supports only `event_type`/`connection`/`min_depth`/`min_oldest_seconds`/`severity`/`transports`/`cooldown_seconds` (7 of the 15 `AlertRule` fields). `ide/src/test/suite/` has **no** alert test file (35 suites, none for the alert editor) | An operator on the supported GUI authoring path cannot rule on **14 of the 18** signals, nor set `id`, `recipients`, `mute`, `escalate`, `schedule`, `content_label`, `control_action`, `control_target` | **No** — the `ide` CI leg runs and tests nothing here | **P1** | From 88b852f0a8afd7368d9ccbcdf1c69ef6c948ac03 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:32:42 -0500 Subject: [PATCH 49/66] docs(test-plan): ch15 G5 -- one write site, not two, and all three anchors drifted (BACKLOG #1100) Substance CONFIRMED and stronger than the row states. Anchors fixed: the assignment is api/app.py:1134 (cited :1120), /config/reload is at :2791 (cited :2741, now blank), and the cited :5485 lands on an unrelated phi_read_hop_secure parameter. The row claims TWO write sites. There is exactly ONE. 'app.state.alerts_settings =' occurs once in the module (:1134); every other occurrence is a constructor parameter (:1038, :5453), a call argument (:1546, :5580), or a READ via getattr(request.app.state, 'alerts_settings', None) or AlertsSettings() (:1516, :2517, :4517). The 'lifespan startup' site is not a second write, it is the plumbing feeding the one write. That makes the gap CLEANER to test than the row implies -- a single assignment is a single point to prove unreachable from reload. And the read pattern matters: 'or AlertsSettings()' means a reload-time change does not merely fail to apply, it silently falls back to DEFAULTS at every read site, so the operator sees alerting behaviour that matches neither the old config nor the new one. --- docs/testing/master-test-plan/15-alerting-and-observability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/15-alerting-and-observability.md b/docs/testing/master-test-plan/15-alerting-and-observability.md index f36e0b9c3..a865138d3 100644 --- a/docs/testing/master-test-plan/15-alerting-and-observability.md +++ b/docs/testing/master-test-plan/15-alerting-and-observability.md @@ -136,7 +136,7 @@ ALERT-13 and ALERT-11. | **G2 — `content_match` has no Handler-reachable surface** | ADR 0133 AC-3 says "WHEN a Handler emits a `content_match`" — but there is no export in `messagefoundry/__init__.py` (unlike `db_lookup`/`fhir_lookup` at lines 32-33/150/153 — **all four anchors RE-VERIFIED EXACT 2026-08-15, BACKLOG #1100**: the imports are at `:32`/`:33` and the `__all__` entries at `:150`/`:153`, while **`content_match` appears NOWHERE in the module**. **This row carries its own positive control and that is why it is trustworthy**: the same search that finds zero for `content_match` finds both comparators, so the zero is a fact about the code and not about the probe), no injected sink on a Handler context, and no dry-run path. Every test calls `sink.content_match(...)` directly | The differentiating Corepoint "Action Point" parity capability is unusable in practice; the PHI-free-by-contract guarantee has never been exercised through real Handler code | **No** — AC-3/AC-4 pass at the sink so nothing fails | **P0** | | **G3 — `connection_started` is mapped but emitted nowhere** | `_AUTO_RESOLVE["connection_started"] = "connection_stopped"` ([`alert_sinks.py`](../../../messagefoundry/pipeline/alert_sinks.py) — **RE-POINTED 2026-08-15 (BACKLOG #1100): the mapping is at `:101`, not the cited `:100`, which holds the sibling `"connection_restored": "connection_error"` entry**) but a repo-wide search finds **no emit site**. **CONFIRMED WITH A POSITIVE CONTROL**: `"connection_started"` occurs in exactly ONE place in `messagefoundry/` — the `_AUTO_RESOLVE` mapping itself — whereas the same search finds `"connection_stopped"` genuinely emitted at `alert_sinks.py:682` (`self._emit({"type": "connection_stopped", ...})`) plus a `settings.py:2601` reference. **So the search does find emit sites when they exist, and the asymmetry IS the defect: one half of an auto-resolve PAIR can never fire.** `connection_stopped` raises the alert and `connection_started` is only ever a resolution TARGET, so a lane that stops on an internal error and is later restarted never clears — the alert is raisable but not resolvable, which is worse than an alert that was never wired at all because the operator sees a live alert for a healthy lane. A lane that STOPs on an internal error and is later restarted (by hand or by a #144 `control_action`) leaves its `alert_instance` permanently `open` | `alerts_active` on the connections dashboard stays non-zero forever; `/alerts/active` accumulates; the nav bell's `list_active_alerts(limit=200)` ([`webconsole/routes/status.py:166`](../../../messagefoundry_webconsole/routes/status.py)) saturates. Textbook alert fatigue — the operator learns to ignore the list and the next real stop is missed | **Partially** — `tests/test_phi_logging_inventory.py:439` *documents* the dead key; nothing asserts the operator consequence | **P0** | | **G4 — CLOSED by #323 (2026-08-02). The alert/security-notify SMTP hop is VERIFIED.** This row asserted the hop was encrypted but unauthenticated and that `docs/PHI.md` and `docs/BACKLOG.md` contradicted each other about it. **Both halves are false at HEAD** and were already false when this plan was written (BACKLOG #1100). | `send_plain_email` builds an explicit verifying context via `tls_policy.build_smtp_tls_context()` and passes it — `smtp.starttls(context=tls_context)` ([`alert_sinks.py:430-431`](../../../messagefoundry/pipeline/alert_sinks.py)), whose comment reads *"context= is REQUIRED (#323): starttls()'s own default verifies NOTHING"*. The `CERT_NONE`/`check_hostname=False` text at `:387-388` is a **historical note about the fixed defect**, not the current posture — reading it as current is the mistake this row made. | — | **Yes.** `docs/PHI.md` row 11 states the verifying posture and records the pre-#323 state explicitly as history; `tests/test_alert_smtp_tls.py` exists with 21 tests. The documents agree with the code and with each other. | **CLOSED** | -| **G5 — `[alerts]` is startup-only** | `app.state.alerts_settings` is assigned only at app construction ([`api/app.py:1120`](../../../messagefoundry/api/app.py)) and lifespan startup (`:5485`). `POST /config/reload` (`:2741`) re-runs the `--config` graph, never the service-settings TOML | An operator adds a suppression rule mid-incident via the IDE or `messagefoundry alert add`; the IDE re-lists from the **file** and shows it; `/alerts/rules` still shows the **startup** set; the running notifier keeps paging until a restart. The requirement is documented only in `alerts_edit.py:19-21` and the CLI docstring — nowhere an operator looks | **No** | **P0** | +| **G5 — `[alerts]` is startup-only** | `app.state.alerts_settings` is assigned only at app construction. **RE-VERIFIED 2026-08-15 (BACKLOG #1100) — substance CONFIRMED AND STRONGER THAN WRITTEN, all three anchors drifted.** Corrected: the assignment is [`api/app.py:1134`](../../../messagefoundry/api/app.py) (cited `:1120`), and `POST /config/reload` is at `:2791` (cited `:2741`, now a blank line). The cited `:5485` lands on an unrelated `phi_read_hop_secure` parameter. **The row claims TWO write sites; there is exactly ONE.** `app.state.alerts_settings =` occurs once in the whole module (`:1134`) — every other occurrence is either a constructor parameter (`:1038`, `:5453`), a call argument (`:1546`, `:5580`), or a READ via `getattr(request.app.state, "alerts_settings", None) or AlertsSettings()` (`:1516`, `:2517`, `:4517`). So the "lifespan startup" site is not a second write, it is the plumbing that feeds the one write. **That makes the gap cleaner to test than the row implies: a single assignment means a single point to prove unreachable from reload**, and the `getattr(..., None) or AlertsSettings()` read pattern means a reload-time change does not merely fail to apply, it silently falls back to defaults at every read site | An operator adds a suppression rule mid-incident via the IDE or `messagefoundry alert add`; the IDE re-lists from the **file** and shows it; `/alerts/rules` still shows the **startup** set; the running notifier keeps paging until a restart. The requirement is documented only in `alerts_edit.py:19-21` and the CLI docstring — nowhere an operator looks | **No** | **P0** | | **G6 — IDE alert editor offers 4 of the 18 event types and 7 of the 15 fields** | [`ide/src/alertEditor.ts:13-19`](../../../ide/src/alertEditor.ts) offers a 5-entry dropdown — `any` plus only `connection_stopped`/`queue_buildup`/`storage_threshold`/`cert_expiry`; `:25-32` supports only `event_type`/`connection`/`min_depth`/`min_oldest_seconds`/`severity`/`transports`/`cooldown_seconds` (7 of the 15 `AlertRule` fields). `ide/src/test/suite/` has **no** alert test file (35 suites, none for the alert editor) | An operator on the supported GUI authoring path cannot rule on **14 of the 18** signals, nor set `id`, `recipients`, `mute`, `escalate`, `schedule`, `content_label`, `control_action`, `control_target` | **No** — the `ide` CI leg runs and tests nothing here | **P1** | | **G7 — no end-to-end drive of the runner's buildup / stall / saturation emit sites** | `_maybe_alert_buildup` (`:5401`), `_maybe_alert_saturation` (`:5444`), `_maybe_alert_stall` (`:5511`) in `wiring_runner.py`. Only the pure `SaturationDetector` and the **engine-shard** non-owned-lane watchdog are tested | These are the three alerts an operator relies on to notice a stalled or drowning feed. Threshold resolution, the `_outbound_paused` suppression guard, the per-`(stage,lane)` `_BUILDUP_REALERT_SECONDS` throttle and the `pending_depth` read could all break silently | **No** — FEATURE-COVERAGE-PLAN `FCP:ALERT-10` flagged it and it is still open | **P1** | | **G8 — the alert-storm bound is untested** | `_MAX_QUEUE = 1000` with drop-with-warning ([`alert_sinks.py:126-131`](../../../messagefoundry/pipeline/alert_sinks.py)). With a wedged webhook (a hung POST inside the 10 s timeout) and a large estate, a burst silently exceeds the bound and the excess is dropped with only a `WARNING` — there is **no dropped-alert counter or metric** | The operator sees neither the alerts nor a countable drop signal. A regression that lowers the bound, blocks the drain loop, or turns the drop into a *stall* on the emitting delivery worker is invisible | **No** — `tests/test_communications_inventory.py:372` pins `_MAX_QUEUE` as documentation only | **P1** | From e05cb94655ca07bfd50f6b89c0a07bfec3e87754 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:35:43 -0500 Subject: [PATCH 50/66] docs(test-plan): ch15 G1 -- no live drift, and a naive-guard trap I walked into first (BACKLOG #1100) The row's own assessment is CONFIRMED structurally. AST census: _ALERT_EVENT_TYPES holds 18 types, messagefoundry/ emits 22 distinct {'type': ...} literals, giving 4 emitted-but-not-routable and 0 routable-but-never-emitted. All 4 are deliberate -- connection_restored, leadership_lost and dr_released are three of the four _AUTO_RESOLVE KEYS (resolution events, not raisable alerts), and test_email is the API's diagnostic probe. So there is no live type-level drift, exactly as the row already said. RECORDED A SECOND NAIVE-GUARD TRAP, worse than the method-name-vs- event-type one the row already names, because I hit it: the constant is frozenset({...}), NOT a bare set literal, so a regex expecting '{' after '=' matches nothing and yields an EMPTY routable set. Every emitted type then reads as unroutable. My first pass produced a confident 22-of-22 'EMITTED BUT NOT ROUTABLE' list -- indistinguishable from a catastrophic real finding, and refuted only by noticing that 0 routable-but-never-emitted cannot coexist with it. This matters because the row's whole point is that someone must WRITE this guard. Parse with AST and assert the routable set is non-empty before comparing; the failure mode is to fail LOUD and WRONG. Independent corroboration of G3 fell out of the same census, from a probe built for a different purpose: exactly 3 of the 4 _AUTO_RESOLVE keys are emitted, and the missing one is connection_started. --- docs/testing/master-test-plan/15-alerting-and-observability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/15-alerting-and-observability.md b/docs/testing/master-test-plan/15-alerting-and-observability.md index a865138d3..696c2049c 100644 --- a/docs/testing/master-test-plan/15-alerting-and-observability.md +++ b/docs/testing/master-test-plan/15-alerting-and-observability.md @@ -132,7 +132,7 @@ ALERT-13 and ALERT-11. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| **G1 — three alert vocabularies bound only by convention** | An emit site's `type` literal, `_ALERT_EVENT_TYPES`, and the `AlertSink` Protocol / `LoggingAlertSink` / `NotifierAlertSink` method sets drift. A type absent from the routable set can never be escalated, routed, suppressed or muted by any operator rule; a method present only on `NotifierAlertSink` raises `AttributeError` on the logging fallback | Every deployment. Drift has fired **twice** already: `lane_stuck` + `rcsi_off_degraded` once shipped unroutable — **both are members today** ([`settings.py:2515-2516`](../../../messagefoundry/config/settings.py)), rule-targetable end to end at `test_alert_rules.py:357/370/395`, so the only live drift is that no guard stops the next one; `content_match` is **still** divergent (only on `NotifierAlertSink` at [`alert_sinks.py:676`](../../../messagefoundry/pipeline/alert_sinks.py), absent from the Protocol and the fallback). Note the non-obvious third form: `AlertSink.saturation_rising()` emits type `"saturation"` — method name ≠ event type, so a naive guard would false-fail | **No.** No guard exists in either direction | **P0** | +| **G1 — three alert vocabularies bound only by convention** | An emit site's `type` literal, `_ALERT_EVENT_TYPES`, and the `AlertSink` Protocol / `LoggingAlertSink` / `NotifierAlertSink` method sets drift. A type absent from the routable set can never be escalated, routed, suppressed or muted by any operator rule; a method present only on `NotifierAlertSink` raises `AttributeError` on the logging fallback | Every deployment. Drift has fired **twice** already: `lane_stuck` + `rcsi_off_degraded` once shipped unroutable — **both are members today** ([`settings.py:2515-2516`](../../../messagefoundry/config/settings.py)), rule-targetable end to end at `test_alert_rules.py:357/370/395`, so the only live drift is that no guard stops the next one; `content_match` is **still** divergent (only on `NotifierAlertSink` at [`alert_sinks.py:676`](../../../messagefoundry/pipeline/alert_sinks.py), absent from the Protocol and the fallback). Note the non-obvious third form: `AlertSink.saturation_rising()` emits type `"saturation"` — method name ≠ event type, so a naive guard would false-fail. **STRUCTURALLY RE-VERIFIED 2026-08-15 (BACKLOG #1100): the row's own "no LIVE type-level drift" assessment is CONFIRMED.** An AST census of `_ALERT_EVENT_TYPES` (**18** types) against every `{"type": "..."}` literal in `messagefoundry/` (**22** distinct) yields **4** emitted-but-not-routable and **0** routable-but-never-emitted — and all 4 are deliberate: `connection_restored`, `leadership_lost` and `dr_released` are three of the four `_AUTO_RESOLVE` **keys** (`alert_sinks.py:99-106`), i.e. resolution events rather than raisable alerts, and `test_email` is the API's diagnostic transport probe (`app.py:2579`, documented "NO message body, NO PHI"). **A SECOND NAIVE-GUARD TRAP, MEASURED THE HARD WAY HERE AND MORE DANGEROUS THAN THE method-name-vs-event-type ONE THIS ROW ALREADY NAMES:** `_ALERT_EVENT_TYPES` is `frozenset({...})`, **not** a bare set literal, so a regex expecting `{` after the `=` matches nothing and yields an **EMPTY** routable set — whereupon **every** emitted type reads as unroutable. That produced a confident **22-of-22 "unroutable"** result on the first pass here, which is indistinguishable from a catastrophic real finding and is refuted only by noticing that 0 routable-but-never-emitted is impossible alongside it. **Parse this constant with AST, and assert the routable set is non-empty before comparing** — the guard's own failure mode is to fail LOUD and WRONG. *Independent corroboration of G3 fell out of the same census: exactly 3 of the 4 `_AUTO_RESOLVE` keys are emitted, and the missing one is `connection_started`* | **No.** No guard exists in either direction | **P0** | | **G2 — `content_match` has no Handler-reachable surface** | ADR 0133 AC-3 says "WHEN a Handler emits a `content_match`" — but there is no export in `messagefoundry/__init__.py` (unlike `db_lookup`/`fhir_lookup` at lines 32-33/150/153 — **all four anchors RE-VERIFIED EXACT 2026-08-15, BACKLOG #1100**: the imports are at `:32`/`:33` and the `__all__` entries at `:150`/`:153`, while **`content_match` appears NOWHERE in the module**. **This row carries its own positive control and that is why it is trustworthy**: the same search that finds zero for `content_match` finds both comparators, so the zero is a fact about the code and not about the probe), no injected sink on a Handler context, and no dry-run path. Every test calls `sink.content_match(...)` directly | The differentiating Corepoint "Action Point" parity capability is unusable in practice; the PHI-free-by-contract guarantee has never been exercised through real Handler code | **No** — AC-3/AC-4 pass at the sink so nothing fails | **P0** | | **G3 — `connection_started` is mapped but emitted nowhere** | `_AUTO_RESOLVE["connection_started"] = "connection_stopped"` ([`alert_sinks.py`](../../../messagefoundry/pipeline/alert_sinks.py) — **RE-POINTED 2026-08-15 (BACKLOG #1100): the mapping is at `:101`, not the cited `:100`, which holds the sibling `"connection_restored": "connection_error"` entry**) but a repo-wide search finds **no emit site**. **CONFIRMED WITH A POSITIVE CONTROL**: `"connection_started"` occurs in exactly ONE place in `messagefoundry/` — the `_AUTO_RESOLVE` mapping itself — whereas the same search finds `"connection_stopped"` genuinely emitted at `alert_sinks.py:682` (`self._emit({"type": "connection_stopped", ...})`) plus a `settings.py:2601` reference. **So the search does find emit sites when they exist, and the asymmetry IS the defect: one half of an auto-resolve PAIR can never fire.** `connection_stopped` raises the alert and `connection_started` is only ever a resolution TARGET, so a lane that stops on an internal error and is later restarted never clears — the alert is raisable but not resolvable, which is worse than an alert that was never wired at all because the operator sees a live alert for a healthy lane. A lane that STOPs on an internal error and is later restarted (by hand or by a #144 `control_action`) leaves its `alert_instance` permanently `open` | `alerts_active` on the connections dashboard stays non-zero forever; `/alerts/active` accumulates; the nav bell's `list_active_alerts(limit=200)` ([`webconsole/routes/status.py:166`](../../../messagefoundry_webconsole/routes/status.py)) saturates. Textbook alert fatigue — the operator learns to ignore the list and the next real stop is missed | **Partially** — `tests/test_phi_logging_inventory.py:439` *documents* the dead key; nothing asserts the operator consequence | **P0** | | **G4 — CLOSED by #323 (2026-08-02). The alert/security-notify SMTP hop is VERIFIED.** This row asserted the hop was encrypted but unauthenticated and that `docs/PHI.md` and `docs/BACKLOG.md` contradicted each other about it. **Both halves are false at HEAD** and were already false when this plan was written (BACKLOG #1100). | `send_plain_email` builds an explicit verifying context via `tls_policy.build_smtp_tls_context()` and passes it — `smtp.starttls(context=tls_context)` ([`alert_sinks.py:430-431`](../../../messagefoundry/pipeline/alert_sinks.py)), whose comment reads *"context= is REQUIRED (#323): starttls()'s own default verifies NOTHING"*. The `CERT_NONE`/`check_hostname=False` text at `:387-388` is a **historical note about the fixed defect**, not the current posture — reading it as current is the mistake this row made. | — | **Yes.** `docs/PHI.md` row 11 states the verifying posture and records the pre-#323 state explicitly as history; `tests/test_alert_smtp_tls.py` exists with 21 tests. The documents agree with the code and with each other. | **CLOSED** | From bde94fada6898c822f9b27463709162933df462e Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:40:10 -0500 Subject: [PATCH 51/66] docs(test-plan): ch05 accounting verified exact, and the two-table P0 trap recorded (BACKLOG #1100) Every figure in the chapter-05 summary paragraph checks out when the tables are PARSED rather than read: 67 CONN-nn rows, the 63 T / 4 C / 0 A split, CONN-34/35/38/49 present as the C rows, and the nine claimed P0 T rows all present and all P0 with nothing claimed that is missing. A clean accounting row, which is worth recording so nobody re-derives it. Recorded the trap that caught me, because it will catch the next counter: a naive scan for a P0 priority cell returns ELEVEN, not nine. The two extras are RISK-TABLE rows -- the risk table carries its own priority column. The chapter has TWO tables with P0 cells and only ONE is the T-row matrix this sentence counts, so a discrepancy against 9 is the counter's error and not the document's. This is the same shape as the false enumerations I committed earlier in this sweep: rows from two tables, counted together, then named after one of them. --- docs/testing/master-test-plan/05-connections-and-transports.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/05-connections-and-transports.md b/docs/testing/master-test-plan/05-connections-and-transports.md index f5248b039..d54bf1c32 100644 --- a/docs/testing/master-test-plan/05-connections-and-transports.md +++ b/docs/testing/master-test-plan/05-connections-and-transports.md @@ -108,7 +108,7 @@ This chapter covers **every registered inbound/outbound connector and the connec **Row class.** Every row carries a **Cls**: **T** = *Test* (a falsifiable assertion with an observable pass criterion — **only T rows count toward the release gate**); **C** = *Characterisation* (produces a recorded measurement, finding or dated decision, with no threshold yet — legitimate work that **cannot fail**, so it never gates a release, and becomes a T row the day its threshold is recorded); **A** = *Assurance* (an external engagement — pen test, third-party review, DAST — blocking only for an off-loopback / production-exposure release, and excluded from the ordinary P0 count). -This chapter has **67 rows: 63 T, 4 C (CONN-34, CONN-35, CONN-38, CONN-49 — all editorial doc/ledger reconciliations), 0 A.** Nine of the 63 T rows are **P0**: CONN-01 through CONN-08, plus the CONN-37 pointer row (P0 because its owner SEC-01 is P0). **Foreign IDs carry a prefix** — `FCP:` for a `docs/testing/FEATURE-COVERAGE-PLAN.md` gap ID and `W25:` for a WIN2025 test/matrix ID; a bare `CONN-nn`, `MIG-nn`, `SEC-nn`, `CFG-nn` etc. is always a row of **this** plan. Four rows are **pointers** (Method `—`, no work scoped here, kept so the deliverable is still counted): CONN-32 and CONN-33 → **MIG-74**, CONN-37 → **SEC-01**, CONN-54 → `FCP:CFG-3`/`FCP:CFG-4` + this plan's CFG-19/CFG-20. +This chapter has **67 rows: 63 T, 4 C (CONN-34, CONN-35, CONN-38, CONN-49 — all editorial doc/ledger reconciliations), 0 A.** Nine of the 63 T rows are **P0**: CONN-01 through CONN-08, plus the CONN-37 pointer row (P0 because its owner SEC-01 is P0). **EVERY FIGURE IN THIS PARAGRAPH RE-VERIFIED EXACT 2026-08-15 (BACKLOG #1100)** by parsing the tables rather than reading them: 67 `CONN-nn` rows, the 63/4/0 split holds, CONN-34/35/38/49 are all present as the C rows, and the nine claimed P0 T rows are all present and all P0 with **nothing claimed that is missing**. **A caution for anyone re-checking it, because it caught me:** a naive scan for a `P0` priority cell across this chapter returns **11**, not 9 — the two extras are *risk-table* rows (first cell "FrameDecoder treats SB inside an open frame as payload" and "WIN2025 acceptance rows bound to tests that do not exercise the named connector"), because the risk table carries its own priority column. **The chapter has TWO tables with P0 cells and only ONE of them is the T-row matrix this sentence counts**, so a discrepancy against 9 is the counter's error, not the doc's. **Foreign IDs carry a prefix** — `FCP:` for a `docs/testing/FEATURE-COVERAGE-PLAN.md` gap ID and `W25:` for a WIN2025 test/matrix ID; a bare `CONN-nn`, `MIG-nn`, `SEC-nn`, `CFG-nn` etc. is always a row of **this** plan. Four rows are **pointers** (Method `—`, no work scoped here, kept so the deliverable is still counted): CONN-32 and CONN-33 → **MIG-74**, CONN-37 → **SEC-01**, CONN-54 → `FCP:CFG-3`/`FCP:CFG-4` + this plan's CFG-19/CFG-20. | ID | Test | Type | Method | Env | Backend | Cls | Pri | Pass criteria | |---|---|---|---|---|---|---|---|---| From c572fdac90ddb720b54feef0669a674748f67822 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:42:30 -0500 Subject: [PATCH 52/66] docs(test-plan): audit my OWN corrections -- two welded claims I carried forward unchecked (BACKLOG #1100) Ran the Steward's audit method against my own work: a correction that replaces a wrong NUMBER while leaving the surrounding STORY standing. Two hits, both mine, both blessed by me earlier today. ch14 r113 -- the coverage claim was FALSE. I corrected TrayApp's line count (154 -> 113) and carried 'test_tray_shell.py only asserts imports + non-zero struct sizes' forward without opening the file. It has SIX tests; only ONE is the imports/structs one. Two of the other five cover this row's OWN stated risk: test_disabled_action_is_not_dispatchable (an action routed to the wrong handler) and test_assign_command_ids_maps_actionable_items_only (TRAY-20's subject). The gap is real but much narrower than 'behaviourally untested' -- what is genuinely uncovered is the Win32 pump (which icon paints, whether a toast fires, WM_DESTROY/WM_ENDSESSION teardown). Re-scoped TRAY-19/20/ 22 to the pump rather than to action routing. ch17 r112 -- conclusion survives, its SUPPORT does not. I kept 'no doc-vs-measurement guard exists' as still-true, including its enumeration. test_feature_map_claims.py does not check 'only' three things; it has six tests. None compares a PUBLISHED number against a MEASURED one, which is WHY the conclusion survives -- but a reader quoting the enumeration to argue the file is nearly empty would be wrong, and the enumeration is what gets quoted. Both are the same shape I have been writing into other people's rows all afternoon: I checked the part that was false and carried the adjacent part because it sat next to something I had just verified. --- .../master-test-plan/14-tray-service-and-distribution.md | 2 +- docs/testing/master-test-plan/17-performance-and-scale.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/testing/master-test-plan/14-tray-service-and-distribution.md b/docs/testing/master-test-plan/14-tray-service-and-distribution.md index 40aa7a9b6..7a0d65f73 100644 --- a/docs/testing/master-test-plan/14-tray-service-and-distribution.md +++ b/docs/testing/master-test-plan/14-tray-service-and-distribution.md @@ -110,7 +110,7 @@ icon state rendering and light/dark iconset (`theme.py`, `iconset.py`, 18 `.ico` | ADR 0113 §1's import contract unenforced | A future edit imports `messagefoundry.config.settings` into the tray to "read TLS properly" | Pulls the engine + pydantic into an unprivileged, always-running process; breaches the boundary the whole ADR rests on | **No** — `tests/test_dependency_boundaries.py:14` lists only `pipeline/transports/parsing/store/config`; nothing scans `messagefoundry/tray/` | **P1** | | `tray.log` destroyed by httpx per-tick noise | `_setup_logging` attaches the handler to the **root** logger at INFO (`__main__.py:29-31`); nothing raises the `httpx` logger, which logs one INFO line per request — two requests every 5 s | ~34k lines/day churn a 1 MB × 3 rotation, so transitions, user actions and elevation outcomes rotate out within hours — destroying the only forensic record of who stopped a clinical interface. Contradicts `docs/TRAY.md:162` ("state transitions, never per-tick") | **No** | **P1** | | `poll_seconds` is inert | Parsed and clamped (`config.py:257-259`), documented (`config.py:359-360`, `docs/TRAY.md:89`), but `StatusPoller._run` (`poller.py:169`) always calls `next_poll_seconds()`, which returns the hardcoded `POLL_BASE_S = 5.0` (`state.py:24,166-172`) | An operator throttling a busy interface box sees no change and gets no signal the setting was ignored | **No** — tests assert only that the value lands in `TrayConfig` (`test_tray_config.py:126,236`) | **P1** | -| The tray's Win32 layer is behaviourally untested | `TrayApp` (**`tray/app.py:42-154`, i.e. 113 lines — CORRECTED 2026-08-15 (BACKLOG #1100); the row said "154 lines", which is the END LINE NUMBER relabelled as a LENGTH, overstating the surface by 36%**) and the pump (`winshell.py:299-451` — **anchor VERIFIED EXACT: that range is precisely the ten dispatch functions `_add_icon` through `_message_loop`, which ends at :451**) decide which icon paints, whether a toast fires, whether Exit tears down, whether the icon is removed on `WM_DESTROY`/`WM_ENDSESSION` | A ghost icon after logoff, a frozen icon on a state change, an action routed to the wrong handler — the exact failure class ADR 0113 named as the design's main risk | **No** — `test_tray_shell.py` only asserts imports + non-zero struct sizes | **P0** — the design's *named* main risk cannot be priced below the install rows it sits beside; closed by TRAY-19, TRAY-20 and TRAY-22 | +| The tray's Win32 layer is behaviourally untested | `TrayApp` (**`tray/app.py:42-154`, i.e. 113 lines — CORRECTED 2026-08-15 (BACKLOG #1100); the row said "154 lines", which is the END LINE NUMBER relabelled as a LENGTH, overstating the surface by 36%**) and the pump (`winshell.py:299-451` — **anchor VERIFIED EXACT: that range is precisely the ten dispatch functions `_add_icon` through `_message_loop`, which ends at :451**) decide which icon paints, whether a toast fires, whether Exit tears down, whether the icon is removed on `WM_DESTROY`/`WM_ENDSESSION` | A ghost icon after logoff, a frozen icon on a state change, an action routed to the wrong handler — the exact failure class ADR 0113 named as the design's main risk | ~~**No** — `test_tray_shell.py` only asserts imports + non-zero struct sizes~~ **PARTLY. CORRECTED 2026-08-15 (BACKLOG #1100) — the coverage claim was FALSE and I carried it forward once before checking it.** `test_tray_shell.py` has **six** test functions, of which only `test_tray_windows_modules_import_and_build_structs` is the imports-and-structs test this cell describes. The other five are behavioural, and **two of them cover this row's own stated risk**: `test_disabled_action_is_not_dispatchable` (an action routed to the wrong handler) and `test_assign_command_ids_maps_actionable_items_only` (menu command-id mapping, i.e. TRAY-20's subject). Also present: `test_launcher_command_quotes_absolute_pythonw`, `test_pythonw_executable_falls_back_to_given_path`, `test_single_instance_second_acquire_detects_running`. **The gap is real but MUCH narrower than "behaviourally untested"**: what remains uncovered is the Win32 message-pump surface — which icon paints, whether a toast fires, whether the icon is removed on `WM_DESTROY`/`WM_ENDSESSION` — none of which any of the six exercises. Re-scope TRAY-19/20/22 to the pump, not to action routing | **P0** — the design's *named* main risk cannot be priced below the install rows it sits beside; closed by TRAY-19, TRAY-20 and TRAY-22 | | Autostart writes are untested | `set_autostart` / `is_autostart_enabled` (`autostart.py:34-64`) write and read the real `HKCU\…\Run` value; only the pure `launcher_command` helper is covered | A stale or wrong-interpreter command silently breaks Start-at-Login (no tray after reboot) or strands a value pointing at a deleted venv; the menu checkbox reads the same untested getter | **No** | **P1** | | Hardened box + non-admin operator = a lying tray | Under `-LockConfigDir`, a standard user cannot read the engine's settings TOML, so `service_toml_uses_tls` sees nothing, `build_engine_url` yields `http://`, the https probe fails, and a healthy engine renders `WEDGED`/`STOPPED`. **CONFIRMED END-TO-END IN CODE 2026-08-15 (BACKLOG #1100)**, every hop: `_read_toml` (`tray/config.py:279-286`) is documented "``None`` on any read/parse failure. Never raises" and catches `(OSError, ValueError)` at `:284` — **`PermissionError` IS an `OSError`**, so the lock yields `None`; `service_toml_uses_tls` then hits `if not service_toml: return False` (`:212-213`); `build_engine_url` renders `f"{'https' if tls else 'http'}://..."` (`:188`) as **`http://`**. **The root defect is a THREE-STATE COLLAPSE: parsed-and-no-TLS, file-absent, and file-present-but-UNREADABLE all become the same `None`, then the same `False`. The distinguishing information is destroyed at `:284` and cannot be recovered by any caller**, so no downstream fix is possible and the test must drive the unreadable case specifically. Instructively, the fleet's own `seat-tick.ps1` handles the identical distinction correctly and says so in comment — a roster that could not be read is not evidence that nobody is live — carrying a separate roster-blind flag rather than folding it into the empty case | The 2026-07-22 amendment's bug, resurfacing for the least-privileged user on the most-hardened box, with no on-screen path to the fix — a healthy clinical interface rendered as down, on the exact configuration the security guidance prescribes | **No** — the fail-soft-to-http behaviour is asserted as *correct* (`test_tray_config.py:274-287`); the operational consequence is untested and undocumented | **P0** — a tray that lies about a healthy engine is worse than no tray; closed by TRAY-27 (+ TRAY-43 on the box) | | Windows-only lock resolution failure | `pip install --require-hashes -r requirements.lock` — prescribed for Windows production at `docs/SERVICE.md:26-33` — runs only on ubuntu (`security.yml:71-77`) | A platform-marker gap, an sdist-only transitive, or a missing `win_amd64` wheel breaks the documented production install on the primary supported platform | **No** | **P1** | diff --git a/docs/testing/master-test-plan/17-performance-and-scale.md b/docs/testing/master-test-plan/17-performance-and-scale.md index 467a3a2b3..90b76a9c3 100644 --- a/docs/testing/master-test-plan/17-performance-and-scale.md +++ b/docs/testing/master-test-plan/17-performance-and-scale.md @@ -109,7 +109,7 @@ Nothing below re-tests any of that; the rows below attack the **verdict**, the * | The `--load` sustainable-rate verdict has no filling / backlog-slope term | A phase whose in-flight backlog grows through the whole hold passes `zero_loss` + eventual drain and reads as sustained. `report.py:503 _run_slos` checks only zero-loss, drain seconds, dead letters, dup rate — no slope, no latency divergence | Every published `--load` ceiling; every adopter sizing run; a cutover sized 3–5.5x over capacity drops clinical messages | **No.** The gate exists only on the co-located shardcert path (`shardcert.py:1758`); `shardcert_ladder.py:1092` documents its own absence on the two-box path in capitals | **P0** | | `/stats` poller-zero contamination | `enginepoll.py:556 await_drain` declares drained on `backlog == 0 and queue_depth == 0 and in_pipeline == 0` plus two equal `read`/`written` samples. A zeroed or frozen `/stats` under exactly the overload the gate exists for satisfies all four | A saturating, lossy run reports PASS and drained | No staleness precondition anywhere | **P0** | | ~~Engine sharding has zero CI execution~~ **WITHDRAWN — FALSE. Corrected 2026-08-15 (BACKLOG #1100)** | **This row asserted a total coverage gap over work that has DEDICATED CI STEPS BUILT TO COVER IT, and it would have driven someone to build tests that already exist and already run.** Two of its three clauses are refuted: (1) "no workflow file contains the string `shard`" is **FALSE** — `ci.yml` carries it at least seven times, including the named steps "Run the engine-shard + statement-dispatch suites on real SQL Server" (`:1348`) and "Run the failover + engine-shard recovery suites on real Postgres" (`:1551`), plus `shard_recovery`/`shard_cert` in the changed-path gate at `:993` (positive control: `test` matched 24 of 24 workflow files, so the search was sound); (2) "named by no CI step, so they execute nowhere" is **FALSE** — all three are named at `ci.yml:1371`, `:1372`, `:1571`, and the steps set **exactly the variable the modules gate on** (`MEFOR_TEST_SQLSERVER: "1"` against `not os.getenv("MEFOR_TEST_SQLSERVER")`), so the gating makes them run rather than skip. The ci.yml comments record that these steps were ADDED to close this very gap ("before this step they were" dark), and `tests/test_serverdb_ci_coverage.py` now fails if a module-gated suite is added without being named — a meta-gate against this exact regression. **What SURVIVES, and it is real but far narrower than P0:** clause (3) holds — `tests/test_supervisor.py` uses `_FakeProcess` (11 references, docstring "without launching real engine subprocesses (Windows-safe)") with no `Popen`/`create_subprocess` anywhere, so the supervisor's process management is exercised only against a stand-in. **Re-scope to that: engine-shard RECOVERY is covered on both server backends; SUPERVISOR PROCESS HANDLING against a real subprocess is not** | ADR 0073 ownership-scoped recovery and single-delivery-consumer-per-lane are what keep per-lane FIFO and at-least-once correct on the only built multicore topology. A regression yields duplicate delivery, cross-engine-shard FIFO inversion, or permanently stranded rows | No | **P0** | -| ~~Published sizing contradicts the measured record and itself~~ **WITHDRAWN — STALE. Corrected 2026-08-15 (BACKLOG #1100)** | **The document was FIXED after this row was written; the over-claim it describes is gone.** The "~500 – low-thousands msg/s / ~40M+/day" tier **no longer exists in `SYSTEM-REQUIREMENTS.md`**. The current tier table (`:216-219`) tops out at **~165 msg/s at 4 engine shards**, and `:219` now carries the caveat inline — "on per-shard SQLite — a topology that is not supported above one shard". The cited anchors have also drifted: `:163` is **blank**, and `:195-198` is bench/active-passive guidance, not tiers. The alleged self-contradiction cannot be reproduced either: **"future direction, not built" appears NOWHERE in `docs/`**, so there is no counterpart to the `is built` statements (which are real, at `:184`, `:187`, `:193`, `:297`). **The "7.23x short" arithmetic is therefore moot — but flagging it anyway, because ~500/~72 is 6.94x, not 7.23x, so the multiplier did not derive from the two numbers the row put beside it.** What is still worth keeping from this row is its LAST column, which remains true: no doc-vs-measurement guard exists, so nothing would have caught the over-claim while it was live, and nothing will catch the next one | The document an adopter sizes hardware from. Order-of-magnitude over-claim + a self-contradiction about whether the scale-out path exists | No doc-vs-measurement guard exists; `tests/test_feature_map_claims.py` checks only the ASVS score, private-path links and superseded docs | **P0** | +| ~~Published sizing contradicts the measured record and itself~~ **WITHDRAWN — STALE. Corrected 2026-08-15 (BACKLOG #1100)** | **The document was FIXED after this row was written; the over-claim it describes is gone.** The "~500 – low-thousands msg/s / ~40M+/day" tier **no longer exists in `SYSTEM-REQUIREMENTS.md`**. The current tier table (`:216-219`) tops out at **~165 msg/s at 4 engine shards**, and `:219` now carries the caveat inline — "on per-shard SQLite — a topology that is not supported above one shard". The cited anchors have also drifted: `:163` is **blank**, and `:195-198` is bench/active-passive guidance, not tiers. The alleged self-contradiction cannot be reproduced either: **"future direction, not built" appears NOWHERE in `docs/`**, so there is no counterpart to the `is built` statements (which are real, at `:184`, `:187`, `:193`, `:297`). **The "7.23x short" arithmetic is therefore moot — but flagging it anyway, because ~500/~72 is 6.94x, not 7.23x, so the multiplier did not derive from the two numbers the row put beside it.** What is still worth keeping from this row is its LAST column — **but its SUPPORT is wrong, and I carried that forward unchecked before auditing it.** The **conclusion** stands: no doc-vs-measurement guard exists, so nothing would have caught the over-claim while it was live and nothing will catch the next one. The **enumeration** behind it does not: `tests/test_feature_map_claims.py` does not check "only the ASVS score, private-path links and superseded docs" — it has **six** tests, the three named plus `test_link_check_catches_a_missing_target_without_a_denylist`, `test_feature_map_is_actually_published` and `test_asvs_row_keeps_its_framing`. **None of the six compares a PUBLISHED number against a MEASURED one, which is why the conclusion survives the correction** — but a reader who quotes the enumeration to argue the file is nearly empty would be wrong, and the enumeration is the part that gets quoted | The document an adopter sizes hardware from. Order-of-magnitude over-claim + a self-contradiction about whether the scale-out path exists | No doc-vs-measurement guard exists; `tests/test_feature_map_claims.py` checks only the ASVS score, private-path links and superseded docs | **P0** | | The one published multi-process scale-out number was measured on a now-forbidden topology | `docs/benchmarks/TUNING-BASELINE.md:150` records the η ≈ 0.85 / E_core ≈ 42 msg/s **engine-sharding** result on the store line "**per-shard SQLite** (one store file per shard — **no shared DB**, no shared-DB commit contention by design)" (quoted verbatim from the source table), measured 2026-06-27. **RE-VERIFIED 2026-08-15 (BACKLOG #1100) — CONFIRMED, and the anchor is EXACT**, which is worth stating because the two rows above it in this same chapter were withdrawn as false and stale: `TUNING-BASELINE.md:150` is verbatim that store line, and `E_core ≈ 42 msg/s/shard` / `η ≈ 0.85` sit at `:161-162`. **One qualification to the row's framing: the topology is now DISCLOSED in both published files, so this is a reproducibility problem rather than a hidden-caveat one.** `TUNING-BASELINE.md:165-171` already separates the non-portable absolute from the portable "speedup shape", and `SYSTEM-REQUIREMENTS.md:219` names the unsupported topology inline. ADR 0063 (2026-07-01) and `sharding.py:81 require_unified_store` now **refuse** >1 engine shard on any non-server backend | The only published **engine-sharding** speedup — the shape adopters are told to multiply by their own `E_core` — is not reproducible with the shipped code, and it was measured with the shared-store contention deliberately absent | No | **P0** | | No throughput/latency regression gate anywhere | `.github/workflows/benchmark.yml` is `workflow_dispatch`-only, and each run wraps the harness in `set +e` … `set -e` so **even the harness's own SLO exit code is discarded**. Numbers are hand-transcribed into TUNING-BASELINE. `--baseline`/`--tolerance` exist (`report.py:753`) but are invoked by no workflow. `docs/CI-QUALITY.md` has no perf section | A 5x throughput or p99 regression from a claim-path, store or connector change merges green and is found on the next manual rig run | No | **P1** | | The published "reference performance floor" is enforced nowhere | TUNING-BASELINE:29 states the floor "≥ 200 msg/s sustained · ACK p99 ≤ 50 ms · e2e p99 ≤ 5 s". `harness/load/profiles/reference.toml` `[load.slo]` carries only `max_error_rate`, `max_dead_letters`, `zero_loss` — deliberately, but nothing else asserts the floor either | A release can clear the two-tier gate's performance tier by assertion rather than measurement | No | **P1** | From a09b094c129b41186a9aced21c1643a198858ab7 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:45:08 -0500 Subject: [PATCH 53/66] docs(test-plan): finish the self-audit -- two withdrawn rows were contradicting themselves (BACKLOG #1100) Both rows I withdrew earlier today kept their ORIGINAL coverage and priority columns, which I never touched. As published in between, ch17 r111 CONTRADICTED ITSELF: its evidence column said the suites run in CI on both server backends while the coverage column beside it still said 'No' and the priority still said P0. r112 carried the false 'checks only three things' enumeration verbatim in its coverage column even after I corrected that same enumeration two columns to the left. Fixed both. Coverage columns now state what is actually covered. Priorities are FLAGGED as needing re-grading rather than re-graded by me: I am not the party to assign the number, but a P0 left standing on a withdrawn finding reads as an open defect forever, which is the worse of the two errors. Also re-verified the ch06 r74 claim I had carried while re-pointing its anchors, and that one HOLDS verbatim: _eq() at :74 is 'return type(expected) is type(got)' with the docstring 'exceptions compare by type', and _EMPTY_FIELDS at :183-189 contains a literal blank-segment line whose own comment names it. The parity suite really does certify the defect as a PASS, via a type-only exception comparison that is correct for its purpose and wrong for this case. Not every carried claim was wrong. The point of the audit is that I had not checked any of them -- I verified what a row got wrong and then took its neighbours on trust because they sat next to something I had just confirmed. --- docs/testing/master-test-plan/06-parsing-and-codecs.md | 2 +- docs/testing/master-test-plan/17-performance-and-scale.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/testing/master-test-plan/06-parsing-and-codecs.md b/docs/testing/master-test-plan/06-parsing-and-codecs.md index 2a7eb2a34..d35cb1dc9 100644 --- a/docs/testing/master-test-plan/06-parsing-and-codecs.md +++ b/docs/testing/master-test-plan/06-parsing-and-codecs.md @@ -71,7 +71,7 @@ | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| Blank segment in any feed (i.e. any partner writing `\r\n\r\n`) | `Peek.field`, every routing property and `summarize()` raise a bare `IndexError` from `_builtin_hl7.raise_if_blank_segment_scan`. The reads sit **outside every catch**, so it unwinds out of `_handle_inbound`. **RE-VERIFIED STRUCTURALLY 2026-08-15 (BACKLOG #1100) — the citations had drifted ~230 lines, and the claim is re-established by AST rather than by line number so it survives the next move.** `_handle_inbound` is now **`wiring_runner.py:3950-4248`**; walking every `ast.Try` inside it and testing membership: **`control_id=peek.control_id` (`:4232`) and `summary=summarize(peek)` (`:4235`) are both OUTSIDE every `try` body**, as are all three `enqueue_ingress` calls (`:3977`, `:4077`, `:4229`). The raising mechanism is real — `_builtin_hl7.raise_if_blank_segment_scan`, called from `parsing/peek.py:265`. **The cited `:3721`/`:3732` now land on a blank line and on `record_received` respectively**, so a spot-check of this row would have found nothing wrong and nothing right; `transports/mllp.py` logs, emits `framing_error` and drops the TCP connection | Every message on that connection: 0 `messages` rows, 0 `queue` rows, no ACK/NAK. Direct count-and-log break **plus** a one-packet, infinitely repeatable per-connection DoS | **No — worse than no.** `tests/test_builtin_hl7_parity.py:183-189` (`adv:empty-fields`) *contains* a blank segment, and `_eq()` at `:74` compares exceptions **by type**, so both backends raising `IndexError` scores a PASS. `samples/messages/hapi-hl7v2/oml_o21.hl7` (26 interior blank lines) is in the globbed parity corpus and is certified this way | **P0** | +| Blank segment in any feed (i.e. any partner writing `\r\n\r\n`) | `Peek.field`, every routing property and `summarize()` raise a bare `IndexError` from `_builtin_hl7.raise_if_blank_segment_scan`. The reads sit **outside every catch**, so it unwinds out of `_handle_inbound`. **RE-VERIFIED STRUCTURALLY 2026-08-15 (BACKLOG #1100) — the citations had drifted ~230 lines, and the claim is re-established by AST rather than by line number so it survives the next move.** `_handle_inbound` is now **`wiring_runner.py:3950-4248`**; walking every `ast.Try` inside it and testing membership: **`control_id=peek.control_id` (`:4232`) and `summary=summarize(peek)` (`:4235`) are both OUTSIDE every `try` body**, as are all three `enqueue_ingress` calls (`:3977`, `:4077`, `:4229`). The raising mechanism is real — `_builtin_hl7.raise_if_blank_segment_scan`, called from `parsing/peek.py:265`. **The cited `:3721`/`:3732` now land on a blank line and on `record_received` respectively**, so a spot-check of this row would have found nothing wrong and nothing right; `transports/mllp.py` logs, emits `framing_error` and drops the TCP connection | Every message on that connection: 0 `messages` rows, 0 `queue` rows, no ACK/NAK. Direct count-and-log break **plus** a one-packet, infinitely repeatable per-connection DoS | **No — worse than no.** `tests/test_builtin_hl7_parity.py:183-189` (`adv:empty-fields`) *contains* a blank segment, and `_eq()` at `:74` compares exceptions **by type**, so both backends raising `IndexError` scores a PASS. `samples/messages/hapi-hl7v2/oml_o21.hl7` (26 interior blank lines) is in the globbed parity corpus and is certified this way. **RE-VERIFIED VERBATIM 2026-08-15 (BACKLOG #1100) as part of an audit of the claims I had carried forward WITHOUT checking while correcting this row's anchors** — this one holds exactly: `_eq()` at `:74` is `return type(expected) is type(got)`, its docstring reading "exceptions compare by **type** (message text differs)", and `_EMPTY_FIELDS` at `:183-189` contains a literal `"\r" # blank segment` line whose own comment names it. **So the parity suite does certify the defect as a PASS**, and it does so by a deliberate design choice (type-only exception comparison) that is correct for its purpose and wrong for this case | **P0** | | `\X00\` in any summarized field | Unescapes to a real U+0000 **inside `summarize()`** — after the `FCP:INGEST-4` post-decode NUL guard which inspects only the raw decoded text (the raw carries the 5-char escape, no NUL). **RE-VERIFIED AND RE-POINTED 2026-08-15 (BACKLOG #1100) — anchors had drifted ~500 lines; substance CONFIRMED BY LIVE REPRODUCTION, not by reading.** The guard is now `wiring_runner.py:4026` (`if "\x00" in text:`); the NUL rides `summary=summarize(peek)` (`:4235`) into `enqueue_ingress` (`:4229`), a **pre-ACK** commit that the same AST pass used for the row above confirms is **outside every `try`**. **The guard's OWN COMMENT is the false premise (SDS-3.7): `:4031-4033` states it dead-letters "BEFORE Peek.parse and any store write, so text (and control_id/summary/strict-fail errors derived from it) is NUL-free for the rest of this handler" — `summary` is precisely a derived value for which that is UNTRUE.** Re-run 2026-08-15 against `parsing/summary.py:27` (note: `summarize` lives there, NOT in `parsing/peek.py`): body containing `DOE^JANE\X00\SMITH` gives `"\x00" in text` = **False** at the guard, and `summarize()` then returns `'MRN MRN123 \xb7 DOE, JANE\x00SMITH'` — **a literal U+0000 manufactured after the control that promised there was none** | Postgres rejects a NUL at bind (`DataError`/SQLSTATE 22021) — **attributed, NOT independently tested here: this is the engine's own statement at `wiring_runner.py:410`, "Postgres REJECTS it at bind (the raise is uncaught)". A live Postgres reproduction is still owed** — the raise unwinds out of `_handle_inbound` exactly as the ADR 0028 `FCP:INGEST-4` amendment describes, dropping the connection with **no ERROR row**. SQLite/SQL Server truncate the summary → operator blind spot in list/search | No. `grep -r 'X00' tests/` returns nothing. The whole `FCP:INGEST-4` retrofit was built for this class and missed the derived-value path | **P0** | | ADR 0054 Phase 2 drops python-hl7 (`pyproject.toml:48` still pins `hl7>=0.4.5`) | The dual-backend oracle structurally requires python-hl7 installed. On removal `test_builtin_hl7_parity.py` becomes vacuous or uncollectable | Every subsequent edit to a 991-line from-scratch reimplementation of python-hl7's tolerant semantics ships unguarded | No golden vectors exist. Already flagged as `FCP:PARSE-12` (row at `FEATURE-COVERAGE-PLAN.md:857`, recommendation at `:874`) | P1 | | HL7 version breadth is 2.5.1-only | Generators are hl7apy `v2_5_1`-driven (`generators/_core.py:24,142,495`); `hl7schema.py:22` and `hl7structures.py:31` are pinned to `SUPPORTED_VERSION = "2.5.1"`; no test drives `validate()` over a 2.3/2.3.1/2.4/2.6/2.7 body | Real hospital feeds are dominated by 2.3/2.4. An hl7apy upgrade could start NAKing an entire production feed synchronously with no failing test | No | P1 | diff --git a/docs/testing/master-test-plan/17-performance-and-scale.md b/docs/testing/master-test-plan/17-performance-and-scale.md index 90b76a9c3..1aa143ad6 100644 --- a/docs/testing/master-test-plan/17-performance-and-scale.md +++ b/docs/testing/master-test-plan/17-performance-and-scale.md @@ -108,8 +108,8 @@ Nothing below re-tests any of that; the rows below attack the **verdict**, the * |---|---|---|---|---| | The `--load` sustainable-rate verdict has no filling / backlog-slope term | A phase whose in-flight backlog grows through the whole hold passes `zero_loss` + eventual drain and reads as sustained. `report.py:503 _run_slos` checks only zero-loss, drain seconds, dead letters, dup rate — no slope, no latency divergence | Every published `--load` ceiling; every adopter sizing run; a cutover sized 3–5.5x over capacity drops clinical messages | **No.** The gate exists only on the co-located shardcert path (`shardcert.py:1758`); `shardcert_ladder.py:1092` documents its own absence on the two-box path in capitals | **P0** | | `/stats` poller-zero contamination | `enginepoll.py:556 await_drain` declares drained on `backlog == 0 and queue_depth == 0 and in_pipeline == 0` plus two equal `read`/`written` samples. A zeroed or frozen `/stats` under exactly the overload the gate exists for satisfies all four | A saturating, lossy run reports PASS and drained | No staleness precondition anywhere | **P0** | -| ~~Engine sharding has zero CI execution~~ **WITHDRAWN — FALSE. Corrected 2026-08-15 (BACKLOG #1100)** | **This row asserted a total coverage gap over work that has DEDICATED CI STEPS BUILT TO COVER IT, and it would have driven someone to build tests that already exist and already run.** Two of its three clauses are refuted: (1) "no workflow file contains the string `shard`" is **FALSE** — `ci.yml` carries it at least seven times, including the named steps "Run the engine-shard + statement-dispatch suites on real SQL Server" (`:1348`) and "Run the failover + engine-shard recovery suites on real Postgres" (`:1551`), plus `shard_recovery`/`shard_cert` in the changed-path gate at `:993` (positive control: `test` matched 24 of 24 workflow files, so the search was sound); (2) "named by no CI step, so they execute nowhere" is **FALSE** — all three are named at `ci.yml:1371`, `:1372`, `:1571`, and the steps set **exactly the variable the modules gate on** (`MEFOR_TEST_SQLSERVER: "1"` against `not os.getenv("MEFOR_TEST_SQLSERVER")`), so the gating makes them run rather than skip. The ci.yml comments record that these steps were ADDED to close this very gap ("before this step they were" dark), and `tests/test_serverdb_ci_coverage.py` now fails if a module-gated suite is added without being named — a meta-gate against this exact regression. **What SURVIVES, and it is real but far narrower than P0:** clause (3) holds — `tests/test_supervisor.py` uses `_FakeProcess` (11 references, docstring "without launching real engine subprocesses (Windows-safe)") with no `Popen`/`create_subprocess` anywhere, so the supervisor's process management is exercised only against a stand-in. **Re-scope to that: engine-shard RECOVERY is covered on both server backends; SUPERVISOR PROCESS HANDLING against a real subprocess is not** | ADR 0073 ownership-scoped recovery and single-delivery-consumer-per-lane are what keep per-lane FIFO and at-least-once correct on the only built multicore topology. A regression yields duplicate delivery, cross-engine-shard FIFO inversion, or permanently stranded rows | No | **P0** | -| ~~Published sizing contradicts the measured record and itself~~ **WITHDRAWN — STALE. Corrected 2026-08-15 (BACKLOG #1100)** | **The document was FIXED after this row was written; the over-claim it describes is gone.** The "~500 – low-thousands msg/s / ~40M+/day" tier **no longer exists in `SYSTEM-REQUIREMENTS.md`**. The current tier table (`:216-219`) tops out at **~165 msg/s at 4 engine shards**, and `:219` now carries the caveat inline — "on per-shard SQLite — a topology that is not supported above one shard". The cited anchors have also drifted: `:163` is **blank**, and `:195-198` is bench/active-passive guidance, not tiers. The alleged self-contradiction cannot be reproduced either: **"future direction, not built" appears NOWHERE in `docs/`**, so there is no counterpart to the `is built` statements (which are real, at `:184`, `:187`, `:193`, `:297`). **The "7.23x short" arithmetic is therefore moot — but flagging it anyway, because ~500/~72 is 6.94x, not 7.23x, so the multiplier did not derive from the two numbers the row put beside it.** What is still worth keeping from this row is its LAST column — **but its SUPPORT is wrong, and I carried that forward unchecked before auditing it.** The **conclusion** stands: no doc-vs-measurement guard exists, so nothing would have caught the over-claim while it was live and nothing will catch the next one. The **enumeration** behind it does not: `tests/test_feature_map_claims.py` does not check "only the ASVS score, private-path links and superseded docs" — it has **six** tests, the three named plus `test_link_check_catches_a_missing_target_without_a_denylist`, `test_feature_map_is_actually_published` and `test_asvs_row_keeps_its_framing`. **None of the six compares a PUBLISHED number against a MEASURED one, which is why the conclusion survives the correction** — but a reader who quotes the enumeration to argue the file is nearly empty would be wrong, and the enumeration is the part that gets quoted | The document an adopter sizes hardware from. Order-of-magnitude over-claim + a self-contradiction about whether the scale-out path exists | No doc-vs-measurement guard exists; `tests/test_feature_map_claims.py` checks only the ASVS score, private-path links and superseded docs | **P0** | +| ~~Engine sharding has zero CI execution~~ **WITHDRAWN — FALSE. Corrected 2026-08-15 (BACKLOG #1100)** | **This row asserted a total coverage gap over work that has DEDICATED CI STEPS BUILT TO COVER IT, and it would have driven someone to build tests that already exist and already run.** Two of its three clauses are refuted: (1) "no workflow file contains the string `shard`" is **FALSE** — `ci.yml` carries it at least seven times, including the named steps "Run the engine-shard + statement-dispatch suites on real SQL Server" (`:1348`) and "Run the failover + engine-shard recovery suites on real Postgres" (`:1551`), plus `shard_recovery`/`shard_cert` in the changed-path gate at `:993` (positive control: `test` matched 24 of 24 workflow files, so the search was sound); (2) "named by no CI step, so they execute nowhere" is **FALSE** — all three are named at `ci.yml:1371`, `:1372`, `:1571`, and the steps set **exactly the variable the modules gate on** (`MEFOR_TEST_SQLSERVER: "1"` against `not os.getenv("MEFOR_TEST_SQLSERVER")`), so the gating makes them run rather than skip. The ci.yml comments record that these steps were ADDED to close this very gap ("before this step they were" dark), and `tests/test_serverdb_ci_coverage.py` now fails if a module-gated suite is added without being named — a meta-gate against this exact regression. **What SURVIVES, and it is real but far narrower than P0:** clause (3) holds — `tests/test_supervisor.py` uses `_FakeProcess` (11 references, docstring "without launching real engine subprocesses (Windows-safe)") with no `Popen`/`create_subprocess` anywhere, so the supervisor's process management is exercised only against a stand-in. **Re-scope to that: engine-shard RECOVERY is covered on both server backends; SUPERVISOR PROCESS HANDLING against a real subprocess is not** | ADR 0073 ownership-scoped recovery and single-delivery-consumer-per-lane are what keep per-lane FIFO and at-least-once correct on the only built multicore topology. A regression yields duplicate delivery, cross-engine-shard FIFO inversion, or permanently stranded rows. **THESE LAST THREE COLUMNS WERE WRITTEN FOR THE WITHDRAWN CLAIM AND I LEFT THEM STANDING WHEN I WITHDREW IT — corrected 2026-08-15 (BACKLOG #1100).** As published between those two edits the row CONTRADICTED ITSELF: the evidence column said the suites run in CI on both server backends, while the coverage column beside it still said "No" and the priority still said P0. The consequence above remains a true statement about what ADR 0073 protects; it is simply **no longer a consequence of THIS row**, because the recovery paths it names are the ones now shown to be exercised | **Was "No" — actually COVERED for engine-shard recovery** on real SQL Server and real Postgres (`ci.yml:1371`, `:1372`, `:1571`, each under a step that sets the gating `MEFOR_TEST_*` variable). **NOT covered: supervisor process handling against a real subprocess** (`test_supervisor.py` drives `_FakeProcess` only) | **NEEDS RE-GRADING, deliberately not re-graded here.** The P0 priced a total absence of engine-shard CI. The residual gap is narrower and I am not the right party to assign it a number — but **leaving P0 on a withdrawn finding is the worse error**, so it is flagged rather than silently carried | +| ~~Published sizing contradicts the measured record and itself~~ **WITHDRAWN — STALE. Corrected 2026-08-15 (BACKLOG #1100)** | **The document was FIXED after this row was written; the over-claim it describes is gone.** The "~500 – low-thousands msg/s / ~40M+/day" tier **no longer exists in `SYSTEM-REQUIREMENTS.md`**. The current tier table (`:216-219`) tops out at **~165 msg/s at 4 engine shards**, and `:219` now carries the caveat inline — "on per-shard SQLite — a topology that is not supported above one shard". The cited anchors have also drifted: `:163` is **blank**, and `:195-198` is bench/active-passive guidance, not tiers. The alleged self-contradiction cannot be reproduced either: **"future direction, not built" appears NOWHERE in `docs/`**, so there is no counterpart to the `is built` statements (which are real, at `:184`, `:187`, `:193`, `:297`). **The "7.23x short" arithmetic is therefore moot — but flagging it anyway, because ~500/~72 is 6.94x, not 7.23x, so the multiplier did not derive from the two numbers the row put beside it.** What is still worth keeping from this row is its LAST column — **but its SUPPORT is wrong, and I carried that forward unchecked before auditing it.** The **conclusion** stands: no doc-vs-measurement guard exists, so nothing would have caught the over-claim while it was live and nothing will catch the next one. The **enumeration** behind it does not: `tests/test_feature_map_claims.py` does not check "only the ASVS score, private-path links and superseded docs" — it has **six** tests, the three named plus `test_link_check_catches_a_missing_target_without_a_denylist`, `test_feature_map_is_actually_published` and `test_asvs_row_keeps_its_framing`. **None of the six compares a PUBLISHED number against a MEASURED one, which is why the conclusion survives the correction** — but a reader who quotes the enumeration to argue the file is nearly empty would be wrong, and the enumeration is the part that gets quoted | The document an adopter sizes hardware from. Order-of-magnitude over-claim + a self-contradiction about whether the scale-out path exists | No doc-vs-measurement guard exists — **TRUE, and the reason it survives is that none of `test_feature_map_claims.py`'s tests compares a PUBLISHED figure against a MEASURED one.** But the enumeration originally here was wrong and is corrected: the file does **not** check "only the ASVS score, private-path links and superseded docs" — it has **six** tests, adding `test_link_check_catches_a_missing_target_without_a_denylist`, `test_feature_map_is_actually_published` and `test_asvs_row_keeps_its_framing` | **NEEDS RE-GRADING — the finding above is WITHDRAWN as stale, so this P0 no longer prices anything live.** Flagged rather than re-graded, for the same reason as the row above: I am not the party to assign the number, but a P0 left standing on a withdrawn row reads as an open defect forever | | The one published multi-process scale-out number was measured on a now-forbidden topology | `docs/benchmarks/TUNING-BASELINE.md:150` records the η ≈ 0.85 / E_core ≈ 42 msg/s **engine-sharding** result on the store line "**per-shard SQLite** (one store file per shard — **no shared DB**, no shared-DB commit contention by design)" (quoted verbatim from the source table), measured 2026-06-27. **RE-VERIFIED 2026-08-15 (BACKLOG #1100) — CONFIRMED, and the anchor is EXACT**, which is worth stating because the two rows above it in this same chapter were withdrawn as false and stale: `TUNING-BASELINE.md:150` is verbatim that store line, and `E_core ≈ 42 msg/s/shard` / `η ≈ 0.85` sit at `:161-162`. **One qualification to the row's framing: the topology is now DISCLOSED in both published files, so this is a reproducibility problem rather than a hidden-caveat one.** `TUNING-BASELINE.md:165-171` already separates the non-portable absolute from the portable "speedup shape", and `SYSTEM-REQUIREMENTS.md:219` names the unsupported topology inline. ADR 0063 (2026-07-01) and `sharding.py:81 require_unified_store` now **refuse** >1 engine shard on any non-server backend | The only published **engine-sharding** speedup — the shape adopters are told to multiply by their own `E_core` — is not reproducible with the shipped code, and it was measured with the shared-store contention deliberately absent | No | **P0** | | No throughput/latency regression gate anywhere | `.github/workflows/benchmark.yml` is `workflow_dispatch`-only, and each run wraps the harness in `set +e` … `set -e` so **even the harness's own SLO exit code is discarded**. Numbers are hand-transcribed into TUNING-BASELINE. `--baseline`/`--tolerance` exist (`report.py:753`) but are invoked by no workflow. `docs/CI-QUALITY.md` has no perf section | A 5x throughput or p99 regression from a claim-path, store or connector change merges green and is found on the next manual rig run | No | **P1** | | The published "reference performance floor" is enforced nowhere | TUNING-BASELINE:29 states the floor "≥ 200 msg/s sustained · ACK p99 ≤ 50 ms · e2e p99 ≤ 5 s". `harness/load/profiles/reference.toml` `[load.slo]` carries only `max_error_rate`, `max_dead_letters`, `zero_loss` — deliberately, but nothing else asserts the floor either | A release can clear the two-tier gate's performance tier by assertion rather than measurement | No | **P1** | From ffcc2beb2e0c3afca587d87237b60ceb6ee991f0 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:47:56 -0500 Subject: [PATCH 54/66] docs(test-plan): ch12 advisory-ide-leg P0 confirmed, both anchors re-pointed (BACKLOG #1100) The chapter I wrongly skipped, and the row is right. ci-gate is at ci.yml:2132 and its needs: (:2135-2141) parses to exactly changes, sqlserver-store, postgres-store, load-test, load-test-sqlserver, windows-service-smoke. 'ide' is ABSENT (that job is :818), so the leg is not rolled up and a red IDE leg does not block a merge. Anchors re-pointed: the cited :265-274 is now a uv pip install block and :1386-1392 is a DR seed-gate step, so neither landed on what it claimed. The distinction is easy to invert and I had it inverted in my own project memory, which lumped 'ide' in with the rolled-up legs. Fixed there too. 'Not individually required' and 'cannot block a merge' are DIFFERENT properties: the server-DB and load legs have the first and not the second; ide is the only leg with both. --- docs/testing/master-test-plan/12-vs-code-ide-extension.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/12-vs-code-ide-extension.md b/docs/testing/master-test-plan/12-vs-code-ide-extension.md index daa0734d0..b993e5817 100644 --- a/docs/testing/master-test-plan/12-vs-code-ide-extension.md +++ b/docs/testing/master-test-plan/12-vs-code-ide-extension.md @@ -119,7 +119,7 @@ TypeScript and Python, and the **delivery vehicle**. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| `ide` leg is advisory only (`ci.yml:265-274`, not in `ci-gate` `needs:` at `ci.yml:1386-1392`) | A red IDE leg merges; a pure-Python PR renaming a CLI JSON field never runs the leg at all | The whole authoring surface stops working with a fully green CI | **No** — by design today | P0 | +| `ide` leg is advisory only — **CONFIRMED 2026-08-15 (BACKLOG #1100), both anchors RE-POINTED.** The `ide` job is `ci.yml:818` (cited `:265-274`, now a `uv pip install` block) and `ci-gate` is `:2132` with its `needs:` at **`:2135-2141`** (cited `:1386-1392`, now a DR seed-gate step). Parsed rather than read, that `needs:` list is exactly **`changes`, `sqlserver-store`, `postgres-store`, `load-test`, `load-test-sqlserver`, `windows-service-smoke`** — **`ide` is absent**, so the leg is genuinely not rolled up and a red IDE leg does not block a merge. **The distinction is easy to get backwards and I had it backwards in my own notes**: `ci-gate` DOES roll up the server-DB and load legs, which are also not individually required checks, so "not individually required" and "cannot block a merge" are different properties and only `ide` has both | A red IDE leg merges; a pure-Python PR renaming a CLI JSON field never runs the leg at all | The whole authoring surface stops working with a fully green CI | **No** — by design today | P0 | | `auth.ts` has zero tests | `signIn`'s pre-prompt `assertTargetAllowed` (`auth.ts:110-115`) is dropped in a refactor; credentials + bearer go in clear to a non-loopback host | Credential exfil (the exact `ADR0035:SEC-005` hole ADR 0035 closed), on a shared clinical workstation | No | P0 | | `signOut` revoke is unpinned (`auth.ts:73-86`; ADR 0110 AC-9 links to `auth.ts` itself, not a test) | Only the local token is dropped; the engine session lives until the 30-min idle cap | "Signed out" is a lie on a walked-away-from workstation | No | P0 | | The `@messagefoundry` prompt is assembled inline in the handler (`chat.ts:117-146`), not behind a testable function | A future `parts.push()` of a dry-run row, a Test Bench payload or a message body ships green | **PHI egress to a third-party model provider** — the extension's headline safety claim | No — `chat.test.ts` covers only `capCode` + command wiring | P0 | From 06130559b92f07277b034ec71b656f7442d49e4e Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:49:56 -0500 Subject: [PATCH 55/66] docs(test-plan): my ch17 r112 WITHDRAWAL REASON was itself unverified (BACKLOG #1100) Applied to myself the standard the Lander named: a retraction needs the same evidence as an assertion, and the asymmetry feels like humility while being the same laziness in the other direction. I withdrew this P0 as STALE and wrote 'the document was FIXED after this row was written'. I never checked that. 'The claimed text is not in this file' and 'the text was removed from this file' are different statements and I published the stronger one. git log -S settles it, and the answer is a THIRD verdict: 'low-thousands' NEVER appears in docs/SYSTEM-REQUIREMENTS.md at any point in that path's history -- nothing was removed from it it DID exist elsewhere in the repo before this plan (5fa6db9f, abedf865, 39990f8c 'bring the documentation set back in line with the code') today it survives in docs/ ONLY in this row the file's current ~165 msg/s table arrived via 959821a7 (#79) So the over-claim was real in SOME document and this row attributes it to a file that did not contain it. Whether it was ever live in THIS file is UNDETERMINED and a path-scoped query cannot settle it. MIS-CITATION, not staleness -- and the worse of the two: staleness is blameless drift, while a quotation that cannot be sourced to the file it names discredits the row carrying it. Three passes to get here ('stale', then 'never existed', then this). The first two were reached faster and on thinner evidence than I would have accepted for an assertion. --- docs/testing/master-test-plan/17-performance-and-scale.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/17-performance-and-scale.md b/docs/testing/master-test-plan/17-performance-and-scale.md index 1aa143ad6..0c453dbd3 100644 --- a/docs/testing/master-test-plan/17-performance-and-scale.md +++ b/docs/testing/master-test-plan/17-performance-and-scale.md @@ -109,7 +109,7 @@ Nothing below re-tests any of that; the rows below attack the **verdict**, the * | The `--load` sustainable-rate verdict has no filling / backlog-slope term | A phase whose in-flight backlog grows through the whole hold passes `zero_loss` + eventual drain and reads as sustained. `report.py:503 _run_slos` checks only zero-loss, drain seconds, dead letters, dup rate — no slope, no latency divergence | Every published `--load` ceiling; every adopter sizing run; a cutover sized 3–5.5x over capacity drops clinical messages | **No.** The gate exists only on the co-located shardcert path (`shardcert.py:1758`); `shardcert_ladder.py:1092` documents its own absence on the two-box path in capitals | **P0** | | `/stats` poller-zero contamination | `enginepoll.py:556 await_drain` declares drained on `backlog == 0 and queue_depth == 0 and in_pipeline == 0` plus two equal `read`/`written` samples. A zeroed or frozen `/stats` under exactly the overload the gate exists for satisfies all four | A saturating, lossy run reports PASS and drained | No staleness precondition anywhere | **P0** | | ~~Engine sharding has zero CI execution~~ **WITHDRAWN — FALSE. Corrected 2026-08-15 (BACKLOG #1100)** | **This row asserted a total coverage gap over work that has DEDICATED CI STEPS BUILT TO COVER IT, and it would have driven someone to build tests that already exist and already run.** Two of its three clauses are refuted: (1) "no workflow file contains the string `shard`" is **FALSE** — `ci.yml` carries it at least seven times, including the named steps "Run the engine-shard + statement-dispatch suites on real SQL Server" (`:1348`) and "Run the failover + engine-shard recovery suites on real Postgres" (`:1551`), plus `shard_recovery`/`shard_cert` in the changed-path gate at `:993` (positive control: `test` matched 24 of 24 workflow files, so the search was sound); (2) "named by no CI step, so they execute nowhere" is **FALSE** — all three are named at `ci.yml:1371`, `:1372`, `:1571`, and the steps set **exactly the variable the modules gate on** (`MEFOR_TEST_SQLSERVER: "1"` against `not os.getenv("MEFOR_TEST_SQLSERVER")`), so the gating makes them run rather than skip. The ci.yml comments record that these steps were ADDED to close this very gap ("before this step they were" dark), and `tests/test_serverdb_ci_coverage.py` now fails if a module-gated suite is added without being named — a meta-gate against this exact regression. **What SURVIVES, and it is real but far narrower than P0:** clause (3) holds — `tests/test_supervisor.py` uses `_FakeProcess` (11 references, docstring "without launching real engine subprocesses (Windows-safe)") with no `Popen`/`create_subprocess` anywhere, so the supervisor's process management is exercised only against a stand-in. **Re-scope to that: engine-shard RECOVERY is covered on both server backends; SUPERVISOR PROCESS HANDLING against a real subprocess is not** | ADR 0073 ownership-scoped recovery and single-delivery-consumer-per-lane are what keep per-lane FIFO and at-least-once correct on the only built multicore topology. A regression yields duplicate delivery, cross-engine-shard FIFO inversion, or permanently stranded rows. **THESE LAST THREE COLUMNS WERE WRITTEN FOR THE WITHDRAWN CLAIM AND I LEFT THEM STANDING WHEN I WITHDREW IT — corrected 2026-08-15 (BACKLOG #1100).** As published between those two edits the row CONTRADICTED ITSELF: the evidence column said the suites run in CI on both server backends, while the coverage column beside it still said "No" and the priority still said P0. The consequence above remains a true statement about what ADR 0073 protects; it is simply **no longer a consequence of THIS row**, because the recovery paths it names are the ones now shown to be exercised | **Was "No" — actually COVERED for engine-shard recovery** on real SQL Server and real Postgres (`ci.yml:1371`, `:1372`, `:1571`, each under a step that sets the gating `MEFOR_TEST_*` variable). **NOT covered: supervisor process handling against a real subprocess** (`test_supervisor.py` drives `_FakeProcess` only) | **NEEDS RE-GRADING, deliberately not re-graded here.** The P0 priced a total absence of engine-shard CI. The residual gap is narrower and I am not the right party to assign it a number — but **leaving P0 on a withdrawn finding is the worse error**, so it is flagged rather than silently carried | -| ~~Published sizing contradicts the measured record and itself~~ **WITHDRAWN — STALE. Corrected 2026-08-15 (BACKLOG #1100)** | **The document was FIXED after this row was written; the over-claim it describes is gone.** The "~500 – low-thousands msg/s / ~40M+/day" tier **no longer exists in `SYSTEM-REQUIREMENTS.md`**. The current tier table (`:216-219`) tops out at **~165 msg/s at 4 engine shards**, and `:219` now carries the caveat inline — "on per-shard SQLite — a topology that is not supported above one shard". The cited anchors have also drifted: `:163` is **blank**, and `:195-198` is bench/active-passive guidance, not tiers. The alleged self-contradiction cannot be reproduced either: **"future direction, not built" appears NOWHERE in `docs/`**, so there is no counterpart to the `is built` statements (which are real, at `:184`, `:187`, `:193`, `:297`). **The "7.23x short" arithmetic is therefore moot — but flagging it anyway, because ~500/~72 is 6.94x, not 7.23x, so the multiplier did not derive from the two numbers the row put beside it.** What is still worth keeping from this row is its LAST column — **but its SUPPORT is wrong, and I carried that forward unchecked before auditing it.** The **conclusion** stands: no doc-vs-measurement guard exists, so nothing would have caught the over-claim while it was live and nothing will catch the next one. The **enumeration** behind it does not: `tests/test_feature_map_claims.py` does not check "only the ASVS score, private-path links and superseded docs" — it has **six** tests, the three named plus `test_link_check_catches_a_missing_target_without_a_denylist`, `test_feature_map_is_actually_published` and `test_asvs_row_keeps_its_framing`. **None of the six compares a PUBLISHED number against a MEASURED one, which is why the conclusion survives the correction** — but a reader who quotes the enumeration to argue the file is nearly empty would be wrong, and the enumeration is the part that gets quoted | The document an adopter sizes hardware from. Order-of-magnitude over-claim + a self-contradiction about whether the scale-out path exists | No doc-vs-measurement guard exists — **TRUE, and the reason it survives is that none of `test_feature_map_claims.py`'s tests compares a PUBLISHED figure against a MEASURED one.** But the enumeration originally here was wrong and is corrected: the file does **not** check "only the ASVS score, private-path links and superseded docs" — it has **six** tests, adding `test_link_check_catches_a_missing_target_without_a_denylist`, `test_feature_map_is_actually_published` and `test_asvs_row_keeps_its_framing` | **NEEDS RE-GRADING — the finding above is WITHDRAWN as stale, so this P0 no longer prices anything live.** Flagged rather than re-graded, for the same reason as the row above: I am not the party to assign the number, but a P0 left standing on a withdrawn row reads as an open defect forever | +| ~~Published sizing contradicts the measured record and itself~~ **WITHDRAWN — STALE. Corrected 2026-08-15 (BACKLOG #1100)** | **CORRECTED TWICE — my first withdrawal reason ("the document was fixed") was itself unverified, and the true verdict is MIS-CITATION rather than staleness.** Measured 2026-08-15 with `git log -S`: the phrase "low-thousands" **never appears in `docs/SYSTEM-REQUIREMENTS.md` at any point in that path's recorded history**, so nothing was removed from it. The phrase **did** exist elsewhere in the repo before this plan was written (`5fa6db9f`, `abedf865`, and `39990f8c` "bring the documentation set back in line with the code"), and today it survives in `docs/` **only in this row** — while the file's current `~165 msg/s` tier table arrived via `959821a7` (#79, 2026-07-30). **So the over-claim was real in SOME document and this row attributes it to a file that did not contain it; whether it was ever live in THIS file is UNDETERMINED and my path-scoped query cannot settle it.** A mis-citation is worse than staleness: staleness is blameless drift, whereas a quotation that cannot be sourced to the file it names discredits the row that carries it. *Recorded at length because the correction took three passes — the first two were "stale" and "never existed", and both were reached faster and on thinner evidence than I would have accepted for an assertion.* The "~500 – low-thousands msg/s / ~40M+/day" tier **no longer exists in `SYSTEM-REQUIREMENTS.md`**. The current tier table (`:216-219`) tops out at **~165 msg/s at 4 engine shards**, and `:219` now carries the caveat inline — "on per-shard SQLite — a topology that is not supported above one shard". The cited anchors have also drifted: `:163` is **blank**, and `:195-198` is bench/active-passive guidance, not tiers. The alleged self-contradiction cannot be reproduced either: **"future direction, not built" appears NOWHERE in `docs/`**, so there is no counterpart to the `is built` statements (which are real, at `:184`, `:187`, `:193`, `:297`). **The "7.23x short" arithmetic is therefore moot — but flagging it anyway, because ~500/~72 is 6.94x, not 7.23x, so the multiplier did not derive from the two numbers the row put beside it.** What is still worth keeping from this row is its LAST column — **but its SUPPORT is wrong, and I carried that forward unchecked before auditing it.** The **conclusion** stands: no doc-vs-measurement guard exists, so nothing would have caught the over-claim while it was live and nothing will catch the next one. The **enumeration** behind it does not: `tests/test_feature_map_claims.py` does not check "only the ASVS score, private-path links and superseded docs" — it has **six** tests, the three named plus `test_link_check_catches_a_missing_target_without_a_denylist`, `test_feature_map_is_actually_published` and `test_asvs_row_keeps_its_framing`. **None of the six compares a PUBLISHED number against a MEASURED one, which is why the conclusion survives the correction** — but a reader who quotes the enumeration to argue the file is nearly empty would be wrong, and the enumeration is the part that gets quoted | The document an adopter sizes hardware from. Order-of-magnitude over-claim + a self-contradiction about whether the scale-out path exists | No doc-vs-measurement guard exists — **TRUE, and the reason it survives is that none of `test_feature_map_claims.py`'s tests compares a PUBLISHED figure against a MEASURED one.** But the enumeration originally here was wrong and is corrected: the file does **not** check "only the ASVS score, private-path links and superseded docs" — it has **six** tests, adding `test_link_check_catches_a_missing_target_without_a_denylist`, `test_feature_map_is_actually_published` and `test_asvs_row_keeps_its_framing` | **NEEDS RE-GRADING — the finding above is WITHDRAWN as stale, so this P0 no longer prices anything live.** Flagged rather than re-graded, for the same reason as the row above: I am not the party to assign the number, but a P0 left standing on a withdrawn row reads as an open defect forever | | The one published multi-process scale-out number was measured on a now-forbidden topology | `docs/benchmarks/TUNING-BASELINE.md:150` records the η ≈ 0.85 / E_core ≈ 42 msg/s **engine-sharding** result on the store line "**per-shard SQLite** (one store file per shard — **no shared DB**, no shared-DB commit contention by design)" (quoted verbatim from the source table), measured 2026-06-27. **RE-VERIFIED 2026-08-15 (BACKLOG #1100) — CONFIRMED, and the anchor is EXACT**, which is worth stating because the two rows above it in this same chapter were withdrawn as false and stale: `TUNING-BASELINE.md:150` is verbatim that store line, and `E_core ≈ 42 msg/s/shard` / `η ≈ 0.85` sit at `:161-162`. **One qualification to the row's framing: the topology is now DISCLOSED in both published files, so this is a reproducibility problem rather than a hidden-caveat one.** `TUNING-BASELINE.md:165-171` already separates the non-portable absolute from the portable "speedup shape", and `SYSTEM-REQUIREMENTS.md:219` names the unsupported topology inline. ADR 0063 (2026-07-01) and `sharding.py:81 require_unified_store` now **refuse** >1 engine shard on any non-server backend | The only published **engine-sharding** speedup — the shape adopters are told to multiply by their own `E_core` — is not reproducible with the shipped code, and it was measured with the shared-store contention deliberately absent | No | **P0** | | No throughput/latency regression gate anywhere | `.github/workflows/benchmark.yml` is `workflow_dispatch`-only, and each run wraps the harness in `set +e` … `set -e` so **even the harness's own SLO exit code is discarded**. Numbers are hand-transcribed into TUNING-BASELINE. `--baseline`/`--tolerance` exist (`report.py:753`) but are invoked by no workflow. `docs/CI-QUALITY.md` has no perf section | A 5x throughput or p99 regression from a claim-path, store or connector change merges green and is found on the next manual rig run | No | **P1** | | The published "reference performance floor" is enforced nowhere | TUNING-BASELINE:29 states the floor "≥ 200 msg/s sustained · ACK p99 ≤ 50 ms · e2e p99 ≤ 5 s". `harness/load/profiles/reference.toml` `[load.slo]` carries only `max_error_rate`, `max_dead_letters`, `zero_loss` — deliberately, but nothing else asserts the floor either | A release can clear the two-tier gate's performance tier by assertion rather than measurement | No | **P1** | From cd0965ff46587eb801d1b289f702219f868b2ea4 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:54:40 -0500 Subject: [PATCH 56/66] docs(test-plan): ch12 exec-gate and full-replace P0s confirmed (BACKLOG #1100) Row 128 CONFIRMED, and its ranges were right where row 122's were not: isExecGated() is defined at cli.ts:67 and guards at :152 and :186, both inside the cited :147-152 / :183-189. The 'nothing asserts' half holds with a POSITIVE CONTROL -- isExecGated is referenced by NO file under the IDE test directories, while 8 files there mention 'exec', so the directories exist and the search works. The zero is about coverage, not about the probe. A CWE-426 untrusted-workspace exec gate with no test is exactly as bad as the row prices it. Row 131 CONFIRMED VERBATIM, and the module states both halves in one comment: invariant 1 opens 'NOTHING ON THE RECORD IS EVER DROPPED' and then explains why that must be engineered -- 'connection upsert is a FULL REPLACE of the table, so a key the form fails to render is a key the form DELETES.' The invariant is the MITIGATION; the full replace is the HAZARD that makes it load-bearing. The row's real point is that the tested part (planSave) is the mitigation while the untested part is what the mitigation depends on: the webview posting every key. Recorded a caution in the row because it nearly caught me. Reading ONLY the cited first line makes the module look like it CONTRADICTS the row. The qualifying sentence is two lines below. I had the false finding half-written before reading the whole block -- a byte-window read where a structural one was needed, which is the same mistake this sweep keeps correcting in the plan. --- docs/testing/master-test-plan/12-vs-code-ide-extension.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/testing/master-test-plan/12-vs-code-ide-extension.md b/docs/testing/master-test-plan/12-vs-code-ide-extension.md index b993e5817..b4940720e 100644 --- a/docs/testing/master-test-plan/12-vs-code-ide-extension.md +++ b/docs/testing/master-test-plan/12-vs-code-ide-extension.md @@ -125,10 +125,10 @@ TypeScript and Python, and the **delivery vehicle**. | The `@messagefoundry` prompt is assembled inline in the handler (`chat.ts:117-146`), not behind a testable function | A future `parts.push()` of a dry-run row, a Test Bench payload or a message body ships green | **PHI egress to a third-party model provider** — the extension's headline safety claim | No — `chat.test.ts` covers only `capCode` + command wiring | P0 | | Test Bench collections persist message bodies (`testBench.ts:23` `COLLECTIONS_KEY`, `:136-142`); ADR 0121 AC-4 is explicitly *design/review-enforced* | One word changed from `workspaceState` to `globalState` pushes saved case bodies into VS Code Settings Sync | PHI off-box to the user's cloud profile | No | P0 | | Transient PHI temp dir (`testBench.ts:233-236` `mkdtempSync("mefor-testbench-")`, `:271-279` `rmSync` in `finally`) | A cleanup regression leaves plaintext message bodies in `os.tmpdir()` | PHI at rest outside the store, unaudited | No | P0 | -| Nothing asserts `run()`/`runWithStdin()`/`runJson()` actually short-circuit (`cli.ts:147-152`, `:183-189`) or that activation execs nothing (`extension.ts:433-461`) | One dropped `isExecGated()` call ⇒ opening a cloned "starter config" repo runs a trojaned `.venv` interpreter on first open | Arbitrary code execution (CWE-426) from merely opening a folder | No — only the pure `resolvePythonPath` is pinned | P0 | +| Nothing asserts `run()`/`runWithStdin()`/`runJson()` actually short-circuit (`cli.ts:147-152`, `:183-189` — **RE-VERIFIED 2026-08-15 (BACKLOG #1100): both cited RANGES are correct**, the guards being `if (isExecGated()) {` at `:152` and `:186`, with the function defined at `:67`) — **and the "nothing asserts" half is CONFIRMED WITH A POSITIVE CONTROL: `isExecGated` is referenced by NO file under the IDE test directories, while 8 files there mention `exec`, so the directories exist and the search works. The zero is a fact about coverage, not about the probe** or that activation execs nothing (`extension.ts:433-461`) | One dropped `isExecGated()` call ⇒ opening a cloned "starter config" repo runs a trojaned `.venv` interpreter on first open | Arbitrary code execution (CWE-426) from merely opening a folder | No — only the pure `resolvePythonPath` is pinned | P0 | | The packaged VSIX is never built, installed or smoke-tested; every test runs from `extensionDevelopmentPath` (`runTest.ts:13`) | A `.vscodeignore` or asset regression drops `media/hl7schema.json`, `media/hl7structures.json`, `media/stepsWebview.js`, `snippets/`, `media/walkthrough/` or the copied `LICENSE` | An extension that installs and then silently has no autocomplete / no field picker / no walkthrough | No | P0 | | The IDE↔CLI JSON contract is hand-mirrored in TypeScript; only the two HL7 media artifacts are pinned (`tests/test_ide_artifacts.py`) | A field rename in `graph --json`, `connection schema --json`, `dryrun --json`/`--trace json`, `codeset list\|show`, `alert list`, `security show` or `ai-policy --json` breaks the consumer | Tree, form, Test Bench, code-set grid, alert/security editors — with both suites green | No | P0 | -| The webview → CLI → file → tree write path is never exercised end to end | `connection upsert` is a **FULL REPLACE** (`connectionForm.ts:12-17`); a key the *webview* fails to post is a key the save DELETES, even though `planSave` is unit-tested | Silent loss of `schedule` / `shard` (the engine-shard partition tag, `connections_edit.py:86-88`) / allowlist / retry keys from a live `connections.toml` | No — `planSave` is tested only against synthetic objects | P0 | +| The webview → CLI → file → tree write path is never exercised end to end | `connection upsert` is a **FULL REPLACE** (`connectionForm.ts:12-17`); a key the *webview* fails to post is a key the save DELETES, even though `planSave` is unit-tested. **CONFIRMED VERBATIM 2026-08-15 (BACKLOG #1100) — the module states BOTH halves itself**, in one comment: invariant 1 opens "NOTHING ON THE RECORD IS EVER DROPPED" and then explains why that has to be engineered — "`connection upsert` is a FULL REPLACE of the table, so a key the form fails to render is a key the form DELETES." **The invariant is the MITIGATION; the full-replace is the HAZARD that makes it load-bearing, and this row's point is that the tested part is the mitigation (`planSave`) while the untested part is what the mitigation depends on — the webview actually posting every key.** *Caution for a re-checker, because it nearly caught me: reading only the cited first line makes the module look like it CONTRADICTS this row. It does not — the qualifying sentence is two lines below, and the invariant is a claim about what the module achieves, not about what `upsert` does* | Silent loss of `schedule` / `shard` (the engine-shard partition tag, `connections_edit.py:86-88`) / allowlist / retry keys from a live `connections.toml` | No — `planSave` is tested only against synthetic objects | P0 | | No plan artifact owns the IDE | Every gap above is unassigned | The largest single UI surface has no accountable owner | N/A | P0 | | `runTest.ts:15` passes no `version` to `runTests()` while `engines.vscode` is `^1.95.0` | A runtime-only newer API breaks every user on the declared minimum | Install-quality gate for the pending Marketplace publish | No | P1 | | `newRoute.ts` generates the Python the engine executes; `q()` (`:76`) escapes only `\` and `"` | A newline or control char in a name emits a syntactically broken `.py`; a wrong `ibSpec`/`obSpec`/router binding emits a valid-but-miswired graph | Mis-routed clinical messages from an authored artifact the user then promotes | No — zero tests | P1 | From 4594e2b9a5d623e6e17841b15b2da9b631e18de5 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:56:42 -0500 Subject: [PATCH 57/66] docs(test-plan): ch12 -- the auth gate is TESTED, its CALL SITE is not (BACKLOG #1100) Row 123's title 'auth.ts has zero tests' is literally false (ai-policy.test.ts:14 imports peekToken from it) and its framing understates the trap. Corrected to name the real gap. Measured: signIn is auth.ts:108; the guard is assertTargetAllowed(url) at :111, inside the cited :110-115 -- anchor EXACT assertTargetAllowed is NOT untested. engine-target.test.ts imports it from ../../engineTarget and asserts at least six cases, including 'non-loopback over plain http is REFUSED (the core SEC-005 case)' and 'an unparseable URL fails safe' NO test exercises signIn, so nothing asserts :111 is ever reached So deleting that one line leaves all six gate tests GREEN. The control is proven correct and its call site is unpinned -- the same shape as the planSave/full-replace row, and the reason 'write tests for auth.ts' is the wrong instruction. The test to write asserts that signIn CONSULTS the gate BEFORE prompting: an ORDERING property, which no test of the gate alone can express. Second time today a 'zero tests' claim was not zero (after test_tray_shell.py). Both times the coverage that existed sat one import away from the thing the row was worried about. --- docs/testing/master-test-plan/12-vs-code-ide-extension.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/12-vs-code-ide-extension.md b/docs/testing/master-test-plan/12-vs-code-ide-extension.md index b4940720e..e451f0beb 100644 --- a/docs/testing/master-test-plan/12-vs-code-ide-extension.md +++ b/docs/testing/master-test-plan/12-vs-code-ide-extension.md @@ -120,7 +120,7 @@ TypeScript and Python, and the **delivery vehicle**. | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | `ide` leg is advisory only — **CONFIRMED 2026-08-15 (BACKLOG #1100), both anchors RE-POINTED.** The `ide` job is `ci.yml:818` (cited `:265-274`, now a `uv pip install` block) and `ci-gate` is `:2132` with its `needs:` at **`:2135-2141`** (cited `:1386-1392`, now a DR seed-gate step). Parsed rather than read, that `needs:` list is exactly **`changes`, `sqlserver-store`, `postgres-store`, `load-test`, `load-test-sqlserver`, `windows-service-smoke`** — **`ide` is absent**, so the leg is genuinely not rolled up and a red IDE leg does not block a merge. **The distinction is easy to get backwards and I had it backwards in my own notes**: `ci-gate` DOES roll up the server-DB and load legs, which are also not individually required checks, so "not individually required" and "cannot block a merge" are different properties and only `ide` has both | A red IDE leg merges; a pure-Python PR renaming a CLI JSON field never runs the leg at all | The whole authoring surface stops working with a fully green CI | **No** — by design today | P0 | -| `auth.ts` has zero tests | `signIn`'s pre-prompt `assertTargetAllowed` (`auth.ts:110-115`) is dropped in a refactor; credentials + bearer go in clear to a non-loopback host | Credential exfil (the exact `ADR0035:SEC-005` hole ADR 0035 closed), on a shared clinical workstation | No | P0 | +| ~~`auth.ts` has zero tests~~ **`auth.ts`'s SIGN-IN PATH is unexercised — corrected 2026-08-15 (BACKLOG #1100); "zero tests" is literally false, `ai-policy.test.ts:14` imports `peekToken` from it** | `signIn`'s pre-prompt `assertTargetAllowed` (`auth.ts:110-115`) is dropped in a refactor; credentials + bearer go in clear to a non-loopback host. **RE-VERIFIED 2026-08-15 (BACKLOG #1100) — anchor EXACT and the risk is REAL, but the row's framing understates the trap and its title is wrong.** `signIn` is at `auth.ts:108` and the guard is `const gate = assertTargetAllowed(url);` at **`:111`**, inside the cited range. **`assertTargetAllowed` is NOT untested — it is thoroughly tested**, in `ide/src/test/suite/engine-target.test.ts`, which imports it from `../../engineTarget` (not from `auth`) and asserts at least six cases including "non-loopback over plain http is REFUSED (the core SEC-005 case)" and "an unparseable URL fails safe". **WHAT IS UNTESTED IS THE INVOCATION, NOT THE GATE: no test exercises `signIn`, so nothing asserts that `:111` is ever reached. Deleting that one line leaves every one of those six tests GREEN** — the control is proven correct and its call site is unpinned, which is the same shape as the `planSave` row below and the reason "write tests for `auth.ts`" is the wrong instruction. **The test to write asserts that `signIn` consults the gate BEFORE prompting** — an ordering property, which no test of the gate alone can express | Credential exfil (the exact `ADR0035:SEC-005` hole ADR 0035 closed), on a shared clinical workstation | No | P0 | | `signOut` revoke is unpinned (`auth.ts:73-86`; ADR 0110 AC-9 links to `auth.ts` itself, not a test) | Only the local token is dropped; the engine session lives until the 30-min idle cap | "Signed out" is a lie on a walked-away-from workstation | No | P0 | | The `@messagefoundry` prompt is assembled inline in the handler (`chat.ts:117-146`), not behind a testable function | A future `parts.push()` of a dry-run row, a Test Bench payload or a message body ships green | **PHI egress to a third-party model provider** — the extension's headline safety claim | No — `chat.test.ts` covers only `capCode` + command wiring | P0 | | Test Bench collections persist message bodies (`testBench.ts:23` `COLLECTIONS_KEY`, `:136-142`); ADR 0121 AC-4 is explicitly *design/review-enforced* | One word changed from `workspaceState` to `globalState` pushes saved case bodies into VS Code Settings Sync | PHI off-box to the user's cloud profile | No | P0 | From dd4f9597cd96a649adb434e0371068994da7dcfe Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 10:58:43 -0500 Subject: [PATCH 58/66] docs(test-plan): ch12 rows 124/125/129/130 confirmed, every anchor exact (BACKLOG #1100) Four for four, which is worth stating because it is the opposite of what ch02-10 produced against wiring_runner.py. Drift really does track the churn of the cited file. 124 signOut revoke -- the code states it in its own comments: :76 is 'return true; // nothing to revoke' and :82 is 'return false; // could not reach the engine to revoke -- the session lives until it times out'. BOTH non-revoking paths return a value the caller cannot distinguish from success, which is what makes the missing test load-bearing rather than merely absent. 125 chat prompt -- the cited span really is four bare parts.push() calls in the handler body (summary, active editor code, task, user request). Adding a fifth is a one-line edit with NO seam to assert against. That is why it is a PHI risk and not a style complaint: CLAUDE.md section 9 confines the assistant to code_only, and that confinement currently exists only as the ABSENCE of a parts.push of a message body. 129 runTest.ts:13 is exactly 'const extensionDevelopmentPath = path.resolve(__dirname, "../../");' so the suite loads the source tree, never the packaged artefact. 130 test_ide_artifacts.py contains exactly TWO tests, both HL7 media artifacts -- count and subject both right. Flagged as a CLEAN enumeration, because several counts elsewhere in this plan were not. --- docs/testing/master-test-plan/12-vs-code-ide-extension.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/testing/master-test-plan/12-vs-code-ide-extension.md b/docs/testing/master-test-plan/12-vs-code-ide-extension.md index e451f0beb..69d9750a0 100644 --- a/docs/testing/master-test-plan/12-vs-code-ide-extension.md +++ b/docs/testing/master-test-plan/12-vs-code-ide-extension.md @@ -121,13 +121,13 @@ TypeScript and Python, and the **delivery vehicle**. |---|---|---|---|---| | `ide` leg is advisory only — **CONFIRMED 2026-08-15 (BACKLOG #1100), both anchors RE-POINTED.** The `ide` job is `ci.yml:818` (cited `:265-274`, now a `uv pip install` block) and `ci-gate` is `:2132` with its `needs:` at **`:2135-2141`** (cited `:1386-1392`, now a DR seed-gate step). Parsed rather than read, that `needs:` list is exactly **`changes`, `sqlserver-store`, `postgres-store`, `load-test`, `load-test-sqlserver`, `windows-service-smoke`** — **`ide` is absent**, so the leg is genuinely not rolled up and a red IDE leg does not block a merge. **The distinction is easy to get backwards and I had it backwards in my own notes**: `ci-gate` DOES roll up the server-DB and load legs, which are also not individually required checks, so "not individually required" and "cannot block a merge" are different properties and only `ide` has both | A red IDE leg merges; a pure-Python PR renaming a CLI JSON field never runs the leg at all | The whole authoring surface stops working with a fully green CI | **No** — by design today | P0 | | ~~`auth.ts` has zero tests~~ **`auth.ts`'s SIGN-IN PATH is unexercised — corrected 2026-08-15 (BACKLOG #1100); "zero tests" is literally false, `ai-policy.test.ts:14` imports `peekToken` from it** | `signIn`'s pre-prompt `assertTargetAllowed` (`auth.ts:110-115`) is dropped in a refactor; credentials + bearer go in clear to a non-loopback host. **RE-VERIFIED 2026-08-15 (BACKLOG #1100) — anchor EXACT and the risk is REAL, but the row's framing understates the trap and its title is wrong.** `signIn` is at `auth.ts:108` and the guard is `const gate = assertTargetAllowed(url);` at **`:111`**, inside the cited range. **`assertTargetAllowed` is NOT untested — it is thoroughly tested**, in `ide/src/test/suite/engine-target.test.ts`, which imports it from `../../engineTarget` (not from `auth`) and asserts at least six cases including "non-loopback over plain http is REFUSED (the core SEC-005 case)" and "an unparseable URL fails safe". **WHAT IS UNTESTED IS THE INVOCATION, NOT THE GATE: no test exercises `signIn`, so nothing asserts that `:111` is ever reached. Deleting that one line leaves every one of those six tests GREEN** — the control is proven correct and its call site is unpinned, which is the same shape as the `planSave` row below and the reason "write tests for `auth.ts`" is the wrong instruction. **The test to write asserts that `signIn` consults the gate BEFORE prompting** — an ordering property, which no test of the gate alone can express | Credential exfil (the exact `ADR0035:SEC-005` hole ADR 0035 closed), on a shared clinical workstation | No | P0 | -| `signOut` revoke is unpinned (`auth.ts:73-86`; ADR 0110 AC-9 links to `auth.ts` itself, not a test) | Only the local token is dropped; the engine session lives until the 30-min idle cap | "Signed out" is a lie on a walked-away-from workstation | No | P0 | -| The `@messagefoundry` prompt is assembled inline in the handler (`chat.ts:117-146`), not behind a testable function | A future `parts.push()` of a dry-run row, a Test Bench payload or a message body ships green | **PHI egress to a third-party model provider** — the extension's headline safety claim | No — `chat.test.ts` covers only `capCode` + command wiring | P0 | +| `signOut` revoke is unpinned (`auth.ts:73-86`; ADR 0110 AC-9 links to `auth.ts` itself, not a test) — **CONFIRMED 2026-08-15 (BACKLOG #1100), anchor EXACT, and the code says it in its own comments**: `:76` is `return true; // nothing to revoke` and `:82` is `return false; // could not reach the engine to revoke — the session lives until it times out`. **Both non-revoking paths return a value the caller cannot distinguish from success**, which is what makes the missing test load-bearing rather than merely absent | Only the local token is dropped; the engine session lives until the 30-min idle cap | "Signed out" is a lie on a walked-away-from workstation | No | P0 | +| The `@messagefoundry` prompt is assembled inline in the handler (`chat.ts:117-146`), not behind a testable function — **CONFIRMED 2026-08-15 (BACKLOG #1100), anchor EXACT, and the shape is precisely the hazard**: the cited span is four bare `parts.push(...)` calls in the handler body — `summary`, `"Active editor code:\n\`\`\`python\n" + capped.text`, `"Task: " + task`, `"User request: " + request.prompt`. **Adding a fifth is a one-line edit with no seam to assert against**, which is why this is a PHI risk and not a style complaint: CLAUDE.md §9 confines the MVP assistant to `code_only`, and that confinement currently exists only as the absence of a `parts.push` of a message body | A future `parts.push()` of a dry-run row, a Test Bench payload or a message body ships green | **PHI egress to a third-party model provider** — the extension's headline safety claim | No — `chat.test.ts` covers only `capCode` + command wiring | P0 | | Test Bench collections persist message bodies (`testBench.ts:23` `COLLECTIONS_KEY`, `:136-142`); ADR 0121 AC-4 is explicitly *design/review-enforced* | One word changed from `workspaceState` to `globalState` pushes saved case bodies into VS Code Settings Sync | PHI off-box to the user's cloud profile | No | P0 | | Transient PHI temp dir (`testBench.ts:233-236` `mkdtempSync("mefor-testbench-")`, `:271-279` `rmSync` in `finally`) | A cleanup regression leaves plaintext message bodies in `os.tmpdir()` | PHI at rest outside the store, unaudited | No | P0 | | Nothing asserts `run()`/`runWithStdin()`/`runJson()` actually short-circuit (`cli.ts:147-152`, `:183-189` — **RE-VERIFIED 2026-08-15 (BACKLOG #1100): both cited RANGES are correct**, the guards being `if (isExecGated()) {` at `:152` and `:186`, with the function defined at `:67`) — **and the "nothing asserts" half is CONFIRMED WITH A POSITIVE CONTROL: `isExecGated` is referenced by NO file under the IDE test directories, while 8 files there mention `exec`, so the directories exist and the search works. The zero is a fact about coverage, not about the probe** or that activation execs nothing (`extension.ts:433-461`) | One dropped `isExecGated()` call ⇒ opening a cloned "starter config" repo runs a trojaned `.venv` interpreter on first open | Arbitrary code execution (CWE-426) from merely opening a folder | No — only the pure `resolvePythonPath` is pinned | P0 | -| The packaged VSIX is never built, installed or smoke-tested; every test runs from `extensionDevelopmentPath` (`runTest.ts:13`) | A `.vscodeignore` or asset regression drops `media/hl7schema.json`, `media/hl7structures.json`, `media/stepsWebview.js`, `snippets/`, `media/walkthrough/` or the copied `LICENSE` | An extension that installs and then silently has no autocomplete / no field picker / no walkthrough | No | P0 | -| The IDE↔CLI JSON contract is hand-mirrored in TypeScript; only the two HL7 media artifacts are pinned (`tests/test_ide_artifacts.py`) | A field rename in `graph --json`, `connection schema --json`, `dryrun --json`/`--trace json`, `codeset list\|show`, `alert list`, `security show` or `ai-policy --json` breaks the consumer | Tree, form, Test Bench, code-set grid, alert/security editors — with both suites green | No | P0 | +| The packaged VSIX is never built, installed or smoke-tested; every test runs from `extensionDevelopmentPath` (`runTest.ts:13` — **CONFIRMED EXACT 2026-08-15 (BACKLOG #1100): that line is `const extensionDevelopmentPath = path.resolve(__dirname, "../../");`**, so the suite loads the source tree and never the packaged artefact, and no `.vscodeignore` or asset regression is observable from it) | A `.vscodeignore` or asset regression drops `media/hl7schema.json`, `media/hl7structures.json`, `media/stepsWebview.js`, `snippets/`, `media/walkthrough/` or the copied `LICENSE` | An extension that installs and then silently has no autocomplete / no field picker / no walkthrough | No | P0 | +| The IDE↔CLI JSON contract is hand-mirrored in TypeScript; only the two HL7 media artifacts are pinned (`tests/test_ide_artifacts.py`) — **CONFIRMED EXACT 2026-08-15 (BACKLOG #1100): that file contains exactly TWO tests, `test_hl7structures_artifact_in_sync` and `test_hl7schema_artifact_in_sync`, so both the count and the subject are right.** Worth flagging as a clean enumeration, because several counts elsewhere in this plan were not | A field rename in `graph --json`, `connection schema --json`, `dryrun --json`/`--trace json`, `codeset list\|show`, `alert list`, `security show` or `ai-policy --json` breaks the consumer | Tree, form, Test Bench, code-set grid, alert/security editors — with both suites green | No | P0 | | The webview → CLI → file → tree write path is never exercised end to end | `connection upsert` is a **FULL REPLACE** (`connectionForm.ts:12-17`); a key the *webview* fails to post is a key the save DELETES, even though `planSave` is unit-tested. **CONFIRMED VERBATIM 2026-08-15 (BACKLOG #1100) — the module states BOTH halves itself**, in one comment: invariant 1 opens "NOTHING ON THE RECORD IS EVER DROPPED" and then explains why that has to be engineered — "`connection upsert` is a FULL REPLACE of the table, so a key the form fails to render is a key the form DELETES." **The invariant is the MITIGATION; the full-replace is the HAZARD that makes it load-bearing, and this row's point is that the tested part is the mitigation (`planSave`) while the untested part is what the mitigation depends on — the webview actually posting every key.** *Caution for a re-checker, because it nearly caught me: reading only the cited first line makes the module look like it CONTRADICTS this row. It does not — the qualifying sentence is two lines below, and the invariant is a claim about what the module achieves, not about what `upsert` does* | Silent loss of `schedule` / `shard` (the engine-shard partition tag, `connections_edit.py:86-88`) / allowlist / retry keys from a live `connections.toml` | No — `planSave` is tested only against synthetic objects | P0 | | No plan artifact owns the IDE | Every gap above is unassigned | The largest single UI surface has no accountable owner | N/A | P0 | | `runTest.ts:15` passes no `version` to `runTests()` while `engines.vscode` is `^1.95.0` | A runtime-only newer API breaks every user on the declared minimum | Install-quality gate for the pending Marketplace publish | No | P1 | From ee76b664a13e0f9357a576f60afe9a289acf6500 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 11:05:46 -0500 Subject: [PATCH 59/66] docs(test-plan): ch13 webview-mirror anchors re-pointed; the drift is monotonic (BACKLOG #1100) All ten mirror anchors resolved against the 33 function declarations in ide/media/stepsWebview.js. blockExtent:68 and captureBlock:83 were already exact; the rest drifted +1, +17, +18 or ~+22. The drift is MONOTONIC IN LINE NUMBER (+0, +0, +1, +1, +17, +17, +18, +18, +18, ~+22), which is the signature of insertions ABOVE rather than edits within. A single uniform offset would NOT have re-pointed these correctly, so each was resolved individually. Worth recording as a re-pointing method: check whether the offset is constant before applying one. The row's SUBSTANCE needed nothing -- it is already self-maintained, recording 'Yes, as of BACKLOG #233', the steps-mirror.test.ts jsdom parity suite, that both original grounds are spent, and the single live divergence that suite found (canDropRow accepting a read-only code row the webview refused) now closed. Also correcting myself: I first read the anchor as clipLabel:30 and diagnosed a dropped-digit transcription error. It is :309 -- MY OWN TRUNCATED PRINT cut the last digit, and I diagnosed from the truncation. Reading the full row dissolved it. Same byte-window mistake this sweep keeps finding in the plan, committed against the plan. --- docs/testing/master-test-plan/13-steps-editor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/13-steps-editor.md b/docs/testing/master-test-plan/13-steps-editor.md index 2e52b2d54..c95d39be2 100644 --- a/docs/testing/master-test-plan/13-steps-editor.md +++ b/docs/testing/master-test-plan/13-steps-editor.md @@ -63,7 +63,7 @@ | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| | Row-contract drift Python→TypeScript | `lens.py` renames a field or changes the partition; the IDE keeps parsing the *frozen* fixture snapshot, so every `ide` test stays green while the live view mis-projects rows. A mis-projected line range means a byte-stable edit splices into the **wrong statement** | Silent wrong transform → wrong clinical data on every message through that Handler | **No.** Verified on this checkout: all 7 committed fixtures are stale — `suite` is missing from all of them, `label`/`operand` from `adt.json` and `IB_RADIOLOGY_SR.json`. `suite` is load-bearing for drag/drop scoping (`stepsModel.ts:29-32`) | P0 | -| Webview mirror divergence | `ide/media/stepsWebview.js` re-implements 10 pure model functions (`blockExtent:68`, `captureBlock:83`, `buildDropSlots:100`, `walkMove:126`, `clipLabel:309` — the `blockLabel` mirror, `canDrop:384`, `scopeLabel:397`, `resolveDrop:404`, `barAnchor:433`, menu enablement ~`:590`) and, until BACKLOG #233, **was** explicitly not unit-tested (`steps-edit.test.ts` said so in as many words: "verified manually"). A diverged mirror computes a wrong move/drop target; the engine then applies it byte-stably and it re-parses clean | Moving a `msg.set` out of an `if` guard, or into the wrong branch, is a semantic change the byte-stability gates structurally cannot see | **Yes, as of BACKLOG #233** — `ide/src/test/suite/steps-mirror.test.ts` loads the webview script under jsdom and asserts every mirror against its model counterpart on every `ide` leg. Both of this row's original grounds are now spent: `buildDropSlots` **is** exported from `stepsModel.ts`, and the drop/clipboard mirrors are **no longer** "verified manually" (`steps-edit.test.ts` now points at the parity suite; what genuinely stays manual is the menu's positioning/dismissal/keyboard wiring, STEPS-76). The suite found exactly **one** live divergence — the model's `canDropRow` accepted a read-only `code` row as a drop target while the webview refused it, contradicting the model's own stated contract — and it is now closed. ADR 0108's "model and mirror in agreement" acceptance line is a gate rather than a claim | P0 | +| Webview mirror divergence | `ide/media/stepsWebview.js` re-implements 10 pure model functions (`blockExtent:68`, `captureBlock:83`, `buildDropSlots:101` (was `:100`), `walkMove:127` (was `:126`), `clipLabel:326` (was `:309`) — the `blockLabel` mirror, `canDrop:401` (was `:384`), `scopeLabel:415` (was `:397`), `resolveDrop:422` (was `:404`), `barAnchor:451` (was `:433`), menu enablement ~`:612` (was ~`:590`) — **ALL RE-POINTED 2026-08-15 (BACKLOG #1100) against the 33 function declarations in that file; `blockExtent:68` and `captureBlock:83` were already exact.** The drift is **monotonic in line number** (+0, +0, +1, +1, +17, +17, +18, +18, +18, ~+22), which is the signature of insertions above rather than of edits within, so **a single uniform offset would NOT have re-pointed these correctly** and each had to be resolved individually) and, until BACKLOG #233, **was** explicitly not unit-tested (`steps-edit.test.ts` said so in as many words: "verified manually"). A diverged mirror computes a wrong move/drop target; the engine then applies it byte-stably and it re-parses clean | Moving a `msg.set` out of an `if` guard, or into the wrong branch, is a semantic change the byte-stability gates structurally cannot see | **Yes, as of BACKLOG #233** — `ide/src/test/suite/steps-mirror.test.ts` loads the webview script under jsdom and asserts every mirror against its model counterpart on every `ide` leg. Both of this row's original grounds are now spent: `buildDropSlots` **is** exported from `stepsModel.ts`, and the drop/clipboard mirrors are **no longer** "verified manually" (`steps-edit.test.ts` now points at the parity suite; what genuinely stays manual is the menu's positioning/dismissal/keyboard wiring, STEPS-76). The suite found exactly **one** live divergence — the model's `canDropRow` accepted a read-only `code` row as a drop target while the webview refused it, contradicting the model's own stated contract — and it is now closed. ADR 0108's "model and mirror in agreement" acceptance line is a gate rather than a claim | P0 | | Engine change never triggers the IDE tests | The `ide` job's PR path filter is `^(ide/\|\.github/workflows/ci\.yml)` (`ci.yml:448`). A PR touching `messagefoundry/lens.py` — the exact contract the Steps view consumes — does not run it at all. And `ci-gate` deliberately does **not** `needs: ide` (`ci.yml:265`), so even a red `ide` leg cannot block a merge | The whole analyst-facing surface can regress green. (It does re-run on push-to-main, `ci.yml:410` — after the merge, when it can no longer block anything) | **No** | P0 | | Zero action rows in the tested corpus | Census on this checkout: `lens parse` over all 12 `samples/config` handlers yields **12 code rows, 12 send rows, 4 control rows and 0 action/lookup/diagnostic rows**. ADR 0076 gate 1's named corpus therefore proves nothing about the action-row, param-edit or Add-palette surface — the part an analyst actually uses. The IDE fixtures inherit the same hole | Every projection/edit path for the editable surface is only ever tested against ad-hoc inline strings written by whoever wrote the test — no shared, reviewed adversarial corpus | Partially (inline strings in `test_lens_native/palette/fanout`) — but no corpus-level partition/byte-stability/ruff/`check` sweep over action rows | P1 | | `{"expr": …}` splice writes arbitrary, unnormalized Python | `_validated_expr` (`lens.py:1869`) checks only "parses as one expression" and "is exactly one call argument". **Verified on this checkout:** `set_field(msg, "PID-3", __import__("os").popen("whoami").read())` is accepted and written into the Handler body. **Also newly verified:** an expr is spliced **verbatim**, so `foo( 1,2 )` produces output that **fails `ruff format --check`** and **fails `ruff check --select F` (F821 undefined name)** — a direct breach of ADR 0076 gate 3 ("emitted code is first-class") that no existing test covers | Handlers execute in the engine process. ADR 0144's lint runs only inside `messagefoundry check`, never on the rewrite path, and the Steps view gives no in-editor signal — while pitching a form field at an analyst who does not know Python. It also silently breaks the purity invariant the at-least-once contract depends on | **No** on all three counts | P1 | From 8988816c82133c72f5f57da2f4ed064640b25aef Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 11:12:59 -0500 Subject: [PATCH 60/66] docs(test-plan): ch13 rows 65/67 confirmed; fixture staleness is WIDER than stated (BACKLOG #1100) Row 65 CONFIRMED and the row UNDERSTATES its own finding. All 7 committed fixtures under ide/src/test/fixtures/lens/ lack 'suite', and they ALSO all lack 'label' and 'operand' -- not merely adt.json and IB_RADIOLOGY_SR.json as written. adt.json's complete key set is 13 keys (control, def_line, handler, handlers, kind, line_end, line_start, module, nesting, outbounds, recognized, rows, test_src) with none of the three at any depth, while messagefoundry/lens.py emits all three. Positive control: the same walker that reports them absent returns those 13 keys, so the absence is about the fixtures, not the probe. Anchor re-pointed: suite is stepsModel.ts:42-44, not :29-32 (which is now a row-KIND union). The comment at :42 confirms the drag/drop scoping role -- it greys an up/down at a suite edge so a reorder never crosses into or out of an if/for body -- and :552-553 consumes it. Row 67 CONFIRMED, all three anchors drifted. The real path filter is ci.yml:1007, grep -qE '^(ide/|\.github/workflows/ci\.yml)', so a PR touching messagefoundry/lens.py does not match and the leg never runs. The ide job is :818 and ci-gate's needs: is :2135-2141 without ide. Best evidence is the workflow's own comment at :820: 'ide is NOT a required check and ci-gate does NOT needs: ide'. The row's claim is written into the file it describes. --- docs/testing/master-test-plan/13-steps-editor.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/testing/master-test-plan/13-steps-editor.md b/docs/testing/master-test-plan/13-steps-editor.md index c95d39be2..4d4385c09 100644 --- a/docs/testing/master-test-plan/13-steps-editor.md +++ b/docs/testing/master-test-plan/13-steps-editor.md @@ -62,9 +62,9 @@ | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| Row-contract drift Python→TypeScript | `lens.py` renames a field or changes the partition; the IDE keeps parsing the *frozen* fixture snapshot, so every `ide` test stays green while the live view mis-projects rows. A mis-projected line range means a byte-stable edit splices into the **wrong statement** | Silent wrong transform → wrong clinical data on every message through that Handler | **No.** Verified on this checkout: all 7 committed fixtures are stale — `suite` is missing from all of them, `label`/`operand` from `adt.json` and `IB_RADIOLOGY_SR.json`. `suite` is load-bearing for drag/drop scoping (`stepsModel.ts:29-32`) | P0 | +| Row-contract drift Python→TypeScript | `lens.py` renames a field or changes the partition; the IDE keeps parsing the *frozen* fixture snapshot, so every `ide` test stays green while the live view mis-projects rows. A mis-projected line range means a byte-stable edit splices into the **wrong statement** | Silent wrong transform → wrong clinical data on every message through that Handler | **No.** Verified on this checkout: all 7 committed fixtures are stale — `suite` is missing from all of them, `label`/`operand` from `adt.json` and `IB_RADIOLOGY_SR.json`. `suite` is load-bearing for drag/drop scoping (**`stepsModel.ts:42-44`, RE-POINTED 2026-08-15 (BACKLOG #1100) — the cited `:29-32` is now a row-KIND union (`"note" | ... | "route"`)**; the real declaration is `suite?: string` at `:44` under the comment at `:42` "greys an up/down at a suite edge (a reorder never crosses into/out of an if/for body)", and it is consumed at `:552-553`). **THE STALENESS IS WIDER THAN THIS ROW STATES — measured, not read.** All **7** committed fixtures under `ide/src/test/fixtures/lens/` lack `suite`, and they also **all** lack `label` and `operand`, not merely `adt.json` and `IB_RADIOLOGY_SR.json`. `adt.json`'s complete key set is 13 keys — `control, def_line, handler, handlers, kind, line_end, line_start, module, nesting, outbounds, recognized, rows, test_src` — with none of the three present at any depth, while `messagefoundry/lens.py` emits all three. *(Positive control: the same walker that reports the three absent returns those 13 keys, so the absence is a fact about the fixtures and not about the probe.)* | P0 | | Webview mirror divergence | `ide/media/stepsWebview.js` re-implements 10 pure model functions (`blockExtent:68`, `captureBlock:83`, `buildDropSlots:101` (was `:100`), `walkMove:127` (was `:126`), `clipLabel:326` (was `:309`) — the `blockLabel` mirror, `canDrop:401` (was `:384`), `scopeLabel:415` (was `:397`), `resolveDrop:422` (was `:404`), `barAnchor:451` (was `:433`), menu enablement ~`:612` (was ~`:590`) — **ALL RE-POINTED 2026-08-15 (BACKLOG #1100) against the 33 function declarations in that file; `blockExtent:68` and `captureBlock:83` were already exact.** The drift is **monotonic in line number** (+0, +0, +1, +1, +17, +17, +18, +18, +18, ~+22), which is the signature of insertions above rather than of edits within, so **a single uniform offset would NOT have re-pointed these correctly** and each had to be resolved individually) and, until BACKLOG #233, **was** explicitly not unit-tested (`steps-edit.test.ts` said so in as many words: "verified manually"). A diverged mirror computes a wrong move/drop target; the engine then applies it byte-stably and it re-parses clean | Moving a `msg.set` out of an `if` guard, or into the wrong branch, is a semantic change the byte-stability gates structurally cannot see | **Yes, as of BACKLOG #233** — `ide/src/test/suite/steps-mirror.test.ts` loads the webview script under jsdom and asserts every mirror against its model counterpart on every `ide` leg. Both of this row's original grounds are now spent: `buildDropSlots` **is** exported from `stepsModel.ts`, and the drop/clipboard mirrors are **no longer** "verified manually" (`steps-edit.test.ts` now points at the parity suite; what genuinely stays manual is the menu's positioning/dismissal/keyboard wiring, STEPS-76). The suite found exactly **one** live divergence — the model's `canDropRow` accepted a read-only `code` row as a drop target while the webview refused it, contradicting the model's own stated contract — and it is now closed. ADR 0108's "model and mirror in agreement" acceptance line is a gate rather than a claim | P0 | -| Engine change never triggers the IDE tests | The `ide` job's PR path filter is `^(ide/\|\.github/workflows/ci\.yml)` (`ci.yml:448`). A PR touching `messagefoundry/lens.py` — the exact contract the Steps view consumes — does not run it at all. And `ci-gate` deliberately does **not** `needs: ide` (`ci.yml:265`), so even a red `ide` leg cannot block a merge | The whole analyst-facing surface can regress green. (It does re-run on push-to-main, `ci.yml:410` — after the merge, when it can no longer block anything) | **No** | P0 | +| Engine change never triggers the IDE tests | The `ide` job's PR path filter is `^(ide/\|\.github/workflows/ci\.yml)` (**`ci.yml:1007`, RE-POINTED 2026-08-15 (BACKLOG #1100)** — the filter is `grep -qE '^(ide/|\.github/workflows/ci\.yml)'`; the cited `:448` is now an unrelated benchmark comment). A PR touching `messagefoundry/lens.py` — the exact contract the Steps view consumes — does not run it at all. And `ci-gate` deliberately does **not** `needs: ide` (**RE-POINTED: the `ide` job is `ci.yml:818` and `ci-gate`'s `needs:` list is `:2135-2141`, containing `changes`, `sqlserver-store`, `postgres-store`, `load-test`, `load-test-sqlserver`, `windows-service-smoke` and NOT `ide`; the cited `:265` is a `uv pip install` comment. CONFIRMED, and `ci.yml:820` states it outright in its own words — "`ide` is NOT a required check and ci-gate does NOT `needs: ide`"**), so even a red `ide` leg cannot block a merge | The whole analyst-facing surface can regress green. (It does re-run on push-to-main, `ci.yml:410` — after the merge, when it can no longer block anything) | **No** | P0 | | Zero action rows in the tested corpus | Census on this checkout: `lens parse` over all 12 `samples/config` handlers yields **12 code rows, 12 send rows, 4 control rows and 0 action/lookup/diagnostic rows**. ADR 0076 gate 1's named corpus therefore proves nothing about the action-row, param-edit or Add-palette surface — the part an analyst actually uses. The IDE fixtures inherit the same hole | Every projection/edit path for the editable surface is only ever tested against ad-hoc inline strings written by whoever wrote the test — no shared, reviewed adversarial corpus | Partially (inline strings in `test_lens_native/palette/fanout`) — but no corpus-level partition/byte-stability/ruff/`check` sweep over action rows | P1 | | `{"expr": …}` splice writes arbitrary, unnormalized Python | `_validated_expr` (`lens.py:1869`) checks only "parses as one expression" and "is exactly one call argument". **Verified on this checkout:** `set_field(msg, "PID-3", __import__("os").popen("whoami").read())` is accepted and written into the Handler body. **Also newly verified:** an expr is spliced **verbatim**, so `foo( 1,2 )` produces output that **fails `ruff format --check`** and **fails `ruff check --select F` (F821 undefined name)** — a direct breach of ADR 0076 gate 3 ("emitted code is first-class") that no existing test covers | Handlers execute in the engine process. ADR 0144's lint runs only inside `messagefoundry check`, never on the rewrite path, and the Steps view gives no in-editor signal — while pitching a form field at an analyst who does not know Python. It also silently breaks the purity invariant the at-least-once contract depends on | **No** on all three counts | P1 | | False completeness: helper-body writes invisible and unmarked | ADR 0089 Phase D (helper descent) is unbuilt — `_msh(msg)` renders as an opaque `code` row. ADR 0104 AC-10's **"unmodeled code present"** marker does not exist: grep for `unmodeled` across `ide/`, `messagefoundry/`, `tests/` returns only the ADR and `docs/research/message-model-eval.md`; the named test is absent | An analyst edits a PID mapping in the Steps view, sees no other write to that field, saves — and a helper's later write silently overrides it. Wrong clinical data, no failing test, no operator signal | **No** | P1 | From bda30f5885bd5c2c979040285802dc704a594e1a Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 11:13:41 -0500 Subject: [PATCH 61/66] fix(test-plan): repair two ch13 rows I broke with unescaped pipes in the prior commit (BACKLOG #1100) 60 broke lines 65 and 67 -- 7 and 6 columns against a 5-column header. My replacement text contained bare | characters: a TypeScript union rendered as 'note | ... | route', and a grep -qE alternation quoted verbatim. Both are cell separators in a markdown table. Replaced both with prose that carries the same meaning and no pipes. THE GATE CAUGHT THIS AND I COMMITTED ANYWAY. I batched the column check and the commit into one command block, so the MISALIGNED lines printed and the commit ran regardless. That is the exact failure I hit early in this session and wrote a rule against: RUN THE GATE AS ITS OWN STEP AND READ IT BEFORE STAGING. Having the rule and having written it down did not help, because batching is what makes the mistake convenient. The information was on screen the whole time. It was not missing, it was unread -- which is worse, and is the same shape as several findings in the plan itself. --- docs/testing/master-test-plan/13-steps-editor.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/testing/master-test-plan/13-steps-editor.md b/docs/testing/master-test-plan/13-steps-editor.md index 4d4385c09..6ebd534f4 100644 --- a/docs/testing/master-test-plan/13-steps-editor.md +++ b/docs/testing/master-test-plan/13-steps-editor.md @@ -62,9 +62,9 @@ | Risk | Failure mode | Blast radius | Detected today? | Priority | |---|---|---|---|---| -| Row-contract drift Python→TypeScript | `lens.py` renames a field or changes the partition; the IDE keeps parsing the *frozen* fixture snapshot, so every `ide` test stays green while the live view mis-projects rows. A mis-projected line range means a byte-stable edit splices into the **wrong statement** | Silent wrong transform → wrong clinical data on every message through that Handler | **No.** Verified on this checkout: all 7 committed fixtures are stale — `suite` is missing from all of them, `label`/`operand` from `adt.json` and `IB_RADIOLOGY_SR.json`. `suite` is load-bearing for drag/drop scoping (**`stepsModel.ts:42-44`, RE-POINTED 2026-08-15 (BACKLOG #1100) — the cited `:29-32` is now a row-KIND union (`"note" | ... | "route"`)**; the real declaration is `suite?: string` at `:44` under the comment at `:42` "greys an up/down at a suite edge (a reorder never crosses into/out of an if/for body)", and it is consumed at `:552-553`). **THE STALENESS IS WIDER THAN THIS ROW STATES — measured, not read.** All **7** committed fixtures under `ide/src/test/fixtures/lens/` lack `suite`, and they also **all** lack `label` and `operand`, not merely `adt.json` and `IB_RADIOLOGY_SR.json`. `adt.json`'s complete key set is 13 keys — `control, def_line, handler, handlers, kind, line_end, line_start, module, nesting, outbounds, recognized, rows, test_src` — with none of the three present at any depth, while `messagefoundry/lens.py` emits all three. *(Positive control: the same walker that reports the three absent returns those 13 keys, so the absence is a fact about the fixtures and not about the probe.)* | P0 | +| Row-contract drift Python→TypeScript | `lens.py` renames a field or changes the partition; the IDE keeps parsing the *frozen* fixture snapshot, so every `ide` test stays green while the live view mis-projects rows. A mis-projected line range means a byte-stable edit splices into the **wrong statement** | Silent wrong transform → wrong clinical data on every message through that Handler | **No.** Verified on this checkout: all 7 committed fixtures are stale — `suite` is missing from all of them, `label`/`operand` from `adt.json` and `IB_RADIOLOGY_SR.json`. `suite` is load-bearing for drag/drop scoping (**`stepsModel.ts:42-44`, RE-POINTED 2026-08-15 (BACKLOG #1100) — the cited `:29-32` is now a row-KIND union ending in `"route"`**; the real declaration is `suite?: string` at `:44` under the comment at `:42` "greys an up/down at a suite edge (a reorder never crosses into/out of an if/for body)", and it is consumed at `:552-553`). **THE STALENESS IS WIDER THAN THIS ROW STATES — measured, not read.** All **7** committed fixtures under `ide/src/test/fixtures/lens/` lack `suite`, and they also **all** lack `label` and `operand`, not merely `adt.json` and `IB_RADIOLOGY_SR.json`. `adt.json`'s complete key set is 13 keys — `control, def_line, handler, handlers, kind, line_end, line_start, module, nesting, outbounds, recognized, rows, test_src` — with none of the three present at any depth, while `messagefoundry/lens.py` emits all three. *(Positive control: the same walker that reports the three absent returns those 13 keys, so the absence is a fact about the fixtures and not about the probe.)* | P0 | | Webview mirror divergence | `ide/media/stepsWebview.js` re-implements 10 pure model functions (`blockExtent:68`, `captureBlock:83`, `buildDropSlots:101` (was `:100`), `walkMove:127` (was `:126`), `clipLabel:326` (was `:309`) — the `blockLabel` mirror, `canDrop:401` (was `:384`), `scopeLabel:415` (was `:397`), `resolveDrop:422` (was `:404`), `barAnchor:451` (was `:433`), menu enablement ~`:612` (was ~`:590`) — **ALL RE-POINTED 2026-08-15 (BACKLOG #1100) against the 33 function declarations in that file; `blockExtent:68` and `captureBlock:83` were already exact.** The drift is **monotonic in line number** (+0, +0, +1, +1, +17, +17, +18, +18, +18, ~+22), which is the signature of insertions above rather than of edits within, so **a single uniform offset would NOT have re-pointed these correctly** and each had to be resolved individually) and, until BACKLOG #233, **was** explicitly not unit-tested (`steps-edit.test.ts` said so in as many words: "verified manually"). A diverged mirror computes a wrong move/drop target; the engine then applies it byte-stably and it re-parses clean | Moving a `msg.set` out of an `if` guard, or into the wrong branch, is a semantic change the byte-stability gates structurally cannot see | **Yes, as of BACKLOG #233** — `ide/src/test/suite/steps-mirror.test.ts` loads the webview script under jsdom and asserts every mirror against its model counterpart on every `ide` leg. Both of this row's original grounds are now spent: `buildDropSlots` **is** exported from `stepsModel.ts`, and the drop/clipboard mirrors are **no longer** "verified manually" (`steps-edit.test.ts` now points at the parity suite; what genuinely stays manual is the menu's positioning/dismissal/keyboard wiring, STEPS-76). The suite found exactly **one** live divergence — the model's `canDropRow` accepted a read-only `code` row as a drop target while the webview refused it, contradicting the model's own stated contract — and it is now closed. ADR 0108's "model and mirror in agreement" acceptance line is a gate rather than a claim | P0 | -| Engine change never triggers the IDE tests | The `ide` job's PR path filter is `^(ide/\|\.github/workflows/ci\.yml)` (**`ci.yml:1007`, RE-POINTED 2026-08-15 (BACKLOG #1100)** — the filter is `grep -qE '^(ide/|\.github/workflows/ci\.yml)'`; the cited `:448` is now an unrelated benchmark comment). A PR touching `messagefoundry/lens.py` — the exact contract the Steps view consumes — does not run it at all. And `ci-gate` deliberately does **not** `needs: ide` (**RE-POINTED: the `ide` job is `ci.yml:818` and `ci-gate`'s `needs:` list is `:2135-2141`, containing `changes`, `sqlserver-store`, `postgres-store`, `load-test`, `load-test-sqlserver`, `windows-service-smoke` and NOT `ide`; the cited `:265` is a `uv pip install` comment. CONFIRMED, and `ci.yml:820` states it outright in its own words — "`ide` is NOT a required check and ci-gate does NOT `needs: ide`"**), so even a red `ide` leg cannot block a merge | The whole analyst-facing surface can regress green. (It does re-run on push-to-main, `ci.yml:410` — after the merge, when it can no longer block anything) | **No** | P0 | +| Engine change never triggers the IDE tests | The `ide` job's PR path filter is `^(ide/\|\.github/workflows/ci\.yml)` (**`ci.yml:1007`, RE-POINTED 2026-08-15 (BACKLOG #1100)** — the filter matches paths starting `ide/` or the workflow file itself; the cited `:448` is now an unrelated benchmark comment). A PR touching `messagefoundry/lens.py` — the exact contract the Steps view consumes — does not run it at all. And `ci-gate` deliberately does **not** `needs: ide` (**RE-POINTED: the `ide` job is `ci.yml:818` and `ci-gate`'s `needs:` list is `:2135-2141`, containing `changes`, `sqlserver-store`, `postgres-store`, `load-test`, `load-test-sqlserver`, `windows-service-smoke` and NOT `ide`; the cited `:265` is a `uv pip install` comment. CONFIRMED, and `ci.yml:820` states it outright in its own words — "`ide` is NOT a required check and ci-gate does NOT `needs: ide`"**), so even a red `ide` leg cannot block a merge | The whole analyst-facing surface can regress green. (It does re-run on push-to-main, `ci.yml:410` — after the merge, when it can no longer block anything) | **No** | P0 | | Zero action rows in the tested corpus | Census on this checkout: `lens parse` over all 12 `samples/config` handlers yields **12 code rows, 12 send rows, 4 control rows and 0 action/lookup/diagnostic rows**. ADR 0076 gate 1's named corpus therefore proves nothing about the action-row, param-edit or Add-palette surface — the part an analyst actually uses. The IDE fixtures inherit the same hole | Every projection/edit path for the editable surface is only ever tested against ad-hoc inline strings written by whoever wrote the test — no shared, reviewed adversarial corpus | Partially (inline strings in `test_lens_native/palette/fanout`) — but no corpus-level partition/byte-stability/ruff/`check` sweep over action rows | P1 | | `{"expr": …}` splice writes arbitrary, unnormalized Python | `_validated_expr` (`lens.py:1869`) checks only "parses as one expression" and "is exactly one call argument". **Verified on this checkout:** `set_field(msg, "PID-3", __import__("os").popen("whoami").read())` is accepted and written into the Handler body. **Also newly verified:** an expr is spliced **verbatim**, so `foo( 1,2 )` produces output that **fails `ruff format --check`** and **fails `ruff check --select F` (F821 undefined name)** — a direct breach of ADR 0076 gate 3 ("emitted code is first-class") that no existing test covers | Handlers execute in the engine process. ADR 0144's lint runs only inside `messagefoundry check`, never on the rewrite path, and the Steps view gives no in-editor signal — while pitching a form field at an analyst who does not know Python. It also silently breaks the purity invariant the at-least-once contract depends on | **No** on all three counts | P1 | | False completeness: helper-body writes invisible and unmarked | ADR 0089 Phase D (helper descent) is unbuilt — `_msh(msg)` renders as an opaque `code` row. ADR 0104 AC-10's **"unmodeled code present"** marker does not exist: grep for `unmodeled` across `ide/`, `messagefoundry/`, `tests/` returns only the ADR and `docs/research/message-model-eval.md`; the named test is absent | An analyst edits a PID mapping in the Steps view, sees no other write to that field, saves — and a helper's later write silently overrides it. Wrong clinical data, no failing test, no operator signal | **No** | P1 | From b11e8053c622c898ea3b449336fdd306c6b9985b Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 11:17:05 -0500 Subject: [PATCH 62/66] docs(test-plan): sweep ch01, the chapter I nearly wrote off on a P0 count (BACKLOG #1100) I had ch01 down as out of scope because it has ZERO P0 rows. It has THIRTEEN file:line citations, which is the sweep's ACTUAL subject. 'Chapters with P0 rows' and 'chapters with verifiable citations' are different sets, and I had been prioritising by the wrong one all session -- the same measure-one-thing-name-another error this sweep keeps finding, now in my triage rather than my probes. The chapter itself is in good shape. Checked: __main__.py:98 --shard EXACT (serve.add_argument( at :98, '--shard' at :99) __main__.py:115 supervise exact release.yml:477 release-harness EXACT __main__.py:349 generate DRIFTED +1 -> :350 Re-pointed the one drift. The generate subparser is 'generate = sub.add_parser(' at :350 with the name on :351. Worth recording how nearly I got that wrong: my first probe searched for add_parser("generate" and returned ZERO, which reads as 'the subcommand was removed'. It failed because the name is on the NEXT line. A control (38 add_parser calls exist in the file) plus a broad search found it immediately. Fourth misleading zero today from a pattern-shaped probe, and the first one I caught BEFORE drawing the conclusion rather than after. --- .../master-test-plan/01-environments-data-and-tooling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/01-environments-data-and-tooling.md b/docs/testing/master-test-plan/01-environments-data-and-tooling.md index df81684c8..93adb2c6e 100644 --- a/docs/testing/master-test-plan/01-environments-data-and-tooling.md +++ b/docs/testing/master-test-plan/01-environments-data-and-tooling.md @@ -126,7 +126,7 @@ is never redirected to a committed file, a ticket, or a CI log. message families the engine handles: `adt`, `oru`, `orm`, `oml`, `orl`, `mdm`, `mfn`, `dft`, `bar`, `ras`, `rde`, `siu`, `vxu`, `documents`, with `all_types.py` as the registry. Two entry points: -- `messagefoundry generate` — the CLI subcommand ([`__main__.py:349`](../../../messagefoundry/__main__.py)). +- `messagefoundry generate` — the CLI subcommand ([`__main__.py:350`](../../../messagefoundry/__main__.py) — **RE-POINTED 2026-08-15 (BACKLOG #1100), was `:349`, which is now blank**; the subparser is `generate = sub.add_parser(` at `:350` with the name `"generate"` on `:351`, and its flags follow at `:353-361`). - `python -m messagefoundry.generators.adt [--triggers A01,A04] [--count N] [--out DIR]` — the ADT corpus builder: **57 triggers across 25 message structures** (A01–A62 excluding the A19 query event and reserved A56–A59), with segment order and the allowed segment set driven by **hl7apy's own From 388ec0953e044e052dbd20e5605b8775999120f0 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 11:41:12 -0500 Subject: [PATCH 63/66] docs(test-plan): re-point the first anchor off the blank-line worklist (BACKLOG #1100) ch07 cites the lens Steps backend at lens.py:270/285/1406/1423. :270 had drifted onto a BLANK LINE; the function is def _native_action_row( at :271. Re-pointed. Deliberately did NOT re-point the other three. They land on real code -- :285 an assignment inside a params loop, :1406 a 'return (', :1423 an 'if field is not None:' -- but all three are MID-FUNCTION rather than at a definition, and the row states no intent to check them against. Marked NOT VERIFIED, ONLY NOT OBVIOUSLY WRONG. That distinction is the whole limit of the blank-line scan and it is worth carrying in the document rather than only in the handoff: the scan finds anchors that drifted onto WHITESPACE and is blind to ones that drifted onto PLAUSIBLE CODE. Silently 'fixing' the other three to the nearest def would have manufactured precision I do not have. One anchor rather than several, on purpose: ~8 minutes remained before the projected window closed, and an uncommitted edit at cutoff is the one state worth avoiding. --- docs/testing/master-test-plan/07-config-wiring-and-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/07-config-wiring-and-cli.md b/docs/testing/master-test-plan/07-config-wiring-and-cli.md index f35bd65e5..86d6b3447 100644 --- a/docs/testing/master-test-plan/07-config-wiring-and-cli.md +++ b/docs/testing/master-test-plan/07-config-wiring-and-cli.md @@ -20,7 +20,7 @@ This chapter covers the **configuration bundle and the command-line surface that - **The `messagefoundry check` gate** — 5 **required** checks (`validate`, `dryrun`, `posture`, `build-check`, `reference-backend`) and 9 **advisory** checks (`cleartext-accepted`, `ruff`, `mypy`, `ruff-security`, `raise-fstring`, `accepts-candidate`, `dead-config`, `send-target`, `handler-security`), assembled in `run_checks` (`checks.py:109`, roster at `:132-188`); the ADR 0144 handler-security lint (`:863`) with `--strict-handler-security` / `--handler-security-allow`. - **`messagefoundry init` scaffold** (`scaffold.py:353/370`) including the generated adopter CI workflow. - **The full CLI surface** — the 30 subcommands in `_DISPATCH` (`__main__.py:4302`) and their flags. -- **The ADR 0076 typed action vocabulary** (`actions.py`, 15 pure `Message` helpers) and the **lens Steps backend** (`lens.py:270/285/1406/1423`) that the VS Code Steps view consumes. +- **The ADR 0076 typed action vocabulary** (`actions.py`, 15 pure `Message` helpers) and the **lens Steps backend** (`lens.py:271/285/1406/1423` — **`:270` RE-POINTED to `:271` 2026-08-15 (BACKLOG #1100): it had drifted onto a BLANK LINE, and `:271` is `def _native_action_row(`.** The other three land on real code (`:285` an assignment inside a params loop, `:1406` a `return (`, `:1423` an `if field is not None:`) but all three are MID-FUNCTION rather than at a definition, and this row states no intent to check them against — **so they are NOT VERIFIED, only NOT OBVIOUSLY WRONG**, which is the limit of a blank-line scan) that the VS Code Steps view consumes. **Explicitly NOT in scope here.** *Promotion, publishing, config reload and environment targeting belong to the **PUB** chapter* — `POST /config/reload`, the IDE Stage→Promote flow, the promote pre-flight and env-aware targeting are cross-referenced, never re-planned here. Also out of scope and owned elsewhere: From 17f5aa9c7952977ec1402e6e4c9eaf7de5e0f536 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 15 Aug 2026 11:53:19 -0500 Subject: [PATCH 64/66] docs(test-plan): second anchor off the blank-line worklist (BACKLOG #1100) ch07 cites EnvRef decode and bounded casts at wiring.py:195/226. :195 had drifted onto a BLANK LINE between the end of the EnvRef dataclass and the factory below it. Re-pointed to :197, which is def env(key, *, default=_UNSET, cast=...) -> EnvRef -- the definition carrying the 'cast' this row names. EnvRef's own cast field is :194. :226 NOT CHECKED and deliberately untouched. It lands on code, and the row states no intent I can test it against, so re-pointing it would manufacture precision I do not have. Same call as the three lens.py anchors in the previous commit, and for the same reason: a confident wrong anchor is worse than a visibly stale one, because a spot-check reads it as fine. Two anchors now off the 66-item worklist. Both were the whitespace kind, which is the only kind the scan can find -- the plan's remaining exposure is anchors that drifted onto PLAUSIBLE code, and no cheap signal exists for those. --- docs/testing/master-test-plan/07-config-wiring-and-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/07-config-wiring-and-cli.md b/docs/testing/master-test-plan/07-config-wiring-and-cli.md index 86d6b3447..277fc8964 100644 --- a/docs/testing/master-test-plan/07-config-wiring-and-cli.md +++ b/docs/testing/master-test-plan/07-config-wiring-and-cli.md @@ -15,7 +15,7 @@ This chapter covers the **configuration bundle and the command-line surface that - **Config-source trust** — POSIX group/world-writable + foreign-owner refusal (`wiring.py:4147`, `:4124`) and the Windows NTFS-DACL guard (`_evaluate_config_dacl:3889`, `_assert_safe_config_source_windows:3917`, ADR 0036), plus the `MEFOR_ALLOW_INSECURE_CONFIG_SOURCE` dev escape (`settings.py:244/247`). - **`connections.toml` config-as-data (ADR 0007)** — the read path (`connections_file.py:156/273`, `_TRANSPORTS:70`), the comment-preserving write path (`connections_edit.py:161/202/349/363`), the `connection list|upsert|remove|schema` CLI (`__main__.py:3788`), the `connection schema` form contract (`connection_schema.py:56`), and the console→TOML `flagged` write seam (ADR 0007 amendment). - **Service settings** — precedence CLI > env(`MEFOR_*`) > toml > default (`settings.py:4128`), the `MEFOR_
_` parser (`:3777`) against `_SECTIONS` (`:109`), the ADR 0118 `[security]` desugar + relocated-key rejection (`:3861`, `:3848`), secrets-in-file warning (`:3789`), and `extra="ignore"` tolerance (`ServiceSettings:3623`). -- **Environments + deferred `env()`** — `environments/.toml` + `MEFOR_VALUE_*` overlay (`environments.py`), `EnvRef` inline-table decode and bounded casts (`wiring.py:195/226`), fail-loud-at-build resolution (`wiring.py:563`). +- **Environments + deferred `env()`** — `environments/.toml` + `MEFOR_VALUE_*` overlay (`environments.py`), `EnvRef` inline-table decode and bounded casts (`wiring.py:197/226` — **`:195` RE-POINTED to `:197` 2026-08-15 (BACKLOG #1100): it had drifted onto a BLANK LINE between the end of the `EnvRef` dataclass and the factory.** `:197` is `def env(key, *, default=_UNSET, cast=...) -> EnvRef:`, which carries the `cast` this row names; `EnvRef`'s own `cast` field is `:194`. **`:226` NOT CHECKED and deliberately left alone** — it lands on code and I have no stated intent to test it against, so re-pointing it would manufacture precision I do not have), fail-loud-at-build resolution (`wiring.py:563`). - **Single project-root anchoring (ADR 0050)** — `anchor.py` (`resolve_project_root`, `anchor_under_root`, `graph_references_env`), the offline anchor trio on `validate`/`graph`/`dryrun`/`check` (`__main__.py:754/783`), the AC-3 scoped fail-loud (`:846`), and the drive-relative member guard. - **The `messagefoundry check` gate** — 5 **required** checks (`validate`, `dryrun`, `posture`, `build-check`, `reference-backend`) and 9 **advisory** checks (`cleartext-accepted`, `ruff`, `mypy`, `ruff-security`, `raise-fstring`, `accepts-candidate`, `dead-config`, `send-target`, `handler-security`), assembled in `run_checks` (`checks.py:109`, roster at `:132-188`); the ADR 0144 handler-security lint (`:863`) with `--strict-handler-security` / `--handler-security-allow`. - **`messagefoundry init` scaffold** (`scaffold.py:353/370`) including the generated adopter CI workflow. From 627b8dde1f782ac9b8c24308aa0036f096fab305 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Wed, 26 Aug 2026 10:54:27 -0500 Subject: [PATCH 65/66] docs(test-plan): re-point six decayed ci.yml anchors in the ch17 sharding row The BACKLOG #1100 audit's flagship withdrawn row (engine sharding CI coverage) cited ci.yml at :1348, :1551, :993, :1371, :1372, :1571 when written on 2026-08-15. ci.yml has grown roughly 490 lines underneath it since. Every anchor re-verified against the current file and re-pointed: :1836, :2039, :1407, :1859, :1860, :2059. The substantive claims (three named CI steps, gating on MEFOR_TEST_SQLSERVER/MEFOR_TEST_POSTGRES, the meta-gate in test_serverdb_ci_coverage.py) all still hold -- only the coordinates moved. Two of the six citations appear twice in the row (once in the clause-2 discussion, once in the evidence column); both occurrences of each are re-pointed. Co-Authored-By: Claude Opus 5 --- docs/testing/master-test-plan/17-performance-and-scale.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/testing/master-test-plan/17-performance-and-scale.md b/docs/testing/master-test-plan/17-performance-and-scale.md index 0c453dbd3..72aa208e9 100644 --- a/docs/testing/master-test-plan/17-performance-and-scale.md +++ b/docs/testing/master-test-plan/17-performance-and-scale.md @@ -108,7 +108,7 @@ Nothing below re-tests any of that; the rows below attack the **verdict**, the * |---|---|---|---|---| | The `--load` sustainable-rate verdict has no filling / backlog-slope term | A phase whose in-flight backlog grows through the whole hold passes `zero_loss` + eventual drain and reads as sustained. `report.py:503 _run_slos` checks only zero-loss, drain seconds, dead letters, dup rate — no slope, no latency divergence | Every published `--load` ceiling; every adopter sizing run; a cutover sized 3–5.5x over capacity drops clinical messages | **No.** The gate exists only on the co-located shardcert path (`shardcert.py:1758`); `shardcert_ladder.py:1092` documents its own absence on the two-box path in capitals | **P0** | | `/stats` poller-zero contamination | `enginepoll.py:556 await_drain` declares drained on `backlog == 0 and queue_depth == 0 and in_pipeline == 0` plus two equal `read`/`written` samples. A zeroed or frozen `/stats` under exactly the overload the gate exists for satisfies all four | A saturating, lossy run reports PASS and drained | No staleness precondition anywhere | **P0** | -| ~~Engine sharding has zero CI execution~~ **WITHDRAWN — FALSE. Corrected 2026-08-15 (BACKLOG #1100)** | **This row asserted a total coverage gap over work that has DEDICATED CI STEPS BUILT TO COVER IT, and it would have driven someone to build tests that already exist and already run.** Two of its three clauses are refuted: (1) "no workflow file contains the string `shard`" is **FALSE** — `ci.yml` carries it at least seven times, including the named steps "Run the engine-shard + statement-dispatch suites on real SQL Server" (`:1348`) and "Run the failover + engine-shard recovery suites on real Postgres" (`:1551`), plus `shard_recovery`/`shard_cert` in the changed-path gate at `:993` (positive control: `test` matched 24 of 24 workflow files, so the search was sound); (2) "named by no CI step, so they execute nowhere" is **FALSE** — all three are named at `ci.yml:1371`, `:1372`, `:1571`, and the steps set **exactly the variable the modules gate on** (`MEFOR_TEST_SQLSERVER: "1"` against `not os.getenv("MEFOR_TEST_SQLSERVER")`), so the gating makes them run rather than skip. The ci.yml comments record that these steps were ADDED to close this very gap ("before this step they were" dark), and `tests/test_serverdb_ci_coverage.py` now fails if a module-gated suite is added without being named — a meta-gate against this exact regression. **What SURVIVES, and it is real but far narrower than P0:** clause (3) holds — `tests/test_supervisor.py` uses `_FakeProcess` (11 references, docstring "without launching real engine subprocesses (Windows-safe)") with no `Popen`/`create_subprocess` anywhere, so the supervisor's process management is exercised only against a stand-in. **Re-scope to that: engine-shard RECOVERY is covered on both server backends; SUPERVISOR PROCESS HANDLING against a real subprocess is not** | ADR 0073 ownership-scoped recovery and single-delivery-consumer-per-lane are what keep per-lane FIFO and at-least-once correct on the only built multicore topology. A regression yields duplicate delivery, cross-engine-shard FIFO inversion, or permanently stranded rows. **THESE LAST THREE COLUMNS WERE WRITTEN FOR THE WITHDRAWN CLAIM AND I LEFT THEM STANDING WHEN I WITHDREW IT — corrected 2026-08-15 (BACKLOG #1100).** As published between those two edits the row CONTRADICTED ITSELF: the evidence column said the suites run in CI on both server backends, while the coverage column beside it still said "No" and the priority still said P0. The consequence above remains a true statement about what ADR 0073 protects; it is simply **no longer a consequence of THIS row**, because the recovery paths it names are the ones now shown to be exercised | **Was "No" — actually COVERED for engine-shard recovery** on real SQL Server and real Postgres (`ci.yml:1371`, `:1372`, `:1571`, each under a step that sets the gating `MEFOR_TEST_*` variable). **NOT covered: supervisor process handling against a real subprocess** (`test_supervisor.py` drives `_FakeProcess` only) | **NEEDS RE-GRADING, deliberately not re-graded here.** The P0 priced a total absence of engine-shard CI. The residual gap is narrower and I am not the right party to assign it a number — but **leaving P0 on a withdrawn finding is the worse error**, so it is flagged rather than silently carried | +| ~~Engine sharding has zero CI execution~~ **WITHDRAWN — FALSE. Corrected 2026-08-15 (BACKLOG #1100)** | **This row asserted a total coverage gap over work that has DEDICATED CI STEPS BUILT TO COVER IT, and it would have driven someone to build tests that already exist and already run.** Two of its three clauses are refuted: (1) "no workflow file contains the string `shard`" is **FALSE** — `ci.yml` carries it at least seven times, including the named steps "Run the engine-shard + statement-dispatch suites on real SQL Server" (`:1836`) and "Run the failover + engine-shard recovery suites on real Postgres" (`:2039`), plus `shard_recovery`/`shard_cert` in the changed-path gate at `:1407` (positive control: `test` matched 24 of 24 workflow files, so the search was sound); (2) "named by no CI step, so they execute nowhere" is **FALSE** — all three are named at `ci.yml:1859`, `:1860`, `:2059`, and the steps set **exactly the variable the modules gate on** (`MEFOR_TEST_SQLSERVER: "1"` against `not os.getenv("MEFOR_TEST_SQLSERVER")`), so the gating makes them run rather than skip. The ci.yml comments record that these steps were ADDED to close this very gap ("before this step they were" dark), and `tests/test_serverdb_ci_coverage.py` now fails if a module-gated suite is added without being named — a meta-gate against this exact regression. **What SURVIVES, and it is real but far narrower than P0:** clause (3) holds — `tests/test_supervisor.py` uses `_FakeProcess` (11 references, docstring "without launching real engine subprocesses (Windows-safe)") with no `Popen`/`create_subprocess` anywhere, so the supervisor's process management is exercised only against a stand-in. **Re-scope to that: engine-shard RECOVERY is covered on both server backends; SUPERVISOR PROCESS HANDLING against a real subprocess is not** | ADR 0073 ownership-scoped recovery and single-delivery-consumer-per-lane are what keep per-lane FIFO and at-least-once correct on the only built multicore topology. A regression yields duplicate delivery, cross-engine-shard FIFO inversion, or permanently stranded rows. **THESE LAST THREE COLUMNS WERE WRITTEN FOR THE WITHDRAWN CLAIM AND I LEFT THEM STANDING WHEN I WITHDREW IT — corrected 2026-08-15 (BACKLOG #1100).** As published between those two edits the row CONTRADICTED ITSELF: the evidence column said the suites run in CI on both server backends, while the coverage column beside it still said "No" and the priority still said P0. The consequence above remains a true statement about what ADR 0073 protects; it is simply **no longer a consequence of THIS row**, because the recovery paths it names are the ones now shown to be exercised | **Was "No" — actually COVERED for engine-shard recovery** on real SQL Server and real Postgres (`ci.yml:1859`, `:1860`, `:2059`, each under a step that sets the gating `MEFOR_TEST_*` variable). **NOT covered: supervisor process handling against a real subprocess** (`test_supervisor.py` drives `_FakeProcess` only) | **NEEDS RE-GRADING, deliberately not re-graded here.** The P0 priced a total absence of engine-shard CI. The residual gap is narrower and I am not the right party to assign it a number — but **leaving P0 on a withdrawn finding is the worse error**, so it is flagged rather than silently carried | | ~~Published sizing contradicts the measured record and itself~~ **WITHDRAWN — STALE. Corrected 2026-08-15 (BACKLOG #1100)** | **CORRECTED TWICE — my first withdrawal reason ("the document was fixed") was itself unverified, and the true verdict is MIS-CITATION rather than staleness.** Measured 2026-08-15 with `git log -S`: the phrase "low-thousands" **never appears in `docs/SYSTEM-REQUIREMENTS.md` at any point in that path's recorded history**, so nothing was removed from it. The phrase **did** exist elsewhere in the repo before this plan was written (`5fa6db9f`, `abedf865`, and `39990f8c` "bring the documentation set back in line with the code"), and today it survives in `docs/` **only in this row** — while the file's current `~165 msg/s` tier table arrived via `959821a7` (#79, 2026-07-30). **So the over-claim was real in SOME document and this row attributes it to a file that did not contain it; whether it was ever live in THIS file is UNDETERMINED and my path-scoped query cannot settle it.** A mis-citation is worse than staleness: staleness is blameless drift, whereas a quotation that cannot be sourced to the file it names discredits the row that carries it. *Recorded at length because the correction took three passes — the first two were "stale" and "never existed", and both were reached faster and on thinner evidence than I would have accepted for an assertion.* The "~500 – low-thousands msg/s / ~40M+/day" tier **no longer exists in `SYSTEM-REQUIREMENTS.md`**. The current tier table (`:216-219`) tops out at **~165 msg/s at 4 engine shards**, and `:219` now carries the caveat inline — "on per-shard SQLite — a topology that is not supported above one shard". The cited anchors have also drifted: `:163` is **blank**, and `:195-198` is bench/active-passive guidance, not tiers. The alleged self-contradiction cannot be reproduced either: **"future direction, not built" appears NOWHERE in `docs/`**, so there is no counterpart to the `is built` statements (which are real, at `:184`, `:187`, `:193`, `:297`). **The "7.23x short" arithmetic is therefore moot — but flagging it anyway, because ~500/~72 is 6.94x, not 7.23x, so the multiplier did not derive from the two numbers the row put beside it.** What is still worth keeping from this row is its LAST column — **but its SUPPORT is wrong, and I carried that forward unchecked before auditing it.** The **conclusion** stands: no doc-vs-measurement guard exists, so nothing would have caught the over-claim while it was live and nothing will catch the next one. The **enumeration** behind it does not: `tests/test_feature_map_claims.py` does not check "only the ASVS score, private-path links and superseded docs" — it has **six** tests, the three named plus `test_link_check_catches_a_missing_target_without_a_denylist`, `test_feature_map_is_actually_published` and `test_asvs_row_keeps_its_framing`. **None of the six compares a PUBLISHED number against a MEASURED one, which is why the conclusion survives the correction** — but a reader who quotes the enumeration to argue the file is nearly empty would be wrong, and the enumeration is the part that gets quoted | The document an adopter sizes hardware from. Order-of-magnitude over-claim + a self-contradiction about whether the scale-out path exists | No doc-vs-measurement guard exists — **TRUE, and the reason it survives is that none of `test_feature_map_claims.py`'s tests compares a PUBLISHED figure against a MEASURED one.** But the enumeration originally here was wrong and is corrected: the file does **not** check "only the ASVS score, private-path links and superseded docs" — it has **six** tests, adding `test_link_check_catches_a_missing_target_without_a_denylist`, `test_feature_map_is_actually_published` and `test_asvs_row_keeps_its_framing` | **NEEDS RE-GRADING — the finding above is WITHDRAWN as stale, so this P0 no longer prices anything live.** Flagged rather than re-graded, for the same reason as the row above: I am not the party to assign the number, but a P0 left standing on a withdrawn row reads as an open defect forever | | The one published multi-process scale-out number was measured on a now-forbidden topology | `docs/benchmarks/TUNING-BASELINE.md:150` records the η ≈ 0.85 / E_core ≈ 42 msg/s **engine-sharding** result on the store line "**per-shard SQLite** (one store file per shard — **no shared DB**, no shared-DB commit contention by design)" (quoted verbatim from the source table), measured 2026-06-27. **RE-VERIFIED 2026-08-15 (BACKLOG #1100) — CONFIRMED, and the anchor is EXACT**, which is worth stating because the two rows above it in this same chapter were withdrawn as false and stale: `TUNING-BASELINE.md:150` is verbatim that store line, and `E_core ≈ 42 msg/s/shard` / `η ≈ 0.85` sit at `:161-162`. **One qualification to the row's framing: the topology is now DISCLOSED in both published files, so this is a reproducibility problem rather than a hidden-caveat one.** `TUNING-BASELINE.md:165-171` already separates the non-portable absolute from the portable "speedup shape", and `SYSTEM-REQUIREMENTS.md:219` names the unsupported topology inline. ADR 0063 (2026-07-01) and `sharding.py:81 require_unified_store` now **refuse** >1 engine shard on any non-server backend | The only published **engine-sharding** speedup — the shape adopters are told to multiply by their own `E_core` — is not reproducible with the shipped code, and it was measured with the shared-store contention deliberately absent | No | **P0** | | No throughput/latency regression gate anywhere | `.github/workflows/benchmark.yml` is `workflow_dispatch`-only, and each run wraps the harness in `set +e` … `set -e` so **even the harness's own SLO exit code is discarded**. Numbers are hand-transcribed into TUNING-BASELINE. `--baseline`/`--tolerance` exist (`report.py:753`) but are invoked by no workflow. `docs/CI-QUALITY.md` has no perf section | A 5x throughput or p99 regression from a claim-path, store or connector change merges green and is found on the next manual rig run | No | **P1** | From 0467580b6f62b8eed5ea379a58625be57c7cbe2d Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Wed, 26 Aug 2026 10:56:03 -0500 Subject: [PATCH 66/66] backlog: record #1100's progress without claiming it is finished (BACKLOG #1100) Lander-authored per ADR 0165: the branch this landed from (stranded PR 433) had its own progress note at 394e2a32e, but that commit is builder-authored ledger content, which the owner's 2026-08-13 ruling reserves to the Dispatcher or Lander. Re-authored here rather than cherry-picked, with the counts independently re-verified against the actual landed diff (18 files, +90/-75, matching the branch's own claim exactly) rather than carried forward unverified. The item stays OPEN: nobody has verified the nine sites it names are among the revised chapters, which the note says plainly. Co-Authored-By: Claude Opus 5 --- docs/BACKLOG.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 83bef1c4d..31ff6a3d6 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -6588,6 +6588,18 @@ contention; recorded in the #1095 handoff note rather than lost. ## 1100. The master test plan asserts document contradictions that were resolved before it was written +> **PROGRESS 2026-08-26 (lander), NOT A CLOSURE -- this item stays OPEN.** Landed the documentation +> half of stranded PR #433 on a fresh branch (the original ref carried an owner-armed auto-merge that +> would have fused it to six unrelated engine commits; disarmed and split per the owner's 2026-08-26 +> ruling, ADR 0165 authorship note below). Measured directly against this landed diff, not carried +> forward from the branch's own claim: 18 chapter files, +90/-75, matching the branch's own count +> exactly. **WHAT IS NOT ESTABLISHED, and it is why this stays open: nobody has verified that the NINE +> SITES THIS ITEM NAMES are among those revised.** Closing needs a pass that walks those nine sites +> and checks each against the now-landed chapters. **Author's note, ADR 0165:** this progress note is +> Lander-authored on the builder's behalf -- the 64 underlying commits are the builder's own verified +> work, cherry-picked unmodified; only this ledger paragraph is mine, because ledger authorship on an +> open item is reserved to the Dispatcher or Lander. +> > 🔢 **Re-scored 2026-08-20 -> P2.** Value **5/10** · Difficulty **3/10** · _fill-in_. The stale claims are not confined to narrative: :593 is item 4 of the chapter's numbered release sign-off list (the three contradicting documents agree with the code) and :606 declares Blocks: ALERT-08, 09, 58, 67 as the chapter's only P0 with a real security consequence, so a plan executor is held on a question the code answered on 2026-08-02. Difficulty is nine content edits, each requiring the claim re-checked against code first, at least one possibly warranting deletion, with the item forbidding a scripted pass. _(was 6/10 · 3/10.)_ > > **Filed 2026-08-07 - not started.** Nine sites in the master test