diff --git a/docs/CI-FAILURE-LOG.md b/docs/CI-FAILURE-LOG.md index cd97fe3f4..da2c193a5 100644 --- a/docs/CI-FAILURE-LOG.md +++ b/docs/CI-FAILURE-LOG.md @@ -67,7 +67,7 @@ which is which, is worse than either alone. | 2026-08-26 22:56 | 625 | `sql server (store + connector) 2022`, reaching `CI gate` through its `needs:` | `test_adr0070_4_fifo_head_first_after_stop[sqlserver]` ERROR on a 60s `pytest-timeout`, hung in async fixture setup; `test_adr0070_3b_streak_resets_on_forward_progress[sqlserver]` FAILED | `flake` | **Queue branch only.** The leg RAN and PASSED on 625's own head (268s, success). I checked for a real merge-result interaction rather than assuming: main's tip was 619, which touches only `transports/wincred.py`, a **Windows** connector, while this job runs on Ubuntu under docker where that module cannot load. 625 adds five lines to `config/settings.py`. No mechanism connects either to `test_stage_dispatcher.py`. Third attempt passed and 625 merged at `50300e583`. | | 2026-08-26 22:25 | 625 | `load test (smoke, sqlserver) 2025`, reaching `CI gate` through its `needs:` | `Sqlcmd: Login timeout expired` / `TCP Provider: Error code 0x2749` / `Server is not found or not accessible` | `infra` | SQL Server container never became reachable. Distinct from the 22:56 row: **different job, different symptom** (container absent vs container healthy and slow). Two unlike failures across two attempts is the signature that made a third attempt worth taking. | | 2026-08-26 21:39 | 627, 626 | `sql server (store + connector) 2022`, reaching `CI gate` | `apt-get failed 3 times` on the Ubuntu runner mirror | `infra` | The workflow's own error text names it: *"This is the UBUNTU RUNNER MIRROR, not the change under test."* Both PRs were green at their heads. Evicted from the queue with no code defect. | -| 2026-08-26 21:31 | 618 | `test (windows-2025, py3.14)` | `empty_claims_monotonic` -- `fixed_per_conn@N=24` below `prior * 0.75` | `flake` | Same assert as the rows below, on the queue branch. **Rate now measured: 19% per RUN, 10.7% per LEG across 21 real windows-2022 runs / 84 legs; the two reconcile as 1-(1-0.107)^2 = 20.3% vs 19.0% measured.** builder-2's truncation analysis puts the mean ratio at 1.05 with sd 0.243 against a 0.25 band -- **the band is the defect, not the direction**; all 9 failures sit in 9 distinct runs, where a wrong direction would fail both legs together. | +| 2026-08-26 21:31 | 618 | `test (windows-2025, py3.14)` | `empty_claims_monotonic` -- `fixed_per_conn@N=24` below `prior * 0.75` | `flake` | Same assert as the rows below, on the queue branch. **Rate now measured: 19% per RUN, 10.7% per LEG across 21 real windows-2022 runs / 84 legs; the two reconcile as 1-(1-0.107)^2 = 20.3% vs 19.0% measured.** ~~builder-2's truncation analysis puts the mean ratio at 1.05 with sd 0.243 against a 0.25 band -- the band is the defect, not the direction~~ **CORRECTED 2026-08-28: THOSE FIGURES ARE RETRACTED. BACKLOG #1211 withdrew them the same day they were written -- the truncation estimate inverted the censoring under a single-normal assumption, and #1211 states the model is misspecified and its numbers should not be used. The DIRECTION conclusion survived; the mechanism and every number did not. Read #1211 for the current model rather than any figure restated here -- it is under active review and this log should not carry a second copy of it.**; all 9 failures sit in 9 distinct runs, where a wrong direction would fail both legs together. | | 2026-08-26 22:40 | several | the queue itself | I reported that 627 and 626 were "green on all 15 required contexts and were evicted anyway", and attributed five evictions to a `docs/BACKLOG.md` tail-append collision | `instrument` | **Wrong, and broadcast to seven mailboxes; three seats repeated it back.** They were green on their PR **heads** and red on their **queue branches** -- separate shas, separate run sets. Zero of the five evictions was a collision. The `git merge-tree` collision result is real as a *mechanism* and was never evidence about what *happened*. Corrected in place at the claim. **Check `gh run list --event merge_group` before calling a PR landable.** | | 2026-08-26 22:48 | 630 | `test (windows-2022, py3.14)` | I computed the connscale failure rate as 100% (3 of 3), having published 22% (4 of 18) shortly before | `instrument` | **Both wrong; the figure is 19% (4 of 21).** The 22% denominator included a path-filtered no-op leg. The 100% came from classifying "did this leg run the test?" by grepping the log for `connscale` -- **pytest only prints a test name when it FAILS**, so the detector could only ever see failures and the sample became the condition. The tell I read past: 19 legs the classifier called "did not run" had durations of 449-565 seconds. | | 2026-08-26 19:20 | all | the merge queue itself | PR 619 sat `AWAITING_CHECKS` 40 min with `main` unmoved; **32 check-runs green on the queue branch, 0 failing, 0 pending** | `gate-artifact` | The three `CodeQL` contexts were made required, and `codeql.yml` has **no `merge_group` trigger** -- PR 616 added one to `ci.yml`, `backlog-hygiene.yml`, `cla.yml`, `security.yml` and not to codeql. A merge_group event cannot start it, so those contexts can never report on a queue branch and **every entry waits forever**. `security.yml`'s own `on:` block predicted this verbatim in the PR that added the other four. Fix: PR 629. | diff --git a/docs/testing/master-test-plan/17-performance-and-scale.md b/docs/testing/master-test-plan/17-performance-and-scale.md index 72aa208e9..fc359d0ec 100644 --- a/docs/testing/master-test-plan/17-performance-and-scale.md +++ b/docs/testing/master-test-plan/17-performance-and-scale.md @@ -73,7 +73,7 @@ measured record. | `tests/test_load_profile.py`, `tests/test_load_config.py` | Profile schema fails loud on a typo'd key; shipped profiles and the load config carry no real partner/site/host tokens | | `tests/test_load_failover_postgres.py` + `test_load_failover_sqlserver.py` as steps in `postgres store` (ci.yml:803) and `sql server (store + connector)` (ci.yml:658) | Two-node SIGKILL-under-load on real server DBs: no acknowledged loss, no split-brain, bounded dups, promotion observed, per-lane FIFO 0 inversions | | `tests/test_load_failover_unit.py`, `tests/_failover_load_support.py` | Failover orchestrator logic offline — lease timing invariants, kill fraction, verdict composition | -| `tests/test_connscale_smoke.py`; `tests/test_connscale_postgres.py` at `MEFOR_STORE_POOL_SIZE=4` (ci.yml:853) | In-process connection-scale N=12→24 on SQLite (no-loss, FD + empty-claim monotonicity, executor shim, reload probe); pool-acquire-wait wall on a real server DB with a forced tiny pool | +| `tests/test_connscale_smoke.py`; `tests/test_connscale_postgres.py` at `MEFOR_STORE_POOL_SIZE=4` (ci.yml:853) | In-process connection-scale N=12→24 on SQLite (no-loss, FD + empty-claim monotonicity **[COVERAGE PIN, added 2026-08-28: the empty-claim monotonicity assertion in `test_connscale_smoke.py` is under active review as a known-noisy leg. IT IS CORRECT TODAY. IF IT IS DISARMED, SKIPPED OR DELETED, THIS ROW MUST BE EDITED IN THE SAME CHANGE -- otherwise this plan keeps asserting coverage that nothing provides, and the disarm looks complete while the false claim survives here.]**, executor shim, reload probe); pool-acquire-wait wall on a real server DB with a forced tiny pool | | `tests/test_connscale_{profile,config,report,driver,cpu_probe,compare,batch,fuse,fuse_replay}.py` | Profile parsing, report shape, driver pacing, the BACKLOG #220 same-PID-set CPU fold, A/B compare, batch/fuse arm wiring | | `tests/test_estate_{driver,profile,shape}.py` | Event-rate calibration (hubs driven slower than simples), the rate identity, fraction/fan-out bounds, heterogeneous graph shape — **unit level only** | | `tests/test_multishard_smoke.py` | Two real `serve` subprocesses on ONE SQLite store: orchestration mechanics + no cross-engine lane steal (`foreign_rows == 0`) |