Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/CI-FAILURE-LOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ which is which, is worse than either alone.
| 2026-08-26 22:56 | 625 | `sql server (store + connector) 2022`, reaching `CI gate` through its `needs:` | `test_adr0070_4_fifo_head_first_after_stop[sqlserver]` ERROR on a 60s `pytest-timeout`, hung in async fixture setup; `test_adr0070_3b_streak_resets_on_forward_progress[sqlserver]` FAILED | `flake` | **Queue branch only.** The leg RAN and PASSED on 625's own head (268s, success). I checked for a real merge-result interaction rather than assuming: main's tip was 619, which touches only `transports/wincred.py`, a **Windows** connector, while this job runs on Ubuntu under docker where that module cannot load. 625 adds five lines to `config/settings.py`. No mechanism connects either to `test_stage_dispatcher.py`. Third attempt passed and 625 merged at `50300e583`. |
| 2026-08-26 22:25 | 625 | `load test (smoke, sqlserver) 2025`, reaching `CI gate` through its `needs:` | `Sqlcmd: Login timeout expired` / `TCP Provider: Error code 0x2749` / `Server is not found or not accessible` | `infra` | SQL Server container never became reachable. Distinct from the 22:56 row: **different job, different symptom** (container absent vs container healthy and slow). Two unlike failures across two attempts is the signature that made a third attempt worth taking. |
| 2026-08-26 21:39 | 627, 626 | `sql server (store + connector) 2022`, reaching `CI gate` | `apt-get failed 3 times` on the Ubuntu runner mirror | `infra` | The workflow's own error text names it: *"This is the UBUNTU RUNNER MIRROR, not the change under test."* Both PRs were green at their heads. Evicted from the queue with no code defect. |
| 2026-08-26 21:31 | 618 | `test (windows-2025, py3.14)` | `empty_claims_monotonic` -- `fixed_per_conn@N=24` below `prior * 0.75` | `flake` | Same assert as the rows below, on the queue branch. **Rate now measured: 19% per RUN, 10.7% per LEG across 21 real windows-2022 runs / 84 legs; the two reconcile as 1-(1-0.107)^2 = 20.3% vs 19.0% measured.** builder-2's truncation analysis puts the mean ratio at 1.05 with sd 0.243 against a 0.25 band -- **the band is the defect, not the direction**; all 9 failures sit in 9 distinct runs, where a wrong direction would fail both legs together. |
| 2026-08-26 21:31 | 618 | `test (windows-2025, py3.14)` | `empty_claims_monotonic` -- `fixed_per_conn@N=24` below `prior * 0.75` | `flake` | Same assert as the rows below, on the queue branch. **Rate now measured: 19% per RUN, 10.7% per LEG across 21 real windows-2022 runs / 84 legs; the two reconcile as 1-(1-0.107)^2 = 20.3% vs 19.0% measured.** ~~builder-2's truncation analysis puts the mean ratio at 1.05 with sd 0.243 against a 0.25 band -- the band is the defect, not the direction~~ **CORRECTED 2026-08-28: THOSE FIGURES ARE RETRACTED. BACKLOG #1211 withdrew them the same day they were written -- the truncation estimate inverted the censoring under a single-normal assumption, and #1211 states the model is misspecified and its numbers should not be used. The DIRECTION conclusion survived; the mechanism and every number did not. Read #1211 for the current model rather than any figure restated here -- it is under active review and this log should not carry a second copy of it.**; all 9 failures sit in 9 distinct runs, where a wrong direction would fail both legs together. |
| 2026-08-26 22:40 | several | the queue itself | I reported that 627 and 626 were "green on all 15 required contexts and were evicted anyway", and attributed five evictions to a `docs/BACKLOG.md` tail-append collision | `instrument` | **Wrong, and broadcast to seven mailboxes; three seats repeated it back.** They were green on their PR **heads** and red on their **queue branches** -- separate shas, separate run sets. Zero of the five evictions was a collision. The `git merge-tree` collision result is real as a *mechanism* and was never evidence about what *happened*. Corrected in place at the claim. **Check `gh run list --event merge_group` before calling a PR landable.** |
| 2026-08-26 22:48 | 630 | `test (windows-2022, py3.14)` | I computed the connscale failure rate as 100% (3 of 3), having published 22% (4 of 18) shortly before | `instrument` | **Both wrong; the figure is 19% (4 of 21).** The 22% denominator included a path-filtered no-op leg. The 100% came from classifying "did this leg run the test?" by grepping the log for `connscale` -- **pytest only prints a test name when it FAILS**, so the detector could only ever see failures and the sample became the condition. The tell I read past: 19 legs the classifier called "did not run" had durations of 449-565 seconds. |
| 2026-08-26 19:20 | all | the merge queue itself | PR 619 sat `AWAITING_CHECKS` 40 min with `main` unmoved; **32 check-runs green on the queue branch, 0 failing, 0 pending** | `gate-artifact` | The three `CodeQL` contexts were made required, and `codeql.yml` has **no `merge_group` trigger** -- PR 616 added one to `ci.yml`, `backlog-hygiene.yml`, `cla.yml`, `security.yml` and not to codeql. A merge_group event cannot start it, so those contexts can never report on a queue branch and **every entry waits forever**. `security.yml`'s own `on:` block predicted this verbatim in the PR that added the other four. Fix: PR 629. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ measured record.
| `tests/test_load_profile.py`, `tests/test_load_config.py` | Profile schema fails loud on a typo'd key; shipped profiles and the load config carry no real partner/site/host tokens |
| `tests/test_load_failover_postgres.py` + `test_load_failover_sqlserver.py` as steps in `postgres store` (ci.yml:803) and `sql server (store + connector)` (ci.yml:658) | Two-node SIGKILL-under-load on real server DBs: no acknowledged loss, no split-brain, bounded dups, promotion observed, per-lane FIFO 0 inversions |
| `tests/test_load_failover_unit.py`, `tests/_failover_load_support.py` | Failover orchestrator logic offline — lease timing invariants, kill fraction, verdict composition |
| `tests/test_connscale_smoke.py`; `tests/test_connscale_postgres.py` at `MEFOR_STORE_POOL_SIZE=4` (ci.yml:853) | In-process connection-scale N=12→24 on SQLite (no-loss, FD + empty-claim monotonicity, executor shim, reload probe); pool-acquire-wait wall on a real server DB with a forced tiny pool |
| `tests/test_connscale_smoke.py`; `tests/test_connscale_postgres.py` at `MEFOR_STORE_POOL_SIZE=4` (ci.yml:853) | In-process connection-scale N=12→24 on SQLite (no-loss, FD + empty-claim monotonicity **[COVERAGE PIN, added 2026-08-28: the empty-claim monotonicity assertion in `test_connscale_smoke.py` is under active review as a known-noisy leg. IT IS CORRECT TODAY. IF IT IS DISARMED, SKIPPED OR DELETED, THIS ROW MUST BE EDITED IN THE SAME CHANGE -- otherwise this plan keeps asserting coverage that nothing provides, and the disarm looks complete while the false claim survives here.]**, executor shim, reload probe); pool-acquire-wait wall on a real server DB with a forced tiny pool |
| `tests/test_connscale_{profile,config,report,driver,cpu_probe,compare,batch,fuse,fuse_replay}.py` | Profile parsing, report shape, driver pacing, the BACKLOG #220 same-PID-set CPU fold, A/B compare, batch/fuse arm wiring |
| `tests/test_estate_{driver,profile,shape}.py` | Event-rate calibration (hubs driven slower than simples), the rate identity, fraction/fan-out bounds, heterogeneous graph shape — **unit level only** |
| `tests/test_multishard_smoke.py` | Two real `serve` subprocesses on ONE SQLite store: orchestration mechanics + no cross-engine lane steal (`foreign_rows == 0`) |
Expand Down
Loading