Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions docs/BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16843,3 +16843,37 @@ four branches returning the same text would pass every other assertion in the bl
**Driven against the PURE verdict function with injected hashes**, never by mutating the installed
gate: that file is machine-global and every PreToolUse hook on this box reads it, so a test may not
take it out from under a concurrent session. Same reasoning the existing negative control gives.

## 1366. the four connscale fields that discriminate the surviving failure hypotheses never escape the test job

> 🔢 **Filed 2026-08-26 (builder 2) - BUILT IN THIS COMMIT, not yet landed.** Successor in subject to [#1211](#1211-empty_claims_per_msg-is-not-contention-immune-the-ratio-form-excursions-past-its-own-slo-band-on-a-hosted-runner), whose limb one made the RATIO survive a passing run; this makes the readings that EXPLAIN the ratio survive one too.
> Verdict: build
> Closing-act: code

**Cluster:** Developer Experience & CI. **Priority:** P2. **Verdict:** build.
**Severity:** no engine effect, no PHI axis, no deployment axis (sec. 0). The cost is diagnostic: a connscale failure on a hosted runner cannot be attributed from CI alone, so each occurrence burns a full cycle and ends in a re-run rather than a cause.

**What:** `tests/test_connscale_smoke.py` runs inside the `test` job. **That job uploads no artifacts** -- ci.yml's three `upload-artifact` steps are in `load-test` (:2162), `load-test-sqlserver` (:2331) and `windows-service-smoke` (:2521). So every reading on a `ConnScaleRecord` that is not printed dies with the job. #1211 limb one made ONE metric survive, `empty_claims_per_msg`, via `$GITHUB_STEP_SUMMARY`. The fields that say WHICH explanation is right were not part of that scope.

**The four the investigation actually turns on**, measured absent from the emission on `origin/main`:

```
drain_seconds 0 reload_seconds 0
fd_probe_ticks 0 fd_probe_degraded_ticks 0
cpu_util_cores_mean 0
```

* **drain-tail vs reload-probe separate ONLY on `drain_seconds` against `reload_seconds`.**
* **contention vs probe-cost separate ONLY on the FD probe's tick counts** -- `fd_probe_degraded_ticks` non-zero means the walk could not measure; zero means it measured cleanly and a wrong reading is a wrong SUBJECT rather than a failed sample. That distinction is what a live investigation into the connscale FD gauge turned on, and it was unavailable from CI.

**THE DESIGN CONSTRAINT THAT DECIDES THE SHAPE, and it is why this is a second renderer rather than a parameter on the first.** Only **two** metrics have a monotonic SLO -- `empty_claims_monotonic` and `fd_count_monotonic`. Every field above has **no band**. `render_readings_markdown` emits `prior` / `band floor` / `margin`, which for a band-less field would be **a threshold computed from whichever reading happened to precede it** -- false precision manufactured by the renderer and indistinguishable, in a job summary, from a measured one. So the table carries no band, no threshold and no verdict column, and says so in its own preamble.

**Emitted from the FIXTURE, before any assertion**, for the same reason #1211's readings are: a field recorded only on failure cannot establish its own normal range. That is the selection bias #1211 exists to fix, one metric family over.

**`None` renders as a dash and never as `0`.** "the probe did not measure" and "the probe measured zero" are different verdicts, and telling them apart is the whole purpose of `fd_probe_degraded_ticks`.

**Not in scope, deliberately:** artifact upload (the step-summary channel already reaches every run and needs no ci.yml change), and any change to the `empty_claims_per_msg` band -- that is #1211 limb two and it stays blocked until samples exist.

**Related:** #1211 (limb one shipped the channel this reuses; limb two blocked), #1101 (the per-message form these annotate), #320 (windows-2025 leg timing, one of the explanations these fields separate).

**Source:** scoped by the Dispatcher, corrected by this lane's scouting -- the original scoping described artifact-upload work already done and a selection bias already fixed by #1211 limb one; what remained was the band-less fields and the constraint above.
128 changes: 128 additions & 0 deletions harness/load/connscale/report.py
Original file line number Diff line number Diff line change
Expand Up @@ -361,6 +361,61 @@ def monotonic_pairs(
return pairs


@dataclass(frozen=True)
class DiagnosticField:
"""One reading emitted for DIAGNOSIS, with no band and therefore no verdict.

Deliberately separate from :class:`MonotonicPair`. A pair carries `prior`, `threshold` and an `ok`
flag because its metric HAS an SLO band; these fields do not. Rendering a floor for a band-less
field would print a threshold computed from an adjacent reading -- a number that looks measured and
is manufactured by the renderer. Two shapes, because there are two kinds of reading.
"""

label: str
read: Callable[[ConnScaleRecord], object]
#: Why this field is here -- which competing explanation it separates. Emitted in the table's
#: preamble so a reader meeting it in a job summary knows what it is FOR, not just what it is.
discriminates: str


#: The band-less readings emitted on every run (BACKLOG #1366).
#:
#: THESE ARE EXACTLY THE FIELDS THAT SEPARATE THE SURVIVING EXPLANATIONS for a connscale failure, which
#: is why the set is small and named rather than "everything on the record". Without them a failure is
#: permanently undiagnosable: the ratio says THAT something moved, and nothing says WHICH.
#:
#: Single definition -- the emitter and its tests both read this tuple, so a field added here is
#: covered without editing a second list.
DIAGNOSTIC_FIELDS: tuple[DiagnosticField, ...] = (
DiagnosticField(
"drain_seconds",
lambda r: r.drain_seconds,
"drain-tail vs reload-probe: these two separate ONLY on drain_seconds against reload_seconds",
),
DiagnosticField(
"reload_seconds",
lambda r: r.reload_seconds,
"the other half of that pair; None means the reload probe did not measure this step",
),
DiagnosticField(
"fd_probe_ticks",
lambda r: r.fd_probe_ticks,
"how many intervals the FD walk actually sampled -- a low count is a coarse gauge, not a fault",
),
DiagnosticField(
"fd_probe_degraded_ticks",
lambda r: r.fd_probe_degraded_ticks,
"contention vs probe-cost: NON-ZERO means the walk could not measure, ZERO means it measured "
"cleanly and any wrong reading is a wrong SUBJECT rather than a failed sample",
),
DiagnosticField(
"cpu_util_cores_mean",
lambda r: r.cpu_util_cores_mean,
"how loaded the box was across the hold, which is the input every contention story needs",
),
)


@dataclass(frozen=True)
class ConnScaleReport:
profile: str
Expand Down Expand Up @@ -557,6 +612,79 @@ def readings_payload(
"readings": readings,
}

def render_diagnostics_markdown(
self,
fields: tuple[DiagnosticField, ...] = DIAGNOSTIC_FIELDS,
*,
context: dict[str, str] | None = None,
max_rows: int = _MAX_SUMMARY_ROWS,
) -> str:
"""The band-less diagnostic table, emitted on every run (BACKLOG #1366).

NOT a variant of :meth:`render_readings_markdown`, and the separation is the point. That one
renders `prior`, `band floor` and `margin` because its metric has an SLO band. ONLY
``empty_claims_monotonic`` and ``fd_count_monotonic`` have one; every field here has none, so
those columns would be a floor computed from whichever reading happened to precede it --
false precision manufactured by the renderer rather than measured by anything.

So this table carries NO verdict column and NO threshold. It says what was observed and what
each field is FOR, and leaves the judgement to a reader who has the competing explanations in
front of them.

Pure: returns text, writes nothing. Row count capped for the same reason as the banded table --
an oversized ``$GITHUB_STEP_SUMMARY`` write is dropped in full rather than trimmed.
"""
head = ["### connscale diagnostics (no band, no verdict)"]
ctx = {
"profile": self.profile,
"db_backend": self.db_backend or "sqlite",
**(context or {}),
}
head.append("")
head.append(" | ".join(f"{k}: {v}" for k, v in ctx.items()))
head.append("")
head.append(
"Recorded on every run, pass or fail. **None of these has an SLO band**, so there is no "
"threshold here and nothing below is a verdict -- they are the readings that separate "
"competing explanations for a failure (BACKLOG #1366)."
)
head.append("")
for f in fields:
head.append(f"- `{f.label}` -- {f.discriminates}")
head.append("")

if not self.records:
head.append("No record was produced by this run.")
return "\n".join(head) + "\n"

head.append("| lane | N | " + " | ".join(f.label for f in fields) + " |")
head.append("|---|---|" + "---|" * len(fields))

rows: list[str] = []
dropped = 0
for r in sorted(self.records, key=lambda r: (r.sweep_mode, r.claim_mode, r.count)):
if len(rows) >= max_rows:
dropped += 1
continue
cells = []
for f in fields:
v = f.read(r)
# `None` renders as an explicit dash, NEVER as 0. "the probe did not measure" and
# "the probe measured zero" are different verdicts and the whole point of these
# fields is telling them apart.
cells.append("-" if v is None else (f"{v:.4g}" if isinstance(v, float) else str(v)))
rows.append(
f"| {lane_label(r.sweep_mode, r.claim_mode)} | {r.count} | "
+ " | ".join(cells)
+ " |"
)

out = head + rows
if dropped:
out.append("")
out.append(f"{dropped} further row(s) not shown: capped at {max_rows}.")
return "\n".join(out) + "\n"

def to_csv(self) -> str:
"""One row per (sweep_mode, N) step — for spreadsheet curve plotting."""
buf = io.StringIO()
Expand Down
Loading
Loading