feat(validation): report celeris's engine error classes, six in the series and all twelve in the tally

[FumingPower3925]

Wires the validation tier to celeris's new engine error classes, so the next nightly can name the branch behind celeris#645's adaptive column instead of bounding it.

celeris#646 (celeris main 91def85) turned EngineMetrics.ErrorCount from one atomic a dozen unrelated branches incremented into the derived sum of eleven cause buckets, plus StandbyErrorCount, the adaptive engine's share-by-sub-engine split. probatorium published the total and nothing else, which is why celeris#645 could say only that the two sub-engines alone do not account for 421 errors in a 112-second cell.

The repin

Now carried by main, not by this PR. probatorium#381 (d0cb24c) pinned celeris 9aa258c, a direct descendant of the 91def85 this branch originally pinned, so the branch's own 20-file repin commit was dropped on the rebase as redundant — its resolved tree was byte-identical to main.

The swap is safe by measurement, not by assumption: the Error* / Standby* field set of engine.EngineMetrics is identical at 91def85 and 9aa258c (celeris#647 added Transplant* counters, no new error bucket), and validation/refapp/internal/debugvars was re-run with -race against 9aa258c — green. That test reflects over celeris's own EngineMetrics rather than a hard-coded name list, so it is the direct proof that the twelve-key publisher still matches the newer pin.

The wire

debugvars publishes all twelve keys → checker.ParseDebugVars reads them into properties.Snapshot → the tally keeps each one's peak → report.Tier1Summary carries engine_error_count, engine_error_classes and engine_standby_error_count → six of the twelve also become per-cell series columns.

EngineErrorClasses is a map[string]int64, not eleven named fields, for the reason EngineZeroWitness is one: celeris#646 leaves no generic "bump the error count" call, so a new error branch has to name a bucket and this set will grow — and a field-by-field literal is how celeris#627 silently dropped ten fields. The standby share is deliberately not a member of the map: it cuts the same total along the other axis, so summing the map with it in would double count. It travels as its own scalar next to PeakStandbyActiveConns.

Because celeris assigns the total from the buckets in engine.FillErrorClasses and keeps no separate running total (and the adaptive engine sums both sub-engines bucket by bucket), sum(engine_error_classes) == engine_error_count holds inside any single /debug/vars document on every engine. The artifact now records both, so the split can be checked rather than trusted.

Six columns in the series, six totals in the tally

Not eleven columns. The series is written once a second for a whole cell and already carried thirty-one columns. A bucket earns one when the question asked of it is "when", and the artifact carries something timestamped to join that against — the promotion instant (adaptive_switches), the hand-off (engine_transplant_adopted), a walker's slow-read or handshake-fail fire, or the per-second denominators a rate needs (engine_requests_total, active). A bucket whose question is "how many, over the cell" does not: 3,600 rows of zeros and one step is not a better answer than the total.

The call is recorded per bucket in report.ErrorClasses (a Series flag plus a prose Why), and validation.TestTheSeriesCarriesExactlyTheBucketsDeclaredForIt fails if the declaration and the CSV header ever disagree in either direction.

In the series (engine_error_accept_fd_limit, engine_error_accept_cancelled, engine_error_accept_other, engine_error_conn_table_cap, engine_error_send_peer_gone, engine_standby_error_count):

bucket	why the timestamp is the answer
accept_fd_limit, accept_other	a connection refused at accept leaves no other trace anywhere — no conn-table entry, no close hook, no request. A walker's handshake-fail-timeout instant has nothing else to join against (celeris#588), which is the whole reason engine_error_count was put in the series in the first place.
accept_cancelled	steps deterministically at the promotion — celeris#646 measured exactly two per io_uring worker per PauseAccept — so joined against adaptive_switches in the same row it is what separates a switch transient from sustained accept-side loss.
conn_table_cap	fires on two different paths at two different instants, accept and transplant adoption, and only the timestamp joined against engine_transplant_adopted says which one a cell hit.
send_peer_gone	the bucket celeris#645 has to size, and "proportionate to the load" is a rate: it needs the per-second denominators only the series carries in the same row. It is also what drowns the total — 88,010 of 88,776 accepts on io_uring — so without its own column no step in engine_error_count is readable at all.
standby_error_count	the direct analogue of engine_standby_active_conns and engine_standby_close_count, both series columns for exactly this reason. "The standby's accepts were cancelled at the promotion" needs accept_cancelled and this at the same second.

Tally only (conn_register, listener_recreate, transplant_adopt, send, request_body, handler):

• conn_register — an epoll_ctl ADD refusal. A kernel refusal of one syscall, not a load-shaped rate, and co-timed with either an accept or an adoption, both already timestamped in the series.
• listener_recreate — celeris's own comment at both increment sites: "the bump is not a rate: it is the one record that the engine lost a listener". At most one per loop, and the loop returns immediately after. Worth noting against the obvious objection: EngineMetrics.Workers is len(e.loops), a static slice length that does not drop when a loop dies, so the tally entry is the only record of the capacity loss that exists — which is an argument for recording it, not for sampling it 3,600 times.
• transplant_adopt — tracks TransplantAdoptSlotOccupied one for one by celeris's own construction, and engine_transplant_adopt_slot_occupied is already a series column and a gated zero witness. A second column with provably identical values buys no timestamp. In the tally it earns its keep as the cross-check: the two disagreeing means one of the two accountings is broken.
• send — nonzero at all is a defect, and when one fires the walker's slow-read ring (schema 5.9) already carries the instant, both socket addresses and the verbatim error, at far higher fidelity than a 1 Hz column. Promote it if a run ever shows it moving in bulk.
• request_body, handler — std only, request-shaped, decided on the request path: every instance is a request the walker sent and got a status back for, so the walker's own record timestamps them at per-request resolution. And the standing question about handler — why every std cell in nightly 34918161309 sat at exactly 45 — is a question about a total, not a trajectory.

Nothing here is gated, on purpose

The must-stay-zero witnesses these sit next to were gated because each counts an event that cannot happen in a correct engine, so one is a failure. Every bucket here counts something that legitimately happens: a client that abandoned a response, an accept cancelled by the PauseAccept a promotion performs, a handler that returned an error. celeris#646 measured 88,010 SendPeerGone over 88,776 accepts on a healthy io_uring abandon-churn load. Until a run says what normal looks like per engine, any threshold would be a number nobody measured.

checker.TestNoErrorClassIsAZeroWitness keeps it that way: a bucket that quietly acquires a ZeroWitnessMeaning entry fails the build rather than the next nightly's normal traffic.

The sum identity is recorded in the artifact and not gated either, for a concrete reason: celeris documents its bucket snapshot as individually atomic but not mutually consistent, and the adaptive engine reads two sub-engines in sequence, so a live cell can legitimately publish a document whose parts are a few errors behind its total. Failing a cell on that would be failing on a race in the measurement.

Schema

SchemaVersion → 5.14, with a changelog entry. Rebased onto d0cb24c, which is still at 5.13: probatorium#369 took 5.12 (per-cell run outcome) and probatorium#379 took 5.13 (resume provenance), so this is renumbered from 5.13 to 5.14 and all three changelog entries are kept.

The literal is pinned as a bare string in three places, and they are the trap: on the rebase, report/schema_test.go, mage_bench_sutenv_test.go and validation/runner_test.go auto-merged with no conflict because both sides had written "5.13" — and so did const SchemaVersion itself. Only the changelog hunk conflicted. All four were re-pinned to 5.14 by hand, and control M below proves each one now bites. The two stale want 5.9 messages next to two of them are corrected while there. The 5.0 / 5.2 / 5.3 literals elsewhere are deliberate legacy-decode fixtures and are left alone.

Rebase notes

Rebased onto d0cb24c. Twenty conflicts, every one the same two-line celeris pin in a go.mod/go.sum, all resolved to main's newer 9aa258c; the branch's repin commit then had nothing left to say and was dropped. The feature commit applied with no conflict, because d0cb24c touches nothing but pins.

go mod tidy was re-run in all ten modules afterwards and produced zero drift, so the go.sum hashes are the real ones rather than hand-edited. Against the pre-rebase tree the only change anywhere is those 20 pin lines: every Go source file is byte-identical, which is why the control table below still stands (M and I were re-run on this tree regardless — see Verification).

• fix(remote): make the child-output fan-in line-atomic (#382) #383 (line-atomic child-output fan-in) — checked, no interaction. The fan-in merges a candidate's stdout and stderr pipes into one captured stream via fanInLines(mergedW, &lineMu, …). Nothing in this change goes near it: the series is a dedicated per-cell file (os.Create of properties_series.csv, its own handle, its own csv.Writer), the buckets reach the checker as an HTTP JSON document over a socket, and the tally reaches validate-results.json. No added or removed line in this diff mentions stdout or stderr. Nothing here assumed the old byte-oriented merge.
• fix(resume): rescue the results before the purge, and merge them into the resumed run's verdict (#376) #379 (resume provenance) and fix(matrix): collect every cell's failure instead of reporting the first one (#359) #369 (per-cell Status / FailureReason, matrix loop moved into matrix_run.go) — no overlap. This change does not touch the matrix loop, and the report.Tier1Summary field block it extends is disjoint from ValidationCellResult, where both of those landed. After resolving, the declarations for all three new fields were checked to have survived alongside theirs — that keep-the-comment-drop-the-declaration slip is the failure mode fix(resume): rescue the results before the purge, and merge them into the resumed run's verdict (#376) #379 hit.

Tests, and the controls that prove they bite

The series column-count guard failed loudly as intended (test covers 31 columns but the writer has 37). Beyond extending it, every new column got a distinct nonzero fixture and the test now enforces distinctness — which caught a pre-existing hole: engine_transplant_detached and engine_transplant_adopted were adjacent and both 96, so swapping them passed. They are now 96 and 97.

Thirteen deliberate mis-wires, each reverted. All re-run on the rebased tree; the output below is the post-rebase run, not carried forward:

#	injection	test output
A	swap two adjacent new series columns in the writer	column "engine_error_accept_cancelled" = 31, want 29 / column "engine_error_accept_other" = 29, want 31
B	give two fixtures the same value (the old 96/96 state)	columns "engine_transplant_detached" and "engine_transplant_adopted" both expect 96; swapping them would pass
C	leave a fixture at zero	column "engine_error_send_peer_gone" expects 0; a zero fixture cannot tell a wired column from an unwired one
D	one bucket reads its neighbour's snapshot field	with only EngineErrorAcceptCancelled set: bucket "engine_error_accept_other" = 7, want 0
E	a declared bucket never recorded	bucket "engine_error_handler" = 0 after a snapshot that set every field to 1 + buckets sum to 110 but EngineErrorCount = 132
F	a parse key misspelled	snap.EngineErrorConnTableCap = 0, want 7
G	the refapp stops publishing one bucket	celeris.engine_error_listener_recreate: missing or not a number: <nil>
H	the refapp stops publishing the standby split	celeris.engine_standby_error_count: missing or not a number: <nil>
I	a bucket promoted to Series in the declaration only	report.ErrorClasses marks "engine_error_send" as a series bucket and seriesColumns does not carry it
J	a bucket added to the series with no declaration	seriesColumns carries "engine_error_handler" but report.ErrorClasses declares it tally-only
K	a bucket quietly made a gated zero witness	"engine_error_send_peer_gone" is declared both an error class and a must-stay-zero witness; the gate would fail a cell on load-proportional traffic
L	the runner's field-by-field literal forgets a line	engine_error_classes = map[], want map[...] + projected buckets sum to 0 against engine_error_count 160
M	move const SchemaVersion to 5.15, leave the three bare literals at 5.14	SchemaVersion drift: want 5.14 got "5.15" / missing "\"schema_version\": \"5.14\"" in document / schema_version="5.15" want 5.14

G is the strongest of them: the refapp test reflects over celeris's own EngineMetrics fields rather than a list of today's names, so a twelfth bucket added upstream fails in the refapp that has to publish it rather than in a nightly that quietly reports one bucket short.

L covers the celeris#627 blind spot directly: TestTier1SummaryExportsEveryTallyField cannot see the property tally because it is a nested struct, and runner.go's report.Tier1Summary{...} literal is projected field by field.

M is new this round, and it exists because the hazard is real rather than hypothetical — see Schema above.

Verification

All post-rebase, on d0cb24c, run in the foreground on a clean worktree:

• go build ./... — clean. go vet ./... — clean. gofmt -l . — clean.
• go test ./report/... ./validation/... ./validation/checker/... ./validation/properties/... — green, 8/8 packages (validation 109.0 s, properties 10.6 s, remote 8.3 s).
• go test -tags mage ./... — green, 23/23 packages. Flagged because probatorium#384 exists precisely because these run in no workflow: they are green here, on this branch, today. The magefiles carry //go:build mage, so plain go test skips mage_bench_sutenv_test.go entirely and the lint workflow only compiles it; its schema pin was therefore exercised explicitly, both at 5.14 and under control M.
• validation/refapp/internal/debugvars — go test -count=1 -race -timeout=5m ./... green against main's 9aa258c, not the 91def85 this branch was written on. All eight refapps go build ./... && go vet ./... — clean. (Both are what test.yml's refapps job runs.)
• go mod tidy in all ten celeris-dependent modules — zero drift.
• golangci-lint run ./... (v2.13.2) — 0 issues, root module and debugvars module both.
• Control M re-run on this rebase and firing, all three literals, with const SchemaVersion moved to 5.15 and restored from a cp backup afterwards (checksum verified identical):
  • report/schema_test.go:399: SchemaVersion drift: want 5.14 got "5.15"
  • validation/runner_test.go:78: missing "\"schema_version\": \"5.14\"" in document
  • mage_bench_sutenv_test.go:295: schema_version="5.15" want 5.14
• Control I re-run and firing: series_test.go:198: report.ErrorClasses marks "engine_error_send" as a series bucket and seriesColumns does not carry it + series_test.go:212: series carries 5 declared buckets, want 6.
• The other eleven controls are carried forward from the previous rebase rather than re-run, which is sound here for a checkable reason: the only difference between that tree and this one is the 20 celeris pin lines, and every Go source file is byte-identical.
• No cluster workflow dispatched; this PR carries no cluster-ok label and a race tier is live.

Refs celeris#645, celeris#646.