Perf checkpoint dccb839 vs 9f4d89b: arm64 RPS is MARGINAL on epoll-auto+upg-async (-1.76%) and adaptive-auto+upg-async (-1.43%) against a 1.33% floor; one replication, then bisect if it holds

[FumingPower3925]

Summary

The perf checkpoint of main dccb839 against 9f4d89b ends MARGINAL. On arm64, two engines lose RPS by slightly more than the in-run noise floor:

Engine (arm64, RPS)	Median d	Engine floor	Margin past the floor
epoll-auto+upg-async	−1.76%	1.33%	0.43 pp
adaptive-auto+upg-async	−1.43%	1.33%	0.10 pp

Both margins are inside the pre-registered 0.5 pp near-floor band. The rule for that case is to re-run the same checkpoint once before acting. This issue tracks that replication, and the bisect if the flags hold. It is not yet a confirmed regression.

Everything else is NO CHANGE:

• amd64: all 9 engines, on both RPS and CPU/req.
• arm64: CPU/req on all 9 engines, and RPS on the other 7.
• No single cell passed its class floor on either arch or metric.
• All 648 cells were ok (324 per arch; no suspect, no dnf).

What was measured

• Run: goceleris/probatorium Benchmark Tier run 36340530922, branch perf/checkpoint-dccb839-vs-9f4d89b at aa4a5ea9, profile fast, both arches in parallel (msa2-server amd64, msr1 arm64). Bench window: 2026-09-27 19:24Z to 23:57Z.
• Design: four columns per engine in the order A1 B1 B2 A2. A = dccb839 (v1.5.12-0.20260927173448-dccb83925a50), B = 9f4d89b (v1.5.12-0.20260920100717-9f4d89b171db). Nine engines × nine scenarios = 324 cells per arch.
• d = (mean A − mean B) / mean B, sign-normalised so that negative means main is worse, for both RPS and CPU/req.
• Engine floor = max(1.0%, 2 × the largest per-engine same-binary median |A1 vs A2| or |B1 vs B2|). A verdict is the engine's median d over its 9 cells against that floor. The rule and the reader were pre-registered and hash-checked before the read.
• Integrity:
  • Every column's version label matches its arm, and every column bound its own binary (bind guard 36/36 per host).
  • One server process per column, exit rc=0, no restart.
  • No WARN, ERROR or panic line in any server log.
  • All 16 adaptive columns promote epoll → io_uring exactly once, at +253 to +255 s, the same in every arm. So the adaptive flag is not a placement difference.

The two flagged engines, cell by cell (arm64 RPS)

"A<B" means every A arm is below every B arm.

Scenario	adaptive-auto+upg-async	epoll-auto+upg-async
churn-close	−1.03%	−0.05%
driver-pg-read	−0.12% A<B	−0.25% A<B
driver-redis-pipeline	−1.82% (known arm64 async stall)	−11.13% (known arm64 async stall)
get-json-1k	−4.39% A<B	−3.02% A<B
get-simple	−1.43%	−5.86% A<B
get-simple-512c	−0.77% A<B	+0.38%
post-4k	−7.12% (A/A +25%)	−3.51% A<B
sse-fanout-128	−0.99% A<B	−1.76% A<B
ws-echo	−1.83%	−1.74% A<B
median	−1.43%	−1.76%

Evidence either way

For a real effect:

• The sign is consistent: 9/9 cells negative for adaptive-auto, 8/9 for epoll-auto.
• With the floor recomputed without the engine that set it (1.29%), both flags still hold, at +0.14 and +0.47 pp.
• epoll-auto+upg-async also flags against either B column alone. adaptive-auto+upg-async flags only against B2.
• A sub-floor main-worse tilt shows on both arches, not only arm64. Counting cells where every A is worse than every B, against the reverse:
  • amd64 CPU/req: 30 against 6 (p≈7e-5). In-run control std-h1: 0 against 3.
  • arm64 RPS: 28 against 11 (p≈0.009).
• epoll-auto+upg-async CPU/req is also negative on amd64 (−0.72%, 7/9 cells strict), which backs the arm64 RPS flag from the other arch.
• iouring-h1-sync is negative on all four engine medians: RPS −0.33% amd64, −0.09% arm64; CPU/req −1.08% amd64, −0.23% arm64. amd64 RPS is 9/9 cells negative. The previous checkpoint flagged this engine and its bisect closed it as noise.

For noise:

• The arm64 RPS floor of this run (1.33%) is unusually low. The expected value is a median of 1.85% (p90 3.84%), and at that floor neither engine would flag.
• The movement is arm64-wide: 54/81 arm64 RPS cells are negative (amd64 42/81), and the median of the engine medians is −0.29%. The two flagged engines are the tail of that distribution.
• The h1-async siblings of both flagged engines are flat: adaptive-h1-async +0.94%, epoll-h1-async +0.08%.

A limit of the design. d = (A1 − B1 − B2 + A2)/2 is exactly the quadratic slot contrast (+1, −1, −1, +1). A humped drift that favours the middle slots, where B sits, reads as "main is worse". The previous checkpoint (A1 B A2) showed the same A-worse tilt on non-streaming rows, std-h1 included. A replication in the same order therefore reproduces any such positional effect and cannot tell it apart from a real shift.

Plan

1. Replicate once with checkpoint 698bed6 vs 9f4d89b: same cells, same four-column design, same unchanged comparison script.

   • Its A arm is engine-identical to dccb839 on the bench server's import graph. dccb839..698bed6 is fix(session): keep a copy of a loaded session's ID, so a write-behind save goes to its own session (celeris#731) #734 and fix: give net/http, Prometheus, OTel and slog copies of the request strings they keep, and give Adapt's request its headers (#732, #720) #736, and neither touches code the bench server runs.
   • It waits for the branch cut, the pre-flight, and a free cluster.

2. Before any replication result exists, its pre-registration must:

   • define "holds": the same engine gets an arm64 RPS REGRESSION again under the same rule, and say what happens if only one of the two engines repeats;
   • pre-register a pooled strict-ordering secondary across both arches, with std-h1 as the control;
   • add a reversed-order arm (B1 A1 A2 B2), or state that a same-order replication cannot separate a middle-slot effect from a real shift.

3. If it holds, bisect inside 9f4d89b..dccb839:

   • epoll: 7c123da (fix(epoll): never park the loop on a running async handler, and leave the live set to the loop on an async hijack (celeris#669, celeris#668) #698) and 49d2726 (fix(engine): keep serving on a paused listener for 1.5 s with TCP_DEFER_ACCEPT cleared, so a switch or PauseAccept no longer resets clients that had not yet sent a request (celeris#662, #675) #674);
   • adaptive: 0046b6f (fix(adaptive): treat io_uring capped to none as not viable; correct stale io_uring env docs (celeris#679) #694) and 49d2726 (fix(engine): keep serving on a paused listener for 1.5 s with TCP_DEFER_ACCEPT cleared, so a switch or PauseAccept no longer resets clients that had not yet sent a request (celeris#662, #675) #674).

   fix(engine): keep serving on a paused listener for 1.5 s with TCP_DEFER_ACCEPT cleared, so a switch or PauseAccept no longer resets clients that had not yet sent a request (celeris#662, #675) #674 is the change both flagged engines share. Before calling it a regression, weigh the by-design costs:

   • fix(engine): keep serving on a paused listener for 1.5 s with TCP_DEFER_ACCEPT cleared, so a switch or PauseAccept no longer resets clients that had not yet sent a request (celeris#662, #675) #674 keeps a paused listener serving for about 1.5 s with TCP_DEFER_ACCEPT cleared;
   • fix(epoll): never park the loop on a running async handler, and leave the live set to the loop on an async hijack (celeris#669, celeris#668) #698 stops parking the epoll loop on running async handlers.

4. If it does not hold, close this as run noise. The skeptic review notes a risk here: re-basing every checkpoint's B to the latest main while sub-floor main-worse tilts keep recurring means a cumulative cost would never flag. The pooled secondary in step 2 is there to catch that.

Watch items for the replication (below every floor, not findings)

• arm64 ws-echo RPS: 7 of the 8 non-std engines are negative. Every A is below every B on epoll-auto+upg-async (−1.74%), epoll-h1-sync (−2.23%) and iouring-auto+upg-async (−2.83%). The ws class floor is 8.77%. fix(websocket): apply pause/resume under the lock that decides them, and lift a stale pause (celeris#667, celeris#672) #671 and fix: clone the request values a detached stream keeps (websocket Conn.Query, SSE Last-Event-ID, Context.Detach, requestid and otel context values) (#714, #717, #718) #723 touch this path.
• iouring-h1-sync ws-echo: the read against B1 alone flags it on both arches: amd64 RPS −2.22%, arm64 CPU/req −2.74%.
• arm64 sse-fanout-128: median −0.60% against a 3.79% floor. 5 cells have every A below every B.
• arm64 driver-pg-read: |d| ≤ 0.52%, but 8/9 cells are negative and 5/9 have every A below every B, all on epoll and adaptive engines.

Known apparatus notes

• The comparison script's "status mismatches: 0" check is vacuous on this artifact, because the merged results carry no cell_statuses key. The statuses were tabulated from the raw per-host files instead: 324/324 ok per arch.
• arm64 async driver-redis-pipeline is the known stall (same-binary dispersion up to 78%) and cannot be interpreted. It is absorbed by its class floor.

[FumingPower3925]

Replication result: the two arm64 flags do not hold. The replication was run exactly as pre-registered. Benchmark Tier 36376407767, 2026-09-28, both arches: 6 engines × 8 columns in the slot order A1 B1 B2 A2 B3 A3 A4 B4 (both orders, so slot-position effects cancel). A = celeris 698bed6, whose engine code is identical to dccb839; B = 9f4d89b. 432 cells per arch, all ok.

The reader is the pre-registered read/r01_read.sh: PREREG sha 9a4fc1e5; compare_arms4.py unchanged, b83bc640. It passed its self-check, the identity check (0 failures) and the raw-status check (0 non-ok).

engine (arm64 saturation RPS)	order 1	order 2	order-averaged d	floor	verdict
epoll-auto+upg-async	flat +0.11%	flat +0.33%	+0.10%	3.43%	does not hold
adaptive-auto+upg-async	flat -0.23%	flat +0.08%	+1.83%	3.43%	does not hold

• Pre-registered decision: close Perf checkpoint dccb839 vs 9f4d89b: arm64 RPS is MARGINAL on epoll-auto+upg-async (-1.76%) and adaptive-auto+upg-async (-1.43%) against a 1.33% floor; one replication, then bisect if it holds #789 as not replicated at checkpoint resolution. This is not proof there is no cost. The power simulation gives about a 5% chance of confirming a real -1.5% effect.
• Other engines: no engine × arch × metric regression replicated on the other 4 engines here. The other 3 engines come from run 36340530922, whose A build is engine-identical.
• Pooled strict-ordering check (a secondary check), counting cells where every A arm is worse than every B arm against the reverse, out of 90 cells:
  • amd64 CPU/req: 28 main-worse vs 11 main-better, p = 0.0095. The std-h1 control is 3 vs 4, p = 1. So this is a main-worse tilt below every floor. It also appeared in 36340530922 (30 vs 6).
  • The other three (arch × metric) combinations show no tilt.
  • Under the PREREG, the next checkpoint keeps B = 9f4d89b and does not re-base to main, so the tilt keeps accumulating evidence.
• Next: an independent skeptic check of this read runs before the issue is closed.

Evidence: evidence/perf-checkpoint-20260927/replication/read-36376407767.log and run-36376407767/.

[FumingPower3925]

Skeptic re-check of the replication read

Two independent skeptics re-checked the read posted in issuecomment-5868126349:

• re-derive: a fresh download, a re-run of the pre-registered reader, and an independent reimplementation.
• method: the unit of replication, the slot terms, and whether the rules were fixed before the data.

Every number below is regenerated by a script in the local evidence tree evidence/perf-checkpoint-20260927/replication/skeptic-20260929/{re-derive,method}/. Each has a MANIFEST.txt, and the re-derive SHA256SUMS.txt checks OK on all 44 files.

Runs

• First checkpoint: 36340530922, dccb839 vs 9f4d89b, columns A1 B1 B2 A2.
• Replication: 36376407767, probatorium @ 4d2bbb3, 698bed6 vs 9f4d89b, 6 engines, columns A1 B1 B2 A2 B3 A3 A4 B4, PREREG sha 9a4fc1e5.

What survived

1. The read reproduces.

• A fresh download matches the loop's copy on all 2215 files (sha256), and the bench job log is identical.
• The pre-registered reader, re-run with only its OUT path changed, gives 23 of 23 outputs byte-identical.
• An independent reimplementation that shares no code with the reader reproduces every floor, median and verdict.
• It also passes a positive control on the first run: -1.758% and -1.434% against F 1.331%.

2. Neither arm64 RPS flag holds.

engine	first run, order 1	this run, order 1	this run, order 2	order-averaged median	Fbar
epoll-auto+upg-async	-1.758%	+0.107%	+0.326%	+0.101%	3.43%
adaptive-auto+upg-async	-1.434%	-0.229%	+0.078%	+1.835%	3.43%

• In at least one order, each flagged engine's per-order median is at or above zero. So neither can be REGRESSION in both orders under any non-negative floor, and the verdict does not depend on the 3.43% floor.
• For epoll-auto, leaving out any one scenario keeps the median between +0.06% and +0.41%.
• The adaptive-auto +1.83% should not be read as "main is 1.8% faster". It is the middle value of nine bimodal scenario averages (sorted: -1.14, -0.63, -0.20, -0.12, +1.83, +3.54, +3.81, +20.68, +59.18). Its per-order medians are -0.23% and +0.08%, which is flat.

3. The checkpoint passes on all 6 engines.

• No per-order REGRESSION verdict appears on any of the 24 rows (6 engines x 2 arches x RPS and CPU/req).
• The worst per-order median is -0.99% (arm64 CPU/req, adaptive-h1-async, order 2). That is below the rule's minimum floor of 1.0%.
• All 864 raw statuses are ok, and every row has n = 9.

4. Arm identity is correct, checked from the run's own logs. Both skeptics found 0 identity failures.

• Construction: at 4d2bbb3, servers/celeris pins celeris 698bed6 and servers/baseline_celeris pins 9f4d89b.
• bench-vars: the A columns run the celeris binary and the B columns run baseline_celeris.
• Bind guard: "wait for OUR server to own the bench port" is ok 48/48 on each host, with 0 fatal or failed lines.
• Per column: each has its own pid, equal to the observer pid, and server.log names the right engine.
• Order: the time order is A1 B1 B2 A2 B3 A3 A4 B4 in all 12 (engine, host) blocks.
• Limits: the binary prints no marker of its own, framework_version is only a registry label (it is empty in the A columns' own results.json), and the deployed binaries' hashes are not logged. Identity rests on how the run was built and on the launch guard, not on anything the binary reports.

5. The rules were fixed before the data.

• The PREREG sha was recorded at 01:48:43Z and SHA256SUMS at 01:49:55Z. The run was created at 04:08:00Z.
• All 62 registered files still match.
• This anchor is local only (transcripts and file mtimes).

What was refuted: "the amd64 CPU/req main-worse tilt REPLICATED"

This corrects issuecomment-5868126349. The arithmetic is right: 28 vs 11, p = 0.0095, control 3 vs 4, so the registered test reads TILT MAIN-WORSE. But calling the tilt "replicated" does not survive:

• One cell decides it. 27 vs 11 gives p = 0.0139 and 28 vs 12 gives p = 0.0166. Both read NO TILT.

• The cells are not independent. One server process serves all 9 scenarios of a column, so the 90 cells come from 40 processes in 10 (engine, order) blocks. The design effect is 2.27, or about 40 effective cells. Tests that respect the blocks:

  • shuffling A/B labels within each block: p = 0.044;
  • sign test over blocks (7 main-worse vs 1 main-better): p = 0.070;
  • flipping signs per block: p = 0.14;
  • t-test over blocks (mean -0.19%, t = -1.57, 9 df): p = 0.15.

  None of these passes the registered 0.01.

• The first-run comparator is from a different pool. "30 vs 6" counts 9 engines, including the std-h1 control, in order 1. On the same 5 non-control engines in order 1, the first run is 27 vs 2 and this run is 12 vs 8 (p = 0.5). A Fisher test says the two runs differ (p = 0.0093). This run's pooled count comes mostly from order 2 (16 vs 3). With std-h1 included on the same basis, this run is 31 vs 15 (p = 0.026; p = 0.078 with blocks respected).

• iouring-h1-sync reversed. It was in the run because of its repeated CPU/req tilt. In order 1 its block went from +6 to -6, and it is 3 vs 6 pooled.

• CPU/req is a ratio of two counters, and both move. Of the 28 main-worse cells, 15 have A's CPU% strictly higher and 8 have A's RPS strictly lower. Across the 90 cells, the CPU and RPS log-ratios correlate at 0.87, and their medians are +0.13% and -0.16%.

What does survive is a weak main-worse direction. In 12 of 15 blocks across both runs, the 5 engines lean main-worse (block mean -0.34%). The size is about 0.1 to 0.4%, roughly an order of magnitude below the 2.6% amd64 CPU/req floor. Its status is an unconfirmed sub-floor direction, not a replicated one. The consequence does not change: the next checkpoint keeps B = 9f4d89b.

Not flagged by the rule: amd64 RPS. The registered test reads NO TILT (25 vs 12, p = 0.047). By block-level tests it is this run's more consistent lean:

• 9 of 10 blocks are negative (t = -3.06, p = 0.014), and the cluster null gives p = 0.018;
• the std-h1 control also leans main-worse (9 vs 1).

Because the control carries it too, it points at the run or at shared code, not at engine code. It was absent in the first run (block mean +0.05%).

Wording and method corrections

• Say "not replicated at checkpoint resolution", not "noise" or "excluded".

  • The PREREG's C8 power check gives a real -1.5% arm64 RPS cost about a 5% chance of holding.
  • A changed between runs (dccb839, then 698bed6).
  • The same order moved 1.2 to 1.9 pp between runs.

  This run shows that 698bed6 has no such cost here. It does not show that the dccb839 flags were noise.

• The slot terms. The reversal cancels the linear term and every even term. The first term it leaves is the odd cubic, which biases both orders in the same direction. The data show no systematic cubic: about ±0.1% on amd64, and +0.6 ± 0.5% on arm64 (toward main-better).

Next checkpoint (queue row 29)

• Keep B = 9f4d89b.
• Register a block-level decision statistic (n = blocks: a cluster randomisation or a block sign-flip) for both amd64 CPU/req and amd64 RPS. Report std-h1, and match the counts to both earlier runs.
• Name the cubic as the term the design does not cancel.
• Post the PREREG sha on the issue before dispatch.

Status

The close gate is met. Both skeptics report flags_replicate = no, checkpoint_passes = yes and arm_identity_ok = true, and the PREREG's outcome for that case is to close as not replicated at checkpoint resolution.

This issue stays open anyway, because a claim posted here (the tilt "REPLICATED") was refuted; this comment is the correction. No open issue in celeris, probatorium, loadgen or docs tracks the amd64 CPU/req direction. It is unconfirmed, so no new issue was filed. It stays tracked here until the next checkpoint's PREREG carries it, and then this can close.