Follow-ups from #767: a named CI interlock for the #712 tests, and the bare-metal failing-first

[FumingPower3925]

The review of #767 (celeris#712) had one blocking finding, fixed on the branch in round 2 (head d9307b5). One enter before the park ran at most 20 deferred completions per local-work pass (IO_LOCAL_TW_DEFAULT_MAX, kernel 6.13 and later). With 64 connections closed in the parking iteration, 20 of them got their FIN. The park now waits until every closed connection's kernel ops have delivered their terminal CQEs, and a 64-connection arm checks it.

The review also left the minor findings and nits below. Items 1 and 2 are still open; each says what to do.

1. No named-test interlock for the #712 tests in CI (nit)

CI covers the #712 tests only through the engine/iouring package step's tally of unlisted SKIP lines. That step names only the two celeris#639 tests (.github/workflows/ci.yml lines 166-181 in the merged tree). grep -n 'celeris712\|park_close' .github/workflows/*.yml finds nothing, so a rename or a removal of these tests would still go green. #766 added its tests to the named interlocks.

To do: add the six tests in engine/iouring/park_close_fin_test.go to a named interlock, with want N, ran N, passed N, SKIP lines 0:

• TestParkedWorkerSendsFINForAHeaderTimeoutClose
• TestParkedWorkerSendsFINForAReadTimeoutClose
• TestRunningWorkerSendsFINForAHeaderTimeoutClose
• TestParkedAsyncWorkerSendsFINForAHeaderTimeoutClose
• TestParkedWorkerSendsFINToManyReadTimeoutCloses
• TestParkedWorkerSendsFINToManyHeaderTimeoutCloses

Doing this in #767 would touch the lines of ci.yml that #766 changes.

2. The failing-first exists only on the laptop kernel (minor, open until the cluster rows run)

The defect has been measured failing only on the laptop's 7.0.12-linuxkit. CI's 6.17-azure runs only the fixed head.

The bare-metal rows in evidence/_queue/cluster.tsv (lane EP-2) run:

• the fixed head;
• 8013189, the new arms on the round-1 fix, where the expected result is red: fin_seen_while_parked of 20 per worker (20/64 with one worker, 40/64 with two), the kernel's cap;
• d655435, main plus the round-1 tests.

Their PASS criteria now key on defer_taskrun=true, which the arms log since round 2 (item 3).

To do: read those rows' results into #767 or celeris#712 when they run.

3. Items the round-2 push dealt with (for the record)

• The tests did not show that the ring defers its task work (minor). A COOP_TASKRUN ring logs tier=high too, and on such a ring the park arms passed on unfixed code (the reviewer's NODEFER mutant, 20/20). Every arm now logs defer_taskrun. Under CELERIS_REQUIRE_IOURING_WORKERS=1 (CI and the cluster), a ring without it fails the premise.
• The flush made with nothing pending was untested (nit, MPENDING). Moot: round 2 removed the pre-park flush and Ring.SubmitAndFlush. The A5 submit is main's again.
• The io-wq exclusion was argued (nit). Moot for the same reason. The park now waits for the terminal CQE of every op kernelInflight counts, whatever runs it, bounded by pendingReleaseHoldNanos (5 s) plus one ring wait.

To do: nothing further.

[FumingPower3925]

Update from merge train 8: #767 is now tests-only

#767's park change in engine/iouring/worker.go has been dropped. #793 (celeris#685, 3e7abba) fixed #712 on main: a close path that still owes an op keeps its descriptor until the op's terminal CQE, and the worker does not park while closeFDOwed is non-zero.

The six tests in engine/iouring/park_close_fin_test.go stay as the regression guard:

• On main before fix(iouring): never release a descriptor number while an op can still resolve it: close paths, hijack, shutdown (celeris#685) #793 (1fdfcd4), the four park arms FAIL 12/12.
• On main after fix(iouring): never release a descriptor number while an op can still resolve it: close paths, hijack, shutdown (celeris#685) #793 (3e7abba), the four park arms PASS 12/12 without the worker.go change.
• The two controls PASS on both.

Status of the items above:

• Item 3 describes the dropped park change. It is superseded.
• Item 1 (a named CI interlock) is still open.
• Item 2 now applies to main. The cluster rows for d9307b5 and 8013189 test a change that will not ship. The failing-first to keep is main before fix(iouring): never release a descriptor number while an op can still resolve it: close paths, hijack, shutdown (celeris#685) #793 against main after it.

4. The client-close poller counts a reset as a close (minor, CodeRabbit on ab1e9f2)

clientSeesClose returns success on any read error that is not a timeout, so an RST counts as the close arriving. When the engine closes with request bytes still unread, the client gets a reset instead of a FIN.

That is still enough to show #712 is fixed: the defect was no close of either kind while the worker stayed parked. It is also narrower than it looks:

• The many-connection arms wait until every partial header is read (BytesRead) before the pause.
• Every single-connection line on record logs (EOF). The many-connection arms do not log which one arrived.

The single-connection arms do not wait for BytesRead.

To do:

• Accept only io.EOF as the FIN (return failure on ECONNRESET), or log which one arrived and assert EOF.
• In the single-connection arms, wait for BytesRead as the many-connection arms do.
• Run the pre-fix(iouring): never release a descriptor number while an op can still resolve it: close paths, hijack, shutdown (celeris#685) #793 (1fdfcd4) arm again to confirm it still fails.

Thread: #767 (comment)

5. Setup deadlines are tight enough to fail the premise on a slow runner (minor, CodeRabbit on ab1e9f2)

Two arms have little slack before their deadline:

• Single-connection header-timer arm. It uses ReadHeaderTimeout 400 ms. The accept and PauseAccept must both happen inside that window, or celeris712 PREMISE: the connection closed ... before the listeners did fails the test.
• 64-connection arms. They use 1 s (ReadHeaderTimeout or ReadTimeout). In that second the arm must make 64 sequential dials and writes, read every header byte and call PauseAccept.

Both failures are loud PREMISE failures, never a false pass. Neither has been hit in any run we have: CI at d9307b5, train 7's runs, and this head's local Docker runs at 8 MiB memlock.

To do:

• Lengthen both deadlines, for example 1 s for the single arm and 3 s for the many arms.
• Lengthen the close-observation bounds and parkWait712 waits that depend on them.
• Confirm the pre-fix(iouring): never release a descriptor number while an op can still resolve it: close paths, hijack, shutdown (celeris#685) #793 arm still fails and the fixed arm still passes.

Thread: #767 (comment)