epoll: a rare close-handshake stall with the receive queue already drained, 1 in 73 (split from #607)

[FumingPower3925]

Split out of #607 so it is not closed along with it. #607's fix touches engine/iouring only and does not address this.

What is observed

The capped WebSocket backpressure suite failed once on the epoll subtest with closeTimeout=1, while io_uring was clean in the same run, and passed on re-run. That is 1 failure in 73 observations under the 8 MiB memlock, one-worker, -race shape.

An epoll close-handshake failure was also seen once in CI, which is what first made #607 look engine-independent.

Why it is a different defect from #607

Measured during the #607 investigation, and this is the discriminator:

• io_uring stalls show the server's kernel receive queue holding 43 KB to 845 KB unread, with the middleware reader empty and the handler blocked. The engine had not read the bytes.
• epoll stalls of the same shape show server rx=0 in every case. The engine had read everything.

Same symptom at the oracle, opposite state at the socket. #607's cause is a recv chained behind a blocked SEND with IOSQE_IO_LINK, which epoll does not do at all, since it has no submission queue and no linked operations.

In 151 runs during that investigation, epoll produced zero close-handshake failures across 16,747 handshakes. So the rate here is at least two orders of magnitude below io_uring's pre-fix rate, which is why it was invisible until the io_uring noise was removed.

Why it still matters

The release bar requires validations 100% clean with the same behaviour on arm64 and x86, and this is a live one-in-seventy-three failure on a required check. It will surface as an unexplained red on unrelated pull requests, which is exactly how #607, #608, #609, #610, #613, #620 and #622 each first appeared.

It is also the kind of rate where a single run discriminates nothing. Any investigation needs a run count in the low hundreds before it can claim either a cause or a cure.

What would move it

• Reproduce with a run count sized for a 1.4% per-run rate, so a cure has something to be measured against. At least 200 runs per arm for a meaningful comparison.
• Capture the same joined evidence io_uring: a linked recv never starts because its chained SEND blocks on a detached WebSocket peer's closed window #607 used: engine-side connection state and /proc/net/tcp for both sockets at the instant of failure, joined by address. server rx=0 with a starved handler is the state to explain.
• Check whether the epoll async dispatch path can leave a connection with its reader drained and its handler parked with no wakeup pending, which is the epoll-shaped analogue of an armed operation that never starts.

Blocked on nothing. Related: #607, #623, #631.

[FumingPower3925]

Two more instances, and the witnesses that were already in place all read zero

Both from 2026-09-15. Recording them here rather than filing a duplicate, because they are the
same symptom — a WebSocket connection that stops delivering and never finishes its close — and
because together they show this is not epoll-only, which is how this issue is currently
titled.

1. io_uring, in celeris CI (this is the one that turned main red at 3f8395c)

middleware/websocket TestBackpressurePauseDoesNotCancelInflightSend/io_uring, job
104569626129: closeTimeout 1/96. Shape: one io_uring worker (8 MiB memlock cap), multishot
recv with provided buffers, SEND_ZC on, kernel 6.17.0-1022-azure. The subtest ran ~21:22:47 to
21:23:14.9; at 21:22:59 it logged SEND_ZC notification reported failure ... err=12, i.e. the
ENOMEM fallback (worker.go:3013-3017 -> 2984 -> completeSend 3185-3215).

Every witness counter reads zero in that job: LINKBLOCK arms=0, RecvSQFull=0, armDeclined=0,
resumeWhileCancelPending=0, doubleArmed=0, cqeUnaccounted=0. That excludes, by measurement rather
than by argument:

• io_uring: a linked recv never starts because its chained SEND blocks on a detached WebSocket peer's closed window #607 (a detached recv chained behind its send) — detached conns no longer chain
  (worker.go:4806-4808) and arms=0;
• a recv owed after a full SQ ring — sqFull=0, so needsRecv was never set (prepareRecv
  4340-4346);
• the io_uring WS: backpressure pause/resume drops buffered inbound bytes, truncating frames #484 window, a resume landing while the pause cancel is still pending (drainDetachQueue
  4527-4549, prepareRecv 4336);
• the ECANCELED and ENOBUFS re-arm paths, which also read correct (2393-2424).

Reading additionally bounds the server-side close path below 10 s once the handler has parsed the
Close frame: the sweep gate is 0x1F while detachedCount > 0 (worker.go:1345-1352) with <= 50 ms
per iteration (1737-1743), so <= ~1.6 s before closeConn; a deferred close is force-finished
after closingDrainTimeoutNanos = 5 s (256, 5033-5045); CloseDrain gives up after 10 ms
(sockopts_linux.go:83-84, 199-220). So the 10 s oracle timeout is not the close path being slow.

2. epoll/arm64, on the cluster

probatorium Race Validation run 35024823796 (probatorium 1a5b439, celeris c40d0cb), cell
cell-05-auth_session_ratelimit-epoll on msr1:
tier_1.ws_echo.ws_echo_timeout = 1 — "a WebSocket echo connection stopped delivering before its
hold expired". That single signal failed the whole tier: ValidateGate FAIL (1 violation),
ValidateDiff strict FAIL with 4 MED cross-arch ws_echo/ws_echo_timeout divergences, and
mage Validate exit 2 on msr1.

The rest of that run was clean, which narrows the field usefully: 66/66 documents show
transplant join (detached - adopted) = 0 (the #658 fix holding on the cluster), 0 race reports
under -race, evaluations > 0 in every cell, and 0 property violations.

What this changes

• Scope: retitle/rescope away from "epoll:" — io_uring reaches the same end state.
• The mechanism is still not determined by reading. The one latent race that fits the symptom
  and leaves no witness behind is the chanReader pause/resume ordering defect, now filed
  separately; it is not claimed to be this cause, and it looks too rare to explain this rate.
• Instrument before fixing. Concretely: (a) clear cs.stallReported = false on every recv CQE
  while recvStallProbeActive (worker.go:2358-2360) — today RECVSILENT fires once per connection
  lifetime and a healthy >= 3 s drain phase uses up that one record before the real stall;
  (b) add RecvPauseNoop / RecvResumeNoop edge counters in the PauseRecv/ResumeRecv closures
  (worker.go:2177-2208), exported next to RecvResumeWhileCancelPending, and stamp
  cs.lastResumeNoopNs into the RECVSILENT line; (c) on the first close timeout, dump per reader
  readerPaused/len(engineReader.ch)/spillLen/closeRecv/closeSent/closed plus handler
  goroutine stacks, joined to the engine's raddr by the client laddr.
• Count clientCloseFail too (The WebSocket backpressure oracle counts clientCloseFail, prints it, and never asserts it: 66 of 96 connections failed and the test passed #623): the same stall caught mid-write lands there instead, and
  io_uring: a linked recv never starts because its chained SEND blocks on a detached WebSocket peer's closed window #607 saw roughly 5x more clientCloseFail than closeTimeout. A repro that counts only
  closeTimeout undercounts.
• Do not raise the 10 s budget here without adding the latency arm, or a permanent stall class
  disappears behind the budget (test(websocket): give the Close handshake a budget that matches the workload (#566) #570 raised it to 30 s only in inbound_sequence_linux_test.go).

[FumingPower3925]

A cheap reproducer for this family, found 2026-09-26 while setting up coverage (celeris#699).

Under go test -race -covermode=atomic on a GitHub-hosted runner, middleware/websocket's TestBackpressurePauseDoesNotCancelInflightSend failed on both engines at 19b4269, a commit whose uninstrumented unit job passed:

engine	conns	closeTimeout	clientCloseFail	otherWriteErr	closedOK	protoErr
epoll	96	4 (10.0-10.2 s after Close)	51	29	41	0
io_uring	96	4 (10.0-10.3 s)	56	1	36	0

otherWriteErr: "the engine failed a WriteMessage on a connection it should have kept alive". Log: evidence/oss-integrations/celeris/pr699/coverage-run1-job.log (Coverage run 36245950077).

Why it matters here:

• The close-timeouts are this issue's symptom, on epoll and on io_uring alike.
• clientCloseFail is the count The WebSocket backpressure oracle counts clientCloseFail, prints it, and never asserts it: 66 of 96 connections failed and the test passed #623 says the oracle never asserts, and it is large.
• Coverage instrumentation slows the schedule, which appears to widen the window.

That makes it a lever for the sized repro this issue needs after #671 (#667 + #672) lands: run the test under -covermode=atomic (or the celeris stress workflow, probatorium#409, once merged) on base and on #671's head, per engine and per arch.

One run is one observation (RULE: one run discriminates nothing). This is a lever, not yet a rate. The coverage workflow in #699 excludes ./middleware/websocket with this reason written in the file. That exclusion is removed when this family is fixed.

[FumingPower3925]

Investigation result (2026-09-27): no celeris defect shows in any class, and the original epoll symptom did not reproduce

All runs are on main 9f4d89b on GitHub runners, kernel 6.17.0-1022-azure on both arches, unless marked otherwise. One process (one go test) is one observation, and intervals are exact 95%. I recounted every number here from the raw shard logs of 14 celeris-stress runs with one script: evidence/celeris-633/investigation-20260927/scripts/report-verify.py writes 60-report-verify.txt. Laptop numbers come from gh-stress/24-laptop-rates.txt.

Short version

• No celeris defect in any captured state. Every class of TestBackpressurePauseDoesNotCancelInflightSend I could reproduce and inspect is a harness or oracle artifact, or a kernel behaviour that the test client triggers.
  • 0 of 2,519 give-ups had the middleware/websocket: a pause decided on a stale depth snapshot wedges the connection permanently when the handler drains to empty first (distinct from #667) #672 or middleware/websocket: a chunk that spills after the handler drained the channel is never promoted, so the connection wedges permanently (stranded spill) #705 wedge shape.
  • ecanceled, clientRST, dial and handshake failures are 0 in all 720 subtests.
• The coverage failure comes from three mechanisms:
  • Coverage slows the echo to 0.05-0.19x, and the client gives up on fixed deadlines (H1).
  • Its give-ups echo back as server write and read errors (H2).
  • A few connections stall for good because the client stops processing ACKs (H3).
  • A client with progress-based waits that also reads while it waits closed 3,840 of 3,840 connections cleanly under coverage.
• Still open:
  • This issue's own symptom, an uninstrumented epoll close-timeout, did not reproduce: 0/100 processes [0, 3.6%]. At the reported 1 in 73 the chance of that is 0.25, so it is not excluded.
  • One uninstrumented x86 io_uring close-timeout has no captured state.
  • Without coverage, the lost-ACK stall hits io_uring and not epoll: 18/100 against 0/100 processes, Fisher 3.3e-6. The cause is not known.
• The fix is test-side, in lane B-2. The WebSocket backpressure oracle counts clientCloseFail, prints it, and never asserts it: 66 of 96 connections failed and the test passed #623's clientCloseFail assertion must ship together with the client fix. Otherwise uninstrumented main fails on io_uring in 18 of 100 processes.
• Side finding, filed as middleware/websocket: on epoll and io_uring, Conn.Query returns garbage once a frame has arrived, because the captured query strings alias the engine's read buffer (200/200 per engine) #714. Conn.Query aliases the engine's read buffer, and was wrong on 200/200 connections per engine.

Rates

Arm (b): no coverage, uninstrumented main. Runs 36281776837 and 36283527124, 40 processes per arch. Each class cell counts the processes where that class was above 0.

arch	engine	subtest failed	closeTimeout	clientCloseFail	otherWriteErr	protoErr
x86	epoll	0/40 [0, 8.8]	0/40	0/40	0/40	0/40
x86	io_uring	1/40 [0.1, 13.2]	1/40 (one connection, 10.001 s)	7/40 [7.3, 32.8]: 15 connections, max 7, all 7 passed	0/40	0/40
arm64	epoll	0/40 [0, 8.8]	0/40	0/40	0/40	0/40
arm64	io_uring	0/40 [0, 8.8]	0/40	5/40 [4.2, 26.8]: 8 connections, max 3, all passed	0/40	0/40

• InboundSequence failed 0/40 on each arch.
• RECVSTALL and LINKBLOCK read all zero in 80/80 io_uring subtests.
• A third uninstrumented x86 run (36292004178, 20 processes) passed everything. Its io_uring clientCloseFail was above 0 in 6/20 processes, 22 connections, 15 of them in one process.

Arm (a): with -covermode=atomic.

celeris-stress refuses the flag: tools/stresstally/plan.go parseExtra has no coverage flag in its allow-list. The stand-in is C1, main plus per-connection instrumentation plus go tool cover -mode=atomic baked into middleware/websocket, with 10 processes per arch. Each class cell reads processes above 0; total connections.

arch	engine	subtest failed	closeTimeout	clientCloseFail	otherWriteErr	protoErr
x86	epoll	10/10	5; 9	10; 471	10; 246	0
x86	io_uring	10/10	10; 382	6; 179	4; 6	7; 61 ("broken pipe")
arm64	epoll	10/10	0	10; 949	10; 432	0
arm64	io_uring	9/10	9; 68	9; 132	2; 2	0

• Echo throughput against no coverage: 0.18x and 0.19x on x86, 0.05x and 0.06x on arm64 (epoll and io_uring).
• Real -covermode in CI: one observation, run 36245950077. This is the table in the comment above.
• Laptop Docker, arm64, kernel 7.0.12, instrumented:
  • With coverage (11 rounds): epoll failed 11/11 [71.5, 100], io_uring 8/11 [39.0, 94.0].
  • Without coverage (16 rounds): both engines 0/16 [0, 20.6]. clientCloseFail was above 0 in 1/16 epoll and 2/16 io_uring rounds.

Mechanism per class

closeTimeout

• With coverage, both engines: H1, confirmed by injection.
  • The client's drain ends at the first 1 s read gap (pause_cancel_linux_test.go:215). The Close frame then queues behind the rest of the echo, and the single absolute 10 s wait (:228) expires while bytes are still arriving.
  • C1 had 459 close-timeouts. 457 of them received 0.29-2.03 MB during the wait, and the last byte came a median 0 ms before the deadline.
  • C2 re-arms the wait on every read. It had 0 close-timeouts, and 1,071 close waits reached EOF more than 10 s after Close.
• Instrumented, no coverage, x86 epoll only: supported.
  • 180 of 180 received 0.77-3.91 MB during the wait, with the last byte at most 504 ms before the deadline.
  • The instrumentation itself induces these: plain main had 0/20 against the instrumented C0's 6/20 (p = 0.02).
• Uninstrumented, no coverage: open.

clientCloseFail

Never asserted (#623). Always "i/o timeout" at the frame-completion write (:206-207). There are two mechanisms.

• (a) The server is behind (H1).
  • Found in 2,108 of 2,190 C1 give-ups: at the give-up the server's window is closed, and the handler has echoed since the flood ended.
  • The client write is a fixed 15 s writeAll. A blocked Go write waits for EPOLLOUT, which the kernel raises only after about a third of the client's ~2.6 MB send queue has drained.
  • epoll: confirmed. In C2, 1,226 of these writes finished more than 15 s after the flood, the slowest at 47.9 s.
  • io_uring: supported. C2's rescue there is its 1 s re-poll, not the longer wait. 369 writes finished 1-5 s after the flood, median 1.2 s, with only 3.6% (x86) or 6.6% (arm64) of the send queue drained. That shows the server was reading, not that a longer fixed wait would have worked.
• (b) The client stops processing ACKs (H3).
  • The client sets SO_RCVBUF to 32 KiB and never reads while it writes. It goes 93-99 s without processing an ACK.
  • Meanwhile the handler is idle in ReadMessage, with an empty, unpaused chanReader and no spill. The server socket has rx=0 and an open window, and holds 1 unacked segment at RTO backoff 8.
  • This state holds in 68/68 and 70/70 of the no-progress give-ups (C2, C2b), and in all 11 give-ups of the instrumented no-coverage runs, which were all io_uring.
  • Pre-registered primary: confirmed. Having the client empty its receive queue while it waits takes no-progress give-ups from 70 to 0 (P = 8.5e-22). Per process: epoll 16/20 against 0/20, io_uring 12/20 against 0/20.
  • The kernel mechanism is supported, not confirmed. Since 9ca48d616ed7 (v6.17), tcp_sequence() drops an end-of-window segment, before tcp_ack(), whenever the receive queue is not empty. Upstream 0e24d17bd966 describes this exact state and is in no release tag.
    • The pre-registered secondary failed by its letter: 17/44 opened within 200 ms, against a 50% kill line.
    • No kernel without the check was run. The laptop's 7.0.12 has it too.
  • The io_uring/epoll asymmetry is open.

otherWriteErr (epoll) and protoErr "broken pipe" (io_uring)

• H2, confirmed by injection. These errors are the server's echo of the client's own close with unread data, which sends an RST.
• Making every 4th client abort after the flood (C3) failed the epoll subtest 10/10, against C0's 3/10 on x86 (p = 0.0031) and 0/10 on arm64 (p = 1.1e-5).
• Across 11 instrumented runs there were 1,165 counted server errors, which equals the :255 counters exactly. None came before its own client's close.
• Removing the give-ups (C2) removed all 747 of C1's.
• io_uring's shows up as "broken pipe" because isCloseErr (inbound_sequence_linux_test.go:499-510) lists ECONNRESET but not EPIPE.

What the skeptic review changed

These points were checked against the raw logs and upheld:

• H1 fails its own pre-registered rule. The rule was no-progress give-ups above 1% of an engine's C2 connections. It fired for x86 epoll (1.15%), arm64 epoll (2.40%) and arm64 io_uring (2.60%).
  • So H1 is not the whole story. What survives is "everything that eventually completes is H1".
• H3's secondary also failed, so its kernel cause is reported as supported, not confirmed.
• The C2 pass depends on the missing assertion. "10/10 against 0/10" holds only because clientCloseFail is not asserted. C2 still had clientCloseFail above 0 in 9/10 x86 and 10/10 arm64 processes. Only C6 passes a complete oracle.
• The H1 kill arm changed two things on io_uring (see (a) above).
• "187 H3-core" came from classifier order. Without the order, C1 splits into 2,108 server-behind, 43 lost-ACK and 39 other.
• The no-coverage C0 vs C7 contrast was overstated. Counted by C0's rule, it is 14/20 against 5/20 (p = 0.01), not 3.3e-6. I do not quote any per-connection P value.

Harness/oracle artifact or product defect

class	verdict	status
closeTimeout with coverage or instrumentation	harness/oracle artifact (1 s drain heuristic plus an absolute 10 s wait)	confirmed (coverage), supported (instrumented)
closeTimeout, uninstrumented, no coverage	unknown: 1 io_uring event with no state; this issue's epoll symptom 0/100	open
clientCloseFail (a)	harness artifact (fixed 15 s behind EPOLLOUT's one-third threshold)	confirmed (epoll), supported (io_uring)
clientCloseFail (b)	kernel behaviour triggered by the test client; no celeris state involved	stall confirmed; kernel cause supported; engine asymmetry open
otherWriteErr / protoErr	oracle artifact (:268-290 blame the engine for the client's own RST)	confirmed

None of the classes is a distinct product defect, so nothing was filed for them. #714 is a separate defect found along the way: the instrumentation joined handler to client through Conn.Query.

Fix plan: lane B-2 (with #611, #623, #705, #706), branched from #671's final head

middleware/websocket/pause_cancel_linux_test.go:

1. Frame-completion and Close writes (:206, :224). Use progress-based waits: write in 1 s slices, and give up only after N s without the client's SIOCOUTQ shrinking, with a cap.
2. Close wait (:228-245). Re-arm it on every byte received, keep a cap, and log the latency, so that a real stall stays visible.
3. Client reads while it waits. It empties its receive queue without blocking between write slices. Keep SO_RCVBUF at 32 KiB, because that is what makes the echo SEND block, which is what the io_uring: WS recv-pause cancels by raw fd, killing the conn's in-flight SEND #482 test is for.
4. The WebSocket backpressure oracle counts clientCloseFail, prints it, and never asserts it: 66 of 96 connections failed and the test passed #623's clientCloseFail assertion lands in the same change as 1-3, never before. It needs a negative control that fails on a build with a real wedge, such as the middleware/websocket: a chunk that spills after the handler drained the channel is never promoted, so the connection wedges permanently (stranded spill) #705 stranded-spill repro or middleware/websocket: a pause decided on a stale depth snapshot wedges the connection permanently when the handler drains to empty first (distinct from #667) #672 without fix(websocket): apply pause/resume under the lock that decides them, and lift a stale pause (celeris#667, celeris#672) #671.
5. otherWriteErr/protoErr. With 1-3 in place, C6 had zero. Also add EPIPE to isCloseErr.

Also in lane B-2:

• The WebSocket inbound oracle reports frames as lost that its own handler abandoned, and never asserts the write errors that explain them #611 in inbound_sequence_linux_test.go: check its client for the same pattern.
• middleware/websocket: a chunk that spills after the handler drained the channel is never promoted, so the connection wedges permanently (stranded spill) #705 in engineread.go: under the new oracle, its wedge surfaces as a no-progress give-up with the server state captured.
• middleware/websocket tests: the native-engine start helper never reads Start's error, so a failed start is reported as a 30 s "server not ready within timeout" #706: the helper that ignores Start's error.

Overlap and follow-ups:

• fix(websocket): apply pause/resume under the lock that decides them, and lift a stale pause (celeris#667, celeris#672) #671 (middleware/websocket: chanReader applies pause/resume to the engine outside the lock that decides them, so a resume can be lost and the connection stops delivering (found by reading, not reproduced) #667 and middleware/websocket: a pause decided on a stale depth snapshot wedges the connection permanently when the handler drains to empty first (distinct from #667) #672) fixes real wedges that this test never showed here, since no give-up had their shape. It left the coverage reproducer at 10/10 failing, and this explains why. B-2 merges after fix(websocket): apply pause/resume under the lock that decides them, and lift a stale pause (celeris#667, celeris#672) #671.
• ci: set up CodeRabbit, Codecov, CodSpeed (celeris#690) #699: drop its ./middleware/websocket coverage exclusion only after B-2 lands, and only once a coverage run shows 0 give-ups.
• probatorium: allow-list -covermode (plus a shard-side -coverprofile) in tools/stresstally/plan.go, so that the coverage arm can run on celeris-stress.

Verification after B-2. Run celeris-stress, uninstrumented, with the new assertion:

• at least 200 processes per arch without coverage, sized for a 1.4% rate;
• the coverage arm;
• pass means 0 give-ups of any class.

This issue's original epoll close-timeout needs that same sized run. Under the new oracle, a real stall arrives as a no-progress give-up with its state captured, instead of a bare counter.

Evidence: evidence/celeris-633/investigation-20260927/

• MANIFEST.txt
• 60-report-verify.txt (+ scripts/report-verify.py)
• gh-stress/
• inject/: pre-registrations 10-, 11-, 12-, kernel reading 03-, lane report 50-.

[FumingPower3925]

Correction to instance 2 in the comment of 2026-09-16: the race tier's ws_echo_timeout is not this issue's symptom, and that instance was a whole-server slowdown

Instance 2 was race run 35024823796, epoll/arm64. It should not count as a data point for this issue, for two reasons. Instance 1, the io_uring CI closeTimeout 1/96, is not affected.

1. The walker cannot see a close-handshake stall. Code is probatorium validation/ws_echo.go at 6f5521c. The file is identical at 1a5b439, the sha that run used.

• ws_echo_timeout is counted when a fire's read reaches its 2 s hold deadline, which is set after dial (:278), while the sender has not finished or echoes are still outstanding (:368).
• The fire then returns without attempting a close (:426).
• A close handshake is attempted only after a fully drained fire, and only its success is counted (closeOK, :439 and :444).
• A failed or timed-out close is not counted anywhere.

So no ws_echo counter measures "never finishes its close". The 2026-09-16 comment described instance 2 with those words, and that description was wrong.

2. That instance coincided with a slowdown of the whole server. The data is from cell-05-auth_session_ratelimit-epoll on msr1, read from the property loop's 1 Hz samples:

interval	requests	requests/s	bytes read
the ten 1 s samples 21:36:55 to 21:37:04	13,605 to 15,299 each	13.6k to 15.3k	25.3 to 26.5 MB each
one 2.0 s sample, 21:37:04 to 21:37:06	9,816	4.9k	18.0 MB (9.0 MB/s)

• The incident was observed at 21:37:06.20.
• During that interval the engine as a whole ran at about a third of its rate.
• That 2 s interval is the only property-loop sample gap longer than 1.5 s in all 192 cells of the three race runs (34961642523, 35024823796, 36280496987).

One connection that stops delivering does not by itself cut the engine's request rate to a third. Something process-wide or host-wide slowed down for those two seconds. The artifact does not say what it was.

The newest race-tier ws_echo_timeout is not recorded here either. It is from run 36280496987 (auth_session_ratelimit/adaptive/amd64, on io_uring after the promotion). It is excluded for reason 1, and its classification is open.

Reproduce. The evidence is on the maintainer host, in evidence/cluster-tiers-20260927/race-36280496987/publish/:

• scripts/epoll_instance_gap.py writes derived/epoll_instance_gap.txt.
• scripts/code-cited.sh writes derived/code-cited.txt.