Skip to content

fix(server): filter benign read-idle-reap WARN on streaming follows#167

Merged
Sootopolis merged 1 commit into
mainfrom
fix/server-read-idle-reap-log-noise
Jul 16, 2026
Merged

fix(server): filter benign read-idle-reap WARN on streaming follows#167
Sootopolis merged 1 commit into
mainfrom
fix/server-read-idle-reap-log-noise

Conversation

@Sootopolis

Copy link
Copy Markdown
Owner

What

The server's idleTimeout installs a read-only Netty ReadTimeoutHandler (resets on inbound reads only). A job-log / progress follow (GET /api/jobs/{id}/{logs,progress}) is write-only server→client once the GET is sent, so the outbound JobLogStream keepalive can never reset that timer — every live follow is reaped on the 60s schedule, which zio-http logs as ZIO.logWarningCause("Fatal exception in Netty", Cause.die(ReadTimeoutException)). The follower already reconnects transparently (#161), so the reap is benign transport noise; the WARN spam (two per cycle since the progress stream landed in #166) was the only real damage.

This drops exactly that WARN at ProgressDisplay's ZLogger:

  • Scoped — matches the exact zio-http message and a cause that is solely a ReadTimeoutException defect (no failure, no other defect), so an unrelated bare read-timeout WARN still logs.
  • Fails open — a zio-http rename/wrap on upgrade just makes the (benign) WARN reappear rather than hiding any genuine fatal-Netty signal.
  • Also fixes the stale CcasServer comment that claimed the keepalive resets the server read timer (it can't — writes don't reset a ReadTimeoutHandler).

Rejected alternatives: removing idleTimeout re-opens the indefinite dead-fd leak #158 closed (zio-http exposes no server SO_KEEPALIVE); blanket logWarningOnFatalError(false) would hide all fatal-Netty WARNs.

Fixes

Log-noise from the read-idle reaper vs. write-only streaming follows (surfaced after #166 doubled it). No linked issue — diagnosed in-session.

Testing

  • TestProgressBar (+1 case): the reaper WARN is dropped; a different fatal-Netty WARN and a bare ReadTimeoutException under a different message both still log. testLoggerSwallowsThrow still passes (throwing thunk still caught).
  • Full sbt test green via pre-push (1048 passed).
  • Verified against zio-http v3.10.1 source: ServerInboundHandler.exceptionCaught logs Cause.die(t) with t = raw ReadTimeoutException.INSTANCE, then ctx.close() — exactly the shape the filter matches.

🤖 Generated with Claude Code

https://claude.ai/code/session_015eWRWek8owPM5eDjatPVit

The server's `idleTimeout` installs a read-only Netty `ReadTimeoutHandler`,
which resets only on inbound reads. A job-log / progress follow is write-only
(server -> client) once the GET is sent, so the outbound `JobLogStream`
keepalive can never reset that timer and every live follow is reaped on the
60s schedule -- surfacing as zio-http's `ZIO.logWarningCause("Fatal exception
in Netty", Cause.die(ReadTimeoutException))`. The follower already reconnects
transparently (#161), so the reap is benign transport noise, not a failure;
the WARN spam (two per cycle since the progress stream landed in #166) was the
only real damage.

Drop exactly that WARN at ProgressDisplay's ZLogger. The match is
over-specified -- the exact zio-http message AND a cause that is solely a
`ReadTimeoutException` defect -- so it stays scoped to the server reaper and
fails open: a zio-http rename or wrap just makes the (benign) WARN reappear
rather than hiding any genuine fatal-Netty signal.

Also fix the stale `CcasServer` comment that claimed the keepalive resets the
server's read timer; it cannot (writes don't reset a `ReadTimeoutHandler`).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015eWRWek8owPM5eDjatPVit
@Sootopolis
Sootopolis merged commit b0680e8 into main Jul 16, 2026
1 check passed
@Sootopolis
Sootopolis deleted the fix/server-read-idle-reap-log-noise branch July 16, 2026 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant