fix(server): filter benign read-idle-reap WARN on streaming follows#167
Merged
Conversation
The server's `idleTimeout` installs a read-only Netty `ReadTimeoutHandler`,
which resets only on inbound reads. A job-log / progress follow is write-only
(server -> client) once the GET is sent, so the outbound `JobLogStream`
keepalive can never reset that timer and every live follow is reaped on the
60s schedule -- surfacing as zio-http's `ZIO.logWarningCause("Fatal exception
in Netty", Cause.die(ReadTimeoutException))`. The follower already reconnects
transparently (#161), so the reap is benign transport noise, not a failure;
the WARN spam (two per cycle since the progress stream landed in #166) was the
only real damage.
Drop exactly that WARN at ProgressDisplay's ZLogger. The match is
over-specified -- the exact zio-http message AND a cause that is solely a
`ReadTimeoutException` defect -- so it stays scoped to the server reaper and
fails open: a zio-http rename or wrap just makes the (benign) WARN reappear
rather than hiding any genuine fatal-Netty signal.
Also fix the stale `CcasServer` comment that claimed the keepalive resets the
server's read timer; it cannot (writes don't reset a `ReadTimeoutHandler`).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015eWRWek8owPM5eDjatPVit
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The server's
idleTimeoutinstalls a read-only NettyReadTimeoutHandler(resets on inbound reads only). A job-log / progress follow (GET /api/jobs/{id}/{logs,progress}) is write-only server→client once the GET is sent, so the outboundJobLogStreamkeepalive can never reset that timer — every live follow is reaped on the 60s schedule, which zio-http logs asZIO.logWarningCause("Fatal exception in Netty", Cause.die(ReadTimeoutException)). The follower already reconnects transparently (#161), so the reap is benign transport noise; the WARN spam (two per cycle since the progress stream landed in #166) was the only real damage.This drops exactly that WARN at
ProgressDisplay'sZLogger:ReadTimeoutExceptiondefect (no failure, no other defect), so an unrelated bare read-timeout WARN still logs.CcasServercomment that claimed the keepalive resets the server read timer (it can't — writes don't reset aReadTimeoutHandler).Rejected alternatives: removing
idleTimeoutre-opens the indefinite dead-fd leak #158 closed (zio-http exposes no serverSO_KEEPALIVE); blanketlogWarningOnFatalError(false)would hide all fatal-Netty WARNs.Fixes
Log-noise from the read-idle reaper vs. write-only streaming follows (surfaced after #166 doubled it). No linked issue — diagnosed in-session.
Testing
TestProgressBar(+1 case): the reaper WARN is dropped; a different fatal-Netty WARN and a bareReadTimeoutExceptionunder a different message both still log.testLoggerSwallowsThrowstill passes (throwing thunk still caught).sbt testgreen via pre-push (1048 passed).v3.10.1source:ServerInboundHandler.exceptionCaughtlogsCause.die(t)witht= rawReadTimeoutException.INSTANCE, thenctx.close()— exactly the shape the filter matches.🤖 Generated with Claude Code
https://claude.ai/code/session_015eWRWek8owPM5eDjatPVit