fix(jellyfin): decide container readiness by healthcheck, not docker logs --since - #120
Merged
Merged
Conversation
… logs --since` Reported as "the database date update keeps failing at starting the docker container". It was not failing to start the container. Stop, backup, SQL update and start all succeeded, and Jellyfin came back healthy - only the readiness verification afterwards failed, and the step covering both was labelled "Starting Docker container", so a readiness timeout was indistinguishable from docker refusing to start. Readiness polled `docker logs --since <timestamp>` for Jellyfin's "Startup complete" line. That primitive does not work on a long-lived container. Measured on the affected container (467k lines, June 6th onward): --since 2026-06-01T00:00:00Z -> 467,127 lines --since 2026-08-01T00:00:00Z -> 32,059 lines --since 2026-08-26T19:00:00Z -> 0 lines <- the burst is right there --tail 5 -> 5 lines Jellyfin logged "Main: Startup complete 0:00:13.53" fourteen seconds after start; the check simply could not see it. Nothing in this codebase changed - the same code succeeded on 2026-06-30 per its own operation log. The container's log grew and --since stopped behaving. Readiness is now two independent signals, whichever lands first: the container's own healthcheck (authoritative when the image defines one, and the Jellyfin image does), and failing that a --tail read whose docker timestamps are compared against the container's start time. That comparison is what --since was really providing: a long-lived log holds the marker from every previous start, and matching one of those would report ready while the server is still booting - worse than the bug being fixed. There is a test pinning exactly that, and one pinning --since out so it cannot creep back. The budget goes 60s -> 120s because a healthcheck only flips on its first passing probe and 30s intervals are common (the Jellyfin image uses 30s). The marker match stays case-sensitive on purpose: the same log carries "Core startup complete" and plugin lines like "MediaBar Startup Completed", neither of which means the server is up. Also removes an unconditional 3s sleep that ran before the first probe. Not changed, deliberately: JellyfinService still shells `docker` as a bare PATH-resolved name. Docker is registered as an External dependency here, and the user has accepted this as a standing exception to Local-Dependencies-Only. 726 tests green in Release; both readiness signals mutation-checked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reported as "the database date update keeps failing at starting the docker container." It was not failing to start the container.
What was actually happening
From your own operation log — every real step succeeded:
The container started and Jellyfin came back healthy. Only the readiness verification failed — and because one step covered both the start and the wait, a readiness timeout was indistinguishable from
docker startrefusing.Root cause
Readiness polled
docker logs --since <timestamp>for Jellyfin'sStartup completeline. That primitive does not work on a long-lived container. Measured on the affected container (467k lines, June 6th onward):Jellyfin logged
Main: Startup complete 0:00:13.5314 seconds after start. The check simply could not see it.Nothing in this codebase changed. The same code succeeded on 2026-06-30 per its own operation log. The container's log grew and
--sincestopped behaving.Fix
Two independent signals, whichever lands first:
curl $HEALTHCHECK_URL, 30s interval).docker logs -t --tail 200, comparing docker's own RFC3339 timestamps against the container's start time.That timestamp comparison is what
--sincewas really providing: a long-lived log holds the marker from every previous start, and matching one of those would report ready while the server is still booting — worse than the bug being fixed. There is a test pinning exactly that case, and another pinning--sinceout so it cannot creep back.Budget goes 60s → 120s, because a healthcheck only flips on its first passing probe. The marker match stays case-sensitive deliberately: the same log carries
Core startup completeand plugin lines likeMediaBar Startup Completed, neither of which means the server is up.Also drops an unconditional 3s sleep before the first probe — the readiness tests went from 15s to 94ms.
Not changed, deliberately
JellyfinServicestill shellsdockeras a bare PATH-resolved name. Docker is registered as an External dependency here, and this is an accepted standing exception to Local-Dependencies-Only.Verification