Skip to content

fix(jellyfin): decide container readiness by healthcheck, not docker logs --since - #120

Merged
bilbospocketses merged 1 commit into
masterfrom
fix/jellyfin-container-readiness
Aug 26, 2026
Merged

fix(jellyfin): decide container readiness by healthcheck, not docker logs --since#120
bilbospocketses merged 1 commit into
masterfrom
fix/jellyfin-container-readiness

Conversation

@bilbospocketses

Copy link
Copy Markdown
Owner

Reported as "the database date update keeps failing at starting the docker container." It was not failing to start the container.

What was actually happening

From your own operation log — every real step succeeded:

14:03:33 OK    Container 6a1bca888abd stopped
14:03:34 OK    Backup saved: jellyfin_20260826_190333.db
14:03:34 OK    SQL update applied: DateCreated = PremiereDate
14:03:34 STEP  Starting Docker container
14:03:35 STEP  Waiting for 'Startup complete' in container logs
14:04:37 FAIL  Jellyfin did not report 'Startup complete' within 60s

The container started and Jellyfin came back healthy. Only the readiness verification failed — and because one step covered both the start and the wait, a readiness timeout was indistinguishable from docker start refusing.

Root cause

Readiness polled docker logs --since <timestamp> for Jellyfin's Startup complete line. That primitive does not work on a long-lived container. Measured on the affected container (467k lines, June 6th onward):

--since 2026-06-01T00:00:00Z  ->  467,127 lines
--since 2026-08-01T00:00:00Z  ->   32,059 lines
--since 2026-08-26T19:00:00Z  ->        0 lines   <- the burst is right there
--since 2026-08-26T19:04:00Z  ->        0 lines
--tail 5                      ->        5 lines

Jellyfin logged Main: Startup complete 0:00:13.53 14 seconds after start. The check simply could not see it.

Nothing in this codebase changed. The same code succeeded on 2026-06-30 per its own operation log. The container's log grew and --since stopped behaving.

Fix

Two independent signals, whichever lands first:

  1. The container's own healthcheck — authoritative when the image defines one, and the Jellyfin image does (curl $HEALTHCHECK_URL, 30s interval).
  2. Fallback: docker logs -t --tail 200, comparing docker's own RFC3339 timestamps against the container's start time.

That timestamp comparison is what --since was really providing: a long-lived log holds the marker from every previous start, and matching one of those would report ready while the server is still booting — worse than the bug being fixed. There is a test pinning exactly that case, and another pinning --since out so it cannot creep back.

Budget goes 60s → 120s, because a healthcheck only flips on its first passing probe. The marker match stays case-sensitive deliberately: the same log carries Core startup complete and plugin lines like MediaBar Startup Completed, neither of which means the server is up.

Also drops an unconditional 3s sleep before the first probe — the readiness tests went from 15s to 94ms.

Not changed, deliberately

JellyfinService still shells docker as a bare PATH-resolved name. Docker is registered as an External dependency here, and this is an accepted standing exception to Local-Dependencies-Only.

Verification

  • 726 tests green in Release.
  • Both signals mutation-checked: removing the timestamp guard fails the stale-marker test; ignoring the healthcheck fails the healthy test.

… logs --since`

Reported as "the database date update keeps failing at starting the docker
container". It was not failing to start the container. Stop, backup, SQL
update and start all succeeded, and Jellyfin came back healthy - only the
readiness verification afterwards failed, and the step covering both was
labelled "Starting Docker container", so a readiness timeout was
indistinguishable from docker refusing to start.

Readiness polled `docker logs --since <timestamp>` for Jellyfin's
"Startup complete" line. That primitive does not work on a long-lived
container. Measured on the affected container (467k lines, June 6th onward):

  --since 2026-06-01T00:00:00Z  ->  467,127 lines
  --since 2026-08-01T00:00:00Z  ->   32,059 lines
  --since 2026-08-26T19:00:00Z  ->        0 lines   <- the burst is right there
  --tail 5                      ->        5 lines

Jellyfin logged "Main: Startup complete 0:00:13.53" fourteen seconds after
start; the check simply could not see it. Nothing in this codebase changed -
the same code succeeded on 2026-06-30 per its own operation log. The
container's log grew and --since stopped behaving.

Readiness is now two independent signals, whichever lands first: the
container's own healthcheck (authoritative when the image defines one, and
the Jellyfin image does), and failing that a --tail read whose docker
timestamps are compared against the container's start time. That comparison
is what --since was really providing: a long-lived log holds the marker from
every previous start, and matching one of those would report ready while the
server is still booting - worse than the bug being fixed. There is a test
pinning exactly that, and one pinning --since out so it cannot creep back.

The budget goes 60s -> 120s because a healthcheck only flips on its first
passing probe and 30s intervals are common (the Jellyfin image uses 30s).
The marker match stays case-sensitive on purpose: the same log carries
"Core startup complete" and plugin lines like "MediaBar Startup Completed",
neither of which means the server is up.

Also removes an unconditional 3s sleep that ran before the first probe.

Not changed, deliberately: JellyfinService still shells `docker` as a bare
PATH-resolved name. Docker is registered as an External dependency here, and
the user has accepted this as a standing exception to Local-Dependencies-Only.

726 tests green in Release; both readiness signals mutation-checked.
@bilbospocketses
bilbospocketses enabled auto-merge (squash) August 26, 2026 19:24
@bilbospocketses
bilbospocketses merged commit e190631 into master Aug 26, 2026
8 checks passed
@bilbospocketses
bilbospocketses deleted the fix/jellyfin-container-readiness branch August 26, 2026 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant