Kubernetes needs liveness and readiness to be distinguishable, and they are not
the same signal. Right now neither exists as a dedicated endpoint.
Why they differ here
- Liveness — the process is not deadlocked. Restarting fixes it.
- Readiness — this instance can serve or consume right now. Postgres and
Redis are reachable. Restarting does NOT fix a dependency being down, so a
failing dependency must fail readiness and NOT liveness. Getting this
backwards produces restart loops during a database blip, which is one of the
most common self-inflicted Kubernetes outages.
Scope
Definition of done
Killing Postgres makes readyz fail while healthz stays green, and the pod is
removed from the Service endpoints without being restarted.
Kubernetes needs liveness and readiness to be distinguishable, and they are not
the same signal. Right now neither exists as a dedicated endpoint.
Why they differ here
Redis are reachable. Restarting does NOT fix a dependency being down, so a
failing dependency must fail readiness and NOT liveness. Getting this
backwards produces restart loops during a database blip, which is one of the
most common self-inflicted Kubernetes outages.
Scope
GET /healthz— liveness. No dependency checks. Must stay cheapGET /readyz— readiness. Checks Postgres and Redis connectivityworkerandscheduler, which have no HTTP servertoday. Decide between a minimal probe server and a file-based liveness marker
SIGTERM: stop accepting work, drain the in-flightBullMQ job, then exit. Kubernetes sends
SIGTERMon every rollout, sowithout this every deploy drops jobs
terminationGracePeriodSecondsand whyDefinition of done
Killing Postgres makes
readyzfail whilehealthzstays green, and the pod isremoved from the Service endpoints without being restarted.