Skip to content

Worker and scheduler readiness reflecting consumer state #16

Description

@FullFran

Kubernetes needs liveness and readiness to be distinguishable, and they are not
the same signal. Right now neither exists as a dedicated endpoint.

Why they differ here

  • Liveness — the process is not deadlocked. Restarting fixes it.
  • Readiness — this instance can serve or consume right now. Postgres and
    Redis are reachable. Restarting does NOT fix a dependency being down, so a
    failing dependency must fail readiness and NOT liveness. Getting this
    backwards produces restart loops during a database blip, which is one of the
    most common self-inflicted Kubernetes outages.

Scope

  • GET /healthz — liveness. No dependency checks. Must stay cheap
  • GET /readyz — readiness. Checks Postgres and Redis connectivity
  • Equivalent probes for worker and scheduler, which have no HTTP server
    today. Decide between a minimal probe server and a file-based liveness marker
  • Graceful shutdown on SIGTERM: stop accepting work, drain the in-flight
    BullMQ job, then exit. Kubernetes sends SIGTERM on every rollout, so
    without this every deploy drops jobs
  • Document the chosen terminationGracePeriodSeconds and why

Definition of done

Killing Postgres makes readyz fail while healthz stays green, and the pod is
removed from the Service endpoints without being restarted.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestkubernetesKubernetes deployment and orchestration

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions