Skip to content

Make queue_worker health check reflect worker liveness, not scheduler - #225

Merged
morcen merged 2 commits into
mainfrom
fix/issue-108-queue-worker-heartbeat
Sep 15, 2026
Merged

morcen merged 2 commits into
mainfrom
fix/issue-108-queue-worker-heartbeat

Conversation

@morcen

@morcen morcen commented Sep 14, 2026

Copy link
Copy Markdown
Owner

What was broken

/health/detailed's queue_worker status is meant to indicate whether webhook deliveries are actively being processed, but it only proved the scheduler container's cron was alive — not that the queue container (which runs queue:work and executes SendWebhook) was running at all.

The queue:heartbeat artisan command wrote the queue:heartbeat Redis key directly, and was itself only ever invoked by the scheduler's cron (routes/console.php). The scheduler and queue containers are separate processes, so if the queue container crashed, OOMed, or its queue:work process died while scheduler kept running, /health/detailed would keep reporting "queue_worker": "ok" — a false positive in exactly the failure mode it exists to catch. Every triggered webhook would silently queue up and never be delivered, with no monitoring signal at all.

What changed

  • Added App\Jobs\WriteQueueHeartbeat, a queued job dispatched onto the same webhooks queue SendWebhook runs on. Its handle() is what now writes the queue:heartbeat Redis key.
  • QueueHeartbeat (the artisan command the scheduler runs every minute) now dispatches WriteQueueHeartbeat::dispatch() instead of writing the key itself.

Since the heartbeat key is only written once an actual queue worker pulls WriteQueueHeartbeat off the webhooks queue and executes it, staleness now reflects whether a worker is alive and processing jobs — not just whether the scheduler container is ticking. A dead queue:work process now correctly surfaces as queue_worker: stale within the existing 2-minute staleness window, even while the scheduler keeps running.

  • Updated CLAUDE.md, ARCHITECTURE.md, DEPLOYMENT.md, and WEBHOOK_README.md, which previously documented the heartbeat as a scheduler-liveness signal.
  • Added tests/Feature/QueueHeartbeatTest.php covering: the command dispatches the job onto the webhooks queue, the command itself never writes the Redis key directly, the job writes the current timestamp when handled, and the job is queued on webhooks.

Testing

  • vendor/bin/pint --dirty — clean
  • php artisan test — full suite passes (256 passed, 1 pre-existing skip)

Fixes #108

morcen and others added 2 commits September 14, 2026 19:20
…duler

The /health/detailed queue_worker status was derived from a Redis
heartbeat key written directly by the queue:heartbeat artisan command,
which runs on the scheduler's cron. Since the scheduler and queue
worker are separate processes/containers, this only proved the
scheduler was alive — if the queue worker crashed or its process died
while the scheduler kept ticking, the heartbeat stayed fresh and
/health/detailed kept reporting queue_worker as ok, giving false
confidence while triggered webhooks silently queued up and were never
delivered.

queue:heartbeat now dispatches a WriteQueueHeartbeat job onto the same
webhooks queue SendWebhook runs on, instead of writing the Redis key
itself. The key is only written once an actual queue worker pulls the
job off that queue and executes it, so staleness now reflects whether
a worker is alive and processing jobs rather than whether the
scheduler container is ticking.

Fixes #108
CI's Lint (Pint) check runs with --preset=psr12, which requires
parentheses on `new ClassName()` expressions. Fixes the two
constructor calls in the new heartbeat test to match.
@morcen
morcen merged commit 37970d2 into main Sep 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

/health "queue_worker" status reflects the scheduler's cron heartbeat, not the actual queue worker process

2 participants