Make queue_worker health check reflect worker liveness, not scheduler - #225
Merged
Merged
Conversation
…duler The /health/detailed queue_worker status was derived from a Redis heartbeat key written directly by the queue:heartbeat artisan command, which runs on the scheduler's cron. Since the scheduler and queue worker are separate processes/containers, this only proved the scheduler was alive — if the queue worker crashed or its process died while the scheduler kept ticking, the heartbeat stayed fresh and /health/detailed kept reporting queue_worker as ok, giving false confidence while triggered webhooks silently queued up and were never delivered. queue:heartbeat now dispatches a WriteQueueHeartbeat job onto the same webhooks queue SendWebhook runs on, instead of writing the Redis key itself. The key is only written once an actual queue worker pulls the job off that queue and executes it, so staleness now reflects whether a worker is alive and processing jobs rather than whether the scheduler container is ticking. Fixes #108
CI's Lint (Pint) check runs with --preset=psr12, which requires parentheses on `new ClassName()` expressions. Fixes the two constructor calls in the new heartbeat test to match.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was broken
/health/detailed'squeue_workerstatus is meant to indicate whether webhook deliveries are actively being processed, but it only proved the scheduler container's cron was alive — not that the queue container (which runsqueue:workand executesSendWebhook) was running at all.The
queue:heartbeatartisan command wrote thequeue:heartbeatRedis key directly, and was itself only ever invoked by the scheduler's cron (routes/console.php). Theschedulerandqueuecontainers are separate processes, so if thequeuecontainer crashed, OOMed, or itsqueue:workprocess died whileschedulerkept running,/health/detailedwould keep reporting"queue_worker": "ok"— a false positive in exactly the failure mode it exists to catch. Every triggered webhook would silently queue up and never be delivered, with no monitoring signal at all.What changed
App\Jobs\WriteQueueHeartbeat, a queued job dispatched onto the samewebhooksqueueSendWebhookruns on. Itshandle()is what now writes thequeue:heartbeatRedis key.QueueHeartbeat(the artisan command the scheduler runs every minute) now dispatchesWriteQueueHeartbeat::dispatch()instead of writing the key itself.Since the heartbeat key is only written once an actual queue worker pulls
WriteQueueHeartbeatoff thewebhooksqueue and executes it, staleness now reflects whether a worker is alive and processing jobs — not just whether the scheduler container is ticking. A deadqueue:workprocess now correctly surfaces asqueue_worker: stalewithin the existing 2-minute staleness window, even while the scheduler keeps running.CLAUDE.md,ARCHITECTURE.md,DEPLOYMENT.md, andWEBHOOK_README.md, which previously documented the heartbeat as a scheduler-liveness signal.tests/Feature/QueueHeartbeatTest.phpcovering: the command dispatches the job onto thewebhooksqueue, the command itself never writes the Redis key directly, the job writes the current timestamp when handled, and the job is queued onwebhooks.Testing
vendor/bin/pint --dirty— cleanphp artisan test— full suite passes (256 passed, 1 pre-existing skip)Fixes #108