Add Prometheus collector service for web_server and dramatiq workers - #1254
Open
anfimovdm wants to merge 1 commit into
Open
Add Prometheus collector service for web_server and dramatiq workers#1254anfimovdm wants to merge 1 commit into
anfimovdm wants to merge 1 commit into
Conversation
Nothing scraped the metrics endpoints added in #1230, so the dramatiq workers' :9191 exposition servers and the web server's /metrics were serving data nobody collected. Add a prometheus compose service, always on like the rest of the stack, with scrape targets in monitoring/prometheus.yml: * web_server:8000/metrics/ - http_request_duration_seconds, http_requests_total, function_duration_seconds * the eight task_queue_* workers on :9191 - dramatiq_* per queue and actor Targets are reached by compose service name over the default network, so no target needs a published host port; only Prometheus' own UI is exposed on :9090. Worker targets carry service and queue labels, so up{queue="sign"} alerts on a dead worker before it has processed anything. The web_server path keeps its trailing slash on purpose: alws.app mounts the metrics ASGI app at "/metrics", so the bare path 307s to "/metrics/". --web.enable-lifecycle is left off since :9090 is published to the host; config edits are applied by restarting the service. Also document in README that worker metrics come from dramatiq's default middleware and must not be registered a second time, which is the crash fixed by dropping the duplicate registration. Resolves: AlmaLinux/build-system#540
anfimovdm
force-pushed
the
issue-540-prometheus-collector
branch
from
August 27, 2026 15:17
2302a25 to
223a584
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves: AlmaLinux/build-system#540
Part 1 — the crash in #540 is already fixed on
masterThe duplicate
Prometheus()registration was removed in c35f672, already merged. No code change was needed here; I verified it instead of assuming:add_middleware(Prometheus())and nodramatiq.middleware.prometheusimport remains inalws/.default_middlewaredoes includePrometheus, so the explanatory comment inalws/dramatiq/__init__.pyis accurate.Part 2 — this PR: the collector
Nothing scraped the endpoints added in #1230, so both the workers'
:9191and the web server's/metricswere serving data nobody collected. This adds aprometheuscompose service (always on, like the rest of the stack) with scrape targets inmonitoring/prometheus.yml:web_server:8000/metrics/http_request_duration_seconds,http_requests_total,function_duration_secondstask_queue_*workers:9191/metricsdramatiq_messages_total,_message_errors_total,_message_retries_total,_message_rejects_total,_messages_inprogress,_message_duration_millisecondsTargets are reached by compose service name over the default network, so no target needs a published host port — only Prometheus' own UI is exposed, on
:9090. Worker targets carryserviceandqueuelabels, soup{queue="sign"} == 0alerts on a dead worker before it has processed anything. 15d retention in theprometheus_datanamed volume.Verification
Verified end-to-end on a throwaway network, not only by config inspection:
promtool check configdocker-compose.yml)queuelabels match the actual-Qargsweb_servertargethttp_requests_total{endpoint="/docs"}ingesteddramatiq_messages_total/_errors_total/_duration_millisecondscame back queryable with correctservice+queuelabelsTwo things the live test caught that config review alone would have missed:
/metrics307-redirects to/metrics/.alws.appmounts the metrics ASGI app, so the bare path always redirects. Prometheus follows it, butmetrics_pathis set to/metrics/to drop a redirect round-trip from every scrape.:9191returns HTTP 200 with an empty body until it processes its first message. That is a healthy target with zero samples, not a broken one — worth knowing before someone reads it as a failure. It is also why thequeuetarget label exists: it lands on the syntheticup/scrape_*series, which are present even when the worker has nodramatiq_*samples yet.Notes for review
--web.enable-lifecycleis deliberately not set::9090is published to the host and the flag would expose remote/-/reloadand/-/quit. Config edits apply withdocker-compose restart prometheus; the TSDB is in a named volume, so nothing is lost. Happy to flip this if you'd rather have hot reload.pulp_*,build_node,gitea_listener, ...) and can be a separate PR if wanted.# Metricssection documenting the targets, how to add a worker, and — to keep Add GCP repos for AlmaLinux 8 and 9 #540 from recurring — that worker metrics come from dramatiq's default middleware and must never be registered a second time.