Skip to content

Add Prometheus collector service for web_server and dramatiq workers - #1254

Open
anfimovdm wants to merge 1 commit into
masterfrom
issue-540-prometheus-collector
Open

Add Prometheus collector service for web_server and dramatiq workers#1254
anfimovdm wants to merge 1 commit into
masterfrom
issue-540-prometheus-collector

Conversation

@anfimovdm

@anfimovdm anfimovdm commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Resolves: AlmaLinux/build-system#540

Part 1 — the crash in #540 is already fixed on master

The duplicate Prometheus() registration was removed in c35f672, already merged. No code change was needed here; I verified it instead of assuming:

  • No add_middleware(Prometheus()) and no dramatiq.middleware.prometheus import remains in alws/.
  • Confirmed against dramatiq v1.17.1 that default_middleware does include Prometheus, so the explanatory comment in alws/dramatiq/__init__.py is accurate.
  • Live check — built the task-queue image and ran a worker against RabbitMQ exactly as compose does. Boot log shows exactly one exposition server and no traceback:
[dramatiq.MainProcess] [INFO] Dramatiq '1.17.1' is booting up.
...
[PID 166] [dramatiq.ForkProcess(0)] [INFO] Fork process
  'dramatiq.middleware.prometheus:_run_exposition_server' is ready for action.

Part 2 — this PR: the collector

Nothing scraped the endpoints added in #1230, so both the workers' :9191 and the web server's /metrics were serving data nobody collected. This adds a prometheus compose service (always on, like the rest of the stack) with scrape targets in monitoring/prometheus.yml:

Target Endpoint Metrics
web_server :8000/metrics/ http_request_duration_seconds, http_requests_total, function_duration_seconds
the 8 task_queue_* workers :9191/metrics dramatiq_messages_total, _message_errors_total, _message_retries_total, _message_rejects_total, _messages_inprogress, _message_duration_milliseconds

Targets are reached by compose service name over the default network, so no target needs a published host port — only Prometheus' own UI is exposed, on :9090. Worker targets carry service and queue labels, so up{queue="sign"} == 0 alerts on a dead worker before it has processed anything. 15d retention in the prometheus_data named volume.

Verification

Verified end-to-end on a throwaway network, not only by config inspection:

Check Result
promtool check config SUCCESS
All 10 targets resolve to real compose service names pass (script-diffed against docker-compose.yml)
All 8 queue labels match the actual -Q args pass (script-diffed)
web_server target UP; real http_requests_total{endpoint="/docs"} ingested
Worker target UP; enqueued a task and dramatiq_messages_total / _errors_total / _duration_milliseconds came back queryable with correct service + queue labels

Two things the live test caught that config review alone would have missed:

  1. /metrics 307-redirects to /metrics/. alws.app mounts the metrics ASGI app, so the bare path always redirects. Prometheus follows it, but metrics_path is set to /metrics/ to drop a redirect round-trip from every scrape.
  2. A worker's :9191 returns HTTP 200 with an empty body until it processes its first message. That is a healthy target with zero samples, not a broken one — worth knowing before someone reads it as a failure. It is also why the queue target label exists: it lands on the synthetic up/scrape_* series, which are present even when the worker has no dramatiq_* samples yet.

Notes for review

  • --web.enable-lifecycle is deliberately not set: :9090 is published to the host and the flag would expose remote /-/reload and /-/quit. Config edits apply with docker-compose restart prometheus; the TSDB is in a named volume, so nothing is lost. Happy to flip this if you'd rather have hot reload.
  • Scope is app-services only — no cAdvisor, node-exporter, or postgres/redis/rabbitmq exporters. Those would cover the remaining compose services (pulp_*, build_node, gitea_listener, ...) and can be a separate PR if wanted.
  • README gains a # Metrics section documenting the targets, how to add a worker, and — to keep Add GCP repos for AlmaLinux 8 and 9 #540 from recurring — that worker metrics come from dramatiq's default middleware and must never be registered a second time.

Nothing scraped the metrics endpoints added in #1230, so the dramatiq
workers' :9191 exposition servers and the web server's /metrics were
serving data nobody collected.

Add a prometheus compose service, always on like the rest of the
stack, with scrape targets in monitoring/prometheus.yml:

  * web_server:8000/metrics/ - http_request_duration_seconds,
    http_requests_total, function_duration_seconds
  * the eight task_queue_* workers on :9191 - dramatiq_* per
    queue and actor

Targets are reached by compose service name over the default
network, so no target needs a published host port; only Prometheus'
own UI is exposed on :9090. Worker targets carry service and queue
labels, so up{queue="sign"} alerts on a dead worker before it has
processed anything.

The web_server path keeps its trailing slash on purpose: alws.app
mounts the metrics ASGI app at "/metrics", so the bare path 307s
to "/metrics/".

--web.enable-lifecycle is left off since :9090 is published to the
host; config edits are applied by restarting the service.

Also document in README that worker metrics come from dramatiq's
default middleware and must not be registered a second time, which
is the crash fixed by dropping the duplicate registration.

Resolves: AlmaLinux/build-system#540
@anfimovdm
anfimovdm force-pushed the issue-540-prometheus-collector branch from 2302a25 to 223a584 Compare August 27, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

albs-web-server: duplicate Prometheus middleware crashes dramatiq workers with "Address already in use" on :9191

1 participant