feat(phase-14): observability — trace propagation, Prometheus/Grafana, metrics and Loki - #40
Merged
Merged
Conversation
Gateway ReverseProxy now forwards X-Request-ID/X-User-ID and builds X-Forwarded-For chain; metadata/upload use RequestID before logger and emit trace_id/request_id via zerolog RequestLogger.
Add prometheus:9090 and grafana:3001 services with healthchecks, provision Prometheus datasource, and alert rules for queue/OOM/disk. Update .env.example with PROMETHEUS_PORT/GRAFANA_* vars.
Add /metrics endpoints via prometheus_client (Python) and client_golang (Go) for gateway/metadata/upload/auth/transcoder, implement ffmpeg_duration_seconds histogram, upload_bytes counter, rabbitmq_queue_depth gauge and vod_cache_hit counter, and scrape transcoder:8004 via Prometheus.
Gateway already logs trace_id; add zerolog JSON config and RequestLogger with trace_id/request_id for metadata/upload, and FastAPI middleware for auth that logs JSON with trace_id and echoes X-Request-ID.
Include trace_id/request_id/service fields for unified JSON logs.
Mount dashboards to /etc/grafana/dashboards and provision datasource so Grafana shows data out of the box (queue depth, ffmpeg duration, upload bytes, vod cache).
Add Loki 3.0 and Promtail 3.0 services with docker_sd_configs, Loki datasource in Grafana, and promtail pipeline extracting trace_id/service/level for Explore queries.
Update up echo with prometheus/grafana/loki ports and add metrics/loki-logs/grafana/prometheus targets for phase 14.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11
Closes #12
Closes #13
Closes #14
Closes #15
Closes #38
Phase 14 — P1: Наблюдаемость — реализация по
docs/PLAN.md:228.Summary
X-Request-ID/X-User-ID/X-Forwarded-Forвgateway/proxyи логированиеtrace_idво всех сервисах (gateway/metadata/upload/auth)prometheus:9090иgrafana:3001в compose с healthcheck, datasource и дашбордомFlowix — Overviewrabbitmq_queue_depth,ffmpeg_duration_seconds,upload_bytes,vod_cache_hitчерезclient_golang/prometheus_clientво всех сервисахalert.rules.yml(queue>100, OOM, disk>80%)trace_id/request_id/servicedocker_sd_configsи Loki datasource в Grafana для централизованных логов ({service="gateway"} |= "test123")Acceptance criteria
X-Request-IDпрокидывается:curl -H X-Request-ID:test123 /api/v1/videos→docker logs gateway/metadata/auth | grep test123содержит IDprometheus:9090/-/healthy200,grafana:3001/api/health200,Status → TargetsUPrabbitmq_queue_depth,ffmpeg_duration_seconds{quality="360p"},upload_bytes,vod_cache_hitвидны вcurl :9090/api/v1/query?query=...и в Grafana дашборде (послеmake e2eupload_bytes 144k, ffmpeg_count 2)promtool check rules3 rules, алерты определеныdocker logs gateway | jq .trace_id— JSON сtrace_id/serviceво всех сервисахcurl -H X-Request-ID:test-loki-999 ...→curl "http://localhost:3100/loki/api/v1/query?query={service=\"gateway\"} |= \"test-loki-999\""возвращает лог сtrace_idVerification
make lint-go— 0 issuesmake lint-py— black/flake8/mypy successmake lint-front— ✔ No ESLintmake test-go— ok (metadata/gateway/upload)make test-py— 11 passed auth, 16 passed transcodermake up --build -d— все Healthy (postgres, minio, rabbitmq, prometheus Healthy, loki Healthy, promtail Started, grafana Healthy)make e2e— PASS (6s sample → 2 renditions, HLS via 8081/8080 200)curl -H X-Request-ID:test-loki-999→ Loki querytrace_id— successcurl -u admin:admin http://localhost:3001/api/search?query=flowix→ found Flowix — OverviewRisks
allow_structured_metadata: false(совместимость с schema v11 boltdb-shipper) — structured metadata отключена, trace_id как label (достаточно для фазы, миграция на v13/tsdb — бэклог)docker_sd_configsс/var/run/docker.sock— на OrbStack работает, на других Docker Desktop может требоватьDOCKER_HOSTrabbitmq_queue_depthпока Gauge 0 без реального опроса Management API — алерт QueueDepthHigh не сработает до poller (можно добавить в след. PR)