feat(metrics): integrate OpenTelemetry for metrics collection and export

[GauravRawat369]

What

Decision Engine now ships its metrics the way Hyperswitch does: pushed over OTLP/gRPC to
the cluster's OpenTelemetry collector, which vmagent already scrapes into VictoriaMetrics.
The separate Prometheus /metrics server and its [metrics] host/port config are removed.

• src/metrics.rs is rebuilt on OpenTelemetry behind a Prometheus-shaped facade
  (with_label_values(..).inc(), start_timer(), observe(), set()), so none of the
  call sites change. Metric names, labels, and histogram buckets are unchanged.
• New [log.telemetry] config (metrics_enabled, ignore_errors,
  otel_exporter_otlp_endpoint, otel_exporter_otlp_timeout), overridable as
  DECISION_ENGINE__LOG__TELEMETRY__*. Off by default; when off every instrument is a
  no-op. The push cadence is fixed at 3s with a 10s timeout, the same as Hyperswitch.
• metrics::init runs right after logger setup, before anything records a metric.
• Dependencies: opentelemetry, opentelemetry_sdk, opentelemetry-otlp at 0.27 (matches
  Hyperswitch and the tonic 0.12 already in the lockfile). prometheus, lazy_static, and
  the metrics listener are gone. internal-logs is enabled so export failures reach tracing.
• Local dev: the compose monitoring profile gains an otel-collector
  (config/otel-collector.yaml); Prometheus scrapes the collector instead of the app.
  oneclick.sh starts the collector and Prometheus and runs the API with metrics enabled.

[log.telemetry]
metrics_enabled = true
ignore_errors = true
otel_exporter_otlp_endpoint = "http://otel-collector-opentelemetry-collector.monitoring.svc.cluster.local:4317"

Flow

flowchart LR
    subgraph app["Decision Engine pod"]
        DE["Instruments in src/metrics.rs<br/>counters · histograms · gauge"]
        RD["PeriodicReader<br/>every 3s, cumulative"]
        DE --> RD
    end

    subgraph col["otel-collector · monitoring namespace"]
        IN["OTLP/gRPC receiver<br/>:4317"]
        K8S["k8sattributes<br/>adds source_namespace, source_pod"]
        OUT["Prometheus exporter<br/>:9898 /metrics"]
        IN --> K8S --> OUT
    end

    subgraph store["VictoriaMetrics"]
        VA["vmagent<br/>scrapes :9898 every 15s<br/>via the collector's ServiceMonitor"]
        VS["vmstorage<br/>persistent volumes, 1y prod / 3M sandbox"]
        VA --> VS
    end

    G["Grafana<br/>VictoriaMetrics datasource<br/>exported_job=&quot;decision-engine&quot;"]

    RD -- "push (outbound only, no port on DE)" --> IN
    OUT -- "pull" --> VA
    VS --> G

    HS["Hyperswitch router"] -. "same push, same collector" .-> IN

Loading

Locally the monitoring Compose profile plays the same roles with an otel-collector container
in place of the cluster collector and Prometheus in place of vmagent + VictoriaMetrics:

flowchart LR
    DE["Decision Engine<br/>(oneclick or Compose)"] -- "OTLP push :4317" --> C["otel-collector container"]
    C -- ":9898 /metrics, scraped every 15s" --> P["Prometheus :9090"]
    P --> GF["Grafana"]

Loading

Why

Nothing in the clusters scraped the old metrics port: no ServiceMonitor, no vmagent job, no
annotation. The metrics existed but never reached Grafana. Hyperswitch avoids the scrape
problem entirely by pushing to the collector, so every link after that (collector →
vmagent → VictoriaMetrics → the Grafana VictoriaMetrics datasource) already exists and DE
inherits it with no monitoring-stack changes. Push also means no port to keep reachable,
no per-pod discovery, and nothing to reconfigure when pods move or scale.

Metrics exported

api_requests_total{endpoint}, api_requests_by_status{endpoint,status},
api_latency_seconds{endpoint}, routing_decisions_total{approach,status},
routing_rule_hits_total{rule_name}, analytics_events_total{flow_type},
analytics_events_dropped_total{stream,reason}, analytics_sink_queue_depth{stream},
analytics_kafka_produce_total{stream,result},
analytics_kafka_delivery_latency_seconds{stream}, plus the two declared-but-unrecorded
analytics_sink_writes_total and analytics_sink_write_latency_seconds.

Notes for dashboards: the collector appends _total to counters that lack it
(api_requests_by_status → api_requests_by_status_total); the app's service.name
arrives as job="decision-engine", which vmagent renames to exported_job because the
collector's ServiceMonitor does not honour labels.

Rollout

Per environment, add to the DE deployment's extraEnvVars in hyperswitch-infra:

DECISION_ENGINE__LOG__TELEMETRY__METRICS_ENABLED: "true"
DECISION_ENGINE__LOG__TELEMETRY__IGNORE_ERRORS: "true"
DECISION_ENGINE__LOG__TELEMETRY__OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector-opentelemetry-collector.monitoring.svc.cluster.local:4317"

The old DECISION_ENGINE__METRICS__* variables and the chart's metrics service port become
unused but harmless. The old us-east-1 cluster has no collector; do not enable there.

Verification

• cargo check, cargo clippy --tests, cargo fmt clean on the postgres backend.
• Two new unit tests: the facade exports the expected names, labels, and values through an
  in-memory exporter; the OTLP reader builds and shuts down cleanly with no collector
  reachable.
• End to end locally: oneclick.sh → decide-gateway and update-gateway-score traffic →
  series visible on the collector, in Prometheus, and on a Grafana dashboard.

[Copilot]

🟡 Changes recommended

There are a few concrete reliability/operability issues (label/value arity enforcement only in debug builds, unvalidated export interval edge cases, and missing port preflight checks in oneclick) that should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR migrates the service’s metrics pipeline from a Prometheus scrape endpoint to OpenTelemetry metrics instruments, exported over OTLP/gRPC to an OpenTelemetry collector (with Prometheus scraping the collector in local/dev setups).

Changes:

• Replaced Prometheus client metrics and /metrics server with an OpenTelemetry-based metrics facade (CounterVec/HistogramVec/IntGaugeVec) and OTLP push exporter initialization.
• Added log.telemetry configuration to control metrics enablement/export endpoint/interval and wired early initialization in open_router startup.
• Updated local/dev tooling and docs (Compose + oneclick + Prometheus config) to include an otel-collector and scrape it instead of the app.

File summaries

File	Description
src/metrics.rs	Replaces Prometheus registry instruments with an OTel metrics facade and OTLP push pipeline initialization + tests.
src/logger/config.rs	Adds log.telemetry config section (LogTelemetry) for OTLP metrics export settings.
src/decider/gatewaydecider/volume_commitment/server.rs	Updates import to use the crate-wide ConfigurationError after metrics server removal.
src/config.rs	Removes now-obsolete metrics server config from GlobalConfig.
src/bin/open_router.rs	Initializes OTel metrics pipeline early during startup and removes metrics server task.
README.md	Updates local oneclick description to include otel-collector/Prometheus and new metrics endpoints.
oneclick.sh	Brings up otel-collector + Prometheus, adds health checks, and enables OTLP metrics for the native run.
docs/local-setup.md	Updates profiles/flow docs to reflect OTel collector + Prometheus and enablement flags.
docs/configuration.md	Replaces [metrics] scrape config with [log.telemetry] OTLP push config and environment variable examples.
docker-compose.yaml	Removes app metrics port, adds otel-collector service, and configures app OTLP env vars.
config/prometheus.yaml	Switches Prometheus scrape target to otel-collector:9898 with honor_labels.
config/otel-collector.yaml	Adds collector config for OTLP ingest (4317) and Prometheus exporter (9898).
config/docker-configuration.toml	Removes [metrics], adds [log.telemetry] defaults for containerized runs.
config/development.toml	Removes [metrics], adds [log.telemetry] defaults for local runs.
config.example.toml	Adds example [log.telemetry] section for OTLP metrics push.
Cargo.toml	Removes Prometheus deps and adds opentelemetry, opentelemetry_sdk, opentelemetry-otlp (plus testing feature for dev).
Cargo.lock	Updates lockfile for OpenTelemetry dependency graph and removes Prometheus crates.

Review details

• Files reviewed: 16/17 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.