feat(observability): configure opt-in OpenTelemetry export

[nerdalert]

What does this PR do?

Add opt-in OpenTelemetry configuration for AGN (AI Gateway Networking) gateways, so operators can export gateway HTTP and routing spans without putting collector settings or credentials in routing overlays.

• Build grid-gateway with the Praxis OTLP exporter and Praxis AI routing-span features; initialize tracing at startup and hold its guard through shutdown.
• Render validated telemetry settings and trace-context handling from both GatewayRef.consumerConfig and the Praxis gateway Helm chart. An omitted or empty OTLP endpoint uses the process environment; invalid sampling rates and credential-bearing endpoint URLs are rejected.
• Support Secret-backed exporter headers, document image and restart requirements, and roll chart-managed pods when telemetry settings change. Telemetry remains disabled by default.
• Document the spans and privacy limits, including serving-overlay revision attributes where the routing filter supplies them.

Merge order: This change can merge before the Praxis framework tracing change. The normal Grid build remains on registry Praxis 0.7.1; no fork dependency is pinned. On that version, opt-in telemetry exports local HTTP and AI routing spans and forwards W3C headers, but edge and provider exported spans are not yet one linked trace. After the framework change is released, Grid must update its dependency, rebuild the gateway image, and verify exported parent IDs against that exact image before claiming cross-gateway linkage.

Which issue(s) does this relate to?

Related to #260. This PR delivers the opt-in configuration and export groundwork. Cross-gateway parentage remains an open acceptance criterion for that issue.

Checklist

• Signed off all commits (git commit -s)
• Tests added or updated
• Documentation updated (if applicable)
• make lint && make test && make test-integration passes locally

make lint, make test, and make helm-lint passed on a clean checkout of this branch; Helm reported 351 passes and one optional API-key runtime skip. A locked gateway image built from the same checkout, and a collector-backed check against that image confirmed local span export, routing revision attributes, and W3C header propagation. It also confirmed the documented cross-gateway linkage gap on Praxis 0.7.1. Grid has no make test-integration target.

Does this introduce a breaking change?

No. Telemetry is opt-in and disabled by default. Enabling it requires a grid-gateway image built with the tracing features; the chart's default Praxis AI image does not provide this Grid-specific build.

Summary by CodeRabbit

• New Features
  • Added optional OpenTelemetry trace export for Grid gateways, with settings for the OTLP endpoint, sampling, service metadata, and batch export.
  • Added W3C trace-context propagation. Collector credentials can be supplied through Secret-backed environment variables.
• Documentation
  • Added setup guidance, configuration examples, image requirements, and notes on current trace-linkage limitations.
• Validation
  • Added checks for telemetry settings, including endpoint formats and sampling and batch limits.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: praxis-proxy/coderabbit/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 615c7b1f-3dc7-4c21-9e99-1bea49de059b

📥 Commits

Reviewing files that changed from the base of the PR and between 145e9db and 30a2938.

⛔ Files ignored due to path filters (1)

• gateway/Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (19)

• charts/grid-operator/templates/crds/gridnetwork.yaml
• charts/praxis-gateway/README.md
• charts/praxis-gateway/templates/_helpers.tpl
• charts/praxis-gateway/templates/deployment.yaml
• charts/praxis-gateway/templates/gateway-config.yaml
• charts/praxis-gateway/tests/telemetry_test.yaml
• charts/praxis-gateway/values.schema.json
• charts/praxis-gateway/values.yaml
• deploy/crds/gridnetwork.yaml
• docs/README.md
• docs/architecture/consumer-config.md
• docs/architecture/opentelemetry.md
• gateway/Cargo.toml
• gateway/src/main.rs
• gateway/tests/startup_log.rs
• operator/src/controller/grid_network.rs
• operator/src/crd/grid_network.rs
• operator/src/resources/consumer_config.rs
• scripts/verify-helm-chart.sh

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

• praxis-proxy/praxis (manual)
• praxis-proxy/conventions (manual)

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

---

📝 Walkthrough

Walkthrough

The change adds opt-in OpenTelemetry configuration for Grid gateways through the operator and Helm chart. It renders Praxis telemetry settings and trace-context propagation, enables gateway exporter features, and documents credential handling and trace-linkage limits.

Changes

Gateway telemetry

Layer / File(s)	Summary
Telemetry configuration contracts operator/src/crd/grid_network.rs, charts/grid-operator/templates/crds/gridnetwork.yaml, deploy/crds/gridnetwork.yaml, charts/praxis-gateway/values.schema.json, charts/praxis-gateway/values.yaml	Operator and Helm configuration add optional telemetry settings. Validation constrains endpoint formats, sampling rates, resource attributes, and batch values.
Operator Praxis configuration operator/src/controller/grid_network.rs, operator/src/resources/consumer_config.rs, docs/architecture/consumer-config.md	The operator passes telemetry settings to the consumer-config renderer. The renderer validates them and adds root-level telemetry settings and a trace_context filter when telemetry is supplied. Tests cover defaults, rendering, and validation.
Helm rendering and deployment charts/praxis-gateway/templates/*, charts/praxis-gateway/tests/telemetry_test.yaml, scripts/verify-helm-chart.sh, charts/praxis-gateway/README.md	The chart renders telemetry settings and trace_context when enabled, checks the image flavor and generated configuration requirements, and adds a telemetry checksum annotation. Tests cover rendering and invalid values.
Gateway exporter support and lifecycle gateway/Cargo.toml, gateway/src/main.rs, gateway/tests/startup_log.rs, docs/README.md, docs/architecture/opentelemetry.md	The gateway enables OpenTelemetry dependency features and explicitly drops the tracing guard after the server returns. Documentation covers exporter configuration, Secret-backed credentials, supported spans, and Praxis trace-linkage limits. A startup test checks that a fatal startup error is logged.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant NetworkAdmin
  participant GridOperator
  participant PraxisConfig
  participant GridGateway
  participant OTLPCollector
  NetworkAdmin->>GridOperator: Set consumerConfig.telemetry
  GridOperator->>PraxisConfig: Render root telemetry and trace_context
  PraxisConfig-->>GridGateway: Supply runtime telemetry configuration
  GridGateway->>OTLPCollector: Export spans through OTLP

Loading

Suggested reviewers: hexfusion

Merge Risk: ⚪ Minimal · up to 30a29

This adds opt-in OpenTelemetry export that is off by default, so existing deployments should not change. No blocking issue was found in the supplied changes.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to 30a29

Telemetry is opt-in, validated, and keeps collector credentials out of generated configuration. However, removing exporter settings can remain unapplied when routing reconciliation is skipped, leaving an old destination available after restart. Production permissions, outbound-network restrictions, and collector-credential isolation remain unverified.

Retained concerns

• Medium · security · inferred: Withdrawal of operator-generated exporter settings depends on successful routing reconciliation. Empty candidates or overlay render/distribution failures skip consumer configuration processing, so a previously generated telemetry endpoint can remain in the ConfigMap even after telemetry is removed from an otherwise enabled consumerConfig. Restarting against that retained ConfigMap can reuse the old destination. The retention mechanism predates this PR, but retaining external trace-export configuration is a new security-relevant consequence. Normal successful removal and the documented restart requirement are counterevidence; the concern applies to skipped transitions, not an expectation of immediate live reconfiguration.

Security review details

Security Blast Radius

• inferred — The evidenced exposure unit is an enabled gateway process and its sampled HTTP and routing activity. A configuration author controlling multiple gateway references could select exporter settings for each gateway whose configuration the operator is authorized to write. Actual tenant, namespace, and network reach remain deployment-dependent; cluster-wide exploitation is not established.

Security Findings and Attack Paths

• inferred — A principal authorized to modify gateway configuration can select an explicit exporter destination. That is a new outbound data channel, but the existing consumerConfig already controls gateway endpoint topology and configuration identity. No less-trusted writer or unauthenticated request-to-exporter-configuration path was established, so endpoint flexibility is not classified as a verified privilege escalation or credential-exfiltration finding.

Trust Boundaries and Controls

• observed — Collector credentials remain deployment-owned: documentation requires Secret-backed OTEL_EXPORTER_OTLP_HEADERS, and Helm forwards the container environment list rather than placing header values in the telemetry ConfigMap. Endpoint validation rejects embedded credential forms but does not bind an endpoint to the Secret’s intended collector. Whether deployment-held headers accompany a changed destination remains unverified.

Resilience and Maintainability Implications

• observed — The existing consumerConfig lifecycle retains prior configuration on failed or skipped reconciliation. Disabling configuration generation records Disabled status without clearing the old ConfigMap. This predates telemetry and is not itself a newly introduced credential-authority defect. Successful enabled reconciliation without telemetry removes the rendered telemetry additions, but activation remains deployment-owned; controller status does not establish that a running exporter stopped.

Hardening Proposals

• proposed — Define an exporter-withdrawal transition independent of routing candidate availability, with explicit deployment-owner activation and confirmation of the effective exporter state. Preserve valid routing state while ensuring a restart cannot silently reuse a withdrawn collector destination.
• proposed — Where gateway configuration authors are less trusted than deployment owners, bind exporter destinations to approved collectors and appropriate TLS and egress controls. Verify destination changes, Secret-header handling, and exported payload privacy against the exact gateway image before relying on those boundaries.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: adding opt-in OpenTelemetry export configuration.
Docstring Coverage	✅ Passed	Docstring coverage is 90.48% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 21 functions across 6 files. (13 skipped: 1…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}