feat(mcp): add native OpenTelemetry observability

[fahreddinozcan]

Summary

• instrument individual dispatched MCP requests and notifications at the public SDK transport boundary, including operations inside valid JSON-RPC batches
• emit the development-status OpenTelemetry MCP server metrics mcp.server.operation.duration and mcp.server.session.duration, plus SERVER spans with standard MCP/GenAI/JSON-RPC/network attributes
• extract traceparent, tracestate, and baggage from params._meta per SEP-414, using the extracted context as parent and linking ambient HTTP context
• add bounded Context7 operation, tool-outcome, upstream, authentication, in-flight, and official Node runtime signals
• expose Prometheus/OpenMetrics on a dedicated HTTP-only :9464/metrics listener while supporting externally preloaded OpenTelemetry providers
• bind the standalone CLI metrics listener to loopback by default, with an explicit 0.0.0.0 override in the production Docker image
• provide OTEL_SDK_DISABLED=true as a true hard-off boundary: no telemetry provider modules, transport wrappers, async scopes, spans, metrics, or runtime collectors are loaded
• document and verify the Envoy/application signal ownership boundary so HTTP and backend-proxy metrics are not duplicated

Why this integration

The MCP TypeScript SDK v2.0.0 does not ship a turnkey OpenTelemetry plugin, tracer/meter, exporter, or completed-operation middleware. It does expose SEP-414 propagation constants and the public transport boundary. Instrumenting that boundary observes protocol operations across HTTP and stdio without subclassing protected SDK internals or mistaking an HTTP envelope for a single MCP operation.

The cluster already has gateway HTTP and Kubernetes/container signals, but those cannot identify MCP methods, individual tool health, authentication outcomes, Context7 API dependency failures, or Node event-loop/heap pressure. This adds those missing bounded semantic signals without duplicating gateway request telemetry.

Envoy non-duplication audit

The checked-in production data path is Envoy Gateway -> HTTPRoute/mcp -> Service/mcp-svc -> MCP pods. Envoy is not a sidecar in the MCP pod.

• keep downstream HTTP envelope totals, status classes, latency, active requests/connections, resets, and gateway timeouts in envoy_http_*_downstream_*
• keep Envoy-to-MCP request/status/latency, pending/active work, retries, resets, connection failures, timeouts, and circuit-breaker overflow in envoy_cluster_upstream_*
• keep pod/container CPU, memory, network, and restarts in the Kubernetes monitoring stack
• the application deliberately registers no generic inbound HTTP server/request instruments
• mcp_server_operation_duration_count is a JSON-RPC operation count after SDK dispatch, not an HTTP request count: one valid batch is one Envoy request and multiple MCP operations; pre-dispatch HTTP rejections are Envoy-only
• context7_mcp_upstream_* observes MCP-to-Context7 API calls, which the ingress gateway cannot see

Signals

• mcp_server_operation_duration — standard MCP server operation histogram; _count is the operation count. Tool calls use standard gen_ai.tool.name plus bounded context7.mcp.tool.outcome (success, not_found, or error), avoiding a duplicate tool counter/histogram.
• mcp_server_session_duration — standard MCP session histogram for real stateful stdio sessions; stateless HTTP request transports are excluded
• standard MCP SERVER spans with SEP-414 parent extraction and ambient transport links
• context7_mcp_operations_active
• context7_mcp_upstream_requests_total, context7_mcp_upstream_request_duration, and context7_mcp_upstream_requests_active
• context7_mcp_authentication_attempts_total, context7_mcp_authentication_duration, and context7_mcp_authentication_active
• official OpenTelemetry Node runtime metrics: nodejs_eventloop_*, v8js_gc_duration, v8js_memory_heap_*, and v8js_resource_active
• standard OpenTelemetry target_info resource metadata

Upstream outcomes distinguish HTTP, response-decoding, network, timeout, and cancellation failures; they include both bounded status class and exact numeric HTTP status. Authentication distinguishes accepted, missing, invalid, and unexpected-error outcomes.

Labels exclude API keys, IPs, client versions, queries, library IDs, session IDs, tool arguments/results, and raw error text. Server-side JSON-RPC caller faults remain visible through rpc.response.status_code without setting error.type; internal, transport, and tool failures set bounded error types.

Operations

• HTTP: embedded Prometheus reader defaults to 127.0.0.1:9464/metrics; the production Docker image explicitly sets OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0 for internal cluster scraping
• stdio: no telemetry listener; the single outer transport owns session lifecycle/version/error capture across SDK modern-probe -> legacy fallback
• graceful EOF/close/SIGHUP awaits SDK close and best-effort flushes an external provider before exit
• provider flush is bounded at 4 seconds and process shutdown independently bounds it at 5 seconds
• OAuth authorization-server metadata fetches are bounded at 10 seconds and return 502 on timeout instead of hanging indefinitely
• OTEL_EXPORTER_PROMETHEUS_HOST and OTEL_EXPORTER_PROMETHEUS_PORT configure the listener
• OTEL_METRICS_EXPORTER=none disables only the embedded exporter, allowing a preloaded provider to receive signals
• OTEL_SDK_DISABLED=true entirely bypasses instrumentation and provider bootstrap
• bind/config/runtime-collector failures are logged and fail open
• an already registered global MeterProvider/TracerProvider takes precedence; external instrumentation owns Node runtime registration in that mode

The deployment repository still needs to expose port 9464 internally and add the Prometheus scrape/ServiceMonitor; it should not route this port through public MCP ingress.

Resource-overhead benchmark

Local Docker benchmark against the telemetry branch's then-current merge-base, using an in-memory/stubbed upstream to make telemetry cost maximally visible: 8 concurrent MCP query-docs clients, 10-second warm-up, 30-second measurement, three repetitions, rotated mode order, and 15-second Prometheus scraping when enabled. Every measured call succeeded. The later master sync and secure bind-default change do not alter the enabled instrumentation hot path.

Mode vs baseline	CPU time / operation (paired median)	Idle container memory	Mean memory under load	Throughput
Current image, OTEL_SDK_DISABLED=true	-0.17% (noise; paired mean +0.78%)	+1.74 MiB	-1.18 MiB (no measurable increase)	+1.04% (noise)
Current image, telemetry enabled + scrape	+12.92%	+7.28 MiB	+22.70 MiB	-0.11% (noise)

Median absolute CPU cost was 0.546 ms/op for baseline, 0.549 ms/op disabled, and 0.639 ms/op enabled. Median throughput was 1815, 1801, and 1803 operations/second respectively. The enabled result is intentionally a worst-case CPU-saturated local test with essentially no upstream latency; production's network-bound requests should have a lower relative percentage, while the absolute memory and CPU measurements remain the useful capacity-planning bounds.

The benchmarked production image grew from 115,663,904 to 118,468,819 bytes: +2,804,915 bytes (+2.43%).

Validation

• pnpm typecheck — passed
• pnpm lint:check — passed
• pnpm format:check — passed
• pnpm build — passed
• pnpm test — 10 files / 97 tests passed after merging current master
• frozen-lockfile production Docker build passed; image metadata contains OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0
• focused coverage includes secure default metrics binding, disabled-path bypass, SEP-414 propagation, server/caller error semantics, cancellation/abort/close lifecycle, one stateful session across SDK fallback, modern/legacy protocol capture, recoverable vs terminal transport errors, bounded stdio close/flush, nested Undici timeout causes, body-phase timeout/cancellation, exact upstream status, not_found, authentication timing/concurrency, runtime metrics, batch counts, active-gauge cleanup, and exporter collision
• local Envoy v1.38.0, Prometheus v3.5.0, and Grafana 12.1.0 stack is healthy
• live scrape validation confirms canonical tools/call operation series exist and removed context7_mcp_tool_calls_* duplicate series do not
• final thermo-nuclear maintainability review approved with no remaining high-confidence findings
• no production configuration, workloads, credentials, or telemetry data were mutated

References

• OpenTelemetry MCP semantic conventions
• MCP SEP-414 trace context propagation
• Envoy HTTP connection manager statistics
• Envoy upstream cluster statistics

[linear-code]

CTX7-2004

[Copilot]

Pull request overview

This PR adds first-class OpenTelemetry observability to the @upstash/context7-mcp server by instrumenting MCP operations at the SDK transport boundary (including batched JSON-RPC messages), exporting MCP server spans/metrics, and exposing Prometheus/OpenMetrics on a dedicated internal listener.

Changes:

• Add MCP operation/session instrumentation (SERVER spans + mcp.server.operation.duration / mcp.server.session.duration) with SEP-414 context extraction from params._meta.
• Add bounded application metrics for upstream Context7 API calls, tool outcomes, authentication outcomes, and Node runtime signals, plus an embedded Prometheus exporter on :9464/metrics.
• Add shutdown handling for stdio to close/flush reliably, and add tests covering lifecycle, disabled-path behavior, batching, and exporter behavior.

Reviewed changes

Copilot reviewed 19 out of 20 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pnpm-lock.yaml	Locks new OpenTelemetry dependencies used by the MCP package.
packages/mcp/package.json	Adds OpenTelemetry API/SDK/exporter and runtime instrumentation dependencies.
packages/mcp/Dockerfile	Exposes the dedicated Prometheus metrics port (9464) in the container image.
packages/mcp/src/index.ts	Wires telemetry into HTTP + stdio serving paths, adds auth/upstream observation, and conditional embedded exporter bootstrapping.
packages/mcp/src/lib/api.ts	Wraps Context7 upstream calls with observeUpstreamRequest and adds tool outcome classification for context fetches.
packages/mcp/src/lib/types.ts	Extends ContextResponse with a bounded outcome for tool telemetry.
packages/mcp/src/lib/tool-names.ts	Centralizes tool names and defines bounded ToolCallOutcome.
packages/mcp/src/lib/telemetry.ts	Implements bounded upstream/auth metrics, tool outcome tagging, and upstream error classification.
packages/mcp/src/lib/telemetry-config.ts	Adds env-based switches for hard-off telemetry and embedded Prometheus enablement.
packages/mcp/src/lib/telemetry-provider.ts	Implements embedded Prometheus MetricReader + Node runtime metrics bootstrap with “fail open” behavior.
packages/mcp/src/lib/mcp-telemetry.ts	Adds transport-boundary MCP operation/session instrumentation and span/metric classification.
packages/mcp/src/lib/mcp-operation-scope.ts	Adds AsyncLocalStorage-based per-operation scope for tool outcome/error marking.
packages/mcp/src/lib/stdio-shutdown.ts	Adds idempotent stdio shutdown with bounded flush behavior.
packages/mcp/README.md	Documents signals, env configuration, scrape endpoint, and gateway non-duplication guidance.
packages/mcp/test/telemetry.test.ts	Unit tests for method/tool normalization, SEP-414 extraction, config flags, and span parenting/linking.
packages/mcp/test/telemetry-disabled.test.ts	Verifies OTEL_SDK_DISABLED=true yields no exported application metrics.
packages/mcp/test/stdio-shutdown.test.ts	Tests idempotent shutdown, error handling, and flush timeout behavior.
packages/mcp/test/mcp-telemetry-lifecycle.test.ts	Exercises MCP lifecycle classification, cancellations, transport errors, and session metric behavior.
packages/mcp/test/integration.test.ts	End-to-end validation of exported Prometheus series (batch counts, bounded labels, runtime metrics, exporter collision).
.changeset/clean-otters-observe.md	Publishes a minor release note for the new observability features.

Files not reviewed (1)

• pnpm-lock.yaml: Generated file

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[enesgules]

Reviewed the full diff. The code is correct and well tested — no blocking bug found. One security-relevant default, one bundled behavior change, and a few non-blocking notes.

1. Metrics listener binds to 0.0.0.0 by default (should be localhost)

telemetry-provider.ts sets DEFAULT_PROMETHEUS_HOST = "0.0.0.0". The OpenTelemetry spec defines the default for OTEL_EXPORTER_PROMETHEUS_HOST as localhost. This package is also a CLI that users run with npx ... --transport http; they would now get an open port on all interfaces with no warning.

Suggestion: default to 127.0.0.1 and add ENV OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0 to the Dockerfile for the container path. This keeps the spec default and the k8s deployment working.

2. Bundled behavior change: OAuth metadata fetch now has a timeout

index.ts adds OAUTH_METADATA_TIMEOUT_MS = 10_000 to the /.well-known/oauth-authorization-server fetch, which previously had no timeout. The change is good, but it's a behavior change (hung auth server → 502 after 10 s instead of hanging) and isn't called out in the PR description. Worth a mention there.

3. Minor: cached rejected import in api.ts

telemetryModule ??= import("./lib/telemetry.js") caches a rejected promise forever if the import fails once, after which every API call fails. In practice main() imports the same module eagerly first so the process would crash earlier — low risk, but a .catch that clears the cache would remove it.

4. Minor: small indirection that could fold away

• tool-names.ts is a 7-line file for two string constants; they could live in mcp-telemetry.ts.
• index.ts and api.ts each define their own observeUpstreamRequest wrapper with a different lazy-load strategy; one shared guard would do.
• @opentelemetry/instrumentation-runtime-node pulls in require-in-the-middle/import-in-the-middle module-hooking machinery for runtime collectors only — the heaviest of the new deps for the least signal. Acceptable per the benchmark, just noting it.

What's done well

• Double-finish/double-decrement paths in InstrumentedTransport are all guarded.
• Cardinality is bounded everywhere; no user data in labels.
• Fail-open is consistent: exporter bind failure, runtime-collector failure, and flush timeout never block MCP serving.
• The test suite is unusually thorough (hard-off module-load verification via a loader hook, batch counting, abort/cancel/close lifecycle, port-collision fail-open).

Recommendation: fix the bind default (1) before merge; the rest are non-blocking.

[fahreddinozcan]

Addressed the latest review in ef69dea:

• Changed the standalone/CLI Prometheus default to 127.0.0.1:9464.
• Added OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0 explicitly to the production Docker image so cluster scraping continues to work.
• Added a real HTTP integration test for the loopback default and verified the built image metadata contains the container override.
• Documented the 10-second OAuth authorization-server metadata timeout in both the README and PR description.
• Reset the cached lazy telemetry import promise after rejection so a transient failure is retryable.

The remaining maintainability notes are intentional:

• tool-names.ts is the canonical source for the runtime tuple, ToolCallOutcome type, and both tool registrations; keeping it separate prevents positional/repeated string taxonomies.
• index.ts performs enabled-at-bootstrap loading, while api.ts must preserve a direct hard-off path for standalone module consumers. Combining them would require mutable cross-module registration or would reintroduce eager OpenTelemetry loading.
• Runtime instrumentation is retained because event-loop, heap, GC, and active-resource saturation are application-owned signals unavailable from Envoy; its measured overhead is documented in the PR.

Validation: 10 test files / 97 tests passed, plus typecheck, ESLint, Prettier, build, frozen-lockfile production Docker build, and git diff --check.

[enesgules]

Re-reviewed the update. All three actionable findings from my previous review are addressed: the metrics listener now defaults to 127.0.0.1 with the Docker image explicitly opting into 0.0.0.0 (plus a loopback-bind integration test), the OAuth metadata timeout is documented, and the lazy telemetry import clears its cache on rejection so a failed import can no longer poison later API calls. The remaining notes were non-blocking. LGTM.

[fahreddinozcan]

Final local production-readiness audit completed on head 378ae6f after merging current master (21c3dd4). The Claude Code plugin-auth conflict was resolved by preserving the new auth policy and routing it through the bounded authentication telemetry. CI is green and GitHub reports the PR clean/mergeable.

Review fixes added:

• centralized hard-off/lazy telemetry ownership in a small runtime facade;
• flush both external metric and trace providers with proxy unwrapping and allSettled semantics;
• instrument MCP v2 subscriptions/listen, which the pinned SDK handles before server transport dispatch;
• separate short operation latency (receipt → ACK) from active subscription lifetime/outcome;
• consolidate stdio subscription state into one typed per-ID record and cover cancel/rejection ID reuse, deferred ACK cancellation, and overlapping deferred ACK/terminal/close races;
• use one idempotent, bounded shutdown coordinator for HTTP and stdio on SIGTERM, SIGINT, SIGHUP, and stdio EOF.

Validation:

• package tests: 11 files / 112 tests passed;
• typecheck, ESLint, Prettier, build, git diff --check, and frozen-lockfile install passed;
• production Docker build passed;
• self-contained MCPB bundle build passed and its HTTP/metrics startup + SIGTERM exit were smoke-tested;
• final Docker image passed enabled, hard-off, and invalid-metrics-port fail-open probes; invalid-port mode also completed 100 MCP requests;
• SIGTERM exited code 0, including under continuous load and with an open MCP v2 subscription; no OOMs or restarts.

Local Prometheus/Grafana was fed only synthetic traffic backed by a local API fixture. Verified operation counts/latency, bounded tool outcomes, upstream success/HTTP-error classes, authentication missing/accepted, active operations/dependencies, active v2 subscriptions, subscription duration/outcome, and Node event-loop/V8 metrics. Active gauges returned to zero. No generic inbound HTTP metrics were emitted, avoiding duplication with Envoy. No query, library ID, API key, client IP, authorization value, or session ID appeared in metric labels.

Final 1 CPU / 256 MiB resource comparison, 3 interleaved fresh-container trials per mode, 2k warmup + 10k modern-v2 tools/list operations:

• hard-off median: 1.0417 ms CPU/op, 957.65 ops/s, 40.23 MiB idle, 82.95 MiB cooled, 101.42 MiB peak;
• enabled median: 1.1711 ms CPU/op, 856.30 ops/s, 44.44 MiB idle, 90.56 MiB cooled, 114.12 MiB peak;
• delta: +12.42% CPU/op, -10.58% throughput, +4.20 MiB idle, +7.61 MiB cooled, +12.70 MiB peak.

This is the worst relative case (tiny CPU-only tools/list operations); real upstream-I/O tool calls should have a smaller proportional cost. Recommended rollout headroom: about 15% CPU and at least 16 MiB memory. Final verdict: ready for production with that headroom and normal canary monitoring.

No production or Kubernetes resources/data were accessed or mutated during this audit.

[fahreddinozcan]

Exact-head follow-up after the final plugin-auth merge/rebuild (378ae6f, image sha256:93036cb…): one fresh paired 1 CPU / 256 MiB run (1k warmup + 5k modern-v2 tools/list operations) remained inside the three-trial envelope above.

• hard-off: 1.1029 ms CPU/op, 911.19 ops/s, 39.77 MiB idle, 82.79 MiB cooled, 102.63 MiB peak;
• enabled: 1.1984 ms CPU/op, 835.88 ops/s, 44.58 MiB idle, 96.15 MiB cooled, 105.22 MiB peak;
• delta: +8.66% CPU/op, -8.27% throughput, +4.81 MiB idle, +13.36 MiB cooled (GC-sensitive), +2.60 MiB peak.

Both completed 5,000/5,000 requests with no OOM/restart and exited code 0 on SIGTERM.

[fahreddinozcan]

Finalized on head 2d03143 after a read-only production-cluster wiring check. No Kubernetes or production resources were mutated.

The application endpoint remains 0.0.0.0:9464/metrics in the production image. The current cluster uses annotation-based kubernetes-pods discovery, has 10 stateless MCP replicas, and currently has zero active MCP scrape targets because the workload pod template has neither the Prometheus annotations nor a declared containerPort 9464. Both existing MCP Services expose only application port 3000. The README now documents the required per-pod annotations/port and explicitly avoids a single load-balanced Service target, which would mix replicas and yield incomplete per-process series. With that workload metadata, Prometheus will scrape every pod at http://:9464/metrics on the existing global 10-second interval.

The existing Envoy scrape remains separate and healthy: five data-plane /stats/prometheus targets and one control-plane /metrics target are up. This PR intentionally does not duplicate Envoy-owned HTTP/proxy metrics.

Final verification: 11 test files / 112 tests passed, Prettier and git diff checks passed, CI is green, and GitHub reports APPROVED, CLEAN, and MERGEABLE. The deployment metadata must be added through the infrastructure rollout before expecting MCP metrics in production.