Skip to content

metrics, otel, logger and celeris.Adapt hand receive-buffer views to Prometheus, the OTel SDK, slog and net/http, which keep them: telemetry can carry another connection's request bytes (Authorization seen in a /metrics label) #732

Description

@FumingPower3925

Summary

On epoll and io_uring, the metrics, otel and logger middleware and celeris.Adapt pass request strings to APIs that are allowed to keep them after the request: Prometheus series, the OpenTelemetry SDK, slog handlers and net/http handlers. Those strings are views of the connection's receive buffer. A kept view reads whatever the engine later receives into that buffer:

  • Same connection. A kept string reads the connection's next request. Measured: 1/1 on each native engine, for each of the four sites.
  • Another connection. When a connection closes, its buffer is pooled and serves the next accepted connection. Measured: a /metrics series label read "ion: SECRET", bytes of another connection's Authorization: SECRETSECRET... header. That was 1 series in 20 rounds on each native engine, and 0 on std.

Telemetry and logs can therefore carry another client's request bytes, including credentials, to anyone who can read them. It was found by the audit that #714 called for (the #485 / #714 class).

Sites (9f4d89b)

  • metrics. lv = append(lv, c.Method(), path, statusStr) (middleware/metrics/metrics.go:196) and lv = append(lv, customLabelFuncs[i](c)) (:198) feed WithLabelValues (:201). client_golang keeps the label values of every new series for the registry's lifetime and does not copy them. The views are: c.Method() for a method the H1 parser does not intern, c.Path() when there is no route pattern (:185), and any LabelFuncs value read from the request.
  • otel. Span attributes url.path (otel.go:204), server.address (:212), user_agent.original (:219), client.address (:209), request.id (:236) and http.request.method_original (:195). Metric attribute server.address (:250), in an attribute.Set (:260) that the metric SDK keeps as an aggregation key for the provider's lifetime. Span processors keep ended spans until export.
  • logger. slog.String("path", ...), "host", "user_agent", "query" and the rest (middleware/logger/logger.go:117-222). slog allows a Handler to keep a Record after Handle if it calls Record.Clone, which shares the strings. Asynchronous and batching handlers do this.
  • Adapt. buildHTTPRequest (bridge.go:55-89) builds the request from c.method, c.path (the URL when there is no query) and every header value, then r.Host. net/http allows a handler to keep the request's strings after ServeHTTP returns. middleware/adapters.buildRequest (adapters.go:106-131) has the same shape. Separately, Adapt drops the headers entirely on the native engines; that is filed publicly as a functional defect.

Reproduction

Evidence tests in the #714 evidence directory run on a snapshot of 9f4d89b. Each sends two requests on one keep-alive connection, with the same layout and different values, then reads what the API kept from request 1:

test std epoll io_uring
metrics TestC714EvidenceLabelValueViews: tenant labels after 2 requests [tenant-aaaa tenant-bbbb] [tenant-bbbb] + Gather: 3 errors same as epoll
metrics TestC714EvidenceLabelCrossConnection: series with another connection's Authorization bytes 0 1 ("ion: SECRET") 1 ("ion: SECRET")
otel TestC714EvidenceAttributeViews: first span url.path / duration series server.address /o/aaaa / [aaaa, bbbb] /o/bbbb / [bbbb, bbbb] same as epoll
logger TestC714EvidenceLoggerAttrViews: first kept record path /l/aaaa /l/bbbb /l/bbbb
Adapt TestC714EvidenceAdaptKeepsViews: request 1's kept URL.Path / Host /a/aaaa / h-aaaa.example /a/bbbb / h-bbbb.example same as epoll

Docker linux/arm64 (--cpus 4), kernel 7.0.12-linuxkit, memlock 8 MiB, go test -race.

Fix direction

Clone at each boundary: every label value, attribute string, log attribute and http.Request string that is handed over. Unlike #714, #717 and #718 this is on the per-request path, so each site needs a cost measurement. Ways to keep the cost down:

  • metrics. Clone only when a label combination is new, for example with a small map of label combinations already seen, keyed by the cloned values.
  • otel. Clone the attribute strings; each span keeps them anyway.
  • logger. Clone when the configured handler is not the built-in synchronous one.
  • Adapt. Clone when the request is built.

Found by the #714 retention audit (2026-09-27). It was briefly held for private reporting under SECURITY.md; the maintainer decided on public handling because celeris has no users yet.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/observabilityMetrics, logging, debug endpointsbugSomething isn't workingmiddlewareMiddleware implementationsecuritySecurity hardening

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions