Make LLM tracing pluggable (currently PostHog-only)

[hev]

Problem

LLM tracing is hardcoded to PostHog. packages/ui/src/observability.ts speaks
PostHog's capture API directly (POST /i/v0/e/ with $ai_generation /
$ai_span / $ai_trace events), and telemetryFromEnv only knows
POSTHOG_KEY / POSTHOG_API_KEY / POSTHOG_HOST / POSTHOG_CAPTURE_CONTENT.
Consumers who run their observability on Datadog, LangSmith, Braintrust,
Langfuse, Honeycomb, etc. get nothing from the agentic path.

We want tracing to be pluggable the same way inference already is. This is a
brainstorm to align on the approach before writing code — there are a lot of
backends and they don't all want the same wire format, so the design choice
matters more than the adapter count.

What's already in our favor

The abstraction seam is already in the right place. The loop and endpoint only
ever touch the Telemetry interface:

interface Telemetry {
  readonly traceId: string;
  generation(event: GenerationEvent): void;
  span(event: SpanEvent): void;
  trace(event: TraceEvent): void;
}

Nothing PostHog-specific leaks past that boundary. The coupling is entirely in
(a) the implementation inside makeTelemetry and (b) the env reading in
telemetryFromEnv. So this is a refactor-behind-a-stable-interface job, not a
rewrite of the call sites in search/loop.ts.

We also already have the exact pattern to copy: the inference provider
registry in packages/ui/src/providers.ts (PROVIDERS record, envKey per
provider, resolveProviderName validator, default-to-anthropic). A tracing
registry should look just like it.

Constraints that shape the answer

These are non-negotiable and they rule out the obvious "just import each
vendor's SDK" approach:

• Edge-runtime friendly, zero runtime deps. observability.ts deliberately
  speaks the capture API over fetch rather than pulling in posthog-node.
  Every adapter has to be the same: fetch-only, no vendor Node SDK
  (@datadog/..., langsmith, braintrust are all heavy/Node-only and would
  break the Cloudflare endpoint).
• Everything degrades, nothing hard-fails. No key configured → no-op sink,
  answer path unaffected. Must stay true per-backend.
• Cloudflare waitUntil. In-flight captures must survive SSE stream close;
  the plumbing for this already exists in resolveTelemetry.
• Docs-first. The ## LLM tracing section in api/endpoint.mdx (and likely
  api/configuration.mdx) ships in the same change, and the digest gets rebuilt.

The key design decision: native-per-vendor vs OTel-as-bridge

This is the crux. Two ways to get breadth:

Option A — a native adapter per vendor. Mirror providers.ts: one module
per backend, each emitting that vendor's own ingest format over fetch
(PostHog $ai_*, Datadog LLM Obs intake, LangSmith /runs/batch, Braintrust
log insert). Highest fidelity in each vendor's native UI, but it's N integrations
to build and maintain, and every new backend is new code.

Option B — one OpenTelemetry (OTLP/HTTP + GenAI semantic conventions)
adapter. Emit GenAI-convention spans (gen_ai.system,
gen_ai.request.model, gen_ai.usage.input_tokens, etc.) over OTLP/HTTP JSON.
The leverage here is large: Datadog, LangSmith, Braintrust, and Langfuse all
now accept OTLP GenAI spans, as do Honeycomb, Grafana, New Relic, Arize
Phoenix, Jaeger. One fetch-based emitter unlocks essentially the whole
ecosystem in a single piece of code, and it stays zero-dep (OTLP/HTTP is just
JSON over POST — no @opentelemetry/* packages needed).

Recommendation: B as the universal bridge, keep PostHog native, add native
adapters only where OTLP loses fidelity. PostHog's $ai_* format isn't OTel,
and we already have it working, so it stays as its own adapter. OTLP covers the
long tail (Datadog / LangSmith / Braintrust / everyone else) for the cost of one
adapter. We only write a native Datadog/LangSmith/Braintrust adapter later if
users report the OTLP mapping is too lossy for that vendor's UI.

OTLP mapping caveats to work through

• Our model is flat events (generation/span/trace); OTLP wants a span tree
  with start/end nanosecond timestamps and proper parent/child links. We emit
  after-the-fact with a latencyMs, so we'd synthesize endTime = now,
  startTime = now − latency.
• IDs: we use UUIDs (PostHog-friendly). OTel needs 16-byte hex trace IDs and
  8-byte hex span IDs. The adapter has to generate/convert these.
• captureContent (off | redacted | full) maps onto whether we attach
  gen_ai.prompt / gen_ai.completion attributes. Keep the existing redaction
  logic.

Proposed plan (phased)

Phase 1 — generalize the seam (no behavior change).

• Add a tracing-backend registry (tracing/ dir or tracing.ts) modeled on
  providers.ts: a record keyed by backend name, each with its env-key(s) and a
  make(opts): Telemetry factory.
• Extract today's PostHog code into tracing/posthog.ts unchanged.
• Generalize telemetryFromEnv → selects backend by config option and/or by
  which env var is present; defaults preserve current PostHog behavior exactly.
• Add a tracing?: { backend, host, captureContent, ... } option to
  types.ts (mirrors provider), env still wins for secrets.
• Existing PostHog users see zero change. Tests in
  test/observability.test.ts stay green; add registry-selection tests.

Phase 2 — OTLP/GenAI adapter (the breadth win).

• tracing/otlp.ts: GenAI semantic-convention spans over OTLP/HTTP JSON.
  Config via OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_HEADERS (the
  standard OTel env vars). This is what gives us Datadog + LangSmith +
  Braintrust + the rest.
• Doc a "point your collector / vendor OTLP endpoint here" recipe per major
  backend.

Phase 3 — native adapters, only where they earn it.

• Revisit once OTLP is in users' hands. Likely candidates if OTLP is too lossy:
  native Datadog LLM Obs, LangSmith /runs, Braintrust logs. Each is a new
  registry entry, no core changes.

Open questions for discussion

1. Multiple simultaneous sinks? The interface makes a composite/fan-out sink
   trivial (ship to PostHog and Datadog at once). Worth supporting in Phase 1
   so backend selection is a list, not a scalar? Leaning yes — cheap now,
   awkward to retrofit.
2. Backend selection: explicit config (tracing.backend: 'otel') vs
   auto-detect from whichever env var is present vs both. providers.ts is
   explicit-with-default; do we match that, or auto-detect for tracing since
   it's optional/secondary?
3. Env naming: keep POSTHOG_KEY for back-compat (must), add a
   backend-neutral selector (HEV_ASK_TRACING=otel|posthog|...?) plus
   per-backend keys. Or lean entirely on standard OTel env vars for the OTLP
   path?
4. Is OTLP genuinely enough for Datadog/LangSmith/Braintrust today, or does
   any of them need its native format to light up the good parts of its UI?
   This decides whether Phase 3 is real or hypothetical — worth a short spike.
5. RFC? This touches the public option surface and the cross-host
   observability story, which is RFC-shaped per our process (docs/rfcs/, next
   number 0007). Propose drafting 0007-pluggable-llm-tracing.md to lock
   the interface + naming before code.

Pointers

• packages/ui/src/observability.ts — the Telemetry interface + PostHog impl
• packages/ui/src/search/loop.ts — the only telemetry.generation() call sites
• packages/ui/src/endpoint.ts (resolveTelemetry) — wiring + waitUntil
• packages/ui/src/providers.ts — the registry pattern to mirror
• packages/ui/src/types.ts — where the tracing option would live
• site/src/content/docs/api/endpoint.mdx (## LLM tracing) — docs to update