feat(evals): add OTEL tracer-provider module + deps (unwired) - #2352
feat(evals): add OTEL tracer-provider module + deps (unwired)#2352miguelg719 wants to merge 1 commit into
Conversation
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 5 files
Confidence score: 5/5
- Safe to merge after the addressed issues were fixed.
Architecture diagram
sequenceDiagram
participant Caller as Eval Runner (future)
participant Builder as buildTracerProvider()
participant Init as initializeTracerProvider()
participant Env as Environment Variables
participant BTExporter as OTLPTraceExporter (Braintrust)
participant LSExporter as LangSmithOTLPTraceExporter
participant Provider as NodeTracerProvider
Note over Caller,Provider: NEW: OTEL Tracer Provider Initialization
Caller->>Builder: buildTracerProvider()
Builder->>Env: EVAL_TRACE_TRANSPORT
alt Transport != "otel"
Env-->>Builder: "native" or other
Builder-->>Caller: null (no-op)
else Transport == "otel"
Env-->>Builder: "otel"
Builder->>Init: initializeTracerProvider() (memoized)
Init->>Env: BRAINTRUST_API_KEY?
alt Key set
Env-->>Init: key present
Init->>Env: BRAINTRUST_OTEL_URL? else default url
Init->>Env: BRAINTRUST_OTEL_PARENT? else project_name:stagehand-dev (or stagehand in CI)
Init->>BTExporter: new OTLPTraceExporter(url, headers)
BTExporter-->>Init: exporter instance
Init->>Init: create BatchSpanProcessor(exporter)
else Key missing
Env-->>Init: no key
end
Init->>Env: LANGSMITH_API_KEY && LANGSMITH_TRACING=="true"?
alt Both set
Env-->>Init: keys set
Init->>LSExporter: new LangSmithOTLPTraceExporter()
LSExporter-->>Init: exporter instance
Init->>Init: create BatchSpanProcessor(exporter)
else Missing
Env-->>Init: not set
end
alt spanProcessors.length > 0
Init->>Provider: new NodeTracerProvider({ spanProcessors })
Provider->>Provider: register() (global)
Provider-->>Init: provider instance
Init-->>Builder: provider
else No processors
Init-->>Builder: null
end
Builder-->>Caller: provider or null
end
Note over Caller,Provider: Shutdown flow
Caller->>Builder: shutdownTracing()
Builder->>Env: EVAL_TRACE_TRANSPORT == "otel"?
alt Not otel or provider null
Builder-->>Caller: return (no-op)
else Valid provider
Builder->>Provider: forceFlush()
Provider-->>Builder: ok
Builder->>Provider: shutdown()
Provider-->>Builder: ok
Builder->>Builder: reset provider to null
Builder-->>Caller: done
end
Note over Caller,Provider: Tracer retrieval
Caller->>Builder: getTracer()
alt provider exists
Provider-->>Builder: tracer from provider
else No provider
Builder-->>Caller: ProxyTracerProvider noop tracer
end
Builder-->>Caller: Tracer
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
840c0e5 to
3055aa4
Compare
There was a problem hiding this comment.
All reported issues were addressed across 9 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
d04c712 to
b4a8994
Compare
Wire the OTEL provider's test-reset through the exported resetTracingStateForTests() helper instead of hand-deleting the globalThis slot (addresses review: dead export).
b4a8994 to
d692cd9
Compare
|
Superseded by the Stagehand 4.0 retarget stack (#2757–#2762). Main moved to the 4.0 monorepo restructure (packages/core → packages/sdk-ts; evals package restructured), so this 3.x-era diff no longer applies. The OTEL/LangSmith tracing work — plus native-span capture via 4.0's global tracer and claude_code/codex agent spans — is reimplemented additively on 4.0 in the new stack. |
1 participant
why
The fan-out core for the opt-in OTEL transport: one
NodeTracerProviderwhose span processors route eval traces to Braintrust and LangSmith, each gated on its own key. Unwired — imported by nothing yet.what changed
framework/otel.ts(new):buildTracerProvider()(memoized; null unlessEVAL_TRACE_TRANSPORT=otel; per-backend gated processors),getTracer(), null-safeshutdownTracing()(forceFlush+shutdown, 10s timeout)@opentelemetry/{api,sdk-trace-base,sdk-trace-node}depstests/framework/otel.test.ts(new): native→null; both keys→2 processors; each alone→1test plan
Stack 2/5 · base: #2351
Summary by cubic
Adds an opt-in OpenTelemetry tracer provider for evals that can fan out spans to Braintrust and/or LangSmith. The module is added but not wired yet; provider state is bundle-safe, and shutdown continues even if
forceFlushfails.New Features
framework/otel.tswithbuildTracerProvider(); returns null unlessEVAL_TRACE_TRANSPORT=oteland uses aglobalThisSymbol-keyed state to avoid duplicate providers in the bundled CLI.BRAINTRUST_API_KEYis set (supportsBRAINTRUST_OTEL_PARENT,BRAINTRUST_OTEL_URL, orbraintrustParentoverride; defaults parent toproject_name:stagehand-devorproject_name:stagehandin CI) and LangSmith whenLANGSMITH_API_KEYis set andLANGSMITH_TRACING=true.getTracer(),shutdownTracing()(10s timeout), andresetTracingStateForTests()for clearing shared state in tests; tests cover native mode, single/dual-backend setups, parent override, bundle-safe state, and shutdown behavior..env.exampledocuments Braintrust OTEL overrides.Dependencies
@opentelemetry/api,@opentelemetry/exporter-trace-otlp-proto,@opentelemetry/sdk-trace-base,@opentelemetry/sdk-trace-node.Written for commit d692cd9. Summary will update on new commits.