evals(v4): OTEL provider + LangSmith gating (unwired)

[miguelg719]

Why

Stack 2/6. Adds the tracing transport surface; no call sites yet.

What

• framework/langsmith.ts: transport gating (EVAL_TRACE_TRANSPORT = native (default) | otel).
• framework/otel.ts: NodeTracerProvider (Braintrust processor + LangSmith exporter) with global provider.register() so Stagehand 4.0 native RPC spans are captured; withHarnessAgentSpan helper. Only builds/registers under otel — native path untouched.
• .env.example: LangSmith + EVAL_TRACE_TRANSPORT vars.

Testing

typecheck + unit + build green.

Base: #2757

---

Summary by cubic

Adds an OpenTelemetry tracing transport for evals, enabling export to Braintrust and LangSmith. Old behavior: native-only tracing. New behavior: when EVAL_TRACE_TRANSPORT="otel", a global NodeTracerProvider is registered and spans export to configured sinks; defaults remain and no call sites yet.

• Transport gating: resolves "native" vs "otel"; warns once on unknown values. LangSmith export requires LANGSMITH_TRACING="true" and LANGSMITH_API_KEY or LANGCHAIN_API_KEY.
• Provider: builds NodeTracerProvider (@opentelemetry/sdk-trace-node) with Braintrust BraintrustSpanProcessor (@braintrust/otel) and optional LangSmith OTLP exporter (langsmith/experimental/otel/exporter); registers globally to capture Stagehand RPC spans; caps JSON span attributes at 200kB; adds shutdownTracing (10s timeout) and resetTracingStateForTests.
• Helper: withHarnessAgentSpan wraps external harness runs in a span; no-op unless OTEL is active.
• To enable OTEL: set EVAL_TRACE_TRANSPORT="otel". For Braintrust, set BRAINTRUST_API_KEY (optional BRAINTRUST_OTEL_PARENT/BRAINTRUST_OTEL_URL). For LangSmith, set LANGSMITH_TRACING="true" and provide an API key. .env.example documents these.

^{Written for commit 5f39300. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 5f39300

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

1 issue found across 3 files

Confidence score: 3/5

• In packages/evals/framework/langsmith.ts, assertLangSmithReady() currently throws generic Errors and includes required environment-variable names, which can leak configuration details in logs and makes callers harder to handle reliably with typed error flows; switch to a dedicated typed configuration error and sanitize outward-facing messages while keeping detailed diagnostics internal.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/framework/langsmith.ts">

<violation number="1" location="packages/evals/framework/langsmith.ts:24">
P2: Custom agent: **Exception and error message sanitization**

`assertLangSmithReady()` raises generic `Error` instances and exposes the required environment-variable names. Throw a dedicated typed configuration error with a sanitized message instead.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant App as Evals App
    participant Gating as langsmith.ts (Transport Gating)
    participant OTEL as otel.ts (Tracer Provider)
    participant NodeTracer as NodeTracerProvider
    participant Braintrust as Braintrust Processor
    participant LangSmith as LangSmith Exporter
    participant Harness as External Harness (claude_code/codex)
    participant Stagehand as Stagehand 4.0 RPC

    Note over App,Stagehand: OTEL Tracing Transport Setup

    App->>Gating: resolveTraceTransport()
    alt EVAL_TRACE_TRANSPORT=otel
        Gating-->>App: "otel"
        App->>OTEL: buildTracerProvider()
        
        opt BRAINTRUST_API_KEY set
            OTEL->>Braintrust: Configure BraintrustSpanProcessor
            Note over Braintrust: project = stagehand (CI) or stagehand-dev
        end
        
        opt LANGSMITH_TRACING="true" AND API key set
            OTEL->>LangSmith: Configure LangSmithOTLPTraceExporter
        end
        
        OTEL->>NodeTracer: provider.register() (GLOBAL)
        Note over NodeTracer: Captures Stagehand 4.0 RPC spans
        NodeTracer-->>Stagehand: Global tracer registration
        
        alt No LangSmith API key
            OTEL-->>App: THROW: Missing API key error
        else LANGSMITH_TRACING != "true"
            OTEL-->>App: THROW: Tracing not enabled error
        end
    else default (native)
        Gating-->>App: "native"
        Note over App: No OTEL setup, native path unchanged
    end

    Note over App,Harness: Runtime Spans

    alt OTEL transport active
        App->>OTEL: withHarnessAgentSpan({harness, model, task})
        OTEL->>OTEL: create span agent.{harness} with 200kB JSON caps
        OTEL->>Harness: execute agent run
        alt Success
            Harness-->>OTEL: result with _success
            OTEL->>OTEL: set output.value + output.mime_type
            OTEL-->>App: return result
        else Error
            Harness-->>OTEL: throws
            OTEL->>OTEL: recordException + SpanStatusCode.ERROR
            OTEL-->>App: re-throw
        end
    else native transport
        App->>Harness: execute directly (no OTEL span)
    end

    Note over App,Stagehand: Shutdown

    App->>OTEL: shutdownTracing()
    OTEL->>NodeTracer: forceFlush() + shutdown()
    alt flush/shutdown success
        NodeTracer-->>OTEL: flushed
    else timeout (10s)
        OTEL->>OTEL: reject, swallow error (never masks eval result)
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>