feat(evals): wire opt-in OTEL transport (EVAL_TRACE_TRANSPORT=otel; native unchanged)

[miguelg719]

why

Turns the OTEL fan-out on behind an opt-in flag. The native (default) path stays byte-identical — no keys/flags means today's behavior exactly.

what changed

• framework/braintrust.ts: tracedSpan gains an otel branch — opens an OTEL active span, adapts span.log({output,scores,metrics,metadata}) → setAttribute/addEvent; native + NOOP_SPAN branches untouched, signature unchanged (verifier/runner call sites need zero edits)
• lib/AISdkClientWrapped.ts: EVAL_TRACE_TRANSPORT=otel → experimental_telemetry (no wrapAISDK); native path verbatim
• framework/runner.ts: provider init at run start; shutdownTracing() after the Braintrust flush
• cli.ts: best-effort flush on signal teardown

test plan

• Unit suite 362/362; braintrust-optional / braintrust-runner-nolog pass unchanged (native invariant)
• Manual: EVAL_TRACE_TRANSPORT=otel LANGSMITH_TRACING=true dual-backend run (env: browserbase) — spans land in both UIs; .trajectories/ identical to native

Stack 3/5 · base: #2352

---

Summary by cubic

Adds an opt-in OpenTelemetry transport for eval traces via EVAL_TRACE_TRANSPORT=otel. Default Braintrust tracing is unchanged; OTEL now emits input/output and metadata attributes that LangSmith reads.

• New Features

  • framework/braintrust.ts: tracedSpan adds an OTEL path that maps inputs/metadata/metrics to attributes, outputs to attributes, and scores to events; records errors; native path untouched.
  • framework/runner.ts: builds the OTEL tracer provider at run start with a parent project_name:<braintrustProjectName> when enabled.
  • lib/AISdkClientWrapped.ts: when EVAL_TRACE_TRANSPORT=otel, call the SDK directly and pass experimental_telemetry for both generateObject and generateText; native wrapping otherwise.

• Bug Fixes

  • framework/braintrust.ts: emit input.value/output.value attributes (mirrored to braintrust.input_json/braintrust.output_json), move metadata to langsmith.metadata.*, and set langsmith.span.kind so LangSmith shows inputs/outputs correctly.
  • framework/runner.ts: always call shutdownTracing() in a finally block, preserving run-scoped trajectory-group stamping and writeExperimentLink cross-linking; cli.ts also triggers best-effort shutdownTracing() on signal teardown.

^{Written for commit 51cf53e. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 51cf53e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 4 files

Confidence score: 5/5

• Safe to merge after the addressed issues were fixed.

Architecture diagram

sequenceDiagram
    participant CLI as CLI / Signal Handler
    participant Runner as runner.ts
    participant Braintrust as braintrust.ts (tracedSpan)
    participant OTEL as @opentelemetry/api (OtelSpan)
    participant AISDK as AISdkClientWrapped
    participant BraintrustAPI as Braintrust Backend
    participant OTELExporter as OTEL Exporter

    Note over CLI,OTELExporter: OTEL Transport (EVAL_TRACE_TRANSPORT=otel)

    Runner->>Runner: resolveTraceTransport() => "otel"
    Runner->>OTEL: buildTracerProvider()
    OTEL-->>Runner: provider ready

    Runner->>Braintrust: tracedSpan(fn, options)
    alt EVAL_TRACE_TRANSPORT equals otel
        Braintrust->>OTEL: getTracer().startActiveSpan(name)
        OTEL-->>Braintrust: active OtelSpan
        Braintrust->>OTEL: setAttribute(type, input.*)
        Braintrust->>fn: call with OTEL adapter
        fn->>AISDK: createChatCompletion()
        AISDK->>AISDK: loadAISDK() (no wrapAISDK)
        AISDK->>AISDK: pass experimental_telemetry block
        Note over AISDK: generates OTEL spans for generateObject/generateText
        AISDK-->>fn: response
        fn->>Braintrust: adapter.log({output, scores, metadata, ...})
        Braintrust->>OTEL: setAttribute(metadata.*, metrics.*)
        Braintrust->>OTEL: addEvent("scores", ...)
        Braintrust->>OTEL: addEvent("output", ...)
        alt error in fn
            Braintrust->>OTEL: setStatus(ERROR)
            Braintrust->>OTEL: recordException(error)
            Braintrust-->>Runner: throw error
        else success
            Braintrust->>OTEL: span.end()
            Braintrust-->>Runner: return result
        end
        Runner->>Braintrust: flush() (Braintrust log)
        Braintrust->>BraintrustAPI: send logs
        Runner->>OTEL: shutdownTracing()
        OTEL->>OTELExporter: export final spans
        OTEL-->>Runner: done
    else native (default)
        Braintrust->>Braintrust: hasBraintrustApiKey()
        alt has key
            Braintrust->>BraintrustAPI: create Braintrust span
            Braintrust-->>fn: Braintrust adapter
            fn->>Braintrust: adapter.log(...)
            Braintrust->>BraintrustAPI: log output/scores
        else no key
            Braintrust-->>fn: NOOP_SPAN
        end
        Braintrust-->>Runner: return result
    end

    Note over CLI,Runner: Signal teardown
    CLI->>CLI: handleSignal(SIGINT)
    CLI->>OTEL: shutdownTracing()
    OTEL->>OTELExporter: final flush
    CLI->>CLI: process.exit(code)

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[miguelg719]

Superseded by the Stagehand 4.0 retarget stack (#2757–#2762). Main moved to the 4.0 monorepo restructure (packages/core → packages/sdk-ts; evals package restructured), so this 3.x-era diff no longer applies. The OTEL/LangSmith tracing work — plus native-span capture via 4.0's global tracer and claude_code/codex agent spans — is reimplemented additively on 4.0 in the new stack.