evals(v4): wire OTEL transport into runEvals

[miguelg719]

Why

Stack 3/6. Wires the provider into the run loop.

What

• buildTracerProvider/shutdownTracing in a try/finally (otel only).
• Gated task-root tracedSpan carrying capForSpan(TaskResult) (UTF-8 byte cap, tail-trim) + path-safe, collision-resistant thread_id.
• Trajectory persistence (writeExperimentLink/persist gate) left byte-identical; native task path unchanged.

Testing

typecheck + unit + build green.

Base: #2758

---

Summary by cubic

Adds OpenTelemetry tracing to runEvals when the trace transport is otel, creating a task‑root span per task. Previously tasks had no tracing; non‑OTel runs and Braintrust logging remain unchanged.

• Gate on resolveTraceTransport() === "otel"; call buildTracerProvider({ braintrustParent: "project_name:<project>" }) and always shutdownTracing() in a finally.
• Wrap each executeTask in tracedSpan (name = task, type = "task", event carries { input: { task, model } }); span logs a capped output subset (_success, error, metrics, logs) and metadata (experiment_name, path‑safe collision‑resistant thread_id, model, task).
• Add capForSpan (2 MB UTF‑8 cap): tail‑trims logs by halving to keep the trailing slice, then drops logs; if still too large or not serializable, replace with a minimal { _truncated }.
• Generate thread_id as <sanitized experimentName>-<base36 timestamp>-<uuid8> for URL‑safe, collision‑resistant threads.
• Add unit tests for capForSpan in packages/evals/tests/framework/capforspan.test.ts.

^{Written for commit fdad0fd. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: fdad0fd

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

3 issues found across 1 file

Confidence score: 3/5

• In packages/evals/framework/runner.ts, the OTEL + LangSmith-only path can route through tracedSpan with NOOP_SPAN, so task-root spans are never created and span propagation can silently fail; this risks losing end-to-end eval trace visibility in that configuration—ensure the gated task-root tracedSpan still creates a real span when BRAINTRUST_API_KEY is absent.
• In packages/evals/framework/runner.ts (runEvals/shutdownTracing), repeated runs in the same process can drop all spans on the second invocation because tracing is shut down but not fully reset for reinit; this creates a concrete regression for long-lived runners—clear/reset tracing state during shutdown so subsequent runs reinitialize cleanly.
• In packages/evals/framework/runner.ts, the new capForSpan truncation and traceThreadId path-sanitization paths lack focused tests, which raises regression risk around byte-cap trimming and safe thread-id formatting—add targeted unit coverage for the cap budget, tail-trim behavior, non-log fallback, and path-safe output.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/framework/runner.ts">

<violation number="1" location="packages/evals/framework/runner.ts:299">
P3: The new `capForSpan` truncation and `traceThreadId` path-sanitization logic ships without unit tests. Add focused tests covering the byte-budget cap, tail-trim halving of logs, the non-log fallback, and the path-safe thread-id regex so the behavior is encoded and future regressions are caught.</violation>

<violation number="2" location="packages/evals/framework/runner.ts:474">
P2: When OTEL transport is active but `BRAINTRUST_API_KEY` is unset (the LangSmith-only case), `tracedSpan` from braintrust.ts returns `fn(NOOP_SPAN)`, so the gated task-root `tracedSpan` never creates a real span and `span?.log({ output: capForSpan(...) })` is a no-op. The task-root span and its payload are therefore silently lost, defeating this PR's OTEL wiring. Create the task-root span via the tracer directly (as `withHarnessAgentSpan` in otel.ts does with `getTracer().startActiveSpan`) rather than reusing the Braintrust-key-gated `tracedSpan`.</violation>

<violation number="3" location="packages/evals/framework/runner.ts:582">
P2: When `runEvals` runs more than once in the same process with OTEL transport, the second run never reinitializes tracing and drops spans. `runEvals` now always shuts tracing down, but `shutdownTracing` does not clear the `providerRegistered` guard that `buildTracerProvider` uses to skip initialization. Reset that guard during shutdown (or remove the guard check) so later runs can register a new provider.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant Runner as runEvals (runner.ts)
    participant OTel as OTel Transport (otel.ts)
    participant Braintrust as Braintrust SDK (braintrust.ts)
    participant Tasks as Task Execution Loop
    participant Env as Environment Config

    Note over Runner,Env: NEW: OTel tracing integration in eval pipeline

    Runner->>Env: resolveTraceTransport()
    Env-->>Runner: traceTransport ("otel" | other)

    Runner->>Runner: Determine braintrustProjectName (CI + core-only logic)

    alt traceTransport === "otel"
        Runner->>OTel: buildTracerProvider({ braintrustParent: "project_name:<name>" })
        OTel-->>Runner: Tracer provider ready
    end

    Runner->>Braintrust: loadBraintrust()
    Braintrust-->>Runner: Eval, flush

    loop For each testcase/task
        Runner->>Tasks: executeTask(input, resolvedTask, options)

        alt traceTransport === "otel"
            Tasks->>Runner: Task result (raw)
            Runner->>Tasks: tracedSpan(callback, spanMetadata)
            Note over Runner,Tasks: NEW: Wrap each task execution in OTel span
            Tasks->>Runner: result (r)
            Runner->>Runner: capForSpan({_success, error?, metrics?, logs?})
            Note over Runner: CHANGED: Enforce 2MB UTF-8 payload cap with tail-trimming
            Runner->>Tasks: span.log({output: cappedResult, metadata: {...}})
            Note over Runner,Tasks: metadata includes experiment_name, thread_id (path-safe + collision-resistant), model, task
        end

        Tasks-->>Runner: Final result
        Runner->>Runner: Track pass/fail counts
        Runner->>Runner: options.onProgress(...)
    end

    alt traceTransport === "otel"
        Runner->>OTel: shutdownTracing()
        Note over Runner,OTel: CHANGED: Guaranteed shutdown in finally block
        OTel-->>Runner: Tracing stopped
    end

    Runner-->>Runner: Return summary (passed, failed, total, results)

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>