[STG-2284] feat(evals): richer OTEL traces — task root, thread grouping, trajectory

[miguelg719]

why

With the transport wired (#2353), a whole eval run rendered in LangSmith as a handful of opaque spans: agent.execute and the verifier spans arrived as separate flat traces with no per-task parent, nothing tied the tasks of one run together, and none of them carried what the agent actually did — so a trace told you almost nothing.

Braintrust looks richer only because its Eval() framework hands the whole TaskResult to its scorers, so the Stagehand logs ride along in the scorer spans (~400KB). OTEL has no equivalent hook, so the same data has to be attached explicitly.

what changed

All three additions are gated on EVAL_TRACE_TRANSPORT=otel; the native Braintrust span tree is untouched.

• Per-task root span — agent.execute / verifier.* now nest under their task instead of arriving as sibling traces.
• Path-safe thread_id — groups every task of one invocation into one LangSmith thread. LangSmith addresses threads as a URL path segment and its gateway answers percent-encoded slashes with a 403 HTML page (which the browser surfaces as a CORS preflight failure, breaking the thread/peek view), so experiment names like agent/onlineMind2Web are sanitized and suffixed per run.
• Trajectory on the root span — the TaskResult (_success, error, metrics, logs). Capped at 2MB with a log-tail trim: LangSmith accepts 3MB payloads and real runs land ~270–640KB, but one oversized span can fail its entire OTLP export batch and silently take every other span in it down.

test plan

• Unit suite green (392); native-path invariants unchanged (braintrust-optional, braintrust-runner-nolog, core-runner).
• Verified end-to-end against live LangSmith on a real local agent run: root + 3 nested children, 59-entry trajectory on the root, all three thread endpoints (stats/traces/messages) returning 200 where the unsanitized id returned 403.
• .trajectories/ remains the complete record and is byte-unchanged.

Stack 6/7 · base: #2355

---

Summary by cubic

Improves OTEL traces for eval runs: per-task root spans, run-level thread grouping with a collision‑resistant, path‑safe thread_id, and the full task trajectory on each task root. Addresses STG-2284. LangSmith traces are now readable and grouped per run; Braintrust spans are unchanged.

• New Features
  • Per-task root span (otel only): agent.execute and verifier.* now nest under each task.
  • Path-safe, collision‑resistant thread_id: sanitizes experiment name and adds timestamp + random suffix to group a run and avoid 403s in LangSmith.
  • Trajectory on the task root: attaches _success, error, metrics, and logs; capped at 2MB (UTF‑8 bytes) with tail-trim; guaranteed within cap even when non‑log fields are oversized; .trajectories/ remains the full record.
  • Tests: unit tests for capForSpan (identity, tail-trim, oversized non-log fields, byte-size enforcement).

^{Written for commit 0dd9b6f. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 0dd9b6f

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 1 file

Architecture diagram

sequenceDiagram
    participant Runner as runEvals
    participant Execute as executeTask
    participant Tracer as tracedSpan (OTEL)
    participant Capper as capForSpan
    participant LangSmith as LangSmith OTLP
    participant Braintrust as Braintrust (native)

    Note over Runner,LangSmith: Per‑run setup
    Runner->>Runner: CHANGED: sanitize experiment name → path‑safe thread_id<br/>(replace "/" with "__", append timestamp)
    
    Note over Runner,Braintrust: Per‑task execution
    alt traceTransport === "otel"
        Runner->>Tracer: NEW: create root span (name, type="task", input)
        Tracer->>Execute: executeTask(input, options)
        Execute-->>Tracer: TaskResult { _success, error, metrics, logs }
        Tracer->>Capper: NEW: capForSpan(TaskResult)
        Capper->>Capper: IF payload >2MB: drop oldest logs entries<br/>until size ≤ MAX_SPAN_PAYLOAD_BYTES
        Capper-->>Tracer: capped output
        Tracer->>LangSmith: NEW: span.log({ output:capped, metadata:{thread_id, ...} })
        Tracer-->>Runner: TaskResult
    else native (Braintrust)
        Runner->>Execute: executeTask(input, options)
        Execute-->>Runner: TaskResult
        Note over Runner,Braintrust: Braintrust tree unchanged
    end

    Note over Runner,LangSmith: Trajectory on root span (OTEL only)
    LangSmith->>LangSmith: Store span with output & metadata<br/>grouped by thread_id for Threads view

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[miguelg719]

Superseded by the Stagehand 4.0 retarget stack (#2757–#2762). Main moved to the 4.0 monorepo restructure (packages/core → packages/sdk-ts; evals package restructured), so this 3.x-era diff no longer applies. The OTEL/LangSmith tracing work — plus native-span capture via 4.0's global tracer and claude_code/codex agent spans — is reimplemented additively on 4.0 in the new stack.