Skip to content

evals(v4): braintrust tracedSpan OTEL branch - #2760

Open
miguelg719 wants to merge 1 commit into
miguelgonzalez/evals-v4-otel-runnerfrom
miguelgonzalez/evals-v4-otel-braintrust
Open

evals(v4): braintrust tracedSpan OTEL branch#2760
miguelg719 wants to merge 1 commit into
miguelgonzalez/evals-v4-otel-runnerfrom
miguelgonzalez/evals-v4-otel-braintrust

Conversation

@miguelg719

@miguelg719 miguelg719 commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Why

Stack 4/6.

What

  • Prepend an otel branch to tracedSpan emitting LangSmith attrs (langsmith.span.kind, input/output.value, metadata).
  • Native traced()/NOOP path below is byte-identical when EVAL_TRACE_TRANSPORT is unset.

Testing

typecheck + unit + build green.

Base: #2759


Summary by cubic

Adds an OTEL branch to tracedSpan in the braintrust evaluator to emit LangSmith-compatible span attributes when EVAL_TRACE_TRANSPORT=otel. When unset or not "otel", behavior is unchanged; the native traced()/NOOP path remains byte-identical.

  • Emits: langsmith.span.kind (llm|chain), input.value/input.mime_type (only when input is present), output.value/output.mime_type (on log), and langsmith.metadata; values are JSON-serialized and size-capped (mime_type is application/json).
  • Records exceptions, sets error status on the OTEL span, and always ends it.
  • To enable, set EVAL_TRACE_TRANSPORT=otel; no other changes required.

Written for commit 78fdf60. Summary will update on new commits.

Review in cubic

@changeset-bot

changeset-bot Bot commented Aug 17, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 78fdf60

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file

Architecture diagram
sequenceDiagram
    participant Eval as Evaluator
    participant TS as tracedSpan()
    participant OTEL as OTel Tracer
    participant Span as OTel Span
    participant LS as LangSmith

    Note over Eval,LS: NEW: OTel transport branch for tracedSpan

    Eval->>TS: tracedSpan(fn, options)

    alt EVAL_TRACE_TRANSPORT === "otel"
        TS->>TS: resolveTraceTransport() === "otel"
        TS->>OTEL: getTracer().startActiveSpan(name, attrs)
        OTEL->>Span: Create span with LangSmith attrs
        Note over Span: langsmith.span.kind (llm|chain)<br/>input.value + input.mime_type (if input)
        
        Span->>Eval: Return span as SpanLike
        Eval->>Span: log(event)
        Alt event.output defined
            Span->>Span: Set output.value + output.mime_type
        end
        Alt event.metadata defined
            Span->>Span: Set langsmith.metadata
        end
        
        alt No exception
            Eval->>Span: fn completes
            Span->>Span: otelSpan.end()
            Span-->>TS: Return result
        else Exception thrown
            Eval->>Span: error thrown
            Span->>Span: recordException(error)
            Span->>Span: otelSpan.end()
            Span-->>TS: Re-throw error
        end
        TS-->>Eval: T (result or error)
        
    else EVAL_TRACE_TRANSPORT unset or other
        Note over TS: Byte-identical native path (unchanged)
        TS->>TS: hasBraintrustApiKey()?
    end
Loading

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/evals/framework/braintrust.ts
Comment thread packages/evals/framework/braintrust.ts Outdated
Comment thread packages/evals/framework/braintrust.ts Outdated
Comment thread packages/evals/framework/braintrust.ts
@miguelg719
miguelg719 force-pushed the miguelgonzalez/evals-v4-otel-braintrust branch from 093bcac to 86312b6 Compare August 17, 2026 19:46
@miguelg719
miguelg719 force-pushed the miguelgonzalez/evals-v4-otel-braintrust branch from 86312b6 to d277c74 Compare August 17, 2026 20:03
@miguelg719
miguelg719 force-pushed the miguelgonzalez/evals-v4-otel-braintrust branch from d277c74 to cd314ce Compare August 17, 2026 21:03
@miguelg719
miguelg719 force-pushed the miguelgonzalez/evals-v4-otel-braintrust branch from cd314ce to 6a402a5 Compare August 17, 2026 22:05
@miguelg719
miguelg719 force-pushed the miguelgonzalez/evals-v4-otel-braintrust branch from 6a402a5 to ee28d3a Compare August 17, 2026 22:07
@miguelg719
miguelg719 force-pushed the miguelgonzalez/evals-v4-otel-braintrust branch from ee28d3a to 78fdf60 Compare August 17, 2026 22:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant