refactor(evals): extract EvalRunner interface (Braintrust Eval() default; no behavior change)

[miguelg719]

why

Isolates the Braintrust Eval() coupling behind a thin seam so a future backend swap is mechanical instead of a rewrite. Pure structural refactor.

what changed

• framework/evalRunner.ts (new): EvalRunner interface + BraintrustEvalRunner wrapping the existing Eval() call verbatim (same progress/reporter/noSendLogs/flush semantics)
• framework/runner.ts: runEvals builds a config and delegates; everything else (trajectories, summary mapping, otel init/shutdown, offline mode, abort handling) unchanged

test plan

• Unit suite 362/362; concurrency/trialCount/results/summary shapes verified unchanged in review
• Offline (noSendLogs) behavior identical

Stack 4/5 · base: #2353

---

Summary by cubic

Extracted an EvalRunner interface with a default BraintrustEvalRunner to decouple eval execution from Braintrust. No behavior change; runEvals now builds a config and delegates to the runner.

• Refactors
  • Added framework/evalRunner.ts with EvalRunner, EvalRunnerConfig, EvalRunnerResult, and BraintrustEvalRunner that wrap Braintrust Eval()/flush, use silent progress/reporter, and honor sendLogs via noSendLogs.
  • Updated framework/runner.ts to remove inline Braintrust wiring, derive sendLogs from hasBraintrustApiKey(), and call the runner; abort handling, offline mode, concurrency/trials, and summary mapping unchanged.

^{Written for commit 044620c. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 044620c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

No issues found across 2 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant Runner as runEvals()
    participant EvalRunner as EvalRunner Interface
    participant Braintrust as BraintrustEvalRunner
    participant BraintrustLib as @braintrust/core (Eval/flush)
    participant Task as task callback
    participant Progress as onProgress callback

    Note over Runner,Progress: Eval execution flow (no behavior change)

    Runner->>EvalRunner: run(config)
    activate EvalRunner
    EvalRunner->>Braintrust: Eval(projectName, {experimentName, metadata, data, task, scores, maxConcurrency, trialCount}, {noSendLogs})

    Braintrust->>Task: invoke for each testcase
    activate Task
    alt cooperative abort active
        Task->>Progress: {type: "failed", error: "aborted"}
        Task-->>Braintrust: {_success: false, error: "aborted by user"}
    else normal execution
        Task->>Progress: {type: "started"}
        Task->>Task: executeTask(input, resolvedTask, options)
        alt success
            Task->>Progress: {type: "passed"}
        else failure
            Task->>Progress: {type: "failed", error}
        end
        Task-->>Braintrust: TaskResult
    end
    deactivate Task

    Braintrust-->>EvalRunner: evalResult
    alt sendLogs == true
        EvalRunner->>Braintrust: flush()
        Braintrust-->>EvalRunner: done
    end
    EvalRunner-->>Runner: EvalRunnerResult
    deactivate EvalRunner

    Note over Runner,Progress: Post-eval (unchanged)
    Runner->>Runner: shutdownTracing() if otel

Loading

_{Re-trigger cubic}

[miguelg719]

Superseded by the Stagehand 4.0 retarget stack (#2757–#2762). Main moved to the 4.0 monorepo restructure (packages/core → packages/sdk-ts; evals package restructured), so this 3.x-era diff no longer applies. The OTEL/LangSmith tracing work — plus native-span capture via 4.0's global tracer and claude_code/codex agent spans — is reimplemented additively on 4.0 in the new stack.