Skip to content

fix(eve): report evals without agent output to Braintrust - #1465

Open
BlackishGreen33 wants to merge 1 commit into
vercel:mainfrom
BlackishGreen33:bg/fix-braintrust-no-output-1405
Open

fix(eve): report evals without agent output to Braintrust#1465
BlackishGreen33 wants to merge 1 commit into
vercel:mainfrom
BlackishGreen33:bg/fix-braintrust-no-output-1405

Conversation

@BlackishGreen33

Copy link
Copy Markdown
Contributor

Closes #1405.

What

Braintrust reporting stopped an eve eval run when an eval finished without producing an agent turn. Eve correctly records that output as null, but Braintrust requires a non-null output value.

How

  • Send an empty string to Braintrust when the eval output is null or undefined.
  • Keep every existing non-null output unchanged.
  • Add a regression test for an eval with no agent output.

Tests / changeset

  • Added focused unit coverage for the Braintrust reporter.
  • Added an eve patch changeset.

Verification

  • pnpm --filter eve exec vitest run --config vitest.unit.config.ts src/evals/runner/reporters/braintrust.test.ts
  • pnpm test
  • pnpm typecheck
  • pnpm fmt
  • pnpm lint
  • pnpm guard:invariants
  • pnpm check:deps
  • pnpm build

PR Checklist

  • I linked an issue with prior discussion confirming this change is wanted
  • I ran the relevant checks from CONTRIBUTING.md
  • I added tests and documentation where relevant
  • I added a changeset because this touches the published eve package
  • DCO sign-off passes for every commit

Signed-off-by: 墨綠BG <s5460703@gmail.com>
@vercel

vercel Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

@BlackishGreen33 is attempting to deploy a commit to the Vercel Team on Vercel.

A member of the Team first needs to authorize it.

@BlackishGreen33
BlackishGreen33 marked this pull request as ready for review July 31, 2026 15:05
Copilot AI review requested due to automatic review settings July 31, 2026 15:05

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@BlackishGreen33

Copy link
Copy Markdown
Contributor Author

CI note: all checks that do not require secrets unavailable to fork PRs passed, including lint, typecheck, unit tests, Ubuntu and Windows integration tests, scenario tests, DCO, Socket, and Postgres E2E.

The remaining failures are external to this patch:

  • Vercel deploy jobs have empty VERCEL_TOKEN, VERCEL_PROJECT_ID, and VERCEL_ORG_ID, then fail with No existing credentials found.
  • Real-model E2E and TUI jobs have an empty AI_GATEWAY_API_KEY, then fail with AI Gateway received no credentials.
  • The docs preview requires Vercel team authorization.

No code changes were made for these fork-only credential failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Braintrust reporter crashes the whole eval run when an eval produces no agent turn (output: null → "output must be specified")

2 participants