Skip to content

feat(integrations): extract Codex SDK harness adapter from evals - #2749

Open
miguelg719 wants to merge 2 commits into
harness/claude-agent-sdkfrom
harness/codex-sdk
Open

feat(integrations): extract Codex SDK harness adapter from evals#2749
miguelg719 wants to merge 2 commits into
harness/claude-agent-sdkfrom
harness/codex-sdk

Conversation

@miguelg719

@miguelg719 miguelg719 commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #2748. Part 3 of the harness consolidation stack. Reshaped after review feedback: thin session layer only.

What

New thin package @browserbasehq/stagehand-integrations-codex-sdk: loadCodexSdk + runCodexSession — the startThread/runStreamed event loop (thread config, tool-step budget with abort + listener detach, usage/stop-reason capture) with explicit options and no EVAL_* env reads. Evals' codex runner and the codex facade example both call it.

Deliberately NOT extracted (stays in evals): the loopback code bridge and mount machinery — they expose evals' in-process tool surfaces to codex and have no integrations consumer. codexCodeBridge.ts is back in evals unchanged (still using the shared redaction from #2746).

The example keeps its posture (read-only sandbox, on-failure approvals, harness-tuned default model) and now exits non-zero on failed sessions instead of printing nothing and exiting 0.

Net diff +674/−420 — scaffolding plus the session module; logic is a move.

Verification

  • Full gates ✅; session tests moved with the code
  • Connected smoke re-run on the reshaped code — results in PR comment; step-budget semantics previously verified identical to main (50 counted command executions)

@changeset-bot

changeset-bot Bot commented Aug 16, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: b8e63d9

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 22 files

Architecture diagram
sequenceDiagram
    participant E as Evals Runner
    participant C as Codex SDK Harness
    participant B as Bridge Server
    participant S as Codex Session
    participant M as MCP Server
    participant A as Codex Example

    Note over E,C: NEW: Evals delegates to shared harness

    E->>C: runCodexSession(prompt, model, thread config)
    C->>C: Create AbortController, forward signal

    alt Handle-based mount (code bridge)
        E->>B: startCodeBridge(handles, context)
        B-->>E: bridge port
        E->>M: Prepare mount session, write browser_run.mjs
        M->>B: POST /run with snippet code
        B->>B: Execute snippet with handles in scope
        B-->>M: Result or error (redacted)
        B->>E: onRunExecuted notification
    else MCP-based mount
        E->>M: Prepare mount via mcp_servers config
        M->>E: codexConfig forwarded
    end

    E->>S: Start Codex thread (sandbox, approval, webSearch disabled)
    S-->>C: Stream events
    C->>C: Count tool steps (command_execution + mcp_tool_call)

    alt Step budget exceeded (default 50)
        C->>S: Abort stream with budget exhaustion reason
    end

    alt Agent produces completion
        C-->>E: finalMessage, token usage, status=completed
        E->>E: Build transcript, parse result
    else SDK error or turn failed
        C-->>E: iterationError, status=sdk_error
        E->>E: Mark eval as failed
    end

    Note over A,C: NEW: Codex example uses shared harness
    
    A->>C: prepareCodexMountSession(mcp mount)
    C-->>A: Prepared session (cwd, codexConfig)
    A->>C: runCodexSession(read-only sandbox, on-failure approval)
    C-->>A: Result
    
    alt Status not completed
        A->>A: Throw error (non-zero exit)
    else Completed
        A->>A: Print final message
    end
Loading

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/integrations/codex-sdk/src/session.ts Outdated
Comment thread packages/integrations/codex/src/agent.ts Outdated
Comment thread turbo.json
Comment thread packages/integrations/codex-sdk/src/session.ts
Comment thread packages/integrations/codex-sdk/src/session.ts
Comment thread packages/integrations/codex-sdk/tests/session.test.ts
Comment thread packages/integrations/codex-sdk/src/session.ts Outdated
Comment thread packages/integrations/codex/src/agent.ts Outdated
Comment thread packages/evals/framework/codexToolAdapter.ts Outdated
Comment thread packages/integrations/codex-sdk/src/session.ts Outdated
@miguelg719

Copy link
Copy Markdown
Collaborator Author

Connected smoke on the reshaped code (restored evals bridge/mount + thin session package): b:webvoyager --harness codex --tool stagehand_code -e browserbase3/3 trials passed (100%), all complete in 23–46 steps with real answers. (Baseline before the stack was 1/3 on this task; the bridge round-trips, step budget, and grading all flow through the new session layer.)

New thin package @browserbasehq/stagehand-integrations-codex-sdk:
loadCodexSdk + runCodexSession — the startThread/runStreamed event loop
(thread config, step budget, abort forwarding, usage/stop-reason) with
explicit options and no EVAL_* env reads. The evals codex runner and
the codex facade example both call it now.

Deliberately NOT extracted: the loopback code bridge and mount
machinery stay in evals — they exist to expose evals' in-process tool
surfaces to codex and have no integrations consumer.
…ards

- constructor failures no longer masquerade as missing-install errors
  (cubic P2)
- error strings passed through sanitizeErrorMessage before logs and the
  returned stopReason (cubic P1)
- positiveInteger clamps fractional inputs to >=1 so maxToolSteps can't
  become zero (cubic P3); budget test now asserts result.status
- delete unused package vitest.config.ts (tests run via repo root)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant