feat(integrations): extract Codex SDK harness adapter from evals - #2749
feat(integrations): extract Codex SDK harness adapter from evals#2749miguelg719 wants to merge 2 commits into
Conversation
|
There was a problem hiding this comment.
All reported issues were addressed across 22 files
Architecture diagram
sequenceDiagram
participant E as Evals Runner
participant C as Codex SDK Harness
participant B as Bridge Server
participant S as Codex Session
participant M as MCP Server
participant A as Codex Example
Note over E,C: NEW: Evals delegates to shared harness
E->>C: runCodexSession(prompt, model, thread config)
C->>C: Create AbortController, forward signal
alt Handle-based mount (code bridge)
E->>B: startCodeBridge(handles, context)
B-->>E: bridge port
E->>M: Prepare mount session, write browser_run.mjs
M->>B: POST /run with snippet code
B->>B: Execute snippet with handles in scope
B-->>M: Result or error (redacted)
B->>E: onRunExecuted notification
else MCP-based mount
E->>M: Prepare mount via mcp_servers config
M->>E: codexConfig forwarded
end
E->>S: Start Codex thread (sandbox, approval, webSearch disabled)
S-->>C: Stream events
C->>C: Count tool steps (command_execution + mcp_tool_call)
alt Step budget exceeded (default 50)
C->>S: Abort stream with budget exhaustion reason
end
alt Agent produces completion
C-->>E: finalMessage, token usage, status=completed
E->>E: Build transcript, parse result
else SDK error or turn failed
C-->>E: iterationError, status=sdk_error
E->>E: Mark eval as failed
end
Note over A,C: NEW: Codex example uses shared harness
A->>C: prepareCodexMountSession(mcp mount)
C-->>A: Prepared session (cwd, codexConfig)
A->>C: runCodexSession(read-only sandbox, on-failure approval)
C-->>A: Result
alt Status not completed
A->>A: Throw error (non-zero exit)
else Completed
A->>A: Print final message
end
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
84c655d to
36fc26c
Compare
|
Connected smoke on the reshaped code (restored evals bridge/mount + thin session package): |
New thin package @browserbasehq/stagehand-integrations-codex-sdk: loadCodexSdk + runCodexSession — the startThread/runStreamed event loop (thread config, step budget, abort forwarding, usage/stop-reason) with explicit options and no EVAL_* env reads. The evals codex runner and the codex facade example both call it now. Deliberately NOT extracted: the loopback code bridge and mount machinery stay in evals — they exist to expose evals' in-process tool surfaces to codex and have no integrations consumer.
…ards - constructor failures no longer masquerade as missing-install errors (cubic P2) - error strings passed through sanitizeErrorMessage before logs and the returned stopReason (cubic P1) - positiveInteger clamps fractional inputs to >=1 so maxToolSteps can't become zero (cubic P3); budget test now asserts result.status - delete unused package vitest.config.ts (tests run via repo root)
36fc26c to
b8e63d9
Compare
Stacked on #2748. Part 3 of the harness consolidation stack. Reshaped after review feedback: thin session layer only.
What
New thin package
@browserbasehq/stagehand-integrations-codex-sdk:loadCodexSdk+runCodexSession— thestartThread/runStreamedevent loop (thread config, tool-step budget with abort + listener detach, usage/stop-reason capture) with explicit options and noEVAL_*env reads. Evals' codex runner and the codex facade example both call it.Deliberately NOT extracted (stays in evals): the loopback code bridge and mount machinery — they expose evals' in-process tool surfaces to codex and have no integrations consumer.
codexCodeBridge.tsis back in evals unchanged (still using the shared redaction from #2746).The example keeps its posture (read-only sandbox, on-failure approvals, harness-tuned default model) and now exits non-zero on failed sessions instead of printing nothing and exiting 0.
Net diff +674/−420 — scaffolding plus the session module; logic is a move.
Verification