Add OpenHands CLI agent - #116
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c18cd240a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| env_status, agent_done = self._map_exit_status( | ||
| returncode, execution_error, has_result=bool(answer) | ||
| ) |
There was a problem hiding this comment.
When OpenHands emits an agent message and then exits nonzero without a JSON error event—for example, after an MCP failure reported only on stderr—answer is truthy, so _map_exit_status(..., has_result=True) returns success/done; the subsequent checks neither inspect the return code nor read stderr. This records a crashed task as completed and admits its partial answer into evaluation. Preserve the partial answer if useful, but mark nonzero exits as failed unless the run timed out.
Useful? React with 👍 / 👎.
| env = {**os.environ, "HOME": str(home_dir), "OH_PERSISTENCE_DIR": str(home_dir / ".openhands")} | ||
| env.setdefault("UV_CACHE_DIR", str(Path.home() / ".cache" / "uv")) |
There was a problem hiding this comment.
When NPM_CONFIG_CACHE is unset, as in the documented non-Docker installation, replacing HOME makes npm resolve its cache to the task-local .openhands-home/.npm. Because every MCP launch uses npx @playwright/mcp@latest, each benchmark task starts with an empty npx cache, repeatedly downloads Playwright MCP, inflates end-to-end latency, and makes otherwise cached runs network-dependent. Preserve the operator's npm cache explicitly, as is already done for UV_CACHE_DIR.
Useful? React with 👍 / 👎.
| from browseruse_bench.agents import openhands # noqa: F401 | ||
| except ImportError as exc: | ||
| logger.warning("Skipping optional agent module browseruse_bench.agents.openhands: %s", exc) |
There was a problem hiding this comment.
The OpenHands module has no optional Python SDK import, so catching every ImportError here also swallows project or interface errors raised by its imports. In such an environment the provider is silently left unregistered and users receive a misleading unknown-agent failure rather than the underlying defect. Import it directly, or move genuinely optional loading into a lazy factory that only tolerates the target package being absent.
AGENTS.md reference: AGENTS.md:L12-L13
Useful? React with 👍 / 👎.
| if agent_config.get("api_key"): | ||
| env["LLM_API_KEY"] = str(agent_config["api_key"]) | ||
| if agent_config.get("base_url"): | ||
| env["LLM_BASE_URL"] = str(agent_config["base_url"]) |
There was a problem hiding this comment.
When the selected model configuration omits api_key or base_url—including when an unset $OPENAI_BASE_URL expands to an empty string—the corresponding inherited LLM_API_KEY or LLM_BASE_URL remains in this copied environment because these branches only overwrite truthy values. Since the command passes --override-with-envs, a stale operator-level OpenHands variable can silently authenticate against or route the benchmark to a different provider than the selected config. Remove both variables from the inherited environment before conditionally adding the resolved values.
Useful? React with 👍 / 👎.
| home_dir = self._write_workspace_config(agent_config, task_workspace, cdp_url) | ||
| cmd = self._build_command(f"{rules}\n\n{prompt}", agent_config) | ||
|
|
||
| env = {**os.environ, "HOME": str(home_dir), "OH_PERSISTENCE_DIR": str(home_dir / ".openhands")} |
There was a problem hiding this comment.
On Windows, setting HOME alone does not redirect ~: Python's Windows home resolution prefers USERPROFILE and then HOMEDRIVE/HOMEPATH. Consequently OpenHands can read the operator's ~/.openhands rather than the task-local directory where this adapter wrote mcp.json, leaving the Playwright MCP server unavailable or exposing mutable user configuration. Set the Windows profile variables as well, or pass an explicit OpenHands config location.
Useful? React with 👍 / 👎.
| && rm -rf /var/lib/apt/lists/* \ | ||
| && if [ "$USE_CN_MIRROR" = "true" ]; then npm config set registry https://registry.npmmirror.com -g; fi \ | ||
| && npm install -g @anthropic-ai/claude-code @openai/codex openclaw \ | ||
| && UV_TOOL_DIR=/opt/uv-tools UV_TOOL_BIN_DIR=/usr/local/bin uv tool install openhands --python 3.12 \ |
There was a problem hiding this comment.
In this python:3.11-slim stage, --python 3.12 makes uv download a managed interpreter; uv python dir shows that the default installation directory is /root/.local/share/uv/python. Setting only UV_TOOL_DIR relocates the tool environment, not that interpreter, so its Python executable references a path beneath /root, which mode 0700 prevents the documented uid-1000 runtime user from traversing. The OpenHands entry point therefore cannot start in the CLI-enabled image. Set UV_PYTHON_INSTALL_DIR to a shared location (and grant traversal), or install Python 3.12 somewhere accessible to the runtime user.
Useful? React with 👍 / 👎.
| "mcpServers": { | ||
| "playwright": { | ||
| "command": agent_config.get("playwright_mcp_command", "npx"), | ||
| "args": build_playwright_mcp_args(agent_config, cdp_url), |
There was a problem hiding this comment.
For managed backends whose CDP URL embeds credentials, this persists the secret verbatim in every task workspace. For example, browseruse_bench/browsers/providers/steel.py lines 52–54 explicitly append the configured apiKey to the CDP URL, and this mcp.json is never removed or scrubbed after the subprocess exits. Anyone receiving or inspecting the experiment directory can recover that provider key. Delete or redact the MCP configuration in a finally cleanup path once OpenHands has stopped.
Useful? React with 👍 / 👎.
1 participant
Summary
Adds an OpenHands CLI agent adapter for browser benchmark runs. The adapter launches
openhands --headless --json, injects Playwright MCP through a task-local OpenHands home, maps OpenHands JSONL events into benchmark actions/results, and registers the agent in config, docs, and Docker optional CLI installs.Gemini CLI is intentionally not included in this PR.
Contribution type
Reproduction or validation
Note: running
uv run pytest tests/with the local disabled Claude key present still reaches the Claude Code external smoke and fails withAPI Error: 403 API Key 所属分组已停用. That is unrelated to this OpenHands adapter; the clean validation above explicitly blanksANTHROPIC_API_KEYso that external smoke is skipped.Result artifacts
OpenHands real smoke output:
/Users/zhu/.codex/worktrees/73b0/browseruse-agent-bench/experiments/LexBench-Browser/All/openhands/gpt-5.5/20260731_143248The smoke result had
env_status=success,agent_done=done, 10 steps, and 4 screenshots.Self-review (mandatory)
See Mandatory Self-Review.
PRs with unticked boxes are returned without review. / 以下勾选项未完成的 PR 会被直接打回。
/ 我逐行读完了自己的 diff,每处改动都是有意为之
no hardcoded config, imports at top) / diff 符合 CONTRIBUTING.md 的硬性红线
/code-reviewand fixed or justified every finding/ 本地跑过
/code-review,findings 已全部修复或说明理由uv run pytest tests/passes and behavior changes have targeted tests/ 测试通过,行为变更配了针对性测试
touch any agent runtime path / 涉及 agent 运行路径的改动已附真实 smoke run 证据,或本
PR 不涉及 agent 运行路径
Notes for reviewers
OpenHands requires Python 3.12+ when installed as a uv tool. The adapter currently supports local/self-launched browsers and CDP-capable managed browser backends via Playwright MCP; non-CDP backends fail fast with an explicit error.
Per user instruction, self-review was performed manually instead of using Claude review automation.