[LLM] Add Terminal-Bench evaluation scripts

[mergennachin]

Add setup and run scripts for evaluating the LLM server with Harbor and mini-SWE-agent. TOML files under evals/configs select the model and task settings. Harbor handles task downloads, attempts, scores, and trajectories; the launcher starts the server, checks container connectivity, and records token counts and timings.

Validated with launcher dry-run, Harbor command validation, Lintrunner, and ShellCheck.

Authored with assistance from OpenAI Codex.

[mergennachin]

Stack from ghstack (oldest at bottom):

• -> [LLM] Add Terminal-Bench evaluation scripts #23384

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23384

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit b527ff7 with merge base 0b3d26d ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[mergennachin]

@claude Review this PR

[claude]

Claude finished @mergennachin's task in 1m 43s —— View job

---

The scripts look well put together, but there are two real problems: if python is left out of the TOML, the server starts with the Harbor venv's interpreter, which doesn't have the server's dependencies, and the server listens on 0.0.0.0 by default with no auth.

Detailed review

PR Review: #23384

Summary

This PR adds macOS setup and run scripts that use Harbor and mini-SWE-agent to run Terminal-Bench 2.0 against the ExecuTorch LLM server. There are two correctness problems in load_config defaults, one exposure issue, and some smaller robustness gaps.

Security

• examples/llm_server/evals/terminal_bench/runner.py:34: host defaults to 0.0.0.0. The server itself defaults to 127.0.0.1. With this default, an unauthenticated server that runs arbitrary prompts listens on every interface, including Wi-Fi/LAN, for the whole run. That can be hours with real task sets. Containers do need to reach the host. Still, please either say this in the README and the example TOML (and suggest a firewall or a trusted network), or bind only to the interface Colima or Docker Desktop actually uses. Users can override host in the TOML, but nothing tells them to.

Code Quality

• runner.py:32: server.setdefault("python", sys.executable) falls back to the wrong interpreter. run.sh starts the runner with the Harbor venv (~/.cache/executorch-evals/terminal-bench/venv/bin/python). That venv only installs harbor and doesn't have fastapi or pydantic. If python is left out of the TOML, the server subprocess fails on import, and the only error is a generic "Server exited; see server.log". Make python required, like worker_bin/model_path, or drop the default. The example TOML sets it, which hides the problem.
• runner.py:36-41: server[key] makes hf_tokenizer required, but server.py:162 makes it optional when --allow-chatml-fallback is set. A config without it fails with KeyError, and main() prints only 'hf_tokenizer'. The same unhelpful one-word error appears when max_context is missing (runner.py:53). Also, the path heuristic only resolves hf_tokenizer/python relative to the config when they start with /, ~ or .. A relative directory like tokenizers/qwen3 is passed through unchanged and resolved against the server's working directory, while worker_bin etc. are resolved against the config. Suggested fix: skip keys that aren't set, and raise a ValueError that names the missing required key.
• runner.py:345-356: metrics() runs inside finally. If it raises (for example a llm_turn_stats line missing a field after a future log-format change), that exception replaces the real failure (preflight/Harbor error), and metrics.json is never written. Wrap the metrics write so it can't hide the original error.
• runner.py:378-380: output.mkdir(exist_ok=False) runs before commands(), which shells out to docker context show (runner.py:81) even with --dry-run. If Docker isn't running, the run leaves an empty results directory and fails with a raw CalledProcessError/FileNotFoundError string. Resolve agent_host before creating the directory, or tolerate a missing Docker during --dry-run.
• setup.sh:36: ln -s isn't idempotent. This branch runs when docker compose version fails. If a stale or dangling docker-compose symlink is the reason for that failure, ln -s fails under set -e and setup can never recover. Use ln -sf.

Testing

• load_config, commands, metrics and summarize are pure functions with non-trivial logic: path resolution, TOML→CLI flag mapping, and llm_turn_stats parsing. None of them have tests. The flag mapping depends on server.py's argparse names, and the metrics parsing depends on the exact log format at serving_chat.py:220. Either can drift silently. A small pytest next to examples/llm_server/python/tests/ (no Docker or Harbor needed) would catch drift: check that the generated serve argv parses with the server's argument parser, and that metrics() handles a sample llm_turn_stats line.

Recommendation

Request Changes: fix the python default and the hf_tokenizer requirement in load_config, and either document or narrow the 0.0.0.0 default. The other items are small robustness fixes.