fix(skill-evals): contain fixture reads inside the eval tree

[potiuk]

Summary

• magpie-run-evals.sh is named in sandbox.excludedCommands, so everything the eval runner reads, it reads unsandboxed. fix(sandbox): use Edit(path) for file denies, add the eval exclusion #1311 left fixtures agent-writable on an explicit, reasoned basis: a fixture "can route repository text to the model, which the session's own API access already allows."
• That reasoning is sound — but nothing enforced the repository half of it. Two escapes, both reachable by a sandboxed agent writing a fixture:
  • A symlink. report.md -> ~/.ssh/id_rsa is followed by read_text() and its contents land in the prompt piped to claude -p. The wrapper's guard checks how its argument is spelled (tools/skill-evals/evals prefix, no ..), not where it leads — and -d follows a directory symlink just as happily.
  • A path string, no symlink needed. step-config.json supplies skill_md, which is joined to the repo root and read with no containment, so enough .. walks straight out.
• Either converts a fixture read into a read of any file the host can see, including the credential paths the sandbox denies the session directly. That is precisely what the exclusion was not supposed to buy.
• Fix: resolve and contain. The wrapper resolves its target against tools/skill-evals/evals; the runner resolves every fixture path it reads. Containment is by resolved path, not a ban on symlinks, so shared fixtures factored out inside the tree keep working.

This is a follow-up to #1311 (merged), not a regression in it — the gap is in the accepted-risk reasoning, which this PR makes true by construction rather than by assumption. The comment block in the wrapper is updated to say what is now enforced instead of assumed.

Type of change

• Skill change (.claude/skills/<name>/) — eval fixtures updated below
• Tool / bridge contract (tools/<system>/*.md)
• Python package (tools/*/ with pyproject.toml) — tools/skill-evals
• Groovy reference impl
• Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM) — sandbox trust boundary
• Documentation (docs/, README.md, CONTRIBUTING.md)
• Project template (projects/_template/)
• CI / dev loop (prek, workflows, validators)

Test plan

• prek run --all-files passes (32 hooks, exit 0)
• uv run pytest tools/skill-evals/tests/ — 170 passed
• TDD: every case was written first and watched fail against the pre-fix tree.
  • load_case returned the symlinked outside file's contents with no error — the escape is confirmed, not theoretical.
  • load_step_config followed both a symlinked system-prompt.md and a ..-escaping skill_md.
  • The wrapper passed a symlinked target through to the runner (it only stopped because the decoy held no eval-shaped cases — it had already traversed outside the repo).
• Negative-space covered too: test_load_case_allows_a_symlink_that_stays_inside_the_eval_tree pins that this is containment, not a symlink ban, and test_lets_a_real_path_inside_the_evals_tree_through pins that the wrapper gate does not cost the normal case.
• New wrapper test file tools/skill-evals/tests/test_run_evals_wrapper.py — the wrapper had no tests before; the three pre-existing gates (one argument, no flags, prefix) are now pinned alongside the new one.

RFC-AI-0004 compliance

• Sandbox — this is the point of the change: it restores the exclusion's intended blast radius. No new host access; strictly narrows what the unsandboxed path may read.
• Write-access discipline — read-path only; no mutation added.

Linked issues

Refs #1311.

Notes for reviewers (optional)

• Provenance: found by a Codex adversarial-review pass. I confirmed it by test before writing any fix — the first run of test_load_case_refuses_a_fixture_symlinked_outside_the_eval_tree returned the planted secret rather than raising. The skill_md path-traversal vector is a second hole in the same class that the original report did not name.
• Containment root choice: load_case contains against case_dir.parent (the fixtures dir, which by construction is the case's parent), load_step_config against fixtures_dir, and skill_md against the repo root — since a legitimate skill_md genuinely points elsewhere in the repo. If you would rather have one root (the evals tree) threaded through all of them, say so; it is a slightly larger change to the call signatures.
• Not addressed here: find_repo_root-relative reads elsewhere in the runner that are not fixture-controlled were left alone deliberately, to keep this to the trust boundary the exclusion created.

🤖 Generated with Claude Code