fix(docker): harness isolation via COPY/PRUNE + GRADE-OUTSIDE. Close the criteria/grader leak

[CarlesUIPath]

Summary

Under --driver docker, the agent-under-test could read its own grading material —
success_criteria, the skills-repo graders (check_*.py), RESOLUTION.md, reference
agents, the staged task.yaml, and the per-task-dir mount — and separately could
overwrite the operator's ~/.uipath/.auth (it was rw-shared with the host). A nightly
audit measured ~2.4% of replicates reading the suite instead of solving the task; an
adversarial "check /work" prompt drives it to ~100%.

This closes the leak by absence, not permissions: the agent container never contains
grading material, grading moves to the host after the container exits, and host state
(auth, checkout) is never rw-shared with the agent.

How it works — before (main) vs. after (this PR)

Before — main: agent and grader share one container/filesystem, and the criteria are staged verbatim into /work/input.

%%{init: {'flowchart': {'useMaxWidth': false, 'padding': 20, 'nodeSpacing': 60, 'rankSpacing': 55}}}%%
flowchart TB
  s1["HOST · stage — task.yaml = FULL criteria; mount RAW skills repo (tests/, check_*.py, reference); ~/.uipath rw-shared"]:::host
  c1["CONTAINER · AGENT + GRADER — same filesystem<br/>agent turn, then grades in-container"]:::leak
  n1["During its turn the agent can read its own success_criteria,<br/>graders, and reference — and overwrite ~/.uipath/.auth"]:::note
  s1 -->|"docker run (TASK_DIR / SKILLS_REPO_PATH point at the above)"| c1
  c1 -.-> n1
  classDef host fill:#eef2ff,stroke:#5566aa,color:#
  classDef leak fill:#ffe0e0,stroke:#cc0000,color:#111;
  classDef note fill:#fff3cd,stroke:#cc9900,color:#

Loading

After — this PR: phase 2 is agent-only; grading is a separate phase 3 on the host.

%%{init: {'flowchart': {'useMaxWidth': false, 'padd'rankSpacing': 55}}}%%
flowchart TB
  s2["HOST · stage (agent-only) — task.yaml strippeared); sanitized skill bundle only — NO raw repo /tests/ / graders / reference / task-dir; ~/.uipath & ~/.claude are throwaway copies"]:::host
  a2["CONTAINER · AGENT ONLY<br/>sees: skill docs ( workspace<br/>cannot see: criteria, check_*.py,tests/, reference, host auth"]:::safe
  g2["HOST · GRADE-OUTSIDE (regrade_on_host)<br/>fua TASK_DIR — agent never shared theirfilesystem"]:::host
  s2 -->|"docker run"| a2
  a2 -->|"container exits → artifacts copied out"| g2
  classDef host fill:#eef2ff,stroke:#5566aa,color:#
  classDef safe fill:#e0ffe0,stroke:#0aa00a,color:#111;

Loading

In one line: split the fused container into ade-on-host (phase 3), and stop mounting the answerkey — or host auth — into the agent's box.

Leaks closed

#	Surface (what the agent could reach on main)	How it's closed
1	Staged criteria — /work/input/task.yaml (full success_criteria) + context.json.source_yaml	task.yaml criteria-stripped (agent_safe_dump), source_yaml nulled, no task_full.json staged; full criteria stay on the host for grading
2	Skills-repo graders — tests/, check_*.py, RESOLUTION.md, reference agents (whole repo mounted)	raw repo not mounted; agent gets only a sanitized :ro bundle (skills/.claude-plugin/commands/agents/hooks)
3	Per-task-dir mount (host_task_dir:host_task_dir)	not mounted; grading reads it on the host via TASK_DIR
4	Reference solution — reference.file / reference.directory	not mounted; template_sources/system_prompt_file auto-mount rejected if it overlaps the task dir
5	Signpost env — TASK_DIR / SKILLS_REPO_PATH	live on the host grader env only; not in the agent's env
6	~/.uipath/.auth (rw-shared → agent could overwrite operator credentials)	agent gets a throwaway copy; the host ~/.uipath (incl. .auth) is never mounted
7	~/.claude/jobs — operator session timelines / conversation copied into every container	excluded from the ~/.claude copy denylist
8	Baked image content — mocks/tooling encoding answers	authoring invariant + baked-image scan test (mounts can't cover this)

Verification

• Docker acceptance (CI-gated): make test-docker-detectors (criteria-absence + host-unchanged proxy + baked-image scan) and the -m live host-unchanged sensor — a real --driver docker run leaves the host byte + metadata identical (also proves ~/.uipath/checkout are untouched).
• Live leak-closure: the adversarial "read your grader" task can no longer obtain the grader sentinel, confirmed on gpt / claude / gemini / kimi.
• Backwards-compat (real runs, all four harnesses): llm_judge, skill_triggered, agent_judge, and simulation (3-turn dialog) all confirmed intact under grade-outside. Early stop is intentionally DISABLED under docker for now (see Limitations).
• make check / typecheck / lint green; full make test = pass modulo pre-existing/environmental failures.

⚠️ The -m live byte-identical check is Linux-authoritative (macOS Docker Desktop's bind-mount remap), but closure is by absence, so it holds on both.

Limitations & follow-ups

• Early stop is a no-op under docker — disabled for now. Criteria are stripped from the container, so the in-container EarlyStopWatcher can't arm. DockerRunner logs a warning; the verdict is unaffected (the host grades the full criteria, and a completed run gates strict-AND). The leak-free follow-up is a host-side watcher over the live event stream (the host already receives the tool-call stream + can signal the container via the heartbeat channel) — see docs/DOCKER_ISOLATION.md § Limitations.
• The durable version of the bundle boundary belongs in the skills repo (an installable bundle that excludes tests/ by construction) — cross-repo follow-up.
• The ~/.claude/~/.uipath copies use a denylist; a follow-up should flip them to an allowlist so new dirs default to excluded.

⚠️ Known blocker — needs decision: pre_run/post_run harness scripts under --driver docker

Context: PR #88 (docker harness isolation, "COPY/PRUNE + GRADE-OUTSIDE"). Kept as draft because of this blocker.

Summary

Grade-outside makes the docker driver leak-safe by no longer mounting the skills-repo tests/ tree into the agent cont
ainer (that tree holds the graders/criteria — the leak we're closing). But many tasks' pre_run/post_run commands in
voke helper scripts that live under tests/ (e.g. python3 $SKILLS_REPO_PATH/tests/tasks/.../seed.py, cleanup_*. py). Those commands currently run inside the container, so under docker they can no longer find their script and f
ail.

Grading itself is unaffected — the skills suite grades workspace-relative (no criterion references an absolute cont
ainer path). This is purely a setup/teardown problem.

Impact

Reproducible counts (grep over $SKILLS_REPO_PATH/tests/tasks):

Metric	Count
Total task YAMLs	1147
Distinct tasks whose pre_run/post_run references $SKILLS_REPO_PATH (the affected set)	243 (~21% of th
e tree)
• with it in post_run	180
• with it in pre_run	123
Tasks running uv sync (must stay in-container)	7

SR="$SKILLS_REPO_PATH/tests/tasks"
{ grep -rl "post_run" "$SR"; grep -rl "pre_run" "$SR"; } | sort -u \
  | xargs grep -l "SKILLS_REPO_PATH" | sort -u | wc -l   # → 243
grep -rlE "uv sync" "$SR" | wc -l                         # → 7

243 is ~21% of the full tree; as a share of the actively-run cloud/e2e suites (uipath-platform, maestro-flow,
maestro-case, agents, ixp, admin, governance, data-fabric) it is a large fraction.

Category split (estimate — derived from reading each distinct script, not a mechanical count):

Phase	Kind	Failure mode under docker today
pre_run	cloud seed / workspace-fixture write	seed with fail_on_error=True (default) → loud task ERROR; `f
ail_on_error: false` → agent runs unseeded →
post_run	teardown (read agent output → delete cloud resources)	non-fatal → **silently leaves cloud resources or
phaned**
pre_run	uv sync venv build (7 tasks)	genuinely must run in-container (venv paths aren't portable)

Does this happen without docker? No — tempdir is fine

It's docker-only, and the reason is where the harness runs, not concurrency:

• tempdir: no container. The sandbox is a plain host tempdir, and pre_run/post_run run on the host (inher
  iting the host env), so $SKILLS_REPO_PATH disk, and creds are present → works.
• docker: pre_run/post_run run inside the container, which grade-outside strips of the tests/ tree → sc
  ripts missing.

Both drivers run a single task's `pre_run → y; the difference is the *isolation boundary
*. So this isn't a new class of bug — docker regressed the location where the harness runs.

Root cause

pre_run/post_run are harness operations (setup/teardown), but they currently execute in the agent's container
. Grade-outside correctly removes the `testsand the harness scripts vanish with it.

Proposed fix — "harness-outside"

Extend the grade-outside principle to the wh/post_runon the host**, where the full rep o + creds live. This does **not** re-open the leak — the scripts never enter the container. In effect, it makes docker do whattempdir` already does.

• post_run → host-only. Teardown/inspepace + cloud is inherently a host concern. No
  new knob.
• **pre_run → new per-command runs_in: ho* agentis the opt-in for setup that must bu ild the agent's *runtime environment* in-container (the 7uv synctasks; image-baked fixtures). Ahost pre_run` th
  at writes workspace files runs into a staginr workspace.
• Migration cost is small: with host as default, ~236 of the affected tasks need zero change; only the ~7 uv sync tasks add one line (runs_in: agent)ma + a ~7-line skills-repo edit.