Skip to content

feat(sandbox): driver-independent protected mock service for CLI fixtures - #90

Open
dmorosanu wants to merge 4 commits into
mainfrom
feat/protected-mock-portable
Open

feat(sandbox): driver-independent protected mock service for CLI fixtures#90
dmorosanu wants to merge 4 commits into
mainfrom
feat/protected-mock-portable

Conversation

@dmorosanu

Copy link
Copy Markdown
Contributor

What

Adds a driver-independent protected_mocks fixture service: task fixtures are loaded by a small host-side per-run server, and the agent's sandbox only ever contains a thin client shim. Extracted from #87 (codex/uid-gid-agent-isolation) and made portable; supersedes the protected_mock portion of that PR, which will rebase onto this and re-add the Docker UID/GID wiring.

  • src/coder_eval/protected_mock/{protocol,server,client,runtime}.py - JSON-over-socket request/response service with per-tool request budgets, bounded payload sizes, and prefix-limited passthrough (unchanged from the donor where it worked).
  • Per-run endpoint: AF_UNIX socket in a run-scoped scratch dir when the platform supports it (probed with a real bind), else TCP on 127.0.0.1 with an ephemeral port. Every request carries a per-run random token the generated shim bakes in - same-user hygiene against other local processes casually querying the service, not a security boundary (documented as such).
  • match_mode: subset added alongside exact/normalized: all rule tokens must appear in the invocation's normalized token set; rules scan in fixture-file order, first match wins; exact/normalized take precedence; duplicates allowed; empty subset argv rejected at load.
  • Driver gate inverted: supported under driver: tempdir; under driver: docker validation fails closed ("requires the UID/GID isolation layer; not yet available") until Add UID/GID isolation for evaluated agents #87 lands.
  • Shims: one protected_mocks/<tool> shim (+ .cmd twin) per entry, PATH-prepended like mock_path_dirs; each shim carries its endpoint/token/call-log path itself and does not rely on the agent process environment.
  • Fixture paths resolve against the task YAML's directory host-side (like uipath_eval.eval_set); fixture bytes are never copied into the sandbox.
  • Call log: cli_called-schema JSONL written to protected_mock_calls.jsonl next to task.json in the run dir, outside the sandbox. Diagnostic-only for now: cli_called resolves its log field sandbox-relative and cannot read the host-side file (documented).
  • Orchestrator wiring: server starts after sandbox setup / before agent start; teardown is unconditional in _cleanup, which runs on every exit path. environment_info records the endpoint kind and a SHA-256 digest of the fixture contents.

Why

Eval fixtures are grading material and must not be readable in the agent workspace. The current approach ships them into the sandbox behind a reversible encoding, and evaluated agents have read them to pass tests they should have failed - an encoding is not a boundary. Host-side loading removes the data from the workspace entirely, and the docker-only version in #87 could not run on the tempdir driver the test suites actually use (or on Windows).

Validation

  • ruff format --check, ruff check, pyright: clean.
  • Custom lint (tests/test_custom_lint.py): 171 passed.
  • Full suite: 3948 passed, 103 skipped, plus the 2 pre-existing Windows symlink failures in tests/test_sandbox.py (WinError 1314 without Developer Mode, unrelated).
  • Coverage: 89.89% (gate 80%).
  • tests/test_protected_mock.py (28 tests): validation gates (tempdir OK, docker fails closed, name uniqueness, record_cli overlap), exact/normalized/subset matching (order, first-wins, precedence, budget exhaustion, duplicate rules), passthrough prefix+cache, TCP loopback end-to-end with token (mismatch rejected), AF_UNIX end-to-end where available, per-run endpoint isolation (two servers), runtime teardown (terminate, scratch dir removed), fixture resolution against task dir, shim generation (endpoint baked in, no fixture bytes anywhere in the sandbox tree), self-sufficient shim execution against a live service, and a full Orchestrator.run() lifecycle test (audit record, seeded call log, server stopped, preserved sandbox carries only the shim).

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown

Claude finished @dmorosanu's task in 1m 21s —— View job


Code Review

Todo List

  • Read review guidelines in .github/code_review.md
  • Read CLAUDE.md for project conventions
  • Get full PR diff
  • Review each changed file with full context
  • Check cross-file consistency
  • Analyze what's missing
  • Provide design-level scrutiny
  • Submit final review

Starting review...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant