Skip to content

fix(sandbox): make tool decisions host-authoritative - #329

Open
Jake Present (jakepresent) wants to merge 3 commits into
jake/fix-sandbox-case-correlationfrom
jake/fix-sandbox-host-action-ledger
Open

fix(sandbox): make tool decisions host-authoritative#329
Jake Present (jakepresent) wants to merge 3 commits into
jake/fix-sandbox-case-correlationfrom
jake/fix-sandbox-host-action-ledger

Conversation

@jakepresent

@jakepresent Jake Present (jakepresent) commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Problem

The stock sandbox currently trusts the evaluated target to choose pass/mock/block and return its own action evidence. Read-only policy files do not make that evidence authoritative: target code can omit or forge endpoint events, and the writable target ledger is not an independent record.

Approach

Add an explicit target.host_action_mediation: true path for stock containers:

  • run policy evaluation, mock resolution, and the action ledger in an authenticated service on the ASSERT host;
  • route the target's AgentHooksToolHost through RemoteActionMediator over the existing narrow relay;
  • require the target client to register the ASSERT-owned case ID during startup;
  • append the attempted call and pass/mock/block decision before returning to the target;
  • keep the ledger outside every target-visible mount;
  • replace target-supplied tool events with host ledger rows in the ASSERT transcript;
  • preserve completed and incomplete action rows when a turn fails;
  • retain attempt order across completed and incomplete calls; and
  • fail startup or the turn rather than silently accepting an opted-in target that did not use the host mediator.

The evidence contract is intentionally split:

  • attempts and pass/mock/block decisions are host-authoritative;
  • mock/block results are host-generated and authoritative;
  • passed tools still execute against disposable state inside the target, so their results are explicitly labeled target_reported and result_authoritative: false;
  • arbitrary in-process calls that bypass the mediator are not claimed as observed. Container isolation, deny-by-default egress, and host-only real credentials remain the containment backstop.

Existing endpoint setups and containers that do not opt in keep the current local/target-reported path.

Review hardening

Adversarial review probes found and this head fixes:

  • target-forged case IDs;
  • duplicate call IDs consuming scenario state;
  • completed and incomplete rows being drained out of attempt order;
  • host evidence being lost when fail-closed validation raised after a turn; and
  • opted-in images silently starting without a host-mediator client;
  • a pass executor without explicit execution tracking could run a side effect behind an incorrect execution claim; and
  • the fail-closed target-event gate depended on a marker inside rendered evidence instead of the structural host-ledger drain.

Tests

  • Exact-head sandbox suites after review hardening: 135 passed, 4 skipped
  • Exact-head Tier 1 CI: passed
  • Focused sandbox/security suites before the follow-up hardening: 186 passed
  • Full Python suite before the follow-up hardening: 1391 passed, 24 skipped, 840 subtests passed
  • Fatal Ruff checks: passed
  • Regression sabotage: the forged-target-event test fails when host replacement is disabled, then passes with the fix restored
  • Live Docker proof using the existing local image with this branch's package overlaid:
    • host decisions observed for lookup_customer and send_message;
    • passed result labeled target_reported;
    • mocked result labeled host_mediator;
    • host ledger outside target mounts; and
    • target-side ledger forgery did not change the host ledger.
  • Zero leftover sandbox containers or networks after the proof.

A fresh stock-image build was blocked locally by an external files.pythonhosted.org TLS handshake failure while pip fetched build requirements. The real-Docker regression is included in tests/test_sandbox_runtime_docker.py; it should run in the dedicated Docker workflow from #328 once this stack is rebased onto it.

Stack and landing

This PR is intentionally stacked on #327 because stable ASSERT case identity is part of the host ledger contract. After #327 merges, rebase this branch onto main and change the base.

Related parallel fixes:

Recommended landing order: #328, #326, #327, then this PR with the Docker workflow rerun against its exact head.

Deliberate follow-up

This PR does not move consequential real tool implementations and credentials behind a host-side adapter. That stronger boundary would make passed execution independently authoritative, but it changes the customer onboarding/tool-packaging contract and should be reviewed as a separate product and architecture extension.

Two hardening fixes on the trust boundary this PR introduces.

RemoteActionMediator defaulted an untracked executor to real_executed=True.
That value drives execution_status in the trusted ledger, which is the
strongest claim in the evidence contract: whether a consequential action
actually ran. An executor that never proved it executed should not be
reported as having executed. Not reachable today (AgentHooksToolHost always
supplies a tracking executor), but it is a fail-open default on the exact
boundary this PR exists to close.

The run_turn fail-closed gate re-derived 'did the host mediate this turn'
by string-matching evidence_source inside a rendered evidence message. That
value is known structurally at drain time, so matching it back out couples
a security gate to a presentation detail: a later change to the evidence
shape could disable the gate with every test still green. Record the count
of drained host ledger rows instead.

Also fixes an existing test that passed a bare lambda where production
passes a tracking executor; it now mirrors the real call shape.

Verified by sabotage: restoring the fail-open default fails the new
execution-claim test, and both breaking the gate and dropping the counter
fail the new gate test.

1346 passed, 23 skipped, 840 subtests outside the viewer suite. The 29
viewer failures are the pre-existing missing viewer/node_modules gap and
reproduce identically on the unmodified PR head.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant