Skip to content

fix(sandbox): the container rule is not asked of Validate, and a make-based binding executes the tree - #418

Merged
tpouyer merged 2 commits into
mainfrom
in-lockstep/implement/410/implement-propose-34408733138-20260909T223934Z-8d0d
Sep 9, 2026
Merged

fix(sandbox): the container rule is not asked of Validate, and a make-based binding executes the tree#418
tpouyer merged 2 commits into
mainfrom
in-lockstep/implement/410/implement-propose-34408733138-20260909T223934Z-8d0d

Conversation

@github-actions

@github-actions github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Makes GATE-SANDBOX-2—the rule that a binding executing model-authored code must run in a container—apply to Validate as it does to Test and Build. The issue: CommandValidate bound to make lint executes recipes, and a model can author config files those recipes read (mypy.ini, ruff.toml, eslint.config.js), turning them into arbitrary execution on the host. The fix adds an EXECUTES_CODE capability declaration to adapters; staged_refusal checks for it before allowing materialisation; and both call sites (_checked and _validate_runner) now refuse bindings that execute without a container. RuffValidate declares READS_REPO only, so it is unaffected. Detection's bindings carry the appropriate declaration.

What changed:

  • CommandValidate declares EXECUTES_CODE alongside READS_REPO, because a repository's own make lint runs recipes and a model can author the files those recipes read.
  • RuffValidate continues to declare READS_REPO only, because ruff is a binary parser and its TOML config cannot execute arbitrary code.
  • The staged_refusal function now checks whether an adapter declares EXECUTES_CODE and skips the container check if it does not, allowing bindings like RuffValidate to proceed unaffected.
  • The strategy's _checked function calls staged_refusal(ctx, Validate) before materialising the tree, refusing any CommandValidate without a container.
  • The run_validate tool's _validate_runner function calls staged_refusal(ctx, Validate) before materialising, refusing the same way and returning an error the model sees.
  • Detection's bindings (make …, npm run …) carry EXECUTES_CODE to indicate they execute model-authored files; pure tool bindings like ruff do not.
  • Documentation in design/gates.md updated to reflect that the container rule now applies to Validate when the binding declares EXECUTES_CODE, citing fix(sandbox): the container rule is not asked of Validate, and a make-based binding executes the tree #410.
  • New test file test_validate_sandbox.py verifies the declaration, the refusal at both call sites, and that read-only bindings are unaffected.

Worth a closer look:

  • The denial of service: any repository currently binding CommandValidate(cmd, sandbox=Sandbox()) without an image will now be refused at validation time. This is the intended breaking change to close a security hole, but deployments using such bindings will need to update them to provide a container image.
  • staged_refusal now requires the container to have a resolve method; test doubles that only implement has will not be interrogated for capabilities and will not trigger the refusal. This is tolerant but worth noting for test design.
  • The diff modifies the capabilities list of an adapter class, not an instance, so all uses of CommandValidate pick up the new declaration immediately—there is no per-binding override here.

Closes #410

The ticket body is untrusted input to a model that held write tools, so review this as you would a change from a stranger who had read your repository — the controls bound where it could write, not what it thought.

Tests: ✅ 2879 passed, 15 skipped, run against the staged change before it was proposed.

Checks: clean — the repository's own checks passed over this change

in-lockstep
{
  "In-Lockstep-Run": "implement-propose-34408733138-20260909T223934Z-8d0d",
  "Ticket": "#410"
}

…-based binding executes the tree

In-Lockstep-Run: implement-propose-34408733138-20260909T223934Z-8d0d
Ticket: #410
…t now requires

Two things the run that opened this pull request could not do.

The binding. `CommandValidate` declaring `EXECUTES_CODE` is correct and makes this repository's own
`Validate` — bound with a bare `Sandbox()` — refused by `staged_refusal`, so every `/implement`
and `/fix` here would have stopped checking what a model wrote. It fails closed, which is the right
direction and the reason nobody would have noticed. It now shares the Test binding's sandbox, named
once and bound twice rather than copied: the image carries `make`, `python`, `python3` and `uv`,
measured rather than assumed, which is what `make lint typecheck` needs.

An agent could not have written that line. `.lockstep/` is tier 1, so the acceptance criterion in
#410 asking for it was one the framework forbids a model from satisfying — my error in writing the
ticket, not the run's in skipping it. `test_own_ceilings.py` now derives the rule from the
declaration instead of restating it, so a verb bound tomorrow to an adapter that executes is
covered without that test being edited.

The fail-open. `staged_refusal` reached `container.resolve` through a `getattr(..., None)` with an
early `return None`, so a container it could not inspect was silently cleared — a crash turned into
a pass on a security control, to spare a test double. The double is the thing that was wrong:
`test_validate_tool.py`'s container answered only `has`, and since #410 the rule asks what the
bound adapter declares. It now resolves to a `READS_REPO`-only adapter, which is the shape those
tests mean, so the rule correctly does not fire and they stay about the tool's worktree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

in-lockstep review — intent

succeeded · $0.0851 · 3 in / 508 out tokens

location finding
⚠️ tests/in_lockstep/test_validate_phase.py:80 Test double gains EXECUTES_CODE but has no sandbox image — the refusal it is supposed to exercise has a different code path.
ℹ️ the lens was given this change's blast radius (2814 chars)

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

in-lockstep review — performance

succeeded · $0.0744 · 3 in / 67 out tokens

No findings.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

in-lockstep review — security

succeeded · $0.0880 · 3 in / 677 out tokens

location finding
⚠️ src/in_lockstep/adapters/ai/strategy.py:898 staged_refusal refusal in _validate_runner returns a string to the model rather than raising, and the model may proceed past it.
⚠️ src/in_lockstep/adapters/worktree.py:158 capabilities_of is imported inside staged_refusal on every call; not a security issue but worth confirming the import cannot be shadowed by model-authored code on sys.path.
ℹ️ the lens was given this change's blast radius (2814 chars)

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

in-lockstep review — tests

succeeded · $0.0912 · 3 in / 912 out tokens

location finding
⚠️ tests/in_lockstep/test_validate_sandbox.py:239 The _validate_runner refusal test asserts on the result string but does not assert that validator.seen is empty when the validator reads.
⚠️ tests/in_lockstep/test_validate_sandbox.py:176 The _Ctx.do dispatcher will KeyError for any verb not in self.bound, masking failures if _checked dispatches an unexpected type.
⚠️ tests/in_lockstep/test_validate_sandbox.py:252 Detection test relies on _detect_facts and detected_bindings being importable from those paths; if those are private APIs, a refactor silently breaks the test.
ℹ️ the lens was given this change's blast radius (2814 chars)

@tpouyer tpouyer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tpouyer
tpouyer merged commit fcd5249 into main Sep 9, 2026
4 checks passed
@tpouyer
tpouyer deleted the in-lockstep/implement/410/implement-propose-34408733138-20260909T223934Z-8d0d branch September 9, 2026 23:10
tpouyer added a commit that referenced this pull request Sep 10, 2026
…not a template for it (#420)

`main` went red on run 34415733664: `run selfcheck`'s container job, `make lint typecheck exited
2`. #418's binding change is the trigger and this restores green.

What it did, in order. `make lint` shells out to `uv run`; `uv run` found `/work/.venv` — the
working tree's venv, writable because a working tree has to be, where `mounts` are `:ro` — and its
interpreter symlink dangled, the image carrying CPython 3.11.14 where the runner had built that
venv with 3.11.16 on a mutable image tag. uv's remedy for a broken venv is to delete and rebuild
it, rebuilding wants a network the sandbox correctly does not have, and the deletion took the suite
down with it in the same job: `test errored (pytest is not installed)`. The identical code was
green on the pull request half an hour earlier; the only difference in the logs is `Downloaded
newer image`.

The general shape, which #419 holds: a contained command that repairs its environment in place can
destroy state the rest of the run depends on, through the working tree rather than through a bind
mount.

And the specific one, recorded beside both bindings because it is the trap that produced this:
`PytestTest` runs `python -m pytest` and site-packages are path-independent, so mounting the host's
`.venv` works. `uv run mypy` executes a console script whose shebang carries the host's absolute
venv path, which nothing mounted makes exist inside the image. Measured under `--network=none`:
`UV_NO_SYNC` does not prevent the delete, and `UV_PROJECT_ENVIRONMENT` inside the image does but
then falls through to `PATH`, which serves `ruff` and not `mypy`.

`test_gate_sandbox_2_every_binding_that_executes_what_a_model_wrote_is_contained` goes with the
binding. It is the right test and this repository does not pass it: `staged_refusal` now refuses
this Validate for a model-staged run, so `/implement` and `/fix` here do not check what they wrote.
A test asserting something the repository does not do is worse than the gap it hides, so the gap is
stated in the module and in #419 and the test returns with the fix.

Refs #419

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(sandbox): the container rule is not asked of Validate, and a make-based binding executes the tree

1 participant