From 55b629082560892e9768c6ff7c3f5263f77744e6 Mon Sep 17 00:00:00 2001 From: sprooty Date: Sat, 8 Aug 2026 09:57:45 +0000 Subject: [PATCH] Stage 2 ran against a real daemon, and both live tests pass MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first execution of any part of this repository against a real Docker daemon. On Node B, in the deployed stack, against a dedicated Docker-in-Docker daemon beside the controller: success: True .. [100%] An agent reads and writes its own worktree, reads a declared mount, cannot reach an undeclared sibling path that exists on the host, receives none of the controller's credentials while still receiving explicitly passed variables, is denied the network under `network=none` and allowed it under `bridge`, and its container is gone after teardown. The evidence package records the image digests, the Komodo update id and the command to run it again, because "it passed" is not reproducible and a tag moves. **The Stage 2 exit is not claimed.** The security profile itself — no-new-privileges, dropped capabilities, read-only rootfs, resource limits — is asserted only in the `docker create` argv and never verified from inside a live container. The nested daemon's own confinement is untested. One sandbox image has been used, and the last defect this suite found was an assumption about exactly that. No workload ran. Also here: the build now pins and bumps the acceptance image by sha. It pinned `:latest`, and `docker compose run` does not re-pull a tag it already holds -- so the live suite silently re-ran a stale image and reproduced a defect that had already been fixed and deployed. An acceptance run that cannot name the image it ran is not evidence. --- .github/workflows/build-deploy.yml | 12 ++ docs/STATUS.md | 40 ++++-- ...6-08-08-stage-2-live-execution-boundary.md | 114 ++++++++++++++++++ 3 files changed, 156 insertions(+), 10 deletions(-) create mode 100644 docs/evidence/2026-08-08-stage-2-live-execution-boundary.md diff --git a/.github/workflows/build-deploy.yml b/.github/workflows/build-deploy.yml index ba74ca4..8e12505 100644 --- a/.github/workflows/build-deploy.yml +++ b/.github/workflows/build-deploy.yml @@ -81,6 +81,7 @@ jobs: STACK_NAME: ${{ env.STACK_NAME }} STACK_DIR: ${{ env.STACK_DIR }} IMAGE_NAME: ${{ env.IMAGE_NAME }} + TEST_IMAGE_NAME: ${{ env.TEST_IMAGE_NAME }} IMAGE_TAG: ${{ github.sha }} GIT_AUTH_TOKEN: ${{ secrets.GIT_AUTH_TOKEN }} KOMODO_API_KEY: ${{ secrets.KOMODO_API_KEY }} @@ -94,4 +95,15 @@ jobs: -H "Authorization: token ${{ secrets.FORGEJO_TOKEN }}" \ "https://repo.indexarr.net/api/v1/repos/indexarr/ops/raw/scripts/komodo-deploy.sh" \ -o komodo-deploy.sh + + # The acceptance image first, then the runtime image. + # + # Both are pinned to this commit's sha, and that is not tidiness. The + # acceptance service used to pin `:latest`, and `docker compose run` + # does not re-pull a tag it already holds -- so the Stage 2 live suite + # silently executed a STALE image and reproduced a defect that had + # already been fixed and deployed. Evidence from an image nobody can + # name is not evidence. The script only rewrites `image: $IMAGE_NAME:*` + # lines, so each call touches exactly its own service. + IMAGE_NAME="${TEST_IMAGE_NAME}" bash komodo-deploy.sh bash komodo-deploy.sh diff --git a/docs/STATUS.md b/docs/STATUS.md index 5d331f4..4fa038d 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -229,17 +229,36 @@ the currently measured delivery failure. Model-call events carry project, item and work-attempt identity; item budgets and terminal policy refusals stop at loop boundaries. Evidence is in [`evidence/2026-08-06-stage-1-role-runner.md`](evidence/2026-08-06-stage-1-role-runner.md). -- **Stage 2 is in implementation; its exit is pending.** The generic - execution-environment contract, metadata-selected Docker backend, disposable - per-item self-contained Git checkout, explicit image/network/mount configuration, - controller-environment allow-list, pre-claim readiness check and teardown path - are implemented and covered by local contract tests. The current host has - Docker CLI but no reachable daemon, so the required tests against the actual - backend — repository-wide work, undeclared sibling/host refusal, declared - mount modes, allowed network access and clean teardown — have not yet run. No - real workload run is authorised by the Stage 2 implementation evidence. See - +- **Stage 2 has run against a real daemon; its exit is still not claimed.** + The generic execution-environment contract, metadata-selected Docker backend, + disposable per-item self-contained Git checkout, explicit image/network/mount + configuration, controller-environment allow-list, pre-claim readiness check + and teardown path are implemented and covered by local contract tests. See [`evidence/2026-08-06-stage-2-execution-environment.md`](evidence/2026-08-06-stage-2-execution-environment.md). + + On 2026-08-08 the live suite ran on Node B against a dedicated Docker-in- + Docker daemon deployed beside the controller, and passed: an agent reads and + writes its own worktree, reads a declared mount, cannot reach an undeclared + sibling path, does not receive controller credentials, is denied the network + under `network=none` and allowed it under `bridge`, and its container is gone + after teardown. Image digests, the update id and the reproduction command are + in + [`evidence/2026-08-08-stage-2-live-execution-boundary.md`](evidence/2026-08-08-stage-2-live-execution-boundary.md). + + **That is a substantial part of the exit, not the whole of it.** §2.7 also + requires the security profile itself — `no-new-privileges`, dropped + capabilities, a read-only root filesystem, resource limits and the recorded + image digest — and those are asserted only in the `docker create` argv, never + verified from inside a live container. The nested daemon's own confinement is + untested. Only one sandbox image has been used (Alpine/BusyBox), and the last + defect this suite found was an assumption about exactly that. **No real + workload run is authorised.** + + The live run earned its cost. Five defects were found that local runs and CI + could not see, the sharpest being that every sandbox command failed against + BusyBox because the harness wrapped them in GNU-only `timeout --signal=TERM` + — a failure that reads as the agent failing rather than the harness, which is + the misattribution class of #216. - **Stage 3 wiring is present but its exit is not claimed.** `serve` can now construct an AIDevEnv-independent local fleet from the metadata-selected role runner and execution backend; readiness and preflight use that executor @@ -788,6 +807,7 @@ evidence and do not build on them. | One plan branch yields exactly one pull request: a correction updates it, an unchanged head touches no remote, an existing PR is adopted, a foreign push is refused, and nothing is merged | **tested** | `tests/test_plan_publication.py` — against a local bare remote and a fake pull-request client, never GitHub | | A fleet publishes that one pull request only once the plan has stopped moving, pushes no item branch, and updates the same PR for a later correction | **tested** | `tests/test_plan_integration.py::test_fleet_publishes_one_plan_pr_only_when_the_plan_is_finished` | | An installed review source gives reviews and review comments distinct immutable identities and decides disposition without a model, defaulting unmarked prose to a hold | **tested** | `tests/test_github_pr_review_source.py` — `gh` is injected; no real pull request has been polled | +| An agent reads and writes its own worktree, reads a declared mount, cannot reach an undeclared sibling path, receives no controller credentials, is denied the network under `none` and allowed it under `bridge`, and its container is removed on teardown — against a real Docker daemon | **observed** | [`evidence/2026-08-08-stage-2-live-execution-boundary.md`](evidence/2026-08-08-stage-2-live-execution-boundary.md) — one run, one host, one sandbox image, with the image digests and update id recorded. The security profile itself is argv-tested only. | | The service runs and is deployed inside AIDevEnv | **observed** | no preserved artefacts | | An earlier supervised NGMS attempt and later direct calls exercised real agents and providers | **observed** | [`evidence/2026-08-03-04-ngms-first-sustained-run-v1.md`](evidence/2026-08-03-04-ngms-first-sustained-run-v1.md) — lacks a common run ID, complete configuration, checksums and a comparable follow-up | | Four executor passes against rdpapp delivered nothing, and why each failed | **observed** | [`evidence/2026-08-05-06-rdpapp-m2-status.md`](evidence/2026-08-05-06-rdpapp-m2-status.md); the pass 3–4 attribution is hindsight and has not been confirmed by re-running against the fix | diff --git a/docs/evidence/2026-08-08-stage-2-live-execution-boundary.md b/docs/evidence/2026-08-08-stage-2-live-execution-boundary.md new file mode 100644 index 0000000..1ab9d69 --- /dev/null +++ b/docs/evidence/2026-08-08-stage-2-live-execution-boundary.md @@ -0,0 +1,114 @@ +# Stage 2 live evidence — the execution boundary, against a real daemon + +**Date:** 2026-08-08 +**Where:** Node B (`winrarhost`), Komodo stack `personal-agent-harness`, against +a dedicated Docker-in-Docker daemon deployed beside the controller. +**Scope:** the two tests in `tests/test_execution_environment_live.py`, run in +the deployed stack. No model was called, no workload was run, and no remote +repository was contacted. + +This is the first time any part of this repository has executed against a real +Docker daemon. Everything before it was a mocked subprocess. + +**It is not the Stage 2 exit.** §"What this does not cover" below says what is +still missing, and that list is not short. + +## What ran, and how to run it again + +```bash +# Komodo: POST /execute/RunStackService, then poll /read/GetUpdate. +{"stack": "personal-agent-harness", "service": "agent-harness-tests"} +``` + +```console +success: True +.. [100%] +``` + +Update `6a76fcfbe1e8f310d4eae2c2`. Two tests, two passes, zero skips — the +suite skips itself unless `HARNESS_STAGE2_IMAGE` names a pullable image and a +daemon answers, so a pass cannot be a silent no-op. + +### The exact artefacts + +Recorded because "it passed" is not reproducible and a tag moves: + +| artefact | identity | +|---|---| +| controller image | `agent-harness:75e76ed5ad5e70d94b68e55c4582cb9efb2f1120` → `sha256:65a4f2073df81572f4e8c4100d89af425dd4e47450f883918471d94ba8d7b527` | +| acceptance image | `agent-harness-test:75e76ed5ad5e70d94b68e55c4582cb9efb2f1120` → `sha256:9808728bd93bf33417ed3151840f3744198c9c3fca22bfb64ab34b640eb652fa` | +| item sandbox image | `alpine:3.21` → `sha256:2607caa9805847fac4de202017bb1b830deb09f4c07dc9964a0157abbc604577` | +| nested daemon | `docker:28-dind` → `sha256:6a68f64cf32d98b09a11c208de78f59f17c0a6fff33c13f11acac853d6aad5ae` | + +## What the two tests prove + +Each row is an assertion that fails if the property stops holding, executed +against the daemon named above. + +| criterion | result | how | +|---|---|---| +| An agent can **read** its own worktree | pass | `cat /workspace/inside.txt` returned `inside`, from a file the controller wrote outside the container | +| An agent can **write** its own worktree | pass | `printf changed > /workspace/result.txt`, then read back by the controller | +| A **declared** dependency mount is readable | pass | `test -f /opt/dependency/readme.txt` | +| An **undeclared** host/sibling path is unreachable | pass | `test ! -e "$HOST_SIBLING"`, where the path exists on the host and holds content | +| Controller **credentials** do not enter the agent environment | pass | `HARNESS_STAGE2_CONTROLLER_SECRET` is set in the controller process and absent in the container | +| An explicitly passed variable **does** arrive | pass | `DECLARED=yes`, so the credential result above is not merely an empty environment | +| `network=none` **denies** outbound access | pass | `wget https://example.com` → `bad address` (DNS and egress both fail) | +| `network=bridge` **allows** outbound access | pass | the same fetch succeeds — P6's "internet is available" is real, not aspirational | +| The container is **gone** after teardown | pass | `docker inspect ` fails after `close()` | + +## What this does not cover + +Named explicitly, because a green run invites over-reading. + +- **The security profile is asserted only in argv.** `--read-only`, + `--cap-drop ALL`, `--security-opt`, the resource limits and the resolved + image digest are covered by `tests/test_execution_environment.py` at the + `docker create` command line, and are **not** verified from inside a live + container. Nothing here proves the kernel applied them. +- **The nested daemon's own confinement is untested.** It is `privileged`, on + an internal network with nothing published, and no test asserts either + property. +- **One image, one shape.** Alpine/BusyBox only. A workload toolchain image — + Rust, for the first real workload — has never been used, and the last defect + this suite found was precisely an assumption about which userland is present. +- **No workload ran.** No model call, no item, no plan. Stage 2 says a real + workload is not authorised until its exit is met, and it is not met. +- **The controller has no model routes**, so the deployed service cannot claim + work at all. That is deliberate for a first deployment, and it means nothing + here exercises the fleet. + +## What the live run found that nothing else did + +Recorded because it is the argument for having deployed at all. Each of these +was invisible to local runs and to CI, and each is fixed: + +| defect | why only a real daemon found it | +|---|---| +| `timeout --signal=TERM 30s` is GNU-only | against BusyBox **every** sandbox command returned 1 with `timeout: unrecognized option`, which reads as the agent failing rather than the harness's wrapper being unportable — the misattribution class of #216 | +| two divergent copies of the daemon check | `DockerItemEnvironment.check()` kept the `--format` template defect after the factory's copy was fixed; an unreachable daemon reported a Go reflect error instead of naming the daemon | +| `serve` exited 2 with a fleet and no routes | the API and GUI never came up to say why, and the supervisor restarted it every 60 seconds | +| mounts owned by root, controller uid 1000 | a mount replaces the image's directory, so the image's `chown` is not the runtime truth; the controller crash-looped on `unable to open database file` | +| the acceptance pinned `:latest` | `docker compose run` does not re-pull a tag it already holds, so the suite silently re-ran a **stale** image and reproduced an already-fixed defect. `agent-harness-test:latest` was `sha256:d73e54bd…` while the current build was `sha256:9808728b…` | + +The last one is worth keeping in mind when reading any evidence produced this +way: an acceptance run that cannot name the image it ran is not evidence. Both +images are now pinned to a commit sha and the build bumps both. + +## The uid that made the difference + +The confinement test first failed with + +``` +/bin/sh: can't create /workspace/result.txt: Permission denied +``` + +That is not a boundary failure. The acceptance container ran as root, so its +fixture worktrees were root-owned, and the item container — uid 1000, which is +`EnvironmentSpec.user`'s default and the uid the deployed controller runs as — +could not write to its own checkout. + +The fix was to make the acceptance mirror the deployment (`user: "1000:1000"`), +not to relax the assertion. An agent being able to write its worktree is one of +the things Stage 2 exists to prove, and weakening it to get a green run would +have produced exactly the kind of evidence this repository refuses to accept.