diff --git a/.github/workflows/build-deploy.yml b/.github/workflows/build-deploy.yml index ba74ca4..8e12505 100644 --- a/.github/workflows/build-deploy.yml +++ b/.github/workflows/build-deploy.yml @@ -81,6 +81,7 @@ jobs: STACK_NAME: ${{ env.STACK_NAME }} STACK_DIR: ${{ env.STACK_DIR }} IMAGE_NAME: ${{ env.IMAGE_NAME }} + TEST_IMAGE_NAME: ${{ env.TEST_IMAGE_NAME }} IMAGE_TAG: ${{ github.sha }} GIT_AUTH_TOKEN: ${{ secrets.GIT_AUTH_TOKEN }} KOMODO_API_KEY: ${{ secrets.KOMODO_API_KEY }} @@ -94,4 +95,15 @@ jobs: -H "Authorization: token ${{ secrets.FORGEJO_TOKEN }}" \ "https://repo.indexarr.net/api/v1/repos/indexarr/ops/raw/scripts/komodo-deploy.sh" \ -o komodo-deploy.sh + + # The acceptance image first, then the runtime image. + # + # Both are pinned to this commit's sha, and that is not tidiness. The + # acceptance service used to pin `:latest`, and `docker compose run` + # does not re-pull a tag it already holds -- so the Stage 2 live suite + # silently executed a STALE image and reproduced a defect that had + # already been fixed and deployed. Evidence from an image nobody can + # name is not evidence. The script only rewrites `image: $IMAGE_NAME:*` + # lines, so each call touches exactly its own service. + IMAGE_NAME="${TEST_IMAGE_NAME}" bash komodo-deploy.sh bash komodo-deploy.sh diff --git a/docs/STATUS.md b/docs/STATUS.md index 5d331f4..4fa038d 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -229,17 +229,36 @@ the currently measured delivery failure. Model-call events carry project, item and work-attempt identity; item budgets and terminal policy refusals stop at loop boundaries. Evidence is in [`evidence/2026-08-06-stage-1-role-runner.md`](evidence/2026-08-06-stage-1-role-runner.md). -- **Stage 2 is in implementation; its exit is pending.** The generic - execution-environment contract, metadata-selected Docker backend, disposable - per-item self-contained Git checkout, explicit image/network/mount configuration, - controller-environment allow-list, pre-claim readiness check and teardown path - are implemented and covered by local contract tests. The current host has - Docker CLI but no reachable daemon, so the required tests against the actual - backend — repository-wide work, undeclared sibling/host refusal, declared - mount modes, allowed network access and clean teardown — have not yet run. No - real workload run is authorised by the Stage 2 implementation evidence. See - +- **Stage 2 has run against a real daemon; its exit is still not claimed.** + The generic execution-environment contract, metadata-selected Docker backend, + disposable per-item self-contained Git checkout, explicit image/network/mount + configuration, controller-environment allow-list, pre-claim readiness check + and teardown path are implemented and covered by local contract tests. See [`evidence/2026-08-06-stage-2-execution-environment.md`](evidence/2026-08-06-stage-2-execution-environment.md). + + On 2026-08-08 the live suite ran on Node B against a dedicated Docker-in- + Docker daemon deployed beside the controller, and passed: an agent reads and + writes its own worktree, reads a declared mount, cannot reach an undeclared + sibling path, does not receive controller credentials, is denied the network + under `network=none` and allowed it under `bridge`, and its container is gone + after teardown. Image digests, the update id and the reproduction command are + in + [`evidence/2026-08-08-stage-2-live-execution-boundary.md`](evidence/2026-08-08-stage-2-live-execution-boundary.md). + + **That is a substantial part of the exit, not the whole of it.** §2.7 also + requires the security profile itself — `no-new-privileges`, dropped + capabilities, a read-only root filesystem, resource limits and the recorded + image digest — and those are asserted only in the `docker create` argv, never + verified from inside a live container. The nested daemon's own confinement is + untested. Only one sandbox image has been used (Alpine/BusyBox), and the last + defect this suite found was an assumption about exactly that. **No real + workload run is authorised.** + + The live run earned its cost. Five defects were found that local runs and CI + could not see, the sharpest being that every sandbox command failed against + BusyBox because the harness wrapped them in GNU-only `timeout --signal=TERM` + — a failure that reads as the agent failing rather than the harness, which is + the misattribution class of #216. - **Stage 3 wiring is present but its exit is not claimed.** `serve` can now construct an AIDevEnv-independent local fleet from the metadata-selected role runner and execution backend; readiness and preflight use that executor @@ -788,6 +807,7 @@ evidence and do not build on them. | One plan branch yields exactly one pull request: a correction updates it, an unchanged head touches no remote, an existing PR is adopted, a foreign push is refused, and nothing is merged | **tested** | `tests/test_plan_publication.py` — against a local bare remote and a fake pull-request client, never GitHub | | A fleet publishes that one pull request only once the plan has stopped moving, pushes no item branch, and updates the same PR for a later correction | **tested** | `tests/test_plan_integration.py::test_fleet_publishes_one_plan_pr_only_when_the_plan_is_finished` | | An installed review source gives reviews and review comments distinct immutable identities and decides disposition without a model, defaulting unmarked prose to a hold | **tested** | `tests/test_github_pr_review_source.py` — `gh` is injected; no real pull request has been polled | +| An agent reads and writes its own worktree, reads a declared mount, cannot reach an undeclared sibling path, receives no controller credentials, is denied the network under `none` and allowed it under `bridge`, and its container is removed on teardown — against a real Docker daemon | **observed** | [`evidence/2026-08-08-stage-2-live-execution-boundary.md`](evidence/2026-08-08-stage-2-live-execution-boundary.md) — one run, one host, one sandbox image, with the image digests and update id recorded. The security profile itself is argv-tested only. | | The service runs and is deployed inside AIDevEnv | **observed** | no preserved artefacts | | An earlier supervised NGMS attempt and later direct calls exercised real agents and providers | **observed** | [`evidence/2026-08-03-04-ngms-first-sustained-run-v1.md`](evidence/2026-08-03-04-ngms-first-sustained-run-v1.md) — lacks a common run ID, complete configuration, checksums and a comparable follow-up | | Four executor passes against rdpapp delivered nothing, and why each failed | **observed** | [`evidence/2026-08-05-06-rdpapp-m2-status.md`](evidence/2026-08-05-06-rdpapp-m2-status.md); the pass 3–4 attribution is hindsight and has not been confirmed by re-running against the fix | diff --git a/docs/evidence/2026-08-08-stage-2-live-execution-boundary.md b/docs/evidence/2026-08-08-stage-2-live-execution-boundary.md new file mode 100644 index 0000000..1ab9d69 --- /dev/null +++ b/docs/evidence/2026-08-08-stage-2-live-execution-boundary.md @@ -0,0 +1,114 @@ +# Stage 2 live evidence — the execution boundary, against a real daemon + +**Date:** 2026-08-08 +**Where:** Node B (`winrarhost`), Komodo stack `personal-agent-harness`, against +a dedicated Docker-in-Docker daemon deployed beside the controller. +**Scope:** the two tests in `tests/test_execution_environment_live.py`, run in +the deployed stack. No model was called, no workload was run, and no remote +repository was contacted. + +This is the first time any part of this repository has executed against a real +Docker daemon. Everything before it was a mocked subprocess. + +**It is not the Stage 2 exit.** §"What this does not cover" below says what is +still missing, and that list is not short. + +## What ran, and how to run it again + +```bash +# Komodo: POST /execute/RunStackService, then poll /read/GetUpdate. +{"stack": "personal-agent-harness", "service": "agent-harness-tests"} +``` + +```console +success: True +.. [100%] +``` + +Update `6a76fcfbe1e8f310d4eae2c2`. Two tests, two passes, zero skips — the +suite skips itself unless `HARNESS_STAGE2_IMAGE` names a pullable image and a +daemon answers, so a pass cannot be a silent no-op. + +### The exact artefacts + +Recorded because "it passed" is not reproducible and a tag moves: + +| artefact | identity | +|---|---| +| controller image | `agent-harness:75e76ed5ad5e70d94b68e55c4582cb9efb2f1120` → `sha256:65a4f2073df81572f4e8c4100d89af425dd4e47450f883918471d94ba8d7b527` | +| acceptance image | `agent-harness-test:75e76ed5ad5e70d94b68e55c4582cb9efb2f1120` → `sha256:9808728bd93bf33417ed3151840f3744198c9c3fca22bfb64ab34b640eb652fa` | +| item sandbox image | `alpine:3.21` → `sha256:2607caa9805847fac4de202017bb1b830deb09f4c07dc9964a0157abbc604577` | +| nested daemon | `docker:28-dind` → `sha256:6a68f64cf32d98b09a11c208de78f59f17c0a6fff33c13f11acac853d6aad5ae` | + +## What the two tests prove + +Each row is an assertion that fails if the property stops holding, executed +against the daemon named above. + +| criterion | result | how | +|---|---|---| +| An agent can **read** its own worktree | pass | `cat /workspace/inside.txt` returned `inside`, from a file the controller wrote outside the container | +| An agent can **write** its own worktree | pass | `printf changed > /workspace/result.txt`, then read back by the controller | +| A **declared** dependency mount is readable | pass | `test -f /opt/dependency/readme.txt` | +| An **undeclared** host/sibling path is unreachable | pass | `test ! -e "$HOST_SIBLING"`, where the path exists on the host and holds content | +| Controller **credentials** do not enter the agent environment | pass | `HARNESS_STAGE2_CONTROLLER_SECRET` is set in the controller process and absent in the container | +| An explicitly passed variable **does** arrive | pass | `DECLARED=yes`, so the credential result above is not merely an empty environment | +| `network=none` **denies** outbound access | pass | `wget https://example.com` → `bad address` (DNS and egress both fail) | +| `network=bridge` **allows** outbound access | pass | the same fetch succeeds — P6's "internet is available" is real, not aspirational | +| The container is **gone** after teardown | pass | `docker inspect ` fails after `close()` | + +## What this does not cover + +Named explicitly, because a green run invites over-reading. + +- **The security profile is asserted only in argv.** `--read-only`, + `--cap-drop ALL`, `--security-opt`, the resource limits and the resolved + image digest are covered by `tests/test_execution_environment.py` at the + `docker create` command line, and are **not** verified from inside a live + container. Nothing here proves the kernel applied them. +- **The nested daemon's own confinement is untested.** It is `privileged`, on + an internal network with nothing published, and no test asserts either + property. +- **One image, one shape.** Alpine/BusyBox only. A workload toolchain image — + Rust, for the first real workload — has never been used, and the last defect + this suite found was precisely an assumption about which userland is present. +- **No workload ran.** No model call, no item, no plan. Stage 2 says a real + workload is not authorised until its exit is met, and it is not met. +- **The controller has no model routes**, so the deployed service cannot claim + work at all. That is deliberate for a first deployment, and it means nothing + here exercises the fleet. + +## What the live run found that nothing else did + +Recorded because it is the argument for having deployed at all. Each of these +was invisible to local runs and to CI, and each is fixed: + +| defect | why only a real daemon found it | +|---|---| +| `timeout --signal=TERM 30s` is GNU-only | against BusyBox **every** sandbox command returned 1 with `timeout: unrecognized option`, which reads as the agent failing rather than the harness's wrapper being unportable — the misattribution class of #216 | +| two divergent copies of the daemon check | `DockerItemEnvironment.check()` kept the `--format` template defect after the factory's copy was fixed; an unreachable daemon reported a Go reflect error instead of naming the daemon | +| `serve` exited 2 with a fleet and no routes | the API and GUI never came up to say why, and the supervisor restarted it every 60 seconds | +| mounts owned by root, controller uid 1000 | a mount replaces the image's directory, so the image's `chown` is not the runtime truth; the controller crash-looped on `unable to open database file` | +| the acceptance pinned `:latest` | `docker compose run` does not re-pull a tag it already holds, so the suite silently re-ran a **stale** image and reproduced an already-fixed defect. `agent-harness-test:latest` was `sha256:d73e54bd…` while the current build was `sha256:9808728b…` | + +The last one is worth keeping in mind when reading any evidence produced this +way: an acceptance run that cannot name the image it ran is not evidence. Both +images are now pinned to a commit sha and the build bumps both. + +## The uid that made the difference + +The confinement test first failed with + +``` +/bin/sh: can't create /workspace/result.txt: Permission denied +``` + +That is not a boundary failure. The acceptance container ran as root, so its +fixture worktrees were root-owned, and the item container — uid 1000, which is +`EnvironmentSpec.user`'s default and the uid the deployed controller runs as — +could not write to its own checkout. + +The fix was to make the acceptance mirror the deployment (`user: "1000:1000"`), +not to relax the assertion. An agent being able to write its worktree is one of +the things Stage 2 exists to prove, and weakening it to get a green run would +have produced exactly the kind of evidence this repository refuses to accept.