feat: add microVM execution environments - #580
Conversation
39f6a33 to
0a8f9b2
Compare
jhrozek
left a comment
jhrozek
left a comment
There was a problem hiding this comment.
Panel review — the consequential subset
Adversarial review of this PR across three orthogonal axes — Spec (does it implement what was asked), Standards (does it follow this repo's documented conventions), and Domain (what specialist reviewers say) — run as three panels over a 202-file diff, with 19 agents total.
Note on scope of this post: the full review produced 84 inline comments. GitHub's secondary rate limiter refuses a review that creates that much content at once, so this posts the 23 consequential ones: every ship-blocker and every hard standards violation. Four further findings are folded into the comment they pair with. The remaining ~57 — mechanical fixes, judgement calls, polish, and the reuse findings (
net: -1,100lines in the module,-162/-194in host wiring) — are summarised below and available in full on request.
Findings are not merged across axes: Spec, Standards and Domain are orthogonal by design, so each comment is tagged with the panel and axis it came from. cross-confirmed marks a finding two or more independent reviewers reached separately — highest confidence. Where two reviewers disagreed, both views are preserved rather than resolved.
The seven I would fix before merge
internal/app/build.go:2254—prepareSessionDiscoveryre-rootscfg.Workspacebefore the project-trust gate is evaluated, so a session-selected root inherits a trust decision made for a different repo. Because the re-root happens first, evenprojectIngestionAdmittedForRoot— the guard written specifically to prevent this — returns true for an arbitrary root. This fires on anyCreateSessionwith a non-launch-root workspace, so it is a live regression on the existing served path, not a microVM-only issue.- Three daemon concurrency defects, none with a test —
environment/microvm/runtime.go:350(unlocked struct fields behind a locked map lookup),registry.go:94(a flock giving no intra-process exclusion on the durable registry),control/multiplex.go:255(an uncancellable write that can pin a Bash tool call forever). .github/workflows/release.yml:211— the release is created non-draft before any artifact exists, on afail-fast: falsematrix where two of three cells are guaranteed to fail (cross-archdocker runwith no QEMU,package-microvm-release.sh:81). The firstv*tag after this merges produces a public, partial, half-signed release — the exact outcome this workflow's own comments say it is designed to prevent.internal/adapter/server/service.go:1897—DeleteSessionremoves the session record without destroying the VM, and the verb that could destroy it has no wire surface, so it becomes permanently unreachable for that id. The docs describe the deletion sequence as working behaviour.internal/adapter/server/environment_profile.go:59— the ADR-0224 §5 egress disclosure crosses an independently-versioned module boundary as an English sentence, re-derived by prefix/suffix parsing, with a mismatch failing session creation. It is already inconsistent in-tree (client_profile_test.go:36,61asserts strings this Service rejects), and the parse verifies grammar rather than policy..github/workflows/microvm-e2e.yml:38—id-token: writeon apull_requestjob that runs PR-authored code and never signs keylessly..github/scripts/install-microvm-release.sh:45— the installer verifies no signature; the digest it checks comes from the same unsigned manifest, which is itself absent fromSHA256SUMS. Two docs claim it verifies.
Cheapest high-value fixes
Four comments below carry one-click suggestion blocks: CombinedOutput() → Output() in gitexec.run() (a git warning currently corrupts a captured tar), both ADR 0108 → 0224 corrections, and the ADR status flip to Accepted. Also cheap: delete three id-token: write lines, and the two-line VersionValid guard at client.go:448.
Not posted inline — the remaining ~57
Spec (Panel 3): docs/usage/http-sse-api.md:125 says the microVM backend is unwired when build.go:1574-1624 wires it · docs/usage/microvm-environments.md:207 documents a deletion sequence with no operator surface · AC8.1's three-platform live matrix is one platform in practice (microvm-e2e.yml:123 — the arm64 and macOS live cells are workflow_dispatch-gated and default to false) · the release builds and signs a mecatl-owned execution image the ADR does not authorise · go-microvm version skew (ADR says v0.0.39, go.mod says v0.0.40, and internal/apicheck/microvm_module_contract_test.go:19 pins the string from a root-module test that the next dependabot bump will break).
Standards: no depguard rule for environment/microvm in .golangci.yml, so the allowlist ADR 0093 says "travels with the modules" does not exist for this one · two missing ADR-0027 List-1 resource rows (Service.sessionEnvironmentInfo, microVMClients.byEndpoint — the latter never closed) · Config.EnvironmentSessionResolver is undocumented and contradicts ADR 0224 §1 · prepareSessionDiscovery is an unsanctioned per-session catalog delta · AGENTS.md not updated for the new environment/ module tier.
Domain, mechanical: dead + lossy LifecycleWorkspace/LifecycleExec proxy arms (~117 lines, and the copy collapses three distinct error codes into "internal") · proxyWorkspace replace mutates before validating · QuotaKind is a parallel vocabulary for admission.Resource with 9 of 11 values never emitted · Stat returns success on a malformed response · no deadline on teardown paths (CloseSession uses bare context.Background()) · Build mutates the caller's maps · unbounded daemon error text reaching API clients · five hand-written cleanup marks · reply-frame echo copied at 7 sites · configgen documents five config keys the strict decoder deliberately rejects.
Domain, judgement calls: egress allowlist is hostname-only with no post-resolution IP denylist (SSRF; flagged because it is the claimed control failing open, not the documented deferral) · two live IdentityAllocators, one hardcoding Generation: 1 · the neutral server layer now speaks microVM vocabulary, and those names have reached the proto · the host-side wire Workspace is the one ADR-0208 implementation with no conformance coverage · Taskfile.yml:232 selects live coverage by a hardcoded 19-name regex, and go test -run '<no match>' exits 0 · the installer's archive preflight has no negative test on any of its ten controls · a third frame-codec copy that re-types the wire shapes as map[string]any, hiding renames.
Checked and deliberately not flagged
Recorded so it is not re-litigated: the environment/ module tier is justified (authn/oidc is the precedent) · the "three-profile fan-out" does not exist — tool profile and placement alias are orthogonal axes pinned by a test — and should not be unified · the mirrored wire structs in internal/adapter/microvm are essential module-graph isolation · guest errorResponse vs host remoteError are deliberate inverses across a process boundary · the +37-line append to frozen ADR-0027 is legitimate (AGENTS.md names that file explicitly) · engine/adapter/fsconformance's additions are the right shape and placement · the 1,739-line e2e suite is real fault injection, not mock theatre · forker.KindRouter is a real abstraction introduced when its second implementation arrived · ~8,000 lines of per-scenario test setup should stay explicit · no host credential crosses into the guest, and the codec framing, artifact supply chain, peer-credential fail-closed behaviour, and operator-tier config gating were all verified sound.
Each axis is orthogonal — verify each independently before shipping.
|
|
||
| go 1.26.6 | ||
|
|
||
| replace github.com/stacklok/mecatl/engine => ../../engine |
There was a problem hiding this comment.
[Panel 1 · Standards · HARD VIOLATION] Committed replace directive in a published module.
ADR 0093 (Decision): "replace directives live in go.work (dev) and the root go.mod (the monorepo consumer), NEVER in a published provider go.mod — a committed replace breaks downstream go get."
Every sibling opt-in module obeys this: authn/oidc/go.mod and all four provider/*/go.mod require engine at a real version with no replace — the Taskfile even notes "Its go.mod deliberately has no replace directive". This PR already adds use ./environment/microvm to go.work, so the replace is redundant in dev and harmful on publish.
Related (not inline-commentable — .golangci.yml is unchanged): there is no depguard rule for environment/microvm. grep 'environment/microvm' .golangci.yml returns zero hits, while the config carries file-scoped strict rules for **/authn/oidc/**.go and each **/provider/*/**.go. ADR 0093 claims "Each provider shares the root .golangci.yml depguard allowlist … so the layering rule travels with the modules" — with no matching rule, this module has no allowlist at all.
There was a problem hiding this comment.
The committed replace and the absent MicroVM-specific depguard allowlist are both still present at this head. The nested module is currently repository-scoped rather than published, so I cannot substantiate the downstream-release consequence, but the standalone-boundary and layering coverage points stand. Leaving this thread open for the module contract decision and matching lint rule.
mecatl/environment/microvm/go.mod
Lines 1 to 15 in 20fa604
Lines 69 to 80 in 20fa604
| if err != nil { | ||
| return err | ||
| } | ||
| if generation.services != nil { |
There was a problem hiding this comment.
[Panel 1 · Domain · HIGH] runtimeGeneration.services is raced.
runtime.go:310-312 writes it under r.mu; Services() (:322), Reattach (:334) and Destroy (:366) read it — and this Detach write clears it — without the lock. composition.go:130 spawns a goroutine per connection, so a LifecycleWorkspace/LifecycleExec proxy can read services while a concurrent LifecycleDetach nils it: nil deref, or use of a closed control.Client. generation() (:386-397) locks only the map lookup; every field access after it is unguarded.
Fix: lock the struct, or make runtimeGeneration immutable after negotiate.
Source: go-architect
There was a problem hiding this comment.
The raced runtimeGeneration.services path is no longer the active production composition: the current runtime is repository-scoped and keeps registrations behind its own control mutex rather than exposing the former per-VM services field. That removes this specific race from the shipped path, although it would not justify retaining the old implementation unreviewed. The repository runtime starts from NewRepositoryRuntime; no legacy runtime.go remains at this head.
mecatl/environment/microvm/repository_runtime.go
Lines 50 to 112 in 20fa604
| return result, err | ||
| } | ||
|
|
||
| func (r *FileRegistry) withLock(ctx context.Context, fn func(*registryDocument) error) error { |
There was a problem hiding this comment.
[Panel 1 · Domain · HIGH] FileRegistry's flock gives no intra-process exclusion — lost updates on the durable registry.
All callers share one *flock.Flock. gofrs/flock@v0.13.0/flock_unix.go:141-147 returns (true, nil) immediately when that instance is already locked, so two goroutines in the same daemon both "acquire", read the same document, and the first defer Unlock() (:108) drops the file lock while the second is still mid-transaction. The type's own doc comment (:17-19) claiming one authoritative set is also wrong for that window. validRecordTransition catches state regressions but not two concurrent create appends.
One sync.Mutex on FileRegistry fixes it.
No test covers this: the concurrent case (lifecycle_reconcile_test.go:185-189) uses an in-memory registry, and the two-FileRegistry case (:197-225) is sequential.
Source: go-architect
There was a problem hiding this comment.
The same-instance flock concern was valid for the original FileRegistry, but that registry is no longer part of the repository-scoped production path. Logical environments now go through RepositoryLogicalManager and the singleton repository registry, so there is no current FileRegistry mutation path to fix here. This does not claim an in-process flock would have been sufficient in the removed design.
mecatl/environment/microvm/repository_logical.go
Lines 189 to 208 in 20fa604
| } | ||
| } | ||
|
|
||
| func (c *Client) write(frame multiplexFrame) error { |
There was a problem hiding this comment.
[Panel 1 · Domain · HIGH] The multiplex client cannot be cancelled if the guest stops reading — this can pin a Bash tool call permanently.
This write has no deadline and no ctx, and Stream holds requestMu across it (:178-194, needed only to satisfy the server's monotonic-ID check at :405). If the guest stops draining, every caller — including the cancel write at :208 — blocks on writeMu indefinitely and ctx is inert.
Worse, :203-252: after sending cancel it sets ctxDone = nil and then waits for an end/error frame forever, so a hung guest handler pins the caller goroutine — and via guestexec/exec.go:174, a Bash tool call — permanently.
Needs a write deadline plus a bounded post-cancel wait.
Sources: go-architect; secure-code-reviewer (CWE-400, on the read-loop half)
There was a problem hiding this comment.
Confirmed on the current multiplex implementation. Stream still performs synchronous writes while holding the request ordering lock, and after a successful cancel write it disables ctxDone and waits for a terminal frame. A guest that stops reading or never terminates can therefore strand the transport work even though the outer host RPC now closes its UDS connection on cancellation. Leaving this open.
mecatl/environment/microvm/control/multiplex.go
Lines 180 to 265 in 20fa604
| return microvm.CreateRequest{Owner: request.Owner, SessionID: request.SessionID, Profile: request.Profile, | ||
| Worktree: worktree.Request{Source: request.SourceCheckout, WorktreePath: names.WorktreePath, MetadataPath: names.MetadataPath, Branch: names.Branch}, | ||
| ArtifactRequests: artifactRequests, Resources: usage, | ||
| ProfileStatus: microvm.EnforcedProfileStatus{Profile: request.Profile, GuestEgress: egress.Status(), HostEgress: "not constrained: LLM providers, WebFetch, WebSearch, MCP, hooks, OCI pulls, telemetry"}}, nil |
There was a problem hiding this comment.
[Panel 1+2 · Domain · HIGH · cross-confirmed] The egress disclosure is a prose sentence used as a load-bearing cross-module protocol constant.
hostServiceEgressStatus (internal/adapter/server/environment_profile.go:59) is a prose literal retyped byte-for-byte here, and network.go:157,159 mints "deny-all (IPv6 disabled)" / "allowlist (%d destinations; IPv6 disabled)" which the host prefix/suffix-parses at environment_profile.go:61-72. A mismatch on either string fails session creation with ErrFailedPrecondition (:141).
The fact is structured — policy mode, destination count, IPv6 state — but crosses an independently-versioned module boundary as an English sentence. Two things a module boundary is supposed to permit (improving a human-facing message; host/daemon version skew) become total loss of the microVM capability.
It is already broken in-tree: internal/adapter/microvm/client_profile_test.go:36,61 asserts a successful Provision with HostEgress: "host services not constrained" and GuestEgress: "daemon enforced deny-all (IPv6 disabled)" — both of which the Service rejects. Nothing in CI ties the two modules together (the daemon's own test asserts only HostEgress != ""; the one default-run cross-layer test uses a hand-rolled fake with the literal hardcoded; the real-daemon suite is behind //go:build microvm_e2e).
It also buys no security: validGuestProcessEgressStatus accepts allowlist (1 destinations; IPv6 disabled) regardless of what was actually configured — the host verifies grammar, not policy.
Recommended: one versioned struct {mode, destinations, ipv6_enabled}; validate the invariant (mode=="allowlist" ⇒ destinations>0) and render prose at the presentation layer. The host-scope sentence is a host-side constant — asking the daemon to echo it back so the host can compare it to itself is a round-trip that can only ever fail.
Sources: software-architect + go-architect (both HIGH)
There was a problem hiding this comment.
The load-bearing prose parser has been removed from the host admission path. The placement client now requires only nonempty daemon-provided disclosure fields while validating the exact binding and fixed guest root; it does not parse or compare egress wording. The daemon still renders a human-readable status, which is presentation metadata rather than a cross-module gate.
mecatl/internal/adapter/microvm/client.go
Lines 229 to 246 in 20fa604
mecatl/environment/microvm/cmd/mecatl-microvmd/main.go
Lines 223 to 234 in 20fa604
| timeout-minutes: 45 | ||
| permissions: | ||
| contents: read | ||
| id-token: write |
There was a problem hiding this comment.
[Panel 3 · Domain · HIGH · cross-confirmed] id-token: write on a pull_request-triggered job that executes PR-authored code and never signs keylessly.
Same unused grant at :128 and :183. The live suite runs cosign in keyed mode with an ephemeral local key — environment/microvm/e2e/prepare.sh:152 does COSIGN_PASSWORD= cosign generate-key-pair — and sign-microvm-release-evidence.sh takes the --key branch, so no step ever reads ACTIONS_ID_TOKEN_REQUEST_URL. (docs/usage/microvm-environments.md claims "live CI cells continue to exercise keyless OIDC"; they do not — this grant looks like a leftover from that intent.)
Impact: PR-authored code (this job runs ./environment/microvm/e2e/prepare.sh and task e2e:microvm verbatim) can mint a Fulcio-backed OIDC token with subject repo:stacklok/mecatl:pull_request, then (a) cosign sign-blob arbitrary bytes with a certificate attributable to this repository in the public Rekor log, and (b) exchange that token against any cloud/vault trust relationship whose subject condition is repo-scoped rather than workflow-and-ref-scoped — the single most common OIDC federation misconfiguration.
Fork PRs are capped read-only by GitHub, so the reachable case is a same-repo head branch — including one pushed by a coding agent running in CI.
Not Critical because the published artifact chain is not forgeable this way: microvmd admission pins an exact --certificate-identity and --certificate-oidc-issuer (environment/microvm/artifact_sigstore.go:86-87), and the documented operator regexp is anchored to release.yml@refs/tags/v*, so certs from this workflow do not satisfy it. That containment is what keeps this HIGH rather than Critical.
Fix: delete the three id-token: write lines (the permissions: blocks then reduce to the inherited contents: read). If a keyless cell is wanted later, put it in a separate job gated if: github.event_name != 'pull_request'.
Verification: add forbid 'id-token: write' "$e2e" to microvm-ci-release_test.sh — it already has a working forbid helper and this is exactly the drift it should catch. Independently, audit every OIDC trust policy naming this repo for a sub condition narrower than repo:stacklok/mecatl:*.
CWE-269 / CWE-250 / CICD-SEC-4 / CICD-SEC-5.
Sources: secure-code-reviewer (HIGH) + devops-expert (M4)
There was a problem hiding this comment.
Fixed by removing every id-token: write grant from the pull-request-triggered MicroVM E2E workflow; the workflow and live-hypervisor job now have only contents: read (workflow). The release contract test also rejects a future OIDC grant or keyless signing command in that job (guard). I have not treated this as evidence of any external OIDC trust-policy audit.
|
|
||
| while IFS=" " read -r kind payload payload_digest reference digest provenance bundle; do | ||
| archive="$assets/$payload" | ||
| test "sha256:$(sha256_file "$archive")" = "$payload_digest" |
There was a problem hiding this comment.
[Panel 3 · Domain · HIGH · cross-confirmed] The installer verifies no signature — the digest it checks comes from the same unsigned manifest.
This compares the archive against $payload_digest, which was read out of the manifest. The script never invokes cosign verify-blob: it reads the provenance / sigstore_bundle names from the manifest and copies their paths into microvmd-artifacts.json — a projection, not a verification. So the trust chain at install time is self-referential.
Compounding it, SHA256SUMS-$platform (package-microvm-release.sh:102,126) lists the two binaries and the three tarballs but not microvm-release-<platform>.json, so the documented sha256sum --check gives an operator zero integrity on the one file that feeds this installer.
Both .github/workflows/README.md:163-166 and docs/usage/microvm-environments.md claim the installer verifies, and the documented sequence places the install step before the cosign verify-blob snippet.
Fix — verify the statement, then bind it to the manifest, before extractall:
cosign verify-blob --bundle "$assets/$bundle" \
--certificate-identity "$MICROVM_RELEASE_IDENTITY" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
"$assets/$provenance"
# then assert the statement's subject digest equals the manifest's tree digestRequire the identity to be supplied (env or --identity) and abort if empty — do not default to a permissive value. Also add the manifest and this script to SHA256SUMS-$platform.
Execution stays fail-closed today because microvmd admission re-verifies the signed in-toto subject against the recomputed tree digest under an exact identity/issuer policy — which is why this is not Critical. But attacker-chosen bytes are still materialized into a privileged host path with no cryptographic check having run.
Verification: flip one byte of a payload tarball and assert the installer exits non-zero; swap a bundle and assert rejection.
CWE-347 / CWE-494 / CICD-SEC-3 / SLSA v1 verification requirements.
Sources: secure-code-reviewer + devops-expert (H2)
Folded in — two more defects in this same script:
:80(HIGH) — the shipped installer cannot run standalone. It resolves the digest tool via"$(dirname -- "$0")/../../environment/microvm", butpackage-microvm-release.sh:41-45copies this script intodist/<platform>/, where that path does not exist. Underset -eit aborts, so it is fail-closed — but the shipped installer is unusable for every operator following the documented flow, and the predictable response to a broken verifying installer is a manualtar -xzfthat skips every control here. The e2e never catches it becauseMECATL_MICROVM_INSTALLER(Taskfile.yml:240) points at the in-repo copy.:78(HIGH) —filter="fully_trusted"is a Python 3.12+ kwarg; stock macOS/usr/bin/python3is 3.9.6 and raisesTypeError, breaking the installer ondarwin-arm64, a platform this release ships. Drop the kwarg —"fully_trusted"is the legacy default, and the preflight at:56-74is the real defence.
There was a problem hiding this comment.
The trust boundary changed since this comment: the installer is executed only from an outer bootstrap bundle whose exact SHA-256 is supplied by the release-stamped host binary and checked before extraction (download verification); artifact signatures are then verified in-process by microvmd, not by this installer. The standalone path issue is fixed by shipping and invoking mecatl-artifact-digest-$platform beside the manifest (installer, digest check). The Python 3.12-only filter="fully_trusted" call remains, however, so that folded portability point is still open; the development fixture is not production release-signature proof.
| # authn/oidc is the opt-in caller-identity adapter module (ADR 0206), | ||
| # separate from both the dependency-free engine and provider modules. | ||
| - cd authn/oidc && go build ./... | ||
| # environment/microvm is the opt-in local microVM runtime module (ADR 0108). |
There was a problem hiding this comment.
[Panel 3 · Standards · HARD VIOLATION] Wrong ADR number.
ADR 0108 is Read skill assets on demand by logical name; the microVM ADR is 0224. docs/design/README.md treats a citation as a load-bearing claim, and docs/lint does not scan Taskfile.yml, so nothing catches this.
| # environment/microvm is the opt-in local microVM runtime module (ADR 0108). | |
| # environment/microvm is the opt-in local microVM runtime module (ADR 0224). |
A grep of ADR 0108|adr/0108|0108- across .github/, Taskfile.yml, docs/ and user-docs/ confirms this and user-docs/deployment/microvm-environments.md:151 are the only two mis-citations — every other 0108 hit is a legitimate reference to the real ADR.
Source: Standards axis (also independently found by the Spec axis)
There was a problem hiding this comment.
The original 0108 citation was removed, but the current Taskfile now says ADR 0350 while the accepted repository-scoped MicroVM decision is ADR 0352 (Taskfile, ADR 0352). The citation defect therefore still exists after renumbering, and I’m leaving this thread open for the mechanical correction to 0352.
|
|
||
| For configuration and recovery details, see the | ||
| [operator guide](https://github.com/stacklok/mecatl/blob/main/docs/usage/microvm-environments.md) | ||
| and [ADR 0108](https://github.com/stacklok/mecatl/blob/main/docs/adr/0108-microvm-execution-environments.md). |
There was a problem hiding this comment.
[Panel 3 · Standards + Spec · HARD VIOLATION] Dead link to an ADR file that does not and will not exist — and both CI gates miss it.
docs/adr/0108-microvm-execution-environments.md is not a file; 0108 is 0108-on-demand-logical-skill-assets.md. The correct ADR is 0224.
Because this is an absolute external GitHub URL, matlatl check . --strict treats it as external and Docusaurus onBrokenLinks: 'throw' does not resolve it either — so neither task docs:check nor task site:build can catch it. That makes AC8.5's proof ("verify: inspection — task docs and task site:build prove the documented surface is linked") vacuous for the one link an operator follows to reach the trust contract.
| and [ADR 0108](https://github.com/stacklok/mecatl/blob/main/docs/adr/0108-microvm-execution-environments.md). | |
| and [ADR 0224](https://github.com/stacklok/mecatl/blob/main/docs/adr/0224-microvm-execution-environments.md). |
The neighbouring docs/usage/microvm-environments.md GitHub URL is correct, for reference.
Sources: Standards axis + Spec axis (independently)
There was a problem hiding this comment.
Fixed through the documentation relocation rather than by retaining the old link. The canonical operator page is now user-docs/building/deployment/microvm-environments.md, the historical docs/usage page is only a compatibility pointer, and the live architecture references resolve to accepted ADR 0352 (canonical guide, compatibility pointer, ADR).
| @@ -0,0 +1,216 @@ | |||
| # ADR 0224 — Local microVM execution environments | |||
|
|
|||
| - Status: Proposed | |||
There was a problem hiding this comment.
[Panel 3 · Standards + Spec · HARD VIOLATION] ADR Status contradicts the shipped state.
This says Proposed (dated 2026-08-14), while docs/acceptance/microvm-execution-environments.md says Status: landed, 2026-08-15, docs/acceptance/README.md says "landed", and the code is merged. Every comparable shipped ADR (0214, 0217, 0218, 0221) is Accepted.
Per AGENTS.md the ADR is the frozen why of shipped behaviour, so shipped work under a Proposed ADR is a documentation-lifecycle contradiction. docs/lint's header gate checks the header exists, not that the value is current, so nothing catches a stale one.
| - Status: Proposed | |
| - Status: Accepted |
The header is otherwise well-formed (Status / Date / Scope / Supersedes / Superseded by, Context / Decision / Consequences / See also).
Sources: Standards axis + Spec axis (independently)
There was a problem hiding this comment.
The original status contradiction is resolved: the repository-scoped Linux decision is now ADR 0352 with Status: Accepted (ADR 0352). The Darwin arm64 delta is separately ADR 0351 and deliberately remains Proposed, matching the user documentation’s explicit statement that native Apple Silicon and signed-candidate qualification are still pending (ADR 0351, qualification boundary). The existing acceptance plan was updated in this PR under the user-authorized process waiver rather than creating a second plan.
9c310d6 to
e1365d5
Compare
38e2267 to
74253a4
Compare
ab8b47c to
6d65d53
Compare
df457f1 to
4908e4a
Compare
60c673f to
8809053
Compare
Co-Authored-By: mecatl <noreply@stacklok.com>
Co-Authored-By: mecatl <noreply@stacklok.com>
Co-Authored-By: mecatl <noreply@mecatl.dev>
Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Record the approved same-PR lifecycle v4 ownership contract and exact Scenario 5 proofs without implementing it. Update the resource inventory, preserve ADR renumbering, and remove the retired readiness-tracker link. Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Preserve explicit guest-source admission, isolated read evidence, child source lifetimes, and successor ownership. The separately approved daemon acquisition amendment follows in the next implementation step. Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Validate complete persisted bindings before runtime mutation, reserve exact refs through resolve publication, and unwind abandoned acquisitions without deleting durable worktrees. Keep fork rollback on the originating socket, release cancelled client owners, make authenticated control I/O cancellable, and permit exact pending-delete retries. Add deterministic authority, publication, cancellation, cleanup, and shutdown regression proofs while preserving standalone module boundaries. Co-Authored-By: OpenAI <noreply@openai.com>
Retire abandoned resolve reservations under the daemon lifetime context. Start bounded cleanup only after operation pins drain, and propagate cancellation through guest detach without changing public attachment APIs. Prove automatic unregister after a real pinned workspace call outlives the cleanup timeout using deterministic fake time, including explicit detach and daemon shutdown cancellation. Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Real KVM session creation exposed Git archive permission normalization when a private source checkout uses a different umask from microvmd. Restore the captured regular-file modes without weakening exact-state verification. Co-Authored-By: OpenAI <noreply@openai.com>
A fresh harness resuming after the compatible local daemon stopped must run the existing readiness path before exact reattachment. Validation and no-filesystem attenuation still precede readiness; no placement or command is replayed. Co-Authored-By: OpenAI <noreply@openai.com>
Close v4 child, explicit binding, and service owners in lifecycle order, then reacquire after daemon restart. Preserve all real guest assertions and recognize only explicit dirty retention during child cleanup. Co-Authored-By: OpenAI <noreply@openai.com>
Exact resolve can validate retained artifacts and boot the repository VM. Keep that acquisition cancellable by its caller instead of imposing a 15-second hot-lookup deadline; independent cleanup bounds remain unchanged. Co-Authored-By: OpenAI <noreply@openai.com>
Distinguish real KVM and live GPT-5.6 Sol functional evidence from deterministic slow-acquisition coverage and unqualified Darwin/release paths. Co-Authored-By: OpenAI <noreply@openai.com>
Remove obsolete lineage migration and read locking resurrected by merge replay, restore exact discovery acquisition and terminal-title wiring, and retain the original Taskfile wording. Link the MicroVM installation instruction to its canonical page to satisfy the new strict documentation graph gate. Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: mecatl <noreply@stacklok.com>
Co-Authored-By: OpenAI <noreply@openai.com>
d5491b7 to
a012760
Compare
Use private portable socket fixtures, restore captured symlink permissions without following targets, and bind Studio publication to the immutable release-ref gate. Cover umask changes and path substitution, and fix opt-in development lint findings. Co-Authored-By: OpenAI <noreply@openai.com>
2 participants
Summary
Adds
microvm-local, an opt-in execution environment for local Git repositories. The deployment default remainshost-localunless trusted operator settings select MicroVM execution. Linux amd64 now has real-KVM source-build and live-model qualification for private lifecycle v4; Darwin arm64/macOS 15+ remains experimental.repositoryinstruction/command sources require explicit operator selection and separate project admission. They borrow the exact authorized guest workspace without a runner or execution read-ledger evidence. Independent sources remain usable without guest attachment, including no-FS execution.65532:65532; Linux uses namespace mapping, while Darwin uses go-microvm v0.0.41 ownership preparation and authenticated launch supervision.Integration and approved amendment
Rebased onto main
17700f1e7a5190141bbf4a784d8990fece338428, including #1875 and #1878, and followed the updated MicroVM HarnessContext handoff. The full final-tree preservation audit included historical merge resolutions, not just patch equivalence. The incremental rebase's tree exactly matched the expected direct merge;environment/microvmandinternal/adapter/microvmremain byte-for-byte identical to the previously qualified candidated5491b7237f5b2c43d23c06e2034e0c706591ab1.The rebase preserved lineage/read-lock behavior and both placement wiring paths. A separately reviewed persisted-approval correction retains the factory GovernanceRoot for permission/memory context and root-session causality in ordinary and scoped paths. MicroVM ADR 0364 was renumbered to 0367 to avoid main's new native-Kubernetes ADR 0364; the native-Kubernetes decision is unchanged.
The directing human explicitly approved the narrow same-PR connection-owned acquisition amendment after reviewing normal multi-terminal and cross-instance scheduling use cases. The amendment was committed separately as
1c3cba775ae75b7bc1d29f17873627fa3efbf694before implementation. It changes the private lifecycle protocol from v3 to v4, not public RPCs, engine interfaces, source-selection authority, SessionLease, or persistent ownership schemas. Mixed private protocol versions are rejected; a mismatch does not automatically replace a serving daemon.Interfaces match the approved amended contract: Yes. No distributed fencing framework, inventory RPC migration, cold-child source reconstruction, or cross-process Git transaction system was added. Human merge remains separate.
docs/acceptance/microvm-execution-environments.mddocs/adr/0367-microvm-execution-environments.mddocs/adr/0365-microvm-darwin-xattr-ownership.mddocs/adr/0359-harness-context-source-authority.mduser-docs/building/deployment/microvm-environments.mdLive validation and corrections
The operator subsequently authorized real Linux KVM and OpenRouter
openai/gpt-5.6-solverification. This exposed three production defects that offline fixtures had missed:0600source files fail exact capture under microvmd's deterministic022umask. Captured regular-file permissions are now restored within the confined worktree; digest verification remains strict.The existing live E2E test was also corrected to finish the run and retire v4 child/binding/Build owners before daemon restart and final deletion. Only the explicit dirty-retention outcome is accepted during child cleanup; other errors remain failures.
Real executed evidence (pre-rebase)
The following live evidence predates this rebase. Runtime/client trees were preserved byte-exact; new-main integration, wiring and the approval-context correction are verified offline, not by a new live-provider or VM run.
MECATL_MICROVM_E2E_PREPARED=1 task e2e:microvmpassed at3d55a8cfa600546fd7c03a177dd2293a1b54aa50: real guest filesystem/Shell, isolation, post-boot child merge/conflict, daemon restart, exact reattachment, and cleanup. The provider in this gate is scripted; the VM is real.9a1c2cd05abca6f101a2b29e467baeda3babdb80against the unchanged compatible, verified v4 runtime bundle from3d55a8cfa600546fd7c03a177dd2293a1b54aa50. The original EnvironmentRef, guest coding fix, proof file and selected context survived; actual Read/Shell results andend_turnconfirmed success. No replacement session or command replay was used.Evidence caveats: the original failed cold request was cut off after 15.008 seconds. The successful corrected run returned HTTP 200 after 12.986 seconds and completed in 19.708 seconds. Its driver returned nonzero solely because an extra greater-than-15-second timing assertion was unmet; functional checks passed. Successful live acquisition beyond 15 seconds was not observed. Deterministic virtual-time regressions prove the old timeout fails a legitimate slow acquisition and the correction preserves caller cancellation/deadlines. One scheduled PresentPlan attempt was correctly denied in headless mode; the read-only fire recovered and completed. The child token was included in its delegated prompt, so token echo alone is not claimed as independent live proof of automatic child-context ingestion; that remains covered by the factory tests.
Provider credentials were supplied only through the provider-process environment, not displayed, placed in prompts, or mounted into the guest. Qualification-owned harnesses and daemon were gracefully stopped; diagnostic fixtures/logs were retained locally, not uploaded. No unrelated local instances were targeted.
Offline verification and review
Rebased candidate:
a012760af3df81aea360a6711cc224777ffdc41fon main17700f1e7a5190141bbf4a784d8990fece338428.Completed again on this assembled post-rebase head, all exit 0 (run serially):
task testtask linttask test:racetask docstask site:buildtask api:checktask ac-trace-strictgo run ./cmd/mecademo— Read, Write permission ask/approval, result, Team, and background Subagenttask testandtask test:racecover the root, engine, authn/oidc, MicroVM and all provider modules, plus their standalone/build-tag proofs through the unchanged Taskfile.task lintpassed with the pinned binary and existing configuration across all eight module scopes; no timeout override or rule disabling was needed. Strict docs checked 582 documents and 5,667 references with zero broken links/anchors, ambiguous targets, or orphan/unreachable pages.Global strict trace exited 0. The owning MicroVM plan resolves all 35 criteria (34 named-test criteria and one inspection method) with zero missing references; shared HarnessContext resolves all 25 criteria. Other proposed/in-progress plans still produce report-only diagnostics. Resolution is not itself live execution evidence.
Focused semantic regressions, mutation checks, race tests and the previously completed independent reviews cover ownership, authority, source isolation, file-mode capture, readiness and cold-acquisition cancellation. Spec, Standards, Test adequacy, Security and Architecture/reuse reviews found no remaining issues; the preceding full integration panel was also clean. The subsequent final-tree preservation audit and persisted-approval correction were independently reviewed clean. No new external review panel is claimed for this final gate run. Generated checksum churn was excluded; there were no tracked site-generation changes. CI must independently verify this pushed head; local gates are not a CI-green claim. This PR is not merged.
Remaining qualification limits
This is Linux source-build qualification using locally verified development artifacts, not qualification of a published signed release. Physical Apple Silicon/HVF and Darwin release qualification remain pending; native unit tests and cross-compilation are not presented as live Darwin proof. Actual TUI keyboard interaction and a naturally slow successful live acquisition exceeding 15 seconds were not tested. The overall acceptance plan remains in-progress for the outstanding platform/release work.
This is a local, single-operator feature supporting multiple harness processes. Linux arm64 live operation, remote/multi-user placement, non-Git sources, repository-VM deletion, and advanced orphan recovery remain outside this slice.
Closes #526
Closes #527
Closes #528
Closes #529
Closes #530
Closes #531
Closes #532
Closes #533
Closes #534
Closes #535
PANEL: ship_blockers=0 important=0 advisory=0 reviewer_failures=0