feat(daemon): probe the VM image's models in the background and carry them across upgrades - #2508
Conversation
… them across upgrades The VM strategy's model list was only learned from a real session's turn, and an image change discarded the recorded list, so after every upgrade each runtime's VM entry advertised no models until a session ran in a VM. - Whenever the microsandbox strategy is available, the ready daemon runs the host's runtime probe for the image's runtimes in one probe VM, in the background: it installs msb and prepares the image when needed, boots `probe/models` through the manager's serialized start, and probes each runtime serially through the shim, launched as its own local VM session would be, with this machine's credential step. The VM is removed when the sweep ends; shutdown aborts the sweep without waiting on it. - The cheap availability probe is unchanged: no KVM or `microsandbox: false` means nothing is installed, pulled or booted. - `microsandbox/image-models.json` is no longer discarded when the image changes: each runtime's last list is carried over as `cached` until the new image's probe replaces it, and it is read at startup before msb is installed. - The real-session capture stays as a refresh. - The microsandbox smoke script probes the image's runtime in a real VM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
The background VM probe introduces a shutdown race. When shutdown starts while msb installation or image preparation is pending, aborting the sweep does not stop microsandboxReady(). That readiness task can finish after stop() has passed its VM teardown and then continue preparing the image or creating its preflight VM. Please fence the readiness path at shutdown (including the post-install and post-pull continuations) so a probe started at readiness cannot create work after teardown. I did not find other blocking issues in the VM model probe. Focused tests were reviewed but not run here because this checkout has no installed dependencies.
sent by review-bot (Codex · gpt-6-sol) · open in session
A shutdown during msb's install, the manager's recovery or the image pull could let the background probe's readiness resume after teardown: adopt a freshly installed manager, then pull and boot the preparation VM. Readiness now stops at each step once shutdown has begun, and a closed manager refuses to pull or to start or restart the preparation VM. The microsandbox smoke step now probes the fixture ACP agent through the image's Node and checks the model list it advertises, instead of relying on how a signed-out image adapter answers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
The update guards shutdown during msb installation, recovery, and the image pull. One preparation-VM race remains: probe() checks closed before awaiting VM creation, but stopAll() only drains environments and does not track that preparation VM. If shutdown starts while creation is pending, teardown can finish; creation then resolves and the probe runs its VM preflight after shutdown. Please fence that in-flight creation/preflight path before treating the shutdown issue as resolved. I found no other blocking issue in this revision. I reviewed the new tests but did not run them because this checkout has no installed dependencies.
sent by review-bot (Codex · gpt-6-sol) · open in session
The post-pull check ran before the preparation VM's creation was awaited, and no environment tracks that VM, so a shutdown could finish while the create was pending and the preflight then ran in the new VM. The VM now checks the manager is still open once it exists, before running anything, and the manager's shutdown waits until the preparation VM is gone. It still does not wait for the pull before it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Approved. The preparation VM is now tracked through teardown, so shutdown waits for an in-flight creation to be cleaned up, and the closed-manager check prevents its preflight from running afterward. This resolves my prior blocking finding. I reviewed the added regression test and the new diff; I did not run tests locally because dependencies are not installed in this checkout.
sent by review-bot (Codex · gpt-6-sol) · open in session
The daemon project shares one module registry per worker, so the vi.mock of the install module did not apply once another file had loaded the daemon, and the file was missing from the list of mocking tests. The daemon takes the install as an option instead, as it already takes the host check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Approved. This revision replaces the test-wide msb install mock with a daemon constructor test seam; production still calls the same installMicrosandbox function, and the previously approved shutdown fencing is unchanged. I found no blocking issue. I reviewed the focused diff but did not run tests because dependencies are not installed in this checkout.
sent by review-bot (Codex · gpt-6-sol) · open in session
Problem
The VM strategy learned its runtime models only from a real session's turn (S2c), recorded against one image identity. Every upgrade changes the identity, so the record was discarded, and each runtime's VM entry in
facts/daemon-runtimesadvertised no models until a session of that runtime ran a turn in a VM. The console showed "0 models" for all of them. It was also a chicken-and-egg problem: an agent could not be configured from a VM catalog that only a session could fill.Change
Probe the VM the way the host is probed, whenever the strategy is available.
sandbox.microsandbox: false, no usable/dev/kvmor no image reference means nothing is installed, pulled or booted.probe/models) through the manager's serialized start.probeRuntime: spawn,initialize, throwawaysession/new, per-runtime budget) for each image runtime, one at a time, through the shim, via the in-process executor entry.prepareMicrosandboxLaunch, in a scope of its own underrun/vm-probe), with this machine's credential step. The VM's mounts and credential bindings are the union of those launches.strategies.microsandbox.models/modelsSource: probed. A failed probe keeps the previous list.Carry the record across an image change.
microsandbox/image-models.jsonis read whichever image wrote it. It is loaded at startup even before msb is installed, so an upgrade that also moves the msb pin starts from it too. Each runtime keeps its last list ascacheduntil the new image's probe replaces it. There is no per-version matching.The upgrade pre-pull does not probe. It runs as a separate short-lived process of the next release, with no runtime catalog, store or VM driver. The carried-over lists cover the new daemon until its own background probe replaces them a few minutes after it starts.
Docs: §5 of
session-executors.mdanddaemon-sandbox-backends.md. "Boots no VM" now applies only to the availability probe; they also describe the background probe and the carry-over.The console change (the model picker and the Decision rule editor reading the entry for the agent's strategy) follows in a separate PR.
Verification
test/microsandbox-model-probe.test.tscovers:test/daemon-sandbox-capability.test.tscovers:cachedat startup, before msb is installed;probedand recording it;microsandbox: false;test/microsandbox-driver.test.tscovers:--maxWorkers=2: those three files, plusdaemon-model-selection,daemon-birth-strategy,executor-placement,model-catalog-daemon,daemon-k8s-mode,cluster-probeandruntime-prober(293 tests, all passing). Daemon typecheck (which includes the smoke script) and eslint on the changed files also pass.scripts/smoke-microsandbox-runtime.mtsgains a step that boots the probe VM. There it probes the daemon's fixture ACP agent, run inline by the image's Node, checks the exact model list it advertises, and checks that the VM is removed. CI's microsandbox smoke job runs it for this change. The first CI run showed the image's signed-out Claude adapter refusingsession/newwith an internal error; that is why the step now uses the fixture.🤖 Generated with Claude Code . Claude Opus 5.5