feat(compute): add AWS Lambda MicroVMs ComputeStrategy backend — P1 (#645)

[dreamorosi]

Implements Phase P1 of ADR-021 (#688): the lambda-microvm compute backend — strategy (start/poll/stop), CDK infrastructure, bootstrap policy, CLI support, and layered regional-availability enforcement. AgentCore remains the default. Suspend/resume is P3; agent-side hook serving and smoke parity are P2 — the ADR and the construct both warn explicitly that a P1-built image is not runnable end to end.

Strategy reports, orchestrator interprets

LambdaMicrovmComputeStrategy.pollSession maps the six-state MicrovmState enum mechanically (SessionStatus gains 'suspended'); the orchestrator cross-references DynamoDB — substrate-terminal + non-terminal status → failed (with a confirm re-read to avoid failing a successful fast teardown), suspended + non-AWAITING_APPROVAL → anomaly event, no fail-fast. RunMicrovm always omits idlePolicy (auto-suspend disabled by construction, asserted by tests) and pins maximumDurationInSeconds at 28 800 s (AgentCore 8 h parity). Strategy errors carry a MicroVM <op> failed marker so the new classifier entries cannot change agentcore/ECS retry semantics — pinned by regression tests against the pre-change UNKNOWN shape.

No image wildcards in IAM

Every grant is scoped to the exact image ARN (bare names are resolved via formatArn; a ':*' sibling covers version-qualified forms — image names cannot contain :). microvmConfig.imageArn is compiler-required, so the account-wide fallback cannot return silently. Orchestrator gets exactly RunMicrovm/GetMicrovm/TerminateMicrovm/PassNetworkConnector + scoped iam:PassRole; the cancel Lambda gets TerminateMicrovm only; no Suspend/Resume/auth-token grants anywhere (P3), asserted by tests including on the generated bootstrap JSON.

Regional availability enforced in layers

Synth-time gate on the 5 launch regions with a microvm_region_override escape hatch (token regions skipped, documented); CLI onboarding probes ListManagedMicrovmImages on the effective compute type (flag ?? stored ?? default) before any write; platform doctor probes when an active blueprint uses the backend; orchestration-time classification is non-retryable with a configuration remedy.

Cancellation ordering fix worth reviewer attention

The new lambda-microvm branch in cancel-task.ts is deliberately placed before the agentRuntimeArn fallback: RUNTIME_ARN is set stack-wide whenever AgentCore stop is wired, so a MicroVM task would otherwise have invoked StopRuntimeSession against an unrelated runtime in mixed deployments. Pinned by a regression test.

ADR-021 refined in place (proposed): six-state poll-mapping table, GetMicrovm NotFound → completed rationale, microvmId/microvmIdentifier seam, per-hook phasing table.

Verification: cdk 141 suites / 2 744 tests, cli 52 / 651, tsc + eslint clean in both, types-sync green, knip at baseline (78), bootstrap generation and docs sync deterministic. DEPLOYMENT_ROLES.md gains golden policy block [5] with test parity.

Refs #645

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 99.90988% with 3 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@d3974f7). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
agent/src/server.py	95.58%	3 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #689   +/-   ##
=======================================
  Coverage        ?   91.69%           
=======================================
  Files           ?      292           
  Lines           ?    80689           
  Branches        ?     7681           
=======================================
  Hits            ?    73984           
  Misses          ?     6705           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[dreamorosi]

I'll address the CI comments tomorrow

[krokoko]

Thanks for this — Phase P1 of ADR-021 is thoughtfully scoped, and the write-up makes the intentional non-goals (P2 hooks / P3 suspend) easy to review against. Strategy-vs-orchestrator split, image-ARN IAM scoping, cancel ordering before the RUNTIME_ARN fallback, and the layered regional gates all look solid. CI green + strong test coverage help a lot.

Request changes

Onboard should gate on ComputeSubstrate for lambda-microvm (parity with ECS)

In cli/src/commands/repo.ts, ECS already refuses onboard when the stack wasn’t deployed with the ECS substrate. MicroVM currently only probes regional availability (ListManagedMicrovmImages), so an operator can still write compute_type=lambda-microvm against an agentcore-only stack and only fail later at RunMicrovm.

Suggest mirroring the ECS check (and the onboard note) for lambda-microvm when ComputeSubstrate is present and not lambda-microvm. The agent stack already emits that output value.

Suggestions (non-blocking / nice-to-have in this PR)

1. microvm_suspend_anomaly spam — reconcileMicrovmSubstrateState emits on every poll (~30s) while suspended outside AWAITING_APPROVAL. Rare in P1 (no Suspend grants), but once it hits it can flood events for a long window. Emitting once per task (e.g. a PollState flag) would keep “no fail-fast” without the noise — especially useful before P3.

2. Orphan MicroVM if persist fails after RunMicrovm — start-session starts the VM, then writes compute_metadata; the catch path failTasks without TerminateMicrovm. Same race as AgentCore/ECS, but MicroVM cost/quota makes the leak more expensive. Best-effort terminate from the in-memory handle in the catch would close it. Fine as a follow-up issue if you’d rather not expand this PR.

3. Shared payload-bucket read — payloadBucket.grantRead(executionRole) is bucket-wide (incl. list), same shape as ECS #502. Scoping to the task key / dropping List* would improve isolation when this backend is enabled; also fine as follow-up.

4. Incomplete RunMicrovm response — the missing microvmId/endpoint throw isn’t wrapped with wrapMicrovmError, so it may miss the MicroVM classifier marker. Small consistency fix.

What looks great

• Exact 16 384 inline/S3 payload split, idlePolicy omitted, 28800s max duration
• Terminal substrate + DDB re-read before fail; suspend + approval treated as healthy
• No Suspend/Resume/auth-token grants; bootstrap deploy-time only with golden policy block
• Packaging script API shapes aligned with the service model
• Knip unexports are fine hygiene

Happy to re-review quickly once the onboard gate lands — and open to deferring 2–3 to tracked follow-ups if you prefer to keep this PR tight.

[dreamorosi]

Thanks @krokoko — all four items addressed in 250ca12, plus important context: a live verification run against the real service (us-east-1) landed between your review and this push, and it invalidated several things both of us believed. Point-by-point first, then the delta.

Blocking — ComputeSubstrate onboard gate: done, with ECS parity via a shared assertComputeSubstrateDeployed() helper (cli/src/compute-substrate.ts) so the two backends can't drift. It preserves the ECS check's deliberate leniencies (agentcore never gated; null output = older stack = allowed), runs before the availability probe (cheaper — reuses the output already fetched; more specific — a substrate-less stack fails regardless of region; an ordering test proves the probe is never reached), and splits on commas testing membership, so the compute_types-list follow-up ADR-021 already names won't silently break it. Onboard notes gained the lambda-microvm entries, including the gap the gate can't cover: ComputeSubstrate proves substrate, not image — the three-state first deploy still needs the packaging step.

Suggestion 1 (anomaly spam): implemented — PollState.microvmSuspendAnomalyReported, once per episode, re-arms on recovery (a second episode is new information; latching forever would hide a flapping suspend loop), carries through poll failures (a GetMicrovm hiccup isn't evidence the anomaly ended). WARN log stays per-poll with an anomaly_already_reported marker.

Suggestion 2 (orphan MicroVM): implemented in-PR — best-effort TerminateMicrovm in the orchestrator catch (handle hoisted out of the try), plus a second window you didn't flag: RunMicrovm returning microvmId without endpoint throws inside the strategy before any handle escapes, so the strategy reaps there too. No new IAM (the orchestrator already holds TerminateMicrovm for finalize). Deliberately not generalized to ECS/AgentCore: their stop permissions live on the cancel Lambda, so the same pattern would add AccessDenied noise — noted in-comment as a follow-up needing the IAM change reviewed together.

Suggestion 3 (bucket-wide read): follow-up issue as you offered — filing it against BOTH backends (it's the #502 shape) with a concrete proposal: drop s3:List* outright, scope GetObject via the aws:PrincipalTag/task_id condition the artifacts bucket already uses.

Suggestion 4 (unwrapped throw): wrapped with wrapMicrovmError, test asserts the marker.

The live-run delta (things your "looks great" list can no longer say): the 16 384 payload split is now 4 096 — the service enforces it; the 16 KB figure in AWS's own docs/SDK prose is wrong and we've filed documentation tickets. Ingress: omitting the field attaches a public HTTP_INGRESS connector, so NO_INGRESS is now an explicit all-or-nothing control with outcome-asserting tests. Memory: minimumMemoryInMiB is a baseline capped at 8 192 MiB (the 32 GB launch figure is the automatic 4× peak) — validated prop, reframed ADR. Hooks: the service refuses lifecycle hooks without /ready, and a hook-less image can't accept runHookPayload at all — so /ready + /run moved into P1 (the agent now serves them, reusing the /invocations spawn path) and the phasing table was rewritten. Also: VPC_EGRESS connectors require an operator role, and image builds need port-80 egress (separate build-time connector; runtime stays 443-only). Full evidence is in the verification findings being posted to #645.

Ready for re-review.

[dreamorosi]

Housekeeping since the last comment, for re-review context:

• bfe895c — merged main (feat(orchestration): land the carved orchestration arc on main (S2–S8) #695 orchestration carve S2–S8; 16 overlapping files). One semantic collision: this branch had un-exported MIME_TO_EXTENSION & friends as knip-dead code, and feat(orchestration): land the carved orchestration arc on main (S2–S8) #695's linear-attachments.ts consumes them — restored. Everything else was disjoint or mechanical; full matrix re-verified post-merge (cdk 3736 / cli 722 / agent 1460; three-config synth; docs/bootstrap determinism). Note: the knip advisory count is 98 vs baseline 78 — all 20 over-baseline findings are in files feat(orchestration): land the carved orchestration arc on main (S2–S8) #695 introduced (this branch alone measures exactly 78); left the baseline un-raised rather than laundering it.
• 3d848ca — cleared 25 osv advisories across agent/uv.lock, the jira-forge lock, and yarn.lock (cryptography 50.0.0, fast-uri, ip-address, undici). These are ecosystem advisories against pre-existing pins — main's weekly security scan currently fails identically — but they gate this PR's scan.

[krokoko]

LGTM ! @dreamorosi thank you for this, could you please just rebase onto current main since there are conflicts and I'll pprove after !

[dreamorosi]

@krokoko conflicts resolved in 4c98618 — merged current main rather than rebasing, since the branch already carries the reviewed #695 merge commit and the repo workflow (AGENTS.md) documents merge-from-main; a squash-merge lands identically either way.

Resolution was clean: the conflicts were confined to the four dependency files where your team's canonical osv remediation (#711) + resolution pruning (#717) + drift guard (#718) collided with this branch's stopgap fix for the same advisories. Took main verbatim for all four, then regenerated yarn.lock under the CI toolchain for the @aws-sdk/client-lambda-microvms entries this branch adds (byte-stable across repeated installs). #718's drift guard passes, osv clean, full matrix green post-merge (cdk 3 736 / cli 722 / agent 1 460 / jira-forge 9; tsc + eslint clean; docs sync idempotent).

[dreamorosi]

@krokoko conflicts with current main resolved again in 42d693b (#345 attribution + #704 digest pins landed under the PR). Beyond the merge itself, the MicroVM-specific SDK clients (cancel handler, strategy, CLI probe) now use #345's attributed-client factories, matching the convention that PR established for every other client. Full matrix green post-merge (cdk 3 776 / cli 736 / agent 1 483 / jira-forge 9; osv + drift guard + docs sync clean). If it's convenient, enabling auto-merge on your approval would spare us racing main a fourth time.