Repository navigation
add ls-on-firecracker: LocalStack on a Firecracker microVM demo - #14
Merged
Merged
Conversation
Boots LocalStack inside a Firecracker microVM (Docker installed in the guest so LocalStack's Lambda executor works normally), with a self-describing Makefile (download -> rootfs -> up -> test -> down) and a GitHub Actions workflow that runs an S3 round-trip plus a real Lambda deploy/invoke assertion on an ubuntu-latest KVM runner. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
… in chroot CI run showed apt-get failing inside the rootfs chroot with confusing "No such file or directory" errors -- /tmp and /run are normally populated by systemd-tmpfiles at boot, which never runs since we chroot into a mounted image rather than booting it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
Confirmed by loop-mounting the actual Firecracker CI base image: it ships apt/dpkg binaries but /var/lib/dpkg is completely empty (no status file, no info/updates/triggers dirs), causing apt-get to fail with "flAbsPath ... realpath: No such file or directory". This is the same bootstrap debootstrap itself does for a fresh root. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
Inspected the actual base image locally with debugfs: /var/cache/apt,
/var/lib/apt and /var/log don't exist at all in this CI artifact (only
/var/lib/{systemd,dpkg} survived whatever stripped the image down).
Recreate the standard set of scratch dirs apt/dpkg expect before
running any apt commands.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
apt-get install was hanging/failing on libpam-modules' postinst asking a Y/I/N/O/D/Z conffile-merge question on stdin (not a TTY in this chroot). --force-confdef + --force-confold resolve it the same way any non-interactive apt-get install does. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
…'s pip apt/docker install now completes cleanly (confirmed in CI). The base image is Ubuntu 22.04, whose bundled pip predates PEP 668 enforcement and doesn't recognize --break-system-packages at all. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
The microVM now boots fully (confirmed in CI: kernel, network, systemd, login all work), but docker.service fails to start and the serial console doesn't show why. The base image ships sshd with a pre-authorized root key; the matching private key is published alongside the rootfs in the same CI bucket. Fetch it, and add `make ssh` / `make diagnose` (systemctl status + journalctl for docker/localstack), wired into the CI failure step so the next run tells us the actual error instead of us guessing blind. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
The id_rsa key is a sibling of the .ext4 file (ubuntu-22.04.id_rsa), not an appended suffix (ubuntu-22.04.ext4.id_rsa) -- the previous path 404'd, so 'make diagnose' had no key and its error went nowhere because the CI step redirected stderr to /dev/null. Fixed both. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
Root cause of the docker.service failure (confirmed via SSH journalctl): Ubuntu 22.04's iptables defaults to the nftables backend, but Firecracker's CI kernel does not compile in nf_tables (only legacy x_tables, which we confirmed is present). Switch both iptables and ip6tables to the legacy alternative, which the kernel does support. Also switches LocalStack's startup from a bare `pip install localstack` to lstk, LocalStack's own CLI, which pulls and runs LocalStack as a container against the guest's Docker daemon -- matching how LocalStack is officially run today, and how real Lambda's own container-inside- Firecracker-microVM architecture is shaped. localstack.service now wraps `lstk start --non-interactive` as a oneshot unit (lstk blocks until the emulator is ready, then exits; the container is the actual long-running thing). Requires a LocalStack CI Auth Token in LOCALSTACK_AUTH_TOKEN, passed into the guest via a systemd.setenv= kernel boot arg. Also documents (README) that the base rootfs is Firecracker's own CI test artifact rather than a general-purpose image, which is the root cause of most of the earlier fixes, and points at the more idiomatic docker-export-based approach and firecracker-containerd as the production-grade alternative. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
…ction - run-vm.sh: bail out as soon as docker.service or localstack.service reaches a terminal "failed" state instead of waiting out the full boot budget every time (Docker itself now starts fine after the iptables-legacy fix, but lstk was failing and getting silently retried by systemd for the full 7 minutes). - build-rootfs.sh: localstack.service now has Restart=no -- repeated auto-restarts were burying the actual lstk failure under a wall of "Failed to start" lines with no visible error. - Makefile: `make diagnose` now prints the unit's exit code/result and the full (untruncated, single-attempt) journal instead of a possibly-truncated tail. - fixtures/handler.py + smoke-test.sh: the Lambda function now creates a bucket and lists all buckets via boto3 (LocalStack auto-injects AWS_ENDPOINT_URL, no endpoint code needed), and the test asserts it sees the CLI-created bucket and that the bucket it creates is visible back on the CLI afterward -- exercising real Lambda <-> S3 API interaction through the same LocalStack backend, not just a bare invoke/response round-trip. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
The previous run's diagnose step itself failed: an echo string with unquoted parentheses broke the remote bash command before it could print the actual lstk failure we need to see. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
Root cause finally visible thanks to the fast-fail + Restart=no fixes: "Error: failed to get user home directory: \$HOME is not defined". systemd services don't inherit HOME the way login shells do, but lstk needs it to resolve its config/cache directory. The unit runs as root, so point HOME at /root. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
Progress: the HOME fix worked -- lstk pulled the image, validated the auth token, and tried to start the container. New failure: "Unable to enable DIRECT ACCESS FILTERING - DROP rule ... iptables raw table: Table does not exist". Docker 28+ added this hardening rule, which needs CONFIG_IP_NF_RAW; our kernel has neither that config nor module loading at all, so it can never be satisfied. Docker 28.0.2 added DOCKER_INSECURE_NO_IPTABLES_RAW=1 as the documented opt-out for exactly this case (moby/moby#49621) -- set via a docker.service systemd drop-in. Documented the security tradeoff in the README. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
The previous run showed real progress: lstk pulled the image, the auth token validated, and the container started successfully -- but our host-side health check never got through. Checked lstk's own source (internal/container/gateway.go, start.go): it publishes the emulator's port bound to the guest's 127.0.0.1 by default, and the bind host only comes from a GATEWAY_LISTEN entry in a config.toml [env.*] profile -- a bare LOCALSTACK_GATEWAY_LISTEN process env var reaches the container too late, after the host-side Docker port binding is already decided. Added a config.toml at /root/.config/lstk/ (matches lstk's config search order once HOME is set) with GATEWAY_LISTEN=0.0.0.0:4566,0.0.0.0:443, so the port is reachable from outside the guest over the tap network. Also: - run-vm.sh: on a health-check timeout, SSH in and dump the actual listening sockets, docker port mappings, and an in-guest curl, so a future networking mismatch is visible immediately instead of requiring another round of guessing. - Split the README: kept it to quick start + prerequisites + layout, moved the "how it works" walkthrough and caveats (stripped-down CI rootfs, the iptables-legacy and DOCKER_INSECURE_NO_IPTABLES_RAW workarounds) into docs/NOTES.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a demo that boots LocalStack inside a Firecracker microVM (the same
tech real AWS Lambda runs on) instead of a Docker container, and exercises
it with an S3 round-trip and a real Lambda deploy + invoke.
ls-on-firecracker/Makefile— self-describing (make help):download->rootfs->up->test->down/cleanscripts/download-assets.sh— fetches thefirecrackerbinary, kernel, and base rootfs from Firecracker's public CI bucketscripts/build-rootfs.sh— installs Docker + LocalStack into the guest rootfs image, registers systemd units to start them on bootscripts/run-vm.sh— sets up the tap device + NAT (so the guest candocker pullthe Lambda runtime image) and boots the microVMscripts/smoke-test.sh— creates/reads an S3 object, then deploys and invokesfixtures/handler.pyas a Lambda function and asserts the response.github/workflows/test-ls-on-firecracker.yml— runs the above on anubuntu-latestKVM runner, using theTEST_LOCALSTACK_AUTH_TOKENsecretWhy not tested locally
Firecracker requires a Linux host with KVM. This was developed on an Apple
Silicon Mac, which can't run KVM (nested virtualization needs an M3+ chip).
Everything here is syntax-checked, but the actual boot -> Docker -> Lambda
pull -> invoke chain is validated by the CI workflow in this PR, not locally.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Ny78ki4drCoyQgDxkSoSdC