add ls-on-firecracker: LocalStack on a Firecracker microVM demo

.github/workflows/test-ls-on-firecracker.yml

@@ -0,0 +1,56 @@
+name: LocalStack on Firecracker
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - 'ls-on-firecracker/**'
+      - '.github/workflows/test-ls-on-firecracker.yml'
+  push:
+    branches: [main]
+    paths:
+      - 'ls-on-firecracker/**'
+      - '.github/workflows/test-ls-on-firecracker.yml'
+  workflow_dispatch:
+
+env:
+  LOCALSTACK_AUTH_TOKEN: ${{ secrets.TEST_LOCALSTACK_AUTH_TOKEN }}
+
+jobs:
+  test-ls-on-firecracker:
+    name: Firecracker microVM + LocalStack + S3/Lambda smoke test
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    defaults:
+      run:
+        working-directory: ls-on-firecracker
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Enable KVM access
+        run: |
+          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
+            | sudo tee /etc/udev/rules.d/99-kvm4all.rules
+          sudo udevadm control --reload-rules
+          sudo udevadm trigger --name-match=kvm
+          ls -l /dev/kvm
+
+      - name: Boot the microVM
+        run: make up
+
+      - name: Run S3 + Lambda smoke test
+        run: make test
+
+      - name: Dump console/boot log on failure
+        if: failure()
+        run: cat work/firecracker.log 2>/dev/null || true
+
+      - name: Dump service diagnostics on failure
+        if: failure()
+        run: make diagnose || true
+
+      - name: Tear down
+        if: always()
+        run: make down

ls-on-firecracker/.gitignore

@@ -0,0 +1 @@
+work/

ls-on-firecracker/Makefile

@@ -0,0 +1,60 @@
+FC_VERSION   := v1.10.1
+CI_TRACK     := v1.10
+UNAME_M      := $(shell uname -m)
+# Firecracker's release/CI artifact names use "aarch64"/"x86_64", not the
+# "arm64"/"amd64" spellings uname reports on macOS or Debian-flavored Linux.
+ARCH         := $(if $(filter arm64,$(UNAME_M)),aarch64,$(if $(filter amd64,$(UNAME_M)),x86_64,$(UNAME_M)))
+WORK_DIR     := work
+TAP_DEV      := fc-ls-tap0
+TAP_IP       := 172.16.0.1
+VM_IP        := 172.16.0.2
+VM_MASK      := 255.255.255.0
+VCPUS        := 2
+MEM_MB       := 4096
+BUCKET       := firecracker-demo
+LAMBDA_FN    := firecracker-demo-fn
+LAMBDA_BUCKET := firecracker-demo-lambda-bucket
+
+export WORK_DIR TAP_DEV TAP_IP VM_IP VM_MASK VCPUS MEM_MB BUCKET LAMBDA_FN LAMBDA_BUCKET FC_VERSION CI_TRACK ARCH LOCALSTACK_AUTH_TOKEN
+
+SSH := ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR \
+         -o BatchMode=yes -o ConnectTimeout=10 \
+         -i $(WORK_DIR)/images/id_rsa root@$(VM_IP)
+
+.PHONY: help download rootfs up test logs down clean ssh diagnose
+
+help: ## Show available targets
+	@grep -E '^[a-zA-Z_-]+:.*##' $(MAKEFILE_LIST) | \
+	  awk 'BEGIN{FS=":.*##"}{printf "  %-10s %s\n", $$1, $$2}'
+
+download: ## Fetch the firecracker binary, guest kernel and base rootfs
+	@bash scripts/download-assets.sh
+
+rootfs: download ## Build a LocalStack-flavored guest rootfs image
+	@bash scripts/build-rootfs.sh
+
+up: rootfs ## Create the tap device and boot the LocalStack microVM
+	@bash scripts/run-vm.sh
+
+test: ## Run S3 + Lambda smoke tests against the running LocalStack instance
+	@bash scripts/smoke-test.sh
+
+logs: ## Tail the microVM's console/boot log
+	@tail -n 100 -f $(WORK_DIR)/firecracker.log
+
+ssh: ## SSH into the running microVM as root
+	@$(SSH)
+
+diagnose: ## Dump docker/localstack service status + logs from inside the microVM
+	@$(SSH) 'systemctl --no-pager status docker.service localstack.service; \
+	  echo ---localstack.service exit info---; \
+	  systemctl show localstack.service -p Result -p ExecMainStatus -p ExecMainCode; \
+	  echo ---docker.journal---; journalctl -u docker --no-pager -n 100; \
+	  echo "---localstack.journal (single attempt, Restart=no)---"; \
+	  journalctl -u localstack --no-pager'
+
+down: ## Stop the microVM and remove the tap device
+	@bash scripts/teardown.sh
+
+clean: down ## Remove all downloaded/built artifacts
+	rm -rf $(WORK_DIR)

ls-on-firecracker/README.md

@@ -0,0 +1,48 @@
+# LocalStack on Firecracker
+
+A demo that boots [LocalStack](https://localstack.cloud) inside a
+[Firecracker](https://firecracker-microvm.github.io/) microVM — the same
+technology AWS Lambda itself runs on — and exercises it over the network with
+an S3 bucket and a Lambda function that itself talks back to S3.
+
+## Quick start
+
+```
+make up      # download -> build a LocalStack-flavored rootfs -> boot the microVM
+make test    # S3 round-trip + Lambda deploy/invoke through it
+make down    # stop the VM
+```
+
+Run `make help` for the full target list, including `make ssh` and
+`make diagnose` for poking around inside the running microVM.
+
+## Prerequisites
+
+- A **Linux host with KVM** (`/dev/kvm`) — doesn't run on macOS or most cloud
+  VMs without nested virtualization. Bare metal, an EC2 `.metal` instance, or
+  a GitHub Actions `ubuntu-latest` runner all work; see
+  `.github/workflows/test-ls-on-firecracker.yml` for a working CI setup.
+- `curl`, `iproute2`, `iptables`, `e2fsprogs`, `zip`, `jq`, `ssh`, and the
+  AWS CLI on the host, plus `sudo` access.
+- A `LOCALSTACK_AUTH_TOKEN` environment variable set to a LocalStack **CI
+  Auth Token** — get one from
+  [your LocalStack workspace](https://app.localstack.cloud/workspace/auth-tokens).
+
+## Layout
+
+```
+Makefile               self-describing entry point (make help)
+scripts/
+  download-assets.sh   fetch firecracker + lstk + kernel + base rootfs
+  build-rootfs.sh       install Docker + lstk into a working rootfs image
+  run-vm.sh             set up networking (incl. NAT) and boot the microVM
+  smoke-test.sh         S3 round-trip + Lambda deploy/invoke against it
+  teardown.sh           stop the VM and remove the tap device / NAT rules
+fixtures/
+  handler.py            the demo Lambda function
+docs/
+  NOTES.md              how it works under the hood, and known caveats
+work/                   downloaded/generated artifacts (git-ignored)
+```
+
+See [docs/NOTES.md](docs/NOTES.md) for the details.

ls-on-firecracker/docs/NOTES.md

@@ -0,0 +1,66 @@
+# How it works
+
+1. **`make download`** grabs the `firecracker` binary, LocalStack's own
+   [`lstk`](https://docs.localstack.cloud/aws/developer-tools/running-localstack/lstk/)
+   CLI, a guest kernel and a base Ubuntu rootfs from Firecracker's public CI
+   artifacts.
+2. **`make rootfs`** clones the base rootfs, grows it, and chroots in to
+   install Docker and drop in the `lstk` binary, then registers `systemd`
+   units so Docker and then `lstk start` come up on boot. Building happens on
+   the host via a loop-mounted image, so the customization step itself
+   doesn't need the guest to be running.
+3. **`make up`** creates a tap network device on the host, NATs the guest out
+   through the host's default interface (`lstk` needs to pull the LocalStack
+   image, and LocalStack itself needs to pull the Lambda runtime image at
+   invoke time), and boots the image with Firecracker. It polls
+   `http://<vm-ip>:4566/_localstack/health` until LocalStack is ready.
+4. **`make test`** creates an S3 bucket and round-trips an object, then
+   deploys a small Python Lambda function and invokes it. The function
+   itself creates a *second* bucket and lists all buckets via `boto3`
+   (LocalStack injects `AWS_ENDPOINT_URL` into the Lambda execution
+   environment automatically, so no endpoint code is needed) — proving the
+   Lambda's own AWS calls land on the same LocalStack backend as the CLI
+   calls above: it sees the first bucket, and the one it creates is visible
+   back on the CLI afterward. `lstk` runs LocalStack as a container against
+   the guest's own Docker daemon, which is also what LocalStack itself uses
+   to spawn the Lambda executor container — the same two-layer shape real
+   Lambda uses (a container runtime inside a Firecracker microVM).
+5. **`make down`** / **`make clean`** tear the VM, NAT rules, and tap device
+   down again.
+
+`make ssh` drops you into a root shell on the running microVM (the base
+image ships a pre-authorized SSH key, fetched by `download-assets.sh`
+alongside the kernel/rootfs). `make diagnose` dumps `systemctl status` and
+`journalctl` for the `docker`/`localstack` services — the same thing the CI
+workflow does automatically on failure.
+
+## What's actually running
+
+Docker runs *inside* the guest OS (installed at rootfs-build time), and
+`lstk` uses it to pull and run the LocalStack container, which in turn uses
+the same Docker daemon as its normal Docker-based Lambda executor. The guest
+needs internet access both to pull the LocalStack image and, at Lambda
+invoke time, the runtime image — which is why `run-vm.sh` sets up NAT
+through the host rather than an isolated host-only network.
+
+## Caveats
+
+The base rootfs comes from Firecracker's own **CI test artifacts** — what
+their integration tests boot, not a general-purpose image. It's stripped
+down accordingly (no `/var/cache/apt`, `/var/log`, or populated dpkg
+database out of the box; `build-rootfs.sh` reconstructs what apt/Docker
+need), and its kernel is minimal too (no loadable modules, no `nf_tables`,
+no `CONFIG_IP_NF_RAW`) — a few of the fixes in `build-rootfs.sh` exist
+specifically to work around that (switching Docker to the legacy iptables
+backend, and opting out of a Docker 28+ hardening rule that needs a table
+this kernel doesn't have via `DOCKER_INSECURE_NO_IPTABLES_RAW=1`, which is
+fine for a single-tenant, ephemeral microVM but not something to carry into
+a shared or long-lived host).
+
+A more idiomatic base for "run a Docker image as a Firecracker rootfs" would
+be `docker export`-ing a real image (e.g. `ubuntu:22.04`) onto a formatted
+ext4 device rather than patching Firecracker's CI artifact. For running
+actual container workloads inside Firecracker in production, see
+[firecracker-containerd](https://github.com/firecracker-microvm/firecracker-containerd)
+(what AWS Lambda/Fargate use) instead of a full Docker-in-VM setup like this
+one.

ls-on-firecracker/fixtures/handler.py

@@ -0,0 +1,17 @@
+import boto3
+
+# LocalStack injects AWS_ENDPOINT_URL into the Lambda execution environment
+# automatically, and boto3 has respected that variable since 1.28.0 -- no
+# endpoint_url override needed here, on LocalStack or on real AWS.
+s3 = boto3.client("s3")
+
+
+def handler(event, context):
+    name = event.get("name", "world")
+    bucket = event.get("bucket")
+
+    if bucket:
+        s3.create_bucket(Bucket=bucket)
+
+    buckets = [b["Name"] for b in s3.list_buckets()["Buckets"]]
+    return {"message": f"hello {name}", "buckets": buckets}