Share one snapshot between runs that differ only in the balloon target by ejc3 · Pull Request #1060 · ejc3/fcvm

ejc3 · 2026-10-04T09:05:39Z

Runs that differ only in --balloon now share the pre-start snapshot, and a cache hit runs at its own target (#1053). The startup snapshot is kept per target.

Follows: #1055, which is merged. This branch is based on its head and targets main.

The Problem

#1054 put the --balloon value in the snapshot key, because nothing set a target on a restored VM. That made the cache correct and fragmented it: every target cold-boots once and keeps its own pre-start and startup snapshots, although the guests differ in one number the VMM can change at any time.

The Solution

Key. FirecrackerConfig::balloon is Option<BalloonDevice>, and BalloonDevice keeps its target out of the JSON the key hashes ("balloon":{}). Whether a device exists stays in the key, because one cannot be added to a restored VM. Keys without --balloon do not move: test_snapshot_key_golden is untouched. Keys with --balloon change once, so each such configuration misses once, and fcvm snapshots prune removes the old entries. Neither type derives Deserialize any more (nothing reads them back, and a reader would get target 0 for every device). What only that derive used is gone with it: two serde default helpers, and default in the 28 skip_serializing_if attributes.
Restore. podman run hands its --balloon to the restore through an internal SnapshotRunArgs field (no new flag on fcvm snapshot run). restore_from_snapshot sends PATCH /balloon after the snapshot load and before the resume, raising or lowering the target the snapshot was made with. A target that cannot be set ends the run. It is not a snapshot load failure, so there is no fallback to a cold boot. Every such failure says what was being done; when Firecracker answered 400 it also says what Firecracker refuses and names --no-snapshot. A target above --mem is refused before the cache is looked up, so a hit and a miss refuse the same run the same way.
Startup snapshots are per target (the second commit, from the review of the first). A startup snapshot (--health-check) holds a workload that initialized under the target of the run that made it, sized by the memory that guest had. Setting the target after the load does not replay that, so restore cannot reconcile it and the snapshot is not shared. Its name carries the target: <key>-balloon<MIB>-startup, and <key>-startup for a run without --balloon, as before. startup_snapshot_key takes the run's balloon, so the lookup and both places that save (the podman run loop after a cold boot, the restore loop after a pre-start hit) name the same snapshot. A run at a target that has none restores the shared pre-start snapshot, sets its target, starts the workload under it and saves a startup snapshot for that target. No cold boot is added. podman prepare installs a startup snapshot, so its content key carries the target the same way, and a --tag prepared at another target is rebuilt like any other changed content key.
The record follows the VMM. The VMM is the authority on its balloon device: the target can be changed on the VM's API socket after boot (the planned fcvm balloon command does exactly that and writes no state), so a record copied from a state file can differ from the device. The record is what GET /vm/config reports. That call answers 200 with or without a device. "No device" is the reply's "balloon": null, which both pinned Firecracker builds send (their reply serializes the Option with no skip), or a reply without the member, which the client reads the same way since the third commit. Any refusal is a failure, and a VM without a balloon gets no error line in its firecracker.log for the read.
- A memory snapshot reads it inside create_snapshot_core, with the VM paused and right before the save. That covers snapshot create and podman run's own pre-start and startup snapshots, and the record is the target the saved device holds.
- A disk-only capture has no pause and reads it under the per-VM snapshot lock.
- Every Firecracker restore reads it before the resume, after the PATCH if there is one. The read stays before the resume: there it cannot race the guest or another caller of the API socket, and it costs about 0.2 ms (139 to 258 us measured) of a restore whose snapshot load takes about 55 ms.
Cloud Hypervisor. fcvm's client for it has no resize call and that backend is a snapshot-cache opt-out, so no caller names a target there. restore_from_snapshot_ch refuses one, by its number, before it prepares the clone, so a later change cannot drop a target without a word. A Cloud Hypervisor clone keeps the snapshot's record, and its snapshots copy it.

One limit (also in DESIGN.md):

A guest that never activated the device. Firecracker refuses a target for a balloon the guest never activated (a custom kernel without the virtio balloon driver). Such a configuration cold-boots once and then fails on every cache hit, because a target that cannot be set ends the run. On an NV2 profile its first run fails too: a miss there tears the cold-booted VM down and restores its own snapshot with the caller's target. The error says what Firecracker refuses and names --no-snapshot, which runs it.

Not changed:

run_args_from_snapshot_metadata keeps its balloon parameter. Finding 6 of the A rebooted clone and a disk-only clone keep the source VM's balloon device #1055 review said it always equals meta.balloon_mib. After this PR it does not: the reboot site passes the clone's state, which is what the VMM reported at restore and can differ from the snapshot's record.
A disk-only clone has no restored VM to ask and still cold-boots from the snapshot's record.
No snapshot_cache_opt_out entry is added, and fc-agent is not touched.

Test Results

Each test was watched failing without its subject. The trees for the first commit: a scaffold with the types, parameters and tests and none of the behaviour; the source as it stood before a round of changes, with that round's tests added; and the commit with one statement mutated. For the second commit: the first commit with the new VM test added, and the second commit with the startup name ignoring the target.

# scaffold, make _test-unit FILTER="-E 'test(/.../)'"
  FAIL commands::podman::tests::the_snapshot_key_says_whether_a_balloon_exists_not_its_target
      runs that differ only in the balloon target must share a snapshot   left: "5f8216e1747d"   right: "7d46cc9d7eb6"
  FAIL firecracker::config::tests::the_key_json_names_a_balloon_device_without_its_target
      left: Some(Object {"target_mib": Number(0)})   right: Some(Object {})
  FAIL commands::podman::tests::a_cache_restore_asks_for_this_runs_balloon_target
      left: None   right: Some(512)
  FAIL commands::podman::tests::a_balloon_target_above_the_guests_memory_is_refused
      a 1025 MiB balloon in a 1024 MiB guest was accepted
  FAIL test_hypervisor_api firecracker_balloon_target_update_sends_the_target_alone   (the client mutated to send the three-member `Balloon`)
      left: Object {"amount_mib": Number(512), "deflate_on_oom": Bool(true), "stats_polling_interval_s": Number(1)}   right: Object {"amount_mib": Number(512)}

# the source before the record was read from GET /vm/config, with the tests for that
  FAIL commands::snapshot::tests::snapshot_create_reads_the_balloon_under_the_vm_snapshot_lock
      snapshot create reads the VM's balloon before it holds the per-VM snapshot lock
  FAIL commands::common::tests::the_restore_sets_the_balloon_target_between_load_and_resume
      the Firecracker restore has no `.balloon_target_mib()`
  FAIL commands::common::tests::a_cloud_hypervisor_restore_refuses_a_balloon_target_before_it_prepares_the_clone
      the Cloud Hypervisor restore does not refuse a balloon target by its number

# the source before a memory snapshot read the balloon in the shared creator, with the tests for that
  FAIL commands::common::tests::a_memory_snapshot_reads_the_balloon_between_the_pause_and_the_save
      create_snapshot_core has no `.balloon_target_mib()`
  FAIL test_balloon_call_sites every_statement_that_sets_or_reads_a_restored_vms_balloon_still_does
      1 of 4 statements that set or read a restored VM's balloon are gone ... a memory snapshot records the device the VMM has

# this commit, one statement mutated
  the client reading any failure of GET /vm/config as "no device":
  FAIL test_hypervisor_api firecracker_vm_config_reports_the_balloon_device_or_none
      called `Result::unwrap_err()` on an `Ok` value: None
  the PATCH failure explained for every failure:
  FAIL commands::common::tests::a_failed_balloon_target_explains_a_refusal_and_nothing_else
      setting the balloon target to 512 MiB on the restored VM (Firecracker refuses a target for a device the guest never activated, ...): Firecracker API error: 500
  the client's 400 not recognisable as a refusal:
  FAIL commands::common::tests::a_failed_balloon_target_explains_a_refusal_and_nothing_else
      setting the balloon target to 512 MiB on the restored VM: Firecracker API error: 400 Bad Request - the reason
  FAIL test_hypervisor_api firecracker_balloon_target_update_sends_the_target_alone
      a 400 is not recognisable as a refusal

# second commit, the startup name ignoring the target
  FAIL commands::podman::tests::two_balloon_targets_share_the_pre_start_snapshot_and_not_the_startup_one
      a 96 MiB run must not find the startup snapshot a 64 MiB run left   left: "849babc9270a-startup"   right: "849babc9270a-startup"

# third commit, the client's old declaration of the reply's `balloon` field
  FAIL firecracker::api::tests::a_vm_config_reply_with_the_balloon_absent_or_null_means_no_device
      left: Err("missing field `balloon` at line 1 column 13")   right: Ok(None)
  FAIL test_hypervisor_api firecracker_vm_config_reports_the_balloon_device_or_none
      missing field `balloon` at line 1 column 13
# and on the third commit
      Summary [   0.014s] 2 tests run: 2 passed, 1433 skipped

VM tests, make _test-root FILTER="--retries 0 -E 'test(=...)'" STREAM=1:

# scaffold: the key still holds the target, so run 2 misses
Error: run 2 differs from run 1 in --balloon alone and must restore from run 1's pre-start snapshot (lineage vm-f0be5f2a93f44949b4f3977b6f84e76d); its lineage is None, so it cold-booted
        FAIL test_balloon_target_honored_on_snapshot_cache_hit

# this commit with the PATCH removed: both hits stay at the snapshot's target
Error: run 2 asked for a 128 MiB balloon; 120s after its device could first be read, the target is 64 MiB and the size 64 MiB
run 3 asked for a 32 MiB balloon; 120s after its device could first be read, the target is 64 MiB and the size 64 MiB
        FAIL test_balloon_target_honored_on_snapshot_cache_hit

# both reads of the VMM removed, before the snapshot's read moved into the shared creator (the restore's read is unchanged since)
Error: run 2's state records balloon target Some(64), not the Some(128) it asked for
        FAIL test_balloon_target_honored_on_snapshot_cache_hit
Error: the snapshot of a VM whose balloon was set to 96 MiB after it booted at 64 records Some(64), not the Some(96) its VMM has
the state of a clone whose device was restored at 96 MiB from a snapshot that records 64 records Some(64), not the Some(96) its VMM has
        FAIL test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has

# the first commit with the new test: run 2 restores the startup snapshot run 1 left at another target
Error: run 2 differs from run 1 in --balloon alone (96 MiB, not 64) and has to restore the pre-start snapshot 58db22792d0e; it chose Startup("58db22792d0e-startup") (run 1's startup snapshot is 58db22792d0e-startup)
run 2 at 96 MiB and run 1 at 64 MiB name one startup snapshot, 58db22792d0e-startup
        FAIL test_startup_snapshot_is_per_balloon_target

Green, on the third commit (one VM with a balloon and one without, each snapshotted and restored through the changed read):

$ make _test-root FILTER="--retries 0 -E 'test(=test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has) | test(=test_restored_clone_reboot_comes_back_healthy) | test(=test_disk_only_clone_keeps_the_balloon)'" STREAM=1
        PASS [  31.640s] fcvm::test_disk_only_snapshot test_disk_only_clone_keeps_the_balloon
        PASS [  33.511s] fcvm::test_reboot test_restored_clone_reboot_comes_back_healthy
        PASS [  23.851s] fcvm::test_snapshot_clone test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has
     Summary [  89.010s] 3 tests run: 3 passed, 1786 skipped

$ make _test-unit (the module filter below)
     Summary [   8.182s] 545 tests run: 545 passed, 890 skipped

Green, on the second commit:

$ make _test-root FILTER="--retries 0 -E 'test(=test_startup_snapshot_is_per_balloon_target) | test(=test_balloon_target_honored_on_snapshot_cache_hit) | test(=test_startup_snapshot_key_generation)'" STREAM=1
        PASS [  24.346s] fcvm::test_snapshot_clone test_balloon_target_honored_on_snapshot_cache_hit
        PASS [  27.605s] fcvm::test_snapshot_clone test_startup_snapshot_is_per_balloon_target
        PASS [   0.025s] fcvm::test_startup_snapshot test_startup_snapshot_key_generation
     Summary [  51.985s] 3 tests run: 3 passed, 1785 skipped

# what each of the four runs of test_startup_snapshot_is_per_balloon_target logged (64, 96, 64, 96 MiB)
run 1 (64 MiB): Snapshot miss, will create snapshot after image load snapshot_key=9ef576e3083b
run 1 (64 MiB): Startup snapshot created successfully snapshot_key=9ef576e3083b-balloon64-startup
run 2 (96 MiB): Pre-start snapshot hit! Restoring from cached snapshot snapshot_key=9ef576e3083b
run 2 (96 MiB): Startup snapshot created successfully snapshot_key=9ef576e3083b-balloon96-startup
run 3 (64 MiB): Startup snapshot hit! Restoring from fully-initialized snapshot snapshot_key=9ef576e3083b-balloon64-startup
run 4 (96 MiB): Startup snapshot hit! Restoring from fully-initialized snapshot snapshot_key=9ef576e3083b-balloon96-startup

$ make _test-unit FILTER="-E 'binary(/^test_(hypervisor_api|balloon_call_sites|documented_make_targets|bench_fixtures|nested_harness_invariants|readme_examples|log_scan|ci_workflow_coverage|ami_hash_inputs|default_kernel_release|fuzz_chaos)/) | test(/^firecracker::|^hypervisor::|^commands::|^cli::|^state::|^storage::/)'"
     Summary [   8.084s] 544 tests run: 544 passed, 890 skipped

$ make fmt      # leaves the tree unchanged
$ make clippy   # clean

Green, on the first commit:

$ make _test-root FILTER="-E 'test(=test_balloon_target_honored_on_snapshot_cache_hit) | test(=test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has) | test(=test_restored_clone_reboot_keeps_its_balloon) | test(=test_disk_only_clone_keeps_the_balloon) | test(=test_restored_clone_reboot_comes_back_healthy)'" STREAM=1
        PASS [  32.065s] fcvm::test_disk_only_snapshot test_disk_only_clone_keeps_the_balloon
        PASS [  33.985s] fcvm::test_reboot test_restored_clone_reboot_comes_back_healthy
        PASS [  35.255s] fcvm::test_reboot test_restored_clone_reboot_keeps_its_balloon
        PASS [  25.615s] fcvm::test_snapshot_clone test_balloon_target_honored_on_snapshot_cache_hit
        PASS [  22.567s] fcvm::test_snapshot_clone test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has
     Summary [ 149.498s] 5 tests run: 5 passed, 1781 skipped

$ make _test-unit FILTER="-E 'binary(/^test_(hypervisor_api|balloon_call_sites|documented_make_targets|bench_fixtures|nested_harness_invariants|readme_examples|log_scan|ci_workflow_coverage|ami_hash_inputs|default_kernel_release|fuzz_chaos)/) | test(/^firecracker::|^hypervisor::|^commands::|^cli::|^state::|^storage::/)'"
     Summary [   8.134s] 543 tests run: 543 passed, 890 skipped

$ make fmt      # leaves the tree unchanged
$ make clippy   # clean

What GET /vm/config reports was checked in both Firecracker refs fcvm pins (the default profile's and the nested profile's): the runtime handler builds the reply from the live devices, and the balloon's amount_mib is its current target, which a restore fills from the saved state and PATCH /balloon changes. The VM tests show it on the default profile's Firecracker: the record test reads 96 MiB, from a VM that booted at 64 and was set to 96, into its snapshot (read while the VM is paused for the save) and into the state of the clone restored from it; the cache-hit test's state is the read made right after each PATCH (128, then 32); test_restored_clone_reboot_comes_back_healthy restores a VM with no balloon and records none. The firecracker.log of a restored VM with no balloon holds one INFO line for the read (The API server received a Get request on "/vm/config".) and no error line.

Convergence after a PATCH on a loaded, paused VM, from the passing cache-hit run (x86_64, default profile's Firecracker, pre-start snapshot restored through the File backend, host load average about 45 on 192 CPUs):

run 2, from 64 to 128 MiB
08:55:58.751434  snapshot load completed duration_ms=27
08:55:58.751935  balloon target settled on the restored VM duration_us=256 balloon_mib=Some(128) set_by_caller=true
08:55:58.752256  VM resume completed duration_ms=0 total_snapshot_ms=27
balloon after run 2 started: first reading 220.280467ms after spawn (target 128 MiB, size 64 MiB); size reached 128 MiB 373.207727ms later, 12 polls in all

run 3, from 64 to 32 MiB
08:56:01.423176  snapshot load completed duration_ms=4
08:56:01.423683  balloon target settled on the restored VM duration_us=230 balloon_mib=Some(32) set_by_caller=true
08:56:01.423968  VM resume completed duration_ms=0 total_snapshot_ms=5
balloon after run 3 started: first reading 162.416723ms after spawn (target 32 MiB, size 64 MiB); size reached 32 MiB 209.001008ms later, 8 polls in all

Each first reading is within about 0.05 s of the resume, so the guest had inflated the balloon within about 0.4 s of first running and deflated it within about 0.25 s. Five earlier hits on the same host, at load averages of 120 to 350, took 1.19 to 1.99 s to inflate.

Firecracker's own log shows the snapshot's read where the code puts it: Paused, GET /vm/config, PUT /snapshot/create, in that order, for snapshot create and for podman run's pre-start snapshot.

Not run: arm64 and the nested profile's Firecracker, and with it the NV2 relaunch path, where every --balloon run is patched while the implicit UFFD server replays. Also not run: the PATCH on any UFFD-served restore (a hugepage restore uses the same implicit server), Cloud Hypervisor, the full make test-root, and cargo audit and cargo deny (the Cargo files are unchanged). The cache-hit test takes no --health-check: it restores from the pre-start snapshot, and nothing in it reads the container's HTTP port. The startup snapshot test takes one and uses routed networking. Also not run for the second commit: the existing startup snapshot tests (their image could not be pulled where this was written, and their rootless health check does not pass on a host with no IPv4, #1056), and podman prepare --balloon (its content key is the startup name, from the code).

Summary by CodeRabbit

New Features
- Added --balloon <MIB> support for restored virtual machines. Cache hits apply the requested target before the guest resumes, so runs with different targets can share a pre-start snapshot.
- Startup snapshots are kept separate for each balloon target.
Bug Fixes
- Balloon targets above guest memory are rejected before snapshot lookup.
- Snapshots and restored clones now reflect the target reported by the virtual machine.
- If a balloon update fails, the run stops with an error rather than falling back to a cold boot.
Documentation
- Clarified balloon behavior and limitations across snapshots, restores, and cache hits.

`podman run --balloon N` put N in the snapshot key, so every target cold-booted once and kept its own pre-start and startup snapshots (#1053). The key now says whether a balloon device exists and nothing about its target, and a restore sets the caller's target on the loaded VM before the guest resumes. Key. `FirecrackerConfig::balloon` is `Option<BalloonDevice>`, and `BalloonDevice` keeps its target out of the JSON the key hashes, so the config serializes as `"balloon":{}`. Whether the device exists stays in the key because a device cannot be added to a restored VM. Keys without --balloon are unchanged (`test_snapshot_key_golden` is untouched). Keys with --balloon change once, so each such configuration misses once; `fcvm snapshots prune` removes the old entries. Neither type derives `Deserialize` any more: nothing reads them back, and a reader would get target 0 for every device. What only that derive used is gone with it: two serde default helpers, and `default` in the 28 `skip_serializing_if` attributes. Restore. `podman run` hands its --balloon to the restore through an internal `SnapshotRunArgs` field. `restore_from_snapshot` sends `PATCH /balloon` after the snapshot load and before the resume, raising or lowering the target the snapshot was made with. A target that cannot be set ends the run: the error is not a snapshot load failure, so there is no fallback to a cold boot. Every such failure says what was being done; when Firecracker answered 400 it also says what Firecracker refuses and names --no-snapshot. A target above --mem is refused before the cache is looked up, so a hit and a miss refuse the same run the same way. A Cloud Hypervisor restore refuses a target before it prepares the clone; that backend is not in the snapshot cache and no caller names one. Record. The VMM is the authority on its balloon device: the target can be changed on the VM's API socket after boot, and nothing writes that to the VM's state. So the record is what `GET /vm/config` reports, not a copy of another record. That call answers 200 with or without a device, so a VM with no balloon is an answer and any refusal is a failure. A memory snapshot reads it inside `create_snapshot_core`, with the VM paused and right before the save, so `snapshot create` and podman's own pre-start and startup snapshots record the target the saved device holds. A disk-only capture has no pause and reads it under the per-VM snapshot lock. Every Firecracker restore reads it before the resume, after the PATCH if there is one. `balloon_stats()` is for the size the guest has reached, and `BalloonStats` carries it. A disk-only clone has no restored VM to ask and cold-boots from the snapshot's record. A Cloud Hypervisor clone keeps the snapshot's record, and its snapshots copy it. Limits, stated in DESIGN.md. Firecracker refuses a target for a device the guest never activated (a kernel without the virtio balloon driver), so such a configuration cold-boots once and then fails on every cache hit. On an NV2 profile its first run fails too, because a miss there tears the cold-booted VM down and restores its own snapshot with the caller's target. A startup-snapshot hit hands over a guest whose workload started under the target of the run that made the snapshot. Tested (x86_64, default kernel profile, rootless networking): make fmt leaves the tree unchanged make clippy clean make _test-unit FILTER="-E 'binary(/^test_(hypervisor_api|balloon_call_sites|documented_make_targets|bench_fixtures|nested_harness_invariants|readme_examples|log_scan|ci_workflow_coverage|ami_hash_inputs|default_kernel_release|fuzz_chaos)/) | test(/^firecracker::|^hypervisor::|^commands::|^cli::|^state::|^storage::/)'" 543 tests run: 543 passed make _test-root FILTER="-E 'test(=test_balloon_target_honored_on_snapshot_cache_hit) | test(=test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has) | test(=test_restored_clone_reboot_keeps_its_balloon) | test(=test_disk_only_clone_keeps_the_balloon) | test(=test_restored_clone_reboot_comes_back_healthy)'" STREAM=1 5 tests run: 5 passed, each on its first try Each test was watched failing without its subject. On a tree with the types, parameters and tests and none of the behaviour: the_snapshot_key_says_whether_a_balloon_exists_not_its_target keys differ: "5f8216e1747d" and "7d46cc9d7eb6" the_key_json_names_a_balloon_device_without_its_target Some(Object {"target_mib": Number(0)}) is not Some(Object {}) a_cache_restore_asks_for_this_runs_balloon_target None is not Some(512) a_balloon_target_above_the_guests_memory_is_refused a 1025 MiB balloon in a 1024 MiB guest was accepted firecracker_balloon_target_update_sends_the_target_alone red by mutation (the client sending the three-member `Balloon`) test_balloon_target_honored_on_snapshot_cache_hit (VM) "run 2 differs from run 1 in --balloon alone and must restore from run 1's pre-start snapshot ...; its lineage is None, so it cold-booted" On the source before the record was read from `GET /vm/config`, with the tests for that: snapshot_create_reads_the_balloon_under_the_vm_snapshot_lock snapshot create reads the VM's balloon before it holds the per-VM snapshot lock the_restore_sets_the_balloon_target_between_load_and_resume the Firecracker restore has no `.balloon_target_mib()` a_cloud_hypervisor_restore_refuses_a_balloon_target_before_it_prepares_the_clone the Cloud Hypervisor restore does not refuse a balloon target by its number On the source before a memory snapshot read the balloon inside `create_snapshot_core`, with the tests for that (4 run: 2 failed, 2 passed): a_memory_snapshot_reads_the_balloon_between_the_pause_and_the_save create_snapshot_core has no `.balloon_target_mib()` every_statement_that_sets_or_reads_a_restored_vms_balloon_still_does 1 of 4 statements gone: a memory snapshot records the device the VMM has On this tree with one thing mutated: the PATCH failure explained for every failure: a_failed_balloon_target_explains_a_refusal_and_nothing_else a 500 came back with the never-activated explanation and --no-snapshot the client's 400 not recognisable as a refusal: a_failed_balloon_target_explains_a_refusal_and_nothing_else a 400 came back with the neutral context only firecracker_balloon_target_update_sends_the_target_alone "a 400 is not recognisable as a refusal" the PATCH removed (VM): test_balloon_target_honored_on_snapshot_cache_hit "run 2 asked for a 128 MiB balloon; 120s after its device could first be read, the target is 64 MiB and the size 64 MiB" and "run 3 asked for a 32 MiB balloon; ... the target is 64 MiB and the size 64 MiB" Run before the last round of changes, on statements that round did not touch: the client reading any failure of GET /vm/config as "no device": firecracker_vm_config_reports_the_balloon_device_or_none called `Result::unwrap_err()` on an `Ok` value: None the restore's read of the VMM removed (VM): test_balloon_target_honored_on_snapshot_cache_hit "run 2's state records balloon target Some(64), not the Some(128) it asked for" test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has the clone's state records Some(64), not the Some(96) the VMM has (and, with the snapshot's read removed where it then was, so does the snapshot) In the passing cache-hit run the PATCH and the read together took 256 and 230 us on the paused VM. The guest inflated the balloon from 64 to 128 MiB within about 0.4 s of the resume and deflated it from 64 to 32 MiB within about 0.25 s (load average about 45 on 192 CPUs). Five earlier hits, at load averages of 120 to 350, took 1.19 to 1.99 s to inflate. The read alone took 139 to 258 us on restores that named no target. Firecracker's log shows Paused, GET /vm/config, PUT /snapshot/create in that order for `snapshot create` and for podman's pre-start snapshot, and a restored VM with no balloon has one INFO line for the read and no error line. Not run: arm64, the nested profile's Firecracker and with it the NV2 relaunch path (every --balloon run patched while the implicit UFFD server replays), the PATCH on any UFFD-served restore, Cloud Hypervisor, the full `make test-root`, `cargo audit` and `cargo deny` (the Cargo files are unchanged).

coderabbitai · 2026-10-04T09:05:50Z

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults
Review profile: CHILL
Plan: Advanced
Run ID: b6737c23-5df5-467a-8f20-f1d77d58960d

📥 Commits

Reviewing files that changed from the base of the PR and between fcd65ba and 71aa944.

📒 Files selected for processing (2)

src/firecracker/api.rs
tests/test_hypervisor_api.rs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Walkthrough

The change updates balloon configuration, snapshot identity, and restore handling. Pre-start snapshots can be shared across balloon targets. Startup snapshots use target-specific keys. Firecracker records the VMM-reported target and applies requested targets to paused restored VMs.

Changes

Balloon Snapshot and Restore Flow

Layer / File(s)	Summary
Balloon API and configuration `src/firecracker/api.rs`, `src/firecracker/config.rs`, `src/firecracker/mod.rs`, `tests/test_hypervisor_api.rs`	Firecracker balloon API support now updates and reads the target, preserves API refusal details, and reports actual balloon size. Configuration uses an optional balloon device and omits its target from serialized snapshot identity.
Cache identity and run configuration `.claude/CLAUDE.md`, `README.md`, `src/cli/args.rs`, `src/commands/podman/mod.rs`, `src/commands/podman/snapshot.rs`, `src/commands/podman/vm_config.rs`, `tests/common/mod.rs`, `tests/test_startup_snapshot.rs`	Balloon presence remains part of pre-start snapshot identity, while startup snapshot keys include the target. The run path validates the target against guest memory and passes it through launch and restore configuration.
Snapshot and restore target state `DESIGN.md`, `src/commands/common.rs`, `src/commands/snapshot.rs`, `tests/test_balloon_call_sites.rs`, `tests/test_snapshot_clone.rs`	Firecracker applies the requested target before resuming a restored VM and records the VMM-reported target in state and snapshot metadata. Cloud Hypervisor retains recorded values and rejects requested targets during restore. Tests cover cache hits, target-specific startup snapshots, and snapshot metadata.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Podman as podman run
  participant SnapshotCache as Snapshot cache
  participant FirecrackerRestore as Firecracker restore
  participant FirecrackerClient
  participant FirecrackerVMM as Firecracker VMM
  Podman->>SnapshotCache: Find snapshot using device-presence key
  SnapshotCache->>FirecrackerRestore: Load matching snapshot
  FirecrackerRestore->>FirecrackerClient: Set requested balloon target
  FirecrackerClient->>FirecrackerVMM: PATCH /balloon
  FirecrackerRestore->>FirecrackerClient: Read balloon target
  FirecrackerClient->>FirecrackerVMM: GET /vm/config
  FirecrackerRestore->>FirecrackerVMM: Resume guest

Merge Risk: ⚪ Minimal · up to 71aa9

No concrete merge-blocking regression is established. The unrun platform and test-matrix items remain validation follow-ups, not evidence of a defect.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 71aa9

Requested balloon targets are applied before restored guests resume, and failures abort rather than silently using a different target. No introduced security vulnerability was established. Remaining uncertainty concerns concurrent direct control-socket updates and interruption outside the normal command lifecycle.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

inferred — The newly exercised mutation is scoped to the restored clone's VMM device. It does not rewrite the shared snapshot's target. Concurrent restores retain the existing shared snapshot-directory lock while opening or copying their inputs; wider host or cross-tenant authority was not established by the reviewed paths.

Trust Boundaries and Controls

observed — The caller-supplied numeric target reaches a fixed local VMM endpoint, and Firecracker remains authoritative for device acceptance and reported state. The public Rust methods are client wrappers, not listening network entrypoints. Deployment-level socket permissions were not established by this review.

Resilience and Maintainability Implications

observed — Ordinary restore failures explicitly kill and reap Firecracker and clean up the namespace holder. The production caller awaits restore directly while signal handling records cancellation, then transfers setup owners exactly once on success. This supports normal failure containment but does not prove cleanup for every possible library-level future drop.

Hardening Proposals

proposed — If independent direct-socket balloon updates are supported during capture, define coordination with snapshot ownership or provide an atomic capture contract. The inspected snapshot lock serializes snapshot creators, but does not itself establish exclusion of direct API writers. This is a hardening proposal, not an established PR-introduced vulnerability.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 88.89% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 72 functions across 14 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: sharing a pre-start snapshot between runs that differ only in balloon target.

✨ Finishing Touches

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

chatgpt-codex-connector · 2026-10-04T09:05:57Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-04T15:18:10.653186Z	`71aa944`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b323f1ec07

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

… shared Runs that differ only in --balloon share one snapshot key, and a cache hit sets its own target on the loaded VM. That is right for the pre-start snapshot, which is taken before the workload starts. It was wrong for the startup snapshot (--health-check): a run at another target restored a workload that had initialized under the creator's target, sized by the memory that guest had, and setting the target after the load does not replay that. The startup snapshot behaves differently for such a run and restore cannot reconcile it, so by the reuse rule it must not be shared. The startup snapshot's name now carries the target: <key>-balloon<MIB>-startup a run with --balloon <MIB> <key>-startup a run without --balloon, as before `startup_snapshot_key` takes the run's balloon, so each place that derives the name has to say which target it means: the lookup in `podman run` and `podman prepare`, the save in the `podman run` loop (a cold boot that turned healthy) and the save in the restore loop (a pre-start hit that turned healthy). A run at a target with no startup snapshot restores the shared pre-start snapshot, sets its target, starts the workload under it and saves a startup snapshot for that target. No cold boot is added, and the snapshot key is unchanged (`test_snapshot_key_golden` is untouched). `podman prepare` installs a startup snapshot, so its content key carries the target the same way, and a --tag whose generation was prepared at another target is rebuilt like any other changed content key. That follows from the code (the content key is the startup name); no prepare was run. Nothing else derives the name. `snapshots prune` selects by snapshot type, the diff parent is read from the VM's state, and the one matcher on the suffix (benches/hugepages.rs, `ends_with("-startup")`) still matches. The cache-hit test's helper made up a `<key>-startup` name for runs that take no health check; it now returns the pre-start key those runs record. Tested, each test first without the change: make _test-unit FILTER="-E 'test(/two_balloon_targets_share_the_pre_start|test_snapshot_key_golden|the_snapshot_key_says_whether_a_balloon/)'" with the name ignoring the target: FAIL commands::podman::tests::two_balloon_targets_share_the_pre_start_snapshot_and_not_the_startup_one a 96 MiB run must not find the startup snapshot a 64 MiB run left left: "849babc9270a-startup" right: "849babc9270a-startup" on this commit: Summary [ 0.016s] 3 tests run: 3 passed, 1431 skipped make _test-root FILTER="--retries 0 -E 'test(=test_startup_snapshot_is_per_balloon_target)'" STREAM=1 on the parent commit with the test added: Error: run 2 differs from run 1 in --balloon alone (96 MiB, not 64) and has to restore the pre-start snapshot 58db22792d0e; it chose Startup("58db22792d0e-startup") (run 1's startup snapshot is 58db22792d0e-startup) run 2 at 96 MiB and run 1 at 64 MiB name one startup snapshot, 58db22792d0e-startup FAIL test_startup_snapshot_is_per_balloon_target on this commit, together with test_balloon_target_honored_on_snapshot_cache_hit and test_startup_snapshot_key_generation: PASS [ 24.346s] fcvm::test_snapshot_clone test_balloon_target_honored_on_snapshot_cache_hit PASS [ 27.605s] fcvm::test_snapshot_clone test_startup_snapshot_is_per_balloon_target PASS [ 0.025s] fcvm::test_startup_snapshot test_startup_snapshot_key_generation Summary [ 51.985s] 3 tests run: 3 passed, 1785 skipped and what the four runs logged: run 1 (64 MiB): Snapshot miss, will create snapshot after image load snapshot_key=9ef576e3083b run 1 (64 MiB): Startup snapshot created successfully snapshot_key=9ef576e3083b-balloon64-startup run 2 (96 MiB): Pre-start snapshot hit! Restoring from cached snapshot snapshot_key=9ef576e3083b run 2 (96 MiB): Startup snapshot created successfully snapshot_key=9ef576e3083b-balloon96-startup run 3 (64 MiB): Startup snapshot hit! Restoring from fully-initialized snapshot snapshot_key=9ef576e3083b-balloon64-startup run 4 (96 MiB): Startup snapshot hit! Restoring from fully-initialized snapshot snapshot_key=9ef576e3083b-balloon96-startup make _test-unit on the changed modules and the test binaries that read the files this change edits: Summary [ 8.084s] 544 tests run: 544 passed, 890 skipped make fmt leaves the tree unchanged. make clippy is clean. Not run: the existing startup snapshot tests (they pull an image this host cannot reach, and their rootless health check does not pass on a host with no IPv4, #1056), arm64, the nested profile, `podman prepare --balloon`, and the full `make test-root`. The new VM test uses routed networking, which is the mode whose health check passes on the host it was written on.

ejc3 · 2026-10-04T10:34:11Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fcd65badbf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

`balloon_target_mib()` reads `GET /vm/config` on every snapshot and every restore. Its `balloon` field was declared so that a reply without the member was an error ("missing field `balloon`"): only `null` meant "no device". A Firecracker that leaves the member out for a VM with no balloon would have failed every snapshot and restore of an ordinary VM. The two Firecracker builds this repository pins do not leave it out. In both, `VmmConfig::balloon` is an `Option` with no `skip_serializing_if`, filled from `resources.balloon.get_config().ok()` and written with `serde_json::to_string`, so a VM with no balloon gets `"balloon":null`. Upstream v1.14.0 declares it the same way. That is why the VM tests with no balloon passed: with the old declaration an absent member would have failed their restores. The client now takes both. The field is a plain `Option` with `#[serde(default)]`, so a null member and an absent one read as None. What stays an error: a refusal (any status that is not success), and a device that does not say its target. Tested, first without the change: make _test-unit FILTER="-E 'test(/a_vm_config_reply_with_the_balloon_absent_or_null/) | test(=firecracker_vm_config_reports_the_balloon_device_or_none)'" with the old declaration (2 tests run: 0 passed, 2 failed, 1433 skipped): FAIL firecracker::api::tests::a_vm_config_reply_with_the_balloon_absent_or_null_means_no_device left: Err("missing field `balloon` at line 1 column 13") right: Ok(None) FAIL test_hypervisor_api firecracker_vm_config_reports_the_balloon_device_or_none missing field `balloon` at line 1 column 13 on this commit: Summary [ 0.014s] 2 tests run: 2 passed, 1433 skipped make _test-root FILTER="--retries 0 -E 'test(=test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has) | test(=test_restored_clone_reboot_comes_back_healthy) | test(=test_disk_only_clone_keeps_the_balloon)'" STREAM=1 (one VM with a balloon and one without, each snapshotted and restored) PASS [ 31.640s] fcvm::test_disk_only_snapshot test_disk_only_clone_keeps_the_balloon PASS [ 33.511s] fcvm::test_reboot test_restored_clone_reboot_comes_back_healthy PASS [ 23.851s] fcvm::test_snapshot_clone test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has Summary [ 89.010s] 3 tests run: 3 passed, 1786 skipped make _test-unit on the changed modules and the test binaries that read the files this change edits: Summary [ 8.182s] 545 tests run: 545 passed, 890 skipped make fmt leaves the tree unchanged. make clippy is clean. Not run: arm64, the nested profile's Firecracker, and the full make test-root. No Firecracker that omits the member was run; none is pinned.

fcvm balloon (--pid PID | --name NAME) [MIB] prints one JSON line, `{"target_mib":96,"actual_mib":64}`: the target the device was asked to reach and the size the guest has given it so far. With MIB it sets the target first (`PATCH /balloon`) and prints right after, so the size is still on its way. Until now the target of a running VM could be changed only by talking to Firecracker's API socket by hand. Refusals. A Cloud Hypervisor VM is refused by name and a target above the VM's memory with both numbers, each from the VM's state and before any request. Whether the VM has a balloon device is read from `GET /vm/config`, which answers 200 either way, so a VM booted without --balloon is told it has no device and that one is attached only at boot. A set that Firecracker answers with a 400 (the typed `ApiRefusal`) says what Firecracker refuses; any other failure says only what was being done. The command writes no state. What follows the new target and what does not, from the code: - The next snapshot of the VM does. A memory snapshot (`snapshot create` and `podman run`'s own) and a disk-only capture read the device from the VMM. A clone of that snapshot records the target in its state, and its relaunch after a guest reboot attaches the device at it. The name of `podman run`'s own startup snapshot does not move: it carries the run's --balloon. - The VM's own state does not: `config.balloon_mib` keeps the target the VM booted with or, for a clone, the one its restore read. Nothing else writes that field. - The VM's own relaunch after a guest reboot does not: a cold-booted VM relaunches from its launch config and a restored clone from its state. Firecracker has exited by then, so there is nothing to ask. Also: `exec`'s lookup of a VM by --pid or --name moves to `commands::common::load_vm_state` and both commands use it; the text of what Firecracker refuses and the test for a 400 are shared with the restore's PATCH; `BalloonStats` serializes, since it is the command's output. Tested, each test first without its subject. The reds are from this change over the first commit of #1060; every green step ran again after the change moved onto its second commit. make _test-unit FILTER="-E 'test(/balloon_names_one_vm|balloon_mib_is_optional|a_cloud_hypervisor_vm_is_refused|a_target_above_the_vms_memory|a_vm_with_no_balloon_device|a_failed_set_explains|the_report_is_one_json_line|a_failed_balloon_target_explains/)'" the command's skeleton with none of its checks (8 tests run: 2 passed, 6 failed, 1432 skipped): FAIL cli::args::tests::balloon_names_one_vm_by_pid_or_by_name --pid and --name were accepted together FAIL commands::balloon::tests::a_cloud_hypervisor_vm_is_refused_by_name_before_any_request FAIL commands::balloon::tests::a_target_above_the_vms_memory_is_refused_before_any_request FAIL commands::balloon::tests::a_vm_with_no_balloon_device_is_told_where_one_comes_from FAIL commands::balloon::tests::a_failed_set_explains_a_refusal_and_nothing_else FAIL commands::balloon::tests::the_report_is_one_json_line MIB made a required argument: FAIL cli::args::tests::balloon_mib_is_optional a report needs no MIB: ErrorInner { kind: MissingRequiredArgument, ... on this commit: Summary [ 0.012s] 8 tests run: 8 passed, 1433 skipped make _test-root FILTER="--retries 0 -E 'test(=test_balloon_command_sets_and_reports_the_target)'" STREAM=1 with the PATCH and the device check removed: Error: `fcvm balloon --pid P 96` printed target 64 MiB after `fcvm balloon --pid P 96` Firecracker reports the device at target 64 MiB a snapshot taken after the target was set to 96 MiB records Some(64) and for the VM booted without --balloon, with and without MIB, Firecracker's own 400 Bad Request ... "Failed perform action on device: Device not found" FAIL test_balloon_command_sets_and_reports_the_target on this commit, with test_restored_clone_reboot_keeps_its_balloon and test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has: PASS [ 23.275s] fcvm::test_balloon test_balloon_command_sets_and_reports_the_target PASS [ 32.589s] fcvm::test_reboot test_restored_clone_reboot_keeps_its_balloon PASS [ 33.016s] fcvm::test_snapshot_clone test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has Summary [ 88.889s] 3 tests run: 3 passed, 1793 skipped make _test-unit on the changed modules and the test binaries that read the files this change edits: Summary [ 8.320s] 562 tests run: 562 passed, 879 skipped make fmt leaves the tree unchanged. make clippy is clean. Not run: arm64, the nested profile's Firecracker, a VM whose guest never activated the device (the 400 explanation is covered by unit tests), Cloud Hypervisor (its refusal is a unit test on the VM's state), and the full make test-root.

ejc3

NOT-A-DEFECT: the review bodies on this pull request carry no finding of their own. Each finding they announce is an inline thread, and each thread is answered there with the test that was watched failing.

ejc3 · 2026-10-04T15:13:11Z

@codex review

chatgpt-codex-connector · 2026-10-04T15:18:09Z

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 71aa94414b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ejc3

NOT-A-DEFECT: the review bot's summary comment was rewritten when this pull request's description changed, and it carries no finding. The findings on this pull request are inline threads, and each is answered there with the test that was watched failing.

chatgpt-codex-connector Bot reviewed Oct 4, 2026

View reviewed changes

Comment thread src/firecracker/config.rs

ejc3 mentioned this pull request Oct 4, 2026

Add fcvm balloon to read and set a running VM's balloon target #1061

Merged

chatgpt-codex-connector Bot reviewed Oct 4, 2026

View reviewed changes

Comment thread src/firecracker/api.rs Outdated

ejc3 commented Oct 4, 2026

View reviewed changes

ejc3 merged commit 565611c into main Oct 4, 2026
14 checks passed

ejc3 deleted the cache-hit-balloon-target branch October 4, 2026 16:07

Navigation Menu

Share one snapshot between runs that differ only in the balloon target - #1060

Share one snapshot between runs that differ only in the balloon target#1060
ejc3 merged 3 commits into
mainfrom
cache-hit-balloon-target

ejc3 commented Oct 4, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 4, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

chatgpt-codex-connector Bot commented Oct 4, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

ejc3 commented Oct 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

ejc3 left a comment

Uh oh!

ejc3 commented Oct 4, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 4, 2026

Uh oh!

ejc3 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ejc3 commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The Problem

The Solution

Test Results

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Security Architecture Review

Uh oh!

chatgpt-codex-connector Bot commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

ejc3 commented Oct 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

ejc3 left a comment

Choose a reason for hiding this comment

Uh oh!

ejc3 commented Oct 4, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 4, 2026

Uh oh!

ejc3 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ejc3 commented Oct 4, 2026 •

edited

Loading

coderabbitai Bot commented Oct 4, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 4, 2026 •

edited

Loading