Repository navigation
Add fcvm balloon to read and set a running VM's balloon target - #1061
Conversation
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (15)
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughAdds ChangesBalloon control
Podman startup snapshots
Priority: ⬇️ Low Estimated code review effort: 4 (Complex) | ~45 minutes Change: Feature Sequence Diagram(s)sequenceDiagram
actor User
participant CLI
participant BalloonCommand as fcvm balloon
participant VmState as VM state
participant SnapshotLock as Snapshot lock
participant Firecracker as Firecracker API
User->>CLI: Select VM and optionally provide target
CLI->>BalloonCommand: Dispatch command
BalloonCommand->>VmState: Load VM state
VmState-->>BalloonCommand: Return VM configuration
alt Target provided
BalloonCommand->>SnapshotLock: Acquire per-VM lock with timeout
BalloonCommand->>Firecracker: Check device and set target
Firecracker-->>BalloonCommand: Return balloon statistics
else No target provided
BalloonCommand->>Firecracker: Read balloon statistics
Firecracker-->>BalloonCommand: Return balloon statistics
end
BalloonCommand-->>User: Print target and actual size as JSON
Merge Risk: ⚪ Minimal · up to The balloon command and startup-snapshot changes are ready to merge after normal checks; no actionable issue remains identified. Security Architecture ReviewSecurity architecture risk: 🔵 Low · up to The new command uses the selected VM’s existing control socket, validates requests and coordinates target changes with snapshots. No introduced security bypass was established. Risk remains bounded but not minimal because deployed access permissions and interrupted-request behavior are not fully verified. Retained concerns Security review detailsSecurity Blast Radius
Trust Boundaries and Controls
Resilience and Maintainability Implications
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 02adee92f4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
fcvm balloon (--pid PID | --name NAME) [MIB]
prints one JSON line, `{"target_mib":96,"actual_mib":64}`: the target the
device was asked to reach and the size the guest has given it so far. With MIB
it sets the target first (`PATCH /balloon`) and prints right after, so the size
is still on its way. Until now the target of a running VM could be changed only
by talking to Firecracker's API socket by hand.
Refusals. A Cloud Hypervisor VM is refused by name and a target above the VM's
memory with both numbers, each from the VM's state and before any request.
Whether the VM has a balloon device is read from `GET /vm/config`, which
answers 200 either way, so a VM booted without --balloon is told it has no
device and that one is attached only at boot. A set that Firecracker answers
with a 400 (the typed `ApiRefusal`) says what Firecracker refuses; any other
failure says only what was being done.
The command writes no state. What follows the new target and what does not,
from the code:
- The next snapshot of the VM does. A memory snapshot (`snapshot create` and
`podman run`'s own) and a disk-only capture read the device from the VMM. A
clone of that snapshot records the target in its state, and its relaunch
after a guest reboot attaches the device at it. The name of `podman run`'s
own startup snapshot does not move: it carries the run's --balloon.
- The VM's own state does not: `config.balloon_mib` keeps the target the VM
booted with or, for a clone, the one its restore read. Nothing else writes
that field.
- The VM's own relaunch after a guest reboot does not: a cold-booted VM
relaunches from its launch config and a restored clone from its state.
Firecracker has exited by then, so there is nothing to ask.
Also: `exec`'s lookup of a VM by --pid or --name moves to
`commands::common::load_vm_state` and both commands use it; the text of what
Firecracker refuses and the test for a 400 are shared with the restore's PATCH;
`BalloonStats` serializes, since it is the command's output.
Tested, each test first without its subject. The reds are from this change over
the first commit of #1060; every green step ran again after the change moved
onto its second commit.
make _test-unit FILTER="-E 'test(/balloon_names_one_vm|balloon_mib_is_optional|a_cloud_hypervisor_vm_is_refused|a_target_above_the_vms_memory|a_vm_with_no_balloon_device|a_failed_set_explains|the_report_is_one_json_line|a_failed_balloon_target_explains/)'"
the command's skeleton with none of its checks (8 tests run: 2 passed, 6 failed, 1432 skipped):
FAIL cli::args::tests::balloon_names_one_vm_by_pid_or_by_name
--pid and --name were accepted together
FAIL commands::balloon::tests::a_cloud_hypervisor_vm_is_refused_by_name_before_any_request
FAIL commands::balloon::tests::a_target_above_the_vms_memory_is_refused_before_any_request
FAIL commands::balloon::tests::a_vm_with_no_balloon_device_is_told_where_one_comes_from
FAIL commands::balloon::tests::a_failed_set_explains_a_refusal_and_nothing_else
FAIL commands::balloon::tests::the_report_is_one_json_line
MIB made a required argument:
FAIL cli::args::tests::balloon_mib_is_optional
a report needs no MIB: ErrorInner { kind: MissingRequiredArgument, ...
on this commit:
Summary [ 0.012s] 8 tests run: 8 passed, 1433 skipped
make _test-root FILTER="--retries 0 -E 'test(=test_balloon_command_sets_and_reports_the_target)'" STREAM=1
with the PATCH and the device check removed:
Error: `fcvm balloon --pid P 96` printed target 64 MiB
after `fcvm balloon --pid P 96` Firecracker reports the device at target 64 MiB
a snapshot taken after the target was set to 96 MiB records Some(64)
and for the VM booted without --balloon, with and without MIB, Firecracker's own
400 Bad Request ... "Failed perform action on device: Device not found"
FAIL test_balloon_command_sets_and_reports_the_target
on this commit, with test_restored_clone_reboot_keeps_its_balloon and
test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has:
PASS [ 23.275s] fcvm::test_balloon test_balloon_command_sets_and_reports_the_target
PASS [ 32.589s] fcvm::test_reboot test_restored_clone_reboot_keeps_its_balloon
PASS [ 33.016s] fcvm::test_snapshot_clone test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has
Summary [ 88.889s] 3 tests run: 3 passed, 1793 skipped
make _test-unit on the changed modules and the test binaries that read the
files this change edits:
Summary [ 8.320s] 562 tests run: 562 passed, 879 skipped
make fmt leaves the tree unchanged. make clippy is clean.
Not run: arm64, the nested profile's Firecracker, a VM whose guest never
activated the device (the 400 explanation is covered by unit tests), Cloud
Hypervisor (its refusal is a unit test on the VM's state), and the full
make test-root.
A target set by `fcvm balloon` could land between a snapshot's read of the
balloon and its save. The shared creator reads `GET /vm/config` with the VM
paused and then saves it, and a pause does not keep a PATCH out: Firecracker
changes the device's target on a paused VM (that is how a restore sets a target
before the resume). The snapshot then recorded one target and saved a device at
another. Two sets could also interleave their PATCH and their statistics read,
so one reported the other's target.
Every snapshot path already holds the per-VM snapshot lock from that read to the
save: `snapshot create`, memory and disk-only, and `create_podman_snapshot`
(`podman run`'s pre-start and startup snapshots, and `podman prepare`'s) take it
before the read and keep it until they return. The command did not take it. A
set now takes that lock before its device check and holds it through the PATCH
and the report.
- The wait is bounded. A snapshot of a large VM holds the lock for minutes, so a
set gives up after 60 s, fails and says the target was not set
(`acquire_vm_snapshot_lock_within`). Snapshots keep their unbounded wait for
one another.
- A report without MIB takes no lock. It changes nothing and does not wait for a
snapshot.
- A PATCH sent to the API socket by anything other than fcvm is outside the lock.
Two failpoints give the tests their seams: `snapshot.post_balloon_read_pre_save`
in the shared creator and `balloon.post_set_pre_report` in the command.
Tested, each test first without its subject:
make _test-root FILTER="--retries 0 -E 'test(=test_balloon_set_cannot_land_between_a_snapshots_read_and_its_save) | test(=test_two_balloon_sets_each_report_their_own_target)'" STREAM=1
with the command setting without the lock (2 tests run: 0 passed, 2 failed, 1799 skipped):
FAIL test_balloon_set_cannot_land_between_a_snapshots_read_and_its_save
Error: the snapshot records balloon Some(64) and the device it saved holds 96 MiB: a target was set between the snapshot's read and its save
FAIL test_two_balloon_sets_each_report_their_own_target
Error: the first set asked for 96 MiB and reported target 80 MiB
on this commit, with test_balloon_command_sets_and_reports_the_target and
test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has:
PASS [ 23.106s] fcvm::test_balloon test_balloon_command_sets_and_reports_the_target
PASS [ 31.618s] fcvm::test_balloon test_balloon_set_cannot_land_between_a_snapshots_read_and_its_save
PASS [ 19.992s] fcvm::test_balloon test_two_balloon_sets_each_report_their_own_target
PASS [ 21.974s] fcvm::test_snapshot_clone test_snapshot_and_restored_clone_record_the_balloon_the_vmm_has
Summary [ 96.697s] 4 tests run: 4 passed, 1797 skipped
make _test-unit FILTER="-E 'test(/a_set_gives_up_while_the_snapshot_lock_stays_held/)'"
with the bounded wait never giving up:
FAIL commands::balloon::tests::a_set_gives_up_while_the_snapshot_lock_stays_held_and_says_nothing_was_set
a set with a 300 ms wait was still waiting for a held lock after 10s
make _test-unit FILTER="-E 'test(/podmans_own_snapshots_hold_the_vm_snapshot_lock/)'"
with create_podman_snapshot giving the lock back as soon as it has it:
FAIL commands::podman::snapshot::tests::podmans_own_snapshots_hold_the_vm_snapshot_lock_through_the_save
create_podman_snapshot does not keep the per-VM snapshot lock for the rest of the function
both, with the command's other unit tests and the balloon call-site pins, on this commit:
Summary [ 0.320s] 13 tests run: 13 passed, 1431 skipped
make _test-unit on the changed modules and the test binaries that read the
files this change edits:
Summary [ 8.342s] 565 tests run: 565 passed, 879 skipped
make fmt leaves the tree unchanged. make clippy is clean.
Not run: arm64, the nested profile's Firecracker, a disk-only capture or one of
`podman run`'s own snapshots racing a set (they hold the same lock; the podman
path is pinned by source, the disk-only path by the existing pin), and the full
make test-root.
02adee9 to
119fe74
Compare
ejc3
left a comment
There was a problem hiding this comment.
NOT-A-DEFECT: the review bodies on this pull request carry no finding of their own. Each finding they announce is an inline thread, and each thread is answered there with the test that was watched failing.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 119fe747ce
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A startup snapshot is named for the balloon target its workload initialized
under (`<key>-balloon<MIB>-startup`), and a restore can set a target but cannot
replay the initialization. `fcvm balloon` can set a target as soon as a run has
published its state, which is before a `--health-check` workload is healthy. The
run still named its startup snapshot from its own `--balloon`, so a workload
that initialized after a set from 64 to 96 MiB was saved as `-balloon64-startup`
and restored by every later 64 MiB run.
`create_podman_snapshot` is now told which target its snapshot is good for
(`BalloonRequirement`). For a startup snapshot that is the target the run
started with: its `--balloon`, which a cold boot attaches the device at and a
restore sets before the guest resumes. The creator reads the device from the
VMM and compares. When the two differ it takes no snapshot and returns
`SnapshotInstall::BalloonTargetChanged`:
- `podman run`, in its own loop and in the restore loop, logs why at info and
goes on with no startup snapshot, under either name. Its state keeps naming
the pre-start snapshot. Later runs restore the shared pre-start snapshot and
make their own startup snapshot.
- `podman prepare` fails and installs nothing, because the snapshot is what it
was asked for.
- The pre-start snapshot asks for no target. It is taken before the workload
starts and is shared between targets.
The comparison runs under the per-VM snapshot lock the creator already holds,
which a set takes too, so a set cannot land between it and the save. It runs
before the pause, so a VM whose snapshot is declined is not paused. Sets are not
refused and no lock is added. A target that was changed and changed back before
the run turned healthy is not seen.
Tested, each test first without its subject. The 1-minute load on the host was
34 to 68 during these runs, from other work on it.
make _test-root FILTER="--retries 0 -E 'test(=test_startup_snapshot_is_not_taken_after_the_balloon_target_was_changed)'" STREAM=1
with create_podman_snapshot taking a startup snapshot without the comparison
(1 test run: 0 passed, 1 failed, 1803 skipped):
FAIL [ 14.452s] test_startup_snapshot_is_not_taken_after_the_balloon_target_was_changed
Error: run 1's balloon target was changed from 64 to 96 MiB before its workload initialized, and the run saved the startup snapshot 51642dabcc92-balloon64-startup, which later 64 MiB runs restore
with only the restore loop asking for its startup snapshot at any target
(1 test run: 0 passed, 1 failed, 1803 skipped):
FAIL [ 24.470s] test_startup_snapshot_is_not_taken_after_the_balloon_target_was_changed
Error: run 2 restored the pre-start snapshot, its balloon target was changed from 64 to 96 MiB before its workload initialized, and the run saved the startup snapshot 7a7501fc333d-balloon64-startup
on this commit, with the command's other VM tests and
test_startup_snapshot_is_per_balloon_target:
PASS [ 22.923s] fcvm::test_balloon test_balloon_command_sets_and_reports_the_target
PASS [ 31.388s] fcvm::test_balloon test_balloon_set_cannot_land_between_a_snapshots_read_and_its_save
PASS [ 24.483s] fcvm::test_balloon test_startup_snapshot_is_not_taken_after_the_balloon_target_was_changed
PASS [ 21.118s] fcvm::test_balloon test_two_balloon_sets_each_report_their_own_target
PASS [ 26.516s] fcvm::test_snapshot_clone test_startup_snapshot_is_per_balloon_target
Summary [ 126.434s] 5 tests run: 5 passed, 1799 skipped
In that run the new test's first two runs logged "Not taking the startup
snapshot" and its third, which nobody set, logged "Startup snapshot created
successfully snapshot_key=c63d748d9082-balloon64-startup".
make _test-unit FILTER="-E 'test(/a_startup_snapshot_compares_the_balloon_target_under/)'"
with the comparison removed:
FAIL commands::podman::snapshot::tests::a_startup_snapshot_compares_the_balloon_target_under_the_vm_snapshot_lock
create_podman_snapshot does not read the VM's balloon target before a startup snapshot
make _test-unit FILTER="-E 'binary(/^test_balloon_call_sites/)'"
with the restore loop asking at any target (3 tests run: 2 passed, 1 failed):
FAIL fcvm::test_balloon_call_sites every_startup_snapshot_is_asked_for_at_the_target_its_run_started_with
1 of 4 statements that keep a startup snapshot to the balloon target its run started with are gone.
both, with the snapshot-lock, snapshot-key and prepare unit tests, on this commit:
Summary [ 0.322s] 26 tests run: 26 passed, 1420 skipped
make _test-unit on the changed modules and the test binaries that read the
files this change edits:
Summary [ 8.263s] 567 tests run: 567 passed, 879 skipped
make fmt leaves the tree unchanged. make clippy is clean.
Not run: `podman prepare` with a set during its preparation in a VM (its request
and its refusal are pinned by source), arm64, the nested profile's Firecracker,
and the full make test-root.
ejc3
left a comment
There was a problem hiding this comment.
NOT-A-DEFECT: the review bot's summary comment was rewritten when this pull request's description changed, and it carries no finding. The findings on this pull request are inline threads, and each is answered there with the test that was watched failing.
|
@codex review |
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
ejc3
left a comment
There was a problem hiding this comment.
NOT-A-DEFECT: the review bot's summary comment was rewritten when this pull request's description changed, and it carries no finding. The findings on this pull request are inline threads, and each is answered there with the test that was watched failing.
fcvm balloon (--pid PID | --name NAME) [MIB]prints a running VM's balloon target and size as one JSON line, and sets the target first when MIB is given.Follows: #1060
The Problem
--balloonattaches the device at boot, and since #1060 a snapshot cache hit sets its own target at restore. After that, the target of a running VM could be changed only by talking to Firecracker's API socket by hand, and nothing in fcvm said what the device was at.The Solution
The target is what the device was asked to reach; the size is what the guest has given it so far. After a set the command prints right away, so the size is still on its way.
GET /vm/config, which answers 200 either way, so a VM booted without--balloonis told it has no device and that one is attached only at boot. Nothing is inferred from a 400.PATCHwith a 400 (the typedApiRefusalfrom Share one snapshot between runs that differ only in the balloon target #1060), the error says what Firecracker refuses: a device the guest never activated, as with a kernel that has no virtio balloon driver. A timeout or any other failure says only what was being done.snapshot create, memory and disk-only, andpodman run's andpodman prepare's own snapshots. A pause does not keep aPATCHout, because Firecracker applies it to a paused VM. So a set that took no lock could land between the read and the save, and the snapshot recorded one target and saved a device at another; two sets could also interleave, and one reported the other's target. A set now takes that lock before its device check and holds it through thePATCHand the report. It waits up to 60 seconds, then fails and says the target was not set, because a snapshot of a large VM holds the lock for minutes. A report without MIB changes nothing, takes no lock and does not wait. APATCHsent to the API socket by anything other than fcvm is outside the lock.<key>-balloon<MIB>-startup, Share one snapshot between runs that differ only in the balloon target #1060), and a restore can set a target but cannot replay the initialization. A set that landed after apodman run --health-checkhad published its state and before the workload was healthy changed the memory the workload initialized with, and the run still saved that workload under the name of the target it started with, where later runs at that target restored it. The creator now reads the device before a startup snapshot and compares it with the target the run started with. When they differ, the run takes no startup snapshot, under either name, and logs why at info; later runs restore the shared pre-start snapshot and make their own.podman preparefails instead and installs nothing, because the snapshot is what it was asked for. The comparison runs under the per-VM snapshot lock, which a set takes too, so a set cannot land between it and the save, and it runs before the pause, so a VM whose snapshot is declined is not paused. The pre-start snapshot needs no check: it is taken before the workload starts and is shared between targets. Sets are not refused and no lock is added. A target that was changed and changed back before the run turned healthy is not seen.The command writes no state. What follows the new target and what does not:
snapshot create,podman run's own pre-start snapshot, a disk-only captureGET /vm/config), not from the statepodman run's own startup snapshot, andpodman prepare'spodman preparefailsfcvm ls --json,config.balloon_mib)The two "No" rows are the gap. Closing it needs a state write under the per-VM lock and a launch config that follows it, which is not in this PR.
Also in this PR:
exec's lookup of a VM by--pidor--namemoves tocommands::common::load_vm_stateand both commands use it; the text of what Firecracker refuses, and the test for a 400, are shared with the restore'sPATCH;BalloonStatsserializes, since it is the command's output. Docs: README CLI table, DESIGN.md command summary and afcvm balloonsection, and the--balloonhelp.Test Results
Each test was watched failing without its subject. The reds are from this change over the first commit of #1060; every green step ran again after it moved onto the second. The second commit's reds and greens ran over #1060's third commit.
The VM test,
make _test-root FILTER="--retries 0 -E 'test(=test_balloon_command_sets_and_reports_the_target)'" STREAM=1, with thePATCHand the device check removed:The second commit,
make _test-root FILTER="--retries 0 -E 'test(=test_balloon_set_cannot_land_between_a_snapshots_read_and_its_save) | test(=test_two_balloon_sets_each_report_their_own_target)'" STREAM=1, with the command setting without the lock:Its unit tests, each with one statement mutated:
The third commit,
make _test-root FILTER="--retries 0 -E 'test(=test_startup_snapshot_is_not_taken_after_the_balloon_target_was_changed)'" STREAM=1, first with the creator taking a startup snapshot without the comparison, then with only the restore loop asking for its startup snapshot at any target:Its source pins, each with one statement mutated:
Green, on the third commit (1-minute load on the host 34 to 68 from other work):
In that run the new test's first two runs logged
Not taking the startup snapshot, and its third, whose target nobody changed, loggedStartup snapshot created successfully.Green, on the second commit:
test_balloon_set_cannot_land_between_a_snapshots_read_and_its_saveholdssnapshot createwith a failpoint between its read of the balloon (64 MiB) and its save, startsfcvm balloon --pid P 96while it is held, and then restores a clone from the snapshot: the clone's device has to be at the 64 the snapshot records, and the set has to report 96 once the snapshot is done.test_two_balloon_sets_each_report_their_own_targetholds one set between itsPATCHand its report and starts a second with another target: each has to report its own.Green, on the first commit, before it moved onto #1060's third commit:
test_balloon_command_sets_and_reports_the_targetboots one VM with--balloon 64and one without. It reads the report by PID and by name, sets 96 with the command, and reads 96 back from the command's own output, from Firecracker'sGET /balloon/statisticsand from a later report once the guest has brought the balloon to that size. It then checks that the VM's state still records 64, that a snapshot taken afterwards records 96, that a target above the VM's memory is refused, and that the VM without a balloon is told it has no balloon device (with and without MIB). The "no balloon device" answer is tested against the real VM: Firecracker itself answers a VM without the device400 Bad Request ... Device not found(the last two lines of the red run above), and the command readsGET /vm/configfirst, so that VM is told it has no balloon device and where one comes from. The other two VM tests are in the run becauseexec's lookup moved (the reboot test drives the guest throughfcvm exec --pid) andBalloonStatschanged.What is pinned and what is read: the "Yes" rows for
snapshot createand for a clone's state are pinned by this test and bytest_snapshot_and_restored_clone_record_the_balloon_the_vmm_has(#1060), and the state row by this test. The two reboot rows and the disk-only row are from reading the code.tests/test_balloon_call_sites.rspins the statements the clone's reboot plan and the disk-only boot rest on; the cold-booted VM's relaunch from its launch config is not pinned.Not run: a disk-only capture or one of
podman run's own snapshots racing a set (they hold the same lock; the podman path is pinned by source and the disk-only path by #1060's pin), arm64, the nested profile's Firecracker, a VM whose guest never activated the device (the 400 explanation is covered by unit tests), Cloud Hypervisor (its refusal is a unit test on the VM's state), the fullmake test-root, andcargo auditandcargo deny(the Cargo files are unchanged).Summary by CodeRabbit
fcvm balloonto view a running VM’s balloon target and actual size, or set a new target.podman preparereports an error instead of installing an unusable snapshot.