Repository navigation
kernel: build the CPU controller's bandwidth limit for cpu.max - #1299
Closed
Pedro Henrique Penna (ppenna) wants to merge 1 commit into
Closed
Pedro Henrique Penna (ppenna) wants to merge 1 commit into
Pedro Henrique Penna (ppenna) wants to merge 1 commit into
Conversation
A guest agent that runs executions in cgroups of their own could cap their memory.max and pids.max but not their CPU time: the guest kernel had no CPU controller, so the root cgroup offered only io, memory, and pids, enabling +cpu in its cgroup.subtree_control failed with EINVAL, and an execution's cgroup had no cpu.max to write (#1193). The kernel feature inventory had left the controller off as one that the sandbox never enables. The inventory's new execution-cpu-limits feature sets CGROUP_SCHED, FAIR_GROUP_SCHED, and CFS_BANDWIDTH, which selects the GROUP_SCHED_BANDWIDTH that gives cgroups cpu.max, and keeps RT_GROUP_SCHED off, with which cgroup v2 refuses the controller while a real-time process runs outside the root cgroup. NVX's own agents enable the controller for no cgroup, so their tasks stay in the root task group. The feature joins the build's sandbox check group beside the networking of #309, so that no configuration or version change drops it silently, and nvx.py kernel-config regenerated kernel/config-microvm and its derived record in the Docker build's image: 774 symbols, 178 of them inventory settings and 596 derived. The kernel that CI's Docker toolchain builds grows by 18,792 bytes: 21,056 more of .text, 4,656 of .rodata, and 3,520 of .data, and 12,288 fewer of .bss. Its loadable segments still span 18.09 MiB. An e2e benchmark on nvx-kvm-test (Linux/KVM, virtual machine), alternating the release kernel of 4cfe833 and this one with 41 runs each, measured cold boots of 313.9 and 314.1 ms against 310.7 and 314.1 ms, restores of 79.3 and 80.4 ms against 80.5 and 81.5 ms, and the same peak RSS. The build and isolation documents describe the controller, and a test pins its settings to the sandbox group as #309's networking is pinned; it fails without them. With the release artifacts of 4cfe833, a guest on nvx-kvm-test and nvx-mshv-test (Linux/MSHV), both virtual machines, listed only io, memory, and pids in cgroup.controllers and refused +cpu with EINVAL. With this kernel on nvx-kvm-test, it lists cpu, enables it, and accepts "50000 100000" in a child's cpu.max, and every default test-microvm scenario passed with it there, on nvx-mshv-test, and on nvx-whp-test (Windows/WHP, virtual machine). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues were identified with the change enabling guest-agent CPU limits.
No open issues 🎉
What changed in this PR
Enables the guest kernel’s CPU bandwidth controller so guest agents can limit per-execution CPU usage through cgroup v2’s cpu.max, addressing #1193. NVX’s own agents continue to set no CPU limits; runtime throttling coverage follows in #1300.
Changes:
- Adds an
execution-cpu-limitsinventory feature and requires its settings in sandbox configuration checks. - Updates the generated kernel configuration and derived record, keeping real-time group scheduling disabled.
- Adds a regression test and documents CPU-limit support and updated configuration counts.
| File | Description |
|---|---|
scripts/test_kernel_config.py |
Tests required CPU-limit settings and inventory ownership. |
scripts/nvx_tools/kernel_features.py |
Defines CPU-limit support and adds it to sandbox checks. |
kernel/config-microvm.derived |
Records derived bandwidth and weight scheduling settings. |
kernel/config-microvm |
Enables CPU bandwidth control without real-time group scheduling. |
doc/design/4.3-isolation-and-resource-model.md |
Explains guest-agent CPU limits and unchanged NVX defaults. |
doc/build.md |
Updates configuration counts and sandbox feature documentation. |
Review effort: Balanced
Pedro Henrique Penna (ppenna)
requested a review
from Enrique Saurez (esaurez)
November 6, 2026 01:41
Contributor
Author
|
Closing without merging: this will not be implemented, because it goes away with the nvx refactor ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1193.
A guest agent that runs executions in cgroups of their own can cap their
memory.maxandpids.max, but not their CPU time: the guest kernel has no CPU controller. With the release artifacts ofdev's4cfe8330e, the guest's root cgroup offers onlyio memory pids, writing+cputo itscgroup.subtree_controlfails withEINVAL, and a child cgroup hascpu.statandcpu.stat.localbut nocpu.max. The issue predates #1200, whose kernel feature inventory left the CPU controller off as one that the sandbox never enables, so the one-line change that it proposes is now a feature of the inventory.This is the bottom of a stack of two draft pull requests: #1300 adds the
cpu-bandwidthmicroVM scenario, which shows thatcpu.maxthrottles a busy loop on every backend.What changed
execution-cpu-limitsfeature ofkernel_features.pysetsCGROUP_SCHED,FAIR_GROUP_SCHED, andCFS_BANDWIDTH.CFS_BANDWIDTHselectsGROUP_SCHED_BANDWIDTH, under which Linux 6.18 definescpu.maxandcpu.max.burst, so the feature expects it too;FAIR_GROUP_SCHEDselectsGROUP_SCHED_WEIGHT, which givescpu.weight. The feature keepsRT_GROUP_SCHEDoff: with it, cgroup v2 enables the CPU controller only while every real-time process is in the root cgroup. NVX's own agents enable the controller for no cgroup, so every task stays in the root task group unless a guest agent sets a limit.nvx::guest::cgroupneeds no change:ExecutionCgroup::preparealready enables the controller of each resource file that its caller names, such ascpu.max.sandboxcheck group beside the networking of Build VETH into the microVM guest kernel #309, so thatbuild-kernel, packaging, andverifyrequire its settings, andtest_the_cpu_limit_of_1193_is_requiredpins them there, astest_the_networking_of_309_is_requiredpins Build VETH into the microVM guest kernel #309's. Against the inventory of4cfe8330e, it fails on the five settings that the group lacks and on the missing owner ofCFS_BANDWIDTH.nvx.py kernel-config, in the Docker build's image, regeneratedkernel/config-microvm, which now setsCGROUP_SCHED,GROUP_SCHED_WEIGHT,GROUP_SCHED_BANDWIDTH,FAIR_GROUP_SCHED, andCFS_BANDWIDTHand holds# CONFIG_RT_GROUP_SCHED is not set, and its derived record, which gainsGROUP_SCHED_BANDWIDTHandGROUP_SCHED_WEIGHT, both without a prompt. The configuration sets 774 symbols: 178 inventory settings of 26 features and 596 derived, 53 of them with a prompt.doc/build.mdupdates those counts, no longer lists the CPU controller among what the inventory removed, and adds it to the sandbox check group. §4.3.3 of the isolation model says that a guest agent can cap an execution's CPU time withcpu.max, and that NVX's own agents set no CPU limit.A checkout that holds the kernel of an earlier release, such as one that
nvx-rs downloadinstalled, failsverify's check ofbuild/vmlinux.configuntil it builds the kernel or downloads a release with this change, as for any setting that a check group gains. No CI job runsverifyon such artifacts.Cost
The kernel that the Docker build's toolchain builds, against the release kernel of
4cfe8330e:4cfe8330evmlinux.text.rodata.data.bssnvx.py benchmark --suite e2e --processors 1 --warmups 3 --runs 41 --skip-buildonnvx-kvm-test(Linux/KVM,virtual-machine), with one OpenVMM build and one Alpine initramfs, alternating the kernels in the order4cfe8330e, this PR, this PR,4cfe8330e:4cfe8330eA first pass with 21 runs each measured restores of 77.0 and 74.9 ms against 82.7 and 81.9 ms, whose spread, 46 to 131 ms within a run, the 41-run pass narrowed to a difference within its runs of the same kernel. Restores stay far below CI's regression gate, which needs 50% and 10 ms.
Validation
Locally, on Windows: at this branch's head,
check-kernel-config,test_kernel_config.py(27 tests), the shard tests oftest_microvm_tests.py, Ruff, the Markdown link check, andcheck_test_changes.py, which reports no change to an existing test; at the head of #1300, which adds the scenario, the commands ofvalidate-nvx(1,816 tests, 73 skipped;test_adversarial.py;test_hosts.py), Pyright for Linux and Windows, and the test layout check, all passed. Onnvx-kvm-test(Linux), at the head of #1300, withNVX_RSnaming a release build ofnvx-rsand without the Azure setup packages thatvalidate-nvxinstalls: its tests (1,816, 35 skipped), the command help, andcheck-kernel-configpassed.On the test VMs of the host inventory, all
virtual-machinehosts, with the release artifacts of4cfe8330e(v0.1.0-dev.4cfe8330e719) and either their kernel or this PR's, which the Docker build'skernel-artifactstarget built onnvx-kvm-test, after its own inventory check, and whosevmlinux.configdiffers from the release's by the five new symbols alone:4cfe8330envx-kvm-testio memory pids;+cpurefused withEINVAL; nocpu.maxcpu io memory pids;+cpuenabled;cpu.maxreads back50000 100000; the defaulttest-microvmsuite passed (58 scenario runs)nvx-mshv-testcpu-bandwidthnvx-whp-testcpu-bandwidthfailed:the kernel has no CPU controllerThe default suites ran at the first revision of #1300's branch, whose kernel is this PR's, and include the public sandbox scenarios and
managed-lifecycle. The time ABI's evidence of MSHV's two runs matched: 9 and 8 of 76 boot checks over their CPU budget, which never gates, and warp offsets of at most 90 and 88 ns.Left to CI: the guest-artifact builds, and the microVM, sandbox, runtime-adapter, debug-kernel, and benchmark jobs on the shared KVM, MSHV, and WHP runners, which a draft pull request skips.