Skip to content

kernel: build the CPU controller's bandwidth limit for cpu.max - #1299

Closed
Pedro Henrique Penna (ppenna) wants to merge 1 commit into
devfrom
agents/issue-1193-cpu-controller
Closed

Pedro Henrique Penna (ppenna) wants to merge 1 commit into
devfrom
agents/issue-1193-cpu-controller

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1193.

A guest agent that runs executions in cgroups of their own can cap their memory.max and pids.max, but not their CPU time: the guest kernel has no CPU controller. With the release artifacts of dev's 4cfe8330e, the guest's root cgroup offers only io memory pids, writing +cpu to its cgroup.subtree_control fails with EINVAL, and a child cgroup has cpu.stat and cpu.stat.local but no cpu.max. The issue predates #1200, whose kernel feature inventory left the CPU controller off as one that the sandbox never enables, so the one-line change that it proposes is now a feature of the inventory.

This is the bottom of a stack of two draft pull requests: #1300 adds the cpu-bandwidth microVM scenario, which shows that cpu.max throttles a busy loop on every backend.

What changed

  • The inventory. The new execution-cpu-limits feature of kernel_features.py sets CGROUP_SCHED, FAIR_GROUP_SCHED, and CFS_BANDWIDTH. CFS_BANDWIDTH selects GROUP_SCHED_BANDWIDTH, under which Linux 6.18 defines cpu.max and cpu.max.burst, so the feature expects it too; FAIR_GROUP_SCHED selects GROUP_SCHED_WEIGHT, which gives cpu.weight. The feature keeps RT_GROUP_SCHED off: with it, cgroup v2 enables the CPU controller only while every real-time process is in the root cgroup. NVX's own agents enable the controller for no cgroup, so every task stays in the root task group unless a guest agent sets a limit. nvx::guest::cgroup needs no change: ExecutionCgroup::prepare already enables the controller of each resource file that its caller names, such as cpu.max.
  • The build's checks. The feature joins the sandbox check group beside the networking of Build VETH into the microVM guest kernel #309, so that build-kernel, packaging, and verify require its settings, and test_the_cpu_limit_of_1193_is_required pins them there, as test_the_networking_of_309_is_required pins Build VETH into the microVM guest kernel #309's. Against the inventory of 4cfe8330e, it fails on the five settings that the group lacks and on the missing owner of CFS_BANDWIDTH.
  • The configuration. nvx.py kernel-config, in the Docker build's image, regenerated kernel/config-microvm, which now sets CGROUP_SCHED, GROUP_SCHED_WEIGHT, GROUP_SCHED_BANDWIDTH, FAIR_GROUP_SCHED, and CFS_BANDWIDTH and holds # CONFIG_RT_GROUP_SCHED is not set, and its derived record, which gains GROUP_SCHED_BANDWIDTH and GROUP_SCHED_WEIGHT, both without a prompt. The configuration sets 774 symbols: 178 inventory settings of 26 features and 596 derived, 53 of them with a prompt.
  • The documents. doc/build.md updates those counts, no longer lists the CPU controller among what the inventory removed, and adds it to the sandbox check group. §4.3.3 of the isolation model says that a guest agent can cap an execution's CPU time with cpu.max, and that NVX's own agents set no CPU limit.

A checkout that holds the kernel of an earlier release, such as one that nvx-rs download installed, fails verify's check of build/vmlinux.config until it builds the kernel or downloads a release with this change, as for any setting that a check group gains. No CI job runs verify on such artifacts.

Cost

The kernel that the Docker build's toolchain builds, against the release kernel of 4cfe8330e:

4cfe8330e This PR Change
vmlinux 21,594,072 B 21,612,864 B +18,792 B
.text 8,637,128 B 8,658,184 B +21,056 B
.rodata 1,685,896 B 1,690,552 B +4,656 B
.data 1,234,816 B 1,238,336 B +3,520 B
.bss 1,523,712 B 1,511,424 B −12,288 B
Span of the loadable segments 18.09 MiB 18.09 MiB none

nvx.py benchmark --suite e2e --processors 1 --warmups 3 --runs 41 --skip-build on nvx-kvm-test (Linux/KVM, virtual-machine), with one OpenVMM build and one Alpine initramfs, alternating the kernels in the order 4cfe8330e, this PR, this PR, 4cfe8330e:

p50 4cfe8330e This PR
Cold boot 313.9 and 314.1 ms 310.7 and 314.1 ms
Cold boot peak RSS 79.37 and 79.41 MiB 79.34 and 79.36 MiB
Snapshot capture 1.48 and 1.54 ms 1.47 and 1.49 ms
Snapshot restore 79.3 and 80.4 ms 80.5 and 81.5 ms
Restore peak RSS 35.73 and 35.25 MiB 35.32 and 35.64 MiB

A first pass with 21 runs each measured restores of 77.0 and 74.9 ms against 82.7 and 81.9 ms, whose spread, 46 to 131 ms within a run, the 41-run pass narrowed to a difference within its runs of the same kernel. Restores stay far below CI's regression gate, which needs 50% and 10 ms.

Validation

Locally, on Windows: at this branch's head, check-kernel-config, test_kernel_config.py (27 tests), the shard tests of test_microvm_tests.py, Ruff, the Markdown link check, and check_test_changes.py, which reports no change to an existing test; at the head of #1300, which adds the scenario, the commands of validate-nvx (1,816 tests, 73 skipped; test_adversarial.py; test_hosts.py), Pyright for Linux and Windows, and the test layout check, all passed. On nvx-kvm-test (Linux), at the head of #1300, with NVX_RS naming a release build of nvx-rs and without the Azure setup packages that validate-nvx installs: its tests (1,816, 35 skipped), the command help, and check-kernel-config passed.

On the test VMs of the host inventory, all virtual-machine hosts, with the release artifacts of 4cfe8330e (v0.1.0-dev.4cfe8330e719) and either their kernel or this PR's, which the Docker build's kernel-artifacts target built on nvx-kvm-test, after its own inventory check, and whose vmlinux.config differs from the release's by the five new symbols alone:

Profile Backend Release kernel of 4cfe8330e This PR's kernel
nvx-kvm-test KVM controllers io memory pids; +cpu refused with EINVAL; no cpu.max controllers cpu io memory pids; +cpu enabled; cpu.max reads back 50000 100000; the default test-microvm suite passed (58 scenario runs)
nvx-mshv-test MSHV the same as on KVM; the default suite passed but for #1300's cpu-bandwidth the default suite passed (58 runs)
nvx-whp-test WHP cpu-bandwidth failed: the kernel has no CPU controller the default suite passed (58 runs)

The default suites ran at the first revision of #1300's branch, whose kernel is this PR's, and include the public sandbox scenarios and managed-lifecycle. The time ABI's evidence of MSHV's two runs matched: 9 and 8 of 76 boot checks over their CPU budget, which never gates, and warp offsets of at most 90 and 88 ns.

Left to CI: the guest-artifact builds, and the microVM, sandbox, runtime-adapter, debug-kernel, and benchmark jobs on the shared KVM, MSHV, and WHP runners, which a draft pull request skips.

A guest agent that runs executions in cgroups of their own could cap
their memory.max and pids.max but not their CPU time: the guest kernel
had no CPU controller, so the root cgroup offered only io, memory, and
pids, enabling +cpu in its cgroup.subtree_control failed with EINVAL,
and an execution's cgroup had no cpu.max to write (#1193). The kernel
feature inventory had left the controller off as one that the sandbox
never enables.

The inventory's new execution-cpu-limits feature sets CGROUP_SCHED,
FAIR_GROUP_SCHED, and CFS_BANDWIDTH, which selects the
GROUP_SCHED_BANDWIDTH that gives cgroups cpu.max, and keeps
RT_GROUP_SCHED off, with which cgroup v2 refuses the controller while a
real-time process runs outside the root cgroup. NVX's own agents enable
the controller for no cgroup, so their tasks stay in the root task
group. The feature joins the build's sandbox check group beside the
networking of #309, so that no configuration or version change drops
it silently, and nvx.py kernel-config regenerated kernel/config-microvm
and its derived record in the Docker build's image: 774 symbols, 178
of them inventory settings and 596 derived.

The kernel that CI's Docker toolchain builds grows by 18,792 bytes:
21,056 more of .text, 4,656 of .rodata, and 3,520 of .data, and 12,288
fewer of .bss. Its loadable segments still span 18.09 MiB. An e2e
benchmark on nvx-kvm-test (Linux/KVM, virtual machine), alternating
the release kernel of 4cfe833 and this one with 41 runs each,
measured cold boots of 313.9 and 314.1 ms against 310.7 and 314.1 ms,
restores of 79.3 and 80.4 ms against 80.5 and 81.5 ms, and the same
peak RSS.

The build and isolation documents describe the controller, and a test
pins its settings to the sandbox group as #309's networking is pinned;
it fails without them.

With the release artifacts of 4cfe833, a guest on nvx-kvm-test and
nvx-mshv-test (Linux/MSHV), both virtual machines, listed only io,
memory, and pids in cgroup.controllers and refused +cpu with EINVAL.
With this kernel on nvx-kvm-test, it lists cpu, enables it, and accepts
"50000 100000" in a child's cpu.max, and every default test-microvm
scenario passed with it there, on nvx-mshv-test, and on nvx-whp-test
(Windows/WHP, virtual machine).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified with the change enabling guest-agent CPU limits.

No open issues 🎉

What changed in this PR

Enables the guest kernel’s CPU bandwidth controller so guest agents can limit per-execution CPU usage through cgroup v2’s cpu.max, addressing #1193. NVX’s own agents continue to set no CPU limits; runtime throttling coverage follows in #1300.

Changes:

  • Adds an execution-cpu-limits inventory feature and requires its settings in sandbox configuration checks.
  • Updates the generated kernel configuration and derived record, keeping real-time group scheduling disabled.
  • Adds a regression test and documents CPU-limit support and updated configuration counts.
File Description
scripts/​test_kernel_config.py Tests required CPU-limit settings and inventory ownership.
scripts/​nvx_tools/​kernel_features.py Defines CPU-limit support and adds it to sandbox checks.
kernel/​config-microvm.derived Records derived bandwidth and weight scheduling settings.
kernel/​config-microvm Enables CPU bandwidth control without real-time group scheduling.
doc/​design/​4.3-isolation-and-resource-model.md Explains guest-agent CPU limits and unchanged NVX defaults.
doc/​build.md Updates configuration counts and sandbox feature documentation.

Review effort: Balanced

@ppenna

Copy link
Copy Markdown
Contributor Author

Closing without merging: this will not be implemented, because it goes away with the nvx refactor (feature/nvx-refactor). #1193 is closed as not planned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kernel: CONFIG_CFS_BANDWIDTH is off, so cpu.max cannot cap an execution

2 participants