You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: measure the nvx-rs host code that the KVM microVM scenarios reach #1325
Measure which host code of nvx-rs the microVM scenarios and the runtime adapter's suite reach on Linux/KVM, alone and together with the crate's own tests, so that the code that only a real VM reaches is measured too.
Background
The crate's own tests leave most of what they don't reach in code that needs a VM or the network. At 82c15cd, on Windows, they reach 75.8% of the lines of src/host/instance/, 66.2% of the lines of src/host/template/clone/launch.rs, and 65.3% of the branches of src/host/template/store/capture.rs. nvx.py and the microVM scenarios run the nvx-rs that NVX_RS names (doc/usage.md), so they can run an instrumented build.
Tasks
On a Linux/KVM runner, build an instrumented nvx-rs, for example in the environment that cargo llvm-cov show-env prints, point NVX_RS at it, and set LLVM_PROFILE_FILE to a pattern that gives each process a profile of its own, such as %p-%m.profraw.
Run test-microvm --backend kvm and test-runtime-adapter --backend kvm with it, on demand or on a schedule, never as a required check.
Checked against current dev (bf81488): #1393 has merged the optional Linux/KVM scenario-coverage workflow, instrumentation/profile accounting, separate scenarios/crate-tests/together reports, missing-profile process inventory, and doc/ci.md documentation. The workflow runs only on schedule or manual dispatch, so it adds no required check.
Leaving this issue open for now: no runs of nvx-rs-scenario-coverage.yml exist yet, and #1393 explicitly identifies the first acceptance criterion as pending a CI-runner measurement after merge and runner setup. Close once a run produces the reports and verifies profile accounting for the exited nvx-rs processes.
The first run of nvx-rs-scenario-coverage.yml, the scheduled 42460108426 at bf814882b, failed at Check the coverage tools on azure-kvm-4, before it measured anything:
cargo-llvm-cov 0.9.1, which scripts/setup/tool-versions.conf pins, is not installed; rerun scripts/setup/setup-linux.sh --runner-only
No KVM runner has either tool yet. setup-linux.sh --runner-only installs cargo-llvm-cov and the llvm-tools component since #1333, merged on 2026-11-07, but the runners' setup predates it: their toolchain of Rust 1.97.1 dates from 2026-11-06. A read-only probe of the five KVM runners today, as my SSH user, without sudo:
Runner
Labels
Instance
Booted
/opt/nvx/cargo/bin/cargo-llvm-cov
llvm-tools of Rust 1.97.1
azure-kvm-1
general
vmss000000
2026-10-28
Missing
Missing
azure-kvm-2
general
vmss000002
2026-10-28
Missing
Missing
azure-kvm-3
benchmark
vmss000003
2026-10-28
Missing
Missing
azure-kvm-4
general
vmss000004
2026-10-28
Missing
Missing
azure-kvm-5
general
vmss000005
2026-10-28
Missing
Missing
Each has Rust 1.97.1 with the x86_64-unknown-linux-musl, x86_64-unknown-none, and x86_64-unknown-uefi targets, and cargo-nextest 0.9.133. Neither the check nor the setup has changed since bf814882b, so every run on a general runner fails the same way until the runner has both tools.
Nothing else blocks the run. On nvx-kvm-test (linux-kvm, virtual-machine, Azure Linux 3), in a temporary worktree of dev at 995230d0e, with the artifacts and OpenVMM of its release, v0.1.0-dev.995230d0ebb4, the host's llvm-tools of Rust 1.97.1, and cargo-llvm-cov 0.9.1 built with --locked into a temporary directory, I ran the job's steps from Check the coverage tools on, with its commands, twice:
Run
Result
First
Status 1. test-microvm stopped at private-network-ingress, which failed with ConnectionResetError: [Errno 104] Connection reset by peer, as described below. The adapter's suite and the crate's tests passed, the reports were written, and each of the 144 nvx-rs starts wrote its profile
Second
Status 0 in 16 min: test-microvm, all 57 scenarios, 599 s; the adapter's suite 202 s; the crate's tests 110 s, with no retries or stray profiles. Lines: the crate's tests 85.1% of 43,461, the scenarios 35.0%, together 85.3%; functions: 85.8%, 36.3%, and 86.2% of 5,806. The scenarios reach 82 lines that the crate's tests don't, and 57.8% of the 26,363 that their builds compile. Each of the 144 nvx-rs starts wrote its profile
The failure of the first run is a race in refused(), the host-denied probe of private-network-ingress in scripts/nvx_tools/microvm_tests.py. It guards only recv() against the reset that the published listener sends to the host's own connections. With a timeout, though, socket.create_connection() connects without blocking and then reads SO_ERROR, which holds ECONNRESET when the reset lands first. Against a listener that accepts each connection and resets it at once, the same calls raised it from create_connection() in 1 of 5,000 connections on the idle host. The scenario is in the default suite, so it can fail this workflow and CI's microVM jobs alike, and it needs a fix of its own.
What remains needs root on the runners:
On each general KVM runner, run the pass that the setup guide describes, which adds both tools and leaves the runner service alone, then check the runner:
On each runner, /opt/nvx-bootstrap holds only the setup-linux-runner.sh of the provisioning on 2026-10-28, which pins Rust 1.95.0 and installs neither tool. Unless the scale set's model has changed since then, a reimage, as by an automatic OS image upgrade, would bring back runners without the tools and without Rust 1.97.1. To keep both across reimages, regenerate and stage the Linux package that scripts/setup/azure builds.
Dispatch the workflow with gh workflow run nvx-rs-scenario-coverage.yml --repo microsoft/nvx --ref dev, or wait for its scheduled run on Sunday, 2026-11-15, at 07:41 UTC.
The issue needs no repository change, so I opened no pull request. It stays open until that run reports, as its first acceptance criterion requires.
The KVM runners now have the coverage tools, and the workflow passes. 42768407172, dispatched on dev at b6ad70074, ran on azure-kvm-5 and succeeded. Its measurement job took 15 min 33 s: test-microvm 560 s, the adapter's suite 183 s, and the crate's tests 100 s, each with status 0, no retries, and no stray profiles.
Run
Lines
Functions
The crate's tests
85.1% of 43,430
85.8% of 5,802
The scenarios
35.0% of 43,430
36.3% of 5,802
Together
85.3% of 43,430
86.1% of 5,802
The scenarios reach 82 lines that the crate's tests don't, and 57.8% of the 26,336 that their builds compile. The scenarios started 144 nvx-rs processes, and each wrote its profile. The reports are the run's 30-day artifact, nvx-rs-scenario-coverage-42768407172. Both acceptance criteria hold, so I'm closing the issue.
On the four general KVM runners, azure-kvm-1, -2, -4, and -5, I ran the setup pass that the setup guide's Rust toolchain section describes, each while the runner was idle. I used setup-linux.sh --runner-only --backend kvm without a runner name, with the script and manifest of 995230d0e. It added the llvm-tools component of Rust 1.97.1 and cargo-llvm-cov 0.9.1 to the root-owned toolchain in /opt/nvx, and left the runner services alone. Its tdnf install -y also upgraded 16 installed packages to their current Azure Linux releases on each runner, among them perl 5.38.2-517, binutils 2.41-14, rsync 3.5.1, and kernel-headers 6.6.161.1. The running kernel is still 6.6.159.1.
setup-linux.sh --runner-only --backend kvm --runner-name azure-kvm-N --check-only passes on azure-kvm-2, -4, and -5. On azure-kvm-1 it stops at GitHub Actions runner labels are not validated. That runner's /opt/nvx-runner/.nvx-labels, written when it registered on 2026-10-28, reads linux,kvm,virtual-machine, without the general label that GitHub lists for it and that the other three runners' files include. A named pass with a fresh registration token would rewrite the file. The pass's own checks, run as the runner account, confirmed that azure-kvm-1 has both tools. I left azure-kvm-3, the benchmark runner, as it was, since this workflow doesn't run there.
Two follow-ups remain outside this issue:
Each runner's /opt/nvx-bootstrap still holds only the setup-linux-runner.sh from its provisioning, which pins Rust 1.95.0. Unless the scale set's model has changed since then, a reimage would remove both tools again, along with Rust 1.97.1, until the Linux package of scripts/setup/azure is regenerated and staged.
private-network-ingress can still fail intermittently with ConnectionResetError, as my earlier comment describes, which would fail this workflow too.
Activity
ppenna commentedon Nov 8, 2026
Checked against current dev (bf81488): #1393 has merged the optional Linux/KVM scenario-coverage workflow, instrumentation/profile accounting, separate scenarios/crate-tests/together reports, missing-profile process inventory, and doc/ci.md documentation. The workflow runs only on schedule or manual dispatch, so it adds no required check.
Leaving this issue open for now: no runs of nvx-rs-scenario-coverage.yml exist yet, and #1393 explicitly identifies the first acceptance criterion as pending a CI-runner measurement after merge and runner setup. Close once a run produces the reports and verifies profile accounting for the exited nvx-rs processes.
scripts: share nvx_rs_coverage's test run and summary helpers
ppenna commentedon Nov 10, 2026
The first run of
nvx-rs-scenario-coverage.yml, the scheduled 42460108426 atbf814882b, failed at Check the coverage tools onazure-kvm-4, before it measured anything:No KVM runner has either tool yet.
setup-linux.sh --runner-onlyinstalls cargo-llvm-cov and thellvm-toolscomponent since #1333, merged on 2026-11-07, but the runners' setup predates it: their toolchain of Rust 1.97.1 dates from 2026-11-06. A read-only probe of the five KVM runners today, as my SSH user, withoutsudo:/opt/nvx/cargo/bin/cargo-llvm-covllvm-toolsof Rust 1.97.1azure-kvm-1generalvmss000000azure-kvm-2generalvmss000002azure-kvm-3benchmarkvmss000003azure-kvm-4generalvmss000004azure-kvm-5generalvmss000005Each has Rust 1.97.1 with the
x86_64-unknown-linux-musl,x86_64-unknown-none, andx86_64-unknown-uefitargets, and cargo-nextest 0.9.133. Neither the check nor the setup has changed sincebf814882b, so every run on ageneralrunner fails the same way until the runner has both tools.Nothing else blocks the run. On
nvx-kvm-test(linux-kvm,virtual-machine, Azure Linux 3), in a temporary worktree ofdevat995230d0e, with the artifacts and OpenVMM of its release,v0.1.0-dev.995230d0ebb4, the host'sllvm-toolsof Rust 1.97.1, and cargo-llvm-cov 0.9.1 built with--lockedinto a temporary directory, I ran the job's steps from Check the coverage tools on, with its commands, twice:test-microvmstopped atprivate-network-ingress, which failed withConnectionResetError: [Errno 104] Connection reset by peer, as described below. The adapter's suite and the crate's tests passed, the reports were written, and each of the 144nvx-rsstarts wrote its profiletest-microvm, all 57 scenarios, 599 s; the adapter's suite 202 s; the crate's tests 110 s, with no retries or stray profiles. Lines: the crate's tests 85.1% of 43,461, the scenarios 35.0%, together 85.3%; functions: 85.8%, 36.3%, and 86.2% of 5,806. The scenarios reach 82 lines that the crate's tests don't, and 57.8% of the 26,363 that their builds compile. Each of the 144nvx-rsstarts wrote its profileThe failure of the first run is a race in
refused(), the host-denied probe ofprivate-network-ingressinscripts/nvx_tools/microvm_tests.py. It guards onlyrecv()against the reset that the published listener sends to the host's own connections. With a timeout, though,socket.create_connection()connects without blocking and then readsSO_ERROR, which holdsECONNRESETwhen the reset lands first. Against a listener that accepts each connection and resets it at once, the same calls raised it fromcreate_connection()in 1 of 5,000 connections on the idle host. The scenario is in the default suite, so it can fail this workflow and CI's microVM jobs alike, and it needs a fix of its own.What remains needs root on the runners:
On each
generalKVM runner, run the pass that the setup guide describes, which adds both tools and leaves the runner service alone, then check the runner:On each runner,
/opt/nvx-bootstrapholds only thesetup-linux-runner.shof the provisioning on 2026-10-28, which pins Rust 1.95.0 and installs neither tool. Unless the scale set's model has changed since then, a reimage, as by an automatic OS image upgrade, would bring back runners without the tools and without Rust 1.97.1. To keep both across reimages, regenerate and stage the Linux package thatscripts/setup/azurebuilds.Dispatch the workflow with
gh workflow run nvx-rs-scenario-coverage.yml --repo microsoft/nvx --ref dev, or wait for its scheduled run on Sunday, 2026-11-15, at 07:41 UTC.The issue needs no repository change, so I opened no pull request. It stays open until that run reports, as its first acceptance criterion requires.
ppenna commentedon Nov 10, 2026
The KVM runners now have the coverage tools, and the workflow passes. 42768407172, dispatched on
devatb6ad70074, ran onazure-kvm-5and succeeded. Its measurement job took 15 min 33 s:test-microvm560 s, the adapter's suite 183 s, and the crate's tests 100 s, each with status 0, no retries, and no stray profiles.The scenarios reach 82 lines that the crate's tests don't, and 57.8% of the 26,336 that their builds compile. The scenarios started 144
nvx-rsprocesses, and each wrote its profile. The reports are the run's 30-day artifact,nvx-rs-scenario-coverage-42768407172. Both acceptance criteria hold, so I'm closing the issue.On the four
generalKVM runners,azure-kvm-1,-2,-4, and-5, I ran the setup pass that the setup guide's Rust toolchain section describes, each while the runner was idle. I usedsetup-linux.sh --runner-only --backend kvmwithout a runner name, with the script and manifest of995230d0e. It added thellvm-toolscomponent of Rust 1.97.1 and cargo-llvm-cov 0.9.1 to the root-owned toolchain in/opt/nvx, and left the runner services alone. Itstdnf install -yalso upgraded 16 installed packages to their current Azure Linux releases on each runner, among them perl 5.38.2-517, binutils 2.41-14, rsync 3.5.1, and kernel-headers 6.6.161.1. The running kernel is still 6.6.159.1.setup-linux.sh --runner-only --backend kvm --runner-name azure-kvm-N --check-onlypasses onazure-kvm-2,-4, and-5. Onazure-kvm-1it stops atGitHub Actions runner labels are not validated. That runner's/opt/nvx-runner/.nvx-labels, written when it registered on 2026-10-28, readslinux,kvm,virtual-machine, without thegenerallabel that GitHub lists for it and that the other three runners' files include. A named pass with a fresh registration token would rewrite the file. The pass's own checks, run as the runner account, confirmed thatazure-kvm-1has both tools. I leftazure-kvm-3, the benchmark runner, as it was, since this workflow doesn't run there.Two follow-ups remain outside this issue:
/opt/nvx-bootstrapstill holds only thesetup-linux-runner.shfrom its provisioning, which pins Rust 1.95.0. Unless the scale set's model has changed since then, a reimage would remove both tools again, along with Rust 1.97.1, until the Linux package ofscripts/setup/azureis regenerated and staged.private-network-ingresscan still fail intermittently withConnectionResetError, as my earlier comment describes, which would fail this workflow too.