Skip to content

ci: measure the nvx-rs host code that the KVM microVM scenarios reach #1325

Closed

Description

Goal

Measure which host code of nvx-rs the microVM scenarios and the runtime adapter's suite reach on Linux/KVM, alone and together with the crate's own tests, so that the code that only a real VM reaches is measured too.

Background

The crate's own tests leave most of what they don't reach in code that needs a VM or the network. At 82c15cd, on Windows, they reach 75.8% of the lines of src/host/instance/, 66.2% of the lines of src/host/template/clone/launch.rs, and 65.3% of the branches of src/host/template/store/capture.rs. nvx.py and the microVM scenarios run the nvx-rs that NVX_RS names (doc/usage.md), so they can run an instrumented build.

Tasks

  • On a Linux/KVM runner, build an instrumented nvx-rs, for example in the environment that cargo llvm-cov show-env prints, point NVX_RS at it, and set LLVM_PROFILE_FILE to a pattern that gives each process a profile of its own, such as %p-%m.profraw.
  • Run test-microvm --backend kvm and test-runtime-adapter --backend kvm with it, on demand or on a schedule, never as a required check.
  • Merge their profiles, and report what the scenarios reach alone and together with the crate's tests, with the exclusions of ci: report the coverage of the nvx crate's tests on Linux and Windows #1323.
  • List the nvx-rs processes that the scenarios kill, since a killed process writes no profile.
  • Document it in doc/ci.md.

Out of scope

  • The binaries that run inside guests: their profiles would stay in the guest, and the images build them with the guest profile, with LTO and stripped.
  • MSHV and WHP, which can follow once KVM works.
  • OpenVMM, whose own CI owns its coverage.

Acceptance criteria

  • A run reports what the scenarios reach, alone and with the crate's tests, from the profile of every nvx-rs process that exited.
  • No required check changes.

Activity

  1. ppenna commented on Nov 8, 2026

    @ppenna
    ContributorAuthor

    Checked against current dev (bf81488): #1393 has merged the optional Linux/KVM scenario-coverage workflow, instrumentation/profile accounting, separate scenarios/crate-tests/together reports, missing-profile process inventory, and doc/ci.md documentation. The workflow runs only on schedule or manual dispatch, so it adds no required check.

    Leaving this issue open for now: no runs of nvx-rs-scenario-coverage.yml exist yet, and #1393 explicitly identifies the first acceptance criterion as pending a CI-runner measurement after merge and runner setup. Close once a run produces the reports and verifies profile accounting for the exited nvx-rs processes.

  2. ppenna commented on Nov 10, 2026

    @ppenna
    ContributorAuthor

    The first run of nvx-rs-scenario-coverage.yml, the scheduled 42460108426 at bf814882b, failed at Check the coverage tools on azure-kvm-4, before it measured anything:

    cargo-llvm-cov 0.9.1, which scripts/setup/tool-versions.conf pins, is not installed; rerun scripts/setup/setup-linux.sh --runner-only
    

    No KVM runner has either tool yet. setup-linux.sh --runner-only installs cargo-llvm-cov and the llvm-tools component since #1333, merged on 2026-11-07, but the runners' setup predates it: their toolchain of Rust 1.97.1 dates from 2026-11-06. A read-only probe of the five KVM runners today, as my SSH user, without sudo:

    Runner Labels Instance Booted /opt/nvx/cargo/bin/cargo-llvm-cov llvm-tools of Rust 1.97.1
    azure-kvm-1 general vmss000000 2026-10-28 Missing Missing
    azure-kvm-2 general vmss000002 2026-10-28 Missing Missing
    azure-kvm-3 benchmark vmss000003 2026-10-28 Missing Missing
    azure-kvm-4 general vmss000004 2026-10-28 Missing Missing
    azure-kvm-5 general vmss000005 2026-10-28 Missing Missing

    Each has Rust 1.97.1 with the x86_64-unknown-linux-musl, x86_64-unknown-none, and x86_64-unknown-uefi targets, and cargo-nextest 0.9.133. Neither the check nor the setup has changed since bf814882b, so every run on a general runner fails the same way until the runner has both tools.

    Nothing else blocks the run. On nvx-kvm-test (linux-kvm, virtual-machine, Azure Linux 3), in a temporary worktree of dev at 995230d0e, with the artifacts and OpenVMM of its release, v0.1.0-dev.995230d0ebb4, the host's llvm-tools of Rust 1.97.1, and cargo-llvm-cov 0.9.1 built with --locked into a temporary directory, I ran the job's steps from Check the coverage tools on, with its commands, twice:

    Run Result
    First Status 1. test-microvm stopped at private-network-ingress, which failed with ConnectionResetError: [Errno 104] Connection reset by peer, as described below. The adapter's suite and the crate's tests passed, the reports were written, and each of the 144 nvx-rs starts wrote its profile
    Second Status 0 in 16 min: test-microvm, all 57 scenarios, 599 s; the adapter's suite 202 s; the crate's tests 110 s, with no retries or stray profiles. Lines: the crate's tests 85.1% of 43,461, the scenarios 35.0%, together 85.3%; functions: 85.8%, 36.3%, and 86.2% of 5,806. The scenarios reach 82 lines that the crate's tests don't, and 57.8% of the 26,363 that their builds compile. Each of the 144 nvx-rs starts wrote its profile

    The failure of the first run is a race in refused(), the host-denied probe of private-network-ingress in scripts/nvx_tools/microvm_tests.py. It guards only recv() against the reset that the published listener sends to the host's own connections. With a timeout, though, socket.create_connection() connects without blocking and then reads SO_ERROR, which holds ECONNRESET when the reset lands first. Against a listener that accepts each connection and resets it at once, the same calls raised it from create_connection() in 1 of 5,000 connections on the idle host. The scenario is in the default suite, so it can fail this workflow and CI's microVM jobs alike, and it needs a fix of its own.

    What remains needs root on the runners:

    1. On each general KVM runner, run the pass that the setup guide describes, which adds both tools and leaves the runner service alone, then check the runner:

      scp scripts/setup/setup-linux.sh scripts/setup/tool-versions.conf azure-kvm-N:/tmp/
      ssh azure-kvm-N 'sh /tmp/setup-linux.sh --runner-only --backend kvm'
      ssh azure-kvm-N 'sh /tmp/setup-linux.sh --runner-only --backend kvm --runner-name azure-kvm-N --check-only'
    2. On each runner, /opt/nvx-bootstrap holds only the setup-linux-runner.sh of the provisioning on 2026-10-28, which pins Rust 1.95.0 and installs neither tool. Unless the scale set's model has changed since then, a reimage, as by an automatic OS image upgrade, would bring back runners without the tools and without Rust 1.97.1. To keep both across reimages, regenerate and stage the Linux package that scripts/setup/azure builds.

    3. Dispatch the workflow with gh workflow run nvx-rs-scenario-coverage.yml --repo microsoft/nvx --ref dev, or wait for its scheduled run on Sunday, 2026-11-15, at 07:41 UTC.

    The issue needs no repository change, so I opened no pull request. It stays open until that run reports, as its first acceptance criterion requires.

  3. ppenna commented on Nov 10, 2026

    @ppenna
    ContributorAuthor

    The KVM runners now have the coverage tools, and the workflow passes. 42768407172, dispatched on dev at b6ad70074, ran on azure-kvm-5 and succeeded. Its measurement job took 15 min 33 s: test-microvm 560 s, the adapter's suite 183 s, and the crate's tests 100 s, each with status 0, no retries, and no stray profiles.

    Run Lines Functions
    The crate's tests 85.1% of 43,430 85.8% of 5,802
    The scenarios 35.0% of 43,430 36.3% of 5,802
    Together 85.3% of 43,430 86.1% of 5,802

    The scenarios reach 82 lines that the crate's tests don't, and 57.8% of the 26,336 that their builds compile. The scenarios started 144 nvx-rs processes, and each wrote its profile. The reports are the run's 30-day artifact, nvx-rs-scenario-coverage-42768407172. Both acceptance criteria hold, so I'm closing the issue.

    On the four general KVM runners, azure-kvm-1, -2, -4, and -5, I ran the setup pass that the setup guide's Rust toolchain section describes, each while the runner was idle. I used setup-linux.sh --runner-only --backend kvm without a runner name, with the script and manifest of 995230d0e. It added the llvm-tools component of Rust 1.97.1 and cargo-llvm-cov 0.9.1 to the root-owned toolchain in /opt/nvx, and left the runner services alone. Its tdnf install -y also upgraded 16 installed packages to their current Azure Linux releases on each runner, among them perl 5.38.2-517, binutils 2.41-14, rsync 3.5.1, and kernel-headers 6.6.161.1. The running kernel is still 6.6.159.1.

    setup-linux.sh --runner-only --backend kvm --runner-name azure-kvm-N --check-only passes on azure-kvm-2, -4, and -5. On azure-kvm-1 it stops at GitHub Actions runner labels are not validated. That runner's /opt/nvx-runner/.nvx-labels, written when it registered on 2026-10-28, reads linux,kvm,virtual-machine, without the general label that GitHub lists for it and that the other three runners' files include. A named pass with a fresh registration token would rewrite the file. The pass's own checks, run as the runner account, confirmed that azure-kvm-1 has both tools. I left azure-kvm-3, the benchmark runner, as it was, since this workflow doesn't run there.

    Two follow-ups remain outside this issue:

    • Each runner's /opt/nvx-bootstrap still holds only the setup-linux-runner.sh from its provisioning, which pins Rust 1.95.0. Unless the scale set's model has changed since then, a reimage would remove both tools again, along with Rust 1.97.1, until the Linux package of scripts/setup/azure is regenerated and staged.
    • private-network-ingress can still fail intermittently with ConnectionResetError, as my earlier comment describes, which would fail this workflow too.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions