Skip to content

ci: report the coverage of the nvx crate's tests on Linux and Windows #1323

Closed

Description

Goal

Report which lines, functions, and branches of the nvx crate's production code its own tests run, on Linux and Windows. #726 measured this only for the families of golden tests that #722 retired, and 10035af deleted its workflow once the last family was gone.

Tasks

  • Add a workflow, beside nvx-rs-mutants.yml, that runs cargo llvm-cov at a pinned release, with --all-features and NVX_BUNDLE=skip, on ubuntu-latest and windows-latest, for pushes to dev or on a schedule, and on a manual dispatch. No required check depends on it.
  • Choose the compiler: the release that rust-toolchain.toml pins, with its llvm-tools component, measures lines, regions, and functions, while branches still need a nightly compiler (-Z coverage-options=branch), which nvx-rs: measure line, branch, and mutation coverage on Linux and Windows #726 pinned apart from rust-toolchain.toml.
  • Count production code only. cargo-llvm-cov already leaves out tests/ and the tests.rs files of src/, but it counts tests/vectors/openvmm/platform_command_line_token_is_invariant.rs, which a unit test includes, and the fake OpenVMM of src/testing/, a test double of 1,611 lines on Windows. Leave out the first and report the second apart.
  • Keep the profiles of the processes that tests start: nvx-rs, nvx-fake-openvmm, and, on Linux, the guest binaries that tests/guest_tools.rs runs. Both host binaries already return their statuses from main so that Windows writes their profiles (84edb1f).
  • Carry over what nvx-rs: measure line, branch, and mutation coverage on Linux and Windows #726's script handled on Linux: let the owner read and write every profile before llvm-profdata merges them, since the store tests run processes whose umasks clear these permissions and llvm-profdata crashed on their profiles; and run failing tests once more, naming in the summary those that passed only then.
  • Upload each OS's LCOV and HTML reports as artifacts, and write the totals and a table per module to the run's summary.
  • Test the workflow as scripts/test_nvx_rs_mutants.py tests its own, its triggers, pinned tools, and artifact names, and add the new module to validate-nvx.
  • Document it in doc/ci.md, doc/project-structure.md, and the Testing section of nvx-rs/README.md.

Acceptance criteria

  • A run on dev reports both OSes from profiles that include the nvx-rs and nvx-fake-openvmm processes that tests start.
  • Test code, test data, and the fake OpenVMM don't count as production code.
  • No required check changes.

Activity

  1. ppenna commented on Nov 7, 2026

    @ppenna
    ContributorAuthor

    #1333 did the tasks of this issue, and the first runs of nvx-rs-coverage.yml on dev meet its acceptance criteria, so this closes it. As #1333 chose, the workflow measures with the Rust release that rust-toolchain.toml pins and its llvm-tools, so branches go unmeasured until they no longer need a nightly compiler.

    Runs on dev

    Run Commit Event Linux Windows
    42387413841 06e8dd5, the merge of #1333 push success success
    42398285564 faac52e, the merge of #1336 push success success

    Both ran Rust 1.97.1, which rust-toolchain.toml pins, and cargo-llvm-cov 0.9.1, which scripts/setup/tool-versions.conf pins, and uploaded nvx-rs-coverage-<os>-<run-id> with lcov.info, html/, and the summary that the run's page shows. No test ran again and no profile strayed into the crate's directory: retries and stray_profiles are empty in each measurement.json.

    Acceptance criteria

    Checked against the artifacts of run 42387413841:

    • A run on dev reports both OSes from profiles that include the nvx-rs and nvx-fake-openvmm processes that tests start: met. Each OS's summary says that the profiles include processes of nvx-rs and nvx-fake-openvmm, and on Linux of the ten guest binaries of src/bin/ too. coverage.json agrees: a binary's main runs only in its own processes, and the tests reached 5 of the 6 lines of src/bin/nvx-rs.rs and the 3 lines of src/bin/nvx-fake-openvmm.rs on both OSes.

    • Test code, test data, and the fake OpenVMM don't count as production code: met. lcov.info and html/ hold only files under src/, none in a tests directory, a tests.rs file, or src/testing/: 262 files on Linux and 164 on Windows, which no longer counts tests/vectors/openvmm/platform_command_line_token_is_invariant.rs. coverage.json adds only src/testing/fake_openvmm.rs and src/bin/nvx-fake-openvmm.rs, which the summary counts apart:

      OS Production lines Test-double lines
      Linux 84.6% of 45,142 86.4% of 1,655
      Windows 91.3% of 28,008 87.0% of 1,610
    • No required check changes: met. The workflow runs only for pushes to dev and on dispatch, dev's only required check is Required status check, and ci: report the coverage of the nvx crate's tests on Linux and Windows #1333 left ci.yml unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions