Skip to content

ci: report the coverage of the Python host tooling in scripts/ #1326

Closed

Description

Goal

Report which statements and branches of the Python host tooling in scripts/ its unit tests run.

Background

At 82c15cd, on Windows, under coverage.py 7.17.0, the modules that validate-nvx runs reach 84.3% of the 30,449 statements of scripts/ outside its test_*.py modules, and 75.7% of their branches. The least-covered modules are scripts/nvx_tools/microvm_tests.py, benchmark.py, adversarial_oracles.py, adversarial_executor.py, ubuntu.py, and build.py.

Measured with the Python subprocesses that they start, the modules took 16 minutes and reported 7 errors among 1,909 tests. Without coverage, the same host reports 4 errors in subtests of test_host_loopback_allow_script_checks_publication_and_traffic, whose shell script exceeds its 10-second timeout there, and none in test_adversarial.py, which reported one under coverage.

Tasks

  • Pin coverage.py, for example in requirements-dev.txt.
  • Run the modules that validate-nvx runs under coverage run --branch, measuring the Python subprocesses that the tests start (coverage.py's patch = subprocess), in a job of its own on ubuntu-latest, so that the slower tests delay no other job.
  • Leave the test_*.py modules out of the report as well as the measurement, and report the scenario drivers of scripts/nvx_tools/*_tests.py apart, since only a hypervisor runs them end to end.
  • Make every test that passes without coverage pass under it, starting with the one of test_adversarial.py.
  • Write the totals and a table per module to the run's summary, upload the reports, test the wiring with the other script tests, and document it in doc/ci.md.

Acceptance criteria

  • The report counts no test module.
  • Every test that passes without coverage passes under it.
  • No required check changes.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions