You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: report the coverage of the Python host tooling in scripts/ #1326
Report which statements and branches of the Python host tooling in scripts/ its unit tests run.
Background
At 82c15cd, on Windows, under coverage.py 7.17.0, the modules that validate-nvx runs reach 84.3% of the 30,449 statements of scripts/ outside its test_*.py modules, and 75.7% of their branches. The least-covered modules are scripts/nvx_tools/microvm_tests.py, benchmark.py, adversarial_oracles.py, adversarial_executor.py, ubuntu.py, and build.py.
Measured with the Python subprocesses that they start, the modules took 16 minutes and reported 7 errors among 1,909 tests. Without coverage, the same host reports 4 errors in subtests of test_host_loopback_allow_script_checks_publication_and_traffic, whose shell script exceeds its 10-second timeout there, and none in test_adversarial.py, which reported one under coverage.
Tasks
Pin coverage.py, for example in requirements-dev.txt.
Run the modules that validate-nvx runs under coverage run --branch, measuring the Python subprocesses that the tests start (coverage.py's patch = subprocess), in a job of its own on ubuntu-latest, so that the slower tests delay no other job.
Leave the test_*.py modules out of the report as well as the measurement, and report the scenario drivers of scripts/nvx_tools/*_tests.py apart, since only a hypervisor runs them end to end.
Make every test that passes without coverage pass under it, starting with the one of test_adversarial.py.
Write the totals and a table per module to the run's summary, upload the reports, test the wiring with the other script tests, and document it in doc/ci.md.
Acceptance criteria
The report counts no test module.
Every test that passes without coverage passes under it.
Activity
ci: report the coverage of the Python host tooling in scripts/
ci: report the coverage of the Python host tooling in scripts/
adversarial: start the containment tests' timeouts once descendants run