diff --git a/README.rst b/README.rst index 23cf6b3..0f09809 100644 --- a/README.rst +++ b/README.rst @@ -138,6 +138,13 @@ releases. Developers may be interested in reading further about the `Plugin`_ system. +Tests +----- + +The test suite runs with pytest, without root, network or a real console. +What it needs, the commands CI runs and the coverage threshold are in +`tests/README.md`_. + .. _GPLv3: https://www.gnu.org/licenses/gpl-3.0.txt .. _Confconsole documentation source: https://github.com/turnkeylinux/confconsole/blob/master/docs/Readme.rst .. _Plugin: ./docs/Plugins.rst @@ -148,5 +155,6 @@ Developers may be interested in reading further about the `Plugin`_ system. .. _Region config: ./docs/Region_config.rst .. _System settings: ./docs/System_settings.rst .. _Instance: ./docs/Instance.rst +.. _tests/README.md: ./tests/README.md .. _TurnKey Linux Appliances: https://www.turnkeylinux.org/all .. _support forums: https://www.turnkeylinux.org/forum/support diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000..9603747 --- /dev/null +++ b/tests/README.md @@ -0,0 +1,82 @@ +# Tests + +## What is needed + +Python 3.13 (the version CI runs) with +`pytest`, `coverage` and `pyyaml`: + + python3 -m pip install pytest coverage pyyaml + +Nothing else: no root, no network, no real console and no installed +confconsole package. `tests/conftest.py` puts the repository root on +`sys.path` and, when they are not importable, registers stand-ins for the +Debian-only modules the code imports: `netinfo` (turnkey-netinfo), +`dialog` (pythondialog), `systemd.journal` and `requests`. No dialog +opens: the screens are driven by a scripted fake console. No test touches +`/etc` or the live network. + +The tests target Linux, as CI does (`ubuntu-latest`): some use +`os.getuid()` and Unix file modes, so on Windows run them under WSL. + +Where `/tmp` carries an ACL that makes new files writable by others +(GitHub Codespaces does: `ls -ld /tmp` ends in `+`), the tests that +write the Keel Cloud flag fail with "writable by group or others". Give +pytest a directory without one by adding `--basetemp=$HOME/pytmp/run` to +the commands below. + +## Running them as CI does + +From the repository root. The modules to measure are read from `package:` +in `.github/workflows/tests.yml`, the list CI uses, so the command stays +right when a module is added: + + MODULES=$(python -c "import yaml; print(yaml.safe_load(open('.github/workflows/tests.yml'))['jobs']['tests']['with']['package'])") + PYTHONPATH=. python -m coverage run --branch --source="$MODULES" -m pytest -q tests + python -m coverage report --show-missing + +CI fails when the report is under its threshold; to check that locally: + + python -m coverage report --show-missing --fail-under=100 + +The threshold is 100 percent, lines and branches, for the measured modules +(set in `.github/workflows/tests.yml`, only ever raised). Which modules are +measured, why `confconsole.py` is not, and the plan for the rest live in +`COVERAGE.md`. + +To run the tests without coverage, or one file or test: + + python -m pytest -q tests + python -m pytest -q tests/test_keelfit.py + python -m pytest -q tests -k static6 + +## Skipped tests + +A few tests skip unless a real keel is present: the +`TestWithTheRealKeel` classes in `test_keelmenu.py` (needs the `keel` +Python package) and in `test_first_boot.py` and `test_overlay_screen.py` +(needs a `keel` command of version 0.11 or later, found on `PATH` or named +by the `KEEL` environment variable). `python -m pytest -rs tests` lists +them with the reason. + +## Layout + +- `conftest.py`: shared fixtures, the module stand-ins and the scripted + fake console. +- Configuration file: `test_conf.py` (`confconsole.conf`, read and + `default_nic` written back). +- Network: `test_ifutil_parse.py` (stanza parsing and rendering), + `test_ifutil_interfaces.py` (the interfaces file, on a scratch copy), + `test_ifutil_system.py` (ifup/ifdown and resolvconf wrappers, stubbed), + `test_ifutil_static6.py` (the static IPv6 writer), + `test_confconsole_ifconf6.py` (the static IPv6 dialog). +- Main screen and boxes: `test_confconsole_usage.py` (the usage screen), + `test_keelbanner.py` (the Keel mark above it), `test_keelfit.py` and + `test_confconsole_boxes.py` (box sizes on the terminal), `test_brand.py` + (Keel Linux, not TurnKey, on screen). +- Keel screens: `test_keelcli.py` (the keel client), `test_keelmenu.py` + (which screens a machine shows), `test_instance_menu.py` (the Instance + menu entries), `test_database_mode.py` and `test_database_handout.py` + (database mode), `test_overlay_screen.py` (the overlay network), + `test_first_boot.py` (first boot role and Keel Cloud key). +- Let's Encrypt: `test_lets_encrypt.py` (the certificate screen and the + instance description).