Skip to content

Simplify test infrastructure - #664

Open
ehelms wants to merge 4 commits into
theforeman:masterfrom
ehelms:simplify-test-infrastructure
Open

Simplify test infrastructure#664
ehelms wants to merge 4 commits into
theforeman:masterfrom
ehelms:simplify-test-infrastructure

Conversation

@ehelms

@ehelms ehelms commented Jul 15, 2026

Copy link
Copy Markdown
Member

Why are you introducing these changes? (Problem description, related links)

Running tests required going through forge test, an Ansible playbook whose
only jobs were installing nmap on the VM, generating an ssh-config from the
inventory, and shelling out to pytest. This indirection made it harder to run
tests directly, required a VM-side package install before every test run, and
obscured what the test runner was actually doing.

What are the changes introduced in this pull request?

  • Replace nmap --script ssl-enum-ciphers in candlepin_test.py with
    openssl s_client to verify TLS version support. openssl is already
    present on the VM as a Candlepin prerequisite, eliminating the only reason
    forge test needed to install a package on the VM before running tests.
  • Move ssh-config generation from forge test into a session-scoped autouse
    pytest fixture (generate_ssh_config in tests/conftest.py). The fixture
    reads the static inventory (defaulting to inventories/local_vagrant,
    overridable via OBSAH_INVENTORY) and writes .tmp/ssh-config before any
    test runs, so pytest can be invoked directly without going through forge.
  • Simplify development/playbooks/test/test.yaml to just run pytest now that
    the nmap install and ssh-config generation are handled elsewhere.
  • Add make test and make test-unit targets as a direct interface to the
    test suite. make test runs the full integration suite; make test-unit
    runs only tests/unit/ with no VM required.
  • Update CI to use make test instead of ./forge test.

How to test this pull request

Steps to reproduce:

  • Run unit tests locally without a VM:

    make test-unit
    

    All tests in tests/unit/ should pass with no VM or deployed environment.

  • Run the full integration suite against a deployed environment:

    make test
    

    Confirm tests/feature/katello/candlepin_test.py::test_tls passes without
    nmap installed on the VM.

  • Confirm forge test still works (now a thin wrapper around pytest):

    ./forge test
    

Checklist

  • Tests added/updated (if applicable)
  • Documentation updated (if applicable)

ehelms and others added 4 commits July 15, 2026 11:53
nmap was the only test dependency requiring a package install on the VM.
openssl is already present as a Candlepin prerequisite and can probe
specific TLS versions directly.

Co-Authored-By: Claude <noreply@anthropic.com>
Move ssh-config generation from the forge test Ansible playbook into a
session-scoped pytest fixture. This allows running tests directly with
pytest or make without going through forge, and removes the only
remaining non-trivial work the forge test playbook was doing.

Co-Authored-By: Claude <noreply@anthropic.com>
make test runs the full integration suite via pytest.
make test-unit runs only the local unit tests with no VM dependency.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@ehelms

ehelms commented Jul 15, 2026

Copy link
Copy Markdown
Member Author

There are things in here I think we should do for sure, the open question is testing interface:

  • forge for consistency with development actions
  • make since it will stream the testing output real-time rather than dump it at the end
  • pytest directly which has the benefits of direct interaction, but the downsides of having to "remember" the baseline set of options we run it with today: --durations=10 -vv

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant