Skip to content

Improve test suite performance via module-scoped fixtures and batched… - #666

Open
ehelms wants to merge 1 commit into
theforeman:masterfrom
ehelms:improve-test-fixture-scoping
Open

Improve test suite performance via module-scoped fixtures and batched…#666
ehelms wants to merge 1 commit into
theforeman:masterfrom
ehelms:improve-test-fixture-scoping

Conversation

@ehelms

@ehelms ehelms commented Jul 15, 2026

Copy link
Copy Markdown
Member

Why are you introducing these changes? (Problem description, related links)

Test runs were spending a large fraction of their wall-clock time recreating
the same remote resources for every test. The ten Foreman/Katello resource
fixtures (organizations, products, repositories, lifecycle environments, content
views, activation keys) were all function-scoped by default, so each individual
test paid the full create-over-API cost. Additionally, two pulp-manager tests
each spawned a separate Django process inside the container, doubling that
startup cost, and inherently-slow tests (backup, target lifecycle) were mixed
with fast ones making it impossible to exclude them during development.

Fixture call reduction (measured with --fixture-durations):

Fixture Before calls Before total After calls After total Saved
Pulp manager Django startup¹ 2 87.5s 1 24.3s 63.2s
yum_repository 4 61.2s 2 10.8s 50.4s
client_environment 2 35.2s 1 13.6s 21.6s
organization 10 23.1s 3 7.9s 15.2s
product 7 3.5s 2 1.1s 2.4s
Total 210.5s 57.7s 152.8s

¹ Before, each test launched its own pulpcore-manager shell process (45.0s + 42.5s in test call time); after, both tests share one module-scoped fixture.

Test phase overhead (pytest --durations grand totals):

Phase Before After Saved
Test setup 156s 2s 154s
Test teardown 120s <1s ~120s

Skippable slow tests (with -m "not slow"):

Test module Duration
backup_test.py ~2m 1s
target_lifecycle_test.py ~2m 46s
Skippable total ~4m 47s

What are the changes introduced in this pull request?

  • tests/conftest.py — Promote all remote resource fixtures to
    scope="module" so resources are created once per test module, not once per
    test. Expand content_view teardown to handle version cleanup (previously
    left to test bodies, which broke with shared fixtures). Simplify
    client_environment teardown to delegate version cleanup to content_view.
    Change wait_for_metadata_generate to filter with state != stopped instead
    of fetching all historical tasks. Register the slow marker in
    pytest_configure.
  • tests/pulp_test.py — Add _PULP_MANAGER_SCRIPT constant and module-scoped
    pulp_manager_info fixture that runs one Django startup via
    pulpcore-manager shell (base64-encoded to avoid shell quoting issues).
    test_pulp_manager_check and test_pulp_import_export_settings now consume
    the shared result instead of each launching their own container process.
  • tests/feature/katello/api_test.py — Remove per-test content view version
    cleanup loop from test_foreman_content_view; the content_view fixture
    teardown now handles this.
  • tests/backup_test.py — Add pytestmark = pytest.mark.slow.
  • tests/target_lifecycle_test.py — Add pytestmark = pytest.mark.slow.

How to test this pull request

Steps to reproduce:

  • Deploy a foremanctl environment and run the full test suite:
    python -m pytest tests/ --durations=20 --fixture-durations=10 -v
    
  • Verify resource fixtures appear once per module in the setup column, not once
    per test
  • Deselect slow tests and confirm the run completes faster:
    python -m pytest tests/ -m "not slow"
    
  • Confirm the content_view teardown cleans up published versions without
    error when the katello feature is enabled

Checklist

  • Tests added/updated (if applicable)
  • Documentation updated (if applicable)

Comment thread tests/pulp_test.py
PULP_API_SOCKET = '/run/httpd.pulp-api.sock'
PULP_CONTENT_SOCKET = '/run/httpd.pulp-content.sock'

# Run both check --deploy and settings inspection in a single Django startup.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's got to be a better / cleaner way to do this, consider this just a "hack" for now.

@ehelms
ehelms force-pushed the improve-test-fixture-scoping branch from dcef7e0 to e936b6f Compare July 16, 2026 14:44
… calls

Promote all remote resource fixtures (organization, product, repositories,
lifecycle_environment, content_view, activation_key) from function scope to
module scope so each is created once per test module rather than once per test.
Batch two pulp-manager Django startup costs into a single podman exec call.
Mark inherently-slow tests (backup, target lifecycle) with pytest.mark.slow
so they can be excluded during development with -m "not slow".

Co-Authored-By: Claude <noreply@anthropic.com>
@ehelms
ehelms force-pushed the improve-test-fixture-scoping branch from e936b6f to 5b2ea19 Compare July 24, 2026 18:31
@ehelms
ehelms marked this pull request as ready for review July 24, 2026 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant