Skip to content

Implement contest-atex-testingfarm based contest-all Packit test #15126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
comps wants to merge 1 commit into
base: master
Choose a base branch
from
+63 −7
Open

Implement contest-atex-testingfarm based contest-all Packit test #15126

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 35 additions & 7 deletions .packit.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,17 @@

actions:
get-current-version:
- bash utils/version.sh

Check failure on line 8 in .packit.yaml

View workflow job for this annotation

GitHub Actions / Yaml Lint on Changed yaml files 

8:3 [indentation] wrong indentation: expected 4 but found 2

srpm_build_deps:
- bash

jobs:
- &build

Check failure on line 14 in .packit.yaml

View workflow job for this annotation

GitHub Actions / Yaml Lint on Changed yaml files 

14:1 [indentation] wrong indentation: expected 2 but found 0
job: copr_build
trigger: pull_request
targets:
- fedora-all-x86_64

Check failure on line 18 in .packit.yaml

View workflow job for this annotation

GitHub Actions / Yaml Lint on Changed yaml files 

18:3 [indentation] wrong indentation: expected 4 but found 2
- centos-stream-8-x86_64
- centos-stream-9-x86_64
- centos-stream-10-x86_64
Expand All @@ -24,13 +24,19 @@
trigger: commit
branch: "gh-readonly-queue/.*"

- &contest-oscap
# Contest-based /hardening/host-os testing
# - this is a minimal set of tests that can run directly on the reserved OS
# by remediating + scanning it
# - let Packit handle it natively by installing built scap-security-guide RPM
# and calling Contest to test that OS-installed RPM content

- &contest-host-os-oscap
job: tests
trigger: pull_request
fmf_url: https://github.com/RHSecurityCompliance/contest.git
fmf_ref: main
tmt_plan: /plans/upstream-parallel/oscap
identifier: contest-oscap
identifier: contest-host-os-oscap
targets:
centos-stream-8: {}
centos-stream-9: {}
Expand All @@ -40,15 +46,37 @@
pipeline:
parallel-limit: 32

- <<: *contest-oscap
- <<: *contest-host-os-oscap
tmt_plan: /plans/upstream-parallel/ansible
identifier: contest-ansible
identifier: contest-host-os-ansible

- <<: *contest-oscap
- <<: *contest-host-os-oscap
tmt_plan: /plans/upstream-parallel/other
identifier: contest-other
identifier: contest-host-os-other

# Contest-based testing where the contest-atex-testingfarm is run via Packit
# on a huge VM host, setting up many containers/VMs to parallelize the testing,
# also allowing it to run tests that require VMs
# - this effectively allows full Contest suite execution (all tests that make
# sense for PR testing here)

- job: tests
trigger: pull_request
fmf_url: https://github.com/RHSecurityCompliance/contest-atex-testingfarm.git
fmf_ref: main
identifier: contest-all
targets: [fedora-latest-stable]
manual_trigger: true
skip_build: true
tf_extra_params:
# increase default, per README.md of the above repo
# - this includes time spent waiting in queued
settings:
pipeline:
timeout: 1200
Comment on lines +57 to +76

@macko1 macko1 Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Contest-based testing where the contest-atex-testingfarm is run via Packit
# on a huge VM host, setting up many containers/VMs to parallelize the testing,
# also allowing it to run tests that require VMs
# - this effectively allows full Contest suite execution (all tests that make
# sense for PR testing here)
- job: tests
trigger: pull_request
fmf_url: https://github.com/RHSecurityCompliance/contest-atex-testingfarm.git
fmf_ref: main
identifier: contest-all
targets: [fedora-latest-stable]
manual_trigger: true
skip_build: true
tf_extra_params:
# increase default, per README.md of the above repo
# - this includes time spent waiting in queued
settings:
pipeline:
timeout: 1200
# Contest-based testing where the contest-atex-testingfarm is run via Packit
# on a huge VM host, setting up containers/VMs to parallelize the testing,
# also allowing it to run tests that require VMs
# - this effectively allows full Contest suite execution (all tests that make
# sense for PR testing here)
- job: tests
trigger: pull_request
fmf_url: https://github.com/RHSecurityCompliance/contest-atex-testingfarm.git
fmf_ref: main
identifier: contest-all
targets: [fedora-latest-stable]
# Do not run this expensive job automatically for every pull request update.
# To trigger it, post a github comment: `/packit test -i contest-all`
manual_trigger: true
skip_build: true # Skip the Copr RPM build; The content is built in containers in this plan.
# Uncomment to run the selected Contest plan without the excluded tests defined in
# contest-atex-testingfarm.
# env:
# NO_EXCLUDES: "1"
tf_extra_params:
settings:
# Allow up to 20 hours once Testing Farm starts running the request.
pipeline:
timeout: 1200


# when modifying anything below, modify also tests/tmt/
# tests present locally in this repo, under tests/
# - when modifying these, modify also tests/tmt/

- job: tests
trigger: pull_request
Expand Down
28 changes: 28 additions & 0 deletions docs/manual/developer/08_content_tests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,3 +104,31 @@ You should replace `0.1.76` with the latest release of the project.

The test `test_stig_rules_in_srg_gpos.py` ensures that all rules selected in RHEL 10 STIG profile are also selected in SRG GPOS control files.
The test prevents data inconsistencies and verifies that the STIG profile remains based on SRG GPOS controls.

## Packit (on Github)

On Github, this project uses [Packit](https://packit.dev/) to build content as a scratch `scap-security-guide` RPM and run additional tests, all configured via `.packit.yaml` in the root of the repository.

Some of these tests execute [Contest](https://github.com/RHSecurityCompliance/contest), an extensive test suite for RHEL / CentOS Stream. Simple test sets are run automatically on a Pull Request push, but bigger runs require manual triggering.

To trigger a full set of Contest tests, write a comment that **begins with**:

```
/packit test -i contest-all
```

You can also parametrize it further with:

- `PLAN` to override the default `/plans/daily` (tmt plan in Contest)
- `TESTS` as comma-separated test name fmf-style expressions
- `RERUNS` to override the default 1 automatic rerun of every failed test
- `CONTEST_PR` to test a specific Contest PR instead of the `main` branch
- `NO_EXCLUDES=1` to run even tests normally incompatible with containers or unsuitable for PR CI

For example, to test all CIS profile variants (incl. non-daily):

```
/packit test -i contest-all --env RERUNS=0 --env PLAN=/plans/weekly --env TESTS=/cis$,/cis_server,/cis_workstation
```

Please **use this test sparingly** as it uses a LOT of resources and will take several hours to run, ideally run it only on a final version of a Pull Request, not on every small change. Limiting by `TESTS` helps only slightly - the bulk of the runtime is spent setting up the testing environment (a constant cost).
Comment on lines +107 to +134

@macko1 macko1 Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## Packit (on Github)
On Github, this project uses [Packit](https://packit.dev/) to build content as a scratch `scap-security-guide` RPM and run additional tests, all configured via `.packit.yaml` in the root of the repository.
Some of these tests execute [Contest](https://github.com/RHSecurityCompliance/contest), an extensive test suite for RHEL / CentOS Stream. Simple test sets are run automatically on a Pull Request push, but bigger runs require manual triggering.
To trigger a full set of Contest tests, write a comment that **begins with**:
```
/packit test -i contest-all
```
You can also parametrize it further with:
- `PLAN` to override the default `/plans/daily` (tmt plan in Contest)
- `TESTS` as comma-separated test name fmf-style expressions
- `RERUNS` to override the default 1 automatic rerun of every failed test
- `CONTEST_PR` to test a specific Contest PR instead of the `main` branch
- `NO_EXCLUDES=1` to run even tests normally incompatible with containers or unsuitable for PR CI
For example, to test all CIS profile variants (incl. non-daily):
```
/packit test -i contest-all --env RERUNS=0 --env PLAN=/plans/weekly --env TESTS=/cis$,/cis_server,/cis_workstation
```
Please **use this test sparingly** as it uses a LOT of resources and will take several hours to run, ideally run it only on a final version of a Pull Request, not on every small change. Limiting by `TESTS` helps only slightly - the bulk of the runtime is spent setting up the testing environment (a constant cost).
## Contest tests running on Testing Farm with Packit
The repository defines its [Packit](https://packit.dev/) jobs in
[`.packit.yaml`](../../../.packit.yaml).
The logs of the test runs are available in the pull request as Testing Farm artifacts.
Some of the jobs have to be triggered manually by posting a comment in the pull request.
### Trigger all Packit test jobs
Post `/packit test` as a pull request comment to trigger all Packit test jobs.
IMPORTANT: This also triggers the [`contest-all` job](#contest-all-job-modified-plansdaily-plan) job
### Trigger a specific Packit job with modified options
Post `/packit test -i JOB_ID` as a pull request comment to trigger one job. Replace `JOB_ID` with the
job `identifier` from [`.packit.yaml`](../../../.packit.yaml). Pass job-specific settings with Packit's
`--env NAME=value` syntax. For example, this runs `contest-host-os-ansible` without automatic retries:
```text
/packit test -i contest-host-os-ansible --env RERUNS=0
```
Some of the available options are:
- `PLAN=/plans/name` selects a Contest TMT plan. The default is `/plans/daily`.
- `TESTS=regex1,regex2` selects test names matching one of the comma-separated Python regular expressions.
- `RERUNS=N` sets automatic retries for failed tests. The default is `1`.
- `CONTEST_PR=N` tests Contest pull request `N` instead of Contest's `main` branch.
For example, this runs selected CIS tests from the weekly plan without automatic retries:
```text
/packit test -i contest-all --env RERUNS=0 --env PLAN=/plans/weekly --env TESTS=/cis$,/cis_server,/cis_workstation
```
### `contest-all` job
The `contest-all` job uses a modified Contest's `/plans/daily` plan with some tests excluded. The plan is run in containers or in VMs for tests when containers cannot be used. The excluded tests are hardcoded in `contest-atex-testingfarm` repo. The `NO_EXCLUDES=1` option includes these tests (but does not guarantee that all tests will pass).
IMPORTANT: `contest-all` can take several hours to complete and consume significant resources, so use it sparingly.
Post this comment in the pull request to trigger the `contest-all` job:
```text
/packit test -i contest-all
```
Add `--env NO_EXCLUDES=1` to run tests excluded by default.

Loading