Conversation
Signed-off-by: Jiri Jaburek <comps@nomail.dom>
Collaborator
Author
|
/packit test -i contest-all |
|
@comps: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Collaborator
Author
|
https://artifacts.dev.testing-farm.io/7c8cdb4a-de7d-4c81-99d4-700df449cc98/work-plan7_usds_l/plan/execute/data/guest/default-0/test-1/data/index.html?q=status+NOT+IN+%28%27pass%27%2C+%27warn%27%2C+%27skip%27%29+ORDER+BY+platform%2C+test are the results from the |
macko1
requested changes
Sep 17, 2026
macko1
self-requested a review
September 17, 2026 11:37
Collaborator
Author
|
Regarding excluded tests - we'd need to copy/paste duplicate the list between CaC/content and Contest (both use the Packit action) and only some of the excludes list would be relevant to the "users" of the testing (you) like "not running disa-alignment", ... some other exclude items are technical limitations (tests that cannot run in containers for now). I considered using a Contest tmt plan for this, but it would be copy/pasting |
macko1
requested changes
Sep 18, 2026
macko1
left a comment
macko1
left a comment
Collaborator
There was a problem hiding this comment.
Code seems correct.
After research, I'd like to suggest these docs updates.
Comment on lines
+57
to
+76
| # Contest-based testing where the contest-atex-testingfarm is run via Packit | ||
| # on a huge VM host, setting up many containers/VMs to parallelize the testing, | ||
| # also allowing it to run tests that require VMs | ||
| # - this effectively allows full Contest suite execution (all tests that make | ||
| # sense for PR testing here) | ||
|
|
||
| - job: tests | ||
| trigger: pull_request | ||
| fmf_url: https://github.com/RHSecurityCompliance/contest-atex-testingfarm.git | ||
| fmf_ref: main | ||
| identifier: contest-all | ||
| targets: [fedora-latest-stable] | ||
| manual_trigger: true | ||
| skip_build: true | ||
| tf_extra_params: | ||
| # increase default, per README.md of the above repo | ||
| # - this includes time spent waiting in queued | ||
| settings: | ||
| pipeline: | ||
| timeout: 1200 |
Collaborator
There was a problem hiding this comment.
Suggested change
| # Contest-based testing where the contest-atex-testingfarm is run via Packit | |
| # on a huge VM host, setting up many containers/VMs to parallelize the testing, | |
| # also allowing it to run tests that require VMs | |
| # - this effectively allows full Contest suite execution (all tests that make | |
| # sense for PR testing here) | |
| - job: tests | |
| trigger: pull_request | |
| fmf_url: https://github.com/RHSecurityCompliance/contest-atex-testingfarm.git | |
| fmf_ref: main | |
| identifier: contest-all | |
| targets: [fedora-latest-stable] | |
| manual_trigger: true | |
| skip_build: true | |
| tf_extra_params: | |
| # increase default, per README.md of the above repo | |
| # - this includes time spent waiting in queued | |
| settings: | |
| pipeline: | |
| timeout: 1200 | |
| # Contest-based testing where the contest-atex-testingfarm is run via Packit | |
| # on a huge VM host, setting up containers/VMs to parallelize the testing, | |
| # also allowing it to run tests that require VMs | |
| # - this effectively allows full Contest suite execution (all tests that make | |
| # sense for PR testing here) | |
| - job: tests | |
| trigger: pull_request | |
| fmf_url: https://github.com/RHSecurityCompliance/contest-atex-testingfarm.git | |
| fmf_ref: main | |
| identifier: contest-all | |
| targets: [fedora-latest-stable] | |
| # Do not run this expensive job automatically for every pull request update. | |
| # To trigger it, post a github comment: `/packit test -i contest-all` | |
| manual_trigger: true | |
| skip_build: true # Skip the Copr RPM build; The content is built in containers in this plan. | |
| # Uncomment to run the selected Contest plan without the excluded tests defined in | |
| # contest-atex-testingfarm. | |
| # env: | |
| # NO_EXCLUDES: "1" | |
| tf_extra_params: | |
| settings: | |
| # Allow up to 20 hours once Testing Farm starts running the request. | |
| pipeline: | |
| timeout: 1200 |
Comment on lines
+107
to
+134
|
|
||
| ## Packit (on Github) | ||
|
|
||
| On Github, this project uses [Packit](https://packit.dev/) to build content as a scratch `scap-security-guide` RPM and run additional tests, all configured via `.packit.yaml` in the root of the repository. | ||
|
|
||
| Some of these tests execute [Contest](https://github.com/RHSecurityCompliance/contest), an extensive test suite for RHEL / CentOS Stream. Simple test sets are run automatically on a Pull Request push, but bigger runs require manual triggering. | ||
|
|
||
| To trigger a full set of Contest tests, write a comment that **begins with**: | ||
|
|
||
| ``` | ||
| /packit test -i contest-all | ||
| ``` | ||
|
|
||
| You can also parametrize it further with: | ||
|
|
||
| - `PLAN` to override the default `/plans/daily` (tmt plan in Contest) | ||
| - `TESTS` as comma-separated test name fmf-style expressions | ||
| - `RERUNS` to override the default 1 automatic rerun of every failed test | ||
| - `CONTEST_PR` to test a specific Contest PR instead of the `main` branch | ||
| - `NO_EXCLUDES=1` to run even tests normally incompatible with containers or unsuitable for PR CI | ||
|
|
||
| For example, to test all CIS profile variants (incl. non-daily): | ||
|
|
||
| ``` | ||
| /packit test -i contest-all --env RERUNS=0 --env PLAN=/plans/weekly --env TESTS=/cis$,/cis_server,/cis_workstation | ||
| ``` | ||
|
|
||
| Please **use this test sparingly** as it uses a LOT of resources and will take several hours to run, ideally run it only on a final version of a Pull Request, not on every small change. Limiting by `TESTS` helps only slightly - the bulk of the runtime is spent setting up the testing environment (a constant cost). |
Collaborator
There was a problem hiding this comment.
Suggested change
| ## Packit (on Github) | |
| On Github, this project uses [Packit](https://packit.dev/) to build content as a scratch `scap-security-guide` RPM and run additional tests, all configured via `.packit.yaml` in the root of the repository. | |
| Some of these tests execute [Contest](https://github.com/RHSecurityCompliance/contest), an extensive test suite for RHEL / CentOS Stream. Simple test sets are run automatically on a Pull Request push, but bigger runs require manual triggering. | |
| To trigger a full set of Contest tests, write a comment that **begins with**: | |
| ``` | |
| /packit test -i contest-all | |
| ``` | |
| You can also parametrize it further with: | |
| - `PLAN` to override the default `/plans/daily` (tmt plan in Contest) | |
| - `TESTS` as comma-separated test name fmf-style expressions | |
| - `RERUNS` to override the default 1 automatic rerun of every failed test | |
| - `CONTEST_PR` to test a specific Contest PR instead of the `main` branch | |
| - `NO_EXCLUDES=1` to run even tests normally incompatible with containers or unsuitable for PR CI | |
| For example, to test all CIS profile variants (incl. non-daily): | |
| ``` | |
| /packit test -i contest-all --env RERUNS=0 --env PLAN=/plans/weekly --env TESTS=/cis$,/cis_server,/cis_workstation | |
| ``` | |
| Please **use this test sparingly** as it uses a LOT of resources and will take several hours to run, ideally run it only on a final version of a Pull Request, not on every small change. Limiting by `TESTS` helps only slightly - the bulk of the runtime is spent setting up the testing environment (a constant cost). | |
| ## Contest tests running on Testing Farm with Packit | |
| The repository defines its [Packit](https://packit.dev/) jobs in | |
| [`.packit.yaml`](../../../.packit.yaml). | |
| The logs of the test runs are available in the pull request as Testing Farm artifacts. | |
| Some of the jobs have to be triggered manually by posting a comment in the pull request. | |
| ### Trigger all Packit test jobs | |
| Post `/packit test` as a pull request comment to trigger all Packit test jobs. | |
| IMPORTANT: This also triggers the [`contest-all` job](#contest-all-job-modified-plansdaily-plan) job | |
| ### Trigger a specific Packit job with modified options | |
| Post `/packit test -i JOB_ID` as a pull request comment to trigger one job. Replace `JOB_ID` with the | |
| job `identifier` from [`.packit.yaml`](../../../.packit.yaml). Pass job-specific settings with Packit's | |
| `--env NAME=value` syntax. For example, this runs `contest-host-os-ansible` without automatic retries: | |
| ```text | |
| /packit test -i contest-host-os-ansible --env RERUNS=0 | |
| ``` | |
| Some of the available options are: | |
| - `PLAN=/plans/name` selects a Contest TMT plan. The default is `/plans/daily`. | |
| - `TESTS=regex1,regex2` selects test names matching one of the comma-separated Python regular expressions. | |
| - `RERUNS=N` sets automatic retries for failed tests. The default is `1`. | |
| - `CONTEST_PR=N` tests Contest pull request `N` instead of Contest's `main` branch. | |
| For example, this runs selected CIS tests from the weekly plan without automatic retries: | |
| ```text | |
| /packit test -i contest-all --env RERUNS=0 --env PLAN=/plans/weekly --env TESTS=/cis$,/cis_server,/cis_workstation | |
| ``` | |
| ### `contest-all` job | |
| The `contest-all` job uses a modified Contest's `/plans/daily` plan with some tests excluded. The plan is run in containers or in VMs for tests when containers cannot be used. The excluded tests are hardcoded in `contest-atex-testingfarm` repo. The `NO_EXCLUDES=1` option includes these tests (but does not guarantee that all tests will pass). | |
| IMPORTANT: `contest-all` can take several hours to complete and consume significant resources, so use it sparingly. | |
| Post this comment in the pull request to trigger the `contest-all` job: | |
| ```text | |
| /packit test -i contest-all | |
| ``` | |
| Add `--env NO_EXCLUDES=1` to run tests excluded by default. | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
This adds
/packit test -i contest-allto run nearly the full Contest test suite against CentOS Stream 8, 9 and 10.This is basically almost the entire "daily productization" we do for RHEL, minus a few tests that either don't make sense in a PR CI by default (DISA alignment, which fails mainly due to external random reasons) or cannot run on containers yet (see Contest upstream GH issues).
The big idea of this test is to reserve a giant "bare metal" x86_64 machine from Testing Farm (via Packit), sponsored by AWS, and set up VM-capable containers for CS8/9/10 on it, and run tests in parallel. This means Testing Farm sees just 1 "job" instead of 60-80 jobs per PR, like the older attempt at doing this kind of testing (which landed us in hot water w.r.t. Testing Farm folks).
Currently, some of the test fail and I believe that's due to content bugs, but this PR at least gives the option to run the testing manually and hopefully eliminate the issues over time.
The PR also renames the existing
contest-*Packit tests tocontest-host-os-*to reflect that they (mainly) run/hardnening/host-os, in contrast tocontest-allrunning everything else except/hardening/host-os. We could perhaps renamecontest-host-os-tocontest-smoke-or something like that if you want.Rationale:
More testing for an upstream PR avoids finding these issues downstream when a PR is already merged into
master. Lessproductization-issueGH issues, more stablemaster.Review Hints:
tests/run_tests_testingfarm.pyalready inmasterfrom a previous attempt at doing this. This PR uses a completely different approach, so we might want to remove the unusedtests/run_tests_testingfarm.py, maybe @ggbecker can chime in.