From 62d3b5bf3bcb9aa416987c5547b9efd9129e8f14 Mon Sep 17 00:00:00 2001 From: politerealism Date: Sun, 27 Sep 2026 09:08:28 -0400 Subject: [PATCH 1/3] test(podman): wire podman_preflight into CI and add coverage drift check podman_preflight (added in #3690) never ran in CI: it needs only the standalone openshell-driver-podman binary, but the external-driver job that already builds and exports that artifact disables cargo test entirely for its own unrelated purpose. Chain a new e2e:podman:preflight mise task onto that job's existing command instead of routing through the gateway-backed PODMAN_CI_TESTS harness this test doesn't need. Add tasks/scripts/check-podman-e2e-coverage.sh (podman-e2e-coverage job in branch-checks.yml) so a new e2e-podman-eligible test target can't silently go unselected the way the 17 targets in #3712 did. It enumerates eligible targets via cargo metadata plus each auto-discovered file's own #![cfg(feature = ...)] gate, and requires every one to be accounted for in PODMAN_CI_TESTS, a perf-benchmark ignore list (verified via source inspection that every test in those files actually carries #[ignore], since cargo test -- --list does not distinguish ignored tests), a separately-wired list for tests like podman_preflight that don't fit the gateway harness, or a temporary, itemized known-gaps list tied to #3712 for the pre-existing gaps this check surfaces on introduction. The check also fails on stale known-gaps entries so that list can't rot in the other direction. Signed-off-by: politerealism --- .github/workflows/branch-checks.yml | 17 ++ .github/workflows/branch-e2e.yml | 2 +- tasks/scripts/check-podman-e2e-coverage.sh | 228 +++++++++++++++++++++ tasks/test.toml | 8 + 4 files changed, 254 insertions(+), 1 deletion(-) create mode 100755 tasks/scripts/check-podman-e2e-coverage.sh diff --git a/.github/workflows/branch-checks.yml b/.github/workflows/branch-checks.yml index 04f1dc87ea..2b5a7ef94f 100644 --- a/.github/workflows/branch-checks.yml +++ b/.github/workflows/branch-checks.yml @@ -143,6 +143,23 @@ jobs: shell: nix develop -c bash -euo pipefail {0} run: tasks/scripts/check-cargo-lockfiles.sh + podman-e2e-coverage: + name: Podman e2e coverage + needs: pr_metadata + if: needs.pr_metadata.outputs.should_run == 'true' + runs-on: linux-amd64-cpu8 + timeout-minutes: 15 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - uses: ./.github/actions/setup-nix + with: + cachix-auth-token: ${{ secrets.CACHIX_AUTH_TOKEN }} + + - name: Verify Podman e2e coverage + shell: nix develop -c bash -euo pipefail {0} + run: tasks/scripts/check-podman-e2e-coverage.sh + rust-lint: name: Rust lint (${{ matrix.system }}) needs: pr_metadata diff --git a/.github/workflows/branch-e2e.yml b/.github/workflows/branch-e2e.yml index 4efff1f6c8..28f44c8d0e 100644 --- a/.github/workflows/branch-e2e.yml +++ b/.github/workflows/branch-e2e.yml @@ -333,7 +333,7 @@ jobs: external-driver-binary: openshell-driver-podman conformance-artifact-prefix: openshell-conformance suite-matrix: >- - [{"suite":"external-driver","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:external-driver"}] + [{"suite":"external-driver","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:external-driver && mise run --no-deps --skip-deps e2e:podman:preflight"}] vm-external-driver-e2e: needs: [pr_metadata, build-binaries, build-gateway-plain, build-vm-driver] diff --git a/tasks/scripts/check-podman-e2e-coverage.sh b/tasks/scripts/check-podman-e2e-coverage.sh new file mode 100755 index 0000000000..9194af86ae --- /dev/null +++ b/tasks/scripts/check-podman-e2e-coverage.sh @@ -0,0 +1,228 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Every e2e/rust test target eligible under the `e2e-podman` Cargo feature +# must be accounted for somewhere: selected by the required-CI test set in +# e2e/rust/e2e-podman.sh, or explicitly ignored as a perf benchmark below. A +# target that is eligible but appears in neither list is a silent CI coverage +# gap (see https://github.com/NVIDIA/OpenShell/issues/3712) and fails this +# check. + +set -euo pipefail + +ROOT="$(git rev-parse --show-toplevel)" +E2E_MANIFEST="${ROOT}/e2e/rust/Cargo.toml" +PODMAN_SCRIPT="${ROOT}/e2e/rust/e2e-podman.sh" + +# Features enabled when building with --features e2e-podman (the feature +# itself plus everything it implies; see e2e/rust/Cargo.toml's [features]). +ENABLED_FEATURES=(e2e-podman e2e e2e-host-gateway e2e-local-container-driver) + +# Perf benchmarks are intentionally excluded from required CI and run +# manually. Every test function in these binaries must be #[ignore]d so a +# newly added, non-ignored test can't silently ride along unexamined. +PERF_BENCHMARK_TARGETS=(internet_network_perf live_internet_traffic_perf) + +# Targets wired into required CI through a mechanism other than +# PODMAN_CI_TESTS, with where and why noted so this doesn't just become a +# second silent-omission list. +# +# - podman_preflight: needs only the standalone openshell-driver-podman +# binary, not a gateway, so it runs as a chained step in the +# podman-external-driver-e2e job (branch-e2e.yml) via the +# e2e:podman:preflight mise task instead of through the gateway-backed +# PODMAN_CI_TESTS array. +SEPARATELY_WIRED_TARGETS=(podman_preflight) + +# Pre-existing gaps this check finds on introduction, tracked collectively by +# https://github.com/NVIDIA/OpenShell/issues/3712 pending the follow-up work +# that either wires each one into CI or gives it its own permanent, justified +# exclusion. Remove an entry here only when it is actually accounted for +# above (selected, ignored, or separately wired) -- do not grow this list for +# new gaps; new eligible targets must be handled properly from the start. +KNOWN_GAP_TARGETS=( + # Superseded by the rootful/rootless driver-podman tmachine suite; slated + # for removal once that migration lands (tracked in #3712). + podman_userns + # Deliberately distinct-scope exclusions needing their own follow-up work, + # per #3712's triage: SPIFFE fixture plumbing, a purpose-built long-running + # rotation suite, and a multi-case conformance migration respectively. + podman_oci_identity + provider_refresh_handles + sandbox_lifecycle + # Already tracked separately in #3009 (needs a prebuilt musl DNS probe in + # guest artifact mode); not duplicated here. + transparent_tcp + # Plausible quick fixes once triaged against a live Podman run; not yet + # investigated. + podman_resource_limits + port_forward + provider_auto_create + proxy_egress_pipeline + sandbox_labels + sandbox_templates + settings_management + sync + upload_create + websocket_conformance + workspace_lifecycle +) + +is_enabled_feature() { + local feature="$1" + local candidate + for candidate in "${ENABLED_FEATURES[@]}"; do + [[ "${candidate}" == "${feature}" ]] && return 0 + done + return 1 +} + +is_in_array() { + local needle="$1" + shift + local candidate + for candidate in "$@"; do + [[ "${candidate}" == "${needle}" ]] && return 0 + done + return 1 +} + +# Read the PODMAN_CI_TESTS bash array literal out of e2e-podman.sh without +# sourcing the whole script (which also runs the gateway harness). +read_podman_ci_tests() { + sed -n '/^PODMAN_CI_TESTS=(/,/^)/p' "${PODMAN_SCRIPT}" \ + | sed '1d;$d' \ + | tr -d ' \t' +} + +# Eligibility under e2e-podman for one cargo-metadata test target: explicit +# required-features must all be enabled-by-e2e-podman features; targets with +# no manifest-level required-features (auto-discovered files) fall back to +# the file's own #![cfg(feature = "...")] gate, or are eligible unconditionally +# if the file has no such gate at all. +target_is_eligible() { + local name="$1" + local src_path="$2" + shift 2 + local required_features=("$@") + + if [[ "${#required_features[@]}" -gt 0 ]]; then + local feature + for feature in "${required_features[@]}"; do + is_enabled_feature "${feature}" || return 1 + done + return 0 + fi + + local cfg_feature + cfg_feature="$(head -n 20 "${src_path}" \ + | grep -m1 -oP '^#!\[cfg\(feature\s*=\s*"\K[^"]+' || true)" + if [[ -z "${cfg_feature}" ]]; then + return 0 + fi + is_enabled_feature "${cfg_feature}" +} + +verify_perf_benchmarks_are_ignored() { + # cargo test -- --list does not distinguish #[ignore]d tests from runnable + # ones in its plain output, so check the source directly: every #[test]/ + # #[tokio::test] attribute must be immediately followed by #[ignore...]. + local target + local src_path + local failed=0 + for target in "${PERF_BENCHMARK_TARGETS[@]}"; do + src_path="${ROOT}/e2e/rust/tests/${target}.rs" + local non_ignored + non_ignored="$(awk ' + /^[[:space:]]*#\[(tokio::)?test\][[:space:]]*$/ { pending = 1; next } + pending && /^[[:space:]]*#\[ignore/ { pending = 0; next } + pending { print; pending = 0 } + ' "${src_path}" | grep -c . || true)" + if [[ "${non_ignored}" -ne 0 ]]; then + printf 'error: perf benchmark target "%s" (%s) has %s non-#[ignore]d test(s); a new test there would silently skip required-CI accounting\n' \ + "${target}" "${src_path#"${ROOT}"/}" "${non_ignored}" >&2 + failed=1 + fi + done + return "${failed}" +} + +main() { + cd "${ROOT}" + + local -a ci_tests + mapfile -t ci_tests < <(read_podman_ci_tests) + + local -a unaccounted=() + local -a stale_known_gaps=() + local -a seen_eligible=() + local accounted_count=0 + + while IFS=$'\t' read -r name src_path required_features_csv; do + local -a required_features=() + if [[ -n "${required_features_csv}" ]]; then + IFS=',' read -r -a required_features <<<"${required_features_csv}" + fi + + if ! target_is_eligible "${name}" "${src_path}" "${required_features[@]}"; then + continue + fi + seen_eligible+=("${name}") + + if is_in_array "${name}" "${ci_tests[@]}" \ + || is_in_array "${name}" "${PERF_BENCHMARK_TARGETS[@]}" \ + || is_in_array "${name}" "${SEPARATELY_WIRED_TARGETS[@]}"; then + accounted_count=$((accounted_count + 1)) + is_in_array "${name}" "${KNOWN_GAP_TARGETS[@]}" && stale_known_gaps+=("${name}") + elif is_in_array "${name}" "${KNOWN_GAP_TARGETS[@]}"; then + accounted_count=$((accounted_count + 1)) + else + unaccounted+=("${name}") + fi + done < <(cargo metadata --manifest-path "${E2E_MANIFEST}" --no-deps --format-version 1 \ + | jq -r ' + .packages[].targets[] + | select(.kind == ["test"]) + | [.name, .src_path, (."required-features" // [] | join(","))] + | @tsv + ') + + local gap + for gap in "${KNOWN_GAP_TARGETS[@]}"; do + is_in_array "${gap}" "${seen_eligible[@]}" || stale_known_gaps+=("${gap}") + done + + local eligible_count=$((accounted_count + ${#unaccounted[@]})) + + if [[ "${#unaccounted[@]}" -gt 0 ]]; then + printf 'error: %d e2e-podman-eligible test target(s) are not accounted for in required Podman CI:\n' \ + "${#unaccounted[@]}" >&2 + local name + for name in "${unaccounted[@]}"; do + printf ' - %s\n' "${name}" >&2 + done + printf 'Add each to PODMAN_CI_TESTS in %s, or mark it an ignored perf benchmark with a rationale.\n' \ + "${PODMAN_SCRIPT#"${ROOT}"/}" >&2 + return 1 + fi + + if [[ "${#stale_known_gaps[@]}" -gt 0 ]]; then + printf 'error: %d entries in KNOWN_GAP_TARGETS no longer belong there (already accounted for elsewhere, or no longer an eligible target) -- remove them:\n' \ + "${#stale_known_gaps[@]}" >&2 + local name + for name in "${stale_known_gaps[@]}"; do + printf ' - %s\n' "${name}" >&2 + done + return 1 + fi + + if ! verify_perf_benchmarks_are_ignored; then + return 1 + fi + + printf 'Podman e2e coverage: %d eligible target(s), %d accounted for (%d selected, %d perf-ignored, %d separately wired)\n' \ + "${eligible_count}" "${accounted_count}" "${#ci_tests[@]}" "${#PERF_BENCHMARK_TARGETS[@]}" "${#SEPARATELY_WIRED_TARGETS[@]}" +} + +main "$@" diff --git a/tasks/test.toml b/tasks/test.toml index 39dd070204..10690b7fdd 100644 --- a/tasks/test.toml +++ b/tasks/test.toml @@ -268,6 +268,14 @@ env = { OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER = "1", OPENSHELL_E2E_PODMAN_FEATUR depends = ["e2e:conformance:build"] run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-podman.sh" +["e2e:podman:preflight"] +description = "Run the Podman driver daemon-unavailable preflight e2e test against the standalone driver artifact" +run = "cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-podman --test podman_preflight -- --nocapture" + +["e2e:podman:coverage:check"] +description = "Verify every e2e-podman-eligible test target is accounted for in required Podman CI" +run = "tasks/scripts/check-podman-e2e-coverage.sh" + ["e2e:vm:external-driver"] description = "Run VM E2E with a driver-free gateway and external VM driver binary" env = { OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER = "1" } From 3838d16eda11ced290b36813bd89c2d524993199 Mon Sep 17 00:00:00 2001 From: politerealism Date: Sun, 27 Sep 2026 09:08:36 -0400 Subject: [PATCH 2/3] docs(ci): document the Podman e2e test selection contract No file described which e2e-podman-eligible test targets run where or how the exclusion mechanism works, making the gap in #3712 invisible from the docs as well as the code. Document PODMAN_CI_TESTS, the perf-ignore and known-gaps lists, and the new coverage-check script that enforces them. Signed-off-by: politerealism --- CI.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/CI.md b/CI.md index 1eb087f2a6..481e13053a 100644 --- a/CI.md +++ b/CI.md @@ -415,6 +415,19 @@ The bot's full administrator documentation is internal to NVIDIA. The only comma | `.github/workflows/trivy-changes.yml` | Blocks pull requests and merge groups that introduce new High or Critical Helm or Dockerfile misconfigurations. | | `.github/workflows/trivy-scan.yml` | Manual or reusable scan of supplied OCI image/chart references and deployment configuration. Findings are informational by default and can be configured to fail the workflow. | +## Podman e2e test selection + +`e2e/rust/tests/` has 39 test targets eligible under the `e2e-podman` Cargo feature (explicit `[[test]] required-features = ["e2e-podman"]` entries in `e2e/rust/Cargo.toml`, or auto-discovered files gated by `#![cfg(feature = "e2e")]`/`#![cfg(feature = "e2e-host-gateway")]`, both implied by `e2e-podman`). Required branch CI runs a curated subset selected by the `PODMAN_CI_TESTS` array in `e2e/rust/e2e-podman.sh` (`podman-e2e` job, `branch-e2e.yml`), not the full unfiltered set. + +Every eligible target must be accounted for in one of: + +- `PODMAN_CI_TESTS` in `e2e/rust/e2e-podman.sh`. +- A small, checked-in perf-benchmark ignore list, for targets that are manual benchmarks rather than CI-gated regressions. Every test in those files must carry `#[ignore]`. +- A separate CI mechanism, when a target's dependencies don't fit the gateway-backed `PODMAN_CI_TESTS` harness (for example `podman_preflight`, which needs only the standalone `openshell-driver-podman` binary and runs as a chained step in the `podman-external-driver-e2e` job instead). +- A temporary, itemized known-gaps list, each entry tied to a tracking issue, for pre-existing gaps found when this check was introduced (see #3712). + +`tasks/scripts/check-podman-e2e-coverage.sh` (the `podman-e2e-coverage` job in `branch-checks.yml`, and `mise run e2e:podman:coverage:check` locally) enforces this: it fails if a target is eligible but appears in none of the above, and fails if a known-gaps entry no longer belongs there. A new `e2e-podman`-eligible test file must be added to one of these lists or the check fails the PR. + ## Release workflows These workflows run after merge to publish dev/tagged artifacts and verify them. They are not PR-gated. From b4fd8be97d18e333fb304e46a19d319208e7c463 Mon Sep 17 00:00:00 2001 From: politerealism Date: Sun, 27 Sep 2026 09:18:17 -0400 Subject: [PATCH 3/3] fix(ci): don't let podman_preflight get masked by external-driver flakiness The chained "&&" command meant a failing or flaky e2e:podman:external-driver run would silently prevent e2e:podman:preflight from ever executing, recreating the exact invisible-coverage problem #3712 describes, just relocated to this job. Run both unconditionally and combine their exit codes instead. Signed-off-by: politerealism --- .github/workflows/branch-e2e.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/branch-e2e.yml b/.github/workflows/branch-e2e.yml index 28f44c8d0e..2f279dd015 100644 --- a/.github/workflows/branch-e2e.yml +++ b/.github/workflows/branch-e2e.yml @@ -333,7 +333,7 @@ jobs: external-driver-binary: openshell-driver-podman conformance-artifact-prefix: openshell-conformance suite-matrix: >- - [{"suite":"external-driver","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:external-driver && mise run --no-deps --skip-deps e2e:podman:preflight"}] + [{"suite":"external-driver","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:external-driver; s1=$?; mise run --no-deps --skip-deps e2e:podman:preflight; s2=$?; [ \"$s1\" -eq 0 ] && [ \"$s2\" -eq 0 ]"}] vm-external-driver-e2e: needs: [pr_metadata, build-binaries, build-gateway-plain, build-vm-driver]