Add new criu kokkos test to the rocm-tests coverage

[arunkumar-AMD]

Motivation

CRIU (Checkpoint/Restore In Userspace) with the AMD amdgpu_plugin is what lets a running GPU workload be checkpointed to disk and later restored — the foundation for fault tolerance, preemption, and migration of long-running HPC jobs on AMD GPUs. That checkpoint/restore path over a real HIP compute workload built from a large CMake project was not covered by our end-to-end framework. This PR automates checkpoint/restore of the Kokkos performance benchmark as a marker-compliant rocm-tests e2e test, so the GPU C/R path (criu dump → criu restore with amdgpu_plugin) is validated against a live HIP process as part of our recovery coverage. This is a correctness check, not a throughput benchmark: "good" means Kokkos builds with HIP, a running Kokkos_PerformanceTest_Benchmark is checkpointed cleanly, and the restored process resumes as the same workload on the GPU.

Technical Details

• Extended the existing CRIU recovery suite: tests/e2e/recovery/criu/ (new test_criu_kokkos.py; a session-scoped kokkos_build fixture added to the existing conftest.py, whose module docstring now documents it as shared across all CRIU test modules).
• Session fixtures own provisioning; the tests own execution. kokkos_build clones Kokkos at a pinned ref (ROCM_TEST_KOKKOS_REF, default 4.2.01) through the remote-transparent external_build.clone_repo with a license-presence guard, then CMake configure (CMAKE_CXX_COMPILER=hipcc, Kokkos_ENABLE_HIP=On, Kokkos_ENABLE_BENCHMARKS=On, Kokkos_ENABLE_SERIAL=on, multiple-kernel-instantiations on, RDC off) + make -j, producing build/core/perf_test/Kokkos_PerformanceTest_Benchmark. criu_runtime (shared with the cuda_memtest tests) ensures CRIU + amdgpu_plugin are ready, auto-installing via scripts/install_criu.py when missing. The tests only launch/checkpoint/restore via target_executor, so the same test runs unchanged on local bare-metal, remote SSH, and container profiles.
• Two tests sharing one checkpoint: test_criu_check_point_kokkos launches Kokkos_PerformanceTest_Benchmark -- benchmark_counters_tabular=true, captures the exact PID ($!), and criu dumps it; test_criu_restore_kokkos criu restores the image and verifies the workload resumed. The checkpoint is materialized once per run and reused, so either test also runs standalone.
• Reused suite infrastructure: the generic criu dump / criu restore / log-attach / kill helpers in _criu_steps.py, the criu_runtime
  readiness+auto-install fixture, and the shared _checkpoint / _restore step helpers — no CRIU logic was forked.
• Build only the benchmark target: make builds just Kokkos_PerformanceTest_Benchmark (not the full tree / unit tests), which is run in place with no make install. The Kokkos unit-test executables link the system libgtest.a, a non-PIC static archive on some hosts that fails with R_X86_64_32... recompile with -fPIC; the benchmark links google-benchmark, not gtest, so this sidesteps the failure with the configure line unchanged. Kokkos sources are not modified.
• Offload arch: left to the compiler's auto-detection on the node — no Kokkos_ARCH flag is passed for discrete GPUs — except the gfx942 APU, which is detected via rocminfo and built with -DKokkos_ARCH_AMD_GFX942_APU=On, HSA_XNACK=1, and a Release build (HSA_XNACK=1 is also exported at launch). A preflight checks the cloned Kokkos tree for the detected GFX target and skips with an actionable hint (bump ROCM_TEST_KOKKOS_REF) if the pinned ref can't build it — self-adapting as the ref is bumped, with no static arch list to maintain.
• Backstop: the dump step requires CRIU to report OK and the checkpointed PID to be gone (PID_GONE); the restore step requires CRIU's Restore finished successfully marker and an exact-PID liveness check plus a /proc//cmdline match for Kokkos_PerformanceTest_Benchmark — so a dump that leaves the process alive, or a restore that doesn't bring back the real workload, is never reported as a pass.
• The build tree is namespaced under the framework-managed external checkout (recovery/kokkos/kokkos-/build). Markers are injected by the CATEGORY_PROFILES entry for tests/e2e/recovery/criu in taxonomy.py — hw.gpu, layer.runtime, ci.weekly, e2e.stack, os.linux; runtime.medium is declared on the test functions. NOTICES.md updated with the Kokkos component (Apache-2.0 WITH LLVM-exception; cloned/built at runtime, unmodified, not redistributed).

Test Plan

Ran checkpoint/restore of the Kokkos benchmark end-to-end through the framework on an AMD GPU node (gfx90a, ROCm 7.14.0):

Required

• --rock-dir /opt/rocm-7.14.0 (or ROCK_DIR / rocm-test.toml) — else the build skips
• Linux AMD GPU node with passwordless sudo (CRIU needs root) — else the suite skips cleanly
• CRIU + amdgpu_plugin (auto-installed on the node if absent)

Optional env

• ROCM_TEST_KOKKOS_URL (default https://github.com/kokkos/kokkos.git)
• ROCM_TEST_KOKKOS_REF (default 4.2.01; set to pin a tag/branch/SHA — newer parts such as gfx950/gfx1250 need a newer Kokkos)
• ROCM_TEST_CRIU_AUTO_INSTALL (default 1), ROCM_TEST_CRIU_VERSION (default v4.1)

Optional flags

• -s (live CRIU log output), --pre-install pkg=

Invocation
pytest tests/e2e/recovery/criu/test_criu_kokkos.py -v --rock-dir <therock directory>

Verified the framework clones + builds Kokkos with the HIP backend, launches the benchmark on an acquired GPU, checkpoints the running process (dump → OK + PID_GONE), restores it (Restore finished successfully), and confirms the resumed PID is still the Kokkos_PerformanceTest_Benchmark workload — with the exit/marker backstops enforced.

Test Result

CRIU Kokkos checkpoint/restore: 2 passed, 0 failed, 0 skipped, 0 error (2 passed in 71.66s).

• criu dump of the running Kokkos_PerformanceTest_Benchmark: Dumping finished successfully → OK + PID_GONE (checkpoint test, ~6.8 s).
• criu restore: Restore finished successfully. Tasks resumed. → RESUMED_OK (/proc//cmdline still Kokkos_PerformanceTest_Benchmark) (restore test, ~2.7 s).
• Time taken: ~72 s for clone → CMake/make build (+ CRIU readiness) → launch → checkpoint → restore, on a 2×GPU (gfx90a) node.

[mparamas-amd]

most comments for PR#63, PR#65-66 applies for this PR.

[mparamas-amd]

xdist gap