Add criu memory testcase by arunkumar-AMD · Pull Request #63 · ROCm/rocm-tests

arunkumar-AMD · 2026-07-28T10:23:05Z

Motivation

CRIU (Checkpoint/Restore In Userspace) freezes a running process and checkpoints its full state to disk so it can later be restored and resume where it left off. On AMD GPUs this relies on the ROCm KFD checkpoint/restore hooks plus AMD's amdgpu_plugin.so, which serialize per-process GPU state (VRAM BOs, user-mode queues, events). That surface had no e2e coverage. This PR automates the CRIU cuda_memtest checkpoint/restore path — a correctness check, not a benchmark: "good" means a running HIP workload can be criu dumped and criu restored and resumes correctly on the GPU, including when the image is archived and restored from a different directory.

Technical Details

New test area tests/e2e/recovery/criu/: test_criu_cuda_memtest.py, test_criu_zip_unzip.py, plus conftest.py, _criu_steps.py,
scripts/install_criu.py, NOTICES.md, init.py.

Provisioning (session fixtures). conftest.py clones cuda_memtest at a pinned ref, hipifies with hipify-perl, applies the hipHostGetDevicePointer cast patch, builds via hipcc -DENABLE_NVML=0 … -lpthread; criu_runtime ensures CRIU + amdgpu_plugin.so are ready (criu check), auto-installing via install_criu.py when missing. Tests run through target_executor (local bare-metal or remote SSH unchanged). Build tree namespaced under compiler_build_dir/recovery/cuda_memtest.
test_criu_cuda_memtest.py (2 tests). test_criu_check_point_* (launch → criu dump) and test_criu_restore_* (criu restore → verify resumed). Launch+dump is materialized once (module cache) and reused by restore, so a run does launch ×1 / dump ×1 / restore ×1; either test still runs standalone.
test_criu_zip_unzip.py (1 test). Spec ROCM_CRIU_SYS_0012: launch → criu dump → tar -cvf the image → tar -xvf into a fresh dir → criu restore from there → verify resumed. Reuses _launch/_checkpoint from the checkpoint suite and helpers from _criu_steps; tarball lives outside the workdir so it never archives itself.
Reusable additions. _criu_steps.py (criu_dump, criu_restore w/ backgrounded restore + restore.log poll, attach_criu_log, kill_pid); hardened install_criu.py (build prereqs incl. asciidoc/xmlto, builds the amdgpu plugin, verifies with criu check).
Sizing & privilege. --max_num_blocks = round(GB)*1000 − 2000 from total VRAM via rocm-smi/amd-smi (<rock_dir>/bin on PATH). CRIU runs under sudo -n env PATH=… criu; without passwordless sudo -n the suite skips cleanly.
Backstops. dump → OK + PID_GONE; zip/unzip also asserts ZIP_OK, UNZIP_OK, .img count > 0; restore → RESTORE_OK + resumed PID alive with /proc//cmdline still the binary (RESUMED_OK). PID captured via $!. cuda_memtest (NCSA) and CRIU (GPL-2.0) fetched/built at runtime, never vendored (NOTICES.md).
Markers (via CATEGORY_PROFILES): hw.gpu, layer.runtime, ci.weekly, e2e.stack, os.linux; runtime.medium per function.

Test Plan

Required: --rock-dir (or ROCK_DIR/rocm-test.toml); Linux AMD GPU node with passwordless sudo; supported GFX (gfx90a/908/942/950, gfx110x/gfx120x).

Optional env: ROCM_TEST_CUDA_MEMTEST_URL/_REF, ROCM_TEST_CRIU_AUTO_INSTALL (default 1), ROCM_TEST_CRIU_VERSION (default v4.1).

Optional flags: -s (full logs inline), --remote-node, --pre-install pkg=…. Run single-process (no -n) — the shared checkpoint isn't xdist-safe.

pytest tests/e2e/recovery/criu/test_criu_cuda_memtest.py --rock-dir <therock path>
pytest tests/e2e/recovery/criu/test_criu_zip_unzip.py   --rock-dir <therock path>

Test Result

Run on an AMD gfx90a node →

test_criu_cuda_memtest.py — 2 passed (66.94 s). checkpoint PASSED (35.5 s: dump OK, PID_GONE); restore PASSED (23.0 s: RESTORE_OK, RESUMED_OK).

- test_criu_zip_unzip.py — 1 passed (74.29 s). dump OK/PID_GONE → ZIP_OK (CRIU_TAR_BYTES=307,271,680) → UNZIP_OK (CRIU_RESTORE_IMG_COUNT=47) → RESTORE_OK → RESUMED_OK.

mparamas-amd · 2026-08-03T08:27:27Z

+# (materialized on first use so either test still runs standalone) and the on-disk image is reused
+# by the restore test. NOTE: not xdist-safe -- with -n the tests may split across workers, each
+# re-dumping; run these single-process.
+_CHECKPOINT: dict = {}


Add @pytest.mark.xdist_group("criu_cuda_memtest_serial") to both test functions, or add a pytest.ini_options addopts entry scoped to this directory. CI runs may use parallel

Done. Both test functions now carry @pytest.mark.xdist_group("criu_cuda_memtest_serial"), so under -n they land on the same worker and run serially — no cross-worker re-dump. The "not xdist-safe" note was removed.

mparamas-amd · 2026-08-03T08:50:15Z

+        _log("Install CRIU build prerequisites manually, then re-run.")
+
+
+def build_and_install(version: str) -> None:


external_build.clone_repo()

Done. installer.build_and_install(version, src_dir=…) no longer clones inline — the fixture clones CRIU via external_build.clone_repo() and passes the checkout through --src-dir; the installer builds/installs from it (falling back to a tagged clone only if no src-dir is given).

mparamas-amd · 2026-08-03T09:59:14Z

+def test_criu_check_point_cuda_memtest(
+    target_executor, ld_path, cuda_memtest_build, criu_runtime, gpu_arch, rock_dir, request
+):
+    """Checkpoint a running cuda_memtest process with ``criu dump``."""
+    _skip_if_unsupported_arch(gpu_arch)
+    executor, ld = target_executor, ld_path["LD_LIBRARY_PATH"]
+    full_log = request.config.getoption("capture") == "no"
+    _ensure_checkpoint(executor, criu_runtime, cuda_memtest_build, ld, rock_dir, full_log)
+
+
+@pytest.mark.runtime.medium
+def test_criu_restore_cuda_memtest(
+    target_executor, ld_path, cuda_memtest_build, criu_runtime, gpu_arch, rock_dir, request


See If the same GPU slots to be enforced for both the tests.
for single-GPU node this may work as the later restore test acquires the same GPU, but when run on multi GPU node restore may be attempted on different GPU slot altogether.

does it makes sense to merge both test : launch → dump → restore → verify

Merged into a single launch → dump → restore → verify test under one target_executor, so the restore always uses the checkpoint's GPU — no multi-GPU slot mismatch, and the _CHECKPOINT global is gone. Kept the criu_cuda_memtest_serial xdist_group so it stays serialized with the zip/unzip test (shared build workdir); honored under --dist loadgroup.

mparamas-amd · 2026-08-03T10:10:00Z

+        vram_mb = _total_vram_mb(executor, ld, rock_dir)
+        if not vram_mb:
+            pytest.skip("Could not determine total GPU VRAM to size cuda_memtest.")
+        blocks = round(vram_mb / 1024) * 1000


pls recheck If this round-off is required.
may reduce the memory foot print

Done — removed entirely. The VRAM probe and round(vram_mb/1024)*1000 sizing are gone; the workload now uses a fixed _MAX_NUM_BLOCKS = 2000 (~2 GB), which both drops the round-off and keeps the footprint small (CRIU's amdgpu_plugin drains each buffer via sDMA within a fence timeout, so a large allocation would time out the dump).

mparamas-amd · 2026-08-03T10:43:15Z

+    poll = (
+        f"for i in $(seq 1 {int(timeout // 2)}); do "
+        f"if sudo -n grep -q '{_RESTORE_MARKER}' {workdir}/{log} 2>/dev/null; then echo RESTORE_OK; break; fi; "
+        "sleep 2; done"


What if the poll fails w/ value != RESTORE_OK ?

Handled. The poll now emits an explicit sentinel either way: RESTORE_OK when the marker appears, else RESTORE_FAIL (including on timeout). _restore asserts RESTORE_OK in stdout, so a failed or timed-out restore fails the test with the restore.log tail attached to Allure — never a silent/false pass.

mparamas-amd · 2026-08-07T10:00:37Z

+_CRIU_VERSION_RE = re.compile(r"^[A-Za-z0-9._/\-]+$")
+
+# Path the installer module is transferred to inside a remote/target environment.
+_REMOTE_INSTALLER = "/tmp/rocm_test_criu_installer.py"


pls use persistent path supporting both baremetal (local, remote) and container(")

mparamas-amd · 2026-08-07T10:02:10Z

+# Third-Party Notices — CRIU
+
+The shared CRIU machinery under `tests/common/criu/` **fetches, builds, and installs** the
+third-party CRIU project **on the test node at test time**. CRIU's source and binaries are not


Suggested change

third-party CRIU project **on the test node at test time**. CRIU's source and binaries are not

third-party CRIU project **on the test node at test run time**. CRIU's source and binaries are not

arunkumar-AMD added 3 commits July 27, 2026 17:10

Add criu memory testcase

26d7e25

Added criu cuda mem stress test

d25d326

Update

834aace

arunkumar-AMD requested review from a team, madkasul and mparamas-amd as code owners July 28, 2026 10:23

arunkumar-AMD assigned mparamas-amd Jul 28, 2026

arunkumar-AMD added the Review PR ready for review label Jul 28, 2026

arunkumar-AMD added 3 commits July 29, 2026 14:04

Updated criu steps and installation prereqs

b0b5568

Added new zip unzip criu test

d6a29a9

ran black, ruff etc

2f17d46

mparamas-amd requested changes Aug 3, 2026

View reviewed changes

mparamas-amd added Changes Requested Review comments to and removed Review PR ready for review labels Aug 3, 2026

arunkumar-AMD added 6 commits August 5, 2026 12:37

Fixed review commments

6396e09

removed vram logic and resolved the comments

60f63bc

Merge branch 'main' into users/arunkumar-AMD/criu_cuda_memtest

8510ada

ran black and other precheck tools

3f1deb3

Update test_criu_zip_unzip.py

07ee6a4

resolved comments

ae977be

arunkumar-AMD added Review PR ready for review and removed Changes Requested Review comments to labels Aug 5, 2026

Update conftest.py

5718847

arunkumar-AMD changed the title ~~Users/arunkumar amd/criu cuda memtest~~ Add criu memory testcase Aug 5, 2026

arunkumar-AMD mentioned this pull request Aug 5, 2026

Added new criu mnist testcase #65

Open

arunkumar-AMD added 2 commits August 5, 2026 21:46

Update test_criu_cuda_memtest.py

356be90

Update test_criu_zip_unzip.py

cd092dd

mparamas-amd requested changes Aug 7, 2026

View reviewed changes

		_log("Install CRIU build prerequisites manually, then re-run.")


		def build_and_install(version: str) -> None:

	third-party CRIU project on the test node at test time. CRIU's source and binaries are not
	third-party CRIU project on the test node at test run time. CRIU's source and binaries are not

Uh oh!

Conversation

arunkumar-AMD commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arunkumar-AMD Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

arunkumar-AMD commented Jul 28, 2026 •

edited

Loading

arunkumar-AMD Aug 5, 2026 •

edited

Loading