Added new criu mnist testcase

[arunkumar-AMD]

Motivation

CRIU (Checkpoint/Restore In Userspace) freezes a running process and checkpoints its full state to disk so it can later be restored and resume where it left off. On AMD GPUs this relies on the ROCm KFD checkpoint/restore hooks plus AMD's amdgpu_plugin.so, which serialize per-process GPU state (VRAM BOs, user-mode queues, events). This PR adds e2e coverage for checkpoint/restore of a real ML training workload — the upstream PyTorch MNIST example — running under ROCm PyTorch on the GPU. It is a correctness check, not a benchmark: "good" means a live training process can be criu dumped and criu restored and then keeps making forward progress, guarding specifically against the failure mode where restore reports success but the process silently hangs.

Technical Details

New test tests/e2e/recovery/criu/test_criu_pyt_ex_mnist.py, plus the supporting fixtures and framework infra that let a workload be checkpointed inside a container.

• Workload & provisioning (function fixture). pyt_mnist_setup git-clones the upstream pytorch/examples repo into the target environment and resolves the ambient ROCm (HIP) PyTorch interpreter (probes torch.version.hip; override via ROCM_TEST_MNIST_PYTHON). No pip install / ML provisioner — main.py runs against the container's ambient torch. Skips cleanly if git or a ROCm-torch interpreter is absent.
• Container execution (new parametric marker). @pytest.mark.container(ipc="host", privileged=True) runs all target_executor commands inside a persistent container on the acquired GPU node, created from --container-image. GPUs are acquired via NodePool first, then injected (device passthrough + ROCR env). The container is started detached with --init (tini as PID 1) so the workload SIGKILLed by criu dump is reaped and its PID is freed for restore.
• CRIU runtime. criu_runtime_target ensures CRIU + amdgpu_plugin.so are ready inside the target (criu check), auto-installing via scripts/install_criu.py when missing (make install-criu, then builds/installs the amdgpu plugin). The criu invocation forwards both PATH and LD_LIBRARY_PATH through sudo -n env … so the elevated process resolves the container's ROCm libraries.
• Flow. launch main.py (PID via $!) → wait for the first Train Epoch and record training.out byte size → criu dump → criu restore → assert the original PID is alive and still running main.py → poll that training.out grows beyond the pre-dump size. The launched PID is always killed in teardown.
• Backstops. dump → OK + PID_GONE (polls for the process to be reaped); restore → RESTORE_OK; resume → RESUMED_OK (/proc//cmdline still main.py); forward progress → post-restore output size strictly greater than pre-dump (a restored but-hung process fails). Metrics: MNIST_OUTPUT_BYTES_PRE_DUMP, MNIST_OUTPUT_BYTES_POST_RESTORE.
• Reusable additions. Shared CRIU steps in _criu_steps.py (criu_dump with post-dump PID-gone poll, backgrounded criu_restore + restore.log poll, attach_criu_log, kill_pid); install_criu.py builds the binary (install-criu) and the amdgpu plugin and verifies with criu check. pytorch/examples (BSD-3-Clause) and CRIU (GPL-2.0 / LGPL-2.1 for lib/) are fetched/built at runtime, never vendored (see NOTICES.md).
• Privilege / arch. CRIU runs under sudo -n; without passwordless sudo the suite skips cleanly. Skips on GFX targets outside the supported set when --gpu-arch is given.
• Markers (via CATEGORY_PROFILES): hw.gpu, layer.runtime, ci.weekly, e2e.stack, os.linux; runtime.medium per function; plus the new parametric container marker.

Test Plan

Test Result

Required:
--rock-dir (or ROCK_DIR / rocm-test.toml)
--container-image (must provide torch + torchvision)
Linux AMD GPU node with passwordless sudo -n
supported GFX (gfx90a/908/942/950, gfx110x/gfx120x).

Optional env:
ROCM_TEST_MNIST_PYTHON, ROCM_TEST_PYT_EXAMPLES_URL, ROCM_TEST_PYT_WORKDIR, ROCM_TEST_CRIU_AUTO_INSTALL (default 1), ROCM_TEST_CRIU_VERSION (default v4.1).

Optional flags:
-s (full CRIU logs inline), --remote-node. Run single-process (no -n).

pytest tests/e2e/recovery/criu/test_criu_pyt_ex_mnist.py -v --container-image <rocm-pytorch-image> --rock-dir <therock path>

Test Result

Run inside a ROCm PyTorch container on an AMD GPU node → 1 passed (129.80 s).

• CRIU auto-installed inside the container; criu check → "Looks good."
• Training launched (PID 7050); first Train Epoch reached — MNIST_OUTPUT_BYTES_PRE_DUMP = 6973.
• criu dump → OK + PID_GONE ("Dumping finished successfully").
• criu restore → RESTORE_OK ("Restore finished successfully. Tasks resumed.").
• RESUMED_OK — PID 7050 alive, /proc/7050/cmdline still main.py.
• Forward progress confirmed — MNIST_OUTPUT_BYTES_POST_RESTORE = 15186 (> 6973): training resumed and continued rather than hanging.

[mparamas-amd]

add container_opts

[arunkumar-AMD]

Removed complete remote node plugin changes on this PR and updated on #48

[mparamas-amd]

pls update the below 2 fixtures to read the container_opts and pass to _acquire_and_yield.
though not related to this single gpu scenario, this may remain as silent no-op for containerExecutor
multi_gpu_fixture
multi_node_fixture

[arunkumar-AMD]

Removed complete remote node plugin changes on this PR and updated on #48

[mparamas-amd]

LGTM, thanks! pls merge after PR#63