Skip to content

Latest commit

 

History

History
259 lines (208 loc) · 9.29 KB

File metadata and controls

259 lines (208 loc) · 9.29 KB

GENESIS Benchkit Integration Notes

This directory owns GENESIS-specific build, run, profiler, and estimation settings. Shared Benchkit CI and estimation packages should not depend on the environment variables documented here. They are local conveniences for programs/genesis/*.sh only; the app wrapper passes information to the shared layers through common artifacts such as results/result, SECTION: metadata, and padata*.tgz.

NVIDIA GPU Run And NCU Collection

For NVIDIA GPU systems, run.sh first runs GENESIS without a profiler and uses that run to measure the app FOM and section timings. It then runs additional NCU acquisition passes by default; those passes are used only to derive GPU kernel source/target time ratios for estimation. MiyabiG and RC_GH200 default to manual NCU windows, while RIKYU defaults to automatic NSYS discovery followed by NCU. Set BK_PROFILER=none, BK_GENESIS_NCU_PROFILE=false, or GENESIS_PROFILER_TOOL=none to skip the additional profiler runs.

The preferred profiler control variables are shared across applications:

BK_PROFILER=ncu          # none|fapp|ncu
BK_PROFILER_LEVEL=detailed

GENESIS still accepts the app/site-specific overrides below for compatibility and site-local tuning; they override the generic variables when set.

RIKYU is a GB200 system. The default GENESIS RIKYU build path uses the current site development SIF image because it provides CUDA, GNU compilers, GNU-compatible Open MPI Fortran modules, and Nsight tools:

GENESIS_RIKYU_SIF=/shared/software/hpc-dev-container/hpc_dev.sif
GENESIS_RIKYU_COMPILER_FAMILY=gnu
GENESIS_RIKYU_GPU_ARCH=sm_100

The RIKYU SIF image name, path, and contents are still site-local and may change. Set GENESIS_RIKYU_SIF to override the image path. Set GENESIS_RIKYU_APPTAINER to override the Apptainer command and GENESIS_RIKYU_APPTAINER_BINDS to add comma-separated bind mounts. The CI runner environment is expected to make apptainer available on PATH.

The RIKYU run path keeps the benchmark at the p8 default size and launches the container per rank from inside the Slurm allocation:

srun --mpi=pmix -n 8 --ntasks-per-node=4 apptainer exec --nv ... ./spdyn p8.inp.sub

PMIX_MCA_gds=hash is set by default for RIKYU because the containerized Open MPI runtime otherwise fails to read Slurm PMIx shared-memory state on multi-node p8 runs. RIKYU follows the same default additional NCU acquisition policy as the other NVIDIA GPU systems; set BK_PROFILER=none or GENESIS_RIKYU_PROFILER_TOOL=none to run without profiling.

The current GENESIS wrapper can collect multiple NCU windows as separate archives:

results/padata_inter.tgz
results/padata_intra.tgz
results/padata_pairlist.tgz

The default profile names are:

BK_GENESIS_NCU_PROFILE=true
BK_GENESIS_NCU_PROFILE_NAMES="inter intra pairlist"

The default kernel filters and windows are:

BK_GENESIS_NCU_INTER_KERNEL_REGEX='regex:.*force_inter_cell.*'
BK_GENESIS_NCU_INTER_LAUNCH_SKIP=100
BK_GENESIS_NCU_INTRA_KERNEL_REGEX='regex:.*force_intra_cell.*'
BK_GENESIS_NCU_INTRA_LAUNCH_SKIP=100
BK_GENESIS_NCU_PAIRLIST_KERNEL_REGEX='regex:.*build_pairlist.*'
BK_GENESIS_NCU_PAIRLIST_LAUNCH_SKIP=10
BK_GENESIS_NCU_LAUNCH_COUNT=10
BK_GENESIS_NCU_PAIRLIST_LAUNCH_COUNT=10
BK_GENESIS_NCU_NSTEPS=600

BK_GENESIS_NCU_NSTEPS shortens only the additional NCU acquisition input. The unprofiled benchmark run keeps the original input and remains the source of FOM and section timing. Use BK_GENESIS_NCU_NSTEPS=off when the full input should be used for NCU as well.

The wrapper also accepts per-profile overrides:

BK_GENESIS_NCU_<PROFILE>_KERNEL_REGEX
BK_GENESIS_NCU_<PROFILE>_LAUNCH_SKIP
BK_GENESIS_NCU_<PROFILE>_LAUNCH_COUNT
BK_GENESIS_NCU_<PROFILE>_NSTEPS

Legacy single-window collection can be requested with BK_GENESIS_NCU_KERNEL_REGEX; the wrapper treats it as a custom profile.

RIKYU's default automatic mode derives candidate NCU windows from an Nsight Systems CUDA kernel summary and then executes the generated NCU plan:

BK_GENESIS_NCU_PROFILE_MODE=discovery

When BK_GENESIS_NCU_PROFILE_MODE=discovery, run.sh keeps the normal unprofiled benchmark run, then runs a short NSYS discovery pass, writes results/kernel_discovery.json and results/ncu_plan.json, and runs the selected NCU windows. The generated NCU profiles default to the top three GPU-time kernels with launch_skip=1 and launch_count=10; the NCU archives are registered as section artifacts and are used by the GPU estimation packages to compute source/target kernel time ratios.

Use BK_GENESIS_NCU_PROFILE_MODE=discovery-only when investigating NSYS output without paying the NCU cost. That mode writes the full discovery summary and plan but skips NCU collection. If the NSYS CSV has already been created by site-local tooling, set BK_GENESIS_NCU_DISCOVERY_CSV=/path/to/cuda_gpu_kern_sum.csv to skip the NSYS pass and only generate the NCU plan. Manual mode still uses the configured inter intra pairlist windows above.

Estimation Sections

GENESIS treats the log dynamics time as the FOM. The app-side parser programs/genesis/parse_timing.sh maps a GENESIS log and dynamics FOM into section/overlap timing rows. This parser is not estimation-specific; it can be used from run.sh, estimate.sh, or local diagnostics.

estimate.sh should stay limited to GENESIS-owned decisions: section names, how to extract section timings from the GENESIS log, and which section package each section should use. Package loading, fallback, FOM composition, and Estimate JSON construction are handled by the shared Benchkit estimation layer.

Current section names are:

pairlist
bond
angle
dihedral
pme_real_wait
pme_real_inter
pme_real_intra
pme_recip
integrator
other

other is a positive residual used to reconstruct dynamics from the measured sections. If a simple sum exceeds dynamics, the wrapper emits an additional overlap instead of a negative residual.

pme real and pme recip are modeled as overlapping parts of nonbond. The wrapper emits an overlap for:

pme_real_wait,pme_real_inter,pme_real_intra,pme_recip

The current temporary split for pme real is:

pme_real_wait  = pme real * 0.8  # wait / CPU / communication-like part
pme_real_inter = pme real * 0.1  # GPU inter-cell kernel family
pme_real_intra = pme real * 0.1  # GPU intra-cell kernel family

The fractions are app-local knobs:

BK_GENESIS_PME_REAL_IDENTITY_FRACTION
BK_GENESIS_PME_REAL_INTER_FRACTION
BK_GENESIS_PME_REAL_INTRA_FRACTION

The current side uses a Fugaku baseline FOM with weak scaling. The MiyabiG section breakdown is used only for the future-side projection toward FugakuNEXT.

GPU Section Package Mapping

GENESIS does not choose individual GPU estimator packages. It marks GPU-related sections as gpu_kernel_ensemble_average; the common section package decides which concrete GPU estimator packages to run. This keeps GENESIS-side ownership limited to app concepts: section names, timing extraction, artifact candidates, and kernel selectors.

Benchkit operators can override the concrete GPU estimator package set with the generic BK_GPU_KERNEL_ENSEMBLE_PACKAGES variable when needed. That knob is not GENESIS-specific and should not be required for normal GENESIS maintenance.

Current GPU section/artifact mapping:

pairlist       -> results/padata_pairlist.tgz -> build_pairlist
pme_real_inter -> results/padata_inter.tgz    -> force_inter_cell
pme_real_intra -> results/padata_intra.tgz    -> force_intra_cell

programs/genesis/run.sh registers these artifact paths when each NCU profile archive is created, then writes them as SECTION: ... artifact:... entries in results/result. scripts/result.sh turns those lines into fom_breakdown.sections[].artifacts[] in the Result JSON. estimate.sh consumes that Result JSON; it should not infer profiler output paths by scanning the filesystem or by knowing profiler archive names.

The app wrapper passes kernel selectors to GPU section packages through the common estimation helper. These selectors are GENESIS-owned because they name GENESIS kernels.

NCU archives contain sampled kernel launches, not full application section timings. The GPU section packages compute source/target kernel time ratios from these samples. Those ratios are applied to app-side section times measured from the profiler-free GENESIS run.

Site-Specific Build/Run Overrides

These variables are app-local knobs consumed by programs/genesis/build.sh and programs/genesis/run.sh.

GENESIS_MIYABIG_MODULE
GENESIS_GH200_MODULE
GENESIS_RIKYU_MODULE
GENESIS_MIYABIG_CUDA_PATH
GENESIS_RIKYU_CUDA_PATH
GENESIS_MIYABIG_FC
GENESIS_MIYABIG_CC
GENESIS_RIKYU_FC
GENESIS_RIKYU_CC
GENESIS_RIKYU_CXX
GENESIS_RIKYU_F77
GENESIS_MIYABIG_CONFIG_ARGS
GENESIS_RIKYU_CONFIG_ARGS
GENESIS_RIKYU_COMPILER_FAMILY
GENESIS_RIKYU_GPU_ARCH
GENESIS_MIYABIG_PROFILER_TOOL
GENESIS_GH200_PROFILER_TOOL
GENESIS_RIKYU_PROFILER_TOOL
GENESIS_MIYABIG_PROFILER_LEVEL
GENESIS_GH200_PROFILER_LEVEL
GENESIS_RIKYU_PROFILER_LEVEL
GENESIS_RIKYU_MPI_CMD
GENESIS_RIKYU_MPI_ARGS
GENESIS_PROFILER_TOOL
GENESIS_PROFILER_LEVEL

GENESIS_*_PROFILER_TOOL=ncu requests additional NCU acquisition. none disables it explicitly. Prefer BK_PROFILER / BK_PROFILER_LEVEL for new generic CI/profile control unless a site needs a GENESIS-specific override.