This directory owns GENESIS-specific build, run, profiler, and estimation
settings. Shared Benchkit CI and estimation packages should not depend on the
environment variables documented here. They are local conveniences for
programs/genesis/*.sh only; the app wrapper passes information to the shared
layers through common artifacts such as results/result, SECTION: metadata,
and padata*.tgz.
For NVIDIA GPU systems, run.sh first runs GENESIS without a profiler and
uses that run to measure the app FOM and section timings. It then runs
additional NCU acquisition passes by default; those passes are used only to
derive GPU kernel source/target time ratios for estimation. MiyabiG and
RC_GH200 default to manual NCU windows, while RIKYU defaults to automatic NSYS
discovery followed by NCU. Set
BK_PROFILER=none, BK_GENESIS_NCU_PROFILE=false, or
GENESIS_PROFILER_TOOL=none to skip the additional profiler runs.
The preferred profiler control variables are shared across applications:
BK_PROFILER=ncu # none|fapp|ncu
BK_PROFILER_LEVEL=detailedGENESIS still accepts the app/site-specific overrides below for compatibility and site-local tuning; they override the generic variables when set.
RIKYU is a GB200 system. The default GENESIS RIKYU build path uses the current site development SIF image because it provides CUDA, GNU compilers, GNU-compatible Open MPI Fortran modules, and Nsight tools:
GENESIS_RIKYU_SIF=/shared/software/hpc-dev-container/hpc_dev.sif
GENESIS_RIKYU_COMPILER_FAMILY=gnu
GENESIS_RIKYU_GPU_ARCH=sm_100The RIKYU SIF image name, path, and contents are still site-local and may
change. Set GENESIS_RIKYU_SIF to override the image path. Set
GENESIS_RIKYU_APPTAINER to override the Apptainer command and
GENESIS_RIKYU_APPTAINER_BINDS to add comma-separated bind mounts. The CI
runner environment is expected to make apptainer available on PATH.
The RIKYU run path keeps the benchmark at the p8 default size and launches the container per rank from inside the Slurm allocation:
srun --mpi=pmix -n 8 --ntasks-per-node=4 apptainer exec --nv ... ./spdyn p8.inp.subPMIX_MCA_gds=hash is set by default for RIKYU because the containerized
Open MPI runtime otherwise fails to read Slurm PMIx shared-memory state on
multi-node p8 runs. RIKYU follows the same default additional NCU acquisition
policy as the other NVIDIA GPU systems; set BK_PROFILER=none or
GENESIS_RIKYU_PROFILER_TOOL=none to run without profiling.
The current GENESIS wrapper can collect multiple NCU windows as separate archives:
results/padata_inter.tgz
results/padata_intra.tgz
results/padata_pairlist.tgz
The default profile names are:
BK_GENESIS_NCU_PROFILE=true
BK_GENESIS_NCU_PROFILE_NAMES="inter intra pairlist"The default kernel filters and windows are:
BK_GENESIS_NCU_INTER_KERNEL_REGEX='regex:.*force_inter_cell.*'
BK_GENESIS_NCU_INTER_LAUNCH_SKIP=100
BK_GENESIS_NCU_INTRA_KERNEL_REGEX='regex:.*force_intra_cell.*'
BK_GENESIS_NCU_INTRA_LAUNCH_SKIP=100
BK_GENESIS_NCU_PAIRLIST_KERNEL_REGEX='regex:.*build_pairlist.*'
BK_GENESIS_NCU_PAIRLIST_LAUNCH_SKIP=10
BK_GENESIS_NCU_LAUNCH_COUNT=10
BK_GENESIS_NCU_PAIRLIST_LAUNCH_COUNT=10
BK_GENESIS_NCU_NSTEPS=600BK_GENESIS_NCU_NSTEPS shortens only the additional NCU acquisition input. The
unprofiled benchmark run keeps the original input and remains the source of FOM
and section timing. Use BK_GENESIS_NCU_NSTEPS=off when the full input should be
used for NCU as well.
The wrapper also accepts per-profile overrides:
BK_GENESIS_NCU_<PROFILE>_KERNEL_REGEX
BK_GENESIS_NCU_<PROFILE>_LAUNCH_SKIP
BK_GENESIS_NCU_<PROFILE>_LAUNCH_COUNT
BK_GENESIS_NCU_<PROFILE>_NSTEPSLegacy single-window collection can be requested with BK_GENESIS_NCU_KERNEL_REGEX;
the wrapper treats it as a custom profile.
RIKYU's default automatic mode derives candidate NCU windows from an Nsight Systems CUDA kernel summary and then executes the generated NCU plan:
BK_GENESIS_NCU_PROFILE_MODE=discoveryWhen BK_GENESIS_NCU_PROFILE_MODE=discovery, run.sh keeps the normal
unprofiled benchmark run, then runs a short NSYS discovery pass, writes
results/kernel_discovery.json and results/ncu_plan.json, and runs the
selected NCU windows. The generated NCU profiles default to the top three
GPU-time kernels with launch_skip=1 and launch_count=10; the NCU archives
are registered as section artifacts and are used by the GPU estimation packages
to compute source/target kernel time ratios.
Use BK_GENESIS_NCU_PROFILE_MODE=discovery-only when investigating NSYS output
without paying the NCU cost. That mode writes the full discovery summary and
plan but skips NCU collection. If the NSYS CSV has already been created by
site-local tooling, set
BK_GENESIS_NCU_DISCOVERY_CSV=/path/to/cuda_gpu_kern_sum.csv to skip the NSYS
pass and only generate the NCU plan. Manual mode still uses the configured
inter intra pairlist windows above.
GENESIS treats the log dynamics time as the FOM. The app-side parser
programs/genesis/parse_timing.sh maps a GENESIS log and dynamics FOM into
section/overlap timing rows. This parser is not estimation-specific; it can be
used from run.sh, estimate.sh, or local diagnostics.
estimate.sh should stay limited to GENESIS-owned decisions: section names,
how to extract section timings from the GENESIS log, and which section package
each section should use. Package loading, fallback, FOM composition, and
Estimate JSON construction are handled by the shared Benchkit estimation layer.
Current section names are:
pairlist
bond
angle
dihedral
pme_real_wait
pme_real_inter
pme_real_intra
pme_recip
integrator
other
other is a positive residual used to reconstruct dynamics from the measured
sections. If a simple sum exceeds dynamics, the wrapper emits an additional
overlap instead of a negative residual.
pme real and pme recip are modeled as overlapping parts of nonbond. The
wrapper emits an overlap for:
pme_real_wait,pme_real_inter,pme_real_intra,pme_recip
The current temporary split for pme real is:
pme_real_wait = pme real * 0.8 # wait / CPU / communication-like part
pme_real_inter = pme real * 0.1 # GPU inter-cell kernel family
pme_real_intra = pme real * 0.1 # GPU intra-cell kernel family
The fractions are app-local knobs:
BK_GENESIS_PME_REAL_IDENTITY_FRACTION
BK_GENESIS_PME_REAL_INTER_FRACTION
BK_GENESIS_PME_REAL_INTRA_FRACTIONThe current side uses a Fugaku baseline FOM with weak scaling. The MiyabiG section breakdown is used only for the future-side projection toward FugakuNEXT.
GENESIS does not choose individual GPU estimator packages. It marks GPU-related
sections as gpu_kernel_ensemble_average; the common section package decides
which concrete GPU estimator packages to run. This keeps GENESIS-side ownership
limited to app concepts: section names, timing extraction, artifact candidates,
and kernel selectors.
Benchkit operators can override the concrete GPU estimator package set with the
generic BK_GPU_KERNEL_ENSEMBLE_PACKAGES variable when needed. That knob is not
GENESIS-specific and should not be required for normal GENESIS maintenance.
Current GPU section/artifact mapping:
pairlist -> results/padata_pairlist.tgz -> build_pairlist
pme_real_inter -> results/padata_inter.tgz -> force_inter_cell
pme_real_intra -> results/padata_intra.tgz -> force_intra_cell
programs/genesis/run.sh registers these artifact paths when each NCU profile
archive is created, then writes them as SECTION: ... artifact:... entries in
results/result. scripts/result.sh turns those lines into
fom_breakdown.sections[].artifacts[] in the Result JSON. estimate.sh
consumes that Result JSON; it should not infer profiler output paths by scanning
the filesystem or by knowing profiler archive names.
The app wrapper passes kernel selectors to GPU section packages through the common estimation helper. These selectors are GENESIS-owned because they name GENESIS kernels.
NCU archives contain sampled kernel launches, not full application section timings. The GPU section packages compute source/target kernel time ratios from these samples. Those ratios are applied to app-side section times measured from the profiler-free GENESIS run.
These variables are app-local knobs consumed by programs/genesis/build.sh and
programs/genesis/run.sh.
GENESIS_MIYABIG_MODULE
GENESIS_GH200_MODULE
GENESIS_RIKYU_MODULE
GENESIS_MIYABIG_CUDA_PATH
GENESIS_RIKYU_CUDA_PATH
GENESIS_MIYABIG_FC
GENESIS_MIYABIG_CC
GENESIS_RIKYU_FC
GENESIS_RIKYU_CC
GENESIS_RIKYU_CXX
GENESIS_RIKYU_F77
GENESIS_MIYABIG_CONFIG_ARGS
GENESIS_RIKYU_CONFIG_ARGS
GENESIS_RIKYU_COMPILER_FAMILY
GENESIS_RIKYU_GPU_ARCH
GENESIS_MIYABIG_PROFILER_TOOL
GENESIS_GH200_PROFILER_TOOL
GENESIS_RIKYU_PROFILER_TOOL
GENESIS_MIYABIG_PROFILER_LEVEL
GENESIS_GH200_PROFILER_LEVEL
GENESIS_RIKYU_PROFILER_LEVEL
GENESIS_RIKYU_MPI_CMD
GENESIS_RIKYU_MPI_ARGS
GENESIS_PROFILER_TOOL
GENESIS_PROFILER_LEVELGENESIS_*_PROFILER_TOOL=ncu requests additional NCU acquisition. none
disables it explicitly. Prefer BK_PROFILER / BK_PROFILER_LEVEL for new
generic CI/profile control unless a site needs a GENESIS-specific override.