TABLE OF CONTENTS
- Welcome to hpcscan
- Overview
- Main features
- Environment set-up
- Compilation
- Validation
- Execution
- Performance benchmarks
- Customization
- Have fun!
Version 1.2
Contact: Vincent Etienne / Email: vetienne@rocketmail.com
Contributors (chronological order) : Vincent Etienne, Suha Kayum, Marcin Rogowski, Laurent Gatineau, Philippe Thierry, Fabrice Dupros, Hugo Barreiro
hpcscan is a tool for benchmarking algorithms/kernels that are found in many scientific applications on various architectures/systems.
It features several categories of test cases aiming to measure memory, computation and communication bandwidths along with electric energy consumption.
- All test cases are validated with embedded reference solutions
- Hybrid OpenMP/MPI parallelism
- Support scalar and vector CPUs, GPUs and other accelerators (depending on compiler/architecture)
- Support double precision (FP64), single precision (FP32) and half precision (FP16) for data storage and/or for computation
- Written in C++ (following C++ Google style code)
- Simple code structure based on individual test cases
- Customizable (new test cases can be added)
There exist several benchmarks commonly used in the HPC community. Just to cite a few, the Stream benchmark, the HPL benchmark and the OSU Micro benchmarks, allow to measure the memory bandwidth, the computation bandwidth and the interconnect bandwidth respectively. In general, these benchmarks target specific characteristics of HPC systems. However, it is not straightforward to transpose these characteristics in the context of a given scientific application.
This is why, HPC vendors used to present throughputs obtained with open source scientific codes such as for instance OpenFOAM (Computational Fluid Dynamics) or SPECFEM3D (Seismology). While these results are important to assess the performance of a given architecture to solve concrete problems, it is again not straightforward to transpose conclusions to other applications. Moreover, every application has been built on technical choices that may hinder performance on a system compared to another. How to overcome these technical biases?
hpcscan has been designed to address these issues 😃
☑️ Lightweight and portable tool that can be easily deployed on a wide range of architectures including CPUs, GPUs and accelerators (see Validated hardware, operating systems and compilers).
☑️ Bridge between HPC architectures and numerical analysis/computational sciences. Beyond accurate performance measurements, hpcscan allows to explore the behavior of numerical kernels and to seek for the optimal configuration on a given architecture.
An example is shown below with a FD Laplacian kernel where accuracy order and spatial sampling are explored to find the optimum (in terms of computation speed vs accuracy). See Performance benchmarks for details on this test case as well as scripts to perform the analysis.
Top left: Error between computed and analytical solutions versus spatial sampling. FD accuracy orders from O2 (blue) to O16 (red) are shown. Top right: Error versus computation time. The black star points to the optimal configuration with an error below the target (1%) and shortest computation time. Bottom left: Kernel bandwidth in GPoint/s versus N (grid size is N x N x N). Bottom right: Kernel bandwidth in GBtye/s.
☑️ Set of representative kernels used in many scientific applications (see List of test cases). Without being too specific, the embedded kernels provide a way to capture the main traits of HPC architectures and identify their bottle-necks and strenghts. With this knowledge, one can re-design or update accordingly specific parts of an application to take full benefit of the target hardware.
☑️ Set of robust protocols to compare architectures. As suggested in the example above, the optimal configuration to solve a given problem might change from an architecture to another. hpcscan provides a solid framework to compare performances between different systems, where one can analyse results from different perspectives and achieve 'apples to apples' comparisons.
☑️ Customizable to fit a specific hardware (see Customization).
☑️ Multi-purpose initiative with benefits at several levels: from computer science students eager to learn to seasoned numerical analysts willing to share their findings or to software engineers reusing kernels of interest to upgrade their applications.
☑️ On-going effort aiming to collect contributions to cover the current offer of HPC systems. More options and kernels will be added with time.
⛔ One-number benchmark to rank HPC systems. However, hpcscan provides a way to perform a complete 'scanning' of architectures and possibly focus on one characteristic.
⛔ Confidential project. Everyone is invited to share results, feedbacks and more important contributions for the benefit of the entire HPC community.
hpcscan is a self-content package that can be easily installed and executed on your system. Just follow the steps:
- Step 1: create the environment script for your system
- Step 2: build the executable
- Step 3: validate the executable
- Step 4: run the performance benchmarks
| Version | Description | Release date |
|---|---|---|
| v1.0 | Initial version with CPU and Vector Engine support |
Nov 28, 2020 |
| v1.1 | GPU support |
May 22, 2021 |
| v1.2 | Energy consumption and half-precision support |
Q4 2026 |
binthis directory is created during compilation and contains hpcscan executablebuildhpcscan can be compiled from hereenvscripts to initialize hpcscan environmentmicsoutput samples and studiesscriptscripts for validation and performance benchmarkssrcall hpcscan source files
| Test case name | Description | Remark |
|---|---|---|
| Comm |
MPI communications bandwidth
|
This case requires at least 2 MPI processes |
| FD_D1 |
Finite-difference - 1st derivative Accuracy and computation bandwidth
|
Accuracy depends on the selected FD order (from to 2 to 16), grid sampling and number of periods in the sine function used to compute the derivative |
| FD_D2 |
Finite-difference - 2nd derivative Accuracy and computation bandwidth
|
Same features as above |
| Grid |
Grid operations bandwidth
|
Operations on grids include selection of specific portions of the grids (for instance, excluding halos) |
| Memory |
Memory operations bandwidth
|
Conversely to Test Case Grid, operations are done on continuous memory arrays |
| Modeling |
Acoustic wave modeling bandwidth Same features as for test case Propa except
|
There is no accuray checking for this test case |
| Propa |
Acoustic wave propagator bandwidth
|
Accuracy is checked against the multi-dimensional analytical solution (Eigen modes) of the wave equation |
| Template | Test case template | Used to create new test cases |
| Util | Utility tests to check internal functions | Reserved for developpers |
All available test modes are listed below. Activation of each test mode depends on the compilers defined in the hpscan environment script, see Environment script (mandatory).
| Test mode name | Target hardware | Description | Remark |
|---|---|---|---|
| Baseline | Generic CPU | Standard implementation without optimization | ➡️ This mode is the reference implementation Default test mode. Always enabled |
| CacheBlk | Generic CPU | Optimized with cache blocking techniques | Always enabled |
| CUDA | NVIDIA GPU | Regular CUDA implementation without optimization | Enabled when compiled with nvcc (NVIDIA CUDA compiler) |
| CUDA_Opt | NVIDIA GPU | Optimized CUDA implementation | Enabled when compiled with nvcc (NVIDIA CUDA compiler) |
| CUDA_Ref | NVIDIA GPU | Reference CUDA implementation (for developpers) | Enabled when compiled with nvcc (NVIDIA CUDA compiler) |
| DPC++ | Intel CPU/GPU/FPGA | Regular DPC++ implementation without optimization | Enabled when compiled with dpcpp (Intel OneAPI DPC++ compiler) |
| HIP | AMD GPU | Regular HIP implementation without optimization | Enabled when compiled with hipcc (AMD HIP compiler) |
| HIP_Opt | AMD GPU | Optimized HIP implementation | Enabled when compiled with hipcc (AMD HIP compiler) |
| NEC | NEC SX-Aurora | With NEC compiler directives | Enabled when compiled with nc++ (NEC C++ compiler) |
| NEC_SCA | NEC SX-Aurora | With NEC Library Stencil Code Accelerator | Enabled when compiled with nc++ (NEC C++ compiler) |
| OpenAcc | NVIDIA GPU | Regular OpenACC implementation without optimization | Enabled when compiled with a C++ compiler that supports OpenACC (not yet operational) |
- Linux operating system
- C++ compiler with OpenMP support
- MPI library
- python and Matlab to plot figures
- NVIDIA CUDA compiler
- Intel DPC++ compiler
- AMD HIP compiler
- NEC C++ compiler
- C++ compiler with OpenACC support
In order to compile and run hpcscan, you need to source one of the files in the directory ./env
cd ./env
Example to set up the environment for hpcscan with GCC:
source ./setEnvGenericGcc.sh
🔔 For a new system, you would need to create a file for your system (take example from one of the existing files)
Go to ./build, and use the command make
Executable can be found in ./bin/hpcscan
🔔 If hpcscan environment has not been set (see Environment script (mandatory)), compilation will abort.
By default, hpcscan is compiled in single precision (FP32) for both data storage and computation.
To specify specific formats: make storage=OPT1 compute=OPT2
Where OPT1 defines the desired format for data storage: FP64, FP32 of FP16 are valid options. Similarly OPT2 defines the format to be used for computations. Example for FP16 data with FP32 computations : make storage=FP16 compute=FP32
To check the test modes that are enabled in your hpcscan binary, use the command
./bin/hpcscan -v
To check that hpcscan has correctly been built and works fine, go to ./script and launch
sh runValidationTests.sh
This script runs a set a light test cases for each test mode enabled (takes few minutes even on a laptop).
Check the summary of the output report (displayed on the terminal) the following:
- All tests marked as PASSED (768 tests passed for each test mode enabled)
- No test marked as FAILED
🔔 These tests are intended for validation purpose only, they do not allow for performance measurements.
hpcscan has been successfully tested on the hardware, operating systems and compilers listed below.
| Operating system | Compiler | MPI | Host | Device | Test modes |
|---|---|---|---|---|---|
| Ubuntu 22.04.1 LTS | g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0 | mpirun (Open MPI) 4.1.2 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz (Intel Kaby Lake) | - | Baseline, CacheBlk |
| Ubuntu 22.04.1 LTS | Intel icpc (ICC) 2021.7.0 20220726 | Intel MPI Version 2021.7 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz (Intel Kaby Lake) | - | Baseline, CacheBlk |
| Red Hat 4.8.5-39 | Intel oneAPI DPC++/C++ Compiler 2022.1.0 | Intel MPI Version 2021.6 | Intel(R) Xeon(R) Gold 6240L CPU @ 2.60GHz (Intel Cascade Lake) | - | Baseline, CacheBlk |
| Red Hat 4.8.5-39 | Intel MPI Version 2021.6 | Intel(R) Xeon(R) Gold 6240L CPU @ 2.60GHz (Intel Cascade Lake) | Tesla V100S-PCI (NVIDIA GPU) | Baseline, CacheBlk, Cuda, Cuda_Opt, Cuda_Ref | |
| Red Hat 8.5.0-10 | NEC nc++ (NCC) 4.0.0 | NEC MPI 3.1.0 | Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz (Intel Skylake) | NEC SX-Aurora TSUBASA 20B-P (NEC Vector Engine) | Baseline, CacheBlk, NEC, NEC_SCA |
| Red Hat 8.5.0-10 | Intel oneAPI DPC++/C++ Compiler 2022.1.0 | Intel MPI Version 2021.6 | Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz (Intel Skylake) | - | Baseline, CacheBlk |
| SUSE Linux Enterprise Server 15 | Intel icpc (ICC) 19.0.5.281 20190815 | - | Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz (Intel Haswell) | - | - |
| Red Hat 4.8.5-39 | Intel icpc version 19.1.2.254 | - | Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz (Intel Cascade Lake) | - | - |
| Ubuntu 20.04.1 LTS | - | Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz (Intel Ice Lake) | GP108M [GeForce MX330] (NVIDIA GPU) | - | |
| CentOS Linux release 7.7.1908 | - | Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz (Intel Skylake) | GV100GL [Tesla V100 SXM2 32GB] (NVIDIA GPU) | - | |
| Ubuntu 20.04.1 LTS | Intel(R) oneAPI DPC++ Compiler 2021.2.0 | - | Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz (Intel Ice Lake) | - | - |
| Ubuntu 20.04.1 LTS | - | AMD EPYC 7742 64-Core Processor @ 2.25GHz (AMD Rome) | [AMD Instinct MI100] (AMD GPU) | - |
hpcscan can be launched from a terminal with all configuration parameters within a single line.
To get help on the parameters
./bin/hpcscan -h
Execution with a unique MPI process
mpirun -n 1 ./bin/hpcscan -testCase <TESTCASE> -testMode <TESTMODE>
where
TESTCASEis the name of the test case (see List of test cases)TESTMODEis the name of the test mode (see List of test modes)
Example
mpirun -n 1 ./bin/hpcscan -testCase Propa -testMode CacheBlk
🔔 If you omit to specify -testMode <TESTMODE>, the Baseline mode is assumed.
Example
mpirun -n 1 ./bin/hpcscan -testCase Propa
Execution with multiple MPI processes
mpirun -n <N> ./bin/hpcscan -testCase <TESTCASE> -testMode <TESTMODE> -nsub1 <NSUB1> -nsub2 <NSUB2> -nsub3 <NSUB3>
🔔 When several MPI processes are used, subdomain decomposition is activated. The product NSUB1 x NSUB2 x NSUB3 must be equal to N (no. of MPI proc.). You may omit to specify the number of subdomains along an axis if that number is 1.
Example
mpirun -n 2 ./bin/hpcscan -testCase Comm -nsub1 2
Configuration of the grid size and dimension
Simply add on the command line
-n1 <N1> -n2 <N2> -n3 <N3> -dim <DIM>
Where N1, N2, N3 are the number of grid points along axis 1, 2 and 3.
And DIM = 1,2 or 3 (1D, 2D or 3D grids).
Example
mpirun -n 1 ../bin/hpcscan -testCase Grid -dim 2 -n1 200 -n2 300
🔔 If you omit to specify -dim <DIM>, 3D grid is assumed.
Input
hpcscan does not require any input file. All data are built internally.
Output on the terminal
During execution, information regarding results validation and performances are sent to the terminal output.
Output performance log file
For every test case, an ASCII file containing all measures in a compact way is created. It can used to plot results with dedicated tools. The name of the log file is as follows
hpcscan.perf.<TESTCASE>.log
If hpcscan is launched several times, results are added to the log file. It is convenient for instance, when you want to analyse the effect of a parameter and plot the serie of results in a graph.
Output grids
Be default, the grids manipulated by hpcscan are not written on disk.
To output the grids, use the option -writeGrid.
When activated, each grid used in a test will generate 2 files:
- An ASCII file with the grid dimensions (name of the file
<GRIDNAME>.proc<ID>.grid.info) - A binary file with the grid data (name of the file
<GRIDNAME>.proc<ID>.grid.bin) where ID is the MPI rank.
Example (this is the command that was used to produce the hpcscan logo on top of this page)
mpirun -n 1 ../../bin/hpcscan -testCase Propa -writeGrid \
-tmax 0.2 -snapDt 0.1 \
-dim 2 -n1 200 -n2 600 \
-param1 4 -param2 8
Outputs the following files: PropaEigenModeRef.proc0.grid.info, PropaEigenModeRef.proc0.grid.bin, PropaEigenModePrn.proc0.grid.info and PropaEigenModePrn.proc0.grid.bin
Output debug traces
The code is equipped with debug traces that can be activated with the option -debug <LEVEL> where LEVEL can be set to light, mid or full (minimum, middle and maximum level of verbosity).
It can be useful to activate them when developping/debugging to understand the behavior of the code.
When activated, debug traces are written by each MPI proc in an ASCII file with name hpcscan.debug.proc<ID>.log where ID is the MPI rank.
🔔 Maximum memory required per node (device) is 20 GB
🔔 At maximum, 8 computing nodes (devices) are used
The benchmarks are independent and can be used as is or configured according to your system if needed.
Test cases description
| Test case | Objectives | Remarks |
|---|---|---|
| Memory | Assess memory bandwidth | Scalability analysis on a single node |
| Grid | Assess bandwidth of grid operations | Analyse effect of the grid size |
| Comm | Assess inter-node communication bandwidth | Analyse effect of subdomain decomposition |
| FD_D1 | Assess FD 1st derivative computation bandwidth | Analyse effect of FD stencil order |
| FD_D2 | Assess FD 2nd derivative computation bandwidth | Analyse effect of FD stencil order |
| Propa | Find optimal configuration for the wave propagator | Explore range of parameters |
| Propa | Scalability analysis of wave propagator on multiple nodes | Analyse effect of the FD stencil order |
➡️ Performance measurements and scripts to reproduce results obtained on various architectures are available in ./misc/hpcscanPerfSlides/hpcscanPerfSlides.pdf
hpcscan is based on a simple yet highly flexible design that relies heavily on the C++ inheritance mechanism.
The main class is Grid (see ./src/grid.cpp).
This class handles all grid data in hpcscan and all operations performed on grids.
It implements the so-called Baseline mode and it is the reference implementation.
💡 All test cases, at some point, call methods of this class. Indeed, test cases (testCase_xxx.cpp) do not implement kernels.
Let us say, you would like to specialize the implementation for a given architecture.
In order to do this, you would need to create a new class that derives from Grid.
Just copy the source files grid_Custom.h and grid_Custom.cpp to another location and re-implement the desired functions that are declared as virtual in Grid.
💡 You can proceed step by step, implementing one function at a time, with the possibility to check the behavior of your implementation against the Baseline reference solution.
Check the grids that are already implemented in hpcscan to get some concrete examples.
💡 To allow hpcscan to use this new class, you need only to indicate the path to your new source files in the environment script (see Environment script (mandatory)) using the environment variable $HPCSCAN_CUSTOM_PATH.
By doing this, you can switch at execution time, to your new grid with the -testMode Custom.
- Issues encountered
- Suggestions of new test cases
- Performance measurements
➡️ If you want to contribute to hpcscan, please contact the project coordinator (vetienne@rocketmail.com).