Porting to Kokkos

DEVELOPER_NOTES

@@ -1,3 +1,7 @@
+XX/XX/26 (TRUST) GPU          : Blackwell B6000 and Cuda 13.0 build (NVHPC 25.11) supported
+XX/XX/26 (TRUST) Matrix       : Introduce Stencil structure to deal with possible nnz larger than 2^31 (possible on future GPU device)
+XX/XX/26 (TRUST) VEF          : Elem_VEF_base::normale replaced by Elem_VEF_base::creer_face_normales
+XX/XX/26 (TRUST) Build        : Add a new target profiling to build a -O3 -g (+specific options) binary to ease the profiler tools (perf, nsys, rocprof,...)
 -------------------------------------------------------------
 Developer notes version 1.9.8_beta : Changes since version 1.9.7 :
 -------------------------------------------------------------

RELEASE_NOTES

@@ -1,3 +1,10 @@
+XX/XX/26 (TRUST) GPU          : AMG solver fixed for gfx1100 AMD card and ROCm 7.2 supported
+XX/XX/26 (TRUST) Keyword      : Add Correlation_triple keyword for computed advanced fields
+XX/XX/26 (TRUST) Keyword      : Add Enstrophie_totale keyword
+XX/XX/26 (TRUST) Bug fix      : Fix Corriger_frontiere_periodique_64 issue (VDF only) + Ecrire_med_64 not available
+XX/XX/26 (TRUST) Change       : Warning, VEF VerifierCoin keyword algorithm has been rewritten which possibly change the numerotation of the mesh and the partitioned mesh
+XX/XX/26 (TRUST) Solver       : Sparskit based solver "solveur gen { solv_elem bicgstab|gmres ... }" removed. It can be replaced by more efficient PETSc equivalent solvers
+XX/XX/26 (TRUST) Tool         : trust -energy JOB_ID returns now on well-configured cluster the energy consumption of your job
 -------------------------------------------------------------------------------------------------
 Release notes version 1.9.8_beta : Enhancements, modifications and corrected bugs since version 1.9.7 :
 --------------------------------------------------------------------------------------------------
@@ -7,7 +14,7 @@ Release notes version 1.9.8_beta : Enhancements, modifications and corrected bug
 30/04/26 (TRUST) New feature  : Introducing an IBM pre-processor + optimisations
 27/04/26 (TRUST) New feature  : Add new colocalised discretization that currently works with a compressible single-phase Euler or two-phase Baer-Nunziato problem integrating a Riemann solver (HLL & Rusanov).
 08/04/26 (TRUST) Change       : 'corriger_frontiere_periodique' becomes 'declarer_bord_perio'. Old keyword remains valid as a synonym. A periodic boundary must now **always** be declared with this keyword. The 'periodic' option in Partitioners is no longer necessary.
-20/04/26 (TRUST) Change       : Warning, VEF VerifierCoin keyword algorithm has been rewritten which possibly change the numerotation of the mesh and the partitioned mesh 
+20/04/26 (TRUST) Change       : Warning, VEF VerifierCoin keyword algorithm has been rewritten which possibly change the numerotation of the mesh and the partitioned mesh
 20/04/26 (TRUST) Solver       : Sparskit based solver "solveur gen { solv_elem bicgstab|gmres ... }" removed. It can be replaced by more efficient PETSc equivalent solvers
 20/04/26 (TRUST) Tool         : trust -energy JOB_ID returns now on well-configured cluster the energy consumption of your job
 14/04/26 (TRUST) New feature  : CGNS supports now reset_time used in the framework of ICoCo
@@ -18,8 +25,8 @@ Release notes version 1.9.8_beta : Enhancements, modifications and corrected bug
 09/04/26 (TRUST) Major change : TrioCFD radiation models (transparent and semi-transparent media) are re-written completely so that they can be used in basic classes of TRUST. See examples and validations in TrioCFD code. For transparent medium, it is possible to define now more than one model in a coupled problem; ie: a model in each fluid problem. It is also possible to use it with a distant problem; ie: without a coupled problem.
 31/03/26 (TRUST) Change       : Change the diffusive time step computation in multiphase VDF. Less restrictive, similar as in PolyMAC now.
 31/03/26 (TRUST) Change       : Change the diffusive time step computation in multiphase VDF. Less restrictive, similar
-as in PolyMAC now. 
-30/03/26 (TRUST) Performance  : Computation overlapped by communication in VEF convective schemes. Convection operator duration reduced by 20% on 128 GPU case on Adastra. 
+as in PolyMAC now.
+30/03/26 (TRUST) Performance  : Computation overlapped by communication in VEF convective schemes. Convection operator duration reduced by 20% on 128 GPU case on Adastra.
 12/03/26 (TRUST) New keyword  : Analyse_angle_64 version of Analyse_angle keyword for large mesh to compute angles histogram especially for tetraedras to detect too much obtuse angles
 24/02/26 (TRUST) GPU          : Single GPU performance on MI250X (adastra, lumi) and MI300A (adastra) improved by 30% after ROCm update (6.4.x)
 24/02/26 (TRUST) Bug fix      : Replace CG by GMRES in AMG solver for better robustness and speed especially on GPU (may not converge)
@@ -51,6 +58,14 @@ as in PolyMAC now.
 13/01/26 (TRUST) New feature  : Add new flag 'adapt_dt_tmax' in time scheme that ensures that the simulation ends at tmax
 08/01/26 (TRUST) Bug fix      : Fix for the case where dt_post is specified once in the header of post-processing block and where a mix of post-processings from files and old fashion is requested
 08/01/26 (TRUST) Bug fix      : Possible memory increase when writing some files (typically opened/closed at each write)
+07/01/26 (TRUST) New feature  : New mechanics module (Meca) integrated, including linear elasticity (Hooke's law), a Newmark solver for elastodynamics, and thermo-elastic source terms.
+07/01/26 (TRUST) New feature  : Enable bidim_axi support in EF and PolyMAC_MPFA.
+07/01/26 (TRUST) New feature  : DP_impose & regular pressure drops: time-dependent driving, regul option, and save/restore; dp_regul parameter renamed from eps to alpha.
+07/01/26 (TRUST) New feature  : New source term Echange_Thermique_Volumique: implicit heat conduction between two domains in VDF/Poly*
+07/01/26 (TRUST) New feature  : New Champ_Morceaux (piecewise fields on sub-domains) and improvements to Champ_Fonc_Tabule_Morceaux (can use post-processed fields).
+07/01/26 (TRUST) Fix          : Postprocessing on deformable/ALE domains: fix weighted_sum/average (face control volumes) and optimize volume_maille.
+07/01/26 (Tools) Fix          : Improve macOS/darwin builds (PDI, darwin_g++).
+07/01/26 (TRUST) Change       : Decouper_multi now splits domains following the order specified in the input file.
 --------------------------------------------------------------------------------------------------
 Release notes version 1.9.7 : Enhancements, modifications and corrected bugs since version 1.9.6 :
 --------------------------------------------------------------------------------------------------

ThirdPart/src/LIBAMGX/install

@@ -51,7 +51,7 @@ then
    # Hack du CMakeLists.txt (vu sur orcus avec Cuda 12) car manque de cublasLt
    sed -i "1,$ s?CUDA::cublas?CUDA::cublas CUDA::cublasLt?g" ../CMakeLists.txt || exit -1
 
-   if [ "$HOST" = jean-zay ]
+   if [ "$HOST" = jean-zay ] || [ $HOST = dalianvl ]
    then
       # ToDo: cublas not found on JeanZay so we simplify and it works now... Generalize to other builds ?
       cmake -DCMAKE_CUDA_COMPILER=$TRUST_NVCC -DAMGX_NO_RPATH=1 -DCMAKE_INSTALL_PREFIX=$AMGX_DIR/$LIB $AMGX_COMPILERS -DCMAKE_CUDA_FLAGS_RELEASE="-DNDEBUG" .. || exit -1

ThirdPart/src/LIBAMGXWRAPPER/install

@@ -67,6 +67,7 @@ then
    cd $AMGX_DIR/$LIB 
    rm -r -f example && mkdir -p example && cd example
    tests="poisson solveFromFiles"
+   tests=""
    for test in $tests
    do
       echo "Building $test test..."

ThirdPart/src/LIBCUDSS/install.sh

@@ -1,5 +1,7 @@
 #!/bin/bash
-build_dir=libcudss-linux-`uname -m`-0.6.0.5_cuda12-archive
+version=0.7.1.4
+cuda=12 && [ "`nvcc --version 2>/dev/null | grep cuda_13`" != "" ] && cuda=13
+build_dir=libcudss-linux-`uname -m`-$version"_cuda"$cuda-archive
 mkdir -p $TRUST_ROOT/lib/src/LIBCUDSS
 tar -xf $TRUST_ROOT/externalpackages/cudss/$build_dir.tar.xz || exit -1
 cp -r $build_dir/* $TRUST_ROOT/lib/src/LIBCUDSS

ThirdPart/src/LIBKOKKOS/install_arborx.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 [ "$TRUST_STDCPP" = c++14 ] && exit 0
 [ "$TRUST_STDCPP" = c++17 ] && exit 0
-archive=$TRUST_ROOT/externalpackages/kokkos/arborx-2.0.1.tar.gz # C++ 20 
+archive=$TRUST_ROOT/externalpackages/kokkos/arborx-2.1.tar.gz # C++ 20 
 
 build_dir=$TRUST_ROOT/build/arborx
 KOKKOS_ROOT_DIR=$TRUST_ROOT/lib/src/LIBKOKKOS

ThirdPart/src/LIBKOKKOS/install_kokkos-kernels.sh

@@ -3,7 +3,7 @@
 [ "$TRUST_USE_GPU" != 1 ] && exit 0
 
 # Kokkos-kernels:
-archive=$TRUST_ROOT/externalpackages/kokkos/kokkos-kernels-release-candidate-5.1.0.tar.gz
+archive=$TRUST_ROOT/externalpackages/kokkos/kokkos-kernels-5.1.1.tar.gz
 build_dir=$TRUST_ROOT/build/kokkos-kernels
 KOKKOS_ROOT_DIR=$TRUST_ROOT/lib/src/LIBKOKKOS
 # Log file of the process:

ThirdPart/src/LIBKOKKOS/install_kokkos.sh

@@ -3,7 +3,7 @@
 if [ "$TRUST_STDCPP" = c++20 ]
 then
    # Kokkos (C++20):
-   archive=$TRUST_ROOT/externalpackages/kokkos/kokkos-release-candidate-5.1.0.tar.gz
+   archive=$TRUST_ROOT/externalpackages/kokkos/kokkos-5.1.1.tar.gz
 elif [ "$TRUST_STDCPP" = c++17 ]
 then
    # Kokkos (C++17):

ThirdPart/src/LIBKOKKOS/makefile

@@ -1,8 +1,9 @@
 # Kokkos
-lib=$(TRUST_LIB)/src/LIBKOKKOS/$(TRUST_ARCH)$(OPT)/lib64/libkokkoscore.a
+lib1=$(TRUST_LIB)/src/LIBKOKKOS/$(TRUST_ARCH)$(OPT)/lib64/libkokkoscore.a
+lib2=$(TRUST_LIB)/src/LIBKOKKOS/$(TRUST_ARCH)$(OPT)/lib64/libkokkoskernels.a
 
-all: $(lib) 
-$(lib): install_kokkos.sh install_kokkos-kernels.sh install_arborx.sh makefile
+all: $(lib1) $(lib2)
+$(lib1) $(lib2): install_kokkos.sh install_kokkos-kernels.sh install_arborx.sh makefile
 	@make clean
 	./install_kokkos.sh && ./install_kokkos-kernels.sh && ./install_arborx.sh
 

ThirdPart/src/LIBLAPACK/Installer

@@ -11,7 +11,7 @@ install_OpenBlas()
    rm -r -f *-OpenBLAS-*
    OPENBLAS_USE_OPENMP=$TRUST_USE_OPENMP
    # Disable OpenMP for PETSc (Probleme avec STRUMPACK sur GPU)
-   OPENBLAS_USE_OPENMP=0 
+   OPENBLAS_USE_OPENMP=0
    gunzip -f -c $package | tar -xf -
    cd OpenBLAS-$version_openblas
    if [ "$TRUST_CC_BASE_EXTP" != "" ]
@@ -39,7 +39,7 @@ install_OpenBlas()
    #   CPU_ARCH=""
    #fi
    # Instructions -mavx512 fait crasher valgrind sur les machines avec instructions avx512 ... On desactive (comme dans TRUST d'ailleurs)
-   options="CC=$CC FC=$FC USE_THREAD=$OPENBLAS_USE_OPENMP USE_OPENMP=$OPENBLAS_USE_OPENMP NO_AVX512=1 $CPU_ARCH"
+   options="CC=$CC FC=$FC USE_THREAD=$OPENBLAS_USE_OPENMP USE_OPENMP=$OPENBLAS_USE_OPENMP NO_AVX512=1 BUILD_BFLOAT16=0 $CPU_ARCH"
    echo "Installation of $package ($options)..."
    if [ `uname -s` = Darwin ]
    then
@@ -144,8 +144,7 @@ version_lapack=3.4.1 && [ "$TRUST_INT64" != "1" ] && [ "$TRUST_USE_MUMPS" != 1 ]
 # [HPC][!Portabilite] Utilisation OpenBlas par defaut (valide sur TRUST/F5/TrioCFD/G3) -> Decomposition LU plus rapide
 if [ "$TRUST_USE_OPENBLAS" = 1 ]
 then
-   #for tag in 0.3.25 0.3.29
-   for tag in 0.3.29
+   for tag in 0.3.33
    do 
      version_openblas=$tag
      echo "version_openblas = $tag"

ThirdPart/src/LIBPETSC/amgx_int32.cxx

@@ -552,30 +552,30 @@ static PetscErrorCode PCView_AMGX(PC pc, PetscViewer viewer)
 }
 
 /*MC
-     PCAMGX - Interface to NVIDIA's AmgX algebraic multigrid
-
-   Options Database Keys:
-+    -pc_amgx_amg_method <CLASSICAL,AGGREGATION> - set the AMG algorithm to use
-.    -pc_amgx_amg_cycle <V,W,F,CG> - set the AMG cycle type
-.    -pc_amgx_smoother <PCG,PCGF,PBICGSTAB,GMRES,FGMRES,JACOBI_L1,BLOCK_JACOBI,GS,MULTICOLOR_GS,MULTICOLOR_ILU,MULTICOLOR_DILU,CHEBYSHEV_POLY,NOSOLVER> - set the AMG pre/post smoother
-.    -pc_amgx_jacobi_relaxation_factor - set the relaxation factor for Jacobi smoothing
-.    -pc_amgx_gs_symmetric - enforce symmetric Gauss-Seidel smoothing (only applies if GS smoothing is selected)
-.    -pc_amgx_selector <SIZE_2,SIZE_4,SIZE_8,MULTI_PAIRWISE,PMIS,HMIS> - set the AMG coarse selector
-.    -pc_amgx_presweeps - set the number of AMG pre-sweeps
-.    -pc_amgx_postsweeps - set the number of AMG post-sweeps
-.    -pc_amgx_max_levels - set the maximum number of levels in the AMG level hierarchy
-.    -pc_amgx_strength_threshold - set the strength threshold for the AMG coarsening
-.    -pc_amgx_aggressive_levels - set the number of levels (from the finest) that should apply aggressive coarsening
-.    -pc_amgx_coarse_solver <DENSE_LU_SOLVER,NOSOLVER> - set the coarse solve
-.    -pc_amgx_print_grid_stats - output the AMG grid hierarchy to stdout
--    -pc_amgx_verbose - enable AmgX output
+  PCAMGX - Interface to NVIDIA's AmgX algebraic multigrid
+
+  Options Database Keys:
++    -pc_amgx_amg_method (CLASSICAL,AGGREGATION)                       - set the AMG algorithm to use
+.    -pc_amgx_amg_cycle (V,W,F,CG)                                     - set the AMG cycle type
+.    -pc_amgx_jacobi_relaxation_factor                                 - set the relaxation factor for Jacobi smoothing
+.    -pc_amgx_gs_symmetric                                             - enforce symmetric Gauss-Seidel smoothing (only applies if GS smoothing is selected)
+. -pc_amgx_selector (SIZE_2|SIZE_4|SIZE_8|MULTI_PAIRWISE|PMIS|HMIS) - set the AMG coarse selector
+.    -pc_amgx_presweeps                                                - set the number of AMG pre-sweeps
+.    -pc_amgx_postsweeps                                               - set the number of AMG post-sweeps
+.    -pc_amgx_max_levels                                               - set the maximum number of levels in the AMG level hierarchy
+.    -pc_amgx_strength_threshold                                       - set the strength threshold for the AMG coarsening
+.    -pc_amgx_aggressive_levels                                        - set the number of levels (from the finest) that should apply aggressive coarsening
+.    -pc_amgx_coarse_solver (DENSE_LU_SOLVER,NOSOLVER)                 - set the coarse solve
+.    -pc_amgx_print_grid_stats                                         - output the AMG grid hierarchy to `stdout`
+-    -pc_amgx_verbose                                                  - enable AmgX verbose output
+-    -pc_amgx_smoother (PCG|PCGF|PBICGSTAB|GMRES|FGMRES|JACOBI_L1|BLOCK_JACOBI|GS|MULTICOLOR_GS|MULTICOLOR_ILU|MULTICOLOR_DILU|CHEBYSHEV_POLY|NOSOLVER) - set the AMG pre/post smoother
 
    Level: intermediate
 
    Note:
-     Implementation will accept host or device pointers, but good performance will require that the `KSP` is also GPU accelerated so that data is not frequently transferred between host and device.
+   Implementation will accept host or device pointers, but good performance will require that the `KSP` is also GPU accelerated so that data is not frequently transferred between host and device.
 
-.seealso: [](ch_ksp), `PCGAMG`, `PCHYPRE`, `PCMG`, `PCAmgXGetResources()`, `PCCreate()`, `PCSetType()`, `PCType` (for list of available types), `PC`
+.seealso: [](ch_ksp), `PCGAMG`, `PCHYPRE`, `PCMG`, `PCAmgXGetResources()`, `PCCreate()`, `PCSetType()`, `PCType`, `PC`
 M*/
 
 PETSC_EXTERN PetscErrorCode PCCreate_AMGX(PC pc)

ThirdPart/src/LIBPETSC/amgx_int64.cxx

@@ -556,30 +556,30 @@ static PetscErrorCode PCView_AMGX(PC pc, PetscViewer viewer)
 }
 
 /*MC
-     PCAMGX - Interface to NVIDIA's AmgX algebraic multigrid
-
-   Options Database Keys:
-+    -pc_amgx_amg_method <CLASSICAL,AGGREGATION> - set the AMG algorithm to use
-.    -pc_amgx_amg_cycle <V,W,F,CG> - set the AMG cycle type
-.    -pc_amgx_smoother <PCG,PCGF,PBICGSTAB,GMRES,FGMRES,JACOBI_L1,BLOCK_JACOBI,GS,MULTICOLOR_GS,MULTICOLOR_ILU,MULTICOLOR_DILU,CHEBYSHEV_POLY,NOSOLVER> - set the AMG pre/post smoother
-.    -pc_amgx_jacobi_relaxation_factor - set the relaxation factor for Jacobi smoothing
-.    -pc_amgx_gs_symmetric - enforce symmetric Gauss-Seidel smoothing (only applies if GS smoothing is selected)
-.    -pc_amgx_selector <SIZE_2,SIZE_4,SIZE_8,MULTI_PAIRWISE,PMIS,HMIS> - set the AMG coarse selector
-.    -pc_amgx_presweeps - set the number of AMG pre-sweeps
-.    -pc_amgx_postsweeps - set the number of AMG post-sweeps
-.    -pc_amgx_max_levels - set the maximum number of levels in the AMG level hierarchy
-.    -pc_amgx_strength_threshold - set the strength threshold for the AMG coarsening
-.    -pc_amgx_aggressive_levels - set the number of levels (from the finest) that should apply aggressive coarsening
-.    -pc_amgx_coarse_solver <DENSE_LU_SOLVER,NOSOLVER> - set the coarse solve
-.    -pc_amgx_print_grid_stats - output the AMG grid hierarchy to stdout
--    -pc_amgx_verbose - enable AmgX output
+  PCAMGX - Interface to NVIDIA's AmgX algebraic multigrid
+
+  Options Database Keys:
++    -pc_amgx_amg_method (CLASSICAL,AGGREGATION)                       - set the AMG algorithm to use
+.    -pc_amgx_amg_cycle (V,W,F,CG)                                     - set the AMG cycle type
+.    -pc_amgx_jacobi_relaxation_factor                                 - set the relaxation factor for Jacobi smoothing
+.    -pc_amgx_gs_symmetric                                             - enforce symmetric Gauss-Seidel smoothing (only applies if GS smoothing is selected)
+. -pc_amgx_selector (SIZE_2|SIZE_4|SIZE_8|MULTI_PAIRWISE|PMIS|HMIS) - set the AMG coarse selector
+.    -pc_amgx_presweeps                                                - set the number of AMG pre-sweeps
+.    -pc_amgx_postsweeps                                               - set the number of AMG post-sweeps
+.    -pc_amgx_max_levels                                               - set the maximum number of levels in the AMG level hierarchy
+.    -pc_amgx_strength_threshold                                       - set the strength threshold for the AMG coarsening
+.    -pc_amgx_aggressive_levels                                        - set the number of levels (from the finest) that should apply aggressive coarsening
+.    -pc_amgx_coarse_solver (DENSE_LU_SOLVER,NOSOLVER)                 - set the coarse solve
+.    -pc_amgx_print_grid_stats                                         - output the AMG grid hierarchy to `stdout`
+-    -pc_amgx_verbose                                                  - enable AmgX verbose output
+-    -pc_amgx_smoother (PCG|PCGF|PBICGSTAB|GMRES|FGMRES|JACOBI_L1|BLOCK_JACOBI|GS|MULTICOLOR_GS|MULTICOLOR_ILU|MULTICOLOR_DILU|CHEBYSHEV_POLY|NOSOLVER) - set the AMG pre/post smoother
 
    Level: intermediate
 
    Note:
-     Implementation will accept host or device pointers, but good performance will require that the `KSP` is also GPU accelerated so that data is not frequently transferred between host and device.
+   Implementation will accept host or device pointers, but good performance will require that the `KSP` is also GPU accelerated so that data is not frequently transferred between host and device.
 
-.seealso: [](ch_ksp), `PCGAMG`, `PCHYPRE`, `PCMG`, `PCAmgXGetResources()`, `PCCreate()`, `PCSetType()`, `PCType` (for list of available types), `PC`
+.seealso: [](ch_ksp), `PCGAMG`, `PCHYPRE`, `PCMG`, `PCAmgXGetResources()`, `PCCreate()`, `PCSetType()`, `PCType`, `PC`
 M*/
 
 PETSC_EXTERN PetscErrorCode PCCreate_AMGX(PC pc)