Table of contents

1. Running Cosmolike projects (Basic instructions)
2. Plotting the example chains
3. Baryonic feedback from the bfmt theory block
4. Running Hybrid Cosmolike-ML emulators
5. Unit tests
6. Exploring notebooks
7. Computing covariances
8. Appendix: Which accuracy settings are available?

Running Cosmolike projects (Basic instructions)

From Cocoa/Readme instructions:

Note

Cocoa provides several cosmolike projects that can be loaded and compiled using setup_cocoa.sh and compile_cocoa.sh scripts. To activate them, comment the following lines on set_installation_options.sh

[Adapted from Cocoa/set_installation_options.sh shell script]
(...)

# ------------------------------------------------------------------------------
# The keys below control which cosmolike projects will be installed and compiled
# ------------------------------------------------------------------------------
#export IGNORE_COSMOLIKE_LSST_Y1_CODE=1
(...)
export IGNORE_COSMOLIKE_DES_Y3_CODE=1

(...)
# ------------------------------------------------------------------------------
# Cosmolike projects below -------------------------------------------
# ------------------------------------------------------------------------------
(...)
export DES_Y3_URL="https://github.com/CosmoLike/cocoa_des_y3.git"
export DES_Y3_NAME="des_y3"
#Pin the project version with at most one of the keys below (COMMIT, BRANCH, or TAG).
#If more than one is set, COMMIT wins over BRANCH, and BRANCH wins over TAG.
#If none is set, Cocoa loads the latest commit on the repository default branch.
#export DES_Y3_GIT_BRANCH="main"
#export DES_Y3_GIT_COMMIT="abc"
export DES_Y3_GIT_TAG="v5.05"

Note

In case users need to rerun setup_cocoa.sh, Cocoa will not download previously installed packages, cosmolike projects, or large datasets, unless the following keys are set on set_installation_options.sh

[Adapted from Cocoa/set_installation_options.sh shell script]
# ------------------------------------------------------------------------------
# OVERWRITE_EXISTING_XXX_CODE=1 -> setup_cocoa overwrites existing PACKAGES ----
# overwrite: delete the existing PACKAGE folder and install it again -----------
# redownload: delete the compressed file and download data again ---------------
# These keys are only relevant if you run setup_cocoa multiple times -----------
# ------------------------------------------------------------------------------
(...)
export OVERWRITE_EXISTING_ALL_PACKAGES=1    # except cosmolike projects
#export OVERWRITE_EXISTING_COSMOLIKE_CODE=1 # dangerous (possible loss of uncommitted work)
                                            # if unset, users must manually delete cosmolike projects
#export REDOWNLOAD_EXISTING_ALL_DATA=1      # warning: some data is many GB

Note

If users want to recompile cosmolike, there is no need to rerun the Cocoa general scripts. Instead, run the following three commands:

 source start_cocoa.sh

and

 source ./installation_scripts/setup_cosmolike_projects.sh

and

  source ./installation_scripts/compile_all_projects.sh

or (in case users just want to compile des_y3 project)

  source ./projects/des_y3/scripts/compile_des_y3.sh

Tip

Assuming Cocoa is installed on a local (not remote!) machine, type the command below after step 2️⃣ to run Jupyter Notebooks.

jupyter notebook --no-browser --port=8888

The terminal will then show a message similar to the following template:

(...)
[... NotebookApp] Jupyter Notebook 6.1.1 is running at:
[... NotebookApp] http://f0a13949f6b5:8888/?token=XXX
[... NotebookApp] or http://127.0.0.1:8888/?token=XXX
[... NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Now go to the local internet browser and type http://127.0.0.1:8888/?token=XXX, where XXX is the previously saved token displayed on the line

[... NotebookApp] or http://127.0.0.1:8888/?token=XXX

The project des_y3 contains jupyter notebook examples located at projects/des_y3.

Note

The example notebooks load their shared support functions from Cocoa/external_modules/code/cosmolike_core/cosmolike_notebook_utils/: the CAMB run packaged for cosmolike, the data-vector plots, and the Fisher-forecast helpers, called through the cnu namespace. The notebooks keep only what is specific to this project: fiducial values, the compiled-interface calls, and thin wrappers binding them to the shared functions.

To run the example

Step 1️⃣: activate the Cocoa Conda environment, and the private Python environment

  conda activate cocoa

and

  source start_cocoa.sh

Step 2️⃣: Select the number of OpenMP cores (8 below).

• Linux

  export OMP_NUM_THREADS=8; export OMP_PROC_BIND=close; \
  export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \
  export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
  

• macOS (arm)

  export OMP_NUM_THREADS=8; export OMP_PROC_BIND=disabled; \
  export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \
  export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
  

Step 3️⃣: The folder projects/des_y3 contains examples. So, run the cobaya-run on the first example following the commands below.

Warning

(Linux only) In some HPC nodes, numa can cause you problems. If that is the case, replace numa with slot

• One model evaluation:

  • Linux

    "${CONDA_PREFIX}"/bin/mpirun -n 1 --oversubscribe \
      --mca pml ob1 --mca btl vader,tcp,self \
      --bind-to core:overload-allowed --report-bindings \
      --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \
      cobaya-run ./projects/des_y3/EXAMPLE_EVALUATE1.yaml -f
    

  • macOS (arm)

    mpirun -n 1 --oversubscribe \
     cobaya-run ./projects/des_y3/EXAMPLE_EVALUATE1.yaml -f
    

• MCMC (Metropolis-Hastings Algorithm):

  • Linux

    "${CONDA_PREFIX}"/bin/mpirun -n 1 --oversubscribe \
      --mca pml ob1 --mca btl vader,tcp,self \
      --bind-to core:overload-allowed --report-bindings \
      --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \
      cobaya-run ./projects/des_y3/EXAMPLE_MCMC1.yaml -f
    

  • macOS (arm)

    mpirun -n 4 --oversubscribe \
      cobaya-run ./projects/des_y3/EXAMPLE_MCMC1.yaml -f
    

The likelihoods of the examples, the parameter files they include, and the parameters each likelihood fixes or must not vary are described in likelihood/README.md.

Plotting the example chains

The scripts in plots/ draw GetDist triangle plots, each comparing two chains. They read the chains from projects/des_y3/chains/, discard the first half of each chain as burn-in, add $S_8=\sigma_8(\Omega_m/0.3)^{0.5}$, and write plotN.pdf into the folder they run in.

Script	Chains compared	Parameters shown
plot1.py	EXAMPLE_MCMC3 (DES-Y1 cosmic shear) and EXAMPLE_MCMC1 (DES-Y3 cosmic shear)	Cosmological
plot2.py	EXAMPLE_MCMC1 (DES-Y3 cosmic shear) and EXAMPLE_MCMC2 (DES-Y3 3x2pt)	Cosmological
plot3.py	EXAMPLE_MCMC1 and EXAMPLE_MCMC2	Intrinsic-alignment amplitudes with $\Omega_m$, $\sigma_8$, $w$, $w_a$
plot4.py	EXAMPLE_MCMC3 (DES-Y1 cosmic shear) and EXAMPLE_MCMC4 (DES-Y1 3x2pt)	Cosmological
plot5.py	EXAMPLE_MCMC3 and EXAMPLE_MCMC4	Intrinsic-alignment amplitudes with $\Omega_m$, $\sigma_8$, $w$, $w_a$

Only EXAMPLE_MCMC1.yaml and EXAMPLE_MCMC2.yaml ship with the project. The DES-Y1 chains come from the same two files with data_file: des_y1_real.dataset and the output renamed to EXAMPLE_MCMC3 and EXAMPLE_MCMC4.

flowchart TD
  A["EXAMPLE_MCMC1.yaml: DES-Y3 cosmic shear"] --> C["plot2.py, plot3.py: Y3 shear vs Y3 3x2pt"]
  B["EXAMPLE_MCMC2.yaml: DES-Y3 3x2pt"] --> C
  A --> D["plot1.py: Y1 shear vs Y3 shear"]
  E["EXAMPLE_MCMC3: MCMC1 on des_y1_real.dataset"] --> D
  E --> F["plot4.py, plot5.py: Y1 shear vs Y1 3x2pt"]
  G["EXAMPLE_MCMC4: MCMC2 on des_y1_real.dataset"] --> F

Loading

We assume users are in the Conda cocoa environment, that the shell is bash, that the current folder is cocoa/Cocoa, and that the chains exist.

Step 1️⃣: activate Cocoa.

  source start_cocoa.sh

Step 2️⃣: enter the plots folder; the scripts read ../chains/ relative to it.

  cd ./projects/des_y3/plots

Step 3️⃣: draw the DES-Y3 comparison.

  python plot2.py

plot_all.sh runs plot1.py, plot2.py and plot3.py with their printed output discarded.

Baryonic feedback from the bfmt theory block

EXAMPLE_EVALUATE1.yaml can apply an external baryonic feedback suppression to the matter power spectrum via the bfmt theory block (SP(k), BCEmu, Flamingo, BACCOemu, or BCemu2025). By default, the example runs without feedback. The 3×2pt notebook compares the block's models in its own section; both need the installation of Step 1️⃣.

Step 1️⃣: ensure the lines below are commented out in set_installation_options.sh before running setup_cocoa.sh and compile_cocoa.sh. By default, these lines should be commented out, but it is worth checking.

  [Adapted from Cocoa/set_installation_options.sh shell script]
  #export IGNORE_PYSPK_CODE=1     # SP(k)
  #export IGNORE_BCEMU_CODE=1     # BCEmu and BCemu2025
  #export IGNORE_FBRE_CODE=1      # FlamingoBaryonResponseEmulator
  #export IGNORE_BACCOEMU_CODE=1  # BACCOemu
  #export IGNORE_BFMT_CODE=1      # Baryon Feedback Theory Block

Step 2️⃣: in EXAMPLE_EVALUATE1.yaml, uncomment the bfmt theory block and select the model:

  theory:
    bfmt:
      baryon_model: 2 # 1 = SP(k), 2 = BCEmu, 3 = FlamingoEmulator, 4 = BACCOemu, 5 = BCemu2025

Step 3️⃣: set external_baryon_suppression: True on the des_y3.cosmic_shear likelihood block.

Step 4️⃣: uncomment the selected model's parameters in the params block and in the sampler: evaluate: override block (the example carries a commented block for each model).

Tip

For the sampled parameters of each model, their validity ranges, and the bfmt options, see Cocoa/external_modules/code/baryon_suppression/README.md.

The bfmt study in EXAMPLE_EVALUATE2.ipynb

The notebook section Baryonic feedback from the bfmt theory block evaluates the block through a minimal Cobaya model (CAMB, bfmt and the one likelihood), so it needs the installation of Step 1️⃣ but none of the YAML edits. It runs six methods at fixed parameter points: the three SP(k) relations, BCEmu, Flamingo and BCemu2025. BACCOemu stays commented out: its omega_baryon training range starts at 0.04001, above the notebook's omegab = 0.04. Each suppression $S(k,z)$ multiplies the nonlinear matter power, as external_baryon_suppression: True does in the likelihood, and two figures draw $C_\ell^{gs}$ and $\gamma_t(\theta)$ relative to no feedback. The section runs CAMB 13 times.

Its table covers the 533 entries of the baseline 3×2pt mask:

Column	Meaning
$\chi^2$	Against the measured DES Y3 data vector.
$\Delta\chi^2$	$\chi^2$ minus the no-feedback $\chi^2$, of either sign.
$\chi^2$ of the shift	$(t-t_{\rm none})^{\mathsf T}C^{-1}(t-t_{\rm none})$: the change of the prediction $t$ in units of the DES Y3 errors, independent of the data.

In the executed notebook, the no-feedback $\chi^2 = 1369.0618$, and the $\chi^2$ of the shift stays below one for every method, from 0.0165 (BCemu2025) to 0.8075 (SP(k) power law): the baseline scale cuts keep these feedback strengths below the DES Y3 errors. $\Delta\chi^2$ reaches $-12.3832$ (SP(k) power law) because the fiducial point is not a fit to the measurements, so a small shift can still move the prediction toward the data.

Running Hybrid Cosmolike-ML emulators

Note

These hybrid examples remain experimental. The checks below verify the workflow; assess emulator accuracy and posterior convergence for your analysis.

The EXAMPLE_EMUL2 examples emulate the background expansion and matter power spectra. CosmoLike still computes the survey projections, bias and intrinsic-alignment contributions. Changing n(z) or nuisance parameters does not require retraining a survey data-vector network.

The shared theory networks live in external_modules/data/emultrf. Install them through the main Cocoa emulator recipe. These networks assume mnu = 0.06 eV; do not sample neutrino mass. Their cold-matter power approximation is not a calibrated massive-neutrino halo model. Check their training range before widening cosmological priors.

We assume Cocoa and this project are installed, the Cocoa Conda environment is active, the shell is Bash, and the current folder is cocoa/Cocoa/.

Step 1️⃣: activate Cocoa.

source start_cocoa.sh

Step 2️⃣: Select the number of OpenMP cores (below, we set it to 4).

• Linux

  export OMP_NUM_THREADS=4; export OMP_PROC_BIND=close; \
  export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \
  export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
  

• macOS (arm)

  export OMP_NUM_THREADS=4; export OMP_PROC_BIND=disabled; \
  export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \
  export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
  

Step 3️⃣: remove GPU access on Linux; these examples use the CPU.

export CUDA_VISIBLE_DEVICES=""

Step 4️⃣: evaluate the first hybrid example.

• Linux

  "${CONDA_PREFIX}"/bin/mpirun -n 1 --oversubscribe \
    --mca pml ob1 --mca btl vader,tcp,self \
    --bind-to core:overload-allowed --report-bindings \
    --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \
    cobaya-run ./projects/des_y3/EXAMPLE_EMUL2_EVALUATE1.yaml --force
  

• macOS (arm)

  mpirun -n 1 --oversubscribe \
    cobaya-run ./projects/des_y3/EXAMPLE_EMUL2_EVALUATE1.yaml --force
  

The YAML selects the CPU for the distance emulator. Keep BLAS at one thread per MPI rank (OPENBLAS_NUM_THREADS=1, MKL_NUM_THREADS=1); on macOS also use VECLIB_MAXIMUM_THREADS=1. The Python sampler entry points set these BLAS limits before importing numerical libraries.

Example	Configuration 1	Configuration 2
Fixed evaluation	EXAMPLE_EMUL2_EVALUATE1.yaml	EXAMPLE_EMUL2_EVALUATE2.yaml
Cobaya MCMC	EXAMPLE_EMUL2_MCMC1.yaml	EXAMPLE_EMUL2_MCMC2.yaml
Annealed minimization	EXAMPLE_EMUL2_MINIMIZE1.py	EXAMPLE_EMUL2_MINIMIZE2.py
Parameter profile	EXAMPLE_EMUL2_PROFILE1.py	EXAMPLE_EMUL2_PROFILE2.py
Nautilus sampling	EXAMPLE_EMUL2_NAUTILUS1.py	EXAMPLE_EMUL2_NAUTILUS2.py

Configuration 1 uses des_y3.cosmic_shear, TATT, and des_y3_real.dataset. Configuration 2 uses des_y3.combo_3x2pt and des_y3_real.dataset, with NLA in EXAMPLE_EMUL2_EVALUATE2.yaml (and the scripts that read it) but TATT in EXAMPLE_EMUL2_MCMC2.yaml.

The minimization, profile and Nautilus scripts read the corresponding EXAMPLE_EMUL2_EVALUATE1.yaml or 2.yaml; --input selects another evaluate YAML. They require cocoa_hybrid_sampling.py from the matching shared core revision. They do not maintain separate embedded cosmologies. --check evaluates the specified fiducial and prints the sampled parameter order without sampling. Use a new --outroot for each run; these scripts refuse to overwrite results.

Cobaya MCMC

With Cocoa activated in Bash from cocoa/Cocoa/, run the first MCMC example. Use configuration 2 for the second likelihood listed above. Check chain convergence before interpreting posterior constraints.

Step 1️⃣: Select the number of OpenMP cores (below, we set it to 8).

• Linux

  export OMP_NUM_THREADS=8; export OMP_PROC_BIND=close; \
  export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \
  export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
  

• macOS (arm)

  export OMP_NUM_THREADS=8; export OMP_PROC_BIND=disabled; \
  export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \
  export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
  

Step 2️⃣: remove GPU access on Linux; these examples use the CPU.

export CUDA_VISIBLE_DEVICES=""

Step 3️⃣: start Cobaya's hybrid MCMC.

• Linux

  "${CONDA_PREFIX}"/bin/mpirun -n 4 --oversubscribe \
    --mca pml ob1 --mca btl vader,tcp,self \
    --bind-to core:overload-allowed --report-bindings \
    --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \
    cobaya-run ./projects/des_y3/EXAMPLE_EMUL2_MCMC1.yaml
  

• macOS (arm)

  mpirun -n 4 --oversubscribe \
    cobaya-run ./projects/des_y3/EXAMPLE_EMUL2_MCMC1.yaml
  

Minimization, profiles and Nautilus

We assume Cocoa and this project are installed, the Cocoa Conda environment is active, the shell is Bash, and the current folder is cocoa/Cocoa/.

Step 1️⃣: Select the number of OpenMP cores (below, we set it to 8).

• Linux

  export OMP_NUM_THREADS=8; export OMP_PROC_BIND=close; \
  export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \
  export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
  

• macOS (arm)

  export OMP_NUM_THREADS=8; export OMP_PROC_BIND=disabled; \
  export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \
  export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
  

Step 2️⃣: remove GPU access on Linux; these examples use the CPU.

export CUDA_VISIBLE_DEVICES=""

Step 3️⃣: check the hybrid setup before a long run.

python ./projects/des_y3/EXAMPLE_EMUL2_MINIMIZE1.py --check

Step 4️⃣: search for a minimum.

• Linux

  "${CONDA_PREFIX}"/bin/mpirun -n 21 --oversubscribe \
    --mca pml ob1 --mca btl vader,tcp,self \
    -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX -x ROOTDIR \
    -x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES -x OMP_DYNAMIC \
    -x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CLIK_PATH -x CLIK_DATA \
    -x CLIK_PLUGIN -x CUDA_VISIBLE_DEVICES --mca mpi_yield_when_idle 1 \
    --mca btl_tcp_if_exclude lo,docker0,virbr0,ib0 \
    --bind-to core:overload-allowed --report-bindings \
    --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \
    python ./projects/des_y3/EXAMPLE_EMUL2_MINIMIZE1.py --nstw 200 --outroot hybrid_min1
  

• macOS (arm)

  mpirun -n 12 --oversubscribe \
    python ./projects/des_y3/EXAMPLE_EMUL2_MINIMIZE1.py --nstw 200 --outroot hybrid_min1
  

Step 5️⃣: profile the first sampled parameter using that saved minimum.

• Linux

  "${CONDA_PREFIX}"/bin/mpirun -n 21 --oversubscribe \
    --mca pml ob1 --mca btl vader,tcp,self \
    -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX -x ROOTDIR \
    -x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES -x OMP_DYNAMIC \
    -x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CLIK_PATH -x CLIK_DATA \
    -x CLIK_PLUGIN -x CUDA_VISIBLE_DEVICES --mca mpi_yield_when_idle 1 \
    --mca btl_tcp_if_exclude lo,docker0,virbr0,ib0 \
    --bind-to core:overload-allowed --report-bindings \
    --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \
    python ./projects/des_y3/EXAMPLE_EMUL2_PROFILE1.py --profile 0 --nstw 200 --numpts 11 --factor 1 --minfile ./projects/des_y3/chains/hybrid_min1.json --outroot hybrid_profile1
  

• macOS (arm)

  mpirun -n 12 --oversubscribe \
    python ./projects/des_y3/EXAMPLE_EMUL2_PROFILE1.py --profile 0 --nstw 200 --numpts 11 --factor 1 --minfile ./projects/des_y3/chains/hybrid_min1.json --outroot hybrid_profile1
  

Step 6️⃣: run Nautilus as an independent sampling example.

• Linux

  "${CONDA_PREFIX}"/bin/mpirun -n 90 --oversubscribe \
    --mca pml ob1 --mca btl vader,tcp,self \
    -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX -x ROOTDIR \
    -x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES -x OMP_DYNAMIC \
    -x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CLIK_PATH -x CLIK_DATA \
    -x CLIK_PLUGIN -x CUDA_VISIBLE_DEVICES --mca mpi_yield_when_idle 1 \
    --mca btl_tcp_if_exclude lo,docker0,virbr0,ib0 \
    --bind-to core:overload-allowed --report-bindings \
    --rank-by slot --map-by numa:pe=${OMP_NUM_THREADS} \
    python ./projects/des_y3/EXAMPLE_EMUL2_NAUTILUS1.py --nlive 1000 --neff 10000 --maxfeval 100000 --outroot hybrid_nautilus1
  

• macOS (arm)

  mpirun -n 12 --oversubscribe \
    python ./projects/des_y3/EXAMPLE_EMUL2_NAUTILUS1.py --nlive 1000 --neff 10000 --maxfeval 100000 --outroot hybrid_nautilus1
  

The annealed Emcee search follows the DES × Planck template. Its objective is −2 log posterior, including nuisance and cosmological priors; the profile is therefore a penalized profile, not a pure likelihood profile. --nstw sets steps per walker per temperature. More steps and independent starts are needed to assess whether a minimum is reliable.

--profile accepts a sampled-parameter name or its printed zero-based index. --factor gives the half-width in proposal standard deviations, clipped to the prior bounds. --cov accepts a covariance whose header lists the sampled parameters in order; without it, the prior covariance sets the proposal. The minimum JSON must come from the same evaluate YAML and parameter order. Older plain-text minimum files are not accepted. Set any additional priors in the input YAML; these scripts do not insert hidden cosmological priors.

Nautilus writes weighted GetDist-compatible rows and a JSON convergence record. Reaching --maxfeval is not convergence. If the budget ends before any posterior samples are retained, only the checkpoint and a JSON record with converged: false are saved. Its prior transform uses Cobaya's one-dimensional prior distributions; external prior factors enter once as additional log weight. Evidence with unnormalized external priors has that normalization limitation. These examples do not certify emulator accuracy or posterior convergence.

Emulator design and optional approximations

Details on the matter power spectrum emulator designs will be presented in the emulator_code repository.

The networks apply standard neural network techniques to generalize the syren-new Eq. 6 of arXiv:2410.14623 formula for the linear power spectrum (w0waCDM with a fixed neutrino mass of $0.06$ eV) to new models, extended ranges, or higher precision. Similar networks generalize the syren-Halofit LCDM nonlinear boost fit (Eq. 11 of arXiv:2402.17492).

MPI across nodes

Note

Running on more than one node.

1. Network interface. TCP must use an interface routable between compute nodes. A common exclusion list is --mca btl_tcp_if_exclude lo,docker0,virbr0,ib0; adapt it to the cluster. Keep ib0 if routable IP-over-InfiniBand is the intended network. These examples exchange parameter vectors and scalar scores, so communication volume is small; actual scaling still depends on the machine.
2. Environment. Remote ranks need the same Cocoa paths and libraries: ROOTDIR, PATH, LD_LIBRARY_PATH, PYTHONPATH, CONDA_PREFIX, OpenMP and BLAS settings, and CLIK_PATH/CLIK_DATA/CLIK_PLUGIN when used. Slurm normally exports the submitting environment (--export=ALL). Explicit -x options also forward these variables with SSH launchers. Activate Cocoa before launching; build/download flags do not replace runtime paths. All nodes must see the same files at the same paths.
3. Slurm geometry. Keep ntasks-per-node × cpus-per-task within the allocated physical cores per node. Set OMP_NUM_THREADS to SLURM_CPUS_PER_TASK and use --map-by numa:pe=${OMP_NUM_THREADS}. The minimization, profile and Nautilus pool reserves one MPI rank as coordinator; the remaining ranks evaluate the model.

Within a Slurm allocation, first activate Cocoa in Bash on the launch node. The following steps assume Open MPI 4 and shared installation/data paths. Omit optional CLIK exports if those variables are not set.

Step 1️⃣: match OpenMP threads to the scheduler allocation.

export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK}"; export OMP_PROC_BIND=close; \
export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \
export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1

Step 2️⃣: launch the hybrid minimizer across the allocated ranks.

"${CONDA_PREFIX}"/bin/mpirun -n "${SLURM_NTASKS}" \
  --mca pml ob1 --mca btl vader,tcp,self \
  --mca btl_tcp_if_exclude lo,docker0,virbr0 \
  --map-by numa:pe=${OMP_NUM_THREADS} --bind-to core --report-bindings \
  -x ROOTDIR -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x CONDA_PREFIX \
  -x OMP_NUM_THREADS -x OMP_PROC_BIND -x OMP_PLACES -x OMP_DYNAMIC \
  -x OPENBLAS_NUM_THREADS -x MKL_NUM_THREADS -x CUDA_VISIBLE_DEVICES \
  python ./projects/des_y3/EXAMPLE_EMUL2_MINIMIZE1.py --nstw 200 --outroot hybrid_multinode

For a Planck likelihood add -x CLIK_PATH -x CLIK_DATA -x CLIK_PLUGIN when those variables are defined. Follow the cluster's MPI module and Slurm launch policy; do not oversubscribe a production allocation. Outside Slurm, supply the hosts and slots with the cluster's --hostfile or --host recipe.

Unit tests

The tests/ folder compares likelihood predictions against stored reference values, checks for race conditions from OpenMP threading, and measures the numerical error of the default accuracy settings. tests/README.md describes the checks, how to run them, the input data snapshots, and how to refresh them.

Exploring notebooks

Armadillo was chosen to make a convenient Python API for notebook exploration. This C++ library provides vectors, matrices and three-dimensional arrays called cubes. A small interface layer connects them to NumPy through pybind11, with CARMA handling array conversion. The notebooks expose intermediate quantities; production calculations use the CLI interfaces.

We assume Cocoa and this project are installed, the Cocoa Conda environment is active, the shell is Bash, and the current folder is cocoa/Cocoa/.

Compile the project first; the covariance notebook also needs the optional covariance build described below.

Step 1️⃣: activate Cocoa.

source start_cocoa.sh

Step 2️⃣: Select the number of OpenMP cores (below, we set it to 8).

• Linux

  export OMP_NUM_THREADS=8; export OMP_PROC_BIND=close; \
  export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \
  export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
  

• macOS (arm)

  export OMP_NUM_THREADS=8; export OMP_PROC_BIND=disabled; \
  export OMP_PLACES=cores; export OMP_DYNAMIC=FALSE; \
  export OPENBLAS_NUM_THREADS=1; export MKL_NUM_THREADS=1
  

Step 3️⃣: start Jupyter.

jupyter notebook --no-browser --port=8888

Step 4️⃣: open the printed URL and choose a notebook below.

Step 5️⃣: select Kernel → Restart Kernel and Run All Cells.

Notebook	Contents
EXAMPLE_EVALUATE1.ipynb	Cosmic shear, $C_\ell^{EE}$ and $\xi_\pm(\theta)$, through wrapper functions defined in the notebook: parameter and binning changes, tabulated-simulation feedback, AccuracyBoost, and $\chi^2$ against the DES Y3 cosmic-shear data (227 entries) with quadrature and interpolation checks.
EXAMPLE_EVALUATE2.ipynb	3×2pt: $C_\ell^{gs}$, $\gamma_t(\theta)$, $C_\ell^{gg}$ with and without the Limber approximation, and $w(\theta)$; tabulated-simulation feedback; the bfmt feedback study, which needs the bfmt installation; AccuracyBoost; and $\chi^2$ against the DES Y3 3×2pt data (533 entries) with quadrature and interpolation checks.
EXAMPLE_EVALUATE_COVARIANCE.ipynb	G, SSC, cNG, total, separate 1h–4h matter trispectra and matrix diagnostics, compared with the supplied DES Y3 covariance after the same mask.

Read EXAMPLE_EVALUATE1.ipynb first: it defines the vocabulary, the five stages of one evaluation and the wrappers that EXAMPLE_EVALUATE2.ipynb extends to the lens galaxies.

flowchart TD
  A["EXAMPLE_EVALUATE1.ipynb: cosmic shear, stages and wrappers"] --> B["EXAMPLE_EVALUATE2.ipynb: 3x2pt, Limber choices, bfmt study"]
  C["data/des_y3_real.dataset: DES Y3 data, covariance and mask"] --> A
  C --> B
  C --> D["EXAMPLE_EVALUATE_COVARIANCE.ipynb: G, SSC and cNG forecast"]
  E["bfmt block and its emulators"] --> B
  F["Optional covariance build"] --> D

Loading

Choose the Python kernel from the activated Cocoa environment and restart it after recompiling. The covariance guide explains the forecast files, figures and refinement workflow.

Computing covariances

The covariance tools compute Gaussian (G), super-sample (SSC), connected non-Gaussian (cNG) and total matrices before scale cuts. covariance/README.md explains compilation, the production CLI, exploratory notebooks, accuracy settings and output files.

Appendix

FAQ: Which accuracy settings are available?

Data-vector options belong to the selected likelihood block. Covariance options belong to the evaluate YAML's covariance block. They use separate names and settings; changing one does not refine the other.

Data-vector setting	What it changes
accuracyboost	Overall interpolation-table resolution.
integration_accuracy	Quadrature resolution; levels 5/6/7 allow 2048/4096/8192 points per Limber panel. Refine independently of interpolation.
internal_accuracyboost	C-FAST-PT convolution grid.
nonlimber_accuracyboost	Non-Limber distance sampling.
pk_z_refinement	Nested redshift refinement of matter-power inputs.
lmax	Real-space angular-transform cutoff, where a real-space transform is used.
kmax_boltzmann	Requested Boltzmann power range; coordinate it with the theory settings.

adopt_limber_gg and adopt_limber_gs choose a projection approximation. photoz_interpolation_type chooses how n(z) is interpolated, while photoz_zmid_convention describes the input coordinates. These are modeling or input-convention choices, not interchangeable accuracy boosts.

photoz_histogram_akima: 0 retains the original midpoint n(z) reconstruction; 1 selects histogram preparation followed by Akima for source and lens samples. photoz_histogram_akima_factor chooses 3 (default), 5 or 7 preparation cells per original uniform bin. See the reconstruction comparisons and area plots. The selected first factor never changes with an accuracy boost.

The default data-vector power grid has 1,500 wavenumbers at boost 1. Covariance alone uses power_accuracyboost: 8 to prepare 11,993 nodes by natural cubic interpolation before C linear lookup. Its accuracy_boost refines tables and cutoffs; its integration_accuracy independently selects quadrature levels 0–4. See the complete covariance accuracy table.

CAMB's theory.camb.extra_args.AccuracyBoost controls CAMB, not CosmoLike. Check interpolation, quadrature, input-power sampling and transform cutoffs separately at fixed cosmology and measurement bins. Narrow n(z) overlaps particularly require a quadrature check; increasing accuracyboost alone is not that check. The data-vector test guide and covariance test guide state what each suite actually verifies. A passing regression or a larger boost is not a general claim of survey or Fisher convergence.