EOS26

Description

All scripts to run the EOS26 simulation with 21cmFASTv4.2. EOS26 is run phase by phase i.e. ICs first, then PFs, then PHFs, and finally coevals.

How to run

All scripts accept an optional --test flag to run a small test box (HII_DIM=200).

Comparison workflow

• Generate the reference once with --test and no --compare. It is written to EOS26_test_HIIDIM200/.
• Run the production workflow with --compare to check each output against that reference.
• For a local workflow check, use both flags: --test --compare. Candidate files then go to EOS26_test_HIIDIM200_compare/, so the reference is never overwritten. Both test modes use the same input parameters and random seed (42).

End-to-end test

• sbatch sbatch_scripts/full_test_job.sh — runs all three pipeline steps sequentially in test mode.
• qsub pbs_scripts/full_test_job.sh — PBS equivalent of the end-to-end test.

Scaling Measurements

• Run bash scaling/scaling_job.sh to measure the default HII_DIM=200 and HII_DIM=300 cases. Each run disables garbage collection, records peak process RSS with psutil, and amortizes the full node-redshift coeval evolution over its outputs, then writes a phase-by-phase JSON record under scaling/results/. To benchmark other dimensions, pass them as arguments, for example bash scaling/scaling_job.sh 200 300 400.

• After at least two measurements, scaling/run_scalingrelation.py fits power laws for time, peak memory, phase storage, and every stored HDF5 structure. It writes plots, README_scaling_values.md, and scaling_fits.json to scaling/reports/. The generated Markdown file is the source for updating the scaling table below. To apply direct measurements from the smallest and largest completed HII_DIM runs plus fitted extrapolation intervals from every available scaling result, run uv run --no-sync scaling/run_scalingrelation.py --update-readme; it relabels the two measured columns and updates both EOS extrapolation sections.

Production run (in order)

1. Initial conditions (ICs):
   • sbatch sbatch_scripts/ICs_job.sh [--test]

• qsub pbs_scripts/ICs_job.sh [--test] on PBS.
• Writes EOS26.toml (full template with embedded node redshifts and random seed) and EOS26_minimal.toml, then runs run_scripts/run_ICs.py.

2. Perturbed fields (PFs):
   • bash sbatch_scripts/submit_PF_jobs.sh [--test] — submits one job per PF (indices 0–91) (not used for production).
   • sbatch sbatch_scripts/N_PF_job.sh <z_idx> [N] [--test] — runs a batch of N PFs (default N=10) starting from redshift index z_idx.

• bash pbs_scripts/submit_PF_jobs.sh [--test] and qsub pbs_scripts/N_PF_job.sh <z_idx> [N] [--test] are the PBS equivalents.

3. Perturbed halo fields (PHFs):
   • sbatch sbatch_scripts/PHFs_job.sh [--test] — runs run_scripts/run_PHFs.py.

• qsub pbs_scripts/PHFs_job.sh [--test] on PBS.

4. Coevals:
   • sbatch sbatch_scripts/N_coeval_job.sh [N] [--test] — runs a batch of N coevals (default N=10) with run_scripts/run_N_coevals.py.

• qsub pbs_scripts/N_coeval_job.sh [N] [--test] on PBS.

Table

EOS25 simulation step	Computation time [hrs]		Memory [Tb]		Storage [Tb]		SUs
	Estimated	Actual	Estimated	Actual	Estimated	Actual	Estimated	Actual
Initial conditions	13.5 + 2.75 for writing to disk		1.1	1.3	652 Gb	747 Gb		864 EM for ICs + PFs
One perturbed field	0.6		25 Gb	26 Gb	25Gb x 92 = 2.3Tb	2.4Tb
Perturbed halo fields	22 hrs		0.71	0.77	~330 G	33 G	720 EM	617 EM
Evolving astrophysics for one coeval	7		3.65		2.6Tb x 92 = 240 Tb 0.215 x 92 = 20Tb without XRS		672 EM x 92 = 62k

Scaling test results

When updated with --update-readme, each extrapolated value is the mean ± 1σ from a regression fit to all available scaling points: peak RSS uses an affine model (overhead + coefficient * HII_DIM^3, physically motivated by fixed per-process overhead plus volume-scaling grid buffers) fit by ordinary least squares, while time and storage use a free-exponent power-law fit in log-log space. In both cases the regression's own confidence interval on the mean is combined in quadrature with a fixed 10% relative per-point measurement-uncertainty floor. EOS-1: HII_DIM = 1400 (1.5 cMpc/cell, 2100 Mpc). EOS-2: HII_DIM = 1200 (1.667 cMpc/cell, 2000 Mpc). Storage for PFs and coevals is the total across all 92 node redshifts; coeval storage excludes IonizedBox.

EOS26 simulation step	Computation time [hrs]		Memory		Storage
	N=100	N=300	N=100	N=300	N=100	N=300
Scaling tests (measured)
Initial conditions	0.00349	0.045	1.58 GB	12.3 GB	0.464 GB	12.5 GB
One perturbed field	0.00252	0.00311	0.425 GB	4.91 GB	0.016 GB	0.432 GB
Perturbed halo fields	0.00693	0.0437	0.786 GB	13.6 GB	0.207 GB	5.49 GB
Evolving astrophysics for one coeval	0.000529	0.0214	1.4 GB	16.5 GB	0.0441 GB x 92 = 4.06 GB	1.19 GB x 92 = 109 GB
Extrapolated to EOS-1 (HII_DIM = 1400, 1.5 cMpc/cell, 2100 Mpc)
Initial conditions	1.24 ± 1.28		1.06 ± 0.317 TB		1.27 ± 0.122 TB
One perturbed field	0.00409 ± 0.000474		0.473 ± 0.0473 TB		0.0438 ± 0.00418 TB
Perturbed halo fields	0.467 ± 0.356		1.34 ± 0.142 TB		0.542 ± 0.0522 TB
Evolving astrophysics for one coeval	3.18 ± 2.15		1.49 ± 0.427 TB		0.12 ± 0.0115 TB x 92 = 11.1 ± 1.06 TB
Extrapolated to EOS-2 (HII_DIM = 1200, 1.667 cMpc/cell, 2000 Mpc)
Initial conditions	0.875 ± 0.824		0.668 ± 0.199 TB		0.802 ± 0.0765 TB
One perturbed field	0.00398 ± 0.00045		0.298 ± 0.0298 TB		0.0276 ± 0.00263 TB
Perturbed halo fields	0.363 ± 0.255		0.846 ± 0.0894 TB		0.342 ± 0.0329 TB
Evolving astrophysics for one coeval	1.91 ± 1.19		0.941 ± 0.268 TB		0.0758 ± 0.00724 TB x 92 = 6.97 ± 0.666 TB

Projected maximum simulation size

Evolving astrophysics for one coeval is the most memory-intensive phase at every measured HII_DIM (see table above), so it sets the ceiling for the largest simulation a single node can run. Peak memory for 3D simulation grids is physically fixed per-process overhead (interpreter, shared libraries, lookup tables) plus a term that scales with box volume (HII_DIM^3), so peak RSS is fit as overhead + coefficient * HII_DIM^3 by ordinary least squares using all measured points (HII_DIM = 100, 200, 300), rather than a free-exponent log-log power law, which is biased low by that same fixed overhead when it is not negligible at small HII_DIM (the same effect documented above for compute time). Resolution is fixed at 1.5 cMpc/cell, matching all current scaling points; only HII_DIM is varied. The "Max HII_DIM" columns require the upper end of the fit's 1-sigma band to stay within the memory budget, so the true peak should stay at or below the stated value.

Memory budget	Max HII_DIM	Box length [Mpc]	Predicted peak RSS (coeval)
3.0 TB	1624	2436	2.33 ± 0.668 TB
2.7 TB (90% safety margin)	1568	2352	2.1 ± 0.601 TB

Predicted coeval peak RSS at the current EOS target sizes (affine overhead + coefficient * HII_DIM^3 fit on all measured points):

EOS target	HII_DIM	Predicted peak RSS (coeval)
EOS-1	1400	1.49 ± 0.427 TB
EOS-2	1200	0.941 ± 0.268 TB

With only 3 measured points, the affine fit above is pulled noticeably off the data by whichever point dominates the linear-space sum of squares. A free-exponent power-law fit (the same form used for time and storage; see fit_power_law) tracks the coeval measurements markedly better, so it is shown here for comparison -- the affine model above remains the one used for the Max HII_DIM budget table:

EOS target	HII_DIM	Predicted peak RSS, affine	Predicted peak RSS, power law
EOS-1	1400	1.49 ± 0.427 TB	0.64 ± 0.453 TB
EOS-2	1200	0.941 ± 0.268 TB	0.45 ± 0.293 TB

Caveats: this projection extrapolates well beyond the largest measured HII_DIM (300) to production scale, so treat it as an order-of-magnitude estimate rather than a precise bound. Run an additional scaling point at HII_DIM = 400 (or larger) on the cluster and check whether the residuals from the affine fit stay small -- growing residuals at larger HII_DIM would mean memory grows faster than the assumed overhead-plus-cubic form and this projection is optimistic. Consider also profiling one run with memray to see the full memory-vs-redshift curve and confirm the RSS sampler is not missing any brief spikes.