Interior refactor: Aragog, Zalmoxis, atmodeller, dummy modules, testing overhaul, version pinning, oxygen accounting

[timlichtenberg]

This PR is not quite ready yet. I am currently preparing the full PR text and inline links, please (still) do not review yet. I will move it out of draft tomorrow (Monday 1 June) after a last pass.

Review timeline. I would like the review to take about a week. Please send feedback by Friday 5 June; I will incorporate comments and aim to merge by Monday 8 June.

Reviewers: @maraattia, @nichollsh, @rdc49, @planetmariana, @EmmaPostolec, @stuitje, @egpbos, @MarijnJ0, @Emeline0110. General comments are welcome, not just on the points I flag below.

How to review. This PR is too large to read line by line, and most of it is documentation and tests. Please do not try to read every diff. Instead, serve the docs locally (zensical serve, see docs/How-to/documentation.md) and follow a few of the tutorials end to end to confirm they work for you. For the implementation itself, please focus only on the specific places where I tag you in the text below; that is where I want eyes on the actual code.

Paired PR and merge order. This PR depends on FormingWorlds/CALLIOPE#20, the chemistry half of the oxygen accounting, which imports cleanly only once that lands. Merge order is CALLIOPE#20 first, then this PR.

Description

tl/interior-refactor has been in development for a few months. The change the branch is named for is the incorporation of two new Python interior backends, Aragog (thermal evolution) and Zalmoxis (interior structure), and the reorganisation of the interior into two clean axes around them. Built on top of that are a one-command automatic installer, central module version pinning, a new config format version, a documentation restructure, a test-framework overhaul, a full set of dummy module backends, atmodeller as a fully supported outgassing backend, and the #677 whole-planet oxygen accounting fix. The branch also merges in the ten PRs that landed on main in the meantime, so it stays current with main rather than diverging from it; that is catch-up, not new capability. Most of the change is the docs overhaul, the migrated config files, and the expanded test suite, not new physics.

Interior backends: Aragog and Zalmoxis

This is the change the branch is named for. The interior is two clean axes: interior_energetics/ for thermal evolution (spider, aragog, boundary, dummy) and interior_struct/ for radial structure (zalmoxis, dummy, plus a SPIDER-internal option). Energy bookkeeping uses frozen-mass conservation (E_residual_cons_J). Architecture overview: docs/Explanations/code_architecture.md.

Aragog (interior energetics, the default). Aragog is a pure-Python interior thermal-evolution model in the entropy formulation: the same magma-ocean physics as the compiled C SPIDER, reimplemented around an entropy solver. The wrapper in src/proteus/interior_energetics/aragog.py drives Aragog's EntropySolver over an entropy-based equation of state (EntropyEOS), with the mantle EOS tables materialised under data/spider_eos and cached by content fingerprint so repeated solves in one process reuse them. Aragog integrates the entropy evolution with a stiff ODE solver (SUNDIALS CVODE) and a JAX-assembled Jacobian, and derives the core density from the mesh. It is the default interior_energetics.module, with spider, the boundary backend, and dummy as the alternatives. The main practical gain is that Aragog needs no PETSc and no C build, so the common interior path is pure Python; SPIDER stays fully supported, and the two can be run at the same initial condition as a cross-implementation check. Aragog installs as an editable sibling checkout (aragog/) inside the PROTEUS root, with fwl-aragog pinned in [project] dependencies as a fallback. Config: [interior_energetics], see docs/Reference/config/interior.md.

Zalmoxis (interior structure, the default). Zalmoxis solves the planet's radial interior structure (density, pressure, gravity, radius) self-consistently. The wrapper in src/proteus/interior_struct/zalmoxis.py calls Zalmoxis' solver over its EOS tables and melting curves and iterates the density profile to convergence with a Picard scheme. To keep that iteration cheap across the many structure solves in a run, the wrapper seeds each solve from the previous converged density profile, keyed on planet identity (mass_tot, core_frac, mantle_mass_fraction) so a multi-planet driver never seeds one planet from another; the seed only accelerates convergence and never changes the converged answer. Zalmoxis is the default interior_struct.module; the alternative dummy structure uses the Noack & Lasbleis (2020) analytic scaling laws (calibrated for 0.8 to 2 M_Earth), and the energetics solver can either take the Zalmoxis structure as its external mesh or use an Adams-Williamson density profile. The interior radius and density profile Zalmoxis returns feed both the energetics solve and the dry-mass target that the volatile and oxygen accounting is computed against. Zalmoxis installs as an editable sibling checkout (Zalmoxis/), with fwl-zalmoxis pinned as a fallback. Config: [interior_struct], see docs/Reference/config/interior.md. @maraattia, please review the Zalmoxis implementation specifically: go through the [interior_struct] parameters in input/all_options.toml and tell me whether each one is understandable and useful as named and documented, or should be renamed, defaulted differently, or dropped.

Outgassing backend: atmodeller

atmodeller is a fully supported alternative to CALLIOPE for the outgassing and equilibrium-chemistry step, selectable with outgas.module = "atmodeller". It wraps the atmodeller package (Bower et al. 2025, ApJ 995:59), a JAX-based solver for thermodynamically consistent magma-ocean and atmosphere equilibrium with real-gas equations of state. Both outgassing backends honour the same planet.fO2_source modes, including the authoritative-O from_O_budget path, so the whole-planet oxygen accounting behaves identically whichever you select. Config lives under [outgas] (shared solver parameters) and [outgas.atmodeller]. @Emeline0110, please run the Earth analogue tutorial twice, once with outgas.module = "atmodeller" and once with outgas.module = "calliope", and tell me whether both work for you. See docs/Reference/config/escape_outgas.md.

Automatic install path

bash install.sh is a single-command installer for the whole ecosystem, in place of the manual sequence of clone, build, and pip-install steps. From a fresh conda environment it runs idempotently through pre-flight checks (OS, disk, Python 3.12, system libraries), Julia setup with a version pin (1.11), the FWL_DATA and RAD_DIR environment variables, the SOCRATES build, AGNI and FastChem, editable installs of every Python submodule and PROTEUS itself, reference-data downloads, and a final proteus doctor verification. It is safe to re-run: completed stages are skipped. Data scope is selectable (--all-data, --no-data, default essential set) and -i runs it interactively. For an environment where PROTEUS already imports, proteus install-all performs the same setup from the CLI (--export-env persists the environment variables); proteus doctor diagnoses an existing install (environment variables, submodule presence and editable git hashes, SOCRATES and AGNI, FWL_DATA); and proteus update-all refreshes the submodules, reference data, and PROTEUS itself in place. @egpbos and @nichollsh, please stress-test the installer, install-all, doctor, and update-all on a clean machine and on a cluster and flag anything that breaks. See docs/How-to/installation.md and docs/How-to/doctor.md.

Editable install layout

Aragog, Zalmoxis, and VULCAN install as editable sibling checkouts inside the PROTEUS root, matching AGNI / MORS / JANUS / CALLIOPE / ZEPHYRUS. The PyPI pins (fwl-aragog, fwl-zalmoxis, fwl-vulcan) stay as a fallback for pip install fwl-proteus without cloning. proteus doctor reports the editable git hash and dirty state next to the installed version, and a new CI gate verifies the editable copy takes precedence over the PyPI fallback on every PR. New setup scripts: tools/get_aragog.sh, tools/get_zalmoxis.sh. See docs/How-to/installation.md and docs/How-to/doctor.md.

Module version pinning

External module versions are pinned in one place and checked at runtime. pyproject.toml carries a [tool.proteus.modules] table that pins every non-PyPI module PROTEUS clones or builds (AGNI, SOCRATES, SPIDER, VULCAN, LovePy, PETSc) to a url and a ref (commit SHA, tag, or branch). The CI composite action and the tools/get_*.sh scripts both read this table (via tools/_module_pins.py), so bumping a module is a one-line ref edit and the same SHA always reproduces the same external state, which keeps branch CI deterministic. The Python submodules (fwl-aragog, fwl-zalmoxis, fwl-janus, fwl-calliope, and the rest) carry minimum-version pins in [project] dependencies. At startup, validate_module_versions (src/proteus/utils/coupler.py) compares the installed version of each active module against its required minimum, with a comparison that handles both semver and CalVer (year/month/day), and refuses to run on an out-of-date module. PROTEUS itself is versioned with setuptools-scm CalVer (version_scheme = "no-guess-dev"), matching the Aragog / Zalmoxis / CALLIOPE ecosystem convention. See docs/Reference/module_versions.md.

Config format and defaults

PROTEUS stamps a config_version on every config. The current format is 3.0 (CURRENT_CONFIG_VERSION in src/proteus/config/_config.py); it defaults to 3.0, and a config that explicitly declares a different version is rejected with an actionable error that points at input/all_options.toml. Version 3.0 covers the current layout: the interior split into [interior_energetics] and [interior_struct], the volatile inventory under [planet.elements], and the current section names. The shipped input/ configs and the test configs are all on 3.0.

Every config parameter has a default. The config dataclasses in src/proteus/config/ define a safe Earth-like or dummy value for each field, so a minimal config loads to a known state and only the fields you want to change need to appear (tests/config/test_defaults.py guards this). The per-parameter defaults are documented in the docs/Reference/config/ pages, and input/all_options.toml is the full enumeration of every option with recommended starting values. See docs/How-to/config.md.

Documentation restructure

The documentation follows a Diataxis layout: How-to guides (install, proteus doctor, configure, run, clusters, development), Tutorials (all-dummy quick start, Earth analogue, parameter-grid sweep, Solar System CHILI intercomparison), Explanations (model description, coupling loop, code architecture reflecting the interior two-axis split, dummy modules, test framework), and Reference (per-section config reference, melting curves, module versions, output format, API, and a new per-source Validation tree). The Validation pages anchor each physics source to a published benchmark or analytical limit and are wired into the nav. Installation, local-machine, and diagnose/update guides are included. The docs build with Zensical (zensical serve / zensical build). Entry points: docs/index.md, docs/getting_started.md.

Test framework and coverage

The branch brings PROTEUS onto the ecosystem test standard. Every test file carries a module-level tier marker (unit, smoke, integration, slow) with a wall-time budget; physics tests additionally carry physics_invariant (asserts conservation, positivity, boundedness, monotonicity, or symmetry) and reference_pinned (pins against a published benchmark, an analytical limit, or a SPIDER-versus-Aragog cross-check). Markers are strict (--strict-markers), and an AST linter (tools/check_test_quality.py) blocks single-assert tests, weak is not None checks, missing docstrings, and float == comparisons against a one-way baseline. CI runs a fast PR gate (unit + smoke on Linux and macOS on every push) and a nightly full gate (adds integration and slow tiers). Line coverage auto-ratchets toward the 90% ecosystem ceiling and is never manually decreased (tools/update_coverage_threshold.py). The suite mirrors src/proteus/ one-to-one, validated by tools/validate_test_structure.sh. See docs/How-to/testing.md and docs/Explanations/test_framework.md.

Dummy modules

Every module slot has a dummy backend: interior energetics and structure, atmosphere climate and chemistry, outgassing, orbit, star, and escape. Each replaces the full physics with a minimal parameterisation that captures the qualitative behaviour (a planet cools, volatiles outgas, the atmosphere radiates) without external solvers, compiled code, or reference data. They serve two purposes. First, testing: an all-dummy run exercises the entire coupling architecture (the helpfile data bus, timestep control, convergence checks, output pipeline) in well under a minute, which is what the fast unit tests and the quick-start tutorial use, and what keeps coupling bugs separable from solver bugs. Second, physics grounding: the dummy backends give analytical end-member behaviour that the production modules should reproduce in the appropriate limits, a zeroth-order sanity check on any full-solver result. They are not meant for quantitative science. See docs/Explanations/dummy_modules.md and the all-dummy quick-start tutorial.

Incorporated main PRs

Each incorporated PR keeps the original author's physics and tests. Please check that your code still behaves as you intended on this branch, and tell me if you have questions or spot anything off.

@nichollsh (seven PRs): #661 (get_socrates script), #659 (CHILI intercomparison data and scripts), #665 (aerosol support and chemistry plot rework), #669 (doctor command and SOCRATES version check), #671 (VULCAN as a Python package), #673 (environment and install simplification), and #675 (BayesOpt rewrite, AGNI grey-gas RT, timestep-overshoot fix, prevent_warming boundary-layer fix, config tolerance migration). #675 is staged into eight atomic commits so each sub-feature is independently verifiable. Two things to confirm: (a) in #665 the aerosol schema is present but the mass mixing ratio is pinned at 0.0 with no source term plumbed, so aerosols are radiatively inert even with aerosols_enabled=True; is that the intended interim state? (b) in #659 the xfail marker describes a cattrs silent-drop, but that case currently passes; can you confirm the original intent? I also deliberately left out a few items from #675 (the dt.adaptive / dt.proportional nested-class refactor, the parent-to-child schema move for p_top / p_obs / spectral_* / num_levels, and several renames and removals that postdate the fork) because they would either rewrite every [params.dt] block or silently revert load-bearing features; happy to walk through that.

@planetmariana: #658 (library of solidus/liquidus parameterizations, selected via melting_dir). Please confirm the parameterization selection behaves as you expect under the new interior split.

@rdc49: #668 (boundary interior module, the fourth energetics backend). It implements the Schaefer et al. (2016) parameterised-convection magma-ocean model: the Rayleigh-Nusselt mantle flux q_m, the mantle potential-temperature evolution (their Eq. 15), and the surface energy balance 4 pi R^2 (q_m - F_atm) (their Eq. 20), all of which match the published equations. One small reporting point to confirm: the helpfile F_int column is set to F_atm, while the convective flux q_m that drives the evolution is written to the backend's own log; is that the reporting you want, or should F_int carry q_m? Otherwise, please confirm it runs as intended on this branch.

(#662, FastChem install docs, is mine.)

Whole-planet oxygen accounting (closes #677)

#677 reported M_atm > M_planet for volatile-rich cases (high H_ppmw), with the summed volatile inventory also inconsistent with the bulk mass.

Oxygen is a tracked element in PROTEUS-side accounting alongside H/C/N/S. The chemistry step is unchanged (CALLIOPE and atmodeller equilibrate against the fO2 buffer), and the atmospheric and dissolved O mass they produce is counted in M_ele, subtracted from the Zalmoxis dry-mass target, and included in the proportional escape distribution. A planet.elements.O_mode field selects how the O budget is set: "ppmw", "kg", "FeO_mantle_wt_pct" (a petrology-friendly unit, sets the volatile O budget only, does not change the PALEOS EOS density), or "ic_chemistry", which defers the IC budget to CALLIOPE's first equilibrium and is the default, so a config that says nothing about oxygen keeps the buffered-mode behaviour. tools/migrate_oxygen_mode.py writes an explicit O_mode into the [planet.elements] blocks under input/ and tests/. A runtime invariant enforces M_atm <= M_planet, and an IC consistency check hard-fails when the user O budget diverges from CALLIOPE's equilibrium value by more than 50%. The chemistry half lives in FormingWorlds/CALLIOPE#20. Config reference: docs/Reference/config/planet.md.

One caveat worth a comment. FeO_mantle_wt_pct sets the volatile O budget only; it does not change the mantle EOS density, since PALEOS still uses its built-in FeO content. That makes it a leaky unit for now, and I would like a view on whether to keep it that way or make it strict once PALEOS density responds to user-set mantle composition.

Please try this once in an extreme volatile-rich regime (water-dominated, high H budget): @IKisvardai, @nichollsh, @EmmaPostolec, @planetmariana. And @IKisvardai, @EmmaPostolec, @planetmariana: please send me one representative TOML from your current work, so I can convert your v2 configs to the v3 layout and run the new accounting on your own planet scenarios.

Radial fO2 via Fe3+/Fe2+ tracking (#653)

@planetmariana, the interface for your radial ferric/ferrous framework is already scaffolded. planet.fO2_source is an enum: "user_constant" (the fO2-buffered source, the default), "from_O_budget" (authoritative O, fO2 derived), and a reserved "from_mantle_redox" member that currently raises a clear "reserved for issue #653, not yet wired into the runtime" error (src/proteus/config/_planet.py). To wire in your branch you add the from_mantle_redox runtime path to the outgas dispatch in src/proteus/outgas/calliope.py (and atmodeller.py), keyed on config.planet.fO2_source, alongside the two existing source branches. Because O is tracked end-to-end, the bookkeeping prerequisite for a self-consistent Fe-derived fO2 is in place. Could you check that this hook works for your approach in principle?

Bug fixes

• Planet-satellite angular-momentum prefactor (src/proteus/orbit/satellite.py): Ltot uses the satellite mass in the orbital square-root, following Korenaga 2023 (Icarus 400, Eq. 60); the planet-mass form would inflate L by M_planet / M_satellite (~81 for Earth-Moon) and corrupt the dω/dt and da/dt evolution. The Earth-Moon value is pinned against Korenaga 2023. @MarijnJ0, this is your module; please confirm the form is right. See docs/Validation/orbit/satellite.md.
• Interior resume: a resumed run restores the evolved entropy and anchors T_magma, so the mantle skin layer stays consistent across the resume boundary; residual ~0.9 K.
• Zalmoxis structure caching: the density seed and table cache are keyed per planet, so a multi-planet driver never seeds one planet's structure solve from another's.
• Interior tolerance alias resolution and assorted atmosphere, config, escape, and mass-accounting edge cases carry explicit guards.

Validation of changes

Unit and smoke tests green on Linux and macOS (Python 3.12) on every push. The full nightly (unit, integration, and slow tiers, including the zalmoxis-coupled structure run on both platforms) is green on the current head, with combined coverage above the 90% ecosystem target. ruff check and ruff format --check clean. End-to-end check of the #677 fix at H_ppmw = 2e5, Earth mass, fO2 = +4: M_planet = mass_tot * M_earth exactly, M_atm / M_planet = 0.7737, mass-conservation invariant holds.

All four tutorials run end to end, in particular the Solar System CHILI intercomparison, which exercises the full interior, structure, outgassing, and atmosphere coupling across the terrestrial planets. Beyond the tutorials, the branch has been run on a range of super-Earth configurations. It has not been stress-tested across every possible configuration combination.

Issues closed

Closing keywords auto-close the PROTEUS issues on merge. GitHub does not auto-close across repositories, so I will close the two Zalmoxis issues by hand once this PR merges.

• Closes Volatile masses and 'M_atm' is larger than 'M_planet' for volatile rich cases #677 (M_atm > M_planet for volatile-rich cases): oxygen is a tracked element in PROTEUS-side mass accounting, so M_planet includes atmospheric and dissolved O and a runtime invariant enforces M_atm <= M_planet. See the oxygen-accounting section above.
• Closes Online M-R structure calculation using Zalmoxis #390 (online M-R structure with Zalmoxis): Zalmoxis is wired in as the online interior-structure backend and is the default interior_struct.module, solving the radial structure at runtime.
• Closes Use structure calculation (M-R) to derive mantle mass or radius #50 (derive mantle mass or radius from the M-R structure calc): Zalmoxis runs an outer mass-radius solver (Newton with brentq bracketing) that derives the interior radius and density profile from total mass, core fraction, and the EOS; with core_frac_mode = "mass" the core/mantle mass split follows self-consistently from the structure and the EOS.
• Closes Calculate volumetric melt fraction properly when using Aragog #537 (volumetric melt fraction with Aragog): the Aragog path records the solver's volumetric melt fraction (Phi_global_vol) directly into the helpfile, so the column carries the true volumetric melt fraction.
• Closes Total flux in Aragog is not equal to user-provided boundary condition #499 (Aragog total flux vs the F_atm boundary condition): the Aragog surface boundary condition runs in prescribed-flux mode (outer BC type 4) driven by F_atm, refreshed each coupling iteration, so the integrated surface flux equals the atmosphere's F_atm. Per-layer fluxes still differ from the surface value, as they physically should.
• Closes Fix codecov nightly/PR-check coverage mislabeling #651 (codecov nightly/PR coverage mislabeling): codecov.yml carries separate unit-tests and nightly flags with carryforward: true, so a fast-suite PR upload is compared on equal footing with a base that also carries nightly coverage. Both codecov checks pass on this PR.
• Closes Adapt PROTEUS to perform consistent interior structure calculations throughout thermal evolution with Zalmoxis Zalmoxis#47 (consistent structure through thermal evolution): when interior_struct.zalmoxis.update_interval > 0, PROTEUS recomputes the Zalmoxis structure as the evolution proceeds, through update_structure_from_interior, gated by elapsed time and changes in T_magma and melt fraction.
• Closes all_options.toml test run fails inside Zalmoxis Zalmoxis#57 (all_options.toml crashes inside Zalmoxis): input/all_options.toml uses the built-in PALEOS EOS, so a fresh install needs no external RTPress100TPa tables, and the file runs as a standard integration test (tests/integration/test_integration_std_config.py) that is green in the nightly.

Related

• Radially resolved evolution of fO2 through ferric/ferrous iron tracking #653 (radial fO2 via Fe3+/Fe2+; interface described above).

Next up

The next major effort is a shared input/output layer for the ecosystem (working name fwl-io): a single library for reading and writing PROTEUS data products against a central data manifest, so I/O, reference-data resolution, and provenance live in one place across all modules. I am deliberately not addressing that here. This PR lays the basis for it: the helpfile data bus, the central module-pin manifest in pyproject.toml, the config-version stamp, and the editable-install layout are the pieces fwl-io will build on. It comes next, as its own PR, starting with #605 (universal downloader utilities).

This also needs to land on main and be stress-tested against the wider set of configurations the group runs. I expect some issues to surface in configs not covered here, and I will work through them as fast as I can over the next few weeks.

Escape modeling needs further development, and that is in progress: a BSc group project and @EmmaPostolec's PhD thesis are tackling it more closely, including a more physical, element-fractionated treatment.

Feature requests in this review are welcome. I will weigh each against scope: some can fold into this PR, others I will defer to a follow-up so this one can land on the timeline above.

Checklist

• I have followed the contributing guidelines
• My code follows the style guidelines of this project
• I have performed a self-review of my code
• My changes generate no new warnings or errors
• I have checked that the tests still pass on my computer
• I have updated the docs, as appropriate
• I have added tests for these changes, as appropriate
• I have checked that all dependencies have been updated, as required

[nichollsh]

Exciting! Looking forward to reviewing this PR when it's ready :)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.09%. Comparing base (534eb28) to head (3a7066a).

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #678       +/-   ##
===========================================
+ Coverage   70.56%   90.09%   +19.53%     
===========================================
  Files         100      108        +8     
  Lines       13675    16511     +2836     
  Branches     2241     3006      +765     
===========================================
+ Hits         9650    14876     +5226     
+ Misses       3875     1635     -2240     
+ Partials      150        0      -150     

Flag	Coverage Δ
nightly	90.83% <ø> (+56.45%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.