From 9d7b6bf9e572541ac6353ae05cf99c661e0c39fa Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 29 Apr 2026 17:57:39 +0000 Subject: [PATCH 01/11] Initial plan From ec76ad72a1385e7f88fca918353678e0c92bce3d Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 29 Apr 2026 18:09:27 +0000 Subject: [PATCH 02/11] docs: remove model-specific references from documentation - Replace CESM-specific active component namelist section with generic guidance - Generalize CESM-ECT documentation to generic ECT documentation - Remove CESM-specific URLs to namelist documentation and verification website - Remove model-specific machine references (cheyenne) from examples - Generalize CESM-specific paths in examples to use placeholders - Add notes to model-specific example outputs for context - Keep model references where they provide useful context (e.g. listing CIME-driven models, historical conventions, concrete examples) Agent-Logs-Url: https://github.com/ESMCI/cime/sessions/2ef12468-32a6-42e3-a377-f05a2a354d5c Co-authored-by: jasonb5 <2191036+jasonb5@users.noreply.github.com> --- doc/.gitignore | 1 + doc/source/ccs/index.rst | 2 +- .../support-a-new-machine.rst | 25 +++--- .../variables/compsets.rst | 2 +- .../model-configuration/variables/grids.rst | 8 +- .../ccs/model-configuration/variables/pes.rst | 2 +- doc/source/ccs/running-a-case.rst | 8 +- doc/source/ccs/setting-up-a-case.rst | 82 +++---------------- doc/source/ccs/timers.rst | 4 + doc/source/ccs/troubleshooting.rst | 2 +- doc/source/contributing-guide.rst | 12 +-- doc/source/system_testing.rst | 12 +-- doc/source/tools/ect.rst | 47 +++++------ 13 files changed, 72 insertions(+), 135 deletions(-) diff --git a/doc/.gitignore b/doc/.gitignore index e8797ee56dc..009affbc7a0 100644 --- a/doc/.gitignore +++ b/doc/.gitignore @@ -4,3 +4,4 @@ source/Tools_api source/CIME_api source/_autosummary source/generated +build/ diff --git a/doc/source/ccs/index.rst b/doc/source/ccs/index.rst index e22d59f2221..6bc2e6fe457 100644 --- a/doc/source/ccs/index.rst +++ b/doc/source/ccs/index.rst @@ -18,7 +18,7 @@ The first thing to do is clone the model repository into ``$SRCROOT``. All examples will be run from ``$CIMEROOT`` which is should exist under ``$SRCROOT`` e.g. (``$CIMEROOT`` would be ``$SRCROOT/cime``). -Next set the ``CIME_MODEL`` evnironment variable for your model, e.g. ``export CIME_MODEL=e3sm``. +Next set the ``CIME_MODEL`` environment variable for your model, e.g. ``export CIME_MODEL=``. .. note:: diff --git a/doc/source/ccs/model-configuration/support-a-new-machine.rst b/doc/source/ccs/model-configuration/support-a-new-machine.rst index 4b0b6952d29..ae0d80ebd16 100644 --- a/doc/source/ccs/model-configuration/support-a-new-machine.rst +++ b/doc/source/ccs/model-configuration/support-a-new-machine.rst @@ -68,10 +68,10 @@ As an example, on a MAC with 2 cores that has mpich with gnu fortran you would i > mpif90 fhello_world_mpi.F90 -o hello_world > mpirun -np 2 ./hello_world -CESM Linux and Mac Support +Linux and Mac Support --------------------------- -The distribution of CESM includes machines called **homebrew** and **centos7-linux** in the file **$CIMEROOT/config/cesm/machines/config_machines.xml**. +Your model distribution may include generic machine definitions (e.g., **homebrew** and **centos7-linux**) in the file **$CIMEROOT/config/$model/machines/config_machines.xml**. Please see the instructions in the file to create the directory structure and use these generic machine definitions. Steps for porting @@ -85,7 +85,7 @@ Porting CIME involves several steps. The first step is to define your machine. Y In particular, you can create a **$HOME/.cime/config_machines.xml** file with the definition for your machine. A template to create this definition is provided in **$CIMEROOT/config/xml_schemas/config_machines_template.xml**. More details are provided in the template file. In addition, if you have a batch system, you will also need to add a **config_batch.xml** file to your **$HOME/.cime** directory. - All files in **$HOME/.cime/** are appended to the xml objects that are read into memory from the **$CIME/config/$model**, where **$model** is either ``e3sm`` or ``cesm``. + All files in **$HOME/.cime/** are appended to the xml objects that are read into memory from the **$CIME/config/$model** directory. .. note:: If you use method (2), you can download CIME updates without affecting your machine definitions in **$HOME/.cime**. @@ -114,7 +114,7 @@ In what follows we outline the process for method (2) above: After running those steps correctly, you are ready to try a case at your target compset and resolution. -Validating a CESM port with prognostic components +Validating a port with prognostic components ------------------------------------------------- The following port validation is recommended for any new machine. @@ -133,15 +133,15 @@ possible. Users are responsible for their own validation process, especially with respect to science validation. -These are the recommended steps for validating a port for the CESM model: +These are the recommended steps for validating a port: -1. Verify basic functionality of your port by performing the cheyenne "prealpha" tests on your machine. This can be done by issuing the following command: +1. Verify basic functionality of your port by performing the "prealpha" tests on your machine. This can be done by issuing the following command: :: - ./create_test --xml-category prealpha --xml-machine cheyenne --xml-compiler intel --machine --compiler + ./create_test --xml-category prealpha --xml-machine --xml-compiler --machine --compiler - This command will run the prealpha tests *defined* for cheyenne with the intel compiler, but will run them on *your* machine with *your* compiler. + This command will run the prealpha tests *defined* for the reference machine with the reference compiler, but will run them on *your* machine with *your* compiler. These tests will be run in the **$CIME_OUTPUT_ROOT**. To see the results of tests, you need to do the following: :: @@ -153,11 +153,11 @@ These are the recommended steps for validating a port for the CESM model: 2. Carry out ensemble consistency tests: This is described in ``$CIMEROOT/tools/statistical_ensemble_test/README``. - The CESM-ECT (CESM Ensemble Consistency Test) determines whether a new simulation set up (new machine, compiler, etc.) is statistically distinguishable from an accepted ensemble. - The ECT process involves comparing several runs (3) generated with the new scenario to an ensemble built on a trusted machine (currently cheyenne). + The Ensemble Consistency Test (ECT) determines whether a new simulation set up (new machine, compiler, etc.) is statistically distinguishable from an accepted ensemble. + The ECT process involves comparing several runs (3) generated with the new scenario to an ensemble built on a trusted machine. The python ECT tools are located in the pyCECT subdirectory ``$CIMEROOT/tools/statistical_ensemble_test/pyCECT``. - The verification tools in the CESM-ECT suite are: + The verification tools in the ECT suite are: ``CAM-ECT``: detects issues in CAM and CLM (12 month runs) @@ -166,9 +166,8 @@ These are the recommended steps for validating a port for the CESM model: ``POP-ECT``: detects issues in POP and CICE (12 month runs) Follow the instructions in the **README** file to generate three ensemble runs for any of the above tests that are most relevant to your port. - Then please go to the `CESM2 ensemble verification website `_, where you can upload your files and subsequently obtain a quick response as to the success or failure of your verification. -Performance tuning of a CESM port +Performance tuning of a port ------------------------------------------------- Once you have performed the verification that your port is successful, diff --git a/doc/source/ccs/model-configuration/variables/compsets.rst b/doc/source/ccs/model-configuration/variables/compsets.rst index 13d5c82aa3d..5719c9de0c4 100644 --- a/doc/source/ccs/model-configuration/variables/compsets.rst +++ b/doc/source/ccs/model-configuration/variables/compsets.rst @@ -48,7 +48,7 @@ Every file listed in ``COMPSETS_SPEC_FILE`` will be searched to compile possible CIME will note which component's config_compsets.xml had the matching compset name and that component will be treated as the **primary component** As an example, the primary component for a compset that has a prognostic atmosphere, -land and cice (in prescribed mode) and a data ocean is the atmosphere component (for cesm this is CAM) because the compset +land and cice (in prescribed mode) and a data ocean is the atmosphere component (e.g. CAM in CESM) because the compset is defined, using the above example, in ``$SRCROOT/components/cam/cime_config/config_compsets.xml`` In a compset where all components are prognostic, the primary component will be **allactive**. diff --git a/doc/source/ccs/model-configuration/variables/grids.rst b/doc/source/ccs/model-configuration/variables/grids.rst index b35c0cc409a..6bee7517cd4 100644 --- a/doc/source/ccs/model-configuration/variables/grids.rst +++ b/doc/source/ccs/model-configuration/variables/grids.rst @@ -387,14 +387,14 @@ The steps for adding a new component grid to the model system follow. This proce At this time, if you are running with a new ocean or runoff grid, please contact Michael Levy (mlevy_AT_ucar_DOT_edu) for assistance. If you are running with standard ocean and runoff grids, the mapping file should already exist and you do not need to generate it. -6. CESM specific: If you are adding a new atmosphere grid, this means you are also generating a new land grid, and you will need to create a new CLM surface dataset. (Otherwise you can skip this step). - You need to first generate mapping files for CLM surface dataset (since this is a non-standard grid). +6. If you are adding a new atmosphere grid, this means you are also generating a new land grid, and you may need to create a new land surface dataset. (Otherwise you can skip this step). + You need to first generate mapping files for the land surface dataset (since this is a non-standard grid). :: > cd $CIMEROOT/../components/clm/tools/mkmapdata > ./mkmapdata.sh --gridfile --res --gridtype global - These mapping files are then used to generate CLM surface dataset. Below is an example for a current day surface dataset (model year 2000). + These mapping files are then used to generate the land surface dataset. Below is an example for a current day surface dataset (model year 2000). :: @@ -403,7 +403,7 @@ The steps for adding a new component grid to the model system follow. This proce 7. Create grid file needed for create_newcase. The next step is to add the necessary new entries in the appropriate ``config_grids.xml`` file. - You will need to modify ``$CIMEROOT/config/cesm/config_grids.xml`` or ``$CIMEROOT/config/e3sm/config_grids.xml`` depending on the value of ``$CIME_MODEL``. + You will need to modify ``$CIMEROOT/config/$model/config_grids.xml`` depending on the value of ``$CIME_MODEL``. You will need to: - add a single ```` entry diff --git a/doc/source/ccs/model-configuration/variables/pes.rst b/doc/source/ccs/model-configuration/variables/pes.rst index e8be806b939..5f4bf26722b 100644 --- a/doc/source/ccs/model-configuration/variables/pes.rst +++ b/doc/source/ccs/model-configuration/variables/pes.rst @@ -436,7 +436,7 @@ help determine the optimal load balance. Changing the pe layout of the model has NO IMPACT on the scientific results. The basic order of operations and calling sequence are hardwired into the driver and do not change with the pe -layout. However, both CESM and E3SM do impose some contraints in the +layout. However, models such as CESM and E3SM may impose some constraints in the tempororal evolution of the components. For example, the prognostic atmosphere model always run sequentially with the ice and land models for scientific reasons. As a result, running the atmosphere diff --git a/doc/source/ccs/running-a-case.rst b/doc/source/ccs/running-a-case.rst index 04c8a709308..19406d868fa 100644 --- a/doc/source/ccs/running-a-case.rst +++ b/doc/source/ccs/running-a-case.rst @@ -19,6 +19,10 @@ Before submitting a case, it is a good idea to preview the run to ensure that th Example output: +.. note:: + + The following is an example output. The specific paths, environment variables, and commands will vary depending on your model and machine configuration. + .. code-block:: bash CASE INFO: @@ -500,7 +504,7 @@ or a hybrid run or to back up to a previous restart date. Long-term Archiving ``````````````````` Users may choose to follow their institution's preferred method for long-term -archiving of model output. Previous releases of CESM provided an external +archiving of model output. Previous releases of some CIME-driven models provided an external long-term archiver tool that supported mass tape storage and HPSS systems. However, with the industry migration away from tape archives, it is no longer feasible for CIME to support all the possible archival schemes available. @@ -516,7 +520,7 @@ Scripts ``````` Variables ``PRERUN_SCRIPT`` and ``POSTRUN_SCRIPT`` can each be used to name a script which should be executed immediately prior starting or -following completion of the CESM executable within the batch +following completion of the model executable within the batch environment. The script is expected to be found in the case directory and will receive one argument which is the full path to that directory. If the script is written in python and contains a diff --git a/doc/source/ccs/setting-up-a-case.rst b/doc/source/ccs/setting-up-a-case.rst index b67f6fffb6e..47cdeeba9f7 100644 --- a/doc/source/ccs/setting-up-a-case.rst +++ b/doc/source/ccs/setting-up-a-case.rst @@ -238,85 +238,23 @@ The namelist file for DROF is ``drof_in`` (or ``drof_in_NNN`` for multiple insta ``chmod 644 user_drof.streams.txt[extension`` 3. Edit the **user_drof.streams.txt.*** file. -.. TODO:: remove cesm specific docs +Customizing active component-specific namelist settings +------------------------------------------------------- -Customizing CESM active component-specific namelist settings ------------------------------------------------------------- +Active components typically provide a ``buildnml`` script in their ``cime_config`` directory that generates the component's namelist variables. Component-specific CIME xml variables are set in the component's ``config_component.xml`` file and are used by the ``buildnml`` script to generate the namelist. -CAM -``` +To modify an active component's namelist settings, add the appropriate keyword/value pair at the end of the **$CASEROOT/user_nl_** file, where ```` is the component name. See the documentation at the top of each ``user_nl`` file for details. -CIME calls **$SRCROOT/components/cam/cime_config/buildnml** to generate the CAM's namelist variables. - -CAM-specific CIME xml variables are set in **$SRCROOT/components/cam/cime_config/config_component.xml** and are used by CAM's **buildnml** script to generate the namelist. - -For complete documentation of namelist settings, see `CAM namelist variables `_. - -To modify CAM namelist settings, add the appropriate keyword/value pair at the end of the **$CASEROOT/user_nl_cam** file. (See the documentation for each file at the top of that file.) - -For example, to change the solar constant to 1363.27, modify **user_nl_cam** file to contain the following line at the end: +For example, to modify a namelist variable for an atmosphere component, edit the corresponding ``user_nl`` file: :: - solar_const=1363.27 - -To see the result, call ``preview_namelists`` and verify that the new value appears in **CaseDocs/atm_in**. - -CLM -``` - -CIME calls **$SRCROOT/components/clm/cime_config/buildnml** to generate the CLM namelist variables. - -CLM-specific CIME xml variables are set in **$SRCROOT/components/clm/cime_config/config_component.xml** and are used by CLM's **buildnml** script to generate the namelist. - -For complete documentation of namelist settings, see `CLM namelist variables `_. - -To modify CLM namelist settings, add the appropriate keyword/value pair at the end of the **$CASEROOT/user_nl_clm** file. - -To see the result, call ``preview_namelists`` and verify that the changes appear correctly in **CaseDocs/lnd_in**. - -MOSART -`````` - -CIME calls **$SRCROOT/components/mosart/cime_config/buildnml** to generate the MOSART namelist variables. + = -To modify MOSART namelist settings, add the appropriate keyword/value pair at the end of the **$CASEROOT/user_nl_rtm** file. +To see the result, call ``preview_namelists`` and verify that the new value appears in the appropriate file under **CaseDocs/**. -To see the result of your change, call ``preview_namelists`` and verify that the changes appear correctly in **CaseDocs/rof_in**. - -CICE -```` - -CIME calls **$SRCROOT/components/cice/cime_config/buildnml** to generate the CICE namelist variables. - -For complete documentation of namelist settings, see `CICE namelist variables `_. - -To modify CICE namelist settings, add the appropriate keyword/value pair at the end of the **$CASEROOT/user_nl_cice** file. -(See the documentation for each file at the top of that file.) -To see the result of your change, call ``preview_namelists`` and verify that the changes appear correctly in **CaseDocs/ice_in**. - -In addition, ``case.setup`` creates CICE's compile time `block decomposition variables `_ in **env_build.xml**. - -POP2 -```` - -CIME calls **$SRCROOT/components/pop2/cime_config/buildnml** to generate the POP2 namelist variables. - -For complete documentation of namelist settings, see `POP2 namelist variables `_. - -To modify POP2 namelist settings, add the appropriate keyword/value pair at the end of the **$CASEROOT/user_nl_pop2** file. -(See the documentation for each file at the top of that file.) -To see the result of your change, call ``preview_namelists`` and verify that the changes appear correctly in **CaseDocs/ocn_in**. - -CISM -```` - -See `CISM namelist variables `_ for a complete description of the CISM runtime namelist variables. This includes variables that appear both in **cism_in** and in **cism.config**. - -To modify any of these settings, add the appropriate keyword/value pair at the end of the **user_nl_cism** file. (See the documentation for each file at the top of that file.) -Note that there is no distinction between variables that will appear in **cism_in** and those that will appear in **cism.config**: simply add a new variable setting in **user_nl_cism**, and it will be added to the appropriate place in **cism_in** or **cism.config**. -To see the result of your change, call ``preview_namelists`` and verify that the changes appear correctly in **CaseDocs/cism_in** and **CaseDocs/cism.config**. +.. note:: -Some CISM runtime settings are sets via **env_run.xml**, as documented in `CISM runtime variables `_. + Refer to your model's documentation for a complete list of namelist variables and their descriptions for each active component. Setting up the Case ------------------- @@ -359,6 +297,6 @@ Depends.* Lists of source code files that need special bui Macros.cmake File containing machine-specific makefile directives for your target platform/compiler. This file is created if it does not already exist. The user can modify the file to change certain aspects of the build, such as compiler flags. Running ``case.setup --clean`` will not remove the file once it has been created. However, if you remove or rename the Macros.make file, running ``case.setup`` recreates it. case.st_archive Script to perform short-term archiving to disk for your case output. Note that this script is run automatically by the normal CIME workflow. cmake_macros/ Directory containing any CMake macros required for the machine/compiler combination. -user_nl_xxx[_NNNN] Files where all user modifications to component namelists are made. **xxx** is any one of the set of components targeted for the case. For example, for a full active CESM compset, **xxx** is cam, clm, or rtm, and so on. NNNN goes from 0001 to the number of instances of that component. (See :ref:`multiple instances`) For a case with 1 instance of each component (default), NNNN will not appear in the user_nl file names. A user_nl file of a given name is created only once. Calling ``case.setup --clean`` will *not remove* any user_nl files. Changing the number of instances in the **env_mach_pes.xml** file will cause only new user_nl files to be added to ``$CASEROOT``. +user_nl_xxx[_NNNN] Files where all user modifications to component namelists are made. **xxx** is any one of the set of components targeted for the case. For example, for a fully active compset, **xxx** could be the atmosphere, land, or river component name, and so on. NNNN goes from 0001 to the number of instances of that component. (See :ref:`multiple instances`) For a case with 1 instance of each component (default), NNNN will not appear in the user_nl file names. A user_nl file of a given name is created only once. Calling ``case.setup --clean`` will *not remove* any user_nl files. Changing the number of instances in the **env_mach_pes.xml** file will cause only new user_nl files to be added to ``$CASEROOT``. software_environment.txt This file records some aspects of the computing system on which the case is built, such as the shell environment. ============================= =============================================================================================================================== diff --git a/doc/source/ccs/timers.rst b/doc/source/ccs/timers.rst index c94d01b29a7..68598cdd494 100644 --- a/doc/source/ccs/timers.rst +++ b/doc/source/ccs/timers.rst @@ -56,6 +56,10 @@ The following describes the most important parts of this timing file: An example timing file of this type is: +.. note:: + + The following is an example timing output. The specific machine, paths, compset, and component names will vary depending on your model and configuration. + .. code-block:: text ---------------- TIMING PROFILE --------------------- diff --git a/doc/source/ccs/troubleshooting.rst b/doc/source/ccs/troubleshooting.rst index cff6a130c5b..225eb797726 100644 --- a/doc/source/ccs/troubleshooting.rst +++ b/doc/source/ccs/troubleshooting.rst @@ -104,7 +104,7 @@ Also, some model configurations read data mid-month or run physics intermittentl In those cases, some variability is expected. The time variation typically is quite erratic and unpredictable if the problem is system performance variability. Sometimes when a job times out or overflows disk space, the restart files will get mangled. -With the exception of the CAM and CLM history files, all the restart files have consistent sizes. +With the exception of component history files, all the restart files have consistent sizes. Compare the restart files against the sizes of a previous restart. If they don't match, remove them and move the previous restart into place before resubmitting the job. See `Restarting a run `_. diff --git a/doc/source/contributing-guide.rst b/doc/source/contributing-guide.rst index e43b1785f92..7e098e32da5 100644 --- a/doc/source/contributing-guide.rst +++ b/doc/source/contributing-guide.rst @@ -161,16 +161,16 @@ following example assumes the model is checked out in ``$SRC_PATH``. .. code-block:: bash - docker run -it --rm --hostname docker -e CIME_MODEL=e3sm -v ${SRC_PATH}:/root/model -v ./storage:/root/storage -w /root/E3SM/cime ghcr.io/esmci/cime:latest bash + docker run -it --rm --hostname docker -e CIME_MODEL= -v ${SRC_PATH}:/root/model -v ./storage:/root/storage -w /root/model/cime ghcr.io/esmci/cime:latest bash This example will drop into a shell where CIME commands or tests can be run. The options are broken down below. - ``--hostname docker`` is required to tell CIME which machine definition to use. -- ``-e CIME_MODEL=e3sm`` defines the model. -- ``-v ${SRC_PATH}:/root/E3SM`` passes through the model source. +- ``-e CIME_MODEL=`` defines the model. +- ``-v ${SRC_PATH}:/root/model`` passes through the model source. - ``-v ./storage:/root/storage`` persist all data; cases, baselines, archive, inputdata. the bind mounts can be broken out if you only want to persist certain input/outputs. -- ``-w /root/E3SM/cime`` set the current working directory to CIME's root. +- ``-w /root/model/cime`` set the current working directory to CIME's root. - ``ghcr.io/esmci/cime:latest`` container image. - ``bash`` the command to run in the container. @@ -178,9 +178,9 @@ You can even run CIME or testing without a shell. .. code-block:: bash - docker run -it --rm --hostname docker -e CIME_MODEL=e3sm -v ${SRC_PATH}:/root/model -v ./storage:/root/storage -w /root/E3SM/cime ghcr.io/esmci/cime:latest pytest CIME/tests/test_unit* + docker run -it --rm --hostname docker -e CIME_MODEL= -v ${SRC_PATH}:/root/model -v ./storage:/root/storage -w /root/model/cime ghcr.io/esmci/cime:latest pytest CIME/tests/test_unit* .. code-block:: bash - docker run -it --rm --hostname docker -e CIME_MODEL=e3sm -v ${SRC_PATH}:/root/model -v ./storage:/root/storage -w /root/E3SM/cime ghcr.io/esmci/cime:latest ./scripts/create_test SMS.f19_g16.S + docker run -it --rm --hostname docker -e CIME_MODEL= -v ${SRC_PATH}:/root/model -v ./storage:/root/storage -w /root/model/cime ghcr.io/esmci/cime:latest ./scripts/create_test SMS.f19_g16.S diff --git a/doc/source/system_testing.rst b/doc/source/system_testing.rst index 24b08f35e63..2f9a87b84e1 100644 --- a/doc/source/system_testing.rst +++ b/doc/source/system_testing.rst @@ -81,7 +81,7 @@ The search for test names can be restricted to a single test list using:: Omitting this results in searching all testlists listed in:: - cime/config/{cesm,e3sm}/config_files.xml + cime/config/$model/config_files.xml The ``./scripts/query_testlists`` tool gathers descriptions of the tests and testlists available in the XML format, the components, and projects. @@ -379,12 +379,12 @@ TESTMODS Name of the directory under ``GROUP`` that contains any combination A test mod can contain any combination of ``user_nl_*``, ``shell_commands``, ``user_mods``, or ``params.py``. -For example, the ``ERP`` test for an E3SM ``F-case`` can be modified to use a different radiation scheme by using ``eam-rrtmgp``:: +For example, the ``ERP`` test for an ``F-case`` can be modified to use a different radiation scheme by using ``eam-rrtmgp``:: ERP_D_Ld3.ne4pg2_oQU480.F2010.pm-cpu_intel.eam-rrtmgp -If ``TESTS_MODS_DIR`` was set to ``$E3SM/components/eam/cime_config/testdefs/testmods_dirs`` then the -directory containing the testmods would be ``$E3SM/components/eam/cime_config/testdefs/testmods_dirs/eam/rrtmpg``. +If ``TESTS_MODS_DIR`` was set to ``$SRCROOT/components/eam/cime_config/testdefs/testmods_dirs`` then the +directory containing the testmods would be ``$SRCROOT/components/eam/cime_config/testdefs/testmods_dirs/eam/rrtmpg``. In this directory, you'd find a `shell_commands`` file containing the following:: @@ -444,7 +444,7 @@ Lets look at the output from ``./scripts/create_test SMS.f19_f19.A``. Here you c .. code-block:: bash - Creating test directory /home/jgfouca/e3sm/scratch/SMS.f19_f19.A.melvin_gnu.20170504_163152_31aahy + Creating test directory /home/user/scratch/SMS.f19_f19.A.melvin_gnu.20170504_163152_31aahy RUNNING TESTS: SMS.f19_f19.A.melvin_gnu Starting CREATE_NEWCASE for test SMS.f19_f19.A.melvin_gnu with 1 procs @@ -461,7 +461,7 @@ Lets look at the output from ``./scripts/create_test SMS.f19_f19.A``. Here you c Finished RUN for test SMS.f19_f19.A.melvin_gnu in 35.068546 seconds (PASS). [COMPLETED 1 of 1] At test-scheduler close, state is: PASS SMS.f19_f19.A.melvin_gnu RUN - Case dir: /home/jgfouca/e3sm/scratch/SMS.f19_f19.A.melvin_gnu.20170504_163152_31aahy + Case dir: /home/user/scratch/SMS.f19_f19.A.melvin_gnu.20170504_163152_31aahy test-scheduler took 154.780044079 seconds The case is created in ``$CASEDIR`` which can be seen in the output above. The test status is stored in ``$CASEDIR/TestStatus``. diff --git a/doc/source/tools/ect.rst b/doc/source/tools/ect.rst index ee2e042440d..5c4b0cc990a 100644 --- a/doc/source/tools/ect.rst +++ b/doc/source/tools/ect.rst @@ -1,49 +1,40 @@ .. _ensemble-consistency-test: ========================================== -CESM-ECT (CESM Ensemble Consistency Test): +Ensemble Consistency Test (ECT): ========================================== .. contents:: :local: -CESM-ECT is a suite of tests to determine whether a new +ECT is a suite of tests to determine whether a new simulation set up (new machine, compiler, etc.) is statistically -distinguishable from an accepted ensemble. The verification tools in -the CESM-ECT suite are: +distinguishable from an accepted ensemble. The verification tools in +the ECT suite are: CAM-ECT - detects issues in CAM and CLM (12 month runs) UF-CAM-ECT - detects issues in CAM and CLM (9 time step runs) POP-ECT - detects issues in POP and CICE (12 month runs) The ECT process involves comparing runs generated with -the new scenario ( 3 for CAM-ECT and UF-CAM-ECT, and 1 for POP-ECT) -to an ensemble built on a trusted machine (currently -cheyenne). The python ECT tools are located in the pyCECT +the new scenario (3 for CAM-ECT and UF-CAM-ECT, and 1 for POP-ECT) +to an ensemble built on a trusted machine. The python ECT tools are located in the pyCECT subdirectory or https://github.com/NCAR/PyCECT/releases. --OR- - -We now provide a web server for CAM-ECT and UF-CAM-ECT, where -you can upload the (3) generated runs for comparison to our ensemble. -Please see the webpage at http://www.cesm.ucar.edu/models/cesm2/verification/ -for further instructions. - ------------------------------------- Creating or obtaining a summary file: ------------------------------------- Before the test can be run, a summary file is needed of the ensemble runs to which the comparison will be made. Ensemble summary files -(NetCDF) for existing tags for CAM-ECT, UF-CAM-ECT, and POP-ECT that -were created by CSEG are located (respectively) in the CESM input data -directories: +(NetCDF) for existing tags for CAM-ECT, UF-CAM-ECT, and POP-ECT are +located in the input data validation directories: -$CESMDATAROOT/inputdata/validation/ensembles -$CESMDATAROOT/inputdata/validation/uf_ensembles -$CESMDATAROOT/inputdata/validation/pop_ensembles +$CIME_INPUT_DATA_ROOT/validation/ensembles +$CIME_INPUT_DATA_ROOT/validation/uf_ensembles +$CIME_INPUT_DATA_ROOT/validation/pop_ensembles -If none of our ensembles are suitable for your needs, then you may create +If none of the available ensembles are suitable for your needs, then you may create your own ensemble (and summary file) using the following instructions: (1) To create a new ensemble, use the ensemble.py script in this directory. @@ -65,16 +56,16 @@ POP-ECT 40 CAM-ECT: -python ensemble.py --case /glade/scratch/cesm_user/cesm_tag/ensemble/ensemble.cesm_tag.000 --mach cheyenne --ensemble 151 --ect cam --project P99999999 +python ensemble.py --case /scratch/user/model_tag/ensemble/ensemble.model_tag.000 --mach --ensemble 151 --ect cam --project UF-CAM-ECT: -python ensemble.py --case /glade/scratch/cesm_user/cesm_tag/uf_ensemble/ensemble.cesm_tag.uf.000 --mach cheyenne --ensemble 350 --uf --ect cam --project P99999999 +python ensemble.py --case /scratch/user/model_tag/uf_ensemble/ensemble.model_tag.uf.000 --mach --ensemble 350 --uf --ect cam --project POP-ECT: -python ensemble.py --case /glade/scratch/cesm_user/cesm_tag/uf_ensemble/ensemble.cesm_tag.000 --mach cheyenne --ensemble 40 --ect pop --project P99999999 +python ensemble.py --case /scratch/user/model_tag/uf_ensemble/ensemble.model_tag.000 --mach --ensemble 40 --ect pop --project Notes: (a) ensemble.py accepts (most of) the argumenets of create_newcase @@ -98,7 +89,7 @@ Creating test runs: ------------------- (1) Once an ensemble summary file has been created or chosen to -use from $CESMDATAROOT/inputdata/validation, the simulation +use from $CIME_INPUT_DATA_ROOT/validation, the simulation run(s) to be verified by ECT must be created via script ensemble.py. NOTE: It is important that the **same** resolution and compset be used in the @@ -107,15 +98,15 @@ attributes give this information. (2) For example, for CAM-ECT: -python ensemble.py --case /glade/scratch/cesm_user/cesm_tag/camcase.cesm_tag.000 --ect cam --mach cheyenne --project P99999999 +python ensemble.py --case /scratch/user/model_tag/camcase.model_tag.000 --ect cam --mach --project --compset F2000climo --res f19_f19 For example, for UF-CAM-ECT: -python ensemble.py --case /glade/scratch/cesm_user/cesm_tag/uf.camcase.cesm_tag.000 --ect cam --uf --mach cheyenne --project P99999999 --compset F2000climo --res f19_f19 +python ensemble.py --case /scratch/user/model_tag/uf.camcase.model_tag.000 --ect cam --uf --mach --project --compset F2000climo --res f19_f19 For example, for POP-ECT: -python ensemble.py --case /glade/scratch/cesm_user/cesm_tag/popcase.cesm_tag.000 --ect pop --mach cheyenne --project P99999999 --compset G --res T62_g17 +python ensemble.py --case /scratch/user/model_tag/popcase.model_tag.000 --ect pop --mach --project --compset G --res T62_g17 (3) Next verify the new simulation(s) with the pyCECT tool pyCECT.py (see README_pyCECT.rst with the pyCECT tools). From f25e52368e83bf2638a3e063127971944e80f067 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 29 Apr 2026 18:10:21 +0000 Subject: [PATCH 03/11] docs: fix typo 'tempororal' -> 'temporal' in pes.rst Agent-Logs-Url: https://github.com/ESMCI/cime/sessions/2ef12468-32a6-42e3-a377-f05a2a354d5c Co-authored-by: jasonb5 <2191036+jasonb5@users.noreply.github.com> --- doc/source/ccs/model-configuration/variables/pes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/ccs/model-configuration/variables/pes.rst b/doc/source/ccs/model-configuration/variables/pes.rst index 5f4bf26722b..733739837fb 100644 --- a/doc/source/ccs/model-configuration/variables/pes.rst +++ b/doc/source/ccs/model-configuration/variables/pes.rst @@ -437,7 +437,7 @@ Changing the pe layout of the model has NO IMPACT on the scientific results. The basic order of operations and calling sequence are hardwired into the driver and do not change with the pe layout. However, models such as CESM and E3SM may impose some constraints in the -tempororal evolution of the components. For example, the prognostic +temporal evolution of the components. For example, the prognostic atmosphere model always run sequentially with the ice and land models for scientific reasons. As a result, running the atmosphere concurrently with the ice and land will result in idle processors at From 96fd71eb3be00cb98e570f16c0a08c8b8238e1c2 Mon Sep 17 00:00:00 2001 From: Jason Boutte Date: Wed, 24 Jun 2026 08:58:41 -0700 Subject: [PATCH 04/11] docs: clarify model-dependent config_machines.xml in port guide The "support a new machine" guide referenced legacy $CIMEROOT/config/$model/... paths that no longer reflect how the machines file is located (via MACHINES_SPEC_FILE in the model's config_files.xml) and described only one schema. Update paths to be model-agnostic, document the v2 (monolithic) and v3 (per-machine directory) layouts with examples, extend the xmllint snippet to cover both XSDs, and drop residual model-specific machine name examples so the page applies regardless of which model ships CIME. --- .../support-a-new-machine.rst | 86 +++++++++++++++++-- 1 file changed, 77 insertions(+), 9 deletions(-) diff --git a/doc/source/ccs/model-configuration/support-a-new-machine.rst b/doc/source/ccs/model-configuration/support-a-new-machine.rst index ae0d80ebd16..bd1d2d2c6ff 100644 --- a/doc/source/ccs/model-configuration/support-a-new-machine.rst +++ b/doc/source/ccs/model-configuration/support-a-new-machine.rst @@ -71,21 +71,85 @@ As an example, on a MAC with 2 cores that has mpich with gnu fortran you would i Linux and Mac Support --------------------------- -Your model distribution may include generic machine definitions (e.g., **homebrew** and **centos7-linux**) in the file **$CIMEROOT/config/$model/machines/config_machines.xml**. +Your model distribution may include generic machine definitions in the model's ``config_machines.xml`` file. +The location of this file is model-dependent and is set by the ``MACHINES_SPEC_FILE`` entry in the model's ``config_files.xml`` (see :ref:`MACHINES_SPEC_FILE `); refer to your model repository for the exact path. Please see the instructions in the file to create the directory structure and use these generic machine definitions. +.. _config-machines-schema-versions: + +About config_machines.xml +--------------------------- + +The ``config_machines.xml`` file describes the machines on which CIME-driven models can be built and run. +Both its location and the XML schema(s) it must conform to are **model-dependent**. + +The schema(s) advertised by a model are declared as ```` children of the ``MACHINES_SPEC_FILE`` entry in that model's ``config_files.xml``. +The XSD applied at validation time is then selected from the ``version`` attribute on the root ```` element of the file being loaded. +The two supported schemas are: + +**Version 2.0 (monolithic)** — ``$CIMEROOT/CIME/data/config/xml_schemas/config_machines.xsd`` + A single file contains the full definition for every supported machine. + Each ```` block carries its own ```` (used by CIME to auto-detect the machine), and ```` must appear at least once at the top level. + + .. code-block:: xml + + + + ... + mymachine.* + LINUX + ... + + + ... + + + +**Version 3.0 (per-machine directory)** — ``$CIMEROOT/CIME/data/config/xml_schemas/config_machines_version3.xsd`` + The top-level ``config_machines.xml`` only holds a registry that maps each machine to its node-name regex. + The full definition for each machine lives in its own subdirectory (``//config_machines.xml``) next to that machine's ``config_batch.xml`` and ``*.cmake`` files. + Top-level ```` blocks are optional, ```` is removed from individual ```` blocks, and the v3 schema additionally accepts a ```` element per machine. + + .. code-block:: xml + + + + mymachine.* + other.* + + + + Per-machine file (``/mymachine/config_machines.xml``): + + .. code-block:: xml + + + + ... + LINUX + ... + + + +Not every model registers both schemas. Refer to the ``MACHINES_SPEC_FILE`` entry in your model's ``config_files.xml`` to see which schema versions are available; if no ``version`` attribute is present on the ```` entry, the model accepts a single unversioned schema matching the **2.0** layout. + +See :ref:`MACHINES_SPEC_FILE ` for the full element reference for both schema versions. + Steps for porting --------------------------- Porting CIME involves several steps. The first step is to define your machine. You can do this in one of two ways: -1. You can edit **$CIMEROOT/config/$model/machines/config_machines.xml** and add an appropriate section for your machine. +1. You can edit the model's ``config_machines.xml`` directly (the path is given by ``MACHINES_SPEC_FILE`` in the model's ``config_files.xml``) and add an appropriate section for your machine. + For a **version 3.0** layout, instead add (or update) a ```` entry under the top-level ```` and create a ``//`` subdirectory containing the per-machine ``config_machines.xml``, ``config_batch.xml`` and ``*.cmake`` files. 2. You can use your **$HOME/.cime** directory (see :ref:`customizing-cime`). In particular, you can create a **$HOME/.cime/config_machines.xml** file with the definition for your machine. - A template to create this definition is provided in **$CIMEROOT/config/xml_schemas/config_machines_template.xml**. More details are provided in the template file. + A template to create this definition is provided in **$CIMEROOT/CIME/data/config/xml_schemas/config_machines_template.xml**. More details are provided in the template file. In addition, if you have a batch system, you will also need to add a **config_batch.xml** file to your **$HOME/.cime** directory. - All files in **$HOME/.cime/** are appended to the xml objects that are read into memory from the **$CIME/config/$model** directory. + All files in **$HOME/.cime/** are appended to the xml objects that are read into memory from the directory containing the model's ``MACHINES_SPEC_FILE``. + + .. note:: When the model uses the **version 3.0** schema, the per-machine override is read from **$HOME/.cime//config_machines.xml** (and a top-level **$HOME/.cime/config_machines.xml** may be used to register the ``NODENAME_REGEX``). .. note:: If you use method (2), you can download CIME updates without affecting your machine definitions in **$HOME/.cime**. @@ -97,18 +161,22 @@ In what follows we outline the process for method (2) above: This file contains all the information you must set in order to configure a new machine to be CIME-compliant. - Fill in the contents of **$HOME/.cime/config_machines.xml** that are specific to your machine. For more details see :ref:`the config_machines.xml file `. + Fill in the contents of **$HOME/.cime/config_machines.xml** that are specific to your machine. For more details see :ref:`MACHINES_SPEC_FILE ` and :ref:`the schema overview above `. - Check to ensure that your **config_machines.xml** file conforms to the CIME schema definition by doing the following: + Check to ensure that your **config_machines.xml** file conforms to the CIME schema definition by validating it against the XSD that matches the ``version`` declared on its root element: :: - xmllint --noout --schema $CIME/config/xml_schemas/config_machines.xsd $HOME/.cime/config_machines.xml + # version 2.0 (also used when the model registers only an unversioned schema) + xmllint --noout --schema $CIMEROOT/CIME/data/config/xml_schemas/config_machines.xsd $HOME/.cime/config_machines.xml + + # version 3.0 + xmllint --noout --schema $CIMEROOT/CIME/data/config/xml_schemas/config_machines_version3.xsd $HOME/.cime/config_machines.xml - If you find that you need to introduce compiler settings specific to your machine, create a **$HOME/.cime/*.cmake** file. - The default compiler settings are defined in **$CIME/config/$model/machines/cmake_macros/**. + The default compiler settings are defined in the ``cmake_macros/`` directory that lives alongside the model's ``MACHINES_SPEC_FILE``. - If you have a batch system, you may also need to create a **$HOME/.cime/config_batch.xml** file. - Out-of-the-box batch settings are set in **$CIME/config/$model/machines/config_batch.xml**. + Out-of-the-box batch settings are set in the ``config_batch.xml`` file that lives alongside the model's ``MACHINES_SPEC_FILE`` (its path is given by ``BATCH_SPEC_FILE`` in the model's ``config_files.xml``). - Once you have defined a basic configuration for your machine in your **$HOME/.cime** xml files, run the :ref:`unittest `. From a9b58425aca41b021450da4d04c0fad8139e6aec Mon Sep 17 00:00:00 2001 From: Jason Boutte Date: Wed, 24 Jun 2026 09:03:15 -0700 Subject: [PATCH 05/11] docs: note CESM/NorESM scope of port validation section The validation workflow uses the prealpha test category and pyCECT ECT tooling that target CAM/CLM/POP/CICE, which are CESM (and NorESM) constructs. Flag this up front so users of other CIME-based models look to their own documentation for the equivalent procedure. --- doc/source/ccs/model-configuration/support-a-new-machine.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/source/ccs/model-configuration/support-a-new-machine.rst b/doc/source/ccs/model-configuration/support-a-new-machine.rst index bd1d2d2c6ff..319e5a6f341 100644 --- a/doc/source/ccs/model-configuration/support-a-new-machine.rst +++ b/doc/source/ccs/model-configuration/support-a-new-machine.rst @@ -185,6 +185,10 @@ After running those steps correctly, you are ready to try a case at your target Validating a port with prognostic components ------------------------------------------------- +.. note:: + + This section is specific to CESM (and NorESM). Consult your model's documentation for the equivalent procedure. + The following port validation is recommended for any new machine. Carrying out these steps does not guarantee the model is running properly in all cases nor that the model is scientifically valid on From 2a8552d56e7fbe1ed2d810f380224f5dbba2b96b Mon Sep 17 00:00:00 2001 From: Jason Boutte Date: Wed, 24 Jun 2026 09:10:59 -0700 Subject: [PATCH 06/11] docs: clarify CIME_MODEL placeholder in CCS quick start Rename the bare placeholder to in CCS docs so readers cannot mistake it for a literal value, add a Sphinx label on the environment-setup section, and cross-reference it from the Quick Start so the valid CIME_MODEL values remain documented in a single authoritative place. --- doc/source/ccs/index.rst | 8 +++++--- doc/source/ccs/user-config.rst | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/source/ccs/index.rst b/doc/source/ccs/index.rst index 6bc2e6fe457..45177d5e74d 100644 --- a/doc/source/ccs/index.rst +++ b/doc/source/ccs/index.rst @@ -18,7 +18,7 @@ The first thing to do is clone the model repository into ``$SRCROOT``. All examples will be run from ``$CIMEROOT`` which is should exist under ``$SRCROOT`` e.g. (``$CIMEROOT`` would be ``$SRCROOT/cime``). -Next set the ``CIME_MODEL`` environment variable for your model, e.g. ``export CIME_MODEL=``. +Next set the ``CIME_MODEL`` environment variable for your model, e.g. ``export CIME_MODEL=`` (see :ref:`Setting up your environment for CCS ` for valid values). .. note:: @@ -155,16 +155,18 @@ Customizations The CCS allows for a number of customizations to be made to a case. +.. _ccs-env-setup: + Setting up your environment for CCS ``````````````````````````````````` After you've cloned the model repository, you'll need to set up your environment to use the CCS. First you'll need to let CIME know which model configuration to use by setting the ``CIME_MODEL`` environment variable. In bash, use **export** as shown and replace -**** with the appropriate text. Current possibilities are "e3sm", "cesm", or "ufs". +**** with the appropriate text. Current possibilities are "e3sm", "cesm", or "ufs". :: - export CIME_MODEL= + export CIME_MODEL= There are a number of possible ways to set CIME variables. For variables that can be set in more than one way, the order of precedence is: diff --git a/doc/source/ccs/user-config.rst b/doc/source/ccs/user-config.rst index 8c3f44acdb0..4da800c1277 100644 --- a/doc/source/ccs/user-config.rst +++ b/doc/source/ccs/user-config.rst @@ -19,7 +19,7 @@ Variables can be defined in the **[main]** section of the config file. .. code-block:: ini [main] - CIME_MODEL= + CIME_MODEL= PROJECT= MAIL_TYPE= SRCROOT= From eddaceb1a9d55c8127cea28559c8290d76bccb4f Mon Sep 17 00:00:00 2001 From: Jason Boutte Date: Wed, 24 Jun 2026 09:19:31 -0700 Subject: [PATCH 07/11] docs: normalize placeholder convention in setting-up-a-case Replace angle-bracket placeholders like = with ALL_CAPS tokens (VARIABLE=VALUE) so readers don't type the brackets literally. Also fix mismatched [extension] brackets in the five data model sections (DATM/DOCN/DICE/DLND/DROF). --- doc/source/ccs/setting-up-a-case.rst | 42 ++++++++++++++-------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/doc/source/ccs/setting-up-a-case.rst b/doc/source/ccs/setting-up-a-case.rst index 47cdeeba9f7..070af038619 100644 --- a/doc/source/ccs/setting-up-a-case.rst +++ b/doc/source/ccs/setting-up-a-case.rst @@ -28,18 +28,18 @@ To print detailed information about variables, including their description, type ./xmlquery --listall --full -To view a specific group, replace ```` with the desired group name. +To view a specific group, replace ``GROUP`` with the desired group name. .. code-block:: bash - ./xmlquery --subgroup --listall + ./xmlquery --subgroup GROUP --listall -You can search for partial matches using the ``-p`` or ``--partial`` option (can take a regex) and replacing ```` with the string you want to search for. This will return all variables that contain the search string in their name or value. +You can search for partial matches using the ``-p`` or ``--partial`` option (can take a regex) and replacing ``SEARCH_STRING`` with the string you want to search for. This will return all variables that contain the search string in their name or value. .. code-block:: bash # e.g. ./xmlquery --partial STOP, ./xmlquery --partial 'STOP_(N|O)' - ./xmlquery --partial + ./xmlquery --partial SEARCH_STRING Variables can be printed without being resolved by using the ``--no-resolve`` option. @@ -61,19 +61,19 @@ The ``xmlchange`` command is used to modify the configuration of a case. The fol .. code-block:: bash - ./xmlchange = + ./xmlchange VARIABLE=VALUE The `value` can be literal or a reference to another value. When using a reference, it must be prefixed with `$`. .. code-block:: bash - ./xmlchange =$ + ./xmlchange VARIABLE=$VARIABLE The reference can also define the `subgroup`. This is usful when a variable exists under multiple subgroups and a specific one needs to be referenced. The `subgroup` and `variable` are delimited with `::`. .. code-block:: bash - ./xmlchange =$:: + ./xmlchange VARIABLE=$SUBGROUP::VARIABLE Some variables can exist in multiple groups. To change a variable in a specific group, use the ``--subgroup`` option. @@ -95,7 +95,7 @@ For example, if a model's atmosphere model (DATM) was located in the directory ` Users can customize a component models's namelist in two ways: -1. By editing the ``$CASEROOT/user_nl_`` files +1. By editing the ``$CASEROOT/user_nl_COMP`` files These files should be modified via keyword-value pairs that correspond to new namelist or input data settings. They use the syntax of Fortran namelists. @@ -159,9 +159,9 @@ The namelist file for DATM is **datm_in** (or **datm_in_NNN** for multiple insta - To modify the contents of a DATM stream file, first run ``preview_namelists`` to list the *streams.txt* files in the **CaseDocs/** directory. Then, in the same directory: 1. Make a *copy* of the file with the string *"user_"* prepended. - ``> cp datm.streams.txt.[extension] user_datm.streams.txt[extension.`` + ``> cp datm.streams.txt.EXTENSION user_datm.streams.txt.EXTENSION`` 2. **Change the permissions of the file to be writeable.** (chmod 644) - ``chmod 644 user_datm.streams.txt[extension`` + ``chmod 644 user_datm.streams.txt.EXTENSION`` 3. Edit the **user_datm.streams.txt.*** file. **Example** @@ -180,9 +180,9 @@ The namelist file for DOCN is **docn_in** (or **docn_in_NNN** for multiple insta - To modify the contents of a DOCN stream file, first run ``preview_namelists`` to list the *streams.txt* files in the **CaseDocs/** directory. Then, in the same directory: 1. Make a *copy* of the file with the string *"user_"* prepended. - ``> cp docn.streams.txt.[extension] user_docn.streams.txt[extension.`` + ``> cp docn.streams.txt.EXTENSION user_docn.streams.txt.EXTENSION`` 2. **Change the permissions of the file to be writeable.** (chmod 644) - ``chmod 644 user_docn.streams.txt[extension`` + ``chmod 644 user_docn.streams.txt.EXTENSION`` 3. Edit the **user_docn.streams.txt.*** file. **Example** @@ -201,9 +201,9 @@ The namelist file for DICE is ``dice_in`` (or ``dice_in_NNN`` for multiple insta - To modify the contents of a DICE stream file, first run ``preview_namelists`` to list the *streams.txt* files in the **CaseDocs/** directory. Then, in the same directory: 1. Make a *copy* of the file with the string *"user_"* prepended. - ``> cp dice.streams.txt.[extension] user_dice.streams.txt[extension.`` + ``> cp dice.streams.txt.EXTENSION user_dice.streams.txt.EXTENSION`` 2. **Change the permissions of the file to be writeable.** (chmod 644) - ``chmod 644 user_dice.streams.txt[extension`` + ``chmod 644 user_dice.streams.txt.EXTENSION`` 3. Edit the **user_dice.streams.txt.*** file. Data Land (DLND) @@ -217,9 +217,9 @@ The namelist file for DLND is ``dlnd_in`` (or ``dlnd_in_NNN`` for multiple insta - To modify the contents of a DLND stream file, first run ``preview_namelists`` to list the *streams.txt* files in the **CaseDocs/** directory. Then, in the same directory: 1. Make a *copy* of the file with the string *"user_"* prepended. - ``> cp dlnd.streams.txt.[extension] user_dlnd.streams.txt[extension.`` + ``> cp dlnd.streams.txt.EXTENSION user_dlnd.streams.txt.EXTENSION`` 2. **Change the permissions of the file to be writeable.** (chmod 644) - ``chmod 644 user_dlnd.streams.txt[extension`` + ``chmod 644 user_dlnd.streams.txt.EXTENSION`` 3. Edit the **user_dlnd.streams.txt.*** file. Data River (DROF) @@ -233,9 +233,9 @@ The namelist file for DROF is ``drof_in`` (or ``drof_in_NNN`` for multiple insta - To modify the contents of a DROF stream file, first run ``preview_namelists`` to list the *streams.txt* files in the **CaseDocs/** directory. Then, in the same directory: 1. Make a *copy* of the file with the string *"user_"* prepended. - ``> cp drof.streams.txt.[extension] user_drof.streams.txt[extension.`` + ``> cp drof.streams.txt.EXTENSION user_drof.streams.txt.EXTENSION`` 2. **Change the permissions of the file to be writeable.** (chmod 644) - ``chmod 644 user_drof.streams.txt[extension`` + ``chmod 644 user_drof.streams.txt.EXTENSION`` 3. Edit the **user_drof.streams.txt.*** file. Customizing active component-specific namelist settings @@ -243,12 +243,12 @@ Customizing active component-specific namelist settings Active components typically provide a ``buildnml`` script in their ``cime_config`` directory that generates the component's namelist variables. Component-specific CIME xml variables are set in the component's ``config_component.xml`` file and are used by the ``buildnml`` script to generate the namelist. -To modify an active component's namelist settings, add the appropriate keyword/value pair at the end of the **$CASEROOT/user_nl_** file, where ```` is the component name. See the documentation at the top of each ``user_nl`` file for details. +To modify an active component's namelist settings, add the appropriate keyword/value pair at the end of the **$CASEROOT/user_nl_COMP** file, where ``COMP`` is the component name. See the documentation at the top of each ``user_nl`` file for details. For example, to modify a namelist variable for an atmosphere component, edit the corresponding ``user_nl`` file: :: - = + VARIABLE_NAME=VALUE To see the result, call ``preview_namelists`` and verify that the new value appears in the appropriate file under **CaseDocs/**. @@ -278,7 +278,7 @@ If you set the ``CASE_GIT_REPOSITORY`` variable to a valid Git repository URL, t .. code-block:: bash - ./xmlchange CASE_GIT_REPOSITORY= + ./xmlchange CASE_GIT_REPOSITORY=REPOSITORY .. note:: From 334d6c842b6896c75720433267f4d228ab758f12 Mon Sep 17 00:00:00 2001 From: Jason Boutte Date: Wed, 24 Jun 2026 09:29:58 -0700 Subject: [PATCH 08/11] docs: refine driver-specific notes in troubleshooting - List all three CIME-accepted termination markers (SUCCESSFUL TERMINATION, END OF MODEL RUN, HAS ENDED) instead of just one; see TERMINATION_TEXT in CIME/case/case_run.py. - Note that the tStamp_write timing-diagnostics format is specific to MCT/cpl7-based drivers; other drivers may differ. - Replace stale external esmci.github.io URL with internal :ref: to the 'Restarting a Run' section, and add the corresponding label in running-a-case.rst. --- doc/source/ccs/running-a-case.rst | 2 ++ doc/source/ccs/troubleshooting.rst | 7 +++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/doc/source/ccs/running-a-case.rst b/doc/source/ccs/running-a-case.rst index 19406d868fa..f721e94821f 100644 --- a/doc/source/ccs/running-a-case.rst +++ b/doc/source/ccs/running-a-case.rst @@ -347,6 +347,8 @@ The XML variable ``$GET_REFCASE`` is a flag that if set will automatically pre-s - If ``$GET_REFCASE`` is ``FALSE``, then the data is assumed to already exist in ``$RUNDIR``. +.. _restarting-a-run: + Restarting a Run ````````````````` Active components (and some data components) write restart files diff --git a/doc/source/ccs/troubleshooting.rst b/doc/source/ccs/troubleshooting.rst index 225eb797726..5e4b94052bc 100644 --- a/doc/source/ccs/troubleshooting.rst +++ b/doc/source/ccs/troubleshooting.rst @@ -37,7 +37,7 @@ Take these steps to check for problems: Runtime problems --------------------------------- -To see if a run completed successfully, check the last several lines of the **cpl.log** file for a string like ``SUCCESSFUL TERMINATION``. A successful job also usually copies the log files to the **$CASEROOT/logs** directory. +To see if a run completed successfully, check the last several lines of the **cpl.log** file for one of the termination markers ``SUCCESSFUL TERMINATION``, ``END OF MODEL RUN``, or ``HAS ENDED``. A successful job also usually copies the log files to the **$CASEROOT/logs** directory. Check these things first when a job fails: @@ -90,6 +90,9 @@ The output looks like this: tStamp_write: model date = 10120 0 wall clock = 2009-09-28 09:10:46 avg dt = 58.58 dt = 58.18 tStamp_write: model date = 10121 0 wall clock = 2009-09-28 09:12:32 avg dt = 60.10 dt = 105.90 +.. note:: + + The ``tStamp_write`` format shown above is produced by MCT/cpl7-based drivers; other drivers may emit different timing diagnostics. Review the run times at the end of each line for each model day. The "avg dt =" is the average time to simulate a model day and "dt = " is the time needed to simulate the latest model day. @@ -107,7 +110,7 @@ Sometimes when a job times out or overflows disk space, the restart files will g With the exception of component history files, all the restart files have consistent sizes. Compare the restart files against the sizes of a previous restart. If they don't match, remove them and move the previous restart into place before resubmitting the job. -See `Restarting a run `_. +See :ref:`restarting-a-run`. It is not uncommon for nodes to fail on HPC systems or for access to large file systems to hang. Before you file a bug report, make sure a case fails consistently in the same place. From 1b10396f50b92e00bebfd73a48d97a5575540871 Mon Sep 17 00:00:00 2001 From: Jason Boutte Date: Wed, 24 Jun 2026 09:43:30 -0700 Subject: [PATCH 09/11] docs: revert ect.rst to original CESM-ECT content ECT (Ensemble Consistency Test) is specific to CESM and targets CAM/CLM/POP/CICE components. Restore the original CESM-specific references, including CESMDATAROOT paths, cheyenne machine examples, and the CSEG-hosted verification web server. --- doc/source/tools/ect.rst | 47 ++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/doc/source/tools/ect.rst b/doc/source/tools/ect.rst index 5c4b0cc990a..ee2e042440d 100644 --- a/doc/source/tools/ect.rst +++ b/doc/source/tools/ect.rst @@ -1,40 +1,49 @@ .. _ensemble-consistency-test: ========================================== -Ensemble Consistency Test (ECT): +CESM-ECT (CESM Ensemble Consistency Test): ========================================== .. contents:: :local: -ECT is a suite of tests to determine whether a new +CESM-ECT is a suite of tests to determine whether a new simulation set up (new machine, compiler, etc.) is statistically -distinguishable from an accepted ensemble. The verification tools in -the ECT suite are: +distinguishable from an accepted ensemble. The verification tools in +the CESM-ECT suite are: CAM-ECT - detects issues in CAM and CLM (12 month runs) UF-CAM-ECT - detects issues in CAM and CLM (9 time step runs) POP-ECT - detects issues in POP and CICE (12 month runs) The ECT process involves comparing runs generated with -the new scenario (3 for CAM-ECT and UF-CAM-ECT, and 1 for POP-ECT) -to an ensemble built on a trusted machine. The python ECT tools are located in the pyCECT +the new scenario ( 3 for CAM-ECT and UF-CAM-ECT, and 1 for POP-ECT) +to an ensemble built on a trusted machine (currently +cheyenne). The python ECT tools are located in the pyCECT subdirectory or https://github.com/NCAR/PyCECT/releases. +-OR- + +We now provide a web server for CAM-ECT and UF-CAM-ECT, where +you can upload the (3) generated runs for comparison to our ensemble. +Please see the webpage at http://www.cesm.ucar.edu/models/cesm2/verification/ +for further instructions. + ------------------------------------- Creating or obtaining a summary file: ------------------------------------- Before the test can be run, a summary file is needed of the ensemble runs to which the comparison will be made. Ensemble summary files -(NetCDF) for existing tags for CAM-ECT, UF-CAM-ECT, and POP-ECT are -located in the input data validation directories: +(NetCDF) for existing tags for CAM-ECT, UF-CAM-ECT, and POP-ECT that +were created by CSEG are located (respectively) in the CESM input data +directories: -$CIME_INPUT_DATA_ROOT/validation/ensembles -$CIME_INPUT_DATA_ROOT/validation/uf_ensembles -$CIME_INPUT_DATA_ROOT/validation/pop_ensembles +$CESMDATAROOT/inputdata/validation/ensembles +$CESMDATAROOT/inputdata/validation/uf_ensembles +$CESMDATAROOT/inputdata/validation/pop_ensembles -If none of the available ensembles are suitable for your needs, then you may create +If none of our ensembles are suitable for your needs, then you may create your own ensemble (and summary file) using the following instructions: (1) To create a new ensemble, use the ensemble.py script in this directory. @@ -56,16 +65,16 @@ POP-ECT 40 CAM-ECT: -python ensemble.py --case /scratch/user/model_tag/ensemble/ensemble.model_tag.000 --mach --ensemble 151 --ect cam --project +python ensemble.py --case /glade/scratch/cesm_user/cesm_tag/ensemble/ensemble.cesm_tag.000 --mach cheyenne --ensemble 151 --ect cam --project P99999999 UF-CAM-ECT: -python ensemble.py --case /scratch/user/model_tag/uf_ensemble/ensemble.model_tag.uf.000 --mach --ensemble 350 --uf --ect cam --project +python ensemble.py --case /glade/scratch/cesm_user/cesm_tag/uf_ensemble/ensemble.cesm_tag.uf.000 --mach cheyenne --ensemble 350 --uf --ect cam --project P99999999 POP-ECT: -python ensemble.py --case /scratch/user/model_tag/uf_ensemble/ensemble.model_tag.000 --mach --ensemble 40 --ect pop --project +python ensemble.py --case /glade/scratch/cesm_user/cesm_tag/uf_ensemble/ensemble.cesm_tag.000 --mach cheyenne --ensemble 40 --ect pop --project P99999999 Notes: (a) ensemble.py accepts (most of) the argumenets of create_newcase @@ -89,7 +98,7 @@ Creating test runs: ------------------- (1) Once an ensemble summary file has been created or chosen to -use from $CIME_INPUT_DATA_ROOT/validation, the simulation +use from $CESMDATAROOT/inputdata/validation, the simulation run(s) to be verified by ECT must be created via script ensemble.py. NOTE: It is important that the **same** resolution and compset be used in the @@ -98,15 +107,15 @@ attributes give this information. (2) For example, for CAM-ECT: -python ensemble.py --case /scratch/user/model_tag/camcase.model_tag.000 --ect cam --mach --project +python ensemble.py --case /glade/scratch/cesm_user/cesm_tag/camcase.cesm_tag.000 --ect cam --mach cheyenne --project P99999999 --compset F2000climo --res f19_f19 For example, for UF-CAM-ECT: -python ensemble.py --case /scratch/user/model_tag/uf.camcase.model_tag.000 --ect cam --uf --mach --project --compset F2000climo --res f19_f19 +python ensemble.py --case /glade/scratch/cesm_user/cesm_tag/uf.camcase.cesm_tag.000 --ect cam --uf --mach cheyenne --project P99999999 --compset F2000climo --res f19_f19 For example, for POP-ECT: -python ensemble.py --case /scratch/user/model_tag/popcase.model_tag.000 --ect pop --mach --project --compset G --res T62_g17 +python ensemble.py --case /glade/scratch/cesm_user/cesm_tag/popcase.cesm_tag.000 --ect pop --mach cheyenne --project P99999999 --compset G --res T62_g17 (3) Next verify the new simulation(s) with the pyCECT tool pyCECT.py (see README_pyCECT.rst with the pyCECT tools). From 1ba45272d1baba7384080a02aa398c8690caf176 Mon Sep 17 00:00:00 2001 From: Jason Boutte Date: Wed, 24 Jun 2026 09:52:10 -0700 Subject: [PATCH 10/11] docs: restore E3SM context in ERP testmods example The ERP F-case example with eam-rrtmgp is E3SM-specific. Restore the E3SM model reference and $E3SM source path, and clarify that pm-cpu_intel is the E3SM-specific Perlmutter CPU + Intel machine/compiler combination. --- doc/source/system_testing.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/source/system_testing.rst b/doc/source/system_testing.rst index 2f9a87b84e1..a7add4421aa 100644 --- a/doc/source/system_testing.rst +++ b/doc/source/system_testing.rst @@ -379,12 +379,12 @@ TESTMODS Name of the directory under ``GROUP`` that contains any combination A test mod can contain any combination of ``user_nl_*``, ``shell_commands``, ``user_mods``, or ``params.py``. -For example, the ``ERP`` test for an ``F-case`` can be modified to use a different radiation scheme by using ``eam-rrtmgp``:: +For example, the ``ERP`` test for an E3SM ``F-case`` can be modified to use a different radiation scheme by using ``eam-rrtmgp``, where ``pm-cpu_intel`` is the E3SM-specific machine/compiler combination (Perlmutter CPU partition with Intel compiler):: ERP_D_Ld3.ne4pg2_oQU480.F2010.pm-cpu_intel.eam-rrtmgp -If ``TESTS_MODS_DIR`` was set to ``$SRCROOT/components/eam/cime_config/testdefs/testmods_dirs`` then the -directory containing the testmods would be ``$SRCROOT/components/eam/cime_config/testdefs/testmods_dirs/eam/rrtmpg``. +If ``TESTS_MODS_DIR`` was set to ``$E3SM/components/eam/cime_config/testdefs/testmods_dirs`` then the +directory containing the testmods would be ``$E3SM/components/eam/cime_config/testdefs/testmods_dirs/eam/rrtmpg``. In this directory, you'd find a `shell_commands`` file containing the following:: From 4aa71164f7a5134d658f9728ae0c1ec7f19d330f Mon Sep 17 00:00:00 2001 From: Jason Boutte Date: Wed, 24 Jun 2026 10:20:45 -0700 Subject: [PATCH 11/11] docs: tighten restart-file mangling note in troubleshooting Consolidate the explanation about mangled restart files into a single sentence for clarity. --- doc/source/ccs/troubleshooting.rst | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/doc/source/ccs/troubleshooting.rst b/doc/source/ccs/troubleshooting.rst index 5e4b94052bc..fc0100504ab 100644 --- a/doc/source/ccs/troubleshooting.rst +++ b/doc/source/ccs/troubleshooting.rst @@ -106,10 +106,7 @@ For instance, on the last day of every simulated month, the model typically writ Also, some model configurations read data mid-month or run physics intermittently at a timestep longer than one day. In those cases, some variability is expected. The time variation typically is quite erratic and unpredictable if the problem is system performance variability. -Sometimes when a job times out or overflows disk space, the restart files will get mangled. -With the exception of component history files, all the restart files have consistent sizes. - -Compare the restart files against the sizes of a previous restart. If they don't match, remove them and move the previous restart into place before resubmitting the job. +Sometimes when a job times out or overflows disk space, the restart files will get mangled. Since restart files normally have consistent sizes across runs, compare them against a previous restart -- if they don't match, remove them and move the previous restart into place before resubmitting the job. See :ref:`restarting-a-run`. It is not uncommon for nodes to fail on HPC systems or for access to large file systems to hang. Before you file a bug report, make sure a case fails consistently in the same place.