Add explanation docs

.markdownlint.yaml

@@ -0,0 +1,12 @@
+# Documentation line-length enforcement.
+#
+# An 88-character limit for documentation, matching [tool.ruff] line-length in
+# pyproject.toml. Only the line-length rule (MD013) is enabled.
+#
+# Files generated by the Copier template are excluded via the
+# markdownlint-cli2 hook in .pre-commit-config.yaml rather than reformatted.
+default: false
+MD013:
+  line_length: 88
+  tables: false # Markdown tables cannot be sensibly wrapped
+  code_blocks: false # fenced code and tree diagrams cannot be wrapped

.pre-commit-config.yaml

@@ -31,6 +31,22 @@ repos:
         entry: uv sync
         files: ^(uv\.lock|pyproject\.toml)$
 
+  - repo: https://github.com/DavidAnson/markdownlint-cli2
+    rev: v0.23.1
+    hooks:
+      - id: markdownlint-cli2
+        name: enforce 88-char line length in docs
+        files: ^docs/.*\.md$
+        # Files generated by the Copier template are excluded.
+        exclude: |
+          (?x)^docs/(
+            index|genindex|how-to|tutorials|explanations|reference|
+            how-to/contribute|how-to/run-container|tutorials/installation|
+            explanations/decisions|
+            explanations/decisions/0001-record-architecture-decisions|
+            explanations/decisions/0002-switched-to-python-copier-template
+          )\.md$
+
   - repo: https://github.com/gitleaks/gitleaks
     rev: v8.28.0
     hooks:

README.md

@@ -43,23 +43,27 @@ deploy-tools validate $deployment_root $config_folder
 # validation as part of determining the required changes
 deploy-tools sync $deployment_root $config_folder
 
-# Compare the previous deployment snapshot against what is actually deployed in the
+# Compare the current deployment snapshot against what is actually deployed in the
 # deployment area. CI/CD should run this before a deploy to confirm a healthy state.
 deploy-tools compare $deployment_root
 
 ```
 
 ## Deployment Steps
 
-There are several key steps that make up the deployment process. Note that these are a
-bit different to the CLI commands; see `deploy-tools --help` for more information.
+There are several key conceptual steps that make up the deployment process. These are
+**not** one-to-one with the CLI commands; the "Run by" column shows which command runs
+each step. See `deploy-tools --help` for more detail.
 
-|**Step**|**Description**                                                                                                                                                                                                                                  |
-|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-|Compare |Compare the snapshot taken of the previous deployment with the modulefiles and built modules that already exist. This ensures that the Deployment Area is in a healthy state                                                                     |
-|Validate|Process the updated configuration. By comparing the new configuration files with a snapshot from the previous deployment, we determine the set of actions that need to be taken                                                                  |
-|Build   |Generate entrypoint scripts, configuration files and environment variables for a given Module. These are output to the Build Area                                                                                                                |
-|Deploy  |Move all built Modules from the Build Area into the Modules Area. A link to the built modulefile is moved to either the Modulefiles Folder or Deprecated Folder, depending on its deprecation status. Update default versions for the modulefiles|
+|**Step**|**Description**                                                                                                                                                                                                                                  |**Run by**|
+|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|
+|Compare |Compare the current deployment snapshot file with the modulefiles and built modules that currently exist. This ensures that the Deployment Area is in a healthy state                                                                     |`compare` |
+|Validate|Process the updated configuration. By comparing the new configuration files with the current deployment snapshot from the last sync, we determine the set of actions that need to be taken                                                                  |`validate`, `sync`|
+|Build   |Generate entrypoint scripts, configuration files and environment variables for a given Module. These are output to the Build Area                                                                                                                |`sync` (`validate --test-build`)|
+|Deploy  |Move all built Modules from the Build Area into the Modules Area. A link to the built modulefile is moved to either the Modulefiles Folder or Deprecated Folder, depending on its deprecation status. Update default versions for the modulefiles|`sync`    |
+
+There is no standalone `build` or `deploy` command: `sync` always runs them together, so
+a half-built Module is never exposed to users.
 
 ## JSON Schema
 
@@ -69,6 +73,9 @@ We strongly recommend that you provide a schema for configuration file validatio
 
 ```# yaml-language-server: $schema=https://raw.githubusercontent.com/DiamondLightSource/deploy-tools/main/src/deploy_tools/models/schemas/release.json```
 
+Use `release.json` for the per-version Module files (`<name>/<version>.yaml`) and
+`deployment-settings.json` for the top-level `settings.yaml`.
+
 As the demo_configuration is used during development, we instead set it to use the locally generated schemas via an absolute workspace path (e.g. `/workspaces/deploy-tools/src/deploy_tools/models/schemas/release.json`). This dev-container-only path should not be used for production configuration.
 
 Note that the 'Generate Schema' VSCode task will update the schemas according to any update of the code, but you need to trigger this manually and check the contents in.
@@ -83,8 +90,14 @@ available as Tasks and Debug configurations for VSCode. These tasks (plus their
 inputs) should create a separate `demo-output` folder at the top-level of the workspace
 folder.
 
-You will need to use the `--from-scratch` argument if starting from a clean Deployment
-Area, as there is no snapshot from a prior Deploy step.
+You will need to use the `--from-scratch` argument when starting from a clean Deployment
+Area, as there is no snapshot from a prior Deploy step:
+
+- `sync`/`validate --from-scratch` work against an empty area without expecting a
+  snapshot, and imply `--allow-all` (there is no prior state to enforce lifecycle rules
+  against).
+- `compare --from-scratch` just checks the Deployment Area exists and is **empty**. Run
+  this in CI before the first deploy, where a normal `compare` has no snapshot to use.
 
 It is recommended that you use `--help` to explore the commands, arguments and options
 in greater detail.

docs/explanations.md

@@ -4,7 +4,10 @@ Explanations of how it works and why it works that way.
 
 ```{toctree}
 :maxdepth: 1
-:glob:
 
-explanations/*
+explanations/deployment-area
+explanations/snapshots-and-compare
+explanations/deprecation-lifecycle
+explanations/default-versions
+explanations/decisions
 ```

docs/explanations/default-versions.md

@@ -0,0 +1,48 @@
+# Default version resolution
+
+When a user runs `module load <name>` without a version, Environment Modules must pick
+one. By default it picks the lexically last version, which is often wrong: `1.10` sorts
+before `1.9`, and an unfinished alpha might also win. `deploy-tools` allows you to make
+an explicit choice, and otherwise uses the `natsort` library for a more reasonable default.
+
+## Every Module has a specific default configured
+
+For every live module name, `deploy-tools` writes a `.version` file in its modulefiles
+folder (`modulefiles/<name>/.version`) containing a `set ModulesVersion <version>` line.
+It is written even if the configuration does not have an explicit default, so Environment
+Modules never guesses.
+
+The version it records is resolved as follows:
+
+1. **An explicit default has priority.** If `settings.default_versions` names a version,
+   it is used. It must be a live (non-deprecated) version that exists, or validation fails.
+2. **Otherwise the highest eligible version is chosen automatically**, by natural version
+   sorting (see below), considering only versions *not* marked `exclude_from_defaults`.
+
+Deprecated versions are never candidates and never get a `.version` file.
+
+## Natural version sorting
+
+Automatic selection uses [`natsort`](https://natsort.readthedocs.io) rather than plain
+string comparison, so `1.10` sorts after `1.9`. It also uses natsort's key for
+pre-release schemes, so pre-release tags sort *before* the release they lead up to:
+
+```text
+1.2alpha1  <  1.2rc1  <  1.2  <  1.2.1
+```
+
+So publishing `1.2rc1` does not make it the default while `1.2` exists, and publishing
+`1.2` makes it the default over its own release candidates. Be aware that `1.3alpha1` still
+sorts after `1.2.1`, however; to keep alphas or similar out of the automatic default, mark
+them `exclude_from_defaults: true`.
+
+## Excluding a version from the automatic default
+
+Marking a version `exclude_from_defaults: true` keeps it out of *automatic* selection
+while still allowing it to be set explicitly. Use this to publish an alpha or beta that
+users can opt into with `module load <name>/<version>` without it ever being handed to
+someone who just typed `module load <name>`.
+
+One safety check applies: if *every* version of a name is excluded and no explicit default
+is given, validation fails rather than leave the Module with no default. Either give an
+explicit default or keep at least one non-excluded version.

docs/explanations/deployment-area.md

@@ -0,0 +1,73 @@
+# The deployment area
+
+The *deployment area* is the directory tree that `deploy-tools` writes into. It is normally
+a shared filesystem location that end users put on their `MODULEPATH`, so its layout is
+important to the end user. This page explains what lives where, and why.
+
+## Layout
+
+Everything below is created and maintained by `deploy-tools`; nothing in the area is meant
+to be edited by hand.
+
+```text
+<deployment root>/
+├── deployment.yaml          # snapshot of the deployed configuration
+├── modules/                 # Modules Area — the built files for every Module
+│   └── <name>/<version>/
+│       ├── modulefile        # the Environment Modules (Tcl) file
+│       ├── entrypoints/      # the executables added to PATH on module load
+│       ├── module.yaml       # this Module's configuration snapshot
+│       └── sif_files/        # Apptainer images, when the Module uses any
+├── modulefiles/             # Modulefiles Folder — placed on MODULEPATH
+│   └── <name>/
+│       ├── <version>         # absolute symlink into modules/<name>/<version>/modulefile
+│       └── .version          # records the default version (see below)
+├── deprecated/
+│   └── modulefiles/         # Deprecated Folder — optionally on MODULEPATH
+│       └── <name>/<version>  # symlink for a deprecated Module version
+└── build/                   # transient build area (default location)
+```
+
+The split between the **Modules Area** (`modules/`) and the **Modulefiles Folder**
+(`modulefiles/`) is the central design choice. The Modules Area holds the real files for
+*every* version that has been deployed and not removed, both live and deprecated. The
+Modulefiles Folder holds only a thin tree of symlinks, one per version, pointing at the
+corresponding `modulefile`. Users put only the *modulefiles* directories on their
+`MODULEPATH`, making each Module visible to `module avail` and `module load`.
+
+Deprecation is therefore cheap and reversible: the built files never move, only the symlink
+moves between `modulefiles/` and `deprecated/modulefiles/`. See
+[the release lifecycle](deprecation-lifecycle.md) for the full set of transitions.
+
+## The build area
+
+Modules are first assembled under the **build area** (`build/` by default), then moved into
+the Modules Area. The move is a filesystem `rename`, which is atomic *only when source and
+destination are on the same filesystem* — hence the default build area sits inside the
+deployment root. Building and deploying separately means a half-built Module is never
+visible: either the rename succeeds and the whole version appears, or nothing changes.
+
+The transient `build/` directory and the large Apptainer `sif_files/` images are
+deliberately kept out of the deployment area's git history (see
+[snapshots and the compare safety net](snapshots-and-compare.md)). They are reference-only
+data used by the `compare` command, not something to revert to.
+
+## Why modulefiles embed absolute paths
+
+A generated `modulefile` adds its Module's executables to the user's path with one line:
+
+```tcl
+prepend-path PATH "<deployment root>/modules/<name>/<version>/entrypoints"
+```
+
+That path is **absolute**, so a deployment area cannot be relocated after the fact.
+This is a deliberate trade-off.
+
+A path relative to the modulefile does not work. Environment Modules sources the file
+*through the symlink* under `modulefiles/` without resolving it first (`[info script]`
+/ `ModulesCurrentModulefile` report the link path). A relative path would be interpreted
+relative to the *link's* location — which changes when a Module is deprecated and the link
+moves to a different location in the tree, silently breaking it.
+
+This allows us to use two separate `modulefiles/` folders, corresponding to live and
+deprecated modules, without needing to edit modulefiles after initial creation.

docs/explanations/deprecation-lifecycle.md

@@ -0,0 +1,41 @@
+# The release lifecycle
+
+A *Release* is a Module version plus one boolean: whether it is `deprecated`. The whole
+lifecycle of a deployed Module — appearing, being updated, retired, restored, and finally
+deleted — is expressed by adding or removing a Release in configuration and toggling its
+`deprecated` flag. This page explains these transitions and the guard rails around them.
+
+## Configuration is declarative
+
+You never tell `deploy-tools` to "deprecate" or "remove" something directly. You describe
+the set of Releases you want, and the tool compares that against the
+[snapshot](snapshots-and-compare.md) of the last `sync` to derive the actions needed. Each
+Release falls into one of these cases:
+
+| Transition | Detected when… | Effect on the deployment area |
+|------------|----------------|-------------------------------|
+| **Add** | the name or version is not in the last snapshot | build the Module, create a live modulefile link |
+| **Update** | the version exists but its Module config changed | rebuild in place (only if `allow_updates` is set) |
+| **Deprecate** | the version was live in the last snapshot, now `deprecated: true` | move the modulefile link into `deprecated/modulefiles/` |
+| **Restore** | was deprecated in the last snapshot, now `deprecated: false` | move the link back into `modulefiles/` |
+| **Remove** | present in the last snapshot, absent from configuration | delete the modulefile link and the built files |
+
+Deprecate and restore only move a symlink; the built files stay in the Modules Area (see
+[the deployment area](deployment-area.md)). If your `MODULEPATH` includes
+`deprecated/modulefiles`, a deprecated Module still loads as before — deprecation indicates
+"you should stop using this", not "this is gone".
+
+## The guard rails
+
+Some transitions are blocked by default so mistakes aren't applied silently:
+
+- **A new version may not be born deprecated.** Adding a Release with `deprecated: true`
+  is rejected.
+- **An existing version may not change without a new version.** If its configuration differs
+  from the last snapshot, validation fails unless the Module sets `allow_updates: true`.
+- **A live version may not be deleted directly.** Deprecate it in one `sync`, then
+  remove it in a later one — unless the Module sets `allow_updates: true`.
+
+`--allow-all` lifts the first and third guards, so you can force those transitions.
+`--from-scratch` implies `--allow-all`. Neither relaxes the integrity checks — the area
+must still be healthy.