feat: add experiment store with registration and Confluence publishing

.gitignore

@@ -17,6 +17,8 @@ _dev
 output*
 _sandbox/
 experiment_report.html
+.local
+.claude
 
 # Jupyter Notebook checkpoints
 .ipynb_checkpoints/

config/forecasters-ich1-oper-fixed.yaml

@@ -13,8 +13,8 @@ dates:
 
 runs:
   - forecaster:
-      checkpoint: https://service.meteoswiss.ch/mlstore#/experiments/602/runs/9f4626a7562a4eb49700aaaaa8607230
-      label: icon_1km_hierarchical_2_level_proc_from_scratch
+      checkpoint: amber-ridge
+      label: amber-ridge
       steps: 0/120/6
       config: resources/inference/configs/sgm-multidataset-forecaster-global-ich1-oper.yaml
       extra_requirements:

config/forecasters-ich1-oper.yaml

@@ -13,8 +13,8 @@ dates:
 
 runs:
   - forecaster:
-      checkpoint: https://servicedepl.meteoswiss.ch/mlstore#/experiments/409/runs/b30acf68520a4bbd8324c44666561696
-      label: stage_C_icon_1km
+      checkpoint: amber-ridge
+      label: amber-ridge
       steps: 0/120/6
       config: resources/inference/configs/sgm-forecaster-global-ich1-oper.yaml
       extra_requirements:

docs/experiment-store.md

@@ -0,0 +1,116 @@
+# The experiment store
+
+Registering promotes a finished evaluation from the working directory into a shared,
+immutable archive at `/store_new/mch/msopr/ml/experiments`, cross-referenced with
+devml's model store at `/store_new/mch/msopr/ml/models`. The design mirrors devml's
+model registration: one owned directory per registration, an append-only JSONL index,
+tombstones that retire names forever, and a Confluence page that is a copy of the
+index — never the truth.
+
+## Commands
+
+```
+evalml register RESULTS_DIR [NAME]    # promote results/<experiment> into the store
+evalml unregister NAME                # take a registration out (tombstones the name)
+evalml list [--json] [--rebuild] [--columns ...]
+evalml publish [--dry-run]            # write the index to the Confluence page
+```
+
+All accept `--store DIR` (and, where relevant, `--models-store DIR`) so tests run
+against temp directories. Register, unregister, and publish have `--dry-run`;
+`list --rebuild` does not — it rewrites the index and recreates missing model-store
+symlinks whenever it runs.
+
+`RESULTS_DIR` is the workflow's `<output_root>/results/{date}_{config-label}_{confighash}`.
+It must contain `config.yaml` (the `experiment_config` rule copies it there); for older
+results, pass `--config`. `NAME` is a slug (`^[a-z0-9][a-z0-9-]*$`), defaulting to
+`{config-label-slug}-{date}`. It is the identity everywhere — directory, index key,
+symlink, Confluence anchor — and is **never reused**: unregistering leaves an
+`.orphaned-<name>` tombstone.
+
+## Layout and atomicity
+
+```
+<store>/                    # 3775: anyone in the group adds, only the owner removes
+  index.jsonl
+  <experiment-name>/        # 0755, owned by whoever registered it
+    experiment.json         # written once, then 0444
+    results/                # copied from the WD, files read-only
+```
+
+Registration builds everything in `<store>/.tmp.<name>` and renames it into place; a
+crash leaves a `.tmp.` directory to sweep, never a half-registered experiment.
+
+## experiment.json — the contract
+
+`name`, `models`, and `baselines` are the contract; the rest is provenance.
+
+```json
+{
+  "name": "forecasters-ich1-oper-20260824",
+  "registered": "2026-08-24T10:00:00+02:00",
+  "by": "fzanetta",
+  "description": "…the config's description…",
+  "models": ["amber-ridge"],
+  "baselines": ["ICON-CH1"],
+  "evalml": {"version": "…", "commit": "…", "dirty": false, "config": "results/config.yaml"},
+  "identity": "20260824_forecasters-ich1-oper_ab12"
+}
+```
+
+- `models`: **registered model-store names only** (stable foreign keys), detected from
+  the config's `checkpoint` entries — a bare name or a path under the model store both
+  count; MLflow/HF URLs and plain paths are not cross-referenced. Each listed name must
+  exist as `<models-store>/<name>/model.json`, or registration refuses.
+- `baselines`: descriptive free text, no existence check.
+- `identity`: the results directory's basename; what "already registered" is judged by,
+  always against the store itself.
+- `evalml.commit`/`dirty` describe the evalml tree **at registration time** (not
+  necessarily the run's) — recorded, never gated on.
+
+## Cross-references
+
+The truth of the model↔experiment edge is `experiment.json`'s `models` list. The
+reverse direction is materialized as relative symlinks, created at registration:
+
+```
+<models-store>/<model>/experiments/<name> -> ../../../experiments/<name>
+```
+
+Symlink failures warn and carry on (`list --rebuild` recreates missing ones); dangling
+symlinks are history — reported, never deleted. Nothing else is ever written into the
+model store. The Models page on Confluence renders these symlinks but is owned by
+devml's publisher: when evalml changes a model's links it prints a reminder that the
+page is stale (`just model-publish` in devml refreshes it).
+
+## The index
+
+`index.jsonl`: one JSON object per line, one line per register/unregister event,
+append-only; replaying it yields the inventory. **The store is the truth; the index is
+derived** — a failed append warns, and `list --rebuild` reconstructs it from the
+experiment directories. `list` cross-checks store and index both ways and reports
+disagreements.
+
+## Unregistering
+
+Only the owner can unregister. Destructive step last: rename to `.orphaned-<name>`,
+write `NOTE.txt`, append the index line, remove this experiment's model-store symlinks,
+and only then delete the copied results (`--keep-results` keeps them).
+
+## Confluence
+
+`evalml publish` renders the index as a table on the
+[Experiments page](https://meteoswiss.atlassian.net/wiki/spaces/MR/pages/2142666865/Experiments)
+(`EXPERIMENTS_PAGE` in `src/evalml/store.py`). The tool owns only its marker-delimited
+section (`<!-- evalml:experiment-index:start/end -->`), located by its "Registered
+experiments" heading if Confluence strips the comments, appended if neither is found.
+Name cells carry anchor macros (bare `#<name>` fragments resolve — verified 2026-08-25
+against the rendered HTML); the Models column links to devml's Models page (id
+`2139488787`), whose `#<model-name>` fragments start landing once devml's publisher
+adds anchors there.
+
+Credentials, exactly like devml: a token in `~/.atlassian-token` (bare or
+`email:token`), or `$ATLASSIAN_TOKEN`/`$ATLASSIAN_EMAIL`, with `git config user.email`
+as the fallback address (`evalml publish --email` overrides it). The token is never
+printed or logged. Register and unregister publish at the end; a publish failure only
+warns, and publish refuses to touch the real page from any non-default `--store`.

src/evalml/cli.py

@@ -288,6 +288,151 @@ def sandbox(
     )
 
 
+def _store_options(func):
+    """Options shared by every experiment-store command, so tests (and the wary) can run
+    against temp directories instead of the shared stores."""
+    from evalml import store as experiment_store
+
+    func = click.option(
+        "--store",
+        "store_dir",
+        type=click.Path(path_type=Path),
+        default=experiment_store.STORE,
+        show_default=True,
+        help="Experiment store directory.",
+    )(func)
+    func = click.option(
+        "--models-store",
+        type=click.Path(path_type=Path),
+        default=experiment_store.MODELS_STORE,
+        show_default=True,
+        help="Model store directory (devml's).",
+    )(func)
+    return func
+
+
+@cli.command(help="Register a finished experiment's results into the experiment store.")
+@click.argument(
+    "results_dir", type=click.Path(exists=True, file_okay=False, path_type=Path)
+)
+@click.argument("name", required=False, default=None)
+@click.option(
+    "--config",
+    "config_path",
+    type=click.Path(exists=True, dir_okay=False, path_type=Path),
+    default=None,
+    help="The config YAML that produced the run, for results that do not contain config.yaml.",
+)
+@click.option(
+    "--dry-run",
+    "-n",
+    is_flag=True,
+    help="Run the checks and print the plan, copy nothing.",
+)
+@click.option(
+    "--no-publish", is_flag=True, help="Do not update the store's Confluence page."
+)
+@_store_options
+def register(
+    results_dir, name, config_path, dry_run, no_publish, store_dir, models_store
+):
+    from evalml import registration
+
+    print(
+        registration.register_results(
+            results_dir,
+            name=name,
+            config_path=config_path,
+            store_dir=store_dir,
+            models_store=models_store,
+            dry_run=dry_run,
+            no_publish=no_publish,
+        )
+    )
+
+
+@cli.command(help="Take a registered experiment back out of the experiment store.")
+@click.argument("name")
+@click.option(
+    "--reason", default="", help="Why, recorded in NOTE.txt and in the index."
+)
+@click.option("--dry-run", "-n", is_flag=True, help="Print the plan, then stop.")
+@click.option("--yes", "-y", is_flag=True, help="Skip the confirmation prompt.")
+@click.option(
+    "--keep-results",
+    is_flag=True,
+    help="Tombstone the directory but keep its contents.",
+)
+@click.option(
+    "--no-publish", is_flag=True, help="Do not update the store's Confluence page."
+)
+@_store_options
+def unregister(
+    name, reason, dry_run, yes, keep_results, no_publish, store_dir, models_store
+):
+    from evalml import store as experiment_store
+
+    experiment_store.cmd_unregister(
+        name,
+        store=store_dir,
+        models_store=models_store,
+        reason=reason,
+        dry_run=dry_run,
+        yes=yes,
+        keep_results=keep_results,
+        no_publish=no_publish,
+    )
+
+
+@cli.command("list", help="List the experiments registered in the experiment store.")
+@click.option("--json", "as_json", is_flag=True, help="Print the inventory as JSON.")
+@click.option(
+    "--rebuild",
+    is_flag=True,
+    help="Rewrite the index from the experiment directories and repair missing symlinks.",
+)
+@click.option("--columns", default=None, help="Comma-separated columns to show.")
+@_store_options
+def list_experiments(as_json, rebuild, columns, store_dir, models_store):
+    from evalml import store as experiment_store
+
+    experiment_store.cmd_list(
+        store=store_dir,
+        models_store=models_store,
+        as_json=as_json,
+        rebuild_index=rebuild,
+        columns=columns or experiment_store.DEFAULT_COLUMNS,
+    )
+
+
+@cli.command(help="Publish the experiment store's index to its Confluence page.")
+@click.option(
+    "--page", default=None, help="Page id or URL (default: the Experiments page)."
+)
+@click.option(
+    "--email", default="", help="The Atlassian account the API token belongs to."
+)
+@click.option(
+    "--dry-run", "-n", is_flag=True, help="Print what would be published, send nothing."
+)
+@click.option(
+    "--store",
+    "store_dir",
+    type=click.Path(path_type=Path),
+    default=None,
+    help="Experiment store directory (default: the shared store).",
+)
+def publish(page, email, dry_run, store_dir):
+    from evalml import store as experiment_store
+
+    experiment_store.cmd_publish(
+        store=store_dir if store_dir is not None else experiment_store.STORE,
+        page=page or experiment_store.EXPERIMENTS_PAGE,
+        email=email,
+        dry_run=dry_run,
+    )
+
+
 @cli.command(help="Make a specific file from a workflow defined in the YAML file.")
 @click.argument(
     "configfile", type=click.Path(exists=True, dir_okay=False, path_type=Path)

src/evalml/config.py

@@ -1,3 +1,4 @@
+import re
 from math import isfinite
 from pathlib import Path
 from typing import Dict, List, Any, ClassVar, FrozenSet, Optional, Union
@@ -6,6 +7,9 @@
 
 PROJECT_ROOT = Path(__file__).parents[2]
 
+MODEL_REGISTRY_ROOT = Path("/store_new/mch/msopr/ml/models")
+REGISTERED_MODEL_PATTERN = re.compile(r"^[a-z]+-[a-z]+$")
+
 PREDEFINED_REGIONS: Dict[str, List[float]] = {
     "global": [-180, 180, -90, 90],
     "icon": [1.5, 16, 43, 49.5],
@@ -77,7 +81,9 @@ class RunConfig(BaseModel):
     )
     checkpoint: str = Field(
         ...,
-        description="The model checkpoint to use. Can be an MLflow run URL, a Hugging Face `.ckpt` URL, or a local checkpoint path.",
+        description="The model checkpoint to use. Can be an MLflow run URL, a Hugging Face `.ckpt` URL, "
+        "a local checkpoint path, or a registered model name (e.g. 'amber-ridge') resolved "
+        f"against {MODEL_REGISTRY_ROOT}.",
     )
     label: str | None = Field(
         None,
@@ -111,6 +117,23 @@ class RunConfig(BaseModel):
 
     model_config = {"extra": "forbid"}
 
+    @field_validator("checkpoint")
+    def resolve_registered_model(cls, v: str) -> str:
+        """Resolve a registered model name (e.g. 'amber-ridge') to its checkpoint path."""
+        if not REGISTERED_MODEL_PATTERN.match(v):
+            return v
+        model_dir = MODEL_REGISTRY_ROOT / v
+        if not model_dir.is_dir():
+            raise ValueError(
+                f"Registered model '{v}' not found in the model registry: {model_dir}"
+            )
+        checkpoint = model_dir / "inference-last.ckpt"
+        if not checkpoint.exists():
+            raise ValueError(
+                f"Registered model '{v}' has no 'inference-last.ckpt' in {model_dir}"
+            )
+        return str(checkpoint)
+
     @field_validator("steps")
     def validate_steps(cls, v: str) -> str:
         if "/" not in v: