feat(lab): pd-cold R2 archive — auto-push finished runs' decomposition item + cross-cluster pull

[claude-spd1]

Stacked on #950 (needs the decomposition|training checkpoint split). Retarget to feature/jax after #950 merges.

What

param_decomp_lab/tools/r2_cold.py — pd-cold R2 archive for finished decomposition runs:

• push <run_dir>: uploads ckpts/<latest step>/decomposition + _CHECKPOINT_METADATA + launch_config.yaml + a provenance manifest.json to the no-expiry archive bucket under <user>/pd-cold/<run_id>/. Never the resume-only training item (~90% of ckpt bytes). Strict completeness gate (latest step == pd.steps); --allow-incomplete for archiving abandoned / strip_checkpoint-migrated runs — this also serves the bulk migration of pre-split runs.
• pull <run_id> [--user] [--dest]: fetches the archive onto any cluster as a consumer-ready run dir ($PARAM_DECOMP_OUT_DIR/runs/<run_id>/: launch_config.yaml + ckpts/<step>/decomposition, the shape restore_decomposition* expects). Owner search across user prefixes when --user is omitted; refuses to overlay a resumable local training run.
• Both directions are aws s3 sync: idempotent, resume-on-rerun. manifest.json uploads LAST as the upload-complete marker; pull asserts it first. Post-upload verification compares remote object count/bytes against local.
• Drives the cluster scripts' r2_* bash functions ($DATA_MOUNT/scripts/{config.sh,r2.sh}) directly — bin/r2 push --name rejects the nested pd-cold/<run_id>/... prefix (^[a-zA-Z0-9._-]+$), and the functions give the same creds + tuned multipart config.

Edge wiring (param_decomp_lab/experiments/lm/run.py): rank 0, after the multihost teardown (so no rank waits on a barrier behind the upload), archives iff the final-step checkpoint exists — a SIGTERM/requeue save or profiling run never fires it — then stamps wandb.config["r2_cold"] = {bucket, prefix, step} so every finished run points at its durable artifact (restores the torch-era model_<step>.pth discoverability contract the JAX migration dropped). Upload failure is LOUD but non-fatal: training itself succeeded; the printed retry command re-runs the same idempotent sync. The engine stays pure — completion is read off the filesystem witness (engine always saves at now_step == pd.steps).

Verified

• 8 unit tests (completeness gating, never-upload-training, manifest-last ordering, remote-verification mismatch, manifest-exclusion in re-push verification, live-run overlay refusal, pull shape, edge-hook skip/non-fatality). Full param_decomp_lab suite: 194 passed. ruff + basedpyright clean.
• Probed on cw-rno2: aws CLI v2 + sbatch + /mnt/data mount present on both h100 and cpu-compute nodes → the inline rank-0 push works from compute nodes. Bash plumbing smoke-tested up to the creds boundary.

Known gaps / follow-ups

• Creds: r2-credentials.sh is goodfire-group readable; pd-user (crew) is not in that group on cw-rno2, so crew-launched runs can't push (Oli's can). Needs group membership or the scoped pd-cold token discussed on the task.
• Real push throughput probe still pending (needs a goodfire-group user).
• andromeda/bitdeer compute-node aws availability unprobed (graceful loud failure + retry command if absent).

Task: bridge r2-cold-storage-auto-upload
Crew-Address: task/r2-cold-storage-auto-upload