Fix MPI on candide (OpenMPI 5 image + latent code bug); containerize & SLURM-ify candide scripts

.felt/docker-uv-revert/docker-uv-revert.md

@@ -6,11 +6,11 @@ tags:
     - docker
     - infra
 created-at: 2026-04-27T11:26:45.677512058+02:00
-outcome: 'PR #719 (chore: switch Dockerfile to slim Python + uv lockfile) opened and CI-green on first try (3m31s); ready for Martin''s review. Drops conda double-install, makes pyproject SSOT + uv.lock the pinned manifest, switches WeightWatcher from sed-patched source build to Debian''s pre-patched 1.12+dfsg-3 package, adds binary smoke tests to deploy-image.yml.'
+outcome: 'PR #719 (chore: switch Dockerfile to slim Python + uv lockfile) opened and CI-green on first try (3m31s); ready for review. Drops conda double-install, makes pyproject SSOT + uv.lock the pinned manifest, switches WeightWatcher from sed-patched source build to Debian''s pre-patched 1.12+dfsg-3 package, adds binary smoke tests to deploy-image.yml.'
 decisions:
     base:
         label: Base image
-        rationale: Conda double-install was the actual problem; cleanest resolution is to drop conda entirely. Martin's canfar concern is satisfied as long as the slim image works on canfar.
+        rationale: Conda double-install was the actual problem; cleanest resolution is to drop conda entirely. The canfar deployment concern is satisfied as long as the slim image works on canfar.
         default: python-slim
         options:
             python-slim:
@@ -50,15 +50,15 @@ decisions:
                 label: uv + pyproject + uv.lock; uv sync --frozen in Dockerfile
     modernize:
         label: Modernize package versions
-        rationale: 'We determined which versions MUST stay pinned: only ngmix (Axel''s stable_version branch — replacement is tracked separately). Everything else can move to current latest because uv resolved cleanly and CI smoke test still passes (3m42s). If a real pipeline run on canfar surfaces a numpy-2 / pandas-3 break, the fix is a targeted constraint + uv lock, not a wholesale revert.'
+        rationale: 'We determined which versions MUST stay pinned: only ngmix (pinned to a stable_version fork branch — replacement is tracked separately). Everything else can move to current latest because uv resolved cleanly and CI smoke test still passes (3m42s). If a real pipeline run on canfar surfaces a numpy-2 / pandas-3 break, the fix is a targeted constraint + uv lock, not a wholesale revert.'
         default: stay-current
         options:
             stay-conservative:
                 label: Keep pre-v2 minimums (numpy 1.26, astropy 6.1, pandas 2.2); only bump when forced
                 excluded: true
                 excluded_reason: Drift between pyproject signal and lockfile reality; loses the chance to surface numpy-2/pandas-3 incompatibilities at PR time when CI is fast
             stay-current:
-                label: Bump pyproject minimums to current major versions (numpy 2, astropy 7, pandas 3, galsim 2.8, mpi4py 4.1, etc.); pin ngmix to Axel's stable_version branch
+                label: Bump pyproject minimums to current major versions (numpy 2, astropy 7, pandas 3, galsim 2.8, mpi4py 4.1, etc.); pin ngmix to its stable_version fork branch
 insights:
     ci-fast:
         claim: 'First CI run on PR #719 went green in 3m31s. uv installed 238 packages in 322ms — everything resolved to prebuilt wheels, no source compilation of galsim/mpi4py/python-pysap/etc. Massive speedup vs. previous build.'
@@ -97,11 +97,10 @@ The `--frozen` flag is the discipline mechanism: a stale lockfile cannot ship.
 ## Followups
 
 - Watch CI on #719. The slim-base apt list is conjectural — galsim/mpi4py/python-pysap pull a lot of system deps and we may need to add more (`libatlas-base-dev`, `libblas-dev`, etc).
-- If CI needs anything beyond what's in the apt block, that's the surface that benefits from a [[shapepipe/prs-in-flight]] note for next time.
-- After this lands, [[shapepipe/prs-in-flight]] PRs #708 and #714 may need a small rebase.
-- Optional: separate `Dockerfile.canfar` building on skaha if there's a concrete deployment reason. Currently conjectural — Martin floated it but we agreed slim should work on canfar.
+- If CI needs anything beyond what's in the apt block, that's worth noting for next time.
+- After this lands, PRs #708 and #714 may need a small rebase.
+- Optional: separate `Dockerfile.canfar` building on skaha if there's a concrete deployment reason. Currently conjectural — floated as a possibility, but slim should work on canfar.
 
 ## Connections
 
 - [[shapepipe]] — root
-- [[shapepipe/prs-in-flight]] — touches the testing-scaffold xfail set and the develop-bugs PR

.felt/ngmix-update/ngmix-update.md

@@ -1,9 +1,9 @@
 ---
-name: ngmix library upgrade + Lucy wrapper sync
+name: ngmix library upgrade + wrapper sync
 tags:
     - shapepipe
     - ngmix
     - future
 created-at: 2026-04-27T11:26:51.026191639+02:00
-outcome: 'Future: replace Axel''s stable_version fork with upstream ngmix; reconcile with Lucy''s cleaned-up wrapper from her visit'
+outcome: 'Replace the pinned ngmix fork (a stable_version branch carrying not-yet-upstreamed fixes) with upstream ngmix once those land; reconcile the wrapper afterward.'
 ---

.felt/shapepipe.md

@@ -1,50 +1,40 @@
 ---
-name: ShapePipe maintenance & PRs
+name: ShapePipe — project knowledge & active threads
 tags:
     - shapepipe
-    - portolan
 created-at: 2026-04-27T11:26:38.71538657+02:00
-outcome: 'Root: collaboration with Martin on ShapePipe — PRs, infra, future ngmix and Fabian work'
+outcome: 'Root of ShapePipe''s felt store: the stack division, repo conventions, and the why behind in-flight infra/cleanup threads.'
 ---
 
-ShapePipe is the UNIONS shape-measurement pipeline. I'm not the primary
-maintainer (that's Martin Kilbinger); my role is collaborator helping
-clean up infra, surface bugs, and keep the merge queue moving while
-Martin focuses on science threads.
+This is the root of ShapePipe's felt store — shared notes on architecture
+decisions, conventions, and in-flight work, for the team and AI agents alike.
+ShapePipe is the UNIONS galaxy shape-measurement pipeline; `CLAUDE.md` covers the
+build / container / CI overview, and the fibers here carry the *why*. Start here,
+then follow the links.
 
-## Working agreement with Martin
+## Stack division
 
-Surfaced over a 2026-04-27 walking conversation. Captured in
-[[shapepipe/prs-in-flight]] and the per-thread fibers below.
+ShapePipe **produces** shear catalogues; `sp_validation` / `cosmo_val`
+**consume** and validate them; `cs_util` holds code shared across both. A concern
+about *validating* catalogues belongs downstream, not in ShapePipe.
 
-- I review and patch his PRs; he reviews mine. Bugs found during review
-  go to a dedicated PR rather than getting bundled into his feature
-  branch (per `feedback_separate_infra_prs`).
-- v2.0 was merged fast (it was ready). The skaha base it brought in is
-  the active source of pain → see [[shapepipe/docker-uv-revert]].
-- I file the issues; Claude usually drafts the PRs in my voice.
-  Disclosure on Claude-only review per
-  `feedback_claude_only_review_disclosure`.
-
-## Active threads
-
-- **[[shapepipe/docker-uv-revert]]** — slim Python + uv lockfile, drop conda. PR #719 (draft).
-- **[[shapepipe/prs-in-flight]]** — tracking #708 (testing scaffold), #714 (develop bugs), #719 (this one).
-
-## Future work
+## Conventions specific to this repo
 
-- **[[shapepipe/ngmix-update]]** — replace Axel's stable_version fork
-  with upstream ngmix; reconcile with Lucy's wrapper.
-- **[[shapepipe/fabian-coord-bug]]** — port Fabian's 1-line coord
-  propagation fix; first need his image-sim code on github.
+- **Rho-statistics are obsolete inside ShapePipe.** PSF-systematics validation
+  moved downstream to `sp_validation` / `cosmo_val` (via `shear_psf_leakage`);
+  the stile/treecorr rho code was removed in #715. But the **meanshapes /
+  ellipticity focal-plane plots** (`mccd_plots_runner`) are *deliberately kept* —
+  they are a general PSF/star-catalogue diagnostic, not rho-stats, and feed
+  catalogue-paper figures. Don't delete that path along with rho-stats; see
+  [[shapepipe/cleanup-rhostats-jobscripts]] for where the boundary actually sits.
+- Run the pipeline through the container; use `python3.12` explicitly inside it.
+- **ngmix** is pinned to a fork branch until fixes land upstream — don't bump
+  that dependency line. [[ngmix-update]] tracks the path back to upstream.
 
-## Conventions specific to this repo
+## Active threads
 
-- Container runs through `app` (apptainer wrapper); use `python3.12`
-  inside the shapepipe container (see `reference_containers`).
-- ShapePipe produces; `sp_validation` consumes; `cs_util` is shared (see
-  `project_stack_division`).
-- Rho stats are obsolete here — sp_validation/cosmo_val took over (see
-  `project_rho_stats_obsolete`).
-- Royal "we" in PR/issue voice; specific findings attributed to Claude
-  by name (see `feedback_writing_voice_on_cails_behalf`).
+- **[[shapepipe/ci-green-on-develop]]** / **[[shapepipe/test-suite]]** — a
+  tiered, in-image test suite and trustworthy CI on `develop`.
+- **[[docker-uv-revert]]** — slim Python base + uv lockfile, dropping conda.
+- **[[shapepipe/mpi-hybrid]]** — running hybrid MPI through the container on candide.
+- **[[ngmix-update]]** — replacing the pinned ngmix fork with upstream.

.felt/shapepipe/ci-develop-trigger/ci-develop-trigger.md

@@ -64,7 +64,7 @@ just CI. Deserves its own issue; #732 doesn't touch it.
 
 ## Knock-on
 
-[[shapepipe/prs-in-flight]]: **#729** (actions group, bumps `setup-miniconda`
+**#729** (actions group, bumps `setup-miniconda`
 v3→v4) hit the layer-1 failure too — confirming the action bump alone
 doesn't fix the path. #729 must rebase on top of #732 once it merges before
 it can go green. The smoke-test work in [[shapepipe/smoke-test-read-only]]