Skip to content

CLAUDE.md

Latest commit

 

History

History
78 lines (66 loc) · 4.06 KB

CLAUDE.md

File metadata and controls

78 lines (66 loc) · 4.06 KB

ShapePipe

ShapePipe is a galaxy shape-measurement pipeline for weak-lensing cosmology. It runs the full chain from raw survey images to calibrated shear catalogues — object detection, PSF modelling, and shape measurement — and was used to produce the first UNIONS cosmic-shear release.

The project is now entering a substantial rework of the shape-measurement pipeline, with the near-term goal of a tight loop between the pipeline and image simulations for validation and calibration. Development is by a small team; reproducibility and clarity matter more than breadth.

Environment — the container is the source of truth

ShapePipe is not a stand-alone library: it needs system tools (Source Extractor, PSFEx, WeightWatcher), MPI, and a specific scientific-Python stack. The supported way to get all of that is the container.

  • Dependencies are declared in pyproject.toml — abstract minimums, the single source of truth for what ShapePipe needs — and pinned exactly in uv.lock, the reproducible manifest generated by uv (never hand-edited). System-level tools live in the Dockerfile.
  • Images publish to ghcr.io/cosmostat/shapepipe on every push to an integration branch, in two targets:
    • :<tag> — the dev image: the full stack plus interactive tooling and the test / lint / doc extras.
    • :<tag>-runtime — a slim image for batch jobs and downstream FROM clauses.
  • On a cluster, run with Apptainer: 
    apptainer build --sandbox shapepipe docker://ghcr.io/cosmostat/shapepipe:develop
apptainer shell --writable shapepipe
cd /app && shapepipe_run -c /app/example/config.ini
  • For development, work inside the dev image — it carries vim, ripgrep, pytest, and the rest. A common pattern is a long-lived --writable Apptainer sandbox with a host clone of the repo bind-mounted in and pip install -e pointed at it, so edits on the host are live inside the container.

Full detail: docs/source/installation.md and docs/source/container.md.

Layout

  • src/shapepipe/ — the package (src-layout). modules/ holds the pipeline modules and their *_runner.py wrappers; pipeline/ is execution and file I/O; utilities/; canfar/ is CANFAR/cluster job orchestration. Console entry points (shapepipe_run, summary_run, canfar_*) are defined under [project.scripts].
  • src/shapepipe/tests/ — unit tests; tests/unit/ holds newer test scaffolding.
  • example/ — a runnable example pipeline (example/config.ini) on a single CFIS tile; doubles as the CI smoke test.
  • scripts/ — shell / Python / notebook helpers (sh/, python/, jupyter/), symlinked onto $PATH inside the image.
  • docs/ — Sphinx sources; the API docs are generated from docstrings.

Development workflow

  • develop is the integration branch — open PRs against it. main / master are release branches.
  • Tests run with pytest. CI runs them inside the dev image, so the suite exercises exactly what ships — run them the same way, in the dev container.
  • CI (.github/workflows/deploy-image.yml): every PR and push builds the image and runs the test suite + the example pipeline + binary smokes; pushes to develop / main / master additionally publish the images. API docs deploy from master (cd.yml).
  • Style: PEP 8; numpydoc docstrings on public modules, classes, and methods. Match the surrounding code.
  • ngmix is pinned to a fork branch (aguinot/ngmix@stable_version) until the fixes land upstream — don't bump that dependency line.

Project knowledge — felt

This repo carries a version-controlled .felt/ store: the team's shared notes on decisions, architecture, and plans, written as markdown "fibers" (readable directly; the felt CLI indexes and searches them). It's where the why lives, for humans and AI agents alike. Start at the shapepipe root fiber; the container/CI architecture and the in-progress test-suite work each have their own. When you make a durable decision or learn something worth keeping, leave a fiber so the next person finds it.