From 3fe632bc025defc49e389754879719908f791ca6 Mon Sep 17 00:00:00 2001 From: Tal Ben-Nun Date: Tue, 25 Aug 2026 08:26:33 -0700 Subject: [PATCH] Replace setup.py/requirements*.txt with pyproject.toml Move all packaging metadata into a single PEP 621 pyproject.toml and drop setup.py, requirements.txt, and requirements-ci.txt. - Distribution name is declared as SpaDA; sdist/wheel filenames still normalize to `spada-0.1.0`, so `pip install spada` is unaffected. - Repository URLs point at the new home, https://github.com/spcl/spada. - Core dependencies are now just click, lark, networkx, and numpy. The previously declared `parse` is imported nowhere. `pydot` is only reachable through a commented-out nx_pydot debug dump in syntax/csl/tasks.py, so it moves to the `dev` extra rather than being required for every install. `pycairo` (which needs system cairo, and was the only difference between requirements.txt and requirements-ci.txt) moves to a `render` extra. - Extras: `placement` (igraph/matplotlib/hilbertcurve, used only by the standalone spada/placement research code and its tests), `render`, `docs`, `dev` (= placement + pydot + pytest/black/isort/flake8/yapf), and `all`. - Version stays dynamic from spada.__version__; license corrected to the actual BSD-3-Clause; package data now also ships the two .lark grammars, which the old setup.py package_data omitted. - Version pins relax to lower bounds; the suite passes on numpy 2.x and networkx 3.x, which the old `~=1.24.4` style pins forbade. - CI installs `-e ".[dev]"` instead of requirements-ci.txt, so pytest no longer needs PYTHONPATH and the CSL job drops its duplicate install. Adds a build-package job running `python -m build` + `twine check`. - Makefile, run-in-lima.sh, and README updated to the new install command. Co-Authored-By: Claude Opus 5 --- .github/workflows/python-app.yml | 31 +++++-- README.md | 37 ++++---- pyproject.toml | 95 ++++++++++++++++++++ requirements-ci.txt | 14 --- requirements.txt | 15 ---- setup.py | 103 ---------------------- tests/csl_runtime/Makefile | 5 +- tests/csl_runtime/lima-ubuntu-x86_64.yaml | 4 +- tests/csl_runtime/run-in-lima.sh | 3 +- 9 files changed, 146 insertions(+), 161 deletions(-) create mode 100644 pyproject.toml delete mode 100644 requirements-ci.txt delete mode 100644 requirements.txt delete mode 100644 setup.py diff --git a/.github/workflows/python-app.yml b/.github/workflows/python-app.yml index 368b5f6a..da167b43 100644 --- a/.github/workflows/python-app.yml +++ b/.github/workflows/python-app.yml @@ -24,16 +24,16 @@ jobs: with: python-version: "3.10" cache: "pip" + cache-dependency-path: pyproject.toml - name: Install Python dependencies run: | python -m pip install --upgrade pip - pip install pytest - if [ -f requirements-ci.txt ]; then pip install -r requirements-ci.txt; fi + pip install -e ".[dev]" - name: Test with pytest run: | - PYTHONPATH=`pwd` pytest + pytest test-csl: runs-on: ubuntu-latest @@ -46,12 +46,12 @@ jobs: with: python-version: "3.10" cache: "pip" + cache-dependency-path: pyproject.toml - name: Install Python dependencies run: | python -m pip install --upgrade pip - pip install pytest - if [ -f requirements-ci.txt ]; then pip install -r requirements-ci.txt; fi + pip install -e ".[dev]" - name: Install CSL dependencies run: | @@ -91,6 +91,25 @@ jobs: - name: Test CSL with simulator run: | - pip install --no-deps -e . export PATH=$PATH:`pwd`/cerebras-sdk ./tests/csl_runtime/run_tests.sh + + build-package: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + + - name: Set up Python 3.10 + uses: actions/setup-python@v6 + with: + python-version: "3.10" + cache: "pip" + cache-dependency-path: pyproject.toml + + - name: Build sdist and wheel + run: | + python -m pip install --upgrade pip + pip install build twine + python -m build + twine check dist/* diff --git a/README.md b/README.md index b40ff77a..eb106154 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,14 @@ -# SPADA — A Spatial Dataflow Architecture Programming Language +# SpaDA — A Spatial Dataflow Architecture Programming Language -SPADA is a programming language and compiler for spatial dataflow architectures such as the [Cerebras Wafer-Scale Engine](https://www.cerebras.net/). It provides precise control over data placement, communication streams, and asynchronous execution while abstracting architecture-specific routing details. SPADA also serves as a compiler intermediate representation (IR) for domain-specific languages; this repository includes a complete end-to-end compilation pipeline from the [GT4Py](https://github.com/GridTools/gt4py) stencil DSL (used in production weather forecasting at CSCS/MeteoSwiss) to Cerebras CSL. +SpaDA is a programming language and compiler for spatial dataflow architectures such as the [Cerebras Wafer-Scale Engine](https://www.cerebras.net/). It provides precise control over data placement, communication streams, and asynchronous execution while abstracting architecture-specific routing details. SpaDA also serves as a compiler intermediate representation (IR) for domain-specific languages; this repository includes a complete end-to-end compilation pipeline from the [GT4Py](https://github.com/GridTools/gt4py) stencil DSL (used in production weather forecasting at CSCS/MeteoSwiss) to Cerebras CSL. -Spatial dataflow architectures achieve exceptional throughput through disaggregated memory: each processing element (PE) holds only fast local SRAM, eliminating cache hierarchies and shared-memory contention. However, programming these architectures demands explicit orchestration of data movement over a circuit-switched network-on-chip (NoC), with limited concurrent communication channels and asynchronous, data-triggered task execution. SPADA addresses this by offering high-level constructs—`place`, `dataflow`, and `compute` blocks; `async`/`await`; `foreach` and `map` loops—alongside a formal dataflow semantics that defines routing correctness, data races, and deadlocks at compile time. +Spatial dataflow architectures achieve exceptional throughput through disaggregated memory: each processing element (PE) holds only fast local SRAM, eliminating cache hierarchies and shared-memory contention. However, programming these architectures demands explicit orchestration of data movement over a circuit-switched network-on-chip (NoC), with limited concurrent communication channels and asynchronous, data-triggered task execution. SpaDA addresses this by offering high-level constructs—`place`, `dataflow`, and `compute` blocks; `async`/`await`; `foreach` and `map` loops—alongside a formal dataflow semantics that defines routing correctness, data races, and deadlocks at compile time. Key capabilities: - **Explicit placement and dataflow**: Declare where data lives and how it moves between PEs. - **Automatic routing assignment**: A checkerboard decomposition algorithm guarantees conflict-free channel allocation by construction, eliminating manual reasoning about hardware routing. -- **Multi-level compilation**: GT4Py stencils → Stencil IR → SPADA IR → Cerebras CSL, with automatic vectorization via Data Structure Descriptors (DSDs) and task fusion. -- **Compact code**: Hand-written SPADA kernels require 6–8× fewer lines than equivalent CSL; GT4Py stencils compile with up to 700× code reduction. +- **Multi-level compilation**: GT4Py stencils → Stencil IR → SpaDA IR → Cerebras CSL, with automatic vectorization via Data Structure Descriptors (DSDs) and task fusion. +- **Compact code**: Hand-written SpaDA kernels require 6–8× fewer lines than equivalent CSL; GT4Py stencils compile with up to 700× code reduction. - **Near-ideal weak scaling**: Compiler-generated stencil kernels achieve >150 TFlop/s on the WSE-2 with near-ideal weak scaling across three orders of magnitude. For full details, see the paper: @@ -21,7 +21,7 @@ For full details, see the paper: ### Prerequisites -- Python ≥ 3.8 +- Python ≥ 3.10 - [Cerebras SDK](https://sdk.cerebras.net/) (required to compile and run generated CSL code on WSE hardware; optional for compiler development) ### Installation @@ -29,20 +29,25 @@ For full details, see the paper: Clone the repository and install the package: ```bash -git clone https://github.com/glukas/spada.git +git clone https://github.com/spcl/spada.git cd spada pip install -e . ``` -To install with development dependencies: +To install with development dependencies (test suite, formatters, and the +standalone placement research code): ```bash pip install -e ".[dev]" ``` -### Compiling a SPADA Program +Other optional dependency groups: `docs` (MkDocs site in `irspec/`), `placement` +(igraph/matplotlib/hilbertcurve), `render` (pycairo, needs system cairo), and +`all`. -The `sptlc` command-line tool compiles a SPADA Spatial IR (`.sptl`) file to Cerebras CSL: +### Compiling a SpaDA Program + +The `sptlc` command-line tool compiles a SpaDA Spatial IR (`.sptl`) file to Cerebras CSL: ```bash sptlc samples/benchmarks/laplacian_128_128_80.sptl output/ --param I=128 --param J=128 @@ -61,7 +66,7 @@ Key options: ### Compiling from GT4Py -To compile a GT4Py stencil file to SPADA IR (`.spst` and `.sptl`): +To compile a GT4Py stencil file to SpaDA IR (`.spst` and `.sptl`): ```bash python -m spada.cli.gt4py_to_spatial samples/stencils.py 128,128,80 output/ --function-name laplacian @@ -94,7 +99,7 @@ The runtime reads `metadata.json` generated by `sptlc` to determine the PE grid ### Example Kernels -Sample SPADA programs are in `samples/`: +Sample SpaDA programs are in `samples/`: | Directory | Contents | |---|---| @@ -125,7 +130,7 @@ pytest tests/ --ignore=tests/csl_runtime ### CSL Runtime Tests (Singularity / Cerebras SDK) -End-to-end tests in `tests/csl_runtime/` compile and simulate SPADA programs using the Cerebras SDK and simulator. The Cerebras SDK ships as a Singularity Image File (`.sif`) and requires Singularity/Apptainer and an x86_64 Linux environment. Follow the Cerebras installation guide for full details: [Installation and Setup](https://sdk.cerebras.net/installation-guide). +End-to-end tests in `tests/csl_runtime/` compile and simulate SpaDA programs using the Cerebras SDK and simulator. The Cerebras SDK ships as a Singularity Image File (`.sif`) and requires Singularity/Apptainer and an x86_64 Linux environment. Follow the Cerebras installation guide for full details: [Installation and Setup](https://sdk.cerebras.net/installation-guide). **Linux or x86_64 VM setup** @@ -141,7 +146,7 @@ This saves the tarball to `tests/csl_runtime/cerebras-sdk.tar.gz` and extracts i 3. Install Python dependencies for the compiler: ```bash -python3 -m pip install -r requirements-ci.txt +python3 -m pip install -e ".[dev]" ``` 4. Verify the toolchain: @@ -217,7 +222,7 @@ make -C tests/csl_runtime clean-sdk # also remove the downloaded SDK Questions, discussions, and feedback are welcome via GitHub Issues: -- **Bug reports and feature requests**: [GitHub Issues](https://github.com/glukas/spada/issues) +- **Bug reports and feature requests**: [GitHub Issues](https://github.com/spcl/spada/issues) --- @@ -241,6 +246,6 @@ For significant changes (new language constructs, compiler passes, or architectu ## Release -SPADA is released under BSD-3-Clause License, see [LICENSE](LICENSE) for details. +SpaDA is released under BSD-3-Clause License, see [LICENSE](LICENSE) for details. LLNL-CODE-2000963 diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 00000000..efcc3e57 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,95 @@ +[build-system] +requires = ["setuptools>=77"] +build-backend = "setuptools.build_meta" + +[project] +name = "SpaDA" +description = "SpaDA: a spatial dataflow architecture programming language and compiler" +readme = "README.md" +requires-python = ">=3.10" +license = "BSD-3-Clause" +license-files = ["LICENSE", "NOTICE"] +authors = [ + { name = "Lukas Gianinazzi" }, + { name = "Tal Ben-Nun" }, + { name = "Torsten Hoefler" }, +] +keywords = [ + "stencil", + "compiler", + "spatial", + "dataflow", + "high-performance-computing", + "csl", + "cerebras", +] +classifiers = [ + "Development Status :: 3 - Alpha", + "Intended Audience :: Developers", + "Intended Audience :: Science/Research", + "Operating System :: OS Independent", + "Programming Language :: Python :: 3", + "Topic :: Scientific/Engineering", + "Topic :: Software Development :: Compilers", +] +dependencies = [ + "click>=8.0", + "lark>=1.3.1", + "networkx>=2.8", + "numpy>=1.24", +] +dynamic = ["version"] + +[project.urls] +Homepage = "https://github.com/spcl/spada" +Repository = "https://github.com/spcl/spada" +Paper = "https://arxiv.org/abs/2511.09447" + +[project.scripts] +sptlc = "spada.cli.compiler:compile_spatial_ir" + +[project.optional-dependencies] +# Standalone placement research code (spada/placement, tests/placement). +placement = [ + "hilbertcurve>=2.0.5", + "igraph>=0.11.4", + "matplotlib>=3.7", +] +# Graph rendering through igraph; needs the system cairo libraries. +render = ["pycairo>=1.26"] +docs = [ + "mkdocs>=1.5.3", + "mkdocs-material>=9.2.7", + "mkdocs-material-extensions>=1.2", + "pymdown-extensions>=10.2.1", +] +dev = [ + "SpaDA[placement]", + "black", + "flake8", + "isort", + # Backs networkx's nx_pydot dot export, used only by commented-out debug + # dumps of the completion DAG (spada/syntax/csl/tasks.py). + "pydot", + "pytest", + "pytest-cov", + "yapf", +] +all = ["SpaDA[dev,docs,placement,render]"] + +[tool.setuptools.dynamic] +version = { attr = "spada.__version__" } + +[tool.setuptools.packages.find] +include = ["spada*"] + +[tool.setuptools.package-data] +spada = ["**/*.lark", "assets/csl/sync/*.csl"] + +[tool.black] +line-length = 120 +target-version = ["py310"] + +[tool.isort] +profile = "black" +line_length = 120 diff --git a/requirements-ci.txt b/requirements-ci.txt deleted file mode 100644 index 0bc73869..00000000 --- a/requirements-ci.txt +++ /dev/null @@ -1,14 +0,0 @@ -lark==1.3.1 -parse==1.20.2 -pytest - -igraph~=0.11.4 -numpy~=1.24.4 -matplotlib~=3.7.5 -hilbertcurve~=2.0.5 -networkx~=2.8.8 - -mkdocs~=1.5.3 -mkdocs-material~=9.2.7 -mkdocs-material-extensions~=1.2 -pymdown-extensions~=10.2.1 diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index e9553366..00000000 --- a/requirements.txt +++ /dev/null @@ -1,15 +0,0 @@ -lark==1.3.1 -parse==1.20.2 -pytest - -igraph~=0.11.4 -numpy~=1.24.4 -matplotlib~=3.7.5 -hilbertcurve~=2.0.5 -pycairo~=1.26.1 -networkx~=2.8.8 - -mkdocs~=1.5.3 -mkdocs-material~=9.2.7 -mkdocs-material-extensions~=1.2 -pymdown-extensions~=10.2.1 diff --git a/setup.py b/setup.py deleted file mode 100644 index dc897eeb..00000000 --- a/setup.py +++ /dev/null @@ -1,103 +0,0 @@ -#!/usr/bin/env python3 -"""Setup script for spada package.""" - -from setuptools import setup, find_packages -import os - - -# Read the README file for long description -def read_readme(): - readme_path = os.path.join(os.path.dirname(__file__), 'README.md') - if os.path.exists(readme_path): - with open(readme_path, 'r', encoding='utf-8') as f: - return f.read() - return "A SpaDA compiler for high-performance computing." - - -# Read version from package -def get_version(): - """Get version from spada package.""" - try: - import spada - return spada.__version__ - except (ImportError, AttributeError): - return "0.1.0" - - -setup( - name="spada", - version=get_version(), - author="SpaDA Team", - author_email="", - description="A SpaDA compiler for high-performance computing", - long_description=read_readme(), - long_description_content_type="text/markdown", - url="https://github.com/glukas/spada", - packages=find_packages(), - classifiers=[ - "Development Status :: 3 - Alpha", - "Intended Audience :: Developers", - "Intended Audience :: Science/Research", - "License :: OSI Approved :: MIT License", - "Operating System :: OS Independent", - "Programming Language :: Python :: 3", - "Topic :: Scientific/Engineering", - "Topic :: Software Development :: Compilers", - ], - python_requires=">=3.10", - install_requires=[ - "numpy", - "pydot", - "networkx", - "lark", - "matplotlib", - "hilbertcurve", - "igraph", - "click", # Added for CLI functionality - "parse", # From requirements.txt - "pycairo", # For rendering graphs - ], - extras_require={ - "docs": [ - "mkdocs", - "mkdocs-material", - "mkdocs-material-extensions", - "pymdown-extensions", - ], - "dev": [ - "pytest", - "pytest-cov", - "black", - "isort", - "flake8", - ], - "all": [ - # Docs dependencies - "mkdocs", - "mkdocs-material", - "mkdocs-material-extensions", - "pymdown-extensions", - # Dev dependencies - "pytest", - "pytest-cov", - "black", - "isort", - "flake8", - ], - }, - entry_points={ - "console_scripts": ["sptlc=spada.cli.compiler:compile_spatial_ir"], - }, - include_package_data=True, - package_data={ - "spada": ["**/*.py", "assets/csl/sync/*.csl"], - }, - keywords=[ - "stencil", - "compiler", - "spatial", - "high-performance-computing", - "csl", - "cerebras", - ], -) diff --git a/tests/csl_runtime/Makefile b/tests/csl_runtime/Makefile index 778841a4..34499e77 100644 --- a/tests/csl_runtime/Makefile +++ b/tests/csl_runtime/Makefile @@ -93,10 +93,9 @@ check-host: .PHONY: check-python check-python: - @python3 -m pip install --quiet -r "$(REPO_ROOT)/requirements-ci.txt" && \ - python3 -m pip install --no-deps --quiet -e "$(REPO_ROOT)" || { \ + @python3 -m pip install --quiet -e "$(REPO_ROOT)[dev]" || { \ echo "ERROR: Failed to install Python dependencies."; \ - echo "Run: python3 -m pip install -r $(REPO_ROOT)/requirements-ci.txt && python3 -m pip install --no-deps -e $(REPO_ROOT)"; \ + echo "Run: python3 -m pip install -e '$(REPO_ROOT)[dev]'"; \ exit 1; \ } diff --git a/tests/csl_runtime/lima-ubuntu-x86_64.yaml b/tests/csl_runtime/lima-ubuntu-x86_64.yaml index 1c9559a5..a1b9b724 100644 --- a/tests/csl_runtime/lima-ubuntu-x86_64.yaml +++ b/tests/csl_runtime/lima-ubuntu-x86_64.yaml @@ -53,7 +53,7 @@ message: | export PATH=/absolute/path/to/cs_sdk:$PATH Verify the SDK: - make -C /absolute/path/to/spatialStencil/tests/csl_runtime check-sdk + make -C /absolute/path/to/spada/tests/csl_runtime check-sdk Run the full CSL runtime suite: - make -C /absolute/path/to/spatialStencil/tests/csl_runtime test + make -C /absolute/path/to/spada/tests/csl_runtime test diff --git a/tests/csl_runtime/run-in-lima.sh b/tests/csl_runtime/run-in-lima.sh index 2eaeeedb..97b89c13 100755 --- a/tests/csl_runtime/run-in-lima.sh +++ b/tests/csl_runtime/run-in-lima.sh @@ -186,8 +186,7 @@ vm "if ! python3 -m pip --version >/dev/null 2>&1; then \ sudo DEBIAN_FRONTEND=noninteractive apt-get update -y && \ sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python3-pip; \ fi && \ - python3 -m pip install --quiet -r '$REPO_ROOT/requirements-ci.txt' && \ - python3 -m pip install --no-deps --quiet -e '$REPO_ROOT'" + python3 -m pip install --quiet -e '$REPO_ROOT[dev]'" # ── Delegate to the Makefile ────────────────────────────────────────────────── MAKE_ARGS="CSL_SDK_DIR=$SDK_DIR"