From 06b9e76ab800a9978fd831a13f28bb4bc76e078d Mon Sep 17 00:00:00 2001 From: Andrey Fedorov Date: Thu, 9 Jul 2026 16:14:15 -0400 Subject: [PATCH 1/4] fix: derive __version__ from package metadata instead of a literal `__version__` hardcoded "3.0.0.dev0" while core/version.py resolves the served version from importlib.metadata. The literal was unused and would have gone stale on the first release bump, leaving two disagreeing answers to "which version is this". Read the distribution metadata instead, so `version` in pyproject.toml is the single source of truth. Kept self-contained rather than importing core.version, which would pull settings/pydantic into `import idc_api`. Co-Authored-By: Claude Opus 4.8 --- src/idc_api/__init__.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/idc_api/__init__.py b/src/idc_api/__init__.py index cbaa1ea..3032ba3 100644 --- a/src/idc_api/__init__.py +++ b/src/idc_api/__init__.py @@ -4,4 +4,12 @@ ``mcp`` packages are thin adapters over it. See ``dev/api_v3_plan.md`` for the design. """ -__version__ = "3.0.0.dev0" +from importlib.metadata import PackageNotFoundError, version as _pkg_version + +try: + # Single source of truth: `version` in pyproject.toml. Do not hardcode it here — the version + # the *running server* advertises resolves through ``core.version``, which reads the same + # distribution metadata, so a literal here would silently drift at release time. + __version__ = _pkg_version("idc-api") +except PackageNotFoundError: # running from a source tree without an install + __version__ = "0.0.0+unknown" From 8420d016521333993d7da92601e7f7c22ceb3f86 Mon Sep 17 00:00:00 2001 From: Andrey Fedorov Date: Thu, 9 Jul 2026 16:14:24 -0400 Subject: [PATCH 2/4] docs: add CONTRIBUTING and CHANGELOG with the release procedure Neither existed, and the conventions were only implicit: Conventional Commits were already de facto (last ~22 commits) but written down nowhere. - CONTRIBUTING.md: branching, commits, the hand-curated changelog rule, versioning (SemVer with MAJOR pinned to the URL prefix, /v3 <-> 3.y.z), and a release runbook. - CHANGELOG.md: Keep a Changelog, seeded with the [3.0.0b1] surface. - README.md: pointers only, per the doc conventions in CLAUDE.md. The runbook records two traps in the existing pipeline: a `v*` tag deploys prod (the glob matches pre-release tags), and it must point at a commit already promoted to test, since prod redeploys test's digest without rebuilding. The version bump therefore needs its own commit through test, or /v3/version reports the previous release. Co-Authored-By: Claude Opus 4.8 --- CHANGELOG.md | 52 ++++++++++++++ CONTRIBUTING.md | 183 ++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 4 +- 3 files changed, 238 insertions(+), 1 deletion(-) create mode 100644 CHANGELOG.md create mode 100644 CONTRIBUTING.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..d84f377 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,52 @@ +# Changelog + +All notable changes to this project are documented here. + +The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project +adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) with one house rule: the +**MAJOR version is pinned to the served URL prefix** — `/v3` ↔ `3.y.z`. A breaking change to the +REST or MCP contract means a new prefix (`/v4`) and a new major (`4.0.0`), never a silent break +under `/v3`. See [CONTRIBUTING.md](CONTRIBUTING.md#versioning) for the full policy. + +Entries describe **user-visible** change — endpoints, MCP tools, response shapes, configuration. +Refactors, CI, and formatting land in the git history, not here. + +## [Unreleased] + +Nothing yet. + +## [3.0.0b1] — unreleased + +First public release of the v3 API: a rewrite that replaces the v1/v2 service with a single +backend-agnostic core behind two thin adapters (REST + MCP), served from the `idc-index` Parquet +index queried locally with DuckDB. + +**This is a beta.** The `/v3` contract may still change in response to feedback before `3.0.0`. +Pin to an exact version if you need stability. Legacy v1/v2 endpoints are unaffected — they are +served by a different backend and v3 lives only under `/v3/*`. + +### Added + +- **REST API**, entirely under the `/v3` prefix: discovery (`/v3/version`, `/v3/stats`, + `/v3/collections`, `/v3/analysis_results`, `/v3/attributes`, `/v3/tables`), cohort building, + retrieval (manifests / cohort URLs, viewer URLs), citations and licenses, guarded SQL, and + `/v3/health`. Interactive docs at `/v3/docs`; the bare domain redirects there. +- **MCP server** over stdio (local) and streamable-http (hosted at `/mcp`), exposing the same + capabilities as tools — `list_collections`, `get_collection`, `list_attributes`, + `get_attribute_values`, `build_cohort`, `get_cohort_urls`, `download_cohort`, `get_viewer_url`, + `run_sql`, `get_citations`, `get_licenses`, and the clinical/table introspection tools — plus + an `idc://guide` resource describing the data model and workflow. +- **Guarded SQL** (`POST /v3/sql`, `run_sql`): read-only DuckDB with external access and + extension loading disabled, single-statement enforcement, a server row cap, and a timeout. + See [SECURITY.md](SECURITY.md). +- **Specialized indices** joinable to `index` on `SeriesInstanceUID` (`seg_index`, `ann_index`, + `ct_index`, `mr_index`, `pt_index`, `sm_index`, …) and per-collection **clinical tables** under + a `clinical` schema, both fetched from `idc-index` releases at build time. +- **Software version reporting**, distinct from the IDC *data* version: `/v3/version` returns + `api_version` (and `build`, when a deploy stamps `IDC_API_BUILD`); the same string appears in + the OpenAPI `info.version` and the MCP `initialize` handshake (`serverInfo.version`). +- **Structured audit logging** — one JSON line per REST request and MCP tool call. + `IDC_API_SQL_LOG_MODE` selects how the guarded SQL query is rendered (`snippet` or `hash`). + +[Unreleased]: https://github.com/ImagingDataCommons/IDC-REST-MCP/compare/v3.0.0b1...HEAD +[3.0.0b1]: https://github.com/ImagingDataCommons/IDC-REST-MCP/releases/tag/v3.0.0b1 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..078d47b --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,183 @@ +# Contributing + +Thanks for helping improve the IDC REST API + MCP server. This document covers how we branch, +commit, changelog, version, and release. For *how the code is laid out* and *how to add a +capability*, see [dev/developer_guide.md](dev/developer_guide.md); for *why*, see +[dev/architecture.md](dev/architecture.md). + +## Getting set up + +```bash +uv venv --python 3.12 +uv pip install -e ".[dev]" +uv run --directory . pytest tests -q +``` + +The first test run downloads the specialized `idc-index` Parquet indices (~40 MB). No GCP +account, network credentials, or authentication are needed — everything queries local Parquet. + +## Before you open a pull request + +Run what CI runs: + +```bash +uv run ruff check src tests +uv run ruff format --check src tests +uv run bandit -q -r src/idc_api +uv run pytest tests -q +``` + +CI also runs `pip-audit` for dependency CVEs. + +## Architecture invariants + +The design depends on a short list of rules — `core/` never imports an adapter, adapters stay +thin, services return Pydantic models, SQL we author is parameterized, untrusted SQL only ever +goes through `backend.run_user_sql`, and errors are raised as `IDCAPIError` subclasses. They are +enumerated in [CLAUDE.md](CLAUDE.md#architecture-invariants-do-not-break) and are what reviewers +check first. Adding a capability touches five places: model → service → REST route → MCP tool → +parity test. + +Documentation is split by audience (user guide vs. the agent-facing `idc://guide` resource vs. +the always-on MCP `INSTRUCTIONS` vs. `dev/`). Keep each in its lane — the conventions, and which +file to touch when, are in [CLAUDE.md](CLAUDE.md#documentation-conventions). + +## Branches + +Branch off `main`, named `/` using the same type vocabulary as commits: + +``` +feat/cohort-size-estimate fix/mcp-trailing-slash +docs/api-endpoint-examples ci/multi-tier-deploy +``` + +Pull requests are merged with a merge commit, so the individual commits on your branch land in +`main`'s history. Make them ones you'd want to read later. + +## Commits + +We follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/): + +``` +(): +``` + +**Types:** `feat`, `fix`, `docs`, `test`, `refactor`, `perf`, `style`, `build`, `ci`, `chore`. +**Scopes** in use: `rest`, `mcp`, `api`, `deploy`, `deps`. + +``` +feat(rest): redirect the bare domain to the interactive docs +fix(mcp): serve /mcp and /mcp/ directly instead of redirecting +docs(api): add OpenAPI summaries/descriptions to every REST route +``` + +This is a convention, not a CI gate — nothing will fail your build if you deviate. It exists so +history stays skimmable and so the changelog is easy to assemble at release time. Dependabot's +commits don't always conform; that's fine. + +Mark a breaking change to the REST or MCP contract with a `!` (`feat(rest)!: …`) and a +`BREAKING CHANGE:` footer. See [Versioning](#versioning) — such a change needs a new URL prefix, +so it is a much bigger conversation than a commit message. + +## Changelog + +[CHANGELOG.md](CHANGELOG.md) is **hand-curated**, in [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) +format. It is not generated from commits: it describes what changed for *callers of the API*, +which is a different thing from what changed in the tree. + +**If your PR changes user-visible behavior, add an entry to `## [Unreleased]` in the same PR.** +User-visible means an endpoint, an MCP tool or its description, a response shape, a configuration +variable, or a default. Use the standard groupings — `Added`, `Changed`, `Deprecated`, `Removed`, +`Fixed`, `Security` — and write for someone consuming the API, not someone reading the diff: + +```markdown +### Fixed +- MCP: `/mcp` and `/mcp/` are both served directly; neither redirects. +``` + +Refactors, test changes, CI, formatting, and dependency bumps do **not** get an entry. The git +history already records them. + +> While `3.0.0b1` is still unreleased, fixes to code that has never shipped are folded into that +> release's `Added` section rather than listed under `Fixed` — there is no released behavior to +> have fixed. Once the beta ships, use the groupings normally. + +## Versioning + +[Semantic Versioning](https://semver.org/spec/v2.0.0.html), with one house rule: + +**MAJOR is pinned to the served URL prefix.** `/v3` ↔ `3.y.z`, always. + +| Change | Version | URL | +|---|---|---| +| Add an endpoint, MCP tool, or optional field | MINOR — `3.1.0` | `/v3` | +| Fix a bug without changing the contract | PATCH — `3.0.1` | `/v3` | +| Break the REST or MCP contract | MAJOR — `4.0.0` | new prefix `/v4` | + +So a breaking change is never a silent break under `/v3`: it is a new prefix served alongside the +old one. This keeps `api_version` predictive of the URL, and matches the clean break v3 already +made from v1/v2. + +Pre-releases use [PEP 440](https://peps.python.org/pep-0440/) spelling so the Python package +version and the git tag agree: `3.0.0b1` → tag `v3.0.0b1`; `3.0.0rc1` → tag `v3.0.0rc1`. + +**The version lives in exactly one place: `version` in [pyproject.toml](pyproject.toml).** +Everything else derives from it — `idc_api.__version__` and `core/version.py:package_version()` +both read the installed distribution metadata. Never hardcode it a second time. + +## Releasing + +> [!IMPORTANT] +> **Pushing a `v*` tag deploys to production.** [promote.yml](.github/workflows/promote.yml) +> triggers on `push: tags: ["v*"]`, and that glob matches pre-release tags too — `v3.0.0b1` goes +> to prod exactly like `v3.0.0`. Never create a `v*` tag for bookkeeping, and be careful with +> `git push --tags`, which can fire a deploy from a stale local tag. + +Two constraints follow from how the pipeline is built (see [dev/deployment.md](dev/deployment.md)): + +1. **Prod deploys test's image, by digest, without rebuilding.** A `v*` tag must therefore point + at a commit that was already promoted to `test` via the `promote.yml` manual dispatch — tag a + commit that never went through test and the deploy fails fast at the digest-resolve step. +2. **The version bump must be its own commit, and it must go through test.** The version is baked + into the image at build time (it comes from the installed package metadata), while + `IDC_API_BUILD` only stamps the git SHA. Tagging `v3.0.0` on the same commit that shipped as + `3.0.0b1` would redeploy an image that still reports `3.0.0b1` at `/v3/version`. + +### Steps + +1. **Bump and curate.** In one PR: set `version` in `pyproject.toml`, and in `CHANGELOG.md` + rename `## [Unreleased]` to `## [X.Y.Z] — YYYY-MM-DD`, open a fresh empty `[Unreleased]`, and + update the link definitions at the foot of the file. Merge it. +2. **Promote to test.** Run `promote.yml` via workflow dispatch against that merge commit. It + builds the canonical image into test's registry and deploys `testing-api.canceridc.dev`. +3. **Verify** against test — `/v3/health`, `/v3/version` (confirm it reports the version you just + set), and the MCP handshake at `/mcp`. +4. **Tag.** `git tag -a v3.0.0 -m "v3.0.0" && git push origin v3.0.0`. This starts + the prod deploy, which waits on the `prod` Environment's required-reviewer gate. +5. **Approve** the deployment, then confirm `api.imaging.datacommons.cancer.gov/v3/version`. +6. **Publish a GitHub Release** on the tag, with the changelog section as its body. Tick + **"Set as a pre-release"** for `bN` / `rcN` tags. + +### The v3 beta + +v3 ships to production as `3.0.0b1` before `3.0.0`. + +The beta is **not** a traffic-safety measure — it can't be one. Prod's load balancer routes only +`/v3/*` to the `idc-api-v3` service; every other path falls through to the legacy backend. No +existing caller reaches v3, so shipping it cannot break them. What the beta buys is the freedom +to **change the `/v3` contract in response to real usage** without spending a major version, and +an honest signal to early adopters that it may move. Exit the beta by tagging `v3.0.0` once the +contract has held under real use and the docs match it. + +Deliberately **not** doing a Cloud Run traffic split (`--tag beta --no-traffic` + a percentage +rollout) for this release. There is no incumbent v3 revision to canary against, and percentage +splits are applied **per request**, not per session — an MCP streamable-http session could have +its requests land on different revisions mid-conversation unless `--session-affinity` is enabled. +If a canary becomes worthwhile for a later release (`3.1.0` onward, once v3 has consumers), use a +**tagged revision** at zero traffic, which gets its own `beta---idc-api-v3-*.run.app` URL that +testers opt into explicitly, rather than a percentage split of the live domain. + +## Reporting security issues + +Please don't open a public issue for a vulnerability — use GitHub's private reporting flow, as +described in [SECURITY.md](SECURITY.md). diff --git a/README.md b/README.md index d40369b..2ad5373 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,9 @@ both. - [`dev/architecture.md`](dev/architecture.md) — internal design (one core, two adapters). - [`dev/api_v3_plan.md`](dev/api_v3_plan.md) — full design rationale + SQL threat model. - [`dev/deployment.md`](dev/deployment.md) — Cloud Run deployment. -- [`dev/developer_guide.md`](dev/developer_guide.md) — contributing / adding capabilities. +- [`dev/developer_guide.md`](dev/developer_guide.md) — codebase tour / adding capabilities. +- [`CONTRIBUTING.md`](CONTRIBUTING.md) — branching, commits, changelog, versioning, releases. +- [`CHANGELOG.md`](CHANGELOG.md) — what changed for callers, per release. - [`SECURITY.md`](SECURITY.md) — threat model, hardening in place, and how to report a vulnerability. From b113dcc8c326ffa1c8bac37aaf46f98cb913d29a Mon Sep 17 00:00:00 2001 From: Andrey Fedorov Date: Thu, 9 Jul 2026 16:50:04 -0400 Subject: [PATCH 3/4] ci: fail a prod tag whose version disagrees with pyproject.toml Prod redeploys test's existing digest and never rebuilds, so the version the server reports is whatever test baked in from pyproject.toml. Tagging v3.0.0 on the commit that shipped as 3.0.0b1 therefore deployed cleanly and quietly served the wrong version at /v3/version. resolve-prod now checks out the tagged commit and asserts that the tag equals "v" + the packaged version, failing before the reviewer gate. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/promote.yml | 21 ++++++++++++++++++++- CONTRIBUTING.md | 4 ++++ 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/.github/workflows/promote.yml b/.github/workflows/promote.yml index d519db7..620c8dd 100644 --- a/.github/workflows/promote.yml +++ b/.github/workflows/promote.yml @@ -7,7 +7,8 @@ name: Promote (test / prod) # - Push a v* tag -> PROD: deploy the exact image TEST already validated, by digest, # referencing TEST's registry directly (no rebuild, no copy). Gated by the `prod` # Environment's required-reviewer rule. prod's deployer SA + Cloud Run service agent must -# have read on TEST's Artifact Registry. +# have read on TEST's Artifact Registry. The tag must equal "v" + pyproject.toml's version +# (checked below): no rebuild happens here, so the version is whatever test baked in. # # For the prod path, test's registry coordinates come from repo-level Variables (test's project # isn't in prod's Environment): TEST_PROJECT_ID (required), TEST_REGION / TEST_AR_REPO (optional). @@ -89,6 +90,24 @@ jobs: build_label: ${{ steps.r.outputs.sha }} source_image: ${{ steps.r.outputs.image }} steps: + - uses: actions/checkout@v7 + + # The tag names the release, but the version the running server reports comes from + # pyproject.toml, baked into the image at build time. Prod redeploys test's existing digest + # and never rebuilds, so a tag that disagrees with the packaged version would deploy fine + # and quietly serve the wrong version at /v3/version. Fail here, before the reviewer gate. + - name: Verify the tag matches the packaged version + env: + TAG: ${{ github.ref_name }} + run: | + set -euo pipefail + VERSION=$(python3 -c 'import tomllib; print(tomllib.load(open("pyproject.toml","rb"))["project"]["version"])') + if [ "${TAG#v}" != "$VERSION" ]; then + echo "::error::Tag '${TAG}' does not match pyproject.toml version '${VERSION}'. The version is baked in when test builds the image, so bump pyproject.toml, promote that commit to test, then tag it. See CONTRIBUTING.md -> Releasing." + exit 1 + fi + echo "Tag ${TAG} matches packaged version ${VERSION}." + - id: r env: TEST_PROJECT: ${{ vars.TEST_PROJECT_ID }} diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 078d47b..c7bc002 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -143,6 +143,10 @@ Two constraints follow from how the pipeline is built (see [dev/deployment.md](d `IDC_API_BUILD` only stamps the git SHA. Tagging `v3.0.0` on the same commit that shipped as `3.0.0b1` would redeploy an image that still reports `3.0.0b1` at `/v3/version`. + `promote.yml` guards this: on a `v*` tag it asserts `tag == "v" + pyproject version` and fails + before the reviewer gate if they disagree. So the tag can't lie about what prod will serve — + but it also means **you cannot tag a release without having bumped `pyproject.toml` first.** + ### Steps 1. **Bump and curate.** In one PR: set `version` in `pyproject.toml`, and in `CHANGELOG.md` From d95af262ec523375fb3285b351ba5952e300ac94 Mon Sep 17 00:00:00 2001 From: Andrey Fedorov Date: Thu, 9 Jul 2026 17:02:39 -0400 Subject: [PATCH 4/4] docs: reconcile CONTRIBUTING with the dev/ docs CONTRIBUTING duplicated content that dev/ already owned, and in one case disagreed with it. - Move the release runbook (and the v3 beta rationale) into deployment.md as "Cutting a release", absorbing the constraint paragraph that already sat under the trigger table. It depends on the tier machinery, the promote dispatch, and the approval gate -- all documented there. Cutting a release is a maintainer task, not a contributor one. - CONTRIBUTING keeps the process a contributor needs -- branching, commits, the changelog rule, the versioning policy -- and points at the developer guide for setup/test/CI/invariants rather than restating them. - developer_guide.md's CI section listed only ruff + pytest; CI also runs ruff format, bandit, and pip-audit. Two docs disagreed on what CI runs. - Register CONTRIBUTING.md and CHANGELOG.md as lanes in CLAUDE.md's documentation conventions, which enumerated every other doc but not these. Co-Authored-By: Claude Opus 4.8 --- CLAUDE.md | 14 +++++- CONTRIBUTING.md | 102 +++++++++-------------------------------- dev/deployment.md | 62 +++++++++++++++++++++++-- dev/developer_guide.md | 13 ++++-- 4 files changed, 99 insertions(+), 92 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 2e85626..bcde1a6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -65,10 +65,20 @@ Docs are split by audience — keep them in their lanes: - **[README.md](README.md)** — kept **lean**: intro, status, install, run one-liners, deploy, and pointers. **Do not** add endpoint/tool reference or usage detail here — that goes in the user guide. +- **[CONTRIBUTING.md](CONTRIBUTING.md)** — the **process**: branching, Conventional Commits, the + changelog rule, and the versioning policy (SemVer with MAJOR pinned to the URL prefix, + `/v3` ↔ `3.y.z`). **Do not** restate setup/test commands or the architecture invariants here — + it points at `dev/developer_guide.md` for those. The release *runbook* lives in + `dev/deployment.md`, not here: it is a maintainer task, not a contributor one. +- **[CHANGELOG.md](CHANGELOG.md)** — hand-curated, [Keep a Changelog](https://keepachangelog.com/) + format. Describes **user-visible** change only (endpoints, MCP tools, response shapes, config + vars, defaults) — never refactors, CI, or dependency bumps. Not generated from commits. - **`dev/`** — design & contributor docs: [architecture.md](dev/architecture.md), [api_v3_plan.md](dev/api_v3_plan.md) (design rationale + SQL threat model), - [deployment.md](dev/deployment.md), [developer_guide.md](dev/developer_guide.md). + [deployment.md](dev/deployment.md) (Cloud Run tiers **and** the release runbook), + [developer_guide.md](dev/developer_guide.md) (setup, test, CI, the invariants, walkthroughs). When you add or change a capability: update `docs/user-guide.md`; mirror any conceptual change into the `idc://guide` resource; check whether `INSTRUCTIONS` needs a one-line touch (it -usually shouldn't, if it stays lean); keep `README.md` a pointer. +usually shouldn't, if it stays lean); keep `README.md` a pointer. If the change is user-visible, +add a `CHANGELOG.md` entry under `## [Unreleased]` in the same PR. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c7bc002..a60f80a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,42 +1,21 @@ # Contributing -Thanks for helping improve the IDC REST API + MCP server. This document covers how we branch, -commit, changelog, version, and release. For *how the code is laid out* and *how to add a -capability*, see [dev/developer_guide.md](dev/developer_guide.md); for *why*, see -[dev/architecture.md](dev/architecture.md). +Thanks for helping improve the IDC REST API + MCP server. This document is the **process**: how we +branch, commit, changelog, and version. -## Getting set up +The **code** is documented elsewhere, and this file does not repeat it: -```bash -uv venv --python 3.12 -uv pip install -e ".[dev]" -uv run --directory . pytest tests -q -``` - -The first test run downloads the specialized `idc-index` Parquet indices (~40 MB). No GCP -account, network credentials, or authentication are needed — everything queries local Parquet. - -## Before you open a pull request - -Run what CI runs: - -```bash -uv run ruff check src tests -uv run ruff format --check src tests -uv run bandit -q -r src/idc_api -uv run pytest tests -q -``` - -CI also runs `pip-audit` for dependency CVEs. - -## Architecture invariants +| You want to | Read | +|---|---| +| Set up, run, and test the project | [dev/developer_guide.md](dev/developer_guide.md) — *Setup*, *Run*, *Test* | +| Know what CI will run against your PR | [dev/developer_guide.md](dev/developer_guide.md) — *Continuous integration* | +| Understand the invariants reviewers check | [dev/developer_guide.md](dev/developer_guide.md) — *Conventions* | +| Add a capability (model → service → REST → MCP → parity test) | [dev/developer_guide.md](dev/developer_guide.md) — *Walkthrough* | +| Understand *why* it's built this way | [dev/architecture.md](dev/architecture.md) | +| Cut a release / deploy | [dev/deployment.md](dev/deployment.md) — *Cutting a release* | -The design depends on a short list of rules — `core/` never imports an adapter, adapters stay -thin, services return Pydantic models, SQL we author is parameterized, untrusted SQL only ever -goes through `backend.run_user_sql`, and errors are raised as `IDCAPIError` subclasses. They are -enumerated in [CLAUDE.md](CLAUDE.md#architecture-invariants-do-not-break) and are what reviewers -check first. Adding a capability touches five places: model → service → REST route → MCP tool → -parity test. +In short: `uv pip install -e ".[dev]"`, then make sure `ruff`, `bandit`, and `pytest` pass before +you push — the developer guide has the exact commands CI uses. Documentation is split by audience (user guide vs. the agent-facing `idc://guide` resource vs. the always-on MCP `INSTRUCTIONS` vs. `dev/`). Keep each in its lane — the conventions, and which @@ -127,59 +106,20 @@ both read the installed distribution metadata. Never hardcode it a second time. ## Releasing +Cutting a release is a **maintainer** task that deploys to production. The runbook — including the +v3 beta plan — lives with the deploy machinery it depends on: +[dev/deployment.md § Cutting a release](dev/deployment.md#cutting-a-release). + +Two things every contributor should know: + > [!IMPORTANT] > **Pushing a `v*` tag deploys to production.** [promote.yml](.github/workflows/promote.yml) > triggers on `push: tags: ["v*"]`, and that glob matches pre-release tags too — `v3.0.0b1` goes > to prod exactly like `v3.0.0`. Never create a `v*` tag for bookkeeping, and be careful with > `git push --tags`, which can fire a deploy from a stale local tag. -Two constraints follow from how the pipeline is built (see [dev/deployment.md](dev/deployment.md)): - -1. **Prod deploys test's image, by digest, without rebuilding.** A `v*` tag must therefore point - at a commit that was already promoted to `test` via the `promote.yml` manual dispatch — tag a - commit that never went through test and the deploy fails fast at the digest-resolve step. -2. **The version bump must be its own commit, and it must go through test.** The version is baked - into the image at build time (it comes from the installed package metadata), while - `IDC_API_BUILD` only stamps the git SHA. Tagging `v3.0.0` on the same commit that shipped as - `3.0.0b1` would redeploy an image that still reports `3.0.0b1` at `/v3/version`. - - `promote.yml` guards this: on a `v*` tag it asserts `tag == "v" + pyproject version` and fails - before the reviewer gate if they disagree. So the tag can't lie about what prod will serve — - but it also means **you cannot tag a release without having bumped `pyproject.toml` first.** - -### Steps - -1. **Bump and curate.** In one PR: set `version` in `pyproject.toml`, and in `CHANGELOG.md` - rename `## [Unreleased]` to `## [X.Y.Z] — YYYY-MM-DD`, open a fresh empty `[Unreleased]`, and - update the link definitions at the foot of the file. Merge it. -2. **Promote to test.** Run `promote.yml` via workflow dispatch against that merge commit. It - builds the canonical image into test's registry and deploys `testing-api.canceridc.dev`. -3. **Verify** against test — `/v3/health`, `/v3/version` (confirm it reports the version you just - set), and the MCP handshake at `/mcp`. -4. **Tag.** `git tag -a v3.0.0 -m "v3.0.0" && git push origin v3.0.0`. This starts - the prod deploy, which waits on the `prod` Environment's required-reviewer gate. -5. **Approve** the deployment, then confirm `api.imaging.datacommons.cancer.gov/v3/version`. -6. **Publish a GitHub Release** on the tag, with the changelog section as its body. Tick - **"Set as a pre-release"** for `bN` / `rcN` tags. - -### The v3 beta - -v3 ships to production as `3.0.0b1` before `3.0.0`. - -The beta is **not** a traffic-safety measure — it can't be one. Prod's load balancer routes only -`/v3/*` to the `idc-api-v3` service; every other path falls through to the legacy backend. No -existing caller reaches v3, so shipping it cannot break them. What the beta buys is the freedom -to **change the `/v3` contract in response to real usage** without spending a major version, and -an honest signal to early adopters that it may move. Exit the beta by tagging `v3.0.0` once the -contract has held under real use and the docs match it. - -Deliberately **not** doing a Cloud Run traffic split (`--tag beta --no-traffic` + a percentage -rollout) for this release. There is no incumbent v3 revision to canary against, and percentage -splits are applied **per request**, not per session — an MCP streamable-http session could have -its requests land on different revisions mid-conversation unless `--session-affinity` is enabled. -If a canary becomes worthwhile for a later release (`3.1.0` onward, once v3 has consumers), use a -**tagged revision** at zero traffic, which gets its own `beta---idc-api-v3-*.run.app` URL that -testers opt into explicitly, rather than a percentage split of the live domain. +And you cannot tag a release without bumping `pyproject.toml` first: `promote.yml` asserts that +the tag equals `"v"` + the packaged version, and fails before the reviewer gate if they disagree. ## Reporting security issues diff --git a/dev/deployment.md b/dev/deployment.md index 785c704..829f4fe 100644 --- a/dev/deployment.md +++ b/dev/deployment.md @@ -248,10 +248,64 @@ file — the first becomes the *trigger*, the second becomes an *Environment set | [promote.yml](../.github/workflows/promote.yml) (tag) | push a `v*` tag | deploy test's digest to **prod** (behind the required-reviewer gate) | | [deploy.yml](../.github/workflows/deploy.yml) | reusable (`workflow_call`) | the shared deploy job the callers invoke | -Prod deploys **test's** image, so a `v*` tag must point at a commit that was **already promoted to -test** (its image exists in test's registry). Cut releases from a ref you've dispatched to test; -tag a commit that never went through test and the prod deploy fails fast at the digest-resolve -step. +### Cutting a release + +Versioning policy — what a version *number* means, and when to bump which part — is in +[CONTRIBUTING.md](../CONTRIBUTING.md#versioning). This section is the mechanics. + +> [!IMPORTANT] +> **Pushing a `v*` tag deploys to production**, and the glob matches pre-release tags too — +> `v3.0.0b1` goes to prod exactly like `v3.0.0`. Never create a `v*` tag for bookkeeping, and +> beware `git push --tags` firing a deploy from a stale local tag. + +Two constraints fall out of the security boundary above — prod runs test's exact bytes and never +rebuilds: + +1. **A `v*` tag must point at a commit already promoted to test** (its image exists in test's + registry). Cut releases from a ref you've dispatched to test; tag a commit that never went + through test and the prod deploy fails fast at the digest-resolve step. +2. **The version bump must be its own commit, and it must go through test.** The version comes from + the installed package metadata, baked into the image when *test* builds it; `IDC_API_BUILD` only + stamps the git SHA on top. Tagging `v3.0.0` on the same commit that shipped as `3.0.0b1` would + redeploy an image that still reports `3.0.0b1` at `/v3/version`. `promote.yml` guards this: on a + `v*` tag it asserts `tag == "v" + pyproject version` and fails **before** the reviewer gate if + they disagree. + +#### Steps + +1. **Bump and curate.** In one PR: set `version` in [pyproject.toml](../pyproject.toml), and in + [CHANGELOG.md](../CHANGELOG.md) rename `## [Unreleased]` to `## [X.Y.Z] — YYYY-MM-DD`, open a + fresh empty `[Unreleased]`, and update the link definitions at the foot of the file. Merge it. +2. **Promote to test.** Run [promote.yml](../.github/workflows/promote.yml) via workflow dispatch + against that merge commit. It builds the canonical image into test's registry and deploys + `testing-api.canceridc.dev`. +3. **Verify** against test — `/v3/health`, `/v3/version` (confirm it reports the version you just + set), and the MCP handshake at `/mcp`. +4. **Tag.** `git tag -a v3.0.0 -m "v3.0.0" && git push origin v3.0.0`. This starts the + prod deploy, which waits on the `prod` Environment's required-reviewer gate. +5. **Approve** the deployment, then confirm `api.imaging.datacommons.cancer.gov/v3/version`. +6. **Publish a GitHub Release** on the tag, with the changelog section as its body. Tick **"Set as a + pre-release"** for `bN` / `rcN` tags. + +#### The v3 beta + +v3 ships to production as `3.0.0b1` before `3.0.0`. + +The beta is **not** a traffic-safety measure — it can't be one. The prod load balancer routes only +`/v3/*` to the `idc-api-v3` service (see [Shared-domain path routing](#shared-domain-path-routing-as-deployed--the-glob-gotcha) +below); every other path falls through to the legacy ESP backend. No existing caller reaches v3, so +shipping it cannot break them. What the beta buys is the freedom to **change the `/v3` contract in +response to real usage** without spending a major version — which, under the versioning policy, +would otherwise cost a whole new `/v4` prefix — plus an honest signal to early adopters that the +surface may move. Exit the beta by tagging `v3.0.0` once the contract has held under real use. + +Deliberately **not** doing a Cloud Run traffic split (`--tag beta --no-traffic` plus a percentage +rollout) for this release. There is no incumbent v3 revision to canary against, and percentage +splits are applied **per request**, not per session — an MCP streamable-http session could have its +requests land on different revisions mid-conversation unless `--session-affinity` is enabled. If a +canary becomes worthwhile for a later release (`3.1.0` onward, once v3 has consumers), use a +**tagged revision** at zero traffic, which gets its own `beta---idc-api-v3-*.run.app` URL that +testers opt into explicitly, rather than a percentage split of the live domain. ### One-time setup diff --git a/dev/developer_guide.md b/dev/developer_guide.md index 1d2a2b7..5dbb795 100644 --- a/dev/developer_guide.md +++ b/dev/developer_guide.md @@ -64,14 +64,17 @@ Fixtures live in [tests/conftest.py](../tests/conftest.py): `ctx` (the core [`.github/workflows/ci.yml`](../.github/workflows/ci.yml) runs on pushes to `main` (and manual dispatch) and on PRs that touch `src/idc_api/**`, `tests/**`, `pyproject.toml`, -or `uv.lock`. It installs locked deps with `uv sync --extra dev`, then runs `ruff check` and -`pytest tests` on Python 3.11 and 3.12. CI needs no secrets or GCP; its first test run +or `uv.lock`. It installs locked deps with `uv sync --extra dev`, then lints, scans, and tests on +Python 3.11 and 3.12. CI needs no secrets or GCP; its first test run downloads the specialized indices (`IDC_API_INCLUDE_INDICES=all` by default — set `none` to run -from bundled data only). Before pushing, run the same two commands locally: +from bundled data only). Before pushing, run the same checks locally: ```bash -uv run ruff check src tests -uv run pytest tests -q +uv run ruff check src tests # lint +uv run ruff format --check src tests # formatting +uv run bandit -q -r src/idc_api # static security lint +uv run pip-audit # dependency CVEs +uv run pytest tests -q # tests ``` ## Project layout