Feat/dump v2 prefix

API.md

@@ -75,9 +75,9 @@ token (except under `anonymous`).
 | POST | `/datastore/api/v2/datastore_delete` | Delete rows, drop columns, or drop the table |
 | GET | `/datastore/api/v2/datastore_search` | Search a resource (streaming) |
 | GET | `/datastore/api/v2/datastore_search_sql` | Run a read-only SQL `SELECT` (streaming) |
-| GET | `/datastore/dump/query` | Download the result of a SQL `SELECT` as a file |
+| GET | `/datastore/api/v2/dump/query` | Download the result of a SQL `SELECT` as a file |
 | GET | `/datastore/api/v2/datastore_info` | Schema + row stats for a resource |
-| GET | `/datastore/dump/{resource_id}` | Download a whole resource (CSV/NDJSON/Parquet) |
+| GET | `/datastore/api/v2/dump/{resource_id}` | Download a whole resource (CSV/NDJSON/Parquet) |
 | GET | `/datastore/api/health` · `/datastore/api/ready` | Liveness / readiness |
 
 ---
@@ -314,7 +314,7 @@ GET /datastore/api/v2/datastore_search
 Run a single read-only `SELECT` / `WITH` statement and stream the result. Tables
 are referenced by `resource_id`; each is authorized individually, and functions
 are checked against the engine's allow-list. Include a `LIMIT` (required).
-To export the result as a file instead, use [`GET /datastore/dump/query`](#get-datastoredumpquery).
+To export the result as a file instead, use [`GET /datastore/api/v2/dump/query`](#get-datastoreapiv2dumpquery).
 
 ### Query parameters
 
@@ -386,11 +386,22 @@ row stats — a column-level metadata catalog without a side store.
 
 ---
 
-## `GET /datastore/dump/{resource_id}`
+## `GET /datastore/api/v2/dump/{resource_id}`
 
 Download an entire resource. Pick the format with `?format=csv` (default),
 `gzip`, `ndjson`, or `parquet`.
 
+This route returns a **file**, never the JSON envelope — the only JSON it can
+produce is an error. The content type comes from the signed GCS URL and
+follows `format`:
+
+| `format` | Content-Type | Extension |
+|---|---|---|
+| `csv` | `text/csv` | `.csv` |
+| `gzip` | `application/gzip` | `.csv.gz` |
+| `ndjson` | `application/x-ndjson` | `.json` |
+| `parquet` | `application/vnd.apache.parquet` | `.parquet` |
+
 - **csv / gzip / ndjson** — `302` redirect to a signed GCS URL, at any size.
   Shards from a large export are stitched into one object server-side, so the
   bytes go straight from storage to the client (resumable, no server
@@ -405,13 +416,13 @@ Download an entire resource. Pick the format with `?format=csv` (default),
 Requires `read` permission on the resource and a configured export bucket
 (`BIGQUERY_EXPORT_BUCKET`).
 
-`query` is a **reserved name** on this route — `/datastore/dump/query` is the SQL
+`query` is a **reserved name** on this route — `/datastore/api/v2/dump/query` is the SQL
 download endpoint below, so a resource literally named `query` can't be dumped
 by this URL.
 
 ---
 
-## `GET /datastore/dump/query`
+## `GET /datastore/api/v2/dump/query`
 
 Download the result of a **SQL `SELECT`** as a single file — filtered
 downloads at any size. Same validation as `datastore_search_sql` (single
@@ -428,14 +439,14 @@ file itself, not the CKAN envelope.
 ### Example
 
 ```http
-GET /datastore/dump/query
+GET /datastore/api/v2/dump/query
     ?sql=SELECT * FROM "c6153a74-43cb-4edf-8bdf-bb664feca937" WHERE accepted = true
     &format=csv
 ```
 
 ### Response
 
-Identical to `/datastore/dump/{resource_id}` above:
+Identical to `/datastore/api/v2/dump/{resource_id}` above:
 
 - **csv / gzip / ndjson** — `302` to a signed GCS URL at any size (shards are
   composed into one object). The URL expires after

CLAUDE.md

@@ -430,9 +430,9 @@ Each endpoint takes a single `ContextDep`. The handler calls `context.authorize(
 | POST | `/datastore/api/v2/datastore_delete` | **implemented** | `DatastoreDeleteRequest` | `DatastoreDeleteResponse` |
 | GET  | `/datastore/api/v2/datastore_search` | **implemented** (streaming) | `DatastoreSearchRequest` | `DatastoreSearchResponse` |
 | GET  | `/datastore/api/v2/datastore_search_sql` | **implemented** (streaming) | `DatastoreSearchSQLRequest` | `DatastoreSearchResponse` |
-| GET  | `/datastore/dump/query` | **implemented** | `sql=<SELECT…>`, `format=csv\|gzip\|ndjson\|parquet` | 302 → GCS *or* streaming body (see §5.3) |
+| GET  | `/datastore/api/v2/dump/query` | **implemented** | `sql=<SELECT…>`, `format=csv\|gzip\|ndjson\|parquet` | 302 → GCS *or* streaming body (see §5.3) |
 | GET  | `/datastore/api/v2/datastore_info` | **implemented** | `DatastoreInfoRequest` | `DatastoreInfoResponse` |
-| GET  | `/datastore/dump/{resource_id}` | **implemented** | `format=csv\|ndjson\|parquet` | 302 → GCS *or* streaming body (see §5.3) |
+| GET  | `/datastore/api/v2/dump/{resource_id}` | **implemented** | `format=csv\|ndjson\|parquet` | 302 → GCS *or* streaming body (see §5.3) |
 
 The BigQuery engine is wired end-to-end: DDL, MERGE-based upsert, DML delete, parameterised search, native table-level metadata (the Frictionless schema + unique_key are JSON-encoded into the table's own `description` OPTION) for the schema round-trip, a row-count fast path via `INFORMATION_SCHEMA.TABLE_STORAGE`, and `EXPORT DATA`-backed dump with `table.modified`-keyed GCS caching. The DuckLake engine is the next concrete adapter — see §7.
 
@@ -443,7 +443,7 @@ The BigQuery engine is wired end-to-end: DDL, MERGE-based upsert, DML delete, pa
 
 **Read-only guard (`AUTH_TYPE=ckan` only).** `datastore_create`, `datastore_upsert`, and `datastore_delete` refuse to write a resource whose CKAN record carries `url_type="datastore"` unless the request sets `force: true` — a `Validation Error` ("Cannot update a read-only resource. Use \"force\" to force update.") otherwise. This mirrors CKAN's protection against clobbering datastore-managed data by accident. The guard is gated on `AUTH_TYPE=ckan` and skipped entirely under any other provider (only the CKAN provider attaches a resource record).
 
-### 5.3 `GET /datastore/dump/{resource_id}`
+### 5.3 `GET /datastore/api/v2/dump/{resource_id}`
 
 Full-table download, **one URL → one file** from the caller's point of
 view. Bytes never pass through API memory — the one exception is a
@@ -497,9 +497,9 @@ A single SA works if both perm sets land on the same identity — `BIGQUERY_CRED
 
 A 24h object-lifecycle rule on the bucket is **required** in practice: the engine GCs older revs already, but lifecycle is the only thing that cleans abandoned `dumps/<qhash>/` prefixes (SQL downloads whose query is never re-issued — see below) and anything stranded by a crashed dump.
 
-### SQL download (`GET /datastore/dump/query`)
+### SQL download (`GET /datastore/api/v2/dump/query`)
 
-`GET /datastore/dump/query?sql=<SELECT…>&format=csv|gzip|ndjson|parquet` exports the result of an arbitrary vetted SELECT through the same pipeline as `/datastore/dump/{resource_id}` — engine method `dump_sql` in [bigquery/export.py](datastore/infrastructure/engines/bigquery/export.py), response shaping shared via `download_response` in [api/endpoints/dump.py](datastore/api/endpoints/dump.py) (302 for the composed file · gzip streamed · JSON URL list for multi-file parquet). Same SQL validation + per-table auth as `datastore_search_sql` (`DatastoreDumpSQLRequest` subclasses its request schema); the action API itself stays pure JSON envelope. The route is declared before `/datastore/dump/{resource_id}`, making `query` a reserved resource name on the dump family.
+`GET /datastore/api/v2/dump/query?sql=<SELECT…>&format=csv|gzip|ndjson|parquet` exports the result of an arbitrary vetted SELECT through the same pipeline as `/datastore/api/v2/dump/{resource_id}` — engine method `dump_sql` in [bigquery/export.py](datastore/infrastructure/engines/bigquery/export.py), response shaping shared via `download_response` in [api/endpoints/dump.py](datastore/api/endpoints/dump.py) (302 for the composed file · gzip streamed · JSON URL list for multi-file parquet). Same SQL validation + per-table auth as `datastore_search_sql` (`DatastoreDumpSQLRequest` subclasses its request schema); the action API itself stays pure JSON envelope. The route is declared before `/datastore/api/v2/dump/{resource_id}`, making `query` a reserved resource name on the dump family.
 
 Deltas vs the whole-table dump:
 
@@ -799,7 +799,7 @@ Optional fields appear in `result` only when requested:
 
 ### 6.4 `GET /datastore/api/v2/datastore_search_sql`
 
-**Query params**: `sql` (required; must carry a `LIMIT` literal). To export the result as a file instead of the JSON envelope, use `GET /datastore/dump/query?sql=…&format=…` (LIMIT optional + uncapped there — see §5.3 "SQL download").
+**Query params**: `sql` (required; must carry a `LIMIT` literal). To export the result as a file instead of the JSON envelope, use `GET /datastore/api/v2/dump/query?sql=…&format=…` (LIMIT optional + uncapped there — see §5.3 "SQL download").
 
 **Example request — daily clearing-price summary**
 ```

README.md

@@ -31,7 +31,7 @@ Copy [.env.example](.env.example) and fill it in. The essentials:
 | `AUTH_TYPE` | `ckan` | Auth provider: `ckan` · `jwt` · `anonymous` |
 | `CKAN_URL` | — | CKAN base URL (required when `AUTH_TYPE=ckan`) |
 | `BIGQUERY_PROJECT` / `BIGQUERY_DATASET` | — | Required when `DATASTORE_ENGINE=bigquery` |
-| `BIGQUERY_EXPORT_BUCKET` | — | GCS bucket for downloads (`/datastore/dump/{resource_id}`, `/datastore/dump/query`) |
+| `BIGQUERY_EXPORT_BUCKET` | — | GCS bucket for downloads (`/datastore/api/v2/dump/{resource_id}`, `/datastore/api/v2/dump/query`) |
 | `REDIS_URL` | — | Cache backend; empty → in-process cache |
 | `API_URL` | `https://example.com` | Public base URL; sets the host in the OpenAPI examples |
 | `DOCS_PRIMARY_COLOR` / `DOCS_HEADER_COLOR` | — | Swagger UI branding (see [Documentation](#documentation)) |

datastore/api/docs.py

@@ -284,10 +284,12 @@ def api_description(auth_type: str) -> str:
     },
     {
         "name": "Datastore",
-        "description": ("API endpoint - create, upsert, delete, searchsearch_sql, and info."),
+        "description": "JSON action API — every response is the CKAN envelope.",
     },
     {
-        "name": "Datastore Download",
-        "description": "Bulk download of an entire resource in available formats.",
+        "name": "Datastore Downloads",
+        "description": (
+            "File downloads — a `302` to a signed URL, not the JSON envelope."
+        ),
     },
 ]

datastore/api/endpoints/dump.py

@@ -9,22 +9,54 @@
 
 from __future__ import annotations
 
-from typing import Annotated
+from typing import Annotated, Any
 
 from fastapi import APIRouter, Query
 from starlette.requests import Request
 from starlette.responses import RedirectResponse, StreamingResponse
 
 from datastore.api.context import Context
 from datastore.api.responses import ERROR_RESPONSES
-from datastore.core.constants import DUMP_EXTENSIONS, DumpFormat
+from datastore.core.constants import DUMP_EXTENSIONS, DUMP_MEDIA_TYPES, DumpFormat
 from datastore.core.exceptions import ServerError
 from datastore.infrastructure.engines import get_datastore_engine
 from datastore.schemas.request import DatastoreDumpSQLRequest
 from datastore.services.read import dump_sql_datastore
 from datastore.services.streaming import zip_archive_writer
 
-router = APIRouter(tags=["Datastore Download"], responses=ERROR_RESPONSES)
+router = APIRouter(tags=["Datastore Downloads"], responses=ERROR_RESPONSES)
+
+
+# What a download actually returns, for OpenAPI. Without an explicit
+# `response_class` FastAPI documents a JSON body on the 200 — wrong for
+# every path here, and it would have a generated client parsing a parquet
+# file as an envelope. The 302 is the usual answer; the 200 only happens
+# when a parquet export shards.
+_DOWNLOAD_RESPONSES: dict[int | str, dict[str, Any]] = {
+    302: {
+        "description": (
+            "The download. Redirects to a short-lived signed GCS URL — the "
+            "bytes stream from GCS, not through this API, so the transfer "
+            "is resumable. Content type follows `format`: "
+            + ", ".join(f"`{f}` → `{m}`" for f, m in DUMP_MEDIA_TYPES.items())
+            + ". The URL carries a `Content-Disposition` naming the file."
+        ),
+        "headers": {
+            "Location": {
+                "description": "Signed GCS URL holding the exported file.",
+                "schema": {"type": "string", "format": "uri"},
+            },
+        },
+    },
+    200: {
+        "description": (
+            "A parquet export that sharded: one streamed zip of the parts, "
+            "served by this API rather than redirected. Chunked, so there "
+            "is no `Content-Length` and no range support."
+        ),
+        "content": {"application/zip": {}},
+    },
+}
 
 
 def download_response(
@@ -67,13 +99,9 @@ def download_response(
 @router.get(
     "/query",
     summary="Download the result of a SQL SELECT",
-    responses={
-        302: {"description": "Redirect to the signed download URL."},
-        200: {
-            "description": ("Sharded parquet export - one streamed zip of the parts."),
-            "content": {"application/zip": {}},
-        },
-    },
+    response_class=RedirectResponse,
+    status_code=302,
+    responses=_DOWNLOAD_RESPONSES,
 )
 async def dump_sql(
     request: Request,
@@ -99,13 +127,9 @@ async def dump_sql(
 @router.get(
     "/{resource_id}",
     summary="Download an entire table",
-    responses={
-        302: {"description": "Redirect to the signed Download URL."},
-        200: {
-            "description": ("Multi-file parquet export — one streamed zip."),
-            "content": {"application/zip": {}},
-        },
-    },
+    response_class=RedirectResponse,
+    status_code=302,
+    responses=_DOWNLOAD_RESPONSES,
 )
 async def dump(
     request: Request,

datastore/api/routes.py

@@ -8,8 +8,9 @@
 api_router = APIRouter()
 # Unversioned: a probe URL is stable regardless of the action contract.
 api_router.include_router(health.probe_router, prefix=API_BASE_PREFIX)
-# Downloads sit outside `/api` — a dump URL is opened in a browser or fed to
-# `curl`, not driven by an API client.
+# Downloads: versioned alongside the actions, since their query params and
+# output layout are part of the same contract. Mounted before the action
+# router so `/dump/...` can't be read as an action name.
 api_router.include_router(dump.router, prefix=DUMP_PREFIX)
 # Versioned: the datastore actions are the compatibility contract.
 api_router.include_router(datastore.router, prefix=API_PREFIX)

datastore/core/constants.py

@@ -14,8 +14,10 @@
 #     the compatibility contract;
 #   - the health probes sit outside it, so an orchestrator's probe URL
 #     doesn't churn when the contract gets a new version;
-#   - downloads sit outside the `/api` segment entirely, since a dump URL is
-#     handed to a browser or a `curl` rather than driven by an API client.
+#   - downloads sit under the versioned prefix alongside the actions: a dump
+#     URL is handed to a browser or a `curl` rather than driven by an API
+#     client, but its query params and output layout are part of the same
+#     compatibility contract, so it versions with them.
 #
 # Note this is NOT the prefix used to *call* an upstream CKAN — those
 # requests go to CKAN's own `/api/3/action/` (see
@@ -24,7 +26,7 @@
 API_BASE_PREFIX = f"{SERVICE_PREFIX}/api"
 API_VERSION = "v2"
 API_PREFIX = f"{API_BASE_PREFIX}/{API_VERSION}"
-DUMP_PREFIX = f"{SERVICE_PREFIX}/dump"
+DUMP_PREFIX = f"{API_PREFIX}/dump"
 # Download formats served by the export pipeline (`<DUMP_PREFIX>/…` and
 # `datastore_search_sql?download=…`). Lives here — not in `api/` — because
 # both the request schemas (pydantic layer) and the endpoints (starlette
@@ -41,6 +43,17 @@
     "parquet": "parquet",
 }
 
+# Media type per dump format. Only ever reaches the client through the
+# signed GCS URL's own headers — the API redirects rather than serving the
+# bytes — but OpenAPI needs them to document what a download actually is,
+# so a generated client doesn't assume the JSON envelope.
+DUMP_MEDIA_TYPES: dict[str, str] = {
+    "csv": "text/csv",
+    "gzip": "application/gzip",
+    "ndjson": "application/x-ndjson",
+    "parquet": "application/vnd.apache.parquet",
+}
+
 POSTGRES_TYPES: dict[str, str] = {
     # integer
     "int2": "int2",

datastore/infrastructure/engines/bigquery/export.py

@@ -648,7 +648,7 @@ def _get_export_bucket(backend: Any) -> str:
     if not bucket:
         raise ServerError(
             "BIGQUERY_EXPORT_BUCKET is not configured — "
-            "/datastore/dump cannot run without an export bucket."
+            "/datastore/api/v2/dump cannot run without an export bucket."
         )
     return bucket
 

datastore/schemas/request.py

@@ -327,7 +327,7 @@ class DatastoreSearchSQLRequest(BaseModel):
     _REQUIRE_LIMIT: ClassVar[bool] = True
 
     sql: str = Field(
-        description=("A  Datastore read API with `SELECT` / `WITH` statement."),
+        description=("A read-only `SELECT` / `WITH` statement."),
         examples=['SELECT * FROM "balancing_auction_results_2025" WHERE accepted = true LIMIT 100'],
     )
 
@@ -434,7 +434,7 @@ class DatastoreDumpSQLRequest(DatastoreSearchSQLRequest):
     _REQUIRE_LIMIT: ClassVar[bool] = False
 
     sql: str = Field(
-        description=("A  Datastore read API with`SELECT` / `WITH` statement."),
+        description=("A read-only `SELECT` / `WITH` statement to export."),
         examples=['SELECT * FROM "balancing_auction_results_2025" WHERE accepted = true'],
     )
 

postman/README.md

@@ -6,9 +6,9 @@ Auto-generated from [`example_payload/`](../example_payload/) by
 
 ## Import
 
-In Postman: **File → Import** → `collection.json`. Seven folders appear:
+In Postman: **File → Import** → `collection.json`. Eight folders appear:
 `health`, `datastore_create`, `datastore_upsert`, `datastore_info`,
-`datastore_search`, `datastore_search_sql`, `datastore_delete`.
+`datastore_search`, `datastore_search_sql`, `datastore_delete`, `dump`.
 
 ## Variables
 
@@ -29,17 +29,28 @@ Run folders top-to-bottom on a fresh resource:
 3. **`datastore_info`** — confirm schema + row count.
 4. **`datastore_search`** — filter / full-text / paginated.
 5. **`datastore_search_sql`** — raw SQL; `LIMIT` required. JOIN/UNION variants need a second resource `balancing_auction_results_2024`.
-6. **`datastore_delete`** — row delete (`auction_id=1`) → drop column (`bidder_metadata`) → drop table.
+6. **`dump`** — download the table or a SQL result as a file. Run before
+   `datastore_delete`, while the rows still exist.
+7. **`datastore_delete`** — row delete (`auction_id=1`) → drop column (`bidder_metadata`) → drop table.
 
 `health` is independent — hit any time to check the server.
 
+**On the `dump` folder:** these return `302` to a signed GCS URL rather than a
+body. Postman follows the redirect by default and downloads the file; switch
+off **Settings → Automatically follow redirects** to inspect the `Location`
+header instead. A sharded parquet export returns `200` + a zip of the parts
+instead of a redirect. The server needs `BIGQUERY_EXPORT_BUCKET` set —
+without it every dump request is a `500`.
+
 ## Regenerate
 
 ```sh
 python postman/generate_postman.py
 ```
 
 Drop new files under `example_payload/<action>/<name>.json` to add requests.
+The `dump` folder is the exception: its requests take a path parameter and no
+body, so they're declared inline as `DUMP_REQUESTS` in the generator.
 
 ## Auth