From 3a070398bc62981e6481b1fcfbc9cf5b60fe91b2 Mon Sep 17 00:00:00 2001 From: Neil Macneale IV Date: Thu, 20 Aug 2026 23:31:59 +0000 Subject: [PATCH] Document the s3:// remote for dolt and doltgres Adds an S3-compatible section to the remote options on both sites, between AWS and GCS, covering what the protocol is, how credentials are resolved, and how to address a provider that is not AWS. The three query parameters get a table and worked R2 and MinIO examples, because the interesting parts are not obvious: endpoint and region fall back to the SDK chain but path-style does not and can only be set in the url; MinIO generally needs path-style and R2 does not; and an unrecognized parameter is an error rather than being ignored, so a typo is reported when the remote is added rather than surfacing later as a connection failure. The dolt page also warns to quote the url, since an unquoted ampersand backgrounds the command. The doltgres page is written in its own idiom rather than copied: sql function calls instead of cli, credentials framed as what the server process sees, and no shell quoting advice, which does not apply inside a quoted sql string. Doltgres inherits the scheme through dolt's dbfactory once its dolt dependency is bumped. cli.md is generated by dump-docs, so it is untouched here; its scheme list and remote help text come from the dolt source and will update when it is regenerated. Co-Authored-By: Claude Opus 5 (1M context) --- .../reference/sql/version-control/remotes.md | 68 +++++++++++++++++++ .../reference/sql/version-control/remotes.md | 58 ++++++++++++++++ 2 files changed, 126 insertions(+) diff --git a/site/dolt/src/content/reference/sql/version-control/remotes.md b/site/dolt/src/content/reference/sql/version-control/remotes.md index d79bd72..5dc351e 100644 --- a/site/dolt/src/content/reference/sql/version-control/remotes.md +++ b/site/dolt/src/content/reference/sql/version-control/remotes.md @@ -247,6 +247,74 @@ or dolt clone --aws-creds-profile prod-profile --aws-region us-west-2 origin aws://[dolt_dynamo_table:dolt_remotes_s3_storage]/menus ``` +If you would rather not run a DynamoDB table, the `s3://` protocol below stores +the manifest in the bucket itself and works against S3 as well. + +### S3-compatible + +The `s3://` protocol works against any object store that implements the S3 API +with conditional writes, including AWS S3 itself, Cloudflare R2, and MinIO. It +stores the manifest as an ordinary object and uses conditional writes to update +it atomically, so unlike `aws://` it needs no DynamoDB table. + +```bash +dolt remote add origin s3://BUCKET/path/for/remote +``` + +Credentials always come from the standard AWS SDK chain: environment variables, +the shared credentials file, SSO, or an instance role. They are never accepted +in the URL, since the URL is stored with the remote in plain text. + +Against AWS S3 that is all you need, because the SDK resolves the endpoint from +your region: + +```bash +export AWS_REGION=us-west-2 +dolt remote add origin s3://my-bucket/menus +dolt push origin main +``` + +#### Addressing another provider + +A non-AWS provider is reached by adding query parameters to the URL: + +| Parameter | Meaning | +| ------------ | ------------------------------------------------------------------------ | +| `endpoint` | The provider's S3 API host, e.g. an R2 or MinIO endpoint | +| `region` | The signing region. Some providers want a fixed value; R2 uses `auto` | +| `path-style` | `true` to address buckets as a path segment instead of a hostname prefix | + +`endpoint` and `region` fall back to the SDK's own resolution when omitted, so +`AWS_ENDPOINT_URL_S3`, `AWS_REGION`, and the shared config file all work as +usual. `path-style` has no such fallback and can only be set in the URL. Leave +it off unless your provider needs it; MinIO generally does, because it has no +wildcard DNS to serve `bucket.host` addresses, while R2 does not. + +Because the parameters travel in the URL, they are stored with the remote and +survive `clone`, `push`, `pull`, and backups without any extra flags. One +repository can therefore address several providers at once. + +Cloudflare R2, where the endpoint is your account's S3 API host: + +```bash +export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... +dolt remote add origin 's3://my-bucket/menus?endpoint=https://ACCOUNT_ID.r2.cloudflarestorage.com®ion=auto' +dolt push origin main +``` + +MinIO, which needs path-style addressing: + +```bash +dolt clone 's3://my-bucket/menus?endpoint=http://minio.internal:9000®ion=us-east-1&path-style=true' +``` + +Quote the URL in your shell. An unquoted `&` will otherwise be read as a +request to run the command in the background. + +An unrecognized parameter is an error rather than being ignored, so a typo such +as `entpoint=` is reported when the remote is added instead of failing later +with a confusing connection error. + ### GCS Google Cloud Platform remotes use Google Cloud Storage (GCS). You can create or use an existing GCS bucket to host one or more [Dolt](https://doltdb.com) remotes. To add a GCP remote provide a URL with the `gs://` protocol like so: diff --git a/site/doltgres/src/content/reference/sql/version-control/remotes.md b/site/doltgres/src/content/reference/sql/version-control/remotes.md index 9ee3e5d..66d269d 100644 --- a/site/doltgres/src/content/reference/sql/version-control/remotes.md +++ b/site/doltgres/src/content/reference/sql/version-control/remotes.md @@ -107,6 +107,64 @@ This same URL can then be used to clone this database by another user. select dolt_clone('aws://[dolt_remotes:dolt_remotes_storage]/menus') ``` +If you would rather not run a DynamoDB table, the `s3://` protocol below stores +the manifest in the bucket itself and works against S3 as well. + +## S3-compatible + +The `s3://` protocol works against any object store that implements the S3 API +with conditional writes, including AWS S3 itself, Cloudflare R2, and MinIO. It +stores the manifest as an ordinary object and uses conditional writes to update +it atomically, so unlike `aws://` it needs no DynamoDB table. + +```sql +select dolt_remote('add', 'origin', 's3://BUCKET/path/for/remote') +``` + +Credentials come from the standard AWS SDK chain as seen by the Doltgres server +process: environment variables, the shared credentials file, SSO, or an +instance role. They are never accepted in the URL, since the URL is stored with +the remote in plain text. + +Against AWS S3 nothing further is needed, because the SDK resolves the endpoint +from the server's configured region. + +### Addressing another provider + +A non-AWS provider is reached by adding query parameters to the URL: + +| Parameter | Meaning | +| ------------ | ------------------------------------------------------------------------ | +| `endpoint` | The provider's S3 API host, e.g. an R2 or MinIO endpoint | +| `region` | The signing region. Some providers want a fixed value; R2 uses `auto` | +| `path-style` | `true` to address buckets as a path segment instead of a hostname prefix | + +`endpoint` and `region` fall back to the SDK's own resolution when omitted, so +`AWS_ENDPOINT_URL_S3`, `AWS_REGION`, and the shared config file all work as +usual. `path-style` has no such fallback and can only be set in the URL. Leave +it off unless your provider needs it; MinIO generally does, because it has no +wildcard DNS to serve `bucket.host` addresses, while R2 does not. + +Because the parameters travel in the URL, they are stored with the remote and +survive clone, push, and pull without any extra configuration. One database can +therefore address several providers at once. + +Cloudflare R2, where the endpoint is your account's S3 API host: + +```sql +select dolt_remote('add', 'origin', 's3://my-bucket/menus?endpoint=https://ACCOUNT_ID.r2.cloudflarestorage.com®ion=auto') +``` + +MinIO, which needs path-style addressing: + +```sql +select dolt_clone('s3://my-bucket/menus?endpoint=http://minio.internal:9000®ion=us-east-1&path-style=true') +``` + +An unrecognized parameter is an error rather than being ignored, so a typo such +as `entpoint=` is reported when the remote is added instead of failing later +with a confusing connection error. + ## GCS Google Cloud Platform remotes use Google Cloud Storage (GCS). You can create or use an existing GCS bucket to host one or more Doltgres remotes. To add a GCP remote provide a URL with the `gs://` protocol like so: