Skip to content

Improve Docs for IRC Catalog Connection #2175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kevinjqliu merged 6 commits into from
Jul 8, 2025
Merged

Improve Docs for IRC Catalog Connection #2175

Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
086aef5
improve REST catalog documentation with examples and configuration de…
james5418 Jul 6, 2025
136a7be
adjust the position of the auth configuration in the docs
james5418 Jul 7, 2025
09a3846
refactor auth options in doc
james5418 Jul 8, 2025
710426e
Update mkdocs/docs/configuration.md
kevinjqliu Jul 8, 2025
8cbb3a7
Update mkdocs/docs/configuration.md
kevinjqliu Jul 8, 2025
0055ac1
make lint
kevinjqliu Jul 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 82 additions & 15 deletions mkdocs/docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -339,25 +339,19 @@ catalog:

| Key | Example | Description |
| ------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------- |
| uri | <https://rest-catalog/ws> | URI identifying the REST Server |
| uri | <https://rest-catalog/ws> | URI identifying the REST Server |
| ugi | t-1234:secret | Hadoop UGI for Hive client. |
Comment thread
james5418 marked this conversation as resolved.
Outdated
| credential | t-1234:secret | Credential to use for OAuth2 credential flow when initializing the catalog |
| token | FEW23.DFSDF.FSDF | Bearer token value to use for `Authorization` header |
| scope | openid offline corpds:ds:profile | Desired scope of the requested security token (default : catalog) |
| resource | rest_catalog.iceberg.com | URI for the target resource or service |
| audience | rest_catalog | Logical name of target resource or service |
Comment thread
james5418 marked this conversation as resolved.
Outdated
| rest.sigv4-enabled | true | Sign requests to the REST Server using AWS SigV4 protocol |
| rest.signing-region | us-east-1 | The region to use when SigV4 signing a request |
| rest.signing-name | execute-api | The service signing name to use when SigV4 signing a request |
| oauth2-server-uri | <https://auth-service/cc> | Authentication URL to use for client credentials authentication (default: uri + 'v1/oauth/tokens') |
| snapshot-loading-mode | refs | The snapshots to return in the body of the metadata. Setting the value to `all` would return the full set of snapshots currently valid for the table. Setting the value to `refs` would load all snapshots referenced by branches or tags. |
| warehouse | myWarehouse | Warehouse location or identifier to request from the catalog service. May be used to determine server-side overrides, such as the warehouse location. |
| snapshot-loading-mode | refs | The snapshots to return in the body of the metadata. Setting the value to `all` would return the full set of snapshots currently valid for the table. Setting the value to `refs` would load all snapshots referenced by branches or tags. |
| warehouse | myWarehouse | Warehouse location or identifier to request from the catalog service. May be used to determine server-side overrides, such as the warehouse location. |
Comment thread
james5418 marked this conversation as resolved.
| `header.X-Iceberg-Access-Delegation` | `vended-credentials` | Signal to the server that the client supports delegated access via a comma-separated list of access mechanisms. The server may choose to supply access via any or none of the requested mechanisms. When using `vended-credentials`, the server provides temporary credentials to the client. When using `remote-signing`, the server signs requests on behalf of the client. (default: `vended-credentials`) |

<!-- markdown-link-check-enable-->

#### Headers in RESTCatalog

To configure custom headers in RESTCatalog, include them in the catalog properties with the prefix `header.`. This
To configure custom headers in RESTCatalog, include them in the catalog properties with `header.<Header-Name>`. This
ensures that all HTTP requests to the REST service include the specified headers.

```yaml
Expand All @@ -368,11 +362,84 @@ catalog:
header.content-type: application/vnd.api+json
```

Specific headers defined by the RESTCatalog spec include:

| Key | Options | Default | Description |
| ------------------------------------ | ------------------------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `header.X-Iceberg-Access-Delegation` | `{vended-credentials,remote-signing}` | `vended-credentials` | Signal to the server that the client supports delegated access via a comma-separated list of access mechanisms. The server may choose to supply access via any or none of the requested mechanisms |
#### Authentication Options
Comment thread
james5418 marked this conversation as resolved.
Outdated
- **SigV4**: For AWS services that require SigV4 signing.
- **OAuth2**: For services that require OAuth2 authentication.
Comment thread
james5418 marked this conversation as resolved.
Outdated
- **Bearer Token**: Use the `token` property to pass a bearer token directly for services that accept token-based authentication.
- **Client Credentials**: Use the `credential` property with the format `client_id:client_secret` to perform the OAuth2 client credentials flow. Optionally, use the `oauth2-server-uri` property to specify a custom OAuth2 endpoint for client credentials authentication.

| Key | Example | Description |
| ------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------- |
| rest.sigv4-enabled | true | Sign requests to the REST Server using AWS SigV4 protocol |
| rest.signing-region | us-east-1 | The region to use when SigV4 signing a request |
| rest.signing-name | execute-api | The service signing name to use when SigV4 signing a request |
| oauth2-server-uri | <https://auth-service/cc> | Authentication URL to use for client credentials authentication (default: uri + 'v1/oauth/tokens') |
| token | FEW23.DFSDF.FSDF | Bearer token value to use for `Authorization` header |
| credential | t-1234:secret | Credential to use for OAuth2 credential flow when initializing the catalog |

<!-- markdown-link-check-enable-->


#### Common Integrations & Examples

##### AWS Glue
```yaml
catalog:
s3_tables_catalog:
type: rest
uri: https://glue.<region>.amazonaws.com/iceberg
warehouse: <account-id>:s3tablescatalog/<table-bucket-name>
rest.sigv4-enabled: true
rest.signing-name: glue
rest.signing-region: <region>
```

##### Unity Catalog
```yaml
catalog:
unity_catalog:
type: rest
uri: https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest
warehouse: <uc-catalog-name>
token: <databricks-pat-token>
```

##### R2 Data Catalog
```yaml
catalog:
r2_catalog:
type: rest
uri: <r2-catalog-uri>
warehouse: <r2-warehouse-name>
token: <r2-token>
```

##### Lakekeeper
```yaml
catalog:
lakekeeper_catalog:
type: rest
uri: <lakekeeper-catalog-uri>
warehouse: <lakekeeper-warehouse-name>
credential: <client-id>:<client-secret>
oauth2-server-uri: http://localhost:30080/realms/<keycloak-realm-name>/protocol/openid-connect/token
scope: lakekeeper
```

##### Apache Polaris
```yaml
catalog:
polaris_catalog:
type: rest
uri: https://<account>.snowflakecomputing.com/polaris/api/catalog
warehouse: <polaris-catalog-name>
credential: <client-id>:<client-secret>
header.X-Iceberg-Access-Delegation: vended-credentials
scope: PRINCIPAL_ROLE:ALL
token-refresh-enabled: true
py-io-impl: pyiceberg.io.fsspec.FsspecFileIO
```

### SQL Catalog

Expand Down