Skip to content

Add BigQuery warehouse adapter #57

Description

@carolsimone

Add a BigQuery warehouse engine: one WarehouseAdapter implementation, published as continuo-bigquery-adapter, with a continuo-python-runtime-bigquery base image.

Part of the warehouse-engines rollout tracked in carolsimone/continuo#517. Sibling engines: Snowflake (#58), Redshift (#59).

Scope

Mirror adapters/postgres/ and adapters/trino/. A new engine is one WarehouseAdapter subclass serving both roles (data plane + validation) and one image that runs both.

  • New uv workspace member adapters/bigquery/: continuo_bigquery_adapter/ (the adapter class), pyproject.toml, tests/, README.md. Add it to [tool.uv.workspace] in the root pyproject.toml.
  • Implement the data-plane methods: fetch, ensure_table, load.
  • Implement the validation methods: ensure_schema, drop_schema, build_empty_from_sql, build_empty_from_columns, clone_empty_from_prod, check_binds.
  • Register the adapter under the continuo_engine.adapters entry-point group (so the runtime discovers it at run time).
  • Publish continuo-bigquery-adapter on the shared vX.Y.Z tag (wire into publish-pypi.yml).
  • Add Dockerfile.bigquery + image-requirements-bigquery.txt; wire into images.yml → base image continuo-python-runtime-bigquery (installs the pinned continuo-bigquery-adapter==X.Y.Z).
  • Tests: adapter unit tests + the shared adapter conformance suite.

Engine notes

  • Auth: Application Default Credentials / service account; project + GOOGLE_APPLICATION_CREDENTIALS.
  • BigQuery dataset maps to the adapter's notion of schema (ensure_schema / drop_schema create/drop a dataset).
  • clone_empty_from_prod: CREATE TABLE ... LIKE is not available the same way; use CREATE TABLE ... AS SELECT ... WHERE FALSE or a schema-only copy.
  • Types: map the contract type set to BigQuery standard-SQL types.

Do not touch

The result-block wire contract is frozen (docs/boundary-contract.md, continuo_engine_contract.result). The Go side (pkg/validationresult in carolsimone/continuo) parses it byte-for-byte. Reuse it as-is.

Reference

  • adapters/trino/ and adapters/postgres/ — the two existing engines to copy from.
  • docs/boundary-contract.md — the external boundary contract.
  • continuo-engine-contract (contract/) — the WarehouseAdapter port and the shared SQL/type/config guards.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions