fix(bigquery): make partitioning configurable, detected and repairable (v0.5.3)

[anaselmhamdi]

Three related BigQuery partitioning defects, all silent — no error, just tables that scan in full forever.

What was wrong

1. No self-heal for legacy tables. The batch destination publishes with a copy job (_copy_temp_to_main). A copy job into a table that already exists keeps that table's own partitioning spec, so a table first created by the pre-0.4.0 CREATE TABLE AS SELECT is unpartitioned and stays that way forever, however time_partitioning is configured, with nothing in the logs.

2. The partition column was hardcoded. _build_load_job_config() always passed field="_bizon_loaded_at", and time_partitioning was a bare DAY|HOUR|MONTH|YEAR enum. With unnest: true the table holds only record_schemas columns, so that column does not exist and the load job failed mid-run.

3. bigquery_streaming_v2 threw away the layout on publish. The staging table was created with time_partitioning and clustering_fields, then finalize() published with CREATE OR REPLACE TABLE main AS SELECT * FROM temp — CTAS inherits neither. Its incremental path was worse: INSERT INTO 404s when the main table does not yet exist.

Verified BigQuery behavior

The design hinged on facts I could not take on trust, so I ran them against the API on a throwaway dataset in gorgias-growth-development:

Experiment	Result
cp -f (WRITE_TRUNCATE) partitioned → existing unpartitioned	Succeeds silently, destination stays unpartitioned
cp -f partitioned+clustered → nonexistent destination	Inherits both
cp -a (WRITE_APPEND) partitioned → existing unpartitioned	Succeeds silently, spec unchanged
CREATE OR REPLACE TABLE ... PARTITION BY over existing unpartitioned table	ERRORS: Cannot replace a table with a different partitioning spec. Instead, DROP the table, and then recreate it.
Same DDL when the target does not exist	Creates it partitioned + clustered
Plain CTAS over an existing partitioned table	ERRORS symmetrically

Two consequences that shaped the implementation:

• BigQuery cannot change a table's partitioning at all. Dropping and recreating is the only mechanism, which is why the repair is opt-in.
• Defect 3's mechanism is not what it looked like. streaming_v2's CTAS does not strip partitioning every run — it creates the table unpartitioned on the first run and BigQuery pins it there. So naively adding PARTITION BY would make every existing streaming_v2 pipeline fail on its next run. It therefore only emits the clauses when the table is absent or already matches, and otherwise falls back to the historical plain CTAS with a warning.

What changed

• time_partitioning takes the {type, field} shape the streaming destinations already had, shared by all three (they had three byte-identical copies that had drifted). A bare time_partitioning: DAY still validates.
• field is validated at config load and against the resolved schema at runtime — the stream runner injects record_schemas after validation, so neither layer alone is sufficient. field: null selects ingestion-time partitioning.
• finalize() compares the staging and destination specs and warns, naming both.
• enforce_partitioning (default false, on bigquery and bigquery_streaming_v2): a full refresh drops and rebuilds a mismatched table. Restricted to full refreshes, where the staging table already holds every row, so the rebuild is lossless and free. Incremental never rebuilds — it warns and points at bizon stream reset.
• streaming_v2's publish DDL carries PARTITION BY/CLUSTER BY; incremental creates the main table first.
• bigquery_streaming (v1) turns out not to have defect 3 — no finalize(), no staging table, and it already sets partitioning and clustering on the table it appends to. Config model only.

Testing

65 new mock tests (test_bigquery_partitioning.py, test_bigquery_streaming_v2_partitioning.py) plus a shared conftest.py harness; two existing tests that were passing by accident made explicit. Full live end-to-end runs against real BigQuery:

1. default full refresh → partitioned on _bizon_loaded_at/DAY ✅
2. legacy unpartitioned table, flag off → warns, run succeeds, table unchanged ✅
3. same table, flag on → dropped and republished partitioned ✅
4. {type: HOUR, field: _bizon_extracted_at} → applied ✅
5. bare time_partitioning: MONTH → applied (back-compat) ✅
6. streaming_v2 full refresh → published partitioned ✅
7. streaming_v2 over a legacy table, flag off → warns, run does not fail ✅
8. streaming_v2 incremental first run → creates the table, no 404 (control confirms the old path did 404) ✅

Also confirmed pip install bizon with no extras still imports (config.py stays free of google.cloud.bigquery — the 0.5.1 regression class).

Pre-existing test failures on this machine (tests/cli, tests/alerting, tests/engine/backend) are a local Postgres credential conflict; confirmed identical on pristine v0.5.2.

Not included

Per review scope: batch clustering_keys remains a silent no-op for destination.name: bigquery; pytest.yml still --ignores the whole BigQuery test tree, so these new mock tests will not run in CI. Both worth follow-ups.

🤖 Generated with Claude Code

[anaselmhamdi]

Follow-up commit: CREATE OR REPLACE is rejected in both directions

Reviewing my own diff I found a real bug in the first commit. I had treated BigQuery's restriction as "you cannot add partitioning to an unpartitioned table," so the streaming_v2 fallback dropped the clauses and emitted a bare CTAS. But the restriction is symmetric — a bare CTAS against a table partitioned on a different column fails just as hard:

New partitioning spec is none and existing spec is interval(type:day,field:ts)

That path is reachable the moment someone changes time_partitioning.field on a live pipeline — which this release is what makes possible.

_publish_spec() now decides what the DDL declares:

Destination table	Declared spec
absent	configured
matches config	configured
differs, enforce_partitioning: false	current (run keeps working, warns)
differs, enforce_partitioning: true	configured (table dropped first)

Verified live: main table on _bizon_extracted_at, config asking for _bizon_loaded_at → publishes successfully with

Partitioning mismatch on …v2_drift: the table is partitioned by _bizon_extracted_at (DAY), but the config asks for partitioned by _bizon_loaded_at (DAY). BigQuery cannot change an existing table's partitioning.
Publishing …v2_drift as partitioned by _bizon_extracted_at (DAY) to keep the run working. Set enforce_partitioning: true …

Three tests added for the new direction; 131 BigQuery mock tests green.

Unrelated pre-existing bug noticed while testing

bigquery_streaming_v2 intermittently fails a full-refresh run that starts shortly after a previous one on the same table:

404 Table …:dataset.table_temp is re-created.
Entity: projects/…/tables/table_temp/_default

finalize() deletes the staging table and the next run recreates it, and the Storage Write API's _default stream metadata cache lags behind. It is timing-dependent — back-to-back runs succeed as often as not. This PR does not change the staging-table lifecycle, so it is not introduced here; flagging it as a separate issue worth filing.