fix(bigquery): run the partition-field check on all three destinations (v0.5.4)

[anaselmhamdi]

The gap

0.5.3 added a config-time check that time_partitioning.field exists in the schema the destination writes — and wired it to one of the three BigQuery destinations. BigQueryStreamingConfigDetails and BigQueryStreamingV2ConfigDetails have no model_validator at all, so unnest: true streaming pipelines were never covered by the guardrail the release note described.

Why the obvious fix would have been worse

• In STREAM sync mode bigquery_streaming_v2.finalize() returns immediately (destination.py:560), so _partition_clause() — the only thing that validates the field on v2 — is never reached. bigquery_streaming v1 has no finalize() at all.
• On both, the field is therefore only ever touched by create_table, which raises Conflict on an existing table and silently discards the spec.

So a v1 or v2 stream pipeline can carry an arbitrarily wrong time_partitioning.field and run correctly, forever. After the first run every table exists — that's the normal state of every streaming pipeline, not a corner case. Extending 0.5.3's hard config-time check to the streaming destinations would have stopped all of them on the bump.

The rule

Raise only where BigQuery itself would raise; warn everywhere else — equivalently, check the spec that is actually about to be applied to a table, not the config in the abstract.

situation	today	after
batch bigquery, bad field — every load job stamps the spec onto the temp table it creates	config error (0.5.3)	config error, unchanged
batch, field present but STRING/JSON (_bizon_id, _source_data)	passes config, load job fails every run	config error
streaming, bad field, table exists	works (spec discarded on Conflict)	works, plus a warning naming both specs
streaming, bad field, table absent	opaque BigQuery error on create_table	raises, naming the field and the columns
v2 full refresh, bad field, table mismatched, flag off	_publish_spec() emits the table's spec; works	works, plus warning
unnest: true, no time_partitioning written	batch: config error. streaming: fails at create_table	ingestion-time partitioning; works
non-unnest default / bare time_partitioning: DAY	DAY / _bizon_loaded_at	identical

No currently-working pipeline changes behaviour. Every new raise replaces an existing opaque failure.

Changes

• describe_partition_problem() (bigquery/src/config.py) returns a message instead of raising, so each of the six call sites picks its own severity. It now covers the column's type as well as its presence — a STRING partition column or HOUR on a DATE column is caught rather than left to BigQuery.
• resolve_partition_field() settles the implicit default: field defaults to _bizon_loaded_at, which unnest: true can never produce. An unwritten field falls back to ingestion-time partitioning and logs it; a field the user actually wrote is still checked. Nobody has to touch a config to upgrade.
• should_apply_partitioning() (partitioning.py) is the absent-vs-exists adjudicator shared by both streaming destinations. The get_table probe only runs when there is a problem, and only once per table per process.
• bigquery_streaming's Conflict handler reconciled the schema while ignoring partitioning entirely, so a config that had drifted from its live table produced no signal, run after run. It now compares both specs and warns. (BigQuery still cannot repartition in place; rebuilding remains the only fix.)
• Both streaming configs declared time_partitioning with default=TimePartitioning(...) — pydantic v2 does not copy a BaseModel passed as default, so every config in the process shared one instance. Latent until the new validator mutated it. Both now use default_factory, which also makes model_fields_set truthful — that's how "did the user write field?" is answered.
• CLAUDE.md gains a BigQuery partitioning semantics section, since the severity split is exactly the kind of thing a later change "unifies" and breaks.

Verification

• New tests/connectors/destinations/test_partition_field_validation.py — 44 tests parametrized over all three destinations, covering both the config layer and the runtime absent/present split. It sits outside the bigquery*/ directories that CI ignores, so unlike 0.5.3's partitioning tests it actually runs in CI.
• The shipped kafka_streams.example.yml (the production shape: bigquery_streaming_v2, unnest: true, field: __inserted_at, TIMESTAMP in both streams) validates untouched and is now a regression test, along with a case that drops the column from one stream and asserts the warning names it.
• Ran the exact CI pytest command before and after: identical 18-failure set (no local Postgres, Slack needs network). Same for tests/connectors/destinations: identical 14, all live-BigQuery. No new failures.
• Validated all 26 shipped example configs before and after: identical status for every one.

Not in scope

kafka_debezium.example.yml sets table_id: and record_schema: (singular) directly on the destination config — both unknown keys under extra="forbid", so that example cannot parse. Pre-existing, fails identically before and after this change. Worth its own fix.

🤖 Generated with Claude Code