fix: mark primary key columns as non-nullable in output manifests - #133
Draft
devin-ai-integration[bot] wants to merge 1 commit into
Draft
devin-ai-integration[bot] wants to merge 1 commit into
devin-ai-integration[bot] wants to merge 1 commit into
Conversation
Co-Authored-By: Adam Zetocha <adam.zetocha@keboola.com>
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Author
|
CI is red for a pre-existing reason unrelated to this change: the Google OAuth Verified by pushing an empty commit off unmodified Locally (Docker, php 8.3): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Output manifests declared every column
nullable: true, including the primary key column. Storage forces a PK column non-nullable when the primary key is created (on BigQuery: column modeREQUIRED), so from the second run on, output mapping saw a diff and issuedupdateTableColumnDefinition(tableId, 'id', ['nullable' => true]). BigQuery refuses to relax a key column and the job fails permanently — recreating the table doesn't help, since applying the PK makes the columnREQUIREDagain:The fix ties nullability to the primary key flag in the three places that build manifest schemas —
Output::createManifest(report tables, PKid) and theprofiles(PKid) /properties(PKpropertyKey) system tables inComponent:Resulting manifest for a report table (both formats, PK
id):{"schema":[{"nullable":false,"primary_key":true,"name":"id","data_type":{"base":{"type":"STRING"}}}, ...]} {"primary_key":["id"],"column_metadata":{"id":[{"key":"KBC.datatype.nullable","value":false}, ...]}}Only the PK column changes; all other columns stay nullable. Functional-test expected manifests updated accordingly.
Reported for a configuration on
europe-west3.gcp(BigQuery) whose every run failed with the error above.Release Notes
Justification, description
Configurations of
keboola.ex-google-analytics-v4writing to BigQuery-backed projects fail on every run once the output table's primary key exists, because the manifest asks Storage to make the key column nullable. This makes the manifest self-consistent (PK ⇒ non-nullable).Plans for Customer Communication
N/A
Impact Analysis
Affects the
nullableflag of primary key columns (id,propertyKey) in output manifests only. Existing typed tables already have those columns non-nullable (Storage enforces it for PK columns), so no ALTER is triggered by the new value; the previously attempted (and failing) relax-to-nullable ALTER stops being requested. Untyped/legacy tables are unaffected in structure — only theKBC.datatype.nullablemetadata value changes.Deployment Plan
Standard component release via Developer Portal (tag on
master).Rollback Plan
Revert the commit and re-release the previous tag.
Post-Release Support Plan
Re-run an affected configuration and confirm the job finishes without the
KBC.datatype.nullableALTER error.Link to Devin session: https://app.devin.ai/sessions/394588d8be5546c3af6f4625e1a08710