Skip to content

fix: mark primary key columns as non-nullable in output manifests - #133

Draft
devin-ai-integration[bot] wants to merge 1 commit into
masterfrom
devin/1785833556-pk-non-nullable
Draft

devin-ai-integration[bot] wants to merge 1 commit into
masterfrom
devin/1785833556-pk-non-nullable

Conversation

@devin-ai-integration

Copy link
Copy Markdown

Summary

Output manifests declared every column nullable: true, including the primary key column. Storage forces a PK column non-nullable when the primary key is created (on BigQuery: column mode REQUIRED), so from the second run on, output mapping saw a diff and issued updateTableColumnDefinition(tableId, 'id', ['nullable' => true]). BigQuery refuses to relax a key column and the job fails permanently — recreating the table doesn't help, since applying the PK makes the column REQUIRED again:

ALTER operation partially failed: 1/0.
 Succeeded: none
Failed: "KBC.datatype.nullable": Key column id cannot be modified or removed.
        column's mode changed: REQUIRED -> NULLABLE

The fix ties nullability to the primary key flag in the three places that build manifest schemas — Output::createManifest (report tables, PK id) and the profiles (PK id) / properties (PK propertyKey) system tables in Component:

-    true,                                    // nullable
-    isset($primaryKey) && in_array($column, $primaryKey, true),
+    !$isPrimaryKey,
+    $isPrimaryKey,

Resulting manifest for a report table (both formats, PK id):

{"schema":[{"nullable":false,"primary_key":true,"name":"id","data_type":{"base":{"type":"STRING"}}}, ...]}
{"primary_key":["id"],"column_metadata":{"id":[{"key":"KBC.datatype.nullable","value":false}, ...]}}

Only the PK column changes; all other columns stay nullable. Functional-test expected manifests updated accordingly.

Reported for a configuration on europe-west3.gcp (BigQuery) whose every run failed with the error above.

Release Notes

Justification, description

Configurations of keboola.ex-google-analytics-v4 writing to BigQuery-backed projects fail on every run once the output table's primary key exists, because the manifest asks Storage to make the key column nullable. This makes the manifest self-consistent (PK ⇒ non-nullable).

Plans for Customer Communication

N/A

Impact Analysis

Affects the nullable flag of primary key columns (id, propertyKey) in output manifests only. Existing typed tables already have those columns non-nullable (Storage enforces it for PK columns), so no ALTER is triggered by the new value; the previously attempted (and failing) relax-to-nullable ALTER stops being requested. Untyped/legacy tables are unaffected in structure — only the KBC.datatype.nullable metadata value changes.

Deployment Plan

Standard component release via Developer Portal (tag on master).

Rollback Plan

Revert the commit and re-release the previous tag.

Post-Release Support Plan

Re-run an affected configuration and confirm the job finishes without the KBC.datatype.nullable ALTER error.

Link to Devin session: https://app.devin.ai/sessions/394588d8be5546c3af6f4625e1a08710

Co-Authored-By: Adam Zetocha <adam.zetocha@keboola.com>
@devin-ai-integration

Copy link
Copy Markdown
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration

Copy link
Copy Markdown
Author

CI is red for a pre-existing reason unrelated to this change: the Google OAuth REFRESH_TOKEN secret is expired, so every test that talks to the API fails with invalid_grant on POST https://www.googleapis.com/oauth2/v4/token.

Verified by pushing an empty commit off unmodified master — identical result (Tests: 30, Assertions: 40, Errors: 3, Failures: 4, Skipped: 2) in both runs. The credentials need refreshing before this repo's CI can pass again.

Locally (Docker, php 8.3): phplint, phpcs and phpstan --level=max pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants