Repository navigation
feat(review): incremental and snapshot semantics that fit any dbt project - #1725
Draft
RaghwendraSingh wants to merge 4 commits into
Draft
RaghwendraSingh wants to merge 4 commits into
RaghwendraSingh wants to merge 4 commits into
Conversation
…apter-dependent rules read from the capability table
- `incremental-semantics.ts`: config from the manifest (dbt-merged) overlaid by the head's inline `{{ config() }}`; a new `modelConfig` runner method
- incremental model or snapshot logic changed with no stated plan for history (`--full-refresh`, backfill, forward-only) in the PR
- `partition_by` changed on an incremental model
- `on_schema_change='fail'` with added or removed output columns
- `delete+insert`/`merge` whose `unique_key` lacks the column a `var()` filter slices runs by (one tenant's run deleting another's rows)
- snapshot `check_cols`/`strategy` change, and snapshot query change without a plan
- `row_number()` dedup ordered only by a load timestamp
- capability table: Trino/Athena concurrent Iceberg commits conflict (microbatch without `concurrent_batches=false`); Trino `on_table_exists='replace'` + enforced contract is create-then-insert and `drop` removes the table during the build (warning when the model has consumers); sources cite dbt-trino and Iceberg docs
- all findings are advisory (catalog origin); unchanged real-world corpus: 15/15 caught, 0/5 false positives
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a; dialect packs only on their own engine Found by the local end-to-end run on a real DuckDB dbt project: - the partial-schema `TableNotFound` filter read `category`, but the engine reports the type under `kind`, so every model reading an undocumented upstream still got an engine `validate` warning (50 new models → 50 warnings → the gate blocked). The runner test now uses the engine's real shape - BigQuery/Snowflake/Redshift/Databricks pack rules (titled `Engine: …`) fire only on that engine once the project's engine is known; unknown engine keeps today's behavior Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ory rule, opt-in slice vars, no microbatch guess, yml snapshots From an adversarial generality review: - the history rule fires only when an incremental model gains a column or its `is_incremental()` filter changes, compares tokens (formatting never counts), defers to the `on_schema_change='fail'` finding, and is a suggestion - the slice-column `unique_key` rule is opt-in: only vars listed in `incrementalSliceVars` count (a `reporting_currency` var filter is not a slice) - the Trino/Athena microbatch rule and its capability entry are removed: dbt-trino's concurrency default is changing and the connector/table format isn't knowable from the adapter - the non-atomic Trino rebuild is always a suggestion and speaks to readers outside dbt, not downstream dbt models - load-timestamp dedup ignores loader sync columns (`_fivetran_synced`, `_airbyte_*`, `_sdc_*`) and is a suggestion - snapshot `check_cols`/`strategy` changes are also caught in dbt 1.9+ yml snapshots; snapshot query changes compare tokens Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the local end-to-end run: column lineage only reports columns that read a source column, so `current_date as load_date` or `1 as one` were invisible and adding one under `on_schema_change='fail'` went unreported. Output columns now add the final SELECT's own names (`select-list.ts`: an item counts only when aliased after a complete expression or a bare column; `*` defers to lineage). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
6 of 7 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Closes #
Stacked on #1724 (← #1723 ← … ← #1719); merge in that order. Draft: the new rules are not yet measured on a public-repo benchmark.
Type of change
What does this PR do?
dbt incremental and snapshot semantics. Rules follow dbt-core behavior, which is the same on every adapter. The one adapter-dependent rule reads the capability table and stays silent where the behavior isn't established. Config comes from the manifest (project, yml and inline, as dbt merges them), with the head's inline
{{ config() }}on top. Every finding is advisory and never blocks.Rules
is_incremental()filter changes, and the PR doesn't say how existing rows are handledpartition_bychangeon_schema_change='fail'check_colsorstrategychanges on an existing snapshot.sqland dbt 1.9+ yml snapshotsrow_number()ordered only byloaded_at/ingestion_timestamp_fivetran_synced,_airbyte_*,_sdc_*) are not flaggedunique_key(opt-in:incrementalSliceVars)delete+insert/mergemodel filtered by a listed var (e.g. a tenant) lacks that column inunique_keyon_table_exists='replace'with an enforced contract, ordropreplace+ contract runscreate or replacetheninsert;dropremoves the table during the build. Checked against the dbt-trino 1.10.6 sourceunique_keyand strategy changes are left to the engine'sDBT011/DBT012.Also in this PR
BigQuery: …,Snowflake: …,Redshift: …,Databricks: …) fire only on their own engine once the project's engine is known. A DuckDB project was getting BigQuery partition advice.current_date as load_date). Column lineage only reports columns that read a source column.select-list.tsadds the final SELECT's own names: an item counts only when it's aliased after a complete expression or is a bare column, and*defers to lineage.Changed after an adversarial generality review, before opening. The goal was rules that fit any dbt project, not one monorepo:
How did you verify your code works?
test/altimate/review-incremental-semantics.test.tscovers the generic rules on two adapters, formatting-only edits, the opt-ins, yml snapshots and orchestrator wiring.test/altimate/review-select-list.test.tscovers the select-list names.bun typecheckis clean, and the real-world corpus is unchanged (15/15 caught, 0/5 false positives).check_colschange and adding a column underon_schema_change='fail'are reported; 0.12.10 reports neither. A model reading a tenant var with noincrementalSliceVarsconfigured stays quiet.on_table_exists: replace.Cost side effects
None. No model calls or warehouse queries.
Screenshots / recordings
N/A (CLI).
Checklist
🤖 Generated with Claude Code
Summary by cubic
Adds dbt incremental and snapshot semantics to PR review as advisory findings, following dbt-core behavior that is the same on every adapter. Config is read from the manifest (project, yml, inline, as dbt merges them) with the head's inline
{{ config() }}on top; formatting-only edits never count as changes.Rules fire when an incremental model gains a column or its
is_incremental()filter changes without a stated plan for existing rows,partition_bychanges,on_schema_change='fail'meets added or removed columns, a snapshot'scheck_cols/strategyor query changes, or dedup orders only by a load timestamp. Per-slice runs are opt-in viaincrementalSliceVars, and each must include the slice column inunique_key. On Trino,on_table_exists='replace'with an enforced contract anddropget a suggestion because readers can see an empty or missing table. Adapter-dependent advice reads the capability table and stays silent where behavior isn't established, dialect pack rules fire only on their own engine, and column-set rules now see literal and function-only columns (current_date as load_date) via the final SELECT's own names. Also fixes a false "table not found" warning against a manifest-only schema (category→kind).Written for commit fe31dc6. Summary will update on new commits.