Skip to content

feat(review): incremental and snapshot semantics that fit any dbt project - #1725

Draft
RaghwendraSingh wants to merge 4 commits into
feat/review-contracts-dialectfrom
feat/review-incremental-semantics
Draft

RaghwendraSingh wants to merge 4 commits into
feat/review-contracts-dialectfrom
feat/review-incremental-semantics

Conversation

@RaghwendraSingh

@RaghwendraSingh RaghwendraSingh commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Issue for this PR

Closes #

Stacked on #1724 (← #1723 ← … ← #1719); merge in that order. Draft: the new rules are not yet measured on a public-repo benchmark.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

dbt incremental and snapshot semantics. Rules follow dbt-core behavior, which is the same on every adapter. The one adapter-dependent rule reads the capability table and stays silent where the behavior isn't established. Config comes from the manifest (project, yml and inline, as dbt merges them), with the head's inline {{ config() }} on top. Every finding is advisory and never blocks.

Rules

Rule Fires when Notes
History plan (suggestion) An existing incremental model gains a column (old rows get NULL) or its is_incremental() filter changes, and the PR doesn't say how existing rows are handled Tokens are compared, so formatting never counts
partition_by change Changed on an incremental model Needs a full refresh
on_schema_change='fail' Columns are added or removed The first incremental run after merge fails
Snapshot tracking change check_cols or strategy changes on an existing snapshot .sql and dbt 1.9+ yml snapshots
Load-timestamp dedup (suggestion) row_number() ordered only by loaded_at / ingestion_timestamp Loader sync columns (_fivetran_synced, _airbyte_*, _sdc_*) are not flagged
Per-slice unique_key (opt-in: incrementalSliceVars) A delete+insert/merge model filtered by a listed var (e.g. a tenant) lacks that column in unique_key Otherwise one slice's run deletes or overwrites another's rows
Trino non-atomic rebuild (suggestion) on_table_exists='replace' with an enforced contract, or drop replace + contract runs create or replace then insert; drop removes the table during the build. Checked against the dbt-trino 1.10.6 source

unique_key and strategy changes are left to the engine's DBT011/DBT012.

Also in this PR

  • Dialect-pack rules (BigQuery: …, Snowflake: …, Redshift: …, Databricks: …) fire only on their own engine once the project's engine is known. A DuckDB project was getting BigQuery partition advice.
  • Column-set rules now see literal and function-only columns (current_date as load_date). Column lineage only reports columns that read a source column. select-list.ts adds the final SELECT's own names: an item counts only when it's aliased after a complete expression or is a bare column, and * defers to lineage.

Changed after an adversarial generality review, before opening. The goal was rules that fit any dbt project, not one monorepo:

  • the history rule no longer fires on every incremental edit
  • the slice rule is opt-in
  • a Trino/Athena microbatch-concurrency rule was removed: dbt-trino's concurrency default is changing, and the table format can't be known from the adapter
  • loader sync columns are no longer treated as load timestamps

How did you verify your code works?

  • New tests: test/altimate/review-incremental-semantics.test.ts covers the generic rules on two adapters, formatting-only edits, the opt-ins, yml snapshots and orchestrator wiring. test/altimate/review-select-list.test.ts covers the select-list names.
  • Suites: 495 tests across the review files pass, bun typecheck is clean, and the real-world corpus is unchanged (15/15 caught, 0/5 false positives).
  • Local end-to-end run. A real dbt 1.10 DuckDB project, released 0.12.10 vs this stack. A snapshot check_cols change and adding a column under on_schema_change='fail' are reported; 0.12.10 reports neither. A model reading a tenant var with no incrementalSliceVars configured stays quiet.
  • Replay of 7 real PRs from a private dbt monorepo. 0 off-diff findings, no earlier catch lost. One new true positive: a contracted mart on Trino with project-level on_table_exists: replace.

Cost side effects

None. No model calls or warehouse queries.

Screenshots / recordings

N/A (CLI).

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code


Summary by cubic

Adds dbt incremental and snapshot semantics to PR review as advisory findings, following dbt-core behavior that is the same on every adapter. Config is read from the manifest (project, yml, inline, as dbt merges them) with the head's inline {{ config() }} on top; formatting-only edits never count as changes.

Rules fire when an incremental model gains a column or its is_incremental() filter changes without a stated plan for existing rows, partition_by changes, on_schema_change='fail' meets added or removed columns, a snapshot's check_cols/strategy or query changes, or dedup orders only by a load timestamp. Per-slice runs are opt-in via incrementalSliceVars, and each must include the slice column in unique_key. On Trino, on_table_exists='replace' with an enforced contract and drop get a suggestion because readers can see an empty or missing table. Adapter-dependent advice reads the capability table and stays silent where behavior isn't established, dialect pack rules fire only on their own engine, and column-set rules now see literal and function-only columns (current_date as load_date) via the final SELECT's own names. Also fixes a false "table not found" warning against a manifest-only schema (category → kind).

Written for commit fe31dc6. Summary will update on new commits.

View guided diff Turn on auto-fix

RaghwendraSingh and others added 4 commits November 13, 2026 11:12
…apter-dependent rules read from the capability table

- `incremental-semantics.ts`: config from the manifest (dbt-merged) overlaid by the head's inline `{{ config() }}`; a new `modelConfig` runner method
- incremental model or snapshot logic changed with no stated plan for history (`--full-refresh`, backfill, forward-only) in the PR
- `partition_by` changed on an incremental model
- `on_schema_change='fail'` with added or removed output columns
- `delete+insert`/`merge` whose `unique_key` lacks the column a `var()` filter slices runs by (one tenant's run deleting another's rows)
- snapshot `check_cols`/`strategy` change, and snapshot query change without a plan
- `row_number()` dedup ordered only by a load timestamp
- capability table: Trino/Athena concurrent Iceberg commits conflict (microbatch without `concurrent_batches=false`); Trino `on_table_exists='replace'` + enforced contract is create-then-insert and `drop` removes the table during the build (warning when the model has consumers); sources cite dbt-trino and Iceberg docs
- all findings are advisory (catalog origin); unchanged real-world corpus: 15/15 caught, 0/5 false positives

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a; dialect packs only on their own engine

Found by the local end-to-end run on a real DuckDB dbt project:

- the partial-schema `TableNotFound` filter read `category`, but the engine reports the type under `kind`, so every model reading an undocumented upstream still got an engine `validate` warning (50 new models → 50 warnings → the gate blocked). The runner test now uses the engine's real shape
- BigQuery/Snowflake/Redshift/Databricks pack rules (titled `Engine: …`) fire only on that engine once the project's engine is known; unknown engine keeps today's behavior

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ory rule, opt-in slice vars, no microbatch guess, yml snapshots

From an adversarial generality review:

- the history rule fires only when an incremental model gains a column or its `is_incremental()` filter changes, compares tokens (formatting never counts), defers to the `on_schema_change='fail'` finding, and is a suggestion
- the slice-column `unique_key` rule is opt-in: only vars listed in `incrementalSliceVars` count (a `reporting_currency` var filter is not a slice)
- the Trino/Athena microbatch rule and its capability entry are removed: dbt-trino's concurrency default is changing and the connector/table format isn't knowable from the adapter
- the non-atomic Trino rebuild is always a suggestion and speaks to readers outside dbt, not downstream dbt models
- load-timestamp dedup ignores loader sync columns (`_fivetran_synced`, `_airbyte_*`, `_sdc_*`) and is a suggestion
- snapshot `check_cols`/`strategy` changes are also caught in dbt 1.9+ yml snapshots; snapshot query changes compare tokens

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the local end-to-end run: column lineage only reports columns that read a source column, so `current_date as load_date` or `1 as one` were invisible and adding one under `on_schema_change='fail'` went unreported. Output columns now add the final SELECT's own names (`select-list.ts`: an item counts only when aliased after a complete expression or a bare column; `*` defers to lineage).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 13, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

👋 This PR was automatically closed by our quality checks.

Common reasons:

  • New GitHub account with limited contribution history
  • PR description doesn't meet our guidelines
  • Contribution appears to be AI-generated without meaningful review

If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant