Repository navigation
fix(review): precision and new rules from a replay of human-caught defects on public dbt PRs - #1739
RaghwendraSingh wants to merge 10 commits into
Conversation
- `var_no_default`: skip vars defined in `dbt_project.yml` (top level or project-scoped) or defaulted elsewhere in the same model - `nested-subqueries-deep`, `nested-case`: count real nesting with a paren / CASE stack instead of counting occurrences (sibling CTEs, two CASEs) - `case-when-no-end`: check the whole model, abstain on Jinja branches - `select-star-in-cte-final`: fire only when the final SELECT * reads straight from another relation, not from a CTE - partition-pruning: only flag a function that wraps a column, not a literal, a var or the clock - schema.yml test removal: tests moved to a renamed column are not removals - incremental filter change: compare predicates of the `is_incremental()` branch only, not select-list edits or the else branch - grain not_null finding is a suggestion with accurate wording Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- monorepo: per-file hunk diffs came back empty when the dbt project sits in a
subdirectory (`--name-status` paths are repo-root relative, the pathspec was
resolved against the project dir), so every diff-anchored rule and the risk
tier saw 0 changed lines; anchor pathspecs with `:(top)`
- join rules (`join-on-cast`, `join-or-condition`) match each JOIN's whole ON
clause, so a predicate on a continuation line counts; `join-on-cast` also
covers `date()`/`timestamp()`/`lower()` and similar around a column
- `config(**cfg)` with a literal `{% set cfg = {...} %}` dict is inlined before
the config lint, so incremental/materialization rules see it
- changed macros are reviewed: catalog rules marked `macros` and `var()` read
with no default
- a column removed from a final-layer model (marts, reporting, dimensional,
extracts) with no declared consumers is a suggestion
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eclared uniqueness test - `runner.ts`: `declaredUniqueKeys(model)` reads `unique` / `unique_combination_of_columns` tests attached to the model in the manifest - `orchestrate.ts`: `uniqueKeyMismatchLane` warns when an incremental model's `unique_key` matches none of its declared uniqueness keys - tests for match, mismatch, case-insensitivity and non-incremental models Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er a precision sample Measured on 423 replayed public dbt PRs: the modified-model fan-out finding was 1,717 of 7,820 findings (22%) and matched none of the 391 real reviewer-found defects. - `orchestrate.ts`: a modified model's downstream count no longer emits a finding; it already drives risk tiering and appears in the tier reasons. Deleted or renamed models with consumers still report the break. - `rule-catalog.ts` `join-or-condition`: skip an OR whose disjunct is an IS [NOT] NULL guard (open-ended interval joins, null-safe keys). A sample of 8 hits had 6 such guards. The body now names the duplicate-row risk the human reviewers raised. - `orchestrate.ts`: intermediate/staging models nested under a final-layer folder are not treated as final-layer for the removed-column suggestion. - tests for each Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e advisory AI lane The AI lane only received changed model SQL. In a labeled sample of 206 defects that only judgment can catch, 104 were in other files (82 macros, 15 yml, 7 singular tests), and a macro-only PR skipped the lane with "no changed model SQL to review". - `orchestrate.ts`: changed macros, singular tests, snapshots, schema/source yml, seeds and project config join the lane's input with their current content (seeds: diff only) - `ai-review.ts`: each file carries its dbt `kind`; `selectAiFiles` puts models first, then macros, tests, snapshots, yml, seeds, config under the 20-file cap; headers name the role and content is fenced by type; skip reason is now "no changed dbt files to review" - tests for the inputs, ordering, cap and prompt layout; docs note what the lane sees Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hape On the 377-PR compile-mode replay the check fired 41 times across 27 models; most were a declared key plus a partition column (`block_month`, `effective_date`). - silent when the merge key is a declared key plus only date/partition columns - `suggestion` when the merge key is wider than a declared key (duplicates only if an extra column changes for an existing row); `warning` when it is narrower or a different key - tests for each shape Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ublic dbt PR replay Ranked by volume on 413 compile-mode replays and checked against which findings matched any of 391 real reviewer-found defects: - engine check: injection rules (`union_injection`, `comment_injection`, `unbalanced_quote`) are skipped in models; `validate` errors from the engine's planner (planning/execution/not implemented) are skipped and parser syntax errors are suggestions; unresolved columns stay - undecidable equivalence is a suggestion to data-diff instead of a warning (426 hits, 1 partial match) - `union-select-star` needs a UNION branch that is `select *` - `order-by-in-cte` needs an ORDER BY at the CTE's top level that is not a top-N - `unqualified-column-multi-table` needs a changed bare-column line and a joining SELECT block with a bare column - `ref-double-quotes` removed: quoting style only - window without PARTITION BY reads the whole OVER clause, so a PARTITION BY on the next line counts - new join before an aggregate is a suggestion (1 partial match in 254) - tests for each shape Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Written from the defects in a 391-defect replay corpus that a deterministic rule can catch. On their own commits (no AI, no manifest), 50 of the 59 targeted defects now produce the matching finding; the join-key, missing-ref and table-`unique_key` rules need a manifest or installed packages and are measured on the full replay. New module `project-rules.ts` (files outside the SELECT): - packages.yml: a dependency this PR points at a git branch instead of a release/tag/sha - dbt_project.yml: version going down, a duplicate grantee, newly disabled resources - Jinja: unbalanced blocks (stray `endif`, missing `endmacro`, `macro x` without `()`), macro parameters moved position, a var() key renamed without the old key as fallback, code its own comment marks temporary, naive `datetime.utcnow()` - model config: `unique_key` on a table/view, BigQuery microbatch `batch_size` vs partition granularity, partition column missing from the merge key, an incremental filter that never constrains the partition column, a creation-time filter on a model whose rows change - `ref()` to a name no model/seed/snapshot carries (repo + installed packages; suggests a near match) - a new join whose key no uniqueness test on the joined model covers (manifest) - a not_null/relationships test on a column the model sets to a literal NULL Catalog: LEFT JOIN filtered in WHERE, case-folding one side of a join, null-safe comparison replaced by `=`, window ORDER BY repeating the partition key, duplicate CASE WHEN, QUALIFY / in-call IGNORE NULLS / MIN_BY-MAX_BY portability, async ClickHouse mutations, language-dependent DATENAME, hardcoded year filters. Also: portability advice is skipped in an adapter's own macro variant when that adapter supports the construct; the config dict-splat inliner handles nested dicts; `run.ts` lists resource names at head for the ref check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Join-key rule: only a LEFT JOIN onto a model with a declared uniqueness test whose key the join doesn't cover (an SCD2 dimension without `is_current`, a natural key instead of the declared one). Inner joins, models with no declared key, range / `OR` / Jinja-generated ON conditions are skipped; keys inside `cast(...)` / function wrappers are read. - Portability: a project whose own macros carry adapter variants (`postgres__x`, `snowflake__x`, ...) is judged against each of those adapters, so `min_by` in a package that supports Postgres is reported even when this checkout runs on DuckDB. `presto__` counts as Trino. - Missing `ref()`: snapshots defined only in YAML (dbt 1.9+) count as targets. - `var()` rename: skipped when the old key is still passed as a string literal (handed to a macro that reads it). - `grepMatchesAt` git helper; paths are repo-root relative. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the summary - `mergeRepeats`: the same rule several times in one file (one finding per column of a grain test, one per join onto the same kind of key) becomes one finding whose body lists each instance. Critical findings stay separate. - `ruleTitle` moves to `finding.ts` and strips any leading `<model or file>: `, so yml-based findings group across files too. - The summary groups a rule on two or more models into one bullet (was three). On the public dbt PR replay this takes findings per PR from 17.0 to 15.9 and summary bullets per PR from 9.1 to 8.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
full receipts (1 session)
orchestrator ·
|
| subagent | cost |
|---|---|
| You are judging a dbt PR-review benchmark, blind. Two versions of a code-review… | 497,477 tokens |
| You are labeling real dbt PR review comments for a benchmark. Label by READING… | 894,538 tokens |
| You are judging whether an automated dbt PR reviewer caught real defects that h… | 561,451 tokens |
| You are auditing feature coverage for a dbt PR-review tool. For each real revie… | 1,085,864 tokens |
| You are judging whether code-review findings are real problems, for a dbt PR-re… | 2,185,267 tokens |
| You are labeling real dbt PR review comments for a benchmark. Label by READING… | 1,064,110 tokens |
| Goal: an evidence-backed, cited landscape of (a) dbt PR review tools, (b) data-… | 2,134,743 tokens |
| You are auditing feature coverage for a dbt PR-review tool. For each real revie… | 1,332,813 tokens |
| You are judging whether an automated dbt PR reviewer caught real defects that h… | 315,327 tokens |
| You are judging whether an automated dbt PR reviewer caught real defects that h… | 493,007 tokens |
| Goal: determine what gaps altimate-code's dbt PR review has, by comparing what… | 5,303,299 tokens |
| You are judging a dbt PR-review benchmark, blind. Two versions of a code-review… | 414,573 tokens |
| Label dbt PR review comments by reading each one. Instructions: read /private/t… | 2,794,842 tokens |
| Label dbt PR review comments by reading each one. Instructions: read /private/t… | 3,244,430 tokens |
| You are judging whether an automated dbt PR reviewer caught real defects that h… | 326,356 tokens |
| Label escaped-defect candidates by READING them (no keyword bucketing). Directo… | 2,570,116 tokens |
| Repo: /Users/raghwendrasingh/code/altimateai/altimate-code (current branch feat… | 4,836,719 tokens |
| You are doing an ADVERSARIAL review of a set of Jira tickets that plan improvem… | 763,761 tokens |
| You are judging whether an automated dbt PR reviewer caught real defects that h… | 407,802 tokens |
| 27 more subagents | 62,717,734 unpriced tokens |
Generated by aireceipts
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
There was a problem hiding this comment.
35 issues found across 21 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/opencode/test/altimate/review-run-stale.test.ts">
<violation number="1" location="packages/opencode/test/altimate/review-run-stale.test.ts:68">
P3: These tests leave every temporary Git repository behind after completion, causing `/tmp` accumulation across repeated local and CI runs. Register cleanup with `afterEach`/`finally`, or use the test fixture with `await using` so each directory is disposed automatically.</violation>
</file>
<file name="docs/docs/usage/dbt-pr-review.md">
<violation number="1" location="docs/docs/usage/dbt-pr-review.md:586">
P2: The AI lane does not see every changed dbt file: deleted files are filtered out, and `analysis`/other classified dbt files are never added to its input. Say “eligible non-deleted files in these supported roles” or expand the implementation to cover all changed dbt kinds.</violation>
</file>
<file name="packages/opencode/src/altimate/review/git.ts">
<violation number="1" location="packages/opencode/src/altimate/review/git.ts:163">
P2: With no `head` (the documented working-tree mode), this helper always scans `HEAD`, so staged additions are absent from `projectResourceNames` and valid new `ref()` targets can be reported as missing. Make the undefined-ref path inspect the index/working tree rather than substituting `HEAD`, and keep `grepMatchesAt` consistent.</violation>
</file>
<file name="packages/opencode/src/altimate/review/runner.ts">
<violation number="1" location="packages/opencode/src/altimate/review/runner.ts:454">
P2: This requires `attached_node`, but older dbt manifests identify the test owner only through `depends_on.nodes`. Reuse that fallback or uniqueness tests disappear from both new key-based review lanes.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/runner.ts:457">
P2: `declaredUniqueKeys` misses standard dbt `unique` tests because `column_name` is nested under `test_metadata.kwargs`, so the new join and incremental-key checks silently skip models with ordinary uniqueness tests. Read `kwargs.column_name` like the existing manifest helper does.</violation>
</file>
<file name="packages/opencode/src/altimate/review/dbt-patterns.ts">
<violation number="1" location="packages/opencode/src/altimate/review/dbt-patterns.ts:318">
P2: Qualified columns written directly after a Jinja relation are no longer detected, so common `date({{ ref('events') }}.event_date)` filters evade the partition-pruning rule. Allow the identifier matcher to start after a qualification dot.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/dbt-patterns.ts:710">
P2: This prefix-based association can report an unchanged whole-table window when the PR adds another window with the same `row_number() over (` prefix. Associate the finding with the actual added window span rather than matching duplicate prefixes.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/dbt-patterns.ts:863">
P2: Add `mysql` to `ADAPTERS`; otherwise MySQL-specific macros produce portability noise and project adapter discovery misses `mysql__...` variants.</violation>
<violation number="4" location="packages/opencode/src/altimate/review/dbt-patterns.ts:896">
P2: A default on another `var('window', ...)` call suppresses the warning for `var('window')`, but dbt does not make that per-call default global. Check each invocation independently, or only suppress when the project defines the variable.</violation>
<violation number="5" location="packages/opencode/src/altimate/review/dbt-patterns.ts:908">
P2: Distinct missing variables in one macro are deduplicated into a single finding because their `ruleKey` omits the variable name. Include `name` in the key so every compile-breaking variable is reported.</violation>
<violation number="6" location="packages/opencode/src/altimate/review/dbt-patterns.ts:1535">
P2: `movedTo` assumes any matching test on a newly tested column represents a column rename, so it hides genuine removals when a guardrail is intentionally reassigned. Only suppress the finding when a rename is established; otherwise retain the old-column removal.</violation>
</file>
<file name="packages/opencode/src/altimate/review/orchestrate.ts">
<violation number="1" location="packages/opencode/src/altimate/review/orchestrate.ts:1166">
P2: This suppresses a `var_no_default` finding when the same variable has a default at a different call site. The default is not global, so `var('x')` can still fail even if `var('x', 1)` appears elsewhere; match the reported call itself instead of searching the whole SQL.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/orchestrate.ts:1854">
P2: This reports missing targets for commented-out refs because the new lane passes raw added lines to `missingRefFindings`. Ignore SQL/Jinja comments before matching, otherwise a harmless comment can produce a warning that dbt cannot parse the model.</violation>
</file>
<file name="packages/opencode/src/altimate/review/finding.ts">
<violation number="1" location="packages/opencode/src/altimate/review/finding.ts:211">
P3: `ruleTitle` does not normalize model or file labels containing spaces or non-ASCII characters, so equivalent findings fail to merge or group. Strip the exact model/file prefix instead of restricting the prefix to `\w`, dots, slashes, and hyphens.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/finding.ts:229">
P1: `mergeRepeats` collapses distinct same-rule lint issues even though `qualityLane` deliberately fingerprints each message separately. The merged finding keeps only the first line, body, and evidence, hiding the other defect from the envelope and inline review.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/finding.ts:229">
P3: This delimiter-joined grouping key is ambiguous when a file or title contains `|`, allowing unrelated findings to merge. Serialize the tuple with `JSON.stringify` or use a collision-safe key.</violation>
</file>
<file name="packages/opencode/src/altimate/review/run.ts">
<violation number="1" location="packages/opencode/src/altimate/review/run.ts:568">
P2: Working-tree reviews pass `head` as undefined, but this call makes resource discovery read `HEAD`; current snapshot names and newly added resources are invisible. Use a working-tree-aware listing and content path when `head` is absent.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/run.ts:572">
P2: `projectResourceNames` treats every matching file as a dbt resource, so `ref('foo')` is accepted when only `tests/foo.sql`, `macros/foo.sql`, or an arbitrary CSV exists. Restrict both scans to configured model, seed, and snapshot paths, including package layouts.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/run.ts:574">
P2: `projectResourceNames` searches only `*.yml`, although dbt file handling supports `.yaml`; references to YAML-only snapshots in that extension produce false `ref-target-missing` findings. Search both extensions.</violation>
</file>
<file name="packages/opencode/src/altimate/review/project-rules.ts">
<violation number="1" location="packages/opencode/src/altimate/review/project-rules.ts:165">
P2: Valid Jinja `with` blocks currently produce a `jinja-unbalanced` finding because `with` is missing from `BLOCK_OPEN`. Add the supported Jinja block tags before treating their `end*` tags as errors.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/project-rules.ts:329">
P2: Inherited `incremental_strategy` values are ignored even though `manifestConfig` is provided to fill omitted file config. Fall back to `manifestConfig.incremental_strategy` here so inherited merge and microbatch models are checked.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/project-rules.ts:331">
P2: Check that `cfg.unique_key` is a meaningful value before reporting it. Raw `None`, `null`, empty-string, and empty-list values are truthy strings here but do not configure a dbt unique key.</violation>
<violation number="4" location="packages/opencode/src/altimate/review/project-rules.ts:365">
P2: This rule reports a missing partition merge key without checking `incremental_predicates`, even though those predicates can prune the destination partitions and are offered as the fix. Suppress the finding when the configured predicates constrain the partition column.</violation>
<violation number="5" location="packages/opencode/src/altimate/review/project-rules.ts:559">
P2: `relationships` does not fail merely because every child value is NULL; dbt excludes NULL child keys from the referential-integrity check. Restrict this detector to `not_null` or model the test's NULL semantics separately.</violation>
</file>
<file name="packages/opencode/src/altimate/review/rule-catalog.ts">
<violation number="1" location="packages/opencode/src/altimate/review/rule-catalog.ts:475">
P2: This relevance pattern is too broad: adding an unrelated `AND`/`WHERE` causes the whole-file check to report an existing LEFT JOIN filter. Anchor the match to a changed predicate on the joined alias, or otherwise prove the changed line introduced the nullification.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/rule-catalog.ts:1171">
P3: This structural check can attribute a pre-existing final `SELECT *` to an unrelated added `SELECT *` in a CTE. Correlate the added star with the final top-level SELECT before reporting it, so unchanged debt does not create a new finding.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/rule-catalog.ts:2365">
P2: This whole-file UNION rule misses the common branch-edit case because its relevance regex requires `UNION` in the changed lines. Add an `added` condition for the changed `select *` branch and retain the whole-file check.</violation>
<violation number="4" location="packages/opencode/src/altimate/review/rule-catalog.ts:2477">
P2: The relevance regex misses multiline CASE additions: `case` and `when` are commonly written on separate lines, so the structural check never runs for a newly added unbalanced CASE. Anchor relevance to an added `CASE` or `WHEN` token instead of requiring `case when` on one line.</violation>
</file>
<file name="packages/opencode/src/altimate/review/incremental-semantics.ts">
<violation number="1" location="packages/opencode/src/altimate/review/incremental-semantics.ts:143">
P2: `CLAUSE_END` treats `left` and `right` as clause starts even when they are SQL functions. A change from `left(name, 3)` to `left(name, 4)` can be reduced to the same `where` token and bypass the history check.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/incremental-semantics.ts:152">
P2: `predicates` returns a `where`/`having` branch before applying `CLAUSE_END`, so trailing `GROUP BY` or `ORDER BY` edits are reported as filter changes. Restrict this early return to bare `and`/`or` fragments.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/incremental-semantics.ts:154">
P2: `predicates` finds clause keywords inside quoted SQL literals, so a projection containing `'where'` becomes a fake filter. Changing that projection can emit an `incremental-history-plan` warning even though row selection is unchanged.</violation>
<violation number="4" location="packages/opencode/src/altimate/review/incremental-semantics.ts:154">
P2: `predicates` documents `on` clauses but never scans them, and it also omits `qualify`; join-only or qualify-only incremental changes therefore bypass the history-plan check.</violation>
<violation number="5" location="packages/opencode/src/altimate/review/incremental-semantics.ts:171">
P2: `split` treats a nested Jinja `{% else %}` as the guard's branch separator. Changes in the nested else arm are dropped from `incrementalFilter`, so the history-plan check can miss a real incremental change.</violation>
</file>
<file name="packages/opencode/src/altimate/review/ai-review.ts">
<violation number="1" location="packages/opencode/src/altimate/review/ai-review.ts:39">
P3: The new `source_yml` label is unreachable because `classifyDbtFile` never produces that kind; source declaration YAML is labeled as `schema_yml` instead. Add source-YAML classification before the generic schema branch, or remove this unused role so the AI receives an accurate file label.</violation>
</file>
<file name="packages/opencode/src/altimate/review/format.ts">
<violation number="1" location="packages/opencode/src/altimate/review/format.ts:42">
P3: `GROUP_MIN` is documented as a model threshold, but the grouping check counts findings. Two occurrences for one model now collapse into one bullet and hide their separate details; require `names.length >= GROUP_MIN` or otherwise require distinct model/file names.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| out.push(f) | ||
| continue | ||
| } | ||
| const key = [f.file, f.category, f.severity, !!f.degraded, ruleTitle(f)].join("|") |
There was a problem hiding this comment.
P1: mergeRepeats collapses distinct same-rule lint issues even though qualityLane deliberately fingerprints each message separately. The merged finding keeps only the first line, body, and evidence, hiding the other defect from the envelope and inline review.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/finding.ts, line 229:
<comment>`mergeRepeats` collapses distinct same-rule lint issues even though `qualityLane` deliberately fingerprints each message separately. The merged finding keeps only the first line, body, and evidence, hiding the other defect from the envelope and inline review.</comment>
<file context>
@@ -203,3 +203,46 @@ export function dedupe(findings: Finding[]): Finding[] {
+ out.push(f)
+ continue
+ }
+ const key = [f.file, f.category, f.severity, !!f.degraded, ruleTitle(f)].join("|")
+ const g = groups.get(key)
+ if (g) g.push(f)
</file context>
| 3. **AI reviewer lane (layer 3).** When a model is configured, the LLM reviewer | ||
| runs **after** the deterministic lanes, grounded in their findings, and adds | ||
| advisory contextual comments (intent, naming, business logic, test coverage). | ||
| It sees every changed dbt file, not only models: macros, singular data tests, |
There was a problem hiding this comment.
P2: The AI lane does not see every changed dbt file: deleted files are filtered out, and analysis/other classified dbt files are never added to its input. Say “eligible non-deleted files in these supported roles” or expand the implementation to cover all changed dbt kinds.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/usage/dbt-pr-review.md, line 586:
<comment>The AI lane does not see every changed dbt file: deleted files are filtered out, and `analysis`/other classified dbt files are never added to its input. Say “eligible non-deleted files in these supported roles” or expand the implementation to cover all changed dbt kinds.</comment>
<file context>
@@ -579,6 +583,9 @@ across 12 dialects all plan; anything still unrecognized stays *undecidable*, ne
3. **AI reviewer lane (layer 3).** When a model is configured, the LLM reviewer
runs **after** the deterministic lanes, grounded in their findings, and adds
advisory contextual comments (intent, naming, business logic, test coverage).
+ It sees every changed dbt file, not only models: macros, singular data tests,
+ snapshots, schema/source yml, seeds (diff only) and project config, each
+ labeled with its role, with models first under the 20-file cap.
</file context>
| It sees every changed dbt file, not only models: macros, singular data tests, | |
| It sees eligible non-deleted dbt files, not only models: macros, singular data tests, |
| /** Every tracked file at `ref` (default HEAD), repo-root relative; undefined when git can't list it. */ | ||
| export async function listFilesAt(cwd: string, ref?: string): Promise<string[] | undefined> { | ||
| try { | ||
| const out = await git(["ls-tree", "-r", "--full-tree", "--name-only", ref || "HEAD"], cwd) |
There was a problem hiding this comment.
P2: With no head (the documented working-tree mode), this helper always scans HEAD, so staged additions are absent from projectResourceNames and valid new ref() targets can be reported as missing. Make the undefined-ref path inspect the index/working tree rather than substituting HEAD, and keep grepMatchesAt consistent.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/git.ts, line 163:
<comment>With no `head` (the documented working-tree mode), this helper always scans `HEAD`, so staged additions are absent from `projectResourceNames` and valid new `ref()` targets can be reported as missing. Make the undefined-ref path inspect the index/working tree rather than substituting `HEAD`, and keep `grepMatchesAt` consistent.</comment>
<file context>
@@ -154,6 +157,27 @@ export async function gitRepoRoot(cwd: string): Promise<string | undefined> {
+/** Every tracked file at `ref` (default HEAD), repo-root relative; undefined when git can't list it. */
+export async function listFilesAt(cwd: string, ref?: string): Promise<string[] | undefined> {
+ try {
+ const out = await git(["ls-tree", "-r", "--full-tree", "--name-only", ref || "HEAD"], cwd)
+ return out.split("\n").filter(Boolean)
+ } catch {
</file context>
| if (!uid) return [] | ||
| const keys: string[][] = [] | ||
| for (const [, n] of nodes) { | ||
| if (n?.resource_type !== "test" || n.attached_node !== uid) continue |
There was a problem hiding this comment.
P2: This requires attached_node, but older dbt manifests identify the test owner only through depends_on.nodes. Reuse that fallback or uniqueness tests disappear from both new key-based review lanes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/runner.ts, line 454:
<comment>This requires `attached_node`, but older dbt manifests identify the test owner only through `depends_on.nodes`. Reuse that fallback or uniqueness tests disappear from both new key-based review lanes.</comment>
<file context>
@@ -442,6 +442,29 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun
+ if (!uid) return []
+ const keys: string[][] = []
+ for (const [, n] of nodes) {
+ if (n?.resource_type !== "test" || n.attached_node !== uid) continue
+ const name = String(n.test_metadata?.name ?? "")
+ const kw = n.test_metadata?.kwargs ?? {}
</file context>
| if (n?.resource_type !== "test" || n.attached_node !== uid) continue | |
| if (n?.resource_type !== "test" || (n.attached_node || (n.depends_on?.nodes ?? []).find((id: string) => id.startsWith("model."))) !== uid) continue |
| * default when `x` isn't set in dbt_project.yml (in a package, every project that | ||
| * doesn't set it fails to compile). | ||
| */ | ||
| export const ADAPTERS = ["bigquery", "snowflake", "databricks", "spark", "redshift", "postgres", "trino", "athena", "duckdb", "exasol", "synapse", "sqlserver", "fabric", "oracle", "clickhouse", "presto"] |
There was a problem hiding this comment.
P2: Add mysql to ADAPTERS; otherwise MySQL-specific macros produce portability noise and project adapter discovery misses mysql__... variants.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 863:
<comment>Add `mysql` to `ADAPTERS`; otherwise MySQL-specific macros produce portability noise and project adapter discovery misses `mysql__...` variants.</comment>
<file context>
@@ -824,6 +855,64 @@ const MODEL_DETECTORS: Detector[] = [
+ * default when `x` isn't set in dbt_project.yml (in a package, every project that
+ * doesn't set it fails to compile).
+ */
+export const ADAPTERS = ["bigquery", "snowflake", "databricks", "spark", "redshift", "postgres", "trino", "athena", "duckdb", "exasol", "synapse", "sqlserver", "fabric", "oracle", "clickhouse", "presto"]
+
+/** The adapter a macro file is specific to, from its directory or its macros' `<adapter>__` prefix. */
</file context>
| export const ADAPTERS = ["bigquery", "snowflake", "databricks", "spark", "redshift", "postgres", "trino", "athena", "duckdb", "exasol", "synapse", "sqlserver", "fabric", "oracle", "clickhouse", "presto"] | |
| export const ADAPTERS = ["bigquery", "snowflake", "databricks", "spark", "redshift", "postgres", "trino", "athena", "mysql", "duckdb", "exasol", "synapse", "sqlserver", "fabric", "oracle", "clickhouse", "presto"] |
| test: "singular data test", | ||
| snapshot: "snapshot", | ||
| schema_yml: "schema yml (tests, contracts, descriptions)", | ||
| source_yml: "sources yml", |
There was a problem hiding this comment.
P3: The new source_yml label is unreachable because classifyDbtFile never produces that kind; source declaration YAML is labeled as schema_yml instead. Add source-YAML classification before the generic schema branch, or remove this unused role so the AI receives an accurate file label.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/ai-review.ts, line 39:
<comment>The new `source_yml` label is unreachable because `classifyDbtFile` never produces that kind; source declaration YAML is labeled as `schema_yml` instead. Add source-YAML classification before the generic schema branch, or remove this unused role so the AI receives an accurate file label.</comment>
<file context>
@@ -19,13 +19,45 @@ const MAX_FILES = 20
+ test: "singular data test",
+ snapshot: "snapshot",
+ schema_yml: "schema yml (tests, contracts, descriptions)",
+ source_yml: "sources yml",
+ seed: "seed CSV",
+ project_config: "project config",
</file context>
| export const SUMMARY_BUDGET_CHARS = 5_000 | ||
| /** The same rule on this many models or more collapses into one bullet. */ | ||
| export const GROUP_MIN = 3 | ||
| export const GROUP_MIN = 2 |
There was a problem hiding this comment.
P3: GROUP_MIN is documented as a model threshold, but the grouping check counts findings. Two occurrences for one model now collapse into one bullet and hide their separate details; require names.length >= GROUP_MIN or otherwise require distinct model/file names.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/format.ts, line 42:
<comment>`GROUP_MIN` is documented as a model threshold, but the grouping check counts findings. Two occurrences for one model now collapse into one bullet and hide their separate details; require `names.length >= GROUP_MIN` or otherwise require distinct model/file names.</comment>
<file context>
@@ -39,7 +39,7 @@ export function verdictHeadline(env: VerdictEnvelope): string {
export const SUMMARY_BUDGET_CHARS = 5_000
/** The same rule on this many models or more collapses into one bullet. */
-export const GROUP_MIN = 3
+export const GROUP_MIN = 2
export interface RenderOptions {
</file context>
| { add: "select *" }, | ||
| { add: "select id, name" }, | ||
| "The model's final `select *` reads straight from another relation, so every upstream column (including ones added later) lands in this model.", | ||
| { added: /\bselect\s+(distinct\s+)?\*/i, check: finalSelectStarFromRelation }, |
There was a problem hiding this comment.
P3: This structural check can attribute a pre-existing final SELECT * to an unrelated added SELECT * in a CTE. Correlate the added star with the final top-level SELECT before reporting it, so unchanged debt does not create a new finding.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/rule-catalog.ts, line 1171:
<comment>This structural check can attribute a pre-existing final `SELECT *` to an unrelated added `SELECT *` in a CTE. Correlate the added star with the final top-level SELECT before reporting it, so unchanged debt does not create a new finding.</comment>
<file context>
@@ -817,10 +1167,10 @@ const PERF: Rule[] = [
- { add: "select *" },
- { add: "select id, name" },
+ "The model's final `select *` reads straight from another relation, so every upstream column (including ones added later) lands in this model.",
+ { added: /\bselect\s+(distinct\s+)?\*/i, check: finalSelectStarFromRelation },
+ { add: "select * from {{ ref('orders') }}" },
+ { add: "with final as (select id from {{ ref('orders') }})\nselect * from final" },
</file context>
| out.push(f) | ||
| continue | ||
| } | ||
| const key = [f.file, f.category, f.severity, !!f.degraded, ruleTitle(f)].join("|") |
There was a problem hiding this comment.
P3: This delimiter-joined grouping key is ambiguous when a file or title contains |, allowing unrelated findings to merge. Serialize the tuple with JSON.stringify or use a collision-safe key.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/finding.ts, line 229:
<comment>This delimiter-joined grouping key is ambiguous when a file or title contains `|`, allowing unrelated findings to merge. Serialize the tuple with `JSON.stringify` or use a collision-safe key.</comment>
<file context>
@@ -203,3 +203,46 @@ export function dedupe(findings: Finding[]): Finding[] {
+ out.push(f)
+ continue
+ }
+ const key = [f.file, f.category, f.severity, !!f.degraded, ruleTitle(f)].join("|")
+ const g = groups.get(key)
+ if (g) g.push(f)
</file context>
| const key = [f.file, f.category, f.severity, !!f.degraded, ruleTitle(f)].join("|") | |
| const key = JSON.stringify([f.file, f.category, f.severity, !!f.degraded, ruleTitle(f)]) |
| * names blanked, so the same rule on different models/columns shares a key. */ | ||
| export function ruleTitle(f: Pick<Finding, "title">): string { | ||
| return f.title | ||
| .replace(/^[\w.\-/]+: /, "") |
There was a problem hiding this comment.
P3: ruleTitle does not normalize model or file labels containing spaces or non-ASCII characters, so equivalent findings fail to merge or group. Strip the exact model/file prefix instead of restricting the prefix to \w, dots, slashes, and hyphens.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/finding.ts, line 211:
<comment>`ruleTitle` does not normalize model or file labels containing spaces or non-ASCII characters, so equivalent findings fail to merge or group. Strip the exact model/file prefix instead of restricting the prefix to `\w`, dots, slashes, and hyphens.</comment>
<file context>
@@ -203,3 +203,46 @@ export function dedupe(findings: Finding[]): Finding[] {
+ * names blanked, so the same rule on different models/columns shares a key. */
+export function ruleTitle(f: Pick<Finding, "title">): string {
+ return f.title
+ .replace(/^[\w.\-/]+: /, "")
+ .replace(/ \(\+\d+ more\)$/, "")
+ .replace(/`[^`]*`/g, "`…`")
</file context>
| })) | ||
| // Macros, data tests, snapshots, yml and seeds carry defects no model diff shows. | ||
| // Seeds go as a diff only: the full CSV is data, not logic. | ||
| const supporting = reviewable.filter((f) => AI_SUPPORTING_KINDS.has(f.kind) && f.status !== "deleted") |
There was a problem hiding this comment.
WARNING: project_config includes profiles.yml, and its full content is sent to the model provider
classifyDbtFile maps profiles, packages and dbt_project all to kind: "project_config" (diff-filter.ts:122), so this filter picks up a committed profiles.yml and the getContent(...) just below passes its whole text to the advisory AI lane as sql. That lane runs by default in the lite/full tiers (TIER_LANES), and a committed profiles file routinely carries password / token / private_key / client_secret or a key-file path, so warehouse credentials egress to a third-party model whenever the PR touches it. Seeds are correctly diff-only here; profiles.yml should be excluded from the payload entirely (or credential keys redacted) while dbt_project.yml / packages.yml continue to be sent.
| { added: /\bcase\s+when\b[^;]*$/i, unless: /\bend\b|\{\{|\{%/i }, | ||
| "The model has more `CASE` than `END` keywords, which is a syntax error.", | ||
| // Whole-file balance, not per line: a CASE whose END sits on a later line is fine. | ||
| // Jinja branches can emit different CASE/END counts per branch, so they abstain. |
There was a problem hiding this comment.
WARNING: this abstention disables the CASE/END check on almost every real model
JINJA_BRANCH matches any {% if %}, {% for %}, {% elif %} or {% else %} in the whole file, so caseDepth(s) === -1 && !JINJA_BRANCH.test(s) can only fire on a model with no Jinja control flow at all. Every incremental model has {% if is_incremental() %}, so an added CASE with no END — the exact syntax error this rule targets — is never reported there. The previous per-line version did fire in that case because it only inspected the added line. Consider abstaining only when a Jinja tag actually wraps or emits CASE/END (or compare the balance per branch) rather than on the presence of any block anywhere in the file.
| const col = String(c?.name ?? "").toLowerCase() | ||
| if (!nulls.has(col)) continue | ||
| const failing = [...testNames(c.data_tests), ...testNames(c.tests)].filter((t) => /(^|\.)(not_null|relationships)$/.test(t)) | ||
| if (!failing.length || !new RegExp(`\\b${col}\\b|not_null|relationships`, "i").test(added)) continue |
There was a problem hiding this comment.
SUGGESTION: the not_null|relationships alternatives make this "added in this diff" gate vacuous
The regex is an OR tested against the joined added text, so any added line containing not_null or relationships — i.e. almost every schema.yml test edit — satisfies it for every literal-NULL column in the file. A PR that adds a not_null test to column B will therefore also report a pre-existing test on column A, which defeats anchoring this finding to the change that introduced it. Test only an escaped \b<col>\b, or resolve the added lines per model/column.
| (p) => p && typeof p.git === "string" && (p.revision === undefined || !PINNED_REVISION.test(String(p.revision))), | ||
| ) | ||
| // Only entries this PR touched: a long-standing branch pin is not this change's decision. | ||
| const touched = branchPinned.filter((p) => added.includes(p.git) || (p.revision !== undefined && added.includes(String(p.revision)))) |
There was a problem hiding this comment.
SUGGESTION: added.includes(...) matches a short revision anywhere in the diff
added is the whole concatenated added text, so added.includes(String(p.revision)) is true for a pre-existing revision: main / dev / v2 pin as soon as any unrelated added line happens to contain that substring — the long-standing pin is then reported as this PR's decision, which is what this filter exists to prevent. Check the individual addedLines(file.diff) for the git URL or an exact revision: value instead of substring-matching the joined blob.
| const oldVars = varNames(oldContent) | ||
| const newVars = varNames(newContent) | ||
| // A key still named as a string literal is usually still read (handed to a macro that reads it). | ||
| const gone = [...oldVars].filter((v) => !newVars.has(v) && !new RegExp(`['"]${v}['"]`).test(newContent)) |
There was a problem hiding this comment.
SUGGESTION: unescaped new RegExp built from file content can throw and sink the file's whole lane
v comes from var('…') in the old file, so a name with an unbalanced regex metacharacter (var('a(b'), var('a[b'), or a trailing backslash) makes new RegExp throw SyntaxError. jinjaFindings is chained with modelConfigFindings in a single task (orchestrate.ts:1766-1771), so the throw marks the model unverified and drops that whole lane's findings for the file. detectMacroPatterns and dbtConfigLane already escape content-derived names into an esc variable for exactly this hazard — reuse that idiom here (it also stops names containing | / ^ from silently over-matching and suppressing real findings).
Code Review SummaryStatus: 5 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (21 files)
Fix these issues in Kilo Cloud Reviewed by deepseek-v4.1-flash-1017 · Input: 359.8K · Output: 74K · Cached: 8M Review guidance: REVIEW.md from base branch |
Stacked on #1726. This PR covers precision fixes and new rules for the no-AI review lanes, measured by replaying public dbt PRs that had defects found by human reviewers.
How it was measured
Corpus: 391 defects that human reviewers caught on public OSS dbt PRs, replayed with
altimate reviewin compile mode on about 377–423 PRs.A judge labelled each defect caught, partial or missed against the findings.
Each defect was also tagged by what can reach it:
Results (no AI lane)
What changed
False positives and precision
var_no_default, nesting counts,case-when-no-end,select-star-in-cte-final, partition pruning, test removal on renamed columns, and incremental filter changes.union-select-star,order-by-in-cte,unqualified-column-multi-tableandjoin-or-conditionref-double-quotesis removedunique_keyvs declared uniqueness test check is graded by key shape.Rules that should have fired but didn't
:(top).config(**cfg)is inlined before the config lint.New rules (
project-rules.tsplus additions to the catalog)packages.yml: a dependency pinned to a branch.dbt_project.yml: the version going down, a duplicate grantee, or resources newly disabled.var()renamed without the old key as a fallback, andutcnow().unique_keyon a table or viewbatch_sizevs the partition granularityref()to a name that doesn't exist. YAML snapshots and installed packages count, and a near match is suggested.is_current.postgres__x,snowflake__x, ...).PR comment
AI lane input
Tests
tscis clean, apart from existingtui/errors.🤖 Generated with Claude Code
Summary by cubic
Fixes false positives and adds rules to the no-AI review lanes, measured by replaying 391 defects human reviewers caught on public dbt PRs. The rule set now catches 51 of the 53 defects it targets, and the modified-model fan-out finding (22% of all findings, matched none of the defects) is removed.
Precision and noise
var_no_default, nesting counts,case-when-no-end,select-star-in-cte-final, partition pruning, test removal on renamed columns, and incremental filter changes.ref-double-quotesis removed.config(**cfg)is inlined, changed macros are reviewed, and pathspecs are anchored with:(top)so monorepo hunk diffs aren't empty.New rules
packages.yml: dependency pinned to a branch.dbt_project.yml: version going down, duplicate grantee, newly disabled resources.var()without a fallback,utcnow().unique_keyon a table/view, BigQuery microbatchbatch_size, partition column missing from the merge key, an incremental filter that never filters on the partition column.ref()to a nonexistent name; a LEFT JOIN onto a model whose declared unique key the join doesn't cover; portability judged against every adapter the project's own macros target.Written for commit 502f835. Summary will update on new commits.