From 88e7f119696d37eca953fb324622b6c4a649d3db Mon Sep 17 00:00:00 2001 From: RaghwendraSingh Date: Fri, 13 Nov 2026 09:51:51 +0530 Subject: [PATCH 1/2] feat(review): consumers outside dbt, contract vs `SELECT`, engine-aware integer division, empty unit-test fixtures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - `adapter-capabilities.ts`: per-adapter behavior table with doc citations; engine-dependent rules read it instead of assuming one engine - integer division: correctness finding on Postgres/Redshift/Trino/Athena/SQL Server; portability advice (packages only) on Snowflake/BigQuery/Databricks/DuckDB; also matches `count(…) / count(…)` and underscore literals - a column removed from a model with `exposures` or an enforced contract names those consumers (`dbt.manifest` evidence, via a new `modelConsumers` runner method) - a new column missing from an enforced contract is reported unless `ciRunsDbtBuild: true`; a column declared in a changed yml counts - opt-in `consumerColumnLists` in `review.yml`: explicit column lists in loaders/exports must follow a model's added and removed columns - dbt unit tests whose `given` inputs are all `rows: []` are reported (cannot fail, or expect rows from nothing) - column sets come from column lineage; a `*` in the output skips the checks rather than guessing - column lineage retries without the schema when a manifest-only (partial) schema yields nothing: undocumented upstream tables made it return no columns, silently disabling column-breakage checks on projects without `catalog.json` - a manifest-only schema no longer reports `validate: Table '' not found` for undocumented models (an engine warning that could gate a PR); kept when a complete `catalog.json` schema is loaded Co-Authored-By: Claude Opus 5.5 --- docs/docs/usage/dbt-pr-review.md | 36 +++ .../altimate/review/adapter-capabilities.ts | 79 ++++++ .../opencode/src/altimate/review/config.ts | 15 ++ .../src/altimate/review/consumer-lists.ts | 88 ++++++ .../src/altimate/review/dbt-patterns.ts | 11 +- .../src/altimate/review/orchestrate.ts | 187 ++++++++++++- .../src/altimate/review/rule-catalog.ts | 55 +++- packages/opencode/src/altimate/review/run.ts | 12 + .../opencode/src/altimate/review/runner.ts | 47 +++- .../src/altimate/review/unit-tests.ts | 77 ++++++ .../altimate/review-contracts-dialect.test.ts | 253 ++++++++++++++++++ .../test/altimate/review-runner.test.ts | 83 ++++++ 12 files changed, 923 insertions(+), 20 deletions(-) create mode 100644 packages/opencode/src/altimate/review/adapter-capabilities.ts create mode 100644 packages/opencode/src/altimate/review/consumer-lists.ts create mode 100644 packages/opencode/src/altimate/review/unit-tests.ts create mode 100644 packages/opencode/test/altimate/review-contracts-dialect.test.ts diff --git a/docs/docs/usage/dbt-pr-review.md b/docs/docs/usage/dbt-pr-review.md index 78e5df7115..fb7a0a5379 100644 --- a/docs/docs/usage/dbt-pr-review.md +++ b/docs/docs/usage/dbt-pr-review.md @@ -272,6 +272,8 @@ riskTierPathTokens: # OFF by default — see "Risk-tier path tokens" b finops: [preset:finops] # promote any FinOps-named path to `full` tier pii: [ssn, email, phone] # or supply your own case-insensitive tokens allowForceTier: true # set false in gate CI: --force-tier is then rejected +ciRunsDbtBuild: false # true when CI runs `dbt build`: dbt then enforces contracts itself +consumerColumnLists: [] # OFF by default — see "Consumers outside dbt" below rubric: blockOn: [lineage_breakage, contract_violation, pii_exposure, semantic_change] warningPatternThreshold: 3 @@ -303,6 +305,31 @@ rubric: review is dismissed only by a later run that completed: when an engine call or a file failed, the earlier block stays. +### Consumers outside dbt + +A column removed or renamed can break readers the dbt DAG doesn't show. With a +manifest, the review reports: + +- **Exposures and contracts.** Removing a column from a model that has + `exposures` (dashboards, apps, ML) or an enforced contract names those + consumers and suggests a deprecation period or a new model version. + Downstream dbt models that select the column are reported separately, with + the models named. +- **Contract vs `SELECT`.** A new column missing from an enforced contract + fails `dbt build`. The review reports it unless `ciRunsDbtBuild: true`, in + which case dbt already catches it in CI. A column the PR declares in a changed + yml counts as declared even when the manifest predates the PR. +- **Explicit column lists (opt-in).** Projects whose loaders or exports keep + their own column list can have it checked: when a PR adds or removes a + model's columns, each matching list must follow. + +```yaml +consumerColumnLists: + - files: "loaders/**/*.py" # repo-relative glob + pattern: "MART_COLUMNS\\s*=\\s*\\[([^\\]]*)\\]" # first capture group = the list + model: fct_orders # omit: files that mention the model +``` + ### What the PR comment shows - **One summary comment**, updated in place on every push, capped at about @@ -448,6 +475,15 @@ The deterministic rules group into four families: cartesian (`JOIN` without a condition), `NOT IN (… NULL)`, sum-of-a-ratio, non-portable dialect functions, integer-multiplication overflow, PK-aware join fan-out, timezone-in-hash-key. +- **Engine-dependent rules read an adapter table.** Where behavior differs by + engine, a rule checks the project's adapter (from the manifest or `dialect`) + instead of assuming one. Example: `int / int` (and `count(…) / count(…)`) + truncates on Postgres, Redshift, Trino/Athena and SQL Server, so it is a + correctness finding there. On Snowflake, BigQuery, Databricks and DuckDB it is + only portability advice, shown for packages (`portability: true`). +- **dbt unit tests.** A unit test whose `given` inputs are all `rows: []` is + reported: it either can't fail, or expects rows the model has nothing to + build from. - **dbt config / Jinja (`DBT0xx`).** A core **minijinja** parser reads `{{ config(...) }}` and the model body: incremental model without an `is_incremental()` guard, `merge`/`delete+insert` without a `unique_key`, diff --git a/packages/opencode/src/altimate/review/adapter-capabilities.ts b/packages/opencode/src/altimate/review/adapter-capabilities.ts new file mode 100644 index 0000000000..cb75636dea --- /dev/null +++ b/packages/opencode/src/altimate/review/adapter-capabilities.ts @@ -0,0 +1,79 @@ +/** + * Per-adapter behavior that review rules depend on. A rule whose correctness + * differs by engine reads this table instead of hard-coding one engine, so it + * fires where the behavior is real and stays silent elsewhere. Every entry + * cites the engine's documentation. An adapter missing from the table gets the + * generic (engine-agnostic) form of each rule only. + * + * Keys are the review's dialect names (see run.ts `ADAPTER_DIALECT`). + */ + +export interface AdapterCapability { + /** `int / int` returns an integer, dropping the fraction (7 / 2 = 3). */ + integerDivisionTruncates: boolean + /** Documentation backing the entries above. */ + sources: string[] +} + +export const ADAPTER_CAPABILITIES: Record = { + postgres: { + integerDivisionTruncates: true, + sources: ["https://www.postgresql.org/docs/current/functions-math.html"], + }, + redshift: { + integerDivisionTruncates: true, + sources: ["https://docs.aws.amazon.com/redshift/latest/dg/r_numeric_computations201.html"], + }, + trino: { + integerDivisionTruncates: true, + sources: ["https://trino.io/docs/current/functions/math.html"], + }, + athena: { + // Athena SQL runs on Trino. + integerDivisionTruncates: true, + sources: ["https://trino.io/docs/current/functions/math.html"], + }, + tsql: { + integerDivisionTruncates: true, + sources: ["https://learn.microsoft.com/en-us/sql/t-sql/language-elements/divide-transact-sql"], + }, + fabric: { + integerDivisionTruncates: true, + sources: ["https://learn.microsoft.com/en-us/sql/t-sql/language-elements/divide-transact-sql"], + }, + snowflake: { + integerDivisionTruncates: false, + sources: ["https://docs.snowflake.com/en/sql-reference/operators-arithmetic"], + }, + bigquery: { + integerDivisionTruncates: false, + sources: ["https://cloud.google.com/bigquery/docs/reference/standard-sql/operators#arithmetic_operators"], + }, + databricks: { + integerDivisionTruncates: false, + sources: ["https://docs.databricks.com/en/sql/language-manual/functions/slashsign.html"], + }, + duckdb: { + integerDivisionTruncates: false, + sources: ["https://duckdb.org/docs/sql/functions/numeric"], + }, + mysql: { + integerDivisionTruncates: false, + sources: ["https://dev.mysql.com/doc/refman/8.0/en/arithmetic-functions.html"], + }, + oracle: { + integerDivisionTruncates: false, + sources: ["https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/Arithmetic-Operators.html"], + }, + clickhouse: { + integerDivisionTruncates: false, + sources: ["https://clickhouse.com/docs/en/sql-reference/functions/arithmetic-functions"], + }, +} + +export function capabilityFor(dialect: string | undefined): AdapterCapability | undefined { + return dialect ? ADAPTER_CAPABILITIES[dialect.toLowerCase()] : undefined +} + +/** Display names of the engines where `int / int` truncates. */ +export const TRUNCATING_DIVISION_ENGINES = "Postgres, Redshift, Trino/Athena and SQL Server" diff --git a/packages/opencode/src/altimate/review/config.ts b/packages/opencode/src/altimate/review/config.ts index d67cd7462c..cc098268bc 100644 --- a/packages/opencode/src/altimate/review/config.ts +++ b/packages/opencode/src/altimate/review/config.ts @@ -35,6 +35,21 @@ export const ReviewConfig = z.object({ * Functions missing on the project's own engine are always reported. */ portability: z.boolean().optional(), + /** + * CI runs `dbt build` (or `dbt run`) on the PR, so dbt itself fails a model + * whose SELECT doesn't match its enforced contract. The review then skips its + * own contract-vs-SELECT check instead of repeating CI. + */ + ciRunsDbtBuild: z.boolean().default(false), + /** + * Opt-in: explicit column lists outside dbt (loaders, exports) that must + * track a model's columns. `files` is a repo-relative glob, `pattern` a regex + * whose first capture group is the list, `model` the model it mirrors + * (omitted = files that mention the model by name). + */ + consumerColumnLists: z + .array(z.object({ files: z.string(), pattern: z.string(), model: z.string().optional() })) + .default([]), /** Allow `--force-tier` (a debug bypass of the risk classifier). Set false * in gate CI so a run cannot skip checks by forcing a lower tier. */ allowForceTier: z.boolean().default(true), diff --git a/packages/opencode/src/altimate/review/consumer-lists.ts b/packages/opencode/src/altimate/review/consumer-lists.ts new file mode 100644 index 0000000000..b7c6846ed2 --- /dev/null +++ b/packages/opencode/src/altimate/review/consumer-lists.ts @@ -0,0 +1,88 @@ +import { execFile } from "node:child_process" +import { promisify } from "node:util" + +/** + * Opt-in: explicit column lists outside dbt (loaders, exports, reverse-ETL + * configs) that must stay in step with a model's columns. A project names the + * files and a regex whose first capture group holds the list in + * `.altimate/review.yml`; the review then reports model columns a list is + * missing, and list entries the model no longer selects. Off unless configured, + * since no such convention is universal. + */ + +export interface ConsumerColumnListConfig { + /** Repository-relative glob of the files holding the lists. */ + files: string + /** Regex whose first capture group is the column list. */ + pattern: string + /** The model the lists mirror; omitted = files that mention the model by name. */ + model?: string +} + +export interface ConsumerColumnList { + path: string + model?: string + columns: string[] + /** File text, to match lists to models by name when `model` is omitted. */ + content: string +} + +const exec = promisify(execFile) +const IDENT = /^[A-Za-z_][A-Za-z0-9_$]*$/ + +/** Column names from every match of `pattern` in `content` (lowercased, de-duplicated). */ +export function extractColumnList(content: string, pattern: RegExp): string[] { + const flags = pattern.flags.includes("g") ? pattern.flags : pattern.flags + "g" + const re = new RegExp(pattern.source, flags) + const out = new Set() + for (const m of content.matchAll(re)) { + for (const token of (m[1] ?? "").split(/[,\s]+/)) { + const name = token.replace(/^["'`]+|["'`]+$/g, "") + if (IDENT.test(name)) out.add(name.toLowerCase()) + } + } + return [...out] +} + +/** True when a list belongs to `model`: named explicitly, or the file mentions the model. */ +export function listMirrorsModel(list: ConsumerColumnList, model: string): boolean { + if (list.model) return list.model.toLowerCase() === model.toLowerCase() + return new RegExp(`\\b${model.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}\\b`, "i").test(list.content) +} + +/** + * Read the configured lists at the head (`head` ref, or the working tree). + * An invalid pattern is reported on stderr and skipped, never fatal. + */ +export async function loadConsumerLists(opts: { + configs: ConsumerColumnListConfig[] + gitRoot: string + head?: string + read: (repoRelPath: string) => Promise +}): Promise { + const out: ConsumerColumnList[] = [] + for (const cfg of opts.configs) { + let re: RegExp + try { + re = new RegExp(cfg.pattern, "g") + } catch (e) { + process.stderr.write(`⚠️ consumerColumnLists: invalid pattern ${JSON.stringify(cfg.pattern)} (${String(e)}); skipped.\n`) + continue + } + const pathspec = `:(glob)${cfg.files}` + const args = opts.head ? ["ls-tree", "-r", "--name-only", opts.head, "--", pathspec] : ["ls-files", "--", pathspec] + let files: string[] = [] + try { + files = (await exec("git", args, { cwd: opts.gitRoot, maxBuffer: 16 * 1024 * 1024 })).stdout.split("\n").filter(Boolean) + } catch { + files = [] + } + for (const file of files) { + const content = await opts.read(file) + if (!content) continue + const columns = extractColumnList(content, re) + if (columns.length) out.push({ path: file, model: cfg.model, columns, content }) + } + } + return out +} diff --git a/packages/opencode/src/altimate/review/dbt-patterns.ts b/packages/opencode/src/altimate/review/dbt-patterns.ts index b9b99c1f8b..1ecc2765dd 100644 --- a/packages/opencode/src/altimate/review/dbt-patterns.ts +++ b/packages/opencode/src/altimate/review/dbt-patterns.ts @@ -4,7 +4,7 @@ import { Log } from "@/altimate/util/log" import { type Finding, type Severity, type ReviewCategory, makeFinding } from "./finding" import { type ChangedFile, fileKind } from "./diff-filter" import { type Rubric, clampSeverity, exclusionReason } from "./rubric" -import { evaluateCatalog } from "./rule-catalog" +import { evaluateCatalog, type CatalogContext } from "./rule-catalog" /** Local copy to avoid a cycle with orchestrate.ts. */ function modelNameFromPath(p: string): string { @@ -765,7 +765,12 @@ const MODEL_DETECTORS: Detector[] = [ ] /** Run the dbt anti-pattern detectors over a changed MODEL file. */ -export function detectModelPatterns(file: ChangedFile, newSql: string | undefined, rubric: Rubric): Finding[] { +export function detectModelPatterns( + file: ChangedFile, + newSql: string | undefined, + rubric: Rubric, + catalogCtx?: CatalogContext, +): Finding[] { const kind = fileKind(file) if (!isModelSql(kind) || file.status === "deleted" || !newSql) return [] const { added, removed } = splitDiff(file.diff) @@ -784,7 +789,7 @@ export function detectModelPatterns(file: ChangedFile, newSql: string | undefine if (f) out.push({ ...f, severity: clampSeverity(f.category, f.severity, f.confidence) }) } // Declarative rule catalog (data-driven checks) complements the programmatic detectors. - out.push(...evaluateCatalog(file, newSql, added, removed, rubric)) + out.push(...evaluateCatalog(file, newSql, added, removed, rubric, catalogCtx)) return out.filter((f) => !exclusionReason(f, rubric)) } diff --git a/packages/opencode/src/altimate/review/orchestrate.ts b/packages/opencode/src/altimate/review/orchestrate.ts index 05b7f5f306..5326f1cb34 100644 --- a/packages/opencode/src/altimate/review/orchestrate.ts +++ b/packages/opencode/src/altimate/review/orchestrate.ts @@ -14,6 +14,8 @@ import { type Rubric, exclusionReason, clampSeverity } from "./rubric" import { type ReviewConfig } from "./config" import { type ReviewMode, type ReviewScope, type VerdictEnvelope, buildEnvelope, signEnvelope } from "./verdict" import { detectModelPatterns, detectSchemaYmlPatterns, splitDiff } from "./dbt-patterns" +import { listMirrorsModel, type ConsumerColumnList } from "./consumer-lists" +import { detectUnitTestFixtures } from "./unit-tests" import { type AiLaneStatus, type AiReviewInput, type AiReviewResult } from "./ai-review" /** @@ -132,6 +134,16 @@ export interface ReviewRunner { lexicalScan?(addedLines: string[]): Promise> /** Direct downstream models (name + file path) of a model, from the manifest DAG. */ downstreamModels?(model: string): Promise> + /** + * Consumers of a model that dbt refs don't show: exposures that depend on it, + * and whether it has an enforced contract (with the declared columns). + * Undefined when the model is not in the manifest. + */ + modelConsumers?( + model: string, + ): Promise< + { exposures: Array<{ name: string; type?: string }>; contractEnforced: boolean; contractColumns: string[] } | undefined + > /** Column-lineage edges for a query: source `"table"."col"` → target output column. */ columnLineage?(sql: string, dialect: string): Promise> /** Classify a set of column names as PII (Email/SSN/…) with confidence + masking. */ @@ -213,6 +225,10 @@ export interface OrchestrateInput { scope?: Omit /** Drop the regex catalog's cross-engine portability findings (single known engine). */ suppressPortability?: boolean + /** The dialect came from config or the manifest, not the default guess. */ + dialectKnown?: boolean + /** Opt-in explicit column lists outside dbt (see consumer-lists.ts). */ + consumerLists?: ConsumerColumnList[] /** Which review policy was applied (see config.ts loadReviewPolicy). */ policy?: VerdictEnvelope["policy"] } @@ -606,6 +622,156 @@ async function columnBreakageLane( ] } +/** A query's output columns from column lineage (lowercased); empty when unknown or `*`. */ +async function outputColumns(runner: ReviewRunner, sql: string, dialect: string): Promise> { + if (!runner.columnLineage) return new Set() + const cols = (await runner.columnLineage(sql, dialect)).map((e) => e.target.toLowerCase()).filter(Boolean) + // A star in the output means the list is incomplete: report nothing rather than guess. + if (cols.some((c) => c === "*" || c.endsWith(".*"))) return new Set() + return new Set(cols) +} + +/** Output columns added and removed by the change (both empty when either side is unknown). */ +async function columnDelta( + ctx: ModelContext, + runner: ReviewRunner, + dialect: string, +): Promise<{ added: string[]; removed: string[]; current: Set }> { + const none = { added: [], removed: [], current: new Set() } + const { file, engineNewSql, engineOldSql } = ctx + if (!engineNewSql || file.status === "deleted") return none + const current = await outputColumns(runner, engineNewSql, dialect) + if (!current.size) return none + if (!hasOldSide(file)) return { added: [...current], removed: [], current } + if (!engineOldSql) return none + const before = await outputColumns(runner, engineOldSql, dialect) + if (!before.size) return none + return { + added: [...current].filter((c) => !before.has(c)), + removed: [...before].filter((c) => !current.has(c)), + current, + } +} + +const ticks = (xs: string[]) => xs.map((x) => `\`${x}\``).join(", ") + +/** + * Consumers dbt refs don't show. A column removed or renamed on a model that + * feeds exposures (dashboards, apps, ML) or has an enforced contract breaks + * readers outside the DAG; the finding names them. And a new column missing + * from an enforced contract fails `dbt build` — reported unless the project + * says CI already runs the build (`ciRunsDbtBuild`), where dbt catches it. + */ +async function consumerContractLane( + ctx: ModelContext, + runner: ReviewRunner, + dialect: string, + opts: { ciRunsDbtBuild: boolean; changedYmlText: () => Promise }, +): Promise { + if (!runner.modelConsumers || !runner.columnLineage) return [] + const model = modelNameFromPath(ctx.file.path) + const consumers = await runner.modelConsumers(model) + if (!consumers || (!consumers.exposures.length && !consumers.contractEnforced)) return [] + const { added, removed } = await columnDelta(ctx, runner, dialect) + const out: Finding[] = [] + + if (removed.length) { + const who: string[] = [] + if (consumers.exposures.length) + who.push(`exposure(s) ${consumers.exposures.map((e) => `\`${e.name}\`${e.type ? ` (${e.type})` : ""}`).join(", ")}`) + if (consumers.contractEnforced) who.push("an enforced contract, so readers outside dbt rely on its columns") + out.push( + makeFinding({ + severity: "warning", + category: "lineage_breakage", + title: `${model}: removed column(s) on a model with declared consumers`, + body: + `This change removes ${ticks(removed)}. \`${model}\` has ${who.join(" and ")}. ` + + `Keep the column for a deprecation period, or publish a new model version and migrate the consumers.`, + file: ctx.file.path, + model, + confidence: "high", + evidence: { + tool: "dbt.manifest", + result: { removed, exposures: consumers.exposures.map((e) => e.name), contractEnforced: consumers.contractEnforced }, + }, + ruleKey: "lineage_breakage:consumer-column-removed", + }), + ) + } + + if (consumers.contractEnforced && !opts.ciRunsDbtBuild && consumers.contractColumns.length && added.length) { + const declared = new Set(consumers.contractColumns) + const yml = await opts.changedYmlText() + // A column the PR adds to a changed yml is declared even when the manifest predates the PR. + const inChangedYml = (c: string) => + yml.some((t) => new RegExp(`name:\\s*["']?${c.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}["']?\\s*$`, "im").test(t)) + const missing = added.filter((c) => !declared.has(c) && !inChangedYml(c)) + if (missing.length) + out.push( + makeFinding({ + severity: "warning", + category: "contract_violation", + title: `${model}: new column(s) missing from the enforced contract`, + body: + `${ticks(missing)} ${missing.length === 1 ? "is" : "are"} selected but not declared in \`${model}\`'s contract, ` + + `so \`dbt build\` fails the contract check. Declare ${missing.length === 1 ? "it" : "them"} (with \`data_type\`) in the model's yml.`, + file: ctx.file.path, + model, + confidence: "high", + evidence: { tool: "dbt.manifest", result: { missing } }, + ruleKey: "contract_violation:contract-missing-column", + }), + ) + } + return out +} + +/** + * Opt-in consumer column lists (`consumerColumnLists` in review.yml): when the + * PR changes a model's columns, every configured list that mirrors the model + * must gain the new columns and drop the removed ones. + */ +async function consumerListLane( + ctx: ModelContext, + runner: ReviewRunner, + dialect: string, + lists: ConsumerColumnList[], +): Promise { + if (!lists.length) return [] + const model = modelNameFromPath(ctx.file.path) + const mirrors = lists.filter((l) => listMirrorsModel(l, model)) + if (!mirrors.length || !hasOldSide(ctx.file)) return [] + const { added, removed } = await columnDelta(ctx, runner, dialect) + if (!added.length && !removed.length) return [] + const out: Finding[] = [] + for (const list of mirrors) { + const have = new Set(list.columns) + const missing = added.filter((c) => !have.has(c)) + const stale = removed.filter((c) => have.has(c)) + if (!missing.length && !stale.length) continue + const parts: string[] = [] + if (missing.length) parts.push(`lacks the new column(s) ${ticks(missing)}`) + if (stale.length) parts.push(`still lists the removed column(s) ${ticks(stale)}`) + out.push( + makeFinding({ + severity: "warning", + category: "lineage_breakage", + title: `${model}: column list in ${list.path} is out of step`, + body: + `\`${list.path}\` declares an explicit column list for \`${model}\` that ${parts.join(" and ")}. ` + + `Readers of that list will drop or fail on these columns; update it in this PR.`, + file: ctx.file.path, + model, + confidence: "high", + evidence: { tool: "consumer-column-list", result: { list: list.path, missing, stale } }, + ruleKey: `lineage_breakage:consumer-list:${list.path}`, + }), + ) + } + return out +} + /** * DATA-DIFF — the showcase: actually run base vs head against the warehouse * (core DataParity) and report row/value deltas a linter can never see. Needs a @@ -1262,6 +1428,14 @@ export async function runReview(input: OrchestrateInput): Promise | undefined + const changedYmlText = () => + (ymlText ??= Promise.all( + reviewable + .filter((f) => f.kind === "schema_yml" && f.status !== "deleted") + .map((f) => (getContent?.(f.path, "new") ?? Promise.resolve(undefined)).then((t) => t ?? "").catch(() => "")), + )) const all: Finding[][] = [] // Files where the diff-scoped PII classifier completed (looked at the change), @@ -1297,6 +1471,12 @@ export async function runReview(input: OrchestrateInput): Promise + portabilityBody?: string example: RuleExample counter?: RuleExample } @@ -107,6 +116,22 @@ export function ruleMatches(rule: Rule, input: RuleInput): boolean { return relevant } +/** The project's engine, for engine-dependent rules. Omitted = unknown. */ +export interface CatalogContext { + dialect: string + /** False when the dialect is a default guess, not read from config or the manifest. */ + dialectKnown: boolean +} + +/** The body an engine-dependent rule reports for this project, or null to stay silent. */ +function gatedBody(rule: Rule, ctx: CatalogContext | undefined): string | null { + if (!rule.dialectGate || !ctx?.dialectKnown) return rule.body + const cap = capabilityFor(ctx.dialect) + if (!cap) return rule.body + if (cap[rule.dialectGate]) return rule.body + return rule.portabilityBody?.replace("{dialect}", ctx.dialect) ?? null +} + /** Evaluate the catalog against a changed file. */ export function evaluateCatalog( file: ChangedFile, @@ -114,23 +139,29 @@ export function evaluateCatalog( added: string[], removed: string[], rubric: Rubric, + ctx?: CatalogContext, ): Finding[] { const input: RuleInput = { file, newSql, added, removed } const model = modelName(file.path) const matched: { rule: Rule; finding: Finding }[] = [] for (const rule of CATALOG) { if (!ruleMatches(rule, input)) continue + const body = gatedBody(rule, ctx) + if (body === null) continue const f = makeFinding({ severity: clampSeverity(rule.category, rule.severity, "high"), category: rule.category, title: `${model}: ${rule.title}`, - body: rule.body, + body, file: file.path, model, confidence: "high", evidence: { tool: "rule-catalog", - result: rule.portability ? { rule: rule.id, portability: true, worksOn: rule.portability.worksOn } : { rule: rule.id }, + result: + rule.portability || body !== rule.body + ? { rule: rule.id, portability: true, worksOn: rule.portability?.worksOn } + : { rule: rule.id }, }, ruleKey: `${rule.category}:${rule.id}`, }) @@ -176,7 +207,10 @@ const r = ( severity: Severity, title: string, body: string, - match: Pick, + match: Pick< + Rule, + "added" | "removed" | "newSql" | "absentNewSql" | "unless" | "kind" | "pathGlob" | "portability" | "dialectGate" | "portabilityBody" + >, example: RuleExample, counter?: RuleExample, ): Rule => ({ id, category, severity, title, body, ...match, example, counter }) @@ -1824,10 +1858,17 @@ const MISC: Rule[] = [ "sql_correctness", "suggestion", "integer division truncates", - "`5 / 2` does integer division (=2) on many warehouses; cast a side to numeric for a real ratio.", - { added: /(? getContent(p, "new"), + }) + : [] + return runReview({ changedFiles, config, @@ -412,6 +422,8 @@ export async function reviewPullRequest(opts: ReviewPullRequestOptions): Promise staleManifest, layout, suppressPortability: config.portability === undefined ? dialectKnown : !config.portability, + dialectKnown, + consumerLists, policy: { source: policy.source, hash: policy.hash, changedInPr: policy.changedInPr.length ? policy.changedInPr : undefined }, scope: { baseRef: resolved?.ref ?? opts.base, diff --git a/packages/opencode/src/altimate/review/runner.ts b/packages/opencode/src/altimate/review/runner.ts index aa59b398ca..40e01408e4 100644 --- a/packages/opencode/src/altimate/review/runner.ts +++ b/packages/opencode/src/altimate/review/runner.ts @@ -1,6 +1,7 @@ import { Dispatcher } from "../native" import { EngineCoerce } from "../native/engine-coerce" import { parseManifest } from "../native/dbt/manifest" +import { loadRawManifest } from "../native/dbt/helpers" import type { CheckResult, EquivalenceResult, GradeResult, ImpactResult, ReviewRunner } from "./orchestrate" import { buildReviewSchemaContext, type SchemaContext } from "./schema-context" @@ -219,8 +220,13 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun // Only surface validation errors when a REAL schema was supplied — // in lint-only mode the throwaway `_altimate_lint_` schema makes // every real table "unknown" and would flood the review. + // A manifest-derived schema lists only models documented in yml, so + // "table not found" there means "undocumented", not "missing": drop + // it unless the schema came from a complete catalog.json. hasTables - ? asArray(data.validation?.errors).map((e: any) => ({ ...e, rule: "validate", severity: "error" })) + ? asArray(data.validation?.errors) + .filter((e: any) => opts.schemaContext || e?.category?.type !== "TableNotFound") + .map((e: any) => ({ ...e, rule: "validate", severity: "error" })) : [], ) .concat( @@ -414,6 +420,28 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun return out }, + async modelConsumers(model: string) { + try { + const raw = loadRawManifest(opts.manifestPath) + if (!raw) return undefined + const nodes = Object.entries(raw.nodes ?? {}) + const hit = nodes.find(([id, n]) => id.startsWith("model.") && n?.name === model) + if (!hit) return undefined + const [uid, node] = hit + const exposures = Object.values(raw.exposures ?? {}) + .filter((e) => Array.isArray(e?.depends_on?.nodes) && e.depends_on.nodes.includes(uid)) + .map((e) => ({ name: String(e.name ?? ""), type: e.type ? String(e.type) : undefined })) + const contractEnforced = node.config?.contract?.enforced === true || node.contract?.enforced === true + const contractColumns = contractEnforced + ? Object.values(node.columns ?? {}).map((c) => String(c?.name ?? "").toLowerCase()).filter(Boolean) + : [] + return { exposures, contractEnforced, contractColumns } + } catch (err) { + fail("modelConsumers", err) + return undefined + } + }, + async isComplex(sql: string): Promise { try { const res = await Dispatcher.call("altimate_core.metadata", { sql }) @@ -481,18 +509,21 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun }, async columnLineage(sql: string, dialect?: string): Promise> { - try { - const res = await Dispatcher.call("altimate_core.column_lineage", { - sql, - schema_path: "", - schema_context: await resolveSchema(), - dialect, - }) + const lineage = async (schema_context: Record | undefined) => { + const res = await Dispatcher.call("altimate_core.column_lineage", { sql, schema_path: "", schema_context, dialect }) const data = (res.data ?? {}) as Record return asArray(data.column_lineage).map((e) => ({ source: String(e.source ?? ""), target: String(e.target ?? ""), })) + } + try { + const schema = await resolveSchema() + const edges = await lineage(schema) + // A partial schema (the manifest only types documented models) makes + // the engine drop columns read from tables it doesn't know, returning + // nothing. Output column names need only the SQL: retry without it. + return edges.length || !schema ? edges : await lineage(undefined) } catch (err) { fail("columnLineage", err) return [] diff --git a/packages/opencode/src/altimate/review/unit-tests.ts b/packages/opencode/src/altimate/review/unit-tests.ts new file mode 100644 index 0000000000..c139a5cd4b --- /dev/null +++ b/packages/opencode/src/altimate/review/unit-tests.ts @@ -0,0 +1,77 @@ +import YAML from "yaml" +import { makeFinding, type Finding } from "./finding" +import type { ChangedFile } from "./diff-filter" + +/** + * dbt unit tests whose fixtures cannot exercise the model. When every `given` + * input has `rows: []`: + * - `expect` with no rows: the test passes whatever the model does; + * - `expect` with rows: the model has nothing to transform, so the test can + * only pass if the model emits constant rows. + * One empty input is normal (an empty `this` for an `is_incremental` test), so + * the rule needs all of them empty. Only unit tests added or changed in this + * PR are checked; `format: csv`/`sql` fixtures are skipped. + */ + +interface UnitTest { + name?: string + model?: string + given?: Array<{ input?: string; rows?: unknown; format?: string }> + expect?: { rows?: unknown; format?: string } +} + +function unitTestsOf(doc: unknown): Map { + const out = new Map() + const list = (doc as { unit_tests?: unknown })?.unit_tests + if (!Array.isArray(list)) return out + for (const t of list) { + if (t && typeof t === "object" && typeof (t as UnitTest).name === "string") out.set((t as UnitTest).name!, t as UnitTest) + } + return out +} + +const isDict = (f?: string) => f === undefined || f === "dict" + +export function detectUnitTestFixtures(file: ChangedFile, oldContent?: string, newContent?: string): Finding[] { + if (file.status === "deleted" || !newContent) return [] + let newDoc: unknown + let oldDoc: unknown + try { + newDoc = YAML.parse(newContent) + oldDoc = oldContent ? YAML.parse(oldContent) : undefined + } catch { + return [] + } + const before = unitTestsOf(oldDoc) + const out: Finding[] = [] + for (const [name, test] of unitTestsOf(newDoc)) { + const prev = before.get(name) + if (prev && JSON.stringify(prev) === JSON.stringify(test)) continue + const given = Array.isArray(test.given) ? test.given : [] + const allInputsEmpty = given.length > 0 && given.every((g) => isDict(g.format) && Array.isArray(g.rows) && g.rows.length === 0) + const expectRows = test.expect?.rows + if (!allInputsEmpty || !isDict(test.expect?.format) || !Array.isArray(expectRows)) continue + const model = test.model ?? "" + const passesAnyway = expectRows.length === 0 + out.push( + makeFinding({ + severity: passesAnyway ? "suggestion" : "warning", + category: "test_coverage", + title: passesAnyway + ? `${model || name}: unit test \`${name}\` cannot fail` + : `${model || name}: unit test \`${name}\` expects rows from empty inputs`, + body: passesAnyway + ? "Every `given` input has `rows: []` and `expect` has no rows, so the test passes whatever the model does. " + + "Add input rows that exercise the logic under test." + : `Every \`given\` input has \`rows: []\` but \`expect\` has ${expectRows.length} row(s): the model has nothing to transform. ` + + "Add input rows that exercise the changed logic.", + file: file.path, + model: model || undefined, + confidence: "high", + evidence: { tool: "unit-test-fixtures", result: { rule: passesAnyway ? "unit_test_cannot_fail" : "unit_test_empty_inputs", test: name } }, + ruleKey: `test_coverage:unit-test-empty-inputs:${name}`, + }), + ) + } + return out +} diff --git a/packages/opencode/test/altimate/review-contracts-dialect.test.ts b/packages/opencode/test/altimate/review-contracts-dialect.test.ts new file mode 100644 index 0000000000..4d762deedd --- /dev/null +++ b/packages/opencode/test/altimate/review-contracts-dialect.test.ts @@ -0,0 +1,253 @@ +import { describe, test, expect } from "bun:test" +import { promises as fs } from "node:fs" +import path from "node:path" +import { tmpdir } from "node:os" +import { execFile } from "node:child_process" +import { promisify } from "node:util" +import { evaluateCatalog } from "../../src/altimate/review/rule-catalog" +import { isCatalogPortability, runReview, type ReviewRunner } from "../../src/altimate/review/orchestrate" +import { DEFAULT_REVIEW_CONFIG } from "../../src/altimate/review/config" +import { DEFAULT_RUBRIC } from "../../src/altimate/review/rubric" +import { extractColumnList, listMirrorsModel, loadConsumerLists } from "../../src/altimate/review/consumer-lists" +import { detectUnitTestFixtures } from "../../src/altimate/review/unit-tests" +import { capabilityFor } from "../../src/altimate/review/adapter-capabilities" + +/** + * Checks that work on any dbt project from the manifest and the diff: + * engine-aware integer division, columns removed under exposures or an + * enforced contract, contract-vs-SELECT, unit-test fixtures that can't test + * anything, and opt-in consumer column lists outside dbt. + */ + +const exec = promisify(execFile) + +describe("integer division follows the adapter", () => { + const file = { path: "models/marts/m.sql", status: "modified" as const } + const run = (line: string, ctx?: { dialect: string; dialectKnown: boolean }) => + evaluateCatalog(file, line, [line], [], DEFAULT_RUBRIC, ctx).filter((f) => f.title.includes("integer division")) + + test("truncating engines get the correctness finding", () => { + for (const dialect of ["trino", "postgres", "redshift", "tsql"]) { + expect(capabilityFor(dialect)?.integerDivisionTruncates).toBe(true) + const [f] = run("select count(*) / count(distinct user_id) as r", { dialect, dialectKnown: true }) + expect(f.body).toContain("returns 3, not 3.5") + expect(isCatalogPortability(f)).toBe(false) + } + }) + + test("non-truncating engines get a portability finding (dropped for single-engine projects)", () => { + for (const dialect of ["snowflake", "bigquery", "databricks", "duckdb"]) { + const [f] = run("select 7 / 2 as r", { dialect, dialectKnown: true }) + expect(f.body).toContain(`returns a decimal on ${dialect}`) + expect(isCatalogPortability(f)).toBe(true) + } + }) + + test("unknown engine keeps the generic finding", () => { + expect(run("select 7 / 2 as r", { dialect: "snowflake", dialectKnown: false })[0].body).toContain("returns 3") + expect(run("select 7 / 2 as r")[0].body).toContain("returns 3") + }) + + test("shapes: count ratio and underscore literals match; casts and decimals don't", () => { + expect(run("select 1_000 / 3 as r")).toHaveLength(1) + expect(run("select count(*) / nullif(count(x), 0) as r")).toHaveLength(1) + expect(run("select count(*) * 1.0 / count(x) as r")).toHaveLength(0) + expect(run("select cast(count(*) as double) / count(x) as r")).toHaveLength(0) + expect(run("select 7.0 / 2 as r")).toHaveLength(0) + expect(run("select col1 / 2 as r")).toHaveLength(0) + }) + + test("end to end: a Snowflake project drops it, a multi-adapter package keeps it", async () => { + const base = { + changedFiles: [{ path: "models/m.sql", status: "added" as const, diff: "+select 7 / 2 as r\n" }], + config: { ...DEFAULT_REVIEW_CONFIG, reviewers: ["dbt_patterns"], dialect: "snowflake" }, + rubric: DEFAULT_RUBRIC, + mode: "comment" as const, + runner: fakeRunner(), + getContent: async () => "select 7 / 2 as r", + dialectKnown: true, + } + const has = (fs: { title: string }[]) => fs.some((f) => f.title.includes("integer division")) + expect(has((await runReview({ ...base, suppressPortability: true })).findings)).toBe(false) + expect(has((await runReview({ ...base, suppressPortability: false })).findings)).toBe(true) + const trino = { ...base, config: { ...base.config, dialect: "trino" } } + expect(has((await runReview({ ...trino, suppressPortability: true })).findings)).toBe(true) + }) +}) + +/** Output columns are read from a simple `select a, b from …`. */ +function fakeRunner(consumers?: Awaited>>): ReviewRunner { + return { + async impact() { + return { hasManifest: true, severity: "SAFE", directCount: 0, transitiveCount: 0, testCount: 0 } + }, + async grade() { + return { grade: "B" } + }, + async check() { + return { issues: [], ran: true } + }, + async equivalence() { + return { decided: true, equivalent: false } + }, + async detectPii() { + return { columns: [] } + }, + async columnLineage(sql: string) { + const m = /select\s+([\s\S]*?)\s+from/i.exec(sql) + if (!m) return [] + return m[1].split(",").map((c) => ({ source: `"src"."${c.trim()}"`, target: c.trim().split(/\s+as\s+/i).pop()! })) + }, + async modelConsumers() { + return consumers + }, + } as ReviewRunner +} + +const OLD = "select id, email, amount from src" +const NEW = "select id, amount, region from src" +const changed = [{ path: "models/marts/fct.sql", status: "modified" as const, diff: "-select id, email, amount from src\n+select id, amount, region from src\n" }] + +function review(consumers: Parameters[0], extra: Partial[0]> = {}) { + return runReview({ + changedFiles: changed, + config: { ...DEFAULT_REVIEW_CONFIG, reviewers: ["lineage_breakage", "contract_violation"] }, + rubric: DEFAULT_RUBRIC, + mode: "comment", + runner: fakeRunner(consumers), + getContent: async (_f, side) => (side === "old" ? OLD : NEW), + getCompiled: async (_f, side) => (side === "old" ? OLD : NEW), + ...extra, + }) +} + +describe("consumers outside the DAG", () => { + test("a removed column on a model with exposures names them", async () => { + const env = await review({ exposures: [{ name: "revenue_dashboard", type: "dashboard" }], contractEnforced: false, contractColumns: [] }) + const f = env.findings.find((x) => x.title.includes("declared consumers")) + expect(f?.body).toContain("`email`") + expect(f?.body).toContain("`revenue_dashboard` (dashboard)") + expect(f?.origin).toBe("engine") + }) + + test("a removed column on a contracted model is flagged; a model with neither is not", async () => { + const contracted = await review({ exposures: [], contractEnforced: true, contractColumns: ["id", "amount", "region"] }) + expect(contracted.findings.find((x) => x.title.includes("declared consumers"))?.body).toContain("enforced contract") + const plain = await review({ exposures: [], contractEnforced: false, contractColumns: [] }) + expect(plain.findings.some((x) => x.title.includes("declared consumers"))).toBe(false) + }) + + test("a new column missing from the enforced contract fails dbt build", async () => { + const env = await review({ exposures: [], contractEnforced: true, contractColumns: ["id", "email", "amount"] }) + expect(env.findings.find((x) => x.title.includes("missing from the enforced contract"))?.body).toContain("`region`") + }) + + test("skipped when CI runs dbt build, or when the PR declares the column in a changed yml", async () => { + const consumers = { exposures: [], contractEnforced: true, contractColumns: ["id", "email", "amount"] } + const ci = await review(consumers, { config: { ...DEFAULT_REVIEW_CONFIG, reviewers: ["contract_violation"], ciRunsDbtBuild: true } }) + expect(ci.findings.some((x) => x.title.includes("missing from the enforced contract"))).toBe(false) + const yml = await review(consumers, { + changedFiles: [...changed, { path: "models/marts/_fct.yml", status: "modified", diff: "+ - name: region\n" }], + getContent: async (f, side) => (f.endsWith(".yml") ? "models:\n - name: fct\n columns:\n - name: region\n" : side === "old" ? OLD : NEW), + }) + expect(yml.findings.some((x) => x.title.includes("missing from the enforced contract"))).toBe(false) + }) + + test("a star select is never guessed at", async () => { + const env = await review( + { exposures: [{ name: "d" }], contractEnforced: false, contractColumns: [] }, + { getCompiled: async (_f, side) => (side === "old" ? "select * from src" : NEW) }, + ) + expect(env.findings.some((x) => x.title.includes("declared consumers"))).toBe(false) + }) +}) + +describe("opt-in consumer column lists", () => { + test("lists are extracted from the capture group; models matched by name", () => { + const py = 'MART_COLUMNS = [\n "id",\n "amount",\n \'email\',\n]\nTABLE = "fct"\n' + expect(extractColumnList(py, /MART_COLUMNS\s*=\s*\[([^\]]*)\]/)).toEqual(["id", "amount", "email"]) + const list = { path: "loaders/fct.py", columns: ["id"], content: py } + expect(listMirrorsModel(list, "fct")).toBe(true) + expect(listMirrorsModel(list, "fct_orders")).toBe(false) + expect(listMirrorsModel({ ...list, model: "other" }, "fct")).toBe(false) + }) + + test("a list missing the new column and keeping the removed one is reported", async () => { + const env = await review(undefined, { + consumerLists: [{ path: "loaders/fct.py", columns: ["id", "email", "amount"], content: "fct" }], + }) + const f = env.findings.find((x) => x.title.includes("loaders/fct.py")) + expect(f?.body).toContain("lacks the new column(s) `region`") + expect(f?.body).toContain("still lists the removed column(s) `email`") + expect(f?.origin).toBe("catalog") + }) + + test("an up-to-date list, or no lists configured, reports nothing", async () => { + const ok = await review(undefined, { + consumerLists: [{ path: "loaders/fct.py", columns: ["id", "amount", "region"], content: "fct" }], + }) + expect(ok.findings.some((x) => x.title.includes("loaders/fct.py"))).toBe(false) + }) + + test("lists are read from the head with git; a bad pattern is skipped", async () => { + const root = await fs.realpath(await fs.mkdtemp(path.join(tmpdir(), "review-lists-"))) + const g = (...a: string[]) => exec("git", a, { cwd: root }) + await g("init", "-q", "-b", "main") + await fs.mkdir(path.join(root, "loaders"), { recursive: true }) + await fs.writeFile(path.join(root, "loaders", "fct.py"), 'COLS = ["id", "amount"]\n') + await fs.writeFile(path.join(root, "loaders", "other.txt"), 'COLS = ["x"]\n') + await g("add", "-A") + const lists = await loadConsumerLists({ + configs: [ + { files: "loaders/**/*.py", pattern: "COLS\\s*=\\s*\\[([^\\]]*)\\]", model: "fct" }, + { files: "loaders/**", pattern: "([", model: "fct" }, + ], + gitRoot: root, + read: (p) => fs.readFile(path.join(root, p), "utf8"), + }) + expect(lists.map((l) => [l.path, l.columns])).toEqual([["loaders/fct.py", ["id", "amount"]]]) + }) +}) + +describe("unit tests that cannot exercise the model", () => { + const file = { path: "models/marts/_fct.yml", status: "modified" as const } + const yml = (expectRows: string) => `unit_tests: + - name: t1 + model: fct + given: + - input: ref('a') + rows: [] + - input: this + rows: [] + expect: + rows: ${expectRows} +` + test("all inputs empty and nothing expected: cannot fail", () => { + const [f] = detectUnitTestFixtures(file, undefined, yml("[]")) + expect(f.title).toContain("cannot fail") + expect(f.severity).toBe("suggestion") + }) + + test("all inputs empty but rows expected", () => { + const [f] = detectUnitTestFixtures(file, undefined, yml("\n - {id: 1}")) + expect(f.title).toContain("expects rows from empty inputs") + }) + + test("an empty `this` next to a populated input is fine; unchanged tests are skipped", () => { + const ok = `unit_tests: + - name: t1 + model: fct + given: + - input: ref('a') + rows: + - {id: 1} + - input: this + rows: [] + expect: + rows: + - {id: 1} +` + expect(detectUnitTestFixtures(file, undefined, ok)).toHaveLength(0) + expect(detectUnitTestFixtures(file, yml("[]"), yml("[]"))).toHaveLength(0) + }) +}) diff --git a/packages/opencode/test/altimate/review-runner.test.ts b/packages/opencode/test/altimate/review-runner.test.ts index 538b2d7631..b7d20ecdbe 100644 --- a/packages/opencode/test/altimate/review-runner.test.ts +++ b/packages/opencode/test/altimate/review-runner.test.ts @@ -162,3 +162,86 @@ describe("runner honors engine `decidable` flag (core 0.5.1)", () => { await tmp[Symbol.asyncDispose]?.() }) }) + +describe("column lineage and consumers from the manifest", () => { + function manifest(dir: string) { + const p = path.join(dir, "manifest.json") + writeFileSync( + p, + JSON.stringify({ + metadata: { adapter_type: "duckdb" }, + nodes: { + "model.demo.fct": { + resource_type: "model", + name: "fct", + original_file_path: "models/fct.sql", + config: { materialized: "table", contract: { enforced: true } }, + depends_on: { nodes: [] }, + columns: { id: { name: "id", data_type: "integer" }, amount: { name: "amount", data_type: "double" } }, + }, + }, + sources: {}, + exposures: { + "exposure.demo.dash": { name: "dash", type: "dashboard", depends_on: { nodes: ["model.demo.fct"] } }, + "exposure.demo.other": { name: "other", type: "ml", depends_on: { nodes: ["model.demo.x"] } }, + }, + }), + ) + return p + } + + test("a partial schema that yields no lineage is retried without it", async () => { + await using tmp = await tmpdir() + const calls: unknown[] = [] + dispatcherSpy = spyOn(Dispatcher, "call").mockImplementation((async (_m: string, params: any) => { + calls.push(params.schema_context) + return { data: { column_lineage: params.schema_context ? [] : [{ source: '"stg"."id"', target: "id" }] } } + }) as any) + const runner = createDispatcherRunner({ manifestPath: manifest(tmp.path) }) + expect(await runner.columnLineage!("select id from stg", "duckdb")).toEqual([{ source: '"stg"."id"', target: "id" }]) + expect(calls).toHaveLength(2) + expect(calls[1]).toBeUndefined() + }) + + test("exposures and the enforced contract are read for a model", async () => { + await using tmp = await tmpdir() + const runner = createDispatcherRunner({ manifestPath: manifest(tmp.path) }) + expect(await runner.modelConsumers!("fct")).toEqual({ + exposures: [{ name: "dash", type: "dashboard" }], + contractEnforced: true, + contractColumns: ["id", "amount"], + }) + expect(await runner.modelConsumers!("missing")).toBeUndefined() + }) +}) + +describe("validation against a partial schema", () => { + const notFound = { data: { validation: { errors: [{ message: "Table 'stg' not found", category: { type: "TableNotFound", table: "stg" } }] } } } + + test("manifest-only schema: an undocumented table is not reported as missing", async () => { + await using tmp = await tmpdir() + const manifestPath = path.join(tmp.path, "manifest.json") + writeFileSync( + manifestPath, + JSON.stringify({ + nodes: { + "model.d.fct": { resource_type: "model", name: "fct", depends_on: { nodes: [] }, columns: { id: { name: "id", data_type: "int" } } }, + }, + sources: {}, + }), + ) + dispatcherSpy = spyOn(Dispatcher, "call").mockImplementation((async () => notFound) as any) + const runner = createDispatcherRunner({ manifestPath }) + expect((await runner.check("select id from stg", "duckdb")).issues?.some((i) => i.rule === "validate")).toBe(false) + }) + + test("catalog schema: a missing table is still reported", async () => { + await using tmp = await tmpdir() + dispatcherSpy = spyOn(Dispatcher, "call").mockImplementation((async () => notFound) as any) + const runner = createDispatcherRunner({ + manifestPath: path.join(tmp.path, "none.json"), + schemaContext: { tables: { fct: { columns: [{ name: "id", type: "int" }] } }, version: "1" }, + }) + expect((await runner.check("select id from stg", "duckdb")).issues?.some((i) => i.rule === "validate")).toBe(true) + }) +}) From 1522b5a2f170e3eff3e478ab08857149d0dc04e3 Mon Sep 17 00:00:00 2001 From: RaghwendraSingh Date: Fri, 13 Nov 2026 11:12:24 +0530 Subject: [PATCH 2/2] fix(review): contract check opt-in and non-blocking, no division matches inside strings, generic docs From an adversarial generality review: - the contract-vs-`SELECT` check restates what `dbt build` enforces, so it is now opt-in (`checkContractColumns`, replaces `ciRunsDbtBuild`) and comments only (catalog origin) - integer division ignores text inside string literals (`'2024/01/15'` is not a division) - the empty-inputs unit-test finding no longer claims the test "cannot fail" (models emitting constant rows still produce output) - an adapter the review doesn't map (a community adapter) no longer counts as a known dialect, so generic advice isn't suppressed on a guess - docs: the column-list example is a generic export script, not one project's loader convention Co-Authored-By: Claude Opus 5.5 --- docs/docs/usage/dbt-pr-review.md | 24 ++++++++++--------- .../opencode/src/altimate/review/config.ts | 8 +++---- .../src/altimate/review/orchestrate.ts | 10 ++++---- .../src/altimate/review/rule-catalog.ts | 22 +++++++++++++++-- packages/opencode/src/altimate/review/run.ts | 5 +++- .../src/altimate/review/unit-tests.ts | 8 +++---- .../altimate/review-contracts-dialect.test.ts | 21 ++++++++++------ 7 files changed, 64 insertions(+), 34 deletions(-) diff --git a/docs/docs/usage/dbt-pr-review.md b/docs/docs/usage/dbt-pr-review.md index fb7a0a5379..fe40ad1e94 100644 --- a/docs/docs/usage/dbt-pr-review.md +++ b/docs/docs/usage/dbt-pr-review.md @@ -272,7 +272,7 @@ riskTierPathTokens: # OFF by default — see "Risk-tier path tokens" b finops: [preset:finops] # promote any FinOps-named path to `full` tier pii: [ssn, email, phone] # or supply your own case-insensitive tokens allowForceTier: true # set false in gate CI: --force-tier is then rejected -ciRunsDbtBuild: false # true when CI runs `dbt build`: dbt then enforces contracts itself +checkContractColumns: false # opt-in: new columns missing from an enforced contract (CI that doesn't run `dbt build`) consumerColumnLists: [] # OFF by default — see "Consumers outside dbt" below rubric: blockOn: [lineage_breakage, contract_violation, pii_exposure, semantic_change] @@ -315,18 +315,20 @@ manifest, the review reports: consumers and suggests a deprecation period or a new model version. Downstream dbt models that select the column are reported separately, with the models named. -- **Contract vs `SELECT`.** A new column missing from an enforced contract - fails `dbt build`. The review reports it unless `ciRunsDbtBuild: true`, in - which case dbt already catches it in CI. A column the PR declares in a changed - yml counts as declared even when the manifest predates the PR. -- **Explicit column lists (opt-in).** Projects whose loaders or exports keep - their own column list can have it checked: when a PR adds or removes a - model's columns, each matching list must follow. +- **Contract vs `SELECT` (opt-in).** A new column missing from an enforced + contract fails `dbt build`, so this is off by default. Turn it on with + `checkContractColumns: true` when CI runs only `dbt parse` or `dbt compile`. + A column the PR declares in a changed yml counts as declared even when the + manifest predates the PR. It comments; it never blocks. +- **Explicit column lists (opt-in).** Some projects copy a model's columns + into code outside dbt: an export or reverse-ETL script, a feature list, an + API schema. Name those files and the list's pattern, and when a PR adds or + removes a model's columns, each matching list must follow. ```yaml consumerColumnLists: - - files: "loaders/**/*.py" # repo-relative glob - pattern: "MART_COLUMNS\\s*=\\s*\\[([^\\]]*)\\]" # first capture group = the list + - files: "exports/**/*.py" # repo-relative glob + pattern: "COLUMNS\\s*=\\s*\\[([^\\]]*)\\]" # first capture group = the list model: fct_orders # omit: files that mention the model ``` @@ -482,7 +484,7 @@ The deterministic rules group into four families: correctness finding there. On Snowflake, BigQuery, Databricks and DuckDB it is only portability advice, shown for packages (`portability: true`). - **dbt unit tests.** A unit test whose `given` inputs are all `rows: []` is - reported: it either can't fail, or expects rows the model has nothing to + reported: it either never tests the model's logic, or expects rows the model has nothing to build from. - **dbt config / Jinja (`DBT0xx`).** A core **minijinja** parser reads `{{ config(...) }}` and the model body: incremental model without an diff --git a/packages/opencode/src/altimate/review/config.ts b/packages/opencode/src/altimate/review/config.ts index cc098268bc..c9ff8d3d2f 100644 --- a/packages/opencode/src/altimate/review/config.ts +++ b/packages/opencode/src/altimate/review/config.ts @@ -36,11 +36,11 @@ export const ReviewConfig = z.object({ */ portability: z.boolean().optional(), /** - * CI runs `dbt build` (or `dbt run`) on the PR, so dbt itself fails a model - * whose SELECT doesn't match its enforced contract. The review then skips its - * own contract-vs-SELECT check instead of repeating CI. + * Opt-in: report a new column missing from an enforced contract. `dbt build` + * already fails on it, so this only helps CI that runs `dbt parse`/`compile` + * without building. */ - ciRunsDbtBuild: z.boolean().default(false), + checkContractColumns: z.boolean().default(false), /** * Opt-in: explicit column lists outside dbt (loaders, exports) that must * track a model's columns. `files` is a repo-relative glob, `pattern` a regex diff --git a/packages/opencode/src/altimate/review/orchestrate.ts b/packages/opencode/src/altimate/review/orchestrate.ts index 5326f1cb34..c492427aba 100644 --- a/packages/opencode/src/altimate/review/orchestrate.ts +++ b/packages/opencode/src/altimate/review/orchestrate.ts @@ -660,13 +660,13 @@ const ticks = (xs: string[]) => xs.map((x) => `\`${x}\``).join(", ") * feeds exposures (dashboards, apps, ML) or has an enforced contract breaks * readers outside the DAG; the finding names them. And a new column missing * from an enforced contract fails `dbt build` — reported unless the project - * says CI already runs the build (`ciRunsDbtBuild`), where dbt catches it. + * opts in (`checkContractColumns`); `dbt build` already enforces it. */ async function consumerContractLane( ctx: ModelContext, runner: ReviewRunner, dialect: string, - opts: { ciRunsDbtBuild: boolean; changedYmlText: () => Promise }, + opts: { checkContractColumns: boolean; changedYmlText: () => Promise }, ): Promise { if (!runner.modelConsumers || !runner.columnLineage) return [] const model = modelNameFromPath(ctx.file.path) @@ -700,7 +700,7 @@ async function consumerContractLane( ) } - if (consumers.contractEnforced && !opts.ciRunsDbtBuild && consumers.contractColumns.length && added.length) { + if (consumers.contractEnforced && opts.checkContractColumns && consumers.contractColumns.length && added.length) { const declared = new Set(consumers.contractColumns) const yml = await opts.changedYmlText() // A column the PR adds to a changed yml is declared even when the manifest predates the PR. @@ -719,7 +719,7 @@ async function consumerContractLane( file: ctx.file.path, model, confidence: "high", - evidence: { tool: "dbt.manifest", result: { missing } }, + evidence: { tool: "contract-check", result: { missing } }, ruleKey: "contract_violation:contract-missing-column", }), ) @@ -1473,7 +1473,7 @@ export async function runReview(input: OrchestrateInput): Promise portabilityBody?: string + /** Match `added` outside quoted strings only (e.g. `'2024/01/15'` is not a division). */ + ignoreStrings?: boolean example: RuleExample counter?: RuleExample } @@ -67,6 +69,11 @@ function modelName(p: string): string { return path.basename(p).replace(/\.(sql|py)$/i, "") } +/** Replace quoted string contents so text inside literals can't match. */ +function stripStrings(s: string): string { + return s.replace(/'(?:[^']|'')*'/g, "''").replace(/"(?:[^"\\]|\\.)*"/g, '""') +} + function stripComments(s: string): string { return s .replace(/\/\*[\s\S]*?\*\//g, " ") @@ -96,7 +103,7 @@ export function ruleMatches(rule: Rule, input: RuleInput): boolean { let relevant = false if (rule.added) { const hit = input.added.some((l) => { - const s = stripComments(l) + const s = rule.ignoreStrings ? stripStrings(stripComments(l)) : stripComments(l) return rule.added!.test(s) && !(rule.unless && rule.unless.test(s)) }) if (!hit) return false @@ -209,7 +216,17 @@ const r = ( body: string, match: Pick< Rule, - "added" | "removed" | "newSql" | "absentNewSql" | "unless" | "kind" | "pathGlob" | "portability" | "dialectGate" | "portabilityBody" + | "added" + | "removed" + | "newSql" + | "absentNewSql" + | "unless" + | "kind" + | "pathGlob" + | "portability" + | "dialectGate" + | "portabilityBody" + | "ignoreStrings" >, example: RuleExample, counter?: RuleExample, @@ -1863,6 +1880,7 @@ const MISC: Rule[] = [ { added: /(? { expect(run("select cast(count(*) as double) / count(x) as r")).toHaveLength(0) expect(run("select 7.0 / 2 as r")).toHaveLength(0) expect(run("select col1 / 2 as r")).toHaveLength(0) + expect(run("where dt >= '2024/01/15'")).toHaveLength(0) + expect(run("select 'a 1/2 b' as label, 3 / 4 as r")).toHaveLength(1) }) test("end to end: a Snowflake project drops it, a multi-adapter package keeps it", async () => { @@ -137,16 +139,21 @@ describe("consumers outside the DAG", () => { expect(plain.findings.some((x) => x.title.includes("declared consumers"))).toBe(false) }) - test("a new column missing from the enforced contract fails dbt build", async () => { - const env = await review({ exposures: [], contractEnforced: true, contractColumns: ["id", "email", "amount"] }) - expect(env.findings.find((x) => x.title.includes("missing from the enforced contract"))?.body).toContain("`region`") + const optIn = { config: { ...DEFAULT_REVIEW_CONFIG, reviewers: ["lineage_breakage", "contract_violation"], checkContractColumns: true } } + + test("opt-in: a new column missing from the enforced contract is reported, and never blocks", async () => { + const env = await review({ exposures: [], contractEnforced: true, contractColumns: ["id", "email", "amount"] }, optIn) + const f = env.findings.find((x) => x.title.includes("missing from the enforced contract")) + expect(f?.body).toContain("`region`") + expect(f?.origin).toBe("catalog") }) - test("skipped when CI runs dbt build, or when the PR declares the column in a changed yml", async () => { + test("off by default (dbt build enforces it), and skipped when the PR declares the column in a changed yml", async () => { const consumers = { exposures: [], contractEnforced: true, contractColumns: ["id", "email", "amount"] } - const ci = await review(consumers, { config: { ...DEFAULT_REVIEW_CONFIG, reviewers: ["contract_violation"], ciRunsDbtBuild: true } }) - expect(ci.findings.some((x) => x.title.includes("missing from the enforced contract"))).toBe(false) + const def = await review(consumers) + expect(def.findings.some((x) => x.title.includes("missing from the enforced contract"))).toBe(false) const yml = await review(consumers, { + ...optIn, changedFiles: [...changed, { path: "models/marts/_fct.yml", status: "modified", diff: "+ - name: region\n" }], getContent: async (f, side) => (f.endsWith(".yml") ? "models:\n - name: fct\n columns:\n - name: region\n" : side === "old" ? OLD : NEW), }) @@ -224,7 +231,7 @@ describe("unit tests that cannot exercise the model", () => { ` test("all inputs empty and nothing expected: cannot fail", () => { const [f] = detectUnitTestFixtures(file, undefined, yml("[]")) - expect(f.title).toContain("cannot fail") + expect(f.title).toContain("doesn't exercise the model") expect(f.severity).toBe("suggestion") })