Repository navigation
fix(review): an honest PR comment: real lint-only banner, AI lane status, PR intent, project_dir #1720
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: fix/review-scope-diff
Are you sure you want to change the base?
fix(review): an honest PR comment: real lint-only banner, AI lane status, PR intent, project_dir #1720
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -199,6 +199,34 @@ jobs: | |
| # model_api_key: ${{ secrets.ANTHROPIC_API_KEY }} | ||
| ``` | ||
|
|
||
| ### What the review needs (and what each input unlocks) | ||
|
|
||
| | Input | How to produce it | Unlocks | | ||
| |---|---|---| | ||
| | Full git history of the PR's target branch | `actions/checkout` with `fetch-depth: 0` | Correct base: the diff starts at the merge-base with the PR's target branch. A target branch that is not fetched is an error, never a silent fallback to `main`. | | ||
| | `target/manifest.json` | `dbt compile` (or `dbt parse` for lint + lineage only) | Blast radius, lineage breakage, tiering. Without it the run is lint-only. | | ||
| | `target/compiled/…` | `dbt compile` | Engine checks on the head (lint, grade, PII, grain). | | ||
| | `target-base/compiled/…` | Compile the base ref into `target-base/` (below) | Equivalence proofs and column-breakage for modified models. | | ||
| | `target/catalog.json` | `dbt docs generate` | Complete warehouse columns for column-lineage breakage. | | ||
|
|
||
| Compiling the base (optional, for equivalence and column-breakage): | ||
|
|
||
| ```yaml | ||
| - name: Compile the base into target-base/ | ||
| run: | | ||
| git worktree add ../base "origin/${{ github.base_ref }}" | ||
| (cd ../base && dbt deps && dbt compile --target-path "$GITHUB_WORKSPACE/target-base") | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. SUGGESTION: This snippet does not work in the monorepo layout the paragraph below documents. It writes |
||
| ``` | ||
|
|
||
| **Monorepos:** set `project_dir` to the folder holding `dbt_project.yml`. Only | ||
| changes inside it are reviewed, `manifest_path` is resolved relative to it, and | ||
| changes elsewhere are counted in the comment. | ||
|
|
||
| **What the comment tells you it could not check:** findings the engine could not | ||
| decide, models without compiled SQL, files outside the project, and whether the | ||
| advisory AI lane ran, was skipped, timed out or failed (with the reason). The | ||
| same facts are in the envelope (`scope`, `ai`) and the `review_run` telemetry. | ||
|
|
||
| ### Model & credentials for the advisory lane | ||
|
|
||
| The deterministic engine (lineage, equivalence, PII, grade, lint — the only layer | ||
|
|
@@ -356,8 +384,9 @@ reviewer does **not** re-implement Jinja — it consumes dbt's own compiled outp | |
| from `target/compiled/<project>/…` (HEAD) and `target-base/compiled/…` (BASE), | ||
| produced by `dbt compile`. To enable full equivalence verdicts, compile both | ||
| the base and head refs in CI (the base into `target-base/`, the Recce | ||
| convention). Without compiled SQL the engine lanes fall back to raw and stay | ||
| *undecidable* (never fabricated) — the `dbt-patterns` lane still runs. | ||
| convention). A model without compiled SQL is never sent to the engine as raw | ||
| Jinja: it is listed as **not verified** (with the reason) in the comment and in | ||
| the envelope's `scope.unverified`, and the `dbt-patterns` lane still runs. | ||
|
|
||
| ## What it checks — deterministic rule catalog | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -9,8 +9,12 @@ inputs: | |||||||||||||||||||
| description: "comment (never blocks) | gate (blocks the merge on REQUEST_CHANGES)" | ||||||||||||||||||||
| required: false | ||||||||||||||||||||
| default: "comment" | ||||||||||||||||||||
| project_dir: | ||||||||||||||||||||
| description: "Directory of the dbt project (the one holding dbt_project.yml), relative to the repository root. Only changes inside it are reviewed — set it in a monorepo. `manifest_path` is resolved relative to it." | ||||||||||||||||||||
| required: false | ||||||||||||||||||||
| default: "." | ||||||||||||||||||||
| manifest_path: | ||||||||||||||||||||
| description: "Path to the compiled dbt manifest.json. Run `dbt compile` before this step for the full (non-degraded) verdict." | ||||||||||||||||||||
| description: "Path to the compiled dbt manifest.json, relative to `project_dir`. Run `dbt compile` before this step for the full (non-degraded) verdict; `dbt docs generate` (catalog.json) and a base compile into `target-base/` enable column-lineage breakage and equivalence proofs." | ||||||||||||||||||||
| required: false | ||||||||||||||||||||
| default: "target/manifest.json" | ||||||||||||||||||||
| severity_threshold: | ||||||||||||||||||||
|
|
@@ -170,6 +174,7 @@ runs: | |||||||||||||||||||
| # from `${{ }}` would be a shell-injection vector. Build an arg array instead. | ||||||||||||||||||||
| env: | ||||||||||||||||||||
| IN_MODE: ${{ inputs.mode }} | ||||||||||||||||||||
| IN_PROJECT_DIR: ${{ inputs.project_dir }} | ||||||||||||||||||||
| IN_MANIFEST: ${{ inputs.manifest_path }} | ||||||||||||||||||||
| IN_SEVERITY: ${{ inputs.severity_threshold }} | ||||||||||||||||||||
| IN_BASE: ${{ inputs.base }} | ||||||||||||||||||||
|
|
@@ -181,6 +186,7 @@ runs: | |||||||||||||||||||
| ALTIMATE_REVIEW_SIGNING_KEY: ${{ inputs.signing_key }} | ||||||||||||||||||||
| run: | | ||||||||||||||||||||
| args=(--mode "$IN_MODE" --manifest "$IN_MANIFEST" --severity "$IN_SEVERITY") | ||||||||||||||||||||
| [[ -n "$IN_PROJECT_DIR" && "$IN_PROJECT_DIR" != "." ]] && args+=(--cwd "$IN_PROJECT_DIR") | ||||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: A misspelled Prompt for AI agents
Suggested change
|
||||||||||||||||||||
| [[ -n "$IN_BASE" ]] && args+=(--base "$IN_BASE") | ||||||||||||||||||||
| [[ -n "$IN_HEAD" ]] && args+=(--head "$IN_HEAD") | ||||||||||||||||||||
| [[ "$IN_POST" == "true" ]] && args+=(--post) | ||||||||||||||||||||
|
|
||||||||||||||||||||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -11,7 +11,8 @@ import { readStreamText } from "../llm/stream-text" | |||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
| const log = Log.create({ service: "ai-review" }) | ||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
| const AI_TIMEOUT_MS = 60_000 | ||||||||||||||||||||||||||||||||||||
| const AI_TIMEOUT_BASE_MS = 60_000 | ||||||||||||||||||||||||||||||||||||
| const AI_TIMEOUT_MAX_MS = 300_000 | ||||||||||||||||||||||||||||||||||||
| const MAX_DIFF_CHARS = 6_000 // per file, keep the prompt bounded | ||||||||||||||||||||||||||||||||||||
| const MAX_FILES = 20 | ||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
|
|
@@ -31,6 +32,39 @@ export interface AiReviewInput { | |||||||||||||||||||||||||||||||||||
| grounding: Finding[] | ||||||||||||||||||||||||||||||||||||
| prTitle?: string | ||||||||||||||||||||||||||||||||||||
| prBody?: string | ||||||||||||||||||||||||||||||||||||
| /** Explicit `provider/model`; omit to use the configured default model. */ | ||||||||||||||||||||||||||||||||||||
| model?: string | ||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
| /** What happened in the advisory lane, reported on the envelope and in the comment. */ | ||||||||||||||||||||||||||||||||||||
| export type AiLaneState = "ok" | "skipped" | "timeout" | "error" | ||||||||||||||||||||||||||||||||||||
| export interface AiLaneStatus { | ||||||||||||||||||||||||||||||||||||
| state: AiLaneState | ||||||||||||||||||||||||||||||||||||
| /** Why the lane was skipped or failed (never contains credentials). */ | ||||||||||||||||||||||||||||||||||||
| reason?: string | ||||||||||||||||||||||||||||||||||||
| /** `provider/model` the lane ran with. */ | ||||||||||||||||||||||||||||||||||||
| model?: string | ||||||||||||||||||||||||||||||||||||
| durationMs?: number | ||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||
| export interface AiReviewResult { | ||||||||||||||||||||||||||||||||||||
| findings: Finding[] | ||||||||||||||||||||||||||||||||||||
| status: AiLaneStatus | ||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
| /** Timeout scaled to the prompt: 60 s plus 1 s per 2,000 characters, capped at 5 minutes. */ | ||||||||||||||||||||||||||||||||||||
| export function aiTimeoutMs(promptChars: number): number { | ||||||||||||||||||||||||||||||||||||
| return Math.min(AI_TIMEOUT_MAX_MS, AI_TIMEOUT_BASE_MS + Math.ceil(promptChars / 2_000) * 1_000) | ||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
| /** Error text safe to show in a PR comment: one line, bounded, with anything | ||||||||||||||||||||||||||||||||||||
| * that looks like a key or bearer token masked. */ | ||||||||||||||||||||||||||||||||||||
| function safeReason(err: unknown): string { | ||||||||||||||||||||||||||||||||||||
| const raw = err instanceof Error ? err.message : String(err) | ||||||||||||||||||||||||||||||||||||
| return raw | ||||||||||||||||||||||||||||||||||||
| .replace(/\s+/g, " ") | ||||||||||||||||||||||||||||||||||||
| .replace(/(bearer\s+)[^\s]+/gi, "$1***") | ||||||||||||||||||||||||||||||||||||
| .replace(/\b(sk|pk|key|token)[-_][A-Za-z0-9_-]{8,}/gi, "***") | ||||||||||||||||||||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: (Based on your team's feedback about masking credential-stem error values.) [a5c5a10f-6a4a-4e42-8c2e-f8e01b48683d] Prompt for AI agents |
||||||||||||||||||||||||||||||||||||
| .slice(0, 200) | ||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||
|
Comment on lines
+61
to
68
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [🟠 MEDIUM] While this effectively sanitizes Consider expanding the regex to cover these patterns for more robust redaction. Suggested change:
Suggested change
|
||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
| /** | ||||||||||||||||||||||||||||||||||||
|
|
@@ -73,23 +107,39 @@ function buildUserMessage(input: AiReviewInput): string { | |||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
| /** | ||||||||||||||||||||||||||||||||||||
| * Run the LLM reviewer lane. Returns advisory findings (severity ≤ warning, | ||||||||||||||||||||||||||||||||||||
| * clamped by core), or [] if no model / core is available or the call fails — | ||||||||||||||||||||||||||||||||||||
| * a review must never crash because the AI layer is unavailable. | ||||||||||||||||||||||||||||||||||||
| * clamped by core) and a status saying whether the lane ran, was skipped, timed | ||||||||||||||||||||||||||||||||||||
| * out or failed. Never throws — a review must not crash because the AI layer is | ||||||||||||||||||||||||||||||||||||
| * unavailable — but never fails silently either: the status reaches the | ||||||||||||||||||||||||||||||||||||
| * envelope and the PR comment. | ||||||||||||||||||||||||||||||||||||
| */ | ||||||||||||||||||||||||||||||||||||
| export async function runAiReview(input: AiReviewInput): Promise<Finding[]> { | ||||||||||||||||||||||||||||||||||||
| export async function runAiReview(input: AiReviewInput): Promise<AiReviewResult> { | ||||||||||||||||||||||||||||||||||||
| const startedAt = Date.now() | ||||||||||||||||||||||||||||||||||||
| const done = (findings: Finding[], status: Omit<AiLaneStatus, "durationMs">): AiReviewResult => ({ | ||||||||||||||||||||||||||||||||||||
| findings, | ||||||||||||||||||||||||||||||||||||
| status: { ...status, durationMs: Date.now() - startedAt }, | ||||||||||||||||||||||||||||||||||||
| }) | ||||||||||||||||||||||||||||||||||||
| const files = input.files.filter((f) => f.status !== "deleted" && (f.diff || f.sql)) | ||||||||||||||||||||||||||||||||||||
| if (!files.length) return [] | ||||||||||||||||||||||||||||||||||||
| if (!files.length) return done([], { state: "skipped", reason: "no changed model SQL to review" }) | ||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
| const message = buildUserMessage({ ...input, files }) | ||||||||||||||||||||||||||||||||||||
| const controller = new AbortController() | ||||||||||||||||||||||||||||||||||||
| const timeout = setTimeout(() => controller.abort(), AI_TIMEOUT_MS) | ||||||||||||||||||||||||||||||||||||
| const timeoutMs = aiTimeoutMs(message.length) | ||||||||||||||||||||||||||||||||||||
| const timeout = setTimeout(() => controller.abort(), timeoutMs) | ||||||||||||||||||||||||||||||||||||
| let modelLabel: string | undefined | ||||||||||||||||||||||||||||||||||||
| try { | ||||||||||||||||||||||||||||||||||||
|
Comment on lines
+124
to
129
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [🔴 HIGH] Moving Consider moving the message generation and timeout setup inside the Suggested change:
Suggested change
|
||||||||||||||||||||||||||||||||||||
| // Prompt comes from the compiled core, not this file. | ||||||||||||||||||||||||||||||||||||
| const promptRes = await Dispatcher.call("altimate_core.review_ai_prompt", {}) | ||||||||||||||||||||||||||||||||||||
| const system = ((promptRes.data ?? {}) as Record<string, unknown>).prompt as string | undefined | ||||||||||||||||||||||||||||||||||||
| if (!system) return [] | ||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
| const defaultModel = await Provider.defaultModel() | ||||||||||||||||||||||||||||||||||||
| const model = await Provider.getModel(defaultModel.providerID, defaultModel.modelID) | ||||||||||||||||||||||||||||||||||||
| if (!system) return done([], { state: "error", reason: "the core reviewer prompt is unavailable" }) | ||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
| let model: Awaited<ReturnType<typeof Provider.getModel>> | ||||||||||||||||||||||||||||||||||||
| try { | ||||||||||||||||||||||||||||||||||||
| const chosen = input.model ? Provider.parseModel(input.model) : await Provider.defaultModel() | ||||||||||||||||||||||||||||||||||||
| model = await Provider.getModel(chosen.providerID, chosen.modelID) | ||||||||||||||||||||||||||||||||||||
| } catch (err) { | ||||||||||||||||||||||||||||||||||||
| return done([], { state: "skipped", reason: `no usable model: ${safeReason(err)}`, model: input.model }) | ||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||
| modelLabel = `${model.providerID}/${model.id}` | ||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
| const agent: Agent.Info = { | ||||||||||||||||||||||||||||||||||||
| name: "dbt-ai-reviewer", | ||||||||||||||||||||||||||||||||||||
|
|
@@ -119,10 +169,12 @@ export async function runAiReview(input: AiReviewInput): Promise<Finding[]> { | |||||||||||||||||||||||||||||||||||
| abort: controller.signal, | ||||||||||||||||||||||||||||||||||||
| sessionID: user.sessionID, | ||||||||||||||||||||||||||||||||||||
| retries: 1, | ||||||||||||||||||||||||||||||||||||
| messages: [{ role: "user", content: buildUserMessage({ ...input, files }) }], | ||||||||||||||||||||||||||||||||||||
| messages: [{ role: "user", content: message }], | ||||||||||||||||||||||||||||||||||||
| }) | ||||||||||||||||||||||||||||||||||||
| const text = await readStreamText(stream) | ||||||||||||||||||||||||||||||||||||
| if (!text) return [] | ||||||||||||||||||||||||||||||||||||
| if (controller.signal.aborted) | ||||||||||||||||||||||||||||||||||||
| return done([], { state: "timeout", reason: `no answer within ${Math.round(timeoutMs / 1000)} s`, model: modelLabel }) | ||||||||||||||||||||||||||||||||||||
| if (!text) return done([], { state: "error", reason: "the model returned no text", model: modelLabel }) | ||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
| // Parse + clamp in core (the prompt-injection-resistant, advisory-only | ||||||||||||||||||||||||||||||||||||
| // contract). Returns already-validated, severity-clamped, file-checked items. | ||||||||||||||||||||||||||||||||||||
|
|
@@ -165,10 +217,12 @@ export async function runAiReview(input: AiReviewInput): Promise<Finding[]> { | |||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||
| log.info("ai review complete", { findings: out.length }) | ||||||||||||||||||||||||||||||||||||
| return out | ||||||||||||||||||||||||||||||||||||
| return done(out, { state: "ok", model: modelLabel }) | ||||||||||||||||||||||||||||||||||||
| } catch (err) { | ||||||||||||||||||||||||||||||||||||
| log.error("ai review failed", { error: err }) | ||||||||||||||||||||||||||||||||||||
| return [] | ||||||||||||||||||||||||||||||||||||
| if (controller.signal.aborted) | ||||||||||||||||||||||||||||||||||||
| return done([], { state: "timeout", reason: `no answer within ${Math.round(timeoutMs / 1000)} s`, model: modelLabel }) | ||||||||||||||||||||||||||||||||||||
| return done([], { state: "error", reason: safeReason(err), model: modelLabel }) | ||||||||||||||||||||||||||||||||||||
| } finally { | ||||||||||||||||||||||||||||||||||||
| clearTimeout(timeout) | ||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -42,11 +42,12 @@ export function renderSummary(env: VerdictEnvelope): string { | |
|
|
||
| if (env.summary.degraded) { | ||
| lines.push( | ||
| "> ⚙️ **Lint-only run** — no dbt manifest/warehouse was available, so lineage, equivalence and", | ||
| "> ⚙️ **Lint-only run** — no changed model resolved against a dbt manifest, so lineage, equivalence and", | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. WARNING: The banner now asserts a cause that is only one of the two ways
|
||
| "> data-impact checks were skipped. Wire `manifest_path` (and optionally warehouse creds) for the full verdict.", | ||
| "", | ||
| ) | ||
| } | ||
| lines.push(...coverageNotes(env)) | ||
|
|
||
| // G1 (Round 18) — surface classifier reasons when --explain-tier populated | ||
| // them, so a customer can see why the review ran at this tier. Also surfaces | ||
|
|
@@ -110,11 +111,41 @@ export function renderSummary(env: VerdictEnvelope): string { | |
| `<sub>altimate dbt-pr-review · verdict \`${env.verdict}\`` + | ||
| (env.signature ? ` · signed \`${env.signature.slice(0, 18)}…\`` : "") + | ||
| (env.manifestHash ? ` · manifest \`${env.manifestHash.slice(0, 10)}\`` : "") + | ||
| (env.scope?.baseRef ? ` · base \`${env.scope.baseRef}\`` : "") + | ||
| (env.ai?.state === "ok" && env.ai.model ? ` · AI \`${env.ai.model}\`` : "") + | ||
| "</sub>", | ||
| ) | ||
| return lines.join("\n") | ||
| } | ||
|
|
||
| /** | ||
| * What the review could not fully check, each on its own line: undecidable | ||
| * findings, models without compiled SQL, files outside the dbt project, and | ||
| * the advisory AI lane when it did not run. Kept separate from the lint-only | ||
| * banner so one undecidable finding never mislabels a manifest-backed run. | ||
| */ | ||
| export function coverageNotes(env: VerdictEnvelope): string[] { | ||
| const notes: string[] = [] | ||
| const undecided = env.findings.filter((f) => f.degraded).length | ||
| if (undecided) notes.push(`${undecided} finding(s) could not be decided by the engine and are marked _unverified_.`) | ||
| const unverified = env.scope?.unverified ?? [] | ||
| if (unverified.length) { | ||
| const head = unverified.slice(0, 5).map((u) => `\`${u.path}\``).join(", ") | ||
| const more = unverified.length > 5 ? ` and ${unverified.length - 5} more` : "" | ||
| notes.push( | ||
| `${unverified.length} model(s) not verified by the engine — ${unverified[0].reason}: ${head}${more}. ` + | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: Mixed head/base compile failures are rendered with the first model’s reason for every model, which misstates review coverage. Group the entries by reason or render each model with its own reason. Prompt for AI agentsThere was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. SUGGESTION: Only the first model's reason is reported for the whole set.
|
||
| "Run `dbt compile` (and compile the base into `target-base/`) for equivalence and lineage checks.", | ||
| ) | ||
| } | ||
| if (env.scope?.outOfScopeFiles) | ||
| notes.push(`${env.scope.outOfScopeFiles} changed file(s) outside the dbt project were not reviewed.`) | ||
| if (env.ai && env.ai.state !== "ok") { | ||
| const what = { skipped: "skipped", timeout: "timed out", error: "failed" }[env.ai.state] | ||
| notes.push(`Advisory AI review ${what}${env.ai.reason ? `: ${env.ai.reason}` : ""}.`) | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: The PR comment does not report the AI lane’s available model and duration: successful runs show only the model, while failed or timed-out runs show only state and reason. Include model and duration in the rendered AI status. Prompt for AI agentsThere was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. SUGGESTION:
|
||
| } | ||
| return notes.length ? [...notes.map((n) => `> ℹ️ ${n}`), ""] : [] | ||
| } | ||
|
|
||
| /** GitHub `pulls.createReview` inline comments — only findings with a line. */ | ||
| export interface InlineComment { | ||
| path: string | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,7 +14,7 @@ import { type Rubric, exclusionReason, clampSeverity } from "./rubric" | |
| import { type ReviewConfig } from "./config" | ||
| import { type ReviewMode, type ReviewScope, type VerdictEnvelope, buildEnvelope, signEnvelope } from "./verdict" | ||
| import { detectModelPatterns, detectSchemaYmlPatterns, splitDiff } from "./dbt-patterns" | ||
| import { type AiReviewInput } from "./ai-review" | ||
| import { type AiLaneStatus, type AiReviewInput, type AiReviewResult } from "./ai-review" | ||
|
|
||
| /** | ||
| * The deterministic review recipe. | ||
|
|
@@ -188,7 +188,11 @@ export interface OrchestrateInput { | |
| * tests pass a fake or omit it (the lane is then skipped). Receives the | ||
| * deterministic findings as grounding and returns ADVISORY findings only. | ||
| */ | ||
| aiReview?: (input: AiReviewInput) => Promise<Finding[]> | ||
| aiReview?: (input: AiReviewInput) => Promise<AiReviewResult | Finding[]> | ||
| /** Why the AI lane is off when `aiReview` is not injected (shown in the comment). */ | ||
| aiDisabledReason?: string | ||
| /** Explicit `provider/model` for the AI lane. */ | ||
| aiModel?: string | ||
| /** PR metadata passed to the AI reviewer for intent checking. */ | ||
| prTitle?: string | ||
| prBody?: string | ||
|
|
@@ -1410,7 +1414,12 @@ export async function runReview(input: OrchestrateInput): Promise<VerdictEnvelop | |
| // findings are clamped to ≤ warning upstream, so they enrich the review but | ||
| // never change what blocks. Skipped when no aiReview fn is injected (tests / | ||
| // headless-without-model) — the review degrades to deterministic-only. | ||
| if (lanes.has("ai_review") && input.aiReview) { | ||
| let aiStatus: AiLaneStatus | undefined | ||
| if (!lanes.has("ai_review")) { | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: Empty-scope reviews now emit an AI Prompt for AI agents |
||
| aiStatus = { state: "skipped", reason: `not part of the ${tier} tier` } | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. SUGGESTION: This reason can be false.
|
||
| } else if (!input.aiReview) { | ||
| aiStatus = { state: "skipped", reason: input.aiDisabledReason ?? "disabled" } | ||
| } else { | ||
| const aiFiles = [...ctxByPath.values()].map((ctx) => ({ | ||
| path: ctx.file.path, | ||
| status: ctx.file.status, | ||
|
|
@@ -1419,16 +1428,20 @@ export async function runReview(input: OrchestrateInput): Promise<VerdictEnvelop | |
| sql: ctx.engineNewSql ?? ctx.newSql, | ||
| })) | ||
| try { | ||
| const aiFindings = await input.aiReview({ | ||
| const res = await input.aiReview({ | ||
| files: aiFiles, | ||
| grounding: merged, | ||
| prTitle: input.prTitle, | ||
| prBody: input.prBody, | ||
| model: input.aiModel, | ||
| }) | ||
| const aiFindings = Array.isArray(res) ? res : res.findings | ||
| aiStatus = Array.isArray(res) ? { state: "ok" } : res.status | ||
| // Defense in depth: the AI never blocks, regardless of what it returns. | ||
| for (const f of aiFindings) merged.push(f.severity === "critical" ? { ...f, severity: "warning" } : f) | ||
| } catch { | ||
| // A failed AI lane must never fail the review. | ||
| } catch (err) { | ||
| // A failed AI lane must never fail the review — but it must say so. | ||
| aiStatus = { state: "error", reason: String((err as Error)?.message ?? err).slice(0, 200) } | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: AI callback failures bypass the redaction promised by (Based on your team's feedback about masking credential-bearing error text.) [173edeff-f795-42ca-9d7d-86524f63e58a] Prompt for AI agentsThere was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. SUGGESTION: This reason reaches the signed envelope and the PR comment unmasked. Unlike |
||
| } | ||
| } | ||
|
|
||
|
|
@@ -1440,8 +1453,12 @@ export async function runReview(input: OrchestrateInput): Promise<VerdictEnvelop | |
| // Sort by severity desc, then file. | ||
| findings.sort((a, b) => SEVERITY_ORDER[b.severity] - SEVERITY_ORDER[a.severity] || a.file.localeCompare(b.file)) | ||
|
|
||
| const degraded = runDegraded || findings.some((f) => f.degraded) | ||
| // Lint-only means no changed model resolved against a manifest. A single | ||
| // undecidable finding does not make the whole run lint-only; those findings | ||
| // carry their own `degraded` flag and are counted separately in the summary. | ||
| const degraded = runDegraded | ||
| const envelope = buildEnvelope({ | ||
| ai: aiStatus, | ||
| findings, | ||
| tier, | ||
| mode: input.mode, | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -53,6 +53,8 @@ export interface ReviewPullRequestOptions { | |||||||||||||||||||||||||
| /** PR metadata for the AI reviewer's intent check. */ | ||||||||||||||||||||||||||
| prTitle?: string | ||||||||||||||||||||||||||
| prBody?: string | ||||||||||||||||||||||||||
| /** Explicit `provider/model` for the advisory AI lane. */ | ||||||||||||||||||||||||||
| aiModel?: string | ||||||||||||||||||||||||||
| /** G1 — surface the tier classifier's reasons on the verdict envelope. */ | ||||||||||||||||||||||||||
| explainTier?: boolean | ||||||||||||||||||||||||||
| /** G2 — [EXPERIMENTAL] bypass the tier classifier with the supplied tier. | ||||||||||||||||||||||||||
|
|
@@ -379,6 +381,8 @@ export async function reviewPullRequest(opts: ReviewPullRequestOptions): Promise | |||||||||||||||||||||||||
| // path verdicts (cubic-review PR #1041). | ||||||||||||||||||||||||||
| cliVersion: opts.cliVersion ?? Installation.VERSION, | ||||||||||||||||||||||||||
| aiReview: opts.noAi || config.ai === false ? undefined : runAiReview, | ||||||||||||||||||||||||||
| aiDisabledReason: opts.noAi ? "disabled by --no-ai" : config.ai === false ? "disabled by `ai: false` in .altimate/review.yml" : undefined, | ||||||||||||||||||||||||||
| aiModel: opts.aiModel, | ||||||||||||||||||||||||||
|
Comment on lines
382
to
+385
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [🔵 LOW] The checklist prohibits the use of nested ternary expressions. Please refactor this to avoid nesting, perhaps by computing the disabled reason in a separate variable before using it here. Suggested change:
Suggested change
|
||||||||||||||||||||||||||
| prTitle: opts.prTitle, | ||||||||||||||||||||||||||
| prBody: opts.prBody, | ||||||||||||||||||||||||||
| explainTier: opts.explainTier, | ||||||||||||||||||||||||||
|
|
@@ -396,6 +400,32 @@ export async function reviewPullRequest(opts: ReviewPullRequestOptions): Promise | |||||||||||||||||||||||||
| }) | ||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| /** | ||||||||||||||||||||||||||
| * The pull/merge request's title and description from the CI environment, so | ||||||||||||||||||||||||||
| * the AI lane can check the change against its stated intent. GitHub: the | ||||||||||||||||||||||||||
| * `pull_request` object in GITHUB_EVENT_PATH. GitLab: CI_MERGE_REQUEST_TITLE / | ||||||||||||||||||||||||||
| * CI_MERGE_REQUEST_DESCRIPTION. Empty outside a PR/MR run. | ||||||||||||||||||||||||||
| */ | ||||||||||||||||||||||||||
| export async function prContextFromEnv(env: NodeJS.ProcessEnv = process.env): Promise<{ prTitle?: string; prBody?: string }> { | ||||||||||||||||||||||||||
| if (env.GITHUB_EVENT_PATH) { | ||||||||||||||||||||||||||
| try { | ||||||||||||||||||||||||||
| const pr = JSON.parse(await readFile(env.GITHUB_EVENT_PATH, "utf8"))?.pull_request | ||||||||||||||||||||||||||
| if (pr) { | ||||||||||||||||||||||||||
| return { | ||||||||||||||||||||||||||
| prTitle: typeof pr.title === "string" && pr.title ? pr.title : undefined, | ||||||||||||||||||||||||||
| prBody: typeof pr.body === "string" && pr.body ? pr.body : undefined, | ||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
| } catch { | ||||||||||||||||||||||||||
| /* unreadable event file: try GitLab */ | ||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
| return { | ||||||||||||||||||||||||||
| prTitle: env.CI_MERGE_REQUEST_TITLE || undefined, | ||||||||||||||||||||||||||
| prBody: env.CI_MERGE_REQUEST_DESCRIPTION || undefined, | ||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| /** | ||||||||||||||||||||||||||
| * The dbt project being reviewed: the nearest directory at or above `cwd` | ||||||||||||||||||||||||||
| * (not above the repo root) holding dbt_project.yml; else the project that | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
P2: In a monorepo the base-compile example writes
target-base/at the repo root and runsdbtfrom the repo root, but the reviewer readstarget-base/relative toproject_dir. Equivalence and column-breakage checks would silently fall back to not verified. Note that the base compile must target<project_dir>/target-baseand run inside the project folder, or the Monorepos paragraph should say so.Prompt for AI agents