Skip to content

fix(review): verdict trust: only the engine blocks, policy from the base branch, allowForceTier - #1722

Open
RaghwendraSingh wants to merge 1 commit into
fix/review-noisefrom
fix/review-verdict-trust
Open

RaghwendraSingh wants to merge 1 commit into
fix/review-noisefrom
fix/review-verdict-trust

Conversation

@RaghwendraSingh

@RaghwendraSingh RaghwendraSingh commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Issue for this PR

Closes #

Stacked on #1721, which is stacked on #1720 and #1719; merge in that order.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

The README and the docs promise "only the deterministic engine blocks". In the code, any three non-AI warnings, or one regex-catalog critical (join-no-on, create-or-replace, pii_into_mart, …), produced REQUEST_CHANGES. Separately, .altimate/review.yml was read from the PR's own checkout. So a PR could set mode: comment, drop a blockOn category, exclude its own paths, or narrow reviewers, and pass under its own relaxed rules.

  • Who can block: findings now carry origin: engine | catalog | ai, derived from evidence.tool. Unknown tools count as catalog, so a new detector can't block until someone classifies it. Only engine findings count toward REQUEST_CHANGES. A pattern rule a team trusts can be allowed to block via rubric.blockingRules: [<rule id>]. origin is stamped in buildEnvelope and covered by the signature.
  • Policy source: .altimate/review.yml is read at the merge-base commit (git show <base>:<path>). The PR's version is parsed only to list what it changes; those settings show in the comment and in policy.changedInPr and apply once merged. The envelope signs policy.source and policy.hash. Callers without git (pre-supplied files and content) still read the working tree.
  • --force-tier: allowForceTier: false in the base policy rejects the flag. Forced runs were already recorded as tier_forced in review_run.

Behavior changes:

  • Pattern-rule criticals and warning counts no longer block by default.
  • A PR's edits to review.yml no longer apply to that PR.
  • The tests that asserted the old behavior are updated: the two pii_into_mart regex-twin tests now expect COMMENT, and the verdict unit tests use engine evidence.

Not covered here: CLI flags in the workflow file (--mode, --severity) still override the policy. On pull_request events GitHub runs the workflow from the PR, so a PR could edit those flags too. Branch protection on .github/ (CODEOWNERS) is the usual control there.

How did you verify your code works?

  • New test/altimate/review-verdict-trust.test.ts (11 tests):
    • origin classification, including unknown tools
    • 1 catalog critical + 3 catalog warnings → COMMENT
    • blockingRules opt-in, and engine criticals and warning patterns still blocking
    • origin covered by the signature (tamper fails verification)
    • policy hash and change listing
    • on real git repos with GITHUB_BASE_REF: a PR relaxing mode/exclude/ai still judged in gate mode with the change shown; a PR adding review.yml not applied; an unchanged policy applied with no note; allowForceTier: false rejecting --force-tier
  • 1,883 tests across the review files and release gates pass. bun typecheck is clean.

Cost side effects

None. One extra local git show per run.

Screenshots / recordings

N/A (CLI).

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code


Summary by cubic

Makes the review verdict deterministic and tamper-proof: only the deterministic engine can block, and a PR can no longer relax the policy it is reviewed under.

Old behavior: any three non-AI warnings, or one regex-catalog critical, produced REQUEST_CHANGES, and .altimate/review.yml was read from the PR's own checkout. Now pattern-rule criticals and warning counts only comment unless rubric.blockingRules opts a catalog rule in, and the policy comes from the merge-base commit so a PR's edits are listed but only take effect after merge. allowForceTier: false in the base policy rejects --force-tier.

Behavior changes

  • Findings carry an origin (engine | catalog | ai) derived from evidence.tool, stamped in the envelope and covered by the signature; unknown tools count as catalog, so a new detector can't block until classified.
  • Only engine findings count toward REQUEST_CHANGES; the AI lane never blocks.
  • A PR's changes to review.yml show in the comment and policy.changedInPr, apply after merge, and the applied policy's source and hash are signed into the envelope. Non-git callers still read the working tree.
  • CLI flags in the workflow file still override the policy on pull_request events.

Written for commit 4004ec6. Summary will update on new commits.

View guided diff Turn on auto-fix

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 096d0e9a-51a6-4da2-9feb-2b8dd4b32b2e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

  1. Open an issue describing the bug/feature (if one doesn't exist)
  2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8 issues found across 12 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/opencode/test/altimate/review.test.ts">

<violation number="1" location="packages/opencode/test/altimate/review.test.ts:145">
P2: This fixture update leaves the unknown-confidence regression test ineffective: `u()` has no engine evidence, so its findings are classified as catalog and excluded before confidence is evaluated. Add deterministic-engine evidence to `u()` (or reuse the engine fixture) so the test proves three unknown engine warnings cannot block.</violation>
</file>

<file name="packages/opencode/src/altimate/review/format.ts">

<violation number="1" location="packages/opencode/src/altimate/review/format.ts:221">
P2: This note misidentifies the policy file when the repository uses the supported `.altimate/review.yaml` or `.altimate/review.json` filename. Use a generic `review policy` label or carry the selected filename into the envelope before rendering.</violation>
</file>

<file name="packages/opencode/test/altimate/review-verdict-trust.test.ts">

<violation number="1" location="packages/opencode/test/altimate/review-verdict-trust.test.ts:115">
P3: These integration tests leak one temporary Git repository per case. Use the repository test fixture or remove each root in a `finally` block so repeated test runs do not accumulate temp directories.</violation>
</file>

<file name="packages/opencode/src/altimate/review/finding.ts">

<violation number="1" location="packages/opencode/src/altimate/review/finding.ts:104">
P1: `findingOrigin` trusts caller-supplied `origin`, so a catalog or untrusted finding can label itself `engine` and block. Always derive the origin from `evidence.tool` and overwrite any supplied origin before computing and signing the verdict.</violation>

<violation number="2" location="packages/opencode/src/altimate/review/finding.ts:107">
P1: The namespace prefix is too broad for the trust boundary: an unrecognized `altimate_core.*` tool is promoted to `engine` and can block. Replace the prefix check with an explicit allowlist of the known deterministic evidence tools so new tools remain catalog until classified.</violation>
</file>

<file name="packages/opencode/src/altimate/review/run.ts">

<violation number="1" location="packages/opencode/src/altimate/review/run.ts:415">
P2: `policy.hash` is stale whenever a CLI override changes the effective configuration, especially `--severity`, whose value is not signed elsewhere. Recompute the hash from the effective user-configured settings after applying overrides so the signed policy metadata identifies the settings that produced the verdict.

(Based on your team's feedback about hashing effective review settings.)</violation>
</file>

<file name="docs/docs/usage/dbt-pr-review.md">

<violation number="1" location="docs/docs/usage/dbt-pr-review.md:289">
P1: The base policy does not prevent workflow flags from relaxing the review: `--mode` and `--severity` override it after loading. Document these overrides and tell users to protect the workflow separately.</violation>
</file>

<file name="packages/opencode/src/altimate/review/git.ts">

<violation number="1" location="packages/opencode/src/altimate/review/git.ts:269">
P1: This catch treats every `git show` failure as a missing policy, so an unavailable or corrupt base object silently falls back to `DEFAULT_REVIEW_CONFIG` and can bypass the base branch's restrictions. Only suppress the expected missing-path error; rethrow other Git failures.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

if (f.origin) return f.origin
const tool = f.evidence?.tool ?? ""
if (tool === "ai-review") return "ai"
if (tool.startsWith("altimate_core.") || ENGINE_TOOLS.has(tool)) return "engine"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: The namespace prefix is too broad for the trust boundary: an unrecognized altimate_core.* tool is promoted to engine and can block. Replace the prefix check with an explicit allowlist of the known deterministic evidence tools so new tools remain catalog until classified.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/finding.ts, line 107:

<comment>The namespace prefix is too broad for the trust boundary: an unrecognized `altimate_core.*` tool is promoted to `engine` and can block. Replace the prefix check with an explicit allowlist of the known deterministic evidence tools so new tools remain catalog until classified.</comment>

<file context>
@@ -84,9 +84,36 @@ export const Finding = z.object({
+  if (f.origin) return f.origin
+  const tool = f.evidence?.tool ?? ""
+  if (tool === "ai-review") return "ai"
+  if (tool.startsWith("altimate_core.") || ENGINE_TOOLS.has(tool)) return "engine"
+  return "catalog"
+}
</file context>

Comment on lines +289 to +293
commit the PR branched from, so a PR cannot relax the rules it is reviewed
under (switch to `comment` mode, drop a `blockOn` category, exclude its own
paths). If the PR edits the file, the comment lists the changed settings; they
take effect after merge. The applied policy's hash is signed into the envelope
(`policy`).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: The base policy does not prevent workflow flags from relaxing the review: --mode and --severity override it after loading. Document these overrides and tell users to protect the workflow separately.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/usage/dbt-pr-review.md, line 289:

<comment>The base policy does not prevent workflow flags from relaxing the review: `--mode` and `--severity` override it after loading. Document these overrides and tell users to protect the workflow separately.</comment>

<file context>
@@ -271,16 +271,32 @@ exclude:
+### Which policy applies, and what can block
+
+- **The base branch's `.altimate/review.yml` applies.** The file is read from the
+  commit the PR branched from, so a PR cannot relax the rules it is reviewed
+  under (switch to `comment` mode, drop a `blockOn` category, exclude its own
+  paths). If the PR edits the file, the comment lists the changed settings; they
</file context>
Suggested change
commit the PR branched from, so a PR cannot relax the rules it is reviewed
under (switch to `comment` mode, drop a `blockOn` category, exclude its own
paths). If the PR edits the file, the comment lists the changed settings; they
take effect after merge. The applied policy's hash is signed into the envelope
(`policy`).
merge-base, so changing `.altimate/review.yml` in a PR cannot relax the
config-derived policy (for example, by dropping `blockOn` categories or
excluding its own paths). Workflow flags such as `--mode` and `--severity`
still override policy, so protect the workflow separately. If the PR edits the
file, the comment lists the changed settings; they take effect after merge.
The applied policy's hash is signed into the envelope (`policy`).

/** The layer a finding came from. Unknown or missing evidence counts as
* `catalog`, so a new detector can never block until it is classified. */
export function findingOrigin(f: Pick<Finding, "evidence" | "origin">): FindingOrigin {
if (f.origin) return f.origin

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: findingOrigin trusts caller-supplied origin, so a catalog or untrusted finding can label itself engine and block. Always derive the origin from evidence.tool and overwrite any supplied origin before computing and signing the verdict.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/finding.ts, line 104:

<comment>`findingOrigin` trusts caller-supplied `origin`, so a catalog or untrusted finding can label itself `engine` and block. Always derive the origin from `evidence.tool` and overwrite any supplied origin before computing and signing the verdict.</comment>

<file context>
@@ -84,9 +84,36 @@ export const Finding = z.object({
+/** The layer a finding came from. Unknown or missing evidence counts as
+ *  `catalog`, so a new detector can never block until it is classified. */
+export function findingOrigin(f: Pick<Finding, "evidence" | "origin">): FindingOrigin {
+  if (f.origin) return f.origin
+  const tool = f.evidence?.tool ?? ""
+  if (tool === "ai-review") return "ai"
</file context>

export async function showAtRef(cwd: string, ref: string, rel: string): Promise<string | undefined> {
try {
return await git(["show", `${ref}:${rel}`], cwd)
} catch {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: This catch treats every git show failure as a missing policy, so an unavailable or corrupt base object silently falls back to DEFAULT_REVIEW_CONFIG and can bypass the base branch's restrictions. Only suppress the expected missing-path error; rethrow other Git failures.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/git.ts, line 269:

<comment>This catch treats every `git show` failure as a missing policy, so an unavailable or corrupt base object silently falls back to `DEFAULT_REVIEW_CONFIG` and can bypass the base branch's restrictions. Only suppress the expected missing-path error; rethrow other Git failures.</comment>

<file context>
@@ -261,6 +261,16 @@ export async function resolveReviewBase(opts: ResolveBaseOptions): Promise<Resol
+export async function showAtRef(cwd: string, ref: string, rel: string): Promise<string | undefined> {
+  try {
+    return await git(["show", `${ref}:${rel}`], cwd)
+  } catch {
+    return undefined
+  }
</file context>

notes.push(`${env.scope.outOfScopeFiles} changed file(s) outside the dbt project were not reviewed.`)
if (env.policy?.changedInPr?.length) {
notes.push(
`This PR changes the review policy (\`.altimate/review.yml\`: ${env.policy.changedInPr.join(", ")}). ` +

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This note misidentifies the policy file when the repository uses the supported .altimate/review.yaml or .altimate/review.json filename. Use a generic review policy label or carry the selected filename into the envelope before rendering.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/format.ts, line 221:

<comment>This note misidentifies the policy file when the repository uses the supported `.altimate/review.yaml` or `.altimate/review.json` filename. Use a generic `review policy` label or carry the selected filename into the envelope before rendering.</comment>

<file context>
@@ -216,6 +216,12 @@ export function coverageNotes(env: VerdictEnvelope): string[] {
     notes.push(`${env.scope.outOfScopeFiles} changed file(s) outside the dbt project were not reviewed.`)
+  if (env.policy?.changedInPr?.length) {
+    notes.push(
+      `This PR changes the review policy (\`.altimate/review.yml\`: ${env.policy.changedInPr.join(", ")}). ` +
+        "The base branch's policy was applied; the change takes effect after merge.",
+    )
</file context>
Suggested change
`This PR changes the review policy (\`.altimate/review.yml\`: ${env.policy.changedInPr.join(", ")}). ` +
`This PR changes the review policy (${env.policy.changedInPr.join(", ")}). ` +

Comment thread docs/docs/usage/dbt-pr-review.md Resolved
staleManifest,
layout,
suppressPortability: config.portability === undefined ? dialectKnown : !config.portability,
policy: { source: policy.source, hash: policy.hash, changedInPr: policy.changedInPr.length ? policy.changedInPr : undefined },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: policy.hash is stale whenever a CLI override changes the effective configuration, especially --severity, whose value is not signed elsewhere. Recompute the hash from the effective user-configured settings after applying overrides so the signed policy metadata identifies the settings that produced the verdict.

(Based on your team's feedback about hashing effective review settings.)

View Feedback

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/run.ts, line 415:

<comment>`policy.hash` is stale whenever a CLI override changes the effective configuration, especially `--severity`, whose value is not signed elsewhere. Recompute the hash from the effective user-configured settings after applying overrides so the signed policy metadata identifies the settings that produced the verdict.

(Based on your team's feedback about hashing effective review settings.) </comment>

<file context>
@@ -392,6 +412,7 @@ export async function reviewPullRequest(opts: ReviewPullRequestOptions): Promise
     staleManifest,
     layout,
     suppressPortability: config.portability === undefined ? dialectKnown : !config.portability,
+    policy: { source: policy.source, hash: policy.hash, changedInPr: policy.changedInPr.length ? policy.changedInPr : undefined },
     scope: {
       baseRef: resolved?.ref ?? opts.base,
</file context>

})

async function repo(baseYml: string | undefined, headYml: string | undefined) {
const root = await fs.realpath(await fs.mkdtemp(path.join(tmpdir(), "review-policy-")))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: These integration tests leak one temporary Git repository per case. Use the repository test fixture or remove each root in a finally block so repeated test runs do not accumulate temp directories.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/test/altimate/review-verdict-trust.test.ts, line 115:

<comment>These integration tests leak one temporary Git repository per case. Use the repository test fixture or remove each root in a `finally` block so repeated test runs do not accumulate temp directories.</comment>

<file context>
@@ -0,0 +1,174 @@
+  })
+
+  async function repo(baseYml: string | undefined, headYml: string | undefined) {
+    const root = await fs.realpath(await fs.mkdtemp(path.join(tmpdir(), "review-policy-")))
+    await g(root, "init", "-q", "-b", "main")
+    await g(root, "config", "user.email", "t@test")
</file context>


/** File content at a commit (`git show <ref>:<path>`), or undefined when the
* path does not exist there. `rel` is repository-relative, POSIX. */
export async function showAtRef(cwd: string, ref: string, rel: string): Promise<string | undefined> {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: A failed git show is silently treated as "no config at the base commit"

readFirst treats the undefined returned here as "candidate absent" and moves on, so loadReviewPolicy falls back to DEFAULT_REVIEW_CONFIG (mode: comment, allowForceTier: true) and stamps policy.source: "default" — indistinguishable from "the base branch has no config". Any git-level failure (bad object, missing shallow history, permissions, git erroring for a reason other than the path being absent) therefore silently demotes a gate policy to the permissive default instead of surfacing it, which is the opposite of the trust property this PR adds. git cat-file -e <ref>:<path> (or checking the exit code/stderr) would distinguish "path missing at that commit" from "git failed" and let the latter throw.

if (origin === "engine") return true
if (origin === "ai") return false
const rule = catalogRuleId(f)
return !!rule && rubric.blockingRules.includes(rule)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: blockingRules entries are matched by exact string and never validated

Rule ids are emitted by two catalogs with different conventions (rule-catalog uses hardcoded-credential, dbt-patterns uses pii_into_mart), and nothing checks that a configured id exists. A typo, a case difference, or the other convention therefore leaves the gate silently non-blocking — the exact failure mode a team opting a rule in wants to avoid — with no error and no note in the comment. Consider validating rubric.blockingRules against the known catalog ids when the rubric is resolved (warning when an entry matches nothing) and documenting where teams can find the ids.

changedInPr: string[]
}

function canonical(value: unknown): string {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: canonical duplicates stableStringify in verdict.ts

verdict.ts:278 already implements this exact algorithm (keys sorted at every depth, array order preserved, undefined dropped) for the signature body. Export that helper (or move it to a shared module) and use it for policyHash too: policy.hash is signed alongside the signature body, so the two canonicalizations should stay identical by construction instead of being two copies that can drift.

}): Promise<ReviewPolicy> {
if (!opts.readBase) {
const config = await loadReviewConfig(opts.cwd)
const present = await readFirst((rel) => fs.readFile(path.join(opts.cwd, rel), "utf8").catch(() => undefined))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: The no-git branch reads the same candidate file twice

loadReviewConfig above has already scanned CANDIDATE_FILES and read the first existing file; this second readFirst repeats the same read sequence only to learn whether a file was present, and the candidate loop is now written a third time here. Folding both into one helper that returns { text, rel } | undefined (and parsing that text once) removes the extra I/O and the chance of the two passes disagreeing if the file changes between them.

const out: string[] = []
const keys = new Set([...Object.keys(base), ...Object.keys(head)]) as Set<keyof ReviewConfig>
for (const k of [...keys].sort()) {
if (canonical(base[k]) === canonical(head[k])) continue

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Comparing parsed partial configs reports settings spelled out at their default values as changes

Representation, not effective policy, drives this comparison. A base file with mode: gate parses to rubric: {}; a PR that adds rubric:\n thresholds:\n warehouseCostMinRows: 1000000 (the default) parses to a full rubric.thresholds object, so the PR comment claims it "changes the review policy (rubric.thresholds)" that "takes effect after merge" even though the resolved rubric is identical. Comparing resolveRubric(...) results (and normalized top-level values) instead of the raw partials would keep the note to genuine behavior changes.

throw new Error("--force-tier is disabled by `allowForceTier: false` in the base branch's .altimate/review.yml")
}
if (opts.manifestPath) config.manifestPath = opts.manifestPath
if (opts.mode) config.mode = opts.mode

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: policy.hash is computed before these overrides, so it can describe a different policy than the envelope

loadReviewPolicy hashes the file config, then this line and the severityThreshold override mutate config, and the envelope later records policy.hash next to mode: config.mode / severityThreshold. With --mode gate (which github/review/action.yml always passes) the signed hash says comment while the signed mode says gate, so an auditor cannot check that the hashed policy is the one that was applied. Hashing after the overrides, or documenting that the hash covers only the config file, would remove the ambiguity.

title: "t",
body: "b",
file: `m_${Math.random()}.sql`,
evidence: { tool: "altimate_core.check", result: {} },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: This makes the unknown-confidence test pass for the wrong reason

Adding engine evidence to mk is right for the blocking assertions, but "unknown-confidence warnings do NOT accumulate into a block" (u() around line 168) builds findings with raw makeFinding and no evidence. Those are now catalog origin and ineligible regardless of confidence, so the f.confidence !== "unknown" clause in computeIdealVerdict is no longer exercised by that test. Giving u() engine evidence (altimate_core.equivalence) keeps it guarding the confidence rule.

### Which policy applies, and what can block

- **The base branch's `.altimate/review.yml` applies.** The file is read from the
commit the PR branched from, so a PR cannot relax the rules it is reviewed

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: The docs state a stronger trust boundary than the shipped path enforces

The policy file is base-scoped, but github/review/action.yml always passes --mode/--severity from the workflow (args=(--mode "$IN_MODE" ...)), and on pull_request events that workflow comes from the PR. A PR can therefore still switch the gate to comment mode by editing .github/workflows/*.yml — the PR description notes this as out of scope, but this bullet reads as if no PR-side edit can relax the gate. Adding that caveat (and pointing at CODEOWNERS / branch protection on .github/) would keep the doc accurate.

@kilo-code-bot

kilo-code-bot Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Code Review Summary

Status: 8 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 6

The design is sound and the core claims check out: origin is derived from evidence.tool (altimate_core.* plus impact_analysis / dbt.manifest / data.diff / schema.detect_pii), every engine lane that emits findings sets evidence, the AI lane still hardcodes tool: "ai-review" and never sets origin (so it cannot become verdict-eligible), buildEnvelope stamps origin before signEnvelope so it is covered by the signature, and both shipped callers (cli/cmd/review.ts, tools/dbt-pr-review.ts) resolve a base, so they now read the policy from the merge-base commit rather than the PR checkout.

Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/altimate/review/git.ts 266 showAtRef swallows every git error as "path absent", so a git failure silently falls back to the default policy (mode comment, allowForceTier: true) instead of surfacing the failure
packages/opencode/src/altimate/review/verdict.ts 52 blockingRules ids are matched by exact string with no validation; ids come from two catalogs with different conventions, so a typo silently leaves the gate non-blocking

SUGGESTION

File Line Issue
packages/opencode/src/altimate/review/config.ts 133 canonical duplicates stableStringify from verdict.ts; reuse one implementation for the hash and the signature
packages/opencode/src/altimate/review/config.ts 194 No-git branch reads the same candidate config file twice; fold into a single read that also reports presence
packages/opencode/src/altimate/review/config.ts 158 policyChanges compares parsed partials, so settings spelled out at default values (e.g. rubric.thresholds) are reported as policy changes
packages/opencode/src/altimate/review/run.ts 261 policy.hash is computed before --mode / --severity overrides, so the signed hash can disagree with the signed mode
packages/opencode/test/altimate/review.test.ts 145 Adding engine evidence to mk makes the unknown-confidence test pass because those findings are now ineligible catalog, not because of confidence
docs/docs/usage/dbt-pr-review.md 289 Docs omit that workflow-level --mode / --severity flags (from the PR-editable workflow file) still override the base policy

Notes / assumptions: this review found no path where the new code makes a non-engine finding block, and the AI lane remains non-blocking. The documented residual gap (pre-supplied changedFiles + getContent, and CLI flags in the workflow) is consistent with the PR description; only the missing user-doc caveat is flagged. Tests could not be executed in this read-only review, so the test-quality finding is derived from reading the assertions.

Files Reviewed (12 files)
  • docs/docs/usage/dbt-pr-review.md - 1 issue
  • packages/opencode/src/altimate/review/config.ts - 3 issues
  • packages/opencode/src/altimate/review/finding.ts - 0 issues
  • packages/opencode/src/altimate/review/format.ts - 0 issues
  • packages/opencode/src/altimate/review/git.ts - 1 issue
  • packages/opencode/src/altimate/review/orchestrate.ts - 0 issues
  • packages/opencode/src/altimate/review/rubric.ts - 0 issues
  • packages/opencode/src/altimate/review/run.ts - 1 issue
  • packages/opencode/src/altimate/review/verdict.ts - 1 issue
  • packages/opencode/test/altimate/review-verdict-trust.test.ts - 0 issues
  • packages/opencode/test/altimate/review.test.ts - 1 issue
  • packages/opencode/test/skill/release-v0.8.1-adversarial.test.ts - 0 issues

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4.1-flash-1017 · Input: 253.8K · Output: 46K · Cached: 3.5M

Review guidance: REVIEW.md from base branch fix/review-noise (truncated)

…ase branch, `allowForceTier`

- Findings carry `origin: engine | catalog | ai` (from `evidence.tool`;
  unknown tools count as catalog). Only engine findings count toward
  REQUEST_CHANGES, matching the documented promise; a pattern rule can be
  opted in by id via `rubric.blockingRules`. The AI lane never blocks.
- `.altimate/review.yml` is read from the merge-base commit, so a PR cannot
  relax the rules it is judged by. Settings the PR changes are listed in the
  comment and envelope (`policy.changedInPr`) and take effect after merge.
  The applied policy's source and hash are signed into the envelope.
- `allowForceTier: false` in the base policy rejects `--force-tier`
  (forced runs were already recorded as `tier_forced` in `review_run`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RaghwendraSingh
RaghwendraSingh force-pushed the fix/review-verdict-trust branch from 8ce9cec to 4004ec6 Compare November 13, 2026 06:36
@kilo-code-bot

kilo-code-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

Kilo Code Review could not run — your account is out of credits.

Add credits or switch to a free model to enable reviews on this change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant