Skip to content

[Feature]: Admit trusted same-revision CI outcomes as bounded review evidence #161

Description

@xeonvs

Problem

A code review can make a falsifiable runtime or compatibility claim even though an earlier mandatory CI job already executed the exact changed path successfully on the same source revision. When review runs in a separate pipeline or job, the toolkit currently supplies immutable repository evidence but no trusted, revision-bound outcome from sibling or earlier pipelines.

The reviewer must then infer behavior from source and model knowledge alone. This is particularly weak for SQL dialect, generated-code, compiler, integration, and platform claims where a successful targeted job is materially stronger than a static guess.

This is not a request to treat "pipeline green" as proof that all code is correct. It is a request to expose narrowly scoped execution evidence with explicit provenance and limitations.

Synthetic scenario

  • Revision H changes a functional-test helper to execute a database expression.
  • Mandatory job functional-tests for revision H runs the complete test package and succeeds.
  • A later review-only pipeline for the same revision H has no dependency edge to that job and receives no result metadata.
  • OCR reports that the expression is syntactically invalid and would make the same functional suite fail.

The already-observed execution does not prove every semantic property, but it directly contradicts the claimed unconditional syntax failure on the executed path.

Proposed behavior

Introduce an optional provider-neutral evidence record for verified CI outcomes, populated by a forge adapter before OCR starts. A candidate contract could include:

{
  "kind": "review.ci_outcome",
  "revision": "<exact immutable head>",
  "check": "functional-tests",
  "status": "passed",
  "required": true,
  "scope": {
    "mode": "declared",
    "paths": ["tests/**"]
  },
  "source": "forge-api",
  "completed_at": "<bounded timestamp>"
}

The exact schema should distinguish:

  • exact revision identity;
  • required/advisory/skipped/canceled/failed/unknown status;
  • provider-declared check identity;
  • trustworthy scope versus unknown scope;
  • pipeline/job origin and freshness without exposing private URLs or logs;
  • whether the result came from the current pipeline or another pipeline for the same revision.

Expose records through the existing read-only MCP and include only compact counts/status hints in the bootstrap.

Trust and safety constraints

  • Accept only forge API data authenticated for the selected project and bound to the exact immutable head revision.
  • Never accept repository-authored status text as authoritative execution evidence.
  • Do not download or send job logs by default.
  • Do not expose project names, private URLs, runner names, user identities, raw IDs, variables, artifacts, or failure traces to the model/public summary.
  • Normalize provider-specific statuses into a closed schema and retain unknown values as unknown.
  • A passing job proves only that its declared/known scope completed; it does not prove absence of unrelated defects.
  • A skipped, canceled, allowed-to-fail, stale, different-revision, or unknown-scope job must not be presented as successful validation.
  • Do not make CI data an authorization channel for posting, approval, merge, or suppression.

Acceptance criteria

  • Synthetic GitLab-style and provider-neutral fixtures admit a required successful check only when its revision exactly matches the reviewed head.
  • Same-name checks on another revision, stale pipelines, ambiguous retries, and untrusted repository payloads are rejected or marked unknown.
  • Current-pipeline and separate same-revision-pipeline outcomes have explicit provenance.
  • Scope is explicit; unknown scope cannot prove that a finding's path executed.
  • MCP summary/list/get can retrieve bounded outcome records with completeness state.
  • Bootstrap remains within OCR-owned limits and contains no private identifiers.
  • DLP and persistence readback reject private URLs, raw logs, credentials, identities, malformed status values, and oversized fields.
  • Approval remains conservative: a passing check is contextual evidence, not automatic finding suppression or clean-review authority.
  • A synthetic review prompt can use a same-revision functional-test pass to challenge an unconditional "this executed syntax always fails" claim.

Non-goals

  • Arbitrary CI log ingestion.
  • Executing repository tests from the evidence collector.
  • Treating all green checks as proof of correctness.
  • Provider-specific policy embedded in review prompts.
  • Changing pipeline gating or allow-failure behavior.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions