[kingsleytorlowei]

ReviewOutcome.gate(fail_on) checks a review against a WARNING or SERIOUS threshold using the rules of proofline ci review --fail-on. It returns REACHED, NOT_REACHED, NOT_COMPLETE or UNAVAILABLE, which correspond to the CLI's exit codes 3, 0, 2 and 4.

outcome = client.wait_for_review(organization_id, review_id)
result = outcome.gate(fail_on="SERIOUS")
if result != "NOT_REACHED":
    sys.exit({"REACHED": 3, "NOT_COMPLETE": 2, "UNAVAILABLE": 4}[result])

Who this helps, and the evidence

This is for Python pipelines and deploy tools that already call wait_for_review and want to fail on findings.

• The CLI encodes this rule twice: in ci review --fail-on and in the Git hooks' failOn setting. The comments on ci_verdict and code_reaches define it precisely. Add --fail-on to proofline review cli#46 is open and extends it to proofline review.
• Neither SDK has it. A Python caller gets a ReviewOutcome and has to rebuild the rule from the CLI source. Each part of it is easy to get wrong:
  • completeness has to decide first;
  • missing counts must not read as a pass;
  • an unknown severity counts as reaching every threshold;
  • CODE findings count only from a complete record of the review's own commits.
• What I didn't find: a user asking for this in Python. A Python pipeline can run the CLI. Once it has a ReviewOutcome in hand, though, it would have to start the CLI as a separate program and read its exit code. With this change it gets the same answer in one call.

Decisions

• Port the CLI's rule, not a new one. The order, counts and edge cases all follow ci_verdict, code_reaches, code_assessment and Severity::reached_by. A Python job and a CLI job therefore agree on the same review.
• Four results, not a boolean. "Incomplete" and "can't be measured" are different failures, and the CLI gives them different exit codes. A bool would merge them, and the shortest misuse would turn an incomplete review into a pass.
• The server's counts decide, not the findings on the page. open_items.open_findings and serious_findings already exclude dismissed findings and unexpired risk acceptances, and they include earlier unresolved findings of the same pull request. Recounting from assessments[].findings would get decisions wrong.
• Fail closed where the CLI does.
  • Missing or malformed counts give UNAVAILABLE. The generated model doesn't type-check them, so a string, bool or negative count is rejected here.
  • An unknown severity reaches every threshold.
  • A timed-out wait gives UNAVAILABLE.
• CODE findings reuse the SDK's own CODE checks. The commit-match and withheld-finding checks that landed in _code_assessment in 3.3.0 now live in _recorded_code. _code_assessment keeps its rule that a CODE result never stands in for an environment result, and calls _recorded_code. The gate calls _recorded_code directly, because the CLI measures a complete CODE assessment's findings even beside complete environments. There is one definition of a usable CODE record, and wait_for_review's behavior is unchanged.
  • Beside complete environments, a partial or unusable CODE record supplies no findings, as in the CLI.
  • A code-only review with such a record is already INCOMPLETE, so the gate gives NOT_COMPLETE.
• Separate from the server's merge_gate. merge_gate in the contract is an opt-in server policy. It records a frozen ALLOW or BLOCK decision for a pull request head, with bypass receipts. The CLI's --fail-on doesn't read it, and neither does this. Here the threshold is the caller's, and it works for any review, including organizations without a merge gate. The live review below has merge_gate: null. The README says which is which.
• A method on ReviewOutcome, keyword fail_on. It sits where the review is already in hand, and the name matches --fail-on and failOn. No new client method, so scripts/contracts.py is unchanged. The names avoid the forbidden outcome words.
• An unknown threshold raises ValueError. A typo like "HIGH" should fail loudly rather than gate on nothing.

Verification

• Against a real review. I installed Proofline on my fork and opened this branch there as Measure a review against a severity threshold like ci review --fail-on kingsleytorlowei/sdk-python#1. Proofline reviewed it: "no new findings".

  • wait_for_review(...).gate(...) on that review gave NOT_REACHED for both thresholds, matching the console.
  • The review was code-only, since no environment is configured. Its CODE record named this branch's head and base, so the gate's CODE path ran on real data.
  • It also confirmed what the gate depends on: the server sends open_items on a completed code-only review.

• python3 scripts/check.py passes end to end on Python 3.13: generation check, contracts, ruff check, ruff format --check, mypy, pytest (120 passed: 91 existing, 29 new), and both the wheel and sdist install smoke tests.

• tests/test_gate.py mirrors the CLI's ci_measures_only_a_complete_review_against_the_open_finding_counts table case for case. It adds:

  • malformed and missing counts;
  • CODE findings by severity and lifecycle (NEW, UNCHANGED, REINTRODUCED, RESOLVED, and the unknown CRITICAL), for both environment and code-only reviews;
  • each unusable CODE record (incomplete, incompleteness kind, other head, other base, withheld), shown to supply no findings beside environments and to give NOT_COMPLETE when code-only;
  • commits compared only where the review recorded them;
  • an end-to-end wait_for_review(...).gate(...) against the loopback server, sync and async.

• Mutation check. I broke each rule in turn and confirmed a test fails every time:

  • missing counts read as not reached;
  • malformed counts accepted;
  • unknown severity below the threshold;
  • RESOLVED counts;
  • partial CODE used, once by status and once by incompleteness;
  • CODE ignored beside environments;
  • SERIOUS uses all findings;
  • timeout reads as not complete;
  • incomplete measured anyway;
  • shared reader stops comparing commits;
  • shared reader uses withheld findings.

  All 12 were caught.

🤖 Generated with Claude Code