Skip to content

merge-clearance: a rate-limited CodeRabbit reports status success, and the gate scores it as a real review #65

Description

@mujtaba3B

The gap

merge-clearance.sh recognizes three rate-limit shapes, keyed off CodeRabbit's commit status on HEAD:

CR status signal gate function
missing marker comment anywhere on the PR mc_cr_rate_limited
pending marker is CR's LATEST comment mc_cr_rate_limited_latest
failure status DESCRIPTION says rate limited, or marker is latest mc_cr_failure_rate_limited

There is a fourth shape none of them cover: status success whose DESCRIPTION says Review rate limited. CodeRabbit posts this when it hits its review limit but still closes out the check as successful. The gate reads the success state, finds no rate-limit handling for it, and scores the dimension as a genuine clean review.

Observed

mujtaba3B/mutwo-skills PR #146, HEAD 866ed41b374431e50c10ba6a1f5ba56f37ed86fc:

$ gh api repos/mujtaba3B/mutwo-skills/commits/866ed41.../statuses \
    -q 'first(.[]|select(.context=="CodeRabbit")|.description)'
Review rate limited

$ gh pr checks 146
CodeRabbit   pass   0    Review rate limited

CodeRabbit posted a rate limited by coderabbit.ai marker comment and zero review comments; it never reviewed that HEAD. merge-clearance check scored it:

- [x] **CodeRabbit** - ✅ verdict=clear, status=success, reviewed-head=yes

reviewed-head=yes is wrong. Nothing reviewed that head.

Why it matters

The three existing shapes all end in the same safe place: the rate limit is only satisfied when a current local /eng:cr review backstops the HEAD. That backstop is the whole point. In this fourth shape the gate never learns a rate limit happened, so it never asks for the backstop, and the CodeRabbit dimension passes with no review of any kind behind it.

It happened to be safe on #146 because a current /eng:cr stamp existed anyway and I noted the situation in the PR. A PR without that stamp would have merged with the CodeRabbit dimension satisfied by nothing.

Suggested fix

Check the status DESCRIPTION for a rate-limit match on success too, not only on failure. mc_cr_failure_rate_limited already does exactly this parse; the same non-negated rate limit description match applied to a success state would route it into the existing backstop path rather than into a clean pass.

Worth deciding whether success should use the strict marker rule (marker must be CR's LATEST comment, as pending and failure require, since CR demonstrably ran) or accept the description alone. The description is the stronger signal here: it is CR's own account of what it did on this HEAD.

Related but distinct: #54 is about an incremental review that emits no artifact. This one is about a review that never happened being reported as success.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions