Skip to content

feat(review): capture per-reviewer AI votes with swap-proof attribution#8265

Merged
JSONbored merged 1 commit into
mainfrom
feat/reviewer-vote-capture
Jul 23, 2026
Merged

feat(review): capture per-reviewer AI votes with swap-proof attribution#8265
JSONbored merged 1 commit into
mainfrom
feat/reviewer-vote-capture

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Advances #8229 (stage 0 of 3 — the capture substrate; the would-have-routed report and flag-gated weighting follow once votes accumulate).

The dual-AI combiner never persisted which provider voted which way, and its disagreement tie-break judge re-runs the pair order-swapped — so any capture at slot-interpretation time would misattribute votes. This PR attaches votes where slot↔model is unambiguous by construction:

  • LoopOverAiReviewResult ok-branch gains reviewerVotes: { reviewer, votedFail }[], populated at leg production time (a.reviewprimary.model, b.reviewsecondary.model) in the dual path and on the single-reviewer block path. Advisory-only runs and unparseable legs cast no votes — never a fabricated stance.
  • Orchestration persists each vote as a best-effort reviewer_vote audit event (actor = model id, target = owner/repo#N, stance in metadata) after an ok result; a rejecting write is swallowed and never touches the review outcome.
  • The stored shape feeds computeProviderTrackRecords (calibration: per-provider reviewer track records from the consensus corpus #8228) directly in stage 1.

Validation

  • Full local gate npm run test:ci green (TEST_CI_EXIT=0) on a branch rebased onto current main, plus npm audit --audit-level=moderate.
  • 100% line+branch coverage on every changed range in src/services/ai-review.ts and src/queue/ai-review-orchestration.ts (verified via lcov against the exact diff hunks): split-case attribution, unparseable-leg no-vote, advisory-empty, persistence rows, and the swallowed-rejection arm each have both sides exercised.
  • New suite test/unit/reviewer-vote-capture.test.ts pins the persistence contract; runner-side invariants live beside the existing combiner tests in test/unit/ai-review.test.ts.

…on (#8229)

Block-mode dual review now attaches one vote per reviewer leg at production
time (a.review <-> primary.model, b.review <-> secondary.model), immune to the
tie-break judge's order swap. Ok results expose reviewerVotes; orchestration
persists each as a best-effort reviewer_vote audit event (actor = model id,
stance in metadata) for the later per-provider track-record stages. Advisory
runs and unparseable legs cast no votes.
@JSONbored JSONbored self-assigned this Jul 23, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@github-actions

Copy link
Copy Markdown
Contributor

Logic backtest

Replayed 0 historical case(s) for linked_issue_scope_mismatch through the base (264f26a) and head (d3d913f) versions of its detection logic (corpus checksum 4f53cda18c2b).

Backtest comparison: linked_issue_scope_mismatch

Verdict: unchanged — no comparable axis moved.

Advisory only — this check never blocks merge (#8105).

@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.62%. Comparing base (904ec7d) to head (d3d913f).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8265      +/-   ##
==========================================
- Coverage   92.15%   91.62%   -0.53%     
==========================================
  Files         786      786              
  Lines       78814    78823       +9     
  Branches    23790    23796       +6     
==========================================
- Hits        72630    72224     -406     
- Misses       5062     5524     +462     
+ Partials     1122     1075      -47     
Flag Coverage Δ
shard-1 56.10% <88.88%> (+2.24%) ⬆️
shard-2 49.48% <22.22%> (-0.79%) ⬇️
shard-3 54.66% <88.88%> (-2.64%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/queue/ai-review-orchestration.ts 100.00% <100.00%> (ø)
src/services/ai-review.ts 96.87% <100.00%> (+0.01%) ⬆️

... and 3 files with indirect coverage changes

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 23, 2026
@loopover-orb

loopover-orb Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Warning

⏸️ LoopOver review result - manual review recommended

Review updated: 2026-07-23 17:06:25 UTC

4 files · 1 AI reviewer · 2 blockers · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This PR adds a `reviewerVotes` array to `LoopOverAiReviewResult`'s ok-branch, populated at leg-production time in `runLoopOverAiReview` (src/services/ai-review.ts:2381-2383, 2444-2445, 2509) so slot↔model attribution is fixed before the order-swapped tie-break judge could ever muddy it, and persists each vote as a best-effort audit event in `runAiReviewForAdvisory` (src/queue/ai-review-orchestration.ts:724-734) after the ok-result check, guarded by `.catch(() => undefined)` so a write failure never touches the review outcome. The attribution logic is correct: votes are only pushed when `a.review`/`b.review` is present (never fabricated for an unparseable leg), and stay empty in advisory-only runs since the vote-pushing code only runs inside the `mode === "block"` branch. Tests cover the split-attribution, no-vote-on-unparseable, advisory-empty, and swallowed-rejection cases against the real production paths, not fabricated states.

Nits — 4 non-blocking
  • src/queue/ai-review-orchestration.ts:725 awaits each `recordAuditEvent` sequentially inside the `for` loop instead of firing both writes in parallel (e.g. `Promise.allSettled`) — with at most 2 votes this is a negligible cost, not worth blocking on.
  • Both changed files are now well over the repo's stated 400-line long-file threshold per the external size-smell brief; consider flagging for a future extraction pass rather than growing them further.
  • Consider batching the reviewer_vote writes with `Promise.allSettled` in src/queue/ai-review-orchestration.ts's new loop for a minor latency win when both reviewers vote.
  • If `recordAuditEvent`'s signature doesn't already support this call shape elsewhere, add a short comment noting where an existing calibration write uses the identical pattern, to make the 'best-effort like every calibration write' claim easy to verify for the next reader.

Concerns raised — review before merging

  • No linked issue detected: No closing reference or linked issue number was found in the PR metadata/body. — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue: This repo's maintainer focus manifest requires every PR to reference a tracked issue. — Link the relevant issue (for example Closes #123) before opening the PR.
📋 Copy for AI agents — paste into your coding agent
Fix the following blocker(s) from this PR review:

1. No linked issue detected: No closing reference or linked issue number was found in the PR metadata/body. — If this PR is intended to solve an issue, link it explicitly in the PR body.

2. Maintainer requires a linked issue: This repo's maintainer focus manifest requires every PR to reference a tracked issue. — Link the relevant issue (for example `Closes #123`) before opening the PR.

Decision drivers

  • ❌ Code review — 2 blockers (1 reviewer)
  • ❌ Gate result — Blocking (Repo-configured hard blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 14 registered-repo PR(s), 14 merged, 247 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 14 PR(s), 247 issue(s).
Improvement ✅ Minor risk: clean · value: minor · LLM: moderate
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
  • Official Gittensor activity: 14 PR(s), 247 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Then work through the remaining 2 steps in the Signals table above.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 23, 2026
@JSONbored
JSONbored merged commit d9cf5e1 into main Jul 23, 2026
16 checks passed
@JSONbored
JSONbored deleted the feat/reviewer-vote-capture branch July 23, 2026 21:20
JSONbored added a commit that referenced this pull request Jul 23, 2026
orb-v3.3.0 shipped stable, so the beta cutter is idle until the manifest's
declared target moves forward. The calibration-expansion epic's merges since
(per-repo loosening #8264, reviewer votes #8265, tightening #8267, AMS loop
#8270, counterfactual replay #8271) are feats — the inferred bump is minor,
so the next automated snapshot targets orb-v3.4.0-beta.1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant