Skip to content

Spike: issue triage — should Vera listen to issues and label/dedupe when filed by an admin? #15

Description

@mabry1985

Spike, not a build. Time-boxed investigation to decide whether this belongs to Vera at all, and if so what v1 is.

The issues webhook event is being subscribed on the GitHub App alongside issue_comment and pull_request_review_comment (which the summon surface needs — pr-reviewer-plugin#28). So the events will arrive; nothing consumes them yet. Until a handler exists they drop as not-a-pr-event, which is harmless but should be silenced once this is decided either way.

The idea

When an admin files an issue, Vera triages it: proposes labels, flags likely duplicates, maybe asks for a repro.

Why it is not just "the panel, but for issues"

The review panel is five finders reading a diff, deduped, adversarially verified, mapped to a verdict. An issue has no diff. None of that machinery transfers — triage is a different recipe with a different shape.

What should transfer is the discipline, which is hard-won and repo-specific:

  • identity and permission resolved server-side, never from the webhook payload
  • typed drops, every decision telemetered (including declining — see pr-reviewer-plugin#34)
  • fail-closed on unreadable state
  • the chokepoint, so a burst of issue edits doesn't spawn N runs

summon.is_admin() is reusable as-is.

The stakes argument (this shapes v1)

A wrong label is reversible and costs seconds. A wrong FAIL blocks a merge. So triage can afford to be more liberal than review.

But the inverse matters more: auto-close, auto-assign, and auto-transfer should be out of scope for v1. Those are the actions that hurt when a model is confidently wrong, and 2026-07-22/23 produced three separate cases of exactly that — a hallucinated blocker confirmed twice, a fabricated prefix claim, and a real finding downgraded by an over-eager filter. Label + comment are recoverable; closing someone's issue is not.

The genuinely valuable question is dedupe

"Has this been filed before?" is the highest-value triage question and the one humans are worst at. Concretely: projectBoard-plugin#88's eight-round loop and the later PRs rhymed, and nobody noticed until it was analysed by hand.

The fleet already has the infrastructure — qwen3-embedding is on the gateway — so this is plausible rather than speculative.

What the spike should answer

  • Does this belong to Vera or to a different seat? She is the QA reviewer; triage may fit projectBoard or a new agent better. Cheapest wrong answer to correct now.
  • What does v1 do? Proposed: comment with suggested labels + likely duplicates, apply labels only if that proves accurate. Explicitly not: close, assign, transfer.
  • What is the trigger? issues.opened by an admin only? Also edited? Does a @vera triage summon make more sense than automatic (reuses pr-reviewer-plugin#28's surface and is opt-in per issue)?
  • How is dedupe done? Embedding search over open issues, title/body similarity, or a model pass over recent issues. What is the false-positive cost of "this looks like a duplicate of #X" when it isn't?
  • Where does it live? A new plugin, or the pr-reviewer plugin growing an issue surface? The name argues against the latter; the shared discipline argues for it.
  • How is it evaluated? The PR reviewer's precision is measurable because findings can be grounded against blobs. What is the equivalent for a label suggestion — and if there isn't one, is that a reason not to build it?

Refs

pr-reviewer-plugin#28 (the summon surface and its App-event requirement), pr-reviewer-plugin#34 (guard-decision telemetry — the observability posture any new surface should inherit), protoLab#24 (the hand-grounded findings dataset, as a model for how a new surface would be evaluated).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions