Skip to content

agent-team: reviewer-agent should run the test command and fail-closed on skip, not just parse the PR body #53

Description

@verkyyi

Summary

Reviewer prompt today says "Re-run the tests yourself if feasible to confirm" and grades ## Test status from the PR body. In the verkyyi/agentfolio#105 review (run 24682194207), tests weren't re-run — the reviewer trusted the implementer's self-report. For single-contributor agent-team runs this works (the implementer is honest), but the system is only as trustworthy as its weakest loop. Trusting self-report undermines the whole point of a review gate.

Proposal

  • Reviewer agent detects the test command by reading package.json / Makefile / pyproject.toml / go.mod / existing CI files, in that order. If found, it must run it against the PR branch checkout.
  • If the repo has no detectable test command: mark ## Tests as ⚠ no test command detected — skipped by design and proceed.
  • If detection succeeds but the command errors (real failure, not infra): verdict is kickback regardless of other checks.
  • If detection succeeds but command is skipped without a detectable reason: verdict is block.

Acceptance

  • reviewer-agent.md prompt gates approve on a successful test run.
  • Reviewer ## Review comment includes the exact command and a ✅/❌ outcome.
  • catalog/agent-team/README.md documents the test-command detection order and how to override via .agent-team.yml (see sibling issue on per-repo config).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions