Skip to content

Use compact default agent guidance with evaluation evidence - #22

Merged
Raithlin merged 2 commits into
masterfrom
feature/compact-agent-guidance
Sep 28, 2026
Merged

Raithlin merged 2 commits into
masterfrom
feature/compact-agent-guidance

Conversation

@Raithlin

Copy link
Copy Markdown
Owner

Summary

  • Make the always-present GHA guidance compact while keeping the detailed skill installed for explicit use.
  • Add reproducible agent evaluations and sanitized results to show quality, token use, and command effort.

Changes

  • Replace the mandatory skill-loading instruction with JSON capability discovery, uncertainty handling, and dry-run guidance.
  • Document the git/gh control: both arms answered 14/14; short-guide GHA used 30% fewer tokens on these GHA-derived tasks. PR inspection favored git/gh.
  • Add a held-out local pilot: compact guidance with the skill installed answered 4/4 branch questions with roughly the short-guide token cost. It used GHA on 4/4 separate PR/release checks. A third local task had a bad expected clean-tree answer because the harness added AGENTS.md; its strict failures are retained and excluded from the pilot comparison.

Validation

  • GOCACHE=/tmp/gha-go-build GOLANGCI_LINT_CACHE=/tmp/gha-lint-cache make check — lint, race tests, and 95.1% coverage passed.
  • make test-eval — 30 tests passed.
  • Disposable gha agent install/uninstall smoke test — compact block and detailed skill installed and removed correctly.

Checklist

  • Tests cover the changed behavior.
  • README, evaluation method, and roadmap updated.
  • Structured output contracts remain unchanged.
  • Existing write previews and confirmations remain unchanged.

@Raithlin
Raithlin merged commit 5097fbb into master Sep 28, 2026
2 checks passed
@Raithlin
Raithlin deleted the feature/compact-agent-guidance branch September 28, 2026 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant