docs(benchmarking): Add Sonnet 5 Pi benchmark#418
Merged
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 54ba1bf. Configure here.
dcramer
force-pushed
the
benchmark/sonnet-5-pi
branch
from
54ba1bf to
dd670d1
Compare
dcramer
force-pushed
the
benchmark/sonnet-5-pi
branch
from
dd670d1 to
3ef75b3
Compare
Record the no-skip Sonnet 5 high-effort Pi benchmark against the Sentry corpus and update the comparison docs with the final numbers. Update the benchmark runbook so corpus runs use Warden workspace Pi dependencies, strict skipped-target validation, and explicit scan limits until local file-target runs stop sharing PR scan budgets. Co-Authored-By: GPT-5 Codex <noreply@anthropic.com>
dcramer
force-pushed
the
benchmark/sonnet-5-pi
branch
from
3ef75b3 to
601f705
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the Claude Sonnet 5 high-effort Pi benchmark result for the Sentry corpus and update the benchmark overview with the final comparison against Sonnet 4.6.
Benchmark Result
The committed result scans all 79 corpus target files as 156 traced chunks, with zero failed chunks. The docs now state the final Sonnet 5 result: 22/86 known findings, 27 total findings, and $23.46 total recorded cost.
Runbook Guardrails
The benchmark runbook now documents that Warden uses workspace Pi dependencies, includes the Sonnet 5 direct and OpenRouter selectors, raises corpus scan limits for complete target-file coverage, and validates the shard output. It also links the follow-up issue for making explicit local file-target runs stop sharing the PR-wide changed-line budget.
Pi Support
Update Warden's Pi dependencies and compatibility imports so the workspace Pi registry exposes Claude Sonnet 5, and add Sonnet 5 pricing for usage accounting.
Refs #419