Replace ScaffBench with Fixproof, a sealed benchmark graded by hidden tests - #421
Replace ScaffBench with Fixproof, a sealed benchmark graded by hidden tests#421Marve10s wants to merge 10 commits into
Conversation
|
Too many files changed for review (263 files, 100 file limit). Bypass the limit by tagging |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Important Review skippedToo many files! This PR contains 262 files, which is 112 over the limit of 150. To get a review, reduce the PR to 150 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (262)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
vouch:trusted
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3401cd0e63
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… MCP speed claim, keep old benchmark redirects
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 24c794c281
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 40b833a60d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… stats grid, call to action, and changelog wording
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a4a034b13c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 65ed6ec374
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e10efd208d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c41fd5fcd1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Both models now have all ten tasks graded, so the board carries two rows instead of one partial row. Luna resolves 4 of 10 at a 9.5 minute median, Gemini 2 of 10 at 27.5 minutes. Add the solved-with-regressions outcome: on task 08 Luna's patch passes the hidden oracle but the task's own regression suite goes red, so it must not count as resolved. Rows are generated from the harness scorer, not by hand.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b8502e6ec
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "permanent": true | ||
| }, | ||
| { | ||
| "source": "/run", |
There was a problem hiding this comment.
For users who bookmarked a localized route such as /de/run or /fr/run, this exact /run rule does not match, and the removed TanStack route consequently returns a 404 after Paraglide delocalizes the request. Although this revision adds the base-language redirect, the fresh evidence is that the generated URL patterns expose eight non-English prefixes; make the compatibility redirect locale-aware so those previously valid URLs reach the localized benchmark page too. docs/guidelines/public-docs-i18n-and-seo.mdL77-L79
Useful? React with 👍 / 👎.
1 participant
Problem
ScaffBench stopped telling us anything. Its board had 13 specs at one trial each, so a single flipped spec moved the index by about 8 points, and the last published cohort sat inside one such band: the smallest GPT tier tied or beat the frontier models. The tasks measured toolchain recall rather than reasoning, agents could iterate on the build until it went green, and a full run cost a day of serial machine time for one noisy number. The harness, its reports, and the page kept growing around a benchmark we no longer trust.
Solution
This retires ScaffBench completely and replaces it with Fixproof, a benchmark built the way the repo-issue benchmarks are: real issues mined from private production codebases, each one a repository at a base commit plus a short symptom statement, graded by hidden tests that were proven red at base and green with the maintainers' fix. The agent gets the repository with history hidden and design notes stripped, a 30-minute cap, and its own test edits reverted before grading.
Everything task-related is anonymous and sealed. The task statements, the hidden oracles, the requirement maps, the harness, and the raw run artefacts live in a separate private repository and never enter this one. Publicly a task is only an id, a category, a difficulty tier, and whether it came from a private or the public Better-Fullstack repo. That is deliberate: publishing tasks puts them into training data and invites tuning against the tests.
The new
/benchmarkpage shows what we actually have from the first dry run rather than a projection. Two indexes are reported, both difficulty-weighted: Resolved, which counts a task only when every hidden check passes with no regression, and Progress, which credits the share of a task's requirements that were failing at base and pass after the patch, so near misses are visible without letting check counts inflate the score. The page also carries the run conditions in plain words, because these numbers come from real unattended agent runs on a dedicated Linux bench machine, not from anyone's estimate: one model, one trial per task, ten tasks, with three not yet run and one voided by a provider quota error and excluded.The cleanup removes the ScaffBench harness, its published reports, the
/runpage, the home teaser, the CI solvability job, the retired i18n keys and generated message modules, and every doc and script reference. The rule that benchmark results never count as product verification evidence stays, reworded for Fixproof, and its validator still enforces it.What this PR does not do: it does not publish any task, oracle, or private-codebase detail, and it does not claim the index is stable. A single trial on one model is a dry run of the pipeline. More trials, more models, and the alternative-fix check for each oracle come next.