Repository navigation
[Feat] Add a reproducible public231 decision benchmark - #1
Draft
linear3735 wants to merge 13 commits into
Draft
linear3735 wants to merge 13 commits into
linear3735 wants to merge 13 commits into
Conversation
This was referenced Oct 8, 2026
The 2026-10-07/08 review passes approved two pull requests (ThinkFlowLab#31, ThinkFlowLab#33) that were CONFLICTING with main, and both approvals had to be withdrawn in public threads: an approval issued without a mergeability check is an invalid signal, and a green CI run pinned to an older commit proves nothing about the head under review. review-pr now verifies the merge gates before any approve or ready statement (the branch reports mergeable, the required checks ran green on the exact head commit, the diff comes from a freshly fetched default branch; fork workflow runs pending maintainer approval count as pending, not absent). It also reads for readability - names that say what they do, reviewable function length, the house idioms - and the old blanket ban on style comments narrows to formatter-settled formatting, so naming and length are judged against the repository's own conventions rather than personal preference. self-review asks authors for the same gates before requesting review and after every push, with the head SHA recorded in the report.
…ibutor-intent-20261011 [Docs] Gate contributions on explicit human intent
…l-merge-gates-and-readability [Docs] Review skills: merge-readiness gates and a readability focus
(Reapplies 7c1120b dropped in the branch rebuild: pin utf-8 on the remaining benchmark fixture reads/writes so mock/loopback tests behave identically under cp-1252 and C-locale checkouts.) Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[Feat] Add a reproducible public231 decision benchmark
Why
Compare decision services on the same public inputs, with quality and request latency reported together.
How
predictsaves raw requests and responses.aggregatechecks the evidence and calls the pinned JevBench scorer. Failures and missing requests remain in the 231-question denominator.What
benchmarks/.Core: 483 added lines. Tests and documentation are counted separately. Read
benchmarks/legs/leg1_public231.pyfirst, thenclient.pyandevidence.py.Verification
tests/benchmarks/test_public231.py: 82 passed with the pinned JevBench source.The final benchmark suite also passed 193/193 tests with Torch on CPU.
All five real configurations completed 693/693 measured requests. Independent
raw-response scoring and a replay through this code agree within 1e-12.
Format, lint, types, build, lock and smoke checks pass on the integrated tree;
the full suite has 838 passed and 46 local-environment skips.
Diff: 483 core lines, 569 test lines and 178 documentation/changelog lines.
Demo / evidence
Measured comparison: five models on public231
One H800, one request at a time, five warmups and three measured rounds per model.
Accuracy uses 231 unique questions; latency pools the 693 measured requests.
All five arms completed 693/693 requests without failed or invalid responses.
Intern used official HF at temperature 1; Open-Jev used Omni native Rust/CUDA.
This compares those service configurations on public231.
The prepared recorded replay follows three ticket-routing decisions through System1-Agents and System1-Omni's Open-Jev 9B worker. All three routes match the fixture; there are no chat-model calls. Reading pauses are added; the clip is not a latency measurement.
The benchmark uses 231 public questions, five warmups and three measured rounds per model. Intern runs the official HF service; Open-Jev runs native Rust/CUDA. These are deployed-configuration comparisons, not isolated kernel speedups or a reproduction of the article's 248-question leaderboard.
The recorded application uses Agents
9790b52, Omni4a79980, native Rust/CUDAand Open-Jev 9B checkpoint
47e9668. Video, trace and output/timing evidenceare prepared as attachments outside Git.
The public copy omits input text and response legends; complete raw responses stay
with the original run and were used for the independent replay.