Arrange the hero atoms on a sphere and tighten the contribution rules - #16
Conversation
Replace the drifting lattice with a geodesic sphere: an icosahedron subdivided three times and pushed onto the unit sphere, 642 atoms bonded along the subdivided triangle edges. Every vertex is equivalent to every other, so the mesh stays regular from any angle where a lat/long grid would bunch at the poles. Depth is carried by perspective scale and opacity alone. The cursor well returns, now applied to the projected points, so the bonds bend with the atoms and the mesh dimples rather than only the dots moving. Cut the top bar to Tasks, Contribute and Docs, with Docs pointing at the repository's own docs directory — read there rather than mirrored here, so there is one copy and it cannot go stale. State the contribution rules the benchmark actually enforces. A detailed PR description now reports three things against explicit minimums: a two-week project span, 40 working hours, and 10 hours for a first-year PhD to reproduce the results. That last one is a backward estimate rather than a measurement, and it is the closest single proxy for whether a task is substantial enough to grade. The test-results report spells out its three checks: the oracle at reward 1.0, and a state-of-the-art agent with and without skills. Add the two rules contributors could not previously discover: reviewing opens up once your first task merges, and only tasks merged by 20 August 2026 score. Because merging takes days of review, both the guide and the site now push contributors to open a draft PR early rather than polish in private. Credit SkillsBench as prior work by the same team, linked to its paper. That required narrowing the documentation linter, which banned the bare product name outright: it now rejects only the copied branding forms — the domain and the repository path — so earned attribution is possible while stale branding still fails. Verified in both directions. Also fix two anchors in CONTRIBUTING.md left dangling by earlier section renames, and drop the closing call-to-action cards from both pages.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Require a strong agent in the testing summary
When contributors follow this four-step summary, “at least one agent” permits trials with a weak or outdated agent, even though the submission checks later on this page require a state-of-the-art agent and the repository review rule requires a strong current agent. Make the summary carry the same qualification so contributors do not discover after costly runs that their evidence is ineligible.
AGENTS.md reference: AGENTS.md:L51-L53
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Open the PR as a **draft** as soon as you have the task idea and a skeleton — | ||
| do not wait until it is polished. Reviewing and revising a task takes days of | ||
| back-and-forth, so iterating with a maintainer in a draft is both faster than | ||
| guessing and the only reliable way to merge before the deadline. |
There was a problem hiding this comment.
This newly tells contributors to open a draft as soon as they have a skeleton, but the same guide still says at lines 190–198 that they must pass the structural and oracle checks, finish multiple agent trials, and have every task artifact ready before opening the PR. A contributor cannot follow both sequences; the later prerequisite should apply before marking the draft ready for review rather than before opening it.
Useful? React with 👍 / 👎.
1 participant
Two threads: the hero background becomes a sphere, and the contribution rules say what the benchmark actually enforces.
Background
The atoms now sit on a geodesic sphere — an icosahedron subdivided three times and pushed onto the unit sphere, giving 642 atoms bonded along the subdivided triangle edges. Every vertex is equivalent to every other, so the mesh stays regular from any angle; a lat/long grid would visibly bunch at the poles. Depth comes from perspective scale and opacity alone, so it reads as solid without shading.
The cursor well is back, and now applies to the projected points — the bonds bend along with the atoms, so the mesh dimples rather than only the dots moving.
Nothing else about the hero changed: same atom glyph, same theme-derived colour, same vignette, same
prefers-reduced-motionstatic frame.Navigation
Cut to Tasks · Contribute · Docs. Docs points at
/tree/main/docsand opens in a new tab — read from the repository rather than mirrored into the site, so there is exactly one copy and it cannot go stale. Handled in both the desktop and mobile menus.Contribution rules
A detailed PR description now reports three things against explicit minimums:
The third is a backward estimate, not a measurement — assume a capable first-year PhD already handed the prompt and data. It is the closest single proxy for whether a task is substantial enough to grade, and the guide and PR template both say so, since otherwise it reads as a duplicate of row two.
A local test results report now spells out its three checks: the oracle at reward 1.0, and a state-of-the-art agent with and without skills.
Two rules contributors previously had no way to discover:
Since merging takes days of review, both the guide and the site now push contributors to open a draft PR early rather than polish in private.
Prior work
The contribute page and footer credit SkillsBench as prior work by the same team, linked to its paper, phrased as "passed 100 citations within three months of release" — time-bounded so it stays true rather than silently ageing.
That required narrowing the documentation linter. It banned the bare product name outright, which would have failed CI the moment this attribution reached a
.md. It now rejects only the copied-branding forms —skillsbench.aiandbenchflow/skillsbench— so earned attribution is possible while stale branding still fails. Verified both directions: still errors on the domain, passes on "From the team behind SkillsBench."Alignment and fixes
CONTRIBUTING.mdwas checked claim-by-claim against the site — points, deadline, reviewer rule, all three minimums, oracle reward, SOTA wording, eligibility — and two things were wrong:#what-makes-a-good-taskand#pr-requirementsboth pointed at headings that no longer exist.Also removed the closing call-to-action cards from both pages, the
benchcommand block, and the "Read the contributor guide" button.Verification
eslintand production build; all routes prerender static.prefers-reduced-motionpaints a static frame rather than going blank.ruffon the changed linter.CONTRIBUTING.mdre-validated against its headings — none broken.