Define the ideal task and tighten the site copy - #15
Conversation
State what an ideal task is, in the three terms that actually decide it: work the contributor did themselves, weeks of their own effort, and a result a script can grade. Drop the 100-steps/80-tool-calls proxy, which measured the agent rather than the task and had drifted into the quality rubric. Add the contributor bar — a PhD or PhD candidate in physics, EECS, or an adjacent field, or extensive lab or equivalent industry experience. Restructure submission around the three deliverables: a PR from a fork, a description carrying the task's history (what the work was, start and end dates, hours spent), and a local test results report. The bench commands move below those as a pre-flight note instead of interrupting them, and the PR template gains a Task history table so the dates and hours are actually collected rather than merely requested. Cut roughly a third of the site's prose. Gone: the "small and early / no leaderboard yet" paragraph, the two-week banner that repeated its own cards, the credentialism justification, and both "No AI background required" lines. Every section now carries one short declarative line under its heading. Also align the criteria rule in every card so the divider sits at a fixed height whether or not the question wraps, and link PR #2 from the test-report step as a worked example.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 94fa2e5e63
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| 2. Complex | ||
| Require substantial domain expertise and effort. An agent without mentoring skills have to use at least 80 tool calls and over 100 steps, and is very likely to fail the task. | ||
| # What makes an ideal task |
There was a problem hiding this comment.
Update the existing [What makes a good task](#what-makes-a-good-task) link near the top of this document when renaming this heading. GitHub now generates #what-makes-an-ideal-task, so clicking the authorship-policy link no longer navigates to the quality criteria.
Useful? React with 👍 / 👎.
| 4. **Submit**: Fork this repository and open a PR against `main` here. See | ||
| [PR Requirements](#pr-requirements). |
There was a problem hiding this comment.
Replace #pr-requirements with the fragment generated by the new heading, #the-final-submission. In the contributor workflow, clicking “PR Requirements” currently targets a heading removed by this change and therefore leaves readers at a nonexistent anchor.
Useful? React with 👍 / 👎.
1 participant
Makes the contribution bar explicit, and cuts the site's prose back to what earns its place.
What an ideal task is
Stated in the three terms that actually decide it, on both the site and in
CONTRIBUTING.md:Plus a contributor bar: a PhD or PhD candidate in physics, EECS, or an adjacent field, or extensive lab or equivalent industry experience.
Removed the "100+ steps and 80+ tool calls" proxy. It measured the agent rather than the task, and it had drifted into the quality rubric as a merge criterion. The rubric now reads "took the author two weeks or more, and agents without skills are likely to fail it".
Submission, restructured
The doc and the page were organised around how to submit plus what the description must contain, which buried the actual structure. Both now read as three deliverables:
main, one task per PRThe
benchcommands moved below those as a pre-flight note.PULL_REQUEST_TEMPLATE.mdgains a Task history table, so dates and hours are collected rather than merely asked for.PR #2 is now linked from the test-report step as a worked example.
Copy audit
Cut roughly a third of the prose. Contribute page 4327 → ~3400px, home 2869 → 2745px. Removed:
Section heading renamed to Example tasks. Every section now carries one short declarative line under its heading.
One layout fix
The criteria cards pin their check question to the card bottom, so the two-line question raised its own divider by exactly one line height while the other two sat level. Measured before: 1056 / 1036 / 1056. Two lines are now reserved in every card, so the rule sits at a fixed offset regardless of wrapping — 1037 / 1036 / 1037, the 1px being sub-pixel line-box rounding. Used
min-hrather than a fixed height so a future three-line question grows the card instead of overflowing it.Verification
eslintand production build; all four routes prerender static.