Repository navigation
Add comp-lab v0.1: template, skills, agents and hooks - #1
Merged
Merged
Conversation
A harness each new competition copies: comp.toml describes the competition, and the comp command scores every experiment twice, on cross-validation and on a shadow holdout built to look like the test. noise measures how much the score moves by the random seed, compare keeps a change only if it beats twice that on both checks, and submit uploads only files that are logged with their commit. The defaults fit tabular data; data.py, cv.py, metric.py and shadow.py are meant to be rewritten for other shapes.
In repos with a comp.toml, every comp submit asks the person through Claude Code's own permission prompt, so an upload can't happen on an agent's word alone. Direct platform uploads and uploads from subagents are denied, and so is running a formatter over the whole folder, which once rewrote a document that wasn't ours.
Six skills cover the stages of a competition: comp-start builds and proves the validation first, experiment and research run ideas, submit plans scarce uploads, postmortem checks the leaderboard against our own checks, and explain writes optional learning notes. The researcher agent works in its own worktree and reports a fixed table; the leak auditor reads code and changes nothing. The seed lessons come from the Madinah competition.
The repo is its own marketplace, so it installs with /plugin marketplace add A7med7x7/comp-lab. CI lints and tests the template and the hook, and checks that the JSON manifests parse.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
This adds the first version of the bench plugin, extracted from the Madinah competition repo.
template/is the project harness each competition copies.comp runscores an experiment on cross-validation and on a shadow holdout built to look like the test.comp noisemeasures how much the score moves with the random seed.comp comparesays keep only if a change is better on both checks by more than twice that noise.comp submituploads only files logged insubmissions.mdwith their commit. Kaggle uploads go through its CLI. Zindi uploads are manual for now.hooks/guard.pyruns in repos that have acomp.toml. Everycomp submitasks the person in Claude Code's permission prompt. It denies direct platform uploads, uploads from subagents, and formatter runs over the whole folder.skills/covers the stages of a competition: comp-start, experiment, research, submit, postmortem and explain.agents/has a researcher that works in a worktree and a read-only leak auditor.lessons/seed.mdstarts the lessons shared across competitions, with what we learned in Madinah.Why
In Madinah, the method decided the result more than any single model did. Validation came first, and every result was checked twice against the noise. Research agents explored ideas and the main agent kept only the winners. This plugin packages that method so the next competition starts with it.
How it was tested
--submit, the manual Zindi upload steps, and recording a score.claude plugin validate .passes. Its warnings are for a missing licence and icon, which matter only for the public plugin directory.Left to do by hand
/plugin marketplace add A7med7x7/benchand then/plugin install bench@bench.isolation: worktree.