Skip to content

Add comp-lab v0.1: template, skills, agents and hooks - #1

Merged
A7med7x7 merged 5 commits into
mainfrom
feature/v0.1
Nov 11, 2026
Merged

A7med7x7 merged 5 commits into
mainfrom
feature/v0.1

Conversation

@A7med7x7

@A7med7x7 A7med7x7 commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

What changed

This adds the first version of the bench plugin, extracted from the Madinah competition repo.

  • template/ is the project harness each competition copies. comp run scores an experiment on cross-validation and on a shadow holdout built to look like the test. comp noise measures how much the score moves with the random seed. comp compare says keep only if a change is better on both checks by more than twice that noise. comp submit uploads only files logged in submissions.md with their commit. Kaggle uploads go through its CLI. Zindi uploads are manual for now.
  • hooks/guard.py runs in repos that have a comp.toml. Every comp submit asks the person in Claude Code's permission prompt. It denies direct platform uploads, uploads from subagents, and formatter runs over the whole folder.
  • skills/ covers the stages of a competition: comp-start, experiment, research, submit, postmortem and explain. agents/ has a researcher that works in a worktree and a read-only leak auditor.
  • lessons/seed.md starts the lessons shared across competitions, with what we learned in Madinah.

Why

In Madinah, the method decided the result more than any single model did. Validation came first, and every result was checked twice against the noise. Research agents explored ideas and the main agent kept only the winners. This plugin packages that method so the next competition starts with it.

How it was tested

  • The template has 7 tests on a fake tabular competition: CV ranks a real model above the mean, folds never show held-out targets, the shadow holdout picks test-like rows, and the submission log builds, writes and maps files.
  • The hook has 14 tests, including the JSON it prints.
  • I ran an end-to-end check on a scratch copy: check, run, noise, compare, run --submit, the manual Zindi upload steps, and recording a score.
  • claude plugin validate . passes. Its warnings are for a missing licence and icon, which matter only for the public plugin directory.

Left to do by hand

  • Install it with /plugin marketplace add A7med7x7/bench and then /plugin install bench@bench.
  • Check that the plugin's researcher agent really runs in a worktree. The docs don't say whether plugin agents honour isolation: worktree.

A harness each new competition copies: comp.toml describes the
competition, and the comp command scores every experiment twice, on
cross-validation and on a shadow holdout built to look like the test.
noise measures how much the score moves by the random seed, compare
keeps a change only if it beats twice that on both checks, and submit
uploads only files that are logged with their commit.

The defaults fit tabular data; data.py, cv.py, metric.py and shadow.py
are meant to be rewritten for other shapes.
In repos with a comp.toml, every comp submit asks the person through
Claude Code's own permission prompt, so an upload can't happen on an
agent's word alone. Direct platform uploads and uploads from subagents
are denied, and so is running a formatter over the whole folder, which
once rewrote a document that wasn't ours.
Six skills cover the stages of a competition: comp-start builds and
proves the validation first, experiment and research run ideas,
submit plans scarce uploads, postmortem checks the leaderboard against
our own checks, and explain writes optional learning notes.

The researcher agent works in its own worktree and reports a fixed
table; the leak auditor reads code and changes nothing. The seed
lessons come from the Madinah competition.
The repo is its own marketplace, so it installs with /plugin
marketplace add A7med7x7/comp-lab. CI lints and tests the template and
the hook, and checks that the JSON manifests parse.
@A7med7x7
A7med7x7 merged commit a3d6b72 into main Nov 11, 2026
1 check passed
@A7med7x7 A7med7x7 changed the title Add comp-lab v0.1: template, skills, agents and hooks Add bench v0.1: template, skills, agents, hooks and theme Nov 11, 2026
@A7med7x7 A7med7x7 changed the title Add bench v0.1: template, skills, agents, hooks and theme Add comp-lab v0.1: template, skills, agents and hooks Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant