diff --git a/.claude-plugin/icon.png b/.claude-plugin/icon.png new file mode 100644 index 0000000..a56d465 Binary files /dev/null and b/.claude-plugin/icon.png differ diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 339a0fa..d27ba7d 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -1,9 +1,9 @@ { - "name": "comp-lab", + "name": "bench", "owner": { "name": "Ahmed Alghali" }, "plugins": [ { - "name": "comp-lab", + "name": "bench", "source": "./", "description": "Run machine learning competitions with Claude: validation first, research agents, tracked results, approved uploads." } diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 12f30e5..7d8d010 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,8 +1,8 @@ { - "name": "comp-lab", + "name": "bench", "version": "0.1.0", "description": "Run machine learning competitions (Kaggle, Zindi) with Claude: validation first, a main agent plus research agents, every result tracked, uploads only with your approval.", "author": { "name": "Ahmed Alghali" }, - "repository": "https://github.com/A7med7x7/comp-lab", + "repository": "https://github.com/A7med7x7/bench", "keywords": ["kaggle", "zindi", "machine-learning", "competition", "mlflow"] } diff --git a/README.md b/README.md index 0c112ed..f539d17 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,20 @@ -# comp-lab +

+ + + bench + +

-[![CI](https://github.com/A7med7x7/comp-lab/actions/workflows/ci.yml/badge.svg)](https://github.com/A7med7x7/comp-lab/actions/workflows/ci.yml) +

Validation first. Every result measured twice.

-A Claude Code plugin for machine learning competitions on Kaggle and Zindi. It has one goal: the best final score. It does not try to get there by guessing faster. It makes the validation trustworthy first, then lets a main agent and research agents run many experiments that are all measured the same way. +

CI

+ +bench is a Claude Code plugin for machine learning competitions on Kaggle and Zindi. It has one goal: the best final score. It does not try to get there by guessing faster. It makes the validation trustworthy first, then lets a main agent and research agents run many experiments that are all measured the same way. It grew out of the Madinah crowd-count competition (Kaggle, 2026). There, one model scored 0.821 in cross-validation and 1.022 on the leaderboard. A second check built to look like the test caught the problem, and a research agent then found three improvements in one evening. This plugin packages that way of working. +![The bench loop: comp-start proves the validation, experiment tries one idea on two checks, research sends agents to try many, submit uploads only with your approval, postmortem checks whether the leaderboard agrees, and lessons feed the next experiment. A change is kept only if CV and the shadow test both beat twice the noise.](assets/brand/loop.svg) + ## What is in it | part | what it does | @@ -21,14 +30,15 @@ It grew out of the Madinah crowd-count competition (Kaggle, 2026). There, one mo | `agents/leak-auditor` | reads an experiment and reports any information leak | | `hooks/guard.py` | in competition repos: asks you before every upload, blocks uploads that skip the log, blocks whole-folder formatting | | `lessons/seed.md` | lessons across competitions, read at the start of each new one | +| `assets/brand/` | the bench logo kit (symbol, wordmark, lockups, icons), the workflow diagram, `notes.css` for themed learning notes, and `GUIDELINES.md` | ## Install The repo is private, so your git must be able to read it (`gh auth login` and `gh auth setup-git`, or an SSH key). ```text -/plugin marketplace add A7med7x7/comp-lab -/plugin install comp-lab@comp-lab +/plugin marketplace add A7med7x7/bench +/plugin install bench@bench ``` > [!NOTE] @@ -39,16 +49,16 @@ The repo is private, so your git must be able to read it (`gh auth login` and `g Start a competition by telling Claude, or run the skill directly: ```text -/comp-lab:comp-start https://www.kaggle.com/competitions/ +/bench:comp-start https://www.kaggle.com/competitions/ ``` Then work in that repo: ```text -/comp-lab:experiment try adding lag features to 003_baseline -/comp-lab:research test the ideas in this playbook against our best model -/comp-lab:submit upload the best file when the limit resets -/comp-lab:postmortem the score came back: 0.912 +/bench:experiment try adding lag features to 003_baseline +/bench:research test the ideas in this playbook against our best model +/bench:submit upload the best file when the limit resets +/bench:postmortem the score came back: 0.912 ``` Every experiment is scored twice. `uv run comp run` gives the CV score and the shadow score, and `uv run comp compare` keeps a change only if it is better on both by more than twice the noise that `uv run comp noise` measured. The leaderboard is a third check and never decides alone. @@ -59,7 +69,7 @@ Every experiment is scored twice. `uv run comp run` gives the CV score and the s - Claude is the orchestrator. The skills say what to do at each stage, the agents have fixed roles and limits, and the hook enforces the two rules that must never depend on memory: uploads need your approval, and nothing formats the whole folder. - The template is copied into each competition and then belongs to it. Change `data.py`, `cv.py`, `metric.py` and `shadow.py` freely for a competition with a different shape; keep their meaning. -- Lessons live in `~/.comp-lab/lessons.md` on your machine, seeded from `lessons/seed.md`. Add the general ones back to `lessons/seed.md` here so every new competition starts from them. +- Lessons live in `~/.bench/lessons.md` on your machine, seeded from `lessons/seed.md`. Add the general ones back to `lessons/seed.md` here so every new competition starts from them. ## Develop diff --git a/agents/leak-auditor.md b/agents/leak-auditor.md index 388d25a..3df577d 100644 --- a/agents/leak-auditor.md +++ b/agents/leak-auditor.md @@ -1,6 +1,6 @@ --- name: leak-auditor -description: Reads an experiment's code in a comp-lab competition repo and reports any way it could see information the real test would not have (target leakage, future values, held-out rows, test labels). Read-only. Use before submitting a new method and on every research agent's winning code. +description: Reads an experiment's code in a bench competition repo and reports any way it could see information the real test would not have (target leakage, future values, held-out rows, test labels). Read-only. Use before submitting a new method and on every research agent's winning code. tools: Read, Grep, Glob, Bash --- diff --git a/agents/researcher.md b/agents/researcher.md index e06acc8..cceb8ad 100644 --- a/agents/researcher.md +++ b/agents/researcher.md @@ -1,6 +1,6 @@ --- name: researcher -description: Tests many variants of one ML direction in a comp-lab competition repo, in its own git worktree, and returns a standard report table with CV and shadow scores against the baseline. Use with a filled brief from the research skill. +description: Tests many variants of one ML direction in a bench competition repo, in its own git worktree, and returns a standard report table with CV and shadow scores against the baseline. Use with a filled brief from the research skill. isolation: worktree --- diff --git a/assets/brand/GUIDELINES.md b/assets/brand/GUIDELINES.md new file mode 100644 index 0000000..4a6774b --- /dev/null +++ b/assets/brand/GUIDELINES.md @@ -0,0 +1,53 @@ +# bench logo guidelines + +## 1. The logo + +- Idea: a bench seen from the side is also the letter b. The back post is the stem; the seat, front leg and stretcher close the bowl. The mark is the first letter of the name, so the icon and the wordmark are one idea. +- Versions: + - horizontal (`bench-horizontal.svg`): the wordmark with a green b. The primary version. + - stacked (`bench-stacked.svg`): the symbol above the wordmark. + - symbol (`bench-symbol.svg`) for 64 px and up. + - small symbol (`bench-symbol-small.svg`) for 16 to 48 px. + - wordmark (`bench-wordmark.svg`): all ink. +- Files: SVG for screens and print. `bench-horizontal-dark.svg` keeps the green b for dark backgrounds. The web icons are in `icons/`, the presentation slides in `board/`, the workflow diagram is `loop.svg`, and `notes.css` themes learning notes. + +## 2. Clear space + +Keep a clear zone of one wordmark stroke width on all sides: the thickness of the b's stem, which is 28 of the wordmark's 256-unit height. It scales with the logo. + +## 3. Minimum size + +| Version | Screen | +| --- | --- | +| Horizontal | 96 px wide | +| Symbol | 64 px; below that use the small symbol, down to 16 px | + +## 4. Colour + +| Name | HEX | RGB | Use | +| --- | --- | --- | --- | +| Signal green | #0B7A4B | 11, 122, 75 | the b, app icon tile, "keep" | +| Ink | #15181C | 21, 24, 28 | wordmark letters, text | +| Ground | #F6F7F5 | 246, 247, 245 | light backgrounds | +| Muted | #5B636B | 91, 99, 107 | secondary text | +| Caution | #B45309 | 180, 83, 9 | uploads and warnings in diagrams, never in the logo | + +Green on white has a contrast of 5.4:1 and ink on white 17.8:1, so both pass for text. + +Approved pairs: green b and ink on white or Ground; all white on green; all white on Ink; all black on white. + +## 5. Typography + +The wordmark is drawn, not typed: one 28-unit stroke and the bench seat's rounded top-right corner on every letter. For text next to it, use Space Grotesk 700 for headings, Inter for body and JetBrains Mono for numbers and commands. All three are open licence (SIL OFL) from Google Fonts. + +## 6. Don'ts + +- Don't stretch, rotate, outline or add shadows or gradients. +- Don't recolour outside the palette, and don't make the b any colour but green or the one-colour version. +- Don't type "bench" in a font in place of the wordmark. +- Don't use the large symbol below 64 px; use the small one. + +## 7. Notes + +- Trademark: no search was done. Check before any public launch. +- Made with the logo-design skill (github.com/kaankiziltug/logo-design-skill): SVG audit 100/100 on every file, renders checked at 16 to 512 px. diff --git a/assets/brand/bench-horizontal-black.svg b/assets/brand/bench-horizontal-black.svg new file mode 100644 index 0000000..76ff07f --- /dev/null +++ b/assets/brand/bench-horizontal-black.svg @@ -0,0 +1,8 @@ +bench + bench + + + + + + diff --git a/assets/brand/bench-horizontal-dark.svg b/assets/brand/bench-horizontal-dark.svg new file mode 100644 index 0000000..5c83b43 --- /dev/null +++ b/assets/brand/bench-horizontal-dark.svg @@ -0,0 +1,8 @@ + + bench + + + + + + diff --git a/assets/brand/bench-horizontal-mono-0b7a4b.svg b/assets/brand/bench-horizontal-mono-0b7a4b.svg new file mode 100644 index 0000000..d9eef18 --- /dev/null +++ b/assets/brand/bench-horizontal-mono-0b7a4b.svg @@ -0,0 +1,8 @@ +bench + bench + + + + + + diff --git a/assets/brand/bench-horizontal-white.svg b/assets/brand/bench-horizontal-white.svg new file mode 100644 index 0000000..e92db5a --- /dev/null +++ b/assets/brand/bench-horizontal-white.svg @@ -0,0 +1,8 @@ +bench + bench + + + + + + diff --git a/assets/brand/bench-horizontal.svg b/assets/brand/bench-horizontal.svg new file mode 100644 index 0000000..a1f58be --- /dev/null +++ b/assets/brand/bench-horizontal.svg @@ -0,0 +1,8 @@ + + bench + + + + + + diff --git a/assets/brand/bench-stacked.svg b/assets/brand/bench-stacked.svg new file mode 100644 index 0000000..f7e695b --- /dev/null +++ b/assets/brand/bench-stacked.svg @@ -0,0 +1,11 @@ + + bench stacked lockup + + + + + + + + + diff --git a/assets/brand/bench-symbol-app-icon.svg b/assets/brand/bench-symbol-app-icon.svg new file mode 100644 index 0000000..38a1e20 --- /dev/null +++ b/assets/brand/bench-symbol-app-icon.svg @@ -0,0 +1,5 @@ +bench + bench symbol, small sizes + + + \ No newline at end of file diff --git a/assets/brand/bench-symbol-black.svg b/assets/brand/bench-symbol-black.svg new file mode 100644 index 0000000..d40499b --- /dev/null +++ b/assets/brand/bench-symbol-black.svg @@ -0,0 +1,5 @@ +bench + bench symbol + + + diff --git a/assets/brand/bench-symbol-favicon.svg b/assets/brand/bench-symbol-favicon.svg new file mode 100644 index 0000000..c564c1b --- /dev/null +++ b/assets/brand/bench-symbol-favicon.svg @@ -0,0 +1,5 @@ +bench + bench symbol, small sizes + + + \ No newline at end of file diff --git a/assets/brand/bench-symbol-mono-0b7a4b.svg b/assets/brand/bench-symbol-mono-0b7a4b.svg new file mode 100644 index 0000000..8d8509a --- /dev/null +++ b/assets/brand/bench-symbol-mono-0b7a4b.svg @@ -0,0 +1,5 @@ +bench + bench symbol + + + diff --git a/assets/brand/bench-symbol-small.svg b/assets/brand/bench-symbol-small.svg new file mode 100644 index 0000000..8776fcb --- /dev/null +++ b/assets/brand/bench-symbol-small.svg @@ -0,0 +1,5 @@ + + bench symbol, small sizes + + + diff --git a/assets/brand/bench-symbol-square.svg b/assets/brand/bench-symbol-square.svg new file mode 100644 index 0000000..285552e --- /dev/null +++ b/assets/brand/bench-symbol-square.svg @@ -0,0 +1,5 @@ +bench + bench symbol + + + \ No newline at end of file diff --git a/assets/brand/bench-symbol-white.svg b/assets/brand/bench-symbol-white.svg new file mode 100644 index 0000000..dd8386f --- /dev/null +++ b/assets/brand/bench-symbol-white.svg @@ -0,0 +1,5 @@ +bench + bench symbol + + + diff --git a/assets/brand/bench-symbol.svg b/assets/brand/bench-symbol.svg new file mode 100644 index 0000000..d3f9099 --- /dev/null +++ b/assets/brand/bench-symbol.svg @@ -0,0 +1,5 @@ + + bench symbol + + + diff --git a/assets/brand/bench-wordmark.svg b/assets/brand/bench-wordmark.svg new file mode 100644 index 0000000..f3cfa12 --- /dev/null +++ b/assets/brand/bench-wordmark.svg @@ -0,0 +1,8 @@ + + bench + + + + + + diff --git a/assets/brand/board/slide-01.png b/assets/brand/board/slide-01.png new file mode 100644 index 0000000..538a8b7 Binary files /dev/null and b/assets/brand/board/slide-01.png differ diff --git a/assets/brand/board/slide-02.png b/assets/brand/board/slide-02.png new file mode 100644 index 0000000..c59079b Binary files /dev/null and b/assets/brand/board/slide-02.png differ diff --git a/assets/brand/board/slide-03.png b/assets/brand/board/slide-03.png new file mode 100644 index 0000000..7c88f4e Binary files /dev/null and b/assets/brand/board/slide-03.png differ diff --git a/assets/brand/board/slide-04.png b/assets/brand/board/slide-04.png new file mode 100644 index 0000000..4b45c33 Binary files /dev/null and b/assets/brand/board/slide-04.png differ diff --git a/assets/brand/board/slide-05.png b/assets/brand/board/slide-05.png new file mode 100644 index 0000000..19c3760 Binary files /dev/null and b/assets/brand/board/slide-05.png differ diff --git a/assets/brand/favicon.svg b/assets/brand/favicon.svg new file mode 100644 index 0000000..c564c1b --- /dev/null +++ b/assets/brand/favicon.svg @@ -0,0 +1,5 @@ +bench + bench symbol, small sizes + + + \ No newline at end of file diff --git a/assets/brand/icons/apple-touch-icon.png b/assets/brand/icons/apple-touch-icon.png new file mode 100644 index 0000000..61a5808 Binary files /dev/null and b/assets/brand/icons/apple-touch-icon.png differ diff --git a/assets/brand/icons/favicon-16.png b/assets/brand/icons/favicon-16.png new file mode 100644 index 0000000..e008239 Binary files /dev/null and b/assets/brand/icons/favicon-16.png differ diff --git a/assets/brand/icons/favicon-32.png b/assets/brand/icons/favicon-32.png new file mode 100644 index 0000000..5d8a1b1 Binary files /dev/null and b/assets/brand/icons/favicon-32.png differ diff --git a/assets/brand/icons/favicon-48.png b/assets/brand/icons/favicon-48.png new file mode 100644 index 0000000..09a48a4 Binary files /dev/null and b/assets/brand/icons/favicon-48.png differ diff --git a/assets/brand/icons/favicon.ico b/assets/brand/icons/favicon.ico new file mode 100644 index 0000000..fd52fda Binary files /dev/null and b/assets/brand/icons/favicon.ico differ diff --git a/assets/brand/icons/head-snippet.html b/assets/brand/icons/head-snippet.html new file mode 100644 index 0000000..bc97ea6 --- /dev/null +++ b/assets/brand/icons/head-snippet.html @@ -0,0 +1,5 @@ + + + + + diff --git a/assets/brand/icons/icon-192.png b/assets/brand/icons/icon-192.png new file mode 100644 index 0000000..e02d510 Binary files /dev/null and b/assets/brand/icons/icon-192.png differ diff --git a/assets/brand/icons/icon-512.png b/assets/brand/icons/icon-512.png new file mode 100644 index 0000000..a56d465 Binary files /dev/null and b/assets/brand/icons/icon-512.png differ diff --git a/assets/brand/icons/maskable-512.png b/assets/brand/icons/maskable-512.png new file mode 100644 index 0000000..4c47b1e Binary files /dev/null and b/assets/brand/icons/maskable-512.png differ diff --git a/assets/brand/icons/site.webmanifest b/assets/brand/icons/site.webmanifest new file mode 100644 index 0000000..f24c1e1 --- /dev/null +++ b/assets/brand/icons/site.webmanifest @@ -0,0 +1,24 @@ +{ + "name": "bench", + "icons": [ + { + "src": "/icon-192.png", + "sizes": "192x192", + "type": "image/png" + }, + { + "src": "/icon-512.png", + "sizes": "512x512", + "type": "image/png" + }, + { + "src": "/maskable-512.png", + "sizes": "512x512", + "type": "image/png", + "purpose": "maskable" + } + ], + "theme_color": "#0B7A4B", + "background_color": "#0B7A4B", + "display": "standalone" +} \ No newline at end of file diff --git a/assets/brand/loop.svg b/assets/brand/loop.svg new file mode 100644 index 0000000..ce4c40b --- /dev/null +++ b/assets/brand/loop.svg @@ -0,0 +1,57 @@ + + The bench loop + comp-start proves the validation, experiment tries one idea on two checks, research sends agents to try many, submit uploads only with your approval, postmortem checks whether the leaderboard agrees, and its lessons feed the next experiment. A change is kept only if CV and the shadow test both beat twice the noise. + + + + + + + comp-start + prove the validation + + + + experiment + one idea, two checks + + + + research + agents try many + + + + submit + you approve each one + + + + postmortem + do the checks agree? + + + + + + + + + + + + + lessons feed the next experiment + + + + + KEEP + only if CV and the shadow test both beat 2 × noise + the leaderboard never decides alone + diff --git a/assets/brand/notes.css b/assets/brand/notes.css new file mode 100644 index 0000000..be5a231 --- /dev/null +++ b/assets/brand/notes.css @@ -0,0 +1,87 @@ +/* bench theme for learning notes (notes/*.html). Copy it to notes/bench.css and link it: + + Colours and fonts follow assets/brand/GUIDELINES.md. Works offline: the fonts fall back to system ones. */ + +:root { + --ground: #F6F7F5; + --surface: #FFFFFF; + --ink: #15181C; + --muted: #5B636B; + --line: #DCE0DC; + --keep: #0B7A4B; /* signal green: the b, "keep", good results */ + --keep-soft: #E7F3EC; + --caution: #B45309; /* uploads and warnings */ + --caution-soft: #FFF4EB; + --heading: "Space Grotesk", Inter, -apple-system, "Segoe UI", Helvetica, Arial, sans-serif; + --body: Inter, -apple-system, "Segoe UI", Helvetica, Arial, sans-serif; + --mono: "JetBrains Mono", ui-monospace, Menlo, Consolas, monospace; +} + +@media (prefers-color-scheme: dark) { + :root:not([data-theme="light"]) { + --ground: #101316; + --surface: #171B1F; + --ink: #E8ECEA; + --muted: #A3ACB4; + --line: #2A3036; + --keep: #3FBF83; + --keep-soft: #12281E; + --caution: #F0A35E; + --caution-soft: #2B1E12; + } +} + +:root[data-theme="dark"] { + --ground: #101316; + --surface: #171B1F; + --ink: #E8ECEA; + --muted: #A3ACB4; + --line: #2A3036; + --keep: #3FBF83; + --keep-soft: #12281E; + --caution: #F0A35E; + --caution-soft: #2B1E12; +} + +body { + margin: 0; + background: var(--ground); + color: var(--ink); + font: 17px/1.6 var(--body); +} + +main { max-width: 760px; margin: 0 auto; padding: 48px 16px 96px; } + +h1, h2, h3 { font-family: var(--heading); line-height: 1.2; letter-spacing: -0.01em; } +h1 { font-size: 2.2rem; margin: 0 0 0.4em; } +h2 { font-size: 1.5rem; margin: 2em 0 0.5em; } +h3 { font-size: 1.15rem; margin: 1.6em 0 0.4em; } + +a { color: var(--keep); } +a:hover { text-decoration-thickness: 2px; } + +code, pre, .num { font-family: var(--mono); font-size: 0.92em; } +pre { background: var(--surface); border: 1px solid var(--line); border-radius: 10px; padding: 14px 16px; overflow-x: auto; } + +table { border-collapse: collapse; width: 100%; margin: 1em 0; } +th, td { border-bottom: 1px solid var(--line); padding: 8px 10px; text-align: left; } +th { color: var(--muted); font-weight: 600; font-size: 0.9em; } +td.num, th.num { text-align: right; } + +/* The status box at the top of the where-we-stand page. */ +.status { background: var(--surface); border: 1px solid var(--line); border-radius: 14px; padding: 16px 20px; } + +/* Three layers of an explanation. */ +.layer { background: var(--surface); border: 1px solid var(--line); border-radius: 14px; padding: 4px 20px 12px; margin: 16px 0; } +.layer > h3::before { color: var(--muted); font-weight: 500; margin-right: 0.4em; } +.layer.idea > h3::before { content: "1 · idea"; } +.layer.example > h3::before { content: "2 · example"; } +.layer.formula > h3::before { content: "3 · formula"; } + +/* Verdict chips: KEEP / DROP. */ +.keep, .drop { display: inline-block; border-radius: 999px; padding: 2px 12px; font-weight: 600; font-size: 0.85em; } +.keep { background: var(--keep); color: var(--surface); } /* white on green in light, dark on green in dark: both pass 4.5:1 */ +.drop { background: var(--caution-soft); color: var(--caution); border: 1px solid var(--caution); } + +.glossary dt { font-weight: 600; margin-top: 12px; } +.glossary dd { margin: 2px 0 0 0; color: var(--muted); } diff --git a/hooks/hooks.json b/hooks/hooks.json index f85c978..458a0be 100644 --- a/hooks/hooks.json +++ b/hooks/hooks.json @@ -1,5 +1,5 @@ { - "description": "comp-lab guards: uploads need the person's approval, and no whole-folder formatting, in competition repos only", + "description": "bench guards: uploads need the person's approval, and no whole-folder formatting, in competition repos only", "hooks": { "PreToolUse": [ { diff --git a/lessons/seed.md b/lessons/seed.md index 32c3155..944540d 100644 --- a/lessons/seed.md +++ b/lessons/seed.md @@ -1,8 +1,8 @@ # Lessons across competitions Lessons about the method, grouped by problem type. Each one says where it was -learned. `/comp-lab:comp-start` reads this file before planning a new -competition, and `/comp-lab:postmortem` adds to it. +learned. `/bench:comp-start` reads this file before planning a new +competition, and `/bench:postmortem` adds to it. ## Any competition diff --git a/skills/comp-start/SKILL.md b/skills/comp-start/SKILL.md index 49e7856..76d4585 100644 --- a/skills/comp-start/SKILL.md +++ b/skills/comp-start/SKILL.md @@ -1,6 +1,6 @@ --- name: comp-start -description: Start a new machine learning competition (Kaggle or Zindi) from the comp-lab template - create the repo, fill comp.toml, and build and prove the validation before any model. Use when the person wants to begin or set up a competition or ML project. +description: Start a new machine learning competition (Kaggle or Zindi) from the bench template - create the repo, fill comp.toml, and build and prove the validation before any model. Use when the person wants to begin or set up a competition or ML project. --- # Start a competition @@ -19,7 +19,7 @@ Ask only for what you can't find yourself: the competition URL, and solo or team - daily upload limit, final pick count, deadline (with time zone) - rules on external data and pretrained models -Save the page text in `docs/competition.md`. Read the shared lessons, `~/.comp-lab/lessons.md` (create it from `${CLAUDE_PLUGIN_ROOT}/lessons/seed.md` if it is missing), and note which ones apply. +Save the page text in `docs/competition.md`. Read the shared lessons, `~/.bench/lessons.md` (create it from `${CLAUDE_PLUGIN_ROOT}/lessons/seed.md` if it is missing), and note which ones apply. ## 2. Create the repo diff --git a/skills/experiment/SKILL.md b/skills/experiment/SKILL.md index 32a8c17..bbcee4e 100644 --- a/skills/experiment/SKILL.md +++ b/skills/experiment/SKILL.md @@ -1,6 +1,6 @@ --- name: experiment -description: Run one experiment in a comp-lab competition repo - one idea, scored on CV and the shadow test against the noise, logged, kept or dropped. Use when trying an idea, a feature, a model or a setting to improve the score. +description: Run one experiment in a bench competition repo - one idea, scored on CV and the shadow test against the noise, logged, kept or dropped. Use when trying an idea, a feature, a model or a setting to improve the score. --- # One experiment diff --git a/skills/explain/SKILL.md b/skills/explain/SKILL.md index ce2f514..bb591a6 100644 --- a/skills/explain/SKILL.md +++ b/skills/explain/SKILL.md @@ -1,6 +1,6 @@ --- name: explain -description: Write learning notes in a comp-lab competition repo - explain a finding, technique or metric detail in three layers (plain idea, worked example, formula or code) in simple English, and keep a glossary. Use when notes are enabled in comp.toml and a finding changes the plan, a new technique is introduced, or the person asks why something happened. +description: Write learning notes in a bench competition repo - explain a finding, technique or metric detail in three layers (plain idea, worked example, formula or code) in simple English, and keep a glossary. Use when notes are enabled in comp.toml and a finding changes the plan, a new technique is introduced, or the person asks why something happened. --- # Explain a finding @@ -13,7 +13,7 @@ Use this only when `[notes] enabled = true` in `comp.toml`. The notes are for th - One page per topic: `notes/NN-short-topic.html`. Keep a status page (`notes/NN-where-we-stand.html`) with a status box at the top, a section per finding, and a log table at the bottom; update it after every upload, leaderboard score and finding that changes the plan. - `notes/glossary.html`: every special word, the first time it appears anywhere. -Plain HTML with one small inline stylesheet, readable offline. Diagrams as inline SVG when a picture says it faster. +Plain HTML, readable offline, in the bench theme: copy `${CLAUDE_PLUGIN_ROOT}/assets/brand/notes.css` to `notes/bench.css` once and link it from every page. It gives light and dark modes, `.status`, `.layer.idea` / `.layer.example` / `.layer.formula` boxes for the three layers, `.keep` / `.drop` chips, and `.glossary`. Diagrams as inline SVG when a picture says it faster, in the same colours: ink #15181C, green #0B7A4B for what is kept or good, amber #B45309 for uploads and warnings, muted #5B636B for labels. ## Three layers, always in this order diff --git a/skills/postmortem/SKILL.md b/skills/postmortem/SKILL.md index 41dce28..bd82c56 100644 --- a/skills/postmortem/SKILL.md +++ b/skills/postmortem/SKILL.md @@ -1,6 +1,6 @@ --- name: postmortem -description: After a leaderboard score comes back in a comp-lab repo, check whether CV, the shadow test and the leaderboard agree, find out why if they don't, and record lessons for this and future competitions. Use after every upload's score, and at the end of a competition. +description: After a leaderboard score comes back in a bench repo, check whether CV, the shadow test and the leaderboard agree, find out why if they don't, and record lessons for this and future competitions. Use after every upload's score, and at the end of a competition. --- # Postmortem @@ -26,4 +26,4 @@ The leaderboard is the only check you can't build yourself. Use every score to t Add a lesson to `lessons.md` in the repo whenever the method itself worked or failed: a check that caught a problem, a check that missed one, an agent mistake, a workflow that saved time. Write the lesson, the evidence (numbers), and what to do next time. -At the end of a competition, or when a lesson is clearly general, copy it to the shared file `~/.comp-lab/lessons.md` under its problem type (create the file from `${CLAUDE_PLUGIN_ROOT}/lessons/seed.md` if missing), with the competition's name and year. Tell the person, so they can also add it to the plugin's `lessons/seed.md` for everyone. +At the end of a competition, or when a lesson is clearly general, copy it to the shared file `~/.bench/lessons.md` under its problem type (create the file from `${CLAUDE_PLUGIN_ROOT}/lessons/seed.md` if missing), with the competition's name and year. Tell the person, so they can also add it to the plugin's `lessons/seed.md` for everyone. diff --git a/skills/research/SKILL.md b/skills/research/SKILL.md index 8776eae..6815f75 100644 --- a/skills/research/SKILL.md +++ b/skills/research/SKILL.md @@ -1,6 +1,6 @@ --- name: research -description: Send research agents to try many ML ideas at once in a comp-lab competition repo, each in its own worktree, and collect a standard report table. Use when the person wants to explore ideas, apply a playbook or paper, or try many combinations, or when single experiments stop improving. +description: Send research agents to try many ML ideas at once in a bench competition repo, each in its own worktree, and collect a standard report table. Use when the person wants to explore ideas, apply a playbook or paper, or try many combinations, or when single experiments stop improving. --- # Research with agents @@ -9,7 +9,7 @@ Machine learning is experimental: most ideas fail, and the ones that work often ## 1. Choose directions -List three to six directions. Good sources: the person's request, `~/.comp-lab/lessons.md`, the `experiments.md` failures (to avoid repeating them), a playbook or paper the person names, and the parts of the score that are worst. Give each direction a clear question, for example "Does a LightGBM model of the current forecast's error help, and with which mix weight?" +List three to six directions. Good sources: the person's request, `~/.bench/lessons.md`, the `experiments.md` failures (to avoid repeating them), a playbook or paper the person names, and the parts of the score that are worst. Give each direction a clear question, for example "Does a LightGBM model of the current forecast's error help, and with which mix weight?" ## 2. Write the brief @@ -17,7 +17,7 @@ Fill `${CLAUDE_SKILL_DIR}/brief.md` once per agent. Send at most three agents at ## 3. Launch -Use the `comp-lab:researcher` agent with `run_in_background: true`. It runs in its own git worktree, so it can't change the main working tree. Keep working or waiting while it runs, and don't guess its results before it reports. +Use the `bench:researcher` agent with `run_in_background: true`. It runs in its own git worktree, so it can't change the main working tree. Keep working or waiting while it runs, and don't guess its results before it reports. The worktree starts from the default branch, not from your current branch. Commit the baseline first, and if it is not on `main`, put the branch name in the brief so the agent checks it out before it starts. The data files are not in git: give the agent the absolute path of the main repo's `data/` folder and tell it to link it into its worktree (`rmdir data && ln -s
/data data`). Don't point `COMP_ROOT` at the main repo: the agent would then write into its MLflow database and submissions. @@ -27,7 +27,7 @@ Every report ends with the table from `${CLAUDE_SKILL_DIR}/report.md`. Before yo - Does the baseline row reproduce the known scores exactly? If not, its harness differs, and its numbers can't be compared. - Is every KEEP better on CV and on shadow by more than twice the noise? -- Ask the `comp-lab:leak-auditor` agent to read the candidate code: researchers make leakage mistakes too. +- Ask the `bench:leak-auditor` agent to read the candidate code: researchers make leakage mistakes too. ## 5. Move the winners into the repo diff --git a/skills/submit/SKILL.md b/skills/submit/SKILL.md index b8cdc9b..9c17463 100644 --- a/skills/submit/SKILL.md +++ b/skills/submit/SKILL.md @@ -1,6 +1,6 @@ --- name: submit -description: Plan and make leaderboard uploads in a comp-lab competition repo - choose which files to spend the day's few uploads on, upload with the person's approval, record the score, and pick the final submissions near the deadline. Use when the person wants to submit, upload, check the leaderboard, or choose final submissions. +description: Plan and make leaderboard uploads in a bench competition repo - choose which files to spend the day's few uploads on, upload with the person's approval, record the score, and pick the final submissions near the deadline. Use when the person wants to submit, upload, check the leaderboard, or choose final submissions. --- # Submit @@ -27,7 +27,7 @@ Upload only a file the person asked you to upload, or that falls under a standin uv run comp submit submissions/submission.csv -m "" --yes ``` -The comp-lab hook shows the person a permission prompt for every `comp submit`; their approval there is the confirmation, which is why `--yes` is fine after it. Never upload through the platform's own CLI, and never from a research agent. +The bench hook shows the person a permission prompt for every `comp submit`; their approval there is the confirmation, which is why `--yes` is fine after it. Never upload through the platform's own CLI, and never from a research agent. Zindi has no upload API here: `comp submit` checks the file and prints what to upload and the comment to use. After the person uploads it, record the score with `uv run comp scores --set `. diff --git a/template/AGENTS.md b/template/AGENTS.md index f290bf2..1a0907f 100644 --- a/template/AGENTS.md +++ b/template/AGENTS.md @@ -1,6 +1,6 @@ # Rules for coding agents -This is a machine learning competition repo made from the comp-lab template. Its one goal is the best final score. `comp.toml` says which competition, which metric, solo or team, and whether to write learning notes. Read it, `README.md` and `experiments.md` before you change anything. +This is a machine learning competition repo made from the bench template. Its one goal is the best final score. `comp.toml` says which competition, which metric, solo or team, and whether to write learning notes. Read it, `README.md` and `experiments.md` before you change anything. ## How a score is trusted diff --git a/template/README.md b/template/README.md index 2b7e0a9..e317839 100644 --- a/template/README.md +++ b/template/README.md @@ -2,7 +2,7 @@ CHANGE-ME: what the competition asks for, in one or two sentences, and its deadline. -Made from the comp-lab template. `comp.toml` describes the competition; `AGENTS.md` has the rules for coding agents. +Made from the bench template. `comp.toml` describes the competition; `AGENTS.md` has the rules for coding agents. ## Setup diff --git a/template/lessons.md b/template/lessons.md index 2685818..180e396 100644 --- a/template/lessons.md +++ b/template/lessons.md @@ -1,5 +1,5 @@ # Lessons from this competition Lessons about the method, not the model: what we would do the same or -differently next time. `/comp-lab:postmortem` adds to this file and copies +differently next time. `/bench:postmortem` adds to this file and copies the general lessons to the shared lessons file.