Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .claude-plugin/icon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
{
"name": "comp-lab",
"name": "bench",
"owner": { "name": "Ahmed Alghali" },
"plugins": [
{
"name": "comp-lab",
"name": "bench",
"source": "./",
"description": "Run machine learning competitions with Claude: validation first, research agents, tracked results, approved uploads."
}
Expand Down
4 changes: 2 additions & 2 deletions .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{
"name": "comp-lab",
"name": "bench",
"version": "0.1.0",
"description": "Run machine learning competitions (Kaggle, Zindi) with Claude: validation first, a main agent plus research agents, every result tracked, uploads only with your approval.",
"author": { "name": "Ahmed Alghali" },
"repository": "https://github.com/A7med7x7/comp-lab",
"repository": "https://github.com/A7med7x7/bench",
"keywords": ["kaggle", "zindi", "machine-learning", "competition", "mlflow"]
}
32 changes: 21 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,20 @@
# comp-lab
<p align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="assets/brand/bench-horizontal-dark.svg">
<img src="assets/brand/bench-horizontal.svg" alt="bench" width="360">
</picture>
</p>

[![CI](https://github.com/A7med7x7/comp-lab/actions/workflows/ci.yml/badge.svg)](https://github.com/A7med7x7/comp-lab/actions/workflows/ci.yml)
<p align="center"><strong>Validation first. Every result measured twice.</strong></p>

A Claude Code plugin for machine learning competitions on Kaggle and Zindi. It has one goal: the best final score. It does not try to get there by guessing faster. It makes the validation trustworthy first, then lets a main agent and research agents run many experiments that are all measured the same way.
<p align="center"><a href="https://github.com/A7med7x7/bench/actions/workflows/ci.yml"><img src="https://github.com/A7med7x7/bench/actions/workflows/ci.yml/badge.svg" alt="CI"></a></p>

bench is a Claude Code plugin for machine learning competitions on Kaggle and Zindi. It has one goal: the best final score. It does not try to get there by guessing faster. It makes the validation trustworthy first, then lets a main agent and research agents run many experiments that are all measured the same way.

It grew out of the Madinah crowd-count competition (Kaggle, 2026). There, one model scored 0.821 in cross-validation and 1.022 on the leaderboard. A second check built to look like the test caught the problem, and a research agent then found three improvements in one evening. This plugin packages that way of working.

![The bench loop: comp-start proves the validation, experiment tries one idea on two checks, research sends agents to try many, submit uploads only with your approval, postmortem checks whether the leaderboard agrees, and lessons feed the next experiment. A change is kept only if CV and the shadow test both beat twice the noise.](assets/brand/loop.svg)

## What is in it

| part | what it does |
Expand All @@ -21,14 +30,15 @@ It grew out of the Madinah crowd-count competition (Kaggle, 2026). There, one mo
| `agents/leak-auditor` | reads an experiment and reports any information leak |
| `hooks/guard.py` | in competition repos: asks you before every upload, blocks uploads that skip the log, blocks whole-folder formatting |
| `lessons/seed.md` | lessons across competitions, read at the start of each new one |
| `assets/brand/` | the bench logo kit (symbol, wordmark, lockups, icons), the workflow diagram, `notes.css` for themed learning notes, and `GUIDELINES.md` |

## Install

The repo is private, so your git must be able to read it (`gh auth login` and `gh auth setup-git`, or an SSH key).

```text
/plugin marketplace add A7med7x7/comp-lab
/plugin install comp-lab@comp-lab
/plugin marketplace add A7med7x7/bench
/plugin install bench@bench
```

> [!NOTE]
Expand All @@ -39,16 +49,16 @@ The repo is private, so your git must be able to read it (`gh auth login` and `g
Start a competition by telling Claude, or run the skill directly:

```text
/comp-lab:comp-start https://www.kaggle.com/competitions/<name>
/bench:comp-start https://www.kaggle.com/competitions/<name>
```

Then work in that repo:

```text
/comp-lab:experiment try adding lag features to 003_baseline
/comp-lab:research test the ideas in this playbook against our best model
/comp-lab:submit upload the best file when the limit resets
/comp-lab:postmortem the score came back: 0.912
/bench:experiment try adding lag features to 003_baseline
/bench:research test the ideas in this playbook against our best model
/bench:submit upload the best file when the limit resets
/bench:postmortem the score came back: 0.912
```

Every experiment is scored twice. `uv run comp run` gives the CV score and the shadow score, and `uv run comp compare` keeps a change only if it is better on both by more than twice the noise that `uv run comp noise` measured. The leaderboard is a third check and never decides alone.
Expand All @@ -59,7 +69,7 @@ Every experiment is scored twice. `uv run comp run` gives the CV score and the s

- Claude is the orchestrator. The skills say what to do at each stage, the agents have fixed roles and limits, and the hook enforces the two rules that must never depend on memory: uploads need your approval, and nothing formats the whole folder.
- The template is copied into each competition and then belongs to it. Change `data.py`, `cv.py`, `metric.py` and `shadow.py` freely for a competition with a different shape; keep their meaning.
- Lessons live in `~/.comp-lab/lessons.md` on your machine, seeded from `lessons/seed.md`. Add the general ones back to `lessons/seed.md` here so every new competition starts from them.
- Lessons live in `~/.bench/lessons.md` on your machine, seeded from `lessons/seed.md`. Add the general ones back to `lessons/seed.md` here so every new competition starts from them.

## Develop

Expand Down
2 changes: 1 addition & 1 deletion agents/leak-auditor.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: leak-auditor
description: Reads an experiment's code in a comp-lab competition repo and reports any way it could see information the real test would not have (target leakage, future values, held-out rows, test labels). Read-only. Use before submitting a new method and on every research agent's winning code.
description: Reads an experiment's code in a bench competition repo and reports any way it could see information the real test would not have (target leakage, future values, held-out rows, test labels). Read-only. Use before submitting a new method and on every research agent's winning code.
tools: Read, Grep, Glob, Bash
---

Expand Down
2 changes: 1 addition & 1 deletion agents/researcher.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: researcher
description: Tests many variants of one ML direction in a comp-lab competition repo, in its own git worktree, and returns a standard report table with CV and shadow scores against the baseline. Use with a filled brief from the research skill.
description: Tests many variants of one ML direction in a bench competition repo, in its own git worktree, and returns a standard report table with CV and shadow scores against the baseline. Use with a filled brief from the research skill.
isolation: worktree
---

Expand Down
53 changes: 53 additions & 0 deletions assets/brand/GUIDELINES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# bench logo guidelines

## 1. The logo

- Idea: a bench seen from the side is also the letter b. The back post is the stem; the seat, front leg and stretcher close the bowl. The mark is the first letter of the name, so the icon and the wordmark are one idea.
- Versions:
- horizontal (`bench-horizontal.svg`): the wordmark with a green b. The primary version.
- stacked (`bench-stacked.svg`): the symbol above the wordmark.
- symbol (`bench-symbol.svg`) for 64 px and up.
- small symbol (`bench-symbol-small.svg`) for 16 to 48 px.
- wordmark (`bench-wordmark.svg`): all ink.
- Files: SVG for screens and print. `bench-horizontal-dark.svg` keeps the green b for dark backgrounds. The web icons are in `icons/`, the presentation slides in `board/`, the workflow diagram is `loop.svg`, and `notes.css` themes learning notes.

## 2. Clear space

Keep a clear zone of one wordmark stroke width on all sides: the thickness of the b's stem, which is 28 of the wordmark's 256-unit height. It scales with the logo.

## 3. Minimum size

| Version | Screen |
| --- | --- |
| Horizontal | 96 px wide |
| Symbol | 64 px; below that use the small symbol, down to 16 px |

## 4. Colour

| Name | HEX | RGB | Use |
| --- | --- | --- | --- |
| Signal green | #0B7A4B | 11, 122, 75 | the b, app icon tile, "keep" |
| Ink | #15181C | 21, 24, 28 | wordmark letters, text |
| Ground | #F6F7F5 | 246, 247, 245 | light backgrounds |
| Muted | #5B636B | 91, 99, 107 | secondary text |
| Caution | #B45309 | 180, 83, 9 | uploads and warnings in diagrams, never in the logo |

Green on white has a contrast of 5.4:1 and ink on white 17.8:1, so both pass for text.

Approved pairs: green b and ink on white or Ground; all white on green; all white on Ink; all black on white.

## 5. Typography

The wordmark is drawn, not typed: one 28-unit stroke and the bench seat's rounded top-right corner on every letter. For text next to it, use Space Grotesk 700 for headings, Inter for body and JetBrains Mono for numbers and commands. All three are open licence (SIL OFL) from Google Fonts.

## 6. Don'ts

- Don't stretch, rotate, outline or add shadows or gradients.
- Don't recolour outside the palette, and don't make the b any colour but green or the one-colour version.
- Don't type "bench" in a font in place of the wordmark.
- Don't use the large symbol below 64 px; use the small one.

## 7. Notes

- Trademark: no search was done. Check before any public launch.
- Made with the logo-design skill (github.com/kaankiziltug/logo-design-skill): SVG audit 100/100 on every file, renders checked at 16 to 512 px.
8 changes: 8 additions & 0 deletions assets/brand/bench-horizontal-black.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 8 additions & 0 deletions assets/brand/bench-horizontal-dark.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 8 additions & 0 deletions assets/brand/bench-horizontal-mono-0b7a4b.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 8 additions & 0 deletions assets/brand/bench-horizontal-white.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 8 additions & 0 deletions assets/brand/bench-horizontal.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
11 changes: 11 additions & 0 deletions assets/brand/bench-stacked.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions assets/brand/bench-symbol-app-icon.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions assets/brand/bench-symbol-black.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions assets/brand/bench-symbol-favicon.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions assets/brand/bench-symbol-mono-0b7a4b.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions assets/brand/bench-symbol-small.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions assets/brand/bench-symbol-square.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions assets/brand/bench-symbol-white.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions assets/brand/bench-symbol.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 8 additions & 0 deletions assets/brand/bench-wordmark.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/brand/board/slide-01.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/brand/board/slide-02.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/brand/board/slide-03.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/brand/board/slide-04.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/brand/board/slide-05.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions assets/brand/favicon.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/brand/icons/apple-touch-icon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/brand/icons/favicon-16.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/brand/icons/favicon-32.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/brand/icons/favicon-48.png
Binary file added assets/brand/icons/favicon.ico
Binary file not shown.
Loading
Loading