Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"name": "comp-lab",
"owner": { "name": "Ahmed Alghali" },
"plugins": [
{
"name": "comp-lab",
"source": "./",
"description": "Run machine learning competitions with Claude: validation first, research agents, tracked results, approved uploads."
}
]
}
8 changes: 8 additions & 0 deletions .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"name": "comp-lab",
"version": "0.1.0",
"description": "Run machine learning competitions (Kaggle, Zindi) with Claude: validation first, a main agent plus research agents, every result tracked, uploads only with your approval.",
"author": { "name": "Ahmed Alghali" },
"repository": "https://github.com/A7med7x7/comp-lab",
"keywords": ["kaggle", "zindi", "machine-learning", "competition", "mlflow"]
}
36 changes: 36 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: CI

on:
push:
branches: [main]
pull_request:

concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true

jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
with:
enable-cache: true
- name: Install the template's environment
run: uv sync --locked
working-directory: template
- name: Lint the template and the hook
run: |
uv run --project template ruff check template hooks tests
uv run --project template ruff format --check template hooks tests
- name: Test the template
run: uv run pytest
working-directory: template
- name: Test the hook
run: uv run --project template pytest -q tests
- name: Check the JSON manifests
run: |
for f in .claude-plugin/plugin.json .claude-plugin/marketplace.json hooks/hooks.json; do
python3 -m json.tool "$f" > /dev/null && echo "$f ok"
done
70 changes: 70 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# comp-lab

[![CI](https://github.com/A7med7x7/comp-lab/actions/workflows/ci.yml/badge.svg)](https://github.com/A7med7x7/comp-lab/actions/workflows/ci.yml)

A Claude Code plugin for machine learning competitions on Kaggle and Zindi. It has one goal: the best final score. It does not try to get there by guessing faster. It makes the validation trustworthy first, then lets a main agent and research agents run many experiments that are all measured the same way.

It grew out of the Madinah crowd-count competition (Kaggle, 2026). There, one model scored 0.821 in cross-validation and 1.022 on the leaderboard. A second check built to look like the test caught the problem, and a research agent then found three improvements in one evening. This plugin packages that way of working.

## What is in it

| part | what it does |
| --- | --- |
| `template/` | the project harness each competition starts from: `comp.toml`, data loading, CV, a shadow test, the metric, MLflow tracking, a submissions log tied to commits, and Kaggle/Zindi upload |
| `skills/comp-start` | set up a new competition and build and prove the validation before any model |
| `skills/experiment` | one idea, scored on CV and the shadow test against the noise, logged, kept or dropped |
| `skills/research` | send research agents to try many ideas in their own worktrees, with a fixed brief and report |
| `skills/submit` | plan the day's few uploads, upload with your approval, pick the final submissions |
| `skills/postmortem` | after each leaderboard score: do the checks agree? Record lessons |
| `skills/explain` | optional learning notes in three layers, in simple English |
| `agents/researcher` | tests one direction in a git worktree and reports a table |
| `agents/leak-auditor` | reads an experiment and reports any information leak |
| `hooks/guard.py` | in competition repos: asks you before every upload, blocks uploads that skip the log, blocks whole-folder formatting |
| `lessons/seed.md` | lessons across competitions, read at the start of each new one |

## Install

The repo is private, so your git must be able to read it (`gh auth login` and `gh auth setup-git`, or an SSH key).

```text
/plugin marketplace add A7med7x7/comp-lab
/plugin install comp-lab@comp-lab
```

> [!NOTE]
> If you have no SSH key for GitHub, set `CLAUDE_CODE_PLUGIN_PREFER_HTTPS=1` before starting Claude Code, so the install uses HTTPS.

## Use

Start a competition by telling Claude, or run the skill directly:

```text
/comp-lab:comp-start https://www.kaggle.com/competitions/<name>
```

Then work in that repo:

```text
/comp-lab:experiment try adding lag features to 003_baseline
/comp-lab:research test the ideas in this playbook against our best model
/comp-lab:submit upload the best file when the limit resets
/comp-lab:postmortem the score came back: 0.912
```

Every experiment is scored twice. `uv run comp run` gives the CV score and the shadow score, and `uv run comp compare` keeps a change only if it is better on both by more than twice the noise that `uv run comp noise` measured. The leaderboard is a third check and never decides alone.

`comp.toml` in each competition repo sets solo or team mode (team mode adds per-person experiment folders and reviewers) and whether learning notes are written.

## How it is built

- Claude is the orchestrator. The skills say what to do at each stage, the agents have fixed roles and limits, and the hook enforces the two rules that must never depend on memory: uploads need your approval, and nothing formats the whole folder.
- The template is copied into each competition and then belongs to it. Change `data.py`, `cv.py`, `metric.py` and `shadow.py` freely for a competition with a different shape; keep their meaning.
- Lessons live in `~/.comp-lab/lessons.md` on your machine, seeded from `lessons/seed.md`. Add the general ones back to `lessons/seed.md` here so every new competition starts from them.

## Develop

```bash
cd template && uv sync && uv run pytest && cd ..
uv run --project template pytest tests # the hook's tests
claude plugin validate .
```
20 changes: 20 additions & 0 deletions agents/leak-auditor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
name: leak-auditor
description: Reads an experiment's code in a comp-lab competition repo and reports any way it could see information the real test would not have (target leakage, future values, held-out rows, test labels). Read-only. Use before submitting a new method and on every research agent's winning code.
tools: Read, Grep, Glob, Bash
---

You audit one experiment for leakage. You change nothing. Leakage makes CV look better than the leaderboard will be, and it is the most common reason a good CV score fails.

Read `comp.toml`, `AGENTS.md`, `src/comp/data.py`, `src/comp/cv.py`, `src/comp/shadow.py`, and the experiment file (and anything it imports). Then check:

1. `solve(task)` uses only `task`. It never reads `data/`, global caches filled from other folds, or files written by an earlier run.
2. Everything fitted (models, scalers, encoders, target means, imputation values) is fitted on `task.train` inside `solve`, never on train plus test together, unless the competition allows unlabelled test features and the code uses no test labels.
3. Time: no feature for a row uses values after that row's time. Look closely at rolling windows with `center=True`, `shift(-k)`, `bfill`, `interpolate`, `fillna` over the time axis, and resampling. Each can pull the future in.
4. Groups: when CV keeps groups apart, the features don't carry group-level statistics computed across folds.
5. Target: no feature is the target in disguise (a column built from it, an id that sorts by it).
6. The shadow test and CV use the same `solve` path as the real submission.

You may run code to confirm a suspicion (for example, change a future value and see if a past prediction moves). Don't edit tracked files.

Report: for each problem, the file and line, what leaks, how much it could matter, and the fix. If you find nothing, say what you checked. End with one line: CLEAN, or LEAKS FOUND (n).
20 changes: 20 additions & 0 deletions agents/researcher.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
name: researcher
description: Tests many variants of one ML direction in a comp-lab competition repo, in its own git worktree, and returns a standard report table with CV and shadow scores against the baseline. Use with a filled brief from the research skill.
isolation: worktree
---

You are a research agent in a machine learning competition. The main agent owns the repo, the decisions and the uploads. Your job is to find out, with numbers, which variants of your direction improve the score.

How you work:

- Follow the brief. Read the files it names before you write code.
- You are in your own git worktree. Write only under `scratch/`. Don't commit, push, or change files outside `scratch/`.
- First reproduce the baseline's scores exactly with your harness. If you can't, stop and report why: numbers from a different harness can't be compared.
- Change one thing at a time. Try many variants and combinations: most ideas fail, and some only work together. Average random seeds when the model is random, and compare every gain with the noise the brief gives.
- `solve(task)` may use only `task`. No feature may use information the real test would not have, such as values after the row's time or test labels. Check every rolling window, `shift`, `bfill`, `interpolate` and centred window for the future leaking in.
- Never upload, never run `comp submit`, never call a platform's submit command.
- Format only files you wrote. Don't run a formatter on the whole folder.
- Before you finish, stop every background process you started.

End your reply with the report format from the brief: summary, the table of every idea tried (with the baseline row first), noise, how to build the winners, what doesn't apply, and what you could not test. Be plain about failures and noise.
80 changes: 80 additions & 0 deletions hooks/guard.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
"""PreToolUse guard for Bash commands in competition repos.

Claude Code sends the tool call as JSON on stdin. This script answers with a
permission decision on stdout, or prints nothing to leave the call alone.
It acts only inside a competition repo: a folder with comp.toml in it or
above it.

direct platform upload (kaggle competitions submit ...) deny
comp submit, from a subagent deny
comp submit, from the main agent ask the person
a formatter run over the whole folder (ruff format .) deny

Uploads use a few daily submissions and can't be taken back, so the person
approves each one in Claude Code's own permission prompt. That approval is
real in a way that an agent passing --yes is not.
"""

import json
import re
import sys
from pathlib import Path

DIRECT_UPLOAD = re.compile(r"\bkaggle\s+(competitions|c)\s+submit\b")
COMP_SUBMIT = re.compile(r"\bcomp\s+submit\b")
# `ruff format` or `black` with no path, or with `.` as the path.
WHOLE_FOLDER_FORMAT = re.compile(r"\b(ruff\s+format|black)(\s+-{1,2}[\w-]+)*\s*(\.\s*)?($|[;&|])")


def in_competition_repo(cwd: str) -> bool:
folder = Path(cwd or ".").resolve()
return any((p / "comp.toml").exists() for p in (folder, *folder.parents))


def decide(event: dict) -> tuple[str, str] | None:
"""(decision, reason) for this tool call, or None to leave it alone."""
if event.get("tool_name") != "Bash" or not in_competition_repo(event.get("cwd", "")):
return None
command = event.get("tool_input", {}).get("command", "")
if DIRECT_UPLOAD.search(command):
return "deny", (
"Upload through `uv run comp submit <file>`, not the platform CLI: it checks the "
"file, ties it to a commit, and writes the score into submissions.md."
)
if COMP_SUBMIT.search(command):
if event.get("agent_id"):
return "deny", "Only the main agent uploads. Report the file to it instead."
return "ask", (
"This uploads a submission and uses one of today's few tries. "
"Approve only if you asked for this specific file."
)
match = WHOLE_FOLDER_FORMAT.search(command)
if match and not re.search(r"--(check|diff)\b", match.group(0)): # read-only runs are fine
return "deny", (
"Don't format the whole folder: it rewrites notes and other people's files. "
"Use `make format` (tracked Python files only) or name the files you changed."
)
return None


def main() -> None:
try:
event = json.load(sys.stdin)
except json.JSONDecodeError:
return # not our business: never block on a bad payload
decision = decide(event)
if decision is None:
return
verdict, reason = decision
output = {
"hookSpecificOutput": {
"hookEventName": "PreToolUse",
"permissionDecision": verdict,
"permissionDecisionReason": reason,
}
}
print(json.dumps(output))


if __name__ == "__main__":
main()
17 changes: 17 additions & 0 deletions hooks/hooks.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"description": "comp-lab guards: uploads need the person's approval, and no whole-folder formatting, in competition repos only",
"hooks": {
"PreToolUse": [
{
"matcher": "Bash",
"hooks": [
{
"type": "command",
"command": "python3 \"${CLAUDE_PLUGIN_ROOT}/hooks/guard.py\"",
"timeout": 10
}
]
}
]
}
}
19 changes: 19 additions & 0 deletions lessons/seed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Lessons across competitions

Lessons about the method, grouped by problem type. Each one says where it was
learned. `/comp-lab:comp-start` reads this file before planning a new
competition, and `/comp-lab:postmortem` adds to it.

## Any competition

- Build and prove the validation before any model. Reproduce something the organisers published (the sample submission, a baseline score) with your own code first. (Madinah 2026)
- One check is not enough. Ask "what does the test have that my CV does not?" and build a second check for that difference. In Madinah, CV trained on the day next to the held-out day, while the test days were a week away. A model scored 0.821 in CV and 1.022 on the leaderboard. A test-day check built from the released test rows caught it. (Madinah 2026)
- Measure the noise before you believe a gain. Changing only the random seed of a LightGBM model moved the score by about 0.01, which was as large as most tuning gains. Average a few seeds, and keep a change only if it beats about twice the noise on every check. (Madinah 2026)
- A main agent plus research agents works. Researchers try many variants in scratch or a worktree and report a table; the main agent moves only the winners into the repo. About 30 variants in one evening, with no conflicts. (Madinah 2026)
- Never run a formatter over the whole folder. It rewrote code blocks inside a Markdown document that was not ours. (Madinah 2026)

## Time series and forecasting

- A gradient boosting model that predicts the whole value did worse than simple rules. The same model trained to predict the rules' error (a residual), and mixed half and half with them, improved CV by 0.04 and both test-day checks. Start from strong hand-made rules, then let a model correct them. (Madinah 2026)
- Calendar features matter, and their strength must be checked on data far from the training days. Lining up daily crowd curves on prayer times helped only at the mild setting; the strong setting overfit the CV. (Madinah 2026)
- Pseudo-labels made from the test's own released rows did not help here. (Madinah 2026)
73 changes: 73 additions & 0 deletions skills/comp-start/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
---
name: comp-start
description: Start a new machine learning competition (Kaggle or Zindi) from the comp-lab template - create the repo, fill comp.toml, and build and prove the validation before any model. Use when the person wants to begin or set up a competition or ML project.
---

# Start a competition

The goal of every competition is the best final score. Most lost competitions are lost here, in validation, not later in modelling. So this skill ends with a validation you can trust and one baseline, and nothing more.

Work through the steps in order. Tell the person what you found after steps 2, 4 and 6.

## 1. Gather the facts

Ask only for what you can't find yourself: the competition URL, and solo or team (and the teammates' GitHub logins for a team). Then read, from the competition page and data page:

- the task, the target, the metric (exact formula, and whether lower is better)
- the files and their columns, the id column, the sample submission
- the public/private leaderboard split, if stated
- daily upload limit, final pick count, deadline (with time zone)
- rules on external data and pretrained models

Save the page text in `docs/competition.md`. Read the shared lessons, `~/.comp-lab/lessons.md` (create it from `${CLAUDE_PLUGIN_ROOT}/lessons/seed.md` if it is missing), and note which ones apply.

## 2. Create the repo

```bash
cp -R "${CLAUDE_PLUGIN_ROOT}/template" <new-folder>
cd <new-folder> && git init -b main
```

- Fill `comp.toml`, the `description` in `pyproject.toml`, and the top of `README.md`.
- Create `experiments/<name>/` for each person in `[team] people` and remove `experiments/me/` if unused.
- If `[notes] enabled`, create `notes/index.html` and `notes/glossary.html` (see the `explain` skill for the style).
- `uv sync`, `make data` (Kaggle) or download by hand (Zindi), then `make test`.
- Make the first commit on `main`, create a private GitHub repo with `gh repo create --private --source . --push`, and check that CI passes.

## 3. Make the metric exact

If the metric is not one of the built-in names in `src/comp/metric.py`, write it there, with a test using numbers you compute by hand. If the score averages several parts (several targets, several kinds of rows), return each part too.

Prove it: reproduce any score the organisers published (a baseline, the sample submission's score) to several decimals. Write in `README.md` what you reproduced.

## 4. Answer: how does the test differ from the training data?

This is the most important question in the whole competition. Look at the data, not just the description:

- Run `uv run comp check`. An adversarial AUC near 0.5 means train and test look alike; above about 0.75 means they differ.
- Time: does the test come after the training data? Then `[cv] scheme = "time"` and set `time_column`.
- Groups: does the test contain new users, patients, places or days? Then `scheme = "group"` and set `group_column`.
- Distance: how far is the test from the nearest training row (in days, in groups)? CV must not be closer than that.
- New categories in the test that never appear in training.

Write the answer in `README.md` under "How we validate". If the data has a shape the default `data.py` and `cv.py` don't fit (time series with holes, images, several tables), rewrite those two files now, keeping `Task`, `folds` and `cross_validate` with the same meaning.

## 5. Build the shadow test

`src/comp/shadow.py` holds out the most test-like training rows, or the latest rows by time. Replace `holdout` if you know something better, for example rows the organisers released from the test period. The rule: the shadow test must copy the difference you found in step 4 more closely than CV does.

## 6. Baseline and noise

- Copy `experiments/_template.py` to `experiments/<name>/001_baseline.py` and make it a simple, honest model.
- `uv run comp run experiments/<name>/001_baseline.py`
- `uv run comp noise experiments/<name>/001_baseline.py`
- Submit the baseline once (with the person's approval, see the `submit` skill). Check that the leaderboard score is close to CV and shadow. If not, the validation is wrong: go back to step 4 before any modelling.

## Done when

- the metric is reproduced, CV and shadow are built, and the reason for each is written down
- the baseline's leaderboard score agrees with CV and shadow
- the noise is measured
- CI is green

Then continue with the `experiment` and `research` skills.
Loading
Loading