phi4-experiments: Anti-Overfitting Hypothesis Testing on Phi-4-mini

A workflow for making code changes to Phi-4-mini and testing hypotheses rigorously — with structural safeguards against benchmark overfitting.

The Problem This Solves

Classic ML-experimentation failure mode:

1. Make a change → run benchmark → see score
2. Tweak based on score → re-run → iterate
3. Eventually hit a "good" number on the held-out set
4. Publish
5. Doesn't replicate on other datasets

The Structural Cure

This project enforces the following workflow via file structure:

1. Pre-register before running

Every experiment starts with a hypotheses/HYPOTHESIS_<name>.md that specifies:

• The claim (what you expect to happen)
• The modification (exact code diff)
• The metrics (pre-committed)
• The decision rule (what counts as pass/fail BEFORE you see results)

You must commit this file before running eval.py --test. The test script refuses to run if the hypothesis file is newer than its git commit.

2. Dev/test separation

• protocol/dev_samples.json — 200 samples per benchmark, iterate freely
• protocol/test_samples.json — 500 held-out samples, accessed exactly once
• protocol/splits.py generates these with a fixed seed, then freezes them

Touching test_samples.json while iterating is detected (file mtime tracked in git).

3. Noise floor first

Before evaluating ANY modification, measure how much the baseline score varies across:

• Repeated runs (CUDA non-determinism)
• Different decoding settings
• Different sample subsets

protocol/noise_floor.py reports the 95% CI of baseline scores. No claimed improvement is meaningful unless it exceeds this CI.

4. Distribution-level comparison

For numerically-equivalent modifications (like Bakhshali rsqrt), we expect:

• Logit KL-divergence ≈ 0
• Benchmark scores: identical within noise
• Speed: measurably different (the actual thing we care about)

src/compare_logits.py computes KL divergence over a reference prompt set — often more informative than benchmark scores.

5. Multiple benchmarks, one decision

All 5 benchmarks run simultaneously. The hypothesis passes only if the modification performs consistently across them. Cherry-picking "my change helps math but hurts code" is the red flag that kills most bogus claims.

Directory Structure

phi4-experiments/
├── README.md                    (this file)
├── hypotheses/                  (pre-registered claims)
│   └── HYPOTHESIS_<name>.md
├── src/
│   ├── load_model.py            (load Phi-4-mini)
│   ├── patch_rmsnorm.py         (monkey-patch RMSNorm)
│   ├── bakhshali_rsqrt.py       (the proposed modification)
│   ├── eval.py                  (benchmark runner — the ONE eval script)
│   └── compare_logits.py        (KL divergence measurement)
├── benchmarks/
│   ├── mmlu_mini.py             (knowledge)
│   ├── gsm8k_mini.py            (math)
│   ├── humaneval_mini.py        (code)
│   ├── hellaswag_mini.py        (commonsense)
│   └── ifeval_mini.py           (instruction following)
├── protocol/
│   ├── splits.py                (generate dev/test splits)
│   ├── noise_floor.py           (measure baseline variance)
│   ├── dev_samples.json         (frozen dev set)
│   └── test_samples.json        (frozen test set — touch once)
└── results/
    └── <experiment_name>/
        ├── dev_runs/            (OK to iterate here)
        └── test_run/            (one file only — the final answer)

Workflow for a New Hypothesis

# 1. Write the hypothesis file
cp hypotheses/TEMPLATE.md hypotheses/HYPOTHESIS_bakhshali_rsqrt.md
# edit it, commit it

# 2. Measure noise floor (only once per environment)
python protocol/noise_floor.py

# 3. Implement the modification in src/
# (the patch_rmsnorm.py file gets edited with your change)

# 4. Iterate on DEV set
python src/eval.py --hypothesis bakhshali_rsqrt --split dev

# 5. When you're ready, ONE test run
python src/eval.py --hypothesis bakhshali_rsqrt --split test
# (refuses to run twice — output file is immutable after creation)

# 6. Write up result — whatever it is

Why This Structure Prevents Cheating

• You cannot iterate on test data — the eval script refuses to re-run on test
• You cannot change the hypothesis after seeing test results — the hypothesis file mtime is tracked
• You cannot cherry-pick benchmarks — the runner reports all 5 together
• You cannot hide null results — the results directory is version-controlled

Requirements

pip install torch transformers accelerate datasets lm-eval evaluate

GPU recommended (Phi-4-mini needs ~8GB VRAM at FP16).

First Hypothesis Pre-Loaded

hypotheses/HYPOTHESIS_bakhshali_rsqrt.md tests whether replacing the rsqrt in RMSNorm with Bakhshali's closed-form 2-step Newton expression changes model behavior. Expected result: no measurable change (it's algebraically equivalent). This is a sanity check of the whole infrastructure before testing more interesting modifications.