New benchmark: Zoey-MetalGate on matbench_expt_gap (MAE 0.2775)

[JeetuSK0808]

New benchmark submission: Zoey-MetalGate

Task: matbench_expt_gap (composition-only, 4,604 samples)
Score: MAE 0.2775 ± 0.0051 eV, RMSE 0.6825, on the official five-fold protocol
Authors: Satyajeeth Suresh Kannan, Tushar Chatterjee (high school students, Atlanta GA; independent work, no lab affiliation)

What it is

A two-stage metal-classifier-gated gradient-boosting blend. 53% of the labels in
matbench_expt_gap are exactly 0.0 eV because those compounds are metals, so a single
regressor spends much of its error budget smearing small positive predictions across that
zero mass. Instead:

1. A LightGBM classifier predicts P(metal) from composition, trained on the binary
   indicator gap == 0 derived from the training fold's own labels.
2. A gradient-boosted regressor is fit on the non-metal rows only of the training fold.
3. The two are combined per compound, softly (1 - P_metal) * max(reg, 0) or hard
   (zero where P_metal > 0.5).

The submitted model averages three such gated models with fixed weights (0.50 / 0.25 / 0.25),
chosen by greedy forward selection on out-of-fold predictions from the training folds.
Features are 493 stateless matminer composition descriptors.

For reference, a plain single LightGBM on the same features with no gate and no blend
scores 0.3322 ± 0.0117.

The classify-then-regress idea is not new — two-stage metal/non-metal handling appears
in the band gap literature. What this submission adds is a carefully preregistered
evaluation of that combination on this benchmark plus an error analysis of where the gain
actually comes from.

Evaluation hygiene

The campaign was preregistered before any results existed, fixing the metric, success
criteria, model-selection procedure and leakage checks in advance.

• All model selection (features, model families, hyperparameters, blend weights) used
  internal 5-fold CV inside the official training folds only.
• Test data was loaded only via get_test_data(fold, include_target=False) and scored only
  via task.record(). The finalist was frozen to a file from validation scores before any
  test evaluation, and the test splits were evaluated exactly once for it.
• model_selection.py in the submission directory documents and reproduces the selection
  procedure without touching test data.

Because this result came in below the existing leaderboard entries, our own preregistration
required an investigation before reporting it. Three checks:

1. Independent rescore. Recomputing every fold's MAE directly from the raw prediction
   arrays against dataset ground truth, bypassing matbench's scorer entirely, gives
   0.27751 — matching the recorded score to five decimals.
2. Split integrity. The five official folds are index-disjoint, cover the dataset
   exactly, and share zero normalized reduced formulas between any fold's train and test
   splits. Per-fold test counts 921/921/921/921/920.
3. Mechanism. The gain decomposes arithmetically: the classifier agrees with the true
   metal/non-metal label ~92% of the time, metals then carry ~0.15 eV residual error over
   53% of rows and non-metals ~0.45 eV over the rest, and 0.53*0.15 + 0.47*0.45
   reconstructs the validation score.

Where the gain comes from, and what it costs

true gap range	n	plain boosted ensemble	Zoey-MetalGate
metals (gap = 0)	2450	0.241	0.145
0–1 eV	493	0.228	0.295
1–2 eV	615	0.323	0.340
2–4 eV	855	0.540	0.426
> 4 eV	191	1.486	1.069

The improvement is concentrated in the zero-gap population and in wide-gap insulators; the
soft gate measurably hurts 0–1 eV semiconductors. We think that is the most useful thing
to know about this entry, so it is stated in info.json as well.

Compute

Whole campaign (featurization + 103-config search + final evaluation) ran overnight on 8 CPU
cores, no GPU. Reproducing just the submitted result via run.py takes ~45 min, mostly
featurization, which caches after the first run.

Files

benchmarks/matbench_v0.1_zoey_metalgate/
├── info.json
├── results.json.gz
├── run.py                 # self-contained reproduction of the submitted result
└── model_selection.py     # how the config was chosen, training folds only

Local run of scripts/test_submission.py checks passes: is_valid true, all required
info.json fields present, source .py files present, no non-results file over 10 MB.

Limitations

Composition-only, so polymorphs are indistinguishable and part of the remaining error is an
irreducible floor in the data that we have not quantified. No uncertainty estimates. Only
matbench_expt_gap was run; nothing here demonstrates transfer to other tasks.

Could a maintainer please apply the new_benchmark label — we don't have permission to set
it ourselves. Happy to make any changes needed.