New benchmark: Zoey-MetalGate on matbench_expt_gap (MAE 0.2775)

benchmarks/matbench_v0.1_zoey_metalgate/info.json

@@ -0,0 +1,20 @@
+{
+  "authors": "Satyajeeth Suresh Kannan, Tushar Chatterjee",
+  "algorithm": "Zoey-MetalGate",
+  "algorithm_long": "Zoey-MetalGate is a two-stage, metal-classifier-gated gradient-boosting blend for composition-only band gap prediction. The name refers to the two ideas it is built from: 'Zoey' for the zero-inflated label distribution of matbench_expt_gap (53% of the experimental gaps in this dataset are exactly 0.0 eV because those compounds are metals), and 'MetalGate' for the mechanism that exploits it. Stage 1 is a LightGBM classifier that predicts P(metal) from composition, trained on the binary indicator (gap == 0) derived from the training fold's own labels. Stage 2 is a gradient-boosted regressor (XGBoost or LightGBM) fit only on the non-metal rows of the training fold, so that it models the shape of real band gaps without being pulled toward the zero mass. The two stages are combined per compound either softly, prediction = (1 - P_metal) * max(regressor, 0), or hard, prediction = 0 where P_metal > 0.5 and max(regressor, 0) otherwise. The submitted model is a fixed-weight average of three such gated models (0.50 soft-gate/XGBoost, 0.25 hard-gate/XGBoost, 0.25 hard-gate/LightGBM), with the members and weights chosen by greedy forward selection on out-of-fold predictions computed inside the official training folds. Inputs are 493 stateless composition descriptors from matminer: Magpie element-property statistics, valence-orbital fractions, atomic HOMO/LUMO energies, electronegativity band center, stoichiometry p-norms, element fractions, and the deml, Meredig, Miedema, Yang solid-solution, atomic packing efficiency, transition-metal fraction and ion-property families. Every descriptor is a tabulated element-property lookup combined by stoichiometry, so nothing is fit to the dataset during featurization. On the official five-fold protocol the blend scores MAE 0.2775 +/- 0.0051 eV, and a plain single LightGBM on the same features, with no gate and no blend, scores 0.3322 +/- 0.0117 eV. We want to be explicit that the classify-then-regress idea is not new: two-stage metal/non-metal handling appears in the band gap prediction literature. The contribution here is a careful, preregistered evaluation of that combination on this benchmark, together with an error analysis showing where the gain actually comes from.",
+  "bibtex_refs": "@article{Dunn2020,\n doi = {10.1038/s41524-020-00406-3},\n url = {https://doi.org/10.1038/s41524-020-00406-3},\n year = {2020},\n month = sep,\n publisher = {Springer Science and Business Media {LLC}},\n volume = {6},\n number = {1},\n author = {Alexander Dunn and Qi Wang and Alex Ganose and Daniel Dopp and Anubhav Jain},\n title = {Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm},\n journal = {npj Computational Materials}\n}\n\n@article{Zhuo2018,\n doi = {10.1021/acs.jpclett.8b00124},\n url = {https://doi.org/10.1021/acs.jpclett.8b00124},\n year = {2018},\n publisher = {American Chemical Society},\n volume = {9},\n number = {7},\n pages = {1668--1673},\n author = {Ya Zhuo and Aria Mansouri Tehrani and Jakoah Brgoch},\n title = {Predicting the Band Gaps of Inorganic Solids by Machine Learning},\n journal = {The Journal of Physical Chemistry Letters}\n}\n\n@article{Ward2018,\n doi = {10.1016/j.commatsci.2018.05.018},\n url = {https://doi.org/10.1016/j.commatsci.2018.05.018},\n year = {2018},\n publisher = {Elsevier},\n volume = {152},\n pages = {60--69},\n author = {Logan Ward and Alexander Dunn and Alireza Faghaninia and Nils E. R. Zimmermann and Saurabh Bajaj and Qi Wang and Joseph Montoya and Jiming Chen and Kyle Bystrom and Maxwell Dylla and Kyle Chard and Mark Asta and Kristin A. Persson and G. Jeffrey Snyder and Ian Foster and Anubhav Jain},\n title = {Matminer: An open source toolkit for materials data mining},\n journal = {Computational Materials Science}\n}\n\n@inproceedings{Chen2016,\n doi = {10.1145/2939672.2939785},\n url = {https://doi.org/10.1145/2939672.2939785},\n year = {2016},\n publisher = {ACM},\n pages = {785--794},\n author = {Tianqi Chen and Carlos Guestrin},\n title = {{XGBoost}: A Scalable Tree Boosting System},\n booktitle = {Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining}\n}\n\n@inproceedings{Ke2017,\n year = {2017},\n publisher = {Curran Associates, Inc.},\n volume = {30},\n author = {Guolin Ke and Qi Meng and Thomas Finley and Taifeng Wang and Wei Chen and Weidong Ma and Qiwei Ye and Tie-Yan Liu},\n title = {{LightGBM}: A Highly Efficient Gradient Boosting Decision Tree},\n booktitle = {Advances in Neural Information Processing Systems}\n}",
+  "notes": "AUTHORS: Satyajeeth Suresh Kannan and Tushar Chatterjee, high school students in Atlanta, GA (contact: jeetusk08@gmail.com, GitHub @JeetuSK0808). This is independent work with no institutional affiliation or lab resources.\n\nCOMPUTE: The entire campaign, including featurization, a 103-configuration hyperparameter search and the final evaluation, ran overnight on 8 CPU cores (AMD Ryzen Z1 Extreme, 15 GB RAM) with no GPU. Reproducing just the submitted result with run.py takes roughly 45 minutes, of which about 30 minutes is featurization (dominated by AtomicPackingEfficiency) and is cached to disk after the first run.\n\nEVALUATION HYGIENE: The campaign was preregistered before any results existed, fixing the baseline, metric, success criteria, model-selection procedure and leakage checks in advance. All model selection (feature sets, model families, hyperparameters, blend membership and weights) was done by internal 5-fold cross-validation inside the official training folds only; test data was loaded exclusively through get_test_data(fold, include_target=False) and scored exclusively through task.record(). The finalist configuration was frozen to a file based on validation scores before any test evaluation, and the test splits were evaluated exactly once for it. model_selection.py in this directory documents and reproduces that procedure.\n\nVERIFICATION: Because this result came in below the published leaderboard entries, our preregistration required us to investigate before reporting it. Three checks were run. (1) Recomputing every fold's MAE directly from the raw prediction arrays against the dataset ground truth, bypassing matbench's scorer entirely, reproduces 0.27751, matching the recorded score to five decimals. (2) The five official folds are index-disjoint, cover the dataset exactly, and share zero normalized reduced formulas between any fold's training and test splits. (3) The improvement decomposes arithmetically: the metal classifier agrees with the true label about 92% of the time, metals then carry about 0.15 eV of residual error over 53% of the data and non-metals about 0.45 eV over the rest, and 0.53*0.15 + 0.47*0.45 reconstructs the validation score.\n\nWHERE THE GAIN COMES FROM, AND WHAT IT COSTS: Relative to a plain boosted-tree ensemble on the same data, the gate improves metals (MAE 0.241 to 0.145) and the expanded feature set improves wide-gap insulators above 4 eV (1.486 to 1.069), but the soft gate measurably HURTS compounds with true gaps between 0 and 1 eV (0.228 to 0.295). The headline improvement is therefore concentrated in the zero-gap population rather than spread evenly across the chemistry, and we think that is the most useful thing to know about this entry.\n\nLIMITATIONS: Composition-only, so polymorphs are indistinguishable and some remaining error is an irreducible floor in the data itself, which we have not quantified. The model provides no uncertainty estimates. Only matbench_expt_gap was run; nothing here demonstrates transfer to other tasks.",
+  "requirements": {
+    "python": [
+      "matbench==0.6",
+      "matminer==0.10.1",
+      "pymatgen==2026.5.4",
+      "scikit-learn==1.9.0",
+      "xgboost==3.2.0",
+      "lightgbm==4.7.0",
+      "pandas==2.3.3",
+      "numpy==2.4.6",
+      "pyarrow==25.0.0"
+    ]
+  }
+}

benchmarks/matbench_v0.1_zoey_metalgate/model_selection.py

@@ -0,0 +1,97 @@
+"""
+How the Zoey-MetalGate configuration was chosen, WITHOUT touching test data.
+
+This file documents and reproduces the selection procedure that produced the
+blend in run.py. It never loads test targets: every score computed here comes
+from internal cross-validation inside the official TRAINING folds.
+
+PROCEDURE
+---------
+Stage 1 "screen": each candidate configuration is scored by internal 5-fold CV
+on the training split of official fold 0 only. This is a cheap ranking pass;
+103 configurations were screened across two campaigns (XGBoost, LightGBM and
+CatBoost hyperparameters, three feature-set sizes, plain regressors, seed-bagged
+regressors, and metal-gated two-stage models with soft and hard gates).
+
+Stage 2 "confirm": the best configurations are re-scored by internal 5-fold CV
+inside EACH of the five official training folds (25 model fits per config), and
+their out-of-fold predictions are saved. A family-diversity cap (at most four
+configurations per model family) keeps the candidate pool from collapsing onto
+near-identical models.
+
+Stage 3 "blend": greedy forward selection with replacement over the confirmed
+out-of-fold predictions, minimising mean out-of-fold MAE across the five
+training folds. This chose three metal-gated models with weights 0.5/0.25/0.25.
+
+Only after all of the above was the finalist frozen to a file and evaluated
+once on the official test splits by run.py.
+
+VALIDATION SCORES THAT DROVE THE CHOICE (internal CV, training folds only)
+-------------------------------------------------------------------------
+    plain LightGBM, 493 features .................... 0.3465
+    plain XGBoost, 493 features ..................... 0.3596
+    metal-gated, hard gate, LightGBM regressor ...... 0.2965
+    metal-gated, hard gate, XGBoost regressor ....... 0.2963
+    metal-gated, soft gate, XGBoost regressor ....... 0.2915
+    three-member blend (submitted) .................. 0.2903
+
+For reference, the corresponding official test score of the submitted blend is
+0.2775, and of the plain LightGBM 0.3322. The gap between validation and test
+is consistent with training on the full fold rather than 4/5 of it, an effect
+that was measured on an earlier model before this campaign ran.
+
+WHY THE GATE HELPS, ARITHMETICALLY
+----------------------------------
+On out-of-fold predictions the classifier agrees with the true metal/nonmetal
+label about 92% of the time. Metals (53% of rows) then carry roughly 0.15 eV of
+residual error and non-metals (47%) roughly 0.45 eV, and
+0.53 * 0.15 + 0.47 * 0.45 is about 0.29, which is the validation score above.
+The improvement is concentrated in the metal population and in wide-gap
+insulators; the soft gate slightly HURTS compounds with true gaps in the
+0 to 1 eV range, which is the honest cost of the mechanism.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+from matbench.bench import MatbenchBenchmark
+from sklearn.model_selection import KFold
+
+from run import BLEND, build_features
+
+CV_SEED = 18012019
+N_INTERNAL_FOLDS = 5
+
+
+def internal_cv_score():
+    """Reproduce the validation score of the submitted blend. No test data."""
+    mb = MatbenchBenchmark(autoload=False, subset=["matbench_expt_gap"])
+    task = mb.tasks_map["matbench_expt_gap"]
+    task.load()
+    features = build_features(task.df["composition"])
+
+    fold_maes = []
+    for fold in task.folds:
+        X_train, y_train = task.get_train_and_val_data(fold)
+        X = features.loc[X_train.index].values
+        y = np.asarray(y_train.values, dtype=float)
+
+        oof = np.zeros(len(y))
+        kf = KFold(n_splits=N_INTERNAL_FOLDS, shuffle=True, random_state=CV_SEED)
+        for inner_train, inner_val in kf.split(X):
+            pred = np.zeros(len(inner_val))
+            for weight, make_model in BLEND:
+                model = make_model().fit(X[inner_train], y[inner_train])
+                pred += weight * model.predict(X[inner_val])
+            oof[inner_val] = pred
+
+        mae = float(np.mean(np.abs(oof - y)))
+        fold_maes.append(mae)
+        print(f"official training fold {fold}: internal-CV MAE {mae:.4f}")
+
+    print(f"\nmean internal-CV MAE: {np.mean(fold_maes):.4f} "
+          f"(+/- {np.std(fold_maes):.4f})")
+
+
+if __name__ == "__main__":
+    internal_cv_score()