Skip to content

Add composition and composition+persistent-homology benchmark submissions - #363

Open
nkwork9999 wants to merge 1 commit into
materialsproject:mainfrom
nkwork9999:matbench-pr-submission-20260718
Open

Add composition and composition+persistent-homology benchmark submissions#363
nkwork9999 wants to merge 1 commit into
materialsproject:mainfrom
nkwork9999:matbench-pr-submission-20260718

Conversation

@nkwork9999

Copy link
Copy Markdown

Summary

Two new algorithm folders under benchmarks/, both scored on the official
Matbench v0.1 fold IDs and metrics:

  • matbench_v0.1_composition_gbmmatbench_glass (glass-forming
    ability), composition-only features (156-dim: element fractions + global
    composition statistics) with HistGradientBoostingClassifier. Mean
    official-fold ROC-AUC 0.8658 (± fold std ~0.017).
  • matbench_v0.1_composition_ph_gbmmatbench_log_gvrh and
    matbench_log_kvrh (shear/bulk modulus), the same composition features
    plus a 51-dim persistent-homology structure descriptor (H0/H1
    Vietoris-Rips diagrams on the periodic point cloud, via ripser+persim)
    with HistGradientBoostingRegressor. Mean official-fold MAE 0.0813
    (log_gvrh) and 0.0636 (log_kvrh); the PH features improved over a
    composition-only control in all 5 folds on both tasks.

Methodology notes

  • All three tasks use the official matbench_v0.1_validation.json fold IDs
    (the local environment could not build matbench==0.6's pinned
    scikit-learn==1.0.1, so results were constructed to the
    MatbenchBenchmark JSON schema directly from the raw datasets + official
    fold JSON — same fold IDs, same metric functions).
  • Hyperparameter selection used only an inner split inside each official
    training fold (never the fold's test data); the selected model was refit
    on the complete official training fold before recording test predictions.
  • Each folder's src/ is standalone and was verified to reproduce the
    recorded results.json.gz exactly (byte-identical per-fold predictions
    and scores) by running fresh from inside that directory alone.
  • Full details, feature definitions, and reproduction commands are in each
    folder's info.json and notebook.md.

Test plan

  • results.json.gz in both folders validated against the
    MatbenchBenchmark schema (gzip integrity, official fold IDs/order,
    metric keys, finite predictions).
  • src/ reproduction verified standalone (fresh dataset download +
    refit + score) to match the committed results.json.gz exactly for all
    three tasks.
  • Maintainer review / merge — flagging that I could not get the
    matbench package itself to build locally (see above), so if the schema
    needs any adjustment for merge I'm happy to iterate.

- matbench_v0.1_composition_gbm: matbench_glass, composition-only
  HistGradientBoostingClassifier (mean rocauc 0.8658).
- matbench_v0.1_composition_ph_gbm: matbench_log_gvrh and
  matbench_log_kvrh, composition + persistent-homology structure features
  with HistGradientBoostingRegressor (mean MAE 0.0813 / 0.0636).

All results use official Matbench v0.1 fold IDs and metrics. Each folder's
src/ is a standalone, verified-runnable reproduction (dataset download,
refit, and scoring reproduce the recorded results.json.gz exactly).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant