Skip to content

Add AlchemBERT results and fix classification scoring compatibility - #358

Open
liuxiaotong15 wants to merge 7 commits into
materialsproject:mainfrom
liuxiaotong15:main
Open

Add AlchemBERT results and fix classification scoring compatibility#358
liuxiaotong15 wants to merge 7 commits into
materialsproject:mainfrom
liuxiaotong15:main

Conversation

@liuxiaotong15

Copy link
Copy Markdown

Benchmark submissions

Brief description of your algorithm

We test BERT on all tasks of Matbench.

Included files

-- benchmarks
---- matbench_v0.1_AlchemBERT
------ results.json.gz
------ readme.py              # please visit our repo to get all codes
------ info.json                 

@liuxiaotong15
liuxiaotong15 force-pushed the main branch 2 times, most recently from c1b0b0c to 053d026 Compare December 11, 2024 23:26
Fix ROC-AUC scoring for probability predictions
Keep ROC‑AUC probabilities and accept numpy types in classifier helpers; bump GH Actions artifact actions to v4
Keep ROC-AUC probabilities, accept numpy types in clf homogenization, and bump upload/download-artifact to v4
@liuxiaotong15 liuxiaotong15 changed the title add AlchemBERT results #new_benchmark Add AlchemBERT results and fix classification scoring compatibility Jul 29, 2026
@liuxiaotong15

Copy link
Copy Markdown
Author

A note on the scope of this PR:

This PR was originally opened as an AlchemBERT benchmark submission. While
bringing the branch up to date and getting the submission checks to pass, it
also became necessary to include several compatibility and scoring fixes:

  1. score_array() now preserves the original probability predictions for
    ROC-AUC while using separately thresholded labels for accuracy, balanced
    accuracy, and F1. This fixes the issue described in [MUST FIX] How does ROCAUC work in score_array()? #137 and follows up on
    the classification probability discussion in Error when recording group probability results to classification tasks #40.

  2. The classification helpers and tests now support NumPy scalar types and
    NumPy 2. The removed np.float_ alias was replaced by dtype-category checks
    using np.issubdtype.

  3. The coverage artifact workflow now uses
    actions/upload-artifact@v4 and actions/download-artifact@v4, because
    GitHub rejects the deprecated v3 actions.

I considered splitting the benchmark submission and core compatibility fixes
into separate PRs. However, the original standalone AlchemBERT branch no
longer passes against the repository's current dependency and GitHub Actions
environment. I have therefore kept the changes together in this PR so the
submission remains reproducible and all required checks can pass.

Current checks:

  • test (3.9): passed
  • benchmark_submission: passed
  • code_coverage: passed

The ROC-AUC change affects future scoring and does not rewrite historical
leaderboard artifacts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant