ci: publish the benchmark table as a reproducible artifact ( - #145
Merged
shivamlalakiya merged 5 commits intoAug 24, 2026
Merged
Conversation
…oducible artifact ( Signed-off-by: Mr-Neutr0n <64578610+Mr-Neutr0n@users.noreply.github.com>
Contributor
Author
|
CI caught the golden-file comparison being too strict: the same commit produced 0.732 vs 0.731 in third decimals across Python versions and macOS, because the metrics come from fitted sklearn models. Replaced the exact string equality with a structural comparison: the non-numeric skeleton must match exactly and every metric must stay within 0.01 of the committed table (45e4ed6). Real drift still fails; build noise no longer does. |
Contributor
Author
|
The 0.01 flat tolerance was still too tight for py3.9/3.10: older sklearn shifts a seed range endpoint by 0.013 (0.752 vs 0.765) while the means stay within 0.001. Split the bands: means within 0.02, range endpoints within 0.05, skeleton still exact (d984740). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7
Connected
scripts/benchmark_models.pyto a committed Linux golden file via a pytest regression test, removed the deadRANDOM_STATEconstant, and documented the check; the issue’s separate.github/workflowsbenchmark job was not added because workflow edits were forbidden.Local tests pass.
This change was prepared with AI assistance under human direction and review.