Add gated OpenBenchmarking.org publish path for the engine benchmark suite - #1067
Merged
Merged
Conversation
…ds, bad --smoke arg Each of the 8 skainet-engine-* PTS test profiles was silently broken and had never actually been executed by real Phoronix Test Suite before now (CI only ever validated XML shape or ran the jar directly, bypassing PTS's own install/execute path): - install.sh wrote the wrapper executable as `engine-<scenario>`, but PTS's fallback executable-resolution convention looks for a file matching the profile identifier, `skainet-engine-<scenario>` — every profile installed fine but PTS could never find anything to run. - <Executable> was declared under the wrong XML parent (TestProfile instead of TestInformation), so it was silently ignored by the schema/parser rather than fixing the mismatch above. - <SoftwareType>Library</SoftwareType>, <License>MIT</License>, and <Status>Testing</Status> aren't valid enum values in the current PTS test-profile schema; validate-test-profile logs these as warnings but still exits 0, so validate_pts_profiles.sh never caught them. - Default/Option arguments passed `--smoke false`, but --smoke is a bare boolean flag in the harness CLI (kotlinx-cli) — passing "false" as a positional argument made every full-mode invocation fail immediately. - The wrapper only printed its result line to stdout, but PTS's results parser exclusively reads from the file at $LOG_FILE (an env var PTS passes to the child process) — it never inspects captured stdout for parsing. Wrapper now tees its output to $LOG_FILE. - Dropped the redundant Mode (Full/Smoke) option from the 5 profiles that had it: smoke mode is already served directly by run_engine_smoke.sh, and exposing it as a PTS-selectable option only added an unwanted extra result when combined with RunAllTestCombinations. Verified by actually installing and running each fixed profile through real `phoronix-test-suite benchmark`/`batch-benchmark` (not just the JSON harness) — this surfaced all of the above one at a time.
…suite scripts/upload_openbenchmarking.sh: validates a full-mode JSON run against the BenchmarkRecord schema, refuses to proceed if any record is "unstable": true (CoV over the manifest's stability limit), re-runs the suite through PTS itself in non-interactively-configured batch mode (PTS needs its own native result file to upload — the JSON harness output is a parallel record, not a PTS result), then uploads. The actual upload only runs with --confirm/CONFIRM_PUBLISH=yes; without it the script prints what it would do and exits, since publishing is public and effectively irreversible. Wired as the last step of the full-self-hosted CI job, gated on a manual workflow_dispatch run with a new confirm_publish input — a release publish event alone still runs benchmarks and uploads the JSON artifact, but never triggers a publish to OpenBenchmarking.org on its own. OpenBenchmarking.org login is intentionally left to a one-time, interactive `phoronix-test-suite openbenchmarking-login` on the bench host — the login flow does a live account handshake, so there's no static credential this script could take as a secret. Verified locally end-to-end (schema/stability gate, PTS batch-mode config, non-interactive batch-benchmark run, saved result ready for upload-result) short of the final live upload, which needs real OpenBenchmarking.org credentials on the actual bench host.
- benchmarks.adoc: cross-link the publish section from Lanes, explain why the OpenBenchmarking.org login stays a manual one-time step (no static credential exists to hand CI), and link to register-bench-runner.adoc for the host-setup side of the same step. - register-bench-runner.adoc: add the openbenchmarking-login step to the one-time operator setup (prerequisites + a new step 5), with the gh workflow run ... -f confirm_publish=true invocation. - contributing/index.adoc: mention publishing explicitly in the benchmarks page's one-line summary so it's discoverable from the section index.
michalharakal
marked this pull request as draft
August 23, 2026 14:15
|
📖 Documentation Preview The documentation has been built successfully for this PR. Generated Files:
Artifacts:
This comment will be updated automatically when the PR is updated. |
michalharakal
changed the title
michalharakal
marked this pull request as ready for review
August 23, 2026 14:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
skainet-engine-*PTS test profiles (unresolvable install executable, invalidtest-profile.xsdmetadata, a--smoke falsearg the harness CLI can't parse, missing$LOG_FILEoutput) — none of these profiles had ever actually been executed by real Phoronix Test Suite before this branch; CI only validated XML shape or ran the jar directly.scripts/upload_openbenchmarking.sh: validates a full-mode JSON run against theBenchmarkRecordschema, blocks on any"unstable": truerecord, runs the suite through PTS's own batch mode, then uploads — the actual upload only fires with--confirm/CONFIRM_PUBLISH=yes.full-self-hostedCI job (workflow_dispatchwithconfirm_publish: true) — areleaseevent alone still runs benchmarks and uploads the JSON artifact, but never publishes to the public leaderboard on its own.benchmarks.adoc,register-bench-runner.adoc,contributing/index.adoc) to document the one-time OpenBenchmarking.org login and the publish steps.Note: the
<License>MIT</License>→<License>Free</License>change in eachtest-definition.xmlis a PTS test-profile metadata field fix, not a change to SKaiNET's project license (still MIT) —MITisn't a valid value in PTS's own{Free, Non-Free, Retail, Restricted}enum.Test plan
:skainet-backends:benchmarks:jvm-cpu-publish:shadowJarand ran the full engine suite locally (11/11 scenarios produced validBenchmarkRecordJSON)scripts/validate_pts_profiles.shpasses with zero schema errors (previously 32 silent errors across the 8 profiles)phoronix-test-suite benchmark/batch-benchmark(not just the JSON harness) — confirmed results are now correctly parsed and savedscripts/upload_openbenchmarking.shexercised end-to-end locally through the PTS batch-run stage (schema/stability gate, non-interactive batch config, saved result ready forupload-result) — stopped short of the live upload, which needs real OpenBenchmarking.org credentials on the actual self-hosted bench hostphoronix-test-suite openbenchmarking-loginis configured there🤖 Generated with Claude Code