Skip to content

feat: SPEED-Bench and GreenSpec arms on aiperf #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Noridom1 wants to merge 8 commits into from
Closed

feat: SPEED-Bench and GreenSpec arms on aiperf #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
0292096
feat(speedbench): shared SPEED-Bench client on aiperf
Sep 29, 2026
4f0bd06
feat(greenspec): GreenSpec arm on aiperf
Noridom1 Oct 28, 2026
60c5a2f
fix(greenspec): no preset for a lite bucket selection
Noridom1 Oct 28, 2026
0c6498a
docs(greenspec): GreenSpec procedures
Noridom1 Oct 28, 2026
8e8d434
feat(greenspec): progress lines in the job log
Noridom1 Oct 28, 2026
02e9561
feat(greenspec): stream aiperf's console to the job log, filtered
Noridom1 Oct 28, 2026
611667b
fix(greenspec): swap the server log stub in, don't truncate in place
Noridom1 Oct 28, 2026
4b28842
fix(greennode): keep RUN_ENV comment free of parens
Noridom1 Oct 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/workflows/benchmark-tmpl.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@
# recipe ignores it, and the SPEED-Bench path falls back to HF_TOKEN.
SPEEDBENCH_HF_TOKEN:
required: false
# Read-only token for the private GreenSpec repo pulled by
# resolve_greenspec_dataset (benchmark_lib.sh). Optional, falls back to HF_TOKEN.
GREENSPEC_HF_TOKEN:
required: false
inputs:
config:
description: "Benchmark configuration as JSON"
Expand Down Expand Up @@ -169,6 +173,7 @@
HF_TOKEN: ${{ secrets.INFERENCEX_OFFICIAL_RO_HF_TOKEN }}
HF_HUB_CACHE: '/mnt/hf_hub_cache/'
SPEEDBENCH_HF_TOKEN: ${{ secrets.SPEEDBENCH_HF_TOKEN }}
GREENSPEC_HF_TOKEN: ${{ secrets.GREENSPEC_HF_TOKEN }}
EXP_NAME: ${{ fromJSON(inputs.config).exp-name }}
RECIPE_FINGERPRINT: ${{ fromJSON(inputs.config).recipe-fingerprint || '' }}
MODEL: ${{ fromJSON(inputs.config).model }}
Expand Down Expand Up @@ -227,11 +232,11 @@
# scenario in one run). scenario-map is keyed by the row's exp-name; the flat
# serving.scenario is the single-scenario fallback, empty for scenario lists
# so a missing map entry fails loudly in serve_generic.sh (check_env_vars).
SERVING_SCENARIO: ${{ inputs.serving != '' && (fromJSON(inputs.serving).scenario-map[fromJSON(inputs.config).exp-name].scenario || fromJSON(inputs.serving).scenario) || '' }}

Check warning on line 235 in .github/workflows/benchmark-tmpl.yml

View workflow job for this annotation

GitHub Actions / Zizmor

obfuscation 

benchmark-tmpl.yml:235: obfuscated usage of GitHub Actions features: index expression is computed
SERVING_LOADER: ${{ inputs.serving != '' && (fromJSON(inputs.serving).scenario-map[fromJSON(inputs.config).exp-name].loader || fromJSON(inputs.serving).loader) || '' }}

Check warning on line 236 in .github/workflows/benchmark-tmpl.yml

View workflow job for this annotation

GitHub Actions / Zizmor

obfuscation 

benchmark-tmpl.yml:236: obfuscated usage of GitHub Actions features: index expression is computed
# '1' for sweep scenarios (adaptive_search), '0' otherwise — never '': check_env_vars
# rejects empty values, so '' would abort every non-sweep serving run in serve_generic.sh.
SERVING_SEARCH: ${{ inputs.serving != '' && ((fromJSON(inputs.serving).scenario-map[fromJSON(inputs.config).exp-name].search || fromJSON(inputs.serving).search) && '1' || '0') }}

Check warning on line 239 in .github/workflows/benchmark-tmpl.yml

View workflow job for this annotation

GitHub Actions / Zizmor

obfuscation 

benchmark-tmpl.yml:239: obfuscated usage of GitHub Actions features: index expression is computed
SERVING_YAML: ${{ inputs.serving != '' && fromJSON(inputs.serving).yaml || '' }}
# Router sidecar (any router, spec 2026-10-14); only runners/launch_hardware-hcm.sh reads these.
ROUTER_IMAGE: ${{ inputs.serving != '' && fromJSON(inputs.serving).router-image || '' }}
Expand Down Expand Up @@ -261,7 +266,7 @@

jobs:
benchmark:
runs-on: >-

Check warning on line 269 in .github/workflows/benchmark-tmpl.yml

View workflow job for this annotation

GitHub Actions / Zizmor

self-hosted-runner 

benchmark-tmpl.yml:269: runs on a self-hosted runner: expression may expand into a self-hosted runner
${{ fromJSON(
vars.PRIORITY_SCHEDULER_ENABLED == 'true' &&
(
Expand Down Expand Up @@ -535,6 +540,23 @@
!results/speedbench_aiperf/**/inputs.json
if-no-files-found: ignore

# GreenSpec recipes (run_greenspec_aiperf) hold internal prod traffic. The
# client's guard already pruned results/greenspec_aiperf to aggregate files
# and checked them for row ids; upload by allowlist all the same, so a file
# a future aiperf adds never ships by default.
- name: Upload GreenSpec aiperf artifacts
if: ${{ always() && inputs.scenario-type != 'agentic-coding' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: greenspec_aiperf_${{ env.RESULT_FILENAME }}
path: |
results/greenspec_aiperf/profile_export_aiperf.json
results/greenspec_aiperf/result.json
results/greenspec_aiperf/spec_metrics_before.json
results/greenspec_aiperf/spec_metrics_after.json
results/greenspec_aiperf/benchmark_command.txt
if-no-files-found: ignore

- name: Upload GPU metrics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/e2e-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,8 @@ on: # zizmor: ignore[concurrency-limits]
required: false
SPEEDBENCH_HF_TOKEN:
required: false
GREENSPEC_HF_TOKEN:
required: false
inputs:
generate-cli-command:
description: "Command passed to generate matrix script"
Expand Down Expand Up @@ -438,6 +440,7 @@ jobs:
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
SPEEDBENCH_HF_TOKEN: ${{ secrets.SPEEDBENCH_HF_TOKEN }}
GREENSPEC_HF_TOKEN: ${{ secrets.GREENSPEC_HF_TOKEN }}
with:
config: ${{ toJSON(matrix.config) }}
serving: ${{ needs.get-jobs.outputs.serving }}
Expand Down Expand Up @@ -468,6 +471,7 @@ jobs:
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
SPEEDBENCH_HF_TOKEN: ${{ secrets.SPEEDBENCH_HF_TOKEN }}
GREENSPEC_HF_TOKEN: ${{ secrets.GREENSPEC_HF_TOKEN }}
with:
config: ${{ toJSON(matrix.config) }}
klaud-run: ${{ inputs.klaud-run }}
Expand Down Expand Up @@ -566,6 +570,7 @@ jobs:
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
SPEEDBENCH_HF_TOKEN: ${{ secrets.SPEEDBENCH_HF_TOKEN }}
GREENSPEC_HF_TOKEN: ${{ secrets.GREENSPEC_HF_TOKEN }}
with:
config: ${{ toJSON(matrix.config) }}
klaud-run: ${{ inputs.klaud-run }}
Expand All @@ -592,6 +597,7 @@ jobs:
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
SPEEDBENCH_HF_TOKEN: ${{ secrets.SPEEDBENCH_HF_TOKEN }}
GREENSPEC_HF_TOKEN: ${{ secrets.GREENSPEC_HF_TOKEN }}
with:
config: ${{ toJSON(matrix.config) }}
klaud-run: ${{ inputs.klaud-run }}
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/run-sweep.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -434,6 +434,7 @@ jobs:
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
SPEEDBENCH_HF_TOKEN: ${{ secrets.SPEEDBENCH_HF_TOKEN }}
GREENSPEC_HF_TOKEN: ${{ secrets.GREENSPEC_HF_TOKEN }}
with:
config: ${{ toJSON(matrix.config) }}
# Only same-repository PRs owned by Klaud receive background priority.
Expand Down Expand Up @@ -525,6 +526,7 @@ jobs:
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
SPEEDBENCH_HF_TOKEN: ${{ secrets.SPEEDBENCH_HF_TOKEN }}
GREENSPEC_HF_TOKEN: ${{ secrets.GREENSPEC_HF_TOKEN }}
with: &single-node-inputs
config: ${{ toJSON(matrix.config) }}
klaud-run: *klaud-run
Expand Down Expand Up @@ -559,6 +561,7 @@ jobs:
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
SPEEDBENCH_HF_TOKEN: ${{ secrets.SPEEDBENCH_HF_TOKEN }}
GREENSPEC_HF_TOKEN: ${{ secrets.GREENSPEC_HF_TOKEN }}
with: *single-node-inputs

sweep-agentic:
Expand All @@ -583,6 +586,7 @@ jobs:
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
SPEEDBENCH_HF_TOKEN: ${{ secrets.SPEEDBENCH_HF_TOKEN }}
GREENSPEC_HF_TOKEN: ${{ secrets.GREENSPEC_HF_TOKEN }}
with:
config: ${{ toJSON(matrix.config) }}
klaud-run: *klaud-run
Expand Down Expand Up @@ -656,6 +660,7 @@ jobs:
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
SPEEDBENCH_HF_TOKEN: ${{ secrets.SPEEDBENCH_HF_TOKEN }}
GREENSPEC_HF_TOKEN: ${{ secrets.GREENSPEC_HF_TOKEN }}
with:
config: ${{ toJSON(matrix.config) }}
klaud-run: *klaud-run
Expand Down Expand Up @@ -695,6 +700,7 @@ jobs:
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
SPEEDBENCH_HF_TOKEN: ${{ secrets.SPEEDBENCH_HF_TOKEN }}
GREENSPEC_HF_TOKEN: ${{ secrets.GREENSPEC_HF_TOKEN }}
with:
config: ${{ toJSON(matrix.config) }}
klaud-run: *klaud-run
Expand Down
3 changes: 2 additions & 1 deletion .gitmodules
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[submodule "utils/aiperf"]
path = utils/aiperf
url = https://github.com/SemiAnalysisAI/aiperf.git
url = https://github.com/Noridom1/aiperf.git
branch = feat/greenspec-loader
[submodule "utils/srt-slurm"]
path = utils/srt-slurm
url = https://github.com/NVIDIA/srt-slurm.git
Loading
Loading