Skip to content

Update a few techreport v2 plots #54

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
priyakasimbeg merged 2 commits into from
Sep 10, 2026
Merged

Update a few techreport v2 plots #54

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cbd1f4b
[algoperf] Update plotting code
andyliu14 Aug 31, 2026
d4c8ab8
[ruff] fix format
andyliu14 Aug 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions ...ts/tech_report_v1/section_4_leaderboards/out/byproducts/newly_reached_targets_relaxed.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
Submission,criteo1tb,fastmri,finewebedu_lm,imagenet_resnet,imagenet_vit,librispeech_conformer,librispeech_deepspeech,ogbg,wmt
AdEMAMix (PyTorch),False,False,False,False,False,False,True,False,False
Cautious NAdamW (JAX),True,False,False,True,True,False,False,False,False
Lion (PyTorch),True,False,False,False,False,False,True,False,False
Muon (JAX),True,False,False,True,False,False,False,True,True
Muon (PyTorch),False,True,False,False,False,True,False,False,False
NAdamW (JAX),False,False,False,True,False,False,False,True,True
NAdamW (Baseline AlgoPerf v0.5) (JAX),False,False,False,True,True,False,False,False,False
NAdamW (Tuned for ResNet) (JAX),True,False,False,True,True,False,False,False,True
Schedule-Free AdamW (PyTorch),False,False,False,False,False,True,False,False,False
Schedule-Free AdamW (JAX),True,False,False,False,False,False,False,False,False
Schedule-Free AdamW v2 (JAX),False,False,False,False,False,False,False,False,True
Schedule-Free AdamW v2 (PyTorch),False,False,True,False,False,False,False,False,False
Single Worker DiLoCo (JAX),True,False,False,True,True,False,False,True,True
Single Worker DiLoCo v2 (JAX),True,False,False,True,False,False,False,True,True
15 changes: 15 additions & 0 deletions ...facts/tech_report_v1/section_4_leaderboards/out/byproducts/scores_standard_vs_relaxed.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
submission,standard_score,relaxed_score,standard_rank,relaxed_rank,rank_shift
Schedule-Free AdamW v2 (PyTorch),0.5679012345679016,0.607182940516274,1,1,0
NAdamW (Baseline AlgoPerf v0.5) (JAX),0.4506172839506181,0.6010101010101013,4,2,2
Muon (PyTorch),0.4231200897867556,0.5993265993265987,6,3,3
Schedule-Free AdamW (PyTorch),0.466891133557801,0.5785634118967448,3,4,-1
Schedule-Free AdamW v2 (JAX),0.4405162738496074,0.5740740740740736,5,5,0
Schedule-Free AdamW (JAX),0.4107744107744111,0.5561167227833891,7,6,1
Muon (JAX),0.2845117845117842,0.5521885521885517,10,7,3
AdEMAMix (PyTorch),0.5190796857463527,0.5213243546576883,2,8,-6
Cautious NAdamW (JAX),0.31425364758698043,0.5089786756453424,9,9,0
NAdamW (JAX),0.384399551066217,0.4814814814814808,8,10,-2
Lion (PyTorch),0.27833894500561146,0.4438832772166114,11,11,0
NAdamW (Tuned for ResNet) (JAX),0.18574635241301896,0.4343434343434347,12,12,0
Single Worker DiLoCo (JAX),0.1369248035914701,0.42592592592592626,13,13,0
Single Worker DiLoCo v2 (JAX),0.1301907968574634,0.3232323232323228,14,14,0
38 changes: 20 additions & 18 deletions artifacts/tech_report_v1/section_4_leaderboards/out/results/scores_relaxed_table.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,27 @@
\begin{table}[h]
\begin{table}[htbp]
\centering
\caption{AlgoPerf Self-Tuning Leaderboard (targets relaxed 10%)}
\caption{Standard vs.\ 10\% relaxed-target AlgoPerf self-tuning leaderboard. $\Delta$ is the rank change after relaxing targets.}
\label{tab:scores_relaxed}
\begin{tabular}{rlr}
\begin{tabular}{lrrrrc}
\toprule
Rank & Submission & Score \\
& \multicolumn{2}{c}{Standard} & \multicolumn{2}{c}{10\% relaxed} & \\
\cmidrule(lr){2-3}\cmidrule(lr){4-5}
Submission & Score & Rank & Score & Rank & $\Delta$ \\
\midrule
1 & \sfadamwii & \textbf{0.6072} \\
2 & \nadamwbase & 0.6010 \\
3 & \muonpt & 0.5993 \\
4 & \sfadamw & 0.5786 \\
5 & \sfadamwjaxii & 0.5741 \\
6 & \sfadamwjax & 0.5561 \\
7 & \muonjax & 0.5522 \\
8 & \ademamix & 0.5213 \\
9 & \cautiousnadamw & 0.5090 \\
10 & \nadamw & 0.4815 \\
11 & \lion & 0.4439 \\
12 & \nadamwresnet & 0.4343 \\
13 & \dilocosw & 0.4259 \\
14 & \dilocoswii & 0.3232 \\
\sfadamwii & \textbf{0.5679} & 1 & \textbf{0.6072} & 1 & -- \\
\nadamwbase & 0.4506 & 4 & 0.6010 & 2 & $\uparrow$2 \\
\muonpt & 0.4231 & 6 & 0.5993 & 3 & $\uparrow$3 \\
\sfadamw & 0.4669 & 3 & 0.5786 & 4 & $\downarrow$1 \\
\sfadamwjaxii & 0.4405 & 5 & 0.5741 & 5 & -- \\
\sfadamwjax & 0.4108 & 7 & 0.5561 & 6 & $\uparrow$1 \\
\muonjax & 0.2845 & 10 & 0.5522 & 7 & $\uparrow$3 \\
\ademamix & 0.5191 & 2 & 0.5213 & 8 & $\downarrow$6 \\
\cautiousnadamw & 0.3143 & 9 & 0.5090 & 9 & -- \\
\nadamw & 0.3844 & 8 & 0.4815 & 10 & $\downarrow$2 \\
\lion & 0.2783 & 11 & 0.4439 & 11 & -- \\
\nadamwresnet & 0.1857 & 12 & 0.4343 & 12 & -- \\
\dilocosw & 0.1369 & 13 & 0.4259 & 13 & -- \\
\dilocoswii & 0.1302 & 14 & 0.3232 & 14 & -- \\
\bottomrule
\end{tabular}
\end{table}
2 changes: 1 addition & 1 deletion artifacts/tech_report_v1/section_4_leaderboards/out/results/time_to_target_table.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
\begin{table}[htbp]
\centering
\caption{Time to target as a fraction of each workload's self-tuning-ruleset time budget (1.5\times the external-tuning maximum allowed runtime). \textemdash{} = target not reached.}
\caption{Time to target as a fraction of each workload's self-tuning-ruleset time budget ($1.5\times$ the external-tuning maximum allowed runtime). \textemdash{} = target not reached.}
\label{tab:time_to_target}
\resizebox{\textwidth}{!}{%
\begin{tabular}{lrrrrrrrrr}
Expand Down
30 changes: 15 additions & 15 deletions artifacts/tech_report_v1/section_4_leaderboards/out/results/time_to_target_table_relaxed.tex
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,26 @@
\begin{table}[htbp]
\centering
\caption{Time to target with targets relaxed 10%, as a fraction of each workload's self-tuning-ruleset time budget (1.5\times the external-tuning maximum allowed runtime). \textemdash{} = target not reached.}
\caption{Time to target with targets relaxed 10\%, as a fraction of each workload's self-tuning-ruleset time budget ($1.5\times$ the external-tuning maximum allowed runtime). \textbf{Bold} values mark targets reached only after relaxation; \textemdash{} = target not reached.}
\label{tab:time_to_target_relaxed}
\resizebox{\textwidth}{!}{%
\begin{tabular}{lrrrrrrrrr}
\toprule
Submission & \criteo & \fastmri & \finewebedu & \resnet & \vit & \conformer & \deepspeech & \ogbg & \wmt \\
\midrule
\sfadamwii & 0.03 & 0.06 & 0.59 & \textemdash{} & \textemdash{} & 0.61 & 0.42 & 0.16 & 0.06 \\
\nadamwbase & 0.03 & 0.07 & 0.32 & 0.43 & 0.56 & \textemdash{} & \textemdash{} & 0.13 & 0.11 \\
\muonpt & 0.03 & 0.08 & 0.22 & \textemdash{} & \textemdash{} & 0.72 & \textemdash{} & 0.08 & 0.08 \\
\sfadamw & 0.03 & 0.08 & \textemdash{} & \textemdash{} & \textemdash{} & 0.72 & 0.42 & 0.13 & 0.08 \\
\sfadamwjaxii & 0.03 & 0.07 & 0.21 & \textemdash{} & 0.27 & \textemdash{} & \textemdash{} & 0.13 & 0.08 \\
\sfadamwjax & 0.03 & 0.09 & 0.16 & \textemdash{} & 0.27 & \textemdash{} & \textemdash{} & 0.16 & 0.08 \\
\muonjax & 0.05 & 0.08 & 0.32 & 0.40 & 0.48 & \textemdash{} & \textemdash{} & 0.05 & 0.27 \\
\ademamix & 0.03 & 0.08 & 0.32 & \textemdash{} & \textemdash{} & 0.67 & 0.47 & 0.19 & 0.24 \\
\cautiousnadamw & 0.03 & 0.07 & 0.48 & 0.43 & 0.59 & \textemdash{} & \textemdash{} & 0.16 & 0.16 \\
\nadamw & 0.03 & 0.06 & 0.48 & 0.45 & 0.51 & \textemdash{} & \textemdash{} & 0.16 & 0.27 \\
\lion & 0.03 & 0.07 & 0.38 & \textemdash{} & \textemdash{} & \textemdash{} & 0.44 & 0.27 & 0.13 \\
\nadamwresnet & 0.03 & 0.07 & 0.38 & 0.43 & 0.77 & \textemdash{} & \textemdash{} & 0.21 & 0.27 \\
\dilocosw & 0.05 & 0.08 & 0.48 & 0.45 & 0.72 & \textemdash{} & \textemdash{} & 0.16 & 0.19 \\
\dilocoswii & 0.03 & 0.09 & 0.54 & 0.61 & \textemdash{} & \textemdash{} & \textemdash{} & 0.27 & 0.27 \\
\sfadamwii & 0.03 & 0.06 & \textbf{0.59} & \textemdash{} & \textemdash{} & 0.61 & 0.42 & 0.16 & 0.06 \\
\nadamwbase & 0.03 & 0.07 & 0.32 & \textbf{0.43} & \textbf{0.56} & \textemdash{} & \textemdash{} & 0.13 & 0.11 \\
\muonpt & 0.03 & \textbf{0.08} & 0.22 & \textemdash{} & \textemdash{} & \textbf{0.72} & \textemdash{} & 0.08 & 0.08 \\
\sfadamw & 0.03 & 0.08 & \textemdash{} & \textemdash{} & \textemdash{} & \textbf{0.72} & 0.42 & 0.13 & 0.08 \\
\sfadamwjaxii & 0.03 & 0.07 & 0.21 & \textemdash{} & 0.27 & \textemdash{} & \textemdash{} & 0.13 & \textbf{0.08} \\
\sfadamwjax & \textbf{0.03} & 0.09 & 0.16 & \textemdash{} & 0.27 & \textemdash{} & \textemdash{} & 0.16 & 0.08 \\
\muonjax & \textbf{0.05} & 0.08 & 0.32 & \textbf{0.40} & 0.48 & \textemdash{} & \textemdash{} & \textbf{0.05} & \textbf{0.27} \\
\ademamix & 0.03 & 0.08 & 0.32 & \textemdash{} & \textemdash{} & 0.67 & \textbf{0.47} & 0.19 & 0.24 \\
\cautiousnadamw & \textbf{0.03} & 0.07 & 0.48 & \textbf{0.43} & \textbf{0.59} & \textemdash{} & \textemdash{} & 0.16 & 0.16 \\
\nadamw & 0.03 & 0.06 & 0.48 & \textbf{0.45} & 0.51 & \textemdash{} & \textemdash{} & \textbf{0.16} & \textbf{0.27} \\
\lion & \textbf{0.03} & 0.07 & 0.38 & \textemdash{} & \textemdash{} & \textemdash{} & \textbf{0.44} & 0.27 & 0.13 \\
\nadamwresnet & \textbf{0.03} & 0.07 & 0.38 & \textbf{0.43} & \textbf{0.77} & \textemdash{} & \textemdash{} & 0.21 & \textbf{0.27} \\
\dilocosw & \textbf{0.05} & 0.08 & 0.48 & \textbf{0.45} & \textbf{0.72} & \textemdash{} & \textemdash{} & \textbf{0.16} & \textbf{0.19} \\
\dilocoswii & \textbf{0.03} & 0.09 & 0.54 & \textbf{0.61} & \textemdash{} & \textemdash{} & \textemdash{} & \textbf{0.27} & \textbf{0.27} \\
\bottomrule
\end{tabular}%
}
Expand Down
Binary file modified artifacts/tech_report_v1/section_4_leaderboards/out/results/wallclock_vs_steps.pdf
View file
Open in desktop
Binary file not shown.
Binary file modified artifacts/tech_report_v1/section_4_leaderboards/out/results/wallclock_vs_steps.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading