Skip to content
BranchesTags

Repository files navigation

ROCm-Agent: Agentic RL for AMD GPU Kernel Optimization

This project provides a working ROCm-native agent prototype inspired by CUDA-Agent, adapted for AMD GPUs and PyTorch ROCm builds.

It includes:

  • A HIP/PyTorch extension workspace (agent_workdir/) with a single end-to-end fused kernel path.
  • Verification and profiling tools.
  • An optimization agent that uses robust reward shaping and UCB search to select the best runtime kernel config.

Why this is different from a plain kernel demo

The implementation is agent-oriented, not just a single kernel:

  • Skill loop: compile → verify → benchmark → optimize.
  • ROCm-aware backend handling: auto-detects gcnArchName and sets PYTORCH_ROCM_ARCH.
  • Robust rewards: discrete milestone rewards plus jitter penalty.
  • Bandit search: UCB exploration over runtime kernel configurations.

Layout

ROCm-Agent/
├── CUDA_AGENT_DEEP_ANALYSIS.md
├── run_agent.py
├── rocm_agent/
│   ├── __init__.py
│   ├── agent.py
│   ├── device.py
│   ├── reward.py
│   └── search.py
└── agent_workdir/
    ├── binding.cpp
    ├── binding_registry.h
    ├── model.py
    ├── model_new.py
    ├── kernels/
    │   ├── fused_e2e_pipeline.cu
    │   └── fused_e2e_pipeline_binding.cpp
    └── utils/
        ├── __init__.py
        ├── compile.py
        ├── compile.sh
        ├── profiling.py
        └── verification.py

Requirements

  • Python with torch ROCm build (tested with torch 2.9.0 + rocm 7.2.0 via rocm/vllm-dev:rocm7.2_navi_ubuntu22.04_py3.10_pytorch_2.9_vllm_0.14.0rc0 Docker image).
  • ROCm toolchain (hipcc).
  • AMD GPU (tested with Radeon Pro W7900).

Quickstart

From ROCm-Agent/:

python3 run_agent.py

This will:

  1. Compile the HIP extension in agent_workdir/.
  2. Profile eager and torch.compile baselines.
  3. Search configs via UCB.
  4. Validate correctness.
  5. Save the selected config to agent_workdir/best_config.json.

Manual commands

cd agent_workdir
bash utils/compile.sh
python3 -m utils.verification --config-e2e 0
python3 -m utils.profiling --config-e2e 0

KernelBench ROCm Optimization Report

Generated: 2026-03-06 03:58 UTC

Scope

  • Dataset: KernelBench local repository
  • Hardware backend: ROCm via torch HIP runtime
  • Benchmark mode: full
  • Evaluated levels: 1,2,3,4
  • Evaluated tasks: 270
  • Baselines: torch eager and torch.compile
  • Custom optimized tasks: 10/270
  • Compile guard: skip compile timing when total input elements exceed 200000000
  • Correctness guard: reuse initial inputs for very large tasks; regenerate random inputs only below 50000000 elements.

Aggregate Metrics (CUDA-Agent Style)

Split #Tasks Completed Timed(E) Timed(C) Pass Rate Faster vs Eager Faster vs Compile Geomean vs Eager Geomean vs Compile
Overall 270 270 270 201 100.0% 47.8% 11.9% 1.01x 0.83x
Level-1 100 100 100 40 100.0% 46.0% 11.0% 0.99x 0.89x
Level-2 100 100 100 93 100.0% 53.0% 4.0% 1.04x 0.76x
Level-3 50 50 50 48 100.0% 40.0% 22.0% 0.99x 0.82x
Level-4 20 20 20 20 100.0% 50.0% 30.0% 1.00x 1.07x

Per-Task Results

Task Opt Kind Cfg Correct Eager (us) Compile (us) Compile Status Optimized (us) S/E S/C Error
L1-1 1_Square_matrix_multiplication_ fallback_eager - Yes 53254.67 49138.26 measured 52793.29 1.01x 0.93x -
L1-2 2_Standard_matrix_multiplication_ fallback_eager - Yes 77058.46 70538.97 measured 75935.31 1.01x 0.93x -
L1-3 3_Batched_matrix_multiplication fallback_eager - Yes 106864.93 NA skipped_large_input 105740.75 1.01x NA -
L1-4 4_Matrix_vector_multiplication_ fallback_eager - Yes 33539.81 NA skipped_large_input 32936.42 1.02x NA -
L1-5 5_Matrix_scalar_multiplication custom 1 Yes 13670.90 NA skipped_large_input 13901.82 0.98x NA -
L1-6 6_Matmul_with_large_K_dimension_ fallback_eager - Yes 19946.65 NA skipped_large_input 19690.32 1.01x NA -
L1-7 7_Matmul_with_small_K_dimension_ fallback_eager - Yes 61020.66 53020.63 measured 59242.96 1.03x 0.89x -
L1-8 8_Matmul_with_irregular_shapes_ fallback_eager - Yes 116651.14 112384.28 measured 116307.56 1.00x 0.97x -
L1-9 9_Tall_skinny_matrix_multiplication_ fallback_eager - Yes 28418.67 26672.77 measured 28016.30 1.01x 0.95x -
L1-10 10_3D_tensor_matrix_multiplication fallback_eager - Yes 18158.64 18612.39 measured 18154.20 1.00x 1.03x -
L1-11 11_4D_tensor_matrix_multiplication fallback_eager - Yes 136046.47 NA skipped_large_input 136379.01 1.00x NA -
L1-12 12_Matmul_with_diagonal_matrices_ fallback_eager - Yes 213.47 242.57 measured 222.35 0.96x 1.09x -
L1-13 13_Matmul_for_symmetric_matrices fallback_eager - Yes 55690.38 51642.14 measured 55370.66 1.01x 0.93x -
L1-14 14_Matmul_for_upper_triangular_matrices fallback_eager - Yes 51651.92 50365.08 measured 51978.12 0.99x 0.97x -
L1-15 15_Matmul_for_lower_triangular_matrices fallback_eager - Yes 54478.91 50594.34 measured 52479.04 1.04x 0.96x -
L1-16 16_Matmul_with_transposed_A fallback_eager - Yes 34605.39 30234.06 measured 33956.77 1.02x 0.89x -
L1-17 17_Matmul_with_transposed_B fallback_eager - Yes 53050.44 49534.51 measured 52352.06 1.01x 0.95x -
L1-18 18_Matmul_with_transposed_both fallback_eager - Yes 113584.77 113075.16 measured 114159.89 0.99x 0.99x -
L1-19 19_ReLU custom 1 Yes 20764.50 NA skipped_large_input 20686.38 1.00x NA -
L1-20 20_LeakyReLU fallback_eager - Yes 20754.27 NA skipped_large_input 20736.05 1.00x NA -
L1-21 21_Sigmoid fallback_eager - Yes 20788.38 NA skipped_large_input 20791.97 1.00x NA -
L1-22 22_Tanh fallback_eager - Yes 20747.79 NA skipped_large_input 20728.54 1.00x NA -
L1-23 23_Softmax fallback_eager - Yes 41178.62 NA skipped_large_input 41358.77 1.00x NA -
L1-24 24_LogSoftmax fallback_eager - Yes 41762.05 NA skipped_large_input 41652.32 1.00x NA -
L1-25 25_Swish fallback_eager - Yes 52275.10 NA skipped_large_input 52260.60 1.00x NA -
L1-26 26_GELU_ fallback_eager - Yes 20788.89 NA skipped_large_input 20767.06 1.00x NA -
L1-27 27_SELU_ fallback_eager - Yes 20771.68 NA skipped_large_input 20792.61 1.00x NA -
L1-28 28_HardSigmoid fallback_eager - Yes 20780.99 NA skipped_large_input 20798.60 1.00x NA -
L1-29 29_Softplus fallback_eager - Yes 28782.73 NA skipped_large_input 28826.24 1.00x NA -
L1-30 30_Softsign fallback_eager - Yes 72986.01 NA skipped_large_input 73043.04 1.00x NA -
L1-31 31_ELU fallback_eager - Yes 20762.52 NA skipped_large_input 20764.16 1.00x NA -
L1-32 32_HardTanh fallback_eager - Yes 20796.97 NA skipped_large_input 20786.49 1.00x NA -
L1-33 33_BatchNorm fallback_eager - Yes 13917.13 NA skipped_large_input 13946.06 1.00x NA -
L1-34 34_InstanceNorm fallback_eager - Yes 45050.45 NA skipped_large_input 45043.67 1.00x NA -
L1-35 35_GroupNorm_ fallback_eager - Yes 35759.91 NA skipped_large_input 35724.88 1.00x NA -
L1-36 36_RMSNorm_ fallback_eager - Yes 61487.08 NA skipped_large_input 61489.00 1.00x NA -
L1-37 37_FrobeniusNorm_ fallback_eager - Yes 35125.77 NA skipped_large_input 35099.64 1.00x NA -
L1-38 38_L1Norm_ fallback_eager - Yes 67557.33 NA skipped_large_input 67564.25 1.00x NA -
L1-39 39_L2Norm_ fallback_eager - Yes 40213.82 NA skipped_large_input 40189.60 1.00x NA -
L1-40 40_LayerNorm fallback_eager - Yes 3020.85 1407.87 measured 3030.23 1.00x 0.46x -
L1-41 41_Max_Pooling_1D fallback_eager - Yes 26195.10 NA skipped_large_input 27020.37 0.97x NA -
L1-42 42_Max_Pooling_2D fallback_eager - Yes 21932.49 NA skipped_large_input 24001.62 0.91x NA -
L1-43 43_Max_Pooling_3D fallback_eager - Yes 12545.83 NA skipped_large_input 12589.73 1.00x NA -
L1-44 44_Average_Pooling_1D fallback_eager - Yes 12023.25 NA skipped_large_input 12143.51 0.99x NA -
L1-45 45_Average_Pooling_2D fallback_eager - Yes 28070.74 NA skipped_large_input 29218.78 0.96x NA -
L1-46 46_Average_Pooling_3D fallback_eager - Yes 19332.80 NA skipped_large_input 20353.05 0.95x NA -
L1-47 47_Sum_reduction_over_a_dimension fallback_eager - Yes 13417.97 NA skipped_large_input 13419.33 1.00x NA -
L1-48 48_Mean_reduction_over_a_dimension fallback_eager - Yes 13440.79 NA skipped_large_input 13414.72 1.00x NA -
L1-49 49_Max_reduction_over_a_dimension fallback_eager - Yes 13459.82 NA skipped_large_input 13434.91 1.00x NA -
L1-50 50_conv_standard_2D__square_input__square_kernel fallback_eager - Yes 10067.39 9846.02 measured 10008.97 1.01x 0.98x -
L1-51 51_Argmax_over_a_dimension fallback_eager - Yes 13415.03 NA skipped_large_input 13417.20 1.00x NA -
L1-52 52_Argmin_over_a_dimension fallback_eager - Yes 13455.35 NA skipped_large_input 13422.63 1.00x NA -
L1-53 53_Min_reduction_over_a_dimension fallback_eager - Yes 13470.35 NA skipped_large_input 13439.41 1.00x NA -
L1-54 54_conv_standard_3D__square_input__square_kernel fallback_eager - Yes 9585.58 9917.24 measured 10411.06 0.92x 0.95x -
L1-55 55_conv_standard_2D__asymmetric_input__square_kernel fallback_eager - Yes 14664.80 NA skipped_large_input 17201.22 0.85x NA -
L1-56 56_conv_standard_2D__asymmetric_input__asymmetric_kernel fallback_eager - Yes 15537.94 15748.15 measured 17191.82 0.90x 0.92x -
L1-57 57_conv_transposed_2D__square_input__square_kernel fallback_eager - Yes 15356.35 NA skipped_large_input 18087.39 0.85x NA -
L1-58 58_conv_transposed_3D__asymmetric_input__asymmetric_kernel fallback_eager - Yes 14204.60 14444.35 measured 14172.54 1.00x 1.02x -
L1-59 59_conv_standard_3D__asymmetric_input__square_kernel fallback_eager - Yes 13058.53 12640.85 measured 13745.28 0.95x 0.92x -
L1-60 60_conv_standard_3D__square_input__asymmetric_kernel fallback_eager - Yes 30059.20 27201.55 measured 30307.08 0.99x 0.90x -
L1-61 61_conv_transposed_3D__square_input__square_kernel fallback_eager - Yes 44781.99 42697.82 measured 44882.63 1.00x 0.95x -
L1-62 62_conv_standard_2D__square_input__asymmetric_kernel fallback_eager - Yes 12069.81 11837.89 measured 12336.88 0.98x 0.96x -
L1-63 63_conv_standard_2D__square_input__square_kernel fallback_eager - Yes 42448.69 NA skipped_large_input 41350.91 1.03x NA -
L1-64 64_conv_transposed_1D fallback_eager - Yes 41164.38 NA skipped_large_input 40940.85 1.01x NA -
L1-65 65_conv_transposed_2D__square_input__asymmetric_kernel fallback_eager - Yes 11312.92 11623.78 measured 11313.93 1.00x 1.03x -
L1-66 66_conv_standard_3D__asymmetric_input__asymmetric_kernel fallback_eager - Yes 13310.51 13495.85 measured 14926.14 0.89x 0.90x -
L1-67 67_conv_standard_1D fallback_eager - Yes 37730.86 NA skipped_large_input 38567.18 0.98x NA -
L1-68 68_conv_transposed_3D__square_input__asymmetric_kernel fallback_eager - Yes 307633.07 306852.78 measured 308265.91 1.00x 1.00x -
L1-69 69_conv_transposed_2D__asymmetric_input__asymmetric_kernel fallback_eager - Yes 12993.64 13720.57 measured 14820.41 0.88x 0.93x -
L1-70 70_conv_transposed_3D__asymmetric_input__square_kernel fallback_eager - Yes 89250.36 NA skipped_large_input 89029.19 1.00x NA -
L1-71 71_conv_transposed_2D__asymmetric_input__square_kernel fallback_eager - Yes 3337.15 3154.66 measured 3083.13 1.08x 1.02x -
L1-72 72_conv_transposed_3D_asymmetric_input_asymmetric_kernel___strided_padded_grouped_ fallback_eager - Yes 5813.38 6947.70 measured 5828.20 1.00x 1.19x -
L1-73 73_conv_transposed_3D_asymmetric_input_square_kernel__strided_padded__grouped fallback_eager - Yes 21432.37 20074.16 measured 21980.61 0.98x 0.91x -
L1-74 74_conv_transposed_1D_dilated fallback_eager - Yes 455910.93 455355.13 measured 475064.09 0.96x 0.96x -
L1-75 75_conv_transposed_2D_asymmetric_input_asymmetric_kernel_strided__grouped____padded____dilated__ fallback_eager - Yes 19996.84 20716.42 measured 19829.35 1.01x 1.04x -
L1-76 76_conv_standard_1D_dilated_strided__ fallback_eager - Yes 2199671.63 NA skipped_large_input 2203386.66 1.00x NA -
L1-77 77_conv_transposed_3D_square_input_square_kernel___padded____dilated____strided__ fallback_eager - Yes 1029124.30 1035464.72 measured 1022323.03 1.01x 1.01x -
L1-78 78_conv_transposed_2D_asymmetric_input_asymmetric_kernel___padded__ fallback_eager - Yes 6074.36 6548.38 measured 6546.08 0.93x 1.00x -
L1-79 79_conv_transposed_1D_asymmetric_input_square_kernel___padded____strided____dilated__ fallback_eager - Yes 296520.88 298658.80 measured 297894.96 1.00x 1.00x -
L1-80 80_conv_standard_2D_square_input_asymmetric_kernel___dilated____padded__ fallback_eager - Yes 1462136.96 1483206.85 measured 1484511.47 0.98x 1.00x -
L1-81 81_conv_transposed_2D_asymmetric_input_square_kernel___dilated____padded____strided__ fallback_eager - Yes 341085.14 339191.36 measured 340379.34 1.00x 1.00x -
L1-82 82_conv_depthwise_2D_square_input_square_kernel fallback_eager - Yes 22098.76 NA skipped_large_input 21359.04 1.03x NA -
L1-83 83_conv_depthwise_2D_square_input_asymmetric_kernel fallback_eager - Yes 5322.37 5377.00 measured 5457.62 0.98x 0.99x -
L1-84 84_conv_depthwise_2D_asymmetric_input_square_kernel fallback_eager - Yes 82304.15 NA skipped_large_input 82242.26 1.00x NA -
L1-85 85_conv_depthwise_2D_asymmetric_input_asymmetric_kernel fallback_eager - Yes 18968.36 20185.19 measured 18677.70 1.02x 1.08x -
L1-86 86_conv_depthwise_separable_2D fallback_eager - Yes 28436.64 NA skipped_large_input 28772.22 0.99x NA -
L1-87 87_conv_pointwise_2D fallback_eager - Yes 25901.52 NA skipped_large_input 25999.06 1.00x NA -
L1-88 88_MinGPTNewGelu fallback_eager - Yes 7867.39 865.40 measured 7873.32 1.00x 0.11x -
L1-89 89_cumsum fallback_eager - Yes 16432.43 NA skipped_large_input 16519.14 0.99x NA -
L1-90 90_cumprod fallback_eager - Yes 16403.35 NA skipped_large_input 16534.69 0.99x NA -
L1-91 91_cumsum_reverse fallback_eager - Yes 43736.18 NA skipped_large_input 43712.18 1.00x NA -
L1-92 92_cumsum_exclusive fallback_eager - Yes 62939.71 NA skipped_large_input 62891.50 1.00x NA -
L1-93 93_masked_cumsum fallback_eager - Yes 31475.25 NA skipped_large_input 31513.33 1.00x NA -
L1-94 94_MSELoss fallback_eager - Yes 40886.37 NA skipped_large_input 40935.73 1.00x NA -
L1-95 95_CrossEntropyLoss fallback_eager - Yes 1980.61 793.77 measured 1981.05 1.00x 0.40x -
L1-96 96_HuberLoss fallback_eager - Yes 27083.21 NA skipped_large_input 27082.69 1.00x NA -
L1-97 97_ScaledDotProductAttention fallback_eager - Yes 361434.86 NA skipped_large_input 359729.48 1.00x NA -
L1-98 98_KLDivLoss fallback_eager - Yes 19565.60 NA skipped_large_input 19534.74 1.00x NA -
L1-99 99_TripletMarginLoss fallback_eager - Yes 21313.43 NA skipped_large_input 21303.68 1.00x NA -
L1-100 100_HingeLoss fallback_eager - Yes 47447.11 NA skipped_large_input 47471.66 1.00x NA -
L2-1 1_Conv2D_ReLU_BiasAdd fallback_eager - Yes 17163.38 10157.11 measured 16799.32 1.02x 0.60x -
L2-2 2_ConvTranspose2d_BiasAdd_Clamp_Scaling_Clamp_Divide fallback_eager - Yes 48806.70 14997.46 measured 48704.80 1.00x 0.31x -
L2-3 3_ConvTranspose3d_Sum_LayerNorm_AvgPool_GELU fallback_eager - Yes 47306.04 24645.92 measured 47194.60 1.00x 0.52x -
L2-4 4_Conv2d_Mish_Mish fallback_eager - Yes 41588.65 NA skipped_large_input 43163.08 0.96x NA -
L2-5 5_ConvTranspose2d_Subtract_Tanh fallback_eager - Yes 30164.79 18823.06 measured 29766.96 1.01x 0.63x -
L2-6 6_Conv3d_Softmax_MaxPool_MaxPool fallback_eager - Yes 5478.88 5057.92 measured 5430.05 1.01x 0.93x -
L2-7 7_Conv3d_ReLU_LeakyReLU_GELU_Sigmoid_BiasAdd fallback_eager - Yes 65308.15 46711.03 measured 65329.76 1.00x 0.72x -
L2-8 8_Conv3d_Divide_Max_GlobalAvgPool_BiasAdd_Sum fallback_eager - Yes 41170.96 37722.30 measured 41165.90 1.00x 0.92x -
L2-9 9_Matmul_Subtract_Multiply_ReLU custom 1 Yes 56026.56 49837.36 measured 55936.20 1.00x 0.89x -
L2-10 10_ConvTranspose2d_MaxPool_Hardtanh_Mean_Tanh fallback_eager - Yes 29824.23 NA skipped_large_input 31665.14 0.94x NA -
L2-11 11_ConvTranspose2d_BatchNorm_Tanh_MaxPool_GroupNorm fallback_eager - Yes 10676.54 8551.20 measured 10628.17 1.00x 0.80x -
L2-12 12_Gemm_Multiply_LeakyReLU custom 0 Yes 56463.41 51071.78 measured 55804.43 1.01x 0.92x -
L2-13 13_ConvTranspose3d_Mean_Add_Softmax_Tanh_Scaling fallback_eager - Yes 223413.65 216028.77 measured 223702.51 1.00x 0.97x -
L2-14 14_Gemm_Divide_Sum_Scaling custom 1 Yes 56305.83 50989.86 measured 720.63 78.13x 70.76x -
L2-15 15_ConvTranspose3d_BatchNorm_Subtract fallback_eager - Yes 7062.43 7175.86 measured 7123.70 0.99x 1.01x -
L2-16 16_ConvTranspose2d_Mish_Add_Hardtanh_Scaling fallback_eager - Yes 43705.86 17270.48 measured 43590.35 1.00x 0.40x -
L2-17 17_Conv2d_InstanceNorm_Divide fallback_eager - Yes 19751.24 10687.94 measured 20220.15 0.98x 0.53x -
L2-18 18_Matmul_Sum_Max_AvgPool_LogSumExp_LogSumExp fallback_eager - Yes 55733.72 50613.20 measured 56131.01 0.99x 0.90x -
L2-19 19_ConvTranspose2d_GELU_GroupNorm fallback_eager - Yes 39842.21 NA skipped_large_input 40528.04 0.98x NA -
L2-20 20_ConvTranspose3d_Sum_ResidualAdd_Multiply_ResidualAdd fallback_eager - Yes 23133.66 11568.95 measured 23496.48 0.98x 0.49x -
L2-21 21_Conv2d_Add_Scale_Sigmoid_GroupNorm fallback_eager - Yes 23807.04 9798.75 measured 23650.28 1.01x 0.41x -
L2-22 22_Matmul_Scale_ResidualAdd_Clamp_LogSumExp_Mish fallback_eager - Yes 56119.42 50216.93 measured 55830.54 1.01x 0.90x -
L2-23 23_Conv3d_GroupNorm_Mean fallback_eager - Yes 8034.85 6356.80 measured 7955.58 1.01x 0.80x -
L2-24 24_Conv3d_Min_Softmax fallback_eager - Yes 7047.31 6023.18 measured 7026.14 1.00x 0.86x -
L2-25 25_Conv2d_Min_Tanh_Tanh fallback_eager - Yes 20603.25 13117.91 measured 20077.83 1.03x 0.65x -
L2-26 26_ConvTranspose3d_Add_HardSwish fallback_eager - Yes 37009.95 NA skipped_large_input 37628.83 0.98x NA -
L2-27 27_Conv3d_HardSwish_GroupNorm_Mean fallback_eager - Yes 81115.20 72195.86 measured 81083.61 1.00x 0.89x -
L2-28 28_BMM_InstanceNorm_Sum_ResidualAdd_Multiply fallback_eager - Yes 56077.75 51935.05 measured 57431.73 0.98x 0.90x -
L2-29 29_Matmul_Mish_Mish fallback_eager - Yes 56641.06 50981.87 measured 56370.08 1.00x 0.90x -
L2-30 30_Gemm_GroupNorm_Hardtanh fallback_eager - Yes 55661.33 52062.98 measured 56434.02 0.99x 0.92x -
L2-31 31_Conv2d_Min_Add_Multiply fallback_eager - Yes 20394.23 10218.32 measured 20140.83 1.01x 0.51x -
L2-32 32_Conv2d_Scaling_Min fallback_eager - Yes 31523.39 NA skipped_large_input 32202.21 0.98x NA -
L2-33 33_Gemm_Scale_BatchNorm fallback_eager - Yes 56356.04 51786.27 measured 57024.72 0.99x 0.91x -
L2-34 34_ConvTranspose3d_LayerNorm_GELU_Scaling fallback_eager - Yes 65236.19 38211.61 measured 65410.91 1.00x 0.58x -
L2-35 35_Conv2d_Subtract_HardSwish_MaxPool_Mish fallback_eager - Yes 21382.37 12406.15 measured 20975.63 1.02x 0.59x -
L2-36 36_ConvTranspose2d_Min_Sum_GELU_Add fallback_eager - Yes 4683.01 3359.29 measured 4695.69 1.00x 0.72x -
L2-37 37_Matmul_Swish_Sum_GroupNorm fallback_eager - Yes 105156.00 95868.51 measured 104594.53 1.01x 0.92x -
L2-38 38_ConvTranspose3d_AvgPool_Clamp_Softmax_Multiply fallback_eager - Yes 37114.96 24620.55 measured 36969.23 1.00x 0.67x -
L2-39 39_Gemm_Scale_BatchNorm fallback_eager - Yes 202053.60 200336.09 measured 195835.26 1.03x 1.02x -
L2-40 40_Matmul_Scaling_ResidualAdd custom 2 Yes 201194.14 195760.99 measured 197515.38 1.02x 0.99x -
L2-41 41_Gemm_BatchNorm_GELU_ReLU fallback_eager - Yes 199783.49 198358.62 measured 196195.90 1.02x 1.01x -
L2-42 42_ConvTranspose2d_GlobalAvgPool_BiasAdd_LogSumExp_Sum_Multiply fallback_eager - Yes 25039.30 NA skipped_large_input 26881.28 0.93x NA -
L2-43 43_Conv3d_Max_LogSumExp_ReLU fallback_eager - Yes 59156.62 54245.97 measured 59183.79 1.00x 0.92x -
L2-44 44_ConvTranspose2d_Multiply_GlobalAvgPool_GlobalAvgPool_Mean fallback_eager - Yes 6535.62 3195.30 measured 6476.29 1.01x 0.49x -
L2-45 45_Gemm_Sigmoid_LogSumExp fallback_eager - Yes 150187.63 147754.99 measured 149118.05 1.01x 0.99x -
L2-46 46_Conv2d_Subtract_Tanh_Subtract_AvgPool fallback_eager - Yes 22797.33 12601.61 measured 23092.72 0.99x 0.55x -
L2-47 47_Conv3d_Mish_Tanh fallback_eager - Yes 57428.83 50354.53 measured 57436.48 1.00x 0.88x -
L2-48 48_Conv3d_Scaling_Tanh_Multiply_Sigmoid fallback_eager - Yes 22152.16 16352.67 measured 23388.03 0.95x 0.70x -
L2-49 49_ConvTranspose3d_Softmax_Sigmoid fallback_eager - Yes 15764.16 12069.38 measured 16767.13 0.94x 0.72x -
L2-50 50_ConvTranspose3d_Scaling_AvgPool_BiasAdd_Scaling fallback_eager - Yes 28691.61 22856.14 measured 28790.98 1.00x 0.79x -
L2-51 51_Gemm_Subtract_GlobalAvgPool_LogSumExp_GELU_ResidualAdd fallback_eager - Yes 116293.42 113508.88 measured 116794.69 1.00x 0.97x -
L2-52 52_Conv2d_Activation_BatchNorm fallback_eager - Yes 13982.41 6415.10 measured 13792.99 1.01x 0.47x -
L2-53 53_Gemm_Scaling_Hardtanh_GELU fallback_eager - Yes 115802.47 112976.80 measured 115038.09 1.01x 0.98x -
L2-54 54_Conv2d_Multiply_LeakyReLU_GELU fallback_eager - Yes 21514.00 NA skipped_large_input 20663.20 1.04x NA -
L2-55 55_Matmul_MaxPool_Sum_Scale fallback_eager - Yes 128324.13 124462.59 measured 128709.97 1.00x 0.97x -
L2-56 56_Matmul_Sigmoid_Sum fallback_eager - Yes 129326.32 122417.93 measured 129306.89 1.00x 0.95x -
L2-57 57_Conv2d_ReLU_HardSwish fallback_eager - Yes 13411.11 3695.00 measured 13369.35 1.00x 0.28x -
L2-58 58_ConvTranspose3d_LogSumExp_HardSwish_Subtract_Clamp fallback_eager - Yes 34888.58 19399.09 measured 34918.21 1.00x 0.56x -
L2-59 59_Matmul_Swish_Scaling fallback_eager - Yes 127954.41 124037.21 measured 129136.04 0.99x 0.96x -
L2-60 60_ConvTranspose3d_Swish_GroupNorm_HardSwish fallback_eager - Yes 38222.81 22683.76 measured 38285.44 1.00x 0.59x -
L2-61 61_ConvTranspose3d_ReLU_GroupNorm fallback_eager - Yes 32690.95 29562.70 measured 33392.64 0.98x 0.89x -
L2-62 62_Matmul_GroupNorm_LeakyReLU_Sum fallback_eager - Yes 56824.56 51389.72 measured 56539.51 1.01x 0.91x -
L2-63 63_Gemm_ReLU_Divide custom 1 Yes 56886.75 50932.46 measured 55941.63 1.02x 0.91x -
L2-64 64_Gemm_LogSumExp_LeakyReLU_LeakyReLU_GELU_GELU fallback_eager - Yes 57228.79 51703.70 measured 56865.76 1.01x 0.91x -
L2-65 65_Conv2d_AvgPool_Sigmoid_Sum fallback_eager - Yes 42340.19 38011.33 measured 42408.55 1.00x 0.90x -
L2-66 66_Matmul_Dropout_Softmax fallback_eager - Yes 31900.35 28575.59 measured 32388.71 0.98x 0.88x -
L2-67 67_Conv2d_GELU_GlobalAvgPool fallback_eager - Yes 24567.52 11264.26 measured 24851.05 0.99x 0.45x -
L2-68 68_Matmul_Min_Subtract custom 1 Yes 33029.25 28353.98 measured 32085.54 1.03x 0.88x -
L2-69 69_Conv2d_HardSwish_ReLU fallback_eager - Yes 7379.19 3760.45 measured 7389.41 1.00x 0.51x -
L2-70 70_Gemm_Sigmoid_Scaling_ResidualAdd custom 2 Yes 57231.52 51146.64 measured 56640.95 1.01x 0.90x -
L2-71 71_Conv2d_Divide_LeakyReLU fallback_eager - Yes 7451.76 3752.04 measured 7398.07 1.01x 0.51x -
L2-72 72_ConvTranspose3d_BatchNorm_AvgPool_AvgPool fallback_eager - Yes 28241.44 22809.04 measured 28455.67 0.99x 0.80x -
L2-73 73_Conv2d_BatchNorm_Scaling fallback_eager - Yes 7971.48 3751.80 measured 8019.38 0.99x 0.47x -
L2-74 74_ConvTranspose3d_LeakyReLU_Multiply_LeakyReLU_Max fallback_eager - Yes 8566.00 6102.80 measured 8535.52 1.00x 0.71x -
L2-75 75_Gemm_GroupNorm_Min_BiasAdd fallback_eager - Yes 57643.75 53825.75 measured 57411.56 1.00x 0.94x -
L2-76 76_Gemm_Add_ReLU custom 0 Yes 57864.88 52966.96 measured 57660.84 1.00x 0.92x -
L2-77 77_ConvTranspose3d_Scale_BatchNorm_GlobalAvgPool fallback_eager - Yes 72186.71 67159.13 measured 72660.54 0.99x 0.92x -
L2-78 78_ConvTranspose3d_Max_Max_Sum fallback_eager - Yes 78743.56 76148.10 measured 78569.45 1.00x 0.97x -
L2-79 79_Conv3d_Multiply_InstanceNorm_Clamp_Multiply_Max fallback_eager - Yes 6625.37 4490.09 measured 6620.58 1.00x 0.68x -
L2-80 80_Gemm_Max_Subtract_GELU fallback_eager - Yes 58040.28 50817.01 measured 58563.22 0.99x 0.87x -
L2-81 81_Gemm_Swish_Divide_Clamp_Tanh_Clamp fallback_eager - Yes 58001.16 52392.35 measured 58645.11 0.99x 0.89x -
L2-82 82_Conv2d_Tanh_Scaling_BiasAdd_Max fallback_eager - Yes 38924.16 18068.43 measured 38957.55 1.00x 0.46x -
L2-83 83_Conv3d_GroupNorm_Min_Clamp_Dropout fallback_eager - Yes 21572.48 17056.51 measured 22209.96 0.97x 0.77x -
L2-84 84_Gemm_BatchNorm_Scaling_Softmax fallback_eager - Yes 58749.12 53437.59 measured 58531.31 1.00x 0.91x -
L2-85 85_Conv2d_GroupNorm_Scale_MaxPool_Clamp fallback_eager - Yes 10044.36 5154.12 measured 10053.53 1.00x 0.51x -
L2-86 86_Matmul_Divide_GELU fallback_eager - Yes 58436.63 51446.40 measured 58296.69 1.00x 0.88x -
L2-87 87_Conv2d_Subtract_Subtract_Mish fallback_eager - Yes 36737.02 16373.53 measured 36652.38 1.00x 0.45x -
L2-88 88_Gemm_GroupNorm_Swish_Multiply_Swish fallback_eager - Yes 59202.22 52575.42 measured 59686.21 0.99x 0.88x -
L2-89 89_ConvTranspose3d_MaxPool_Softmax_Subtract_Swish_Max fallback_eager - Yes 28903.11 25165.16 measured 29246.64 0.99x 0.86x -
L2-90 90_Conv3d_LeakyReLU_Sum_Clamp_GELU fallback_eager - Yes 66325.26 41795.42 measured 65988.22 1.01x 0.63x -
L2-91 91_ConvTranspose2d_Softmax_BiasAdd_Scaling_Sigmoid fallback_eager - Yes 26602.30 10826.80 measured 26689.46 1.00x 0.41x -
L2-92 92_Conv2d_GroupNorm_Tanh_HardSwish_ResidualAdd_LogSumExp fallback_eager - Yes 18253.87 6144.27 measured 18126.12 1.01x 0.34x -
L2-93 93_ConvTranspose2d_Add_Min_GELU_Multiply fallback_eager - Yes 22506.77 8421.65 measured 22490.27 1.00x 0.37x -
L2-94 94_Gemm_BiasAdd_Hardtanh_Mish_GroupNorm fallback_eager - Yes 58614.86 52142.52 measured 58925.80 0.99x 0.88x -
L2-95 95_Matmul_Add_Swish_Tanh_GELU_Hardtanh fallback_eager - Yes 60277.60 51573.49 measured 59829.94 1.01x 0.86x -
L2-96 96_ConvTranspose3d_Multiply_Max_GlobalAvgPool_Clamp fallback_eager - Yes 28798.85 27663.10 measured 28675.90 1.00x 0.96x -
L2-97 97_Matmul_BatchNorm_BiasAdd_Divide_Swish fallback_eager - Yes 58691.32 52390.00 measured 59651.02 0.98x 0.88x -
L2-98 98_Matmul_AvgPool_GELU_Scale_Max fallback_eager - Yes 58980.27 51388.56 measured 59327.78 0.99x 0.87x -
L2-99 99_Matmul_GELU_Softmax fallback_eager - Yes 58761.76 54850.60 measured 58579.23 1.00x 0.94x -
L2-100 100_ConvTranspose3d_Clamp_Min_Divide fallback_eager - Yes 110352.72 85609.70 measured 109676.89 1.01x 0.78x -
L3-1 1_MLP fallback_eager - Yes 79250.62 71503.27 measured 80371.65 0.99x 0.89x -
L3-2 2_ShallowWideMLP fallback_eager - Yes 255423.90 258222.89 measured 257338.39 0.99x 1.00x -
L3-3 3_DeepNarrowMLP fallback_eager - Yes 22921.43 22591.02 measured 22200.71 1.03x 1.02x -
L3-4 4_LeNet5 fallback_eager - Yes 1507.15 1071.20 measured 1454.59 1.04x 0.74x -
L3-5 5_AlexNet fallback_eager - Yes 160722.72 146880.10 measured 162655.01 0.99x 0.90x -
L3-6 6_GoogleNetInceptionModule fallback_eager - Yes 34183.69 NA skipped_large_input 38882.46 0.88x NA -
L3-7 7_GoogleNetInceptionV1 fallback_eager - Yes 3692.80 3783.66 measured 3634.22 1.02x 1.04x -
L3-8 8_ResNetBasicBlock fallback_eager - Yes 3893.59 2395.27 measured 3890.46 1.00x 0.62x -
L3-9 9_ResNet18 fallback_eager - Yes 1415.82 1356.07 measured 1350.34 1.05x 1.00x -
L3-10 10_ResNet101 fallback_eager - Yes 12233.35 10157.09 measured 12161.68 1.01x 0.84x -
L3-11 11_VGG16 fallback_eager - Yes 10983.48 9743.38 measured 10939.12 1.00x 0.89x -
L3-12 12_VGG19 fallback_eager - Yes 12771.92 11584.84 measured 12782.47 1.00x 0.91x -
L3-13 13_DenseNet121TransitionLayer fallback_eager - Yes 17605.84 NA skipped_large_input 17619.36 1.00x NA -
L3-14 14_DenseNet121DenseBlock fallback_eager - Yes 18726.56 9941.71 measured 18755.16 1.00x 0.53x -
L3-15 15_DenseNet121 fallback_eager - Yes 8014.62 6539.39 measured 8064.94 0.99x 0.81x -
L3-16 16_DenseNet201 fallback_eager - Yes 18880.05 17801.49 measured 18644.17 1.01x 0.95x -
L3-17 17_SqueezeNetFireModule fallback_eager - Yes 57470.87 25235.35 measured 57496.78 1.00x 0.44x -
L3-18 18_SqueezeNet fallback_eager - Yes 96389.77 58835.63 measured 96139.70 1.00x 0.61x -
L3-19 19_MobileNetV1 fallback_eager - Yes 3828.23 3348.97 measured 3850.25 0.99x 0.87x -
L3-20 20_MobileNetV2 fallback_eager - Yes 4457.25 3883.57 measured 4548.00 0.98x 0.85x -
L3-21 21_EfficientNetMBConv fallback_eager - Yes 30597.62 26211.31 measured 32010.49 0.96x 0.82x -
L3-22 22_EfficientNetB0 fallback_eager - Yes 4885.13 4307.70 measured 4889.37 1.00x 0.88x -
L3-23 23_EfficientNetB1 fallback_eager - Yes 2510.11 2285.57 measured 2567.49 0.98x 0.89x -
L3-24 24_EfficientNetB2 fallback_eager - Yes 1807.24 1836.44 measured 1797.11 1.01x 1.02x -
L3-25 25_ShuffleNetUnit fallback_eager - Yes 31747.78 20852.05 measured 32941.31 0.96x 0.63x -
L3-26 26_ShuffleNet fallback_eager - Yes 24292.47 18315.90 measured 25738.24 0.94x 0.71x -
L3-27 27_RegNet fallback_eager - Yes 5056.05 4097.23 measured 5080.96 1.00x 0.81x -
L3-28 28_VisionTransformer fallback_eager - Yes 4192.72 3982.96 measured 4267.52 0.98x 0.93x -
L3-29 29_SwinMLP fallback_eager - Yes 21591.39 20564.28 measured 21226.48 1.02x 0.97x -
L3-30 30_SwinTransformerV2 fallback_eager - Yes 33415.75 30414.49 measured 34231.66 0.98x 0.89x -
L3-31 31_VisionAttention fallback_eager - Yes 208174.23 57153.06 measured 208257.52 1.00x 0.27x -
L3-32 32_ConvolutionalVisionTransformer fallback_eager - Yes 931.00 842.18 measured 931.17 1.00x 0.90x -
L3-33 33_VanillaRNN fallback_eager - Yes 98446.13 105582.97 measured 97914.46 1.01x 1.08x -
L3-34 34_VanillaRNNHidden fallback_eager - Yes 15777.13 23560.90 measured 15805.56 1.00x 1.49x -
L3-35 35_LSTM fallback_eager - Yes 100214.97 100218.51 measured 100331.25 1.00x 1.00x -
L3-36 36_LSTMHn fallback_eager - Yes 100595.03 100407.62 measured 100355.77 1.00x 1.00x -
L3-37 37_LSTMCn fallback_eager - Yes 100504.63 100271.72 measured 100373.57 1.00x 1.00x -
L3-38 38_LSTMBidirectional fallback_eager - Yes 193574.50 193365.46 measured 193913.51 1.00x 1.00x -
L3-39 39_GRU fallback_eager - Yes 98688.84 98865.94 measured 98675.78 1.00x 1.00x -
L3-40 40_GRUHidden fallback_eager - Yes 98697.64 98729.87 measured 98627.23 1.00x 1.00x -
L3-41 41_GRUBidirectional fallback_eager - Yes 197567.07 197589.12 measured 197551.10 1.00x 1.00x -
L3-42 42_GRUBidirectionalHidden fallback_eager - Yes 197839.82 197752.04 measured 197786.39 1.00x 1.00x -
L3-43 43_MinGPTCausalAttention fallback_eager - Yes 138119.83 127761.24 measured 138016.11 1.00x 0.93x -
L3-44 44_MiniGPTBlock fallback_eager - Yes 344168.09 311143.71 measured 345956.57 0.99x 0.90x -
L3-45 45_UNetSoftmax fallback_eager - Yes 13903.03 10954.76 measured 13963.12 1.00x 0.78x -
L3-46 46_NetVladWithGhostClusters fallback_eager - Yes 7531.62 5683.13 measured 7716.85 0.98x 0.74x -
L3-47 47_NetVladNoGhostClusters fallback_eager - Yes 6031.49 4452.65 measured 6238.71 0.97x 0.71x -
L3-48 48_Mamba2ReturnY fallback_eager - Yes 42852.97 24998.37 measured 43158.02 0.99x 0.58x -
L3-49 49_Mamba2ReturnFinalState fallback_eager - Yes 32377.35 6966.03 measured 33634.73 0.96x 0.21x -
L3-50 50_ReLUSelfAttention fallback_eager - Yes 49300.34 42194.31 measured 48073.62 1.03x 0.88x -
L4-1 1_EleutherAI-gpt-neo-2p7B_bs32_seq256 fallback_eager - Yes 17292571.29 17101309.57 measured 17295829.10 1.00x 0.99x -
L4-2 2_facebook-opt-1p3b_bs1_seq2047 fallback_eager - Yes 2035886.84 2093837.65 measured 2046484.01 0.99x 1.02x -
L4-3 3_EleutherAI-gpt-neo-2p7B_bs1_seq2047 fallback_eager - Yes 4793280.27 4508970.70 measured 4768139.16 1.01x 0.95x -
L4-4 4_facebook-opt-1p3b_bs32_seq256 fallback_eager - Yes 7808687.99 7737530.03 measured 7802303.71 1.00x 0.99x -
L4-5 5_google-bigbird-roberta-base_bs1_seq4095 fallback_eager - Yes 427896.24 400130.33 measured 427785.39 1.00x 0.94x -
L4-6 6_facebook-bart-large_bs1_seq1023 fallback_eager - Yes 159253.89 165324.92 measured 159015.29 1.00x 1.04x -
L4-7 7_gpt2_bs32_seq256 fallback_eager - Yes 886613.92 859340.85 measured 891631.90 0.99x 0.96x -
L4-8 8_facebook-opt-1p3b_bs512_seq32 fallback_eager - Yes 15500717.29 15528676.27 measured 15476937.01 1.00x 1.00x -
L4-9 9_google-bigbird-roberta-base_bs32_seq256 fallback_eager - Yes 774042.39 4771057.56 measured 775286.50 1.00x 6.15x -
L4-10 10_google-bigbird-roberta-base_bs1024_seq32 fallback_eager - Yes 2331434.57 2135189.21 measured 2337182.25 1.00x 0.91x -
L4-11 11_google-electra-small-discriminator_bs1_seq511 fallback_eager - Yes 7252.56 7869.66 measured 7281.23 1.00x 1.08x -
L4-12 12_google-electra-small-discriminator_bs1024_seq32 fallback_eager - Yes 292614.81 283364.14 measured 293488.04 1.00x 0.97x -
L4-13 13_google-reformer-enwik8_bs32_seq256 fallback_eager - Yes 885036.50 857011.81 measured 883632.11 1.00x 0.97x -
L4-14 14_google-electra-small-discriminator_bs32_seq256 fallback_eager - Yes 80403.84 75321.78 measured 79802.39 1.01x 0.94x -
L4-15 15_google-reformer-enwik8_bs1024_seq32 fallback_eager - Yes 3446508.91 3312108.15 measured 3437517.94 1.00x 0.96x -
L4-16 16_gpt2_bs1_seq1023 fallback_eager - Yes 114855.35 116690.05 measured 116618.79 0.98x 1.00x -
L4-17 17_facebook-bart-large_bs1024_seq32 fallback_eager - Yes 4270042.72 4148121.34 measured 4242575.93 1.01x 0.98x -
L4-18 18_EleutherAI-gpt-neo-2p7B_bs512_seq32 fallback_eager - Yes 34234019.53 33632361.33 measured 34143773.44 1.00x 0.99x -
L4-19 19_gpt2_bs1024_seq32 fallback_eager - Yes 2973443.97 2754085.57 measured 3000537.35 0.99x 0.92x -
L4-20 20_facebook-bart-large_bs32_seq256 fallback_eager - Yes 1241902.10 1233769.96 measured 1262559.39 0.98x 0.98x -

Technical Insights

Diagnostic vs Eager vs Compile
Geomean (timed tasks) 1.01x 0.83x
Median (timed tasks) 1.00x 0.91x
Geomean (compile, excluding >10x outliers) NA 0.81x
  • Best vs compile: L2-14 14_Gemm_Divide_Sum_Scaling at 70.76x.
  • Worst vs compile: L1-88 88_MinGPTNewGelu at 0.11x.
  • Custom-kernel geomean vs compile (8 tasks): 1.58x.
  • Fallback (no custom kernel) geomean vs compile (193 tasks): 0.81x.
  • Performance pattern: algebraic graph rewrites drive the largest gains; pure pointwise fusion after large GEMMs trends near parity with torch.compile.

Optimization Notes

  • L1-5 (Matrix_scalar_multiplication): vectorized scale kernel with runtime config tuning.
  • L1-19 (ReLU): vectorized ReLU kernel with chunk-safe correctness checks for giant tensors.
  • L2-9 (Matmul_Subtract_Multiply_ReLU): fused affine + ReLU epilogue.
  • L2-12 (Gemm_Multiply_LeakyReLU): fused multiply + LeakyReLU epilogue.
  • L2-14 (Gemm_Divide_Sum_Scaling): algebraic rewrite from GEMM+reduction to GEMV-like path.
  • L2-40 (Matmul_Scaling_ResidualAdd): residual algebra simplification to a single scale.
  • L2-63 (Gemm_ReLU_Divide): fused ReLU + divide epilogue.
  • L2-68 (Matmul_Min_Subtract): fused clamp-to-constant plus subtract.
  • L2-70 (Gemm_Sigmoid_Scaling_ResidualAdd): fused sigmoid + scale + residual add.
  • L2-76 (Gemm_Add_ReLU): fused bias add + ReLU epilogue.

Reproducibility

  • KernelBench root: /mnt/extra/lshen/Research/KernelBench
  • Command: python3 -m kernelbench_rocm.benchmark_kernelbench_rocm --kernelbench-root /mnt/extra/lshen/Research/KernelBench --output-json /mnt/extra/lshen/Research/ROCm-Agent/kernelbench_rocm/results/kernelbench_rocm_report.json --readme /mnt/extra/lshen/Research/ROCm-Agent/README.md --tune-warmup 1 --tune-iters 3 --bench-warmup 1 --bench-iters 4 --check-trials 2 --compile-max-input-elems 200000000 --correctness-regen-max-input-elems 50000000 --full-kernelbench --levels 1,2,3,4

About

ROCm-Agent: Agentic RL for AMD GPU Kernel Optimization, inspired by https://cuda-agent.github.io/

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages