Merge cuda-perf.yml into cuda.yml and add 5% perf regression gate (#23385)

[Gasoonjia]

Summary:

cuda-perf.yml is retired. Its benchmark + upload pipeline now runs inside
cuda.yml's test-model-cuda-e2e cells, reusing each cell's exported
pte/ptd and built runner in place (new helper
.ci/scripts/cuda_benchmark_from_e2e.sh, allowlisted to the 7 perf
models; whisper-medium backfilled into the accuracy matrix).

New jobs:

• upload-benchmark-results: aggregates cuda-bench-* artifacts to the
  unchanged s3://gha-artifacts/executorch-cuda-perf/ prefix (history
  preserved) + HUD dashboard v3 upload.
• check-perf-regression: fails the PR when decode or prefill tok/s
  drops >5% vs the median of the last 5 successful main runs (same
  model, quant, GPU, min 3 history points; new
  .ci/scripts/cuda_check_regression.py). Bypass with the
  bypass-perf-regression PR label.

Cleanup: delete cuda-perf.yml, rename trigger_cuda_perf.sh to
trigger_cuda_benchmark.sh (retargeted at cuda.yml dispatch inputs),
drop ciflow/cuda-perf from pytorch-probot.yml, update the
_ci-run-decision.yml caller comment.

Differential Revision: D123135327

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23385

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 9 New Failures, 1 Pending, 1 Unrelated Failure, 19 Unclassified Failures

As of commit 8214424 with merge base 0b3d26d ():

NEW FAILURES - The following jobs have failed:

• pull / unittest / linux / linux-job (gh)
  .ci/scripts/tests/test_cuda_workflow.py::CudaWorkflowTest::test_model_e2e_does_not_transfer_artifacts
• pull / unittest / macos / macos-job (gh)
  .ci/scripts/tests/test_cuda_workflow.py::CudaWorkflowTest::test_model_e2e_does_not_transfer_artifacts
• pull / unittest / windows / windows-job (gh)
  .ci/scripts/tests/test_cuda_workflow.py::CudaWorkflowTest::test_model_e2e_does_not_transfer_artifacts
• pull / unittest-editable / linux / linux-job (gh)
  .ci/scripts/tests/test_cuda_workflow.py::CudaWorkflowTest::test_model_e2e_does_not_transfer_artifacts
• pull / unittest-editable / macos / macos-job (gh)
  .ci/scripts/tests/test_cuda_workflow.py::CudaWorkflowTest::test_model_e2e_does_not_transfer_artifacts
• pull / unittest-editable / windows / windows-job (gh)
  .ci/scripts/tests/test_cuda_workflow.py::CudaWorkflowTest::test_model_e2e_does_not_transfer_artifacts
• trunk / unittest-release / linux / linux-job (gh)
  .ci/scripts/tests/test_cuda_workflow.py::CudaWorkflowTest::test_model_e2e_does_not_transfer_artifacts
• trunk / unittest-release / macos / macos-job (gh)
  .ci/scripts/tests/test_cuda_workflow.py::CudaWorkflowTest::test_model_e2e_does_not_transfer_artifacts
• trunk / unittest-release / windows / windows-job (gh)
  .ci/scripts/tests/test_cuda_workflow.py::CudaWorkflowTest::test_model_e2e_does_not_transfer_artifacts

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Test CUDA Builds / test-model-cuda-e2e-diar_streaming_sortformer_4spk-v2-non-quantized / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-dinov2-small-imagenet1k-1-layer-non-quantized / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-gemma-3-4b-it-non-quantized / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-gemma-3-4b-it-quantized-int4-tile-packed / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-gemma-4-31B-it-GGUF-quantized-int4-tile-packed / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-parakeet-tdt-non-quantized / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-parakeet-tdt-quantized-int4-tile-packed / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-Qwen3-0.6B-non-quantized / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-Qwen3-0.6B-quantized-int4-tile-packed / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-Qwen3.5-35B-A3B-HQQ-INT4-quantized-int4-tile-packed / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-Voxtral-Mini-3B-2507-non-quantized / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-Voxtral-Mini-3B-2507-quantized-int4-tile-packed / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-Voxtral-Mini-3B-2507-quantized-int4-weight-only / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-Voxtral-Mini-4B-Realtime-2602-quantized-int4-tile-packed / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-whisper-large-v3-turbo-quantized-int4-tile-packed / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-whisper-large-v3-turbo-quantized-int4-weight-only / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-whisper-medium-non-quantized / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-whisper-small-non-quantized / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test ROCm AOTI / unittest-rocm-gfx950-rocm7.2 / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  backends/cuda/tests/test_sdpa_splitk_replacement.py::TestUnsupportedSDPAStaysWithRegularLowering::test_float_mask_is_not_replaced

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• Test CUDA Builds / upload-benchmark-results (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[meta-codesync]

@Gasoonjia has exported this pull request. If you are a Meta employee, you can view the originating Diff in D123135327.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.