From 382f9e0c9b0e83de714bfb527751a422f4cee353 Mon Sep 17 00:00:00 2001 From: Ken VanDine Date: Tue, 28 Jul 2026 22:01:27 -0400 Subject: [PATCH 1/2] ci: CUDA build improvements Two changes, both measured against run 30332616539. Cache the windows-latest-cmake build. Its cuda12 leg compiles nine CUDA architectures in a single job and had no ccache step at all, making it the second-longest job in the run at 90.6 min of Build. windows-latest-cuda runs the same MSVC/Ninja/Release setup with ccache and averages 1.6 min per leg at 97.96% hit rate. No cmake flags are needed: ggml's src/CMakeLists.txt sets RULE_LAUNCH_COMPILE when it finds ccache on PATH (GGML_CCACHE defaults to ON), which is how windows-latest-cuda already gets its hits. The cuda12 leg gets max-size 2G because nine architectures plus GGML_CPU_ALL_VARIANTS will not fit the 500M default that suits the cpu and vulkan legs. Parallelize xz when packing the Linux CUDA artifacts. tar -cJf drives a single-threaded xz and was the largest cost in those jobs, 50.9 min across the eight x64 legs plus 5.9 min on arm64, against only 12.8 min of Build. Measured on a 2.9 GB binary payload: 12m41s single-threaded versus 47.7s on 32 cores, with the archive 1.8% larger because threaded mode compresses independent blocks. The runners are 4-core, so expect roughly 3-4x there. ccache itself was already healthy everywhere it was configured, at 98-100% hit rates, and the release job's artifact download is fine on a full ubuntu-latest VM. Neither needed changing. Co-Authored-By: Claude Opus 5 --- .github/workflows/build.yml | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 9d308c896..78d18b724 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -247,7 +247,7 @@ jobs: run: | cp ggml/LICENSE ./build/bin/ggml.txt cp LICENSE ./build/bin/stable-diffusion.cpp.txt - tar -cJf sd-${{ env.BRANCH_NAME }}-${{ steps.commit.outputs.short }}-ubuntu-cuda-${{ matrix.sm }}-x64.tar.xz -C ./build/bin . + tar -I 'xz -T0' -cf sd-${{ env.BRANCH_NAME }}-${{ steps.commit.outputs.short }}-ubuntu-cuda-${{ matrix.sm }}-x64.tar.xz -C ./build/bin . - name: Upload artifacts if: ${{ github.event_name == 'schedule' || github.event.inputs.create_release == 'true' }} @@ -429,7 +429,7 @@ jobs: run: | cp ggml/LICENSE ./build/bin/ggml.txt cp LICENSE ./build/bin/stable-diffusion.cpp.txt - tar -cJf sd-${{ env.BRANCH_NAME }}-${{ steps.commit.outputs.short }}-ubuntu-cuda-${{ matrix.sm }}-arm64.tar.xz -C ./build/bin . + tar -I 'xz -T0' -cf sd-${{ env.BRANCH_NAME }}-${{ steps.commit.outputs.short }}-ubuntu-cuda-${{ matrix.sm }}-arm64.tar.xz -C ./build/bin . - name: Upload artifacts if: ${{ github.event_name == 'schedule' || github.event.inputs.create_release == 'true' }} @@ -489,6 +489,14 @@ jobs: Add-Content $env:GITHUB_ENV "VULKAN_SDK=C:\VulkanSDK\${env:VULKAN_VERSION}" Add-Content $env:GITHUB_PATH "C:\VulkanSDK\${env:VULKAN_VERSION}\bin" + - name: ccache + uses: ggml-org/ccache-action@v1.2.16 + with: + key: windows-cmake-${{ matrix.build }} + variant: ccache + evict-old-files: 1d + max-size: ${{ matrix.build == 'cuda12' && '2G' || '500M' }} + - name: Activate MSVC environment id: msvc_dev_cmd uses: ilammy/msvc-dev-cmd@v1 From aeaf146d5c7939d33a6d4ad2de2cf97d31884904 Mon Sep 17 00:00:00 2001 From: Ken VanDine Date: Tue, 28 Jul 2026 22:07:42 -0400 Subject: [PATCH 2/2] ci: install only the CUDA components the Linux builds use Both Linux CUDA jobs installed the cuda-toolkit-12-9 metapackage, which apt reports as 3935 MB of archives per job. That is 16 jobs per nightly run on the x64 matrix plus the arm64 leg, and it made "Install CUDA Toolkit" the second-largest cost in those jobs at 36.5 min across the x64 legs. Most of it is never used. The metapackage pulls Nsight Systems and Compute, cuFFT, cuSPARSE, cuSOLVER and NPP; ggml-cuda links only CUDA::cudart, CUDA::cublas and CUDA::cuda_driver plus CCCL for thrust/cub, and stable-diffusion.cpp links only ggml. Install that subset instead, ~595 MB of direct archives, and add a guard that names the missing file if NVIDIA ever repackages one of them rather than failing partway through the compile. Package names verified present in both developer.download.nvidia.com repos the jobs use, ubuntu2204/x86_64 and ubuntu2404/sbsa. The Windows CUDA installs already select sub-packages through Jimver/cuda-toolkit and were left alone. Co-Authored-By: Claude Opus 5 --- .github/workflows/build.yml | 50 +++++++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 78d18b724..3d3d8a852 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -192,7 +192,33 @@ jobs: wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt-get update - sudo apt-get install -y cuda-toolkit-12-9 cmake ninja-build patchelf + # The cuda-toolkit-12-9 metapackage is ~3.9 GB of archives and drags + # in Nsight, cuFFT, cuSPARSE, cuSOLVER and NPP, none of which + # ggml-cuda links or this workflow ships. Install just what the Build + # and Bundle steps consume (~1.2 GB): nvcc, cudart-dev (also supplies + # the libcuda stub behind CUDA::cuda_driver), cccl for thrust/cub, + # cublas, and the curand/nvjitlink libs the bundle step copies. + sudo apt-get install -y \ + cuda-nvcc-12-9 \ + cuda-cudart-dev-12-9 \ + cuda-cccl-12-9 \ + libcublas-dev-12-9 \ + libcurand-dev-12-9 \ + libnvjitlink-dev-12-9 \ + cmake ninja-build patchelf + + # Fail here, naming the missing file, rather than partway through the + # compile if NVIDIA repackages any of the above. + for f in /usr/local/cuda/bin/nvcc \ + /usr/local/cuda/include/cublas_v2.h \ + /usr/local/cuda/include/cub/cub.cuh \ + /usr/local/cuda/lib64/libcudart.so \ + /usr/local/cuda/lib64/libcublas.so \ + /usr/local/cuda/lib64/libcublasLt.so \ + /usr/local/cuda/lib64/libcurand.so \ + /usr/local/cuda/lib64/libnvJitLink.so; do + [ -e "$f" ] || { echo "::error::CUDA install is missing $f"; exit 1; } + done - name: Set CUDA environment run: | @@ -376,7 +402,27 @@ jobs: wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt-get update - sudo apt-get install -y cuda-toolkit-12-9 cmake ninja-build patchelf + # See the x64 job above for why this is a hand-picked subset rather + # than the cuda-toolkit-12-9 metapackage. + sudo apt-get install -y \ + cuda-nvcc-12-9 \ + cuda-cudart-dev-12-9 \ + cuda-cccl-12-9 \ + libcublas-dev-12-9 \ + libcurand-dev-12-9 \ + libnvjitlink-dev-12-9 \ + cmake ninja-build patchelf + + for f in /usr/local/cuda/bin/nvcc \ + /usr/local/cuda/include/cublas_v2.h \ + /usr/local/cuda/include/cub/cub.cuh \ + /usr/local/cuda/lib64/libcudart.so \ + /usr/local/cuda/lib64/libcublas.so \ + /usr/local/cuda/lib64/libcublasLt.so \ + /usr/local/cuda/lib64/libcurand.so \ + /usr/local/cuda/lib64/libnvJitLink.so; do + [ -e "$f" ] || { echo "::error::CUDA install is missing $f"; exit 1; } + done - name: Set CUDA environment run: |