Skip to content

chore: adjust comment in Dockerfile regarding RTX5090 support #316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kyteinsky wants to merge 1 commit into
base: master
Choose a base branch
from
Open

chore: adjust comment in Dockerfile regarding RTX5090 support #316

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ RUN /opt/venv/bin/python -m pip wheel \
# ============================================================
# CUDA (NVIDIA) builder
# Builds llama_cpp_python with CUDA support.
# CUDA 12.8 supports up to sm_100 (Blackwell / B100, B200).
# CUDA 12.8 supports up to sm_120 (consumer Blackwell / RTX 5090).
# gcc-14 is used for consistency with the other build stages and
# because CUDA 12.6+ accepts gcc-14 natively on Ubuntu 24.04.
# ============================================================
Expand All @@ -91,8 +91,12 @@ RUN ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/lib/libcuda.so \
ENV LD_LIBRARY_PATH="/usr/local/lib:/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:${LD_LIBRARY_PATH}"
ENV CC=gcc-14 CXX=g++-14

# Real cubins for all shipping GPU generations through Blackwell (sm_100),
# plus one forward-compatible PTX target to keep wheel size manageable.
# CMAKE_CUDA_ARCHITECTURES is intentionally not set here. llama.cpp's CMake
# selects sensible defaults based on the detected CUDA toolkit version:

@marcelklehr marcelklehr Jun 17, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mmh, but this is at build time, right? so it does not depend on the CUDA version that the admin has but on the CUDA version of the CUDA_DEVEL_IMAGE, correct?

@kyteinsky kyteinsky Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, and we depend on llama.cpp to make the correct choice of cubins to bake in, so we don't have to check and adjust it when the cuda version changes.
we could but it already exists so I didn't bother.

# with CUDA 12.8 this yields real cubins for sm_50..sm_89, sm_90, sm_120a
# (all shipping GPU generations through consumer Blackwell / RTX 5090) plus
# PTX virtual targets for forward compatibility. See:
# https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cuda/CMakeLists.txt
ENV CMAKE_ARGS="-DGGML_CUDA=ON -DGGML_CUDA_FORCE_MMQ=ON -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF \
-DGGML_AVX=ON -DGGML_AVX2=ON \
-DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16"
Expand Down
Loading