RTX5090 Blackwell support

[HKNDerRollo]

Summary
The official context_chat_backend Docker images do not support NVIDIA Blackwell GPUs (RTX 5090, RTX 5080, etc.) because they are compiled against CUDA 12.2 without sm_120 architecture support.

Environment

• GPU: NVIDIA GeForce RTX 5090 (compute capability 12.0 / sm_120)
• Driver: 595.58.03
• CUDA: 13.2
• context_chat_backend: 5.3.0
• context_chat: 5.3.1
• Nextcloud: 33.x

Problem
The official image ghcr.io/nextcloud/context_chat_backend:5.3.0 is based on nvidia/cuda:12.2.2-runtime-ubuntu22.04 and downloads a prebuilt llama-cpp-python wheel compiled for CUDA 12.2 without Blackwell support:

CUDA : ARCHS = 500,520,530,600,610,620,700,720,750,800,860,870,890,900
# Missing: 1000 (sm_100), 1200 (sm_120)

As a result:

• GPU utilization stays at 0%
• Only 746 MiB of 32607 MiB VRAM is used
• Power draw: 7W of 575W
• All embedding computation falls back to CPU

The RTX 5090 requires CUDA 12.8+ and sm_120 for native Blackwell support.

What already works
The master branch (5.4.0-beta0) has a new multi-stage Dockerfile using CUDA 12.8 which is much better. However, the final stage still defaults to FROM runtime-cpu AS final instead of FROM runtime-cuda AS final.

With two small fixes to the master Dockerfile:

# 1. Add Blackwell architecture
ENV CMAKE_CUDA_ARCHITECTURES="89;90;100;120"

# 2. Use CUDA runtime as final stage
FROM runtime-cuda AS final  # was: runtime-cpu

...the RTX 5090 works correctly:

CUDA : ARCHS = 500,610,700,750,800,860,890,1200 | BLACKWELL_NATIVE_FP4 = 1
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0

Request

1. Fix the FROM runtime-cpu AS final → FROM runtime-cuda AS final in the master Dockerfile (this seems like a bug)
2. Add sm_120 to CMAKE_CUDA_ARCHITECTURES for Blackwell support
3. Consider publishing a CUDA-specific image tag e.g. context_chat_backend:5.3.x-cuda for GPU users
4. Publish a v5.3.1 git tag to match the context_chat frontend app version

Additional Notes

• The RTX 5090 has 32 GB VRAM which is ideal for large embedding workloads
• With proper Blackwell support, indexing performance should improve dramatically
• BLACKWELL_NATIVE_FP4 = 1 confirms native FP4 Tensor Core support is available

Thank you for the great work on this project!

[kyteinsky]

hello,
v5.4.0-beta0 is already available to install and comes with cuda 12.8. It should also have support for all supported architectures in cuda 12.8 so blackwell should be covered.
would you mind trying it out and check if it works with your GPU? To use the cuda build of the image, set the compute device to CUDA in app_api's deploy daemon configuration.
if you don't want to install the beta, we will release a stable version in some time.

[HKNDerRollo]

Thanks for the quick response!

I already tested v5.4.0-beta0 (master branch). The CUDA 12.8 base is great, but there are two issues:

1. FROM runtime-cpu AS final — the last line of the Dockerfile defaults to the CPU stage, so the CUDA build is never used regardless of the deploy daemon setting.

2. CMAKE_CUDA_ARCHITECTURES missing sm_120 — without explicitly adding 120 to the arch list, Blackwell falls back to PTX JIT compilation instead of native cubins.

With these two fixes applied to the master Dockerfile:

sed -i 's/^FROM runtime-cpu AS final$/FROM runtime-cuda AS final/' Dockerfile
sed -i '/^ENV CMAKE_ARGS="-DGGML_CUDA=ON/i ENV CMAKE_CUDA_ARCHITECTURES="89;90;100;120"' Dockerfile

The RTX 5090 works perfectly:

CUDA : ARCHS = 500,610,700,750,800,860,890,1200 | BLACKWELL_NATIVE_FP4 = 1
Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VRAM: 32110 MiB

The FROM runtime-cpu AS final line looks like a bug — could you fix it before the stable release? Happy to open a PR if helpful.

[kyteinsky]

thanks for testing it out.

1. FROM runtime-cpu AS final — the last line of the Dockerfile defaults to the CPU stage, so the CUDA build is never used regardless of the deploy daemon setting.

that's the default build target, for cuda --target runtime-cuda should be used in the docker build command. So the full command could look like: docker build --target=runtime-cpu -t ccb-cuda . inside the context_chat_backend folder.
but that shouldn't be necessary as we have a github actions workflow that builds it so it can be directly used from https://github.com/nextcloud-releases/context_chat_backend/pkgs/container/context_chat_backend/892407489?tag=latest-cuda
or just by installing the latest beta release of context_chat_backend from the appstore, but make sure to set the compute device to CUDA in the AppAPI's deploy daemon config otherwise it would pick the cpu image as default.

2. CMAKE_CUDA_ARCHITECTURES missing sm_120 — without explicitly adding 120 to the arch list, Blackwell falls back to PTX JIT compilation instead of native cubins.

It would be nice to know how this conclusion was reached. Was it something different in the logs about the PTX JIT compilation? Or a performance difference?
Do try the cuda image from above to see if the difference still exists.

also, llama.cpp's cuda build doesn't seem to compile cubins for any architectures by default: https://github.com/ggml-org/llama.cpp/blob/d7be46189f4a9ed59c5d98f71a87f8a16954a6eb/.devops/cuda.Dockerfile#L15-L31
maybe the difference is just at the startup since it's JIT compiled and cached.
not really against the idea though, just curious how things work and how much the performance differs.