CUDA error: forward compatibility was attempted on non supported HW

[ayan4m1]

I am getting an error running in Docker using the docker-compose.yml from the documentation site after upgrading to this nvidia driver in Ubuntu:

CUDA error: forward compatibility was attempted on non supported HW 

ii  nvidia-driver-550                              550.120-0ubuntu0.24.04.1                    amd64        NVIDIA driver metapackage

[giladgd]

Many CUDA errors happen due to mismatching versions of the drivers and the rest of the CUDA tools, so try to upgrade everything CUDA-related to the latest version to ensure all the versions match.
Also, it's important for the CUDA version used in the docker image to match or be older than the CUDA version installed on the host machine, so consider downgrading it if you need.

[ayan4m1]

Yeah, unfortunately I'm caught between two pieces of software - a game that needs the latest version of nvidia-drivers, and node-llama-cpp needing older ones. At least it doesn't seem to stop inferencing outright. Thanks.

[giladgd]

What's the version of CUDA installed on your machine?
node-llama-cpp prebuilt binaries are compiled with CUDA 12.2 to ensure all newer CUDA drivers can use them.

[ayan4m1]

12.6, but I'm manually building CUDA binaries in my Dockerfile according to the docs, so I'm not using prebuilt binaries afaik. How would I rule it out?

[giladgd]

Do you build the binaries outside the docker image or inside of it?
The prebuilt CUDA binaries have better compatibility with various hardware and machine configurations than binaries you can build from source on your machine, thus it's recommended to use them if possible.
I'll release a new version that significantly improves the performance of the prebuilt CUDA binaries in the next week or so, so you shouldn't notice a performance difference between the prebuilt binaries and a local build anymore after that release, if that's the reason you build from source.

[ayan4m1]

Inside the docker image.

Gotcha - the reason I build from source is just that I assumed it was necessary for CUDA (e.g. non-CPU). I can try relying on prebuilt binaries but I don't know how to ensure CUDA ones will be selected. I will go back to the docs.

[ayan4m1]

Sorry, still having trouble here... I'm using the dockerfile example for CUDA on the documentation site (I assume this means using prebuilt binaries), my app is now using CPU for inference and throwing the forward compatibility error periodically.

Do I need to downgrade to a specific nvidia-driver version? @giladgd

OS: Debian GNU/Linux 12 (bookworm) (x64)
Node: 22.12.0 (x64)
TypeScript: 5.6.3
node-llama-cpp: 3.2.0

CUDA error: forward compatibility was attempted on non supported HWCUDA error: forward compatibility was attempted on non supported HWggml_cuda_init: failed to initialize CUDA: forward compatibility was attempted on non supported HW
CUDA: available

CUDA used VRAM: 0% (0B/0B)
CUDA free VRAM: 0% (0B/0B)

CPU model: AMD Ryzen 9 5900X 12-Core Processor
Math cores: 12
Used RAM: 12.41% (7.79GB/62.72GB)
Free RAM: 87.58% (54.93GB/62.72GB)

[giladgd]

Try to update to the latest CUDA version on the host machine and ensure you setup the NVIDIA Container Toolkit correctly.
After you do this, restart the machine since some of these issues don't go away until a full restart is done.

[ayan4m1]

For some reason, I am unable to install drivers which provide CUDA 12.6 - I get 12.4 in nvidia-smi - so I changed the dockerfile to install 12.4 and source from the ubuntu 22.04 packages and was able to "downgrade" my Docker image that way to resolve the issue. Thanks for your help!