Prebuilds and build scripts

[iimez]

Currently windows and linux x64 prebuilds with CUDA are available. It would be nice to support more backends and arch/platform combinations - but also leave an escape hatch for anybody wanting to do builds themselves, for greater flexibility.

Some of the difficulties I know of are:

• CUDA prebuilds are huge and can easily hit npm package size limit.
• Total size is multiplied by the platform/arch combinations we want to ship (I think the download on install alleviates this, but they're still pretty large?)
• It looks like separate Vulkan and CUDA builds are necessary as well, so that would multiply the amount of prebuilds again (but Vulkan bins are not that large fortunately)
• Upstream changes frequently and prebuilds might be permanently out of date.

@giladgd created a very thoughtful (best of all worlds?) solution for that over at withcatai/node-llama-cpp - could take inspiration from it. Maybe you're interested in collaborating or sharing some wisdom on this, Gilad?

The gist of it, as far as I read it:

• node-llama-cpp publishes x separate prebuild packages, all of them containing platform/arch conditionals in their package.json's. Backends are split into different packages as well.
• At runtime user decides which prebuild - or custom build - should be require'd
• For custom builds theres cli commands to download a llama.cpp release and build it with cmake-js.

[giladgd]

@iimez I have plans to refactor node-llama-cpp to extract a lot of functionality from it that's responsible for handling the native parts as well as ggml-related custom integrations to an another library to facilitate making great bindings for other AI libraries.
I still have a lot planned for node-llama-cpp which takes most of my time, but I started building many things in a way that I can later convert to be more generic and interoperable with other libraries.
It's not practical to reuse most of the code in node-llama-cpp yet though, as most of it is still extremely integrated around llama.cpp specifically.

A major feature I'm currently working on is smart and automatic resource management - models consume a lot of memory and are compute-intensive, and people just want to use models without messing with loading and unloading things from memory or customizing parameters.
I'm working on a solution that can orchestrate and prioritize work, and balance many parameters (such as GPU layers, context size, etc.) automatically and dynamically, and load and unload things from memory as needed to do all the things the user want it to do in a performant and optimized way.
node-llama-cpp already does the part of balancing all the parameters, but it doesn't do all the orchestration stuff yet.
Ideally, I'd like to have all ggml-based libraries share the same orchestrator so you can use models from different libraries simultaneously on a single machine, even if the machine is resource-constrained.

I've already looked at the stable-diffusion.cpp codebase and started playing around with it to create node-stable-diffusion, but stopped since I realized that the orchestration thing has to be finished first.
Once I do the aforementioned refactor to node-llama-cpp, I'd be happy to help you use it to bind to stable-diffusion.cpp.

Regarding the prebuilds, it takes a lot of trial and error to ensure the prebuilds can actually run on all platforms.
I had to ensure there are no local libraries that are dynamically linked, and that the libraries that are used are compiled with an older version that's available on most platforms (like glibc, for example).
There's no alternative to manually testing on many machines and machine configurations, and it's important to retain access to those machines so you can ensure changes to the build process don't break the prebuilt binaries on such machines.
It's hard to set up tests for this as part of the CI, since CIs with GPUs are expensive and CIs are also usually more capable machines that don't represent the average user.

I constantly keep up with changes in llama.cpp to track external library dependency changes and ensure no additional dependencies are added if they can be circumvented.
For example, there was a PR that was about to introduce the use of Python in the build process and I helped replace it with cpp code to ensure there's no need for Python.

I hope this helps.

[lmagder]

Thanks guys, yeah it's a challenging problem for sure. I would totally be open to using an existing solution like @giladgd 's once it's working, since TBH dealing with building and packaging has been like 90% of the work of getting this stood up. I wrote the NAPI code in like a day, everything else was all GH actions and npm wrangling.

Linux ABI problems are definitely an issue with prebuilds. I'm a C++ dev in my day job so I've dealt with issues on shipping binaries. I think to really make it robust you would need to use something like https://github.com/ValveSoftware/steam-runtime. But maybe that's overkill? Windows is actually not too bad luckily since the ABI is very stable, but I have no idea about macOS (or maybe nobody practically cares about that?)

I noticed pkg-prebuilds lets you specify different binary flavors so I was thinking of exploring using that to support Vulkan vs CUDA, etc in one package but the NPM package size limits are definitely an issue there. I'm already downloading the BLAS binaries from https://developer.download.nvidia.com/compute/cuda/redist in postinstall to avoid hitting the max size.

Also, just full disclosure this is a part time hobby project for me that I created just because there was no existing node bindings and I wanted to mess around with using it in a personal project. I'm happy to take contributions but I don't want to overpromise how much time I'll have, so this repo doesn't have to be the canonical node stable-diffusion.cpp bindings. You are welcome to go your own direction (via fork or from scratch), that was also why I didn't publish it to npm as an unscoped package. I didn't want to steal the name from somebody doing a more serious project.

[iimez]

@giladgd It does help, and is interesting. Thanks taking the time sharing. I believe you're right in that orchestration is crucial. Specific solutions that wire up combinations of smaller specialist models often seem to beat big generalists in cost, sometimes in speed or accuracy, some cases in all regards. And being able to do that efficiently on a single machine has a whole list of further upsides. Looking forward to all the things to be cooked and appreciate your work!

@lmagder No worries, not meaning to put more load on you. I don't think there is need to go the node-llama-cpp distance. I just found your bindings useful and while I'm on it I might as well help improving them. Steam runtime would add a virtualization layer? Seems like a risky experiment, but I don't know enough to judge haha

Been thinking a bit, a good effort/usability compromise might be:

• Limit to prebuilds we can test on our machines and only as many as package limit allows. (I'm wondering, what is the limit exactly? I read 120M but couldnt find a reliable confirmation. @node-llama-cpp/linux-x64-cuda is 360M)
• Offer an easy way to run a custom build with cmake-js (and/or, improve the postinstall script), allowing to specify the backend explicitly or detecting the best one automatically.

This should allow to use all the backends stable-diffusion.cpp supports without having to fork in order to modify the CMakeLists.

Let me know if you like the direction, happy to dig a bit deeper and contribute a PR for that sometime.

One thing that this doesnt solve is that users still wouldn't be able to choose between CPU/GPU at runtime (would have to decide at buildtime). That could be a future upgrade though.

[lmagder]

Yeah it would be overkill :) I don't think using the Steam runtime directly necessarily is the way to go, just more of an example of one way to solve this problem since Steam has the same issue of distributing GPU-using binaries.

I think the original steam runtime version was like a container image you build in which has very old versions of things like glibc, since often Linux libraries are compatible with older binaries, but not newer ones, so you need to build against the oldest version you want to run on even if you would never want to use that for a development machine. Really glibc is the only library you can't statically link, but they have certain guarantees. Practically for us, also the NAPI, but luckily that is similarly backward-ABI-stable based on NAPI_VERSION. The GLIBC_x.y symbols are basically their version of NAPI_VERSION. This is kind of viral when it comes to other binaries since they all depend on glibc which is why SR has more stuff.

That said, I think they have moved to actual containers even at runtime in later versions, so maybe this approach doesn't scale? although they need to support GUI apps which is harder.

Limit to prebuilds we can test on our machines and only as many as package limit allows. (I'm wondering, what is the limit exactly? I read 120M but couldnt find a reliable confirmation. @node-llama-cpp/linux-x64-cuda is 360M)

I think it's around 100MB at least that's what I found. Weirdly it's not actually a hard limit but at some point it stores the tarball as Base64 in a JS string while uploading it and the limit actually comes from the max length of a string in Node. At least that was where it failed for me. One part here which is weird specifically is that 90% of our size here is actually CUDA binaries embedded in the overall binary which have some unique properties:

1. There are mostly the same platform to platform so if we could compress the Windows .dll and Linux .so together somehow it would be a lot smaller. We're shipping copies of the same bytes I think.
2. The size depends a lot on what architectures you target. CUDA binaries are fat binaries so they can contain any number of copies of the code for different GPU architectures (eg RTX 30x0 vs RTX 40x0). We could ship PTX-only binaries (which is something akin to SPRIV under the hood) which is more compatible at the cost of some runtime performance/loading time. Not sure what the tradeoff here is though. Unfortunately PTX is still versioned similar to D3D shader models so you either need to pick a conservative baseline and possibly leave performance on the table due to not using new features or ship multiple PTX blobs.
3. the NV BLAS libraries are massive (which is why I'm downloading them on postinstall) because they contain blobs for every supported GPU. NV has a tool: nvprune to remove some blobs from a binary but unfortunately it doesn't work on Windows DLLs and they don't provide a static BLAS lib on Windows. If we didn't use cuBLAS this would be a huge space savings, but unfortunately as the GGML folks found, cuBLAS is actually really fast and it's not trivial to replicate it. (and maybe it's big because it has to be?)

Open to ideas here. Also if the VK backend performs basically the same then maybe we just don't bother with CUDA? I'm not sure, usually it's not the case though unfortunately.

I think we could build on a runner that has a older Ubuntu version with an old enough glibc and if we try to statically link as much as we can then possibly it's fairly compatible? I know the CUDA Runtime ABI (libcudart.so) is not stable, which is why you see like PyTorch 118 vs 12, etc. but the CUDA driver API (libnvcuda.so) is stable since it ships as part of the driver so an app linking against an old libnvcuda.so will work on all future ones as long as you statically link the runtime.

Offer an easy way to run a custom build with cmake-js (and/or, improve the postinstall script), allowing to specify the backend explicitly or detecting the best one automatically.

Sounds good to me!

One thing that this doesnt solve is that users still wouldn't be able to choose between CPU/GPU at runtime (would have to decide at buildtime). That could be a future upgrade though.

Weirdly this is the one config the GGML lib already supports dynamically since it always builds the CPU fallback. That's why I did the delay load stuff with implib-so so if you run the current release on a machine with no CUDA libs it will still work and just fall back to the CPU since it can gracefully skip that part if the init fails.

But yeah overall the direction you're proposing sounds good to me if you want to work on it 👍