CPU and RAM requirements

[zohaib09]

I am using next cloud 33 on Ubuntu 22 my question is if I add local AI in my next cloud server VM so how much RAM & cpu should I increase currently I have 12 GB ram and 4 core processor and we are only 2 to 3 users. My task are like general text queries and image generations etc?

[sauravsingla]

12 GB RAM and 4 CPU cores can be workable for smaller quantized models, although the practical requirement will depend heavily on model size, context length and backend. Image generation may have very different hardware requirements, especially without a GPU.

I have been experimenting with this exact RAM-constrained inference problem in an open-source C++ project called MemVanta.

For one controlled OpenLLaMA 7B v2 Q4_0 CPU-only comparison using the same GGUF:

• MemVanta peak RSS: ~3.80 GiB
• llama.cpp peak RSS: ~7.24 GiB
• Peak RSS reduction: 47.5%

In a separate cgroup-v2 test with swap disabled, MemVanta also completed under a tested 3584 MiB memory ceiling where the pinned llama.cpp comparison was OOM-killed.

There is an important trade-off: llama.cpp is currently substantially faster. My focus is specifically on lowering resident-memory requirements, then improving throughput without losing that memory advantage.

If anyone here runs LocalAI or GGUF models on RAM-constrained VMs, I would be very interested in independent reproduction or feedback:

https://github.com/sauravsingla/MemVanta