server: bound the 'unlimited' prompt cache by free host memory and log its size

[jasstrong]

What

--cache-ram -1 ("no limit") now means half of the free host memory at server start, and the prompt cache prints one info line per update with its entry count and size.

Why

Every cached prompt is a full copy of a sequence's KV state in host memory. For a 220k-token f16 cache on Qwen3.8-27B that is 14 GiB per entry. LocalAI's llama.cpp backend passes -1 by default, so on a single-slot server every request that displaces a long session (OpenCode's title and summary calls, for example) saves another 14 GiB copy and nothing ever evicts it. On a 60 GB box the backend reached 59.7 GB resident plus 5 GB of swap within an hour; from that point every new GPU process on the machine died with amdgpu: SVM mapping failed, exceeds resident system memory limit, and the session itself fell back to full re-prefills of 220k tokens (seven minutes each).

The cache state was only logged at trace level, which is how this went unnoticed; it is now an info line.

Behaviour

• explicit --cache-ram N: unchanged
• --cache-ram 0: unchanged, disabled
• --cache-ram -1: bounded to free_host / 2 (falls back to 8192 MiB when the host memory cannot be read); the chosen bound is logged at startup

Test

Production, before (bigtime, MI210, Qwen3.8-27B with a 220k-token OpenCode session, cache_ram=-1 via LocalAI): backend at 59.7 GB RSS + 5 GB swap after ~1 h; two 14.2 GB and one 26.5 GB anonymous mappings (full KV copies of the session); 10 LRU slot selections in 2 h; every new HIP process on the box failed with SVM mapping failed.

Standalone, after (moonage, CPU backend to stay off the production GPUs, qwen2.5-coder-14b, -c 8192, a 2.8k-token session interleaved with four unrelated short requests):

setting	startup log	behaviour
--cache-ram -1	bounding it to 32103 MiB (half of the 62.7 GiB of free host memory)	cache holds one entry at a time (~550 MiB); each return of the session restores it and processes 15-19 tokens; RSS flat at 8.8 GB across 9 requests
--cache-ram 64	size limit: 64 MiB	every save skipped (prompt state size 552 MiB exceeds cache size limit), each return re-prefills ~2.8k tokens: the failure mode the bound prevents at scale

The per-update line reads e.g. prompt cache: 1 prompts, 552.0 MiB (limit 32103.0 MiB, 8192 tokens).

🤖 Generated with Claude Code

https://claude.ai/code/session_01NxP6x5bmUDYFvmouceN2mR

[TheTom]

Agree with the change; it is exactly the failure mode in #325 and LocalAI's -1 default makes it the common case. Pushed a one-liner so --cache-ram's help text says what -1 now does instead of "no limit". The Windows failure is the test_completion_unified flake again; merging when CI is back.