Silent context truncation isn't just a hosted-API problem

[behrnt-slatgng]

I'm the founder of Grunz, a hosted inference service — flagging this here because the failure mode isn't actually hosted-specific, and I think it bites self-hosted LocalAI setups just as often, maybe more quietly.

The context length on a model card is a property of the weights. The context length you actually get is a property of whoever serves it — and with LocalAI that's supposed to be you, which is the whole appeal. But in practice there are three numbers in play (the card's advertised window, the backend's own default/max — llama.cpp, vLLM, exllama2, whichever you've wired up — and whatever you actually set in the model's YAML), and when they disagree, nothing errors. The request just gets served against whichever ceiling is lowest, silently, and output quality degrades in a way that looks like "the model got worse" rather than "your effective context is a quarter of what you think it is."

We hit this constantly on the hosted side and had to build the habit of verifying real served context empirically (send a deliberately oversized prompt, watch where it actually starts truncating) rather than trusting any config value. If the point of your setup is running weights locally, LocalAI is the right layer for this precisely because you can go verify and fix it yourself — there's no black box between you and the backend. The catch is that most people don't know to check, because a mismatch here fails silently instead of throwing.

Question for people running this in production: is there a way to get LocalAI (or the backend underneath it) to report the effective context length it's actually honoring per-request, rather than just the configured context_size? Right now the only reliable check I know of is empirical.