Qwen3.8-Flash-Next my config with 9tok/s

[nfouka]

Hi ,

I'm trying to optimize Qwen3.8-Flash-Next on LocalAI and would appreciate some advice from people familiar with LocalAI / llama.cpp and the qwen-exp backend.

My priority is maximum generation speed while keeping the model on the GPU, with no CPU/RAM offloading if possible.

Hardware
NVIDIA Tesla T4 — 16 GB VRAM
16 vCPU
200 GB RAM
NVIDIA driver: 580.159.03
CUDA: 13.0
LocalAI: 4.6.2
Model

Qwen3.8-Flash-Next GGUF:

Unsloth UD-Q4_K_XL

4 GGUF shards
Qwen3.8-Flash-Next
Vision enabled with F16 mmproj

Current LocalAI configuration:

backend: qwen-exp

mmproj: llama-cpp/models/qwen3.8-flash-next/mmproj-F16.gguf

threads: 8

cache_type_k: q4_1
cache_type_v: q4_1

context_size: 262144

debug: false
embeddings: false

f16: true
flash_attention: true
gpu_layers: -1

known_usecases:

• chat
• vision

mirostat: 0
mirostat_eta: 0.1
mirostat_tau: 5

name: Qwen3.8-Flash-Next

parameters:
max_tokens: 50000
min_p: 0
model: llama-cpp/models/qwen3.8-flash-next/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
presence_penalty: 0
repeat_penalty: 1
temperature: 1
tfz: 1
top_k: 20
top_p: 0.95

template:
use_tokenizer_template: true

low_vram: true
mlock: false
mmap: false
Current performance

With this setup, I get approximately:

35–40 tokens/s

For example:

~40 tok/s for ~500 generated tokens
~35 tok/s for ~1300 generated tokens

I have also tested different n_batch / n_ubatch configurations. The best results so far are around:

n_batch = 512
n_ubatch = 128
threads = 8
gpu_layers = -1
parallel = 1
Questions

I would particularly appreciate advice on the following:

1. low_vram: true

Since I have 16 GB VRAM but the model is heavily quantized, is low_vram: true actually beneficial on a Tesla T4?

Could it be hurting performance by changing how buffers / tensors are allocated?

Would you recommend:

low_vram: false

with gpu_layers: -1?

2. mmap: false

I currently have:

mmap: false
mlock: false

Is disabling mmap useful in this configuration?

Could enabling mmap improve model loading or runtime behavior without causing CPU/RAM offloading?

3. f16: true

The model itself is Q4_K_XL.

Is:

f16: true

recommended for Qwen3.8-Flash-Next with the qwen-exp backend?

Could f16: false improve performance or reduce VRAM pressure?

4. KV cache

I'm currently using:

cache_type_k: q4_1
cache_type_v: q4_1

Would you recommend:

cache_type_k: q4_0
cache_type_v: q4_0

or perhaps:

cache_type_k: q4_0
cache_type_v: q8_0

on a T4?

I'm looking for the best quality/performance/VRAM trade-off.

5. Context size

I have:

context_size: 262144

but most of my requests don't actually use the full 262K context.

Does reserving such a large context significantly affect performance or VRAM usage, especially the KV cache?

Would reducing it to something like 64K / 128K significantly improve generation speed?

6. Threads

I'm using:

threads: 8

on a 16-vCPU VM.

Since the model is fully GPU-offloaded, is there any reason to use 8 rather than 16 threads?

Could too many CPU threads actually hurt GPU generation performance?

7. Flash Attention

I have:

flash_attention: true

Is Flash Attention recommended for Qwen3.8-Flash-Next on a Tesla T4?

Does the T4 benefit significantly from it compared with newer GPUs?

8. MTP / speculative decoding

I'm particularly interested in MTP (Multi-Token Prediction) and speculative decoding.

Does the current qwen-exp backend support the relevant Qwen3.8-Flash-Next MTP functionality?

If so, what is the recommended LocalAI configuration?

Can MTP provide a significant speed-up on a Tesla T4?

9. N-gram speculative decoding

Qwen3.8-Flash-Next appears to have mechanisms that can benefit from n-gram/speculative decoding.

Is it possible to keep the n-gram data/index in system RAM while keeping the model itself entirely in VRAM?

I have 200 GB of RAM available, so I would be very interested in using RAM for auxiliary data if this can reduce VRAM pressure without slowing generation.

10. Backend

Finally, is:

backend: qwen-exp

currently the recommended backend for Qwen3.8-Flash-Next in LocalAI?

Or is there a newer / faster implementation that I should test?

My main objective is:

Tesla T4 16 GB → Qwen3.8-Flash-Next UD-Q4_K_XL → maximum tok/s → GPU-only model → minimal quality loss.

If needed, I can provide the complete LocalAI logs, nvidia-smi, llama.cpp parameters and detailed benchmark results.

Thanks in advance for any advice!

[jangatzke]

Have a look at #12514