[BUG] Ternary Bonsai 27B enters infinite prompt-processing loop, never generates response — context grows unbounded (43k+ tokens), session 8h+

[Milor123]

I'm reporting what appears to be a critical bug in the llama.cpp serving path for Ternary Bonsai 27B, not a performance complaint.

System

• OS: Windows 11 25H2
• GPU: RTX 4070 12 GB
• CPU: Intel i5-14500, 48 GB RAM
• Build: PrismML-Eng/llama.cpp, compiled from source today
• Model: Ternary-Bonsai-27B-Q2_0.gguf
• Vision: Ternary-Bonsai-27B-mmproj-BF16.gguf
• Client tested: Hermes Desktop and OpenCode — same behavior in both

llama-prism-b1-62061f9-bin-win-cuda-12.4-x64

llama-prism-b1-62061f9-bin-win-cuda-12.4-x64  .\llama-server.exe -m "C:\Users\User\.vllm\Ternary-Bonsai-27B-dspark\Ternary-Bonsai-27B-Q2_0.gguf" --host 127.0.0.1 --port 10000 -ngl 99 -c 106000 -b 8192 -ub 2048 --temp 0.7 -np 1 --top-k 20 --top-p 0.95 --api-key JORCH_SK_d736nng1mX4OFza5uFcbDfcVmVXKqjQOPV7gTXNIB47LhPD0hXddZKl8sPtsAlszjLZ --cont-batching --metrics -ctk q5_0 -ctv q4_0 --jinja -tb 20 -t 20 --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2 --no-mmap --mmproj "C:\Users\User\.vllm\Ternary-Bonsai-27B-dspark\Ternary-Bonsai-27B-mmproj-BF16.gguf" -fa on

The bug

When given a task that triggers tool use / skill loading (in my case: "load the research skill and do extensive research on tallow consumption and benefits"), the model enters a state where prompt processing never terminates, the context grows without bound, and no response is ever generated.

Evidence from my session (screenshot attached)

Console logs show prompt processing repeating with increasing token counts and degrading speed:

prompt processing, n_tokens = 8192,  progress = 0.19, t = 270.64 s  / 30.27 tokens per second
prompt processing, n_tokens = 16384, progress = 0.38, t = 1004.23 s / 16.31 tokens per second
prompt processing, n_tokens = 24576, progress = 0.57, t = 2266.99 s / 10.84 tokens per second

Meanwhile the UI shows the context growing: 43.2k / 106.2k tokens consumed.

Session timer: 8 hours 26 minutes and still running. The model has never output a single token of response or thinking.

Why this is not "slow inference"

• Not a thinking delay: If this were the model's <think> block, tokens would appear in the output. Nothing appears.
• Not a tool loop: The client shows no tool calls being made. The server is stuck in prompt processing.
• Context explosion: The token count grows from 8k → 16k → 24k → 43k+ without user input. This suggests the server is repeatedly re-processing an ever-growing internal state.
• Degrading throughput: Speed drops from 30 t/s → 16 t/s → 10 t/s as the context bloats, confirming the work is real and unbounded, not a deadlock.

What I expected

The model should either:

1. Generate a response (including thinking tags), or
2. Fail with a clear error, or
3. Timeout — not spin for 8+ hours consuming CPU/GPU and growing context.

What actually happens

Silent, infinite prompt processing with unbounded context growth. The only way to stop it is killing the server.

Additional context

• reasoning-budget: activated, budget=2147483647 tokens appears in logs (unlimited thinking budget).
• --cont-batching is enabled.
• -c 106000 configured.

[khosravipasha]

hmm that should not happen, I need to see if its a build or windows issue.

Can you try running llama-cli directly see if issue happens Or also try without vision tower

If not can you build from source (there is scripts for it on https://github.com/PrismML-Eng/Bonsai-demo)

by the way, there is some api key in your description, probably want to invalidate it if its a real key

[legobyte]

Same problem:
https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf/discussions/19

[legobyte]

Can you try running llama-cli directly see if issue happens Or also try without vision tower

The same problem using llama-cli directly:
(Self-correction during thought: Keep it simple no extraflfl fl flfl fl flfl fl fl fl fl fl fl

[Milor123]

Hmm, eso no debería pasar, necesito ver si es un problema de montaje o de Windows.

¿Puedes probar a usar llama-cli directamente para ver si ocurre algún problema? O también probar sin torre de visión?

Si no, ¿puedes compilar desde el código fuente (hay scripts para ello en https://github.com/PrismML-Eng/Bonsai-demo)

Por cierto, hay alguna clave API en tu descripción, probablemente quieras invalidarla si es una clave real

I will run some tests and let you know. Are you referring to compiling llama.cpp? Actually, I initially used my own llama.cpp compiled from this project, but I thought the problem was that I had compiled it incorrectly, so I discovered that you had pre-compiled binaries and came to test them. I will try again with the source code compiled by myself and without the projector to see what happens, but it seems other users are experiencing strange things as well.

The curious thing I noticed is that Hermes or OpenCode completely fall apart, but for example, AnythingLLM at least talks to me and responds, although the tokens/s drop rapidly to less than 10 t/s, making the model undesirable to use. I suppose that is due to the way these models work and the CUDA kernel, right? Well, I don't understand much about this, but something will have to be done about it. Thanks for the help.

Oh, thanks for the advert about the key, I hadn't noticed!

[Milor123]

@khosravipasha I cloned the repo you shared with the information and had DeepSeek help me adjust my commands, and I also passed it the KV-BIAS guide you uploaded to an issue in this repo. I added $env:LLAMA_ATTN_ROT_DISABLE = "1" and also removed a few things; on the other hand, I left the context on automatic with -c 0.

I tried two things, with vision and without vision in Hermes Desktop, but it never responded. (I'll leave the log below).
After that, I tried talking to the llama server webui, sending it a short YouTube transcript, but it is extremely slow (it's reading at 40t/s, I repeat reading, not writing). To give you a reference, Qwen 35B A3 weighing 20 GB (Q4 with 200k context kv cache q4 or turboquant) on my hybrid GPU with CPU is capable of reading at more than 2000t/s on my PC, while Bonsai is writing (responding) at 12.94 t/s on a super small document, whereas my 20GB model on a large document reaches up to 20t/s to 30t/s approx.

PS: don't worry about the API, I've discarded it.

New command:
.\llama-server.exe -m "C:\Users\User\.vllm\Ternary-Bonsai-27B-dspark\Ternary-Bonsai-27B-Q2_0.gguf" --host 127.0.0.1 --port 10000 -ngl 99 -c 0 -b 8192 -ub 2048 --temp 0.7 --top-p 0.95 --top-k 20 --min-p 0 --jinja --mmproj "C:\Users\User\.vllm\Ternary-Bonsai-27B-dspark\Ternary-Bonsai-27B-mmproj-BF16.gguf" -fa on --cont-batching --metrics -ctk q4_0 -ctv q4_0 -t 20 -tb 20 --kv-mean-center "C:\Users\User\.vllm\Ternary-Bonsai-27B-dspark\Ternary-Bonsai-27B-kv-bias.gguf"

Log with vision, stucked in hermes desktop

$env:LLAMA_ATTN_ROT_DISABLE = "1"
S:\..\..\..\llama-prism-b1-62061f9-bin-win-cuda-12.4-x64  .\llama-server.exe -m "C:\Users\User\.vllm\Ternary-Bonsai-27B-dspark\Ternary-Bonsai-27B-Q2_0.gguf" --host 127.0.0.1 --port 10000 -ngl 99 -c 0 -b 8192 -ub 2048 --temp 0.7 --top-p 0.95 --top-k 20 --min-p 0 --jinja --mmproj "C:\Users\User\.vllm\Ternary-Bonsai-27B-dspark\Ternary-Bonsai-27B-mmproj-BF16.gguf" -fa on --cont-batching --metrics -ctk q4_0 -ctv q4_0 -t 20 -tb 20 --kv-mean-center "C:\Users\User\.vllm\Ternary-Bonsai-27B-dspark\Ternary-Bonsai-27B-kv-bias.gguf" --api-key JORCH_SK_d736nng1mX4OFza5uFcbDfcVmVXKqjQOPV7gTXNIB47LhPD0hXddZKl8sPtsAlszjLZ
>
0.00.027.864 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.027.868 I device_info:
0.00.150.317 I   - CUDA0   : NVIDIA GeForce RTX 4070 (12281 MiB, 11106 MiB free)
0.00.150.328 I   - CPU     : Intel(R) Core(TM) i5-14500 (48445 MiB, 20708 MiB free)
0.00.161.758 I system_info: n_threads = 20 (n_threads_batch = 20) / 20 | CUDA : ARCHS = 500,610,700,750,800,860,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.161.763 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.181.856 I srv          init: running without SSL
0.00.188.849 I srv          init: api_keys: ****zjLZ
0.00.188.863 I srv          init: using 19 threads for HTTP server
0.00.191.632 I srv         start: binding port with default address family
0.00.206.641 I srv  llama_server: loading model
0.00.207.011 I srv    load_model: loading model 'C:\Users\User\.vllm\Ternary-Bonsai-27B-dspark\Ternary-Bonsai-27B-Q2_0.gguf'
0.03.128.302 I srv    load_model: [mtmd] estimated worst-case memory usage of mmproj is 1161.02 MiB
0.03.128.323 I common_init_result: fitting params to device memory ...
0.03.128.324 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.03.664.934 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 99, abort
0.10.873.639 W llama_kv_cache: attention rotation force disabled (LLAMA_ATTN_ROT_DISABLE)
0.11.259.719 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.12.908.800 W load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
0.12.908.805 W load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
0.12.908.805 W load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

0.14.124.615 I srv    load_model: loaded multimodal model, 'C:\Users\User\.vllm\Ternary-Bonsai-27B-dspark\Ternary-Bonsai-27B-mmproj-BF16.gguf'
0.14.124.623 I srv    load_model: initializing slots, n_slots = 4
0.14.504.062 W srv    load_model: speculative decoding will use checkpoints
0.14.521.486 W common_speculative_init: no implementations specified for speculative decoding
0.14.528.365 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 262144
0.14.528.377 I slot   load_model: id  1 | task -1 | new slot, n_ctx = 262144
0.14.528.377 I slot   load_model: id  2 | task -1 | new slot, n_ctx = 262144
0.14.528.378 I slot   load_model: id  3 | task -1 | new slot, n_ctx = 262144
0.14.528.599 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB
0.14.528.599 I srv    load_model: use `--cache-ram 0` to disable the prompt cache
0.14.528.601 I srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
0.14.528.601 I srv    load_model: context checkpoints enabled, max = 32, min spacing = 256
0.14.529.333 I srv          init: idle slots will be saved to prompt cache and cleared upon starting a new task
0.14.605.332 I init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
<think>
'
0.14.627.864 I srv          init: init: chat template, thinking = 1
0.14.632.404 I srv  llama_server: model loaded
0.14.632.409 I srv  llama_server: server is listening on http://127.0.0.1:10000
0.14.641.949 I srv  update_slots: all slots are idle
4.36.512.993 I srv  params_from_: Chat format: peg-native
4.36.558.617 I slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
4.36.558.623 I srv  get_availabl: updating prompt cache
4.36.559.418 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
4.36.559.431 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 262144 tokens, 8589934592 est)
4.36.559.432 I srv  get_availabl: prompt cache update took 0.81 ms
4.36.562.731 I reasoning-budget: activated, budget=2147483647 tokens
4.36.562.763 I slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
Received second interrupt, terminating immediately.

Log part 2 without vision (same result)

S:\..\..\..\llama-prism-b1-62061f9-bin-win-cuda-12.4-x64  .\llama-server.exe -m "C:\Users\User\.vllm\Ternary-Bonsai-27B-dspark\Ternary-Bonsai-27B-Q2_0.gguf" --host 127.0.0.1 --port 10000 -ngl 99 -c 0 -b 8192 -ub 2048 --temp 0.7 --top-p 0.95 --top-k 20 --min-p 0 --jinja -fa on --cont-batching --metrics -ctk q4_0 -ctv q4_0 -t 20 -tb 20 --kv-mean-center "C:\Users\User\.vllm\Ternary-Bonsai-27B-dspark\Ternary-Bonsai-27B-kv-bias.gguf" --api-key JORCH_SK_d736nng1mX4OFza5uFcbDfcVmVXKqjQOPV7gTXNIB47LhPD0hXddZKl8sPtsAlszjLZ
>
0.00.026.119 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.026.125 I device_info:
0.00.114.871 I   - CUDA0   : NVIDIA GeForce RTX 4070 (12281 MiB, 11106 MiB free)
0.00.114.882 I   - CPU     : Intel(R) Core(TM) i5-14500 (48445 MiB, 21006 MiB free)
0.00.114.936 I system_info: n_threads = 20 (n_threads_batch = 20) / 20 | CUDA : ARCHS = 500,610,700,750,800,860,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.114.937 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.114.975 I srv          init: running without SSL
0.00.114.992 I srv          init: api_keys: ****zjLZ
0.00.115.000 I srv          init: using 19 threads for HTTP server
0.00.115.149 I srv         start: binding port with default address family
0.00.122.156 I srv  llama_server: loading model
0.00.122.214 I srv    load_model: loading model 'C:\Users\User\.vllm\Ternary-Bonsai-27B-dspark\Ternary-Bonsai-27B-Q2_0.gguf'
0.00.122.266 I common_init_result: fitting params to device memory ...
0.00.122.266 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.00.598.333 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 99, abort
0.07.423.531 W llama_kv_cache: attention rotation force disabled (LLAMA_ATTN_ROT_DISABLE)
0.07.632.681 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.08.019.109 I srv    load_model: initializing slots, n_slots = 4
0.08.213.630 W srv    load_model: speculative decoding will use checkpoints
0.08.213.665 W common_speculative_init: no implementations specified for speculative decoding
0.08.213.668 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 262144
0.08.213.671 I slot   load_model: id  1 | task -1 | new slot, n_ctx = 262144
0.08.213.672 I slot   load_model: id  2 | task -1 | new slot, n_ctx = 262144
0.08.213.672 I slot   load_model: id  3 | task -1 | new slot, n_ctx = 262144
0.08.213.916 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB
0.08.213.916 I srv    load_model: use `--cache-ram 0` to disable the prompt cache
0.08.213.918 I srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
0.08.213.919 I srv    load_model: context checkpoints enabled, max = 32, min spacing = 256
0.08.213.954 I srv          init: idle slots will be saved to prompt cache and cleared upon starting a new task
0.08.237.319 I init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
<think>
'
0.08.254.486 I srv          init: init: chat template, thinking = 1
0.08.254.509 I srv  llama_server: model loaded
0.08.254.511 I srv  llama_server: server is listening on http://127.0.0.1:10000
0.08.254.526 I srv  update_slots: all slots are idle
0.24.773.245 I srv  params_from_: Chat format: peg-native
0.24.817.503 I slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
0.24.817.511 I srv  get_availabl: updating prompt cache
0.24.817.519 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.24.817.526 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 262144 tokens, 8589934592 est)
0.24.817.528 I srv  get_availabl: prompt cache update took 0.01 ms
0.24.818.845 I reasoning-budget: activated, budget=2147483647 tokens
0.24.818.865 I slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
3.48.035.437 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =   8192, progress = 0.44, t = 203.22 s / 40.31 tokens per second
5.25.379.787 I srv  params_from_: Chat format: peg-native

Log webui small document

.08.254.486 I srv          init: init: chat template, thinking = 1
0.08.254.509 I srv  llama_server: model loaded
0.08.254.511 I srv  llama_server: server is listening on http://127.0.0.1:10000
0.08.254.526 I srv  update_slots: all slots are idle
0.24.773.245 I srv  params_from_: Chat format: peg-native
0.24.817.503 I slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
0.24.817.511 I srv  get_availabl: updating prompt cache
0.24.817.519 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.24.817.526 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 262144 tokens, 8589934592 est)
0.24.817.528 I srv  get_availabl: prompt cache update took 0.01 ms
0.24.818.845 I reasoning-budget: activated, budget=2147483647 tokens
0.24.818.865 I slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
3.48.035.437 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =   8192, progress = 0.44, t = 203.22 s / 40.31 tokens per second
5.25.379.787 I srv  params_from_: Chat format: peg-native
5.33.977.760 W srv          stop: cancel task, id_task = 0
8.44.786.136 I slot      release: id  3 | task 0 | stop processing: n_tokens = 16384, truncated = 0
8.44.786.157 I slot get_availabl: id  2 | task -1 | selected slot by LRU, t_last = -1
8.44.786.157 I srv  get_availabl: updating prompt cache
8.44.786.162 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
8.44.786.166 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 262144 tokens, 8589934592 est)
8.44.786.167 I srv  get_availabl: prompt cache update took 0.01 ms
8.44.786.231 I reasoning-budget: activated, budget=2147483647 tokens
8.44.786.232 I reasoning-budget: deactivated (natural end)
8.44.786.256 I slot launch_slot_: id  2 | task 3 | processing task, is_child = 0
8.44.786.258 I slot slot_save_an: id  3 | task -1 | saving idle slot to prompt cache
8.44.985.572 W srv   prompt_save:  - saving prompt with length 16384, total state size = 437.939 MiB (draft: 0.000 MiB)
9.20.078.285 I slot prompt_clear: id  3 | task -1 | clearing prompt with 16384 tokens
9.20.080.477 I srv        update:  - cache state: 1 prompts, 437.939 MiB (limits: 8192.000 MiB, 262144 tokens, 306475 est)
9.20.080.484 I srv        update:    - prompt 000001C43474A520:   16384 tokens, checkpoints:  0,   437.939 MiB
9.24.145.572 I slot print_timing: id  2 | task 3 | prompt processing, n_tokens =    248, progress = 0.05, t =   4.07 s / 61.01 tokens per second
9.26.665.334 I slot create_check: id  2 | task 3 | created context checkpoint 1 of 32 (pos_min = 247, pos_max = 247, n_tokens = 248, size = 149.626 MiB)
10.22.931.799 I slot print_timing: id  2 | task 3 | prompt processing, n_tokens =   2803, progress = 0.58, t =  62.85 s / 44.60 tokens per second
10.28.791.783 I slot create_check: id  2 | task 3 | created context checkpoint 2 of 32 (pos_min = 2802, pos_max = 2802, n_tokens = 2803, size = 149.626 MiB)
10.59.767.085 I slot print_timing: id  2 | task 3 | prompt processing, n_tokens =   4851, progress = 1.00, t =  99.69 s / 48.66 tokens per second
11.22.752.338 I slot create_check: id  2 | task 3 | created context checkpoint 3 of 32 (pos_min = 4850, pos_max = 4850, n_tokens = 4851, size = 149.626 MiB)
11.31.622.550 I slot print_timing: id  2 | task 3 | n_decoded =    100, tg =  12.98 t/s
11.34.685.952 I slot print_timing: id  2 | task 3 | n_decoded =    140, tg =  13.00 t/s
11.37.761.135 I slot print_timing: id  2 | task 3 | n_decoded =    180, tg =  13.00 t/s
11.40.775.824 I slot print_timing: id  2 | task 3 | n_decoded =    219, tg =  12.99 t/s
11.43.838.619 I slot print_timing: id  2 | task 3 | n_decoded =    259, tg =  13.00 t/s
11.46.863.587 I slot print_timing: id  2 | task 3 | n_decoded =    298, tg =  12.99 t/s
11.49.925.368 I slot print_timing: id  2 | task 3 | n_decoded =    337, tg =  12.96 t/s
11.52.926.323 I slot print_timing: id  2 | task 3 | n_decoded =    376, tg =  12.96 t/s
11.55.992.276 I slot print_timing: id  2 | task 3 | n_decoded =    415, tg =  12.94 t/s
11.59.047.481 I slot print_timing: id  2 | task 3 | n_decoded =    454, tg =  12.92 t/s
12.02.052.212 I slot print_timing: id  2 | task 3 | n_decoded =    493, tg =  12.93 t/s
12.05.061.841 I slot print_timing: id  2 | task 3 | n_decoded =    532, tg =  12.93 t/s
12.08.062.415 I slot print_timing: id  2 | task 3 | n_decoded =    571, tg =  12.93 t/s
12.11.062.722 I slot print_timing: id  2 | task 3 | n_decoded =    610, tg =  12.94 t/s
12.14.134.058 I slot print_timing: id  2 | task 3 | n_decoded =    650, tg =  12.94 t/s
12.17.143.367 I slot print_timing: id  2 | task 3 | n_decoded =    689, tg =  12.94 t/s
12.20.143.687 I slot print_timing: id  2 | task 3 | n_decoded =    728, tg =  12.95 t/s
12.23.211.123 I slot print_timing: id  2 | task 3 | n_decoded =    768, tg =  12.95 t/s
12.26.229.112 I slot print_timing: id  2 | task 3 | n_decoded =    807, tg =  12.95 t/s
12.29.230.858 I slot print_timing: id  2 | task 3 | n_decoded =    846, tg =  12.95 t/s
12.32.243.923 I slot print_timing: id  2 | task 3 | n_decoded =    885, tg =  12.95 t/s
12.35.296.184 I slot print_timing: id  2 | task 3 | n_decoded =    924, tg =  12.94 t/s
12.38.300.395 I slot print_timing: id  2 | task 3 | n_decoded =    963, tg =  12.95 t/s
12.41.315.434 I slot print_timing: id  2 | task 3 | n_decoded =   1002, tg =  12.95 t/s
12.44.363.831 I slot print_timing: id  2 | task 3 | n_decoded =   1041, tg =  12.94 t/s
12.47.413.025 I slot print_timing: id  2 | task 3 | n_decoded =   1080, tg =  12.93 t/s
12.50.455.980 I slot print_timing: id  2 | task 3 | n_decoded =   1119, tg =  12.93 t/s
12.53.479.941 I slot print_timing: id  2 | task 3 | n_decoded =   1157, tg =  12.92 t/s
12.56.541.865 I slot print_timing: id  2 | task 3 | n_decoded =   1195, tg =  12.90 t/s
12.59.624.416 I slot print_timing: id  2 | task 3 | n_decoded =   1234, tg =  12.89 t/s
13.02.625.455 I slot print_timing: id  2 | task 3 | n_decoded =   1271, tg =  12.88 t/s
13.05.655.589 I slot print_timing: id  2 | task 3 | n_decoded =   1309, tg =  12.87 t/s
13.08.713.728 I slot print_timing: id  2 | task 3 | n_decoded =   1348, tg =  12.86 t/s
13.11.750.654 I slot print_timing: id  2 | task 3 | n_decoded =   1387, tg =  12.86 t/s
13.14.802.223 I slot print_timing: id  2 | task 3 | n_decoded =   1426, tg =  12.86 t/s
13.17.841.302 I slot print_timing: id  2 | task 3 | n_decoded =   1465, tg =  12.86 t/s
13.20.891.657 I slot print_timing: id  2 | task 3 | n_decoded =   1504, tg =  12.86 t/s
13.23.930.625 I slot print_timing: id  2 | task 3 | n_decoded =   1543, tg =  12.86 t/s
13.26.971.904 I slot print_timing: id  2 | task 3 | n_decoded =   1582, tg =  12.86 t/s
13.30.009.911 I slot print_timing: id  2 | task 3 | n_decoded =   1621, tg =  12.86 t/s
13.33.050.777 I slot print_timing: id  2 | task 3 | n_decoded =   1660, tg =  12.85 t/s
13.36.074.361 I slot print_timing: id  2 | task 3 | n_decoded =   1699, tg =  12.86 t/s
13.39.080.862 I slot print_timing: id  2 | task 3 | n_decoded =   1737, tg =  12.85 t/s
13.42.114.416 I slot print_timing: id  2 | task 3 | n_decoded =   1776, tg =  12.85 t/s
13.45.155.672 I slot print_timing: id  2 | task 3 | n_decoded =   1815, tg =  12.85 t/s
13.48.194.807 I slot print_timing: id  2 | task 3 | n_decoded =   1854, tg =  12.85 t/s
13.51.231.219 I slot print_timing: id  2 | task 3 | n_decoded =   1893, tg =  12.85 t/s
13.54.264.731 I slot print_timing: id  2 | task 3 | n_decoded =   1932, tg =  12.85 t/s
13.54.342.950 I slot print_timing: id  2 | task 3 | prompt eval time =  123835.17 ms /  4855 tokens (   25.51 ms per token,    39.21 tokens per second)
13.54.342.956 I slot print_timing: id  2 | task 3 |        eval time =  150427.25 ms /  1933 tokens (   77.82 ms per token,    12.85 tokens per second)
13.54.342.957 I slot print_timing: id  2 | task 3 |       total time =  274262.42 ms /  6788 tokens
13.54.342.958 I slot print_timing: id  2 | task 3 |    graphs reused =       1924
13.54.343.355 I slot      release: id  2 | task 3 | stop processing: n_tok