CUDA:Model Ternary-Bonsai-27B-Q2_0.gguf Not loaded in GPU

[LsnavTech]

I am using Ternary-Bonsai-27B-Q2_0.gguf and Model Not loaded in GPU
hero@LOQ-Tower:/media/hero/Apps$ ./prism_llama_dense_server.sh
0.00.166.542 I log_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.166.544 I device_info:
0.00.253.415 I - CUDA0 : NVIDIA GeForce RTX 5060 (7707 MiB, 17807 MiB free)
0.00.253.419 I - CPU : AMD Ryzen 7 8745HX with Radeon Graphics (31266 MiB, 31266 MiB free)
0.00.253.650 I system_info: n_threads = 10 (n_threads_batch = 12) / 16 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.253.662 I srv init: running without SSL
0.00.253.693 I srv init: using 2 threads for HTTP server
0.00.253.761 I srv start: binding port with default address family
0.00.254.931 I srv llama_server: loading model
0.00.254.938 I srv load_model: loading model '/media/hero/Data/gguf/Ternary-Bonsai-27B-Q2_0.gguf'
0.00.574.568 I srv load_model: [mtmd] estimated worst-case memory usage of mmproj is 873.10 MiB
0.00.574.579 I common_init_result: fitting params to device memory ...
0.00.574.579 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.04.179.411 W llama_context: n_ctx_seq (16384) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.04.275.769 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.04.334.276 W load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
0.04.334.278 W load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
0.04.334.279 W load_hparams: more info: ggml-org#16842

0.04.538.972 I srv load_model: loaded multimodal model, '/media/hero/Data/gguf/Ternary-Bonsai-27B-mmproj-Q8_0.gguf'
0.04.538.977 I srv load_model: initializing slots, n_slots = 1
0.04.827.946 W srv load_model: speculative decoding will use checkpoints
0.04.827.951 W common_speculative_init: no implementations specified for speculative decoding
0.04.827.952 I slot load_model: id 0 | task -1 | new slot, n_ctx = 16384
0.04.827.966 I srv load_model: prompt cache is enabled, size limit: 8192 MiB
0.04.827.966 I srv load_model: use --cache-ram 0 to disable the prompt cache
0.04.827.966 I srv load_model: for more info see ggml-org#16391
0.04.827.967 I srv load_model: context checkpoints enabled, max = 32, min spacing = 256
0.04.827.978 W srv init: --cache-idle-slots requires --kv-unified, disabling
0.04.838.235 I init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

'
0.04.846.384 I srv init: init: chat template, thinking = 1
0.04.846.393 I srv llama_server: model loaded
0.04.846.395 I srv llama_server: server is listening on http://127.0.0.1:8081
0.04.846.398 I srv update_slots: all slots are idle

[khosravipasha]

What's was error? Seems your gpu memory is full, is some other process using all of it?
it might be offloading some to gpu.

Also might need to reduce context length so it fits in the 8GB, trying to make it more automated here:
PrismML-Eng/Bonsai-demo#120