Eval bug: Mistral Small 4 (mistral4 arch): repetitive/empty output on Metal

[bunnynode]

Name and Version

llama.cpp: b8390-b6c83aad5

Operating systems

Mac

GGML backends

Metal

Hardware

Environment:

• llama.cpp: b8390-b6c83aad5
• Device: Apple M3 Ultra 512GB
• GGUF: unsloth/Mistral-Small-4-119B-2603-GGUF Q4_K_XL
• FA on/off: no difference

cc @ngxson

Models

Mistral-Small-4

Problem description & steps to reproduce

Symptoms:

1. Chat mode (via server/--jinja): outputs repetitive text
   ("汉书后汉书后汉书后..." endlessly)
2. Think/reasoning mode: no output, hangs
3. Pure completion (--no-conversation): intermittent —
   sometimes correct, sometimes falls into repetition loop

First Bad Commit

No response

Relevant log output

Logs

 ./build/bin/llama-cli \
  -m unsloth/Mistral-Small-4-119B-2603-GGUF/Mistral-Small-4-119B-2603-UD-Q4_K_XL-00001-of-00003.gguf \
  -ngl 99 \
  -c 4096 \
  -n 512 \
  --no-conversation \
  -p '[SYSTEM_PROMPT]You are a helpful assistant[/SYSTEM_PROMPT][MODEL_SETTINGS]{"reasoning_effort": "high"}[/MODEL_SETTINGS][INST]What is the capital of France?[/INST]'
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 0.006 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name:   MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)
ggml_metal_device_init: simdgroup reduction   = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory    = true
ggml_metal_device_init: has bfloat            = true
ggml_metal_device_init: has tensor            = false
ggml_metal_device_init: use residency sets    = true
ggml_metal_device_init: use shared buffers    = true
ggml_metal_device_init: recommendedMaxWorkingSetSize  = 498216.21 MB
--no-conversation is not supported by llama-cli
please use llama-completion instead

Loading model...  


▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b8390-b6c83aad5
model      : Mistral-Small-4-119B-2603-UD-Q4_K_XL-00001-of-00003.gguf
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read               add a text file


> [SYSTEM_PROMPT]You are a helpful assistant[/SYSTEM_PROMPT][MODEL_SETTINGS]{"reasoning_effort": "high"}[/MODEL_SETTINGS][INST]What is the capital of France?[/INST]

 

[ Prompt: 206.2 t/s | Generation: 71.3 t/s ]

> who are you

 

[ Prompt: 128.1 t/s | Generation: 71.3 t/s ]

> 

Exiting...
llama_memory_breakdown_print: | memory breakdown [MiB]    |  total     free     self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - MTL0 (Apple M3 Ultra) | 475136 = 404407 + (70728 = 70374 +      90 +     264) +           0 |
llama_memory_breakdown_print: |   - Host                  |                      568 =   544 +       0 +      24                |
ggml_metal_free: deallocating

[fabioamigo]

Same here:

llama-server -m Mistral-Small-4-119B-2603-GGUF/UD-IQ4_NL/Mistral-Small-4-119B-2603-UD-IQ4_NL-00001-of-00003.gguf -mm Mistral-Small-4-119B-2603-GGUF/mmproj-BF16.gguf --host 192.168.15.34 --jinja

llama.cpp: 8397 (d2ecd2d)
Device: Apple M3 Max 125GB
GGUF: unsloth/Mistral-Small-4-119B-2603-UD-IQ4_NL
FA on/off: no difference

With reasoning_effort High or None ==> no difference

Custom System Prompt crashes llama.cpp

Edit: I have not experencied loops yet, just empty output and crashes

[ggerganov]

Hm, there is something weird going on with the MoE down weights in layer 32:

make -j && ./bin/llama-debug -hf bartowski/mistralai_Mistral-Small-4-119B-2603-GGUF:IQ4_NL -p "[SYSTEM_PROMPT] You are a helpful assistant[/SYSTEM_PROMPT][INST] Hello[/INST]" -n 1 --tensor-filter "ffn_moe_down.*" --no-warmup

common_debug_cb_eval:          ffn_moe_down-32 = (f32) MUL_MAT_ID(blk.32.ffn_down_exps.weight{2048, 4096, 128, 1}, ffn_moe_weighted-32{2048, 4, 30, 1}}) = {4096, 4, 30, 1}
    [
        [
            [  -2559.0359,    -473.9050,    5961.5908,    ...,    -208.0920,    1463.8502,   -1987.4396  ],
            [     -1.5931,       0.8739,      -0.3425,    ...,       1.8437,       0.0360,      -0.3827  ],
            [     -0.0317,      -0.7340,       0.9408,    ...,       0.2199,       0.0402,      -0.9936  ],
            [     -0.9309,       0.0219,      -0.2309,    ...,      -0.5265,      -0.0114,      -1.8943  ],
        ],
        [
            [      1.2770,       0.7773,       2.1354,    ...,      -0.4442,       0.6043,       0.4203  ],
            [      0.0844,       0.8541,      -0.7266,    ...,       0.5161,       1.0015,       0.6863  ],
            [     -0.8610,      -0.5280,       0.7662,    ...,      -0.2359,       0.5828,      -0.2291  ],
            [     -0.4032,      -1.2069,       0.8116,    ...,       0.1623,       0.0540,       0.0307  ],
        ],
        [
            [     -0.2593,       0.1510,      -0.5101,    ...,       0.9137,      -0.7561,      -0.5051  ],
            [     -1.0371,       0.6449,       0.1419,    ...,      -0.3286,       0.1571,      -0.4051  ],
            [     -0.4971,       0.3054,       0.4942,    ...,       0.0746,       0.2564,      -0.3038  ],
            [      0.3244,       0.5080,      -0.4780,    ...,       0.5673,      -0.2737,       0.0558  ],
        ],
        ..., 
        [
            [      0.6855,       1.9788,      -0.1687,    ...,       0.8653,      -0.1563,      -1.7064  ],
            [     -0.4899,       0.3691,      -0.3174,    ...,       0.2751,       0.0518,      -1.6463  ],
            [      0.0921,       0.0161,      -0.0658,    ...,      -0.4107,      -0.4765,      -0.0765  ],
            [     -0.1245,       0.1560,       0.1286,    ...,       0.0675,       0.2849,       0.2750  ],
        ],
        [
            [      0.1914,      -0.7907,      -0.6164,    ...,      -1.3333,       0.7482,       1.3721  ],
            [     -0.2541,      -0.4277,       0.1286,    ...,       0.1658,      -0.4343,      -0.1337  ],
            [     -0.3760,       0.0331,       0.3724,    ...,      -0.0354,      -0.0535,       0.0460  ],
            [      0.1890,       0.0584,       0.3901,    ...,      -0.7189,       0.0697,      -0.0113  ],
        ],
        [
            [      0.2756,       0.0720,       0.0037,    ...,      -0.0402,      -1.1297,       0.4500  ],
            [     -0.0340,      -0.6315,       0.1084,    ...,      -0.5227,       1.6020,      -0.2162  ],
            [      0.0392,      -0.1006,      -0.2890,    ...,      -0.1238,      -0.0815,      -0.0954  ],
            [     -0.2787,       0.0793,       0.1030,    ...,      -0.0651,      -0.2563,       0.1328  ],
        ],
    ]
    sum = -585118.250000

See the large values in the first row. These overflow in some of the matrix Metal kernels leading to infs and nans.

To me, it looks like a problem with the model data. Either some quantization issue, or the source model data is highly unusual.

Here is a debug log for reference: debug.txt

cc @ngxson for viz

Edit: I did some more research after I posted this and it's not the "down" tensor, but I think it was actually the one before it "gate_inp" (I think) that produces the unusual values. I couldn't see any obvious problem though.

[adhusch]

@danielhanchen do you maybe have any insights on this specific layer 31/32 issue in Mistral Small 4 from your expertise? Might be even best to be discussed directly with the Mistral team depending on the further analysis? (i saw a user asked essentially the same question at the HF comment section of your quants). Thank you very much!

[ShivaThomas]

I'm having the same issue here. Mistral Small 4 in Llama.cpp return "<SPECIAL_31>" as soon as the prompt gets more than a couple of lines long. Using Metal MacOS. Tried with and without FA.

Device: Apple M4 Max 128GB
Typical output is non-stop.

<SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31><SPECIAL_31>

[TheTom]

Reproduced the empty / repetitive output on M5 Max 128GB with the published IQ4_NL GGUF, using upstream master (9c69907).

I inspected the GGUF directly and found that several norm tensors contain invalid data (NaN, magnitudes up to ~3e38). For example, blk.0.ffn_norm.weight is corrupted, while blk.0.attn_norm.weight looks normal. Once that happens, the forward pass goes bad at ffn_norm-0 during decode, logits become NaN, and sampling falls into token 31 (<SPECIAL_31>).

The original HF source tensor is clean. Reading post_attention_layernorm.weight directly from the safetensors shard gives:

first5=[0.080, 0.166, 0.119, 0.103, 0.158]

I also ran the current upstream pipeline locally:

HF safetensors -> convert_hf_to_gguf.py -> BF16 GGUF
BF16 GGUF -> llama-quantize -> IQ4_NL GGUF
For the layers/tensors I checked, the norm weights came through correctly in both steps, matching the HF source. I could not reproduce the corruption with the current pipeline.

So my current conclusion is that the broken behavior is coming from corrupted published GGUFs, and that those files were likely produced by an older or otherwise different conversion/quantization pipeline.

@bartowski1182 @danielhanchen if helpful, it may be worth checking blk.0.ffn_norm.weight in the affected files and/or regenerating with current master.

[danielhanchen]

@TheTom Sorry just saw this! I regenerated and changed but I'm not sure if the overflows for the ffn norm are the issue - I rechecked an old SHA before my re-upload but it's fine. I'm checking the new uploads we just did to confirm

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.