Eval bug: on HIP, a sequence whose prompt shares a llama_decode() batch with another sequence's decode row gets corrupted logits, and every call reports success

[dlsniper]

Name and Version

$ ./llama-cli --version
version: 0.3.0-dev (build 10798, commit c390d0abb)
built with GNU 13.3.0 for Linux x86_64

Official release builds, no local patches. First measured on b10760, reproduced on b10798 and on the b10819 release tarball.

Operating systems

Linux — Ubuntu 24.04.4 LTS container (glibc 2.39, g++ 13.3.0) on a Fedora Linux
44 host, kernel 7.1.9-200.fc44.x86_64

GGML backends

HIP

Hardware

Radeon 8060S Graphics, gfx1151 (AMD Ryzen AI MAX+ 395), 57344 MiB, x64.
Backend reports Wave Size: 32, VMM: no. Reproduced under ROCm 7.2.4 and
under ROCm 10.0 (amdrocm-core10.0-gfx1151 10.0.0-4, HIP runtime 7.15.26333),
both in an Ubuntu 24.04.4 container with /dev/kfd and /dev/dri passed
through, on a Fedora 44 host.

Controls on the same GPU, same host, same build, same model: Vulkan
(RADV GFX1151, Mesa 25.2.8) and CPU. Neither ever corrupts a sequence.

Models

Qwen3-1.7B-Q4_K_M, KV cache q8_0 and f16.

Problem description & steps to reproduce

A batch that carries one sequence's prompt rows together with another sequence's
decode row returns corrupted logits for the joining sequence. Its very first
sampled token is already wrong and the generation collapses into a 3-to-7-token
loop that runs to the token cap.

• Nothing reports a failure: llama_decode() returns 0,
  llama_get_logits_ith() returns a pointer for every requested row, and
  sampling proceeds normally — on logits that are garbage.
• llama-server -np N produces exactly this batch shape whenever a slot starts
  while another slot is generating.
• Two sequences whose prompts are in the same batch, decoding in lockstep
  afterwards, are clean. The joining-mid-generation shape is what breaks.

Deterministic reproduction, repro.cpp below, greedy argmax over
llama_get_logits_ith() so no sampler is involved, one context per run,
-stagger K submits sequence 1's prompt rows in the same batch as sequence 0's
K-th decode row:

./repro Qwen3-1.7B-Q4_K_M.gguf -ngl 99 -n 512 -stagger 40

Each run first generates the same prompt alone (n_seq_max = 1), then generates
it on both sequences, and reports for each: first_div, the index at which it
leaves the solo stream, maxrun, the longest run of one repeated token, and
distinct, the number of distinct tokens in 512.

backend	runs	seq 1 corrupted	seq 1 first_div	maxrun	distinct
HIP	10	10	0	509 or 1	3 or 7
Vulkan	10	0	31	1	114
CPU	2	0	8	1	149

• HIP fails on every run, and the joining sequence is wrong from its first
  token. Two collapse shapes alternate across runs: : M: : : : … (6 of 10,
  maxrun 509 of 512) and . The Word: Marmot. repeated (4 of 10). The Vulkan
  control is byte-identical across all ten runs.
• Sequence 0, the one already generating, stays coherent, but its stream leaves
  the solo baseline at exactly the step the second prompt joins — first_div 1,
  5 and 40 for -stagger 1, 5 and 40. On CPU it is byte-identical to the
  solo run over all 512 tokens, which is the semantics being violated: a
  sequence joining the batch should perturb nothing.

Variations, HIP, one context per row, -n 512:

variation	result
-stagger 1	corrupt
-stagger 5	corrupt
-stagger 40	corrupt
-stagger 40 -f16	corrupt
-stagger 40 -f16 -nofa	corrupt
-nseq 4 -stagger 20	3 of 4 sequences corrupt
-stagger 0 (both prompts in one batch)	clean
-nseq 2 -live 1 (n_seq_max = 2, one sequence live)	clean
n_seq_max = 1	clean

• Not flash attention, not the quantized KV cache, not the number of sequences.
  -nseq 2 -live 1 is byte-identical to the n_seq_max = 1 run over all 512
  tokens, so the context configuration alone is not it either.
• The -nofa rows use f16, since a quantized V cache requires flash attention
  enabled or auto.
• Ack: a joining sequence's stream does drift from the solo run on every backend
  a few tokens in — first_div 8 on HIP at -stagger 0, 31 on Vulkan, 8 on
  CPU — because batch shape changes kernel selection and reduction order and a
  greedy near-tie flips. That is expected and is not this bug. The bug is a
  sequence that is wrong at token 0 and collapses to 3 distinct tokens, on HIP
  only.
• Ack: a Qwen3-0.6B-Q8_0 does show the same HIP collapse (maxrun 512), but that
  model loops under greedy decoding on Vulkan too, so it is not a usable control.
  The 1.7B separates the backends cleanly.

The same defect through llama-server, which is how it was first found. Two
identical requests sent concurrently as the first thing a freshly started server
serves:

llama-server -m Qwen3-1.7B-Q4_K_M.gguf -c 8192 -np 2 -ngl 99 -ctk q8_0 -ctv q8_0
# reports: n_slots = 2, n_ctx_slot = 4096, kv_unified = 'false'

POST /v1/chat/completions
{"messages": [{"role": "user", "content": "Echo back the word: Marmot"}],
 "max_tokens": 2048, "temperature": 0, "top_k": 1, "top_p": 1, "seed": 1234}

temperature 0 with top_k 1 makes the expectation exact rather than
statistical: both responses must equal each other, and both must equal what the
same prompt gets from a server started with -np 1. Token counts are
completion_tokens, "matches" means the whole generation hashes equal to that
backend's own -np 1 baseline.

prompt	-np 1 baseline	set A (5 prompts)	set B (8 prompts)
Gorilla	97 tokens	matches	matches
Platypus	92 tokens	matches	matches
Marmot	107 tokens	diverged, 323 tokens	diverged, 323 tokens
Ocelot	99 tokens	diverged, 266 tokens	matches
Narwhal	263 tokens	matches	matches
Ibex	91 tokens	—	matches
Tapir	99 tokens	—	matches
Lemur	208 tokens	—	matches

• Three of thirteen pairs diverged, each starting with a token emitted before
  the chat template's <think> (log output below), and never rejoining.
• Marmot diverged on both cold starts with byte-identical wrong output — same 323
  tokens, same hash. Either slot can be the corrupted one: slot 1 in set A, slot
  0 in set B, same wrong output both times. The clean slot reproduces the -np 1
  baseline exactly.
• Vulkan control, same script: eight cold starts, eight pairs, every generation
  byte-identical to its -np 1 baseline.
• Kronk's Test_BatchChatConcurrent (sdk/kronk/tests/qwen3/batch_test.go), https://github.com/ardanlabs/kronk, is
  the same shape — ten concurrent requests against two slots — and surfaces a
  corrupted sequence as an empty message. Its HIP CI history at b10798:

ROCm	run	result
7.2.4	33928022827	pass
7.2.4	33929435557	fail — 1 empty (request 6)
10.0	33931215130	fail — 2 empty (requests 0 and 9)

• Through the server it is intermittent, because whether a slot joins
  mid-generation depends on arrival timing; repro.cpp pins that timing and
  fails every run, which makes it the better place to test a candidate patch.
• The ROCm 10.0 leg runs gfx1151-specific code objects from AMD's
  amdrocm10.0 packaging rather than 7.2.x's generic build, so that is a
  different execution path reaching the same defect. Neither the runtime nor the
  llama.cpp build carries it.

Expected: llama_decode() on a batch holding rows for two sequences produces,
for each sequence, the logits that sequence would have received had it been
decoded alone, as it does on Vulkan and CPU.

Impact: multi-sequence serving on this backend returns wrong tokens with no
signal that anything went wrong. Every API call reports success, so a caller
cannot detect it, retry it, or fall back. llama-server -np N and any caller
setting n_seq_max > 1 are affected, and the joining request is the one that
loses — it burns its whole token budget on a 3-token loop.

First Bad Commit

Not bisected. Present on b10760, b10798 and b10819; no earlier build
tested, so whether this is a regression is unknown.

Relevant log output

repro.cpp, HIP vs Vulkan, -stagger 40

# HIP, ROCm 10.0, 3 of the 10 runs
cfg: ngl=99 n=512 n_seq_max=2 live=2 stagger=40 type_kv=q8_0 flash_attn=auto
solo:   512 tokens eog=0 maxrun=1    distinct=140  |   Okay, the user wants me to "echo back the word: Marmot". Let me thin...
seq 0:  512 tokens eog=0 maxrun=1    distinct=137  first_div=40   ok         |   Okay, the user wants me to "echo back the word: Marmot". Let me thin...
seq 1:  512 tokens eog=0 maxrun=509  distinct=3    first_div=0    DEGENERATE | : M: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
result: 1 of 2 sequences degenerate

seq 1:  512 tokens eog=0 maxrun=1    distinct=7    first_div=0    DEGENERATE | . The Word: Marmot. The Word: Marmot. The Word: Marmot. The Word: Marm
result: 1 of 2 sequences degenerate

seq 1:  512 tokens eog=0 maxrun=509  distinct=3    first_div=0    DEGENERATE | : M: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
result: 1 of 2 sequences degenerate

# Vulkan, same GPU, same host, same build, same model — all 10 runs identical
cfg: ngl=99 n=512 n_seq_max=2 live=2 stagger=40 type_kv=q8_0 flash_attn=auto
solo:   512 tokens eog=0 maxrun=1    distinct=152  |   Okay, the user wants me to echo back the word "Marmot". Let me think
seq 1:  512 tokens eog=0 maxrun=1    distinct=114  first_div=31   ok         |   Okay, the user wants me to echo back the word "Marmot". Let me think
result: 0 of 2 sequences degenerate

# CPU, same shape — clean, and the resident sequence is untouched by the join
cfg: ngl=99 n=512 n_seq_max=2 live=2 stagger=40 type_kv=q8_0 flash_attn=auto
solo:   512 tokens eog=0 maxrun=1    distinct=183  |   Okay, the user wants me to echo back the word "Marmot". Let me think
seq 0:  512 tokens eog=0 maxrun=1    distinct=183  first_div=512  ok         |   Okay, the user wants me to echo back the word "Marmot". Let me think
seq 1:  512 tokens eog=0 maxrun=1    distinct=149  first_div=8    ok         |   Okay, the user wants me to "echo back the word: Marmot". Let me thin...
result: 0 of 2 sequences degenerate

# HIP, n_seq_max=2 with one sequence live — identical to the n_seq_max=1 run
cfg: ngl=99 n=512 n_seq_max=2 live=1 stagger=40 type_kv=q8_0 flash_attn=auto
seq 0:  512 tokens eog=0 maxrun=1    distinct=140  first_div=512  ok         |   Okay, the user wants me to "echo back the word: Marmot". Let me thin...
result: 0 of 1 sequences degenerate

# HIP, both prompts in one batch — clean
cfg: ngl=99 n=512 n_seq_max=2 live=2 stagger=0 type_kv=q8_0 flash_attn=auto
seq 0:  512 tokens eog=0 maxrun=1    distinct=137  first_div=8    ok         |   Okay, the user wants me to echo back the word "Marmot". Let me think
seq 1:  512 tokens eog=0 maxrun=1    distinct=137  first_div=8    ok         |   Okay, the user wants me to echo back the word "Marmot". Let me think
result: 0 of 2 sequences degenerate

# HIP, four sequences joining 20 tokens apart
cfg: ngl=99 n=512 n_seq_max=4 live=4 stagger=20 type_kv=q8_0 flash_attn=auto
seq 0:  512 tokens eog=0 maxrun=2    distinct=155  first_div=20   ok         |   Okay, the user wants me to "echo back the word: Marmot". Let meal, I
seq 1:  512 tokens eog=0 maxrun=1    distinct=8    first_div=0    DEGENERATE | . The Word: Marmot. The Word: Marmot. The Word: Marm the Word: Marmot....
seq 2:  512 tokens eog=0 maxrun=481  distinct=7    first_div=0    DEGENERATE | .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  Word: Marm
seq 3:  512 tokens eog=0 maxrun=317  distinct=2    first_div=0    DEGENERATE | ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .....
result: 3 of 4 sequences degenerate

llama-server, -np 2, first pair after a cold start

# "Echo back the word: Marmot"
# slot that matches the -np 1 baseline, 107 tokens
expected:   Okay, the user wants me to echo back the word "Marmot". Let me think.

# slot that diverged, 323 tokens, byte-identical across both cold starts
observed:   spiritedly
            <think>
            Okay, the user wants me to echo back the word "Marmot". Let me think.

The corrupted prefix is spirits in the Ocelot case.

Build and run

curl -sLO https://github.com/ggml-org/llama.cpp/releases/download/b10798/llama-b10798-bin-ubuntu-rocm-10.0-x64.tar.gz
curl -sL -o src.tar.gz https://github.com/ggml-org/llama.cpp/archive/refs/tags/b10798.tar.gz
mkdir -p rocm src && tar xzf llama-b10798-bin-ubuntu-rocm-10.0-x64.tar.gz -C rocm && tar xzf src.tar.gz -C src

g++ -O2 -std=c++17 repro.cpp -o rocm/llama-b10798/repro \
  -I src/llama.cpp-b10798/include -I src/llama.cpp-b10798/ggml/include \
  -L rocm/llama-b10798 -lllama -lggml -lggml-base -Wl,-rpath,'$ORIGIN'

./rocm/llama-b10798/repro Qwen3-1.7B-Q4_K_M.gguf -ngl 99 -n 512 -stagger 40

The binary must sit in the release directory — ggml_backend_load_all()
searches the executable's own directory for the backend .so files. Exit status
is the number of corrupted sequences, so -stagger 40 in a loop counts failures
directly.

repro.cpp

#include "llama.h"
#include "ggml-backend.h"
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <string>
#include <vector>
#include <algorithm>

static const char * PROMPT = "Echo back the word: Marmot";

static llama_token argmax(const float * l, int n) {
    int b = 0;
    for (int i = 1; i < n; ++i) if (l[i] > l[b]) b = i;
    return b;
}

static std::string piece(const llama_vocab * vocab, llama_token t) {
    char buf[128];
    const int n = llama_token_to_piece(vocab, t, buf, sizeof(buf), 0, true);
    return n > 0 ? std::string(buf, n) : std::string("?");
}

static std::string text(const llama_vocab * vocab, const std::vector<llama_token> & t, size_t max) {
    std::string s;
    for (size_t i = 0; i < t.size() && s.size() < max; ++i) s += piece(vocab, t[i]);
    for (char & c : s) if (c == '\n') c = ' ';
    return s.size() > max ? s.substr(0, max) + "..." : s;
}

static size_t common(const std::vector<llama_token> & a, const std::vector<llama_token> & b) {
    size_t i = 0;
    while (i < a.size() && i < b.size() && a[i] == b[i]) ++i;
    return i;
}

static size_t distinct(const std::vector<llama_token> & t) {
    std::vector<llama_token> u(t);
    std::sort(u.begin(), u.end());
    u.erase(std::unique(u.begin(), u.end()), u.end());
    return u.size();
}

static size_t maxrun(const std::vector<llama_token> & t) {
    size_t best = t.empty() ? 0 : 1, run = best;
    for (size_t i = 1; i < t.size(); ++i) {
        run = t[i] == t[i-1] ? run + 1 : 1;
        if (run > best) best = run;
    }
    return best;
}

struct seq_state {
    llama_pos pos = 0;
    int  row     = -1;
    bool started = false;
    bool done    = false;
    bool eog     = false;
    std::vector<llama_token> out;
};

static bool run(llama_model * model, int n_seq, int n_live, int stagger, int n_gen, bool f16, bool nofa,
                std::vector<seq_state> & st) {
    const llama_vocab * vocab   = llama_model_get_vocab(model);
    const int           n_vocab = llama_vocab_n_tokens(vocab);

    llama_token toks[64];
    const int np = llama_tokenize(vocab, PROMPT, (int) strlen(PROMPT), toks, 64, true, true);
    if (np <= 0) return false;

    llama_context_params cp = llama_context_default_params();
    cp.n_ctx = 8192; cp.n_batch = 512; cp.n_ubatch = 512; cp.n_seq_max = n_seq; cp.no_perf = true;
    cp.type_k = cp.type_v = f16 ? GGML_TYPE_F16 : GGML_TYPE_Q8_0;
    cp.flash_attn_type = nofa ? LLAMA_FLASH_ATTN_TYPE_DISABLED : LLAMA_FLASH_ATTN_TYPE_AUTO;

    llama_context * ctx = llama_init_from_model(model, cp);
    if (!ctx) return false;

    llama_batch b = llama_batch_init(512, 0, n_seq);
    auto add = [&](llama_token id, llama_pos p, llama_seq_id s, bool l) {
        const int i = b.n_tokens;
        b.token[i] = id; b.pos[i] = p; b.n_seq_id[i] = 1; b.seq_id[i][0] = s; b.logits[i] = l; b.n_tokens++;
    };

    st.assign(n_live, {});

    for (int step = 0; ; ++step) {
        b.n_tokens = 0;
        for (int s = 0; s < n_live; ++s) {
            seq_state & q = st[s];
            if (q.done) continue;
            if (!q.started) {
                if (step < s * stagger) continue;
                for (int i = 0; i < np; ++i) add(toks[i], i, s, i == np - 1);
                q.row = b.n_tokens - 1; q.pos = np; q.started = true;
            } else {
                add(q.out.back(), q.pos++, s, true);
                q.row = b.n_tokens - 1;
            }
        }
        if (b.n_tokens == 0) break;

        const int rc = llama_decode(ctx, b);
        if (rc) { printf("FAIL: decode rc=%d step=%d rows=%d\n", rc, step, b.n_tokens); break; }

        bool any = false;
        for (int s = 0; s < n_live; ++s) {
            seq_state & q = st[s];
            if (q.done || !q.started) { any |= !q.done; continue; }
            const float * lg = llama_get_logits_ith(ctx, q.row);
            if (!lg) { printf("FAIL: logits NULL seq=%d row=%d\n", s, q.row); q.done = true; continue; }
            const llama_token t = argmax(lg, n_vocab);
            q.out.push_back(t);
            if (llama_vocab_is_eog(vocab, t)) { q.eog = true; q.done = true; }
            if ((int) q.out.size() >= n_gen) q.done = true;
            any |= !q.done;
        }
        if (!any) break;
    }

    llama_batch_free(b);
    llama_free(ctx);
    return true;
}

int main(int argc, char ** argv) {
    if (argc < 2) { fprintf(stderr, "usage: %s model.gguf [-ngl N] [-n N] [-nseq N] [-live N] [-stagger N] [-f16] [-nofa]\n", argv[0]); return 2; }
    int ngl = 99, n_gen = 512, n_seq = 2, n_live = 0, stagger = 0;
    bool f16 = false, nofa = false;
    for (int i = 2; i < argc; ++i) {
        if      (!strcmp(argv[i], "-ngl")     && i+1 < argc) ngl     = atoi(argv[++i]);
        else if (!strcmp(argv[i], "-n")       && i+1 < argc) n_gen   = atoi(argv[++i]);
        else if (!strcmp(argv[i], "-nseq")    && i+1 < argc) n_seq   = atoi(argv[++i]);
        else if (!strcmp(argv[i], "-live")    && i+1 < argc) n_live  = atoi(argv[++i]);
        else if (!strcmp(argv[i], "-stagger") && i+1 < argc) stagger = atoi(argv[++i]);
        else if (!strcmp(argv[i], "-f16"))                   f16     = true;
        else if (!strcmp(argv[i], "-nofa"))                  nofa    = true;
        else { fprintf(stderr, "bad arg %s\n", argv[i]); return 2; }
    }

    if (n_live <= 0 || n_live > n_seq) n_live = n_seq;

    ggml_backend_load_all();
    llama_backend_init();

    llama_model_params mp = llama_model_default_params();
    mp.n_gpu_layers = ngl;
    llama_model * model = llama_model_load_from_file(argv[1], mp);
    if (!model) { printf("FAIL: model load\n"); return 2; }
    const llama_vocab * vocab = llama_model_get_vocab(model);

    printf("cfg: ngl=%d n=%d n_seq_max=%d live=%d stagger=%d type_kv=%s flash_attn=%s\n",
           ngl, n_gen, n_seq, n_live, stagger, f16 ? "f16" : "q8_0", nofa ? "disabled" : "auto");

    std::vector<seq_state> solo, test;
    if (!run(model, 1, 1, 0, n_gen, f16, nofa, solo))                { printf("FAIL: solo\n"); return 2; }
    if (!run(model, n_seq, n_live, stagger, n_gen, f16, nofa, test)) { printf("FAIL: shared\n"); return 2; }

    printf("solo:  %4d tokens eog=%d maxrun=%-4zu distinct=%-4zu | %s\n",
           (int) solo[0].out.size(), solo[0].eog, maxrun(solo[0].out), distinct(solo[0].out),
           text(vocab, solo[0].out, 70).c_str());

    int degenerate = 0;
    for (size_t s = 0; s < test.size(); ++s) {
        const std::vector<llama_token> & o = test[s].out;
        const size_t run = maxrun(o), uniq = distinct(o), i = common(o, solo[0].out);
        const bool bad = o.empty() || run >= 8 || uniq <= 8;
        if (bad) degenerate++;
        printf("seq %zu: %4d tokens eog=%d maxrun=%-4zu distinct=%-4zu first_div=%-4zu %s | %s\n",
               s, (int) o.size(), test[s].eog, run, uniq, i, bad ? "DEGENERATE" : "ok        ",
               text(vocab, o, 70).c_str());
    }

    printf("result: %d of %d sequences degenerate\n", degenerate, (int) test.size());

    llama_model_free(model);
    llama_backend_free();
    return degenerate ? 1 : 0;
}

The reproducer and the underlying investigation were AI-assisted.

[dlsniper]

Completing the Vulkan control block above — the seq 0: row was filtered out of that paste. Two consecutive runs, same command (-ngl 99 -n 512 -stagger 40), byte-identical to each other:

cfg: ngl=99 n=512 n_seq_max=2 live=2 stagger=40 type_kv=q8_0 flash_attn=auto
solo:   512 tokens eog=0 maxrun=1    distinct=152  |   Okay, the user wants me to echo back the word "Marmot". Let me think
seq 0:  512 tokens eog=0 maxrun=1    distinct=152  first_div=298  ok         |   Okay, the user wants me to echo back the word "Marmot". Let me think
seq 1:  512 tokens eog=0 maxrun=1    distinct=114  first_div=31   ok         |   Okay, the user wants me to echo back the word "Marmot". Let me think
result: 0 of 2 sequences degenerate

That makes the resident sequence's first_div — how long the already-generating sequence stays on the single-sequence stream after the second prompt joins its batch — the cleanest three-way split in the report:

backend	seq 0 first_div	reading
CPU	512 of 512	the join has no effect at all
Vulkan	298	drifts late, consistent with accumulated numerics
HIP	40	leaves at exactly the step the prompt joins — -stagger 1 and 5 give first_div 1 and 5

seq 1, the joining sequence, is the one that collapses, and only on HIP.

[IMbackK]

should be fixed by #28604