fix: cherry-pick 16 upstream P0/high-P1 fixes (CUDA FA crashes + overflows, server DoS, OOB reads, tool-call parsing) - #202
Merged
Conversation
(cherry picked from commit 3686e9d)
…uring last changes (ggml-org#24653) * chat: fix an "oldie but goodie" grammar generator bug that surfaced during last changes * update erroneous case in PEG parser test (cherry picked from commit 0ae3f45)
* chat: harden peg-native tool call parsing accept an optional leading type: function field in build_json_tools_flat_keys so openai style tool calls parse on templates whose serialization opens on the name field. return a clean error and log the unparsed fragment on a final peg parse failure instead of throwing the raw parser position and input. keep the raw arguments string in func_args_not_string when it is not valid json instead of aborting the prompt render. * chat: surface peg-native parse failures a final peg parse failure threw the raw parser position and input. log the unparsed fragment and raise a clearer error instead, so a model output that does not match the expected format no longer fails silently with an empty assistant turn. minimal change, no behavior change on successful parses. * chat: handle openai style tool calls in peg-native * nits * common: scope OpenAI wrapper grammar trigger via autoparser flag * chat: gate type:function parsing leniency on the analysis flag Thread accept_openai_wrapper from the generator to build_json_tools_flat_keys so the leading "type": "function" field is accepted only when openai_wrapper_trigger is set. (cherry picked from commit 581e8ec)
* server: fix non-bound n_discard value * Update tools/server/server-context.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> (cherry picked from commit 159d093)
…gml-org#24893) line_start -1 normalized to n+1, so append inserted at lines.begin() + n + 1, one past end() -> heap-buffer-overflow in vector::_M_range_insert. Normalize -1 to n (insert at end()), restrict -1 to append mode and reject it for replace/delete instead of silently clobbering the last line. Parenthesize the insert offset so empty-file append computes the position as int first, avoiding a transient begin() - 1 on a null vector data pointer. (cherry picked from commit d0f9d2e)
…gml-org#24706) * ggml : address integer overflows in binary ops CUDA implementation * ggml : add size_t casts to avoid integer overflows * ggml : add more asserts checking integer overflows in binary ops CUDA implementation --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> (cherry picked from commit f728ada)
* CUDA: fix Gemma E4B MTP FlashAttention * remove unused template declaration (cherry picked from commit e495d1e)
…ask strides in flash_attn_mask_to_KV_max kernel (ggml-org#24945) Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> (cherry picked from commit 0eca4d4)
* ggml-cpu: fix SVE leftover path in ggml_vec_dot_f32 2D convolutions with kernel size 9 produced different results on SVE enabled ARM devices. After debugging it turned out that ggml_vec_dot_f32 was using data from inactive lanes. Use svmla_f32_m(pg, sum1, ax1, ay1) so inactive lanes retain sum1. * cont : clean-up --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> (cherry picked from commit c16c35b)
…-org#18750) * fix: OOB reads in UGM tokenizer (precompiled_charsmap handling) - Validate minimum size (4 bytes) before reading xcda_blob_size - Use strnlen with bounds check instead of unsafe strlen Both issues allow heap-buffer-overflow from malicious T5/UGM GGUF files. * Replace unsafe strnlen() with a bounds-checked loop that scans for \0 within the remaining array size. * move bounds checks to load * typo merge fix --------- Co-authored-by: hourhl <hourhl8200@gmail.com> Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> (cherry picked from commit 4a7ee31)
…gml-org#23936) * speculative : fix out-of-bounds read in ngram-map on prompt shrink * speculative : fix ngram-map cleanup cutoff after prompt shrink (cherry picked from commit defa95c)
(cherry picked from commit 86961ef)
Tensor parallelism (-sm tensor) combined with -ncmoe (CPU-offloaded MoE
experts) aborts during warm-up on MoE models with
GGML_ASSERT(ggml_is_contiguous(tensor)) in ggml-backend-meta.cpp.
The failing tensor is the MoE router output (ffn_moe_topk): it is mirrored
(GGML_BACKEND_SPLIT_AXIS_MIRRORED, replicated across backends since routing
must be identical) and happens to be a non-contiguous view.
ggml_backend_meta_buffer_{get,set}_tensor asserted contiguity before
consulting the split state, so a mirrored non-contiguous tensor tripped the
assert even though the GGML_BACKEND_SPLIT_AXIS_MIRRORED case right below
already handles it.
Move the split-state lookup above the assert and allow the mirrored case in
both get_tensor and set_tensor.
Diagnosis credit to the reporter (@nathanmp).
Fixes ggml-org#24886
Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>
(cherry picked from commit 4b2a0cd)
TheTom
changed the title
TheTom
marked this pull request as ready for review
July 9, 2026 13:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picks verified P0/high-P1 bug fixes from upstream. Every pick verified against its actual diff for fork exposure. All apply clean (one resolved); local CUDA build passes; turbo3 16-chunk PPL bit-identical to known-good (9.0307); fused decode intact;
test-chat/test-quantize-fnspass.P0 (crash / corruption on shipped paths)
e495d1e74CUDA: fix Gemma E4B MTP FlashAttention (CUDA: fix Gemma E4B MTP FlashAttention ggml-org/llama.cpp#25148) — hard GGML_ABORT on CUDA +-fa on+ MTP + Gemma-4 E4B0eca4d490CUDA: widen KQ-mask strides to int64 (cuda : prevent integer truncation and overflow errors when using KQ mask strides in flash_attn_mask_to_KV_max kernel ggml-org/llama.cpp#24945) — illegal memory access / wrong FA output at long ctxf728adab6CUDA binbcast: 64-bit index math + asserts (ggml : address integer overflows in binary ops CUDA implementation ggml-org/llama.cpp#24706) — silent wrong add/mul output past 2^314b2a0cdeeggml: fix tensor-parallel +-ncmoewarmup abort on MoE (ggml : fix tensor-parallel + -ncmoe crash on MoE models ggml-org/llama.cpp#25028)86961efd5vulkan: fix 32-bit int overflow in CEIL_DIV (vulkan: fix 32-bit integer overflow in CEIL_DIV ggml-org/llama.cpp#25245) — long-ctx Vulkancb295bf59CUDA FA: validate V-type as well as K-type (CUDA: extend K-type validation to V-types for flash attention ggml-org/llama.cpp#24403) —-ctk q8_0 -ctv iq4_nlcrashed instead of CPU fallback. Resolved against our turbo dispatch: turbo types added to the newggml_cuda_fattn_kv_type_supported()helper, head-dim %64 geometry kept as a separate K-and-V check. Verified: crash case now completes via fallback, turbo PPL bit-identical.High P1
159d093a4server: bound n_discard in ctx shift (server: fix non-bound n_discard value (ctx shifting) ggml-org/llama.cpp#24786) — unauthenticated remote DoS9fa94c2f2+45d195138chat: tool-call whitespace + grammar pair (chat: fix whitespace problems once and for all ggml-org/llama.cpp#24624 + chat: fix an "oldie but goodie" grammar generator bug that surfaced during last changes ggml-org/llama.cpp#24653)ac4105d68Vulkan FA: bias before softmax (vulkan: Apply bias before softmax in FA, to avoid overflow ggml-org/llama.cpp#24909) — fp16 overflow to NaNc16c35b81ggml-cpu: SVE dot-product leftover fix (ggml-cpu: fix SVE leftover path in ggml_vec_dot_f32 ggml-org/llama.cpp#24699)3aab96ef1server: edit_file append heap overflow (server: fix edit_file crash on append at end of file (line_start -1) ggml-org/llama.cpp#24893)75967d392chat: accept OpenAI-wrapper tool calls (chat: harden peg-native tool call parsing ggml-org/llama.cpp#24329)3c2aabdf8CUDA: REPEAT supports_op whitelist (CUDA: only support F32/F16 for GGML_OP_REPEAT ggml-org/llama.cpp#24533)4a7ee3126fix OOB reads in UGM tokenizer (fix: OOB reads in UGM tokenizer (precompiled_charsmap handling) ggml-org/llama.cpp#18750) — malformed-GGUF hardeningdefa95c30speculative: fix OOB read in ngram-map on prompt shrink (speculative : fix out-of-bounds read in ngram-map on prompt shrink ggml-org/llama.cpp#23936)Deliberately skipped (verified not applicable / below bar)
e2e7a9b2d(collides with our Gemma4UA set_size validation), CORS-proxye27f30859(opt-in flag unused), dsv4 KV fix024c46ae4(no DSv4 arch in fork), recurrent parallel-decode230ea9d21(enablement, not a fix), draft fitf5525f7e7(minor over-allocation only)