Repository navigation
Update OV gguf Frontend - #38769
Open
preetha-intel wants to merge 7 commits into
Open
Update OV gguf Frontend #38769preetha-intel wants to merge 7 commits into
preetha-intel wants to merge 7 commits into
Conversation
Add decoder-independent VIEW materialization to the GGUF frontend, driven only by the view's shapes, byte strides, offset and optional dynamic axis: - dense crops with a runtime dynamic axis, rectangular and strided crops - mixed-radix grid views for non-dense layouts (fused QKV, interleaved projections) - views of non-dense sources, re-expressed over the source's logical tensor and verified element by element before use - a "kv_view" descriptor for KV-cache reads with runtime length and first stream - unsupported stride layouts are rejected instead of passed through silently; the inner-row flatten fallback that read row-crossing views as contiguous is removed Also make SSM_CONV index its operand shapes from the innermost axis, so a decoder may describe the conv weight with its own rank, and report the failing node and its operand shapes when a translator throws.
Translators no longer dispatch on a decoder-assigned case number. Every operation is described by named attributes and operand shapes that both the llama.cpp cgraph decoder and the native GGUF builder supply, so the same translator body serves both ingest paths: - VIEW: slice/reshape description, ggml strides/offset, or a kv_view descriptor; a layout reference is expressed as RESHAPE with shape_axes - RESHAPE: reshape_target / reference operand / merge_heads - ROPE: rope_mode (normal, neox, imrope, vision); sin/cos tables are memoized per configuration instead of a shared graph scaffold - FLASH_ATTN_EXT: ggml's [B, n_head, n_tokens, head_size] operands with has_mask/f32_attention; K/V produced by a head/token transpose are tiled before the transpose so stateful-SDPA fusion, MakeStateful and SDPAToPagedAttention keep their pattern - CPY: source_slice, reshape_target and in-place axis updates - GET_ROWS, POOL_2D, PERMUTE, CONT, SCALE, ROLL, SOLVE_TRI, ADD, REPEAT, TOP_K and others read explicit attributes and live operand shapes - SET_ROWS/CPY/SCALE in-place writes rebind storage aliases in TranslateSession; the shared mask/RoPE scaffolding is removed - NodeContext drops get_op_case, restricts by-name inputs to operands and adds a per-translation cache; the builder API drops its op_case argument Tests and docs are updated to the attribute contract.
Port the operations the llama.cpp ggml-openvino backend translated in-tree
so the frontend owns them for both decoders: CONV_2D_DW, CONV_3D,
CONV_TRANSPOSE_1D/2D, IM2COL_3D, MEAN, SUM, DUP (as CONT) and the
HARDSIGMOID/EXPM1/STEP/ROUND unary ops. They read ggml's op parameters as
typed attributes (conv_params, im2col_params) and live operand shapes, and
compute in f32 regardless of device. UPSCALE also accepts a fixed target
size ("upscale_sizes") for single-operand graphs.
preetha-intel
requested review from
mlukasze and
zKulesza
and removed request for
a team
October 12, 2026 10:28
preetha-intel
marked this pull request as draft
October 12, 2026 10:29
GGML_OP_DUP copies its operand into a tensor of the output's type, but it was translated as CONT, which ignores a type change. Add translate_dup: the CONT copy, followed by a Convert when the output's dst_type differs from the operand's type. Add a unit test that fails with the previous mapping.
preetha-intel
marked this pull request as ready for review
October 16, 2026 06:37
Resolve attention.cpp: keep the named-attribute add_op API (no op_case) and apply master's has_own_kv guards for shared-KV layers. Port the dspark builder's CONCAT calls to the same API.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details:
This PR lets the OpenVINO GGUF frontend serve as the translation layer for llama.cpp's ggml-openvino backend, in addition to loading GGUF files natively. llama.cpp hands the frontend a ggml graph through a decoder. The frontend converts that graph into an ov::Model, so the backend no longer needs its own op translators and fusion passes.
To do that, the frontend has to be independent of which decoder feeds it. Each op is now described by named attributes and live operand shapes. Both the native GGUF builder and the llama.cpp cgraph decoder can fill those in, and one translator body serves both.
Llama.cpp PR enabling OV frontend usage : ravi9/llama.cpp#342