Skip to content

Update OV gguf Frontend - #38769

Open
preetha-intel wants to merge 7 commits into
openvinotoolkit:masterfrom
preetha-intel:ov-gguf-passes
Open

preetha-intel wants to merge 7 commits into
openvinotoolkit:masterfrom
preetha-intel:ov-gguf-passes

Conversation

@preetha-intel

@preetha-intel preetha-intel commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Details:

This PR lets the OpenVINO GGUF frontend serve as the translation layer for llama.cpp's ggml-openvino backend, in addition to loading GGUF files natively. llama.cpp hands the frontend a ggml graph through a decoder. The frontend converts that graph into an ov::Model, so the backend no longer needs its own op translators and fusion passes.

To do that, the frontend has to be independent of which decoder feeds it. Each op is now described by named attributes and live operand shapes. Both the native GGUF builder and the llama.cpp cgraph decoder can fill those in, and one translator body serves both.

  1. Materialize strided and KV-cache VIEWs generically (acb90e0)
  • VIEW materialization now depends only on the view's shapes, byte strides, offset and optional dynamic axis, not on which decoder produced it.
  • Supported view kinds:
    • dense crops with a runtime dynamic axis
    • rectangular and strided crops
    • mixed-radix grid views for non-dense layouts such as fused QKV and interleaved projections
    • views of non-dense sources, re-expressed over the source's logical tensor and checked element by element before use
    • a kv_view descriptor for KV-cache reads with a runtime length and first stream
  • Unsupported stride layouts are now rejected. Previously the inner-row flatten fallback could read row-crossing views as if they were contiguous, which was silently wrong.
  • SSM_CONV indexes its operand shapes from the innermost axis, so a decoder can describe the conv weight with its own rank.
  • When a translator throws, the error now reports the failing node and its operand shapes.
  1. Describe operations with named attributes instead of op_case (5440037)
  • Translators no longer dispatch on a decoder-assigned case number. Each op is described by named attributes and operand shapes, which both decoders supply.
    • VIEW and RESHAPE: slice/reshape descriptions, ggml strides and offset, kv_view, reshape_target, shape_axes and merge_heads.
    • ROPE: rope_mode is one of normal, neox, imrope or vision. The sin/cos tables are now memoized per configuration, replacing the shared graph scaffold.
    • FLASH_ATTN_EXT: it takes ggml's [B, n_head, n_tokens, head_size] operands, with has_mask and f32_attention. K/V produced by a head/token transpose are tiled before the transpose, so stateful-SDPA fusion, MakeStateful and SDPAToPagedAttention still match their patterns.
    • CPY: source_slice, reshape_target and in-place axis updates.
    • Other ops: GET_ROWS, POOL_2D, PERMUTE, CONT, SCALE, ROLL, SOLVE_TRI, ADD, REPEAT, TOP_K and others read explicit attributes and live operand shapes.
  • In-place writes (SET_ROWS, CPY, SCALE) now rebind storage aliases in TranslateSession. The shared mask and RoPE scaffolding is removed.
  • NodeContext drops get_op_case, restricts by-name inputs to operands, and adds a per-translation cache. The builder API drops its op_case argument.
  • Tests and docs are updated to the attribute contract.
  1. Translate ops added by the llama.cpp OpenVINO backend (b91ed60)
  • Ports the ops the backend used to translate itself, so the frontend owns them:
    • CONV_2D_DW, CONV_3D, CONV_TRANSPOSE_1D/2D, IM2COL_3D
    • MEAN, SUM, DUP (as CONT)
    • the HARDSIGMOID, EXPM1, STEP and ROUND unary ops
  • These read ggml's op parameters as typed attributes (conv_params, im2col_params) and live operand shapes, and compute in f32 regardless of device.
  • UPSCALE also accepts a fixed target size (upscale_sizes) for single-operand graphs.

Llama.cpp PR enabling OV frontend usage : ravi9/llama.cpp#342

Add decoder-independent VIEW materialization to the GGUF frontend, driven only by
the view's shapes, byte strides, offset and optional dynamic axis:

- dense crops with a runtime dynamic axis, rectangular and strided crops
- mixed-radix grid views for non-dense layouts (fused QKV, interleaved projections)
- views of non-dense sources, re-expressed over the source's logical tensor and
  verified element by element before use
- a "kv_view" descriptor for KV-cache reads with runtime length and first stream
- unsupported stride layouts are rejected instead of passed through silently; the
  inner-row flatten fallback that read row-crossing views as contiguous is removed

Also make SSM_CONV index its operand shapes from the innermost axis, so a decoder
may describe the conv weight with its own rank, and report the failing node and its
operand shapes when a translator throws.
Translators no longer dispatch on a decoder-assigned case number. Every
operation is described by named attributes and operand shapes that both
the llama.cpp cgraph decoder and the native GGUF builder supply, so the
same translator body serves both ingest paths:

- VIEW: slice/reshape description, ggml strides/offset, or a kv_view
  descriptor; a layout reference is expressed as RESHAPE with shape_axes
- RESHAPE: reshape_target / reference operand / merge_heads
- ROPE: rope_mode (normal, neox, imrope, vision); sin/cos tables are
  memoized per configuration instead of a shared graph scaffold
- FLASH_ATTN_EXT: ggml's [B, n_head, n_tokens, head_size] operands with
  has_mask/f32_attention; K/V produced by a head/token transpose are tiled
  before the transpose so stateful-SDPA fusion, MakeStateful and
  SDPAToPagedAttention keep their pattern
- CPY: source_slice, reshape_target and in-place axis updates
- GET_ROWS, POOL_2D, PERMUTE, CONT, SCALE, ROLL, SOLVE_TRI, ADD, REPEAT,
  TOP_K and others read explicit attributes and live operand shapes
- SET_ROWS/CPY/SCALE in-place writes rebind storage aliases in
  TranslateSession; the shared mask/RoPE scaffolding is removed
- NodeContext drops get_op_case, restricts by-name inputs to operands and
  adds a per-translation cache; the builder API drops its op_case argument

Tests and docs are updated to the attribute contract.
Port the operations the llama.cpp ggml-openvino backend translated in-tree
so the frontend owns them for both decoders: CONV_2D_DW, CONV_3D,
CONV_TRANSPOSE_1D/2D, IM2COL_3D, MEAN, SUM, DUP (as CONT) and the
HARDSIGMOID/EXPM1/STEP/ROUND unary ops. They read ggml's op parameters as
typed attributes (conv_params, im2col_params) and live operand shapes, and
compute in f32 regardless of device. UPSCALE also accepts a fixed target
size ("upscale_sizes") for single-operand graphs.
@preetha-intel
preetha-intel requested review from a team as code owners October 12, 2026 10:28
@preetha-intel
preetha-intel requested review from mlukasze and zKulesza and removed request for a team October 12, 2026 10:28
@github-actions github-actions Bot added category: build OpenVINO cmake script / infra category: docs OpenVINO documentation category: GGUF FE labels Oct 12, 2026
@preetha-intel preetha-intel changed the title Ov gguf passes Update OV gguf Frontend Oct 12, 2026
@preetha-intel
preetha-intel marked this pull request as draft October 12, 2026 10:29
GGML_OP_DUP copies its operand into a tensor of the output's type, but it was
translated as CONT, which ignores a type change. Add translate_dup: the CONT
copy, followed by a Convert when the output's dst_type differs from the
operand's type.

Add a unit test that fails with the previous mapping.
Resolve attention.cpp: keep the named-attribute add_op API (no op_case)
and apply master's has_own_kv guards for shared-KV layers. Port the
dspark builder's CONCAT calls to the same API.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: build OpenVINO cmake script / infra category: docs OpenVINO documentation category: GGUF FE

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant