[yzhou51]

Overview

Prefill and decode graphs can share n_nodes/first/last node name, but compute_op_case() bakes shape-dependent structural decisions (e.g. GDN/view slicing) into the compiled model at construction time. Without this, a decode-shaped cgraph could reuse a model/infer_request compiled for a prefill-shaped one (or vice versa) on a graph_key collision.

Additional information

graph_key identifies a cgraph by n_nodes, n_leaves, and the first/last node names. A prefill cgraph (n_tokens > 1) and a decode cgraph (n_tokens == 1) for the same model can have identical values for all three, so they hash and compare equal
and can land in the same cache slot (decoder_cache / infer_request_cache).

compute_op_case() makes shape-dependent structural decisions once, at first compile, and bakes them into the translated/compiled OpenVINO model (e.g. GDN state slicing, view handling). GgmlOvDecoder::update_io() rebinds I/O tensors on a cache hit but never re-derives those decisions. If a prefill-shaped and decode-shaped graph collide under the same key, the cached model/infer_request — compiled under one shape's structural assumptions — would get silently reused for the other.

This was found by code review while investigating an unrelated OV-CPU/NPU output-correctness issue on Qwen3.5. There is no reproducing test case or crash report for it: I have not constructed a concrete cgraph pair that actually collides
in practice, and applying this fix in isolation did not resolve the correctness issue under investigation. It is included here as an independently correct hardening of the caching logic, not as a fix for that issue.

Testing: ov-cpu inference compare

No automated test covers this path yet. Manually confirmed the build compiles cleanly; have not observed a regression or behavior change on existing working models.

ov-cpu inference compare

This is a one-time cost, not a per-token regression.

• Base Log:
  base-llama-simple-cpu.log
• Cache key fix log:
  cache-key-llama-simple-cpu.log

Base log (buggy, shared cache key): only one compile, on the very first call (prefill):

• call 1 (prefill): decoder 960.8ms + conversion 131.1ms + compile 1426.7ms + inference 815.4ms
• call 2 onward (every decode step): no compile at all, just ~25-30ms inference each (cache reused, including the first decode step, which was the actual bug — it reused the prefill-compiled model)

Cache-key-fix log (correct): two compiles — prefill, then a second one for the first decode call:

• call 1 (prefill): decoder 924.8ms + conversion 82.0ms + compile 1252.2ms + inference 866.1ms
• call 2 (first decode step): decoder 1023.8ms + conversion 86.9ms + compile 1147.8ms + inference 790.9ms ← this is the new, previously-missing decode-phase compile
• call 3 onward: back to ~24-28ms inference each, matching the base log's steady state almost exactly

The extra one-time cost is call 2 in the fix log minus what call 2 cost in the base log: (1023.8+86.9+1147.8+790.9) − 31.0 ≈ 3018ms, i.e. about 3.0 seconds, added exactly once per process run. That matches your observed -n 50 slowdown almost exactly: (1/7.00 − 1/12.68) × 50 ≈ 3.2s.

So: steady-state decode throughput is unchanged (~24-28ms/token in both logs) — the entire regression is the mandatory second compile this fix now correctly triggers for the decode-shaped graph, which was silently skipped before (that's the bug). For short runs like -n 50 this one-time cost dominates the measured t/s; for longer generations it amortizes away.

Requirements

• I have read and agree with the contributing guidelines
  Yes. The minial required cases are passed.
  test-data.md

• AI usage disclosure:
  Yes. AI help do the issue analysis.

[mostafafaheem]

Thanks for comtributing!

The cache key in non-static contexts is shared between prefill and decode intentionally, since both phases shares the same graph topology, and shape differences are handled using dynamic shapes. In the logs you pasted, there seems to be no issue with accuracy before your PR. Can you please share why this change is needed?

[wine99]

Second @mostafafaheem 's comments. The compiled model is shared between prefill and decode for CPU and GPU intentionally. Fallback or split-cgraph in general does not work well this strategy, as noted in your code comments. It is a known limitation