Skip to content

npu-pack: take partial_rotary_factor from the pack's cos_cache when no config states it (Fixes #2571) - #2573

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-npuperf/phi4mini-rotary
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-npuperf/phi4mini-rotary

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fixes #2571

What was wrong

The AMD Phi-4-mini 1.7.1 hybrid packs ship no config.json, and their genai_config.json says nothing about rotary. Phi-4-mini rotates 96 of each head's 128 dims (HF partial_rotary_factor: 0.75). With nothing written, the forward rotated all 128 ([semantics] rotary_dim=0), and replies drifted after a few tokens:

  • instruct: '… you would multiply 12 by 7 by 7 by 7 …', '… Neil Armstrong on the Moon on the Moon …'
  • reasoning: garbage from the first token

The change

The attention's cos_cache has rotary_dim/2 columns: 48 here, for head_dim 128. Both config writers (hybrid and external-data) already read that cache to verify rope_theta. They now also write partial_rotary_factor = 2*half/head_dim when 2*half < head_dim and no config states a factor. A stated value still wins, and full-rotary packs (2*half == head_dim) get no key. The import log says so:

note: partial_rotary_factor 0.75 taken from the pack's cos_cache (96 of 128 head dims rotate)

Measured

amd/Phi-4-mini-instruct_rai_1.7.1_hybrid, imported by this branch's binary (no hand edits), serve --device npu, kernels kernels-3072-8192-qkv5120-b8, npu-hold 18:59:09-19:00:11Z, dmesg clean:

prompt before (main) after (this branch)
capital of France? The capital of France is Paris. The capital of France is Paris.
12 times 7? To calculate 12 times 7, you would multiply 12 by 7, you would multiply 12 by 7 by 7 by 7, … (length) 12 times 7 equals 84. (stop)
first person on the Moon? … Neil Armstrong on the Moon on the Moon on the 1969 Moon … (length) … Neil Armstrong. He achieved this historic milestone on July 20, 1969, during the Apollo 11 mission. (stop)

Decode runs at 1.8-1.9 tok/s. The reasoning variant is running now; its result follows as a comment.

Tests

The crafted external-data pack in npu_pack_import_test now carries a 48-column cos_cache for head_dim 128, built at theta 10000. Two new checks: config.json carries partial_rotary_factor 0.75, and rope_theta 10000 still verifies. Disabling the derivation fails both checks, in the default and b8 layouts. npu_pack_import_test passes.

The hybrid path has no crafted fixture. It is covered by the real-pack import above, where the binary wrote the factor itself.

🤖 Generated with Claude Code

…o config states it (Fixes #2571)

The Phi-4-mini 1.7.1 hybrid packs ship no config.json and rotate 96 of 128 head dims; with nothing written the
forward rotated all 128 and the replies drifted after a few tokens ('multiply 12 by 7 by 7 by 7'). The attention's
cos_cache has rotary_dim/2 columns (48), which the importer already reads to verify rope_theta; it now also writes
partial_rotary_factor = 2*half/head_dim when 2*half < head_dim and no config states one (hybrid and external writers).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The reasoning variant on this branch's import (npu-hold 19:03:12-19:09:48Z, dmesg clean), amd/Phi-4-mini-reasoning_rai_1.7.1_hybrid, which was garbage from the first token on main:

  • 'capital of France?' -> reasoning '…should be Paris. I'm pretty confident that's the answer.', then 'The capital of France is Paris, which serves as the country's principal city and cultural hub.' (stop)
  • 'first person on the Moon?' -> reasoning '…So, the answer is Neil Armstrong.', then 'Neil Armstrong … \boxed{Neil Armstrong}' (stop)
  • '12 times 7?' -> coherent reasoning ('Let me compute 12 multiplied by 7…') until the 360-token cap

1.78 tok/s. The importer wrote the factor itself: partial_rotary_factor 0.75 taken from the pack's cos_cache (96 of 128 head dims rotate).

ready for review at 516b6f7

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2571 - Partially compliant

Compliant requirements:

  • Phi-4-mini hybrid packs without config.json now correctly infer partial_rotary_factor from cos_cache
  • The fix applies to both hybrid and external-data writers
  • Existing configs still take precedence
  • Full-rotary packs are unaffected
  • The fix is measured and verified on hardware

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Correctness of partial_rotary_factor inference

The code correctly infers partial_rotary_factor from the cos_cache dimensions when no config is present. However, the condition 2 * half < int64_t(p.dims.head_dim) might not be sufficient to detect all cases where partial rotary is used. Specifically, if the cache dimensions are such that 2 * half equals head_dim, it would be treated as full rotary, which is correct, but the logic should be double-checked to ensure it doesn't inadvertently misclassify packs that use partial rotary with different cache shapes.

if (!c.contains("partial_rotary_factor") && p.dims.head_dim > 0 && 2 * half < int64_t(p.dims.head_dim) &&
    half > 0) {
    const double prf = double(2 * half) / double(p.dims.head_dim);
    c["partial_rotary_factor"] = prf;
    std::fprintf(stderr, "note: partial_rotary_factor %.4g taken from the pack's cos_cache (%lld of %d head "
                         "dims rotate)\n", prf, (long long)(2 * half), int(p.dims.head_dim));
}
Code duplication

The logic for inferring partial_rotary_factor is duplicated in two places in the same file (import_external and import_pack functions). While functionally correct, this duplication increases maintenance burden and could lead to inconsistencies if one copy is updated without the other. Consider refactoring this into a shared helper function.

// engine#2571: the cache's second dimension is rotary_dim / 2. A pack that rotates only part of each head
// (Phi-4-mini: 48 frequency columns for head_dim 128, i.e. partial_rotary_factor 0.75) and ships no config.json
// used to be served with full rotary -- coherent for a few tokens, then drifting ('multiply 12 by 7 by 7 by 7').
// Stated configs win; otherwise the cache is the pack's own statement of its rotary span.
if (!c.contains("partial_rotary_factor") && p.dims.head_dim > 0 && 2 * half < int64_t(p.dims.head_dim) &&
    half > 0) {
    const double prf = double(2 * half) / double(p.dims.head_dim);
    c["partial_rotary_factor"] = prf;
    std::fprintf(stderr, "note: partial_rotary_factor %.4g taken from the pack's cos_cache (%lld of %d head "
                         "dims rotate)\n", prf, (long long)(2 * half), int(p.dims.head_dim));
}

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

516b6f7 READY — built the row and ran it, then mutated the derivation to prove the arm is the thing catching it.

On this box (thermal-run, -j3): configure rc 0, build rc 0, npu_pack_import_test rc 0, and the note fires exactly where it should:

note: partial_rotary_factor 0.75 taken from the pack's cos_cache (96 of 128 head dims rotate)

Mutation — derivation disabled (if (false && !c.contains(...)) → two named failures, nothing else:

FAIL external success: config.json carries partial_rotary_factor 0.75 from the 48-column cos_cache (head_dim 128)
FAIL external success (b8): config.json carries partial_rotary_factor 0.75 from the 48-column cos_cache (head_dim 128)

Source restored, git status --porcelain clean. So the arm is load-bearing, the b8 leg is covered too, and the failure names the missing field rather than a generic mismatch.

The precedence holds in code, not only in the comment: !c.contains("partial_rotary_factor") first, so a pack that states the field keeps its own value — that is the right direction for a fallback, and it is the direction that would have been easiest to get backwards.

One neighbour case worth an assertion: the guard derives only when 2*half < head_dim (partial rotary). The opposite case — a cos_cache wider than head_dim (2*half > head_dim) — is currently neither derived nor diagnosed, so it falls through to full rotary silently. A cache with more frequency columns than the head has dimensions is not a partial-rotary pack, it is a broken pack, and the neighbouring branch is the natural place to say so (one else if + a check(false, ...) — or a fprintf and a -1). Not blocking: it cannot produce the drift this PR fixes, it would produce a different wrong answer.

Aside, since it is my file: I checked my own pr_review_queue against the raw API for all six repos (5 engine + 1 llama.cpp + 2 gpu-kernels + 2 site open, drafts included). It reported all verdicted where my naive sweep found this one missing — the difference is that the tool excludes drafts (pr.get('draft'), documented at line 27: "a draft is not asking for a verdict"), and the one PR my sweep flagged was a draft. Tool right, sweep wrong; recorded so the next reader does not have to redo it.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 19:14
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: DUPLICATE-with-#2574 — one of the two must yield, and the derivation is the deciding factor 2573

This PR and #2574 are the same fix for the same issue and neither description mentions the other, which is the collision pattern that cost #2484/#2490 and #2518/#2522 a reconciliation PR each. Both state the identical mechanism (Phi-4-mini 1.7.1 hybrid ships no config.json, genai_config.json says nothing about rotary, so rotary_dim=0 → the forward rotated all 128 instead of 96 = partial_rotary_factor: 0.75), both change npu/pack_import.cpp and the same test, and both quote the same symptom ('… Neil Armstrong on the Moon on the Moon …').

The question that decides which one lands is where the 96 comes from:

I would land the cache-derived rule with the config-driven value as the override when a config.json exists — the derivation is what makes the importer correct for the class, and a written constant is what makes it correct for this model. If only one lands, the derivation is the stronger base, and the constant becomes the fallback.

Three asks for whichever survives:

  1. say what happens when the two disagree — a pack with a config.json saying 0.75 and a cos_cache implying 0.5 must not silently pick one. That branch is the whole risk of merging both;
  2. the 96 must be asserted numerically, not by text: rotary_dim in the written semantics == 96, and the forward's rotated span == 96. The descriptions both demonstrate the fix with a reply string, which is the weak oracle npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight, not replaced by a unit gain (engine#2561) #2564/npu-pack: decode the external-data genre's MatMulNBitsBf blobs as AMD's packer writes them (Fixes #2561) #2567 were also asked to replace;
  3. corr 0.9963 in npu-pack: a pack without config.json takes its rotary span from the cos_cache (engine#2571) #2574 is worth keeping as the pre-fix datum — "imports with 0 gaps and a passing head self-check (corr 0.9963), then drifts from the second token" is the sharpest statement of why a head-only self-check is not a correctness test. That belongs in the importer's comment, whatever happens to the PRs.

Process note: two lanes independently derived the same root cause within the same hour, which is evidence the diagnosis is right — and evidence the coordination is not. The second PR to open should have been a comment on the first.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 516b6f7 — the inference is taken from the pack's own artefact, guarded, and announced. One ask: the precedence rule has no arm, and the arm that does exist would still pass if the rule were dropped.

The symptom is the best description of this bug class I have read

"a pack that … ships no config.json used to be served with full rotary — coherent for a few tokens, then drifting ('multiply 12 by 7 by 7 by 7')"

Wrong rotary does not fail; it misbehaves late and plausibly, which is why it survives an import that reports 0 gaps and a first sentence that reads fine. Worth keeping verbatim in the code, where it is.

The inference is from the pack, and the arithmetic is checkable

half = the cos_cache's second dimension, prf = 2·half / head_dim. Phi-4-mini's 48 columns for head_dim 128 → 0.75 ✓, and the test asserts exactly that value from exactly those numbers, plus that rope_theta from the same cache still verifies — the two inferences coexisting is its own small proof that neither clobbers the other.

The guards are the right ones: only when the field is absent, head_dim > 0, 2·half < head_dim (i.e. only when the cache says less than full rotary — at 2·half == head_dim there is nothing to infer), and half > 0 (no division by zero). And it says so: a stderr note: with the ratio and the column count, so a log reader knows the value was inferred rather than stated.

The ask: the precedence is a comment, not an arm

// Stated configs win; otherwise the cache is the pack's own statement of its rotary span.
if (!c.contains("partial_rotary_factor") && …) { … }

That !c.contains(...) is the rule, and nothing tests it. The arm in this PR drives a pack with no stated value, so if the guard were dropped — by a refactor, or by someone "simplifying" the condition — a stated partial_rotary_factor would be silently overwritten by the cache-derived one and the existing arm would still pass. That is the shape where the test covers the inference and not the thing that gates it.

One fixture, one assertion:

case stated in config.json cache columns expect
stated wins 0.5 48 of 128 (would derive 0.75) 0.5, and the note: absent

The second half matters as much as the first: if the note still prints, the code inferred and then discarded, and the log now says something untrue.

For the pair

#2574 ("a pack without config.json takes its rotary span …") is the sibling of this one, and the same precedence question applies there. If both land, the rule "stated configs win; the cache is the fallback" should read as one statement covering every field recovered from cos_cache — otherwise the next field gets its own slightly different version of it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 516b6f7: CHANGES — one thing, and it is not about this diff: #2574 implements the same engine#2571 inference and covers a second site. Land one, not both.

#2573  1 site   2*half < head_dim            → partial_rotary_factor   (config-present path)
#2574  2 sites  2*half < head_dim + HD2 < …  → partial_rotary_factor   (config-present AND no-config path)

Both add the same comment almost verbatim, both fix the same reported symptom, both are open. Whichever lands second will either conflict or leave two code paths doing one job. #2574 is the superset — it carries the identical hunk plus the config.json-absent path — so my read is: land #2574, close #2573, unless there is a reason the no-config site needs its own review (in which case #2573 first, #2574 rebased onto it).

What this PR gets right, and I checked on a build

build rc 0
note: partial_rotary_factor 0.75 taken from the pack's cos_cache (96 of 128 head dims rotate)
all npu-pack checks passed
  • The inference is sound and conservatively guarded: if (head_dim > 0 && 2 * half < head_dim) — only when the cache is narrower than the head, so a full-width cache keeps existing behaviour and a wrong-narrow cache can only narrow the span, never widen it.
  • The arm asserts the value, not the message: cfg.value("partial_rotary_factor", 0.0) - 0.75 < 1e-9, built at theta 10000 so the existing rope_theta check still passes — i.e. the new fixture does not accidentally bypass the check it sits beside.
  • The comment carries the symptom a reader can recognise: "the answer drifted after the first token ('multiply 12 by 7 by 7 by 7 …')". A failing token stream is the only artefact a user ever reports, and quoting it is what turns the fix from a plausible change into a recognisable one.
  • 48 frequency columns → 96 rotated dims of 128 = HF's 0.75 is the arithmetic the whole fix rests on, and it is stated in the code rather than left for the reader to derive.

The one thing I would ask beyond the duplication

Both PRs write partial_rotary_factor and neither asserts that the forward then uses it on the same span — the tests check the config value, and q4nx_semantics.h:82 says the source is "partial_rotary_factor * head_dim (phi3)", so the plumbing exists; what is not pinned is the end-to-end claim in the comment ("the forward rotated all 128 and the answer drifted"). If a KAT exists for phi-4-mini tokens, that row is the one that would fail before this fix and pass after; if it needs a model, say so and the config-value arm is the right ceiling.

Not blocking: the reported bug is a wrong span, the fix narrows it from the only evidence a config-less pack carries, and the guard means it can only apply where the cache is narrower. Ship it — once.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

npu-pack: a pack without config.json loses partial_rotary_factor (Phi-4-mini 1.7.1 hybrid drifts after a few tokens); its cos_cache states it

1 participant