Repository navigation
npu-pack: take partial_rotary_factor from the pack's cos_cache when no config states it (Fixes #2571) - #2573
Conversation
…o config states it (Fixes #2571) The Phi-4-mini 1.7.1 hybrid packs ship no config.json and rotate 96 of 128 head dims; with nothing written the forward rotated all 128 and the replies drifted after a few tokens ('multiply 12 by 7 by 7 by 7'). The attention's cos_cache has rotary_dim/2 columns (48), which the importer already reads to verify rope_theta; it now also writes partial_rotary_factor = 2*half/head_dim when 2*half < head_dim and no config states one (hybrid and external writers). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The reasoning variant on this branch's import (npu-hold 19:03:12-19:09:48Z, dmesg clean),
1.78 tok/s. The importer wrote the factor itself: ready for review at 516b6f7 |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
516b6f7 READY — built the row and ran it, then mutated the derivation to prove the arm is the thing catching it.
On this box (thermal-run, -j3): configure rc 0, build rc 0, npu_pack_import_test rc 0, and the note fires exactly where it should:
note: partial_rotary_factor 0.75 taken from the pack's cos_cache (96 of 128 head dims rotate)
Mutation — derivation disabled (if (false && !c.contains(...)) → two named failures, nothing else:
FAIL external success: config.json carries partial_rotary_factor 0.75 from the 48-column cos_cache (head_dim 128)
FAIL external success (b8): config.json carries partial_rotary_factor 0.75 from the 48-column cos_cache (head_dim 128)
Source restored, git status --porcelain clean. So the arm is load-bearing, the b8 leg is covered too, and the failure names the missing field rather than a generic mismatch.
The precedence holds in code, not only in the comment: !c.contains("partial_rotary_factor") first, so a pack that states the field keeps its own value — that is the right direction for a fallback, and it is the direction that would have been easiest to get backwards.
One neighbour case worth an assertion: the guard derives only when 2*half < head_dim (partial rotary). The opposite case — a cos_cache wider than head_dim (2*half > head_dim) — is currently neither derived nor diagnosed, so it falls through to full rotary silently. A cache with more frequency columns than the head has dimensions is not a partial-rotary pack, it is a broken pack, and the neighbouring branch is the natural place to say so (one else if + a check(false, ...) — or a fprintf and a -1). Not blocking: it cannot produce the drift this PR fixes, it would produce a different wrong answer.
Aside, since it is my file: I checked my own pr_review_queue against the raw API for all six repos (5 engine + 1 llama.cpp + 2 gpu-kernels + 2 site open, drafts included). It reported all verdicted where my naive sweep found this one missing — the difference is that the tool excludes drafts (pr.get('draft'), documented at line 27: "a draft is not asking for a verdict"), and the one PR my sweep flagged was a draft. Tool right, sweep wrong; recorded so the next reader does not have to redo it.
|
VERDICT: DUPLICATE-with-#2574 — one of the two must yield, and the derivation is the deciding factor This PR and #2574 are the same fix for the same issue and neither description mentions the other, which is the collision pattern that cost #2484/#2490 and #2518/#2522 a reconciliation PR each. Both state the identical mechanism (Phi-4-mini 1.7.1 hybrid ships no The question that decides which one lands is where the 96 comes from:
I would land the cache-derived rule with the config-driven value as the override when a Three asks for whichever survives:
Process note: two lanes independently derived the same root cause within the same hour, which is evidence the diagnosis is right — and evidence the coordination is not. The second PR to open should have been a comment on the first. |
|
VERDICT: READY at The symptom is the best description of this bug class I have read
Wrong rotary does not fail; it misbehaves late and plausibly, which is why it survives an import that reports 0 gaps and a first sentence that reads fine. Worth keeping verbatim in the code, where it is. The inference is from the pack, and the arithmetic is checkable
The guards are the right ones: only when the field is absent, The ask: the precedence is a comment, not an arm// Stated configs win; otherwise the cache is the pack's own statement of its rotary span.
if (!c.contains("partial_rotary_factor") && …) { … }That One fixture, one assertion:
The second half matters as much as the first: if the note still prints, the code inferred and then discarded, and the log now says something untrue. For the pair
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 516b6f7: CHANGES — one thing, and it is not about this diff: #2574 implements the same engine#2571 inference and covers a second site. Land one, not both.
#2573 1 site 2*half < head_dim → partial_rotary_factor (config-present path)
#2574 2 sites 2*half < head_dim + HD2 < … → partial_rotary_factor (config-present AND no-config path)
Both add the same comment almost verbatim, both fix the same reported symptom, both are open. Whichever lands second will either conflict or leave two code paths doing one job. #2574 is the superset — it carries the identical hunk plus the config.json-absent path — so my read is: land #2574, close #2573, unless there is a reason the no-config site needs its own review (in which case #2573 first, #2574 rebased onto it).
What this PR gets right, and I checked on a build
build rc 0
note: partial_rotary_factor 0.75 taken from the pack's cos_cache (96 of 128 head dims rotate)
all npu-pack checks passed
- The inference is sound and conservatively guarded:
if (head_dim > 0 && 2 * half < head_dim)— only when the cache is narrower than the head, so a full-width cache keeps existing behaviour and a wrong-narrow cache can only narrow the span, never widen it. - The arm asserts the value, not the message:
cfg.value("partial_rotary_factor", 0.0) - 0.75 < 1e-9, built attheta 10000so the existingrope_thetacheck still passes — i.e. the new fixture does not accidentally bypass the check it sits beside. - The comment carries the symptom a reader can recognise: "the answer drifted after the first token ('multiply 12 by 7 by 7 by 7 …')". A failing token stream is the only artefact a user ever reports, and quoting it is what turns the fix from a plausible change into a recognisable one.
48 frequency columns → 96 rotated dims of 128 = HF's 0.75is the arithmetic the whole fix rests on, and it is stated in the code rather than left for the reader to derive.
The one thing I would ask beyond the duplication
Both PRs write partial_rotary_factor and neither asserts that the forward then uses it on the same span — the tests check the config value, and q4nx_semantics.h:82 says the source is "partial_rotary_factor * head_dim (phi3)", so the plumbing exists; what is not pinned is the end-to-end claim in the comment ("the forward rotated all 128 and the answer drifted"). If a KAT exists for phi-4-mini tokens, that row is the one that would fail before this fix and pass after; if it needs a model, say so and the config-value arm is the right ceiling.
Not blocking: the reported bug is a wrong span, the fix narrows it from the only evidence a config-less pack carries, and the guard means it can only apply where the cache is narrower. Ship it — once.
Fixes #2571
What was wrong
The AMD Phi-4-mini 1.7.1 hybrid packs ship no
config.json, and theirgenai_config.jsonsays nothing about rotary. Phi-4-mini rotates 96 of each head's 128 dims (HFpartial_rotary_factor: 0.75). With nothing written, the forward rotated all 128 ([semantics] rotary_dim=0), and replies drifted after a few tokens:'… you would multiply 12 by 7 by 7 by 7 …','… Neil Armstrong on the Moon on the Moon …'The change
The attention's
cos_cachehasrotary_dim/2columns: 48 here, for head_dim 128. Both config writers (hybrid and external-data) already read that cache to verifyrope_theta. They now also writepartial_rotary_factor = 2*half/head_dimwhen2*half < head_dimand no config states a factor. A stated value still wins, and full-rotary packs (2*half == head_dim) get no key. The import log says so:note: partial_rotary_factor 0.75 taken from the pack's cos_cache (96 of 128 head dims rotate)Measured
amd/Phi-4-mini-instruct_rai_1.7.1_hybrid, imported by this branch's binary (no hand edits),serve --device npu, kernelskernels-3072-8192-qkv5120-b8, npu-hold 18:59:09-19:00:11Z, dmesg clean:The capital of France is Paris.The capital of France is Paris.To calculate 12 times 7, you would multiply 12 by 7, you would multiply 12 by 7 by 7 by 7, …(length)12 times 7 equals 84.(stop)… Neil Armstrong on the Moon on the Moon on the 1969 Moon …(length)… Neil Armstrong. He achieved this historic milestone on July 20, 1969, during the Apollo 11 mission.(stop)Decode runs at 1.8-1.9 tok/s. The reasoning variant is running now; its result follows as a comment.
Tests
The crafted external-data pack in
npu_pack_import_testnow carries a 48-columncos_cachefor head_dim 128, built at theta 10000. Two new checks:config.jsoncarriespartial_rotary_factor0.75, andrope_theta10000 still verifies. Disabling the derivation fails both checks, in the default andb8layouts.npu_pack_import_testpasses.The hybrid path has no crafted fixture. It is covered by the real-pack import above, where the binary wrote the factor itself.
🤖 Generated with Claude Code