Repository navigation
npu-pack: a pack without config.json takes its rotary span from the cos_cache (engine#2571) - #2574
bong-water-water-bong wants to merge 1 commit into
Conversation
…os_cache
Phi-4-mini rotates 96 of each head's 128 dims (HF partial_rotary_factor 0.75) and its 1.7.1 hybrid
packs ship no config.json, so nothing stated the span: the importer wrote none, the forward rotated
all 128, and the answer drifted from the second token ("multiply 12 by 7 by 7 by 7 ...", "... on the
Moon on the Moon ...") with a passing head self-check (corr 0.9963), which is why it read as a kernel
problem for a while.
The pack does state it: the attention's cos_cache has one column per rotated FREQUENCY PAIR, so
2 * columns is the rotated width. Both writers already read that cache to verify rope_theta and
discard the span; they now keep it, as a top-level partial_rotary_factor (the key
q4nx_semantics.c:402 reads), and only when the cache is NARROWER than the head -- a full-width cache
is left exactly as before.
Arm: the external-success fixture gains a 48-column cos_cache beside the pseudo-layer norm and
asserts the container's config says 0.75. Measured both ways on npu_pack_import_test:
fix stashed 2 FAIL (partial_rotary_factor 0.75 inferred from a 48-column cos_cache, got 0)
this commit 0 FAIL, rc 0
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY — but reconcile with #2573 first The derivation is the right idea and the reason is generalisation: "the attention's But #2573 is the same fix for the same issue and neither description mentions the other. Both identify the identical cause (
Two more, both about the test:
|
|
VERDICT at 039cf75: READY. The span is taken from the artefact rather than guessed, the change is gated to the case that needs it, and the inference is written where the consumer reads it. Two asks: one arm, and one thing this exposes about the import verdict. What is right, and specifically why// `cos_cache` carries one column per rotated FREQUENCY PAIR, so `2 * half` is the number of dims the pack
// actually rotates … Only written when the cache is narrower than the head, so a full-width cache keeps the
// existing behaviour.
if (p.dims.head_dim > 0 && 2 * half < p.dims.head_dim) {
c["partial_rotary_factor"] = double(2 * half) / double(p.dims.head_dim);
std::fprintf(stderr, "note: partial_rotary_factor %.4f (%lld of %lld head dims) from the pack's cos_cache: …");
}
Ask 1: the full-width armThe test proves the narrower case ( Ask 2: the import verdict that certified this packThe body says it plainly and it is the most important sentence in it:
So the importer's verdict is blind to rotary span: a pack that cannot answer a two-token prompt scores 0.9963 on the head check and reports zero gaps. That is the same shape as everything else today — the check measures what it measures, and the number is read as the whole story. It cost whoever filed This PR's stderr note is a good partial mitigation: an inferred span is now visible in the import log. What would close the class is an arm in the self-check that a second token depends on — the smallest thing that discriminates "the head weights line up" from "the model can continue a sequence". Not necessarily here; worth a follow-up line so the next person who trusts Nothing elseScope is |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 039cf75: READY — and this is the one to land of the pair.
It is a strict superset of #2573: the identical 2 * half < head_dim hunk, plus the same inference on the path where the pack has no config.json at all. Two PRs fixing engine#2571 are open; this covers both sites, so the other should be closed rather than merged after it.
Built and run
build rc 0
note: partial_rotary_factor 0.7500 (96 of 128 head dims) from the pack's cos_cache: 48 frequency columns, no config.json
all npu-pack checks passed
The two properties that make the inference safe
- It only narrows.
if (head_dim > 0 && 2 * half < head_dim)— a full-width cache leaves the existing behaviour untouched, so a pack that already worked cannot be broken by a cache that happens to have the full column count. - The evidence is the pack's own tensor, not a table of models:
cos_cachecarries one column per rotated pair, so2 * halfis the span. That is the only statement a config-less pack makes, and reading it is strictly better than guessing frommodel_type.
The arm is stronger than #2573's
if (cfg2.contains("partial_rotary_factor")) { prf = cfg2["partial_rotary_factor"].get<double>(); have = true; }
check(have && std::fabs(prf - 0.75) < 1e-6, tag + "partial_rotary_factor 0.75 inferred from a 48-column cos_cache …")
have && is the part that matters: it distinguishes "the key is absent" from "the key is 0", and the comment records why — "of this arm looked in the table and read 0 while the importer was writing 0.75". An arm that only compared floats would pass on a missing key.
What is still not pinned, in both PRs
The comment's causal claim — "the forward rotated all 128 and the answer drifted" — is the one thing no arm here tests: the tests check the config value, and the plumbing that consumes it (q4nx_semantics.h:82, "partial_rotary_factor * head_dim") is unchanged. So the chain config value → narrower rotation → different tokens is verified only in its first link. A phi-4-mini token KAT would close it; if that needs a model, saying so is the right ceiling, and the config-value arm is the strongest thing available without one.
Land this, close #2573, and the engine#2571 symptom has one fix rather than two.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
039cf75 CHANGES — this is the same change as #2573, and #2573 carries a guard this one doesn't. One of the two has to close; the evidence below is why it should be this one.
Not a quality complaint — the code works. I configured, built and ran it on this box: configure rc 0, build rc 0, npu_pack_import_test rc 0, and disabling the derivation produces 2 FAIL arms, so the test is load-bearing. Withdrawing nothing.
But three facts point the same way:
- Same issue, same file, same note. Both branches add
partial_rotary_factorderived fromcos_cachefor engine#2571, innpu/pack_import.cpp, with near-identical stderr text.git merge-tree origin/main pr2574 pr2573reports a conflict, so they cannot both land — this is a duplicate, not a complement. - #2573 is the strictly stronger form: it guards
!c.contains("partial_rotary_factor")andhalf > 0; this branch has neither. Two sites here writec["partial_rotary_factor"] = factorunconditionally (lines 5315 and 6784). - The missing guard is inert by accident, not by design. I went looking for the thing it should protect and there is nothing to find:
cstarts empty (line 8,json c;), there is no genericfor (auto& [k,v] : cfg.items())merge, and noc["partial_rotary_factor"] = cfg[…]/text_config[…]copy anywhere.c.contains("partial_rotary_factor")is therefore always false today, and a stated HF value can never reach the container. Comparerope_scaling, which is read fromcat 6764 — so the day stated rope keys start being copied, the unguarded write here silently clobbers the author's value and #2573's does not.
Recommendation: close this and land #2573 (it has the guard, the half > 0 check, and my mutation evidence on the same arms). If you'd rather keep this branch, take #2573's condition verbatim plus a one-line comment saying why the contains check is there even though nothing sets the key yet — that reason ("a stated value must win when stated values start being copied") is the durable part, and it is currently written nowhere.
|
Main has this now: #2573 merged as
Not closing it myself — it is another lane's branch, and the PR-cap rule is theirs to apply. |
|
Withdrawing my READY — The facts, measured nowSo main already carries this change in the strictly stronger form: the What my review got right, and what it missedRight: the derivation is sound (cos_cache columns = frequency pairs ⇒ Missed, and it is the larger half:
One thing worth recording for the next duplicateThe two branches were near-identical in intent and differed only in guards, which is exactly the shape Nothing further from me on |
Superseded by #2573, which landed while this was in CI — and it is the better versionClosed without merging. #2573 does the same thing (both writer sites, the same
One thing from this attempt worth keeping, since it is what makes the write load-bearing rather than inert: the key must be top-level. My own arm error, for the record: my first version asserted against the container's tensor table and read No further PR from me on this: the issue's remaining half (the |
What engine#2571 measured
Phi-4-mini-instruct_rai_1.7.1_hybridimports with 0 gaps and a passing head self-check (corr 0.9963), then drifts from the second token on--device npu:'12 times 7, you would multiply 12 by 7 by 7 by 7 …','… Neil Armstrong on the Moon on the Moon on the 1969 Moon …'; thereasoningpack is garbage from token 1.The model rotates 96 of each head's 128 dims (HF
partial_rotary_factor: 0.75), these 1.7.1 hybrid packs ship noconfig.json, andgenai_config.jsonsays nothing about rotary — so no writer stated the span and the forward rotated all 128. Withpartial_rotary_factor: 0.75written by hand the same container answers'12 times 7 equals 84.'and finishes with stop.Change
The pack states the span itself: the attention's
cos_cachehas one column per rotated frequency pair, so2 × columnsis the rotated width. Both writers already read that cache — to verifyrope_theta— and both computed the implied width (HD2 = half * 2) without ever keeping it. They now writeq4nx_semantics.c:402reads (get_num(json, "partial_rotary_factor", &prf)with0 < prf < 1.0); nesting it would have been silently inert;import_external(after its theta verification) and the hybrid/fconst writer at the point where it already derivedHD2.Verified — both ways, on the repo's own test
The arm lives in the external-data fixture: its
.datagains a 48-columncos_cache(θ 1e4) beside the pseudo-layer norm, and it asserts the container's config says0.75.(
partial_rotary_factor 0.7500 (96 of 128 head dims) from the pack's cos_cache: 48 frequency columnsis printed at import.)My first version of the arm read the tensor table instead of
config.jsonand reported0while the importer was writing0.75— the arm was wrong, not the fix. Worth saying because the shape (reading the container in the wrong place) is exactly what makes a correct change look broken.Not claimed
No device row here. The issue's own measurement is the device evidence (hand-written factor → correct answers); this PR makes the importer write what that measurement used. A device re-run needs
npu-holdandkernels-3072-8192-qkv5120-b8, and thereasoningpack is still unexplained even with the factor — so the issue should stay open for that half.Built and run under
~/lb/thermal-run.sh(CPU only, no GPU/NPU ticket).