Skip to content

npu-pack: a pack without config.json takes its rotary span from the cos_cache (engine#2571) - #2574

Closed
bong-water-water-bong wants to merge 1 commit into
mainfrom
npu-pack-partial-rotary-from-cos-cache
Closed

bong-water-water-bong wants to merge 1 commit into
mainfrom
npu-pack-partial-rotary-from-cos-cache

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What engine#2571 measured

Phi-4-mini-instruct_rai_1.7.1_hybrid imports with 0 gaps and a passing head self-check (corr 0.9963), then drifts from the second token on --device npu: '12 times 7, you would multiply 12 by 7 by 7 by 7 …', '… Neil Armstrong on the Moon on the Moon on the 1969 Moon …'; the reasoning pack is garbage from token 1.

The model rotates 96 of each head's 128 dims (HF partial_rotary_factor: 0.75), these 1.7.1 hybrid packs ship no config.json, and genai_config.json says nothing about rotary — so no writer stated the span and the forward rotated all 128. With partial_rotary_factor: 0.75 written by hand the same container answers '12 times 7 equals 84.' and finishes with stop.

Change

The pack states the span itself: the attention's cos_cache has one column per rotated frequency pair, so 2 × columns is the rotated width. Both writers already read that cache — to verify rope_theta — and both computed the implied width (HD2 = half * 2) without ever keeping it. They now write

c["partial_rotary_factor"] = double(2 * half) / double(head_dim);
  • top-level, because that is the key q4nx_semantics.c:402 reads (get_num(json, "partial_rotary_factor", &prf) with 0 < prf < 1.0); nesting it would have been silently inert;
  • only when the cache is narrower than the head, so a full-width cache (every other pack) is byte-for-byte unchanged;
  • in both writers: import_external (after its theta verification) and the hybrid/fconst writer at the point where it already derived HD2.

Verified — both ways, on the repo's own test

The arm lives in the external-data fixture: its .data gains a 48-column cos_cache (θ 1e4) beside the pseudo-layer norm, and it asserts the container's config says 0.75.

fix stashed        2 FAIL    partial_rotary_factor 0.75 inferred from a 48-column cos_cache (engine#2571), got 0
this commit        0 FAIL, npu_pack_import_test rc 0

(partial_rotary_factor 0.7500 (96 of 128 head dims) from the pack's cos_cache: 48 frequency columns is printed at import.)

My first version of the arm read the tensor table instead of config.json and reported 0 while the importer was writing 0.75 — the arm was wrong, not the fix. Worth saying because the shape (reading the container in the wrong place) is exactly what makes a correct change look broken.

Not claimed

No device row here. The issue's own measurement is the device evidence (hand-written factor → correct answers); this PR makes the importer write what that measurement used. A device re-run needs npu-hold and kernels-3072-8192-qkv5120-b8, and the reasoning pack is still unexplained even with the factor — so the issue should stay open for that half.

Built and run under ~/lb/thermal-run.sh (CPU only, no GPU/NPU ticket).

…os_cache

Phi-4-mini rotates 96 of each head's 128 dims (HF partial_rotary_factor 0.75) and its 1.7.1 hybrid
packs ship no config.json, so nothing stated the span: the importer wrote none, the forward rotated
all 128, and the answer drifted from the second token ("multiply 12 by 7 by 7 by 7 ...", "... on the
Moon on the Moon ...") with a passing head self-check (corr 0.9963), which is why it read as a kernel
problem for a while.

The pack does state it: the attention's cos_cache has one column per rotated FREQUENCY PAIR, so
2 * columns is the rotated width. Both writers already read that cache to verify rope_theta and
discard the span; they now keep it, as a top-level partial_rotary_factor (the key
q4nx_semantics.c:402 reads), and only when the cache is NARROWER than the head -- a full-width cache
is left exactly as before.

Arm: the external-success fixture gains a 48-column cos_cache beside the pseudo-layer norm and
asserts the container's config says 0.75. Measured both ways on npu_pack_import_test:
  fix stashed   2 FAIL (partial_rotary_factor 0.75 inferred from a 48-column cos_cache, got 0)
  this commit   0 FAIL, rc 0
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Partial Rotary Factor Inference

The code now infers partial_rotary_factor from the cos_cache when no config.json is present. This is a correct behavior for packs without a config, but it's important to ensure that the inference logic is robust and that the factor is correctly calculated as 2 * half / head_dim where half is the number of frequency columns in the cache. The change should be validated to ensure it doesn't introduce any regressions or incorrect behavior in cases where the cache dimensions do not align with the head dimensions.

if (p.dims.head_dim > 0 && 2 * half < p.dims.head_dim) {
    const double factor = double(2 * half) / double(p.dims.head_dim);
    c["partial_rotary_factor"] = factor;
    std::fprintf(stderr,
                 "note: partial_rotary_factor %.4f (%lld of %lld head dims) from the pack's cos_cache: "
                 "%lld frequency columns, no config.json states the span (engine#2571)\n",
                 factor, (long long) (2 * half), (long long) p.dims.head_dim, (long long) half);
}
Partial Rotary Factor Inference (Second Instance)

Similar to the first instance, this section also infers partial_rotary_factor from the cos_cache when no config.json is present. The logic is consistent with the first instance, but it's important to ensure that both instances are correctly handling the case where the cache is narrower than the head dimension and that the factor is correctly calculated.

if (p.dims.head_dim > 0 && HD2 < p.dims.head_dim) {
    const double factor = double(HD2) / double(p.dims.head_dim);
    c["partial_rotary_factor"] = factor;
    std::fprintf(stderr,
                 "note: partial_rotary_factor %.4f (%d of %lld head dims) from the pack's cos_cache: "
                 "%lld frequency columns, and no config states the span (engine#2571)\n",
                 factor, HD2, (long long) p.dims.head_dim, (long long) half);
}

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY — but reconcile with #2573 first 2574

The derivation is the right idea and the reason is generalisation: "the attention's cos_cache has rotary_dim/2 columns: 48 here, for head_dim 128" means the importer can determine the span from the artefact rather than from a table of known models — so the next partial-rotary pack with no config.json needs no code change. That is the same principle as #2560's "model_type read from each repo rather than inferred from the class name", applied to a numeric parameter, and it is the stronger of the two available bases.

But #2573 is the same fix for the same issue and neither description mentions the other. Both identify the identical cause (partial_rotary_factor: 0.75 → 96 of 128 dims, no config.json, genai_config.json silent on rotary, [semantics] rotary_dim=0 → full rotation), both edit npu/pack_import.cpp plus the same test, and both quote the same '… multiply 12 by 7 by 7 by 7 …'. One must yield, and the criterion should be stated in whichever survives:

  • derive from cos_cache as the base (generalises, measures the artefact);
  • honour config.json's partial_rotary_factor when present as the override (exact when the pack tells us);
  • say what happens when they disagree — that branch is the entire risk of merging both, and silently preferring one would be a new class of bug in the importer.

Two more, both about the test:

  1. assert the number: rotary_dim == 96 in the written semantics, and 96 rotated dims in the forward. The reply-string demonstration is the weak oracle npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight, not replaced by a unit gain (engine#2561) #2564/npu-pack: decode the external-data genre's MatMulNBitsBf blobs as AMD's packer writes them (Fixes #2561) #2567 were asked to replace — and this model has already produced two different wrong answers (on the Moon on the Moon, and garbage from token 1 for the reasoning pack), so text passing is not evidence;
  2. keep corr 0.9963 in the record: "imports with 0 gaps and a passing head self-check (corr 0.9963), then drifts from the second token" is the best one-line argument that a head-only self-check is not a correctness test. That belongs in the importer's own comment regardless of which PR lands.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 039cf75: READY. The span is taken from the artefact rather than guessed, the change is gated to the case that needs it, and the inference is written where the consumer reads it. Two asks: one arm, and one thing this exposes about the import verdict.

What is right, and specifically why

// `cos_cache` carries one column per rotated FREQUENCY PAIR, so `2 * half` is the number of dims the pack
// actually rotates … Only written when the cache is narrower than the head, so a full-width cache keeps the
// existing behaviour.
if (p.dims.head_dim > 0 && 2 * half < p.dims.head_dim) {
    c["partial_rotary_factor"] = double(2 * half) / double(p.dims.head_dim);
    std::fprintf(stderr, "note: partial_rotary_factor %.4f (%lld of %lld head dims) from the pack's cos_cache: …");
}
  • The number comes from the pack, not from a table of known model families. A cos_cache with 48 frequency columns is 96 rotated dims of a 128-dim head; that is a fact carried by the artefact, so it survives a model the importer has never seen — which is the whole class that broke here.
  • Gated to narrower-than-head, so the change cannot move any pack that already worked. That is the difference between a fix and a retune.
  • Announced on stderr with the arithmetic (0.7500, 96 of 128, 48 columns), so an operator reading an import log can see that a span was inferred rather than stated. An inferred value that leaves no trace is how a pack gets certified twice with different behaviour.
  • And the self-correction in the test's comment — "It belongs in the container's CONFIG (which is what q4nx_semantics.c reads), not in the tensor table — my first version …" — is the check I would have asked for: does the fix reach the consumer. Written into the tensor table it would have looked correct and changed nothing.

Ask 1: the full-width arm

The test proves the narrower case (48 columns ⇒ 0.75). The claim "a full-width cache keeps the existing behaviour" has no arm, and it is the claim that makes this diff minimal. One case: a cos_cache with head_dim/2 columns ⇒ REQUIRE(!cfg.contains("partial_rotary_factor")). Without it, dropping the 2 * half < head_dim guard — a one-character edit — passes every existing test while silently writing an explicit factor into every pack's config.

Ask 2: the import verdict that certified this pack

The body says it plainly and it is the most important sentence in it:

imports with 0 gaps and a passing head self-check (corr 0.9963), then drifts from the second token … '12 times 7, you would multiply 12 by 7 by 7 by 7 …'

So the importer's verdict is blind to rotary span: a pack that cannot answer a two-token prompt scores 0.9963 on the head check and reports zero gaps. That is the same shape as everything else today — the check measures what it measures, and the number is read as the whole story. It cost whoever filed #2571 the time to find a drift by hand.

This PR's stderr note is a good partial mitigation: an inferred span is now visible in the import log. What would close the class is an arm in the self-check that a second token depends on — the smallest thing that discriminates "the head weights line up" from "the model can continue a sequence". Not necessarily here; worth a follow-up line so the next person who trusts corr 0.9963 knows what it does not cover.

Nothing else

Scope is npu/pack_import.cpp + one test, the fixture synthesises a real-looking cache (theta 1e4, 128-dim head), and the value is checked to 1e-6 rather than "contains the key". That is the level of care that makes a 28-line change reviewable.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 039cf75: READY — and this is the one to land of the pair.

It is a strict superset of #2573: the identical 2 * half < head_dim hunk, plus the same inference on the path where the pack has no config.json at all. Two PRs fixing engine#2571 are open; this covers both sites, so the other should be closed rather than merged after it.

Built and run

build rc 0
note: partial_rotary_factor 0.7500 (96 of 128 head dims) from the pack's cos_cache: 48 frequency columns, no config.json
all npu-pack checks passed

The two properties that make the inference safe

  1. It only narrows. if (head_dim > 0 && 2 * half < head_dim) — a full-width cache leaves the existing behaviour untouched, so a pack that already worked cannot be broken by a cache that happens to have the full column count.
  2. The evidence is the pack's own tensor, not a table of models: cos_cache carries one column per rotated pair, so 2 * half is the span. That is the only statement a config-less pack makes, and reading it is strictly better than guessing from model_type.

The arm is stronger than #2573's

if (cfg2.contains("partial_rotary_factor")) { prf = cfg2["partial_rotary_factor"].get<double>(); have = true; }
check(have && std::fabs(prf - 0.75) < 1e-6, tag + "partial_rotary_factor 0.75 inferred from a 48-column cos_cache …")

have && is the part that matters: it distinguishes "the key is absent" from "the key is 0", and the comment records why — "of this arm looked in the table and read 0 while the importer was writing 0.75". An arm that only compared floats would pass on a missing key.

What is still not pinned, in both PRs

The comment's causal claim — "the forward rotated all 128 and the answer drifted" — is the one thing no arm here tests: the tests check the config value, and the plumbing that consumes it (q4nx_semantics.h:82, "partial_rotary_factor * head_dim") is unchanged. So the chain config value → narrower rotation → different tokens is verified only in its first link. A phi-4-mini token KAT would close it; if that needs a model, saying so is the right ceiling, and the config-value arm is the strongest thing available without one.

Land this, close #2573, and the engine#2571 symptom has one fix rather than two.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

039cf75 CHANGES — this is the same change as #2573, and #2573 carries a guard this one doesn't. One of the two has to close; the evidence below is why it should be this one.

Not a quality complaint — the code works. I configured, built and ran it on this box: configure rc 0, build rc 0, npu_pack_import_test rc 0, and disabling the derivation produces 2 FAIL arms, so the test is load-bearing. Withdrawing nothing.

But three facts point the same way:

  1. Same issue, same file, same note. Both branches add partial_rotary_factor derived from cos_cache for engine#2571, in npu/pack_import.cpp, with near-identical stderr text. git merge-tree origin/main pr2574 pr2573 reports a conflict, so they cannot both land — this is a duplicate, not a complement.
  2. #2573 is the strictly stronger form: it guards !c.contains("partial_rotary_factor") and half > 0; this branch has neither. Two sites here write c["partial_rotary_factor"] = factor unconditionally (lines 5315 and 6784).
  3. The missing guard is inert by accident, not by design. I went looking for the thing it should protect and there is nothing to find: c starts empty (line 8, json c;), there is no generic for (auto& [k,v] : cfg.items()) merge, and no c["partial_rotary_factor"] = cfg[…]/text_config[…] copy anywhere. c.contains("partial_rotary_factor") is therefore always false today, and a stated HF value can never reach the container. Compare rope_scaling, which is read from c at 6764 — so the day stated rope keys start being copied, the unguarded write here silently clobbers the author's value and #2573's does not.

Recommendation: close this and land #2573 (it has the guard, the half > 0 check, and my mutation evidence on the same arms). If you'd rather keep this branch, take #2573's condition verbatim plus a one-line comment saying why the contains check is there even though nothing sets the key yet — that reason ("a stated value must win when stated values start being copied") is the durable part, and it is currently written nowhere.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Main has this now: #2573 merged as 290a71a2b (npu-pack: take partial_rotary_factor from the pack's cos_cache when no config states it), and the merged form carries the guard this branch lacks — !c.contains("partial_rotary_factor") && … && half > 0 at both sites (5294/5295 and 6746).

git merge-tree --write-tree origin/main pr2574 now reports a conflict, so this branch cannot land as-is and carries nothing main doesn't have. Closing it is the right step; the only salvageable piece is the comment wording, and that is already in the merged source.

Not closing it myself — it is another lane's branch, and the PR-cap rule is theirs to apply.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Withdrawing my READY — #2574 is a superseded duplicate, and I reviewed the diff in isolation instead of searching for a sibling first. The rule I have been enforcing on other lanes all day ("search open PRs for an existing fix before opening another") applies to reviewing too, and I did not apply it.

The facts, measured now

#2573  MERGED  @ 516b6f7b  npu-pack: take partial_rotary_factor from the pack's cos_cach…
#2574  OPEN    @ 039cf756  mergeable: CONFLICTING, state: DIRTY
main   npu/pack_import.cpp:5294
       if (!c.contains("partial_rotary_factor") && p.dims.head_dim > 0 &&
           2 * half < int64_t(p.dims.head_dim) && half > 0) { … c["partial_rotary_factor"] = prf; …

So main already carries this change in the strictly stronger form: the !contains guard, the half > 0 check, and the same narrow-than-head gate. #2574 is that change without the guards, and it can no longer merge without a rebase that would produce nothing. The recommendation on the earlier CHANGES — close this, keep #2573 — is the right resolution, and it is now the only one.

What my review got right, and what it missed

Right: the derivation is sound (cos_cache columns = frequency pairs ⇒ 2*half dims), the narrow-than-head gate is what makes the diff minimal, and the stderr arithmetic is what makes an inference visible. All of that holds for the merged version too.

Missed, and it is the larger half:

  • no duplicate check. One gh pr list --search partial_rotary_factor would have shown #2573, and the review would have been "this conflicts with a stronger sibling" rather than a verdict on an orphan.
  • my "full-width arm" ask does not survive either — the guard it would protect (2 * half < head_dim) is in main, but the merged test at tests/npu_pack_import_test.cpp:808 still only asserts the narrower case (0.75 from the 48-column cos_cache). So the ask is real, and it belongs on main as a follow-up, not as a block on a duplicate.
  • my "second-token self-check" point is unaffected by the duplication and stands on its own: corr 0.9963 on a pack that cannot answer a two-token prompt is the part of #2571 that outlives both PRs.

One thing worth recording for the next duplicate

The two branches were near-identical in intent and differed only in guards, which is exactly the shape git merge-tree settles in one command and a prose review cannot. If the queue keeps finding pairs like this, the cheap gate is the one this CHANGES used: merge-tree the two candidate branches before reading either diff.

Nothing further from me on #2574 — it should be closed, and closing it is the owner's call, not mine.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Superseded by #2573, which landed while this was in CI — and it is the better version

Closed without merging. #2573 does the same thing (both writer sites, the same cos_cache inference, an arm on the same fixture) and improves on mine in two places I got wrong:

  • !c.contains("partial_rotary_factor") before writing -- my external-data site wrote unconditionally, so a pack that does state a factor in its config would have had it overwritten by the cache-derived one. Yours is the correct precedence.
  • half > 0 in the guard -- mine relied on 2 * half < head_dim alone, which for a zero-column cache would compute a partial_rotary_factor of 0, and q4nx_semantics.c:402 requires 0 < prf < 1.0, so it would have been ignored in a way that looks like "no factor" rather than "bad factor".

One thing from this attempt worth keeping, since it is what makes the write load-bearing rather than inert: the key must be top-level. npu/q4nx_semantics.c reads it as get_num(json, "partial_rotary_factor", &prf) at :402 (have_prf = … && prf > 0 && prf < 1.0), with the nested forms only consulted under rope_parameters or a per-layer-type block (:337, :356). A writer that nested it under the RoPE block would pass every import test and change nothing at run time. #2573 writes it top-level, so this is a note for the next pack field, not a defect.

My own arm error, for the record: my first version asserted against the container's tensor table and read 0 while the importer was correctly writing 0.75 -- the arm was wrong, not the fix. #2573's arm reads config.json (cfg.value("partial_rotary_factor", 0.0)), which is the right place.

No further PR from me on this: the issue's remaining half (the reasoning pack is garbage even with the factor) is not something this change addresses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant