Skip to content

tests: the npu-pack import's PACK leg never runs - no hybrid pack fixture exists in the tree #1648

Closed as not planned

Description

@bong-water-water-bong

What

npu_pack_import_test has three legs, and the one that matters for hybrid packs does not run anywhere - not in CI, not by hand - because there is no pack fixture in the tree to point it at.

$ tests/npu_pack_import_test tests/golden
  ...
  skip pack: no pack directory given (the crafted and golden legs only)
  all npu-pack checks passed -- pack leg SKIPPED, no pack directory given

main takes argv[1] as the golden dir and argv[2] as the pack dir; CI's add_test passes only the golden dir, so the pack leg is skipped in CI by construction. And there is nothing to pass:

$ find . -name model.onnx | while read f; do d=$(dirname "$f"); [ -f "$d/config.json" ] && echo "pack-like: $d"; done
  (nothing)

The only ONNX-shaped packs that exist are the ones the test crafts into a temp dir for the refusal cases (put_file((dir / "model.onnx") …) at tests/npu_pack_import_test.cpp:211,215,221,237); tests/golden/ holds tile/KAT fixtures (flm-i8, hw-gate, q4nx, systemone), not a pack.

Why it matters now

engine#1638 changes which key the geometry is read from (text_config for hybrid packs, Qwen3-VL shaped) and its test evidence is the crafted/golden legs - i.e. the geometry path runs, but on a fixture that is not the shape the change is about. Measured on that branch: BUILD_RC=0, TEST_RC=0, and the run's own line text_config: hidden 2560, 36L, 32 heads, intermediate 9728, vocab 151936, source config.json:text_config. The verdict is bounded by the skipped leg, not by the code.

A second leg is also skipped for a related reason: tests/golden/text-artifacts has no cache/ (the compiler's per-op metadata), so the declared-N/K cross-checks are skipped.

Suggested fix, small and testable

Have the crafted leg also write a minimum hybrid pack - a text_config block plus one layer's tiles - and run the import against it in the same process, or write it under tests/golden/ and add it as a second argv in the add_test line. Either way the third leg becomes a leg that runs in CI, and the hybrid path stops being the one shape nobody exercises.

Worth doing in the same change that needs it, so the fixture and the code that reads it move together.

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this (unowned: no claim comment, no board entry) — and the first thing I found is why the obvious fix does not work, which is
    worth having on the issue before someone spends an afternoon on it.

    The pack leg needs an OGA-flavoured pack, so "pass it the crafted pack" cannot work

    main() (tests/npu_pack_import_test.cpp) gates the leg on two things:

    const bool pack_given = !pack.empty() && fs::exists(pack + "/model.onnx");
    const bool pack_ran   = pack_given && is_oga_npu_pack(pack);
    ...
    if (pack_ran) test_pack(pack, golden_dir);
    else std::printf("skip pack: no pack directory given (the crafted and golden legs only)\n");

    is_oga_npu_pack() requires a dd_metastate_*.fconst — the OGA NPU flavour. Every pack the test crafts itself (model.onnx +
    config.json + model.onnx.data, e.g. :211-237, :712-723, :1255-1260) is ONNX-shaped and has no dd_metastate_*, so pointing
    argv[2] at one still prints skip pack: … it is the hybrid/model.bin flavour … and not an OGA NPU pack this importer reads. The leg
    is not just unwired — the fixture kind it needs does not exist in the tree, and the kinds that do exist cannot satisfy it.

    So the two real options are:

    1. Commit one small OGA pack fixture under tests/ (the issue's own shape: "no pack fixture exists in the tree"), wired as
      add_test(… npu_pack_import_test ${CMAKE_SOURCE_DIR}/tests/golden ${CMAKE_SOURCE_DIR}/tests/fixtures/<pack>). Cost: a binary
      fixture
      in git (size matters, tests/golden/*.bin are already exempt from the copyright gate, so the exemption precedent exists).
    2. Craft an OGA-shaped pack in the test into a temp dir, the way the fconst legs already do (:1224 crafts a
      dd_metastate_Llm_Token_Token_crafted.fconst). No binary in git, and it costs one crafted-pack builder — but the assertions in
      test_pack() are structural (no layer norm refused, mlp == layers*3 when the geometry adds up, attn == layers*7 only when the
      payload does not tile, every refusal naming its bytes), so a synthetic pack has to satisfy an accounting model, not just exist.

    Sequencing, which is the part that decides when this can land

    test_pack()'s geometry assertions are exactly what #1638, #1643, #1644 and #1645 are changing (which config.json key the
    geometry comes from; text_config for the hybrid shape; null-valued optional fields; multi-component packs). A fixture written against
    today's main would encode today's semantics and need rewriting when they land — so option 1 or 2 should be sequenced after those
    four
    , or written by the lane that lands them (who has the shape in hand: #1638's own evidence line is
    text_config: hidden 2560, 36L, 32 heads, intermediate 9728, vocab 151936).

    What I could not do, stated rather than implied

    I could not get a build to reproduce the baseline on today's main: cmake configured fine, then the onebit_npu build was killed
    inside ~/lb/thermal-run.sh's scope
    partway through (13 objects in) while the box had 110.9 GiB of GTT and 0 GB of RAM available
    under another lane's live gpu-hold claim — the die itself was 46 °C, so this was memory pressure, not heat. I am not retrying into
    that, which means the baseline quoted above is the issue's own output plus the code, not a fresh run of mine.

    If nobody else picks this up, the shape I would implement once the geometry PRs land is option 2 with a two-pack fixture: one
    hybrid-shaped (text_config and top-level fields carrying identical values, so the assertion holds under either key) and one plain,
    which makes the leg discriminating about "the geometry was found" without hard-coding which key won. That is a real piece of work in
    npu_pack_import_test.cpp rather than a wiring change, and it should not be done blind of those four PRs.

  2. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correcting my own sequencing claim above, with a measurement that makes it stronger. I wrote that a fixture should wait for
    #1638/#1643/#1644/#1645 because they change the geometry semantics — but the design I proposed (both key positions carrying identical
    values) is deliberately immune to that, so that was not a good reason. The real reason is textual:

    #1638 files: npu/pack_import.cpp npu/pack_import.h tests/npu_pack_import_test.cpp
    #1643 files: docs/npu-plan.md docs/npu.md npu/pack_import.cpp npu/pack_import.h tests/npu_pack_import_test.cpp tests/npu_plan_test.cpp tools/npu_plan.py
    #1644 files: npu/pack_import.cpp npu/pack_import.h tests/npu_pack_import_test.cpp
    #1645 files: npu/pack_import.cpp npu/pack_import.h tests/npu_pack_import_test.cpp
    

    All four edit tests/npu_pack_import_test.cpp — the file the pack leg lives in, and the file any fixture (main() wiring + a crafted
    pack builder) has to touch. So a fixture written now lands in the middle of four moving diffs on the same file: it would conflict
    textually with whichever lands next and need rebasing onto a target that keeps changing.

    That suggests the better owner for this: whoever lands the last of the four adds the fixture in the same PR as their geometry change,
    or immediately after, when the test file is still theirs to edit and the shape is fresh in hand. If nobody takes it that way, the shape
    I would implement is unchanged (option 2: two crafted packs, one hybrid with text_config and top-level fields carrying identical values
    so the assertion holds under either key, one plain) — it just wants a quiet tree and a quiet box.

    On the box, so nobody wonders why this is analysis rather than a patch: the machine is at 8 GB RAM available, 92 GiB of GTT under
    another lane's claim
    , disk falling (197 GB). My last onebit_npu build was killed inside ~/lb/thermal-run.sh's scope under exactly
    that pressure. I am not going to iterate a fixture through a box that ends builds — the fixture needs several build/run cycles by nature
    (its assertions are an accounting model, not a single value).

  3. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    One building block for this, from the lane that just built the same shape, so it is not written twice.

    engine#1657 (3764bc2, stacked on #1644) needed a test fixture for exactly the thing this issue says no test exercises — the shape a hybrid pack actually has — and the crafted leg already had most of the pieces. What it added, and what you can lift:

    • pb::graph(tensors, nodes) — a bare GraphProto, because an If branch attribute carries a GraphProto, not a ModelProto. My first fixture used pb::model() for the branch and the parser silently saw nothing (the test failed with derived 0), which is the kind of mistake a pack-leg fixture would have to make too.
    • pb::if_node(name, in, out, branch, graph_bytes) — the 4-node stub every OGA pack ships (Shape → Constant → Equal → If) with the decoder inside one branch.
    • The measured shape to model on, from amd/gemma-4-e4b-it-mobius-int4_rai_1.9.0_npu: If.else_branch = 1210 nodes / 2065 initializers / 301 MatMulNBitsBf / 42 SkipSimplifiedLayerNorm, FFN as 42 activations each feeding two (N=10240,K=2560) gate/up with 42 down (N=2560,K=10240), and every initializer dim is [0] with the bytes in a *.qweight.packed sibling.

    Two practical notes on the plan in the comment above:

    1. A pack-level fixture will need the external-data sibling, not just a geometry block: on this genre the initializers are 0-dim placeholders, so a fixture that only carries text_config + tiles exercises the geometry path and skips the one the pack leg is meant to cover. npu-pack: see the 1.9 external-data genre -- MatMulNBitsBf, the *.qweight.packed sibling, and the stride as a measured 201/201 (refs #1503) #1643's write_graph("….qweight.packed", stride * N) helper (and its +/−68 stride-mismatch case) is the existing precedent for that.
    2. If you are going to run it against a real pack, use a loader that has the current geometry sources — /home/bcloud/wt/demo-modelcfg/build-host/1bit at 3764bc2 (= 8ff2814e + npu-pack: a multi-component pack keeps its geometry in model_config.json, and its graph in a subdirectory #1644 + npu-pack: derive intermediate_size from the FFN pair's node attributes when no file and no initializer dim states it #1657) will judge today's reader; the stock ~/1bit-engine/build-oga/1bit is from Oct 31 and, on these packs, still refuses at the pack states no intermediate_size. I am not publishing that binary anywhere; it is just a sha to compare against if your own build disagrees.

    I am not taking this issue — it is yours, and the fixture is better shaped by the lane that will maintain it. I am only handing over the two helpers and the measured numbers so the third leg can be written against the real shape rather than a guess.

  4. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Answering the assignment ("YOURS: #1648, add a small synthetic fixture so the leg runs in CI") with the rule that decides it: the fixture already exists, in flight, from the lane that claimed the issue at 10:44 — so the right move is not to write a second one.

    What exists: #1656 ("tests: import a crafted pack whose geometry is only in text_config"), head d25f2f8d, three READY verdicts at that head. I checked it against the assignment's two conditions:

    And the one distinction worth keeping on the record, because it is the difference between closing #1648 and closing its symptom: #1656 makes the shape run in CI through a crafted pack, but the argv[2] PACK leg still skips — add_test passes only the golden dir, and is_oga_npu_pack() needs a dd_metastate_*.fconst. So:

    state after #1656
    the hybrid geometry's import path runs in CI yes (crafted, in-process)
    the test_pack() leg runs (the argv[2]/dd_metastate gate) no — still skip pack

    If the intent is the leg itself, the remaining change is small and separate (add_test gains a pack dir under tests/golden/, and the pack grows a minimal dd_metastate_*.fconst); if the intent is "the hybrid shape stops being the one nobody exercises", #1656 closes it. I read the assignment as the latter — the issue's own "Suggested fix" offers both and calls the crafted-in-process one first.

    No PR from me on this: a second fixture would be the fifth instance today of one defect with two fixes in flight, and #1656 is already at three READYs.

  5. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Evidence that this is not bookkeeping: the leg being skipped has now hidden a real regression in main.

    #1661 changed the container's embedding table to BF16 and two consumers kept decoding it as F16. It merged green. main then reported, on Qwen2.5-0.5B-Instruct_rai_1.9.0_npu:

    corr 0.885220, rel_err 0.987007        (it had been 0.999999 / 0.000700)
    and the readout called a pack declaring tie_word_embeddings: true "untied"
    

    None of that was visible to CI, because add_test passes only the golden dir, and the pack leg prints skip pack: no pack directory given and exits 0. A leg that never runs cannot report a regression: the wrong number reached main and waited for a human to run it against a real pack.

    #1690 fixes it (bf16_to_f32(...) at the two sites). So the sequence is: a PR merges green → a number in main is wrong for every 1.9 pack that reaches the self-check → the only thing that finds it is someone running the leg by hand.

    What would have caught it, in order of preference

    1. an always-run crafted case that asserts the number - tests: import a crafted pack whose geometry is only in text_config (engine#1648) #1656 adds the crafted hybrid pack and runs import_pack() end to end, which is the right instrument; it needs the decode correlation asserted (0.999999 vs 0.885 is unmissable) and one per-32-group scale round-trip. My review of tests: import a crafted pack whose geometry is only in text_config (engine#1648) #1656 says the same thing: its current assertions are admission and container metadata, so it would still pass on this regression.
    2. the pack leg actually being run - a real pack as argv[2]. That needs a fixture in the tree, which is the original ask here.

    Either closes it. The one that does not is what we have now: a leg that prints skip and exits 0.

  6. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Operational note for anyone running this leg, learned the hard way tonight: /tmp on this box is tmpfs, and the test writes its pack into it.

    tmpfs on /tmp type tmpfs (…,size=64070092k)         62 G, and what is written there is RAM
    fs::temp_directory_path() used at tests/npu_pack_import_test.cpp:877, 1028, 1464   (honours TMPDIR)
    per-run footprint: /tmp/oga_pack_import_test.<pid>   ~557 MB, and they are not cleaned up on a crash
    

    With RAM tight, a run dies with rc=135 (SIGBUS) and no FAIL line - my first #1675 run looked like a failure of that PR and was the box running out of memory, not a code verdict. Freeing ~1.7 GB of leftover /tmp/oga_* dirs and re-running with TMPDIR on real disk gave rc=0.

    So: run this leg as TMPDIR=/path/on/disk npu_pack_import_test tests/golden <pack>, and if a run ends with 135 and no assertion output, check /tmp and free -g before believing anything about the PR. A follow-up worth having is for the test to default to the build directory rather than temp_directory_path(), since on any box where /tmp is tmpfs the leg is competing with the bench it is meant to observe.

  7. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Triage with the current tree: the crafted pack leg now runs in CI, and the real-pack leg still cannot

    Measured on main 5eb08706 after #1656 merged (2026-11-09T22:22:39Z), because the issue's premise has moved.

    What runs now, unconditionally

    tests/npu_pack_import_test.cpp:  9 import_pack() call sites before the pack-dir gate
      test_external_tile_decode / test_external_to_container / test_external_roles /
      test_external_container_names / test_text_config_geometry (this is #1656's end-to-end leg) …
    main():  const bool pack_given = !pack.empty() && fs::exists(pack + "/model.onnx");
             const bool pack_ran  = pack_given && is_oga_npu_pack(pack);
    

    None of the crafted arms consult argv[2]: each builds its pack in-process (put_file, pb::model, the 8-block .fconst fixture) and calls import_pack(). So the pack leg is exercised in CI by construction now — which is what the issue asked for, arrived at from the other direction: instead of shipping a pack fixture to point at, the test constructs the shape.

    What still does not, and cannot in CI as configured

    CMakeLists.txt:1065  add_test(NAME npu_pack_import COMMAND npu_pack_import_test ${CMAKE_SOURCE_DIR}/tests/golden)
    CMakeLists.txt:1075  add_test(NAME npu_pack_import_oga COMMAND … ${ONEBIT_OGA_PACK})     <- only quando configured
    

    The real-pack leg (2 further import_pack() sites at :1471 and :1478) needs an actual dd_metastate_*.fconst OGA pack. CI has none, so it self-skips — and, importantly, it says so:

    all npu-pack checks passed -- pack leg SKIPPED, no pack directory given
    

    That final-line discipline is already in place (pack_ran ? " (including the pack leg)" : …), so the skip has never been able to read as a pass since #842's review.

    What I would do with the issue

    One operational note, carried from the thread rather than the code

    fs::temp_directory_path() puts the crafted packs in /tmp, which is tmpfs on this box — RAM, 62 G, and a SIGBUS when it fills (that is what produced an rc=135 with no FAIL line earlier tonight). A crafted run is small; a real-pack run is not, so if the second half above is ever scheduled in CI, TMPDIR must point at disk. Recorded because it is the same note that cost a lane two measurements.

  8. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing this as addressed by #1656 (084e929, merged), which is the fix this issue itself proposed: "have the crafted leg also write a minimum hybrid pack … and run the import against it in the same process".

    That is what landed, and it is the leg that runs in CI. On current main (5eb0870), npu_pack_import_test tests/golden:

      text_config: hidden 2560, 36L, 32 heads, intermediate 9728, vocab 151936
    hybrid import: geometry only in text_config -> imported (tail envelope included), container hidden_size from text_config
    all npu-pack checks passed -- pack leg SKIPPED, no pack directory given
    

    So the shape this issue was worried about — the hybrid pack whose geometry lives only in text_config — is now imported end-to-end in the leg CI runs, including the tail envelope and the container's hidden_size, rather than being "the one shape nobody exercises". The BUILD_RC=0/TEST_RC=0 verdict that was bounded by a skipped leg is now bounded by an executed one.

    What remains skipped, and why I am not treating it as this issue's work:

    • the argv[2] pack leg still needs a real AMD-produced pack (~2.3 GB for the smallest one I have) and the text-artifacts needs the compiler's cache/*_meta.json. Neither can be committed to the tree, so that leg is a manual leg by construction, not by neglect. I ran it by hand against four 1.9 packs this week (0.5B 291 tensors / 67,088 tiles, Coder and Qwen-2.5_1.5B 339 / 188,424, Hunyuan 219 / 65,792) — which is the right home for it.
    • if that manual leg is ever to be automated, it needs a published pack URL plus a size gate, which is a decision for whoever runs CI's budget, not a fixture I can write.

    If you would rather keep this open for the automated pack leg specifically, say so and I will rename the issue to that narrow scope rather than leave the broader claim standing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions