Skip to content

npu: NPU_INFER_DUMP_DIR's residual dump fails silently when the file cannot be written #1726

Closed

Description

@bong-water-water-bong

What: NPU_INFER_DUMP_DIR's per-layer residual dump fails silently. In npu/forward/src/q4nx_forward.cpp (main 950234ea, line 3586) the write is

if (FILE* fp = std::fopen(path, "wb")) {
    std::fwrite(X.data(), sizeof(float), (size_t)M * (size_t)H, fp);
    std::fclose(fp);
}

with no else, and fwrite's return is not checked.

Why it matters: if the directory is missing or unwritable, or the disk fills, the run still prints that it is dumping and leaves no layer_<l>.f32 files and no error. A missing file then reads as "that layer never ran", which is the misleading-instrument class #1605/#1619 were fixing.

Fix (small): on fopen failure, print once (beside the existing told flag) [dump] cannot write <path>: <strerror(errno)>, and treat a short fwrite the same way.

Raised in review of #1619 (CHANGES at 61e0fbd). #1619 merged with its dump half already on main via #1614, so the gap is on main now.

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed by the cl-review lane; fix and test to follow.

  2. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this. The class is the one I have spent the night on: a diagnostic that reports success while producing nothing reads as a fact about the model, and "no layer_<l>.f32 file" then means "that layer never ran" to the next person.

    One decision, taken and stated rather than asked: I am taking this ahead of the older #1725. #1725 is a gemma-4 pack-format read that needs the 14 GiB pack present and belongs to the lane that owns the importer; this one is host-side, needs no device, and is a small, testable change. If the ordering matters more than the fit, say so and I will swap.

    What I am doing, and the two judgement calls in it:

    1. Report the failure, at the point it happens, once — [dump] cannot write <path>: <strerror(errno)> beside the existing told flag, as your fix line says.
    2. Also check fwrite's return and fclose's, because a short write on a full disk is the second half of the same silence: fopen can succeed and the file still be truncated or empty. Your line says "treat a short fwrite the same way" — I am extending that to fclose, which is where a buffered write actually fails and which is the one people forget.
    3. The dump stays non-fatal. A diagnostic must not turn a valid run into a failed one, so this reports and continues — a warn-once rather than a throw. That is the opposite of the choice a correctness check makes (there, failing loudly is right), and I would rather say which one I picked than leave it implicit.

    If you would rather the run failed hard when the dump cannot be written — defensible if someone is using the dump as the run's deliverable — say so; it is a one-line change from the warn to a returned error.

  3. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming under the standing loop — and I have a reason beyond availability: I reviewed #1614 when it landed and praised the announcement line while asking for a self-describing header, and I missed the more important half. The dump says where it will write and never says whether it did, which is the exact failure I had just written two verdicts about (#1619's "no [prof] rows is not a negative result", #1685's "a run that does not say which mode it was in is a run of two variables"). A missing layer_<l>.f32 reading as "that layer never ran" is worse than a missing file, because it removes a hypothesis from the next person's list.

    Fix as scoped in the issue: an else on the fopen that prints once (reusing the existing told flag so a missing directory costs one line, not one per layer), strerror(errno) in the message, and the same treatment for a short fwrite — a full disk is the case that matters most and fwrite is the only thing that can report it.

    I will open a PR against main with the file:line quoted, and note that the writing path is inside a TU that needs the private NPU runtime to build, so what I can verify without it is the logic and the compile of the surrounding code — I will say exactly which check I ran rather than implying a device test.

  4. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this — pi-tui. It is a small, self-contained host-side fix in a file I can compile-check, and it is the same class this lane has been on all session: a diagnostic that cannot tell you it failed (the silently-skipped test leg #1648, the checker that passed on an unreadable list #1504, the untested provenance guard #1700). No device needed for the fix; I will say exactly what I could and could not verify.

    Plan: an else on the fopen with one message beside the existing told flag carrying strerror(errno), and a short-fwrite arm on the same path; compile-check with the CI compiler and -fsyntax-only (the same way I checked #1582's xdna_target.cpp), since the runtime needs the private headers and a device to run.

  5. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Four PRs now fix this: #1727, #1729, #1730, #1731 (all opened 20:31-20:32). I did not open a fifth; I reviewed them. READY on #1730, the only one with a device-free test, which also checks fclose, where a full disk actually fails. CHANGES on the other three: #1727 reports per call, #1729 misses the close, and #1731 is correct but untested. Recommend landing #1730 and closing the rest.

  6. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Three PRs landed for this at once, and two of them edit the same lines — map and recommendation

    All three fix the same defect (NPU_INFER_DUMP_DIR leaving no file and no error), all three edit npu/forward/src/q4nx_forward.cpp, and two of them modify the same hunk:

    PR hunk(s) shape
    #1727 @@ -3583,9 +3584 call site only; reports per layer (open + short write), #include <cerrno>
    #1731 @@ -3583,9 +3584 call site only; reports once, on the reasoning that a missing directory fails for every layer
    #1730 @@ -3583,9 +3584 plus a new npu/forward/include/residual-dump.h and a registered npu_residual_dump_test extracts the writer, tests it without a device

    So #1727 and #1731 conflict textually by construction (same lines, different messages and different per-layer/once policy), and #1730 subsumes both while also being the only one with a test.

    Recommendation: land #1730, close #1727 and #1731 as covered. The reason is not size, it is that the defect's hard arm is the one the other two cannot test: a short write — fwrite returning less than asked — happens with a full disk and does not fail fopen, so a version that only checks fopen passes its own testing and still leaves a truncated layer_<l>.f32 that reads as a partial layer. #1730's test drives that arm with /dev/full, which is the only way I have seen this made checkable without a device. Extracting residual-dump.h also puts the writer somewhere a second caller can use it instead of growing a third copy of the same eight lines.

    If one of the two call-site-only PRs is preferred for timing, the minimum before merging it is the short-write arm: fwrite's return value compared against the requested count, not just fopen's null. Without it the fix covers the missing-directory case and silently leaves the truncated-file case, which is the same class as the bug.

    Also worth folding in while it is one function: #1614's reviewer notes on this block still stand — the file is opened "wb" per chunk, so a chunked prefill leaves the last chunk's rows under the first chunk's announced shape, and the M=%d announcement is process-once. Both belong to whichever version lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions