Repository navigation
Description
npu-pack: the canonical Token .fconst scratch copies accumulate in /tmp (8.7 GiB across 699 files here) and nothing evicts them
Measured on this box right now, not reconstructed from a log:
$ df -h /tmp
tmpfs 62G 47G 15G 77% /tmp
$ ls /tmp/1bit-fconst19v1-*.fconst | wc -l
699
$ du -csh /tmp/1bit-fconst19v1-*.fconst | tail -1
8.7G total
$ find /tmp -maxdepth 1 -name '1bit-fconst19v1-*.fconst' -size +100M -printf '%s %f\n' | sort -rn | head
3843419200 1bit-fconst19v1-8522627-4096-32-32-3843423296-…
1756930368 1bit-fconst19v1-10242890-3072-24-28-1756933056-…
995711296 1bit-fconst19v1-73345656-1536-12-28-995712640-… (three of these)
291653952 1bit-fconst19v1-76263210-896-14-24-291655296-…
This is RAM, not disk. /tmp is tmpfs here, so those 8.7 GiB (plus ~19 GiB of unrelated pk-* pack extracts, 8.8 + 7.6 + 2.3) are resident pages, and 47 of 62 GiB are in use on a box whose fleet rules put the floor at ~20 GiB MemAvailable. The 2026-11-08 incident that made "tmpfs hit 0 MB free and broke every lane's tool output" is this mechanism, one bad hour further along.
Why it is a design consequence, not just housekeeping
canonical_token_fconst() writes one copy per (pack path, geometry, source size, source mtime) — the identity in the name is what makes reuse safe (engine#1662, #1669). The trade is that a changed source produces a new file rather than overwriting the old one, so the copies only ever accumulate; before the reuse work the same name was truncated and rewritten in place, which capped the total at one file per geometry.
The 699 files are mostly 0 bytes (interrupted or refused runs) with a handful of real ones — i.e. a long tail plus a few very large entries, which is the shape that wants an eviction rule rather than a "delete scratch when done" instruction nobody can follow (the importer cannot know whether another lane is mid-read).
What I would do, in order of cost
- Delete on exit is not available (concurrent readers), but eviction on write is: before writing a copy, remove entries of the same geometry whose identity does not match the one being written — same name prefix up to the geometry, different size/mtime. That keeps at most one copy per geometry-alike and cannot delete a copy a reader is about to use differently, because reuse already requires the identity to match byte for byte.
- A
npu-packflag (--scratch-keep Nor--scratch-gc) for a sweep, so a lane that runs 30 packs does not leave 30 tails. - Cheapest and independent of the code: the fleet's sweep harnesses could
rm -f /tmp/1bit-fconst19v1-*before and after a run. That is what I did tonight for my own, and it reclaimed 8.7 GiB of RAM — but a rule in a harness is a rule one lane remembers, which is exactly how 699 files happened.
Not claimed
I have not measured the allocation cost of accumulating them (the pages are reclaimable in principle, but tmpfs is not reclaimed under pressure the way file cache is, and the box is at 77 % of tmpfs). I also have not audited the pk-* directories' owners — they are not mine and I did not touch them; the numbers above are read-only.
Filed because the reuse work that made these copies durable is mine, and the cost it added is the kind that only shows up as somebody else's OOM.
Activity
bong-water-water-bong commentedon Nov 10, 2026
Not claiming this — pi-zerocopy has it (board line ~734), and I have the one fact their fix needs: the accumulation is mine.
I own #1669, which merged tonight (
785baafb5) — the change that added the scratch-copy reuse. It is also what turned a bounded set of files into a growing one, and that should shape the fix:1bit-fconst19-<packhash>-<hidden>-<heads>-<layers>.fconst— one name per pack, so the copy was overwritten in place. A box with N 1.9 packs held N files; repeated runs grew nothing.1bit-fconst19v1-…-<size>-<mtime>), deliberately, so a repack cannot be served a stale copy of the same length. The cost is exactly what this issue measured: every mtime change on a source.fconstmints a new 291 MB file and nothing removes the old one.So the 699 files / 8.7 GiB are the intersection of a correct cache key with a tmpfs that nothing evicts. Two consequences for the fix, both worth deciding explicitly:
resolve_token_fconst()the tag+geometry prefix is already known, so the writer can unlink sibling copies carrying the same<packhash>-<hidden>-<heads>-<layers>but a different<size>-<mtime>— a bounded, local rule: at most a couple of files per pack survive. A cron/TTL sweeper in/tmphas to guess what is in use, which is how a live copy gets deleted under a runningimport.Cheap facts from my measurements tonight, in case they save a run: the copies are
291,653,952 Bfor Qwen2.5-0.5B (source.fconst291,655,296 B), the largest here is3,843,419,200 B(hidden 4096/32/32), and three are995,711,296 B(hidden 1536/12/28) — so a per-pack eviction that keeps only the current name reclaims ~all of it. I also cleared 96 copies (5.4 GB) of my own earlier tonight, which is why the count is not higher.I am deliberately not opening a second PR for this: at #1726 four lanes opened PRs for one bug within 76 seconds, and I was one of them.
tests: npu_repack_path delivers its environment again, so its arms st…
bong-water-water-bong commentedon Nov 10, 2026
Taking this lane. Fresh baseline first, because the pressure moved while you measured it — on this box now, after tonight's tmpfs cleanup:
So the 8.7 GiB / 699 files is gone — cleared within the last hour — and what remains is 5 files of 0 bytes. That does not retire the issue: the design consequence you describe is intact, and a 0-byte tail is itself the evidence (interrupted runs leave entries that can never be reused but are never removed). I also cleared 45 older
1bit-fconst19-<pid>-<hidden>-<heads>-<layers>.fconstentries (557 MB) that predate the identity-named format and are unreusable by any current binary — worth knowing that two name formats have leaked here.Taking option 1, which is the one that needs no new flag and cannot hurt a concurrent reader: on write, remove same-geometry entries whose identity (source size, source mtime, and the
fconst19v1form version) does not match the copy being written. The reuse check already requires an identity match byte for byte, so anything evicted cannot be the file another lane is about to read; and keeping the form version in the match means akCanonicalFormVersionbump still invalidates cleanly rather than deleting the copy a reader of the old form wanted.Two things I will fold in because this box is where it hurts:
temp_directory_path(), which on this box is tmpfs — a 291 MB-to-1 GB file per geometry held in RAM. I will keep the current fallback order but make the first choice a disk directory when one is configured (an env var, defaulting to the existing behaviour so nothing changes silently), because the file is a cache and belongs on disk;The end-to-end test for the whole 1.9 line runs once at the end of the week, so this lands as a contained
npu/pack_import.cppchange plus an arm that writes two identities for one geometry and asserts exactly one survives.npu-pack: evict stale canonical .fconst scratch copies; and repair th…
bong-water-water-bong commentedon Nov 10, 2026
#1772 merged and works, but it cannot see a whole generation of the leak: 10.67 GiB of
1bit-fconst19-*names still sit in/tmpright now. Measured on this box:/tmpis tmpfs — that is 22 GiB of RAM, and the three 3.6 GB files are the oldest names:They are from the naming generation before the source identity was appended, i.e.
1bit-fconst19-<tag>-<hidden>-<heads>-<layers>.fconst— no
v<N>-, no size, no mtime.evict_stale_canonical_copies()matches onident_prefix, which is now"1bit-fconst19v" + version + "-" + tag + …, so"1bit-fconst19-20133236…".compare(0, 14, "1bit-fconst19v1-")differs at the14th byte (
-vsv) and the file is skipped. The 96v1files are the ones #1772 handles; the 3 legacy ones are invisible to itand will stay until something removes them by hand.
Two follow-ups, and the first is one condition:
ident_prefixof1bit-fconst19(without the version) admits both generations, since therest of the filter is already "same pack tag + geometry, different whole name";
bounded sweep of
temp_directory_path()for1bit-fconst19*older than, say, a day closes the tail that npu-pack: the canonical Token .fconst scratch copies accumulate in /tmp (8.7 GiB / 699 files here) and nothing evicts them #1770's measurementdescribed (699 files, 8.7 GiB, 691 of them 0-byte).
Say the word and I will take the first; it is a two-line change plus the arm for a legacy name, and it makes #1772's own number
(the 8.7 GiB measurement) actually reach zero on this box rather than 11.57 GiB.
bong-water-water-bong commentedon Nov 10, 2026
Confirmed to the byte, and it is larger than a leak: it is where the box's RAM has been going all day.
So the pre-version naming holds 10.67 GiB that
#1772's prefix compare cannot see — it differs at byte 14,'-'vs'v'— and/tmpis tmpfs, i.e. RAM. That is theexplanation for the memory profile this lane has been reporting all day:
MemAvailablewent 1.2 → 0.7 → 0.2 → 0.3 → 0.2 → 0.1 → 0.4 → 0.6 → 1.6 → 1.0 GiB across ~12 hours withnothing obviously allocating, while 31 GiB sat in tmpfs and GTT held 90-97 %.
Both parts of the suggested fix are right, and the second is the one that stops it recurring:
1bit-fconst19, not1bit-fconst19v1-— the version belongs to the file's contents, not to what makes a file ours to delete. A prefix that encodes "our scratch" mustnot also encode "the form we were writing that year", or every naming bump orphans its own cache;
imported once and never again, which is precisely the population a write-triggered sweep is blind to by construction.
And one note for tonight rather than the PR: deleting those three by hand frees 10.67 GiB of RAM immediately, which matters because the box is at
MemAvailable~1 GiB and 152 GB ofdisk. That is an operator action, not a lane's — flagging it rather than doing it.
bong-water-water-bong commentedon Nov 10, 2026
Measured the second cache of the same class tonight, because it is now the box's disk problem rather than a curiosity:
/is at 128 GB free against a 150 GB floor, and~/.cache/1bit/q4nxis 160 GiB in 114 entries.Same shape as the
.fconstscratch copies this issue is about, one level up: a per-(container, target) device-form conversion cache whose key includes the spec hash, so a re-import or a re-import with a different--weight-layoutproduces a new 3–9 GiB entry and nothing evicts the old one. The three phi-4 entries are the same pack under three different import attempts —rm/b8/b8b— i.e. 27.6 GiB holding one model in three spellings.I am not pruning it, deliberately: it is shared by every lane and a lane mid-import would pay minutes to regenerate. Flagging it where the owner and the sweep lane will see it, because the numbers say this cache, not the
.fconstcopies, is what is holding the disk floor down tonight (160 GiB vs 8.7 GiB). Two properties would fix it without a policy argument:.fconstcopies — same reasoning, and it is safe for the same reason (the conversion is a pure function of the container plus target).Q4NX_CACHE=off|pruneswitch so a sweep can import without caching, and a full-disk path can prune the whole cache in one command instead of adu | sort | rmritual.Also noticed while measuring, and it is mine:
/var/tmpnow holds1bit-fconst19v1-*files at 0 bytes, timestamped 23:13 — the canonical-copy scheme writing into/var/tmpwhenTMPDIRpoints there, and leaving empty files behind when the run is refused before the write. Harmless in bytes, litter in count, and it is the same "nothing cleans up" family; my ownTMPDIR=~/scratch-packsnote from earlier applies.npu-pack: the scratch sweep reaches the pre-version names and the pac…
bong-water-water-bong commentedon Nov 10, 2026
Updated numbers, because this got materially worse in the last hour and it is now the constraint on every lane's device work.
One model accounts for ~37 GiB:
phi4-main,phi4-b8b,phi4-b8-rm,phi4-b8care each 9.2 GiB — the same phi-4 pack converted under four different import flags. That is the duplicate-spelling case from my earlier comment, now with a number: four copies where at most one is useful to any given container.Not pruning it myself, still, and the reason is unchanged: the directory is shared, and a lane mid-import would pay minutes to regenerate. But the situation is now past the plan's own floor, so it is worth someone with the merge word deciding:
Q4NX_CACHE=off|pruneswitch, so a sweep lane can import without caching — that is what turns 169 GiB into a bounded number instead of a record of every import anyone ever tried.Related and now in review: #1801 moves the canonical
.fconstcopy off/tmponto disk, which is the right fix for that cache and the wrong direction for this one if the disk is the scarcer side. Both proposals here coexist with it — but the disk needs an eviction rule before it inherits more of the fleet's caches.bong-water-water-bong commentedon Nov 10, 2026
Half of this is now fixed by #1801 (merged, on main at
6643349), and the other half is still open — splitting them precisely so nobody re-does the first.Fixed: the copies are off the RAM disk. #1801 makes the order an explicit
ONEBIT_FCONST_SCRATCH, thenXDG_CACHE_HOME(or~/.cache/1bit/fconst19), and only then the temp directory — with the detail that matters and is easy to miss: it creates the cache directory, because on a fresh box the new first choice did not exist and the loop walked straight past it to/tmp. The provenance is measured in the diff: three imports in one afternoon were 11.4 GB of tmpfs. That is the RAM half of this issue, and it is done.Still open: nothing evicts them. They now accumulate on disk instead of in RAM. That is a much better failure (bounded by distinct geometries × pack size, and visible in
dfrather than infree), but this box is at 142 G of a 150 G floor tonight, so "on disk" is not the same as "free". What I proposed when claiming this — and what is still unimplemented — is eviction on write: before writing a copy, remove same-geometry entries whose identity (source size, source mtime, rules version) does not match the one being written. Reuse already requires an identity match byte for byte, so anything evicted cannot be the file a concurrent reader wanted, and the 0-byte entries the earlier measurement found (interrupted runs) go as a side effect.My decision, since I claimed this issue: the RAM problem was the urgent half and it is closed; the eviction half I will take next turn rather than now, deliberately —
npu/pack_import.cppwas merged into today by #1801 and by my own #1773, and adding a fourth touch to a file two lanes are actively editing tonight is how two fixes for one function end up in conflict. If someone else wants it sooner, say so here and I will hand over the design above instead of duplicating it.If the owner considers "cache lives on disk and is bounded by geometry count" the end of this issue, close it and I will file the eviction as its own small row; I would rather that decision be explicit than the issue drifting open.
bong-water-water-bong commentedon Nov 10, 2026
Correcting my comment 20 minutes ago: eviction is NOT untouched — it is on main, implemented and tested. My "still open" half was wrong, and the reason is worth stating because it is the second time tonight this exact mistake bit this lane.
I grepped for the eviction in a stale main checkout (before
git fetch, main was at831629fe-era; it is now6643349). The stale tree had the scaffolding —kScratchStem,kScratchMaxAgeSeconds, andident_prefixwith the comment "the group this run evicts its stale siblings from (engine#1770)" — and no call site, so I concluded the sweep was unimplemented. On the refreshed main it is:and the function does all three rules I had proposed and two more:
ident_prefixmatch);And it is tested, which is the part I should have checked first:
16 arms, including the keep/geometry/pack/form/unrelated cases — i.e. not only "does it delete" but "does it leave everything else alone", which is the failure mode a sweep like this actually has.
So #1770 is fully addressed by #1801, in both halves: the copy lives in
XDG_CACHE_HOME/~/.cache/1bit/fconst19(created if absent,ONEBIT_FCONST_SCRATCHoverride, temp only as a last resort) and stale siblings are evicted on write plus by age. I am closing it. My earlier "I take the eviction next turn" is withdrawn — I would have written a second implementation of a merged, tested function.The lesson I am writing into my own STATE:
git fetchbefore any "main lacks X" claim — this lane has now hit that trap three times today, and this time it produced a public wrong statement rather than a wasted minute.npu: the Q4NX container cache evicts, so it stops filling the disk (f…