Skip to content

tmpfs: 17.0 GiB of abandoned fconst19 pack scratch in /tmp (188 files, 2-9 h old) is RAM, not disk #1907

Closed

Description

@bong-water-water-bong

Measured 2026-11-10 11:4xZ, no device and no build involved — this is a filesystem finding.

/tmp on this box is tmpfs, so everything below is RAM, not disk:

$ df -h /tmp
tmpfs  62G  38G  24G  61%  /tmp

$ ls /tmp/1bit-fconst19* | wc -l
188
$ stat -c %s /tmp/1bit-fconst19* | awk '{s+=$1} END{print s/1073741824}'
17.0        # GiB, 188 files

What they are

Per-pack build scratch from the fconst19 pack builder, keyed by a hash and not of the form the #1772 eviction prunes:

/tmp/1bit-fconst19v1-36455404-4096-32-40-4534325824-13803363094301236721.fconst   4.3 G
/tmp/1bit-fconst19v1-96033964-3584-28-28-3830642048-13803372972454282346.fconst   3.6 G
/tmp/1bit-fconst19v1-83421898-3584-28-28-3830642048-13803373531375081270.fconst   3.6 G
/tmp/1bit-fconst19v1-44412148-4096-32-36-3756631872-13803361918437481547.fconst   3.5 G
/tmp/1bit-fconst19v1-71316895-3072-24-28-1756933056-13803369888320294639.fconst   1.7 G

Age: the whole set is 2-9 hours old (oldest 07:13Z, newest 06:31Z — none from the last five hours), so none of it is live work. The remaining ~183 files are small.

Why it matters

The box was at MemAvailable 12.6 GiB an hour ago and 65 GiB now — with 38 G of the 62 G tmpfs already spoken for, a couple of concurrent pack builds are enough to reach the state that took out the 11-08 evening: an OOM that killed the PR-agent LLM, not a disk-full.

It also gets misread as disk pressure. /home free space (152-182 G across the night) is unaffected by any of this; the two numbers move independently and I have been quoting one while chasing the other.

What would fix it

  1. evict by age, not only by key shape — a scratch file with no writer for N minutes and no process holding it is garbage; the builder currently leaves it indefinitely;
  2. a per-owner byte ceiling, so one lane's pack experiment cannot fill the tmpfs for everyone;
  3. or put the scratch on disk, since /home has 155 G free and these files are pure temporaries.

I am not deleting them — other lanes' scratch is not mine to remove (owner, 2026-11-09 night). Flagging the measurement and the mechanism instead.

Related

The builder that writes these is not in the tree — tools/fconst19_measure.py is, but the pack builder is the missing half described in #1843, and ~/t_sweep was removed during the 15.8 GB disk break. So even the writer of these files cannot be re-run from a commit, which is the second reason to record this here rather than patch it silently.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own verdict above, made after measuring instead of eyeballing df:

    I wrote that /tmp held "36 GB of stale .fconst copies". Both halves of that were wrong and I would rather correct them than leave a number on the record:

    • 36 GB is all of /tmp, not the fconst copies. The copies are 145 files / ~16 GB; the rest is unrelated (/tmp/claude-1000 3.1 GB, /tmp/drmmisc-probe 1.7 GB, …).
    • They are fresh, not stale. The newest is 26 seconds old at the time of measurement, and all 145 are under 6 hours — so the existing 6 h age sweep would touch none of them. My "2–9 h old" carried over from the issue text, not from this box.

    That changes what this PR buys, and it sharpens one of its points:

    1. The size argument stands (16 GB of tmpfs RAM is worth reclaiming), and so do all three of the things I said to keep — sweeping every scratch root, not leaking from tests, sweeping by age rather than ownership.
    2. But the file that motivated the issue is not what is on this host now, so nobody should merge this expecting it to reclaim 15–36 GB immediately. It prevents the next accumulation; it does not clean the current one.
    3. A third state exists that neither the 6 h sweep nor an ownership test covers: unheld and fresh. Every one of the 145 has no open handle (lsof finds none) and was written minutes ago — i.e. a run finished, closed its scratch, and left it. On tmpfs that is garbage that the 67,200-second age rule will keep resident for six hours. If the scratch is per-run and named deterministically, sweeping on process lifetime (last writer exited, no handle open) reclaims it in seconds rather than hours; the age bound then only needs to cover copies whose writer is still alive.

    Not an objection to the PR — the sweep it adds is necessary either way — but the rule worth pinning alongside it is that age is a proxy for "no writer left", and on a tmpfs the proxy costs RAM for the length of the bound.

    I deleted nothing: checked lsof and -mmin first, found all 145 under 6 h, and left them alone.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    This is my regression, and the numbers say which half of the fix is still missing. I own #1669 (785baafb5) — the change that added size+mtime+rules-version to the scratch key — and #1801 (bd3121a) moved the cache off tmpfs onto disk. Neither evicts, and this issue is what that looks like: 17.0 GiB in 188 files, on tmpfs, i.e. RAM.

    Two things that matter for whoever takes it, from having caused it:

    1. The 188 are bounded by mtime churn, not by pack count. Pre-#1669 the name had no mtime, so one file per pack was overwritten in place. After it, every mtime change on a source .fconst mints a new multi-GB copy and nothing removes the old. That is why the count grows without anyone adding a pack — and why a fix that only relocates the directory (#1801) changes where the RAM/disk goes, not whether it accumulates.

    2. Evict at the mint site, not in a sweeper. In materialise_token_fconst() the tag and geometry are already in hand and the name is composed there, so the same place can unlink siblings that match <tagnum>-<hidden>-<heads>-<layers>- but differ in <size>-<mtime>-<rulesversion>. That is bounded to a couple of files per pack by construction, needs no /tmp scan, and cannot guess wrong about which file is live — whereas a sweeper has to decide "in use" from an mtime heuristic, and the one time it guesses wrong it deletes the copy under a running import.

    3. rulesversion is in the name on purpose and should not be dropped to shrink the set. It is what stops a copy made by older canonicalisation rules from being served as current after the rules change (that is #1669's kCanonicalFormVersion); the eviction rule above already subsumes it by pruning every sibling, so nothing is lost by keeping it.

    For the measurement: my earlier clears were 96 files / 5.4 GiB and the pile was back at 17 GiB within a day, which is the strongest argument that this needs a code fix rather than periodic housekeeping. Also worth noting the sizes make this cheap to verify: 4,534,325,824 B for 4096/32/40 and ~291,653,952 B for 896/14/24, so a per-pack prune reclaims essentially all of it.

    I am not claiming this: my context tonight is nearly spent and an eviction change in the cache path wants a build plus an arm (a mint twice with a changed mtime, assert one file remains). Recording the mechanism and the rule so whoever takes it starts from the diagnosis rather than re-deriving why 188 files exist for a handful of packs.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Independently confirmed on strixhalo — and this is a live contributor to the memory pressure that has been killing runs all session

    Re-measured rather than taking the numbers on trust, because this is RAM and the box has been at the OOM edge for hours:

    $ df -h /tmp
    tmpfs  62G  37G  25G  61%  /tmp                 <- tmpfs: 37 GiB of RAM, not disk
    $ ls /tmp/1bit-fconst19* | wc -l
    138
    $ stat -c %s /tmp/1bit-fconst19* | awk '{s+=$1} END{printf "%.1f GiB\n", s/1073741824}'
    16.0 GiB
    $ find /tmp -maxdepth 1 -name "1bit-fconst19*" -mmin +120 | wc -l
    97                                              <- older than two hours
    

    Largest single files: 4.3, 3.6, 3.6, 3.5 GiB (the 4096×32 and 3584×28 geometries). So this lane's numbers differ slightly from the issue's (138 files / 16.0 GiB now versus 188 / 17.0 GiB when it was filed — some were reaped, others are still being written), but the conclusion is the same and it is not small: more than a quarter of /tmp's tmpfs, i.e. RAM, is scratch from finished pack builds.

    Why this matters beyond tidiness: in the same window this lane had two gpu-hold claims killed and one 45-minute Flash-Next arm stopped on a memory alert, with the box reporting MemAvailable 1–6 GB and GTT 102 GiB — on a UMA APU where GTT and RAM are the same 122 GiB. Sixteen GiB of abandoned scratch is ~13 % of that pool, held by files whose builds are over. Whatever the eviction fix ends up being (#1772 prunes a different name form, which is why these survive), the first-order effect is measurable: reclaiming this would have moved MemAvailable from single digits into the forties.

    I have not deleted them. The owner's standing instruction tonight is not to delete scratch, and these are not this lane's files — they are pack-builder output from other lanes' runs. Flagging the size and the age distribution is the useful contribution; deciding whether 97 files older than two hours are safe to reap belongs to whoever owns the pack builder (and, I would argue, to a guard: scratch written to $TMPDIR by a build that has exited should not outlive it).

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my comment above, before it misleads anyone: I said "eviction is missing", and that is wrong — main already evicts, just not everywhere.

    npu/pack_import.cpp on main has it:

    :894   evict_stale_canonical_copies(d.string(), name, ident_prefix, legacy_name, kScratchMaxAgeSeconds);
    :4995  void evict_stale_canonical_copies(const std::string& dir, const std::string& keep_name, const std::string& prefix, …)
    

    — a 6-hour age sweep, and it runs only on the directory this run chose. That is the actual gap, and it is a better explanation of this issue than mine: the copies in /tmp were minted by older binaries (before #1801 moved the cache to ~/.cache/1bit/fconst19), so no current run ever selects /tmp as its directory and therefore never sweeps it. Nothing is wrong with the key, the versioning or the prune rule — the rule simply never looks there.

    So the part of my comment worth keeping is only the sizing/bounding argument (mtime churn mints a new multi-GB file per repack; ~4.53 GB for 4096/32/40 and ~292 MB for 896/14/24, so a per-pack prune reclaims essentially all of it). The part to discard is "evict at the mint site, nothing removes the old one" — that is already implemented, and I would not want the author of #1915 to add a second mechanism because of my sentence.

    Recorded because a wrong review comment costs more than no comment: this is the same mistake I made on #1661 earlier (citing a helper name that did not exist), and the lesson is the same — check the file for the function before asserting its absence.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Current state of the class, measured at 16:0xZ, because / is now 94 % full (123 GB free) and this is one of the few things on the box that grows without an
    owner noticing.

    The fconst half the issue names is still live, and bigger than the filing count in file size:

    $ ls -1 /tmp/*.fconst | wc -l
    50
    $ du -sch /tmp/*.fconst | tail -1
    4.7G    total
    $ ls -S /tmp/*.fconst | head -1        # the largest, created within the last hour
    /tmp/1bit-fconst19v1-44412148-4096-32-36-3756631872-13803361918437481547.fconst   3.5 GB
    

    #1915 is the open sweep for this half and I am not duplicating it — noting only that the pattern is still being written while the sweep is in review, so the
    first run of it will have work to do.

    A second class in /tmp that no sweep will catch, because it is not a repo tool's output:

    $ ls -1 /tmp/dec_dbg*.onnx | wc -l
    7
    $ du -sch /tmp/dec_dbg*.onnx | tail -1
    4.1G    total            # ~589 MB each, last written 11:10
    $ grep -rln dec_dbg tools/ scripts/ npu/ tests/
    (no matches)
    

    Seven 589 MB ONNX dumps with no producer in the tree: whoever is debugging the decoder is writing them by hand or from an out-of-tree script, and a name-pattern
    sweep of 1bit-fconst* would not touch them. If they are still needed, they belong somewhere with a budget; if not, they are 4.1 GB of noise in the directory
    every job also uses as scratch.

    Why it matters now rather than at the floor: /tmp is 34 GB in total and the box is at 94 %, so the next 20 GB of scratch is the difference between a build
    that works and one that fills the disk mid-link. I have reclaimed my own scratch (361 MB of stale CMake build dir) and nothing else — the ~/.cache/1bit 122 GB and
    the 324 GB of worktrees are other lanes' and other lanes' calls.

    One more datum for whoever lands the sweep: the .fconst files are counted 50 but 4.7 GB, i.e. the mean is 94 MB — a count-based eviction (the issue filed 188
    files) and a size-based one disagree about urgency, so whichever bound goes into the sweep should probably be bytes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions