0.1.29: MiniMax-H3 bf16 refused on a 64 GB Mac (Metal budget exhausted by mapped weights); unaligned shards fall back to full anonymous copies (+13 GB swap); encoder loads fully resident

[skymars13]

Machine: MacBook Pro M5 Max, 64 GB unified memory, macOS 26.6. vpipe built from tag v0.1.29 (7703614). Checkpoints: official MiniMax-H3 FL2VA bf16, plus an 8-bit pack produced by the in-tree model-quantize (group 64, quant_modulation: true) under v0.1.23. Turbo LoRA applied at run time. All runs with VPIPE_H3_ENC_PIN_FRAC=0.

I upgraded from v0.1.26 (where this machine ran both bf16 and 8-bit cleanly — bf16 544×960, 4 NFE finished in 3:51 with zero swap growth) and hit four related memory problems in the new weight-residency path. Curves below were sampled every 2 s (RSS / free / compressed / swap delta vs run start).

---

A. bf16 cannot run at all: every forward is refused with headroom 0

generate-video, official bf16 FL2VA, 640×480×362 @ 4 NFE (but the geometry does not matter):

[INFO] MiniMax-H3 DiT 0 of 50 blocks pinned at load (none fit beside the other models; the resident set grows into free RAM as the denoise runs)
[ERROR] not enough memory for a 34121-row forward -- it needs ~6960 MB of scratch and there is ~0 MB of GPU working set / ~57896 MB reclaimable. Refusing rather than thrashing ...

Observations:

• Streaming is already active (0 of 50 pinned), so this is not residency growth: the refusal fires at load, before the first step.
• mb.fits_physical passes (~57.9 GB available), mb.fits fails with headroom ~0.
• The message has no after parking suffix, i.e. reclaim_at_least() parked 0 bytes while reporting ~57 GB reclaimable (finding D).
• Reserve value is irrelevant — identical refusal with 8 GB and 22 GB reserves.

My reading: once the shards are 16-byte aligned (see B), the checkpoint is zero-copy mapped into Metal buffers up front, and 66 GB of mapped buffers alone exceeds recommendedMaxWorkingSetSize (~48 GB on this box), so mb.headroom is 0 before a single block is made resident. The 8-bit pack (33 GB) stays under the line and runs perfectly — which matches: same pipeline, same day, 8-bit completed a full 362-frame FL2V run with swap peak +22 MB.

So on v0.1.29 a 64 GB machine cannot run the bf16 checkpoint in any geometry, where v0.1.26 streamed it fine.

B. Shards whose data section is 8 mod 16 fall back to FULL anonymous copies → +13 GB swap in seconds

The official MiniMax-H3 shards (and the 8-bit packs written by pre-v0.1.29 model-quantize) have safetensors data sections starting at 8 mod 16, e.g.:

model-00001-of-00013.safetensors  data @5672 (8 mod 16)

v0.1.29 then logs, per shard:

[WARN] weights: shard 0 is NOT zero-copy mappable -- its data section starts at 15990 (6 mod 16 ...) so EVERY tensor in it is COPIED into anonymous memory instead.

On the 8-bit pack this copied ~33 GB of weights into dirty anonymous memory at load: swap went +13.1 GB in 8 seconds (t=16→24 s of the run), compressor to 9 GB, on a box with 64 GB. Since the official checkpoints are 8-aligned, I believe every v0.1.29 user hits this path today.

Local fix that worked: pad each shard's JSON header with trailing spaces so the data section lands on 16 — after that the warnings disappear. Suggests two upstream fixes: (1) model-quantize should write 16-aligned shards; (2) an unaligned shard should degrade to bounded streaming reads, not a whole-shard anonymous copy.

C. The prompt encoder loads fully resident by default → +12.5 GB swap at prefill

With aligned shards, the load-phase spike moved to the moment diffusion-conditioner started encoding: +12.5 GB swap in ~6 s. Forcing VPIPE_H3_ENC_STREAM=1 dropped the peak to +2.9 GB (and with an 8 GB residency reserve, to +22 MB total for a full 15 s FL2V run). It looks like the conditioner's streaming plan decides the 51 GB Qwen3-VL-32B encoder "fits" without accounting for the dirty working memory already present (video VAE weight copies for the two FL2V keyframes among it), so 51 GB + ~20 GB dirty > 64 GB and the box swaps. Defaulting to streaming whenever encoder + measured dirty memory approaches RAM would avoid it.

D. reclaim_at_least() parks 0 bytes while ~57 GB is reported reclaimable

See the log in A: the refusal path's self-rescue never reclaims anything in this configuration, so even cases that could be saved by parking are refused.

---

Repro

1. A: 64 GB Mac, official bf16 FL2VA (16-aligned shards, or pad them as above), any generate-video geometry on v0.1.29 → refused with headroom 0.
2. B: any pre-v0.1.29 8-bit pack (or the official shards, unpadded) → per-shard NOT zero-copy mappable warnings and a multi-GB swap spike at load.
3. C: aligned 8-bit pack, default env → watch swap during prompt encoding; repeat with VPIPE_H3_ENC_STREAM=1.

What works today on this machine (for other 64 GB users)

8-bit pack + 16-aligned shards + VPIPE_H3_ENC_STREAM=1 + a small residency reserve: full 416×736×362 first-and-last-frame run, 8 NFE, 90–98 s/step, swap peak +22 MB. Happy to run any diagnostic build or dump you need — same machine as #4/#5.

[tgo-app-dev]

Hi skymars13,

Thanks a lot for reporting. In the previous version I was fighting the OS memory management which was quite messy. I just released v0.1.30 with a different memory management subsystem. Please give it a try.

I tried raw bf16 model (Comfy-Org/MiniMax-H3-FL2VA) on a M4 Pro 64 GB with the v0.1.30 release and that works. For 4-8 steps adaLN bake-in will remove a big amount of runtime weight so the full bf16 inference is well under a 64 GB box's capacity.

[skymars13]

Verified on v0.1.30 — same machine (M5 Max, 64 GB), same checkpoints, VPIPE_H3_ENC_PIN_FRAC=0, VPIPE_H3_ENC_STREAM=1, an 8 GB residency reserve, memory sampled every 2 s as before.

bf16 runs again, with zero swap. Official bf16 FL2VA, 544×960×124 @ 4 NFE:

version | outcome | total | swap delta over the run -- | -- | -- | -- v0.1.26 | ran (streamed) | 3:51 | ~0 v0.1.29 | refused at preflight, headroom 0 | — | — v0.1.30 | ran, 50/50 blocks resident inside the wired pool | ~3:20 | 0.00 MB

Step times 35.5–42 s; compressor peaked at 0.5 GB; the log shows residency probe 50 blocks of 50 (735 MB each ...) -- uncapped, the wire budget is the gate and DiT residency ended at 50 of 50 blocks (36751 MB). The pre-denoise stall is gone too — first forward at 19316 rows: 3488 ms before block 0 (gemm autotune 3094) where the old tuner cost about a minute of silence. Findings A–D from the report above all check out as addressed.

One small thing worth a look: with everything at its streaming floor the planner still warns

resource-plan: this graph needs at least 64201 MB resident at its peak -- phase 'condition' ... It does not fit this machine at any setting -- use a smaller model, a smaller geometry, or a quantized checkpoint.

— and then the run completes with 0 MB of swap and ~10 GB idle at the end. Looks like the 'condition' phase peak still counts the encoder and the DiT together at more than the run actually holds, so the warning tells a 64 GB operator to give up on a graph that fits comfortably. Cosmetic, but it points at the wrong repair.

Thanks for the fast turnaround — a day from report to a rework this deep is remarkable.

[tgo-app-dev]

Thanks for verifying! The resource-plan warning message is always printed in this commit. If it really doesn't fit, it'll be terminated by an error message. The bogus warning will be fixed in the next release.