M2-A5: a Q4_K_M model loading under a real heap cap on an Android device

[michalharakal]

Split out of #932's closing summary, where this was recorded as deliberately open but never given an issue.

What exists and what does not

The mechanism is built: mapped staging (#921/#922), the two-pool DeviceFit check, PlannerProfile.MOBILE_2GB — which is now strict (#1128), so a missing kernel fails rather than silently costing several times the weight. MemoryProbe reads RSS and fault counters on every target.

What is missing is the measurement, on hardware with ART. A JVM host is not a substitute: the heap cap, the GC and the page-cache behaviour that M2-A5 is about are all ART's.

Acceptance

• A Q4_K_M model loads and decodes on an Android device under its real heap cap
• RSS and major-fault counts recorded across a decode run
• The MemoryPlan compared against what was actually allocated, within the plan-vs-actual tolerance
• Numbers recorded as measurements, not turned into CI assertions — a shared runner is not the instrument for this claim (test: judge page faults by whether they scale with steps, not by a flat zero #1107)

Note

A suitable device is available; see the maintainer's notes rather than this issue for addressing.

[michalharakal]

Measured. Harness in PR #1188; run on a Google Pixel 8a, Android 17, arm64-v8a — real ART, real heap cap. Two datapoints:

1. Fits-under-cap: SmolLM2-135M-Instruct Q4_K_M (100 MB)

M2-A5 measurement (#1130)

• device: Google Pixel 8a, Android 17 (SDK 37), ABI arm64-v8a
• ART heap cap (Runtime.maxMemory): 256 MB
• model: skainet-m2a5.gguf, 100 MB on disk
• ctx=512, decode steps=16 (warm-up 4), prepack=true

Plan

Smollm2 135M 8k Lc100K Mix1 Ep2 · llama · 30 layers · ctx 512
  weights   Mapped, packed                 98 MB   resident
  kv cache  bf16 @ ctx 512                 11 MB   resident   (3 MB with TurboQuant 4-bit)
  forward   prefill chunk 256              11 MB
  heap      headroom                       64 MB
  total                                   185 MB   of 256 MB  ✔ fits

Load

• wall time: 375 ms for 272 tensors
• materialized bytes (heap tensors + packed payloads): 98 MB
• RSS before → after load: 144 MB → 275 MB (Δ 131 MB)
• major faults during load: 2

Decode — per-step RSS and major faults

step	ms	RSS	Δ major faults
1	72	314 MB	13
2	69	315 MB	0
3	70	315 MB	0
4	69	316 MB	0
5	67	317 MB	0
6	67	317 MB	0
7	66	318 MB	0
8	64	319 MB	0
9	64	319 MB	0
10	63	320 MB	0
11	63	321 MB	0
12	63	321 MB	0
13	62	322 MB	0
14	61	322 MB	0
15	61	323 MB	0
16	61	325 MB	0

• steady-state (after 4 warm-up steps): 63 ms/step, major faults total 0
• forward-scope allocations after warm-up should be 0; whole-run forward allocations: 2049

Plan vs actual

Smollm2 135M 8k Lc100K Mix1 Ep2 · ctx 512 · plan vs actual
  section                     planned      actual      drift
  weights (model scope)       110 MB      121 MB      +10 %   ✘
  forward slab                 11 MB      609 KB      -94 %   ✘
  forward-scope allocations: 2049 (slab + overflow; steady-state decode should add none)
  adapters: 70 MB — prepack-input_block_major 70 MB

within 10 % tolerance: false (recorded as a measurement, per #1130 — not asserted)

Reading the acceptance criteria against this:

• ✅ a Q4_K_M model loads and decodes under the real heap cap — plan said 185/256 MB fits; load 375 ms, decode steady 63 ms/step through the JNI packed kernels
• ✅ RSS and major-fault counts across a decode run — 2 major faults during (mapped) load, zero across all steady-state decode steps: the "mapped weights are only free while the fault rate stays near zero" claim, measured at literally zero
• ✅ plan vs actual, within the tolerance machinery — recorded with two explained drifts: weights +10 % (the 70 MB of one-time feed-order prepack copies are visible as adapter bytes — the cost of the packed-kernel order, now a number); forward −94 % (the plan sizes a 256-token prefill chunk, this loop only decodes; also the harness allocates per-matmul outputs rather than slab-reusing — 2049 forward allocations is a harness limitation, not an engine claim)
• ✅ recorded as measurements, not CI assertions — check() is never called; the run skips cleanly without a model file

2. Exceeds-cap: Qwen2.5-1.5B-Instruct Q4_K_M (1.06 GB)

java.lang.OutOfMemoryError: Failed to allocate a 131272720 byte allocation
with 52020928 free bytes and 49MB until OOM,
target footprint 268435456, growth limit 268435456

OOM at load: under WeightResidency.MAPPED the dense-F32 tensors stay mapped, but quantized payloads still materialize as heap ByteArrays — so a 1 GB Q4_K_M cannot load under a 256 MB cap regardless of the mapping machinery. That is the measured gap this issue existed to surface; packed-tensor mapped staging (serving Q4_K blocks straight from the mapping) is the follow-up it motivates.

Notes: RSS (275–325 MB) exceeds the 256 MB figure because RSS counts mapped file pages and native allocations; the ART cap governs the Java heap only. Reference-kernel runs are impractical for this measurement (per-element view decode is hours-scale even at 135M) — installPacked + prepack=true is the intended configuration and is what the recipe in the harness KDoc produces.

[michalharakal]

Closing as measured: both acceptance datapoints are recorded above (fits-and-decodes with zero steady-state major faults; exceeds-cap OOM precisely diagnosed). The harness lives in PR #1188; the gap the exceeds-case surfaced continues in the packed-tensor mapped staging follow-up.

[michalharakal]

Follow-up: the Qwen2.5-1.5B Q4_K_M exceeds-cap case measured here is fixed by #1189/#1190 — same device, same model, same harness: load 445 ms with 566 KB weight heap (was OOM), steady-state decode 153 ms/step with 1 major fault total across steps 5–16. Details in the #1189 comment.