feat(#1191/#1192/#1193): every GGML block format mapped-servable, on Android and the JVM — and no silent fallbacks - #1197
Merged
Conversation
…droid and JVM; fallbacks visible The 0.50.0 hardening slice. SmolLM2-135M (mixed-quant: k-quant fallback made most of it Q8_0) under MAPPED without prepack went from 48,771 ms/step — silent reference fallback — to 33 ms/step, the fastest configuration measured, with the whole model mapped (98 MB off-heap, 137 KB weight heap). - C: row-major variants for Q8_0/Q4_0/Q5_0/Q5_1/Q5_K, refactored like Q4_K/Q6_K (shared block-term, threaded via skainet_row_threads, #1195). Cross-order results are numerically equivalent, not bit-exact: -ffast-math may contract float accumulation differently per loop shape (q5_1 measured 2 ULP); integer-dot formats happen to match exactly. Docs and the parity suite say so (tolerant cross-order asserts; threaded-vs-solo stays exact). - JNI (#1193): direct-buffer entries for the new formats AND heap-array _rm entries for all seven — JniRowMajorMatmulKernel serves BLOCKED_ROW_MAJOR from mapped, direct, or heap storage with the same C. The 48 s trap was a heap canonical weight with no kernel on its key. - FFM (#1191): FfmRowMajorKernelPack — the same _rm symbols on the JVM, MemorySegment.ofBuffer zero-copy for mapped/direct weights, arena-staged for heap. End-to-end dispatch test proves selection via the trace. - Visibility (#1193): ViewKernel gains a sink-aware run overload; every packed-bridge fallback to the decoding reference emits a KernelRun trace event naming the kernel and reason. The M2-A5 harness reports the count — both device runs now print "reference fallbacks: 0". - Loader/planner: all seven formats in MAPPED_SERVABLE_DEFAULT, the loader's mapped gate, JvmMappedFile, and BufferPackedTensorData; the planner pin for "unmappable stays heap-charged" moves from Q8_0 to TQ2_0 (ternary repack cache is the remaining #1192 follow-up). - Kernel matrix: mapped-serving table now shows all seven formats on both Android (native-jni-direct) and JVM (ffm-rowmajor). Verified on the Pixel 8a (cooled runs): SmolLM2 mapped no-prepack 33 ms/step, 0 faults, 0 fallbacks; Qwen2.5-1.5B mapped 66 ms/step, ✔ fits, 0 fallbacks. JVM: 149 native-cpu tests + FFM dispatch end-to-end green; apiCheck green. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
📖 Documentation Preview The documentation has been built successfully for this PR. Generated Files:
Artifacts:
This comment will be updated automatically when the PR is updated. |
…ization drifts past 1e-5 q80_rm_parity_threaded failed on windows-x86_64: the same shared block-term inlined into two loop shapes auto-vectorizes with different partial-sum splits under MSVC, exceeding 1e-5 relative. 1e-4 remains six orders below any addressing bug (which produces O(1) garbage); threaded-vs-solo comparisons stay bit-exact. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…thful plan, third-party notice The release entry for the Android memory arc, including the explicit Third-party section for the vendored NeoGPU kernel (#1166's last in-repo deliverable — 0.49.0's entry mentioned it inline; this makes the attribution a named section with upstream commit and license). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
📖 Documentation Preview The documentation has been built successfully for this PR. Generated Files:
Artifacts:
This comment will be updated automatically when the PR is updated. |
1 similar comment
|
📖 Documentation Preview The documentation has been built successfully for this PR. Generated Files:
Artifacts:
This comment will be updated automatically when the PR is updated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1191. Closes #1192 (GGML block formats — the ternary repack-cache is re-scoped below). Substantially delivers #1193's points 2–3.
The 0.50.0 hardening slice, motivated by a measured trap: SmolLM2-135M is mostly Q8_0 (its hidden size 576 is not a 256-multiple, so llama.cpp's k-quant fallback kicked in at file creation), and under
MAPPEDwithout prepack those tensors fell to the silent decoding reference at 48,771 ms/step.After this PR (Pixel 8a, cooled runs)
What's in it
_rm(row-major/file-order) variants for Q8_0, Q4_0, Q5_0, Q5_1, Q5_K — same shared-block-term refactor as Q4_K/Q6_K, all threaded via the Thread the packed matmul kernels — ~3× decode on Android, the biggest lever under the bandwidth ceiling #1195 pool. Honest numerics note: cross-order results are numerically equivalent, not bit-exact —-ffast-mathmay contract float accumulation differently per loop shape (q5_1 measured 2 ULP apart; the integer-dot formats happen to match exactly). The parity suite asserts tolerantly cross-order and stays bit-exact for threaded-vs-solo.JniRowMajorMatmulKernelservesBLOCKED_ROW_MAJORweights from mapped, direct, or heap storage — heap-array_rmentries are the one-registration fix that turns the 48 s case into a fast path (mixed-quant models are the common case, not the corner).FfmRowMajorKernelPack— the same_rmsymbols on the JVM,MemorySegment.ofBufferzero-copy for mapped/direct weights. End-to-end dispatch test pins kernel selection via the trace.ViewKernelgains a sink-awarerunoverload; every packed-bridge punt to the decoding reference emits aKernelRuntrace event naming kernel and reason. The M2-A5 harness prints the count.MAPPED_SERVABLE_DEFAULT, loader mapped gate,BufferPackedTensorDataarms; the planner's "unmappable stays heap-charged" pin moves to TQ2_0; the kernel matrix's mapped table now shows all seven on both Android and JVM.Re-scoped follow-up
The single remaining #1192 item — BITNET_B1_58 via a repack-cache sidecar (a repacked copy cannot page from the file it no longer matches) — needs its own design (cache location/invalidation) and moves to a fresh issue rather than holding this release.
JVM: 149 native-cpu tests, FFM dispatch end-to-end, io-gguf + lang-core + backend suites,
jvmApiCheck— all green.🤖 Generated with Claude Code