feat(gemma3n): StableHLO export harness (PLE as graph input) + gemma3n/hybrid-AI docs - #394
Merged
Merged
Conversation
…d vs llama.cpp (#377) Gemma 3n gets a proper DSL definition and stops depending on the hand-rolled Gemma3nRuntime for GGUF inference. The recon showed that runtime was never faithful to real checkpoints: it loaded the PLE tensors but never applied them, had no Laurel block, ignored the AltUp modality router (host-side coefficient math), and its E2B_DEFAULT claimed AltUp/sparsity are E4B-only — the real gemma-3n-E2B GGUF has altup.num_inputs=4 and first-10-layer activation sparsity. Gate was 0/5 for a reason. The new lane, faithful to HF modeling_gemma3n.py (read from source): - Gemma3nAltUpBlock: predict/correct with the tanh modality router (router_norm * 1/hidden -> modality_router -> tanh), per-layer coefficient linears, correct_output_scale. - Gemma3nAltUpGlobals: stream init/merge over the GGUF's 3D altup_proj/altup_unembd_proj stacks with per-token magnitude renormalization (the ggml ne-order vs row-major reinterpret is handled by reshaping to [numExtra, H, H] before slicing). - Gemma3nLaurelBlock; Gemma3nSparseGeGluFFN (in-graph Gaussian-top-k, population std, per-layer std multipliers from the GGUF, -inf = off); Gemma3nPerLayerApply (PLE delta added to the NON-active streams, reusing gemma's PerLayerEmbedding whose math is identical to HF). - gemma3nNetwork() + Gemma3nModel: GemmaModel-pattern orchestrator threading 4 AltUp streams per layer (predict -> attn+Laurel with 1/sqrt2 gating -> sparse FFN -> correct -> PLE), per-type shared KV for the last 10 layers via OwnerReadOnlyKVCache, hybrid sliding/global attention with dual RoPE bases, qk-norm + parameterless v-norm, attention scale 1.0. Everything through ctx.ops — traceable for the StableHLO/IREE mobile path (next PR). - Metadata/parser now read the real llama.cpp GGUF keys: sliding_window_pattern booleans, per-layer activation_sparsity_scale (1.6449/-inf encoding), rope.freq_base fallback (1M global, 10k SWA default), rms_norm_eps, per-layer feed_forward_length. - Gemma3nNetworkLoader: engine loading stays packed/MAPPED (PLE table row-dequants on demand), strict binding both directions. - Routing: kgemma GEMMA3N/GGUF and skainet-cli gemma3n now run the DSL lane; SafeTensors stays on the legacy runtime until the DSL grows that leg. Gate (the last ungated generative family): Gemma3nGoldenTokenParityTest asserts full 32-step greedy text equality vs mainline llama.cpp b10621 on gemma-3n-E2B-it-Q4_K_M — GREEN. Fixture prompt chosen for greedy decisiveness (min top-1/top-2 margin 1.77 nats; "The capital of France is" hits a measured 0.12-nat tie at step 3 that Q4_K noise flips). Wired into smoke-reference (gemma3n_gguf_url + 20g heap arg); smoke-models.json gains a Gemma3n-E2B row. Both CLIs decode " 2, 3, 5, 7, ..." end-to-end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n/hybrid-AI docs The mobile leg of the #377 DSL arc. Gemma3nExportHarness + exportGemma3n (SmolLM2/FunctionGemma redecode pattern): trace gemma3nNetwork() -> StableHLO, bf16 external params, in-graph argMax tail, manifest. The mobile-honest design decision: per_layer_inputs is a graph INPUT ([1, seq, L, pleDim], computed on the CPU from the packed PLE table at runtime) instead of baking the 262k x 7680 table into the accelerator parameters. That is PLE's design point per Google's Gemma 3n guide - per-layer embeddings live off-accelerator - and it keeps the archive trunk-only. PerLayerEmbedding gained a traceable indexSelect path while ctx.isRecording (the eager packed row-dequant gather is host-side and would bake constants); Gemma3nModel.externalPerLayerInputs is the injection point; AltUp's coefficient narrows are rank-general so the trace's [1, seq, H] working rank flows through. Full E2B emission is BLOCKED on engine SKaiNET#1247, precisely characterized there: (a) VoidTensorOps materializes real zero buffers per traced op and constant extraction duplicates every weight - dense originals + zeros + copies OOM a 46 GB heap on a 48 GB host; (b) the HLO converter drops the embedding-gather's weight operand on this trace shape and cascades to a compute-free module while exiting 0. The harness now hard-fails on both: a signature arity check (weights leaked as function args = the packed-tensor mode) and an emission-completeness check (converter failure comments / empty return). GEMMA3N_LAYERS=N truncates the trunk (metadata + tensor filter, PLE table coverage kept full-width) so the whole pipeline is verifiable on smaller hosts the moment the engine lands fixes. Docs: explanation/gemma3n.adoc - why Gemma 3n's mobile-first architecture (MatFormer, PLE, KV sharing per Google's developer guide) and why SKaiNET fits it (one DSL definition serving eager + compiled, packed/MAPPED loading, per-layer heterogeneity, parity-gated faithfulness); wired into nav. README + index verified-matrix rows flip gemma3n to verified. docs/specs/matformer-hybrid-on-device-ai.md - pre-PRD design note: MatFormer slice-view design for SKaiNET and the hybrid on-device/cloud architecture (draft-first, escalate-on-evidence, L0-L3 routing layers, stream-buffer seamlessness), with paper and reference-implementation links. Eager lane unaffected: parity gate re-verified green after the changes; gemma3n/gemma suites + apiCheck green (dumps regenerated). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #393 (the gemma3n DSL lane). The mobile leg of #377, plus the docs.
Export harness —
exportGemma3nSmolLM2/FunctionGemma redecode pattern: trace
gemma3nNetwork()→ StableHLO, bf16 external params, in-graph argMax tail,manifest.json. The mobile-honest design decision, per the "why FP32 on Android?" review comment:per_layer_inputs [1, seq, L, 256]is the graph's second input, computed on the CPU from the packed PLE table at runtime — exactly PLE's design point in Google's Gemma 3n guide (per-layer embeddings live off-accelerator). The archive carries trunk + token embedding only; params emit bf16 (FP32 exists only as a transient host-side form during export; int8 is a mechanical port of FunctionGemma'srewriteGlobalsToInt8, tracked).PerLayerEmbeddinggains a traceableindexSelectpath underctx.isRecording;Gemma3nModel.externalPerLayerInputsis the injection point; AltUp narrows are rank-general for the trace's[1, seq, H]working rank.GEMMA3N_LAYERS=Ntruncates the trunk for end-to-end pipeline verification on smaller hosts (PLE coverage stays full-width so tensors bind shape-exact).Blocked upstream — honestly
Full E2B emission hits two engine defects, filed + precisely characterized in SKaiNET#1247:
VoidTensorOpsmaterializes real zero buffers per traced op and constant extraction duplicates every weight — dense originals + retained zeros + graph copies OOM a 46 GB heap on a 48 GB host.The harness turns both traps into hard errors: a function-signature arity check (weights leaking as args = the packed-tensor mode) and an emission-completeness check (converter failure comments / missing outputs). The moment the engine lands fixes,
exportGemma3nproduces the servable artifact with no further transformers changes expected.Docs
explanation/gemma3n.adoc— why Gemma 3n (MatFormer, PLE, KV sharing, per Google's developer guide) and why SKaiNET fits it: one DSL definition serving eager + compiled, packed/MAPPED loading, per-layer heterogeneity as a plain Kotlin loop, parity-gated faithfulness. In the nav.docs/specs/matformer-hybrid-on-device-ai.md— pre-PRD design note: MatFormer slice-view design for SKaiNET (prefix weight views, slice configs, per-slice vmfbs over one parameter archive) and the hybrid on-device/cloud architecture (draft-first + escalate-on-evidence, L0–L3 routing layers, stream-buffer seamlessness), with papers and reference implementations.dsl-vs-handcodedupdated.Verification
🤖 Generated with Claude Code