Skip to content

feat(io): NameMap — GGUF/HF tensor names ↔ TensorId for Llama, Qwen2, Gemma-3; unmapped reported (SKEEP-003 P1, S0.6) - #1055

Merged
michalharakal merged 1 commit into
developfrom
feature/1011-gguf-namemap
Aug 23, 2026
Merged

michalharakal merged 1 commit into
developfrom
feature/1011-gguf-namemap

Conversation

@michalharakal

Copy link
Copy Markdown
Contributor

Summary

SKEEP-003 slice S0.6 (milestone M0 #1001, PRD M0-F4 / M0-A2): NameMap — checkpoint tensor names ↔ TensorId, bidirectional, for the three reference families, with unmapped names reported, never dropped.

  • io-core sk.ainet.io.weights.NameMap: toTensorId(name), toCheckpointName(id), unmapped(names), toTensorIds(names), asWeightNameResolver() (adapter to the legacy (modulePath, paramName) resolver). RoleTableNameMap: role tables for top-level and per-layer tensors with a {N} layer-prefix pattern. TransformerNameMaps.Gguf (llama, qwen2, gemma3, forArchitecture(general.architecture)) and .Hf (llama, qwen2, gemma3).
  • Canonical ids are family-neutral and GGUF-role based: model.layers[N].attn.{q,k,v,o}_proj.{weight,bias}, model.layers[N].attn.{q,k}_norm.weight, model.layers[N].mlp.{gate,up,down}_proj.weight, model.layers[N].{attn_norm,ffn_norm,post_attention_norm,post_ffw_norm}.weight, model.{embed_tokens,norm,lm_head,rope_freqs}.weight. HF norms are mapped per family (Gemma-3's post_attention_layernorm is a true post-attention norm; Llama/Qwen2's is the pre-FFN ffn_norm).
  • io-gguf: StreamingGGUFReader.nameMap() (from general.architecture), tensorIds(map), NameMap.asTensorNameMapper() (adapter to the legacy role interface).
  • Tests: NameMapTest — the full tensor lists of Llama-3.2-1B (16 layers, rope_freqs), Qwen2.5-0.5B (24 layers, q/k/v biases), Gemma-3-1B (26 layers, q/k norms, four norms) map with zero unmapped and round-trip; family-neutral ids; HF norm mapping; unmapped reporting; resolver adapter. GgufNameMapFixtureTest — the same on real files, fixture-gated (-Dskainet.test.fixturesDir; the Qwen2.5-0.5B Q8_0 GGUF from downloadQwenTokenizerFixtures is picked up automatically; gated Llama/Gemma files run when dropped in).
  • No BCV modules touched (io modules have no dumps).

Stacked on #1054 (S0.5 — TensorId); retarget to develop after it merges.

Test plan

Full local gate (scripts/pr-gate.sh, JDK 25); results in the first comment. Targeted: io-core sk.ainet.io.weights.* 7/7, io-gguf fixture test 3/3 (skips when files absent).

Closes #1011

🤖 Generated with Claude Code

@michalharakal

Copy link
Copy Markdown
Contributor Author

Local gate scripts/pr-gate.sh (JDK 25) on a9d4821 (stacked on #1054):

=== pr-gate: JVM tests ===                                      BUILD SUCCESSFUL (io modules re-run; rest up-to-date)
=== pr-gate: apiCheck ===                                       BUILD SUCCESSFUL
=== pr-gate: verifyNpmPins jsTest wasmJsTest wasmWasiTest ===   BUILD SUCCESSFUL in 1m 35s
=== pr-gate: linuxX64Test ===                                   BUILD SUCCESSFUL in 42s
=== pr-gate: assemble ===                                       BUILD SUCCESSFUL
=== pr-gate: :skainet-test:skainet-test-java:test ===           BUILD SUCCESSFUL
pr-gate: all legs passed.

Targeted: io-core NameMapTest 7/7, io-gguf GgufNameMapFixtureTest 3/3 (skipping when fixture files are absent).

… Gemma-3; unmapped names reported (SKEEP-003 P1)

Milestone M0 (#1001), PRD M0-F4 / M0-A2 (all tensors of Llama-3.2-1B,
Qwen2.5-0.5B, Gemma-3-1B map to TensorIds, zero unmapped).

- io-core sk.ainet.io.weights.NameMap: bidirectional checkpoint name <->
  TensorId, unmapped(names) (never dropped), toTensorIds, and
  asWeightNameResolver() adapter to the legacy (modulePath, paramName)
  resolver. RoleTableNameMap: role tables for top-level and per-layer
  tensors with a {N} layer-prefix pattern; TransformerNameMaps.Gguf
  (llama, qwen2, gemma3; forArchitecture) and .Hf (llama, qwen2,
  gemma3). Canonical ids are family-neutral, GGUF-role based
  (model.layers[N].attn.q_proj.weight, model.layers[N].post_ffw_norm.weight,
  model.embed_tokens.weight, model.lm_head.weight, model.rope_freqs.weight);
  HF norms are mapped per family (Gemma-3's post_attention_layernorm is a
  true post-attention norm, Llama/Qwen2's is the pre-FFN ffn_norm).
- io-gguf: StreamingGGUFReader.nameMap() (from general.architecture),
  tensorIds(map), NameMap.asTensorNameMapper() adapter.
- Tests: NameMapTest (full synthetic tensor lists of the three reference
  GGUFs incl. Qwen2 q/k/v biases and Gemma-3's four norms + q/k norms,
  round trips, family-neutral ids, HF norm mapping, unmapped reporting,
  resolver adapter); GgufNameMapFixtureTest (fixture-gated real files,
  -Dskainet.test.fixturesDir, [skip] when absent).

Closes #1011

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@michalharakal
michalharakal force-pushed the feature/1010-tensor-id branch from e821ffb to 6ca4ed9 Compare August 22, 2026 21:08
@michalharakal
michalharakal force-pushed the feature/1011-gguf-namemap branch from a9d4821 to fae03eb Compare August 22, 2026 21:08
@michalharakal

Copy link
Copy Markdown
Contributor Author

Rebased onto the updated base after #1053 merged into feature/1008-format-and-encoding (lang-core API dump regenerated; no source change). The chain is now: #1051 (Format + AllocationSpec) ← #1054 (TensorId) ← #1055 (NameMap) ← #1056 (MemoryPlan) ← #1057 (skainet-plan CLI); merging each into its base keeps the next one a single-commit diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[S0.6] P1: GGUF NameMap — GGUF/HF names ↔ TensorId for Llama-3, Qwen2.5, Gemma-3; unmapped names listed

1 participant