Product goal: mainstream families run correctly — Qwen, Llama-like, Mistral/Mixtral, Kimi, OpenAI open-weight, and Gemma (the open Gemini-class weights). A user can uaii pull then uaii generate on those checkpoints and get correct tokens, or a precise error that names the missing capability. Silent wrong output is a bug.
This is not “every *ForCausalLM on the Hub”. Unknown families fail closed. Once the families in §1.1 are green, a new Llama-like or Qwen-like card should work without a new runtime; a new paper’s attention still needs a new family handler.
Related: huggingface_support.md · gguf_support.md · architecture.md
This is an implementation contract, not a marketing page. Every item cites the current code path and the change that closes it.
P0 (this document §P0.1–P0.7) is implemented: BF16 ingest, Llama-3/YaRN/linear/dynamic RoPE scaling, byte-level tokenizer.json, sharded uaii pull, uaii inspect / generate --dry-run, Kimi via GGUF MLA, gpt-oss fail-closed (MXFP4 / packing).
P1 family math that used to load wrong is now wired: Gemma RMSNorm (1+w), Qwen3 QK-norm, GPT-2 wpe, sliding window, Gemma2 extra norms + logit cap, Phi partial rotary + longrope factors, unknown hidden_act fail-closed, Qwen2-MoE shared expert. uaii inspect status: ok is no longer a silent-wrong lie for those families. HF MLA and gpt-oss packing stay fail-closed.
A family is supported only when all of the following hold for its dense F16/BF16 Hub folder and its GGUF blk.* (when llama.cpp publishes one):
- Ingest.
config.json+*.safetensors(single file or shards) load into UAII IR. - Weights. Every tensor used by the graph dequantizes to f32 (or stays packed on a proven GEMM path).
- Math. Attention, RoPE, norms, MLP / MoE, and the LM head match the Hugging Face reference for that architecture.
- Tokenizer. Encode / decode match the Hub tokenizer for the same string, including special tokens.
- Chat. Instruct/chat cards use the shipped
chat_template(or a documented family equivalent). - Fail closed. Anything outside this contract returns
NotImplemented/InvalidArgumentwith the tensor name, dtype, or config key that blocked it.
Recognition is model_type / architectures[0] first, then tensor-name binding inside that family. Guessing structure from names alone is how Qwen3, Gemma2, Kimi, and gpt-oss silently go wrong.
Map brand → what actually exists on disk. API-only models are not Hub checkpoints and will never load here.
| Brand (user language) | What we ship | HF model_type / class (examples) |
GGUF arch | Today | Must close to call it supported |
|---|---|---|---|---|---|
| Llama-like | Llama 2/3/3.1/3.2, CodeLlama, TinyLlama, SmolLM, Olmo, Granite (dense) | llama, olmo, olmo2, granite · LlamaForCausalLM |
llama, llama3 |
Layout + BF16 ingest + Llama-3/YaRN/linear/dynamic RoPE + byte-level BPE | Hub greedy golden vs transformers (nightly) |
| Qwen | Qwen2, Qwen2.5, Qwen3 dense; Qwen2-MoE / Qwen3-MoE | qwen2, qwen2_moe, qwen3, qwen3_moe · Qwen2ForCausalLM, Qwen3ForCausalLM |
qwen2, qwen3, qwen2moe |
Dense + QKV bias + BF16 + RoPE scaling + Qwen3 QK-norm + shared expert | Hub greedy golden |
| Mistral | Mistral 7B / Nemo / Small (dense), Mixtral 8x7B / 8x22B | mistral, mixtral · MistralForCausalLM, MixtralForCausalLM |
mistral, mixtral |
Dense + Mixtral experts + BF16 + sliding window | Hub greedy golden |
| Kimi | Moonshot Kimi K2 (and K2-style) open weights | usually deepseek_v3 / Kimi class · MLA + MoE, not Llama |
deepseek2 / deepseekv3 when converted |
HF MLA rejected. GGUF MLA expand path + tiny generate | llama.cpp greedy; absorbed MLA KV; P1.8 HF MLA |
| OpenAI | Open-weight only: GPT-2; gpt-oss-20B / 120B. Not ChatGPT / GPT-4 / o-series APIs (no weights) | gpt2 · GPT2LMHeadModel; gpt_oss · GptOssForCausalLM |
gpt2 (often non-blk); gpt-oss when llama.cpp names it |
GPT-2 layout + wpe. gpt-oss fail-closed (MXFP4 / packing), not silent Llama |
P0.7 gpt-oss family handler |
| Gemini-class | Google Gemma / Gemma 2 / Gemma 3 (the open models). Gemini Pro/Flash APIs have no Hub checkpoint | gemma, gemma2, gemma3 · GemmaForCausalLM, Gemma2ForCausalLM |
gemma, gemma2, gemma3 |
RMSNorm (1+w), Gemma2 extra residual norms, sliding window, logit cap. GGUF Gemma must not add +1 again (llama.cpp already baked it in) |
Hub greedy golden; Unigram JSON still needs SentencePiece |
Vision/audio siblings stay denied: LLaVA, Pixtral, Qwen2-VL, Gemma multimodal towers, Kimi-VL.
No. After BF16 + tokenizer + Llama-3 RoPE:
- Llama-like dense and Mistral dense / Mixtral — expected to generate (short context; sliding window applied when advertised).
- Qwen2 / Qwen2.5 dense — expected to generate; Qwen3 QK-norm is wired.
- Gemma / Gemma2 — RMSNorm
(1+w), extra Gemma2 residuals, sliding window, logit cap. - GPT-2 —
wpeis added. - Kimi — will not load from HF until MLA is wired; use GGUF meanwhile (P0.6).
- OpenAI gpt-oss — will not load as GPT-2; needs the family handler (P0.7).
- Gemini API — never. Point users at Gemma.
That is why the family table is the done list, not “any decoder”.
Mamba, RWKV, Jamba, vision-language towers, and encoder-decoder models stay on the deny list.
The loader does not allowlist class names. It accepts *ForCausalLM / *LMHeadModel, then guesses the block from tensor names (pick_layout in libs/uaii-loaders/src/hf.cpp). That is the right design. The failures are downstream of it: dtypes, missing operators, incomplete tokenizer, and incomplete Hub download.
Four independent gates. A model must pass all four.
| Gate | Code | Passes when | Typical Hub miss |
|---|---|---|---|
| Architecture | hf_architecture_supported + arch_denied |
suffix CausalLM / LMHeadModel, not on deny list, not is_encoder_decoder |
LLaVA, Whisper, T5, Mamba, BERT |
| Layout | pick_layout |
known QKV + MLP / MoE tensor names | DeepSeek MLA, novel packing, GPTQ qweight |
| Weights | safetensors_load_tensor_f32 |
tensor dtype is F32, F16, or BF16 (I8/U8 promoted) | F8, GPTQ qweight, MXFP4 |
| Tokenizer | load_hf_tokenizer |
vocab.json+merges.txt, tokenizer.model (if SentencePiece built), or a BPE tokenizer.json |
byte-level GPT-2 BPE without byte_fallback, Unigram JSON, added tokens |
The website and huggingface_support.md previously described gate 1 as “any decoder folder”. That overclaim is what this document exists to retire.
Work these in order. Later items assume earlier ones exist. Each row is closed only when the tests in §5 pass.
These are the issues users hit first. Closing P0 makes Llama 3, Qwen 2.5, Mistral, and Phi-3 F16/BF16 folders generate instead of erroring.
| Symptom | safetensors dtype not supported for f32 load: BF16 after a successful layout match |
| Cause | safetensors_load_tensor_f32 (libs/uaii-loaders/src/safetensors.cpp) only converts F32 and F16. safetensors_to_dtype already maps BF16. bf16_bits_to_f32 already exists in libs/uaii-quant/src/codec.cpp. The session already coerces BF16 graph tensors to f32 buffers. The read path is the hole. |
| Resolution | Add a conversion table in one place (safetensors_load_tensor_f32 and any streaming alias of it). Required: F32, F16, BF16. Optional same PR: I8 / U8 as integer→f32. F8_E4M3 / F8_E5M2: convert or fail with “convert this checkpoint to BF16”. Never memcpy BF16 bits into an f32 buffer. |
| Tests | Unit: round-trip F32, F16, BF16, and a rejected F8. Integration: load a tiny BF16 Llama-style fixture and run one generate step. |
| Done when | Llama-3.x and Qwen2.5 Hub BF16 folders load without a dtype error. |
| Symptom | Model loads, tokens are garbage at context > ~2k, or even at short context vs Hugging Face |
| Cause | HF loader emits RoPE with theta + mode only (hf.cpp ~875). rope_f32 implements interleaved vs NeoX pairing and optional rotary_dim. It does not read rope_scaling (llama3, yarn, linear, dynamic, longrope). Llama 3.1/3.2 and many Qwen long-context cards are wrong without this. |
| Resolution | Parse rope_scaling in hf_parse_config_json. Pass factor, low/high freq factors, original context, and type into the RoPE op (new attributes; keep defaults = identity). Implement Llama-3 frequency interpolation in rope_f32 to match Hugging Face Llama3ScalingRotaryEmbedding. Yarn / linear / dynamic as follow-on if the same attribute set covers them; otherwise fail closed naming the rope_scaling.type. |
| Tests | Numeric: first 8 RoPE frequencies vs a Python reference for Llama-3.1 8B config. Golden: 32-token greedy decode vs transformers on a 1–2 layer fixture. |
| Done when | Llama 3.1/3.2 short prompts match Hugging Face greedy tokens on a tiny or reduced-layer fixture. |
| Symptom | Model loads; output is wrong from token 1; or encode length ≠ Hugging Face |
| Cause | load_hf_tokenizer pulls model.vocab + model.merges and ignores pre_tokenizer, decoder, byte_fallback, added_tokens, and unk_token. GPT-2 / Llama-3 tokenizer.json is byte-level BPE. Missing the byte map silently mangles UTF-8. |
| Resolution | Parse Hugging Face tokenizer.json as a real pipeline: pre-tokenizer type, byte-level map, added/special tokens (bos, eos, pad, unk, eot), and decode. Prefer using the same vocab Hugging Face ships rather than a parallel BPE. Unigram / SentencePiece inside JSON: either call SentencePiece or fail closed (“rebuild with SentencePiece” / “use tokenizer.model”). |
| Tests | Encode/decode round-trip for English, CJK, and emoji against Hugging Face AutoTokenizer on Llama-3, Qwen2, Mistral, and GPT-2 fixtures (tokenizer files only, no weights). |
| Done when | uaii tokenize on those four tokenizer.json files matches Hugging Face ids. |
| Symptom | uaii pull with only curl writes config.json and maybe one model.safetensors; 7B+ repos are incomplete |
| Cause | Curl fallback downloads a fixed file list and never walks model.safetensors.index.json (libs/uaii-cli/src/commands/pull.cpp). |
| Resolution | If huggingface-cli / hf exists, keep using it. Curl path: download config.json + tokenizer files + *.safetensors.index.json, parse weight_map, download every unique shard, refuse to print “pulled” unless every shard is present and sized. Prefer huggingface_hub snapshot when Python is available. |
| Tests | Mock index with two shards; assert both files exist or the command exits non-zero. |
| Done when | uaii pull of a two-shard fixture is complete without huggingface-cli. |
| Symptom | Users discover gaps at runtime, mid-load |
| Cause | No probe. uaii doctor does not inspect a model directory. |
| Resolution | uaii inspect <hf-dir> (or uaii generate --dry-run) prints: architecture, layout family, weight dtypes, tokenizer kind, RoPE scaling type, MoE, denied features. Unknown dtype / layout / scaling must appear as blocked: lines with the exact error generate would throw. |
| Tests | BF16 Llama dir → weights: BF16 (after P0.1: ok). GPTQ dir → blocked: quantized safetensors. DeepSeek MLA dir → blocked: HF MLA. |
| Done when | Support replies can ask for uaii inspect output instead of a full generate log. |
| Symptom | Users expect kimi / Kimi K2 to run like Llama. HF dir hits the MLA reject. |
| Cause | Kimi K2 is DeepSeek-V3-class (MLA + MoE), not Llama. HF path refuses q_a_proj / kv_a_proj_with_mqa. GGUF already has expand-then-attend (docs/gguf_support.md). |
| Resolution | Treat GGUF as the supported Kimi path in v1: golden uaii generate on a tiny DeepSeek-V3/Kimi blk.* fixture with MLA tensors. Document uaii generate --model kimi.gguf. Do not tell users to uaii pull a Kimi HF folder until P1.8. uaii inspect on a Kimi HF dir must say blocked: HF MLA; use GGUF. |
| Tests | Existing GGUF MLA graph tests plus greedy tokens vs llama.cpp (or a stored blob) on a tiny fixture. |
| Done when | Kimi/DeepSeek GGUF is a documented, tested path. HF still errors clearly. |
| Symptom | GptOssForCausalLM is CausalLM; layout guesser will mis-wire it as Llama or fail on MXFP4 / MoE packing |
| Cause | gpt-oss is MoE + YaRN + attention sinks, often MXFP4 in the official card. It is not GPT-2 and not Llama. |
| Resolution | Allowlist gpt_oss. If weights are BF16/F16 dense (community conversions), implement the published block (router + experts + sinks + YaRN). If weights are MXFP4, fail closed with convert-to-BF16 or GGUF — same policy as GPTQ. GPT-2 stays a separate family (P1.3). ChatGPT API models stay out of scope. |
| Tests | Config parse for gpt_oss; MXFP4 dir rejected with convert hint; BF16 fixture (if we commit one) greedy vs Hugging Face. |
| Done when | uaii inspect names family=gpt_oss and either ok or blocked: mxfp4 convert to BF16/GGUF. |
Status (code): P1.1–P1.7 and P2.3 shared expert are wired. uaii inspect status: ok requires the tensors those graphs need. Remaining: Hub/transformers greedy goldens (§5.2), HF MLA (P1.8, stay fail-closed), Jinja chat_template execution (family equivalent is used).
Closing P1 makes Gemma (Gemini-class), GPT-2, Qwen3, and Kimi HF match reference logits, not merely parse.
| Symptom | Gemma / Gemma2 loads; perplexity is nonsense |
| Cause | rms_norm_f32 multiplies by w[i]. Gemma stores a delta: x * rsqrt * (1 + w). |
| Resolution | RMSNorm attribute offset=1 (default 0). Set it from model_type gemma / gemma2 / gemma3. |
| Tests | One-row RMSNorm vs NumPy for both offset 0 and 1. |
| Symptom | Qwen3, Gemma2, Olmo2, some Llama variants: attention is wrong |
| Cause | Layout never binds q_norm / k_norm. Graph goes Q→RoPE→Attention. |
| Resolution | Detect self_attn.q_norm.weight / k_norm.weight. Insert RMSNorm on Q and K (per-head last axis) before RoPE. |
| Tests | Graph dump contains q_norm / k_norm nodes when tensors exist; fixture logits vs Hugging Face. |
| Symptom | GPT-2 / GPT-J-style absolute position models generate without position signal |
| Cause | Documented. Embed is token table only. transformer.wpe.weight is ignored. |
| Resolution | If wpe (or embed_positions) exists, add tokens_pos + Embedding + Add before the first layer. Positions must follow the sliding KV absolute index (same as RoPE). |
| Tests | Two-token prompt: hidden state after embed+wpe matches Hugging Face. |
| Symptom | Gemma2/3 and Mistral sliding-window cards disagree with Hugging Face on long context |
| Cause | Attention is full causal. Gemma2 also has post-attn and post-MLP RMSNorm and logit softcapping. |
| Resolution | Pass sliding_window into Attention. Add layout for Gemma2 residual norms. Optional final_logit_softcapping / attn_logit_softcapping as tanh clip attributes. Fail closed if use_sliding_window is true and the kernel cannot apply it. |
| Tests | Attention mask with window=8 vs dense causal on a toy QKV. |
| Symptom | Phi / Phi-3, some Falcon: RoPE applied to the full head |
| Cause | rotary_dim is used for MLA GGUF, not wired from HF partial_rotary_factor / rotary_pct. |
| Resolution | rotary_dim = head_dim * partial_rotary_factor (or explicit rotary_dim) on both Q and K RoPE nodes. |
| Tests | Phi-3 mini config fixture: rotary_dim metadata equals expected. |
| Symptom | NeoX-style models: attn and MLP both take the pre-norm hidden state |
| Cause | Graph is always sequential: attn residual then MLP on that residual. |
| Resolution | If config use_parallel_residual or tensor pattern shows one norm feeding both branches, emit out = x + attn(x) + mlp(x). |
| Tests | Graph has both branches reading the same pre-norm tensor. |
| Symptom | hidden_act: gelu_pytorch_tanh / relu2 / swish silently uses SiLU or GELU |
| Cause | use_gelu is a substring check; everything else is SiLU. |
| Resolution | Map known acts to kernels. Unknown act → NotImplemented with the string. |
| Tests | Config with hidden_act: "foo" fails closed. |
| Symptom | Kimi K2 and DeepSeek-V3 HF folders error; GGUF works after P0.6 |
| Cause | HF tensor names (q_a_proj, kv_a_proj_with_mqa, q_b_proj, …) are not bound. GGUF expand-then-attend already exists. |
| Resolution | Reuse the GGUF MLA graph builder with an HF name map. Required for Kimi to be a first-class Hub path. Do not start until P0 Llama/Qwen goldens match Hugging Face. Include shared expert (P2.3) in the same PR if Kimi K2 has one. |
| Tests | Tiny HF MLA fixture greedy vs the GGUF path on identical weights. |
| Done when | uaii generate --model ./Kimi-K2-dir runs on dense BF16. MXFP4/quantized Kimi still convert-or-GGUF. |
Do not pretend these work. Implement or keep the deny/error. After P2, Hub traffic outside §1.1 is either supported or explained.
| Symptom | Layout fail on qweight / qzeros, or garbage if a tensor name collides |
| Cause | No quantization_config handling. HF path is dense F16/BF16/F32. Packed GEMM exists for GGUF quants only. |
| Resolution | Do not unpack GPTQ/AWQ in v1. On quantization_config or qweight tensors, error: HF quantized checkpoints are not loaded. Convert to BF16 safetensors or GGUF (Q4_K / IQ4_NL) and retry. Same message for official MXFP4 gpt-oss / Kimi cards. Optional later: GPTQ unpack-to-f32 for correctness, not speed. |
| Tests | Tiny GPTQ-shaped index is rejected with that message. |
Superseded by P1.8 (Kimi is a product family). Until P1.8 merges, keep the current fail-closed message and the GGUF redirect (P0.6).
| Symptom | Mixtral-style experts load; shared-expert models drop a residual MLP |
| Cause | MoE branch only wires routed experts (block_sparse_moe / mlp.experts). mlp.shared_expert is ignored. |
| Resolution | Detect shared_expert / shared_experts weights; add parallel SwiGLU and add into the MoE output. |
| Tests | Graph contains shared-expert MatMuls when tensors exist. |
| Symptom | rope_mode = -1 and no bias → no position information |
| Cause | Only RoPE and (soon) wpe exist. |
| Resolution | Attention attribute alibi=true + num_heads. Or fail closed if position_embedding_type == "alibi" until the kernel exists. |
| Tests | Fail closed first; kernel test when implemented. |
| Symptom | Denied by arch_denied |
| Cause | No IR ops. |
| Resolution | Keep deny. Error must name the architecture and point at plugins. Do not accept *ForCausalLM for these. |
| Tests | MambaForCausalLM, RwkvForCausalLM, LlavaForConditionalGeneration rejected. |
| Symptom | Folder with only pytorch_model.bin is not an HF decoder load |
| Cause | Loader requires safetensors. |
| Resolution | Keep requiring safetensors. Error: need *.safetensors (convert with transformers / safetensors.torch.save_file). Do not parse pickle. |
| Tests | Dir with only config.json + .bin fails with that message. |
Split detection into named families: llama, qwen2, qwen3, mistral, mixtral, gemma, gemma2, gpt2, gpt_oss, kimi_mla / deepseek_v3. uaii inspect prints the family. A new Hub naming scheme adds a family + tests, not another else if buried in 150 lines.
Unknown layout error must list: first 20 tensor names, expected examples (model.layers.0.self_attn.q_proj.weight), and a link to this document.
See §5. This is what prevents “we fixed Llama and broke Gemma”.
docs/huggingface_support.md and website/app/docs/huggingface/page.tsx list supported families and dtypes, not “any decoder”. When a P-item closes, update both in the same PR.
Users with GPTQ/AWQ/GGUF-only needs get one documented path:
# Dense BF16 Hub → UAII
uaii pull org/model --outdir ./hf-model
uaii generate --model ./hf-model --prompt "Hello"
# Already GGUF
uaii generate --model model.gguf --prompt "Hello"
# Quantized HF (not loaded)
# Convert with the publisher's tool to BF16 or to GGUF, then use the lines above.
Do not add a half-working GPTQ kernel “to say we support it”.
Order is the §1.1 product: Llama + Qwen2 + Mistral first (users think these already work), then Gemma, then Kimi GGUF, then gpt-oss, then Kimi HF.
| Step | Item | Unlocks |
|---|---|---|
| 1 | P0.1 BF16 load | Llama / Qwen / Mistral / Gemma Hub folders ingest |
| 2 | P0.3 tokenizer.json | Tokens are not garbage (Llama-3, Qwen, Mistral, GPT-2) |
| 3 | P0.2 RoPE scaling | Llama 3 and long-context Qwen correctness |
| 4 | P0.5 inspect | Supportable errors (family=…, blocked:…) |
| 5 | P0.4 sharded pull | uaii pull of 7B+ Qwen/Llama repos |
| 6 | P1.1 + P1.4 | Gemma / Gemini-class actually correct (not just loaded) |
| 7 | P1.2 | Qwen3 (and Gemma2 QK-norm) |
| 8 | P0.6 | Kimi via GGUF MLA (ship while HF MLA is open) |
| 9 | P0.7 | OpenAI gpt-oss (BF16 or convert hint) |
| 10 | P1.3 | GPT-2 wpe |
| 11 | P3.1 layout catalog + P3.2 CI matrix | No regressions across the six brands |
| 12 | P1.8 + P2.3 | Kimi / DeepSeek HF + Qwen-MoE shared expert |
| 13 | P1.5–P1.7, P2.1 documented reject | Phi partial rotary, unknown acts, GPTQ/MXFP4 |
Do not start P1.8 until step 3 goldens match Hugging Face. Wrong MLA is worse than a clear GGUF redirect.
Bugs will still exist. What this section prevents is unknown families silently succeeding and known families regressing.
| Test | Asserts |
|---|---|
safetensors_load_tensor_f32 F32 / F16 / BF16 |
max abs error vs NumPy |
| rejected dtype | error contains the dtype string |
hf_parse_config_json Llama 3 rope_scaling |
fields populated |
unknown hidden_act |
NotImplemented |
arch_denied samples |
Mamba, LLaVA, BERT, T5 |
| tokenizer encode/decode | fixtures for Llama-3, Qwen2, Mistral, GPT-2, Gemma (SentencePiece or tokenizer.json) |
| RMSNorm offset 0 vs 1 | numeric |
| QK-norm graph rewrite | nodes present iff weights present |
Commit tiny random safetensors (not 7B weights) with real config.json + tokenizer files for each §1.1 family:
- Llama-style split QKV + SwiGLU (BF16) +
rope_scaling.type=llama3 - Qwen2 with QKV bias + NeoX RoPE
- Qwen3 with QK-norm (after P1.2)
- Mixtral 2-layer 4-expert
- Gemma with RMSNorm offset 1
- GPT-2 Conv1D + GELU +
wpe - DeepSeek/Kimi MLA GGUF (P0.6); HF MLA fixture after P1.8
- gpt-oss config (P0.7): either tiny BF16 MoE or a reject-MXFP4 test
For each: greedy 8 tokens vs a Python transformers script in tests/goldens/ (or precomputed logits blob). Tolerance: exact token match on greedy; logits MAE documented.
| Repo (examples) | Family |
|---|---|
HuggingFaceTB/SmolLM2-135M |
Llama-like BF16 |
Qwen/Qwen2.5-0.5B |
Qwen2 |
Qwen/Qwen3-0.6B (after P1.2) |
Qwen3 QK-norm |
mistralai/Mistral-7B-v0.3 or a tiny Mistral stand-in |
Mistral sliding window |
gpt2 |
OpenAI GPT-2 |
openai/gpt-oss-20b (BF16 conversion if official is MXFP4) |
gpt-oss |
google/gemma-2-2b-it (if license CI allows) |
Gemini-class |
| A small Kimi/DeepSeek GGUF | Kimi via P0.6 |
Nightly may skip gated repos. Merge CI must never depend on Hub or licenses.
Nightly may skip gated repos. Merge CI must never depend on Hub or licenses.
§1.1 is green, not “the Hub is done”.
- Llama-like: P0.1–P0.3 ingest/RoPE/BPE (unit); Hub greedy vs Hugging Face still nightly
- Qwen2/2.5 layout; Qwen3 QK-norm (P1.2) + shared expert (P2.3)
- Mistral dense + Mixtral: sliding window (P1.4)
- Gemma / Gemma2: RMSNorm
(1+w), extra norms, sliding, logit cap (P1.1/P1.4) - Kimi: P0.6 GGUF tiny generate +
blocked: HF MLA; use GGUF; P1.8 HF folders still denied - OpenAI: GPT-2
wpe(P1.3); gpt-oss P0.7 fail-closed (oknever means silent Llama) -
uaii inspectprintsfamily=from the table andblocked:outside it - Website Hugging Face page lists these six brands, dtypes F16/BF16, and the Gemini/ChatGPT API exclusion
- GPTQ/AWQ/MXFP4/SSM either converted or rejected with §6 messages
- No path copies BF16/F16 bits into f32 without conversion
- No path applies RoPE without honouring
rope_scalingwhen present - Hub/
transformersgreedy goldens (§5.2) and optional nightly Hub matrix (§5.3) - Execute shipped Jinja
chat_template(today: family equivalent of Llama-3 / Gemma / Mistral / ChatML) - gpt-oss dense family handler; HF MLA bind (P1.8)
When those boxes are ticked, a new Llama-like or Qwen2-like Hub model is a config problem, not a new runtime. Kimi-class and gpt-oss stay explicit families.
Every message must include what failed and what to do. Implement these strings (or equivalent) so support can grep them.
| Condition | Error |
|---|---|
| Encoder-decoder / deny list | unsupported Hugging Face architecture (decoder-only CausalLM required): {arch} |
| Unknown layout | HF decoder layout not recognized under {dir}. First tensors: … Expected e.g. model.layers.0.self_attn.q_proj.weight. See docs/huggingface_decoder_completeness.md |
| F8 / unknown dtype | safetensors dtype {dtype} is not loaded. Convert this checkpoint to BF16. |
quantization_config |
quantized safetensors (inspect blocked:) |
| MLA HF (before P1.8) | HF MLA; use GGUF |
| gpt-oss MXFP4 | mxfp4 convert to BF16/GGUF |
| gpt-oss dense packing | gpt-oss family handler |
Unknown hidden_act |
unsupported hidden_act={name} |
| Tokenizer missing | no HF tokenizer found in {dir} (need tokenizer.json, vocab.json+merges.txt, or tokenizer.model) |
| SentencePiece off | tokenizer.model found but UAII was built without SentencePiece. Rebuild with -DUAII_WITH_SENTENCEPIECE=ON |
| Unigram JSON | Unigram tokenizer.json is not loaded. Use tokenizer.model (SentencePiece) or a BPE tokenizer.json |
| Rope scaling unknown | unsupported rope_scaling.type={type} |
| ALiBi | unsupported position_embedding_type=alibi |
| Curl pull incomplete | pull incomplete: missing shard {name} |
| Single tensor > RAM | weight {name} is {bytes} which exceeds the memory preset. Use --preset desktop|server or a quantized GGUF. |
| Area | Files |
|---|---|
| Dtype ingest | libs/uaii-loaders/src/safetensors.cpp, tests next to hf_loader_test.cpp |
| Config + layout + graph | libs/uaii-loaders/src/hf.cpp, include/uaii/loaders/hf.hpp |
| RoPE / RMSNorm / Attention | libs/uaii-kernels/src/transformer.cpp, norm.cpp, dispatch.cpp |
| Tokenizer | libs/uaii-tokenizers/src/hf_tokenizer.cpp, bpe_tokenizer.cpp |
| Pull | libs/uaii-cli/src/commands/pull.cpp |
| Inspect | new command or inspect.cpp |
| Docs | this file, huggingface_support.md, website/app/docs/huggingface/page.tsx |
Core session, C API, and Python SDK should not change except where new SessionOptions are required (they should not be for P0).
After this work, UAII still will not:
- Run Gemini or ChatGPT / GPT-4 (no weights; use Gemma and gpt-oss / GPT-2)
- Match llama.cpp tokens/s or GPU coverage
- Run vision, audio, encoder-decoder, SSM, or RWKV (including Kimi-VL, Pixtral, Qwen2-VL)
- Load GPTQ/AWQ/MXFP4 at speed (convert to BF16 or GGUF)
- Guarantee trillion-parameter RSS
- Treat Kimi HF folders as Llama-like
Those stay documented limits, not bugs.
When someone says “this Hub model does not run”:
- Ask for
uaii inspect <dir>(or generate’s first error line). - Match it to §6.
- If
blocked:is a P-item still open, point at this document’s section. Do not patch a one-off in the CLI. - If inspect says
okand generate is wrong, it is a golden-matrix miss: add a fixture, do not tweak prompts. - If the architecture is denied, say so and offer GGUF only when that arch exists in
gguf_support.md.