Read DeepSeek-V3 checkpoints in the converter - #26
Open
fab2s wants to merge 2 commits into
Open
Conversation
fp8 checkpoints (DeepSeek V3/R1, Kimi-K2) store weights as F8_E4M3 with a per-tile `_scale_inv` companion. Both safetensors readers now apply it. The tile size comes from `quantization_config.weight_block_size` rather than being inferred from the two shapes: inferring looks possible and is wrong whenever a dimension is not a multiple of the tile, and a compatible-but-wrong size is undetectable by shape alone. MoE tensor names differ across the family. Mixtral and Kimi use `block_sparse_moe.experts.E.w1/w3/w2`, DeepSeek uses `mlp.experts.E.gate_proj/up_proj/down_proj`. The layout is detected from what is on disk and normalised to one spelling, so the engine sees one name. Without it a conversion finds no experts and reports `0 MB [missing]` for every layer, after the download has already run. `tests/test_convert_resume.py` stubs `mxfp4` with a `types.ModuleType`, which has no `__file__`, so the new import has to resolve there or every check in that file fails at module load.
The two families disagree on MoE config keys as well as on tensor names, and this half fails differently. The engine reads `num_experts`; a DeepSeek config only spells it `n_routed_experts`, so `cfg_from_json` yields 0 experts and the bank loop in `waste_model_load` refuses the container with no diagnostic — after the conversion has completed. `num_experts_per_tok` against `..._per_token` is one letter and leaves `top_k` at 0. The manifest is WASTE's format rather than HF's, so the keys are normalised where the tensor names already are, and written only when absent so a config that already uses the canonical spelling wins. `moe_renormalize` is keyed on the field being present rather than on its value, so a plain alias of DeepSeek's `norm_topk_prob` would turn renormalisation on for a checkpoint that sets it false. It is emitted only when true.
fab2s
pushed a commit
to fab2s/waste
that referenced
this pull request
Aug 7, 2026
The suite stayed green through the whole window in which `src/` applied no rotary, and it would have stayed green after a fix that pairs the wrong dims. Both have the same cause: every container the suite can reach is a Kimi, every Kimi sets `mla_use_nope`, and so nothing in `tests/` ever entered `rope_init` or `rope_apply`. This is the missing half of the previous commit. `make_test_container.py --rope` writes a DeepSeek-V3 at the 1/18 scale the file already builds a Kimi-Linear at: no `mla_use_nope`, `rope_theta` and the YaRN block copied from Kimi-K2-Instruct's config, and no `linear_attn_config` at all, which is what makes every layer MLA. All-MLA is deliberate twice over — it exercises the rotation at depth rather than in the single full-attention layer the Kimi mix leaves, and it is the shape `deepseek_ref.py` can read, since not indexing `linear_attn_config` is exactly what separates it from `kimi_ref.py`. K2's rope block rather than V3's because `beta_fast == beta_slow == 1.0` collapses YaRN's correction range to a two-dim ramp, which is the more awkward of the two to get right. The checks build their own container instead of using `$MODEL`, so they run on every host and do not wait on weights nobody can convert yet — sqliteai#26 is what makes a real V3 container, and the shape is what the engine branches on. - rotated MLA against the PyTorch oracle - chunked prefill == token-at-a-time with rotation, which holds by construction today because `mla_layer` is per-token on both paths, and is exactly the "by construction" a later batched MLA would break quietly - a rope slice wider than `WASTE_MAX_ROPE_HALF` is refused at load The first takes the same two-source shape as the Kimi oracle above it: generate from `deepseek_ref.py` where `uv` exists, fall back to a fixture where it does not, so the Linux image without `uv` runs it rather than skipping it. Unlike that one the fixture ships, because this container is generated rather than converted and so is byte-reproducible at `--seed 0` — the sidecar carries a digest of the container it was made from, so a later change to the generator's weights reads as "regenerate me" and not as an engine bug. The fixture is the reference's logits, never the engine's. `deepseek_ref.py` grows the `--dump` that `kimi_ref.py` already had, so the diff is over whole logit vectors and not a printed top-k. Verified by reverting `src/model.c` and `src/model.h` to their pre-fix state with `tests/` and `tools/` left alone: the oracle check and the refusal check both fail, which is the property that makes them worth having. Both fallback paths were exercised directly — `uv` off `PATH` passes against the fixture, and a corrupted digest skips with the regenerate message instead of reporting a divergence. `set -o pipefail` sank the refusal check on the first run, because a refused load exits non-zero and that is the point; the output is read into a variable now, with a comment saying why. Suite on this commit: 46 passed, 0 failed, 2 skipped against Kimi-Linear and K3 (43/0/2 before), 39/0/9 on the synthetic path CI takes, and `make asan` 33/0/14. Fuzzer and the 168 serve checks unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Read DeepSeek-V3 checkpoints in the converter
Two commits, both needed before
convert.pycan produce a container from aDeepSeek-V3 family checkpoint (V3, R1, Kimi-K2). Independent of #27, which
fixes a separate defect in the engine's attention path — that one builds and tests
without this, but reproducing it needs a container, so this comes first.
fp8 block scales. These checkpoints store weights as
F8_E4M3with a per-tile_scale_invcompanion. Both safetensors readers now apply it —mxfp4.STandconvert.py's ownShardReader, which is a second reader and was easy to miss.The tile size is read from
quantization_config.weight_block_sizerather thaninferred from the weight and scale shapes: inferring looks possible and is wrong
whenever a dimension is not a multiple of the tile, and a compatible-but-wrong
size passes the shape check while placing every scale on the wrong rows.
MoE tensor names. Mixtral and Kimi use
block_sparse_moe.experts.E.w1/w3/w2; DeepSeek usesmlp.experts.E.gate_proj/up_proj/down_proj. The layout is detected from what ison disk and normalised to one spelling, so the engine still sees one name. Without
it the expert probe misses on every layer and a run reports
0 MB [missing]60times after the download has already completed.
MoE config keys. The same split one level up, and this half fails differently:
src/model.creadsnum_experts, a DeepSeek config only spells itn_routed_experts, socfg_from_jsonyields 0 experts and the bank loop inwaste_model_loadrefuses the finished container with no diagnostic. Alsonum_experts_per_tokagainst..._per_token, which is one letter and leavestop_kat 0. Normalised into the manifest where the tensor names already are, andwritten only when absent so a config using the canonical spelling wins.
moe_renormalizeis handled separately becausemodel.ckeys it on the fieldbeing present rather than on its value — a plain alias of
norm_topk_probwould enable renormalisation for a checkpoint that sets it false. Emitted only
when true.
Testing
make checkwithWASTE_REF_MODELpointed at a default VQ3R Kimi-Linearcontainer.
tools/../test_fp8_blocks.pycovers the block mapping on synthetictensors (aligned tiles, partial tiles on both axes, a missing companion, a gross
size mismatch) and cross-checks one real tensor bit-exactly against an
index-array dequant; it also documents the case that cannot be caught, a
compatible-but-wrong tile size, which is why the value is read from config.
The
tests/test_convert_resume.pychange is in the first commit rather than itsown: that file stubs
mxfp4with atypes.ModuleType, so the new import has toresolve there or every check in the file fails at module load, and the commit
would not be green on its own.
Verified end to end by converting
Kimi-K2-Instruct— 61 layers, 384 expertstop-8, VQ3R, 354 GB expert set, 6.9 GB trunk — and opening it with
waste info:1.03 T total, 31.69 B active per token.
Nothing outside
tools/and that one test file changes.docs/LEARNED.md,CHANGELOG.mdandWASTE_VERSION_*are deliberately untouched.