fetch_weights.sh: get_small() drops API-listed files silently — no retry, no log, run still reports success

[GTSUltear]

Summary

get_small() in tools/fetch_weights.sh fetches each small file with a single attempt, and discards the failure silently: no retry, no log line, no effect on exit status. A transient network failure therefore leaves the checkpoint incomplete while the run reports ALL SHARDS COMPLETE and rc=0.

This is not platform-specific. It is still present on main as of today, unchanged from v0.6.6.

What happened

Downloading moonshotai/Kimi-Linear-48B-A3B-Instruct (91.5 GB, 20 shards). Every shard succeeded on try 1. Of the eleven small files the repo lists, one arrived:

2026-08-09 22:34:21  repo lists 11 small files
2026-08-09 22:34:23  got tokenizer_config.json
...
2026-08-09 22:52:49  pass finished (rc=0): 20 / 20 shards complete, 73 GB free
2026-08-09 22:52:49  ALL SHARDS COMPLETE

Missing: config.json, tiktoken.model, chat_template.jinja, special_tokens_map.json, generation_config.json, .gitattributes, README.md, configuration_kimi.py, modeling_kimi.py, tokenization_kimi.py.

All ten were fetchable — probed directly a few minutes later, every one returned 200:

config.json                http=200 size=1751
tiktoken.model             http=200 size=2795286
chat_template.jinja        http=200 size=1850
special_tokens_map.json    http=200 size=5583
generation_config.json     http=200 size=147

So the failures were transient, and nothing in the run said so.

Why it matters

• config.json is required by tools/convert.py. Without it the conversion fails — hours after the download reported success.
• tiktoken.model becomes the container's tokenizer.model. Without it the conversion still completes, and produces a container that cannot tokenize (tokenize: unsupported), which is discovered later still.
• chat_template.jinja is likewise copied through.

The failure mode is "download says it succeeded, conversion fails later for an unrelated-looking reason". On a 1.4 TB K3 pull that gap is a long way from cause to symptom.

Root cause

tools/fetch_weights.sh:108

get_small() {
    for f in "$@"; do
        [ -s "$DEST/$f" ] && continue
        if hcurl -sfL --max-time 300 -o "$DEST/$f.part" "$RAW/$f" 2>/dev/null; then
            mv "$DEST/$f.part" "$DEST/$f"
            log "got $f"
        else
            rm -f "$DEST/$f.part"          # a 404 here is normal: not every
        fi                                  # repo ships every one of these
    done
}

Two things:

1. No retry. The shard path immediately below gets for try in $(seq 1 $max_retry) with exponential backoff and jitter, per the file's own header — "every shard is retried with exponential backoff and jitter". Small files get one attempt.

2. The "a 404 is normal" reasoning does not hold for $SMALL. It is sound for the hardcoded fallback list at L149, where the script is guessing filenames. But $SMALL comes from the repo's own API listing (L129-141) — those files demonstrably exist, so a failure there means a file that exists was not fetched. The comment right above that block makes the point itself:

   "Ask the repo what it contains instead of guessing filenames. The hardcoded list this replaces cost real money: it did not know about encoding_k3.py ... and it silently missed preprocessor_config.json, so the image normalization was the CLIP convention for a day ... A whitelist cannot report what it never knew to ask for."

   Having gone to the trouble of asking the repo what it contains, the script then drops the answer without saying so.

Suggested fix

• Give get_small the same retry/backoff as the shard path when the list came from the API.
• Distinguish a 404 (fine — the fallback list may name files a repo does not ship) from a transport failure (not fine), and log the latter.
• Consider making an unfetched API-listed file affect the exit status, or at least print a summary line: the run currently ends on ALL SHARDS COMPLETE with the checkpoint incomplete.

Workaround

Re-running the script should repair it, since get_small skips files already on disk. On this machine the re-run refused to start for a separate Windows-only reason (reported in the portability issue alongside this one), so the ten files were fetched by hand with curl --retry 5.

Environment

WASTE v0.6.6 (df54209), verified still present on main at d9b919a. Encountered on Windows 11 x86_64 native, MSYS2 UCRT64 — but nothing about the defect looks platform-dependent: the single-attempt path with the discarded error runs everywhere, and the failures here were transient network errors rather than anything Windows did.

—

Reported by Kirin.

🤖 Built, measured and triaged with Claude Code.