Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions config/template-pins.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,25 @@
{
"pins": {
"qwen3-0.6b-npu-1bit": {
"sha256": "a136047ba140347b359d17b6ff8fa1bf8cb29d23f0c07bef7e0bf479b8680793",
"source": "FastFlowLM/Qwen3-0.6B-NPU2"
},
"qwen3-0.6b-npu2": {
"sha256": "a136047ba140347b359d17b6ff8fa1bf8cb29d23f0c07bef7e0bf479b8680793",
"source": "FastFlowLM/Qwen3-0.6B-NPU2"
},
"qwen3-0.6b-q8_0.gguf": {
"sha256": "57f1fd00f0013a2be96aa79b857391f27e23df5b5f847072b524c897e24d0361",
"source": "1bit-MONSTER/Qwen3-0.6B-GGUF"
},
"qwen3-4b-instruct-2507-1bit": {
"sha256": "c979e0e71a3e21b8f208e6ab120d5cb29327885f29d2a8b18fda67a723798e18",
"source": "unsloth/Qwen3-4B-Instruct-2507-GGUF"
},
"qwen3-4b-instruct-2507-q4_k_m.gguf": {
"sha256": "c979e0e71a3e21b8f208e6ab120d5cb29327885f29d2a8b18fda67a723798e18",
"source": "unsloth/Qwen3-4B-Instruct-2507-GGUF"
},
"qwen3-coder-30b-a3b-instruct-q4_k_m.gguf": {
"sha256": "87710339d25b4e789c1d723f93c91ee861a86d305bb3d20a845536f251d6ea8a",
"source": "Qwen/Qwen3-Coder-30B-A3B-Instruct"
Expand Down
6 changes: 6 additions & 0 deletions docs/serve.md
Original file line number Diff line number Diff line change
Expand Up @@ -796,6 +796,12 @@ says nothing, because nothing runs (this keeps the header-only route fixtures on
{"pins": {"zaya1-8b-q4_k_m.gguf": {"sha256": "b55ecfbbbcbb238a…", "source": "Zyphra/ZAYA1-8B"}}}
```

The table carries the rows the appliance itself ships — the stick's `Qwen3-4B-Instruct-2507-1bit` GGUF and
its `Qwen3-0.6B-NPU-1bit` container, whose digest is the `chat_template` string from
`tokenizer_config.json` ([npu-chat-template.md](npu-chat-template.md)) — plus the demo and scoreboard rows and all four `laya-first` routing rows.
A model with no row is logged, not checked, which is why a row is added with its measured digest rather
than guessed (`models/pins.json` records that digest for every file we serve and publish).

A pin is matched against the model id serve serves (`--alias`, i.e. Lemonade's catalog entry) first,
then the file's own name, case-insensitively; `"none"` pins a model that must carry no template. A
model whose pin does not match prints a second line and **still serves**:
Expand Down
10 changes: 9 additions & 1 deletion tests/template_pin.sh
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,9 @@ base = [st("general.architecture", "qwen3"), u32("qwen3.block_count", 28), u32("
u32("qwen3.embedding_length", 1024), u32("qwen3.attention.head_count", 16),
u32("qwen3.attention.head_count_kv", 8)]
for name, kv in (("tpl", [st("tokenizer.chat_template", sys.argv[2])]), ("bare", []),
("zaya1-8b-q4_k_m", [st("tokenizer.chat_template", "{% for m in messages %}{{ m }}{% endfor %}")])):
("zaya1-8b-q4_k_m", [st("tokenizer.chat_template", "{% for m in messages %}{{ m }}{% endfor %}")]),
("qwen3-4b-instruct-2507-q4_k_m",
[st("tokenizer.chat_template", "{% for m in messages %}{{ m }}{% endfor %}")])):
all_kv = base + kv
with open(f"{sys.argv[1]}/{name}.gguf", "wb") as f:
f.write(b"GGUF" + struct.pack("<IQQ", 3, 0, len(all_kv)) + b"".join(all_kv))
Expand Down Expand Up @@ -117,6 +119,12 @@ check "the built-in pins compare without --template-pin (exit $rc)" \
'[ "$rc" = 0 ] && grep -q "template sha256=" "$scratch/err" &&
grep -q "warning: template pin: zaya1-8b-q4_k_m (zaya1-8b-q4_k_m.gguf) expects b55ecfbbbcbb238a011465bfa139366db18790103f9205060cf2855a3f51b30f" "$scratch/err"'

# A row the appliance ships, so the stick's own chat model is covered by the built-in table rather than
# only by the demo/scoreboard names (config/template-pins.json records unsloth/Qwen3-4B-Instruct-2507).
run qwen3-4b-instruct-2507-q4_k_m; rc=$?
check "the appliance's staged chat model is a built-in pin (exit $rc)" \
'[ "$rc" = 0 ] && grep -q "warning: template pin: qwen3-4b-instruct-2507-q4_k_m (qwen3-4b-instruct-2507-q4_k_m.gguf) expects c979e0e71a3e21b8f208e6ab120d5cb29327885f29d2a8b18fda67a723798e18" "$scratch/err"'

# A model directory (a Q4NX / NPU container): the template is tokenizer_config.json's "chat_template",
# or chat_template.jinja beside it. --print-backend on a directory exits 5 (the NPU runs in serve's own
# process), which is after the line, so the line is what these cases check.
Expand Down
Loading