serve/ cannot express tool calls for a chat.json container

[marcobambini]

python3 -m serve refuses to start on any container that is not K3. Reported from a
Kimi-Linear container on a 16 GB MacBook, where waste run and waste chat both work:

WASTE 0.6.6 (container v0, backend NEON, crc32 armv8, arm64)
model kimi_linear — kimi-linear, 27 layers, 256 experts, ctx 4096
<|open|> is not a single token in this container (got 5): its specials.json
does not carry K3's XTML markers: unsupported

What is happening

ServerCore.__init__ resolves the four XTML control tokens up front
(serve/server.py:85 → serve/engine.py:566), and raises WASTE_E_UNSUPPORTED
unless each of <|open|>, <|sep|>, <|close|>, <|end_of_msg|> encodes to
exactly one token. Kimi-Linear's tokenizer has none of them — measured on a
converted container:

$ ./waste tokenize ~/models/kimi-linear.waste '<|open|>'
27 91 4454 91 29
$ ./waste tokenize ~/models/kimi-linear.waste '<|im_user|>'
163587

The refusal is correct and should stay: absent markers encode as ordinary
text, so the model would read its own turn structure as prose and answer
plausibly and wrongly. The bug is not that it refuses; it is that XTML is the
only format serve/ can speak, so a container that has a perfectly good chat
format of its own has no HTTP path at all.

Two consequences worth separating:

1. Kimi-Linear cannot be served. serve/xtml.py is a port of K3's
   encoding_k3.py, and serve/regions.py reads replies back by matching XTML
   token ids. Neither has a second format. chat.json is not read by serve/
   at all, so there is no way to configure around it.
2. /v1/completions is unreachable too, which is the smaller and more
   annoying half. It is raw continuation and needs no chat template, but the
   failure is in the constructor, so the whole server is down before any
   endpoint is dispatched.

What Kimi-Linear does carry

From its specials.json — enough for a conversation and for tool calls, but
there is no think channel, so reasoning_content has no equivalent and thinking
would simply be off for it:

token	id
<|im_system|>	163594
<|im_user|>	163587
<|im_assistant|>	163588
<|im_middle|>	163601
<|im_end|>	163586
<|tool_calls_section_begin|>	163595
<|tool_call_begin|>	163597
<|tool_call_argument_begin|>	163598
<|tool_call_end|>	163599
<|tool_calls_section_end|>	163596

Shape of a fix

Not a small change, and worth doing in two steps that are independently useful:

• Let the server start. Move marker resolution out of __init__ to the
  point of use, so a container without XTML serves /v1/completions and returns
  a clear 400 on /v1/chat/completions instead of failing to boot.
• A second renderer and a second reply parser. regions.py decides
  structure from token ids rather than text, which is the right design and also
  what makes this more than a template swap: it needs a marker vocabulary per
  format, not a hardcoded four. Kimi-Linear's tool-call markers mean tool_calls
  is reachable; reasoning_content is not.

Whether the renderer is driven by the container's chat.json or by a second
Python module in serve/ is the open design question. chat.json is four
prefix/suffix strings and cannot express tool definitions, so it covers plain
chat only — which may be the right first cut.

Workaround today

waste run and waste chat, which read chat.json and are unaffected. A
Kimi-Linear container has no chat.json unless the converter put one there, and
without it the CLI says so and falls back to raw continuation — which is why the
same report also had an instruct model producing odd answers. Write this into
the container directory, next to manifest.json:

{
 "system": ["<|im_system|>system<|im_middle|>", "<|im_end|>"],
 "user": ["<|im_user|>user<|im_middle|>", "<|im_end|>"],
 "assistant": ["<|im_assistant|>assistant<|im_middle|>", "<|im_end|>"],
 "open": "<|im_assistant|>assistant<|im_middle|>"
}

Do not copy examples/chat.json for this: it is generic ChatML, and
<|im_start|> is not in Kimi's vocabulary — it encodes as six ordinary tokens,
which is the same silent failure described above. The converter now installs the
right one per architecture and refuses to install a template whose markup the
release does not carry.

[marcobambini]

Step 1 is done in 37b4c83 — the server no longer refuses to start. What is left
here is the renderer, and the issue should now be read as being about that only.

What now happens on a container without XTML markers (measured on a
Kimi-Linear container):

listening on http://127.0.0.1:8478  (POST /v1/completions)

GET  /health              200
GET  /v1/models           200
POST /v1/chat/completions 400  unsupported_chat_format
POST /v1/completions      200  and generating

ChatServer.__init__ holds the reason rather than raising it; _chat refuses
before the engine lock, so a request that cannot be served never resets state or
builds a prompt. The 400 names /v1/completions and waste chat as the
alternatives. python3 -m serve prints the reason at startup rather than leaving
it for the first client error. 400 rather than 501: for an OpenAI client the
unsupported thing is the model, which is a request parameter, and 501 is the one
status those clients tend to retry.

The refusal itself is unchanged and deliberate, for the reason in the original
report: markers the tokenizer does not have encode as ordinary text, so a
rendered prompt would not fail visibly — the model would read its own turn
structure as prose and answer anyway.

Covered by tests/serve/test_server.py::TestContainerWithoutXTML, where
reaching setUp is half the assertion. Documented in docs/SERVE.md under
"Containers that are not K3".

Still open — step 2, and the design question with it. serve/xtml.py is a
port of K3's encoder and serve/regions.py matches XTML token ids, so a second
format means a marker vocabulary per format rather than a hardcoded four. For
Kimi-Linear specifically: tool_calls is reachable from the markers its
tokenizer carries, reasoning_content is not, since it has no think channel.
Whether the renderer is driven by the container's chat.json or by a second
module in serve/ is undecided — chat.json is four prefix/suffix strings and
cannot express tool definitions, so it covers plain chat only, which may still
be the right first cut.

Unrelated to this issue but relevant to anyone arriving from the same report:
convert.py now installs a per-architecture chat.json, so waste chat and
waste run address a Kimi-Linear container correctly out of the conversion.

[marcobambini]

Plain chat now works, in ccf6cf3 (0.6.7). What is left here is tool calls only —
my previous comment described the whole renderer as open, and that is no longer
right.

serve/chatfmt.py serves a container from its own chat.json — the four
prefix/suffix strings waste chat has always read — when the tokenizer has no
XTML markers. XTML is still asked for first; this is the fallback. On a
Kimi-Linear container:

chat     from ~/models/kimi-linear.waste/chat.json — plain conversation only,
         no tools, no reasoning channel, no images

System / user / assistant turns, blocking and streaming, with the stop token
taken from the template's assistant suffix rather than guessed. A multi-turn
conversation round-trips. Everything four strings cannot express comes back as a
400 naming the field: tools, reasoning_effort, an image part, a tool-result
turn — refused rather than dropped, because a server that ignores
reasoning_effort reports a different amount of reasoning than it did.

Two things make it safe rather than merely convenient. chat.json is validated
harder than the CLI's reader validates it — serving needs an open, a user
turn, and an assistant suffix carrying a control token, since without that last
one every reply runs to max_tokens and reports finish_reason: "length". And
every <|…|> in the file is resolved against the real vocabulary before the
format is used: markup the tokenizer does not have encodes as ordinary text, and
the model then reads its own turn structure as prose and answers anyway. Point
it at the ChatML examples/chat.json and it refuses, naming <|im_start|>.
Rendering keeps the markup/content split, so a user writing <|im_end|> in a
question cannot close their own turn.

Covered by tests/serve/test_chatfmt.py and TestChatFromChatJson in
test_server.py; the serve suite is 211 checks, up from 174. With K3_DIR set
the differential against encoding_k3.py still matches segment for segment, so
the XTML path is unchanged.

Still open: tool calls over chat.json

Two obstacles, and the second is the real one:

1. Four prefix/suffix strings cannot express a tool declaration, an argument
   list, or a result turn. That needs a module, not a template — so the design
   question from the first comment resolves as: chat.json for plain chat
   (done), a real renderer for tools.
2. The markup is not transcribed anywhere in this repo, and cannot be derived
   from the release on disk — that copy ships no chat_template and no
   reference encoder. Kimi-Linear's tokenizer does carry
   <|tool_calls_section_begin|>, <|tool_call_begin|>,
   <|tool_call_argument_begin|>, <|tool_call_end|> and
   <|tool_calls_section_end|>, so it is reachable in principle, but the naming
   convention, the argument encoding, and how tool definitions enter the
   prompt would have to come from an Instruct release's template or model card.

Worth checking whether the local copy is the Base release before concluding the
template is unavailable upstream.

There is also a standing weaker guarantee here, recorded in CHANGELOG 0.6.7:
xtml.py earns its confidence from a segment-for-segment differential against
the release's own encoder, and there is no equivalent program for Kimi-Linear.
This renderer is transcribed and tested, not differentially verified. If an
Instruct release ships a chat_template, HF's Jinja renderer would be a real
oracle and a tools/-side check like the existing ones.

reasoning_content is not on the table at all: these containers have no think
markup in their specials.