Prepare reviewed main-to-zig sync

.github/pull_request_template.md

@@ -0,0 +1,27 @@
+<!-- Read CONTRIBUTING.md first. A small pull request with a complete receipt lands fastest. -->
+
+## What this changes
+
+<!-- One or two sentences, and the issue it closes if there is one. One change to a pull request. -->
+
+## Receipt
+
+<!-- A change to documentation alone needs no receipt. -->
+
+- Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision:
+- Exactness, from `tools/bench_concurrent.py --alone --serial` or `token_sha` against `"draft": false`:
+- Decode speed before and after, from `tools/bench_openai.py`:
+- Prompt speed before and after, from `tools/prefill_cold.py`:
+- Tests run, with pass, fail and skip counts:
+- Not run, and why:
+
+## Checklist
+
+- [ ] Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
+- [ ] Drafted output equals `"draft": false`, a resumed prompt equals a fresh one, and concurrent equals solo.
+- [ ] No precision traded for speed. If the bits change, the description says which sums change and why.
+- [ ] Prompt processing is no slower than the last release.
+- [ ] The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
+- [ ] New tests fail before the change, pass after it, and skip cleanly without their dependency.
+- [ ] Comments and docstrings are one line. No measurements or history in the source.
+- [ ] No personal data, machine names, internal hosts or local paths. No AI attribution lines.

.github/workflows/native-macos.yml

@@ -2,7 +2,7 @@ name: Native macOS bootstrap
 
 on:
   push:
-    branches: [feat/zig]
+    branches: [zig/review-sync, zig/native]
   pull_request:
   workflow_dispatch:
 

.zig-archive.sha256

@@ -1 +1 @@
-98faf6b77e3d681bb27bd65b81dcfc18ee21c437ebb93504eb16731c613fd792  build/toolchains/zig-aarch64-macos-0.17.0-dev.2248+3f6a02acd.tar.xz
+b607e9b9234790a008116ae5bdb71c6243b84b9fb42a53a9e70fde41c06c536a  build/toolchains/zig-aarch64-macos-0.17.0.tar.xz

.zig-version

@@ -1 +1 @@
-0.17.0-dev.2248+3f6a02acd
+0.17.0

CHANGELOG.md

@@ -3,6 +3,124 @@
 `tensorfold update` prints the sections below that are newer than the version you had. Each release's page on
 GitHub has the full notes and the measurements behind them.
 
+## 0.6.3 (2 Oct 2026)
+
+- **Nemotron on M5 Macs: copied text verifies up to 64 tokens a round.** A lone stream's copy window grows from 16 to
+  64 rows while each lands whole, a window attends in one call, and wide windows keep their Mamba states every 8th
+  row. On an M5 Max, one stream editing code decodes about 1.4x faster (550 to 760 tok/s, and up to 1,000 on a cool
+  machine), prose and code writing gain 1.6-3.3%, and drafted replies still equal one-token decoding.
+- **Anthropic Messages API.** `/v1/messages` and `count_tokens` on the Mac and CUDA servers, with streaming, tools,
+  thinking and cache usage (#223, closes #168). Thanks to @kky42.
+- **Control Room.** `tensorfold service` runs a model as a launchd service, and `tensorfold tui` shows its decode and
+  prefill speed and its connections live.
+- **Flash Next on Macs before M5:** `TF_FLASH_DENSE=matrix` runs every dense projection on the matrix units, and the
+  fused DeltaNet stack uses the matrix kernel by default (#149). Thanks to @gilby.
+- **Vision:** `--vision-offload` keeps the CUDA image tower in host RAM between images (#187, closes #185); image
+  parts are accepted inside tool results (#235); `--vision-image-tokens` lets many images share a larger budget
+  (#239); Flash Next on CUDA takes video (#240). Thanks to @barelyworkingcode and @MiaAI-Lab.
+- **CUDA:** `/v1/decisions` on Flash Next scores labels from the prompt's last logits, the questions filling in one
+  pass (#232); `TENSORFOLD_PREFILL_ROWS` sets the prompt piece rows (#238); NVFP4 and FP8 prompt rows add a tile's K
+  slices in one block, with the same bits (#242); `TENSORFOLD_MEMORY_RESERVE_GIB` moves the memory floor the startup
+  keeps free, by default a tenth of the pool and at least 4 GiB as before (#165); Flash Next's chain kernel takes
+  5-17% less time at 2 to 8 rows, with the same bits. Thanks to @Mirrdhyn, @MiaAI-Lab, @jschmied and @eleqtrizit.
+- **Server:** chunked request bodies are decoded before JSON parsing (#244); `/tokenize` and `/detokenize` carry
+  vLLM's fields (#237); `chat_template_kwargs.thinking` is heard as `enable_thinking`, and GLM-5.3 keeps earlier
+  turns' reasoning (#236); streaming usage gets its own chunk when `stream_options.include_usage` asks for it (#216);
+  malformed tool-call history renders safely (#233); `/metrics` times each request's decode (#269); `/health` carries
+  the live decode and prefill speed. Thanks to @JordiPosthumus, @MiaAI-Lab, @salmanarshad321 and @sxuff.
+- **Fixes:** an EXL3 layer with no bias no longer faults in the split-K reduction (#186); an 8-bit `lm_head` keeps its
+  format in Qwen3.6's MTP draft head on CUDA (#270); a lone Flash Next stream's cache growth on CUDA stays inside the
+  explicit memory budget; two ranks refuse to start with different prompt rows. Thanks to @barelyworkingcode and
+  @BHCC2025.
+- **Models moved to the TensorFold Hugging Face org.** `Vontra/<name>` ids redirect, and the tree now names
+  `TensorFold/<name>`.
+- **Contributing.** `CONTRIBUTING.md` says what a pull request needs to land and how it lands, and new pull requests
+  open with a receipt template.
+
+## 0.6.2 (2 Oct 2026)
+
+- **Flash Next on Macs at 64k-128k.** On an M3 Ultra, one stream runs 1.2-3.4% faster at 64k and 3.9-5.5% at 128k,
+  with the same tokens: a window's n-gram ids are hashed on the GPU, and the chain's first step is built while the
+  GPU verifies.
+- **27B with several streams on CUDA.** The GDN tree kernel takes 8-35% less time with the same bits. On an RTX PRO
+  6000 at its 250 W limit, 4 and 8 streams of the NVFP4 27B decode 1.1-4.2% faster.
+- **Fixes:** the config check accepts the FP8 n-gram table in NVIDIA's MIXED_PRECISION Flash Next export; GLM-5.3
+  on CUDA counts its drafts in `/health`, `/metrics` and replies, and names a mixed-bit EXL3 checkpoint when it
+  refuses one; a client that leaves is noticed past file descriptor 1023; the CUDA server prints a line a request, as
+  the Mac server does; a failed snapshot write no longer leaves its partial file; Gemma 4's QKV kernel reserves its
+  1024 threads for M1, M2 and macOS VMs, and a VM's GPU is no longer taken for an M5.
+
+## 0.6.1 (1 Oct 2026)
+
+- **NVFP4 checkpoints in their own math.** `nvidia/Qwen3.8-27B-NVFP4` runs the 4-bit activations its checkpoint
+  names, as vLLM does; `--precision full` runs 16-bit activations against the same weights. On an RTX PRO 6000 at its
+  250 W limit, one stream decodes 1.4-2.0x vLLM and prompts fill at 0.95-0.97x its speed.
+- **Waiting prompts fill together on CUDA.** With `--parallel`, prompts that arrive together now share one prefill
+  forward instead of filling one a round: on an RTX PRO 6000 at its 250 W limit, 8 streams of the 27B run 1.14-1.36x
+  faster and the slowest first token comes in 0.05-0.10 s instead of 0.4-0.8 s (1.6 s to 0.15 s on a DGX Spark), with
+  the same replies.
+- **More 27B tokens with several streams on CUDA.** Streams plan on their measured round cost, and wider lane blocks
+  on RTX PRO and RTX 50 cards add 4-8% at 8 streams, with the same tokens.
+- **Flash Next on Macs at long context.** One stream runs up to 9.6% faster on code at 64k and 10.2% on chat at 128k
+  on an M3 Ultra, with the same tokens.
+- **`/v1/decisions`** scores a choice, a score or a yes/no from the next-token logits, on the shared prompt lanes.
+- **Flash Next on CUDA:** image input, forks that resume from their shared prefix, shared system prompts copied
+  instead of filled again, and short prompts admitted while a long one fills.
+- **RTX cards without Docker:** pip alone installs and builds the CUDA kernels. Native Windows is in as an
+  experimental host layer, not yet run on Windows hardware.
+- **Fixes:** a refused request no longer breaks the next one on its connection; Gemma 4 thought blocks stay out of
+  replies with thinking off; an unnamed reasoning effort goes to the nearest named level; mlx-lm 0.32 support.
+
+## 0.6.0 (30 Sep 2026)
+
+- **RTX 40 cards.** CUDA now runs on compute capability 8.9 (Ada). On one RTX 4090 the 27B serves with DFlash2 in a
+  40,182-token window, exact: drafted replies equal serial ones, and a resume or resend gives a fresh run's reply.
+  Prompts fill at 2.4-2.6k tok/s from 2k to 32k tokens, and replies decode at 64-86 tok/s. On one GPU the 27B's kept
+  prompt states give way, oldest first, when a live reply needs the room, so admission counts one live window.
+- **Prompts fill inside the decode rounds.** With `--parallel` on CUDA, Flash Next prefills a queued prompt in the
+  same forward as the live replies instead of stopping them: on one DGX Spark, first tokens came 2.8-3.1x sooner than
+  on 0.5.0. On Macs several prompts fill side by side, the fewest tokens left first: on an M3 Ultra, short requests
+  queued behind a long prompt got their first token in a median 9.9 s instead of 113 s (the slowest 11.7 s, not 130).
+- **Conversations resume on more engines.** An identical resend or the next thinking turn now resumes from the kept
+  prompt state on Flash Next, Qwen3.6, Nemotron and GLM-5.3 on CUDA, as the 27B did, with a fresh run's reply: an
+  18.7k-token Flash Next resend went from 8.3 s to 0.08 s. On Macs the prompt cache keeps each conversation's newest
+  checkpoint and grows into memory the model leaves idle.
+- **CUDA prompts run at bf16 by default.** Against an fp32 reference the 27B's prompt rows are 20x closer than with
+  FP8 (KL 0.0031 against 0.0624). `--prefill-fp8` keeps 0.5.0's faster FP8 prompts for those who want them.
+- **Tool calls for agents.** The CUDA server streams tool-call arguments as the model writes them (the longest
+  silence in a long call fell from about 30 s to half a second), Python-spelled values like `False` and `None`
+  decode to their schema types, Gemma 4's bare tool calls parse (#121), and a prompt past the context window gets
+  OpenAI's `context_length_exceeded`, so clients compact instead of retrying.
+- **More checkpoints.** GLM-5.3 on Macs reads 8-bit and Q8_0 GGUF checkpoints and takes images, and has an opt-in
+  float32 activation mode (bf16 stays the default). On Macs, Flash Next loads oMLX's oQ checkpoints with scaled
+  n-gram tables; on CUDA it reads consolidated EXL3 tables and block-scaled FP8 linears.
+- **Faster.** Flash Next NVFP4 decodes 4.9-7.1% faster and fills prompts 15-16% faster at 2k-16k, bit-identical. On an
+  M5 Ultra the 27B with DFlash2 on an oQ4e checkpoint went from 33 to 131-160 tok/s. Before M5, a lone stream's copy
+  windows widen to 128 rows (an M3 Ultra edit ran 190 -> 233 tok/s), and Flash Next's concurrent rounds use the matrix
+  units. GLM-5.3's DFlash2 reads only its sliding window: 8% faster at 52k, and 3.8 GiB lighter a rank on two Sparks.
+- **Operations.** Prometheus `/metrics` on both servers, `TENSORFOLD_MEMORY_RESERVE_GIB` for the CUDA startup
+  reserve, `--checkpoint-slots` for the 27B's concurrent decoder on CUDA, and an idle GLM rank no longer spins.
+- **Apache-2.0.** TensorFold is licensed under the Apache License 2.0 from this release, which adds an explicit patent
+  grant from contributors. Releases up to 0.5.0 stay MIT, and code written before 0.6.0 keeps its MIT notice in
+  `LICENSES/MIT.txt`.
+- **Community pull requests.**
+  - Resumed resends and thinking turns on Flash Next (#124). Thanks to @benthecarman, and to @Arminova for the
+    GB10 measurements and the third-resend test.
+  - Streamed tool-call arguments on CUDA (#114), `--alias` on CUDA (#111) and Python-spelled tool parameters
+    (#135). Thanks to @olexale, @philip-pentatonic and @outcastofmusic.
+  - GLM-5.3 on Macs: image input (#101), 8-bit and Q8_0 GGUF checkpoints (#119), float32 activations (#118) and
+    the backbone wiring notes (#120). Thanks to @mgoldwasser and @feni6.
+  - GLM-5.3 on two Sparks: lighter prompt buffers, the EXL3 estimate, the MTP setting, the idle rank and DFlash2's
+    ring (#128, #129, #131, #132, #134), visible-pool selection (#140), and the CUDA startup reserve setting (#133).
+    Thanks to @MiaAI-Lab and @mikolaj92.
+  - Flash Next: scaled n-gram tables (#148, reported in #142) and the M5 draft head fix (#147), consolidated EXL3
+    tables (#145), block-scaled FP8 (#126) and the NVFP4 expert speedups (#102, #105). Thanks to @gilby,
+    @cwschroeder, @shantanugoel, @jschmied and @tournierjc.
+  - On M5 Macs the 27B fuses the projections of 4-bit group-32 checkpoints such as oQ4e too (#164): four streams went
+    from 315-320 to 332-334 tok/s on an M5 Ultra, with the same bits. Thanks to @gilby.
+  - `--checkpoint-slots` (#125), and the DFlash2 drafter's admitted size (#112, from @jkuepker's ROCm work). Thanks
+    to @nood-co1 and @jkuepker.
+
 ## 0.5.0 (29 Sep 2026)
 
 - **OpenAI's Responses API on both servers.** `/v1/responses` runs as a chat completion: every response has its chat
@@ -249,7 +367,7 @@ GitHub has the full notes and the measurements behind them.
 ## 0.3.2 (26 Sep 2026)
 
 - `tensorfold update` installs the newest release.
-- GLM-5.3-Flash reads Mia-AiLab's EXL3 weights on two DGX Sparks (experimental).
+- GLM-5.3-Flash reads Brandon M. Music's EXL3/TR3 weights (re-hosted by Mia-AiLab) on two DGX Sparks (experimental).
 
 ## 0.3.1 (26 Sep 2026)
 

CONTRIBUTING.md

@@ -0,0 +1,124 @@
+# Contributing to TensorFold
+
+Thank you for wanting to make TensorFold faster. This page says what a pull request needs to land in the next
+release, and what happens to it after you open it. A small pull request with its measurements attached lands
+fastest.
+
+## Before you write code
+
+- Read the pinned issue "Landing next" and the open pull requests. Work that is integrated for the next release is
+  listed there before it reaches `main`, so you don't build it twice.
+- Open an issue first for a large change: a new backend, a GPU generation, a model family, or anything that changes
+  the arithmetic a model runs. We will say what it needs to land before you spend the time.
+- Keep one change to a pull request. A fix and an unrelated speed-up land separately. In a stack of dependent pull
+  requests, say which one each sits on.
+
+## What every change must keep
+
+1. **Lanes.** Every model decodes through the shared lane rounds, where one forward verifies the drafted tokens
+   together. No serial path, no feature that works only with drafts off, no second batcher beside the engine.
+2. **Exact output.** A drafted reply equals the same engine's `"draft": false` reply, token for token. A resumed
+   prompt equals a fresh one, and each concurrent stream equals its solo run. The README's "Exact decoding" section
+   has the contract.
+3. **No precision traded for speed.** No lower-precision activations or sums, no skipped keys, no pruned tokens. A
+   reorder, or another kernel at equal or better precision, is fine. Say which sums change and that the bits change.
+   A precision mode that a checkpoint format defines is a separate conversation, so open an issue.
+4. **Prompt processing no slower.** Measure cold prompts against the last release for any change that touches
+   prefill, a kernel, the engine, a family or the server.
+5. **Every platform it touches.** That is Metal on Apple Silicon from M1 to M5, and CUDA. Say which you ran and
+   which you could not.
+6. **Lean code.** One job to a module, files under about 600 lines, and one-line comments and docstrings that say
+   what the code can't. Measurements, history and benchmark tables belong in the pull request, not in the source.
+
+## The receipt
+
+Paste these into the pull request. We rerun what we can, and a complete receipt lets a small change land on review
+without waiting for a machine. A change to documentation alone needs no receipt.
+
+- **Environment.** The TensorFold version or commit, the MLX or PyTorch version, the machine and GPU, the checkpoint
+  and its revision.
+- **Exactness.** Drafted against serial, and concurrent against solo, by token hash:
+
+  ```
+  python3 tools/bench_concurrent.py http://127.0.0.1:8080 MODEL --alone --serial
+  ```
+
+  For a server change, also send one request twice, the second with `"draft": false`, and compare `token_sha`.
+- **Decode speed**, before and after, on the same machine in the same session:
+
+  ```
+  python3 tools/bench_openai.py http://127.0.0.1:8080 MODEL
+  ```
+
+- **Prompt speed**, before and after. The tool builds cold prompts of 2k, 8k, 16k, 32k and 64k tokens:
+
+  ```
+  python3 tools/prefill_cold.py build MODEL_DIR prompts.json
+  python3 tools/prefill_cold.py run http://127.0.0.1:8080 MODEL prompts.json out.json
+  ```
+
+- **Tests** you ran, with the pass, fail and skip counts.
+- **What you did not run**, and why. We need this as much as the rest.
+
+When the difference is a few percent, alternate the runs: before, after, after, before. One run each way cannot tell
+a 2% change from a warm machine.
+
+## Running the tests
+
+TensorFold needs Python 3.11 or newer.
+
+```
+python -m pip install -e '.[test,tui]'
+python -m pytest -q tests
+```
+
+On a Mac that is the host suite. The CUDA tests are `tests/cuda` and `tests/test_cuda_*.py`. Those that need PyTorch
+or a GPU skip without one.
+
+A new test skips cleanly where its dependency is missing. Use `pytest.importorskip`, so the suite still collects on
+a machine without PyTorch or without MLX.
+
+A test should fail before your change and pass after it. Say so in the pull request.
+
+## Commits and identity
+
+- Your commits keep your authorship. We record every author under a GitHub noreply address, such as
+  `12345+yourlogin@users.noreply.github.com`, so no personal email lands in the history. "Keep my email addresses
+  private" in your GitHub email settings does this for you.
+- No AI tool attribution lines, and no co-author trailers for tools. We remove them when we land a commit.
+- No personal data, machine names, internal hosts or local paths in code, tests, fixtures or commit messages.
+- A data file built from text, such as an n-gram table, comes from public text only. Name the corpus in the pull
+  request.
+
+## How a pull request lands
+
+We review it, port it onto the release branch ourselves and run it through the release checks. We do not ask you to
+rebase. If `main` has moved, we carry your commits forward.
+
+Your change lands on `main` as your own commit, under your name, inside a release. We then close the pull request
+with a comment that links the release and says what we changed on our side, if anything. GitHub may show the pull
+request as closed and not merged, because the commit that lands is a port of yours. It still carries your name, and
+you appear in the repository's contributors.
+
+Our changes on top of yours, such as a style pass or an extra test, go in separate commits under our name.
+
+When a pull request overlaps work that is already integrated, we close it with a note that says where the work lives
+and credits what your report or measurements added. That is not a rejection, and the "Landing next" issue exists to
+make it rare.
+
+## What we can check ourselves
+
+We test on an M3 Ultra, an M5 Max, one DGX Spark and two together, and an RTX PRO 6000. A change for other hardware
+needs a complete receipt, because we cannot rerun it, and it takes longer to land. Keep such a change in its own
+files where you can, so it cannot affect the paths we do test.
+
+## Reporting a bug
+
+Open an issue with the TensorFold version, the machine, the checkpoint and its revision, the exact command, the
+startup lines the server printed, and what you expected against what happened. A request that reproduces it is worth
+more than a description.
+
+## License
+
+TensorFold is Apache-2.0. A contribution you submit is under that license, as its section 5 says. Model weights keep
+their own licenses.