mudler · lu-zero · Sep 13, 2026 · Sep 13, 2026
diff --git a/.agents/issues/QUANT-GGUF-IQ-TENSTORRENT/ISSUE-LOCAL-01M2CNC22SYAKJN3YBGNVGSG56.md b/.agents/issues/QUANT-GGUF-IQ-TENSTORRENT/ISSUE-LOCAL-01M2CNC22SYAKJN3YBGNVGSG56.md
@@ -0,0 +1,19 @@
+ID: ISSUE-LOCAL-01M2CNC22SYAKJN3YBGNVGSG56
+Title: TENSTORRENT: serve the APEX-I-Nano IQ-quant family (IQ3_XXS/IQ2_S/IQ2_XXS + Q3_K keep-quant decode)
+Row: QUANT-GGUF-IQ-TENSTORRENT
+State: OPEN
+Kind: feature
+GitHub: -
+Mirror: PENDING
+Availability: FULL
+Created: 2026-09-13
+Updated: 2026-09-13
+Closed: -
+
+## Problem
+
+Ettore's mudler/Qwen3.8-27B-APEX-I-Nano.gguf (10.7 GB) packs the 27B into the P150's 32 GiB with real headroom, but its tensor census (llama-gguf, /tmp/apex-gguf-dump.txt, 2026-09-13) is IQ-dominant: 164 IQ3_XXS + 89 IQ2_S + 44 IQ2_XXS + 78 Q3_K tensors, against only 122 Q4_K + 8 Q8_0 + 1 Q6_K. The TENSTORRENT keep-quant registered set (MatmulBTQuantKernel, tenstorrent_ops.cpp) is exactly {Q4_K, Q5_K, Q6_K, Q8_0}, so the artifact refuses by name on the P150 today. This row tracks the per-encoding device waves (each encoding needs an on-core keep-quant decode chain — the vec_dot semantics — plus route inclusion and the W4a wave-2b test pattern); the CPU-side states live on the per-encoding rows (IQ3_XXS READY, IQ2_XXS ACTIVE, IQ2_S INVENTORIED — its CPU dot is a prerequisite, Q3_K PARTIAL). The e2e gate is the APEX file generating on the P150 against the pinned oracle, with the artifact documented in docs/USAGE.md per the weights rule. Ordering note: the Q4_K_M 27B verdict (W3/W4, in flight) takes the device first — this row starts after that lands or in parallel on the CPU-side prerequisites.
+
+## Resolution
+
+-
diff --git a/.agents/issues/QUANT-GGUF-IQ2_S/ISSUE-LOCAL-01M2CNCPN3TJ2EVR8M4CKKJ89P.md b/.agents/issues/QUANT-GGUF-IQ2_S/ISSUE-LOCAL-01M2CNCPN3TJ2EVR8M4CKKJ89P.md
@@ -0,0 +1,19 @@
+ID: ISSUE-LOCAL-01M2CNCPN3TJ2EVR8M4CKKJ89P
+Title: TENSTORRENT: keep-quant decode chain for IQ2_S (APEX-I-Nano wave)
+Row: QUANT-GGUF-IQ2_S
+State: OPEN
+Kind: feature
+GitHub: -
+Mirror: PENDING
+Availability: FULL
+Created: 2026-09-13
+Updated: 2026-09-13
+Closed: -
+
+## Problem
+
+The APEX-I-Nano census (tracker ISSUE-LOCAL-01M2CNC22SYAKJN3YBGNVGSG56, row QUANT-GGUF-IQ-TENSTORRENT) carries IQ2_S tensors, but the TENSTORRENT keep-quant registered set (MatmulBTQuantKernel) is {Q4_K, Q5_K, Q6_K, Q8_0} — the encoding refuses by name on the P150. Wave: the on-core keep-quant decode chain for IQ2_S (vec_dot semantics mirroring vt::cpu::BlockVecDot), route inclusion (KeepQuantDType/DeviceKeepQuantSupported), the W4a wave-2b test pattern (op-level + bit-exact sweep vs vt::cpu::BlockVecDot), and census-driven reach on mudler/Qwen3.8-27B-APEX-I-Nano. Context: the IQ2_S CPU dot is a PREREQUISITE (the row is INVENTORIED — the device wave cannot start before the CPU vec_dot lands on its own row). The tracker owns ordering; the Q4_K_M 27B verdict (W3/W4, in flight) takes the device first.
+
+## Resolution
+
+-
diff --git a/.agents/issues/QUANT-GGUF-IQ2_XXS/ISSUE-LOCAL-01M2CNCNZPK90NKQKBCK0TS66D.md b/.agents/issues/QUANT-GGUF-IQ2_XXS/ISSUE-LOCAL-01M2CNCNZPK90NKQKBCK0TS66D.md
@@ -0,0 +1,19 @@
+ID: ISSUE-LOCAL-01M2CNCNZPK90NKQKBCK0TS66D
+Title: TENSTORRENT: keep-quant decode chain for IQ2_XXS (APEX-I-Nano wave)
+Row: QUANT-GGUF-IQ2_XXS
+State: OPEN
+Kind: feature
+GitHub: -
+Mirror: PENDING
+Availability: FULL
+Created: 2026-09-13
+Updated: 2026-09-13
+Closed: -
+
+## Problem
+
+The APEX-I-Nano census (tracker ISSUE-LOCAL-01M2CNC22SYAKJN3YBGNVGSG56, row QUANT-GGUF-IQ-TENSTORRENT) carries IQ2_XXS tensors, but the TENSTORRENT keep-quant registered set (MatmulBTQuantKernel) is {Q4_K, Q5_K, Q6_K, Q8_0} — the encoding refuses by name on the P150. Wave: the on-core keep-quant decode chain for IQ2_XXS (vec_dot semantics mirroring vt::cpu::BlockVecDot), route inclusion (KeepQuantDType/DeviceKeepQuantSupported), the W4a wave-2b test pattern (op-level + bit-exact sweep vs vt::cpu::BlockVecDot), and census-driven reach on mudler/Qwen3.8-27B-APEX-I-Nano. Context: the IQ2_XXS CPU side is ACTIVE. The tracker owns ordering; the Q4_K_M 27B verdict (W3/W4, in flight) takes the device first.
+
+## Resolution
+
+-
diff --git a/.agents/issues/QUANT-GGUF-IQ3_XXS/ISSUE-LOCAL-01M2CNCNA30WWNBRA5JBC7MSMY.md b/.agents/issues/QUANT-GGUF-IQ3_XXS/ISSUE-LOCAL-01M2CNCNA30WWNBRA5JBC7MSMY.md
@@ -0,0 +1,19 @@
+ID: ISSUE-LOCAL-01M2CNCNA30WWNBRA5JBC7MSMY
+Title: TENSTORRENT: keep-quant decode chain for IQ3_XXS (APEX-I-Nano wave)
+Row: QUANT-GGUF-IQ3_XXS
+State: OPEN
+Kind: feature
+GitHub: -
+Mirror: PENDING
+Availability: FULL
+Created: 2026-09-13
+Updated: 2026-09-13
+Closed: -
+
+## Problem
+
+The APEX-I-Nano census (tracker ISSUE-LOCAL-01M2CNC22SYAKJN3YBGNVGSG56, row QUANT-GGUF-IQ-TENSTORRENT) carries IQ3_XXS tensors, but the TENSTORRENT keep-quant registered set (MatmulBTQuantKernel) is {Q4_K, Q5_K, Q6_K, Q8_0} — the encoding refuses by name on the P150. Wave: the on-core keep-quant decode chain for IQ3_XXS (vec_dot semantics mirroring vt::cpu::BlockVecDot), route inclusion (KeepQuantDType/DeviceKeepQuantSupported), the W4a wave-2b test pattern (op-level + bit-exact sweep vs vt::cpu::BlockVecDot), and census-driven reach on mudler/Qwen3.8-27B-APEX-I-Nano. Context: the IQ3_XXS CPU side is READY. The tracker owns ordering; the Q4_K_M 27B verdict (W3/W4, in flight) takes the device first.
+
+## Resolution
+
+-
diff --git a/.agents/issues/QUANT-GGUF-Q3_K/ISSUE-LOCAL-01M2CNCQAF1ZDD274C90T1W32H.md b/.agents/issues/QUANT-GGUF-Q3_K/ISSUE-LOCAL-01M2CNCQAF1ZDD274C90T1W32H.md
@@ -0,0 +1,19 @@
+ID: ISSUE-LOCAL-01M2CNCQAF1ZDD274C90T1W32H
+Title: TENSTORRENT: keep-quant decode chain for Q3_K (APEX-I-Nano wave)
+Row: QUANT-GGUF-Q3_K
+State: OPEN
+Kind: feature
+GitHub: -
+Mirror: PENDING
+Availability: FULL
+Created: 2026-09-13
+Updated: 2026-09-13
+Closed: -
+
+## Problem
+
+The APEX-I-Nano census (tracker ISSUE-LOCAL-01M2CNC22SYAKJN3YBGNVGSG56, row QUANT-GGUF-IQ-TENSTORRENT) carries Q3_K tensors, but the TENSTORRENT keep-quant registered set (MatmulBTQuantKernel) is {Q4_K, Q5_K, Q6_K, Q8_0} — the encoding refuses by name on the P150. Wave: the on-core keep-quant decode chain for Q3_K (vec_dot semantics mirroring vt::cpu::BlockVecDot), route inclusion (KeepQuantDType/DeviceKeepQuantSupported), the W4a wave-2b test pattern (op-level + bit-exact sweep vs vt::cpu::BlockVecDot), and census-driven reach on mudler/Qwen3.8-27B-APEX-I-Nano. Context: the Q3_K CPU side is PARTIAL — complete it on its own row first. The tracker owns ordering; the Q4_K_M 27B verdict (W3/W4, in flight) takes the device first.
+
+## Resolution
+
+-
diff --git a/.agents/quantization-matrix.md b/.agents/quantization-matrix.md
@@ -77,6 +77,7 @@ forces the full dequant path back.
 | `QUANT-GGUF-IQ2_S` | IQ2_S | 22 / IQ2_M storage | Y | - | - | - | - | `INVENTORIED` | reader `:229-234`; [explicit rejection](../tests/vllm/test_gguf_dequant.cpp#L223) | leaf open | - |
 | `QUANT-GGUF-IQ4_XS` | IQ4_XS | 23 / output | Y | Y | Y | - | - | `PARTIAL` | reader trait [`FindGgmlTraits` case 23](../src/vllm/model_executor/model_loader/gguf_reader.cpp#L311) (136 B block; the trait predates the decoder, the decoder is [#2240](https://github.com/mudler/vllm.cpp/issues/2240)); [dequant kernel](../src/vt/cpu/cpu_quant_dequant.cpp#L488) `DequantIQ4_XS`, ported 1:1 from llama.cpp `b10451` `ggml/src/ggml-quants.c:2743`; vt block dtype `kIQ4_XS` [geometry](../src/vt/dtype.cpp#L140). NOT a codebook delta from IQ4_NL — it reuses `kValuesIq4nl` unchanged and differs only in the SUPER-BLOCK SCALE LAYOUT (a 6-bit `ls` spliced from a `scales_l` nibble and a `scales_h` bit pair, then biased by -32). **`M` = `Y`, gated BYTE-FOR-BYTE against the oracle** over REAL checkpoint bytes: 4 blocks of `blk.11.ffn_down_exps.weight` from the staged `unsloth/GLM-5.3-Flash-GGUF` UD-Q2_K_XL arm, whose four super-blocks span `ls` 0..61 so both nibbles and both `scales_h` bit pairs and both signs of `dl` are exercised ([goldens + provenance](../tests/vt/iq2xs_iq4xs_golden_vectors.h#L276), [unit](../tests/vllm/test_gguf_dequant.cpp#L576)). [Geometry cross-check](../tests/vt/test_ops_quant_traits.cpp#L172) pins the reader and vt tables against each other. The former `explicit rejection` evidence is GONE by construction — that test now guards Q1_0 (41), the remaining tabulated-but-undecodable id. **`C` = `Y` since [#2247](https://github.com/mudler/vllm.cpp/issues/2247):** keep-quant [`VecDotIQ4_XSQ8_K`](../src/vt/cpu/cpu_quant_dot.cpp#L844), ported 1:1 from llama.cpp `b10451` `ggml/src/ggml-cpu/quants.c:1283` `ggml_vec_dot_iq4_xs_q8_K_generic`, with [`BlockIQ4_XS`](../src/vt/cpu/cpu_quant_blocks.h#L205) (`ggml-common.h:454-460`) and the [traits row](../src/vt/cpu/cpu_quant_traits.cpp#L121). **The activation pairing is `Q8_K`, READ OFF the oracle and not inherited from IQ4_NL:** `type_traits_cpu` at `ggml/src/ggml-cpu/ggml-cpu.c:385-390` carries `.vec_dot = ggml_vec_dot_iq4_xs_q8_K, `.vec_dot_type = GGML_TYPE_Q8_K`, against `:379-384`'s `GGML_TYPE_Q8_0` for IQ4_NL — the 16-entry codebook is shared, the block geometry is not, and a 256-element super-block pairs with the 256-element activation encoding. **Gated BIT FOR BIT against the oracle's OWN KERNEL** on 4 super-blocks of `blk.11.ffn_down_exps.weight`, total and per block ([goldens](../tests/vt/iq2xs_iq4xs_dot_golden.h#L269), [case](../tests/vt/test_ops_quant_dot.cpp#L869)). These 3 tensors stop expanding from 3.59 GiB to 13.50 GiB; the whole-artifact measurement is in the `QUANT-GGUF-IQ2_XS` row above. **It also unblocks a SECOND consumer:** the expert-tower streaming lane is all-or-nothing (`GgufExpertTowersReachSlotLane` returns false on the FIRST tower that does not keep), so the 4 IQ4_XS tensors of the GLM-5.3 `UD-IQ1_S` arm were dropping an arm whose other 221 towers all kept — one tower 6.375 -> 24.000 GiB, and a 4096-slot cache 25.5 -> 96 GiB. Asserted in the `kStackedExpertWeight` role both models store these towers in ([lane case](../tests/vllm/model_executor/test_gguf_device_fit.cpp#L672)). **The CUDA arm now HAS a keep-quant kernel for this dtype too** ([#2260](https://github.com/mudler/vllm.cpp/issues/2260), spec [cuda-keepquant-iq2xs-iq4xs](specs/cuda-keepquant-iq2xs-iq4xs.md)): `src/vt/cuda/cuda_quant_dot.cu::DotIQ4XS` is a 1:1 port of `b10451 quants.c:1283` over the generated `src/vt/cuda/cuda_quant_iq_tables.cuh::d_kvalues_iq4nl`, keeping upstream's EIGHT f32 accumulation steps per super-block rather than one integer core, because that association is what the oracle's golden numbers were produced with. **This completes the device admission of the GLM-5.3 non-flash `UD-IQ1_S` arm**, whose other five encodings (IQ1_S, IQ3_XXS, IQ2_XXS, Q2_K, Q3_K) were already there. Before it, these 3 tensors fit `dgx:gpu0` with their expert GEMM on the CPU fallback behind a `cudaStreamSynchronize` and the fused `src/vt/cuda/cuda_quant_dot.cu::MoeGateUpSwiGLUGroupedCuda` seam throwing. The fit remains RESIDENCY and no speed number is claimed; O19's premise in [glm5-next-flash](specs/glm5-next-flash.md) is discharged, and what still blocks a CUDA run of either model is that model's own host-f32 forward. | [glm5-next-flash](specs/glm5-next-flash.md) | - |
 | `QUANT-GGUF-IQ1_M` | IQ1_M | 29 / output | - | - | - | - | - | `INVENTORIED` | - | leaf open | - |
+| `QUANT-GGUF-IQ-TENSTORRENT` | IQ3_XXS / IQ2_S / IQ2_XXS (+ Q3_K) on TENSTORRENT | 18 / 22 / 16 (+11) / output | Y | Y | Y | - | - | `INVENTORIED` | the APEX-I-Nano census (2026-09-13, `/tmp/apex-gguf-dump.txt`): 297 of 2,598 tensors are IQ3_XXS/IQ2_S/IQ2_XXS and 78 are Q3_K — none in the TENSTORRENT keep-quant registered set {Q4_K, Q5_K, Q6_K, Q8_0} (tenstorrent_ops.cpp `MatmulBTQuantKernel`), so `mudler/Qwen3.8-27B-APEX-I-Nano` refuses by name on the P150. Tracker issue owns the per-encoding waves; CPU-side states live on the per-encoding rows | this spec's tracker issue | - |
 | `QUANT-GGUF-BF16` | BF16 | 30 / output | Y | - | - | - | - | `INVENTORIED` | reader `:261-264`; executable path absent | leaf open | - |
 | `QUANT-GGUF-TQ1_0` | TQ1_0 | 34 / output | - | - | - | - | - | `INVENTORIED` | - | leaf open | - |
 | `QUANT-GGUF-TQ2_0` | TQ2_0 | 35 / output | - | - | - | - | - | `INVENTORIED` | - | leaf open | - |