Add SM72 (Jetson AGX Xavier) CUDA support and TQ2_0 ternary CUDA kernels

[hernandez42]

Problem: Jetson AGX Xavier (SM72 Volta) cannot run Ternary (TQ2_0) with CUDA

Environment

• Device: NVIDIA Jetson AGX Xavier (Tegra194)
• GPU: SM72 Volta gen1 (CUDA 11.4) — not Tesla V100 (SM70)
• Tested: Ternary-Bonsai-27B TQ2_0 loaded in CPU-only mode → 0.8 t/s
• Same board with factory llama.cpp CUDA + 35B MoE Q3_K_M: 19.4 t/s (24× faster)

Two distinct bugs

Bug 1 — SM72 not in CUDA arch list

Jetson AGX Xavier uses SM72 (Volta with gen1 tensor cores). The CMAKE_CUDA_ARCHITECTURES list in ggml/src/ggml-cuda/CMakeLists.txt covers SM70 (V100), SM75 (Turing), SM80 (Ampere) but has no SM72 entry:

list(APPEND CMAKE_CUDA_ARCHITECTURES 50-virtual 61-virtual 70-virtual)
list(APPEND CMAKE_CUDA_ARCHITECTURES 75-virtual 80-virtual 86-real)

SM72 is the only Volta variant still in active deployment (Jetson edge). Desktop V100 (SM70) is practically obsolete.

Adding 72-real (or 72-virtual) is required for Jetson Xavier builds. Without it, CUDA compilation either fails or silently falls back to CPU.

Bug 2 — TQ1_0/TQ2_0 ternary types have zero CUDA kernel support

Ternary types are defined in ggml/include/ggml.h:

GGML_TYPE_TQ1_0   = 34,
GGML_TYPE_TQ2_0   = 35,

CPU implementation exists via ggml-quants.c (quantize_tq2_0, ggml_row_size dispatch). But grep for TQ1_0 or TQ2_0 in ggml/src/ggml-cuda/ returns zero results.

Standard GGML_TYPE_Q2_0 (=42, standard 2-bit) has full CUDA support: dequantize.cuh, getrows.cu, common.cuh, mmq.cu, mmvq.cu, vecdotq.cuh, convert.cu. The ternary variant (TQ2_0) has no equivalent.

Metal backend also lacks TQ1_0/TQ2_0 support, though that's secondary.

Why this matters

Ternary quantization is the whole point of PrismML's fork. On Jetson-class edge devices (Orin, Xavier), being stuck at CPU-only 0.8 t/s vs GPU-accelerated 19 t/s means the difference between "can run" and "is actually usable."

Suggested fix

1. Add 72-real or 72-virtual to CMAKE_CUDA_ARCHITECTURES in ggml/src/ggml-cuda/CMakeLists.txt
2. Create ternary CUDA kernels in ggml-cuda/ — at minimum:
   • template-instances/mmq-instance-tq2_0.cu (analogous to mmq-instance-q2_0.cu)
   • Register GGML_TYPE_TQ1_0 and GGML_TYPE_TQ2_0 in common.cuh with type traits
   • Add dispatch entries in getrows.cu, mmq.cu, mmvq.cu, convert.cu, ggml-cuda.cu
3. A simpler interim path: dequantize ternary → standard Q2_0/f32 and delegate to existing CUDA matmul

We have an AGX Xavier on hand and can test CUDA builds if you provide instructions.