Row: BACKEND-ROCM
T4a adds the VT_GEMV_MMVQ=1 opt-in K-quant decode GEMV arm for MatmulBTQuant, bit-exact vs the CPU oracle. The arm folds activation quant into the MMVQ GEMV prologue and removes the standalone QuantizeQ8KK launch.
The intended arm covers bf16/f16 activations and bf16/f32 outputs. VT_GEMV_MMVQ_FOLD_MAX makes the fold crossover tunable at runtime. The default path remains unchanged when VT_GEMV_MMVQ is unset.
The current reconstruction is limited to the MMVQ arm and its routing evidence. Separate skinny-GEMM work remains outside this issue.
Depends on the landed #2782 keep-quant GEMM infrastructure.
Row:
BACKEND-ROCMT4a adds the
VT_GEMV_MMVQ=1opt-in K-quant decode GEMV arm forMatmulBTQuant, bit-exact vs the CPU oracle. The arm folds activation quant into the MMVQ GEMV prologue and removes the standaloneQuantizeQ8KKlaunch.The intended arm covers bf16/f16 activations and bf16/f32 outputs.
VT_GEMV_MMVQ_FOLD_MAXmakes the fold crossover tunable at runtime. The default path remains unchanged whenVT_GEMV_MMVQis unset.The current reconstruction is limited to the MMVQ arm and its routing evidence. Separate skinny-GEMM work remains outside this issue.
Depends on the landed #2782 keep-quant GEMM infrastructure.