cuda(quant): Q2_K CPU dequant + Q3_K / Q4_1 / Q5_1 CPU+CUDA dequant

[jamesburton]

Summary

Closes #156 — extends K-quant family coverage on both CPU and CUDA paths.

Changes

Area	Files	Net
Core	QuantizationType.cs, QuantizationTypeExtensions.cs	enum + helper additions
CPU	Dequantize.cs, DequantizeKQuants.cs	Q2_K + Q3_K/Q4_1/Q5_1 dequant
CUDA	dequant.cu + .ptx, CudaKernels.cs, CudaModule.cs	Q3_K/Q4_1/Q5_1 device dequant + dispatch + module helper
Tests	DequantizeKQuantTests.cs	hand-calc + RowByteSize + stride + non-aligned per format
Total	~9 files	+913 / -49

Implementation notes

• Q2_K is CPU-only in this PR; CUDA Q2_K lands as part of the upcoming K-quant MMQ rollout.
• Q3_K / Q4_1 / Q5_1 ship CPU + CUDA together.
• Adds CudaModule.TryGetFunction(string) helper (returns 0 on CUDA_ERROR_NOT_FOUND, matches the existing guard pattern used by CudaKernels.cs lookup paths) — required because the new CudaKernels.cs dispatch uses it.

Foundation for follow-ups

This PR is the foundation for the CUDA K-quant MMQ / MMVQ rollout (Q4_K MMQ, Q5_K/Q6_K/Q8_0 MMQ variants, pre-Q8_1 paths) and the i-quant family (IQ1_S, IQ2_, IQ3_, IQ4_*).

Verification

• dotnet build src/DotLLM.Cpu -c Release — 0 warnings, 0 errors.
• dotnet build src/DotLLM.Cuda -c Release — 0 warnings, 0 errors.
• dotnet test --filter "FullyQualifiedName~Dequantize|FullyQualifiedName~Q2_K|FullyQualifiedName~Q3_K|FullyQualifiedName~Q4_1|FullyQualifiedName~Q5_1" — 38 / 38 passing.

Closes #156

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds support for additional quantization formats across CPU and CUDA dequantization paths, including new CUDA kernels and unit tests for K-quant formats.

Changes:

• Add CPU dequantization support for Q2_K, Q3_K, Q4_1, and Q5_1 (plus RowByteSize updates).
• Add CUDA kernels and runtime dispatch for Q4_1, Q5_1, and optional Q3_K (with a “try get” symbol lookup).
• Add unit tests for Q2_K and Q3_K dequantization and stride sizing.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
tests/DotLLM.Tests.Unit/Cpu/Kernels/DequantizeKQuantTests.cs	Adds hand-calculated and sizing/validation tests for Q2_K & Q3_K.
src/DotLLM.Cuda/CudaModule.cs	Adds TryGetFunction to support optional kernels and cache lookups.
src/DotLLM.Cuda/CudaKernels.cs	Adds kernel handles and launch paths for Q4_1/Q5_1/Q3_K dequant.
src/DotLLM.Cpu/Kernels/DequantizeKQuants.cs	Implements Q2_K and Q3_K CPU dequantization.
src/DotLLM.Cpu/Kernels/Dequantize.cs	Adds RowByteSize mappings and scalar dequant for Q4_1/Q5_1; dispatch wiring.
src/DotLLM.Core/Configuration/QuantizationTypeExtensions.cs	Extends byte size mapping (adds Q3_K).
src/DotLLM.Core/Configuration/QuantizationType.cs	Adds enum values for Q2_K and Q3_K.
native/ptx/dequant.ptx	Updates compiled PTX (new targets/version) and adds new kernel entry points.
native/kernels/dequant.cu	Adds CUDA kernels for Q4_1, Q5_1, and Q3_K dequantization.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jamesburton]

Thanks — all seven comments were valid; fixed in 280e7ba.

native/ptx/dequant.ptx:5 + :10 — sm_75 / PTX ISA 9.1 compatibility break

You're right, and it was unintentional: the repo's own native/build.ps1 and build.sh pin -arch=compute_61, so the checked-in file couldn't have come from them. Somebody regenerated it with a CUDA 13.1 toolkit, which defaults to sm_75, and the header change rode along with the real kernel additions.

Rather than ship multiple PTX variants, I restored the documented baseline: regenerated from the unchanged dequant.cu with nvcc 12.8 -arch=compute_61, yielding .target sm_61 / .version 8.7 and the same nine .visible .entry kernels. Validated by assembling the result with ptxas -arch=sm_61 and ptxas -arch=sm_86 (CUDA 12.8) — both clean, confirming nothing in these kernels needs sm_70+.

One residual: .version tracks the toolkit, so this is 8.7 (CUDA 12.8) where the pre-PR file was 8.5 (CUDA 12.6). That's a much smaller driver-floor move than 9.1, and I'd rather not hand-edit a generated artifact — happy to regenerate on a 12.6 toolkit if you want the exact previous floor.

To stop this recurring, added PtxTargetTests, which asserts every checked-in native/ptx/*.ptx declares .target sm_61 and points at build.ps1 in the failure message. A future default-arch regeneration now fails a test instead of shipping.

CudaModule.cs:91 — miss cached as 0 corrupts GetFunction

Confirmed: GetFunction starts with _functions.TryGetValue, so after TryGetFunction("absent") it would return 0 for that name instead of throwing — a null kernel handle passed straight to cuLaunchKernel. Took option (1): misses go into a separate HashSet<string> _missingFunctions, never into _functions, so GetFunction still performs a real lookup and throws. Both are cleared on Dispose.

CudaModule.cs:82 — hard-coded 500

Agreed. There was no CUDA result enum anywhere in the interop layer, so I added a small CudaResult class next to CudaErrorHelper with Success and NotFound, and the check now reads result == CudaResult.NotFound.

QuantizationTypeExtensions.cs:26 — Q2_K missing

Correct — the enum gained Q2_K but ComputeByteCount only gained Q3_K, so Q2_K fell through to the ArgumentOutOfRangeException default. Added QuantizationType.Q2_K => elementCount / 256 * 84, matching Q2_K_BlockBytes, plus a test asserting it agrees with Dequantize.RowByteSize(…, Q2_K) so the two can't drift.

DequantizeKQuants.cs:333 — (int)(destOffset + e) narrowing

Agreed. Note the same pattern is in DequantizeQ2_K ((int)(sb * KQuantGroupSize)), so suggestion (2) alone wouldn't have closed it. Did both halves: DequantizeQ3_KScalar now carries an int destOffset, and both DequantizeQ2_K and DequantizeQ3_K validate dest.Length >= elementCount on entry. Since dest.Length is an int, that guard is what makes the int index provably safe — and it matters because these are reachable directly, not only through ToFloat32 (which already had the check).

Dequantize.cs:94 — no Q4_1 / Q5_1 tests

Fair; the test file only tracked the K-quant additions. Added the coverage you listed for both formats:

• hand-calculated single-block decode, derived from the block layout rather than from the implementation — including the low-nibble→element j / high-nibble→element j+16 split and, for Q5_1, the qh bit j vs bit j+16 split;
• two-block stride tests that specifically catch Q4_0/Q5_0 block-size confusion (18 vs 20, 22 vs 24 bytes);
• RowByteSize and non-aligned-count-throws.
• Q5_1 also gets an all-bits-set case pinning the max 5-bit code at 31.

Tests: --filter "FullyQualifiedName~Dequantize|FullyQualifiedName~QuantizationType|FullyQualifiedName~PtxTarget" → 49 passed, 0 failed.

[jamesburton]

Correction to my own PR: as submitted, this would have shipped a broken Q3_K dequantizer in both backends. Pushed 46af747 to fix it before merge.

The Q3_K decoder here reads a transposed layout. Two independent transpositions, both against ggml-quants.c dequantize_row_q3_K:

1. Scale high bits. I had:

int hiBits = (scales12[8 + (sub >> 2)] >> ((sub & 3) * 2)) & 0x03;   // 8 + sub/4 @ (sub%4)*2

llama.cpp's aux/kmask shuffle packs them the other way round — byte 8 + (sub % 4), shift (sub / 4) * 2:

int hiBits = (scales12[8 + (sub & 3)] >> ((sub >> 2) * 2)) & 0x03;

The two forms agree only for sub ∈ {0, 5, 10, 15}, so 12 of the 16 sub-block scales were wrong. Relatedly, the low-nibble source for sub 8..15 was sub - 4 rather than sub - 8, i.e. it read bytes 8..11 — the high-2-bits bytes — as if they held nibble scales.

2. Element ordering. The 2-bit quants are not stored four consecutive elements per byte. Each 128-element half uses 32 qs bytes and each byte supplies four elements 32 apart: element t is bit-pair (t/32)%4 of qs[(t%32) + 32*(t/128)], with high bit t/32 of hmask[t%32]. I had qs[t/4] >> (t%4)*2 and hmask[t/8] >> t%8, which scatters every element of every super-block into the wrong sub-block scale.

Combined, Q3_K weights came out as effectively noise. Correlating a Q3_K-decoded tensor against the Q8_0 build of the same model: corr 0.006 before the fix, 0.988 after.

Why the existing tests were green

Q3_K_SingleBlock_HandCalculated touches only elements 0 and 1 of sub-block 0, plus an all-zero sub-block 1. That is precisely the degenerate case where the correct and the transposed layouts coincide (sub = 0 is one of the four agreeing sub-blocks; qs/hmask byte 0 bit 0 is the same element under either indexing). It could not have failed, no matter how wrong the layout was — which is exactly how this got as far as a PR.

Added Q3_K_DenseRandomBlocks_MatchLlamaCppReference: dense pseudorandom super-block bytes through Dequantize.ToFloat32, compared for exact float equality against a literal transcription of llama.cpp's dequantize_row_q3_K — the 32-bit aux/kmask scale shuffle and the shift/m loop over 128-element halves, kept in llama.cpp's own control-flow shape on purpose so it is structurally unlike the production kernel's closed-form indexing. Agreement is then evidence rather than a shared-mistake tautology.

Discrimination checked both ways, each half reverted alone:

state	result
scale transposition only	red — element 16 (block 0, sub 1, lane 0): reference -14.25, dotLLM 9.75
element-ordering transposition only	red — element 1 (block 0, sub 0, lane 1): reference 0, dotLLM -25
both fixed	green — 50/50 passing (DequantizeKQuant, Q3_K, DequantizeTests, PtxTarget)

One thing still outstanding: native/ptx/dequant.ptx

The checked-in PTX is not regenerated and still contains the buggy Q3_K kernel. I could not rebuild it and did not want to hand-forge or silently commit a stale artifact.

The blocker is the compute_61 baseline this PR added PtxTargetTests to guard: CUDA 13.x dropped Pascal (nvcc fatal : Unsupported gpu architecture 'compute_61'), so it needs a CUDA 12.x toolkit, and no machine I have access to has one with a working host compiler. The .cu fix is correct and reviewable as-is, but native/build.ps1 needs to be run on a box with CUDA 12.x before the CUDA Q3_K path is trustworthy. Happy for whoever has that toolkit to push the regenerated PTX onto this branch, or I can split it into a follow-up.