llama-quant : fail early on missing imatrix, refactor type selection, code cleanup by ddh0 · Pull Request #19770 · ggml-org/llama.cpp

ddh0 · 2026-02-20T22:49:34Z

Currently, if a quantization requires an importance matrix and one isn't provided, the program doesn't discover this until it reaches the offending tensor during the main quantization loop. Depending on model size and target type, this can mean wasting minutes to hours before the process aborts with a partial GGUF.

This PR adds a preliminary metadata pass over all tensors that determines target types upfront, enabling early validation. The quantization logic in llama-quant.cpp is refactored for clarity and correctness.

Fail early for missing required imatrix

A preliminary pass now computes each tensor's target quantization type before the main loop begins. If an importance matrix is required but missing, quantization fails immediately with an error identifying the offending tensor and its target type:

The old ftype-based imatrix guard in quantize.cpp is removed in favor of this per-tensor check. tensor_requires_imatrix (renamed from tensor_type_requires_imatrix) now uses a switch on dst_type and correctly exempts per_layer_token_embd.weight in addition to token_embd.weight.

Refactoring

Extracted functions to reduce the size of llama_model_quantize_impl and make the logic reusable across the preliminary and main passes, and to improve clarity and maintainability:

tensor_allows_quantization: consolidates all "should we quantize this tensor?" checks (norm tensors, RWKV weights, conv1d, positional embeddings, etc.) previously inlined in the main loop
tensor_category + tensor_get_category: replaces repeated name.find(...) calls with a single categorization pass; used by type selection and the attention-v check (category_is_attn_v)
llama_tensor_get_type / llama_tensor_get_type_impl: splits type resolution into a wrapper (manual overrides, token_embedding/output overrides, fallbacks) and the core mixture/architecture logic
tensor_type_fallback: extracted from the inline incompatible-shape handling block; now also handles the rare case where the fallback type itself is incompatible (falls back to F16 with a warning)
llama_ftype_get_default_type: the ftype -> ggml_type switch, extracted and organized by category
tensor_name_match_token_embd / tensor_name_match_output_weight: small helpers used across multiple call sites

Other changes

Regex patterns for --tensor-type compiled once in the quantize_state_impl constructor instead of compiled per-tensor, per-flag
tensor_quantization struct moved from llama-quant.cpp to the header (shared with quantize.cpp via #include instead of duplicated)
~~tensor_category enum and tensor_metadata struct added to the header~~
has_output replaced with has_tied_embeddings (clearer semantics, inverted logic, same behavior)
n_k_quantized counter removed (fallback count now reported against ml.n_tensors)
Removed dead MXFP4 sanity check code (#if 0 block)
Some logging cleanup

ddh0 · 2026-02-26T06:19:52Z

I am lucky to have had helpful feedback from @ubergarm, @AesSedai, @aldehir, @pwilkin, @compilade, and @bartowski - they all deserve a mention here. Thanks :) 🦙

ddh0 · 2026-02-28T05:14:41Z

Marking back as draft until things have settled down a bit.

ddh0 · 2026-03-04T01:37:56Z

cc @ggerganov, @CISC - sorry for the messy thread, and for the re-marking as draft; there were previously a lot more changes here, but I'm going to leave some of my more aspirational improvements for future PRs.

I would like to get your thought on the changes here, as to whether or not they are acceptable overall.

ggerganov

Overall the refactoring goals are good. Double-check that there aren't any functional changes by comparing a few quantizations before/after and we can merge.

ggerganov · 2026-03-04T10:09:27Z

 #include "common.h"
 #include "llama.h"
 #include "gguf.h"
+#include "../src/llama-quant.h"


Is this needed?

Yes, though right now it's just for the struct tensor_quantization which was previously duplicated between quantize.cpp and llama-quant.cpp.

(edit: replying to ggerganov)

Including internal sources from libllama is not OK, so better to keep the duplicated structs.

Generally, passing C++ structs across the C-style API is also not OK. This API has to be refactored at some point (#12511 (comment)).

I understand, thanks. Will revert this change.

Reverted in 4b7ebed.

@bartowski

In ggml-org#19770, I introduced a regression in the way the `quantize_state_impl` counter values were initialized. I was incrementing and using `n_attention_wv` in the same loop, when it should have been fixed by the time we're deciding tensor types in `llama_tensor_get_type_impl` (for `use_more_bits`). I never observed a difference in any of [my tests](ggml-org#19770 (comment)) - it was only after @bartowski kindly pointed this out that I realized it was incorrect. (Thanks!)

@bartowski

* llama-quant : correct `n_attention_wv` usage In #19770, I introduced a regression in the way the `quantize_state_impl` counter values were initialized. I was incrementing and using `n_attention_wv` in the same loop, when it should have been fixed by the time we're deciding tensor types in `llama_tensor_get_type_impl` (for `use_more_bits`). I never observed a difference in any of [my tests](#19770 (comment)) - it was only after @bartowski kindly pointed this out that I realized it was incorrect. (Thanks!) * simplify

… code cleanup (ggml-org#19770) * quantize : imatrix-fail early + code cleanup * fix manual override printing it's in the preliminary loop now, so needs to be on its own line * revert header changes per ggerganov * remove old #includes * clarify naming rename `tensor_quantization` to `tensor_typo_option` to descirbe its functionality * fix per barto

@bartowski

* llama-quant : correct `n_attention_wv` usage In ggml-org#19770, I introduced a regression in the way the `quantize_state_impl` counter values were initialized. I was incrementing and using `n_attention_wv` in the same loop, when it should have been fixed by the time we're deciding tensor types in `llama_tensor_get_type_impl` (for `use_more_bits`). I never observed a difference in any of [my tests](ggml-org#19770 (comment)) - it was only after @bartowski kindly pointed this out that I realized it was incorrect. (Thanks!) * simplify

… code cleanup (ggml-org#19770) * quantize : imatrix-fail early + code cleanup * fix manual override printing it's in the preliminary loop now, so needs to be on its own line * revert header changes per ggerganov * remove old #includes * clarify naming rename `tensor_quantization` to `tensor_typo_option` to descirbe its functionality * fix per barto

@bartowski

* llama-quant : correct `n_attention_wv` usage In ggml-org#19770, I introduced a regression in the way the `quantize_state_impl` counter values were initialized. I was incrementing and using `n_attention_wv` in the same loop, when it should have been fixed by the time we're deciding tensor types in `llama_tensor_get_type_impl` (for `use_more_bits`). I never observed a difference in any of [my tests](ggml-org#19770 (comment)) - it was only after @bartowski kindly pointed this out that I realized it was incorrect. (Thanks!) * simplify

… code cleanup (ggml-org#19770) * quantize : imatrix-fail early + code cleanup * fix manual override printing it's in the preliminary loop now, so needs to be on its own line * revert header changes per ggerganov * remove old #includes * clarify naming rename `tensor_quantization` to `tensor_typo_option` to descirbe its functionality * fix per barto

@bartowski

* llama-quant : correct `n_attention_wv` usage In ggml-org#19770, I introduced a regression in the way the `quantize_state_impl` counter values were initialized. I was incrementing and using `n_attention_wv` in the same loop, when it should have been fixed by the time we're deciding tensor types in `llama_tensor_get_type_impl` (for `use_more_bits`). I never observed a difference in any of [my tests](ggml-org#19770 (comment)) - it was only after @bartowski kindly pointed this out that I realized it was incorrect. (Thanks!) * simplify

… code cleanup (ggml-org#19770) * quantize : imatrix-fail early + code cleanup * fix manual override printing it's in the preliminary loop now, so needs to be on its own line * revert header changes per ggerganov * remove old #includes * clarify naming rename `tensor_quantization` to `tensor_typo_option` to descirbe its functionality * fix per barto

@bartowski

* llama-quant : correct `n_attention_wv` usage In ggml-org#19770, I introduced a regression in the way the `quantize_state_impl` counter values were initialized. I was incrementing and using `n_attention_wv` in the same loop, when it should have been fixed by the time we're deciding tensor types in `llama_tensor_get_type_impl` (for `use_more_bits`). I never observed a difference in any of [my tests](ggml-org#19770 (comment)) - it was only after @bartowski kindly pointed this out that I realized it was incorrect. (Thanks!) * simplify

… code cleanup (#19770) * quantize : imatrix-fail early + code cleanup * fix manual override printing it's in the preliminary loop now, so needs to be on its own line * revert header changes per ggerganov * remove old #includes * clarify naming rename `tensor_quantization` to `tensor_typo_option` to descirbe its functionality * fix per barto

@bartowski

* llama-quant : correct `n_attention_wv` usage In #19770, I introduced a regression in the way the `quantize_state_impl` counter values were initialized. I was incrementing and using `n_attention_wv` in the same loop, when it should have been fixed by the time we're deciding tensor types in `llama_tensor_get_type_impl` (for `use_more_bits`). I never observed a difference in any of [my tests](ggml-org/llama.cpp#19770 (comment)) - it was only after @bartowski kindly pointed this out that I realized it was incorrect. (Thanks!) * simplify

@bartowski

* llama-quant : correct `n_attention_wv` usage In #19770, I introduced a regression in the way the `quantize_state_impl` counter values were initialized. I was incrementing and using `n_attention_wv` in the same loop, when it should have been fixed by the time we're deciding tensor types in `llama_tensor_get_type_impl` (for `use_more_bits`). I never observed a difference in any of [my tests](ggml-org/llama.cpp#19770 (comment)) - it was only after @bartowski kindly pointed this out that I realized it was incorrect. (Thanks!) * simplify

… code cleanup (ggml-org#19770) * quantize : imatrix-fail early + code cleanup * fix manual override printing it's in the preliminary loop now, so needs to be on its own line * revert header changes per ggerganov * remove old #includes * clarify naming rename `tensor_quantization` to `tensor_typo_option` to descirbe its functionality * fix per barto

@bartowski

* llama-quant : correct `n_attention_wv` usage In ggml-org#19770, I introduced a regression in the way the `quantize_state_impl` counter values were initialized. I was incrementing and using `n_attention_wv` in the same loop, when it should have been fixed by the time we're deciding tensor types in `llama_tensor_get_type_impl` (for `use_more_bits`). I never observed a difference in any of [my tests](ggml-org#19770 (comment)) - it was only after @bartowski kindly pointed this out that I realized it was incorrect. (Thanks!) * simplify

ddh0 mentioned this pull request Feb 20, 2026

quantize : refactor llama-quant.cpp #19616

Closed

ddh0 changed the title ~~quantize : refactor llama-quant.cpp~~ quantize : refactor llama-quant.cpp (imatrix fail-early) Feb 21, 2026

This comment was marked as outdated.

Sign in to view

github-actions Bot added the examples label Feb 25, 2026

This comment was marked as outdated.

Sign in to view

This comment was marked as resolved.

Sign in to view

This comment was marked as outdated.

Sign in to view

ddh0 changed the title ~~quantize : refactor llama-quant.cpp (imatrix fail-early)~~ quantize : fail-early on missing imatrix; refactor + optimize Feb 26, 2026

ddh0 marked this pull request as ready for review February 26, 2026 05:57

ddh0 requested a review from ggerganov as a code owner February 26, 2026 05:57

compilade reviewed Feb 26, 2026

View reviewed changes

Comment thread src/llama-quant.cpp Outdated

bartowski1182 reviewed Feb 26, 2026

View reviewed changes

Comment thread src/llama-quant.cpp Outdated

compilade mentioned this pull request Feb 27, 2026

Quantization laboratory: new per-tensor codebook quants, quantization recipe generator #19941

Draft

loci-dev mentioned this pull request Feb 28, 2026

UPSTREAM PR #19770: quantize : fail-early on missing imatrix; refactor + optimize auroralabs-loci/llama.cpp#1208

Open

This comment was marked as outdated.

Sign in to view

ddh0 marked this pull request as draft February 28, 2026 05:14

bartowski1182 reviewed Mar 3, 2026

View reviewed changes

Comment thread src/llama-quant.cpp

ddh0 changed the title ~~quantize : fail-early on missing imatrix; refactor + optimize~~ quantize : refactor; use quantization work scheduler for faster, more efficient quantization Mar 3, 2026

quantize : imatrix-fail early + code cleanup

decff8b

ddh0 force-pushed the llama-quant-refactor-2 branch from e314fa3 to decff8b Compare March 4, 2026 00:22

ddh0 changed the title ~~quantize : refactor; use quantization work scheduler for faster, more efficient quantization~~ quantize : imatrix fail-early, begin code cleanup Mar 4, 2026

ddh0 changed the title ~~quantize : imatrix fail-early, begin code cleanup~~ llama-quant : fail early on missing imatrix, refactor type selection, code cleanup Mar 4, 2026

ddh0 marked this pull request as ready for review March 4, 2026 01:37

ddh0 requested a review from compilade March 4, 2026 01:42

ggerganov reviewed Mar 4, 2026

View reviewed changes

bartowski1182 mentioned this pull request Mar 4, 2026

Add unit test coverage for llama_tensor_get_type #20112

Merged

ggerganov merged commit 1dab5f5 into ggml-org:master Mar 10, 2026
15 of 75 checks passed

ddh0 deleted the llama-quant-refactor-2 branch March 10, 2026 06:48

ddh0 mentioned this pull request Mar 10, 2026

llama-quant : correct n_attention_wv usage #20357

Merged

ddh0 mentioned this pull request May 1, 2026

llama-quant : fix --tensor-type when default qtype is overriden #22572

Merged

TrevorS mentioned this pull request Jul 28, 2026

quant : do not require imatrix when the tensor keeps its type #26255

Open

qdubious mentioned this pull request Aug 5, 2026

Feature Request: An opt-in flag on llama-quantize to throw error on "required fallback quantization" vs logged warning #26616

Open

4 tasks

Uh oh!

Conversation

ddh0 commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fail early for missing required imatrix

Refactoring

Other changes

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as resolved.

This comment was marked as outdated.

ddh0 commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

ddh0 commented Feb 28, 2026

Uh oh!

Uh oh!

ddh0 commented Mar 4, 2026

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

ggerganov Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

ddh0 Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggerganov Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

ddh0 Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

ddh0 Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ddh0 commented Feb 20, 2026 •

edited

Loading

ddh0 commented Feb 26, 2026 •

edited

Loading

ddh0 Mar 4, 2026 •

edited

Loading