Skip to content

fix(gemma3): don't cast attention mask to packed uint32 weight dtype — padding leaks into embeddings of quantized checkpoints - #73

Open
alekseysotnikov wants to merge 1 commit into
Blaizzy:mainfrom
alekseysotnikov:fix/gemma3-uint32-pad-mask
Open

fix(gemma3): don't cast attention mask to packed uint32 weight dtype — padding leaks into embeddings of quantized checkpoints#73
alekseysotnikov wants to merge 1 commit into
Blaizzy:mainfrom
alekseysotnikov:fix/gemma3-uint32-pad-mask

Conversation

@alekseysotnikov

Copy link
Copy Markdown

Problem

Model.__call__ in mlx_embeddings/models/gemma3_text.py casts the additive attention mask to self.model.embed_tokens.weight.dtype:

extended_attention_mask = extended_attention_mask.astype(
    self.model.embed_tokens.weight.dtype
)

In quantized checkpoints (e.g. mlx-community/embeddinggemma-300m-8bit) embed_tokens.weight.dtype is packed uint32. Casting the 0.0 / -inf additive mask to uint32 silently destroys the -inf entries, so pad tokens leak into attention whenever padded batches are used — which is the documented batch-processing path (batch_encode_plus(padding=True) + attention_mask).

This affects any quantized gemma3 embedding model; fp16-weight checkpoints work only by accident because their weight dtype happens to be floating.

Minimal repro

EmbeddingGemma-300m-8bit, one 11-char sentence:

setup cos vs embedded alone
batched with itself 1.00000
batched with 127 copies of a max-length sentence (current code) 0.884
same, mask cast to mx.float16 instead 0.999997

Quantifying over a full real-world transcript (395 sentences, natural-order batching): mean cosine vs zero-padding ground truth is 0.928 (385/395 sentences below 0.99) with current code — i.e. nearly every embedding is distorted by its batch neighbors' padding.

Fix

Cast the mask to a floating dtype instead of the (possibly packed-integer) weight dtype: pick the first floating parameter dtype (respects bf16 checkpoints), falling back to mx.float16. One-line behavioral change, no API impact.

Verified: parity between per-sentence, natural-order and length-sorted batching goes to cos ≈ 1.0 on both reproductions above.

Fixes #72

…ed weight dtype

Quantized checkpoints pack embed_tokens.weight as uint32. Casting the
0.0/-inf additive attention mask to that dtype silently destroys the
-inf padding entries, so pad tokens leak into attention on every padded
batch (the documented batch-processing path).

On mlx-community/embeddinggemma-300m-8bit, an 11-char sentence embedded
alone vs batched with 127 max-length sentences: cos 0.884 before,
0.999997 after the fix. On a real 395-sentence transcript, mean cosine
vs zero-padding ground truth was 0.928 (385/395 sentences < 0.99) with
the current code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gemma3_text: attention mask cast to packed uint32 weight dtype breaks padding in quantized checkpoints

1 participant