Skip to content

Default recipe: Qwen3-ASR encoder + SmolLM3-3B, padding-free packing, 30 s context - #170

Merged
alexkroman merged 1 commit into
mainfrom
qwen3-asr-smollm3-packing
Oct 21, 2026
Merged

alexkroman merged 1 commit into
mainfrom
qwen3-asr-smollm3-packing

Conversation

@alexkroman

@alexkroman alexkroman commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Restored description: this PR merged with one commit (ca9e808, squashed into main as bb4a4f8): Qwen3-ASR-1.7B encoder + SmolLM3-3B decoder as the default recipe, padding-free packing under flash_attention_2, flash_attention_2 by default with a kernel probe, 30 s training clips / 28 s inference chunks, batch 32 x 2, the hub_model_id fix, and ta analysis compare's deletion/substitution tables (ITN table removed). Follow-up work moved to a new PR.

🤖 Generated with Claude Code

… 30 s context

Encoder: Qwen3-ASR-1.7B's audio tower (tiny_audio/qwen3_asr_encoder.py). Of
the encoders that load natively in our transformers, its system ranks highest
on the Open ASR leaderboard (#9, mean of 9 4.936). transformers'
Qwen3ASREncoder owns its contract (1 s conv chunks, 8 s attention windows,
padding dropped via the mel mask); the wrapper splits its packed output back
into a padded batch and reads only the model.audio_tower.* tensors.
compute_encoder_output_length gains a chunk_frames option (13 tokens per full
100-frame chunk) used by the collator, processor and model.

Decoder: HuggingFaceTB/SmolLM3-3B (same 2048 hidden as Qwen3-1.7B), with
liger's apply_liger_kernel_to_smollm3. Batch 32 x 2 after 64 x 1 OOMed on the
RTX PRO 6000; the effective batch stays 64.

Throughput: labelled batches are packed padding-free under flash_attention_2
(tiny_audio/asr_packing.py), so no decoder compute goes to padding; rows keep
their order and per-token loss. flash_attention_2 is now the default, with a
one-time probe that falls back to sdpa when flash-attn's kernels do not run on
the card; packing switches itself off under sdpa.

Context: data.max_audio_seconds 30 (was the collator's fixed 19 s) and
model.inference_chunk_seconds 28 (was 18); the LibriHeavy length pre-filter is
removed. The tiny preset and older checkpoints keep 19 / 18.

Also: hub_model_id no longer points at the Granite model's repo; ta analysis
compare drops the ITN table (and scripts/itn.py) and adds deletion and
substitution rate tables under the insertion rate table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexkroman
alexkroman merged commit bb4a4f8 into main Oct 21, 2026
6 checks passed
@alexkroman alexkroman changed the title Default recipe: Qwen3-ASR encoder + SmolLM3-3B, padding-free packing, 30 s context Granite NC + full fine-tune Qwen3-0.6B, padding-free packing, 30 s context, MLX eval Oct 21, 2026
@alexkroman alexkroman changed the title Granite NC + full fine-tune Qwen3-0.6B, padding-free packing, 30 s context, MLX eval Default recipe: Qwen3-ASR encoder + SmolLM3-3B, padding-free packing, 30 s context Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants