Skip to content

Granite NC + full fine-tune Qwen3-0.6B recipe, MLX eval, compare cleanup - #171

Merged
alexkroman merged 2 commits into
mainfrom
granite-qwen3-full-ft-mlx
Oct 22, 2026
Merged

alexkroman merged 2 commits into
mainfrom
granite-qwen3-full-ft-mlx

Conversation

@alexkroman

Copy link
Copy Markdown
Owner

Summary

Follows #170 (merged). Moves the default recipe from Qwen3-ASR + SmolLM3 to Granite NC + a fully fine-tuned Qwen3-0.6B: the Qwen3-ASR + SmolLM3 run trailed the Granite stack on every dataset (+1.68 corpus WER, CI [+1.22, +2.15], half-trained).

Default recipe

  • Encoder: ibm-granite/granite-speech-5.0-470m-turboctc-nc, frozen (CC-BY-NC-SA).
  • Decoder: Qwen/Qwen3-0.6B, fully fine-tuned incl. embeddings, at the old stage_1 recipe's measured settings: decoder lr 2e-5, decoder weight decay 0.01, float32 storage so 2e-5 updates are not rounded away under bf16 (compute stays bf16 via autocast).
  • Batch 256 × 1 (measured ~27% of the 95 GiB card at 64), 2 epochs, prefetch 2, eval cap 1000 per dataset; hub_model_id: mazesmazes/tiny-audio-granite-qwen3-0.6b.
  • The tiny laptop preset pins its LoRA setup explicitly.

Serving fix

  • ta serve loads float32-stored checkpoints in bf16 on CUDA/MPS; flash_attention_2 rejects float32 inputs without autocast, and decoding read twice the bytes. DTYPE_CONFIG_FIELDS now lives in scripts/inference.py, shared by serve and eval.

MLX evaluation (ta eval --mlx)

  • Transcribes a Granite Speech 5.0 + fully fine-tuned Qwen3 checkpoint on Apple silicon (scripts/mlx_inference.py): mlx-audio's Granite encoder, an MLX twin of MLPAudioProjector, mlx-lm's Qwen3, plus the PyTorch pipeline's own chunking, lead-in, prompt, 12-gram guard, silence gate and text cleanup.
  • Each run uses the Hub repo's latest revision and converts it once per revision into ~/.cache/tiny-audio/mlx. Results are saved as <model>-mlx.
  • Optional macOS-only mlx dependency group (lock adds packages only); typings/ stubs keep strict pyright passing on Linux (with a .gitignore exception for their models/ dirs); the MLX tests are skipped where MLX is absent.

ta analysis compare

  • Removed the entity table and the corpus WER bootstrap table (the cpWER one remains for speaker datasets).

Test plan

  • ta dev check (29 gates), ta dev test (suite + coverage floors), precommit hook.
  • MLX vs PyTorch on real weights, each stage in float32: features 2.8e-5 max diff, encoder 4.4e-6, projector 1.3e-6, decoder 2.0e-5 relative.
  • ta eval --mlx end to end on a checkpoint with the full-fine-tune layout.
  • ta eval --mlx vs the PyTorch eval on the same clips once the training run's checkpoint is pushed.
  • Record the W&B peak memory for batch 256 in production.yaml.
  • End of run: embed_tokens drift concentrated on common tokens (rare-token drift check).

Known follow-ups (from #170's code review, still open)

  1. asr_modeling.py imports the Qwen3-ASR module unconditionally; make it optional like the Granite import.
  2. Derive encoder conv layers from audio_model_id instead of YAML.
  3. Speaker-streams path still chunks at a fixed 18 s; pass chunk_max_seconds.
  4. Planner's encoder-tape estimate uses Granite's frame rate for every encoder.
  5. Collapse the copies of the 19 s clip default into one constant.
  6. Accept a local directory for the Qwen3-ASR encoder.
  7. Declare pack_sequences in the config schema.

🤖 Generated with Claude Code

alexkroman-assembly and others added 2 commits October 21, 2026 16:41
…eanup

Recipe: back to the Granite Speech 5.0 turboctc-nc encoder, whose CTC-trained
output is close to token-shaped, with Qwen3-0.6B fully fine-tuned instead of
LoRA. Decoder settings are the old stage_1 recipe's measured ones: decoder lr
2e-5 (chosen from attention-weight drift), embeddings trainable, decoder
weight decay 0.01, float32 storage so 2e-5 updates are not rounded away under
bf16 (compute stays bf16). Batch 256 x 1 (a measured ~27% of the card at 64),
2 epochs, prefetch 2, eval cap 1000 per dataset. hub_model_id is named for
the recipe. The tiny preset pins its LoRA setup explicitly.

Serving: `ta serve` loads float32-stored checkpoints in bf16 on CUDA/MPS.
flash_attention_2 rejects float32 inputs with no autocast to cast them, and
decoding read twice the bytes. DTYPE_CONFIG_FIELDS moves to scripts/inference
so serve and eval share one list.

MLX: `ta eval --mlx` transcribes a Granite Speech 5.0 + fully fine-tuned Qwen3
checkpoint on Apple silicon (scripts/mlx_inference.py). mlx-audio's Granite
encoder port, an MLX twin of MLPAudioProjector, mlx-lm's Qwen3 with the
checkpoint's decoder weights, and the PyTorch pipeline's own chunking, lead-in,
prompt, 12-gram guard, silence gate and text cleanup. Each run takes the Hub
repo's latest revision and converts it once per revision into
~/.cache/tiny-audio/mlx. Per stage in float32 it matches PyTorch to <=3e-5.
Optional macOS-only `mlx` dependency group; typings/ stubs keep strict
pyright passing on Linux, and conftest skips the MLX tests where MLX is absent.

Compare: drops the entity table and the corpus WER bootstrap table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Tests job failed the coverage gate's 50% new-file floor: MLX was a
darwin-only optional group, so CI skipped tests/test_mlx_inference.py and
scripts/mlx_inference.py measured 0%. MLX publishes Linux CPU wheels
(mlx + mlx-cpu, manylinux_2_35 x86-64), so the group now installs `mlx[cpu]`
on Linux and the test job runs `poetry install --with dev,mlx`. Every MLX op
the tests use was checked in a linux/amd64 container (features, RMSNorm/
Linear/GELU, the logits mask, Qwen3 generation from embeddings, bf16
safetensors). mlx-audio's sounddevice is never imported on our path. Pods
are unaffected: they install `poetry export --only main`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexkroman
alexkroman merged commit 7897e34 into main Oct 22, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants