Skip to content

Next recipe: one epoch, short clips back, 0 dB noise floor, shuffled eval - #172

Merged
alexkroman merged 1 commit into
mainfrom
next-run-one-epoch-short-clips
Oct 22, 2026
Merged

alexkroman merged 1 commit into
mainfrom
next-run-one-epoch-short-clips

Conversation

@alexkroman

Copy link
Copy Markdown
Owner

Why

granite-qwen-500 (W&B s8jyxw2i, 2 epochs) peaked at the end of epoch 1 and got worse for all of epoch 2:

step eval loss epoch
18,000 0.3998 (best) 0.99
30,000 0.4213 1.65

At the epoch boundary (step 18,200) train loss stepped 0.26 → 0.24 within 50 steps while grad norm held at ~0.85. That's memorization of replayed rows, not instability. stage_1's 2-epoch run ppbo0oqn showed the same thing. The Hub model was reverted to the step-18000 checkpoint.

Its eval (step 18k vs frozen-4) ties on corpus WER (7.04 vs 7.08). But about 0.3% of rows come out as fluent text unrelated to the audio ("he was the 1st member of the group…"). Those clips are short (median 3.6 s) and quiet with low speech/background contrast, but not silent. They cost ~0.6 WER on Earnings22 and ~0.65 on Peoples.

Changes

Training (configs/training/production.yaml)

  • num_train_epochs 2 → 1. Every source is uncapped, so a second pass is an exact replay. One pass measured at ~18.2k steps at batch 256.
  • Decoder weight_decay 0.01 → 0.0. At lr 2e-5 it shrank weights ~0.2% per run: too weak to regularize, and it pulls toward zero rather than toward the pretrained weights. projector_weight_decay 0.01 → 0.0.
  • adam_epsilon 1e-6 → 1e-8. The bf16-underflow rationale doesn't apply, since the trained weights and AdamW state are float32.
  • per_device_train_batch_size 256 → 304. Measured peak at 256 was 73.4 GiB of 95.6 GiB, so ~86 GiB is projected. The GPU was already saturated, so expect fewer steps per pass, not faster training.

Data (configs/data/multiasr.yaml, scripts/train_collator.py)

  • Collator clip floor 0.8 s → 0.1 s. 0.1 s is a technical minimum: under 70 ms the encoder emits zero audio tokens, and under 16 ms the STFT crashes. Granite's own CTC head matches AMI test labels exactly 86–87% of the time from 0.2 s up, as often as at 0.5–0.8 s, so the old floor was dropping ~31% of AMI (backchannels, single words) as supposedly misaligned. Without them the model never saw a short clip get a short answer.
  • Noise augmentation SNR floor −5 → 0 dB. Targets are unchanged, so clips mixed below 0 dB demand the full sentence even when the words are inaudible, which rewards guessing from the decoder's prior. The first degraded run already measured this on clean audio (AMI 13 → 29 hallucinated clips). Tradeoff: sub-0 dB training bought 5.1 WER on URGENT's sub-0 dB clips, so compare URGENT noisy vs clean when scoring.
  • GigaSpeech audiobook tier back in. That's +253K rows (GigaSpeech is uncapped, so they're added, not swapped).
  • Eval splits get a seeded shuffle before the per-dataset cap (scripts/train.py). The old head slice of AMI validation was 1–2 meetings. Cap 1000 → 3000; measured cost ~85 s per eval, ~1% of a run.

Tooling (pyproject.toml): pyright resolves imports from the in-project .venv, as CI's virtualenvs.in-project does. An editor's language server had been picking up Homebrew's Python 3.11 (transformers 4.57) and reporting errors poetry run pyright never saw.

Expected effect on the next run

  • About 530K more clips per pass (~+12%): +253K GigaSpeech rows plus ~280K short AMI/EdAcc clips. Only the GigaSpeech rows add steps, since the collator floor shrank batches rather than the dataset. ~16.2k steps at batch 304.
  • Eval loss is not comparable to s8jyxw2i's 0.3998: the eval set changed (shuffle, 3000 cap, short clips now included). Re-score a reference checkpoint before comparing.
  • Several data changes land together. If results move, attributing them needs ablations.

Ruled out along the way

  • Capping repeats per Common Voice sentence: CV17 en train is 938k rows over 937.7k unique sentences, and none of the 63 hallucinated outputs is a CV train sentence.
  • Adding VoxPopuli back: removed on purpose (edited parliamentary labels vs the leaderboard's verbatim refs). The local VoxPopuli regression is the model no longer imitating the edited style.

Testing

  • ta dev check: all gates pass. ta dev test: 1589 passed, 92.87% coverage, all 151 per-file floors met.
  • New tests: eval-cap sampling is seeded, deterministic and reaches past the head (tests/test_dataset_loader.py); a 0.3 s backchannel is kept (tests/test_data_collator.py).
  • Composed the Hydra config, built the real ASRModel, TrainingArguments and optimizer groups, and diffed against s8jyxw2i's config: only the intended values changed.

🤖 Generated with Claude Code

…eval

granite-qwen-500 (W&B s8jyxw2i) hit its best eval loss (0.3998) at the end
of epoch 1 and rose for all of epoch 2 (0.4213 at 30k) while train loss
stepped down at the boundary: replay memorization, not instability. Its
eval also showed fluent unrelated text on short, quiet clips.

Training config:
- num_train_epochs 2 -> 1; one pass measured at ~18.2k steps at batch 256
- decoder and projector weight decay 0.01 -> 0.0, adam_epsilon 1e-6 -> 1e-8
- batch 256 -> 304 (peak 73.4 GiB of 95.6 at 256; ~86 GiB projected)

Data:
- collator clip floor 0.8 s -> 0.1 s (a technical minimum: under 70 ms the
  encoder emits no audio token). Granite's CTC head matches AMI labels
  86-87% exactly from 0.2 s up, so the floor was dropping ~31% of AMI as
  supposedly misaligned
- noise augmentation SNR floor -5 -> 0 dB, so inaudible-word clips stop
  demanding the full sentence
- GigaSpeech audiobook tier back in (+253K rows)
- eval splits: seeded shuffle before the per-dataset cap (a head slice of
  AMI validation was 1-2 meetings), cap 1000 -> 3000

Tooling: pyright resolves imports from the in-project .venv, as CI does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexkroman
alexkroman merged commit 166ea4e into main Oct 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants