Repository navigation
Next recipe: one epoch, short clips back, 0 dB noise floor, shuffled eval - #172
Merged
Merged
Conversation
…eval granite-qwen-500 (W&B s8jyxw2i) hit its best eval loss (0.3998) at the end of epoch 1 and rose for all of epoch 2 (0.4213 at 30k) while train loss stepped down at the boundary: replay memorization, not instability. Its eval also showed fluent unrelated text on short, quiet clips. Training config: - num_train_epochs 2 -> 1; one pass measured at ~18.2k steps at batch 256 - decoder and projector weight decay 0.01 -> 0.0, adam_epsilon 1e-6 -> 1e-8 - batch 256 -> 304 (peak 73.4 GiB of 95.6 at 256; ~86 GiB projected) Data: - collator clip floor 0.8 s -> 0.1 s (a technical minimum: under 70 ms the encoder emits no audio token). Granite's CTC head matches AMI labels 86-87% exactly from 0.2 s up, so the floor was dropping ~31% of AMI as supposedly misaligned - noise augmentation SNR floor -5 -> 0 dB, so inaudible-word clips stop demanding the full sentence - GigaSpeech audiobook tier back in (+253K rows) - eval splits: seeded shuffle before the per-dataset cap (a head slice of AMI validation was 1-2 meetings), cap 1000 -> 3000 Tooling: pyright resolves imports from the in-project .venv, as CI does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
granite-qwen-500(W&Bs8jyxw2i, 2 epochs) peaked at the end of epoch 1 and got worse for all of epoch 2:At the epoch boundary (step 18,200) train loss stepped 0.26 → 0.24 within 50 steps while grad norm held at ~0.85. That's memorization of replayed rows, not instability. stage_1's 2-epoch run
ppbo0oqnshowed the same thing. The Hub model was reverted to the step-18000 checkpoint.Its eval (step 18k vs
frozen-4) ties on corpus WER (7.04 vs 7.08). But about 0.3% of rows come out as fluent text unrelated to the audio ("he was the 1st member of the group…"). Those clips are short (median 3.6 s) and quiet with low speech/background contrast, but not silent. They cost ~0.6 WER on Earnings22 and ~0.65 on Peoples.Changes
Training (
configs/training/production.yaml)num_train_epochs2 → 1. Every source is uncapped, so a second pass is an exact replay. One pass measured at ~18.2k steps at batch 256.weight_decay0.01 → 0.0. At lr 2e-5 it shrank weights ~0.2% per run: too weak to regularize, and it pulls toward zero rather than toward the pretrained weights.projector_weight_decay0.01 → 0.0.adam_epsilon1e-6 → 1e-8. The bf16-underflow rationale doesn't apply, since the trained weights and AdamW state are float32.per_device_train_batch_size256 → 304. Measured peak at 256 was 73.4 GiB of 95.6 GiB, so ~86 GiB is projected. The GPU was already saturated, so expect fewer steps per pass, not faster training.Data (
configs/data/multiasr.yaml,scripts/train_collator.py)scripts/train.py). The old head slice of AMI validation was 1–2 meetings. Cap 1000 → 3000; measured cost ~85 s per eval, ~1% of a run.Tooling (
pyproject.toml): pyright resolves imports from the in-project.venv, as CI'svirtualenvs.in-projectdoes. An editor's language server had been picking up Homebrew's Python 3.11 (transformers 4.57) and reporting errorspoetry run pyrightnever saw.Expected effect on the next run
s8jyxw2i's 0.3998: the eval set changed (shuffle, 3000 cap, short clips now included). Re-score a reference checkpoint before comparing.Ruled out along the way
Testing
ta dev check: all gates pass.ta dev test: 1589 passed, 92.87% coverage, all 151 per-file floors met.tests/test_dataset_loader.py); a 0.3 s backchannel is kept (tests/test_data_collator.py).ASRModel,TrainingArgumentsand optimizer groups, and diffed againsts8jyxw2i's config: only the intended values changed.🤖 Generated with Claude Code