Add Colab ASR downstream evidence

[Tinnci]

Summary

• pin an external Whisper tiny.en model revision, license, runtime, and transcript normalization policy
• add a Colab-only ASR workflow that consumes the existing LibriSpeech matrices
• generate degraded, sinc, and LavaSR transcripts remotely and feed only precomputed JSON to eval downstream
• keep the base package free of ASR dependencies and model downloads
• record the first downstream WER/CER result and close that roadmap evidence item

Colab T4 evidence

The external evaluator was openai/whisper-tiny.en at revision 87c7102498dcde7456f24cfd30239ca606ed9063 with Transformers 5.13.1 and float16 CUDA inference. It transcribed 24 files: eight degraded, eight sinc-enhanced, and eight LavaSR-enhanced utterances.

Mean results:

• degraded input: WER 0.1594, CER 0.0757
• sinc-resample: WER 0.1594, CER 0.0757, delta 0
• lavasr-compat: WER 0.1594, CER 0.0757, delta 0

This small slice shows neither ASR improvement nor degradation. It does not support stable promotion and leaves blind listening as the remaining P2 evidence gap.

Local lightweight validation

• pixi run lint
• pixi run format-check
• pixi run typecheck
• pixi run test: 163 passed, 6 skipped
• evaluator metadata validation
• script syntax validation
• git diff --check

No ASR model, audio matrix, managed model weight, or GPU inference ran locally.

[Copilot]

Pull request overview

This PR adds reproducible “downstream ASR evidence” by pinning external Whisper tiny.en metadata and providing a Colab-only workflow to generate precomputed transcript JSON that can be evaluated by the existing lightweight WER/CER evaluator—keeping ASR dependencies and model downloads out of the base package.

Changes:

• Added a pinned Whisper evaluator JSON spec (model revision, license, runtime, normalization, and “no package dependency” policy) plus a release artifact test to enforce it.
• Introduced a Colab-only script that transcribes the existing LibriSpeech eval matrices remotely and runs audio-super-res eval downstream over generated transcript datasets.
• Documented and recorded the initial downstream WER/CER results in the roadmap, evaluation docs, accelerators guide, and changelog.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_release_artifacts.py	Adds a regression test to ensure the Whisper evaluator spec remains pinned and explicitly not a package dependency.
ROADMAP.md	Marks the ASR evidence roadmap item complete and records the initial downstream WER/CER outcome.
examples/colab_asr_downstream.py	New Colab-only workflow to generate Whisper transcripts and downstream WER/CER JSON artifacts from existing LibriSpeech matrices.
examples/artifacts/README.md	Documents the new pinned ASR evaluator artifact.
examples/artifacts/asr-evaluator-whisper-tiny-en.json	Adds pinned external Whisper evaluator metadata (revision/license/runtime/normalization/integration policy).
docs/EVALUATION.md	Adds narrative describing the pinned Whisper downstream evidence and results.
docs/ACCELERATORS.md	Documents how to run and download the Colab-only downstream ASR evidence workflow.
CHANGELOG.md	Notes the addition of the pinned Whisper evaluator spec and Colab transcript workflow.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.