Add licensed LibriSpeech eval baseline - #63
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a pinned, license-recorded LibriSpeech dev-clean “tiny” real-speech evaluation baseline designed to be built and evaluated only in a remote Colab runtime, while keeping all audio and model artifacts out of the repository.
Changes:
- Added a JSON spec that pins the LibriSpeech source URL, MD5, CC BY 4.0 license, deterministic speaker selection, and a remote-only storage policy.
- Added a Colab-only Python workflow to download/verify/extract/convert the dataset, run evaluation matrices, and archive evidence for download.
- Updated evaluation docs/roadmap/changelog and added a test validating the new spec is pinned and remote-only.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_release_artifacts.py | Adds a test asserting the LibriSpeech evalset spec is pinned and remote-only. |
| ROADMAP.md | Marks the real-speech baseline deliverables as completed and records initial findings. |
| examples/colab_librispeech_eval.py | New Colab workflow script to build the evalset remotely and run baseline matrices/reports. |
| examples/artifacts/README.md | Documents the new LibriSpeech eval baseline spec artifact. |
| examples/artifacts/librispeech-dev-clean-tiny-v1.json | Adds the pinned LibriSpeech evalset specification (URL/MD5/license/selection/storage policy). |
| docs/EVALUATION.md | Documents the new baseline and records initial T4 evidence metrics and policy implications. |
| docs/ACCELERATORS.md | Adds instructions for running the LibriSpeech Colab workflow and downloading evidence. |
| CHANGELOG.md | Notes addition of the LibriSpeech baseline spec and Colab workflow. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+66
to
+73
| def _extract_archive(archive_path: Path) -> Path: | ||
| EXTRACT_DIR.mkdir(parents=True, exist_ok=True) | ||
| with tarfile.open(archive_path, "r:gz") as archive: | ||
| archive.extractall(EXTRACT_DIR, filter="data") | ||
| source_root = EXTRACT_DIR / "LibriSpeech" | ||
| if not source_root.is_dir(): | ||
| raise FileNotFoundError("LibriSpeech root was not found after extraction") | ||
| return source_root |
Comment on lines
+263
to
+270
| def main() -> int: | ||
| if Path.cwd() != Path("/content"): | ||
| raise RuntimeError("This LibriSpeech workflow must run inside a Colab runtime rooted at /content") | ||
|
|
||
| if WORK_ROOT.exists(): | ||
| shutil.rmtree(WORK_ROOT) | ||
| EVIDENCE_DIR.mkdir(parents=True) | ||
| DOWNLOAD_DIR.mkdir(parents=True) |
Comment on lines
+95
to
+102
| def _transcript_for(source_path: Path) -> str: | ||
| utterance_id = source_path.stem | ||
| transcript_path = next(source_path.parent.glob("*.trans.txt")) | ||
| for line in transcript_path.read_text(encoding="utf-8").splitlines(): | ||
| item_id, transcript = line.split(" ", 1) | ||
| if item_id == utterance_id: | ||
| return transcript | ||
| raise ValueError(f"Transcript not found for {utterance_id}") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Colab T4 evidence
The candidate commit 0ea7f2a evaluated eight speakers and about 61 seconds of licensed real speech with wideband_16k and lowpass_4k. Both backends completed all 16 items without stability failures.
Wideband_16k means:
Lowpass_4k quality metrics were nearly identical, while LavaSR remained slower. The result does not support stable promotion from full-reference metrics alone; downstream ASR and blind listening evidence are now explicit next gates.
Local lightweight validation
No dataset download, model weight, GPU inference, or matrix computation ran locally.