fix(audio): benchmark tagging on pre-staged AMI meetings

[mohammadaaftabv]

Summary

Adds a reproducible AMI-based audio-tagging benchmark setup and hardens the tagging benchmark so nightly runs fail when no useful work is done.

Features Added

• Added benchmarking/data_prep/prepare_audio_tagging_data.py to stage AMI SDM audio plus a local PyAnnote snapshot.
• Added token-free setup defaults, CLI/env mirror overrides, local model copy support, and setup-time validation for manifest, audio files, and model files.
• Added benchmarking/nightly-data-setup.yaml and --data-setup-config runner support so benchmark data prep can run before nightly entries.
• Updated tagging nightly entries to use pre-staged data with --raw-data-dir, --no-auto-download, and a local diarization model path.
• Switched tagging workload to three long AMI meetings, with deterministic repeated IDs for the repeat benchmark.
• Added benchmark correctness checks for row parity, output manifest rows, nonzero stage work, complete segments, WER output, source/tagged duration, and failure metrics.
• Updated audio tagging/runtime stages for local PyAnnote loading, correct second-pass ASR alignment schema, and empty/zero-duration segment rejection.
• Removed default forwarding of host CUDA/debug env vars from run.sh to keep benchmark containers reproducible.
• Updated benchmark docs for Quick Start, setup, standalone debugging, and nightly behavior.

Validation

• GitHub checks passing on PR head: Ruff, pre-commit, secrets, DCO.
• Local checks: ruff check, ruff format --check, py_compile, git diff --check.
• Docker build works with ./benchmarking/tools/build_docker.sh --tag-as-latest.
• README run command reaches Docker locally; this machine cannot run the CUDA 12.9 image because its driver does not satisfy cuda>=12.9.

Checklist

• DCO signoff present.
• No test-file changes included.
• Documentation updated.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This PR replaces the placeholder audio-tagging nightly benchmarks with real AMI meeting audio (three SDM meetings, ~1.56 h). It adds a one-time prepare_audio_tagging_data.py data-prep script, a nightly-data-setup.yaml for orchestration, and a thoroughly reworked audio_tagging_benchmark.py that validates every stage of the 12-step pipeline end-to-end.

• Benchmark redesign: nightly entries now consume pre-staged local paths via --raw-data-dir / --no-auto-download, eliminating the HF_TOKEN dependency and dropping the previous --input-manifest placeholder. Metric requirements cover row-count matching, segment coverage (\u226570\u2009%), 12-stage execution, and throughput floors.
• Runtime fixes: hf_token in PyAnnoteDiarizationStage is made optional (repr=False) to support token-free local pipeline loading; NeMoASRAlignerStage gains correct inputs()/outputs() schema for infer_segment_only mode; PrepareModuleSegmentsStage.is_valid_segment now rejects empty word lists and zero-duration segments; RepeatEntriesStage assigns deterministic per-copy IDs so repeated files cannot collide.
• Tooling: run.py and run.sh gain --data-setup-config to execute data-prep scripts before benchmark entries, with fail-fast semantics.

Confidence Score: 5/5

The change is self-contained to benchmarking infrastructure and a small set of audio-stage fixes; no production data paths are altered.

All runtime fixes are narrowly scoped and locally validated. The new data-prep and benchmark scripts have clear input/output contracts and fail loudly on missing or malformed data. No production pipeline logic is changed in a way that would silently regress existing users.

The row-index-based meeting selection in prepare_audio_tagging_data.py warrants a second look if the upstream diarizers-community/ami dataset is ever updated.

Important Files Changed

Filename	Overview
benchmarking/data_prep/prepare_audio_tagging_data.py	New 473-line data-prep script that downloads three AMI SDM meetings and a local PyAnnote snapshot; correctly validates staged outputs. Row-index-based meeting selection from the streaming dataset is fragile if the HF dataset is reordered.
benchmarking/scripts/audio_tagging_benchmark.py	Major rework replacing HF-token+manifest inputs with pre-staged local paths; adds robust output validation including row-count checks and stage coverage. stage_execution_coverage_ratio is hardcoded 1.0; segment_output_coverage_ratio has a latent divide-by-zero.
benchmarking/scripts/utils.py	RepeatEntriesStage now assigns deterministic unique IDs per copy (fixing the previously reported None/empty-string bug); all copies including index-0 have the _repeat_N suffix, losing the original ID.
nemo_curator/stages/audio/inference/speaker_diarization/pyannote.py	Makes hf_token optional (default None) and excludes it from repr for security; allows the stage to load from a local model directory without a token.
nemo_curator/stages/audio/tagging/inference/nemo_asr_align.py	Adds use_cuda_graphs field and correct inputs()/outputs() schema for infer_segment_only mode used by the second-pass ASR stage.
nemo_curator/stages/audio/tagging/prepare_module_segments.py	Adds early-return validation for empty word lists and zero-duration segments in is_valid_segment, preventing invalid segments from reaching downstream stages.
benchmarking/run.py	Adds --data-setup-config CLI argument and run_data_setups to execute data-prep scripts before benchmark entries; fails fast on first setup failure.

_{Reviews (8): Last reviewed commit: "Fix benchmark ruff lint" | Re-trigger Greptile}

[greptile-apps]

Tutorial now fails hard on empty-segment pipelines

validate_tagging_outputs raises RuntimeError when the pipeline produces zero tagged segments. Because there is no try/except around this call in the tutorial's main, any small or synthetic dataset that passes diarization but produces no PrepareModuleSegments output will crash with an unhandled exception — the PIPELINE COMPLETE block is never reached. Previously the tutorial logged num_tasks = 0 and completed normally, so this is a behavioural break for developers iterating against toy fixtures. Consider documenting in the tutorial that a non-trivial audio dataset producing at least one segment is required, or catching the validation error and printing a user-friendly hint before re-raising.

[greptile-apps]

Use an explicit None/empty-string check so that a legitimate ID value of 0 (or any other falsy non-empty value) does not incorrectly raise.

Suggested change

	source_id = data.get(self._unique_id_key)
	if not source_id:
	msg = f"Cannot repeat entry without '{self._unique_id_key}'"
	raise ValueError(msg)
	source_id = data.get(self._unique_id_key)
	if source_id is None or source_id == "":
	msg = f"Cannot repeat entry without '{self._unique_id_key}'"
	raise ValueError(msg)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[github-actions]

🌿 Preview your docs: https://nvidia-preview-aaftabv-fix-audio-tagging-benchmarks.docs.buildwithfern.com/nemo/curator

Here are the markdown pages you've updated:

• nemo/curator/curate-audio/load-data/fleurs-dataset
• nemo/curator/curate-audio/tutorials/beginner
• nemo/curator/get-started/audio