Audio: add extended performance observability#2223
Draft
mohammadaaftabv wants to merge 47 commits into
Draft
Conversation
Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Signed-off-by: Aaftab V <aaftabv@nvidia.com>
- hoist followup_prompt default into _FOLLOWUP_PROMPT module constant
- move 6 lazy imports to module scope (torch, transformers.Qwen3OmniMoeProcessor,
qwen_omni_utils.process_mm_info, huggingface_hub.{snapshot_download,
hf_hub_download}, yaml); keep existing vllm try/except guard
- drop redundant `language and` short-circuit in _get_prompt_text
- guard tar.extractfile(...) against None before .read() in
NemoTarShardReaderStage; add hard-link regression test
- make ShardedManifestWriterStage.output_dir a required field; drop empty-
string post_init check
- add Apache/NVIDIA copyright headers to inference/__init__.py and
qwen_omni_inprocess.yaml
- drop pip-fallback install block from tutorial README (uv-only)
- remove tests/stages/audio/inference/test_qwen_omni_tutorial.py per
"no pytests for tutorials"
- retarget @patch decorators in test_qwen_omni.py to the use-site
(nemo_curator.stages.audio.inference.qwen_omni.snapshot_download) so
the patches still bind after the import hoist
Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Add qwen-asr and its lazy-imported runtime companions to Curator's audio extras so that the harvest.curator Docker image gets the full qwen-asr stack via Curator's uv sync rather than via post-uv pip installs in NvLLMOps. This honors the Algorithmic vs Data-Mover Dep Ownership Rule: algorithmic libraries belong in Curator, NvLLMOps owns only data-mover clients. audio_common gains the qwen-asr forced-aligner text-norm and audio-feature companions that qwen-asr 0.0.6's qwen3_forced_aligner.py imports lazily: nagisa==0.2.11, soynlp==0.0.493, pyarabic, opencc-python-reimplemented, and nnAudio. audio_cuda12 gains qwen-asr==0.0.6 itself for the Granary v2 Qwen-ASR recovery stage, and fasttext==0.9.3 for the Granary v2 LID stage. fasttext already lives in text_cpu but audio_cuda12 does not pull text_cpu, so the declaration is duplicated here. [tool.uv] override-dependencies replaces the broad huggingface-hub>=0.34,<1.0 override with three exact pins proven against qwen-asr by NvLLMOps commit 68f18e9b: transformers==4.57.6, accelerate==1.12.0, and huggingface-hub==0.36.0. These force-override qwen-asr's declared (incompatible) version pins so the resolver picks the proven-compatible versions for the entire graph. This change extends PR1967's scope from "first-stage Qwen-Omni inference only" to also cover Granary v2 algorithmic-dep self-containment, so that later Granary v2 PRs (Qwen-ASR recovery, text filtering, PnC, ITN, SED) can rely on Curator's audio_cuda12 extra without further pip-after-uv overrides in NvLLMOps. Lock churn: +18 packages including qwen-asr 0.0.6, nagisa, soynlp, pyarabic, opencc-python-reimplemented, nnaudio, and qwen-asr's gradio/flask transitive demo deps. transformers/huggingface-hub/accelerate stayed at the override-pinned versions, so no version drift for the qwen-omni stack. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
NemoTarShardDiscoveryStage and NemoTarredAudioReader previously declared yaml_path as `yaml_path: str = ""` and validated it via a manual empty- string check in __post_init__. Per reviewer feedback, this is cleaner as a required dataclass field: __init__ enforces the requirement directly, removing the sentinel and the manual check. Reordered yaml_path before defaulted fields in both dataclasses (Python requires non-default fields first). NemoTarShardDiscoveryStage's __post_init__ existed only for the empty-string check and is removed; NemoTarredAudioReader's __post_init__ keeps super().__init__() and self._stages wiring, only the check is removed. All three construction sites (the test, hydra _target_ resolution, and NemoTarredAudioReader's own __post_init__) already use keyword arguments, so the required-field reorder is transparent. The observable difference for a caller that forgets yaml_path is now TypeError from __init__ (before any object exists) instead of ValueError from __post_init__. Addresses: - NVIDIA-NeMo#1967 (comment) - NVIDIA-NeMo#1967 (comment) Signed-off-by: Aaftab V <aaftabv@nvidia.com>
…1.0 pins The audio_cuda12 PR previously added three [tool.uv] override-dependencies entries (huggingface-hub==0.36.0, transformers==4.57.6, accelerate==1.12.0) described as overriding qwen-asr's incompatible declared pins. Audit of the resolved graph showed that rationale was inaccurate: - qwen-asr 0.0.6 declares transformers==4.57.6 / accelerate==1.12.0 natively in its requires-dist, so no override is needed for those. - The remaining transformers/accelerate constraints across the graph (nemo-toolkit ~=4.57.0, vllm >=4.56.0,<5, pyannote-audio >=4.48.3, whisperx >=4.48.0) all naturally permit 4.57.6 / 1.12.0. - The hf-hub conflict (data-designer-engine >=1.0.1 vs whisperx <1.0.0) was already resolved on main by hf-hub>=0.34,<1.0; this PR's tightening to ==0.36.0 was redundant. All three entries removed; the pre-existing hf-hub override restored to its original range. Verified via `uv lock` that the resolved versions of transformers, accelerate, huggingface-hub, qwen-asr, nagisa, soynlp, vllm, torch, torchaudio, torchvision, torchcodec, nixl-cu12, xgrammar, fasttext, data-designer-engine, pyannote-audio, whisperx, nemo-toolkit, qwen-omni-utils, fsspec, numpy, protobuf, setuptools are byte-identical before and after (591 packages resolved). This PR now adds zero new override-dependencies entries, eliminating the cross-modality blast radius that was flagged. Also softened the remaining hard `==` pins inside audio_common / audio_cuda12: - nagisa ==0.2.11 -> >=0.2.11,<0.3 (8-year-old stable lib; qwen-asr still transitively pins 0.2.11) - qwen-asr ==0.0.6 -> >=0.0.6,<0.1 (only 0.0.6 exists today, but a future security-patch 0.0.7 auto-flows in if/when published; <0.1 caps speculative jumps) - fasttext ==0.9.3 -> >=0.9.3,<0.10 (10-year-old maintenance-mode lib; releases every 2-4 years) The 7-line preamble comment above qwen-asr was trimmed to one line in the same edit (reviewer asked for less verbose). uv.lock is regenerated; all resolved package versions remain identical to the pre-edit state. Addresses: - NVIDIA-NeMo#1967 (comment) - NVIDIA-NeMo#1967 (comment) - NVIDIA-NeMo#1967 (comment) - NVIDIA-NeMo#1967 (comment) - NVIDIA-NeMo#1967 (comment) Signed-off-by: Aaftab V <aaftabv@nvidia.com>
…scaling Per Praateek's review feedback on tutorials/audio/qwen_omni_inprocess/ qwen_omni_inprocess.yaml (NVIDIA-NeMo#1967), controlled-throughput benchmarking is not a tutorial concern. Remove the explicit worker-count overrides (reader_num_workers, reader_num_workers_per_node, omni_num_workers) from the tutorial config and the NemoTarredAudioReader / NemoTarShardReaderStage / InferenceQwenOmniStage classes. The ProcessingStage base class defaults (num_workers() -> None, xenna_stage_spec() -> {}) let Xenna's autoscaler size each stage from the cluster resources, which is the documented happy-path behaviour of the audio pipelines. Changes: - tutorials/audio/qwen_omni_inprocess/qwen_omni_inprocess.yaml: remove the MULTI-NODE SCALING block (reader_num_workers, reader_num_workers_per_node, omni_num_workers) and the three Hydra interpolations that fed them into the reader and qwen-omni stages. - nemo_curator/stages/audio/io/nemo_tarred_reader.py: drop reader_num_workers / reader_num_workers_per_node fields on the composite NemoTarredAudioReader; drop num_workers_override / num_workers_per_node fields, their __post_init__ validation, the num_workers() override, and the xenna_stage_spec() override on NemoTarShardReaderStage. - nemo_curator/stages/audio/inference/qwen_omni.py: drop the num_workers_override field plus the num_workers() and xenna_stage_spec() overrides on InferenceQwenOmniStage. Remove the now-unused `Any` import. - tutorials/audio/qwen_omni_inprocess/README.md: drop the reader_num_workers / reader_num_workers_per_node / omni_num_workers rows from the configuration-knob table. - tests/stages/audio/inference/test_qwen_omni.py: remove test_worker_override_specs (no longer applicable). Addresses: - NVIDIA-NeMo#1967 (comment) Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Refactors `nemo_curator/stages/audio/metrics/performance.py` around three typed dataclasses and lifts every post-processing helper into a new module `nemo_curator/stages/audio/metrics/performance_utils.py`. Fixes four known anomalies in `perf_summary_merged.json` that the protocol-doc 3-way comparison flagged on the previous validation run (ac6598e). Anomaly fixes ============= P0 — `perf_invocations_counted = 0` (dedup wiring dead-code branch). Root cause: backends attach the same upstream `StagePerfStats` chain to every downstream task an actor invocation emits. The audio summary's dedup branch in `record_stage_perf` keyed on `StagePerfStats.invocation_id`, but `BaseStageAdapter` never populates that field, so the branch was dead and every perf record got summed once per emitted task. Fix (in `performance.py:record_stage_perf`): prefer `StagePerfStats.invocation_id` when the framework wires it, otherwise fall back to a deterministic value-tuple fingerprint (`_fingerprint_perf`) over (stage_name, process_time, actor_idle_time, num_items_processed, sorted custom-metric tuple). Repeat sightings of the same `StagePerfStats` produce byte-equal fingerprints → dedup correctly; two genuinely distinct invocations producing byte-equal tuples is astronomically unlikely. A2 — reader `custom_metrics_sum` over-counted by the fanout factor. Same root cause as P0. With the fingerprint fallback active, the reader's invocation_count collapses from ~68 k (every emitted utterance triple-counts the source invocation) to ~8 (one per unique shard), and `custom_metrics_sum.output_utterances` collapses from ~592 M to ~68 k, matching the top-level `total_utterances`. All per-process_s throughput ratios stay numerically identical because both numerator and denominator were scaled by the same factor. A4 — discovery `total_items_processed = 0` despite work being done. `NemoTarShardDiscoveryStage` is an actor-pattern stage: it synthesises output tasks from configuration (it doesn't consume one input task per output), so the framework's generic `num_items_processed` counter records zero. Adds an explicit `total_items_emitted` custom metric to the discovery stage's `_log_metrics` payload (audio-local; no framework change needed). The summary builder (`_build_stage_summary`) falls back to that value when the framework's `num_items_processed` is zero, so `throughput_items_per_s` and friends become meaningful for actor stages. A3 (`total_input_data_size_mb` mis-naming) remains intentionally out of scope — framework-level fix, recorded in protocol doc. A5 — QwenOmni `inference_time` duplicate + redundant `model_<key>` aliases. In `qwen_omni.py:process_batch` the metrics dict had both `inference_time_s` and `inference_time` for the same value, and the `model_<name>` alias loop emitted `model_audio_duration_s`, `model_waveform_bytes`, `model_output_tokens` etc. on top of the identical base-name keys the stage already emits. Drops the `inference_time` legacy key (kept `inference_time_s`); the alias loop now skips any name already present in `metrics`. Net: 6-7 fewer redundant keys per actor per run; model-side unique counters (`model_turn1_generation_time_s`, `model_turn1_prep_time_s`, `model_turn1_valid_inputs`, `model_utterances_skipped_preprocess`, turn-2 variants) are preserved. Refactor ======== * New `AudioStageMetrics` dataclass — superset of every scalar custom metric any audio stage emits across the codebase (~70 typed fields covering reader, discovery, inference, ALM, split/merge, diarization, VAD, writer) plus a forward-compat `extras: dict[str, float]` for any future emitted key. `from_dict()` / `to_dict()` round-trip cleanly; zeros are stripped from the published JSON so empty fields don't bloat the artifact. * New `AudioStageSamples` dataclass — per-invocation sample lists used to derive p50/p95 percentiles for batch_size, audio_duration_s, invocation_process_time_s, queue_wait_s (from `actor_idle_time`). Populated once per dedup'd invocation. * New `AudioStageCallerContext` dataclass — caller-provided fields the audio accumulator cannot derive itself (actor_count_samples, gpu_util_pct_samples, gpu_hours, setup_time_s_total, wallclock_s). Optional; a writer with NVML / DCGM / Xenna-autoscaler snapshots passes these in via `build_summary(stage_caller_context=…)`. Missing → field omitted from the published JSON (no misleading zeros). * `build_summary()` now also accepts `run_id` and `executor` kwargs and emits both the proposed pipeline-perf top-level shape (run_id, executor, input_hours, output_hours, rows_in, rows_out) AND the backward-compat keys the protocol-doc baselines depend on (total_utterances, total_audio_seconds, total_audio_hours, writer_wall_time_s, pipeline_audio_s_per_wall_s, pipeline_utterances_per_wall_s, perf_invocations_counted, shards, stages). The writer call site (`sharded_manifest_writer.py:_emit_summary`) does not need to change; new kwargs default to None / empty string. * `total_input_data_size_mb` is removed from the per-stage summary output (it was a framework-level message-size metric, misleading at the audio level — see A3 in protocol doc). `performance_utils.py` (new) ============================ Pure post-processing module — every audio-specific maths lives here so it can be unit-tested in isolation and reused by anything that consumes the perf JSON (CI throughput checks, Kratos harvesters, dashboards) without dragging the full accumulator in. Public surface: - `seconds_to_hours`, `hours_to_seconds`, `bytes_to_mb`, `bytes_to_gb` - `safe_ratio(num, denom)` — returns None when either ≤ 0 (consumer omits the field rather than carrying a misleading zero) - `add_ratio(entry, name, num, denom)` — writes to a dict only on positive result - `percentile(values, p)` — linear-interpolation percentile, no numpy dependency (works in writer pods without numpy on the import path) - `summarize_samples(values, name, percentiles)` — returns `{f"{name}_p50": …, f"{name}_p95": …}` - `audio_hours_per_gpu_hour(audio_s, gpu_s)` — domain composite for the proposed pipeline-perf shape - `items_per_hour(items, wall_s)` — generic throughput - `estimate_wallclock_s(total_proc, actor_count?, invocation_times?)` — best-effort stage wall: total_proc/actor_count → max(invocations) → fallback to CPU sum Validation ========== The next Kratos validation run will record (per protocol doc): * P0: perf_invocations_counted was 0 → must be > 0. * A2: reader.invocation_count was 68k → must be ≈ 8 (per shard). * A2: reader.custom_metrics_sum.output_utterances was 592 M → must be ≈ 68 819 (matches total_utterances). * A4: discovery.total_items_emitted was missing → must be ≈ 550 552; total_items_processed must fall back to that value. * A5: QwenOmni custom_metrics_sum must NOT contain `inference_time`, `model_audio_duration_s`, `model_waveform_bytes`, `model_output_tokens`, `model_turn1_output_tokens`, `model_turn2_output_tokens`, `model_utterances_input`. * Throughput-neutral on QwenOmni_inference (|delta| ≤ 0.70 % vs ac6598e baseline). Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Bring the SDP-V2 design doc's stage->adapter split (see "Replaceability: Stage -> Adapter" in the design doc) into PR NVIDIA-NeMo#1967 for the Qwen-Omni first-pass ASR slot. Pure refactor: no behaviour change, no metric-key change, no runtime-performance change expected. Layout changes -------------- New: nemo_curator/adapters/__init__.py nemo_curator/adapters/asr/__init__.py nemo_curator/adapters/asr/base.py - ASRAdapter Protocol + ASRResult dataclass nemo_curator/adapters/asr/qwen_omni.py - QwenOmniASRAdapter (vLLM setup, two-turn generate; core inference logic moved verbatim from the deleted nemo_curator/models/qwen_omni.py) nemo_curator/stages/audio/inference/asr/stage.py - generic ASRStage matching the design doc shape (Tier-1 adapter_target + model_id + stage knobs; Tier-2 opaque adapter_kwargs; adapter class resolved at setup() via hydra.utils.get_class, matching Curator framework convention in nemo_curator/config/run.py). Deleted: nemo_curator/stages/audio/inference/qwen_omni.py - replaced by ASRStage. nemo_curator/models/qwen_omni.py - body moved into QwenOmniASRAdapter. Other audio inference stages (asr_nemo.py, sortformer, pyannote, whisperx_vad, vad_segmentation, nemo_asr_align) are intentionally left unchanged - the doc's broader stage-adapter rollout is a follow-up PR series, not in PR NVIDIA-NeMo#1967's scope. YAML / tutorial --------------- tutorials/audio/qwen_omni_inprocess/qwen_omni_inprocess.yaml rewritten to the design-doc shape: _target_ now ASRStage, with new top-level adapter_target field (Tier-1 swap line) and an adapter_kwargs block (Tier-2 opaque) carrying the Qwen-Omni-specific knobs. The stage's name is explicitly overridden to QwenOmni_inference so perf_summary_merged.json's stages[QwenOmni_inference] key stays stable across the refactor, allowing direct apples-to-apples comparison against the existing 4e8021a / ac6598e / d8b1d751 / 8cd168ea baselines. Tests ----- tests/stages/audio/inference/test_qwen_omni.py rewritten end-to-end: 21 tests covering both ASRStage (process_batch contract, language resolution, skip handling, A5-alias-dedup metric filter preservation, adapter_target validation, outputs() with/without secondary_text_key, setup() via mocked hydra.utils.get_class, setup_on_node prefetch behaviour) and QwenOmniASRAdapter (transcribe_batch packaging contract, single-turn drops secondary_text, empty-vLLM-output helpers). Behaviour preservation ---------------------- Verified equivalent to pre-refactor for every observable surface: - task.data output keys (qwen3_prediction_s1, qwen3_prediction_s2, _skip_me, waveform drop on keep_waveform=False) - vLLM LLM(...) ctor args + generate() call sequence (moved verbatim) - Turn-1 + Turn-2 disfluency flow (gated by followup_prompt) - setup_on_node snapshot_download weight prefetch - Metric key shape including A5-fix alias-dedup filter - All P0 / A2 / A4 / A5 anomaly fixes from earlier commits Throughput-neutrality will be Kratos-validated next. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
…eform, vLLM knobs, batch policy) Builds on the 3c723c3 structural stage-adapter split. Brings the Qwen-Omni ASR stage into full compliance with the SDP-V2 design doc sections §6 (Qwen-Omni stage) and §0.3 (duration-bucketed batching). Doc deltas implemented ---------------------- (b) Stage-side pre-slice + stitch-back (§6 lines 765, 827). Any task whose audio duration exceeds the stage's per-turn cap is cut into contiguous, no-overlap, no-padding sub-chunks BEFORE the adapter sees it. The adapter still gets a flat list of items. The stage carries a parent-task index per item and joins the per-chunk text outputs back with a single-space join post-inference, so each input task still maps to exactly one output row. A new sub_chunks_generated metric surfaces the fan-out factor for transparency. utterances_input / utterances_processed continue to count PARENT tasks - the metric semantics downstream consumers rely on do not change. The parent task is marked skipped (_skip_me="empty_audio") only if EVERY sub-chunk was skipped; partial success keeps the parent live. (c) keep_waveform default flipped to True (§6 line 791). The granary-v2 default of False forced the §7 hallucination filter, §8 ASR-recovery, and §14 forced-alignment stages to re-decode the waveform from tarred shards. The SDP-V2 default lets them reuse the in-memory buffer the stage already holds. Tutorial YAML updated. (d) Elevated 4 vLLM / cache knobs to adapter_kwargs (§6 line 793). enable_prefix_caching (default True), prefix_caching_hash_algo (default "xxhash"), limit_mm_per_prompt_audio (default 2), and seed (default 1234) are now dataclass fields on the adapter and are threaded into the LLM(...) constructor. Defaults reproduce the pre-elevation behaviour byte-for-byte; deployments that want to flip the toggles no longer need a fresh image build. (§0.3) Best-effort within-call duration-bucketed batching. New BatchPolicy dataclass at nemo_curator/stages/audio/batch_policy.py. Left-edge bucketing matches the doc-literal buckets_sec layout. bucketize() re-partitions whatever item list a process_batch call assembles into bucket-homogeneous sub-batches honouring both the per-bucket item cap and a global audio-second budget, so a single vLLM call never mixes a 40-min sub-chunk with 5-sec sub-chunks. Cross-process_batch queueing (the full §0.3 model with a per-bucket flush timer) requires a Curator-framework scheduler hook and is a follow-up PR; flush_interval_ms is recorded on the dataclass for forward-compat. Naming + Tier-1 fields ---------------------- * secondary_text_key -> disfluency_text_key on ASRStage. Matches the doc §6 line 784 YAML knob name verbatim. ASRResult.secondary_text (the protocol-level field name) is unchanged - only the stage's user-facing knob renames. * New Tier-1 stage fields: ideal_inference_segment_s (defaults to 2400 s = 40 min, the Qwen-Omni per-turn cap from the doc), and max_inference_duration_s (defaults to ideal). Validation enforces max <= ideal so the bucket shape always stays anchored on ideal. * Tutorial YAML also exposes the new top-level revision field so the HuggingFace model can be pinned per-deployment. YAML + README ------------- tutorials/audio/qwen_omni_inprocess/qwen_omni_inprocess.yaml gains the batch_policy: block with the doc-literal shape buckets_sec: [0, 600, 1200, 2400] max_items_per_batch_by_bucket: [32, 16, 8, 4] max_audio_sec_per_batch: 2400 flush_interval_ms: 250 Tutorial README documents every new knob in the existing parameter table and adds a dedicated "Duration-aware bucketed batching" section. Tests ----- 24 new tests added (21 -> 45 total) covering: - pre-slice fan-out for an over-long clip ([30,30,30,5] for 95 s) - stitch-back single-space join on pred_text + disfluency_text - partial-skip vs all-skip parent labelling - utterances_input counts parents; sub_chunks_generated counts items - BatchPolicy bucket-homogeneous sub-batches, item-cap splits, audio-sec-cap splits, no-policy single-call fallthrough - BatchPolicy.bucket_for boundary semantics - BatchPolicy strategy/length validation rejects bad configs - new adapter knobs as dataclass fields; default values match doc - new adapter knobs threaded into vLLM LLM(...) ctor - keep_waveform=True default kept; keep_waveform=False still drops - max_inference_duration_s <= ideal_inference_segment_s validation Behaviour preservation ---------------------- All defaults reproduce the pre-elevation Qwen-Omni behaviour byte-for- byte: enable_prefix_caching=True, prefix_caching_hash_algo="xxhash", limit_mm_per_prompt_audio=2, seed=1234. A clip shorter than max_inference_duration_s is NOT sliced (single sub-chunk passes through verbatim). batch_policy=None reproduces the pre-§0.3 single-adapter-call shape. The throughput-neutrality of all four deltas will be Kratos- validated next. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Make adapter_target and model_id required on ASRStage, fix setup_on_node prefetch to call _adapter_class(), forward revision into vLLM and the processor in QwenOmniASRAdapter.setup(), guard optional audio_cuda12 imports with lazy asr package exports, skip bad manifest lines instead of failing the shard, and unpin fasttext in audio_cuda12. Expand unit tests (55-pass suite). Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Reword docstrings, tests, and tutorial docs so they describe behavior without design-doc section numbers or build-history framing. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
…s/inference Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Refresh the audio docs for the new inference architecture and remove the last back-compat surface for the relocated batch policy: - Audio stages developer guide (nemo_curator/stages/audio/README.md): new section on the generic BucketedInferenceStage, the ASRStage + ASRAdapter (Tier-1/Tier-2) split, and the canonical stages/inference/batch_policy home, plus a "which base to use" table. - Qwen-Omni tutorial README + audio tutorials index: surface the swappable-adapter ASR path and duration-bucketed batching. - Point the tutorial YAML batch_policy _target_ at the canonical nemo_curator.stages.inference.batch_policy.BatchPolicy. - Delete nemo_curator/stages/audio/batch_policy.py (the re-export shim); there is now a single canonical import path, no back-compat alias. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Address the latest PR NVIDIA-NeMo#1967 review threads: - Modularize Turn-1/Turn-2 vLLM inference (Sushmitha, r3354153072): extract _pack_vllm_inputs() for the shared prompt/mm-data packing and _infer_turn() for the shared generate -> count-tokens -> scatter-by-index sequence. _prepare_single / _prepare_turn2_single and _run_two_turn now reuse these helpers, removing the duplicated turn logic. Behavior is preserved; covered by new unit tests. - Fail loud on a broken vLLM engine contract (Greptile P1, r3354172564): switch the output-scatter zip() in _infer_turn to strict=True. A count mismatch between inputs and engine outputs now raises ValueError instead of silently emitting empty transcriptions with skipped=False. Added test_qwen_adapter_infer_turn_raises_on_vllm_count_mismatch and test_qwen_adapter_infer_turn_scatters_outputs_by_index. - Empty the package-marker __init__ files per convention (Sarah, r3357852783 / r3357870309): nemo_curator/adapters/__init__.py and nemo_curator/stages/inference/__init__.py. - Remove the duplicated "vLLM or Qwen utilities are missing" install section from the tutorial README (Sarah, r3357891807). Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Stamp actor/node/GPU identity via allocation-first resolution (Xenna allocation, Ray get_gpu_ids, gated CVD fallback). Expand perf_summary with per_gpu scheduling breakdown and pipeline_throughput. Harden tutorial resources overrides (no open-ended hydra instantiate) and discovery YAML structure validation. Reorganize tests by package path; update audio READMEs. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Explain that BaseStageAdapter meters every CPU and GPU stage onto
task._stage_perf, ShardedManifestWriterStage (num_workers=1) serializes
{shard}_perf.jsonl and perf_summary.json, write_perf_stats toggle,
_log_metrics author guide, and CPU vs GPU published JSON differences.
Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Replace internal milestone labels with public-facing perf terminology and remove Kratos/NvLLMOps/harvest-specific validation wording from READMEs, comments, and tests touched by PR NVIDIA-NeMo#1967. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Move actor/node/gpu label resolution into backends/perf_identity.py with separate Xenna (allocation-first) and Ray (get_gpu_ids only) builders. Each adapter stamps WorkerMetadata at setup; BaseStageAdapter copies those fields verbatim. Removes CUDA_VISIBLE_DEVICES and cross-backend fallback chain from base.py. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Split Ray label resolution into small helpers to avoid C901 complexity. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Document Xenna vs Ray resolution paths via backends/perf_identity.py without cross-backend fallback. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Keep gpu_id as the within-run per_gpu join key while stamping physical_address, pod_ip, hostname, gpu_indices, and gpu_uuids at worker setup for cluster-debuggable perf summaries. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Clarify WorkerPerfIdentity scheduling vs cluster-location metadata and the new per_gpu fields (physical_address, pod_ip, gpu_indices). Signed-off-by: Aaftab V <aaftabv@nvidia.com>
…tches Ray Data delivers task batches as numpy ndarrays; truthiness on ndarrays raises ValueError in BucketedInferenceStage. Coerce to list at the Ray adapter boundary and use len(tasks)==0 in the shared bucketed base. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
… fixes - ASRStage: Tier-1 xenna_num_workers / xenna_num_workers_per_node pin (mutually exclusive) to skip the autoscale cold-start ramp on the expensive GPU stage; surfaced via num_workers() (Ray Data) and xenna_stage_spec() (Xenna). - NemoTarShardReaderStage: pin to 1 worker (num_workers + IS_ACTOR_STAGE + per-node Xenna spec) for bounded memory under keep_waveform. - BatchPolicy: default max_audio_sec_per_batch 480 -> 2400 to match the Qwen-Omni docstring buckets. - qwen_omni tutorial: drop driver-side HF_TOKEN/prefetch (does not reach remote workers); rename omni_num_workers* -> asr_num_workers* (Tier-1, adapter-agnostic) with asr_num_workers_per_node: 2. - docs: "pin the GPU stage, autoscale the cheap stages" guidance plus adapter-swap GPU math. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
…ng, perf per-actor/per-GPU, reader/writer hardening Adapters + inference co-location (review NVIDIA-NeMo#3/NVIDIA-NeMo#25): - Move nemo_curator/adapters/asr/* -> stages/audio/inference/asr/adapters/* - Move stages/inference/{batch_policy,bucketed_stage}.py -> stages/audio/inference/ - Update all imports, YAML adapter_target / batch_policy._target_, READMEs, tests (tests mirror the new package paths). vLLM teardown (review NVIDIA-NeMo#15): - _cleanup_gpu() no longer calls torch.distributed.destroy_process_group(); vLLM owns its TP group, so tearing down the global PG could corrupt other components. Tutorial backend (review NVIDIA-NeMo#30): - Default backend -> ray_data (YAML + main.py); trim the "Choosing a Backend" section. - Drop tutorial test tests/tutorials/.../test_main_resources.py (review NVIDIA-NeMo#28). Performance reporting: - Unify metrics into a single per_actor block keyed by actor_id; GPU actors embed physical_address (host_ip:indices, now the canonical per-GPU key), gpu_indices/ gpu_uuids, and per-GPU utilization percentiles (nested gpus map). - Add first-class StagePerfStats.invocation_id for exact dedup; __add__ preserves identity only within the same worker. - Add inference_compute_fraction (inference time / total process time). - New GpuUtilSampler (NVML, bounded deque + lock); harden _collect_gpu_uuids to map physical indices -> visible ordinals under CUDA_VISIBLE_DEVICES. - Remove per-shard <output_dir>/<shard_key>_perf.jsonl output. Reader (nemo_tarred_reader): - Collision-safe _ManifestIndex (exact path, then longest path-suffix; ambiguous basenames resolve to no-match). - Enforce max_duration_s against decoded audio duration; narrow decode except to audio errors; write .done for empty/fully-filtered shards. Writer (sharded_manifest_writer): - Batch manifest writes per shard instead of per-utterance open/close. Misc: - Remove redundant super().setup_on_node() in RayActorPoolStageAdapter.__init__ (executor already runs once-per-node setup). - Fix stale docstring (disfluency_text_key); drop bucketize redundancy; trim adapter duplicate volume math; reduce comment verbosity across PR files. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Add a raw-manifest Qwen-Omni tutorial that reuses the generic Hydra runner with a raw-audio stage graph: ManifestReader -> ResampleAudioStage -> MonoConversionStage -> ASRStage -> ShardedManifestWriterStage. Stamp shard metadata in ManifestReaderStage so raw JSONL input shards can complete through ShardedManifestWriterStage and rebuild final manifests. Add tests covering raw reader shard keys and totals. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Emit 1D numpy waveforms from MonoConversionStage when explicitly requested, then opt the raw Qwen pipeline into that format so it matches the working NeMo tar shard reader contract without changing Qwen Omni or ASR inference code. Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Signed-off-by: Aaftab V <aaftabv@nvidia.com>
Signed-off-by: Mohammad Aaftab <aaftaabv@gmail.com>
Signed-off-by: aaftaabv@gmail.com <aaftaabv@gmail.com>
Keep the generic ASR adapter split and reference sharded I/O while removing superseded bucketing, NeMo ASR, and tutorial surfaces. Signed-off-by: aaftaabv@gmail.com <aaftaabv@gmail.com>
Keep the adapter and observability functionality focused by dropping config redaction, model-input segmentation, mono-conversion extensions, and organization-only files from the PR surface. Signed-off-by: aaftaabv@gmail.com <aaftaabv@gmail.com>
Signed-off-by: aaftaabv@gmail.com <aaftaabv@gmail.com>
Prevent serialization failures and decode errors from corrupting shard completion, isolate malformed discovery entries, and keep unsupported-only ASR metrics accurate. Signed-off-by: aaftaabv@gmail.com <aaftaabv@gmail.com>
Reuse the normal writer metrics path so tarred pipelines preserve actor, GPU, and hardware evidence in their terminal artifacts. Signed-off-by: aaftaabv@gmail.com <aaftaabv@gmail.com>
Keep PR1967 focused on functional inference and sharded I/O while preserving the observability work for a dedicated follow-up. Signed-off-by: aaftaabv@gmail.com <aaftaabv@gmail.com>
Restore backend identity, GPU sampling, audio summaries, and manifest writer performance output as a dedicated follow-up to the core Qwen ASR pipeline. Signed-off-by: aaftaabv@gmail.com <aaftaabv@gmail.com>
Keep backend and pipeline metrics independent of model implementation while emitting terminal summaries through the regular audio manifest writer. Signed-off-by: aaftaabv@gmail.com <aaftaabv@gmail.com>
Signed-off-by: aaftaabv@gmail.com <aaftaabv@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ManifestWriterStageScope boundary
This PR is performance-only. It contains no Qwen adapter, generic ASR implementation, sharded writer, payload lifecycle, dispatch envelope, or global planner.
Test plan
103 passedfinal focused run)main