Skip to content

addon.node: VAD-aligned (faster-whisper-like) timeline with real gaps#2

Merged
buxuku merged 2 commits into
feature/addon-cancelfrom
feature/vad-aligned-timeline
Jun 25, 2026
Merged

addon.node: VAD-aligned (faster-whisper-like) timeline with real gaps#2
buxuku merged 2 commits into
feature/addon-cancelfrom
feature/vad-aligned-timeline

Conversation

@buxuku

@buxuku buxuku commented Jun 25, 2026

Copy link
Copy Markdown
Owner

Summary

With VAD enabled, whisper.cpp concatenates all detected speech into a single stream, so addon.node returned a gap-less timeline where every segment end equals the next segment start. This PR adds an alignment layer that puts timestamps back on the original timeline with real silence gaps, similar to faster-whisper.

New params (all optional, backward compatible)

param default meaning
align_mode "hybrid" hybrid | run | word | legacy
vad_merge_gap_ms 2000 adjacent VAD segments whose silence gap is ≤ this (ms) merge into one decode "run"; a larger gap becomes a real subtitle gap. Negative disables aligned mode.
word_gap_ms 500 (word/hybrid) start a new segment when the gap between two consecutive words exceeds this (ms).

Modes

mode gaps between regions word-level sub-split boundary accuracy notes
hybrid (default) yes yes best VAD-grouped runs, each run sliced into its own buffer and re-segmented by word gaps; every segment end clamped to its last word
run yes no good per-run decode emitting whisper's own segments
word yes yes ok single decode pass + word-gap re-segmentation; uses core VAD via the new whisper_full_get_token_t0/t1 mapping when a VAD model is given
legacy no (original) no original continuous single pass

Implementation notes

  • Each run is decoded from a physically sliced audio buffer, not via offset_ms/duration_ms (those only bound the outer seek loop and let neighbouring speech bleed into short runs). Slice-relative timestamps are shifted back onto the original timeline with a per-run base offset.
  • Progress is rescaled across runs so the JS progress callback still sees a single monotonic 0..100.
  • VAD context is forced onto CPU (matches whisper.cpp internals; the Metal VAD graph aborts).
  • Depends on the core getters whisper_full_get_token_t0/t1 added in this branch.

Test plan

Verified on 4 synthetic audios (multi-gap / sub-split / leading+trailing silence / choppy) × 4 modes:

  • big gaps (> vad_merge_gap_ms) become real empty spans;
  • no text is emitted inside silence (incl. leading/trailing);
  • hybrid has the most accurate boundaries;
  • legacy stays continuous (unchanged behavior).

linxiaodong added 2 commits June 25, 2026 18:03
whisper_full_get_token_data().t0/t1 are in VAD "processed" time when VAD
is enabled (silence removed), so only segment timestamps were mapped back
to the original timeline; callers had no way to get token/word-level times
on the original timeline.

Add whisper_full_get_token_t0/t1 (+ _from_state) which apply the same
vad_mapping_table that the segment getters use. With VAD off, or when no
mapping table exists, they return the raw token times, so existing
behavior is unchanged.
With VAD enabled, whisper.cpp concatenates all detected speech into a
single stream, so addon.node returned a gap-less timeline where every
segment end equals the next segment start. Add an alignment layer that
puts timestamps back on the original timeline with real silence gaps,
controlled by three new params:

  align_mode       "hybrid" (default) | "run" | "word" | "legacy"
  vad_merge_gap_ms adjacent VAD segments whose silence gap is <= this (ms)
                   merge into one run; a larger gap becomes a real gap
                   (default 2000; negative disables aligned mode)
  word_gap_ms      word/hybrid: start a new segment when the gap between
                   two consecutive words exceeds this (default 500)

- hybrid (C): VAD-grouped runs, each run sliced into its own buffer and
  re-segmented by word-level gaps with every segment end clamped to its
  last word. Best boundary accuracy; default.
- run (A): per-run decode emitting whisper's own segments; gaps between runs.
- word (B): single decode pass + word-gap re-segmentation; uses core VAD
  via the new whisper_full_get_token_t0/t1 mapping when a VAD model is given.
- legacy: original continuous single pass.

Each run is decoded from a physically sliced buffer rather than via
offset_ms/duration_ms (which only bound the outer seek loop and let
neighbouring speech bleed into short runs); slice-relative timestamps are
shifted back with a per-run base offset. Progress is rescaled across runs
so the JS callback still sees a single monotonic 0..100.
@buxuku buxuku merged commit f040d8c into feature/addon-cancel Jun 25, 2026
43 of 51 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant