addon.node: VAD-aligned (faster-whisper-like) timeline with real gaps#2
Merged
Merged
Conversation
added 2 commits
June 25, 2026 18:03
whisper_full_get_token_data().t0/t1 are in VAD "processed" time when VAD is enabled (silence removed), so only segment timestamps were mapped back to the original timeline; callers had no way to get token/word-level times on the original timeline. Add whisper_full_get_token_t0/t1 (+ _from_state) which apply the same vad_mapping_table that the segment getters use. With VAD off, or when no mapping table exists, they return the raw token times, so existing behavior is unchanged.
With VAD enabled, whisper.cpp concatenates all detected speech into a
single stream, so addon.node returned a gap-less timeline where every
segment end equals the next segment start. Add an alignment layer that
puts timestamps back on the original timeline with real silence gaps,
controlled by three new params:
align_mode "hybrid" (default) | "run" | "word" | "legacy"
vad_merge_gap_ms adjacent VAD segments whose silence gap is <= this (ms)
merge into one run; a larger gap becomes a real gap
(default 2000; negative disables aligned mode)
word_gap_ms word/hybrid: start a new segment when the gap between
two consecutive words exceeds this (default 500)
- hybrid (C): VAD-grouped runs, each run sliced into its own buffer and
re-segmented by word-level gaps with every segment end clamped to its
last word. Best boundary accuracy; default.
- run (A): per-run decode emitting whisper's own segments; gaps between runs.
- word (B): single decode pass + word-gap re-segmentation; uses core VAD
via the new whisper_full_get_token_t0/t1 mapping when a VAD model is given.
- legacy: original continuous single pass.
Each run is decoded from a physically sliced buffer rather than via
offset_ms/duration_ms (which only bound the outer seek loop and let
neighbouring speech bleed into short runs); slice-relative timestamps are
shifted back with a per-run base offset. Progress is rescaled across runs
so the JS callback still sees a single monotonic 0..100.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With VAD enabled, whisper.cpp concatenates all detected speech into a single stream, so
addon.nodereturned a gap-less timeline where every segment end equals the next segment start. This PR adds an alignment layer that puts timestamps back on the original timeline with real silence gaps, similar to faster-whisper.New params (all optional, backward compatible)
align_mode"hybrid"hybrid|run|word|legacyvad_merge_gap_ms2000word_gap_ms500Modes
whisper_full_get_token_t0/t1mapping when a VAD model is givenImplementation notes
offset_ms/duration_ms(those only bound the outer seek loop and let neighbouring speech bleed into short runs). Slice-relative timestamps are shifted back onto the original timeline with a per-run base offset.0..100.whisper_full_get_token_t0/t1added in this branch.Test plan
Verified on 4 synthetic audios (multi-gap / sub-split / leading+trailing silence / choppy) × 4 modes:
vad_merge_gap_ms) become real empty spans;hybridhas the most accurate boundaries;legacystays continuous (unchanged behavior).