Skip to content

fix(paraformer): correct timestamp alpha and peak argument order - #3716

Open
xx205 wants to merge 1 commit into
modelscope:mainfrom
xx205:fix/paraformer-timestamp-argument-order
Open

xx205 wants to merge 1 commit into
modelscope:mainfrom
xx205:fix/paraformer-timestamp-argument-order

Conversation

@xx205

@xx205 xx205 commented Sep 17, 2026

Copy link
Copy Markdown

Summary

Vanilla Paraformer passes CIF peaks as us_alphas and alpha weights as us_peaks to ts_prediction_lfr6_standard(). This makes the helper search for boundary crossings in alpha weights and, when the boundary count does not match, normalize the CIF peak trajectory as though it were alpha weights. It can silently return the expected number of segments with incorrect boundaries.

The predictor returns (acoustic_embeds, token_num, alphas, cif_peak), while the helper expects (us_alphas, us_peaks, char_list). The reversed call was introduced in ce6b70e and is present at the base revision 615ef05.

  • Correct the caller using explicit us_alphas, us_peaks, and char_list keyword arguments.
  • Preserve per-sample padding trimming, tail handling, VAD offset, and upsample_rate=1.
  • Strengthen the existing padding tests to check distinct tensor contents and per-sample slices, not only shapes.
  • Exercise Paraformer.inference() with the real CIF implementation, timestamp helper, and text postprocessing, while stubbing encoder/predictor/decoder outputs. Cover both sufficient boundary counts and alpha-normalization fallback, with and without a VAD offset.

For the synthetic fallback fixture, the same CIF outputs produce:

Argument order Timestamps (ms)
Before: peaks, alphas [[329, 690], [690, 1080]]
After: alphas, peaks [[90, 510], [510, 1080]]

Both return two segments without raising an exception. Fallback itself is legitimate when a token sequence needs an additional boundary; the defect is feeding it the wrong trajectory.

Type of change

  • Bug fix

Validation

Both sides use the same local environment: macOS 26.5.1 arm64, CPU.

Component Version used for both sides
Python 3.9.6
FunASR package metadata 1.4.15, editable source checkout
PyTorch / torchaudio 2.8.0 / 2.8.0
NumPy / SciPy 2.0.2 / 1.13.1
librosa / soundfile 0.11.0 / 0.13.1
ModelScope 1.37.1
pytest 8.4.2
Matplotlib (figure generation only) 3.9.4

Source baseline: 615ef059c989fca12a546505abb22c5c5b6f329b.
Fix commit: cbce0a2ca1bf8813fa41e3edde7e05fc2b7bbc90.
The version string remains 1.4.15 on both source revisions; this is not a comparison between two PyPI package releases.

python -m pytest tests/test_paraformer_timestamp_contract.py tests/test_paraformer_timestamp_padding.py tests/test_timestamp_tools.py -q --tb=short

Unfixed production code + updated tests: 7 failed, 3 passed
Fixed production code + updated tests:  10 passed
  • python -m compileall -q funasr examples tests
  • git diff --check

The local test run emits one urllib3/LibreSSL environment warning. No real checkpoint/audio alignment accuracy claim is made by these synthetic tests.

Real-audio comparison

Tested clean_0005_0014_output.wav, a sample audio clip (22.8 s, mono, 24 kHz; resampled by FunASR to 16 kHz). SHA-256: f7ac517fa30531430af4d43650e9d2905df07e53ad4c01b1ee0f251f1b457461.

The vanilla checkpoint was iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch, with model-weight SHA-256 5bba782a5e9196166233b9ab12ba04cadff9ef9212b4ff6153ed9290ff679025. Both real-audio runs used the fixed checkout: the before run replayed the old behavior by swapping only the helper's alpha/peak arguments, while the after run used the corrected order. Captured alpha tensors, peak tensors, and token inputs were exactly equal between the two runs in each configuration; the 84-token transcripts were unchanged. This isolates the caller change rather than comparing separately installed package versions.

Vanilla configuration Largest absolute start change Largest absolute end change
Whole audio, no VAD 8.100 s 8.100 s
FSMN-VAD v2.0.4, batch_size_s=300 6.300 s 6.179 s

With VAD enabled, token #57 (zero-based), , starts at 11.690 s before the fix and 17.990 s after it. These are measured timestamp changes, not errors measured against human alignment labels.

Input audio: clean_0005_0014_output.wav

Before text is above the waveform; after text is below it. Both rows use the actual output timestamps (vanilla + FSMN-VAD).

Before and after text aligned to the audio waveform

Download per-token comparison results, exact environment, and reproduction script. Reproduction instructions and checkpoint revisions are provided with the evidence. These attachments are hosted on the contributor fork; no audio or model files are included in the code diff.

User impact

Corrects timestamp generation for vanilla Paraformer when timestamps are requested. The helper API and transcription path with timestamps disabled are unchanged.

Notes for reviewers

Scope is limited to vanilla Paraformer. E-Paraformer has a similar reversed call, but PifPredictor returns cif_peak=None; timestamp support for that path requires separate handling.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant