Skip to content

[Bug] Paraformer silently generates incorrect timestamps because CIF alphas and peaks are passed in reverse order #3723

Description

@xx205

🐛 Bug

Paraformer.inference() passes the two timestamp-related outputs from the CIF predictor to ts_prediction_lfr6_standard() in reverse order.

calc_predictor() returns:

(acoustic_embeds, token_num, alphas, cif_peak)

but the current caller effectively does:

ts_prediction_lfr6_standard(
    timestamp_pre_peak_index,  # cif_peak
    timestamp_alphas,          # alphas
    copy.copy(token),
    vad_offset=kwargs.get("begin_time", 0),
    upsample_rate=1,
)

while the helper contract is:

ts_prediction_lfr6_standard(us_alphas, us_peaks, char_list, ...)

Therefore CIF peaks are treated as alpha weights, and alpha weights are treated as peaks.

This normally does not raise an exception. The helper can enter its normalization fallback and still return the expected number of timestamp segments, so the result looks structurally valid while its boundaries are incorrect.

A fix with regression tests is already available in #3716.

To Reproduce

This deterministic example uses the real CIF implementation and the real timestamp helper:

import torch

from funasr.models.paraformer.cif_predictor import cif
from funasr.utils.timestamp_tools import ts_prediction_lfr6_standard

weights = [0.1, 0.1, 0.4, 0.6] + [0.0] * 4 + [0.1] * 10
alphas = torch.tensor([weights])
hidden = torch.zeros(1, len(weights), 2)

_, peaks = cif(hidden, alphas, threshold=1.0)
tokens = ["你", "你"]

# Current Paraformer argument order
_, current = ts_prediction_lfr6_standard(
    peaks[0],
    alphas[0],
    tokens,
    upsample_rate=1,
)

# Order required by the helper API
_, expected = ts_prediction_lfr6_standard(
    alphas[0],
    peaks[0],
    tokens,
    upsample_rate=1,
)

print("current:", current)
print("expected:", expected)

Observed output:

current:  [[329, 690], [690, 1080]]
expected: [[90, 510], [510, 1080]]

Both calls return two segments. Consequently, tests that verify only output shape or segment count do not detect the defect.

Expected behavior

The caller should use explicit keyword arguments:

timestamp_str, timestamp = ts_prediction_lfr6_standard(
    us_alphas=timestamp_alphas,
    us_peaks=timestamp_pre_peak_index,
    char_list=copy.copy(token),
    vad_offset=kwargs.get("begin_time", 0),
    upsample_rate=1,
)

Per-sample padding trimming, VAD offsets, and the helper's legitimate alpha-normalization fallback should remain unchanged.

Regression tests should validate tensor identity/content and actual timestamp boundaries rather than only tensor shapes.

Impact

A real-audio A/B test used the same:

  • checkpoint and model checkout;
  • input audio;
  • captured alpha tensors;
  • captured peak tensors;
  • token inputs;
  • 84-token transcript.

Only the argument order supplied to the timestamp helper was changed.

Configuration Largest absolute start change Largest absolute end change
Whole audio, no VAD 8.100 s 8.100 s
FSMN-VAD v2.0.4, batch_size_s=300 6.300 s 6.179 s

These are measured timestamp-output changes caused by the argument order, not an alignment-accuracy claim against human labels.

Full reproduction evidence, environment details, audio, and per-token comparisons are available in:

Scope

This issue is limited to the vanilla Paraformer inference timestamp path.

E-Paraformer has a superficially similar reversed call, but its PifPredictor currently returns cif_peak=None; timestamp support for that path requires separate handling.

The transcription path with timestamps disabled is unaffected.

Environment

  • OS: macOS 26.5.1 arm64
  • Python: 3.9.6
  • FunASR package metadata: 1.4.15, editable source checkout
  • PyTorch / torchaudio: 2.8.0 / 2.8.0
  • ModelScope: 1.37.1
  • Device: CPU
  • Validated source baseline: 615ef059c989fca12a546505abb22c5c5b6f329b
  • Fix commit: cbce0a2ca1bf8813fa41e3edde7e05fc2b7bbc90

Related but not duplicate: #1226 reported inaccurate Paraformer timestamps as a user-visible symptom; this issue identifies a specific argument-order defect in the current vanilla Paraformer inference path.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds triageNeeds maintainer triage and routing

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions