AudioToTextProvider: preserve segments and speaker labels in transcript output (verbose_json is already requested)

[khnjrdm]

Describe the feature you'd like to request

The transcribe() method in lib/Service/OpenAiAPIService.php (line ~812 in v4.5.1) sends response_format = 'verbose_json' on every audio transcription request. The comment above the request explains why: "Verbose needed for extraction of audio duration". After using the last segment's end field for quota accounting, the entire segments[] array is discarded and only $response['text'] is returned to callers (line ~838).

Every transcription call therefore pays for:

• Larger response body (segments[] with per-segment start / end / text and, on newer STT providers, speaker labels)
• Extra JSON parsing work

...but the output surface only exposes what a plain response_format = 'json' request would have returned. The extra data is fetched, parsed, and thrown away.

For Talk call recording transcripts specifically, this loses meaningful value:

• No timestamps in Recording <ts> - transcript.md — impossible to jump to "when was this discussed" in a longer meeting
• No speaker attribution — the transcript reads as a flat text blob even when the STT provider returned per-segment speaker labels
• No paragraph structure — one long string without natural breaks makes the transcript harder to skim and harder for the summariser to shape

Framed as a code smell: if the segments data isn't worth exposing to callers, request json and save bandwidth. If it is worth having — which the current implementation seems to concede by requesting verbose_json — please make it available in the output.

Environment where observed:

• Nextcloud 33.0.6 (Hub 26 Winter)
• integration_openai 4.5.1
• Talk 23.0.8
• STT backend: LocalAI-compatible endpoint (LiteLLM proxy + local Whisper on Intel NPU with speaker diarization) — url config points at the LiteLLM endpoint. Same shape applies to the OpenAI cloud Whisper API — its verbose_json response also returns segments[].start / segments[].end, and the OpenAI API is progressively adding richer diarization info to newer models.

Describe the solution you'd like

In OpenAiAPIService::transcribe(), after the existing quota accounting, format the segments into the returned string when they're present. Fall back to $response['text'] when they're not (defensive against non-shim providers or future API changes):

// After the existing quota accounting
if (isset($response['segments']) && is_array($response['segments']) && !empty($response['segments'])) {
    $lines = [];
    $currentSpeaker = null;
    $currentLine = '';
    $lastTsAt = -30.0;  // ensures first ts is always emitted
    foreach ($response['segments'] as $segment) {
        $segText = trim($segment['text'] ?? '');
        if ($segText === '') { continue; }
        $speaker = $segment['speaker'] ?? '';
        $start = floatval($segment['start'] ?? 0);
        $totalSeconds = intval($start);
        $ts = sprintf('%02d:%02d:%02d',
            intdiv($totalSeconds, 3600),
            intdiv($totalSeconds % 3600, 60),
            $totalSeconds % 60);
        $newTurn = ($speaker !== $currentSpeaker) || ($start - $lastTsAt >= 30.0);
        if ($newTurn) {
            if ($currentLine !== '') { $lines[] = $currentLine; }
            $speakerPrefix = $speaker !== '' ? "SPEAKER_{$speaker}: " : '';
            $currentLine = "[{$ts}] {$speakerPrefix}{$segText}";
            $currentSpeaker = $speaker;
            $lastTsAt = $start;
        } else {
            $currentLine .= ' ' . $segText;
        }
    }
    if ($currentLine !== '') { $lines[] = $currentLine; }
    if (!empty($lines)) {
        return implode("\n", $lines);
    }
}

return $response['text'];  // Fallback

Example output for a real 2-speaker Swedish Talk recording (22 minutes, real conversation):

[00:00:00] SPEAKER_A: Tanken var så här...
[00:00:30] SPEAKER_A: Om vi har ett country field som är generiskt, som inte ska vara filtrerat...
[00:01:12] SPEAKER_B: Ja, men exakt.
[00:01:15] SPEAKER_A: Men tanken är så här, jag vill välja en leverantör...

Formatting choices in this sketch — any of them negotiable:

• [HH:MM:SS] prefix — whole seconds are sufficient for meeting-minute reading. Users needing subtitle-grade precision would still use response_format = 'srt' | 'vtt' directly.
• SPEAKER_A: labels only when speaker field is present — fully backward compatible when it isn't (older OpenAI models, non-diarizing providers).
• Consecutive same-speaker segments joined with spaces — avoids one-line-per-Whisper-chunk noise; segments already come pre-punctuated.
• Fresh [HH:MM:SS] every 30s within a long same-speaker turn — lets users navigate long monologues.

Why this isn't asking for admin configuration. Deliberately distinct from #405 (SummaryProvider prompt customization, declined as out of scope): this request asks for consistent output improvement for all users, not admin-level toggles. No .env variables, no UI settings, no per-deployment risk. Same output shape for Talk call recordings, Assistant "Transcribe audio", and every other STT path. Same behaviour whether the backend is LocalAI or cloud OpenAI. If a specific caller ever needs the plain text back, explode("\n", $text) and stripping [HH:MM:SS] SPEAKER_X: prefixes trivially recovers it — this is strictly additive to the current output.

Also affects the Assistant "Transcribe audio" task, not just Talk recordings. Users who paste audio into the Nextcloud Assistant's transcription tool will also get the timestamped, speaker-attributed output. Usually a plus — makes ad-hoc transcriptions much more useful.

Describe alternatives you've considered

1. Hard-patching OpenAiAPIService.php locally on every Nextcloud instance. What I'm doing now. Fragile: patch gets clobbered on every integration_openai app update, has to be manually re-applied. Not maintainable across a fleet.

2. Wrapping the STT endpoint on the shim side to inline timestamps and speaker labels into the plain text field. Technically works, but abuses the plain-text semantics of response_format = 'json' and forces the LocalAI-side toolchain to make Nextcloud-specific formatting decisions. Doesn't help users of the cloud OpenAI API.

3. Adding a new response format parameter and letting callers request AudioToTextProvider-style output. Overkill — it doubles the API surface for one output shape. Simpler to make the improved output the default and provide the escape hatch in the (unlikely) case a caller needs the flat text.

Related:

• RefreshModels cron hits api.openai.com every cycle when use_localai=1 is set #403 — RefreshModels cron hits api.openai.com because per-service URL fallback skips the global url (filed same day, still open)
• Add admin-configurable summary system prompt (SummaryProvider currently hardcodes it) #405 — SummaryProvider hardcodes summary prompt (filed same day, declined as out of scope)

[pmarini-nc]

Thanks for opening this @khnjrdm , this is a much needed improvement to transcription outputs.

I opened a similar request in stt_whisper2 last year, nextcloud/stt_whisper2#40, but we are currently focusing on LocalAI transcriptions, integrated via integration_openai.

[khnjrdm]

Thanks @pmarini-nc, glad this lines up with what you're already working on. Reading through nextcloud/stt_whisper2#40, that's very close to the shape of transcript we ended up building locally.

To share where we landed, at a high level.

User feedback on the current shape. We've had pushback from users on the default output for Talk recordings. What lands in Recording <ts> - transcript.md reads as one long text blob, and the summary Talk stores alongside it is typically a couple of sentences of generic phrasing. In our experience, that hasn't been enough to serve as meeting minutes people are willing to reference the next day. If Talk recordings are meant to be an "attend and reference later" workflow, the output shape probably needs to change.

What we're running locally. Stack:

• Nextcloud 33.0.6, Talk 23.0.8, integration_openai 4.5.1
• LiteLLM as an OpenAI-shape proxy in front of a locally hosted Whisper on Intel NPU
• Custom shim wrapping openvino_genai.WhisperPipeline. Runs Whisper on NPU concurrently with sherpa-onnx speaker diarization on CPU, merges the two, returns OpenAI-compatible verbose_json with per-segment speaker labels
• Multi-sample language detection in the shim. Samples a few windows across the audio, runs each through Whisper autodetect, and majority-votes the language. This lets calls that don't specify a language get resolved server-side, so we're not depending on per-user STT language settings, which we found brittle for mixed-language teams and for users who forget to set it
• Two hot-patches on the Nextcloud side. One in SummaryProvider.php to replace the hardcoded prompt with a meeting-aware structured Markdown prompt (see Add admin-configurable summary system prompt (SummaryProvider currently hardcodes it) #405). One in OpenAiAPIService.php (this ticket) to format segments as [HH:MM:SS] SPEAKER_A: text instead of dropping them.

Both PHP patches get clobbered on every app update, which is why we filed both upstream.

What the output looks like in production. Transcript reads roughly (English rendering of a Swedish sample):

[00:00:00] SPEAKER_A: The idea was that if we have a generic country field, we should be able to filter and search on it...
[00:01:12] SPEAKER_B: Yes, exactly.
[00:01:15] SPEAKER_A: But the thinking is that we should pick a supplier with a strong supply chain and one with a weak one to show the difference...

The summary comes out as structured Markdown minutes with attendees, decisions, action items, open questions, and next steps (via the patched prompt).

What we're still working on. Speaker identity, not just separation. Right now SPEAKER_A stays consistent within a single recording, but there is no cross-recording persistence and no mapping to real user names. Our next step is voice fingerprinting so SPEAKER_A resolves to "John" automatically. That's a bigger piece of work, and the segments plus speaker plumbing this ticket asks for is the foundation everything else would build on.

Happy to share anything from our setup as a data point if it helps shape the upstream design. Also happy to test any candidate patch on the same recording set we've been iterating against.

[lukasdotcom]

Hi @khnjrdm and @pmarini-nc ,

We have discussed improving the transcripts with speaker diarization and timestamps, but haven't had the time to implement this. We would definitely love supporting this though. We do currently have the same limitations as discussed in nextcloud/stt_whisper2#40

[khnjrdm]

Hi @khnjrdm and @pmarini-nc ,

We have discussed improving the transcripts with speaker diarization and timestamps, but haven't had the time to implement this. We would definitely love supporting this though. We do currently have the same limitations as discussed in nextcloud/stt_whisper2#40

Thanks @lukasdotcom I'll head over to the thread mentioned and dive into the topic from there.