ASR: /v1/audio/transcriptions ignores response_format — timestamps are computed internally but discarded

[LalaDango]

Hi! Thanks for FLM — the NPU ASR path has been really useful for me.

While reading the source I noticed that /v1/audio/transcriptions always returns a minimal {model, text} object, and that most of the pieces needed for OpenAI's verbose_json response already exist inside the Whisper engine but are dropped at the REST layer. Filing this in case it's simply "not wired up yet" rather than a deliberate decision — apologies if it's already on the roadmap.

Checked against master @ 8b8399c. Environment: FLM v0.9.45, Windows, AMD Ryzen AI 5 340 — though this comes from reading the source rather than a runtime failure, so the environment probably isn't relevant here.

Current behavior

response_format is never read on the server side (grepping for response_format / verbose_json in src/ returns nothing), so a client asking for verbose_json silently gets plain text back. Anything that needs segment timing has to re-chunk the audio client-side and stitch the offsets back together.

Handler: https://github.com/FastFlowLM/FastFlowLM/blob/8b8399c1f259d31bba2c42c2e3038c2b69856b89/src/server/rest_handler.cpp#L1290-L1301

std::pair<std::string, std::string> audio_result =
    this->whisper_engine->generate(Whisper::whisper_task_type_t::e_transcribe, true, false, std::cout);
//                                                                            ^^^^  ^^^^^
//                                                    enable_time_stamp=true, return_time_stamp=false

std::string audio_context = audio_result.first;
...
response = { {"model", model}, {"text", audio_context} };

What the engine already produces

1. Timestamps are computed. When return_time_stamp is true, decoded timestamps are offset per chunk and appended to the result string:
https://github.com/FastFlowLM/FastFlowLM/blob/8b8399c1f259d31bba2c42c2e3038c2b69856b89/src/common/whisper/modeling_whisper.cpp#L170-L174

2. enable_time_stamp cannot be the exposure switch — it is structurally required for the sliding-window chunking, since the next chunk boundary is derived from the last timestamp:
https://github.com/FastFlowLM/FastFlowLM/blob/8b8399c1f259d31bba2c42c2e3038c2b69856b89/src/common/whisper/modeling_whisper.cpp#L233

So the only knob that controls exposure is return_time_stamp, and it is hardcoded to false at the single call site.

3. The detected language is already returned, but the handler only uses .first:
https://github.com/FastFlowLM/FastFlowLM/blob/8b8399c1f259d31bba2c42c2e3038c2b69856b89/src/common/whisper/modeling_whisper.cpp#L247

return std::make_pair(result, langmap::to_language_name(language_detected));

4. Audio duration is already known — load_audio() logs Length of audio: N seconds.

Suggestion

Read response_format in handle_openai_audio_transcriptions, and for verbose_json pass return_time_stamp=true, parse the <|x.xx|> markers out of the result string into segments[], and fill language / duration from what the engine already hands back:

{
  "task": "transcribe",
  "language": "japanese",
  "duration": 123.4,
  "text": "...",
  "segments": [
    { "id": 0, "start": 0.00, "end": 4.20, "text": "..." }
  ]
}

json and text formats would keep the current behavior, so this should be backward compatible.

Why it matters

Without segments, subtitle generation and long-audio alignment require re-implementing chunking on the client side, which duplicates work the engine is already doing (and can drift from the engine's own chunk boundaries).

Related: #234 — surfacing the detected language would partly address that request as well.

Thanks for the great work!