Add native microphone transcription and Ask with voice

[leehack]

Summary

• add foreground microphone capture to the Flutter chat example with two separate, model-dependent workflows:
  • Qwen3-ASR transcription uses the typed SpeechToTextEngine from Add experimental native speech-to-text support #326 and produces a final transcript
  • Ask with voice sends recorded audio through ordinary Gemma 4 multimodal chat and produces an assistant answer, not a transcript
• expose the SHA-256-pinned Qwen3-ASR 0.6B model/projector across native mobile and desktop, while keeping Web explicitly unsupported
• support native Gemma voice questions through:
  • Gemma 4 E2B LiteRT-LM direct media
  • Gemma 4 GGUF models with a configured, loaded projector that reports runtime audio support
• align Gemma 4 presets with the pinned Unsloth recommendation (temperature=1.0, top_k=64, top_p=0.95, min_p=0, repetition penalty 1.0, thinking off by default) while honoring active user settings
• normalize template-facing OpenAI input_audio parts to Hugging Face/Jinja {type: "audio"} without changing public wire-style toJson() output
• harden native model downloads with immutable revisions, artifact hashes, persisted integrity stamps, and representation-safe resume handling
• retain the selected LiteRT-LM text/vision backend while retrying an incompatible audio executor on CPU
• recover validated app-managed iOS catalog paths after an app-container UUID changes; arbitrary custom paths remain untouched
• split model-library browsing into Mobile, Web, and Desktop filters

Related to #325. This PR does not close it.

Follow-ups:

• live/windowed/stateful transcription: Add live speech-to-text sessions with explicit accumulative, windowed, and stateful modes #327
• typed Qwen3-ASR WebGPU validation: WebGPU: validate and enable typed Qwen3-ASR speech-to-text #329
• TTS: Design experimental native speech synthesis API for Qwen3-TTS #322
• browser Ask with voice remains separate from typed STT and is not covered by WebGPU: validate and enable typed Qwen3-ASR speech-to-text #329

Product behavior

Qwen3-ASR: typed transcription

• selected-file or foreground microphone input
• microphone maximum: five minutes
• output: one final transcript after the whole recording is processed
• no live partials, timestamps, confidence, diarization, or automatic handoff to a second chat model

Gemma 4: Ask with voice

• microphone maximum: 30 seconds
• output: the model's answer to the spoken request, not an ASR transcript
• recorded audio remains byte-backed in in-memory conversation history for regeneration and later text follow-ups
• the temporary WAV is best-effort deleted after it is read
• dedicated STT takes precedence when a selected profile declares both capabilities
• external-projector GGUF models require a loaded projector and a positive runtime audio probe

Platform scope

Platform	Qwen3-ASR typed transcription	Gemma Ask with voice	Validation status
Android native	Catalog, file input, microphone	Code-supported for compatible native audio models	Qwen microphone passed on Pixel 9 Pro; Gemma voice not device-validated
iOS native	Catalog, file input, microphone	Code-supported for compatible native audio models	Qwen and Gemma LiteRT manually exercised on a physical iPad; GGUF was user-smoked but not a scripted acceptance pass
macOS native	File input, microphone	LiteRT direct audio and experimental GGUF + projector	Qwen, LiteRT microphone UI, and GGUF Metal engine paths passed
Windows native	File input, microphone	Code-supported for compatible native audio models	Build/tests pass; real microphone/model behavior not validated
Linux native	Selected-file transcription; microphone disabled	Microphone disabled	Catalog/file contract covered; recorder intentionally unavailable
Web	Disabled	Disabled in the chat app	Typed Qwen Web work is #329; current LiteRT-LM Web bundle is text-only

Platform gating is not runtime proof. Unsupported actions are hidden or disabled, and unsupported typed STT paths fail explicitly.

Model, cache, and runtime correctness

The native Qwen3-ASR, Gemma 4 E2B GGUF/projector, and Gemma 4 E2B LiteRT-LM assets used by these presets are pinned to immutable sources with exact byte sizes and SHA-256 verification.

Download resume now rejects partial files from another source or HTTP representation, uses validators/If-Range where available, and does not promote an unsafe 416 partial. Verified unchanged assets can reuse a persisted integrity stamp instead of being rehashed every time the model screen opens.

On iOS, relocation is limited to a missing catalog path with the recognized app-managed shape:

/var/mobile/Containers/Data/Application/<old-id>/Library/Caches/models/<catalog-file>

The app validates the matching model/projector in the current managed cache before saving rebased paths. Missing or corrupt current artifacts still fail, and custom paths are not rewritten.

The current Gemma LiteRT bundle constrains its audio encoder/adapter to CPU. The app first uses the requested media backend, retries only the audio executor on CPU when compatibility initialization fails, and keeps the requested text/vision backend. This is runtime compatibility handling, not a claim that LiteRT-LM audio universally requires CPU.

Lifecycle and privacy

• permission denial and unsupported platforms surface actionable UI
• one recording/transcription/generation transition owns the active operation
• model, projector, conversation, and lifecycle changes invalidate stale work
• Discard, backgrounding, unload, shutdown, and disposal cancel capture and await cleanup
• empty or malformed recordings are rejected
• raw audio and private fixture paths are not logged
• temporary WAV deletion is best effort; voice audio intentionally remains in in-memory history for follow-up/regeneration

Validation

Final local head: d31eb4fd20a76a19551d4ce12ce60501c752a1dc

example/chat_app: flutter analyze
PASS

example/chat_app: flutter test -r compact
PASS: 267/267

focused chat provider/UI/catalog tests
PASS: 106/106

core audio serialization and Gemma template tests
PASS: 29/29

git diff --check origin/main...HEAD
PASS

Additional focused VM/Chrome, docs, build, cache/resume, recorder, and backend suites passed while developing the combined stack. An independent final review found no remaining P1/P2 issue.

Real model and device evidence

Scenario	Platform/model/backend	Result	Limits
Qwen microphone transcription	macOS arm64 / Qwen3-ASR / native llama.cpp	PASS	English and Korean transcripts; temporary-file cleanup confirmed
Qwen microphone transcription	Pixel 9 Pro / Android 17 / CPU	PASS	Final transcript and no leftover WAV; discard/background not repeated after disconnect
Qwen microphone transcription	iOS Simulator / Qwen3-ASR / CPU	PASS	Host microphone returned Hello.; discard/background cleanup passed; not physical-iPhone evidence
Qwen microphone transcription	Physical iPad / Qwen3-ASR	PASS	User-observed load, recording, and transcription
Gemma voice question	macOS arm64 / Gemma E2B LiteRT-LM	PASS	Packaged microphone UI answered the request; compatible CPU audio executor used
Gemma voice question	Physical iPad / Gemma E2B LiteRT-LM	PASS	User-observed answer; exact latency/memory metrics not captured
GGUF audio chat	macOS arm64 / pinned Gemma E2B GGUF + projector / Metal	PASS	Known WAV returned exact answer 4 in 3/3 runs with recommended sampling
GGUF voice UI	Physical iPad / pinned Gemma E2B GGUF + projector	PARTIAL	User reported the latest build working after model reselection; not a scripted exact-answer/regeneration/cleanup acceptance pass
iOS managed-path recovery	Unit/fake managed cache	PASS	Valid catalog assets relocate; invalid assets and custom paths do not. Automatic recovery has not yet been device-smoked across another reinstall

Out of scope

• live or partial transcription (Add live speech-to-text sessions with explicit accumulative, windowed, and stateful modes #327)
• timestamps, confidence, language detection, diarization, or speaker labels
• automatically feeding a Qwen transcript into a second LLM
• typed STT on WebGPU (WebGPU: validate and enable typed Qwen3-ASR speech-to-text #329)
• browser microphone Ask with voice and generic browser audio-chat validation
• typed STT through the current LiteRT-LM artifacts
• Linux microphone recording
• TTS (Design experimental native speech synthesis API for Qwen3-TTS #322)
• claiming Android or Windows Gemma voice validation from platform gating alone
• migrating arbitrary user-selected model paths

Readiness checklist

• Qwen transcription and Gemma audio-chat behavior are separately labeled and tested
• unsupported platform/model combinations are gated explicitly
• cancellation, stale-context handling, temporary-file cleanup, and negative paths have regression coverage
• native voice artifacts use immutable provenance and SHA-256 verification
• local chat-app, core audio/template, analyze, and diff checks pass
• exact-head CI passes on d31eb4fd20a76a19551d4ce12ce60501c752a1dc (14/14 checks)
• all review threads are resolved (0 unresolved)

[github-actions]

Chat app preview removed for leehack/llamadart-chat-pr-328.

[Copilot]

Pull request overview

Adds native microphone recording to the Flutter chat example’s speech-to-text workflow, expanding the Qwen3-ASR preset to native mobile + desktop while keeping Web explicitly unsupported. This strengthens the end-to-end “record → validate WAV → transcribe → cleanup” flow and updates docs/tests/catalog UX to match the new platform/availability matrix.

Changes:

• Add chat-app microphone capture controls (Record / Stop & transcribe / Discard) that produce a temporary WAV and transcribe it via SpeechToTextEngine, with lifecycle cancellation and best-effort cleanup.
• Expand Qwen3-ASR preset availability from desktop-only to all native platforms; split model availability browsing into Mobile / Web / Desktop filters.
• Add WAV validation + recorder service abstractions and broaden widget/unit test coverage; update docs, changelog, and platform permission/entitlement manifests.

Reviewed changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
website/docs/guides/speech-to-text.md	Updates STT guide to document microphone recording flow, limits, and platform support.
website/docs/examples/chat-app.md	Updates chat-app docs for microphone capture and refined platform availability filters.
tool/testing/test_matrix.dart	Adds a targeted manual smoke row for microphone transcription and cleanup.
example/chat_app/windows/flutter/generated_plugins.cmake	Registers record_windows plugin for Windows builds.
example/chat_app/windows/flutter/generated_plugin_registrant.cc	Registers record_windows plugin in the Windows runner.
example/chat_app/test/wav_audio_validator_test.dart	Adds unit tests for WAV “has audio data” validation.
example/chat_app/test/unit_test.dart	Updates model availability assertions for Qwen3-ASR native (not desktop-only).
example/chat_app/test/model_card_test.dart	Updates/extends model card tests for native-only availability messaging.
example/chat_app/test/manage_models_screen_download_test.dart	Updates platform filter/search tests and asserts Qwen3-ASR visibility across native targets.
example/chat_app/test/chat_input_test.dart	Adds extensive tests for recording UI, recording lifecycle, and transcription integration.
example/chat_app/test/audio_recording_service_test.dart	Adds a basic VM test for recorder “supported platform” gating.
example/chat_app/README.md	Documents new microphone transcription UX and platform support constraints.
example/chat_app/pubspec.yaml	Adds record dependency for microphone capture.
example/chat_app/pubspec.lock	Locks record and platform implementations.
example/chat_app/macos/Runner/Release.entitlements	Enables macOS audio-input entitlement for release builds.
example/chat_app/macos/Runner/Info.plist	Adds macOS NSMicrophoneUsageDescription.
example/chat_app/macos/Runner/DebugProfile.entitlements	Enables macOS audio-input entitlement for debug builds.
example/chat_app/macos/Flutter/GeneratedPluginRegistrant.swift	Registers record_macos plugin for macOS runner.
example/chat_app/linux/flutter/generated_plugins.cmake	Registers record_linux plugin for Linux builds.
example/chat_app/linux/flutter/generated_plugin_registrant.cc	Registers record_linux plugin in the Linux runner.
example/chat_app/lib/widgets/model_card.dart	Updates UI labels and icons for new ModelAvailability values.
example/chat_app/lib/widgets/chat_input.dart	Adds microphone record button + recording status row; disables composer while recording.
example/chat_app/lib/services/wav_audio_validator.dart	Adds RIFF/WAV data-chunk validation used after recording finalization.
example/chat_app/lib/services/audio_recording_service.dart	Introduces typed recorder abstraction + failure taxonomy and conditional platform wiring.
example/chat_app/lib/services/audio_recording_service_stub.dart	Implements unsupported-platform recorder behavior with actionable errors.
example/chat_app/lib/services/audio_recording_service_io.dart	Implements native recording via record, temporary file handling, permission checks, and WAV validation.
example/chat_app/lib/screens/manage_models_screen.dart	Splits model library filtering into Mobile/Web/Desktop and updates search semantics.
example/chat_app/lib/providers/chat_provider.dart	Adds recording state machine, lifecycle cancellation, transcription handoff, and cleanup coordination.
example/chat_app/lib/models/downloadable_model.dart	Adds ModelAvailability.native and updates platform-availability checks and defaults for Qwen3-ASR.
example/chat_app/lib/main.dart	Cancels recording on app hide/pause to keep capture foreground-scoped.
example/chat_app/ios/Runner/Info.plist	Adds iOS NSMicrophoneUsageDescription.
example/chat_app/android/app/src/main/AndroidManifest.xml	Adds Android RECORD_AUDIO permission.
CHANGELOG.md	Updates Unreleased entry to reflect microphone recording + expanded Qwen3-ASR native availability.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 33 changed files in this pull request and generated no new comments.

Suppressed comments (1)

example/chat_app/lib/widgets/chat_input.dart:315

• The mic IconButton is still shown while an audio recording is already active. Tapping it during recording is a no-op (provider guards on hasActiveAudioRecording), which is confusing and adds an extra interactive control while the composer is intentionally locked.

Hide (or disable) the mic button when hasActiveAudioRecording is true, similar to how the attachment menu is gated.

                        if (provider.canAttachMedia && !hasActiveAudioRecording)
                          _buildAttachmentMenu(context, provider),
                        if (provider.canStartAudioRecording)
                          IconButton(

[Copilot]

Pull request overview

Copilot reviewed 51 out of 52 changed files in this pull request and generated no new comments.

Suppressed comments (3)

tool/testing/test_matrix.dart:175

• In the test-matrix entry for gguf-chat-features-smoke, the command string makes --mmproj-path and --image-path look required, but the local E2E runner treats them as optional (only --image-path requires --mmproj-path). This will mislead anyone following the matrix to run the base smoke without image input.
  example/chat_app/lib/main.dart:46
• AppLifecycleListener cancels microphone recording via cancelAudioRecording() without overriding showMessage, so backgrounding/hiding the app will append a “Microphone recording cancelled.” info message to the conversation history. Lifecycle-driven cancellation is usually best kept silent to avoid polluting the chat transcript.

    _listener = AppLifecycleListener(
      onHide: () {
        unawaited(_chatProvider.cancelAudioRecording());
      },
      onPause: () {
        unawaited(_chatProvider.cancelAudioRecording());
      },

example/chat_app/lib/services/model_service_io.dart:492

• _writeDownloadProvenance failures currently bubble up and can abort downloads even though the provenance file is only needed for safe resume/representation binding. If writing the JSON fails (filesystem race/permissions/IO errors), it’s still safe to continue the download with resume effectively disabled on the next run.

  Future<void> _writeDownloadProvenance(
    String savePath,
    _DownloadProvenance provenance,
  ) async {
    await _writeJsonAtomically(_downloadProvenanceFile(savePath), {
      'version': _downloadProvenanceVersion,
      ...provenance.toJson(),
    });
  }

[Copilot]

Pull request overview

Copilot reviewed 56 out of 57 changed files in this pull request and generated no new comments.

Suppressed comments (2)

tool/testing/test_matrix.dart:301

• The test-matrix command for litert-lm-chat-features-smoke includes --audio-path/--expect unconditionally, but the scenario supports a base chat-only run (and useWhen explicitly says to omit those flags unless you’re exercising audio). Making them look mandatory is misleading.
  tool/testing/test_matrix.dart:174
• The test-matrix command string for gguf-chat-features-smoke currently shows --mmproj-path and --image-path as mandatory, but the runner only requires these when you actually run the optional image variant. As written, the matrix command is misleading for the base (text-only) smoke.

This issue also appears on line 297 of the same file.

[Copilot]

Pull request overview

Copilot reviewed 58 out of 59 changed files in this pull request and generated no new comments.

Suppressed comments (3)

tool/testing/test_matrix.dart:174

• The gguf-chat-features-smoke row’s command string currently implies --mmproj-path and --image-path are always required, but the row text says the image variant is optional (and run_local_e2e only requires --mmproj-path when --image-path is provided). This can mislead contributors filling in matrix evidence.
  tool/testing/test_matrix.dart:301
• The litert-lm-chat-features-smoke row’s command shows --audio-path/--expect as mandatory, but the row itself says to omit them for the base chat-only smoke. The command should reflect the optional audio variant to avoid confusing the expected invocation.
  example/chat_app/lib/services/model_service_io.dart:468
• _discardStalePartial is called from cache-state checks and currently does unguarded file operations (tempFile.length() / tempFile.delete()). If the temp file is concurrently removed/locked, this can throw and surface as a UI error. Treat stale-partial cleanup as best-effort (like _deleteIfExists) and gracefully return null when the temp file can’t be inspected.

    final provenance = await _readDownloadProvenance(provenanceFile);
    final partialLength = await tempFile.length();
    if (provenance == null ||
        !provenance.matchesSource(source) ||
        provenance.partialIsInvalid(partialLength)) {
      await tempFile.delete();

[Copilot]

Pull request overview

Copilot reviewed 61 out of 62 changed files in this pull request and generated no new comments.

Suppressed comments (1)

tool/testing/test_matrix.dart:174

• The gguf-chat-features-smoke test-matrix command currently implies --mmproj-path and --image-path are required, but the runner and tool/gguf_chat_features_smoke.dart treat both as optional (with --image-path requiring --mmproj-path). This mismatch can mislead contributors running the matrix row.