Skip to content

[feature] android: voice typing keyboard - #254

Open
YJack0000 wants to merge 1 commit into
mainfrom
feat/android-voice-typing
Open

[feature] android: voice typing keyboard#254
YJack0000 wants to merge 1 commit into
mainfrom
feat/android-voice-typing

Conversation

@YJack0000

Copy link
Copy Markdown
Contributor

Dictation into any app on Android: an InputMethodService that streams the mic to the hosted STT relay and types the result at the cursor. iOS and desktop already have this; Android was the only platform without it.

Targets 0.2.0 — do not merge while the 0.1.0 store release is in flight.

What it reuses

Nothing new in the transcription stack: MicCapture, SttRelayClient (with the previously-unused Feature.VOICE_TYPING), SonioxStreamParser and SegmentBuilder are the same classes a meeting uses. VoiceTypingSession is deliberately MeetingSession minus the encoder and the uploader.

MicCapture ──ByteArray(3200)──▶ SttRelayClient ──▶ segments
                                 feature=voice_typing      │
                          DictationTextAssembler ◀─────────┘
                                 (settled, tail)
                                       ▼
                            TranscriptCommitter ──▶ InputConnection

The keyboard records itself — unlike iOS

An iOS keyboard extension is forbidden from opening the mic, so ios/Keyboard/ hands every dictation to the container app over an App Group. Android has no such rule: an IME runs in its own app's process and may hold RECORD_AUDIO. No channel, no session ids, no app switch per dictation.

The permission hand-off

An IME cannot request a runtime permission — there is no Activity to host the request. This is the classic way third-party dictation keyboards die: the mic button appears to do nothing, forever.

So the keyboard never tries. While the mic grant or the cloud session is missing it says which one in the state line and shows a button that opens parley://voice-typing; MainActivity latches the request and the nav graph lands on a setup screen that can request the permission. The latch (rather than an event) matters because the commonest reason for the hand-off is signed out, and the graph does not exist while the sign-in wall is up — the request survives the sign-in trip the user was just sent on.

The globe key is live in every state, so a user who cannot dictate is never trapped.

The partial/final commit rule

The tentative tail (mix-tail) goes in as composing text; settled runs (mix-0, mix-1, …) are committed as a delta against a high-water mark. commitText implicitly replaces the composing region, so committing the delta erases the stale guess in the same call — which is precisely why a word spoken once is never typed twice.

TranscriptCommitter is pure and has 11 tests; DictationPipelineTest drives 6 more from real Soniox frames through the production SonioxStreamParser/SegmentBuilder/assembler into a fake editor that models InputConnection's composing semantics. Both orders of the end-of-session fold are covered, because they arrive over two separate flows.

Max duration

600 s, mirroring the desktop's HOSTED_VOICE_TYPING_MAX_SECONDS (src/lib/limits.ts). At the cap the session ends itself exactly as a user tap would and the state line says so. (iOS uses 120 s because its extension records under a jetsam limit; an Android IME has the app's own process.)

UI framework: classic Views

Justified in the service's header and in the doc. InputMethodService is not a LifecycleOwner/ViewModelStoreOwner/SavedStateRegistryOwner, so ComposeView needs hand-rolled owner plumbing in the one surface the user cannot escape if it breaks. Material 3 still applies — KeyboardPalette is the app's scheme including dynamic color on API 31+.

Verified on the emulator (Pixel 7 / API 35)

  • IME appears in ime list, enables and selects; renders in light and dark, with the bottom row correctly inset from the API-35 edge-to-edge IME nav strip (this caught a real clipping bug, fixed).
  • Text actually committed into a third-party app. With the (debug-only) ScreenshotDemo seam — mirroring iOS's DictationCoordinator demo path — a scripted transcript runs through the production assembler → committer → InputConnection. Into Contacts' First name, the field ended up with the script exactly once, no duplication, the tail underlined while provisional and settled on stop.
  • Backspace (tap and hold-repeat), Return firing IME_ACTION_NEXT in a field that asks for it, and the hand-off starting the app from inside another app (the background-activity-start exemption holds).
  • No crashes in logcat.

Not verified on-device: anything needing a real microphone or the live relay — real STT partial/final timing, mic-unavailable and quota errors, the 600 s cap in wall-clock, and the zh-TW Simplified→Traditional pass. The manual checklist in android/docs/voice-typing.md covers all of it for a real handset.

Verification

./gradlew assembleDebug :parleykit:test :app:testDebugUnitTest — green, 77 tests (17 new).

Every new user-facing string is in both values/strings.xml and values-zh-rTW/strings.xml. MainActivity, MeetingService and the existing deep links are untouched apart from one added <data> element and one deep-link branch.

Dictation into any app: an InputMethodService that streams the microphone to
the hosted STT relay (feature=voice_typing) and types the result at the cursor.
The Android counterpart of the iOS keyboard extension and the desktop's
push-to-talk, targeting 0.2.0.

Reuses the meeting stack unchanged — MicCapture, SttRelayClient,
SonioxStreamParser, SegmentBuilder. VoiceTypingSession is MeetingSession minus
the encoder and the uploader.

The keyboard records itself, unlike iOS: an Android IME runs in its own app's
process and may hold RECORD_AUDIO, so there is no App Group channel and no app
switch per dictation. What it *cannot* do is request a runtime permission —
there is no Activity to host the request — so a missing mic grant (or a missing
cloud session) surfaces in the state line with a button that opens the app at a
setup screen which can. That hand-off is the single most common way third-party
dictation keyboards break silently.

Partial vs final: the tentative tail goes in as composing text and settled runs
are committed as a delta against a high-water mark. commitText implicitly
replaces the composing region, so committing the delta erases the stale guess in
the same call — which is why a word spoken once is never typed twice. The rule
is pure and covered by 17 tests, including real relay frames driven through
SegmentBuilder into a fake editor with InputConnection's composing semantics.

Sessions are capped at 600 s, mirroring the desktop's
HOSTED_VOICE_TYPING_MAX_SECONDS.

Classic Views rather than Compose (justified in the service's header):
InputMethodService owns none of the ViewTree owners Compose needs, and a
keyboard that fails to draw leaves the user unable to type at all.

Architecture, the hand-off, the commit rule and a manual test checklist are in
android/docs/voice-typing.md.
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant