Skip to content

Quote a cut's words once they are transcribed - #449

Merged
jserv merged 1 commit into
sysprog21:mainfrom
Aatroxo1:cut-line-waits-for-words
Oct 24, 2026
Merged

jserv merged 1 commit into
sysprog21:mainfrom
Aatroxo1:cut-line-waits-for-words

Conversation

@Aatroxo1

@Aatroxo1 Aatroxo1 commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

The line printed when Gemini cuts the interviewer quoted the candidate turn as it stood at the cut, and at that moment the turn still held the answer Jim was replying to, so every cut quoted that answer and a wordless cut could not be told from speech.

Quoting only what followed the reply, the first approach on #188, fails the other way: Interrupted arrives with the start of the candidate's activity, before the words that caused it are transcribed, so speech then read as wordless too. The quote has to wait for the words.

on_interruption now logs the cut where it happens (Gemini cut its own turn at=..., N.Ns of it unplayed), closes the answer's turn, and notes the cut in SpeakerTurns. close_turns prints a second line when the next candidate turn closes: candidate audio since the cut at=..., settled N.Ns later: and the first 80 characters of that turn, or (nothing transcribed; ...) when it has no words. The start is quoted because the words that caused the cut come first, and settled shows how late the quote is. The line is read before the turns close, since closing clears the words, and close_turns_settling_cut keeps that order testable without a room.

A close that lands while the candidate is still speaking leaves the cut waiting, and keeps the words the turn holds, because that close clears them and they are the ones the quote exists to show. After Interrupted, a late transcript of the cut reply makes its TurnComplete reach close_turns before any of the candidate's words arrive, which a Harness test reproduces. A TurnComplete after the words never gets there, since their arrival sets the discard that swallows it; the closes that can land mid-speech are the ones the loop makes outside an event, a pause or a socket replacement, and a Harness test drives close_turns as they do. A cut still waiting when the next one arrives is settled first.

A cut still waiting when the room exits is settled right after the room loop returns in run_room_with_slot. Every return, ? and break out of the loop lands there, so the exits that never call close_turns (the candidate not returning, the billing exits, the event stream ending) no longer drop it, and neither will an exit added later.

One limit remains: a cut whose turn closes before any words are transcribed, as when the interview ends or the tab closes while the candidate is speaking, still reads as wordless.

SpeakerTurn::tail is replaced by head, its only caller being this line. It reads the cleaned turn, so the #hashtag transcription artifact counts as no words. The two doc lines that sat above tail describe segment_id, and now sit there. docs/livekit-connection-troubleshooting.md describes the two lines.

Twelve unit tests in tests/unit/livekit/session.rs cover speech after a cut, no words, the artifact alone, a long reply, a close while the candidate speaks and the words it keeps, a second cut, a cut waiting at teardown, no cut at all, and three through Harness that drive handle_gemini_event and close_turns. Removing the speaking check makes four of them fail, not keeping the words three, and noting the cut before closing the answer's turn three. The call in run_room_with_slot is out of reach of cargo test; the live check below covers it.

Live check with a headset for microphone and playback, on this change:

What I did Result
Speech over a reply, 5 s Cut line at the cut; settled 8.7s later quoted the words
Speech over a reply, 25 s Cut line at the cut; settled 31.5s later quoted the first 80 characters
A second cut before the first settled The second cut printed the first one's quote, settled 14.0s later
Closed the tab while still speaking, 28.6 s after a cut After LiveKit event stream ended, settled 28.6s later read (nothing transcribed; ...), because the words had not arrived
timing: Gemini cut its own turn at=0:57.034, 5.0s of it unplayed
LiveKit event stream ended for room=interview-ne1hze2x; ending with no report, because the room closed before the interview did
timing: candidate audio since the cut at=0:57.034, settled 28.6s later: (nothing transcribed; check speaker echo or background noise)

In one session I tried to cut Jim off with speech three times; two overlapped a reply, and the third began 2 ms after the reply ended (kind=pause), so there was nothing to cut. On the previous revision of this change, a cough or tongue click over a reply (3 times) and a tongue click over the opening (3 times) produced no cut, and this revision only changes what is printed after one.

./scripts/test.sh passes on 82bbdc5 plus this change. Of ten full local runs, two failed on the floor-capture browser tests (follower not ready, a 5 s startup wait) and one on the indent gate over a comment wrap, since fixed; that test file passes alone, five runs in a row on this branch and on a clean upstream/main, and the other seven full runs passed. Not checked locally: Prettier, ruff, eslint, cargo-audit, actionlint, script-requires, and the browser tests that need Chromium or Java.

Changes 2 and 3 from #188 (prefixPaddingMs, and keeping the queued audio in cut_unless_protected until the interruption carries words) are not part of this pull request. They wait for a log from a build with this change that shows a wordless cut, and get their own issue then.

Closes #188

cubic-dev-ai[bot]

This comment was marked as resolved.

Comment thread src/livekit/session.rs Outdated
Comment thread src/livekit/turn.rs Outdated
Comment thread src/livekit/turn.rs
Comment thread src/livekit/session.rs Outdated
Comment thread src/livekit/session.rs Outdated
Comment thread src/livekit/turn.rs Outdated
Comment thread tests/unit/livekit/session.rs Outdated
Comment thread src/livekit/turn.rs Outdated

@jserv jserv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoid the ambiguous reference Refs #188.

Comment thread src/livekit/session.rs Outdated
@Aatroxo1
Aatroxo1 force-pushed the cut-line-waits-for-words branch from 3d37e27 to 2b38301 Compare October 23, 2026 12:53
@Aatroxo1 Aatroxo1 changed the title Print a cut's line once its words arrive Quote a cut's words once they are transcribed Oct 23, 2026
cubic-dev-ai[bot]

This comment was marked as resolved.

@Aatroxo1
Aatroxo1 force-pushed the cut-line-waits-for-words branch from 2b38301 to 2a2b154 Compare October 23, 2026 13:41
cubic-dev-ai[bot]

This comment was marked as resolved.

Comment thread src/livekit/session.rs
Comment thread src/agent.rs Outdated
Gemini sends Interrupted with the start of the candidate's activity,
before the words behind it are transcribed, and the candidate turn
that is still open then holds the answer the reply was answering. The
line naming the cause therefore quoted that answer for every cut, and
quoting only what followed the reply read speech as wordless. The cut
is now logged where it happens, and a second line quotes the start of
the turn that follows it once that turn closes, or when the room ends,
with how long after the cut it settled. A close that lands while the
candidate is still speaking leaves the cut waiting with the words said
so far, so neither the late end of the cut reply nor a pause reads
speech as wordless.

Closes sysprog21#188
@Aatroxo1
Aatroxo1 force-pushed the cut-line-waits-for-words branch from 2a2b154 to 1539ecb Compare October 24, 2026 05:15
@jserv
jserv merged commit 65cee24 into sysprog21:main Oct 24, 2026
11 checks passed
@jserv

jserv commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Thank @Aatroxo1 for contributing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Brief filler sounds unintentionally interrupt the interviewer and cause repeated responses

3 participants