Token tracking bidi - #173
Closed
alfredjimmy-redbus wants to merge 21 commits into
Closed
Conversation
Wire onEventCallback and afterRunCallback into Runner.runLiveImpl so the plugin lifecycle runs for live/bidi sessions, matching the non-live path. Previously live events were appended to the session but never passed through any plugin hook, so token usage emitted by Gemini Live (including audio modality breakdowns) could not be captured by plugins. Add LiveTokenTrackingPlugin which observes usageMetadata on each event and logs per-invocation totals on run completion. Live reports cumulative counts, so the plugin keeps the latest value per field rather than summing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FWJMsUqoQgsxLScGRk7uB2
Runnable example that drives a Gemini Live BIDI session with LiveTokenTrackingPlugin registered, demonstrating that usageMetadata (including audio modality breakdown) reaches onEventCallback and that afterRunCallback receives the aggregated per-session totals. Requires GOOGLE_API_KEY; model selectable via -Dmodel. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FWJMsUqoQgsxLScGRk7uB2
Log every usageMetadata event with a sequence number and run two turns in one session. Empirically Gemini Live emits exactly one usageMetadata event per turn, and each event reports that turn's own totals (total = prompt + candidates) rather than a session-cumulative running total. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FWJMsUqoQgsxLScGRk7uB2
A live two-turn session showed Gemini Live emits one usageMetadata event per turn, each reporting that turn's own usage (total = prompt + candidates) rather than a session-cumulative running total. Keeping the latest value therefore dropped all earlier turns and undercounted multi-turn sessions. Accumulate by summing scalar counts and per-modality token counts across events so the aggregate reflects true session totals. Verified end-to-end against a live session: per-turn totals 892 and 1044 sum to 1936. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FWJMsUqoQgsxLScGRk7uB2
…main PiperOrigin-RevId: 935944702
…main PiperOrigin-RevId: 935957207
PiperOrigin-RevId: 935997601
PiperOrigin-RevId: 936622591
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1.
Runner.runLive— plugin callbacks for live sessionsFile:
core/src/main/java/com/google/adk/runner/Runner.javaProblem: Non-live runs already invoked
onEventCallback/afterRunCallback;runLivedid not. Plugins likeRedBusTrackingPluginandLiveTokenTrackingPluginnever saw live events, so BIDI token usage was never tracked.Change: In
runLiveImpl, for each live event:pluginManager.onEventCallback(invocationContext, event)— per event (e.g. eachusageMetadataturn)pluginManager.afterRunCallback(invocationContext)— once when the live stream completesMirrors the existing non-live
runImplpath. Session persistence (appendEvent) is unchanged; plugin callbacks are observational only.2.
GeminiLlmConnection— handle new Live API control messagesFile:
core/src/main/java/com/google/adk/models/GeminiLlmConnection.javaProblem:
google-genai1.58+ delivers control messages (sessionResumptionUpdate,goAway, voice-activity signals) that ADK did not recognize. They fell through to the “unknown server message” branch, producing errorLlmResponses and fake ADK events — observed in RAE as event#123withhasContent=falseright after:Change: Treat these as no-op control messages (return
Optional.empty(), debug log only):sessionResumptionUpdategoAwayvoiceActivityDetectionSignalvoiceActivityUnchanged: Standalone
usageMetadatamessages still becomeLlmResponses with usage attached (existing behavior at lines 189–197).3. Test coverage
File:
core/src/test/java/com/google/adk/models/GeminiLlmConnectionTest.javaconvertToServerResponse_withSessionResumptionUpdate_returnsEmpty— confirms resumption updates do not emit error events.Existing tests still cover:
usageMetadataWhy this matters for RAE / BIDI voice
Dependency: Requires
google-genai≥ 1.58.0 (RAE already on 1.58.0).Downstream (not in ADK):
genai-sdkRedBusTrackingPluginpersists per-event usage torae_tokens;LiveAudioSessionlogsUSAGE_METADATAfor confirmation.