Skip to content

Token tracking bidi - #173

Closed
alfredjimmy-redbus wants to merge 21 commits into
redbus-labs:mainfrom
alfredjimmy-redbus:token_tracking
Closed

Token tracking bidi#173
alfredjimmy-redbus wants to merge 21 commits into
redbus-labs:mainfrom
alfredjimmy-redbus:token_tracking

Conversation

@alfredjimmy-redbus

Copy link
Copy Markdown

1. Runner.runLive — plugin callbacks for live sessions

File: core/src/main/java/com/google/adk/runner/Runner.java

Problem: Non-live runs already invoked onEventCallback / afterRunCallback; runLive did not. Plugins like RedBusTrackingPlugin and LiveTokenTrackingPlugin never saw live events, so BIDI token usage was never tracked.

Change: In runLiveImpl, for each live event:

  • pluginManager.onEventCallback(invocationContext, event) — per event (e.g. each usageMetadata turn)
  • pluginManager.afterRunCallback(invocationContext) — once when the live stream completes

Mirrors the existing non-live runImpl path. Session persistence (appendEvent) is unchanged; plugin callbacks are observational only.


2. GeminiLlmConnection — handle new Live API control messages

File: core/src/main/java/com/google/adk/models/GeminiLlmConnection.java

Problem: google-genai 1.58+ delivers control messages (sessionResumptionUpdate, goAway, voice-activity signals) that ADK did not recognize. They fell through to the “unknown server message” branch, producing error LlmResponses and fake ADK events — observed in RAE as event #123 with hasContent=false right after:

Received unknown or empty server message: {"sessionResumptionUpdate":{...}}

Change: Treat these as no-op control messages (return Optional.empty(), debug log only):

  • sessionResumptionUpdate
  • goAway
  • voiceActivityDetectionSignal
  • voiceActivity

Unchanged: Standalone usageMetadata messages still become LlmResponses with usage attached (existing behavior at lines 189–197).


3. Test coverage

File: core/src/test/java/com/google/adk/models/GeminiLlmConnectionTest.java

  • Added convertToServerResponse_withSessionResumptionUpdate_returnsEmpty — confirms resumption updates do not emit error events.

Existing tests still cover:

  • Usage-only messages → response with usageMetadata
  • Content + usage in same message
  • Unknown empty message → error response

Why this matters for RAE / BIDI voice

Layer | Before | After -- | -- | -- Runner | Live events bypass plugins | RedBusTrackingPlugin.onEventCallback fires per usageMetadata event GeminiLlmConnection | Resumption updates → spurious error events | Control messages ignored; real usage events pass through cleanly

Dependency: Requires google-genai ≥ 1.58.0 (RAE already on 1.58.0).

Downstream (not in ADK): genai-sdk RedBusTrackingPlugin persists per-event usage to rae_tokens; LiveAudioSession logs USAGE_METADATA for confirmation.

vaidyanath.b and others added 21 commits December 19, 2025 13:07
Wire onEventCallback and afterRunCallback into Runner.runLiveImpl so the
plugin lifecycle runs for live/bidi sessions, matching the non-live path.
Previously live events were appended to the session but never passed
through any plugin hook, so token usage emitted by Gemini Live (including
audio modality breakdowns) could not be captured by plugins.

Add LiveTokenTrackingPlugin which observes usageMetadata on each event and
logs per-invocation totals on run completion. Live reports cumulative
counts, so the plugin keeps the latest value per field rather than summing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWJMsUqoQgsxLScGRk7uB2
Runnable example that drives a Gemini Live BIDI session with
LiveTokenTrackingPlugin registered, demonstrating that usageMetadata
(including audio modality breakdown) reaches onEventCallback and that
afterRunCallback receives the aggregated per-session totals.

Requires GOOGLE_API_KEY; model selectable via -Dmodel.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWJMsUqoQgsxLScGRk7uB2
Log every usageMetadata event with a sequence number and run two turns in
one session. Empirically Gemini Live emits exactly one usageMetadata event
per turn, and each event reports that turn's own totals (total = prompt +
candidates) rather than a session-cumulative running total.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWJMsUqoQgsxLScGRk7uB2
A live two-turn session showed Gemini Live emits one usageMetadata event
per turn, each reporting that turn's own usage (total = prompt +
candidates) rather than a session-cumulative running total. Keeping the
latest value therefore dropped all earlier turns and undercounted
multi-turn sessions.

Accumulate by summing scalar counts and per-modality token counts across
events so the aggregate reflects true session totals. Verified end-to-end
against a live session: per-turn totals 892 and 1044 sum to 1936.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWJMsUqoQgsxLScGRk7uB2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants