Skip to content

Commit 8a294af

Browse files
VedantMadaneDeanChensj
authored andcommitted
fix: Extract grounding_metadata from Live API server_content
Merge #4213 ## Summary Fixes #3542 - VertexAiSearchTool query instability where grounding metadata is intermittently missing from Live API responses. ### Root Cause The Live API's `receive()` method in `gemini_llm_connection.py` extracted various fields from Live API messages (usage_metadata, server_content, tool_call, session_resumption_update) but **never extracted `grounding_metadata`** from `server_content`. This prevented agents from accessing grounding data from Vertex AI Search, even when the backend provided it. ### Changes #### Modified `src/google/adk/models/gemini_llm_connection.py` - Added tracking variable `last_grounding_metadata` to accumulate grounding across messages. - Added tracking variable `tool_call_metadata` to handle grounding metadata for buffered tool calls correctly. - Extract `grounding_metadata` from `message.server_content.grounding_metadata` and accumulate it. - Include accumulated `grounding_metadata` in `LlmResponse` when yielding: - Content responses with parts (only if turn is not complete). - Buffered tool call responses (using `tool_call_metadata` to match the time they were received). - Turn complete responses (using current message's grounding or accumulated). - Interrupted responses. - Full text responses (via `__build_full_text_response`). - Added warning log when incomplete grounding_metadata is detected (has `retrieval_queries` but missing `grounding_chunks`). #### Updated `tests/unittests/models/test_gemini_llm_connection.py` - Fixed existing tests to: - Explicitly set `grounding_metadata = None` on mock server_content objects (in helper). - Use real `types.GroundingMetadata` instead of `autospec` mock in standalone grounding test to avoid `AttributeError` on `retrieval_queries`. - Added four new tests: - `test_receive_extracts_grounding_metadata` - verifies grounding_metadata is extracted and included in content responses. - `test_receive_grounding_metadata_reset_after_tool_call` - verifies grounding_metadata is reset after tool call. - `test_receive_grounding_metadata_accumulates_across_messages` - verifies accumulation. - `test_receive_interrupted_with_pending_text_preserves_flag` - verifies interrupted flag. ### Test Results All 50 tests pass. ### Impact This fix ensures that grounding data from Vertex AI Search is properly extracted and attached to `LlmResponse` events, allowing agents to access `event.grounding_metadata.grounding_chunks` when available. ============= Commits ============== -- cb635f0 by Vedant Madane <vedantnm@gmail.com>: ensure grounding metadata is correctly propagated in live API - Correctly handle search results and citations in grounding metadata. - Fix metadata parsing logic in gemini_llm_connection.py. - Add unit tests for grounding metadata extraction. Signed-off-by: Vedant Madane <vedantnm@gmail.com> Co-authored-by: Shangjie Chen <deanchen@google.com> COPYBARA_INTEGRATE_REVIEW=#4213 from VedantMadane:fix/live-api-grounding-metadata cb635f0 PiperOrigin-RevId: 933291436
1 parent 8b09c48 commit 8a294af

2 files changed

Lines changed: 447 additions & 18 deletions

File tree

‎src/google/adk/models/gemini_llm_connection.py‎

Lines changed: 111 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -186,11 +186,55 @@ async def send_realtime(self, input: RealtimeInput):
186186
else:
187187
raise ValueError('Unsupported input type: %s' % type(input))
188188

189+
@staticmethod
190+
def _merge_grounding_metadata(
191+
existing: types.GroundingMetadata | None,
192+
new: types.GroundingMetadata | None,
193+
) -> types.GroundingMetadata | None:
194+
"""Merges two GroundingMetadata instances, accumulating list fields safely."""
195+
if existing is None:
196+
return new
197+
if new is None:
198+
return existing
199+
existing_data = existing.model_dump(exclude_none=True)
200+
new_data = new.model_dump(exclude_none=True)
201+
202+
# Get offset from existing grounding chunks for shifting support indices
203+
chunk_offset = len(existing_data.get('grounding_chunks', []))
204+
205+
for key, val in new_data.items():
206+
if isinstance(val, list) and all(isinstance(x, str) for x in val):
207+
existing_list = existing_data.get(key, [])
208+
for item in val:
209+
if item not in existing_list:
210+
existing_list.append(item)
211+
existing_data[key] = existing_list
212+
elif key == 'grounding_chunks':
213+
existing_chunks = existing_data.get('grounding_chunks', [])
214+
existing_chunks.extend(val)
215+
existing_data['grounding_chunks'] = existing_chunks
216+
elif key == 'grounding_supports':
217+
existing_supports = existing_data.get('grounding_supports', [])
218+
for support in val:
219+
if (
220+
'grounding_chunk_indices' in support
221+
and support['grounding_chunk_indices']
222+
):
223+
support['grounding_chunk_indices'] = [
224+
idx + chunk_offset for idx in support['grounding_chunk_indices']
225+
]
226+
existing_supports.append(support)
227+
existing_data['grounding_supports'] = existing_supports
228+
else:
229+
existing_data[key] = val
230+
return types.GroundingMetadata(**existing_data)
231+
189232
def __build_full_text_response(
190233
self,
191234
text: str,
192235
is_thought: bool = False,
193236
grounding_metadata: types.GroundingMetadata | None = None,
237+
interrupted: bool = False,
194238
):
195239
"""Builds a full text response.
196240
@@ -201,6 +245,7 @@ def __build_full_text_response(
201245
text: The text to be included in the response.
202246
is_thought: Whether the text is a thought.
203247
grounding_metadata: The grounding metadata to include.
248+
interrupted: Whether this response was interrupted.
204249
205250
Returns:
206251
An LlmResponse containing the full text.
@@ -215,6 +260,7 @@ def __build_full_text_response(
215260
parts=[part],
216261
),
217262
grounding_metadata=grounding_metadata,
263+
interrupted=interrupted,
218264
partial=False,
219265
live_session_id=self._gemini_session.session_id,
220266
)
@@ -258,7 +304,8 @@ async def receive(self) -> AsyncGenerator[LlmResponse, None]:
258304
text = ''
259305
is_thought = False
260306
tool_call_parts = []
261-
pending_grounding_metadata = None
307+
last_grounding_metadata = None
308+
tool_call_metadata = None
262309
async with Aclosing(self._gemini_session.receive()) as agen:
263310
# TODO(b/440101573): Reuse StreamingResponseAggregator to accumulate
264311
# partial content and emit responses as needed.
@@ -276,9 +323,10 @@ async def receive(self) -> AsyncGenerator[LlmResponse, None]:
276323
)
277324
if message.server_content:
278325
content = message.server_content.model_turn
279-
if message.server_content.grounding_metadata:
280-
pending_grounding_metadata = (
281-
message.server_content.grounding_metadata
326+
grounding_metadata = message.server_content.grounding_metadata
327+
if grounding_metadata:
328+
last_grounding_metadata = self._merge_grounding_metadata(
329+
last_grounding_metadata, grounding_metadata
282330
)
283331

284332
# Standalone grounding_metadata event (when content is empty)
@@ -326,9 +374,12 @@ async def receive(self) -> AsyncGenerator[LlmResponse, None]:
326374
llm_response.partial = True
327375
# don't yield the merged text event when receiving audio data
328376
elif text and not content.parts[0].inline_data:
329-
yield self.__build_full_text_response(text, is_thought)
377+
yield self.__build_full_text_response(
378+
text, is_thought, last_grounding_metadata
379+
)
330380
text = ''
331381
is_thought = False
382+
last_grounding_metadata = None
332383
yield llm_response
333384
# Note: in some cases, tool_call may arrive before
334385
# generation_complete, causing transcription to appear after
@@ -430,27 +481,58 @@ async def receive(self) -> AsyncGenerator[LlmResponse, None]:
430481
)
431482
self._output_transcription_text = ''
432483
if message.server_content.turn_complete:
433-
g_metadata_to_yield = pending_grounding_metadata
484+
# Capture final grounding metadata before last_grounding_metadata is cleared in the next block.
485+
final_grounding_metadata = (
486+
grounding_metadata
487+
or last_grounding_metadata
488+
or (
489+
types.GroundingMetadata()
490+
if self._is_gemini_3_1_flash_live
491+
else None
492+
)
493+
)
494+
if (
495+
final_grounding_metadata
496+
and final_grounding_metadata.retrieval_queries
497+
and not final_grounding_metadata.grounding_chunks
498+
):
499+
logger.warning(
500+
'Incomplete grounding_metadata received: retrieval_queries=%s'
501+
' but grounding_chunks is empty. This may indicate a'
502+
' transient issue with the Vertex AI Search backend.',
503+
final_grounding_metadata.retrieval_queries,
504+
)
505+
434506
if text:
435507
yield self.__build_full_text_response(
436-
text, is_thought, g_metadata_to_yield
508+
text,
509+
is_thought,
510+
last_grounding_metadata,
511+
message.server_content.interrupted,
437512
)
438513
text = ''
439514
is_thought = False
440-
g_metadata_to_yield = None
515+
last_grounding_metadata = None
441516
if tool_call_parts:
442517
logger.debug('Returning aggregated tool_call_parts')
443518
yield LlmResponse(
444519
content=types.Content(role='model', parts=tool_call_parts),
520+
grounding_metadata=tool_call_metadata,
445521
model_version=self._model_version,
446522
live_session_id=live_session_id,
447523
)
448524
tool_call_parts = []
525+
if tool_call_metadata is not None:
526+
last_grounding_metadata = None
527+
tool_call_metadata = None
528+
449529
yield LlmResponse(
450530
turn_complete=True,
451531
interrupted=message.server_content.interrupted,
452-
grounding_metadata=message.server_content.grounding_metadata
453-
or g_metadata_to_yield
532+
# If last_grounding_metadata was cleared in the full text yield,
533+
# avoid duplicating it here.
534+
grounding_metadata=grounding_metadata
535+
or last_grounding_metadata
454536
or (
455537
types.GroundingMetadata()
456538
if self._is_gemini_3_1_flash_live
@@ -462,32 +544,47 @@ async def receive(self) -> AsyncGenerator[LlmResponse, None]:
462544
message.server_content, 'turn_complete_reason', None
463545
),
464546
)
547+
last_grounding_metadata = None # Reset after yielding
465548
break
466549
# in case of empty content or parts, we still surface it
467550
# in case it's an interrupted message, we merge the previous partial
468551
# text. Other we don't merge. because content can be none when model
469552
# safety threshold is triggered
470553
if message.server_content.interrupted:
471554
if text:
472-
yield self.__build_full_text_response(text, is_thought)
555+
yield self.__build_full_text_response(
556+
text,
557+
is_thought,
558+
last_grounding_metadata,
559+
interrupted=True,
560+
)
473561
text = ''
474562
is_thought = False
563+
last_grounding_metadata = None
475564
else:
476565
yield LlmResponse(
477566
interrupted=message.server_content.interrupted,
567+
grounding_metadata=last_grounding_metadata,
478568
model_version=self._model_version,
479569
live_session_id=live_session_id,
480570
)
571+
last_grounding_metadata = None
481572
if message.tool_call:
482573
logger.debug('Received tool call: %s', message.tool_call)
483574
if text:
484-
yield self.__build_full_text_response(text, is_thought)
575+
yield self.__build_full_text_response(
576+
text, is_thought, last_grounding_metadata
577+
)
485578
text = ''
486579
is_thought = False
580+
last_grounding_metadata = None
487581
tool_call_parts.extend([
488582
types.Part(function_call=function_call)
489583
for function_call in message.tool_call.function_calls
490584
])
585+
if not self._is_gemini_3_1_flash_live:
586+
if tool_call_metadata is None:
587+
tool_call_metadata = last_grounding_metadata
491588
# Gemini 3.1 does not emit turn_complete until it receives the
492589
# tool response, so yield tool calls immediately to avoid
493590
# deadlocking the conversation. Other models (e.g. 2.5-pro,
@@ -500,10 +597,12 @@ async def receive(self) -> AsyncGenerator[LlmResponse, None]:
500597
)
501598
yield LlmResponse(
502599
content=types.Content(role='model', parts=tool_call_parts),
600+
grounding_metadata=last_grounding_metadata,
503601
model_version=self._model_version,
504602
live_session_id=live_session_id,
505603
)
506604
tool_call_parts = []
605+
last_grounding_metadata = None
507606
if message.session_resumption_update:
508607
logger.debug('Received session resumption message: %s', message)
509608
yield (

0 commit comments

Comments
 (0)