Skip to content

feat: add gemini interactions native tools - #597

Merged
chenyme merged 8 commits into
DEEIX-AI:devfrom
ADabbler:feat/gemini-interactions-native-tools
Aug 16, 2026
Merged

feat: add gemini interactions native tools#597
chenyme merged 8 commits into
DEEIX-AI:devfrom
ADabbler:feat/gemini-interactions-native-tools

Conversation

@ADabbler

Copy link
Copy Markdown
Contributor

Summary

Add native Google Search, Code Execution, and URL Context support to Gemini Interactions, together with a protocol-specific model-capabilities preset and option controls.

Before this change:

  • Native tools enabled for a Gemini Interactions model could pass option-policy filtering but were not included in the provider request payload.
  • The native tool catalog had no Gemini Interactions definitions for Google Search, Code Execution, or URL Context.
  • Generate Content and Interactions tools share catalog keys and types but use different payload formats. The admin UI could merge these definitions without considering their payloads, resulting in incorrect protocol labels or tool configuration.
  • Gemini Interactions had no dedicated preset for thinking level, thinking summaries, maximum output tokens, and its three native tools.

The Interactions adapter previously serialized function tools from GenerateInput.Tools without merging provider-hosted tools from the filtered model options. The frontend also identified catalog entries by tool key and type alone, which was insufficient for tools that have protocol-specific payloads.

This change uses the shared tool-declaration pipeline to merge native and function tools, registers protocol-specific Interactions tool definitions, and keeps Generate Content and Interactions payloads separate. It also adds the Interactions preset, option allowlist updates, migration handling for previous default allowlists, and localized parameter labels and descriptions.

Change type

  • Bug fix
  • Feature

Affected areas

  • Frontend / UI
  • Backend / API
  • Conversations / streaming
  • Model routing / providers
  • MCP / tools
  • Admin console

Verification

  • pnpm check
  • pnpm test
  • pnpm build
  • git diff --check

Screenshots, API examples, or logs

image image image

Configuration, migration, and compatibility notes

  • Adds a gemini_interactions model capability preset; the existing protocol and endpoint routing are unchanged.
  • Adds generation_config.thinking_summaries to the default Interactions option allowlist.
  • Adds generationConfig.thinkingConfig.includeThoughts and generationConfig.thinkingConfig.thinkingLevel to the default Generate Content option allowlist.
  • During settings seeding, installations whose stored allowlist matches a recognized previous default are upgraded automatically. Other custom allowlists remain unchanged.
  • Generate Content and Interactions continue to use separate native tool payload formats.
  • No HTTP API contract, database migration, environment variable, or deployment configuration change is required.

Documentation

  • Documentation is not needed for this change.

Security and privacy

  • No secrets, tokens, credentials, local config, or personal data are included.
  • User data access remains scoped by authenticated user context unless an admin-only path explicitly requires broader access.
  • Security-sensitive behavior was reviewed, including authentication, authorization, provider routing, file processing, billing, and admin APIs where relevant.

Checklist

  • I searched existing issues and pull requests.
  • Changes are focused and do not include unrelated refactors.
  • Tests or static verification were run where practical.
  • User-facing behavior, deployment steps, API contracts, or configuration changes are documented.
  • Generated artifacts are included only when this project explicitly requires them.
  • Caches, build output, .pyc files, .env files, and local storage data are not committed.

Comment thread backend/internal/infra/llm/gemini_interactions.go Fixed
@chenyme
chenyme self-requested a review August 16, 2026 04:22
@chenyme chenyme self-assigned this Aug 16, 2026
@chenyme

chenyme commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Additional changes

  • Align Gemini Interactions streaming with the official event_type and step.* event schema.
  • Preserve stable native-tool call IDs across start, delta, result, and completion events to prevent duplicate traces and usage counts.
  • Parse Google Search, Code Execution, and URL Context call/result steps into unified server-side tool traces.
  • Capture thought and thought_summary output as structured reasoning without leaking it into assistant text.
  • Read cumulative streaming usage from metadata.total_usage, including cached, output, and thought tokens.
  • Keep total_tool_use_tokens in raw usage metadata without adding it again to billable tokens.
  • Generate protocol-specific native-tool payloads for multi-protocol models and remove duplicate default tools.
  • Restrict inferred tool variants to protocols actually supported by the model.
  • Show route-dependent capability status when only part of a model’s protocol set supports an option or native tool.
  • Keep frontend hard-denied model-option paths aligned with backend-managed prompt-cache fields.

Additional verification

  • pnpm check
  • pnpm test
  • Targeted Gemini Interactions, model-option policy, settings migration, and native-tool catalog tests
  • Backend production build
  • Frontend production build
  • git diff --check
  • git diff dev --check

Live verification against the Google API was not run because no provider credentials were used.

@chenyme
chenyme merged commit 78df4e3 into DEEIX-AI:dev Aug 16, 2026
10 checks passed
@ADabbler
ADabbler deleted the feat/gemini-interactions-native-tools branch August 16, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants