Switch default Gemini model to gemini-3.5-flash; de-flake Gemini tests - #105
Merged
Merged
Conversation
gemini-2.5-flash-lite has been hitting sustained 503 "high demand" errors from Google, intermittently failing the Gemini integration tests and blocking the 0.0.36 release. Move the default to gemini-3.5-flash, which is stably available. The Gemini integration tests also carried brittle assertions that check for a literal keyword in the model's free-form response (`"Grafi" in content`, `"weather" in content`). These passed only because 2.5-flash-lite happened to phrase responses a certain way; newer models phrase differently and intermittently fail them. Replace the keyword checks with phrasing-independent non-empty-response assertions. The meaningful assertions are retained: assistant role, response length bounds, and exact event counts (which validate the tool was actually invoked and the workflow ran end-to-end). Also syncs uv.lock to version 0.0.36 (left stale by the version bump). Verified: gemini_tool_example.py and simple_gemini_function_call_assistant_example.py each pass 10/10 runs; full simple_llm_assistant and function_call_assistant integration suites pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
0.0.36release is blocked: CI'ssimple_llm_assistantandfunction_call_assistantintegration jobs fail, which gates the PyPI publish. Both jobs are the only ones that exercise Gemini.Root cause is two-fold:
gemini-2.5-flash-lite(the old default) is under a sustained Google-side503 UNAVAILABLE"high demand" outage, intermittently failing the Gemini tests."Grafi" in content,"weather" in content). These only passed because2.5-flash-litehappened to phrase responses a certain way. Any other model phrases differently and intermittently fails them — so a model switch alone doesn't fix it.What
gemini-3.5-flash(gemini_tool.py), which is stably available. Updated the deserialize fallback and the docstring comment (lite/pro alternatives) to match.< 300), and exact event counts (== 2/== 4/== 24), which validate the tool was actually invoked and the workflow ran end-to-end.uv.lockto0.0.36(left stale by the version-bump PR Bump version to 0.0.36 to trigger PyPI publish #103).Validation
gemini_tool_example.py: 10/10 runs passsimple_gemini_function_call_assistant_example.py: 10/10 runs passsimple_llm_assistant(9/9) andfunction_call_assistant(21/21) integration suites passtests/tools/llms/test_gemini_tool.py+tests/tools/test_tool_factory.py: 26 passRelease impact
Merging this to
mainunblocks the gated0.0.36publish (pyproject0.0.36≠ PyPI0.0.35), so the existing version → publish → release chain will run and ship0.0.36once integration is green.🤖 Generated with Claude Code