modelcards update: refresh generated cards for 9 providers (+353 / ~140) - #12
Merged
Merged
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Provider tool-call precedence currently uses msg.tool_calls or metadata.get("tool_calls"), which can incorrectly resurrect legacy tool calls when msg.tool_calls is an empty list.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (2)
What changed in this PR
This PR primarily refreshes bundled, generated model cards across multiple providers (OpenAI, Google, Poe, OpenRouter, DeepInfra, HuggingFace, etc.). Because it’s stacked on earlier PRs, it also includes supporting runtime changes around prompt registry wiring, tool-calling (R-2) message serialization, and small provider/storage updates.
Changes:
- Regenerates/updates
src/llmcore/model_cards/default_cards/**model card JSONs for many providers. - Introduces first-class
Message.tool_callsand updates several providers/storage backends to round-trip tool-calling data correctly. - Adds/adjusts test fixtures and agent/prompt registry wiring for the grimoire-backed control plane.
| File | Description |
|---|---|
| tools/cardctl/adapters/__init__.py | Registers TypeSafe adapter and jev alias for card generation/diffing. |
| tests/providers/test_kimi_provider.py | Updates/extends Kimi provider tests for non-empty assistant content constraints. |
| tests/conftest.py | Adds a session-scoped bundled prompt registry fixture for tests. |
| tests/agents/test_persona_and_single_agent.py | Injects prompt registry into SingleAgentMode tests; asserts it’s required. |
| tests/agents/test_circuit_breaker_integration.py | Wires prompt registry fixture into cognitive cycle construction. |
| tests/agents/observability/conftest.py | Ensures llmcore.shared_events can be imported in isolated observability tests. |
| src/llmcore/models.py | Adds Message.tool_calls as a first-class field. |
| src/llmcore/storage/sqlite_session.py | Persists tool-protocol fields via metadata so they round-trip without schema changes. |
| src/llmcore/storage/constants.py | Centralizes :memory: token handling to prevent accidental directory creation. |
| src/llmcore/storage/chromadb_vector.py | Treats :memory: as in-memory storage rather than a filesystem path. |
| src/llmcore/providers/kimi_provider.py | Prefers first-class tool calls and enforces non-empty assistant content placeholder for Moonshot/Kimi. |
| src/llmcore/providers/mistral_provider.py | Prefers first-class tool calls over legacy metadata for assistant messages. |
| src/llmcore/providers/zai_provider.py | Prefers first-class tool calls over legacy metadata for assistant messages. |
| src/llmcore/providers/anthropic_provider.py | Accepts tool calls from first-class field or legacy metadata when building tool-use blocks. |
| src/llmcore/providers/poe_provider.py | Flattens tool-role messages for Poe’s text-only protocol when using native path. |
| src/llmcore/providers/manager.py | Registers TypeSafe provider and jev alias for provider lookup/config. |
| src/llmcore/tracing.py | Imports OpenTelemetry trace inside the exception-recording helper. |
| src/llmcore/observability/events.py | Forwards emitted events into the shared-events spine when sinks exist. |
| src/llmcore/agents/observability/logger.py | Forwards agent events into the shared-events spine when sinks exist. |
| src/llmcore/model_cards/__init__.py | Re-exports model_supports_native_search helper. |
| src/llmcore/memory/__init__.py | Re-exports create_semantiscan_memory_backend. |
| src/llmcore/grimoire_pack/** | Adds bundled pack metadata, spells, runes, and promptlets for control-plane prompting. |
| src/llmcore/model_cards/default_cards/** | Large generated refresh of bundled model card JSONs across providers. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+535
to
+538
| # First-class Message.tool_calls (R-2) takes precedence over the | ||
| # legacy metadata["tool_calls"] channel. | ||
| tool_calls = getattr(msg, "tool_calls", None) or metadata.get("tool_calls") | ||
|
|
Comment on lines
+666
to
+670
| # metadata channel so native tool-role results pair correctly. | ||
| if role_str == "assistant": | ||
| if "tool_calls" in metadata: | ||
| msg_dict["tool_calls"] = metadata["tool_calls"] | ||
| if not msg.content: | ||
| msg_dict["content"] = None | ||
| tool_calls = getattr(msg, "tool_calls", None) or metadata.get("tool_calls") | ||
| if tool_calls: | ||
| msg_dict["tool_calls"] = tool_calls |
araray
force-pushed
the
av/modelcards_refresh_2026-09
branch
from
September 19, 2026 04:18
cf16a56 to
19c3bee
Compare
…er, deepinfra, huggingface Data-only refresh via `python -m tools.cardctl generate <provider>` (no --force: hand-written `source: builtin` cards untouched; only generated cards added/updated). 353 new cards, 140 updated: openai +11 google +7 zai +3 deepseek +1 kimi +1 / 2 poe +69 / 10 openrouter +70 / 43 deepinfra +122 / 80 (126 deprecated skipped) huggingface +69 / 5 Updates carry upstream pricing, context, capability and description changes (openrouter/deepinfra/poe) plus Moonshot release_date bumps. Not refreshed: mistral (MISTRAL_API_KEY returns 401), anthropic/xai/qwen (no keys), ollama (local), deepgram (no adapter); typesafe already in sync. Verification: `cardctl validate` passes for every regenerated provider (openai 130, google 42, zai 13, deepseek 7, kimi 16, poe 489, openrouter 586, deepinfra 221, huggingface 297 — 0 failures); the registry loads all 1983 builtin cards; tests/model_cards: 60 passed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
araray
force-pushed
the
av/modelcards_refresh_2026-09
branch
from
September 19, 2026 04:22
19c3bee to
925aac2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Data-only refresh of the bundled model cards via the provider management tool (
python -m tools.cardctl generate <provider>, no--force, so hand-writtensource: builtincards are untouched and onlysource: generatedcards are added or updated).Scope
gpt-5.6-*,gpt-6-astra,gpt-image-2.5-flaregemini-3.5-flash-lite,gemini-3.6/3.7/3.8-flash,gemini-omni-1.1-flashglm-5.3,glm-5.3-flash,glm-5.3-flashxdeepseek-flashkimi-k3; tworelease_datebumps from the Moonshot listingcardctl diff typesafe: no differences)Not refreshed: mistral (the
MISTRAL_API_KEYin the env file returns401 UnauthorizedonGET /v1/models— needs a new key), anthropic / xai / qwen (noANTHROPIC_API_KEY/XAI_API_KEY/DASHSCOPE_API_KEYavailable), ollama (local models are machine-specific), deepgram (no cardctl adapter).Verification
cardctl validate <provider>on every regenerated directory: all pass (openai 130, google 42, zai 13, deepseek 7, kimi 16, poe 489, openrouter 586, deepinfra 221, huggingface 297 — 0 failures).tests/model_cards+tests/providersre-run on the branch (see the last commit message for counts).🤖 Generated with Claude Code