Repository navigation
Add the Gemini 4 Carbon and Gemini 4 Hydrogen model cards and metadata - #266
Merged
Merged
Conversation
…release dates Adds the serving card, list price and release date for Google's Gemini 4 Carbon and Gemini 4 Hydrogen so the board can run them. No result: the models join the board in a later release, and this changes no published artifact. - Release: the Gemini API changelog lists gemini-4-carbon as generally available on 2026-11-10 and gemini-4-hydrogen on 2026-11-12 (read 2026-11-13). Carbon's earlier release to cybersecurity defenders through the Fairwind Program is excluded under the public-availability rule. Route: the first-party Gemini API, gemini/gemini-4-carbon and gemini/gemini-4-hydrogen. - Price: Carbon $2 / $10 per 1M input/output tokens and $0.10 cached input, the rate Google lists through 2026-11-30 (it doubles on 2026-12-01; the override holds the rate through 2026-11-30). Hydrogen $1.50 / $7.50 and $0.075 cached input, with no dated change. - Cards: written from Google's documentation, read 2026-11-13; they record no request to either model. Both take the board's forced answer tool on Gemini 4 Argon's pattern, send no reasoning parameter (Google gives a default thinking level of medium for Carbon and high for Hydrogen) and take the thinking class's 300s timeout. Neither card sets a per-scenario cost. - Ceiling: 32,768 completion tokens for Carbon and 65,536 for Hydrogen, what one non-streaming generateContent request returns per Google's long-decode and thinking guides and the Models API. - The frontier-column list is unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…date the Carbon rate Wording fixes to the Gemini 4 Carbon and Gemini 4 Hydrogen cards and the Carbon price comment. No card field, price, release date or test assertion changes. - Ceiling: 32,768 completion tokens for Carbon and 65,536 for Hydrogen are the Models API's outputTokenLimit for each model, whose entry lists generateContent, the method the harness calls once without streaming. Google's long-decode and thinking guides describe the Interactions API and give the same per-request figures for non-streaming requests there; the card text and the previous commit's message cited them for generateContent. The Carbon card also says that the migration guide's examples use the Interactions API. - Price: the cost reconstruction takes no date, so the Carbon override prices a request at $2 / $10 per 1M and $0.10 cached input whenever it is made. Those are Google's rates only for requests made through 2026-11-30 (they double on 2026-12-01). The comment had said the row is priced at the rate in force on its run day and that the override keeps it priced if the price map changes on 2026-12-01. - Module docstring: cards encode serving-stack facts discovered during onboarding, and a card written before its model's onboarding says so in its notes, as these two do. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…arbon answers as JSON The two cards were written from Google's pages before any request to either model. Both models are now onboarded (2026-11-14), and the cards say what the onboarding found, on Gemini 4 Argon's pattern. No price, release date or ceiling changes, and no board result. - Carbon contract: JSON. The gauntlet, run with no card, sent the forced answer tool first; the API accepted the forced call, but the model reasoned until the 16,384-token budget ran out (16,381 reasoning tokens, finish=length, 0 of 3 outputs parsed, 203s). The JSON contract then passed 3/3 (9,964 tokens, 105s) and 16/16 whole-scenario (7,257 tokens, 79s). The procedure fixed before the runs makes the card JSON when the tool probe fails, and the tool contract was not re-tested. Argon's tool_choice auto rule applies only to a forced-tool row, so Carbon has no auto sensitivity run. - Hydrogen contract: the forced tool, as before. The gauntlet passed 3/3 (10,956 tokens, 48s) and 16/16 (11,947 tokens, 51s). The notes hold one placeholder sentence, marked HPROBE-RESULT, for the harness-path probe's forced-against-auto reasoning result, which is still running. - Timeouts: Carbon 900s. Google recommends client timeouts of at least 15 minutes, and a retry at the 32,768-token ceiling takes about 345 to 410s at the gauntlet's 80 to 95 tokens a second. Hydrogen 600s: a request at its 65,536-token ceiling takes about 280 to 290s at 228 to 234 tokens a second. - Expected cost per scenario: Carbon 0.075, Hydrogen 0.094, each the gauntlet's whole-scenario probe. Carbon's cost-reconstruction row uses that call's exact counts (1,271 prompt, 7,257 completion, $0.075112). Hydrogen's uses 11,947 completion tokens and Argon's 2,762-token prompt for the same request, marked HPROBE-PROMPT-TOKENS until Hydrogen's own count is in. - Reasoning: both models reasoned on all 50 calls of a one-question generateContent probe, in all five request shapes (Argon reasoned on none with JSON or no tools). generateContent reports usageMetadata.thoughtsTokenCount on every response, litellm 1.95.0 maps it to reasoning_tokens, and the harness records it. Every response names the requested model with no dated snapshot. - Ceilings: unchanged (Carbon 32,768, Hydrogen 65,536). Google's generateContent long-decode guide, read 2026-11-14, now backs them for the request the harness sends, one direct non-streaming REST call. The 16,384-token starting budget comes from code, not from the card. - Module docstring: back to main's sentence, since no card is written before onboarding any more. - Tests: Carbon's and Hydrogen's treatments in EXPECTED; card cost locks; the shared request test split so Carbon's case checks the JSON request and Hydrogen's the forced tool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The harness-path probe sent the board's own request for the frozen manifest's scenarios 0 to 15, 16 calls a shape, under the rule fixed before its first call. Calls with reasoning tokens: forced 16 of 16, tool_choice auto 16 of 16 and JSON 16 of 16; forced against auto, two-sided Fisher exact p = 1.0. Forcing the tool did not lower how often the model reasoned, so the rule adds no tool_choice auto sensitivity run. Hydrogen's notes now give that result, and the probe's length finishes: 6 forced, 6 auto and 2 JSON calls ran out of the 16,384-token budget on one request each, which the board escalates. The probe's forced call on scenario 0 is the same request as the gauntlet's tool-full probe (the same messages, answer tool and 16,384-token budget), and Google returned 2,762 prompt tokens for it. The cost-reconstruction row and the card's comment now give that count. With the gauntlet's 11,947 completion tokens at $1.50/$7.50 per 1M it prices at $0.0937455, so the card's 0.094 stands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…the cards' cost to its counts Three readers checked the cards against the onboarding record and Google's saved pages and found every number true. This commit takes their points on what the comments and notes left out. - Timeouts. Google's generateContent long-decode guide says the server closes a non-streaming connection after 10 minutes, and its option table gives 20 minutes per request. Both cards now say what that means for their timeout: no request runs past 600s if the server closes at 10 minutes, and Carbon's 900s also covers the table's 20 minutes. Hydrogen's comment gives the slowest rates in the record (about 215 to 220 tokens a second), at which a request at the 65,536-token ceiling takes about 300s. - Carbon's budget. The harness doubles the 16,384-token budget once, to 32,768, and records any output still missing there as budget_exhausted_at_ceiling. The comment and the notes now say so. - Carbon's price. The override's comment now says to change the entry before any request made on or after 2026-12-01, and both price comments say what the record holds about litellm's remote map (the gauntlet's rates on 2026-11-14) in place of an unsaved read. - Hydrogen's notes name the harness-path probe's households as the frozen manifest's first 16 scenarios, and attach the reasoning-token range to all 14 calls that ran out of budget. - The medians of thought tokens that end in .5 read "about". - test_gemini_4_carbon_and_hydrogen_cards_record_the_gauntlet_cost now reconstructs each gauntlet probe's cost from its token counts at the price override and checks the card against it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…d the code that prices a row late Carbon's card comment said its 900s timeout covers the 20 minutes in Google's option table; 900s is 15 minutes. The comment now says what 900s exceeds: the 345 to 410s the gauntlet's rates predict for a request at the ceiling, and the 620s the slowest one-question probe rate predicts. Both cards now give the long-decode guide's 20-minute read-timeout advice beside its 10-minute close. Carbon's card says the gauntlet's tool probe ran once at 16,384 tokens, half the minimum Google's guide recommends for the model. The price-override comment names the code that prices a row after Google served the response: batch collection, a disk-cache replay and the analysis fallback. Each card's cost comment adds what its smoke households cost. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…cords exhaustion Carbon's card said a household that needs more than 32,768 completion tokens cannot finish, and that the harness records any output still missing at the ceiling as budget_exhausted_at_ceiling. The ceiling bounds each response: the harness keeps the values a truncated response returned and asks again for the outputs still missing, so a household's requests can total more. It records the exhaustion label only when a length-truncated response at the ceiling leaves outputs missing. The comment and the notes now say so. The cost test's docstring says where Gemini 4 Hydrogen's prompt count comes from. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Contributor
Author
|
Review record at a72462a:
Round 2 also asked to remove the description's last line. It stays: it is the attribution line this session puts on every pull request. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We add model cards, list prices and release dates for Google's Gemini 4 Carbon and Gemini 4 Hydrogen, on the pattern of #219 (Gemini 4 Argon's card). We change no published artifact. Anyone who runs
eval-no-toolswith no--modelnow sends requests to both models, along with every other model inconfig.MODELS. We will add both models' scores to the board in the next data release.Facts (read 2026-11-13 and 2026-11-14)
gemini-4-carbonas generally available that day. Google's post of that day says Google had released the model two weeks earlier to cybersecurity defenders through its Fairwind Program. The board dates each model from its public day, as it dates Argon. The changelog listsgemini-4-hydrogenas generally available on 2026-11-12, on the free and the paid tier.gemini/gemini-4-carbon,gemini/gemini-4-hydrogen), as it does every other Gemini row. Each model page lists one stable version and no dated snapshot. Every saved response names the requested model.analysis.model_cost_latencycomputes one from the override as it stands when the analysis runs. Once we change the override, each of these would put the December rate on a request made in November. For Hydrogen the page lists $1.50 / $7.50 and $0.075 cached input, with no dated change. It also lists a free tier; the board prices the row at the paid rates.outputTokenLimitin the Models API. Google's generateContent long-decode guide gives the same caps for a direct non-streaming REST call, which is what the harness sends.The cards
Carbon answers as JSON. The gauntlet, run with no card, sent the board's forced answer tool first. The Gemini API accepted the forced call, but on the 3-variable probe the model reasoned until the 16,384-token budget ran out: 16,381 reasoning tokens, 0 of 3 outputs parsed, 203s. The gauntlet sends each probe once and does not escalate the budget. The tool request therefore ran at 16,384 tokens, half the minimum that Google's Gemini 4 Carbon guide recommends for this model. The gauntlet then passed on the JSON contract: 3/3 (9,964 completion tokens, 105s) and 16/16 whole-scenario (7,257 tokens, 79s), about $0.075 per scenario. Before the runs, we set the rule that a card takes the JSON contract when its tool probe fails. We did not re-test the tool contract, since re-running the gauntlet until the tool probe passes would choose the contract on a retry.
Hydrogen takes the forced answer tool. The gauntlet, run with no card, passed on the tool contract: 3/3 (10,956 completion tokens, 48s) and 16/16 whole-scenario (11,947 tokens, 51s), about $0.094 per scenario.
Reasoning. The harness sends no
thinkingConfig. Google's thinking guide gives Carbon a default thinking level of medium and Hydrogen high. Both models returned thought tokens on every call whose usage we saved, in every request shape:tool_choiceautoNo sensitivity run. Before the first onboarding call to each model, we wrote down Argon's rule for it: we add a
tool_choiceauto sensitivity run only if forcing the tool suppresses reasoning. On the board's own request, Hydrogen returned reasoning tokens on 16 of 16 forced calls and 16 of 16 auto calls (Fisher exact p = 1.0), so its row has no sensitivity run. Carbon's request declares no tool, so the rule has nothing to compare.Budget. The harness gives both rows the Gemini family's 16,384-token starting budget. Our probe of Hydrogen on the board's own request sent each request once, and 6 of its 16 forced calls ran out of that budget. On such a call the harness doubles the budget and asks again, up to 65,536 for Hydrogen. Carbon gets one doubling, to 32,768. If a length-truncated response at that ceiling leaves outputs missing, the harness records those outputs as
budget_exhausted_at_ceiling.Timeout. Carbon's card sets a 900s timeout, since Google's Carbon guide recommends client timeouts of at least 15 minutes for this model. At the gauntlet's 80 to 95 completion tokens a second, a retry at the 32,768-token ceiling takes about 345 to 410s. Hydrogen's card sets 600s. At the gauntlet's 228 to 234 tokens a second, a request at the 65,536-token ceiling takes about 280 to 290s, too close to the thinking class's 300s. Google's long-decode guide says the server closes a non-streaming connection after 10 minutes. The same guide recommends a 20-minute client read timeout, and its option table gives 20 minutes per request. If the server closes at 10 minutes, no request on either row runs past 600s. Carbon's one-question probe ran as slow as 53 tokens a second, and at that rate a request at the ceiling takes about 620s and would not finish. If the table's 20 minutes applies, Carbon's 900s still exceeds that 620s and the 345 to 410s the gauntlet's rates predict for a request at its ceiling.
Smoke. A two-household smoke through
eval-no-toolsfinished both households for each model. In each run all 36 outputs came back with a prediction and an explanation, and every row carries reasoning tokens and a cost. Neither run escalated a budget. The runs cost $0.18 for Carbon and $0.23 for Hydrogen. At about $0.090 and $0.113 a household, they exceed the gauntlet figures the cards carry ($0.075 and $0.094).Tests
tests/test_model_cards.pyholds each card's expected row. It reconstructs each model's whole-scenario gauntlet cost from its token counts at the price override and checks the card's expected cost against it.tests/test_eval_no_tools.pychecks that Carbon's request asks for a JSON object and declares no tool, that Hydrogen's forces the answer tool, and that neither sets a reasoning parameter. It also holds a cost-reconstruction row for each model's whole-scenario gauntlet probe.tests/test_paper_results.pychecks both release dates.Locally, eleven test files that read the cards, the config or the release dates passed at this head (1,634 passed, 13 skipped):
test_model_cards,test_eval_no_tools,test_paper_results,test_disclosures,test_onboard,test_supervisor,test_notes,test_freeze_argon,test_freeze_musewonder,test_snapshot_artifactsandtest_cache. CI runs the suite with slow tests excluded.🤖 Generated with Claude Code