Skip to content

Add the Gemini 4 Carbon and Gemini 4 Hydrogen model cards and metadata - #266

Merged
MaxGhenis merged 7 commits into
mainfrom
gemini-4-carbon-hydrogen-cards
Nov 15, 2026
Merged

MaxGhenis merged 7 commits into
mainfrom
gemini-4-carbon-hydrogen-cards

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

We add model cards, list prices and release dates for Google's Gemini 4 Carbon and Gemini 4 Hydrogen, on the pattern of #219 (Gemini 4 Argon's card). We change no published artifact. Anyone who runs eval-no-tools with no --model now sends requests to both models, along with every other model in config.MODELS. We will add both models' scores to the board in the next data release.

Facts (read 2026-11-13 and 2026-11-14)

  • Release. Google opened Gemini 4 Carbon to the public on 2026-11-10; the Gemini API changelog lists gemini-4-carbon as generally available that day. Google's post of that day says Google had released the model two weeks earlier to cybersecurity defenders through its Fairwind Program. The board dates each model from its public day, as it dates Argon. The changelog lists gemini-4-hydrogen as generally available on 2026-11-12, on the free and the paid tier.
  • Route. The board calls both models on the first-party Gemini API (gemini/gemini-4-carbon, gemini/gemini-4-hydrogen), as it does every other Gemini row. Each model page lists one stable version and no dated snapshot. Every saved response names the requested model.
  • Price. For Carbon, Google's pricing page lists $2 / $10 per 1M input/output tokens (output includes thinking tokens) and $0.10 cached input "through November 30, 2026", and $4 / $20 / $0.20 from December 1. The override prices every request at the first rate, whatever day the board sends it, so it matches Google's price only through 2026-11-30. The harness prices a batch result when it collects the result, and it prices a response that the disk cache replays at the time of the replay. For a model whose rows record no cost, analysis.model_cost_latency computes one from the override as it stands when the analysis runs. Once we change the override, each of these would put the December rate on a request made in November. For Hydrogen the page lists $1.50 / $7.50 and $0.075 cached input, with no dated change. It also lists a free tier; the board prices the row at the paid rates.
  • Ceiling. The cards cap completion at 32,768 tokens for Carbon and 65,536 for Hydrogen, each model's outputTokenLimit in the Models API. Google's generateContent long-decode guide gives the same caps for a direct non-streaming REST call, which is what the harness sends.

The cards

  • Carbon answers as JSON. The gauntlet, run with no card, sent the board's forced answer tool first. The Gemini API accepted the forced call, but on the 3-variable probe the model reasoned until the 16,384-token budget ran out: 16,381 reasoning tokens, 0 of 3 outputs parsed, 203s. The gauntlet sends each probe once and does not escalate the budget. The tool request therefore ran at 16,384 tokens, half the minimum that Google's Gemini 4 Carbon guide recommends for this model. The gauntlet then passed on the JSON contract: 3/3 (9,964 completion tokens, 105s) and 16/16 whole-scenario (7,257 tokens, 79s), about $0.075 per scenario. Before the runs, we set the rule that a card takes the JSON contract when its tool probe fails. We did not re-test the tool contract, since re-running the gauntlet until the tool probe passes would choose the contract on a retry.

  • Hydrogen takes the forced answer tool. The gauntlet, run with no card, passed on the tool contract: 3/3 (10,956 completion tokens, 48s) and 16/16 whole-scenario (11,947 tokens, 51s), about $0.094 per scenario.

  • Reasoning. The harness sends no thinkingConfig. Google's thinking guide gives Carbon a default thinking level of medium and Hydrogen high. Both models returned thought tokens on every call whose usage we saved, in every request shape:

    Request shape Carbon, one question (10 calls each) Hydrogen, one question (10 calls each) Hydrogen, the board's request for 16 frozen households (16 calls each)
    Forced tool 10 with thought tokens 10 16 with reasoning tokens
    tool_choice auto 10 10 16
    Mode ANY, no named function 10 10 not run
    JSON response type 10 10 16
    No tools 10 10 not run
  • No sensitivity run. Before the first onboarding call to each model, we wrote down Argon's rule for it: we add a tool_choice auto sensitivity run only if forcing the tool suppresses reasoning. On the board's own request, Hydrogen returned reasoning tokens on 16 of 16 forced calls and 16 of 16 auto calls (Fisher exact p = 1.0), so its row has no sensitivity run. Carbon's request declares no tool, so the rule has nothing to compare.

  • Budget. The harness gives both rows the Gemini family's 16,384-token starting budget. Our probe of Hydrogen on the board's own request sent each request once, and 6 of its 16 forced calls ran out of that budget. On such a call the harness doubles the budget and asks again, up to 65,536 for Hydrogen. Carbon gets one doubling, to 32,768. If a length-truncated response at that ceiling leaves outputs missing, the harness records those outputs as budget_exhausted_at_ceiling.

  • Timeout. Carbon's card sets a 900s timeout, since Google's Carbon guide recommends client timeouts of at least 15 minutes for this model. At the gauntlet's 80 to 95 completion tokens a second, a retry at the 32,768-token ceiling takes about 345 to 410s. Hydrogen's card sets 600s. At the gauntlet's 228 to 234 tokens a second, a request at the 65,536-token ceiling takes about 280 to 290s, too close to the thinking class's 300s. Google's long-decode guide says the server closes a non-streaming connection after 10 minutes. The same guide recommends a 20-minute client read timeout, and its option table gives 20 minutes per request. If the server closes at 10 minutes, no request on either row runs past 600s. Carbon's one-question probe ran as slow as 53 tokens a second, and at that rate a request at the ceiling takes about 620s and would not finish. If the table's 20 minutes applies, Carbon's 900s still exceeds that 620s and the 345 to 410s the gauntlet's rates predict for a request at its ceiling.

  • Smoke. A two-household smoke through eval-no-tools finished both households for each model. In each run all 36 outputs came back with a prediction and an explanation, and every row carries reasoning tokens and a cost. Neither run escalated a budget. The runs cost $0.18 for Carbon and $0.23 for Hydrogen. At about $0.090 and $0.113 a household, they exceed the gauntlet figures the cards carry ($0.075 and $0.094).

Tests

  • tests/test_model_cards.py holds each card's expected row. It reconstructs each model's whole-scenario gauntlet cost from its token counts at the price override and checks the card's expected cost against it.
  • tests/test_eval_no_tools.py checks that Carbon's request asks for a JSON object and declares no tool, that Hydrogen's forces the answer tool, and that neither sets a reasoning parameter. It also holds a cost-reconstruction row for each model's whole-scenario gauntlet probe.
  • tests/test_paper_results.py checks both release dates.

Locally, eleven test files that read the cards, the config or the release dates passed at this head (1,634 passed, 13 skipped): test_model_cards, test_eval_no_tools, test_paper_results, test_disclosures, test_onboard, test_supervisor, test_notes, test_freeze_argon, test_freeze_musewonder, test_snapshot_artifacts and test_cache. CI runs the suite with slow tests excluded.

🤖 Generated with Claude Code

MaxGhenis and others added 4 commits November 14, 2026 17:35
…release dates

Adds the serving card, list price and release date for Google's Gemini 4 Carbon and Gemini 4 Hydrogen so the board can run them. No result: the models join the board in a later release, and this changes no published artifact.

- Release: the Gemini API changelog lists gemini-4-carbon as generally available on 2026-11-10 and gemini-4-hydrogen on 2026-11-12 (read 2026-11-13). Carbon's earlier release to cybersecurity defenders through the Fairwind Program is excluded under the public-availability rule. Route: the first-party Gemini API, gemini/gemini-4-carbon and gemini/gemini-4-hydrogen.
- Price: Carbon $2 / $10 per 1M input/output tokens and $0.10 cached input, the rate Google lists through 2026-11-30 (it doubles on 2026-12-01; the override holds the rate through 2026-11-30). Hydrogen $1.50 / $7.50 and $0.075 cached input, with no dated change.
- Cards: written from Google's documentation, read 2026-11-13; they record no request to either model. Both take the board's forced answer tool on Gemini 4 Argon's pattern, send no reasoning parameter (Google gives a default thinking level of medium for Carbon and high for Hydrogen) and take the thinking class's 300s timeout. Neither card sets a per-scenario cost.
- Ceiling: 32,768 completion tokens for Carbon and 65,536 for Hydrogen, what one non-streaming generateContent request returns per Google's long-decode and thinking guides and the Models API.
- The frontier-column list is unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…date the Carbon rate

Wording fixes to the Gemini 4 Carbon and Gemini 4 Hydrogen cards and the Carbon price comment. No card field, price, release date or test assertion changes.

- Ceiling: 32,768 completion tokens for Carbon and 65,536 for Hydrogen are the Models API's outputTokenLimit for each model, whose entry lists generateContent, the method the harness calls once without streaming. Google's long-decode and thinking guides describe the Interactions API and give the same per-request figures for non-streaming requests there; the card text and the previous commit's message cited them for generateContent. The Carbon card also says that the migration guide's examples use the Interactions API.
- Price: the cost reconstruction takes no date, so the Carbon override prices a request at $2 / $10 per 1M and $0.10 cached input whenever it is made. Those are Google's rates only for requests made through 2026-11-30 (they double on 2026-12-01). The comment had said the row is priced at the rate in force on its run day and that the override keeps it priced if the price map changes on 2026-12-01.
- Module docstring: cards encode serving-stack facts discovered during onboarding, and a card written before its model's onboarding says so in its notes, as these two do.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…arbon answers as JSON

The two cards were written from Google's pages before any request to either model. Both models are now onboarded (2026-11-14), and the cards say what the onboarding found, on Gemini 4 Argon's pattern. No price, release date or ceiling changes, and no board result.

- Carbon contract: JSON. The gauntlet, run with no card, sent the forced answer tool first; the API accepted the forced call, but the model reasoned until the 16,384-token budget ran out (16,381 reasoning tokens, finish=length, 0 of 3 outputs parsed, 203s). The JSON contract then passed 3/3 (9,964 tokens, 105s) and 16/16 whole-scenario (7,257 tokens, 79s). The procedure fixed before the runs makes the card JSON when the tool probe fails, and the tool contract was not re-tested. Argon's tool_choice auto rule applies only to a forced-tool row, so Carbon has no auto sensitivity run.
- Hydrogen contract: the forced tool, as before. The gauntlet passed 3/3 (10,956 tokens, 48s) and 16/16 (11,947 tokens, 51s). The notes hold one placeholder sentence, marked HPROBE-RESULT, for the harness-path probe's forced-against-auto reasoning result, which is still running.
- Timeouts: Carbon 900s. Google recommends client timeouts of at least 15 minutes, and a retry at the 32,768-token ceiling takes about 345 to 410s at the gauntlet's 80 to 95 tokens a second. Hydrogen 600s: a request at its 65,536-token ceiling takes about 280 to 290s at 228 to 234 tokens a second.
- Expected cost per scenario: Carbon 0.075, Hydrogen 0.094, each the gauntlet's whole-scenario probe. Carbon's cost-reconstruction row uses that call's exact counts (1,271 prompt, 7,257 completion, $0.075112). Hydrogen's uses 11,947 completion tokens and Argon's 2,762-token prompt for the same request, marked HPROBE-PROMPT-TOKENS until Hydrogen's own count is in.
- Reasoning: both models reasoned on all 50 calls of a one-question generateContent probe, in all five request shapes (Argon reasoned on none with JSON or no tools). generateContent reports usageMetadata.thoughtsTokenCount on every response, litellm 1.95.0 maps it to reasoning_tokens, and the harness records it. Every response names the requested model with no dated snapshot.
- Ceilings: unchanged (Carbon 32,768, Hydrogen 65,536). Google's generateContent long-decode guide, read 2026-11-14, now backs them for the request the harness sends, one direct non-streaming REST call. The 16,384-token starting budget comes from code, not from the card.
- Module docstring: back to main's sentence, since no card is written before onboarding any more.
- Tests: Carbon's and Hydrogen's treatments in EXPECTED; card cost locks; the shared request test split so Carbon's case checks the JSON request and Hydrogen's the forced tool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The harness-path probe sent the board's own request for the frozen
manifest's scenarios 0 to 15, 16 calls a shape, under the rule fixed before
its first call. Calls with reasoning tokens: forced 16 of 16, tool_choice
auto 16 of 16 and JSON 16 of 16; forced against auto, two-sided Fisher
exact p = 1.0. Forcing the tool did not lower how often the model reasoned,
so the rule adds no tool_choice auto sensitivity run. Hydrogen's notes now
give that result, and the probe's length finishes: 6 forced, 6 auto and 2
JSON calls ran out of the 16,384-token budget on one request each, which
the board escalates.

The probe's forced call on scenario 0 is the same request as the gauntlet's
tool-full probe (the same messages, answer tool and 16,384-token budget),
and Google returned 2,762 prompt tokens for it. The cost-reconstruction row
and the card's comment now give that count. With the gauntlet's 11,947
completion tokens at $1.50/$7.50 per 1M it prices at $0.0937455, so the
card's 0.094 stands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policybench-site Ready Ready Preview Nov 15, 2026 2:40pm UTC

Request Review

…the cards' cost to its counts

Three readers checked the cards against the onboarding record and Google's
saved pages and found every number true. This commit takes their points on
what the comments and notes left out.

- Timeouts. Google's generateContent long-decode guide says the server
  closes a non-streaming connection after 10 minutes, and its option table
  gives 20 minutes per request. Both cards now say what that means for
  their timeout: no request runs past 600s if the server closes at 10
  minutes, and Carbon's 900s also covers the table's 20 minutes. Hydrogen's
  comment gives the slowest rates in the record (about 215 to 220 tokens a
  second), at which a request at the 65,536-token ceiling takes about 300s.
- Carbon's budget. The harness doubles the 16,384-token budget once, to
  32,768, and records any output still missing there as
  budget_exhausted_at_ceiling. The comment and the notes now say so.
- Carbon's price. The override's comment now says to change the entry
  before any request made on or after 2026-12-01, and both price comments
  say what the record holds about litellm's remote map (the gauntlet's
  rates on 2026-11-14) in place of an unsaved read.
- Hydrogen's notes name the harness-path probe's households as the frozen
  manifest's first 16 scenarios, and attach the reasoning-token range to all
  14 calls that ran out of budget.
- The medians of thought tokens that end in .5 read "about".
- test_gemini_4_carbon_and_hydrogen_cards_record_the_gauntlet_cost now
  reconstructs each gauntlet probe's cost from its token counts at the
  price override and checks the card against it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…d the code that prices a row late

Carbon's card comment said its 900s timeout covers the 20 minutes in
Google's option table; 900s is 15 minutes. The comment now says what 900s
exceeds: the 345 to 410s the gauntlet's rates predict for a request at the
ceiling, and the 620s the slowest one-question probe rate predicts. Both
cards now give the long-decode guide's 20-minute read-timeout advice beside
its 10-minute close.

Carbon's card says the gauntlet's tool probe ran once at 16,384 tokens, half
the minimum Google's guide recommends for the model. The price-override
comment names the code that prices a row after Google served the response:
batch collection, a disk-cache replay and the analysis fallback. Each card's
cost comment adds what its smoke households cost.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…cords exhaustion

Carbon's card said a household that needs more than 32,768 completion tokens
cannot finish, and that the harness records any output still missing at the
ceiling as budget_exhausted_at_ceiling. The ceiling bounds each response: the
harness keeps the values a truncated response returned and asks again for
the outputs still missing, so a household's requests can total more. It
records the exhaustion label only when a length-truncated response at the
ceiling leaves outputs missing. The comment and the notes now say so.

The cost test's docstring says where Gemini 4 Hydrogen's prompt count comes
from.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Review record at a72462a:

  • Round 1 (Subfleet job 20261114-190930-pb266-review-r1, at e0062a9) raised one P1, three P2 and several P3 findings. The P1 was a false clause of mine: the card comment said Carbon's 900s timeout covers the 20 minutes in Google's option table. 3b16074 fixed all of them, and added a third path that prices a row after its request (a disk-cache replay), which two in-session readers found.
  • Round 2 (job 20261114-234719-pb266-review-r2, at 3b16074, a full review with shell access) found every round-1 finding closed and raised one P0 and one P1. The P0: this description said "CI runs the full suite", and CI excludes slow tests. The P1: the card said a household that needs more than 32,768 completion tokens cannot finish on Carbon's row, but the ceiling bounds each response. a72462a fixed both, with its P3 on a test docstring.
  • Round 3 (job 20261115-093853-pb266-review-r3, scoped to those two findings) found both closed and nothing new.

Round 2 also asked to remove the description's last line. It stays: it is the attribution line this session puts on every pull request.

@MaxGhenis
MaxGhenis marked this pull request as ready for review November 15, 2026 15:02
@MaxGhenis
MaxGhenis merged commit 29d3ca9 into main Nov 15, 2026
6 checks passed
@MaxGhenis
MaxGhenis deleted the gemini-4-carbon-hydrogen-cards branch November 15, 2026 15:02

This branch was successfully deployed

1 active deployment
Preview — a72462a4 Deployed Nov 15, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant