Skip to content

Add the Gemini 4 Argon model card and metadata - #219

Merged
MaxGhenis merged 5 commits into
mainfrom
add-gemini-4-argon
Oct 13, 2026
Merged

MaxGhenis merged 5 commits into
mainfrom
add-gemini-4-argon

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

This is the card PR for Google's Gemini 4 Argon, following #216 (Meta Muse Wonder's card). Its scores join the published board in the next data release, so this PR changes no published artifact.

Facts (read 2026-10-13)

  • Release. Google announced Gemini 4 Argon on 2026-09-30 for Fairwind Program cyber partners only (TechCrunch). It opened to the public on 2026-10-13: the Gemini API changelog lists gemini-4-argon as generally available that day, and Google's launch post puts it on the paid tier of the Gemini API and in the Gemini app for Pro and Ultra subscribers. The board's release date is the public day, 2026-10-13; the partner-only day is excluded, as GPT-6 Astra's is.
  • Route. The first-party Gemini API, gemini/gemini-4-argon, like every other Gemini row. GET v1beta/models/gemini-4-argon lists an input limit of 1,048,576 and an output limit of 32,768. The model page lists one stable version and no dated snapshot.
  • Price. $2 / $10 per 1M input/output tokens (output includes thinking tokens) and $0.10 cached input. Google's pricing page (last updated 2026-10-13) lists those rates "through November 30, 2026" and $4 / $20 / $0.20 from December 1. The row is priced at the rate in force on its run day, as Gemini 3.7 Flash and 3.8 Flash are at their dated introductory rates.

The card

  • Contract: the forced answer tool. The Gemini API accepts a forced function call on this model. The gauntlet, run with no card, passed on the tool contract: 3/3 (442 completion tokens, 5s) and 16/16 whole-scenario (2,619 tokens, 24s), about $0.032 per scenario. The older Gemini rows answer as JSON by the family default; this row does not, as Gemini 3.6, 3.7 and 3.8 Flash do not.

  • Reasoning. The harness sends no thinkingConfig (checked on the wire for all 87 harness-probe calls). Google's guide gives a default thinking_level of medium. The model decides call by call whether to think, in every request shape:

    Request shape One question (10 calls each) The board's request, 13 frozen households (29 calls each)
    Forced tool (the board's) 6 with thought tokens 20 with reasoning tokens (480 to 2,737)
    tool_choice auto 7 25 (490 to 2,985)
    Mode ANY, no named function 5 not run
    JSON response type 0 22 (161 to 3,014)
    No tools 0 not run

    All 87 harness-probe calls parsed every output. So a forced tool does not switch this model's reasoning off, as it does Claude Opus 5's.

  • No sensitivity run: a forced tool does not suppress reasoning. The 13-household probe left it open (20 of 29 against 25 of 29, Fisher exact p = 0.21), so the rule written before the full sample (probes/DECISION-RULE.md) sent it to the 100-household runs. The forced-tool board run returned reasoning tokens on 80 households and a tool_choice auto run on 82 (p = 0.86). By household: 70 reasoned in both runs, 10 only when forced, 12 only under auto, 8 in neither. The criterion was the reasoning rate, never the score. The auto run is not published; it stays in the record as evidence.

  • Board runs. A first board run (at d8dd845, 17:19 to 17:30 UTC) answered all 100 households in one request each, 1,984/1,984 parsed, no escalation, $2.77. It recorded a 65,536-token escalation ceiling, the model page's figure (card review round 1, P2, below), and the freeze checks a run's recorded treatment against the registry, so the board row is a second run on the corrected card (18:28 to 18:38 UTC): 100/100 households in one request each, 1,984/1,984 parsed, no escalation, $2.79, reasoning tokens on 85 households. None of its responses replays the first run's. The first run and the auto run stay in the record as the reasoning comparison's evidence.

  • Budget and timeout. The thinking-default class's 300s timeout and the Gemini family's 16,384-token starting budget. The slowest probe call took 41s, and the slowest board household 53s in the first run and 47s in the second; the largest household used 6,374 and 4,872 completion tokens. provider_max_completion_tokens is 32,768. Google's pages disagree: the generateContent guide says one non-streaming request returns up to 32,768 tokens, and the Models API's outputTokenLimit is 32,768 (saved: probes/models-get-gemini-4-argon.json), while the model page lists 65,536 as "default synchronous". litellm sends one non-streaming REST request, so the card follows the guide; 32,768 is safe either way and never binds. Requests asking for up to 131,072 returned 200, which shows only that the parameter is accepted.

  • Smoke. Two scenarios on the run path completed 2/2 with no missing prediction or explanation for $0.05.

Invariants

The price override's cost reconstruction (_reconstruct_token_cost) has a Hypothesis property test: the cost is uncached input at $2, cached input at $0.10 and output at $10 per 1M; it is never negative; and it lies between the all-cached and the uncached cost of the same call.

Evidence

~/reviews/policybench-gemini-4-argon/: probes/ (the one-question probe and its 50 raw responses, the harness-path probe and its 87 raw responses with the body litellm put on the wire, the output-limit probe, the decision rule and its outcome), onboard/ (the gauntlet report), smoke/ (the run) and board-run/ (the forced and auto runs, the comparison script and its output).

Plan

  • Roster entry, price and release date
  • Reasoning probes, card, gauntlet and two-scenario smoke
  • Board run and the tool_choice auto comparison run; the card's notes record the result (5910f79)
  • Independent review (GPT-6.1 Sol): round 1 REQUEST_CHANGES (P2 ceiling 65,536 → 32,768; P3 "does not suppress" wording), fixed in e4b4d05 and d1e7685 with a board re-run; round 2 REQUEST_CHANGES (P2 Google's pages disagree, unsaved Models API record; two P3s), fixed in 0109ea2; round 3 APPROVE at 0109ea2
  • CI green, then merge

🤖 Generated with Claude Code

… accepts

Google opened Gemini 4 Argon to the public on 2026-10-13 (the Gemini API
changelog lists gemini-4-argon as generally available that day; the
2026-09-30 launch was limited to Fairwind Program cyber partners). It runs
on the first-party Gemini API as gemini/gemini-4-argon, like the other
Gemini rows.

Price: $2 / $10 per 1M input/output tokens and $0.10 cached input, the
rates Google's pricing page lists through 2026-11-30 (they double on
2026-12-01; page last updated and read 2026-10-13).

Card: the Gemini API accepts a forced function call on this model, and the
gauntlet, run with no card, passed on the tool contract (3/3 in 5s with 442
completion tokens, 16/16 whole-scenario in 24s with 2,619), so the row
takes the board's forced answer tool, as Gemini 3.6, 3.7 and 3.8 Flash do.

Reasoning: the harness sends no thinkingConfig, and the model decides call
by call whether to think. On the board's own request for 13 frozen
households (29 calls a shape) it returned reasoning tokens on 20 forced
calls, 25 with tool_choice auto and 22 on the JSON contract; all 87 parsed
every output. Forced against auto is within sampling error (Fisher exact
p = 0.21), so the 100-household runs decide whether a tool_choice auto
sensitivity run is published (the rule is fixed in the onboarding record).

A two-scenario smoke on the run path completed 2/2 with no missing
prediction or explanation for $0.05.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policybench-site Ready Ready Preview Oct 13, 2026 7:21pm UTC

Request Review

The 100-household runs decide the question the card left open. The
forced-tool board run returned reasoning tokens on 80 households and a
tool_choice auto run on 82 (Fisher exact p = 0.86; 70 reasoned in both,
10 only when forced, 12 only under auto, 8 in neither). Under the rule
written before the runs, the row gets no tool_choice auto sensitivity run;
the auto run stays in the onboarding record as evidence.

The notes also record the board run: 100/100 households in one request
each, 1,984/1,984 answers parsed, no budget escalation, $2.77.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…est returns (card review P2, P3)

Google's Gemini 4 Argon guide: "One non-streaming request returns up to
32,768 tokens" (read 2026-10-13); the Models API's outputTokenLimit agrees.
The model page's 65,536 is the streaming and SDK figure, reached by
continuing across requests, and litellm sends one non-streaming REST
request. So provider_max_completion_tokens is 32,768, not 65,536. The
output-limit probe's 200s at up to 131,072 showed only that the parameter
is accepted.

The first board run recorded the 65,536 ceiling in its treatment
fingerprint, which the freeze checks against the registry, so the board row
is a second run on this card. The first run never escalated (largest
household 6,374 completion tokens), and it stays in the record, with the
tool_choice auto run, as the reasoning comparison's evidence.

The reasoning sentence now says what the comparison shows: a forced tool
does not switch reasoning off, and no statistically significant reduction
in how often the model reasons was detected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second board run, on the 32,768-token ceiling (e4b4d05): all 100
households in one request each, 1,984/1,984 answers parsed, no budget
escalation, $2.79, reasoning tokens on 85 households. None of its responses
replays the first run's (no shared provider response id).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ved record (card review r2)

P2: the model page lists 65,536 as "default synchronous"; Google's
generateContent guide gives 32,768 for one non-streaming request and the
Models API's outputTokenLimit is 32,768, now saved in the onboarding record
(probes/models-get-gemini-4-argon.json). The card follows the guide because
litellm makes one non-streaming REST request; 32,768 is safe either way.

P3: the timeout comment gives both board runs' slowest households (53s,
47s), and the notes add that the second board run reasoned on 85 households.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 13, 2026 20:23
@MaxGhenis
MaxGhenis merged commit 8ab332d into main Oct 13, 2026
6 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged at reviewed head 0109ea2 (squash 8ab332d). Gates: all checks pass on that head; mergeable, clean; no changes requested. Independent review (GPT-6.1 Sol): round 1 REQUEST_CHANGES (ceiling, wording) fixed in e4b4d05/d1e7685b with a board re-run; round 2 REQUEST_CHANGES (Google's pages disagree on the limit; unsaved Models API record) fixed in 0109ea2; round 3 APPROVE at 0109ea2. Reviews: ~/reviews/policybench-gemini-4-argon/card-review/. The board run joins the published board in release PR #221.

@MaxGhenis
MaxGhenis deleted the add-gemini-4-argon branch October 13, 2026 20:23
MaxGhenis added a commit that referenced this pull request Nov 15, 2026
#266)

We add model cards, list prices and release dates for Google's Gemini 4 Carbon and Gemini 4 Hydrogen, on the pattern of #219 (Gemini 4 Argon's card). We change no published artifact. Anyone who runs `eval-no-tools` with no `--model` now sends requests to both models, along with every other model in `config.MODELS`. We will add both models' scores to the board in the next data release.

## Facts (read 2026-11-13 and 2026-11-14)

- **Release.** Google opened Gemini 4 Carbon to the public on 2026-11-10; the [Gemini API changelog](https://aistudio.google.com/docs/changelog) lists `gemini-4-carbon` as generally available that day. [Google's post](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-4-carbon-now-available) of that day says Google had released the model two weeks earlier to cybersecurity defenders through its Fairwind Program. The board dates each model from its public day, as it dates Argon. The changelog lists `gemini-4-hydrogen` as generally available on 2026-11-12, on the free and the paid tier.
- **Route.** The board calls both models on the first-party Gemini API (`gemini/gemini-4-carbon`, `gemini/gemini-4-hydrogen`), as it does every other Gemini row. Each model page lists one stable version and no dated snapshot. Every saved response names the requested model.
- **Price.** For Carbon, Google's [pricing page](https://aistudio.google.com/docs/pricing) lists $2 / $10 per 1M input/output tokens (output includes thinking tokens) and $0.10 cached input "through November 30, 2026", and $4 / $20 / $0.20 from December 1. The override prices every request at the first rate, whatever day the board sends it, so it matches Google's price only through 2026-11-30. The harness prices a batch result when it collects the result, and it prices a response that the disk cache replays at the time of the replay. For a model whose rows record no cost, `analysis.model_cost_latency` computes one from the override as it stands when the analysis runs. Once we change the override, each of these would put the December rate on a request made in November. For Hydrogen the page lists $1.50 / $7.50 and $0.075 cached input, with no dated change. It also lists a free tier; the board prices the row at the paid rates.
- **Ceiling.** The cards cap completion at 32,768 tokens for Carbon and 65,536 for Hydrogen, each model's `outputTokenLimit` in the Models API. Google's [generateContent long-decode guide](https://aistudio.google.com/docs/long-decode?generate-content=true) gives the same caps for a direct non-streaming REST call, which is what the harness sends.

## The cards

- **Carbon answers as JSON.** The gauntlet, run with no card, sent the board's forced answer tool first. The Gemini API accepted the forced call, but on the 3-variable probe the model reasoned until the 16,384-token budget ran out: 16,381 reasoning tokens, 0 of 3 outputs parsed, 203s. The gauntlet sends each probe once and does not escalate the budget. The tool request therefore ran at 16,384 tokens, half the minimum that Google's [Gemini 4 Carbon guide](https://aistudio.google.com/docs/whats-new-gemini-4-carbon) recommends for this model. The gauntlet then passed on the JSON contract: 3/3 (9,964 completion tokens, 105s) and 16/16 whole-scenario (7,257 tokens, 79s), about $0.075 per scenario. Before the runs, we set the rule that a card takes the JSON contract when its tool probe fails. We did not re-test the tool contract, since re-running the gauntlet until the tool probe passes would choose the contract on a retry.
- **Hydrogen takes the forced answer tool.** The gauntlet, run with no card, passed on the tool contract: 3/3 (10,956 completion tokens, 48s) and 16/16 whole-scenario (11,947 tokens, 51s), about $0.094 per scenario.
- **Reasoning.** The harness sends no `thinkingConfig`. Google's [thinking guide](https://aistudio.google.com/docs/generate-content/thinking) gives Carbon a default thinking level of medium and Hydrogen high. Both models returned thought tokens on every call whose usage we saved, in every request shape:

  | Request shape | Carbon, one question (10 calls each) | Hydrogen, one question (10 calls each) | Hydrogen, the board's request for 16 frozen households (16 calls each) |
  |---|---|---|---|
  | Forced tool | 10 with thought tokens | 10 | 16 with reasoning tokens |
  | `tool_choice` auto | 10 | 10 | 16 |
  | Mode ANY, no named function | 10 | 10 | not run |
  | JSON response type | 10 | 10 | 16 |
  | No tools | 10 | 10 | not run |

- **No sensitivity run.** Before the first onboarding call to each model, we wrote down Argon's rule for it: we add a `tool_choice` auto sensitivity run only if forcing the tool suppresses reasoning. On the board's own request, Hydrogen returned reasoning tokens on 16 of 16 forced calls and 16 of 16 auto calls (Fisher exact p = 1.0), so its row has no sensitivity run. Carbon's request declares no tool, so the rule has nothing to compare.
- **Budget.** The harness gives both rows the Gemini family's 16,384-token starting budget. Our probe of Hydrogen on the board's own request sent each request once, and 6 of its 16 forced calls ran out of that budget. On such a call the harness doubles the budget and asks again, up to 65,536 for Hydrogen. Carbon gets one doubling, to 32,768. If a length-truncated response at that ceiling leaves outputs missing, the harness records those outputs as `budget_exhausted_at_ceiling`.
- **Timeout.** Carbon's card sets a 900s timeout, since Google's Carbon guide recommends client timeouts of at least 15 minutes for this model. At the gauntlet's 80 to 95 completion tokens a second, a retry at the 32,768-token ceiling takes about 345 to 410s. Hydrogen's card sets 600s. At the gauntlet's 228 to 234 tokens a second, a request at the 65,536-token ceiling takes about 280 to 290s, too close to the thinking class's 300s. Google's long-decode guide says the server closes a non-streaming connection after 10 minutes. The same guide recommends a 20-minute client read timeout, and its option table gives 20 minutes per request. If the server closes at 10 minutes, no request on either row runs past 600s. Carbon's one-question probe ran as slow as 53 tokens a second, and at that rate a request at the ceiling takes about 620s and would not finish. If the table's 20 minutes applies, Carbon's 900s still exceeds that 620s and the 345 to 410s the gauntlet's rates predict for a request at its ceiling.
- **Smoke.** A two-household smoke through `eval-no-tools` finished both households for each model. In each run all 36 outputs came back with a prediction and an explanation, and every row carries reasoning tokens and a cost. Neither run escalated a budget. The runs cost $0.18 for Carbon and $0.23 for Hydrogen. At about $0.090 and $0.113 a household, they exceed the gauntlet figures the cards carry ($0.075 and $0.094).

## Tests

- `tests/test_model_cards.py` holds each card's expected row. It reconstructs each model's whole-scenario gauntlet cost from its token counts at the price override and checks the card's expected cost against it.
- `tests/test_eval_no_tools.py` checks that Carbon's request asks for a JSON object and declares no tool, that Hydrogen's forces the answer tool, and that neither sets a reasoning parameter. It also holds a cost-reconstruction row for each model's whole-scenario gauntlet probe.
- `tests/test_paper_results.py` checks both release dates.

Locally, eleven test files that read the cards, the config or the release dates passed at this head (1,634 passed, 13 skipped): `test_model_cards`, `test_eval_no_tools`, `test_paper_results`, `test_disclosures`, `test_onboard`, `test_supervisor`, `test_notes`, `test_freeze_argon`, `test_freeze_musewonder`, `test_snapshot_artifacts` and `test_cache`. CI runs the suite with slow tests excluded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 0109ea2f Deployed Oct 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant