Skip to content

Add Gemini 3.8 Flash extract one-shot pipeline - #36

Merged
boyang-zhang1 merged 1 commit into
mainfrom
add-gemini-3-8-flash-extract
Sep 4, 2026
Merged

Add Gemini 3.8 Flash extract one-shot pipeline#36
boyang-zhang1 merged 1 commit into
mainfrom
add-gemini-3-8-flash-extract

Conversation

@boyang-zhang1

Copy link
Copy Markdown
Member

What

Add Gemini 3.8 Flash as a one-shot structured-output extract pipeline.

Why

Gemini 3.8 Flash is generally available (2026-09-02) at the same introductory price as 3.7 Flash, so the leaderboard should track the latest Flash tier.

How

  • Register gemini_3_8_flash_extract_oneshot_structured_output_file via the gemini_extract provider (one-shot structured output over the uploaded file, thinking_level: low).
  • Add gemini-3.8-flash to the Gemini extract pricing table ($0.75 input / $3.75 output per 1M tokens).

Changes

  • src/extract_bench/inference/pipelines/extract.py — new pipeline spec
  • src/extract_bench/inference/providers/extract/gemini_direct.py — pricing entry
  • docs/pipelines.md — pipeline table row
  • leaderboard.csv / README.md — leaderboard row and regenerated tables

Testing

  • uv run extract-bench pipelines | grep gemini_3_8_flash_extract_oneshot_structured_output_file — registers.
  • uv run ruff check on the changed files — passes.
  • Benchmarked on the full 370-document set: Overall 80.71 (Short 88.14 / Medium 72.23 / Long 28.70). Word and page grounding are 0.00 — a values-only one-shot returns no citation boxes, same as the existing 3.5/3.6/3.7 one-shots.

Notes

Uses the existing Gemini API key env vars (GOOGLE_GEMINI_API_KEY / GEMINI_API_KEY / GOOGLE_API_KEY); no new configuration.

Register gemini_3_8_flash_extract_oneshot_structured_output_file (one-shot structured output, low thinking) via the gemini_extract provider, add its gemini-3.8-flash pricing entry, document it, and add the leaderboard row.
@boyang-zhang1
boyang-zhang1 merged commit 0880af2 into main Sep 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant