Skip to content

Add GPT-6 Astra extract one-shot pipeline - #37

Merged
boyang-zhang1 merged 1 commit into
mainfrom
add-gpt-6-astra-extract
Sep 5, 2026
Merged

Add GPT-6 Astra extract one-shot pipeline#37
boyang-zhang1 merged 1 commit into
mainfrom
add-gpt-6-astra-extract

Conversation

@boyang-zhang1

Copy link
Copy Markdown
Member

What

Adds the GPT-6 Astra one-shot structured-output extract pipeline and its leaderboard entry.

Why

Benchmark GPT-6 Astra on ExtractBench alongside the existing one-shot LLM/VLM extractors.

How

  • Register openai_gpt_6_astra_reasoning_low_extract_oneshot_structured_output_file (provider openai_extract, reasoning_effort: low), matching the existing one-shot sibling config shape.
  • Wire reasoning_effort through the OpenAI Responses provider — forwarded to the API as reasoning.effort only when set, so non-reasoning pipelines are unaffected.
  • Add gpt-6-astra pricing ($10/M input, $50/M output) to the provider pricing table.
  • Add the leaderboard row (Overall 91.91; Short 97.22 / Medium 90.56 / Long 31.70) and regenerate the README tables.

Changes

  • src/extract_bench/inference/pipelines/extract.py — new pipeline spec
  • src/extract_bench/inference/providers/extract/openai_responses.py — pricing entry + reasoning wiring
  • docs/pipelines.md — pipeline row
  • leaderboard.csv — new row (rank 3)
  • README.md — regenerated from the leaderboard

Testing

  • uv run extract-bench pipelines | grep gpt_6_astra resolves the new pipeline
  • uv run ruff check passes on the changed files
  • uv run python scripts/update_readme.py — diff confined to the generated table blocks

Notes

Requires OPENAI_API_KEY (no new env var). Grounding columns are 0.00 — one-shot structured output emits no citation boxes.

Register openai_gpt_6_astra_reasoning_low_extract_oneshot_structured_output_file
with gpt-6-astra pricing, wire reasoning_effort through the OpenAI Responses
provider (sent only when configured), and add its leaderboard row.
@boyang-zhang1
boyang-zhang1 force-pushed the add-gpt-6-astra-extract branch from db8adf5 to ebe7151 Compare September 5, 2026 02:22
@boyang-zhang1
boyang-zhang1 merged commit 94ceac1 into main Sep 5, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant