Add GPT-6 Astra extract one-shot pipeline - #37
Merged
Conversation
Register openai_gpt_6_astra_reasoning_low_extract_oneshot_structured_output_file with gpt-6-astra pricing, wire reasoning_effort through the OpenAI Responses provider (sent only when configured), and add its leaderboard row.
boyang-zhang1
force-pushed
the
add-gpt-6-astra-extract
branch
from
September 5, 2026 02:22
db8adf5 to
ebe7151
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the GPT-6 Astra one-shot structured-output extract pipeline and its leaderboard entry.
Why
Benchmark GPT-6 Astra on ExtractBench alongside the existing one-shot LLM/VLM extractors.
How
openai_gpt_6_astra_reasoning_low_extract_oneshot_structured_output_file(provideropenai_extract,reasoning_effort: low), matching the existing one-shot sibling config shape.reasoning_effortthrough the OpenAI Responses provider — forwarded to the API asreasoning.effortonly when set, so non-reasoning pipelines are unaffected.gpt-6-astrapricing ($10/M input, $50/M output) to the provider pricing table.Changes
src/extract_bench/inference/pipelines/extract.py— new pipeline specsrc/extract_bench/inference/providers/extract/openai_responses.py— pricing entry + reasoning wiringdocs/pipelines.md— pipeline rowleaderboard.csv— new row (rank 3)README.md— regenerated from the leaderboardTesting
uv run extract-bench pipelines | grep gpt_6_astraresolves the new pipelineuv run ruff checkpasses on the changed filesuv run python scripts/update_readme.py— diff confined to the generated table blocksNotes
Requires
OPENAI_API_KEY(no new env var). Grounding columns are 0.00 — one-shot structured output emits no citation boxes.