Repository navigation
Spec forge: draft with Claude, OpenAI or Gemini - #164
Conversation
`npm run forge:spec -- ... --provider anthropic|openai|google [--model NAME]`. Without --provider it uses the first key set: ANTHROPIC_API_KEY, then OPENAI_API_KEY, then GOOGLE_API_KEY. Each provider is a session that keeps its own conversation, so repair rounds continue it (scripts/lib/drafters.ts). OpenAI and Gemini go over fetch, so the dev tool adds no dependency; keys go only in the request header, and errors repeat the provider's message, never a response body. The checks and live calls that decide a draft are unchanged. Measured once with Gemini 2.5 Pro on Open Library's search docs: passed the checks in round 1, and both operations answered live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4a679fa64d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let message = ''; | ||
| try { | ||
| const body = await response.json() as { error?: { message?: unknown } }; | ||
| if (typeof body.error?.message === 'string') message = `: ${body.error.message.slice(0, 200)}`; |
There was a problem hiding this comment.
Prevent provider errors from logging API keys
When OpenAI rejects an invalid or revoked key, its documented 401 message includes the submitted (masked) key; this helper copies up to 200 characters of that message into the thrown error, which forge-spec.ts then writes to stderr. Running the forge with a bad OPENAI_API_KEY therefore puts a value derived from that secret into CI or shell logs; report only the HTTP status (or scrub against the configured key) instead. OpenAI’s documented incorrect-key response
AGENTS.md reference: AGENTS.md:L102-L103
Useful? React with 👍 / 👎.
Objective. Let the spec forge (APEX_PLAN 6.6) use the owner's OpenAI or Google credits, not only Claude.
What changed
scripts/lib/drafters.ts: aDraftSessionper provider that keeps its own native conversation for repair rounds. Claude uses the existing SDK call unchanged:claude-opus-5-5, adaptive thinking, high effort, server-side fallback, refusal and max_tokens checks. OpenAI uses Chat Completions overfetch(defaultgpt-5). Gemini usesgenerateContentoverfetch(defaultgemini-2.5-pro,responseMimeType: application/json, thought parts dropped).chooseProvider:--providerwins; otherwise the first key set (Anthropic, OpenAI, Google); with none, Claude, which can also use anant auth loginprofile.Authorization: Bearer,x-goog-api-key), never the URL or body. An HTTP error reports the status and the provider'serror.message, never the body. A Gemini--modelmust be a plain name, so it cannot change the URL path.refusal/content_filter, GeminiblockReason/SAFETY) areDraftRefused. A cut-off draft is an error.scripts/forge-spec.ts: uses the session. The report names the provider and model.--providerand--modelflags.APEX_PLAN.md6.6 status.What did not change.
checkDraft, the live verification (verifySpec), and the rule that a person reviews a draft before it is curated. No new dependency. The app's own LLM providers are untouched; OpenAI stays out of the in-app persona providers (owner decision), and this is a developer tool only.Tests run
npm run typecheck0 errors;npm run lint0 errors (8 existing warnings).npm test: 1652 passed, 1 failed (fieldHarness.test.ts, known Windows-only CRLF).specDraftProviders.test.ts(8, stubs): provider choice; reply parsing for both providers (text, refusal, cut-off, Gemini thought parts); OpenAI request URL, bearer header, conversation across rounds, key absent from the body; HTTP error message; missing key; Gemini key in the header not the URL, JSON mode; model-name guard.npm run build+npm run scan:bundlePASS;npm run build:next+npm run test:e2e9 passed.Measurements. One real run, 2026-10-12:
npm run forge:spec -- --slug openlibrary-gemini --docs https://openlibrary.org/dev/docs/api/search --provider google. Gemini 2.5 Pro read 1 page (9,262 chars) and producedGET /search.jsonandGET /search/authors.json, with every parameter taken from the page. It passed the mechanical checks in round 1. Both operations answered live, matching their declared output schema. The draft was not committed; drafts are for a person to review.Known failures.
fieldHarness.test.tson Windows only.Claim ceiling. Gemini: measured once. Claude and OpenAI: tested with stubs (no key on this machine).
Architectural decisions introduced. The forge's model is pluggable behind
DraftSession. The checks, not the model, decide a draft.Evidence that could falsify it. An OpenAI run that fails on the request shape (for example, a model that rejects
max_completion_tokens). Any key appearing in a URL, a log line or a report.Next permissible package. The FRED notice and terms link (FRED's terms require it before a house key serves other people).
🤖 Generated with Claude Code