Skip to content

Spec forge: draft with Claude, OpenAI or Gemini - #164

Merged
sykosyber merged 1 commit into
mainfrom
wp/omni/forge-providers
Oct 13, 2026
Merged

sykosyber merged 1 commit into
mainfrom
wp/omni/forge-providers

Conversation

@sykosyber

Copy link
Copy Markdown
Member

Objective. Let the spec forge (APEX_PLAN 6.6) use the owner's OpenAI or Google credits, not only Claude.

What changed

  • scripts/lib/drafters.ts: a DraftSession per provider that keeps its own native conversation for repair rounds. Claude uses the existing SDK call unchanged: claude-opus-5-5, adaptive thinking, high effort, server-side fallback, refusal and max_tokens checks. OpenAI uses Chat Completions over fetch (default gpt-5). Gemini uses generateContent over fetch (default gemini-2.5-pro, responseMimeType: application/json, thought parts dropped).
    • chooseProvider: --provider wins; otherwise the first key set (Anthropic, OpenAI, Google); with none, Claude, which can also use an ant auth login profile.
    • Keys travel only in the request header (Authorization: Bearer, x-goog-api-key), never the URL or body. An HTTP error reports the status and the provider's error.message, never the body. A Gemini --model must be a plain name, so it cannot change the URL path.
    • Refusals (OpenAI refusal/content_filter, Gemini blockReason/SAFETY) are DraftRefused. A cut-off draft is an error.
  • scripts/forge-spec.ts: uses the session. The report names the provider and model. --provider and --model flags.
  • APEX_PLAN.md 6.6 status.

What did not change. checkDraft, the live verification (verifySpec), and the rule that a person reviews a draft before it is curated. No new dependency. The app's own LLM providers are untouched; OpenAI stays out of the in-app persona providers (owner decision), and this is a developer tool only.

Tests run

  • npm run typecheck 0 errors; npm run lint 0 errors (8 existing warnings).
  • npm test: 1652 passed, 1 failed (fieldHarness.test.ts, known Windows-only CRLF).
  • New specDraftProviders.test.ts (8, stubs): provider choice; reply parsing for both providers (text, refusal, cut-off, Gemini thought parts); OpenAI request URL, bearer header, conversation across rounds, key absent from the body; HTTP error message; missing key; Gemini key in the header not the URL, JSON mode; model-name guard.
  • npm run build + npm run scan:bundle PASS; npm run build:next + npm run test:e2e 9 passed.

Measurements. One real run, 2026-10-12: npm run forge:spec -- --slug openlibrary-gemini --docs https://openlibrary.org/dev/docs/api/search --provider google. Gemini 2.5 Pro read 1 page (9,262 chars) and produced GET /search.json and GET /search/authors.json, with every parameter taken from the page. It passed the mechanical checks in round 1. Both operations answered live, matching their declared output schema. The draft was not committed; drafts are for a person to review.

Known failures. fieldHarness.test.ts on Windows only.

Claim ceiling. Gemini: measured once. Claude and OpenAI: tested with stubs (no key on this machine).

Architectural decisions introduced. The forge's model is pluggable behind DraftSession. The checks, not the model, decide a draft.

Evidence that could falsify it. An OpenAI run that fails on the request shape (for example, a model that rejects max_completion_tokens). Any key appearing in a URL, a log line or a report.

Next permissible package. The FRED notice and terms link (FRED's terms require it before a house key serves other people).

🤖 Generated with Claude Code

`npm run forge:spec -- ... --provider anthropic|openai|google [--model NAME]`.
Without --provider it uses the first key set: ANTHROPIC_API_KEY, then
OPENAI_API_KEY, then GOOGLE_API_KEY. Each provider is a session that keeps
its own conversation, so repair rounds continue it (scripts/lib/drafters.ts).
OpenAI and Gemini go over fetch, so the dev tool adds no dependency; keys go
only in the request header, and errors repeat the provider's message, never
a response body. The checks and live calls that decide a draft are unchanged.

Measured once with Gemini 2.5 Pro on Open Library's search docs: passed the
checks in round 1, and both operations answered live.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sykosyber
sykosyber enabled auto-merge October 13, 2026 03:37
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-13T03:42:47.249177Z 4a679fa PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sykosyber
sykosyber merged commit 51549f7 into main Oct 13, 2026
4 checks passed
@sykosyber
sykosyber deleted the wp/omni/forge-providers branch October 13, 2026 03:40

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4a679fa64d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/lib/drafters.ts
let message = '';
try {
const body = await response.json() as { error?: { message?: unknown } };
if (typeof body.error?.message === 'string') message = `: ${body.error.message.slice(0, 200)}`;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Prevent provider errors from logging API keys

When OpenAI rejects an invalid or revoked key, its documented 401 message includes the submitted (masked) key; this helper copies up to 200 characters of that message into the thrown error, which forge-spec.ts then writes to stderr. Running the forge with a bad OPENAI_API_KEY therefore puts a value derived from that secret into CI or shell logs; report only the HTTP status (or scrub against the configured key) instead. OpenAI’s documented incorrect-key response

AGENTS.md reference: AGENTS.md:L102-L103

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant