diff --git a/SCIENT.md b/SCIENT.md index d6efecec41..5cf8178fb5 100644 --- a/SCIENT.md +++ b/SCIENT.md @@ -23,10 +23,12 @@ Everything else is upstream's code and behavior. | Shell traces | The bash tool records no execution trace unless the user turns `display.shellVisualization` on. Scient Agent's host draws no shell visualization, and every live frame of a running command would carry the whole trace, up to megabytes. | | Browser relay | The relay listens on port 9324 (OMP uses 9224), so each product runs its own. | | Next message | `predict_next_message` gives an RPC host the prediction of the user's next message that OMP's terminal composer shows after a turn: OMP's prompt and parsing, on the session's model and context, without changing its history. It runs outside the command queue; a newer request replaces one in flight, and a turn or compaction that starts cancels it. | -| Usage-limit reset | A `prompt_result` for a turn stopped by a usage limit carries `error.resetAt`, the time the provider said the limit resets (epoch ms): from its retry hint, the account's usage report, or an earlier stated block, whichever is latest. It is left out when the provider gave no time; the 30-minute guess for a hintless error is never reported. A host can resume the turn at that time. | +| Usage-limit reset | A `prompt_result` for a turn stopped by a usage limit carries `error.resetAt`, the time the provider said the limit resets (epoch ms): from its retry hint, the account's usage report, or an earlier stated block, whichever is latest. A time the error states is read from "resets at" as well as Oh My Pi's "reset at", by the agent's own retry too. It is left out when the provider gave no time; the 30-minute guess for a hintless error is never reported. A host can resume the turn at that time. A usage-limited turn whose failure came without an HTTP status reports `error.httpStatus` 429, so a host that shows a usage-limit state for a 429 shows it; a status the provider sent is kept. | | Prompt cache | An assistant reply's `message_end` carries `promptCache` (`tier`, `touchedAt`, `ttlMs`): the prompt-cache entry its response confirmed, while the next request would still read it. It is left out for a cache that never expires and for an entry the agent only inferred. A host can compact an idle conversation just before a one-hour entry expires, so the summary reads a warm cache. The agent's own idle compaction stays a terminal feature and never runs over RPC. | | Plan mode | Oh My Pi plans only in its terminal and for ACP clients. Over RPC, the `plan` command (`enter`, `exit`, `get`) turns plan mode on: the agent reads and drafts a plan file but changes nothing else. When it proposes the plan, a `plan_proposed` frame carries the plan, and a select request asks the host to approve it while the run waits. Only an explicit approval leaves plan mode, and the same run then carries out the plan. Refine asks the host what to change, and that answer, like any typed answer, reaches the model as feedback; planning goes on. An answer that comes after plan mode ended never approves, and a new, switched or forked conversation starts outside plan mode. Each time plan mode turns on or off, a `plan_updated` frame tells the host. Planning is refused while a goal is set. | -| Failure kinds | A `prompt_result` for a failed turn carries `error.kinds`, the error classifier's labels for that failure (`auth-failed`, `context-overflow`, `transient`, `usage-limit`, ...), so a host can report why turns fail. It is left out when the classifier recognized no kind; the labels carry no provider text. | +| Failure kinds | A `prompt_result` for a failed turn carries `error.kinds`, the error classifier's labels for that failure (`auth-failed`, `context-overflow`, `transient`, `usage-limit`, ...), so a host can report why turns fail. A request that never reached the provider (no address, a refused connection, no network, a certificate that did not verify) also carries `network`, which the classifier's flags do not tell apart from a provider's outage. It is left out when the classifier recognized no kind; the labels carry no provider text. | +| Access policy | A provider that serves a model only to the apps it lists (OpenRouter: "… is only available on agentic harnesses") refuses other clients with a 403. Oh My Pi reads a bare 403 as a failed sign-in, so the person was sent to fix a key that works. The refusal is an account policy: the agent tries the model's fallback chain, and the turn's `error.kinds` carries `account-policy`. | +| Google tool enums | Cloud Code Assist takes a Claude model's tool schemas in its legacy `parameters` field, whose enums must be strings: one numeric enum (`[-1, 1]`) in any tool made Google refuse every request of the session with a 400. As on the Gemini path, an enum with a value that is not a string is left out and the field keeps its type. | | macOS Accessibility prompt | The agent sends no Accessibility message while macOS doesn't trust it, so a computer call that refuses for lack of a permission never makes macOS ask. Only the host asks for permissions. | | Live voice delegation | `live_connect` with `hostDelegation` sends each request the voice model delegates to the host as a `live_delegation` frame instead of starting a turn, so a host that owns its turns (Scient Desktop) sends it as the user's message. The live session still reads that turn's answer. A host that runs the turn elsewhere, such as in the conversation's own agent while the call runs in a separate one, reports its progress and answer with `live_report`. Without the flag nothing changes. | | Dictation | `scient-agent dictate ` transcribes a recording with ChatGPT's dictation service, the one the ChatGPT and Codex apps use, through the stored ChatGPT sign-in. With `--json` it prints `{ "text" }`, or `{ "error": { "kind", "message" } }` and exits 1, so a host (Scient Desktop's microphone) can show its own message. ChatGPT keeps the uploaded recording as a file for 30 days, as it does for its own apps' dictation. | diff --git a/packages/ai/src/error/flags.ts b/packages/ai/src/error/flags.ts index bacf7d69b4..011f777c7c 100644 --- a/packages/ai/src/error/flags.ts +++ b/packages/ai/src/error/flags.ts @@ -17,6 +17,9 @@ import { parseJsonBody, parseRateLimitReason, } from "./rate-limit"; +// scient(SA-ACCESS-POLICY): begin - A model a provider serves only to its listed apps is an access policy, not a bad key. +import { isAppRestrictedModelText } from "../scient/access-policy"; +// scient(SA-ACCESS-POLICY): end export const Flag = { Class: 0x1000, @@ -672,6 +675,9 @@ function classifyText( if (isAnthropicAccountPolicyText(errorMessage, provider, statusClean)) { kinds |= Flag.AccountPolicy; } + // scient(SA-ACCESS-POLICY): begin - A model a provider serves only to its listed apps is an access policy, not a bad key. + if (isAppRestrictedModelText(errorMessage)) kinds |= Flag.AccountPolicy; + // scient(SA-ACCESS-POLICY): end if (isAuthFailureText(errorMessage)) kinds |= Flag.AuthFailed; const cleanMessage = errorMessage; const isOpaque = isOpaqueStatusBody(cleanMessage); diff --git a/packages/ai/src/scient/access-policy.ts b/packages/ai/src/scient/access-policy.ts new file mode 100644 index 0000000000..63ace7394c --- /dev/null +++ b/packages/ai/src/scient/access-policy.ts @@ -0,0 +1,17 @@ +/** + * Provider refusals that name an access policy, not a bad credential. + * + * OpenRouter serves some free models only to the apps it lists, and refuses any other client with a + * 403: "thinkingmachines/inkling-2-small:free is only available on agentic harnesses. Try plugging + * it into a coding agent or productivity app listed on https://openrouter.ai/apps". The key is + * valid. Oh My Pi reads a bare 403 as a failed sign-in, so the person was told to fix a key that + * works. The classifier marks the refusal an account policy instead, as it does Anthropic's + * organization denials. + */ + +const HARNESS_ONLY_MODEL_PATTERN = /\bis only available on agentic harnesses\b/i; + +/** Whether `text` is a provider's refusal to serve a model outside the apps it lists. */ +export function isAppRestrictedModelText(text: string): boolean { + return HARNESS_ONLY_MODEL_PATTERN.test(text); +} diff --git a/packages/ai/src/utils/schema/normalize.ts b/packages/ai/src/utils/schema/normalize.ts index aafb05c6ec..9cfaed59dc 100644 --- a/packages/ai/src/utils/schema/normalize.ts +++ b/packages/ai/src/utils/schema/normalize.ts @@ -1336,6 +1336,9 @@ export function normalizeSchemaForCCA(value: unknown): unknown { extractNullableFromUnions: true, inferTypeForBareEnum: true, dropNonScalarEnum: false, + // scient(SA-CCA-STRING-ENUMS): begin - Cloud Code Assist's `parameters` take only string enums, as Gemini's do. + stringEnumsOnly: true, + // scient(SA-CCA-STRING-ENUMS): end foldOneOfIntoAnyOf: false, rejectResidualIncompatibilities: ["type-array", "type-null", "nullable", "combiners", "not"], validateAndFallback: { fallback: CLOUD_CODE_ASSIST_CLAUDE_FALLBACK_SCHEMA }, diff --git a/packages/ai/test/scient/access-policy.test.ts b/packages/ai/test/scient/access-policy.test.ts new file mode 100644 index 0000000000..31b81bc41c --- /dev/null +++ b/packages/ai/test/scient/access-policy.test.ts @@ -0,0 +1,40 @@ +import { afterAll, beforeAll, describe, expect, it } from "bun:test"; +import { getBundledModels } from "@oh-my-pi/pi-catalog/models"; +import * as AIError from "@oh-my-pi/pi-ai/error"; +import { complete } from "@oh-my-pi/pi-ai/stream"; + +const APP_RESTRICTED = + "thinkingmachines/inkling-2-small:free is only available on agentic harnesses. Try plugging it into a coding agent or productivity app listed on https://openrouter.ai/apps"; + +describe("a model a provider serves only to its listed apps", () => { + let server: ReturnType; + + beforeAll(() => { + server = Bun.serve({ + port: 0, + fetch: () => Response.json({ error: { message: APP_RESTRICTED, code: 403 } }, { status: 403 }), + }); + }); + + afterAll(() => server.stop(true)); + + it("is an account policy, which a host ranks before a failed sign-in", async () => { + const model = getBundledModels("openrouter")[0]; + if (!model) throw new Error("Expected a bundled OpenRouter model"); + const failed = await complete( + { ...model, baseUrl: `http://localhost:${server.port}/v1` }, + { systemPrompt: ["Test"], messages: [{ role: "user", content: "hi", timestamp: Date.now() }] }, + { apiKey: "openrouter-test-key" }, + ); + + expect(failed.stopReason).toBe("error"); + expect(failed.errorStatus).toBe(403); + expect(AIError.is(AIError.classifyMessage(failed), AIError.Flag.AccountPolicy)).toBe(true); + }); + + it("leaves another 403 a failed sign-in", () => { + const id = AIError.classifyMessage({ provider: "openrouter", errorMessage: "403 Forbidden", errorStatus: 403 }); + expect(AIError.is(id, AIError.Flag.AuthFailed)).toBe(true); + expect(AIError.is(id, AIError.Flag.AccountPolicy)).toBe(false); + }); +}); diff --git a/packages/ai/test/scient/cca-string-enums.test.ts b/packages/ai/test/scient/cca-string-enums.test.ts new file mode 100644 index 0000000000..082caa68d6 --- /dev/null +++ b/packages/ai/test/scient/cca-string-enums.test.ts @@ -0,0 +1,63 @@ +import { describe, expect, it } from "bun:test"; +import { convertTools } from "@oh-my-pi/pi-ai/providers/google-shared"; +import type { Model, TJsonSchema, Tool } from "@oh-my-pi/pi-ai/types"; +import { buildModel } from "@oh-my-pi/pi-catalog/build"; + +function antigravityModel(id: string): Model<"google-gemini-cli"> { + return buildModel({ + id, + name: id, + api: "google-gemini-cli", + provider: "google-antigravity", + baseUrl: "https://example.com", + reasoning: false, + input: ["text"], + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, + contextWindow: 200000, + maxTokens: 8192, + }); +} + +type SchemaNode = { type?: string; enum?: unknown[]; properties?: Record; items?: SchemaNode }; + +describe("Cloud Code Assist tool parameters with enums that are not strings", () => { + it("keep each value's type and drop the enum Google rejects", () => { + // The shape of a real rejection: "Invalid value at 'request.tools[0].function_declarations[45] + // .parameters.properties[2].value.items.properties[6].value.enum[0]' (TYPE_STRING), -1". + const tool: Tool = { + name: "edit_rows", + description: "Edit rows", + parameters: { + type: "object", + properties: { + rows: { + type: "array", + items: { + type: "object", + properties: { + direction: { type: "integer", enum: [-1, 1] }, + step: { enum: [-1, 1] }, + flag: { type: "boolean", enum: [true] }, + mixed: { enum: ["auto", 0] }, + mode: { type: "string", enum: ["read", "write"] }, + }, + }, + }, + }, + } as unknown as TJsonSchema, + }; + const model = antigravityModel("claude-opus-4-6"); + expect(model.compat.ccaLegacyParametersSchema).toBe(true); + + const [declarations] = convertTools([tool], model) ?? []; + const parameters = declarations?.functionDeclarations[0]?.parameters as SchemaNode; + const row = parameters.properties?.rows?.items?.properties; + if (!row) throw new Error("Expected the row schema to survive conversion"); + + expect(row.direction).toEqual({ type: "integer" }); + expect(row.step).toEqual({ type: "number" }); + expect(row.flag).toEqual({ type: "boolean" }); + expect(row.mixed).not.toHaveProperty("enum"); + expect(row.mode).toEqual({ type: "string", enum: ["read", "write"] }); + }); +}); diff --git a/packages/coding-agent/src/modes/rpc/rpc-prompt-results.ts b/packages/coding-agent/src/modes/rpc/rpc-prompt-results.ts index b0e093c6a9..ba1d457b25 100644 --- a/packages/coding-agent/src/modes/rpc/rpc-prompt-results.ts +++ b/packages/coding-agent/src/modes/rpc/rpc-prompt-results.ts @@ -19,7 +19,7 @@ import { isSessionSettled, type SessionSettleState } from "../../session/session import type { RpcScheduledTurnProbe } from "./rpc-session-settle"; import type { RpcPromptError, RpcPromptResultFrame, RpcPromptStatus } from "./rpc-types"; // scient(SA-USAGE-RESET): begin - A host resumes a usage-limited turn when the provider said the limit resets. -import { usageResetAt } from "../../scient/usage-reset"; +import { usageLimitStatus, usageResetAt } from "../../scient/usage-reset"; // scient(SA-USAGE-RESET): end // scient(SA-FAILURE-KIND): begin - A host reports why a turn failed, in the classifier's words. import { failureKinds } from "../../scient/failure-kind"; @@ -233,9 +233,11 @@ function promptError(message: AssistantMessage): RpcPromptError { // scient(SA-USAGE-RESET): begin - A host resumes a usage-limited turn when the provider said the limit resets. const resetAt = usageResetAt(message); if (resetAt !== undefined) error.resetAt = resetAt; + const usageStatus = usageLimitStatus(errorId, message.errorStatus); + if (usageStatus !== undefined) error.httpStatus = usageStatus; // scient(SA-USAGE-RESET): end // scient(SA-FAILURE-KIND): begin - A host reports why a turn failed, in the classifier's words. - const kinds = failureKinds(errorId); + const kinds = failureKinds(errorId, message); if (kinds !== undefined) error.kinds = kinds; // scient(SA-FAILURE-KIND): end return error; diff --git a/packages/coding-agent/src/scient/failure-kind.ts b/packages/coding-agent/src/scient/failure-kind.ts index 2de7d87bac..e026c2cca5 100644 --- a/packages/coding-agent/src/scient/failure-kind.ts +++ b/packages/coding-agent/src/scient/failure-kind.ts @@ -4,9 +4,9 @@ * `promptError` already classifies the failed message with `AIError.classifyMessage` and keeps * only whether it is transient. A host that counts failures needs the rest: an expired sign-in, * a context overflow and an overloaded provider all reach it today as the same opaque error. - * Each label names one classifier flag; together they are a closed vocabulary with no provider - * text. `AIError.stringify` is not used: its labels skip several flags and it renders unclassified - * ids as `status:`. + * Each label names one classifier flag, and `network` marks a failure that never reached the + * provider; together they are a closed vocabulary with no provider text. `AIError.stringify` is + * not used: its labels skip several flags and it renders unclassified ids as `status:`. */ import * as AIError from "@oh-my-pi/pi-ai/error"; @@ -34,12 +34,45 @@ export const FAILURE_KIND_LABELS: Readonly AIError.is(errorId, AIError.Flag[flag as keyof typeof FAILURE_KIND_LABELS])) - .map(([, label]) => label); + const kinds = AIError.is(errorId, AIError.Flag.Class) + ? Object.entries(FAILURE_KIND_LABELS) + .filter(([flag]) => AIError.is(errorId, AIError.Flag[flag as keyof typeof FAILURE_KIND_LABELS])) + .map(([, label]) => label) + : []; + if (isNetworkFailure(failure)) kinds.push(NETWORK_FAILURE_KIND); return kinds.length > 0 ? kinds : undefined; } diff --git a/packages/coding-agent/src/scient/usage-reset.ts b/packages/coding-agent/src/scient/usage-reset.ts index e6ab833ce5..f5a3fade58 100644 --- a/packages/coding-agent/src/scient/usage-reset.ts +++ b/packages/coding-agent/src/scient/usage-reset.ts @@ -8,6 +8,7 @@ * is never reported: a host shows no time rather than a wrong one. */ import type { AssistantMessage, UsageLimitMarkResult } from "@oh-my-pi/pi-ai"; +import * as AIError from "@oh-my-pi/pi-ai/error"; /** Recent resets; a `prompt_result` reads its message's right after recovery records it. */ const MAX_KEPT = 64; @@ -47,3 +48,15 @@ export function recordUsageReset( export function usageResetAt(message: AssistantMessage): number | undefined { return resets.get(messageKey(message)); } + +/** + * The HTTP status to report for a usage-limited failure whose provider sent none: 429. + * + * A host shows its usage-limit state (blocked until the reset) for a 429. Some failures the classifier + * reads as a usage limit carry no status: one a transport delivered without its response, or one + * that arrived as a stream event after the request was accepted. Reported as 429 they reach the + * same state instead of a generic failure. A status the provider sent is never replaced. + */ +export function usageLimitStatus(errorId: number, errorStatus: number | undefined): number | undefined { + return errorStatus === undefined && AIError.is(errorId, AIError.Flag.UsageLimit) ? 429 : undefined; +} diff --git a/packages/coding-agent/test/scient/failure-kind.test.ts b/packages/coding-agent/test/scient/failure-kind.test.ts index 94e734c4ae..6dfcca61dd 100644 --- a/packages/coding-agent/test/scient/failure-kind.test.ts +++ b/packages/coding-agent/test/scient/failure-kind.test.ts @@ -96,6 +96,23 @@ describe("prompt_result failure kinds", () => { expect(frame.error?.kinds).toEqual(["transient", "usage-limit"]); }); + it("tells a provider it could not reach from a provider that failed", async () => { + // The texts of real failures: a DNS lookup that found no address and a refused connection. + for (const failure of [ + "Couldn't reach chatgpt.com (getaddrinfo ENOTFOUND). Check the URL, your network, or proxy.", + "Couldn't connect to chatgpt.com (connection refused). Check that the server is running and reachable.", + ]) { + const frame = await promptResultFor(failure); + expect(frame.error?.kinds).toEqual(["transient", "network"]); + await session?.dispose(); + session = undefined; + } + + const outage = await promptResultFor("503 Service Unavailable: upstream connect error"); + expect(outage.error?.kinds).toContain("transient"); + expect(outage.error?.kinds).not.toContain("network"); + }); + it("leaves kinds out when the classifier recognized none", async () => { const frame = await promptResultFor("418 I'm a teapot"); @@ -114,6 +131,33 @@ describe("failure kind labels", () => { expect(failureKinds(AIError.create(AIError.Flag.CacheControlUnsupported))).toEqual(["cache-control-unsupported"]); }); + it("names every connection failure Oh My Pi describes a network failure", () => { + const failures = [ + { code: "EAI_AGAIN", message: "getaddrinfo EAI_AGAIN chatgpt.com" }, + { code: "ENOTFOUND", message: "getaddrinfo ENOTFOUND chatgpt.com" }, + { code: "ECONNREFUSED", message: "connect ECONNREFUSED 127.0.0.1:443" }, + { code: "SELF_SIGNED_CERT_IN_CHAIN", message: "self signed certificate in certificate chain" }, + ]; + for (const cause of failures) { + const sentence = AIError.networkFailureMessage(Object.assign(new Error(cause.message), { code: cause.code })); + if (sentence === undefined) throw new Error(`Expected a network failure sentence for ${cause.code}`); + expect(failureKinds(AIError.create(AIError.Flag.Transient), { errorMessage: sentence })).toContain("network"); + } + }); + + it("names no network failure for a response the provider sent", () => { + expect( + failureKinds(AIError.create(AIError.Flag.AuthFailed), { + errorMessage: "Unable to verify your API key", + errorStatus: 401, + }), + ).toEqual(["auth-failed"]); + // Without a status, "unable to verify" is about a certificate only when it says so. + expect( + failureKinds(AIError.create(AIError.Flag.AuthFailed), { errorMessage: "Unable to verify your API key" }), + ).toEqual(["auth-failed"]); + }); + it("names no kind for a raw HTTP status or a class with no flag", () => { expect(failureKinds(503)).toBeUndefined(); // 0x800 is also CacheControlUnsupported's bit; without the class bit it is a status lane value. diff --git a/packages/coding-agent/test/scient/usage-reset.test.ts b/packages/coding-agent/test/scient/usage-reset.test.ts index d817d958d1..c4394e2874 100644 --- a/packages/coding-agent/test/scient/usage-reset.test.ts +++ b/packages/coding-agent/test/scient/usage-reset.test.ts @@ -3,6 +3,7 @@ import * as path from "node:path"; import { Agent } from "@oh-my-pi/pi-agent-core"; import { unregisterCustomApis } from "@oh-my-pi/pi-ai/api-registry"; import * as envApiKey from "@oh-my-pi/pi-ai/env-api-key"; +import * as AIError from "@oh-my-pi/pi-ai/error"; import { createMockModel } from "@oh-my-pi/pi-ai/providers/mock"; import { getBundledModel } from "@oh-my-pi/pi-catalog/models"; import { TempDir } from "@oh-my-pi/pi-utils"; @@ -13,6 +14,7 @@ import type { RpcPromptResultFrame } from "../../src/modes/rpc/rpc-types"; import { AgentSession } from "../../src/session/agent-session"; import { AuthStorage } from "../../src/session/auth-storage"; import { SessionManager } from "../../src/session/session-manager"; +import { usageLimitStatus } from "../../src/scient/usage-reset"; const MOCK_API_SOURCE = "scient-usage-reset-test"; @@ -91,6 +93,23 @@ describe("prompt_result reset time of a usage limit", () => { expect(frame.error?.resetAt).toBeLessThanOrEqual(after + 3_600_000); }); + it("reports a long-wait usage limit that came without an HTTP status as a 429 with its reset", async () => { + // The text of a real failure: the wait exceeds retry.maxDelayMs, so recovery gives up at once + // ("Provider requested 119172000ms wait, exceeds retry.maxDelayMs"). The mock transport, like + // some real ones, attaches no HTTP status to the failed message. + const before = Date.now(); + const frame = await promptResultFor( + '429 {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your account\'s rate limit. Please try again later."}} retry-after-ms=119172000', + ); + const after = Date.now(); + + expect(frame.status).toBe("error"); + expect(frame.error?.httpStatus).toBe(429); + expect(frame.error?.kinds).toContain("usage-limit"); + expect(frame.error?.resetAt).toBeGreaterThanOrEqual(before + 119_172_000); + expect(frame.error?.resetAt).toBeLessThanOrEqual(after + 119_172_000); + }); + it("leaves the reset out when the provider gave no time", async () => { const frame = await promptResultFor("429 quota exceeded for this account"); @@ -99,3 +118,11 @@ describe("prompt_result reset time of a usage limit", () => { expect(frame.error).not.toHaveProperty("resetAt"); }); }); + +describe("prompt_result status of a usage limit", () => { + it("never replaces a status the provider sent", () => { + const usageLimit = AIError.create(AIError.Flag.UsageLimit); + expect(usageLimitStatus(usageLimit, 402)).toBeUndefined(); + expect(usageLimitStatus(AIError.create(AIError.Flag.Transient), undefined)).toBeUndefined(); + }); +}); diff --git a/packages/utils/src/fetch-retry.ts b/packages/utils/src/fetch-retry.ts index 879f0970ae..41e1726266 100644 --- a/packages/utils/src/fetch-retry.ts +++ b/packages/utils/src/fetch-retry.ts @@ -1,4 +1,7 @@ import { scheduler } from "node:timers/promises"; +// scient(SA-USAGE-RESET): begin - "Your usage window resets at