Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions SCIENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,12 @@ Everything else is upstream's code and behavior.
| Shell traces | The bash tool records no execution trace unless the user turns `display.shellVisualization` on. Scient Agent's host draws no shell visualization, and every live frame of a running command would carry the whole trace, up to megabytes. |
| Browser relay | The relay listens on port 9324 (OMP uses 9224), so each product runs its own. |
| Next message | `predict_next_message` gives an RPC host the prediction of the user's next message that OMP's terminal composer shows after a turn: OMP's prompt and parsing, on the session's model and context, without changing its history. It runs outside the command queue; a newer request replaces one in flight, and a turn or compaction that starts cancels it. |
| Usage-limit reset | A `prompt_result` for a turn stopped by a usage limit carries `error.resetAt`, the time the provider said the limit resets (epoch ms): from its retry hint, the account's usage report, or an earlier stated block, whichever is latest. It is left out when the provider gave no time; the 30-minute guess for a hintless error is never reported. A host can resume the turn at that time. |
| Usage-limit reset | A `prompt_result` for a turn stopped by a usage limit carries `error.resetAt`, the time the provider said the limit resets (epoch ms): from its retry hint, the account's usage report, or an earlier stated block, whichever is latest. A time the error states is read from "resets at" as well as Oh My Pi's "reset at", by the agent's own retry too. It is left out when the provider gave no time; the 30-minute guess for a hintless error is never reported. A host can resume the turn at that time. A usage-limited turn whose failure came without an HTTP status reports `error.httpStatus` 429, so a host that shows a usage-limit state for a 429 shows it; a status the provider sent is kept. |
| Prompt cache | An assistant reply's `message_end` carries `promptCache` (`tier`, `touchedAt`, `ttlMs`): the prompt-cache entry its response confirmed, while the next request would still read it. It is left out for a cache that never expires and for an entry the agent only inferred. A host can compact an idle conversation just before a one-hour entry expires, so the summary reads a warm cache. The agent's own idle compaction stays a terminal feature and never runs over RPC. |
| Plan mode | Oh My Pi plans only in its terminal and for ACP clients. Over RPC, the `plan` command (`enter`, `exit`, `get`) turns plan mode on: the agent reads and drafts a plan file but changes nothing else. When it proposes the plan, a `plan_proposed` frame carries the plan, and a select request asks the host to approve it while the run waits. Only an explicit approval leaves plan mode, and the same run then carries out the plan. Refine asks the host what to change, and that answer, like any typed answer, reaches the model as feedback; planning goes on. An answer that comes after plan mode ended never approves, and a new, switched or forked conversation starts outside plan mode. Each time plan mode turns on or off, a `plan_updated` frame tells the host. Planning is refused while a goal is set. |
| Failure kinds | A `prompt_result` for a failed turn carries `error.kinds`, the error classifier's labels for that failure (`auth-failed`, `context-overflow`, `transient`, `usage-limit`, ...), so a host can report why turns fail. It is left out when the classifier recognized no kind; the labels carry no provider text. |
| Failure kinds | A `prompt_result` for a failed turn carries `error.kinds`, the error classifier's labels for that failure (`auth-failed`, `context-overflow`, `transient`, `usage-limit`, ...), so a host can report why turns fail. A request that never reached the provider (no address, a refused connection, no network, a certificate that did not verify) also carries `network`, which the classifier's flags do not tell apart from a provider's outage. It is left out when the classifier recognized no kind; the labels carry no provider text. |
| Access policy | A provider that serves a model only to the apps it lists (OpenRouter: "… is only available on agentic harnesses") refuses other clients with a 403. Oh My Pi reads a bare 403 as a failed sign-in, so the person was sent to fix a key that works. The refusal is an account policy: the agent tries the model's fallback chain, and the turn's `error.kinds` carries `account-policy`. |
| Google tool enums | Cloud Code Assist takes a Claude model's tool schemas in its legacy `parameters` field, whose enums must be strings: one numeric enum (`[-1, 1]`) in any tool made Google refuse every request of the session with a 400. As on the Gemini path, an enum with a value that is not a string is left out and the field keeps its type. |
| macOS Accessibility prompt | The agent sends no Accessibility message while macOS doesn't trust it, so a computer call that refuses for lack of a permission never makes macOS ask. Only the host asks for permissions. |
| Live voice delegation | `live_connect` with `hostDelegation` sends each request the voice model delegates to the host as a `live_delegation` frame instead of starting a turn, so a host that owns its turns (Scient Desktop) sends it as the user's message. The live session still reads that turn's answer. A host that runs the turn elsewhere, such as in the conversation's own agent while the call runs in a separate one, reports its progress and answer with `live_report`. Without the flag nothing changes. |
| Dictation | `scient-agent dictate <file>` transcribes a recording with ChatGPT's dictation service, the one the ChatGPT and Codex apps use, through the stored ChatGPT sign-in. With `--json` it prints `{ "text" }`, or `{ "error": { "kind", "message" } }` and exits 1, so a host (Scient Desktop's microphone) can show its own message. ChatGPT keeps the uploaded recording as a file for 30 days, as it does for its own apps' dictation. |
Expand Down
6 changes: 6 additions & 0 deletions packages/ai/src/error/flags.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ import {
parseJsonBody,
parseRateLimitReason,
} from "./rate-limit";
// scient(SA-ACCESS-POLICY): begin - A model a provider serves only to its listed apps is an access policy, not a bad key.
import { isAppRestrictedModelText } from "../scient/access-policy";
// scient(SA-ACCESS-POLICY): end

export const Flag = {
Class: 0x1000,
Expand Down Expand Up @@ -672,6 +675,9 @@ function classifyText(
if (isAnthropicAccountPolicyText(errorMessage, provider, statusClean)) {
kinds |= Flag.AccountPolicy;
}
// scient(SA-ACCESS-POLICY): begin - A model a provider serves only to its listed apps is an access policy, not a bad key.
if (isAppRestrictedModelText(errorMessage)) kinds |= Flag.AccountPolicy;
// scient(SA-ACCESS-POLICY): end
if (isAuthFailureText(errorMessage)) kinds |= Flag.AuthFailed;
const cleanMessage = errorMessage;
const isOpaque = isOpaqueStatusBody(cleanMessage);
Expand Down
17 changes: 17 additions & 0 deletions packages/ai/src/scient/access-policy.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
/**
* Provider refusals that name an access policy, not a bad credential.
*
* OpenRouter serves some free models only to the apps it lists, and refuses any other client with a
* 403: "thinkingmachines/inkling-2-small:free is only available on agentic harnesses. Try plugging
* it into a coding agent or productivity app listed on https://openrouter.ai/apps". The key is
* valid. Oh My Pi reads a bare 403 as a failed sign-in, so the person was told to fix a key that
* works. The classifier marks the refusal an account policy instead, as it does Anthropic's
* organization denials.
*/

const HARNESS_ONLY_MODEL_PATTERN = /\bis only available on agentic harnesses\b/i;

/** Whether `text` is a provider's refusal to serve a model outside the apps it lists. */
export function isAppRestrictedModelText(text: string): boolean {
return HARNESS_ONLY_MODEL_PATTERN.test(text);
}
3 changes: 3 additions & 0 deletions packages/ai/src/utils/schema/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1336,6 +1336,9 @@ export function normalizeSchemaForCCA(value: unknown): unknown {
extractNullableFromUnions: true,
inferTypeForBareEnum: true,
dropNonScalarEnum: false,
// scient(SA-CCA-STRING-ENUMS): begin - Cloud Code Assist's `parameters` take only string enums, as Gemini's do.
stringEnumsOnly: true,
// scient(SA-CCA-STRING-ENUMS): end
foldOneOfIntoAnyOf: false,
rejectResidualIncompatibilities: ["type-array", "type-null", "nullable", "combiners", "not"],
validateAndFallback: { fallback: CLOUD_CODE_ASSIST_CLAUDE_FALLBACK_SCHEMA },
Expand Down
40 changes: 40 additions & 0 deletions packages/ai/test/scient/access-policy.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import { afterAll, beforeAll, describe, expect, it } from "bun:test";
import { getBundledModels } from "@oh-my-pi/pi-catalog/models";
import * as AIError from "@oh-my-pi/pi-ai/error";
import { complete } from "@oh-my-pi/pi-ai/stream";

const APP_RESTRICTED =
"thinkingmachines/inkling-2-small:free is only available on agentic harnesses. Try plugging it into a coding agent or productivity app listed on https://openrouter.ai/apps";

describe("a model a provider serves only to its listed apps", () => {
let server: ReturnType<typeof Bun.serve>;

beforeAll(() => {
server = Bun.serve({
port: 0,
fetch: () => Response.json({ error: { message: APP_RESTRICTED, code: 403 } }, { status: 403 }),
});
});

afterAll(() => server.stop(true));

it("is an account policy, which a host ranks before a failed sign-in", async () => {
const model = getBundledModels("openrouter")[0];
if (!model) throw new Error("Expected a bundled OpenRouter model");
const failed = await complete(
{ ...model, baseUrl: `http://localhost:${server.port}/v1` },
{ systemPrompt: ["Test"], messages: [{ role: "user", content: "hi", timestamp: Date.now() }] },
{ apiKey: "openrouter-test-key" },
);

expect(failed.stopReason).toBe("error");
expect(failed.errorStatus).toBe(403);
expect(AIError.is(AIError.classifyMessage(failed), AIError.Flag.AccountPolicy)).toBe(true);
});

it("leaves another 403 a failed sign-in", () => {
const id = AIError.classifyMessage({ provider: "openrouter", errorMessage: "403 Forbidden", errorStatus: 403 });
expect(AIError.is(id, AIError.Flag.AuthFailed)).toBe(true);
expect(AIError.is(id, AIError.Flag.AccountPolicy)).toBe(false);
});
});
63 changes: 63 additions & 0 deletions packages/ai/test/scient/cca-string-enums.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
import { describe, expect, it } from "bun:test";
import { convertTools } from "@oh-my-pi/pi-ai/providers/google-shared";
import type { Model, TJsonSchema, Tool } from "@oh-my-pi/pi-ai/types";
import { buildModel } from "@oh-my-pi/pi-catalog/build";

function antigravityModel(id: string): Model<"google-gemini-cli"> {
return buildModel({
id,
name: id,
api: "google-gemini-cli",
provider: "google-antigravity",
baseUrl: "https://example.com",
reasoning: false,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 200000,
maxTokens: 8192,
});
}

type SchemaNode = { type?: string; enum?: unknown[]; properties?: Record<string, SchemaNode>; items?: SchemaNode };

describe("Cloud Code Assist tool parameters with enums that are not strings", () => {
it("keep each value's type and drop the enum Google rejects", () => {
// The shape of a real rejection: "Invalid value at 'request.tools[0].function_declarations[45]
// .parameters.properties[2].value.items.properties[6].value.enum[0]' (TYPE_STRING), -1".
const tool: Tool = {
name: "edit_rows",
description: "Edit rows",
parameters: {
type: "object",
properties: {
rows: {
type: "array",
items: {
type: "object",
properties: {
direction: { type: "integer", enum: [-1, 1] },
step: { enum: [-1, 1] },
flag: { type: "boolean", enum: [true] },
mixed: { enum: ["auto", 0] },
mode: { type: "string", enum: ["read", "write"] },
},
},
},
},
} as unknown as TJsonSchema,
};
const model = antigravityModel("claude-opus-4-6");
expect(model.compat.ccaLegacyParametersSchema).toBe(true);

const [declarations] = convertTools([tool], model) ?? [];
const parameters = declarations?.functionDeclarations[0]?.parameters as SchemaNode;
const row = parameters.properties?.rows?.items?.properties;
if (!row) throw new Error("Expected the row schema to survive conversion");

expect(row.direction).toEqual({ type: "integer" });
expect(row.step).toEqual({ type: "number" });
expect(row.flag).toEqual({ type: "boolean" });
expect(row.mixed).not.toHaveProperty("enum");
expect(row.mode).toEqual({ type: "string", enum: ["read", "write"] });
});
});
6 changes: 4 additions & 2 deletions packages/coding-agent/src/modes/rpc/rpc-prompt-results.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import { isSessionSettled, type SessionSettleState } from "../../session/session
import type { RpcScheduledTurnProbe } from "./rpc-session-settle";
import type { RpcPromptError, RpcPromptResultFrame, RpcPromptStatus } from "./rpc-types";
// scient(SA-USAGE-RESET): begin - A host resumes a usage-limited turn when the provider said the limit resets.
import { usageResetAt } from "../../scient/usage-reset";
import { usageLimitStatus, usageResetAt } from "../../scient/usage-reset";
// scient(SA-USAGE-RESET): end
// scient(SA-FAILURE-KIND): begin - A host reports why a turn failed, in the classifier's words.
import { failureKinds } from "../../scient/failure-kind";
Expand Down Expand Up @@ -233,9 +233,11 @@ function promptError(message: AssistantMessage): RpcPromptError {
// scient(SA-USAGE-RESET): begin - A host resumes a usage-limited turn when the provider said the limit resets.
const resetAt = usageResetAt(message);
if (resetAt !== undefined) error.resetAt = resetAt;
const usageStatus = usageLimitStatus(errorId, message.errorStatus);
if (usageStatus !== undefined) error.httpStatus = usageStatus;
// scient(SA-USAGE-RESET): end
// scient(SA-FAILURE-KIND): begin - A host reports why a turn failed, in the classifier's words.
const kinds = failureKinds(errorId);
const kinds = failureKinds(errorId, message);
if (kinds !== undefined) error.kinds = kinds;
// scient(SA-FAILURE-KIND): end
return error;
Expand Down
51 changes: 42 additions & 9 deletions packages/coding-agent/src/scient/failure-kind.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
* `promptError` already classifies the failed message with `AIError.classifyMessage` and keeps
* only whether it is transient. A host that counts failures needs the rest: an expired sign-in,
* a context overflow and an overloaded provider all reach it today as the same opaque error.
* Each label names one classifier flag; together they are a closed vocabulary with no provider
* text. `AIError.stringify` is not used: its labels skip several flags and it renders unclassified
* ids as `status:<n>`.
* Each label names one classifier flag, and `network` marks a failure that never reached the
* provider; together they are a closed vocabulary with no provider text. `AIError.stringify` is
* not used: its labels skip several flags and it renders unclassified ids as `status:<n>`.
*/
import * as AIError from "@oh-my-pi/pi-ai/error";

Expand Down Expand Up @@ -34,12 +34,45 @@ export const FAILURE_KIND_LABELS: Readonly<Record<Exclude<keyof typeof AIError.F
PayloadRejected: "payload-rejected",
};

/** The labels of `errorId`'s flags, or undefined when the classifier recognized no kind. */
export function failureKinds(errorId: number): string[] | undefined {
/**
* The label of a failure to reach the provider at all: no DNS answer, a refused connection, no
* network, a certificate that did not verify. The classifier marks these only `transient`, like a
* provider's own outage; a host tells the two apart by this label. It is Oh My Pi's network
* diagnosis (`classifyNetworkCause`), not a classifier flag.
*/
export const NETWORK_FAILURE_KIND = "network";

/**
* The sentences Oh My Pi's `networkFailureMessage` writes for a failed connection. The refused one
* keeps no error code, so the diagnosis does not recognize it again from the text.
*/
const NETWORK_FAILURE_SENTENCE =
/\b(?:No internet connection: couldn't reach |Couldn't reach .+?(?: \(|\. Check the URL)|Couldn't connect to .+? \(connection refused\)|TLS certificate verification failed for )/;

/** Whether a failure with this text never reached the provider: it has no HTTP status and reads as a transport failure. */
function isNetworkFailure(failure: { errorMessage?: string; errorStatus?: number } | undefined): boolean {
const text = failure?.errorMessage;
if (!text || failure?.errorStatus !== undefined || AIError.status({ message: text }) !== undefined) return false;
if (NETWORK_FAILURE_SENTENCE.test(text)) return true;
const cause = AIError.classifyNetworkCause(text);
// A bare "unable to verify" may be about a token, not a certificate.
return cause !== undefined && (cause.kind !== "tls" || /certificat/i.test(text));
}

/**
* The labels of `errorId`'s flags, plus {@link NETWORK_FAILURE_KIND} when `failure` never reached
* the provider, or undefined when no kind is recognized.
*/
export function failureKinds(
errorId: number,
failure?: { errorMessage?: string; errorStatus?: number },
): string[] | undefined {
// An id without the class bit is a raw HTTP status, whose low bits are not flags.
if (!AIError.is(errorId, AIError.Flag.Class)) return undefined;
const kinds = Object.entries(FAILURE_KIND_LABELS)
.filter(([flag]) => AIError.is(errorId, AIError.Flag[flag as keyof typeof FAILURE_KIND_LABELS]))
.map(([, label]) => label);
const kinds = AIError.is(errorId, AIError.Flag.Class)
? Object.entries(FAILURE_KIND_LABELS)
.filter(([flag]) => AIError.is(errorId, AIError.Flag[flag as keyof typeof FAILURE_KIND_LABELS]))
.map(([, label]) => label)
: [];
if (isNetworkFailure(failure)) kinds.push(NETWORK_FAILURE_KIND);
return kinds.length > 0 ? kinds : undefined;
}
13 changes: 13 additions & 0 deletions packages/coding-agent/src/scient/usage-reset.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
* is never reported: a host shows no time rather than a wrong one.
*/
import type { AssistantMessage, UsageLimitMarkResult } from "@oh-my-pi/pi-ai";
import * as AIError from "@oh-my-pi/pi-ai/error";

/** Recent resets; a `prompt_result` reads its message's right after recovery records it. */
const MAX_KEPT = 64;
Expand Down Expand Up @@ -47,3 +48,15 @@ export function recordUsageReset(
export function usageResetAt(message: AssistantMessage): number | undefined {
return resets.get(messageKey(message));
}

/**
* The HTTP status to report for a usage-limited failure whose provider sent none: 429.
*
* A host shows its usage-limit state (blocked until the reset) for a 429. Some failures the classifier
* reads as a usage limit carry no status: one a transport delivered without its response, or one
* that arrived as a stream event after the request was accepted. Reported as 429 they reach the
* same state instead of a generic failure. A status the provider sent is never replaced.
*/
export function usageLimitStatus(errorId: number, errorStatus: number | undefined): number | undefined {
return errorStatus === undefined && AIError.is(errorId, AIError.Flag.UsageLimit) ? 429 : undefined;
}
Loading
Loading