diff --git a/CLAUDE.md b/CLAUDE.md index d2995ba5..842fe3f1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,12 +5,19 @@ Self-hosted AI workspace built around context management. Monorepo with two work ## Tech stack - **Runtime**: Bun (server + build tooling) -- **Server**: Hono, TypeScript, Claude Agent SDK, podman (sandbox containers) +- **Server**: Hono, TypeScript, Claude Agent SDK, Codex CLI runtime, podman (sandbox containers) - **Web**: React 19, Vite 8, Tailwind CSS v4, Zustand, assistant-ui, xterm.js, CodeMirror, Milkdown - **Infra**: Docker (oven/bun base), rootless podman inside container - **Tests**: Playwright (e2e + dogfood) - **Rust**: Two small binaries in `server/src/serve-rs` and `server/src/port-proxy-rs`, built as part of `web build` +## Agent runtimes + +- Providers declare `runtime: "claude" | "codex"`; legacy provider configs default to Claude. +- Claude runs through the Agent SDK inside podman. Codex runs through the host CLI and must not initialize Claude settings, plugins, MCP servers, containers, or the Anthropic egress gateway. +- Keep Codex command construction, environment mapping, and JSONL parsing in `server/src/codex-cli.ts`. Session code owns lifecycle and UI event translation only. +- Add focused adapter tests when Codex CLI arguments or event parsing changes. Preserve the existing Claude path unless a migration is explicitly required. + ## Development ```bash diff --git a/docs/gitlab-personal-storage.md b/docs/gitlab-personal-storage.md new file mode 100644 index 00000000..00e4ca7b --- /dev/null +++ b/docs/gitlab-personal-storage.md @@ -0,0 +1,150 @@ +# GitLab Personal Storage + +Loopat can use a GitLab-compatible repository for encrypted personal storage. +GitHub remains the default; administrators can select GitLab through workspace +configuration or environment variables. + +```jsonc +{ + "gitHost": { + "provider": "gitlab", + "baseUrl": "https://gitlab.example.com", + "defaultRepo": "team/loopat-personal" + } +} +``` + +Equivalent environment variables are: + +```sh +LOOPAT_GIT_HOST_PROVIDER=gitlab +LOOPAT_GIT_HOST_BASE_URL=https://gitlab.example.com +LOOPAT_GIT_HOST_DEFAULT_REPO=team/loopat-personal +``` + +## Design + +The implementation uses the existing `GitHostProvider` abstraction rather than +adding GitLab conditionals to the personal-storage workflow. Provider resolution +uses the following precedence: + +1. An extension under `LOOPAT_HOME/extensions/providers/`. +2. A provider explicitly supplied by the request. +3. `LOOPAT_GIT_HOST_PROVIDER` or `LOOPAT_GIT_PROVIDER`. +4. Workspace `config.json` field `gitHost.provider`. +5. Built-in `github`. + +`resolveGitHostSettings()` applies the same request, environment, workspace, and +provider-default precedence to `baseUrl` and `defaultRepo`. + +For GitLab, `baseUrl` is the human-facing site root. The provider derives the +REST endpoint as `${baseUrl}/api/v4`; a configured URL that already ends in +`/api/v4` is normalized back to the site root when building web and clone URLs. + +## Authentication + +### Token flow + +The standard flow uses a personal or project access token with these scopes: + +- `api` +- `read_repository` +- `write_repository` + +API requests try the standard `PRIVATE-TOKEN` header first, followed by bearer +authentication and the `private_token` query parameter for compatible +self-hosted deployments. Redirects, HTML responses, and authentication errors +advance to the next mode. + +Git clone and push use the provider's `https-token` mode: + +```text +https://:@gitlab.example.com//.git +``` + +The provider reads standard GitLab user responses and common wrapped enterprise +responses. If the profile endpoint does not expose a namespace username, a +successful project-list request can still validate the token. The user must then +enter a full `namespace/project` path; Loopat uses `oauth2` only as the non-empty +HTTPS Basic Auth username. + +### Administrator-pinned SSH flow + +Some self-hosted installations place the API behind SSO while leaving SSH git +access available. When an administrator configures a full `namespace/project` +as `defaultRepo`, Loopat resolves it to an SSH URL and skips token onboarding: + +```text +git@gitlab.example.com:team/loopat-personal.git +``` + +Direct SSH uses the operating-system account that runs Loopat. Request-body +overrides cannot select another host-SSH repository; the repository must come +from administrator-controlled workspace or environment configuration. Verify +that the process account has write access before enabling this mode. + +The imported repository records `loopat.personalSshAuth=host` in local Git +configuration. Initial push, fetch, pull, delete synchronization, and later +pushes all reuse the same host identity. Host key checking uses +`StrictHostKeyChecking=accept-new` when supported and falls back to `no` for +older OpenSSH versions; administrators using the fallback should pre-seed and +monitor `known_hosts`. + +## Repository identity + +Personal-repository commits select an author in this order: + +1. Provider identity, when both name and email are available. +2. `LOOPAT_GIT_AUTHOR_NAME` and `LOOPAT_GIT_AUTHOR_EMAIL`. +3. Existing repository or process-account Git configuration. +4. Loopat's local fallback identity. + +Installations with commit-email push rules should configure an accepted email +through the environment or the process account's Git configuration. + +## Runtime flow + +1. The UI calls `GET /api/personal/status`. +2. The server resolves the active git host, base URL, and default repository. +3. For token mode, the UI authenticates and lists repositories through the + provider. +4. For direct SSH mode, the UI shows the administrator-pinned repository for + confirmation and skips token entry. +5. The backward-compatible `POST /api/personal/github` route delegates setup to + the active provider. +6. `setupPersonalViaProvider()` clones or initializes the repository, imports + the personal data, and configures git-crypt when needed. + +## Code map + +- `server/src/gitlab.ts` implements the GitLab API adapter, response parsing, + repository creation, project listing, and direct SSH resolution. +- `server/src/providers.ts` registers built-in providers and resolves active git + host settings. +- `server/src/index.ts` routes personal status, repository listing, and setup + through the active provider. +- `server/src/loops.ts` preserves full namespace paths and applies the selected + Git authentication mode to all synchronization operations. +- `web/src/components/dialog/PersonalRepoPanel.tsx` renders token and direct SSH + onboarding from the provider-neutral status response. +- `server/src/git-author.ts` selects a push-rule-compatible commit identity. + +## Validation + +```sh +bun test server/test/gitlab.test.ts server/test/git-author.test.ts +bunx tsc -p server/tsconfig.json --noEmit +bun run build +``` + +End-to-end validation additionally requires either a usable GitLab token or SSH +write access from the Loopat process account. + +## Limitations + +- The public setup route is still named `/api/personal/github` for backward + compatibility. A provider-neutral alias can be added separately. +- Group and subgroup project creation depends on + `GET /api/v4/namespaces/:path` and the token's namespace permissions. +- Repository listing currently reads the first 100 membership projects. +- Direct SSH currently assumes the GitLab SSH service uses its standard port. diff --git a/docs/setup-admin.md b/docs/setup-admin.md index e2100dcc..e82c739f 100644 --- a/docs/setup-admin.md +++ b/docs/setup-admin.md @@ -56,6 +56,34 @@ empty fields after first run. Fill it in: | `repos[]` | each entry → cloned to `context/repos//`. Loops spawn against these. | | `mounts[]` | **operator-level** mounts; `src` is any host path. Shared by every loop on this workspace. See §6. | +Personal storage git host can also be selected here: + +```jsonc +{ + "gitHost": { + "provider": "gitlab", + "baseUrl": "https://gitlab.example.com", + "defaultRepo": "team/loopat-personal" + } +} +``` + +`provider` defaults to `github`; standard GitLab hosts use the GitLab-compatible +`/api/v4` API and HTTPS token git auth. A full, administrator-configured +`namespace/repo` value enables direct SSH onboarding instead. This supports +self-hosted installations whose API is protected by SSO: Loopat skips the API +and uses the operating-system account's existing SSH identity for clone, pull, +and push. Verify that account has write access before exposing the setup UI. The +same values can be supplied with +`LOOPAT_GIT_HOST_PROVIDER`, `LOOPAT_GIT_HOST_BASE_URL`, and +`LOOPAT_GIT_HOST_DEFAULT_REPO`. + +Personal-repo commits use the provider identity when available, then +`LOOPAT_GIT_AUTHOR_NAME` / `LOOPAT_GIT_AUTHOR_EMAIL`, then the operating-system +account's Git configuration. Hosts with commit-email push rules should set a +valid email globally (`git config --global user.email ...`) or +through `LOOPAT_GIT_AUTHOR_EMAIL`. + Provider config (`apiKey`, `model`, `baseUrl`) is **not** here — that lives per-user under `personal//.loopat/config.json`. Admins don't pre-fill API keys for the team. diff --git a/docs/troubleshoot.md b/docs/troubleshoot.md index 2e98fd7d..27b4bb06 100644 --- a/docs/troubleshoot.md +++ b/docs/troubleshoot.md @@ -2,6 +2,23 @@ If chat doesn't work or the UI shows red errors, walk this list top-to-bottom. Most issues land in §1 or §2. +## Codex exits with code 1 + +Loopat records Codex CLI diagnostics in the loop's `stderr.log`. Codex API +failures arrive as JSONL events on stdout; Loopat extracts `error` and +`turn.failed` events and shows their message in the chat. + +If the selected model requires a newer Codex CLI, upgrade the CLI or point +Loopat at a newer binary before restarting: + +```sh +LOOPAT_CODEX_BIN=/path/to/codex bun run dev +``` + +On macOS, ChatGPT may include a newer CLI at +`/Applications/ChatGPT.app/Contents/Resources/codex`. Confirm it with +`codex --version` before configuring the path. + ## 0. The bootstrap banner is the first signal Whatever's wrong, look at the banner `bun run dev` prints first: diff --git a/server/src/auto-name.ts b/server/src/auto-name.ts index b7c0a6c8..f4b5caeb 100644 --- a/server/src/auto-name.ts +++ b/server/src/auto-name.ts @@ -82,7 +82,7 @@ async function resolveProvidersForLoop(meta: { createdBy: string; config?: { def if (seen.has(name)) continue seen.add(name) const p = pCfg.providers[name] ?? wCfg.providers?.[name] - if (p && p.apiKey) result.push(p) + if (p && p.apiKey && p.runtime !== "codex") result.push(p) } return result } diff --git a/server/src/bootstrap.ts b/server/src/bootstrap.ts index 7ce299eb..812e69ce 100644 --- a/server/src/bootstrap.ts +++ b/server/src/bootstrap.ts @@ -17,8 +17,9 @@ import { workspaceTeamClaudeMdPath, } from "./paths" import { listUsers } from "./auth" +import { codexBinary } from "./codex-cli" -type Check = { ok: boolean; label: string; hint?: string } +type Check = { ok: boolean; label: string; hint?: string; required?: boolean } /** The host to print in the "open …" url. HOST=0.0.0.0/:: means "all * interfaces" — localhost works locally but isn't reachable from other @@ -84,6 +85,20 @@ function checkClaudeBinary(): Check { } } +function checkCodexCli(): Check { + const codexBin = codexBinary() + try { + const out = execFileSync(codexBin, ["--version"], { stdio: "pipe" }).toString().trim() + return { ok: true, label: `codex cli: ${out || "available"}` } + } catch { + return { + ok: false, + label: "codex cli", + hint: "install Codex CLI or run `codex login` on the host if you want the Codex runtime", + } + } +} + function checkGitCrypt(): Check { try { const out = execFileSync("git-crypt", ["--version"], { stdio: "pipe" }).toString().trim() @@ -126,6 +141,15 @@ export async function printBootstrapBanner(cfg: WorkspaceConfig) { // notes is declared inside the knowledge repo's .loopat/config.json. The repo // roster is per-user (personal config), so the workspace banner can't list it. const kcfg = await loadKnowledgeConfig() + const podman = checkPodman() + const claude = checkClaudeBinary() + const codex = checkCodexCli() + const hasUsableAgentRuntime = codex.ok || (podman.ok && claude.ok) + if (hasUsableAgentRuntime) { + podman.required = false + claude.required = false + codex.required = false + } const checks: Check[] = [ { ok: true, label: `workspace: ${workspaceDir()}` }, { ok: true, label: `team .claude/CLAUDE.md (${existsSync(workspaceTeamClaudeMdPath()) ? "present" : "absent"})` }, @@ -134,8 +158,9 @@ export async function printBootstrapBanner(cfg: WorkspaceConfig) { { ok: true, label: `repos: (per-user, in personal config)` }, await checkUsers(), { ok: existsSync(configPath()), label: `config: ${configPath()}` }, - checkPodman(), - checkClaudeBinary(), + podman, + claude, + codex, checkGitCrypt(), ] @@ -154,9 +179,9 @@ export async function printBootstrapBanner(cfg: WorkspaceConfig) { if (!c.ok && c.hint) console.log(` ${yellow("→ " + c.hint)}`) } console.log(bar) - const blockers = checks.filter((c) => !c.ok) + const blockers = checks.filter((c) => !c.ok && c.required !== false) if (blockers.length > 0) { - console.log(` ${yellow(`${blockers.length} thing(s) to fix`)} before chat will work — see hints above.\n`) + console.log(` ${yellow(`${blockers.length} thing(s) to fix`)} before all required startup checks pass — see hints above.\n`) return false } // NB: the "ready. open …" line is intentionally NOT printed here. The banner diff --git a/server/src/codex-cli.ts b/server/src/codex-cli.ts new file mode 100644 index 00000000..e8fcbc04 --- /dev/null +++ b/server/src/codex-cli.ts @@ -0,0 +1,150 @@ +type JsonObject = Record + +export type CodexEvent = JsonObject & { type: string } + +export type CodexUsage = { + inputTokens: number + outputTokens: number + cachedInputTokens: number +} + +export type CodexCompletedItem = { + type: string + text?: string +} + +function asObject(value: unknown): JsonObject | null { + return value !== null && typeof value === "object" && !Array.isArray(value) + ? value as JsonObject + : null +} + +function errorMessage(value: unknown): string | null { + let current = value + for (let depth = 0; depth < 4; depth += 1) { + if (typeof current === "string") { + const text = current.trim() + if (!text) return null + try { + current = JSON.parse(text) as unknown + continue + } catch { + return text + } + } + + const record = asObject(current) + if (!record) return null + const nestedError = asObject(record.error) + current = nestedError?.message ?? record.message ?? record.error + } + return typeof current === "string" && current.trim() ? current.trim() : null +} + +export function parseCodexEvent(line: string): CodexEvent | null { + const text = line.trim() + if (!text) return null + try { + const value = asObject(JSON.parse(text) as unknown) + return value && typeof value.type === "string" + ? value as CodexEvent + : null + } catch { + return null + } +} + +/** Codex emits terminal API failures as JSONL events on stdout. */ +export function codexEventError(event: unknown): string | null { + const record = asObject(event) + if (!record) return null + if (record.type === "turn.failed") return errorMessage(record.error) + if (record.type === "error") return errorMessage(record.message ?? record.error) + return null +} + +export function codexThreadId(event: CodexEvent): string | null { + return event.type === "thread.started" && typeof event.thread_id === "string" + ? event.thread_id + : null +} + +export function codexCompletedItem(event: CodexEvent): CodexCompletedItem | null { + if (event.type !== "item.completed") return null + const item = asObject(event.item) + if (!item || typeof item.type !== "string") return null + return { + type: item.type, + ...(typeof item.text === "string" ? { text: item.text } : {}), + } +} + +export function codexTurnUsage(event: CodexEvent): CodexUsage | null { + if (event.type !== "turn.completed") return null + const usage = asObject(event.usage) + if (!usage) return null + return { + inputTokens: typeof usage.input_tokens === "number" ? usage.input_tokens : 0, + outputTokens: typeof usage.output_tokens === "number" ? usage.output_tokens : 0, + cachedInputTokens: typeof usage.cached_input_tokens === "number" ? usage.cached_input_tokens : 0, + } +} + +export function codexBinary(): string { + return process.env.LOOPAT_CODEX_BIN || "codex" +} + +export function codexModelArg(modelId?: string | null): string | undefined { + const value = modelId?.trim() + return value || undefined +} + +export function buildCodexEnv(opts: { + apiKey?: string + baseUrl?: string +}, baseEnv: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEnv { + const env: NodeJS.ProcessEnv = { ...baseEnv } + if (opts.apiKey) { + env.CODEX_API_KEY = opts.apiKey + env.OPENAI_API_KEY = opts.apiKey + } + const baseUrl = opts.baseUrl?.trim().replace(/\/+$/, "") + if (baseUrl) { + env.CODEX_BASE_URL = baseUrl + env.OPENAI_BASE_URL = baseUrl + } + return env +} + +export function buildCodexExecArgs(opts: { + workdir: string + threadId?: string | null + modelArg?: string + sandbox?: "read-only" | "workspace-write" + ephemeral?: boolean +}): string[] { + const modelArgs = opts.modelArg ? ["-m", opts.modelArg] : [] + if (opts.threadId) { + return [ + "exec", + "resume", + "--json", + "--skip-git-repo-check", + ...modelArgs, + opts.threadId, + "-", + ] + } + return [ + "exec", + "--json", + "--sandbox", + opts.sandbox ?? "workspace-write", + ...(opts.ephemeral ? ["--ephemeral"] : []), + "--cd", + opts.workdir, + "--skip-git-repo-check", + ...modelArgs, + "-", + ] +} diff --git a/server/src/config.ts b/server/src/config.ts index 76779660..3c998ec4 100644 --- a/server/src/config.ts +++ b/server/src/config.ts @@ -70,10 +70,27 @@ export function normalizeModelEntry(m: ModelEntryDisk): ModelEntry { return { id: m.id, ...(m.maxContextTokens ? { maxContextTokens: m.maxContextTokens } : {}), ...(m.tier ? { tier: m.tier } : {}) } } +export type ProviderRuntime = "claude" | "codex" + +export function normalizeProviderRuntime(runtime: unknown): ProviderRuntime { + return runtime === "codex" ? "codex" : "claude" +} + +function normalizeProviderModelsFromDisk(p: { models?: ModelEntryDisk[]; model?: string }): ModelEntry[] { + if (Array.isArray(p.models) && p.models.length > 0) { + return p.models.map(m => normalizeModelEntry(m)) + } + if (typeof p.model === "string" && p.model.trim()) { + return [{ id: p.model.trim() }] + } + return [] +} + export type ProviderPreset = { name: string baseUrl: string models: ModelEntryDisk[] + runtime?: ProviderRuntime } export type MiseToolPreset = { @@ -95,9 +112,13 @@ export type PresetsData = { */ export type ProviderConfigDisk = { models?: ModelEntryDisk[] + /** Legacy single-model field accepted from older config.json files and clients. */ + model?: string baseUrl: string apiKey?: string enabled?: boolean + /** Which agent runtime should consume this provider. Defaults to Claude Code. */ + runtime?: ProviderRuntime /** Per-tier model. Written → passed as ANTHROPIC_DEFAULT_*_MODEL; absent → CC native. */ opus_model?: string sonnet_model?: string @@ -112,6 +133,7 @@ export type ProviderConfig = { baseUrl: string apiKey: string enabled: boolean + runtime: ProviderRuntime opus_model?: string sonnet_model?: string haiku_model?: string @@ -312,7 +334,8 @@ export function parseDefault(raw: string): { providerName: string; modelId?: str * 3. workspace config's `default` field * 4. enumeration (personal first, then workspace) * - * `requireKey=true` skips providers with empty apiKey and keeps walking. + * `requireKey=true` skips providers with empty apiKey and keeps walking, except + * Codex runtime providers which may rely on `codex login`. * Returns null when no match found. */ export function pickProvider( @@ -333,28 +356,31 @@ export function pickProvider( if (seen.has(name)) continue seen.add(name) const p = pCfg.providers[name] ?? wCfg.providers?.[name] - if (p && (!requireKey || p.apiKey)) return { name, provider: p } + if (!p || p.enabled === false) continue + if (p && (!requireKey || p.apiKey || p.runtime === "codex")) return { name, provider: p } } return null } -/** Preset providers with Anthropic-compatible endpoints. loopat uses the - * Claude Agent SDK which speaks the Anthropic Messages API — only providers - * that expose an Anthropic-compatible endpoint work directly. - * Each provider is disabled by default; the user supplies an API key. */ +/** Preset providers. Claude runtime entries use Anthropic-compatible endpoints; + * Codex runtime entries use the local Codex CLI and can rely on `codex login`. */ import { DEFAULT_PROVIDER_PRESETS } from "./presets" function buildPresetProviders(): Record { return Object.fromEntries( - DEFAULT_PROVIDER_PRESETS.map(p => [ - p.name, - { - models: p.models.map(m => normalizeModelEntry(m)), - baseUrl: p.baseUrl, - apiKey: "", - enabled: false, - } satisfies ProviderConfig, - ]), + DEFAULT_PROVIDER_PRESETS.map(p => { + const runtime = normalizeProviderRuntime(p.runtime) + return [ + p.name, + { + models: p.models.map(m => normalizeModelEntry(m)), + baseUrl: p.baseUrl, + apiKey: "", + enabled: runtime === "codex", + runtime, + } satisfies ProviderConfig, + ] + }), ) } @@ -378,10 +404,12 @@ const PERSONAL_DISK_TEMPLATE: PersonalConfigDisk = { default: DEFAULT_PROVIDER_PRESETS[0] ? `${DEFAULT_PROVIDER_PRESETS[0].name}/${normalizeModelEntry(DEFAULT_PROVIDER_PRESETS[0].models[0]).id}` : "", } for (const p of DEFAULT_PROVIDER_PRESETS) { + const runtime = normalizeProviderRuntime(p.runtime) providers[p.name] = { models: p.models.map(m => normalizeModelEntry(m)), baseUrl: p.baseUrl, - enabled: false, + runtime, + enabled: runtime === "codex", } } return providers @@ -409,9 +437,12 @@ export async function loadConfig(): Promise { if (cachedWorkspace && mtimeMs === cachedWorkspaceMtimeMs) return cachedWorkspace const raw = await readFile(path, "utf8") const parsed = JSON.parse(raw) as WorkspaceConfig + parsed.providers = { ...buildPresetProviders(), ...(parsed.providers ?? {}) } if (parsed.providers) { for (const [, p] of Object.entries(parsed.providers)) { if (p.enabled === undefined) (p as any).enabled = true + ;(p as any).runtime = normalizeProviderRuntime((p as any).runtime) + ;(p as any).models = normalizeProviderModelsFromDisk(p as any) } } cachedWorkspace = parsed @@ -512,15 +543,14 @@ export async function loadPersonalConfig( const vaultPath = join(personalVaultDir(user, vault), (p.apiKey as any).vault as string) try { apiKey = (await readFile(vaultPath, "utf8")).trim() } catch {} } - // Normalize: mixed string|object → canonical ModelEntry[]. - const models: ModelEntry[] = Array.isArray(p.models) - ? p.models.map(m => normalizeModelEntry(m as ModelEntryDisk)) - : [] + // Normalize both canonical `models` and the legacy single `model`. + const models = normalizeProviderModelsFromDisk(p) providers[name] = { models, baseUrl: p.baseUrl, apiKey, enabled: p.enabled !== false, + runtime: normalizeProviderRuntime(p.runtime), ...(typeof p.opus_model === "string" && p.opus_model ? { opus_model: p.opus_model } : {}), ...(typeof p.sonnet_model === "string" && p.sonnet_model ? { sonnet_model: p.sonnet_model } : {}), ...(typeof p.haiku_model === "string" && p.haiku_model ? { haiku_model: p.haiku_model } : {}), @@ -680,12 +710,17 @@ export async function savePersonalDisk( return { ok: false, error: `provider "${name}" must be an object` } } const p = val as ProviderConfigDisk - if (!Array.isArray(p.models) || p.models.length === 0) { + const models = normalizeProviderModelsFromDisk(p) + if (models.length === 0) { return { ok: false, error: `provider "${name}" missing models` } } + p.models = models if (typeof p.baseUrl !== "string") { return { ok: false, error: `provider "${name}" missing baseUrl` } } + if (p.runtime !== undefined && p.runtime !== "claude" && p.runtime !== "codex") { + return { ok: false, error: `provider "${name}" runtime must be "claude" or "codex"` } + } if (p.apiKey !== undefined && typeof p.apiKey !== "string" && !(typeof p.apiKey === "object" && typeof (p.apiKey as any).vault === "string")) { return { ok: false, error: `provider "${name}" apiKey must be a string or { vault }` } } @@ -705,7 +740,7 @@ export async function savePersonalDisk( for (const [name, val] of Object.entries(patch.providers)) { if (name === "default" || !val || typeof val !== "object") continue const p = val as ProviderConfigDisk - if (p.enabled !== false) { + if (p.enabled !== false && normalizeProviderRuntime(p.runtime) !== "codex") { const hasNewKey = (typeof p.apiKey === "string" && p.apiKey.length > 0) || (p.apiKey && typeof (p.apiKey as any).vault === "string") const existingEntry = disk.providers[name] const existingKey = (existingEntry && typeof existingEntry === "object") ? (existingEntry as ProviderConfigDisk).apiKey : undefined @@ -785,7 +820,7 @@ async function readPersonalDisk(user: string): Promise { */ export async function savePersonalConfig(user: string, cfg: { default?: string - providers?: Record + providers?: Record }): Promise { const disk = await readPersonalDisk(user) const existingDefault = typeof disk.providers.default === "string" ? disk.providers.default : "" @@ -818,14 +853,13 @@ export async function savePersonalConfig(user: string, cfg: { // No new key, no existing key → leave field unset (provider disabled). apiKeyField = undefined } - const models: ModelEntry[] = Array.isArray(p.models) - ? p.models.map(m => normalizeModelEntry(m as ModelEntryDisk)) - : [] + const models = normalizeProviderModelsFromDisk(p) rebuilt[name] = { baseUrl: p.baseUrl, ...(apiKeyField !== undefined ? { apiKey: apiKeyField } : {}), ...(models.length > 0 ? { models } : {}), ...(p.enabled === false ? { enabled: false } : {}), + ...(p.runtime && p.runtime !== "claude" ? { runtime: p.runtime } : {}), ...(p.opus_model ? { opus_model: p.opus_model } : {}), ...(p.sonnet_model ? { sonnet_model: p.sonnet_model } : {}), ...(p.haiku_model ? { haiku_model: p.haiku_model } : {}), @@ -871,14 +905,18 @@ export async function saveWorkspaceConfig(cfg: Partial): Promis for (const [name, p] of Object.entries(cfg.providers)) { const existingProv = merged.providers[name] const incoming = p as any - const models: ModelEntry[] = incoming.models?.length > 0 - ? incoming.models.map((m: any) => normalizeModelEntry(m as ModelEntryDisk)) - : existingProv?.models ?? [] + const incomingModels = normalizeProviderModelsFromDisk(incoming) + const models: ModelEntry[] = incomingModels.length > 0 ? incomingModels : existingProv?.models ?? [] merged.providers[name] = { models, baseUrl: incoming.baseUrl ?? existingProv?.baseUrl ?? "", apiKey: incoming.apiKey || existingProv?.apiKey || "", enabled: incoming.enabled !== undefined ? incoming.enabled : (existingProv?.enabled ?? true), + runtime: normalizeProviderRuntime(incoming.runtime ?? existingProv?.runtime), + ...(incoming.opus_model ?? existingProv?.opus_model ? { opus_model: incoming.opus_model ?? existingProv?.opus_model } : {}), + ...(incoming.sonnet_model ?? existingProv?.sonnet_model ? { sonnet_model: incoming.sonnet_model ?? existingProv?.sonnet_model } : {}), + ...(incoming.haiku_model ?? existingProv?.haiku_model ? { haiku_model: incoming.haiku_model ?? existingProv?.haiku_model } : {}), + ...(incoming.agent_model ?? existingProv?.agent_model ? { agent_model: incoming.agent_model ?? existingProv?.agent_model } : {}), } as any } } diff --git a/server/src/git-author.ts b/server/src/git-author.ts new file mode 100644 index 00000000..95ce95e4 --- /dev/null +++ b/server/src/git-author.ts @@ -0,0 +1,17 @@ +export type GitAuthor = { name?: string; email?: string } + +function firstNonEmpty(...values: Array): string | undefined { + return values.find((value) => typeof value === "string" && value.trim())?.trim() +} + +/** Resolve commit identity without replacing a valid host/repository config. */ +export function selectGitAuthor(opts: { + preferred?: GitAuthor + configured?: GitAuthor + existing?: GitAuthor +}): Required { + return { + name: firstNonEmpty(opts.preferred?.name, opts.configured?.name, opts.existing?.name) ?? "loopat", + email: firstNonEmpty(opts.preferred?.email, opts.configured?.email, opts.existing?.email) ?? "loopat@local", + } +} diff --git a/server/src/git-host.ts b/server/src/git-host.ts index 6bba1aee..a46753b7 100644 --- a/server/src/git-host.ts +++ b/server/src/git-host.ts @@ -10,6 +10,11 @@ export type HostCred = { token: string; baseUrl?: string } export type RepoRef = { owner: string; name: string } +export type DirectRepo = { + url: string + path: string + owner: string +} /** * Onboarding is fully implemented by the provider (see GitHostProvider.onboarding). @@ -95,6 +100,13 @@ export interface GitHostProvider { readonly baseUrl?: string readonly defaultRepo?: string + /** + * Optional pre-configured repository that can be reached without the host + * API. This is intended for self-hosted installations where the loopat host + * already has an SSH identity and an administrator pins one repository. + */ + directRepo?(opts: { baseUrl?: string; repoName: string }): DirectRepo | null + /** * Optional onboarding, FULLY implemented by the provider. When present, loopat * treats onboarding as MANDATORY: until it reports `done`, loop creation is @@ -142,7 +154,7 @@ export interface GitHostProvider { cred: HostCred, name: string, opts?: { private?: boolean }, - ): Promise<{ url: string; created: boolean }> + ): Promise<{ url: string; created: boolean; path?: string }> /** ③ register a deploy key on a repo (only for "ssh-deploy-key" mode). */ registerDeployKey?( diff --git a/server/src/github.ts b/server/src/github.ts index e0c494c9..1c23e662 100644 --- a/server/src/github.ts +++ b/server/src/github.ts @@ -171,6 +171,25 @@ export async function ensureCollaborator( if (r.status !== 201 && r.status !== 204) fail("add collaborator", r) } +type OnboardingProviderConfig = { + apiKey?: unknown + runtime?: unknown + enabled?: unknown + baseUrl?: string + model?: string +} + +function providerConfigs(value: unknown): Record { + return value !== null && typeof value === "object" && !Array.isArray(value) + ? value as Record + : {} +} + +function providerReady(value: OnboardingProviderConfig): boolean { + return (typeof value.apiKey === "string" && value.apiKey.trim().length > 0) || + (value.runtime === "codex" && value.enabled !== false) +} + /** The built-in GitHub provider — adapts the functions above onto GitHostProvider. */ export const githubProvider: GitHostProvider = { id: "github", @@ -183,7 +202,11 @@ export const githubProvider: GitHostProvider = { // the device dance (start/poll) and, on token, provisions the repo. Once the // repo is imported there's nothing left to gate on, so we're done. async onboarding(ctx) { + const wsProviders = providerConfigs(ctx.workspaceConfig?.providers) + const hasWorkspaceKey = Object.values(wsProviders).some(providerReady) if (!ctx.personalRepoImported) { + // Workspace-level AI, including keyless Codex, is enough for local-only loops. + if (hasWorkspaceKey) return { done: true } return { done: false, show: { @@ -196,10 +219,8 @@ export const githubProvider: GitHostProvider = { // Step 2: need at least one usable AI key. Check personal config first // (keys already expanded from vault), then workspace-shared providers so // users can skip the key-entry form when the workspace already has keys. - const providers = (ctx.config?.providers ?? {}) as Record - const hasPersonalKey = Object.values(providers).some((p) => p && typeof p.apiKey === "string" && p.apiKey.trim().length > 0) - const wsProviders = (ctx.workspaceConfig?.providers ?? {}) as Record - const hasWorkspaceKey = Object.values(wsProviders).some((p) => p && typeof p.apiKey === "string" && p.apiKey.trim().length > 0) + const providers = providerConfigs(ctx.config?.providers) + const hasPersonalKey = Object.values(providers).some(providerReady) const hasKey = hasPersonalKey || hasWorkspaceKey const anthropic = providers.anthropic ?? {} if (!hasKey) { diff --git a/server/src/gitlab.ts b/server/src/gitlab.ts new file mode 100644 index 00000000..4862ba53 --- /dev/null +++ b/server/src/gitlab.ts @@ -0,0 +1,455 @@ +/** + * GitLab-compatible integration for loopat personal storage. + * + * `baseUrl` is the human-facing site root by default (for example + * https://gitlab.com or https://gitlab.example.com). API calls derive + * `${baseUrl}/api/v4`; callers may also pass a full `/api/v4` URL for older + * configs and we will recover the site root for links / clone URLs. + */ +import { registerProvider, type DirectRepo, type GitHostProvider } from "./git-host" + +export type GitlabClient = { + token: string + webBaseUrl: string + apiBaseUrl: string +} + +export type GitlabViewer = { + /** Account namespace when available; `oauth2` is the git-auth fallback. */ + login: string + namespaceLogin?: string + id?: number | string + email?: string +} + +const DEFAULT_GITLAB_BASE_URL = process.env.LOOPAT_GITLAB_BASE_URL || "https://gitlab.com" + +type JsonObject = Record + +type GitlabProject = { + id?: number | string + name?: string + web_url?: string + http_url_to_repo?: string + path_with_namespace?: string +} + +function asObject(value: unknown): JsonObject | null { + return value !== null && typeof value === "object" && !Array.isArray(value) + ? value as JsonObject + : null +} + +function trimSlash(s: string): string { + return s.replace(/\/+$/, "") +} + +function normalizeGitlabUrls(baseUrl = DEFAULT_GITLAB_BASE_URL): { webBaseUrl: string; apiBaseUrl: string } { + const raw = trimSlash(baseUrl.trim() || DEFAULT_GITLAB_BASE_URL) + if (/\/api\/v4$/i.test(raw)) { + return { webBaseUrl: raw.replace(/\/api\/v4$/i, ""), apiBaseUrl: raw } + } + return { webBaseUrl: raw, apiBaseUrl: `${raw}/api/v4` } +} + +export function gitlabClient(token: string, baseUrl = DEFAULT_GITLAB_BASE_URL): GitlabClient { + return { token, ...normalizeGitlabUrls(baseUrl) } +} + +type GitlabResponse = { + status: number + data: T + contentType: string +} + +export type GitlabAuthMode = "query" | "private-token" | "bearer" + +export function gitlabAuthModes(): GitlabAuthMode[] { + return ["private-token", "bearer", "query"] +} + +export function gitlabRequestUrl(c: GitlabClient, path: string, authMode: GitlabAuthMode): string { + const url = new URL(`${c.apiBaseUrl}${path}`) + if (authMode === "query") url.searchParams.set("private_token", c.token) + return url.toString() +} + +async function glRequest( + c: GitlabClient, + method: string, + path: string, + authMode: GitlabAuthMode, + body?: unknown, +): Promise> { + const res = await fetch(gitlabRequestUrl(c, path, authMode), { + method, + headers: { + ...(authMode === "private-token" + ? { "PRIVATE-TOKEN": c.token } + : authMode === "bearer" + ? { Authorization: `Bearer ${c.token}` } + : {}), + Accept: "application/json", + ...(body ? { "Content-Type": "application/json" } : {}), + }, + body: body ? JSON.stringify(body) : undefined, + redirect: "manual", + }) + let data: unknown = null + const text = await res.text() + if (text) { + try { data = JSON.parse(text) } catch { data = text } + } + return { + status: res.status, + data: data as T, + contentType: res.headers.get("content-type") ?? "", + } +} + +function shouldRetryAuth(r: GitlabResponse): boolean { + if ([301, 302, 303, 307, 308, 401, 403].includes(r.status)) return true + return r.status >= 200 && r.status < 300 && + (r.contentType.includes("text/html") || typeof r.data === "string") +} + +async function gl( + c: GitlabClient, + method: string, + path: string, + body?: unknown, +): Promise> { + let last: GitlabResponse | null = null + for (const authMode of gitlabAuthModes()) { + last = await glRequest(c, method, path, authMode, body) + if (!shouldRetryAuth(last)) return last + } + if (last) return last + throw new Error("gitlab authentication modes are not configured") +} + +function formatMessage(data: unknown): string { + if (!data) return "" + if (typeof data === "string") return data.replace(/\s+/g, " ").trim().slice(0, 300) + const message = asObject(data)?.message + if (typeof message === "string") return message + if (message) { + try { return JSON.stringify(message) } catch { return String(message) } + } + try { return JSON.stringify(data) } catch { return String(data) } +} + +function fail(op: string, r: { status: number; data: unknown }): never { + const msg = formatMessage(r.data) + throw new Error(`gitlab ${op} failed (${r.status})${msg ? `: ${msg}` : ""}`) +} + +function stripGitSuffix(s: string): string { + return s.replace(/\.git$/i, "") +} + +export function repoPathFromInput(input: string, login?: string): string { + let raw = stripGitSuffix(input.trim()) + if (!raw) { + if (login) return login + throw new Error("repo name required") + } + + const sshMatch = raw.match(/^git@[^:]+:(.+)$/) + if (sshMatch) raw = sshMatch[1] ?? raw + else if (/^https?:\/\//i.test(raw)) { + try { + const url = new URL(raw) + raw = decodeURIComponent(url.pathname.replace(/^\/+/, "")) + } catch { + // Leave raw as-is; the validation below will produce a useful error. + } + } + + raw = stripGitSuffix(raw).replace(/^\/+/, "").replace(/\/+$/, "") + if (raw.includes("/")) return raw + if (!login) { + throw new Error("gitlab user response has no namespace username; enter repository as namespace/project") + } + return `${login}/${raw}` +} + +/** + * Some self-hosted GitLab deployments protect the API behind SSO while normal + * SSH git access remains available. Resolve an administrator-configured full + * project path to an SSH URL so onboarding can skip the API. + */ +export function gitlabDirectRepo(baseUrl: string | undefined, repoName: string): DirectRepo | null { + let site: URL + try { + site = new URL(normalizeGitlabUrls(baseUrl).webBaseUrl) + } catch { + return null + } + if (!site.hostname || (site.protocol !== "https:" && site.protocol !== "http:")) return null + + let path: string + try { + path = repoPathFromInput(repoName) + } catch { + return null + } + const parts = path.split("/").filter(Boolean) + const validPath = parts.length >= 2 && parts.every((part) => + part !== "." && part !== ".." && /^[a-zA-Z0-9_.-]+$/.test(part)) + if (!validPath) return null + return { + url: `git@${site.hostname.toLowerCase()}:${path}.git`, + path, + owner: parts[0]!, + } +} + +function projectName(pathWithNamespace: string): string { + const parts = pathWithNamespace.split("/").filter(Boolean) + return parts.length > 0 ? parts[parts.length - 1] : pathWithNamespace +} + +function encodeProjectId(pathWithNamespace: string): string { + return encodeURIComponent(pathWithNamespace) +} + +function recordsInUserEnvelope(data: unknown): JsonObject[] { + const root = asObject(data) + if (!root) return [] + const records: JsonObject[] = [] + const queue: Array<{ value: JsonObject; depth: number }> = [ + { value: root, depth: 0 }, + ] + const seen = new Set() + const envelopeKeys = ["data", "result", "user", "current_user", "currentUser"] + while (queue.length > 0) { + const item = queue.shift()! + if (seen.has(item.value)) continue + seen.add(item.value) + records.push(item.value) + if (item.depth >= 3) continue + for (const key of envelopeKeys) { + const nested = item.value[key] + const record = asObject(nested) + if (record) queue.push({ value: record, depth: item.depth + 1 }) + } + } + return records +} + +function firstString(records: JsonObject[], keys: string[]): string | undefined { + for (const record of records) { + for (const key of keys) { + const value = record[key] + if (typeof value === "string" && value.trim()) return value.trim() + } + } + return undefined +} + +function namespaceSafe(value: string | undefined): string | undefined { + if (!value) return undefined + return /^[a-zA-Z0-9][a-zA-Z0-9_.-]*$/.test(value) ? value : undefined +} + +/** Parse standard GitLab users and wrapped enterprise responses. */ +export function parseGitlabViewerPayload(data: unknown): GitlabViewer | null { + const records = recordsInUserEnvelope(data) + if (records.length === 0) return null + + const email = firstString(records, ["commit_email", "public_email", "email", "mail"]) + const explicitLogin = namespaceSafe(firstString(records, [ + "username", + "login", + "user_name", + "userName", + "login_name", + "loginName", + "account", + "account_name", + "accountName", + "emp_id", + "empId", + "work_no", + "workNo", + ])) + const emailLogin = namespaceSafe(email?.split("@", 1)[0]) + const nameLogin = namespaceSafe(firstString(records, ["name"])) + const namespaceLogin = explicitLogin ?? emailLogin ?? nameLogin + const id = records + .map((record) => record.id ?? record.user_id ?? record.userId) + .find((value): value is number | string => + typeof value === "number" || (typeof value === "string" && value.trim().length > 0)) + + // A successful response can omit username, but it should still contain + // some identity signal. Do not mistake an HTML/login envelope for a user. + if (!namespaceLogin && !email && id === undefined) return null + return { + login: namespaceLogin ?? "oauth2", + namespaceLogin, + id, + email, + } +} + +function projectItems(value: unknown): GitlabProject[] | null { + if (!Array.isArray(value)) return null + return value.flatMap((item) => { + const record = asObject(item) + return record ? [record as GitlabProject] : [] + }) +} + +function gitlabProjectItems(data: unknown): GitlabProject[] | null { + const direct = projectItems(data) + if (direct) return direct + const records = recordsInUserEnvelope(data) + for (const record of records) { + for (const key of ["data", "result", "projects", "items", "list"]) { + const items = projectItems(record[key]) + if (items) return items + } + } + return null +} + +export async function getGitlabViewer(c: GitlabClient): Promise { + const r = await gl(c, "GET", "/user") + if (r.status === 200) { + const viewer = parseGitlabViewerPayload(r.data) + if (viewer) return viewer + } + + // A self-hosted `/user` route can redirect to SSO even when its repository + // API accepts the token. A successful read-only project-list response proves + // the token without requiring a profile endpoint. Full namespace/project + // inputs remain unambiguous. + const probe = await gl(c, "GET", "/projects?membership=true&simple=true&per_page=1") + if (probe.status === 200 && gitlabProjectItems(probe.data) !== null) { + return { login: "oauth2" } + } + + if (r.status !== 200) fail("authenticate", r) + if (probe.status !== 200) fail("authenticate", probe) + throw new Error("gitlab authenticate failed: unrecognized user response") +} + +export async function ensureGitlabProject( + c: GitlabClient, + nameOrPath: string, + opts: { private?: boolean; description?: string } = {}, +): Promise<{ created: boolean; httpUrl: string; path: string }> { + const me = await getGitlabViewer(c) + const pathWithNamespace = repoPathFromInput(nameOrPath, me.namespaceLogin) + const existing = await gl(c, "GET", `/projects/${encodeProjectId(pathWithNamespace)}`) + if (existing.status === 200) { + const webUrl = existing.data.web_url ? `${existing.data.web_url}.git` : "" + return { + created: false, + httpUrl: existing.data.http_url_to_repo || webUrl, + path: existing.data.path_with_namespace ?? pathWithNamespace, + } + } + if (existing.status !== 404) fail("get project", existing) + + const parts = pathWithNamespace.split("/").filter(Boolean) + const path = parts.pop() + const namespacePath = parts.join("/") + if (!path) throw new Error("repo name required") + + const body: Record = { + name: path, + path, + visibility: opts.private === false ? "public" : "private", + description: opts.description ?? "loopat", + } + + // User namespace creation needs no namespace_id. Group/subgroup creation does. + if (namespacePath && namespacePath !== me.login) { + const ns = await gl<{ id?: number | string }>(c, "GET", `/namespaces/${encodeURIComponent(namespacePath)}`) + if (ns.status !== 200 || !ns.data?.id) fail("get namespace", ns) + body.namespace_id = ns.data.id + } + + const r = await gl(c, "POST", "/projects", body) + if (r.status !== 201) fail("create project", r) + const webUrl = r.data.web_url ? `${r.data.web_url}.git` : "" + return { + created: true, + httpUrl: r.data.http_url_to_repo || webUrl, + path: r.data.path_with_namespace ?? pathWithNamespace, + } +} + +export const gitlabProvider: GitHostProvider = { + id: "gitlab", + label: "GitLab", + gitAuthMode: "https-token", + baseUrl: DEFAULT_GITLAB_BASE_URL, + defaultRepo: "loopat-personal", + tokenHelp: "Create a personal/private token with api, read_repository, and write_repository scopes.", + directRepo({ baseUrl, repoName }) { + return gitlabDirectRepo(baseUrl, repoName) + }, + async authenticate(cred) { + return await getGitlabViewer(gitlabClient(cred.token, cred.baseUrl)) + }, + async ensureRepo(cred, name, opts) { + const r = await ensureGitlabProject(gitlabClient(cred.token, cred.baseUrl), name, { private: opts?.private }) + return { url: r.httpUrl, created: r.created, path: r.path } + }, + async listRepos(cred) { + const c = gitlabClient(cred.token, cred.baseUrl) + const r = await gl(c, "GET", "/projects?membership=true&simple=true&per_page=100&order_by=last_activity_at&sort=desc") + if (r.status !== 200) fail("list projects", r) + const items = gitlabProjectItems(r.data) ?? [] + return items.flatMap((project) => { + const path = project.path_with_namespace || project.name + if (!path) return [] + return [{ name: project.name || projectName(path), path }] + }) + }, + async grantAccess(cred, repo, login, level) { + const c = gitlabClient(cred.token, cred.baseUrl) + const projectPath = `${repo.owner}/${repo.name}` + const users = await gl>(c, "GET", `/users?username=${encodeURIComponent(login)}`) + if (users.status !== 200) fail("find user", users) + const userId = Array.isArray(users.data) ? users.data[0]?.id : undefined + if (!userId) throw new Error(`gitlab user not found: ${login}`) + const r = await gl(c, "POST", `/projects/${encodeProjectId(projectPath)}/members`, { + user_id: userId, + access_level: level === "write" ? 30 : 20, + }) + if (r.status !== 201 && r.status !== 409) fail("add member", r) + }, + async seedDefaults(ctx) { + const fs = await import("node:fs/promises") + const path = await import("node:path") + const { execFile } = await import("node:child_process") + const { promisify } = await import("node:util") + const run = promisify(execFile) + + const config = { + providers: { + default: "Codex/gpt-5-codex", + Codex: { + runtime: "codex", + model: "gpt-5-codex", + baseUrl: "https://api.openai.com/v1", + models: [{ id: "gpt-5-codex", maxContextTokens: 20000000 }], + enabled: true, + }, + }, + } + await fs.mkdir(path.join(ctx.repoDir, ".loopat"), { recursive: true }) + await fs.writeFile(path.join(ctx.repoDir, ".loopat", "config.json"), JSON.stringify(config, null, 2) + "\n") + + const sshDir = path.join(ctx.vaultDir, "mounts", "home", ".ssh") + await fs.mkdir(sshDir, { recursive: true }) + await run("ssh-keygen", ["-t", "ed25519", "-N", "", "-C", `loopat:${ctx.login}`, "-f", path.join(sshDir, "id_ed25519")]) + await fs.writeFile(path.join(sshDir, "config"), "Host *\n StrictHostKeyChecking accept-new\n") + }, +} + +registerProvider(gitlabProvider) diff --git a/server/src/index.ts b/server/src/index.ts index 435b4a8d..dea25743 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -40,6 +40,7 @@ import { LOOPAT_HOME, LOOPAT_INSTALL_DIR, WORKSPACE, + workspaceDir, loopContextKnowledge, loopContextNotes, loopContextPersonal, @@ -57,13 +58,21 @@ import { personalReposDir, loopsDir, } from "./paths" -import { loadConfig, loadPersonalConfig, savePersonalConfig, saveWorkspaceConfig, getActiveProvider, readPersonalDiskRaw, savePersonalDisk, describeApiKeyRef, writeVaultEnv, deleteVaultEnv, loadA2AConfig, saveA2AConfig, type ProviderConfig, type ModelEntry } from "./config" +import { loadConfig, loadPersonalConfig, savePersonalConfig, saveWorkspaceConfig, getActiveProvider, readPersonalDiskRaw, savePersonalDisk, describeApiKeyRef, writeVaultEnv, deleteVaultEnv, loadA2AConfig, saveA2AConfig, type ProviderConfig, type ProviderRuntime, type ModelEntry } from "./config" import { queryUserTokenUsage, queryWorkspaceTokenUsage, queryDailyTokenUsage, queryLoopTokenUsage } from "./usage" import { createApiToken, listApiTokens, revokeApiToken } from "./api-tokens" import { listBoards, createBoard, renameBoard, listKanbanColumns, addCard, toggleCard, deleteCard, moveCard, updateCardMeta, updateCardBlock, reorderCards, createColumn, deleteColumn, readKanbanConfig, saveColumnOrder, setColumnColor, renameColumn, assignDriverForCard, createLoopFromCard, linkLoopToCard, kanbanUserCtx } from "./kanban" import { printBootstrapBanner, printReadyLine } from "./bootstrap" -import { resolveProvider } from "./providers" +import { resolveGitHostSettings } from "./providers" import { ensureSandboxClaudeBinary } from "./claude-binary" +import { + buildCodexEnv, + buildCodexExecArgs, + codexBinary, + codexEventError, + codexModelArg, + parseCodexEvent, +} from "./codex-cli" import { serveHostExec, hostExecSocketPath } from "./host-exec" import { createUser, @@ -267,11 +276,11 @@ app.get("/api/serve/check-port", requireAuth, async (c) => { // (they carry per-user apiKeys via secrets/). Source field indicates origin. app.get("/api/providers", requireAuth, async (c) => { const wCfg = await loadConfig() - const providers: Record = {} + const providers: Record = {} if (wCfg.providers) { for (const [name, p] of Object.entries(wCfg.providers)) { const hasKey = typeof p.apiKey === "string" && p.apiKey.length > 0 - providers[name] = { models: p.models, baseUrl: p.baseUrl, source: "workspace", enabled: hasKey ? p.enabled : false, hasKey } + providers[name] = { models: p.models, baseUrl: p.baseUrl, runtime: p.runtime, source: "workspace", enabled: (hasKey || p.runtime === "codex") ? p.enabled : false, hasKey } } } // Overlay personal providers (they take precedence) @@ -283,8 +292,8 @@ app.get("/api/providers", requireAuth, async (c) => { const hasKey = typeof p.apiKey === "string" && p.apiKey.length > 0 // Only overlay if the user actually configured this provider (has a key). // Template/preset providers without a key should not shadow workspace config. - if (hasKey) { - providers[name] = { models: p.models, baseUrl: p.baseUrl, source: "personal", enabled: p.enabled !== false, hasKey } + if (hasKey || p.runtime === "codex") { + providers[name] = { models: p.models, baseUrl: p.baseUrl, runtime: p.runtime, source: "personal", enabled: p.enabled !== false, hasKey } } } active = pCfg.default || active @@ -292,6 +301,61 @@ app.get("/api/providers", requireAuth, async (c) => { return c.json({ providers, default: active }) }) +async function testCodexConnection(baseUrl: string, apiKey: string, model: string): Promise<{ ok: boolean; error?: string }> { + const args = buildCodexExecArgs({ + workdir: workspaceDir(), + modelArg: codexModelArg(model), + sandbox: "read-only", + ephemeral: true, + }) + + return await new Promise((resolve) => { + const child = spawn(codexBinary(), args, { + env: buildCodexEnv({ apiKey, baseUrl }), + stdio: ["pipe", "pipe", "pipe"], + }) + let stdout = "" + let stderr = "" + let settled = false + let timer: ReturnType | null = null + const finish = (result: { ok: boolean; error?: string }) => { + if (settled) return + settled = true + if (timer) clearTimeout(timer) + resolve(result) + } + child.stdin.end("Respond with exactly: ok") + timer = setTimeout(() => { + try { child.kill("SIGTERM") } catch {} + finish({ ok: false, error: "codex test timed out" }) + }, 60_000) + child.stdout.on("data", (b) => { stdout += b.toString("utf8") }) + child.stderr.on("data", (b) => { stderr += b.toString("utf8") }) + child.on("error", (error: NodeJS.ErrnoException) => { + finish({ + ok: false, + error: error.code === "ENOENT" ? "codex CLI not found on PATH" : error.message, + }) + }) + child.on("exit", (code) => { + let eventError = "" + for (const line of stdout.split("\n")) { + const event = parseCodexEvent(line) + if (!event) continue + eventError = codexEventError(event) ?? eventError + } + if (code !== 0 || eventError) { + finish({ + ok: false, + error: (eventError || stderr.trim() || `codex exited with code ${code}`).slice(0, 400), + }) + return + } + finish({ ok: true }) + }) + }) +} + // Test a provider + model connection by making a minimal Messages API call. // Accepts either a plain apiKey, or a provider name + source to resolve the // key server-side (so tests work for stored/encrypted keys without re-typing). @@ -302,21 +366,31 @@ app.post("/api/providers/test", requireAuth, async (c) => { if (typeof model !== "string" || !model) return c.json({ ok: false, error: "model required" }, 400) let apiKey = typeof rawApiKey === "string" ? rawApiKey.trim() : "" + let runtime: ProviderRuntime = body.runtime === "codex" ? "codex" : "claude" // Resolve key server-side when a stored (encrypted) key is being tested if (!apiKey && typeof provider === "string" && provider) { if (source === "personal") { const userId = c.get("userId") as string try { const pCfg = await loadPersonalConfig(userId) - apiKey = pCfg.providers[provider]?.apiKey ?? "" + const p = pCfg.providers[provider] + apiKey = p?.apiKey ?? "" + runtime = p?.runtime ?? runtime } catch {} } else if (source === "workspace") { try { const wCfg = await loadConfig() - apiKey = (wCfg.providers?.[provider] as any)?.apiKey ?? "" + const p = wCfg.providers?.[provider] + apiKey = (p as any)?.apiKey ?? "" + runtime = p?.runtime ?? runtime } catch {} } } + + if (runtime === "codex") { + return c.json(await testCodexConnection(baseUrl, apiKey, model)) + } + if (!apiKey) return c.json({ ok: false, error: "no API key — enter one or store it first" }, 400) try { @@ -626,11 +700,12 @@ app.get("/api/settings/personal", requireAuth, async (c) => { const userId = c.get("userId") as string const cfg = await loadPersonalConfig(userId) const tokenUsage = queryUserTokenUsage(userId) - const providers: Record = {} + const providers: Record = {} for (const [name, p] of Object.entries(cfg.providers)) { providers[name] = { models: p.models, baseUrl: p.baseUrl, + runtime: p.runtime, hasKey: !!p.apiKey, enabled: p.enabled, } @@ -1007,10 +1082,10 @@ app.post("/api/plugins/refresh", requireAuth, async (c) => { app.get("/api/settings/workspace", requireAuth, requireAdmin, async (c) => { const cfg = await loadConfig() - const providers: Record = {} + const providers: Record = {} if (cfg.providers) { for (const [name, p] of Object.entries(cfg.providers)) { - providers[name] = { models: p.models, baseUrl: p.baseUrl, hasKey: !!(p as any).apiKey, enabled: p.enabled } + providers[name] = { models: p.models, baseUrl: p.baseUrl, runtime: p.runtime, hasKey: !!(p as any).apiKey, enabled: p.enabled } } } const tokenUsage = queryWorkspaceTokenUsage() @@ -1082,9 +1157,11 @@ app.get("/api/personal/status", requireAuth, async (c) => { // Per-vault SSH public keys — what the user registers on the TEAM git host // for knowledge/notes/repos. Distinct from the deploy key (publicKey above). const vaultKeys = await listVaultPublicKeys(userId) - // The active provider comes from the extensions dir (no config.json needed), - // and supplies its own baseUrl / defaultRepo / tokenHelp defaults. - const provider = await resolveProvider() + const gitHost = await resolveGitHostSettings() + const directRepo = gitHost.provider?.directRepo?.({ + baseUrl: gitHost.baseUrl, + repoName: gitHost.defaultRepo, + }) ?? null return c.json({ userId, personalRepo: user.personalRepo ?? null, @@ -1092,10 +1169,11 @@ app.get("/api/personal/status", requireAuth, async (c) => { vaultKeys, imported, gitHost: { - provider: provider?.id ?? "github", - baseUrl: provider?.baseUrl ?? null, - defaultRepo: provider?.defaultRepo ?? "loopat-personal", - tokenHelp: provider?.tokenHelp ?? null, + provider: gitHost.providerId, + baseUrl: gitHost.baseUrl ?? null, + defaultRepo: gitHost.defaultRepo, + tokenHelp: gitHost.provider?.tokenHelp ?? null, + directRepo, }, }) }) @@ -1204,25 +1282,37 @@ app.post("/api/personal/import", requireAuth, async (c) => { return c.json({ ok: true, autoInitialized: !!r.autoInitialized, cryptKey: r.cryptKey ?? null }) }) -// POST /api/personal/github — onboard personal via a GitHub PAT (host-side -// only): create the repo, register the deploy key, clone + git-crypt. The PAT -// never enters a sandbox; runtime git uses the deploy key / vault. See -// docs/identity.md (integration contract). +// POST /api/personal/github — backward-compatible provider onboarding route. +// Token-based providers create/locate the repo through their API. An +// administrator-pinned direct repo instead uses the loopat host's SSH identity +// and never accepts its target repository from the client. app.post("/api/personal/github", requireAuth, async (c) => { const userId = c.get("userId") as string const user = await findUser(userId) if (!user) return c.json({ error: "user missing" }, 500) const body = await c.req.json().catch(() => ({})) const token = typeof body.token === "string" ? body.token.trim() : "" - if (!token) return c.json({ error: "github token required" }, 400) - // Provider + its baseUrl/defaultRepo come from the extensions dir, not - // config.json. A request body may still override any of them. - const p = await resolveProvider(typeof body.provider === "string" && body.provider.trim() ? body.provider.trim() : undefined) - const repoName = (typeof body.repoName === "string" && body.repoName.trim()) || p?.defaultRepo || "loopat-personal" - const baseUrl = (typeof body.baseUrl === "string" && body.baseUrl.trim() ? body.baseUrl.trim() : undefined) ?? p?.baseUrl + // Direct repositories are resolved only from administrator-owned settings. + // Client overrides must never select a repository reached with the host's + // SSH identity. + const configuredGitHost = await resolveGitHostSettings() + const directRepo = configuredGitHost.provider?.directRepo?.({ + baseUrl: configuredGitHost.baseUrl, + repoName: configuredGitHost.defaultRepo, + }) ?? null + const gitHost = directRepo + ? configuredGitHost + : await resolveGitHostSettings({ + provider: typeof body.provider === "string" && body.provider.trim() ? body.provider.trim() : undefined, + baseUrl: typeof body.baseUrl === "string" && body.baseUrl.trim() ? body.baseUrl.trim() : undefined, + defaultRepo: typeof body.repoName === "string" && body.repoName.trim() ? body.repoName.trim() : undefined, + }) + if (!token && !directRepo) return c.json({ error: "git host token required" }, 400) + const repoName = gitHost.defaultRepo + const baseUrl = gitHost.baseUrl const cryptKey = typeof body.cryptKey === "string" && body.cryptKey.trim() ? body.cryptKey.trim() : undefined - const provider = p?.id ?? "github" - const r = await setupPersonalViaProvider({ userId, provider, token, baseUrl, repoName, cryptKey }) + const provider = gitHost.providerId + const r = await setupPersonalViaProvider({ userId, provider, token, baseUrl, repoName, cryptKey, directRepo: directRepo ?? undefined }) if (!r.ok) { if (r.needsCryptKey) return c.json({ error: r.error, needsCryptKey: true }, 409) return c.json({ error: r.error }, 400) @@ -1237,10 +1327,12 @@ app.post("/api/personal/repos", requireAuth, async (c) => { const body = await c.req.json().catch(() => ({})) const token = typeof body.token === "string" ? body.token.trim() : "" if (!token) return c.json({ ok: false, repos: [], error: "token required" }) - // Provider + baseUrl from the extensions dir (no config.json), request may override. - const p = await resolveProvider(typeof body.provider === "string" && body.provider.trim() ? body.provider.trim() : undefined) - const provider = p?.id ?? "github" - const baseUrl = (typeof body.baseUrl === "string" && body.baseUrl.trim() ? body.baseUrl.trim() : undefined) ?? p?.baseUrl + const gitHost = await resolveGitHostSettings({ + provider: typeof body.provider === "string" && body.provider.trim() ? body.provider.trim() : undefined, + baseUrl: typeof body.baseUrl === "string" && body.baseUrl.trim() ? body.baseUrl.trim() : undefined, + }) + const provider = gitHost.providerId + const baseUrl = gitHost.baseUrl try { // Validate the token first so a bad token surfaces as an error in the token // step, instead of an empty (misleading "no repos") picker. diff --git a/server/src/loops.ts b/server/src/loops.ts index 32674a8b..41cd025d 100644 --- a/server/src/loops.ts +++ b/server/src/loops.ts @@ -46,10 +46,11 @@ import { existsSync as existsSyncBase } from "node:fs" import { loadConfig, loadPersonalConfig, loadKnowledgeConfig, writeVaultEnv } from "./config" import { ensurePersonalKeypair } from "./personal-keys" import { composeLoopClaudeConfig, writeLoopSettings } from "./compose" -import { getProvider, type OnboardingView } from "./git-host" +import { getProvider, type DirectRepo, type OnboardingView } from "./git-host" import { loadExtensionProviders, resolveProviderId, resolveProvider } from "./providers" // also registers built-in providers import { loadVaultEnvs } from "./vaults" import { withSpan } from "./tracer" +import { selectGitAuthor, type GitAuthor } from "./git-author" const execFileP = promisify(execFile) @@ -613,6 +614,8 @@ export async function setupPersonalViaProvider(opts: { baseUrl?: string repoName: string cryptKey?: string + /** Server-resolved admin configuration; never accept this from a client. */ + directRepo?: DirectRepo }): Promise< | { ok: true; repo: string; repoUrl: string; created: boolean; autoInitialized?: boolean; cryptKey?: string } | { ok: false; error: string; needsCryptKey?: boolean } @@ -624,24 +627,32 @@ export async function setupPersonalViaProvider(opts: { let login: string let email: string | undefined - try { - const auth = await provider.authenticate(cred) - login = auth.login - email = auth.email - } catch (e: any) { - return { ok: false, error: `${provider.id} auth failed: ${e?.message ?? e}` } - } + let repo: { url: string; created: boolean; path?: string } + if (opts.directRepo) { + login = opts.directRepo.owner + repo = { url: opts.directRepo.url, created: false, path: opts.directRepo.path } + } else { + try { + const auth = await provider.authenticate(cred) + login = auth.login + email = auth.email + } catch (e: any) { + return { ok: false, error: `${provider.id} auth failed: ${e?.message ?? e}` } + } - let repo: { url: string; created: boolean } - try { - repo = await provider.ensureRepo(cred, opts.repoName, { private: true }) - } catch (e: any) { - return { ok: false, error: `ensure repo failed: ${e?.message ?? e}` } + try { + repo = await provider.ensureRepo(cred, opts.repoName, { private: true }) + } catch (e: any) { + return { ok: false, error: `ensure repo failed: ${e?.message ?? e}` } + } } // Set up git auth per the provider's mode. let cloneUrl = repo.url - if (provider.gitAuthMode === "ssh-deploy-key") { + if (opts.directRepo) { + // The configured repository uses the loopat process account's SSH agent or + // default keys. No API call or generated deploy key is involved. + } else if (provider.gitAuthMode === "ssh-deploy-key") { // GitHub-style: register a loopat-generated deploy key; git clones via ssh. const { publicKey } = await ensurePersonalKeypair(opts.userId) if (publicKey && provider.registerDeployKey) { @@ -652,7 +663,7 @@ export async function setupPersonalViaProvider(opts: { } } } else { - // https-token git: https://:@host/path — GitLab/Code use the + // https-token git: https://:@host/path — GitLab uses the // username + private_token as basic auth (GitHub PAT works the same way). // Normalize http→https. (MVP: the token lands in the worktree's .git/config — // fine for a private, user-owned personal repo; a credential-helper pass can @@ -676,11 +687,18 @@ export async function setupPersonalViaProvider(opts: { login, }) : undefined - const imp = await importPersonalFromRepo(opts.userId, cloneUrl, opts.cryptKey, { name: login, email }, seed) + const imp = await importPersonalFromRepo( + opts.userId, + cloneUrl, + opts.cryptKey, + { name: login, email }, + seed, + opts.directRepo ? { sshAuthMode: "host" } : undefined, + ) if (!imp.ok) return { ok: false, error: imp.error, needsCryptKey: imp.needsCryptKey } return { ok: true, - repo: `${login}/${opts.repoName}`, + repo: repo.path ?? `${login}/${opts.repoName}`, repoUrl: repo.url, created: repo.created, autoInitialized: imp.autoInitialized, @@ -920,6 +938,7 @@ export async function importPersonalFromRepo( cryptKey?: string, author?: { name?: string; email?: string }, seed?: (repoDir: string) => Promise, + options?: { sshAuthMode?: "managed-key" | "host" }, ): Promise< | { ok: true; autoInitialized?: boolean; cryptKey?: string } | { @@ -939,22 +958,29 @@ export async function importPersonalFromRepo( return { ok: false, error: "personal/ is not empty — refusing to overwrite" } } - // https-token urls carry their own auth (https://user:token@…) and need no - // ssh deploy key; ssh urls require the loopat-managed deploy key. + // https-token URLs carry their own auth. SSH normally uses loopat's managed + // deploy key; an administrator-pinned direct repository can explicitly use + // the loopat process account's host SSH identity instead. const isHttps = /^https?:\/\//.test(repoUrl) + const sshAuthMode = options?.sshAuthMode ?? "managed-key" const priv = hostDeployKeyPath(userId) - if (!isHttps && !existsSyncBase(priv)) { + if (!isHttps && sshAuthMode === "managed-key" && !existsSyncBase(priv)) { return { ok: false, error: "deploy keypair missing — re-register" } } - // Clone into a tmp dir. ssh uses the deploy key (StrictHostKeyChecking= - // accept-new, no pre-populated known_hosts on first run); https auths via url. + // Clone into a tmp dir. SSH uses the selected managed/host identity and + // HTTPS authenticates via the URL. const tmp = await mkdtemp(join(tmpdir(), `loopat-import-${userId}-`)) - // Bootstrap: first clone of the personal repo uses the host deploy-key (no - // vault key exists yet). Every later op uses the user's vault key. - const cloneEnv = isHttps ? { ...process.env } : { ...process.env, GIT_SSH_COMMAND: personalSshCommand(userId) } + // Persist the SSH choice in local git config after cloning. Later personal + // fetch/pull/push operations read it from there, so auth stays consistent. + const cloneEnv = isHttps + ? { ...process.env } + : { ...process.env, GIT_SSH_COMMAND: await personalSshCommand(userId, sshAuthMode) } try { await execFileP("git", ["clone", "--", repoUrl, tmp], { env: cloneEnv }) + if (!isHttps) { + await execFileP("git", ["-C", tmp, "config", "loopat.personalSshAuth", sshAuthMode]) + } } catch (e: any) { await rm(tmp, { recursive: true, force: true }).catch(() => {}) const msg = (e?.stderr || e?.message || String(e)).toString().trim().split("\n").slice(-3).join(" ") @@ -1067,10 +1093,39 @@ function sshCommandForUser(userId: string, vault: string = "default"): string { * INSIDE the (now-unlocked) personal repo and only reaches the team. This avoids * the recursion of "use a key stored in the repo to reach the repo itself". */ -function personalSshCommand(userId: string): string { +let hostSshStrictMode: Promise<"accept-new" | "no"> | undefined + +async function resolveHostSshStrictMode(): Promise<"accept-new" | "no"> { + if (!hostSshStrictMode) { + hostSshStrictMode = execFileP("ssh", [ + "-G", + "-o", + "StrictHostKeyChecking=accept-new", + "localhost", + ]).then(() => "accept-new" as const, () => "no" as const) + } + return await hostSshStrictMode +} + +async function personalSshCommand( + userId: string, + authMode: "managed-key" | "host" = "managed-key", +): Promise { + if (authMode === "host") { + const strictMode = await resolveHostSshStrictMode() + return `ssh -o BatchMode=yes -o StrictHostKeyChecking=${strictMode}` + } return `ssh -i ${hostDeployKeyPath(userId)} -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=/dev/null` } +async function personalSshCommandForDir(userId: string, dir: string): Promise { + try { + const { stdout } = await execFileP("git", ["-C", dir, "config", "--get", "loopat.personalSshAuth"]) + if (stdout.trim() === "host") return await personalSshCommand(userId, "host") + } catch {} + return await personalSshCommand(userId) +} + async function swapPersonalDir( userId: string, tmp: string, @@ -1116,10 +1171,10 @@ async function autoInitGitCrypt( } } - // Local-only commit author so this doesn't depend on global git config + // Pin the resolved author locally so the bootstrap commit and every later + // personal-repo commit satisfy the same host push rules. try { - await execFileP("git", ["-C", repoDir, "config", "user.email", author?.email ?? "loopat@local"]) - await execFileP("git", ["-C", repoDir, "config", "user.name", author?.name ?? "loopat"]) + await configureGitCommitAuthor(repoDir, author) } catch (e: any) { return { ok: false, error: `git config failed: ${e?.message ?? e}` } } @@ -1231,7 +1286,7 @@ async function autoInitGitCrypt( try { await execFileP("git", ["-C", repoDir, "push", "origin", `HEAD:${branch}`], { - env: { ...process.env, GIT_SSH_COMMAND: personalSshCommand(userId) }, + env: { ...process.env, GIT_SSH_COMMAND: await personalSshCommandForDir(userId, repoDir) }, }) } catch (e: any) { await rollbackSavedKey(userId) @@ -1290,6 +1345,32 @@ export type PersonalDirtyStatus = { hasRemote: boolean } +async function readGitConfig(dir: string, key: string): Promise { + try { + const { stdout } = await execFileP("git", ["-C", dir, "config", "--get", key]) + return stdout.trim() || undefined + } catch { + return undefined + } +} + +async function configureGitCommitAuthor(dir: string, preferred?: GitAuthor): Promise> { + const author = selectGitAuthor({ + preferred, + configured: { + name: process.env.LOOPAT_GIT_AUTHOR_NAME, + email: process.env.LOOPAT_GIT_AUTHOR_EMAIL, + }, + existing: { + name: await readGitConfig(dir, "user.name"), + email: await readGitConfig(dir, "user.email"), + }, + }) + await execFileP("git", ["-C", dir, "config", "user.name", author.name]) + await execFileP("git", ["-C", dir, "config", "user.email", author.email]) + return author +} + /** * Inspect personal//: how many uncommitted worktree changes, how many * commits not reachable from any remote-tracking branch. Used as the @@ -1314,7 +1395,7 @@ export async function inspectPersonalDirty(userId: string): Promise { try { - try { await execFileP("git", ["-C", dir, "config", "user.email"]) } - catch { await execFileP("git", ["-C", dir, "config", "user.email", "loopat@local"]) } - try { await execFileP("git", ["-C", dir, "config", "user.name"]) } - catch { await execFileP("git", ["-C", dir, "config", "user.name", "loopat"]) } + await configureGitCommitAuthor(dir) await execFileP("git", ["-C", dir, "add", "-A"]) } catch (e: any) { return { ok: false, error: `git add failed: ${e?.stderr ?? e?.message ?? e}` } @@ -1537,7 +1613,7 @@ export async function pullPersonalFromRemote( try { await execFileP("git", ["-C", dir, "rebase", "--abort"], { env: silent }) } catch {} try { await execFileP("git", ["-C", dir, "merge", "--abort"], { env: silent }) } catch {} await execFileP("git", ["-C", dir, "fetch", "origin"], { - env: { ...silent, GIT_SSH_COMMAND: personalSshCommand(userId) }, timeout: 30_000, + env: { ...silent, GIT_SSH_COMMAND: await personalSshCommandForDir(userId, dir) }, timeout: 30_000, }) await execFileP("git", ["-C", dir, "reset", "--hard", `origin/${branch}`], { env: silent }) await execFileP("git", ["-C", dir, "clean", "-fd"], { env: silent }) @@ -1550,7 +1626,7 @@ export async function pullPersonalFromRemote( // Normal pull: commit local edits so the tree is clean, then rebase onto origin. const c = await commitLocalChanges(dir, "loopat: local personal edits") if (!c.ok) return { ok: false, error: c.error } - const reb = await rebaseOntoOrigin(dir, branch, personalSshCommand(userId)) + const reb = await rebaseOntoOrigin(dir, branch, await personalSshCommandForDir(userId, dir)) if (!reb.ok) { if ("conflict" in reb) return { ok: false, error: "conflict with remote", conflict: true, files: reb.files } return { ok: false, error: reb.error } @@ -1587,14 +1663,15 @@ export async function pushPersonalToRemote( const c = await commitLocalChanges(dir, "loopat: sync personal vault") if (!c.ok) return { ok: false, error: c.error } - const reb = await rebaseOntoOrigin(dir, branch, personalSshCommand(userId)) + const sshCommand = await personalSshCommandForDir(userId, dir) + const reb = await rebaseOntoOrigin(dir, branch, sshCommand) if (!reb.ok) { if ("conflict" in reb) return { ok: false, error: "conflict with remote", conflict: true, files: reb.files } return { ok: false, error: reb.error } } try { await execFileP("git", ["-C", dir, "push", "origin", `HEAD:${branch}`], { - env: { ...process.env, GIT_SSH_COMMAND: personalSshCommand(userId) }, + env: { ...process.env, GIT_SSH_COMMAND: sshCommand }, }) } catch (e: any) { const stderr = (e?.stderr ?? "").toString().trim() diff --git a/server/src/podman.ts b/server/src/podman.ts index 8c357bef..c7343936 100644 --- a/server/src/podman.ts +++ b/server/src/podman.ts @@ -589,12 +589,17 @@ export async function probePodman(): Promise { try { const { stdout } = await runPodman(["--version"]) const version = stdout.trim() + if (process.platform === "darwin") { + await runPodman(["info"]) + } return { ok: true, version } } catch (e: any) { return { ok: false, hint: e?.message?.includes("not found") - ? "install with: sudo apt install podman uidmap fuse-overlayfs" + ? (process.platform === "darwin" ? "install with: brew install podman" : "install with: sudo apt install podman uidmap fuse-overlayfs") + : process.platform === "darwin" + ? `podman machine unavailable: ${e?.message ?? e}; run: podman machine init && podman machine start` : `podman probe failed: ${e?.message ?? e}`, } } diff --git a/server/src/presets.ts b/server/src/presets.ts index daeb311f..6329cfc4 100644 --- a/server/src/presets.ts +++ b/server/src/presets.ts @@ -1,4 +1,8 @@ -export const DEFAULT_PROVIDER_PRESETS: Array<{ name: string; baseUrl: string; models: Array }> = [ +export const DEFAULT_PROVIDER_PRESETS: Array<{ name: string; baseUrl: string; runtime?: "claude" | "codex"; models: Array }> = [ + { name: "Codex", baseUrl: "https://api.openai.com/v1", runtime: "codex", + models: [ + { id: "gpt-5-codex", maxContextTokens: 20_000_000 }, + ]}, { name: "Anthropic", baseUrl: "https://api.anthropic.com", models: [ "claude-sonnet-4-20250514", @@ -39,4 +43,4 @@ export const DEFAULT_MISE_TOOL_PRESETS: Array<{ name: string; suggestedVersion: { name: "ripgrep", suggestedVersion: "14.1", description: "Line-oriented search tool", backend: "aqua:BurntSushi/ripgrep" }, { name: "fd", suggestedVersion: "10.2", description: "Fast file finder", backend: "aqua:sharkdp/fd" }, { name: "jq", suggestedVersion: "1.7", description: "Command-line JSON processor", backend: "aqua:jqlang/jq" }, -] \ No newline at end of file +] diff --git a/server/src/providers.ts b/server/src/providers.ts index 266b78ba..e745458f 100644 --- a/server/src/providers.ts +++ b/server/src/providers.ts @@ -16,6 +16,7 @@ import { registerProvider, getProvider, type GitHostProvider } from "./git-host" import { extensionsProvidersDir } from "./paths" import "./github" // built-in, open-source +import "./gitlab" // built-in GitLab-compatible host let extLoaded = false // Ids of providers loaded from extension files (NOT the built-in github). A @@ -60,7 +61,17 @@ export async function loadExtensionProviders(): Promise { export async function resolveProviderId(requested?: string): Promise { await loadExtensionProviders() if (extensionProviderIds.length > 0) return extensionProviderIds[0] - return requested || "github" + if (requested) return requested + const envProvider = process.env.LOOPAT_GIT_HOST_PROVIDER || process.env.LOOPAT_GIT_PROVIDER + if (envProvider) return envProvider + try { + const { loadConfig } = await import("./config") + const cfg = await loadConfig() + if (cfg.gitHost?.provider) return cfg.gitHost.provider + } catch { + // Fall back to GitHub below when config is unavailable or malformed. + } + return "github" } /** Resolve and return the active provider object (see resolveProviderId). Its @@ -68,3 +79,35 @@ export async function resolveProviderId(requested?: string): Promise { export async function resolveProvider(requested?: string): Promise { return getProvider(await resolveProviderId(requested)) } + +export type GitHostSettingsRequest = { + provider?: string + baseUrl?: string + defaultRepo?: string +} + +export type GitHostSettings = { + provider: GitHostProvider | undefined + providerId: string + baseUrl?: string + defaultRepo: string +} + +export async function resolveGitHostSettings(requested: GitHostSettingsRequest = {}): Promise { + const envBaseUrl = process.env.LOOPAT_GIT_HOST_BASE_URL || process.env.LOOPAT_GITLAB_BASE_URL + const envDefaultRepo = process.env.LOOPAT_GIT_HOST_DEFAULT_REPO || process.env.LOOPAT_PERSONAL_REPO_NAME + let cfg: { gitHost?: GitHostSettingsRequest } = {} + try { + const { loadConfig } = await import("./config") + cfg = await loadConfig() + } catch { + cfg = {} + } + + const providerId = requested.provider || process.env.LOOPAT_GIT_HOST_PROVIDER || process.env.LOOPAT_GIT_PROVIDER || cfg.gitHost?.provider || "github" + const provider = await resolveProvider(providerId) + const resolvedProviderId = provider?.id ?? providerId + const baseUrl = requested.baseUrl || envBaseUrl || cfg.gitHost?.baseUrl || provider?.baseUrl + const defaultRepo = requested.defaultRepo || envDefaultRepo || cfg.gitHost?.defaultRepo || provider?.defaultRepo || "loopat-personal" + return { provider, providerId: resolvedProviderId, baseUrl, defaultRepo } +} diff --git a/server/src/session.ts b/server/src/session.ts index bd3432f7..daf001b0 100644 --- a/server/src/session.ts +++ b/server/src/session.ts @@ -4,7 +4,7 @@ import { appendFile, readFile, readdir, rm, writeFile, mkdir } from "node:fs/pro import { createWriteStream, mkdirSync, existsSync } from "node:fs" import { randomUUID } from "node:crypto" import { join } from "node:path" -import { loopClaudeDir, loopDir, loopHistoryPath, personalSkillsDir, workspaceTeamSkillsDir } from "./paths" +import { loopClaudeDir, loopDir, loopHistoryPath, loopWorkdir, personalSkillsDir, workspaceTeamSkillsDir } from "./paths" import { appendLoopUsage, appendLoopUsageClear, insertUsageDb, type UsageEntry } from "./usage" import { resolveSandboxClaudeBinary } from "./claude-binary" import { loadConfig, loadPersonalConfig, parseDefault, getModelByTier, pickProvider, type ProviderConfig } from "./config" @@ -19,6 +19,18 @@ import { updateLoopStatus, setLoopPhase } from "./loop-status" import { tracer, withSpan } from "./tracer" import { SpanStatusCode, type Span } from "@opentelemetry/api" import { maybeAutoName } from "./auto-name" +import { + buildCodexEnv, + buildCodexExecArgs, + codexBinary, + codexCompletedItem, + codexEventError, + codexModelArg, + codexThreadId, + codexTurnUsage, + parseCodexEvent, + type CodexUsage, +} from "./codex-cli" // Tests override LOOPAT_CLAUDE_BIN to point at a mock binary (a script that // reads stream-json from stdin and writes canned messages back) so we can @@ -202,6 +214,16 @@ type QueuedMessage = { export type LoopSessionMessageListener = (msg: any) => void +type CodexRuntimeContext = { + loopId: string + driver: string + providerName: string + provider: ProviderConfig + modelId: string + modelArg?: string + loopatAppend: string +} + class LoopSession { id: string private q: Query | null = null @@ -229,6 +251,10 @@ class LoopSession { private usageSession = 0 private currentDriver: string | null = null private gateway: LoopGateway | null = null + private codexRuntime: CodexRuntimeContext | null = null + private codexProc: ReturnType | null = null + private interruptedCodexProcesses = new WeakSet>() + private destroyed = false constructor(id: string) { this.id = id @@ -243,9 +269,10 @@ class LoopSession { } private scheduleIdleCleanup() { + if (this.destroyed) return if (this.idleTimer) return if (this.subscribers.size > 0) return - if (this.consuming) return // never interrupt an active generation + if (this.consuming || this.generating) return // never interrupt an active generation const tag = this.id.slice(0, 8) this.idleTimer = setTimeout(() => { this.idleTimer = null @@ -256,6 +283,22 @@ class LoopSession { }, IDLE_TIMEOUT_MS) } + private autoNameLoop(): void { + maybeAutoName(this.id).then(async (didName) => { + if (!didName) return + const fresh = await getLoop(this.id) + if (fresh) this.broadcast({ type: "meta_updated", meta: fresh }) + }).catch(() => {}) + } + + private stopCodexProcess(): void { + const proc = this.codexProc + this.codexProc = null + if (!proc) return + this.interruptedCodexProcesses.add(proc) + try { proc.kill("SIGTERM") } catch {} + } + private async resolveProvider(meta: { createdBy: string; driver?: string; config?: { vault?: string } }, candidateNames: (string | null | undefined)[], requireKey: boolean): Promise<{ name: string; provider: ProviderConfig } | null> { const pCfg = await loadPersonalConfig(effectiveDriver(meta), meta.config?.vault) const wCfg = await loadConfig() @@ -263,11 +306,8 @@ class LoopSession { } /** - * Set the active provider. Takes effect on the next user message — each - * turn spawns a fresh claude binary via `ensureStarted`, which calls - * `buildLoopEnv` with the current `providerOverride`. No need to - * interrupt the running turn; the new provider applies naturally when - * the current response finishes and the next message triggers a new spawn. + * Set the active provider. The current runtime is stopped so the next user + * message resolves the new provider and starts the matching runtime. */ setProvider(name: string | null) { this.providerOverride = name @@ -290,8 +330,8 @@ class LoopSession { setAt: this.goalSetAt, status: this.goalStatus, }) - if (this.q) { - // Re-compose: the next ensureStarted picks up the goal via buildLoopatAppend. + if (this.q || this.codexRuntime) { + // The next ensureStarted rebuilds the system prompt with the new goal. this.restartOnNextMessage() } } @@ -314,14 +354,9 @@ class LoopSession { } /** - * Interrupt the current `query()` and clear `this.q`, so the next user - * message triggers a fresh `ensureStarted()` — picking up changes to env - * vars, provider config, **mcpServers**, etc. Conversation history is - * preserved because the SDK reads its session JSONL from disk on respawn - * (`continue: true` when `hasPriorSdkSession` is true). - * - * Idempotent: calling on a session that doesn't currently hold a query is - * a no-op. Fire-and-forget; the interrupt runs in the background. + * Stop the active runtime so the next user message re-runs `ensureStarted()`. + * Claude resumes from its SDK JSONL; Codex resumes from its persisted thread. + * Fire-and-forget and idempotent. */ restartOnNextMessage() { if (this.q) { @@ -330,6 +365,8 @@ class LoopSession { this.input = pushIterable() dying.interrupt().catch(() => {}) } + this.stopCodexProcess() + this.codexRuntime = null } private async loadHistoryFromDisk() { @@ -347,10 +384,9 @@ class LoopSession { } private async ensureStarted() { - if (this.q) return + if (this.q || this.codexRuntime) return return withSpan("ensureStarted", async (rootSpan) => { rootSpan.setAttribute("loop.id", this.id.slice(0, 8)) - const shouldContinue = await hasPriorSdkSession(this.id) const meta = await getLoop(this.id) if (!meta) { throw new Error(`loop ${this.id} meta missing`) @@ -368,7 +404,7 @@ class LoopSession { ], true) }) if (!resolved) { - throw new Error(`no provider with a valid apiKey for vault "${meta.config?.vault ?? "default"}" — set one in personal/${driver}/.loopat/vaults/${meta.config?.vault ?? "default"}/envs/`) + throw new Error(`no enabled provider is ready for vault "${meta.config?.vault ?? "default"}" — store an API key or enable a Codex runtime provider`) } const providerName = resolved.name const provider = resolved.provider @@ -376,6 +412,45 @@ class LoopSession { const loopatAppend = await buildLoopatAppend(meta) const loopId = this.id + let modelId: string | undefined = meta.config?.default_model_id + if (!modelId) { + const pCfg = await loadPersonalConfig(driver, meta.config?.vault) + const defaultParsed = parseDefault(pCfg.default) + if (defaultParsed.modelId && defaultParsed.providerName === providerName) { + modelId = defaultParsed.modelId + } + } + const activeModel = (modelId ? provider.models.find(m => m.id === modelId) : undefined) + ?? provider.models[0] + const activeModelId = activeModel?.id ?? modelId ?? "" + const autoCompactWindow = activeModel?.maxContextTokens + + // Codex is a host CLI runtime. It does not need Claude settings, plugins, + // podman, or the Anthropic egress gateway below. + if (provider.runtime === "codex") { + this.codexRuntime = { + loopId, + driver, + providerName, + provider, + modelId: activeModelId, + modelArg: codexModelArg(activeModelId), + loopatAppend, + } + this.broadcast({ + type: "provider", + name: providerName, + model: this.codexRuntime.modelId, + models: provider.models, + contextWindow: resolveContextWindow(provider, this.codexRuntime.modelId), + runtime: "codex", + }) + return + } + + this.codexRuntime = null + const shouldContinue = await hasPriorSdkSession(this.id) + // Compose runs ONCE at loop creation (loops.ts:createLoop). At spawn we // only re-compose if the snapshot is missing — this happens for loops // created before the snapshot model landed, and self-heals on first spawn. @@ -430,17 +505,6 @@ class LoopSession { if (!this.gateway) this.gateway = startLoopGateway(loopId, extraEnv.ANTHROPIC_BASE_URL ?? provider.baseUrl, !!process.env.LOOPAT_EGRESS_TRACE) extraEnv.ANTHROPIC_BASE_URL = `http://host.containers.internal:${this.gateway.port}` - let modelId: string | undefined = meta.config?.default_model_id - if (!modelId) { - const pCfg = await loadPersonalConfig(driver, meta.config?.vault) - const defaultParsed = parseDefault(pCfg.default) - if (defaultParsed.modelId && defaultParsed.providerName === providerName) { - modelId = defaultParsed.modelId - } - } - const activeModel = (modelId ? provider.models.find(m => m.id === modelId) : undefined) - ?? provider.models[0] - const autoCompactWindow = activeModel?.maxContextTokens // Ensure the per-loop podman container exists and is running. Idempotent: // if the container is already up with the same config-hash, no-op. let building = false @@ -759,14 +823,7 @@ class LoopSession { resultReceived = true this.turnSpan?.end() this.turnSpan = null - // Fire-and-forget: try to auto-name the loop if title is still - // "untitled". maybeAutoName() is fully idempotent + best-effort - // (no-ops if title is already set or user has opted out). - maybeAutoName(this.id).then(async (didName) => { - if (!didName) return - const fresh = await getLoop(this.id) - if (fresh) this.broadcast({ type: "meta_updated", meta: fresh }) - }).catch(() => {}) + this.autoNameLoop() } else if ( // Inject queued messages at tool-result boundaries — matching // real Claude Code's per-step queue consumption. @@ -1094,6 +1151,7 @@ class LoopSession { model: activeModelId, models: resolved.provider.models, contextWindow: resolveContextWindow(resolved.provider, activeModelId), + runtime: resolved.provider.runtime, })) } else { console.warn(`[loop:${this.id.slice(0, 8)}] no provider found in personal or workspace config`) @@ -1308,13 +1366,232 @@ class LoopSession { this.history.push(userMsg) this.persist(userMsg) this.broadcast(userMsg) + if (this.codexRuntime) { + await this.runCodexTurn(text, this.codexRuntime, images) + return + } this.input.push(userMsg) } + private codexThreadPath(): string { + return join(loopDir(this.id), "codex-thread.json") + } + + private async loadCodexThreadId(): Promise { + try { + const j = JSON.parse(await readFile(this.codexThreadPath(), "utf8")) + return typeof j?.threadId === "string" && j.threadId ? j.threadId : null + } catch { + return null + } + } + + private async saveCodexThreadId(threadId: string): Promise { + await mkdir(loopDir(this.id), { recursive: true }) + await writeFile(this.codexThreadPath(), JSON.stringify({ threadId }, null, 2) + "\n") + } + + private buildCodexPrompt(text: string, ctx: CodexRuntimeContext, images?: ImageInput[]): string { + const imageNote = images && images.length > 0 + ? `\n\n[loopat] ${images.length} image attachment(s) were omitted because this Codex runtime adapter only forwards text today. Ask the user to describe or reattach them as files if they matter.` + : "" + return [ + ctx.loopatAppend.trim(), + `\n\n[loopat] You are running inside loopat via OpenAI Codex CLI. The workspace root is:\n${loopWorkdir(ctx.loopId)}\n`, + "Reply normally to the user. If you edit files or run commands, summarize the important results.", + "\n\nUser message:\n", + text, + imageNote, + ].join("") + } + + private async runCodexTurn(text: string, ctx: CodexRuntimeContext, images?: ImageInput[]): Promise { + const loopId = ctx.loopId + const tag = loopId.slice(0, 8) + const startedAt = Date.now() + this.generating = true + this.queueProcessing = false + updateLoopStatus(loopId, "Codex running...") + + const workdir = loopWorkdir(loopId) + const prompt = this.buildCodexPrompt(text, ctx, images) + const threadId = await this.loadCodexThreadId() + const codexBin = codexBinary() + const args = buildCodexExecArgs({ workdir, threadId, modelArg: ctx.modelArg }) + + mkdirSync(loopDir(loopId), { recursive: true }) + const stderrLogPath = join(loopDir(loopId), "stderr.log") + const stderrFile = createWriteStream(stderrLogPath, { flags: "a" }) + stderrFile.write(`\n=== ${new Date().toISOString()} codex spawn ===\n`) + stderrFile.write(`binary: ${codexBin}\nargv: ${args.map((a) => (a.includes(" ") ? JSON.stringify(a) : a)).join(" ")}\n`) + + const proc = nodeSpawn(codexBin, args, { + cwd: workdir, + env: buildCodexEnv({ + apiKey: ctx.provider.apiKey, + baseUrl: ctx.provider.baseUrl, + }), + stdio: ["pipe", "pipe", "pipe"], + }) + this.codexProc = proc + + let stdoutBuf = "" + let stderrBuf = "" + let eventError = "" + let sawTurnCompleted = false + const turnUsages: CodexUsage[] = [] + + const handleLine = (line: string) => { + const trimmed = line.trim() + if (!trimmed) return + const event = parseCodexEvent(trimmed) + if (!event) { + if (DEBUG) console.error(`[codex:${tag}:stdout] ${trimmed}`) + return + } + + const parsedError = codexEventError(event) + if (parsedError) { + eventError = parsedError + stderrFile.write(`[codex event] ${parsedError}\n`) + return + } + + const startedThreadId = codexThreadId(event) + if (startedThreadId) { + this.saveCodexThreadId(startedThreadId).catch(() => {}) + return + } + + if (event.type === "turn.started") { + this.broadcast({ type: "system", subtype: "init", runtime: "codex", uuid: randomUUID() }) + return + } + + const item = codexCompletedItem(event) + if (item) { + if (item.type === "agent_message" && item.text?.trim()) { + if (this.ttfbSpan) { + this.ttfbSpan.end() + this.ttfbSpan = null + } + const assistantMsg = { + type: "assistant" as const, + message: { role: "assistant" as const, content: [{ type: "text", text: item.text }] }, + parent_tool_use_id: null, + uuid: randomUUID(), + } + this.history.push(assistantMsg as any) + this.persist(assistantMsg) + this.broadcast(assistantMsg) + this.updateStatus(assistantMsg) + } else if (DEBUG) { + console.error(`[codex:${tag}] item.completed ${item.type}`) + } + return + } + + if (event.type === "turn.completed") { + sawTurnCompleted = true + const usage = codexTurnUsage(event) + if (usage) turnUsages.push(usage) + return + } + } + + proc.stdout?.on("data", (chunk: Buffer) => { + stdoutBuf += chunk.toString("utf8") + const lines = stdoutBuf.split("\n") + stdoutBuf = lines.pop() ?? "" + for (const line of lines) handleLine(line) + }) + + proc.stderr?.on("data", (chunk: Buffer) => { + stderrFile.write(chunk) + stderrBuf += chunk.toString("utf8") + const text = chunk.toString("utf8") + for (const line of text.split("\n")) { + if (line.trim()) console.error(`[codex:${tag}:stderr] ${line}`) + } + }) + + proc.stdin?.end(prompt) + + const code = await new Promise((resolve, reject) => { + proc.on("error", reject) + proc.on("exit", (c) => resolve(c)) + }).catch((error: unknown) => { + const message = error instanceof Error ? error.message : String(error) + stderrFile.write(`spawn error: ${message}\n`) + return -1 + }) + + if (stdoutBuf.trim()) handleLine(stdoutBuf) + stderrFile.end(`=== exit code=${code} ===\n`) + const interrupted = this.interruptedCodexProcesses.delete(proc) + if (this.codexProc === proc) this.codexProc = null + + if (!interrupted && (code !== 0 || eventError)) { + const detail = eventError || stderrBuf.trim() || "see stderr.log" + const err = { + type: "error", + message: code === 0 + ? `Codex turn failed: ${detail.slice(0, 1000)}` + : `Codex exited with code ${code}: ${detail.slice(0, 1000)}`, + } + this.turnSpan?.recordException(new Error(err.message)) + this.turnSpan?.setStatus({ code: SpanStatusCode.ERROR, message: err.message }) + this.history.push(err as any) + this.persist(err) + this.broadcast(err) + } + + const lastUsage = turnUsages.at(-1) + const usage = lastUsage + ? { + input_tokens: lastUsage.inputTokens, + output_tokens: lastUsage.outputTokens, + cache_read_input_tokens: lastUsage.cachedInputTokens, + } + : undefined + const result = { + type: "result" as const, + ...(usage ? { usage } : {}), + ...(lastUsage ? { + modelUsage: { + [ctx.modelId || ctx.providerName]: { + inputTokens: lastUsage.inputTokens, + outputTokens: lastUsage.outputTokens, + cacheReadInputTokens: lastUsage.cachedInputTokens, + cacheCreationInputTokens: 0, + }, + }, + } : {}), + duration_ms: Date.now() - startedAt, + runtime: "codex", + } + this.history.push(result as any) + this.persist(result) + if ((result as any).modelUsage) this.persistUsage(result as any) + this.broadcast(result) + this.turnSpan?.end() + this.turnSpan = null + this.ttfbSpan?.end() + this.ttfbSpan = null + if (sawTurnCompleted) this.autoNameLoop() + + this.generating = false + this.queueProcessing = false + updateLoopStatus(loopId, sawTurnCompleted ? "Ready" : interrupted ? "Interrupted" : "Codex stopped") + this.processNextInQueue() + this.scheduleIdleCleanup() + } + /** Process the next queued message. Called from consume()'s finally block * after each generation completes. Only starts the next message; subsequent * messages are handled recursively by consume()'s finally. */ private processNextInQueue() { + if (this.destroyed) return if (this.queueProcessing) return // already processing if (this.messageQueue.length === 0) { this.broadcast({ type: "queue_update", queue: [] }) @@ -1367,8 +1644,10 @@ class LoopSession { } async interrupt() { - this.generating = false + const codexWasRunning = this.codexProc !== null + if (!codexWasRunning) this.generating = false if (this.q) await this.q.interrupt().catch(() => {}) + this.stopCodexProcess() } /** Background in-flight foreground tasks (Bash commands + subagents) so the @@ -1403,6 +1682,7 @@ class LoopSession { /** Tear down the SDK process and disconnect all subscribers. Used when a * loop is archived so no orphaned processes remain. */ async destroy() { + this.destroyed = true this.cancelIdleCleanup() this.generating = false this.queueProcessing = false @@ -1417,6 +1697,8 @@ class LoopSession { try { await this.q.interrupt() } catch {} this.q = null } + this.stopCodexProcess() + this.codexRuntime = null for (const [, pending] of this.pendingQuestions) { pending.reject(new Error("loop archived")) } @@ -1522,6 +1804,8 @@ class LoopSession { try { await this.q.interrupt() } catch {} this.q = null } + this.stopCodexProcess() + await rm(this.codexThreadPath(), { force: true }).catch(() => {}) // 2. Drop SDK context without deleting history. Touch an empty new // jsonl in each existing encoded-cwd subdir so --continue picks it. // If no subdir exists yet (no SDK has spawned in this loop), the diff --git a/server/test/codex-cli.test.ts b/server/test/codex-cli.test.ts new file mode 100644 index 00000000..38644ccf --- /dev/null +++ b/server/test/codex-cli.test.ts @@ -0,0 +1,116 @@ +import { describe, expect, test } from "bun:test" +import { + buildCodexEnv, + buildCodexExecArgs, + codexModelArg, + codexCompletedItem, + codexEventError, + codexTurnUsage, + parseCodexEvent, +} from "../src/codex-cli" + +describe("Codex CLI adapter", () => { + test("builds fresh and resumed commands for non-git workspaces", () => { + expect(buildCodexExecArgs({ workdir: "/tmp/loop" })).toEqual([ + "exec", + "--json", + "--sandbox", + "workspace-write", + "--cd", + "/tmp/loop", + "--skip-git-repo-check", + "-", + ]) + + expect(buildCodexExecArgs({ + workdir: "/tmp/loop", + threadId: "thread-id", + modelArg: "gpt-test", + })).toEqual([ + "exec", + "resume", + "--json", + "--skip-git-repo-check", + "-m", + "gpt-test", + "thread-id", + "-", + ]) + }) + + test("builds an ephemeral read-only connection probe", () => { + expect(buildCodexExecArgs({ + workdir: "/tmp/workspace", + modelArg: "gpt-test", + sandbox: "read-only", + ephemeral: true, + })).toEqual([ + "exec", + "--json", + "--sandbox", + "read-only", + "--ephemeral", + "--cd", + "/tmp/workspace", + "--skip-git-repo-check", + "-m", + "gpt-test", + "-", + ]) + }) + + test("keeps model selection independent of API-key auth", () => { + expect(codexModelArg("gpt-test")).toBe("gpt-test") + expect(codexModelArg(" gpt-test ")).toBe("gpt-test") + expect(codexModelArg("")).toBeUndefined() + expect(codexModelArg(null)).toBeUndefined() + }) + + test("builds an isolated provider environment", () => { + expect(buildCodexEnv({ + apiKey: "test-key", + baseUrl: "https://example.test/v1/", + }, { PATH: "/bin" })).toEqual({ + PATH: "/bin", + CODEX_API_KEY: "test-key", + OPENAI_API_KEY: "test-key", + CODEX_BASE_URL: "https://example.test/v1", + OPENAI_BASE_URL: "https://example.test/v1", + }) + }) + + test("parses completed items and usage", () => { + const item = parseCodexEvent(JSON.stringify({ + type: "item.completed", + item: { type: "agent_message", text: "done" }, + })) + expect(item && codexCompletedItem(item)).toEqual({ type: "agent_message", text: "done" }) + + const completed = parseCodexEvent(JSON.stringify({ + type: "turn.completed", + usage: { input_tokens: 10, output_tokens: 2, cached_input_tokens: 7 }, + })) + expect(completed && codexTurnUsage(completed)).toEqual({ + inputTokens: 10, + outputTokens: 2, + cachedInputTokens: 7, + }) + }) + + test("extracts nested API failures from stdout JSONL", () => { + expect(codexEventError({ + type: "turn.failed", + error: { + message: JSON.stringify({ + type: "error", + status: 400, + error: { message: "The model requires a newer version of Codex." }, + }), + }, + })).toBe("The model requires a newer version of Codex.") + expect(codexEventError({ type: "error", message: "authentication failed" })) + .toBe("authentication failed") + expect(codexEventError({ type: "turn.completed" })).toBeNull() + expect(parseCodexEvent("not json")).toBeNull() + }) +}) diff --git a/server/test/git-author.test.ts b/server/test/git-author.test.ts new file mode 100644 index 00000000..3b9a0b3c --- /dev/null +++ b/server/test/git-author.test.ts @@ -0,0 +1,27 @@ +import { describe, expect, test } from "bun:test" +import { selectGitAuthor } from "../src/git-author" + +describe("personal repository git author", () => { + test("uses platform identity before admin and host defaults", () => { + expect(selectGitAuthor({ + preferred: { name: "Alice", email: "alice@example.com" }, + configured: { name: "Admin", email: "admin@example.com" }, + existing: { name: "Host", email: "host@example.com" }, + })).toEqual({ name: "Alice", email: "alice@example.com" }) + }) + + test("uses administrator override when the provider omits email", () => { + expect(selectGitAuthor({ + preferred: { name: "Alice" }, + configured: { email: "alice@example.com" }, + existing: { name: "Host", email: "host@example.com" }, + })).toEqual({ name: "Alice", email: "alice@example.com" }) + }) + + test("preserves the repository or host git identity before local fallback", () => { + expect(selectGitAuthor({ + preferred: { name: "Alice" }, + existing: { name: "Host", email: "alice@example.com" }, + })).toEqual({ name: "Alice", email: "alice@example.com" }) + }) +}) diff --git a/server/test/gitlab.test.ts b/server/test/gitlab.test.ts new file mode 100644 index 00000000..9ee87046 --- /dev/null +++ b/server/test/gitlab.test.ts @@ -0,0 +1,96 @@ +import { describe, expect, test } from "bun:test" +import { + gitlabDirectRepo, + gitlabAuthModes, + gitlabClient, + gitlabRequestUrl, + parseGitlabViewerPayload, + repoPathFromInput, +} from "../src/gitlab" + +describe("GitLab viewer compatibility", () => { + test("resolves an administrator-pinned project to host SSH", () => { + expect(gitlabDirectRepo("https://gitlab.example.com", "team/loopat-personal")).toEqual({ + url: "git@gitlab.example.com:team/loopat-personal.git", + path: "team/loopat-personal", + owner: "team", + }) + }) + + test("requires a valid host and qualified project path for host SSH", () => { + expect(gitlabDirectRepo("https://gitlab.example.com", "loopat-personal")).toBeNull() + expect(gitlabDirectRepo("file:///tmp/gitlab", "team/loopat-personal")).toBeNull() + expect(gitlabDirectRepo("https://gitlab.example.com", "team/../loopat-personal")).toBeNull() + }) + + test("parses a standard GitLab user", () => { + expect(parseGitlabViewerPayload({ + id: 17, + username: "alice", + commit_email: "alice@example.com", + })).toEqual({ + login: "alice", + namespaceLogin: "alice", + id: 17, + email: "alice@example.com", + }) + }) + + test("unwraps enterprise user envelopes and alternate login fields", () => { + expect(parseGitlabViewerPayload({ + data: { user: { userName: "alice", id: "123" } }, + })).toEqual({ + login: "alice", + namespaceLogin: "alice", + id: "123", + email: undefined, + }) + }) + + test("derives the namespace from email when username is omitted", () => { + expect(parseGitlabViewerPayload({ + id: 17, + name: "Alice Example", + email: "alice@example.com", + })).toMatchObject({ + login: "alice", + namespaceLogin: "alice", + }) + }) + + test("uses oauth2 only as transport fallback when no namespace is exposed", () => { + expect(parseGitlabViewerPayload({ id: 17, name: "Example User" })).toEqual({ + login: "oauth2", + namespaceLogin: undefined, + id: 17, + email: undefined, + }) + }) + + test("keeps a full namespace/project path without a viewer username", () => { + expect(repoPathFromInput("https://gitlab.example.com/team/loopat-personal.git")) + .toBe("team/loopat-personal") + }) + + test("requires a full path when the account namespace is unavailable", () => { + expect(() => repoPathFromInput("loopat-personal")) + .toThrow("enter repository as namespace/project") + }) + + test("supports private_token query authentication as a fallback", () => { + const client = gitlabClient("test-token", "https://gitlab.example.com") + const url = new URL(gitlabRequestUrl( + client, + "/projects?membership=true&per_page=1", + "query", + )) + expect(url.searchParams.get("membership")).toBe("true") + expect(url.searchParams.get("per_page")).toBe("1") + expect(url.searchParams.get("private_token")).toBe("test-token") + }) + + test("keeps standard GitLab header auth ahead of query auth", () => { + expect(gitlabAuthModes()) + .toEqual(["private-token", "bearer", "query"]) + }) +}) diff --git a/server/test/provider-resolution.test.ts b/server/test/provider-resolution.test.ts index 8cead1c2..39d52da1 100644 --- a/server/test/provider-resolution.test.ts +++ b/server/test/provider-resolution.test.ts @@ -111,6 +111,16 @@ describe("pickProvider — requireKey semantics", () => { expect(r).toBeNull() }) + test("allows Codex runtime providers without apiKey when requireKey=true", () => { + const r = pickProvider( + { default: "codex", providers: { codex: { ...p(""), runtime: "codex" } } }, + {}, + [], + true, + ) + expect(r?.name).toBe("codex") + }) + test("returns null on completely empty configs", () => { expect(pickProvider({ default: "", providers: {} }, {}, [], true)).toBeNull() expect(pickProvider({ default: "", providers: {} }, {}, [], false)).toBeNull() diff --git a/web/src/api.ts b/web/src/api.ts index e1f59935..186a82fe 100644 --- a/web/src/api.ts +++ b/web/src/api.ts @@ -143,7 +143,13 @@ export type PersonalStatus = { * the team git host. */ vaultKeys?: { vault: string; publicKey: string }[] imported: boolean - gitHost?: { provider: string; baseUrl: string | null; defaultRepo?: string; tokenHelp?: string | null } + gitHost?: { + provider: string + baseUrl: string | null + defaultRepo?: string + tokenHelp?: string | null + directRepo?: { url: string; path: string; owner: string } | null + } } // List the user's repos for the onboarding picker ("personal"-named first). @@ -151,11 +157,15 @@ export type PersonalStatus = { // `error` instead of treating it as an empty list. export async function listPersonalRepos( token: string, + opts: { provider?: string; baseUrl?: string | null } = {}, ): Promise<{ ok: boolean; repos: { name: string; path: string }[]; login?: string; error?: string }> { + const payload: Record = { token } + if (opts.provider) payload.provider = opts.provider + if (opts.baseUrl) payload.baseUrl = opts.baseUrl const r = await apiFetch("/api/personal/repos", { method: "POST", headers: { "content-type": "application/json" }, - body: JSON.stringify({ token }), + body: JSON.stringify(payload), }) const j = await r.json().catch(() => ({}) as any) if (!r.ok) return { ok: false, repos: [], error: j?.error ?? "request failed" } @@ -394,7 +404,7 @@ export async function setupPersonalGithub( token: string, repoName?: string, cryptKey?: string, - baseUrl?: string, + opts: { provider?: string; baseUrl?: string | null } = {}, ): Promise<{ ok: boolean error?: string @@ -407,7 +417,8 @@ export async function setupPersonalGithub( const payload: Record = { token } if (repoName) payload.repoName = repoName if (cryptKey) payload.cryptKey = cryptKey - if (baseUrl) payload.baseUrl = baseUrl + if (opts.provider) payload.provider = opts.provider + if (opts.baseUrl) payload.baseUrl = opts.baseUrl const r = await apiFetch("/api/personal/github", { method: "POST", headers: { "content-type": "application/json" }, @@ -1091,7 +1102,8 @@ export async function listTopics(): Promise { } export type ModelEntry = { id: string; maxContextTokens?: number } -export type ProviderInfo = { model?: string; models: ModelEntry[]; baseUrl: string; source: "personal" | "workspace"; enabled: boolean; hasKey: boolean } +export type ProviderRuntime = "claude" | "codex" +export type ProviderInfo = { model?: string; models: ModelEntry[]; baseUrl: string; runtime?: ProviderRuntime; source: "personal" | "workspace"; enabled: boolean; hasKey: boolean } export type ProvidersResponse = { providers: Record; default: string } export async function getProviders(): Promise { const r = await apiFetch("/api/providers") @@ -1105,8 +1117,10 @@ export async function testProviderConnection( model: string, provider?: string, source?: "personal" | "workspace", + runtime?: ProviderRuntime, ): Promise<{ ok: boolean; error?: string }> { const body: Record = { baseUrl, model } + if (runtime) body.runtime = runtime if (apiKey) { body.apiKey = apiKey } else if (provider && source) { @@ -1270,6 +1284,7 @@ export type SettingsProvider = { model?: string models: ModelEntry[] baseUrl: string + runtime?: ProviderRuntime hasKey?: boolean enabled: boolean apiKey?: string @@ -1284,7 +1299,7 @@ export type PersonalSettings = { } export type WorkspaceSettings = { - providers: Record + providers: Record default: string tokenUsage: TokenUsage } @@ -1311,6 +1326,7 @@ export async function updatePersonalSettings(patch: { export type ProviderDisk = { models?: ModelEntry[] baseUrl: string + runtime?: ProviderRuntime /** Plain string; may contain `${VAR}` references resolved against vault envs/ at load. */ apiKey?: string enabled?: boolean @@ -1369,7 +1385,7 @@ export async function getWorkspaceSettings(): Promise { } export async function updateWorkspaceSettings(patch: { - providers?: Record + providers?: Record default?: string }): Promise { const r = await apiFetch("/api/settings/workspace", { @@ -1407,16 +1423,17 @@ export async function getLoopTokenUsage(): Promise { // ── admin presets ── -export type ProviderPresetModel = string | { id: string; tier?: "opus" | "sonnet" | "haiku" } +export type ProviderPresetModel = string | { id: string; tier?: "opus" | "sonnet" | "haiku"; maxContextTokens?: number } export type ProviderPreset = { name: string baseUrl: string + runtime?: ProviderRuntime models: ProviderPresetModel[] } /** Normalize a preset model entry to { id, tier? }. */ -export function normalizePresetModel(m: ProviderPresetModel): { id: string; tier?: "opus" | "sonnet" | "haiku" } { +export function normalizePresetModel(m: ProviderPresetModel): { id: string; tier?: "opus" | "sonnet" | "haiku"; maxContextTokens?: number } { if (typeof m === "string") return { id: m } return m } diff --git a/web/src/components/chat/ModelSelector.tsx b/web/src/components/chat/ModelSelector.tsx index 753b1f36..9759be88 100644 --- a/web/src/components/chat/ModelSelector.tsx +++ b/web/src/components/chat/ModelSelector.tsx @@ -31,7 +31,8 @@ export default function ModelSelector() { if (!providers) return []; const result: FlatModel[] = []; for (const [provName, info] of Object.entries(providers.providers)) { - if (info.enabled === false || !info.hasKey) continue; + if (info.enabled === false) continue; + if (info.runtime !== "codex" && !info.hasKey) continue; for (const m of info.models ?? []) { const q = search.toLowerCase().trim(); diff --git a/web/src/components/dialog/AdminDialog.tsx b/web/src/components/dialog/AdminDialog.tsx index 4c1c6ddb..8f3cd2d0 100644 --- a/web/src/components/dialog/AdminDialog.tsx +++ b/web/src/components/dialog/AdminDialog.tsx @@ -21,6 +21,7 @@ import { type AdminUser, type WorkspaceSettings, type ModelEntry, + type ProviderRuntime, type ProviderPreset, } from "@/api" @@ -245,6 +246,7 @@ type WorkspaceDraft = { providers: Record 0 ? { maxContextTokens: m.maxContextTokens } : {}), })) ?? [], baseUrl: prov.baseUrl ?? "", + runtime: (prov as any).runtime === "codex" ? "codex" : "claude", apiKey: "", keyDirty: false, hasKey: prov.hasKey ?? false, @@ -394,7 +397,7 @@ export function WorkspacePanel() { setDraft((d) => { if (!d) return d return { ...d, providers: { ...d.providers, [n]: { - models: [], baseUrl: "", apiKey: "", keyDirty: false, hasKey: false, enabled: false, + models: [], baseUrl: "", runtime: "claude", apiKey: "", keyDirty: false, hasKey: false, enabled: false, } } } }) setNewName("") @@ -417,6 +420,7 @@ export function WorkspacePanel() { out[name] = { models, baseUrl: p.baseUrl, + runtime: p.runtime, enabled: p.enabled, } if (p.keyDirty && p.apiKey.trim()) out[name].apiKey = p.apiKey.trim() @@ -449,7 +453,7 @@ export function WorkspacePanel() { {names.map((name) => { const p = draft.providers[name] const isAddingModel = addingModel[name] ?? false - const hasKey = p.hasKey || p.apiKey.trim() !== "" + const hasKey = p.runtime === "codex" || p.hasKey || p.apiKey.trim() !== "" return (
{/* Provider header */} @@ -499,6 +503,16 @@ export function WorkspacePanel() { {/* Fields */}
+ + + { const newKey = p.apiKey.trim() const tk = `${name}::${m.id}` - if (!newKey && !p.hasKey) { + if (p.runtime !== "codex" && !newKey && !p.hasKey) { setTestingModel((t) => ({ ...t, [tk]: "error" })) setTestError((t) => ({ ...t, [tk]: "enter an API key first" })) setTimeout(() => { @@ -619,8 +633,8 @@ export function WorkspacePanel() { setTestError((t) => ({ ...t, [tk]: "" })) try { const result = newKey - ? await testProviderConnection(p.baseUrl, newKey, m.id) - : await testProviderConnection(p.baseUrl, "", m.id, name, "workspace") + ? await testProviderConnection(p.baseUrl, newKey, m.id, undefined, undefined, p.runtime) + : await testProviderConnection(p.baseUrl, "", m.id, name, "workspace", p.runtime) setTestingModel((t) => ({ ...t, [tk]: result.ok ? "ok" : "error" })) if (!result.ok) setTestError((t) => ({ ...t, [tk]: result.error ?? "unknown error" })) } catch (e: any) { @@ -632,15 +646,15 @@ export function WorkspacePanel() { setTestError((t) => { const { [tk]: _, ...rest } = t; return rest }) }, 4000) }} - disabled={tmState === "testing" || (!p.hasKey && !p.apiKey.trim())} + disabled={tmState === "testing" || (p.runtime !== "codex" && !p.hasKey && !p.apiKey.trim())} className={`shrink-0 text-[9px] px-1 py-0 rounded transition-colors ${ - !p.hasKey && !p.apiKey.trim() ? "opacity-0 group-hover:opacity-100 text-gray-300" : + p.runtime !== "codex" && !p.hasKey && !p.apiKey.trim() ? "opacity-0 group-hover:opacity-100 text-gray-300" : tmState === "ok" ? "bg-emerald-100 text-emerald-700" : tmState === "error" ? "bg-red-100 text-red-700" : tmState === "testing" ? "bg-gray-100 text-gray-400 animate-pulse" : "text-gray-400 hover:text-gray-700 opacity-0 group-hover:opacity-100" }`} - title={tmErr || (p.apiKey.trim() ? "test connection" : p.hasKey ? "test connection" : "enter an API key first")} + title={tmErr || (p.runtime === "codex" ? "test Codex CLI" : p.apiKey.trim() ? "test connection" : p.hasKey ? "test connection" : "enter an API key first")} > {tmState === "ok" ? "OK" : tmState === "error" ? "FAIL" : tmState === "testing" ? "..." : "test"} @@ -676,12 +690,16 @@ export function WorkspacePanel() { providers: { ...d.providers, [p.name]: { - models: p.models.map((m) => { const n = normalizePresetModel(m); return { id: n.id } }), + models: p.models.map((m) => { + const n = normalizePresetModel(m) + return { id: n.id, ...(n.maxContextTokens ? { maxContextTokens: n.maxContextTokens } : {}) } + }), baseUrl: p.baseUrl, + runtime: p.runtime === "codex" ? "codex" : "claude", apiKey: "", keyDirty: false, hasKey: false, - enabled: false, + enabled: p.runtime === "codex", } satisfies WorkspaceDraft["providers"][string], }, } diff --git a/web/src/components/dialog/PersonalRepoPanel.tsx b/web/src/components/dialog/PersonalRepoPanel.tsx index e045904f..f64bb109 100644 --- a/web/src/components/dialog/PersonalRepoPanel.tsx +++ b/web/src/components/dialog/PersonalRepoPanel.tsx @@ -95,8 +95,14 @@ export function PersonalRepoPanel({ onDone, initialToken }: { onDone?: () => voi getPersonalStatus() .then((s) => { setStatus(s) - setRepoUrl(s?.personalRepo ?? "") - setGhRepoName(s?.gitHost?.defaultRepo ?? "loopat-personal") + const directRepo = s?.gitHost?.directRepo + setRepoUrl(s?.personalRepo ?? directRepo?.url ?? "") + setGhRepoName(directRepo?.path ?? s?.gitHost?.defaultRepo ?? "loopat-personal") + if (directRepo) { + setGhLogin(directRepo.owner) + setGhRepos([{ name: directRepo.path.split("/").at(-1) ?? directRepo.path, path: directRepo.path }]) + setStep("confirm") + } }) .finally(() => setLoading(false)) }, []) @@ -107,7 +113,7 @@ export function PersonalRepoPanel({ onDone, initialToken }: { onDone?: () => voi if (!initialToken || loading) return setGhToken(initialToken) setGhBusy(true) - listPersonalRepos(initialToken) + listPersonalRepos(initialToken, { provider: status?.gitHost?.provider, baseUrl: status?.gitHost?.baseUrl }) .then((res) => { if (!res.ok) { setGhError(res.error ?? "invalid token"); return } setGhRepos(res.repos) @@ -115,7 +121,7 @@ export function PersonalRepoPanel({ onDone, initialToken }: { onDone?: () => voi setStep("repo") }) .finally(() => setGhBusy(false)) - }, [initialToken, loading]) + }, [initialToken, loading, status?.gitHost?.baseUrl, status?.gitHost?.provider]) const copyPub = async () => { if (!status?.publicKey) return @@ -170,7 +176,12 @@ export function PersonalRepoPanel({ onDone, initialToken }: { onDone?: () => voi setGhError(null) setGhBusy(true) try { - const r = await setupPersonalGithub(ghToken.trim(), ghRepoName.trim() || undefined, ghCryptKey.trim() || undefined) + const r = await setupPersonalGithub( + ghToken.trim(), + ghRepoName.trim() || undefined, + ghCryptKey.trim() || undefined, + { provider: status?.gitHost?.provider, baseUrl: status?.gitHost?.baseUrl }, + ) if (!r.ok) { if (r.needsCryptKey) { setGhNeedsCryptKey(true) @@ -197,7 +208,7 @@ export function PersonalRepoPanel({ onDone, initialToken }: { onDone?: () => voi setGhError(null) setGhBusy(true) try { - const res = await listPersonalRepos(ghToken.trim()) + const res = await listPersonalRepos(ghToken.trim(), { provider: status?.gitHost?.provider, baseUrl: status?.gitHost?.baseUrl }) if (!res.ok) { // bad token → stay on the token step and show the error, instead of // advancing to a misleading empty repo picker @@ -213,11 +224,15 @@ export function PersonalRepoPanel({ onDone, initialToken }: { onDone?: () => voi } const ghProvider = status?.gitHost?.provider ?? "github" - const ghProviderLabel = ghProvider === "github" ? "GitHub" : ghProvider - const ghRepoExists = ghRepos.some((r) => r.name === ghRepoName.trim()) + const directRepo = status?.gitHost?.directRepo ?? null + const ghProviderLabel = ghProvider === "github" ? "GitHub" : ghProvider === "gitlab" ? "GitLab" : ghProvider + const ghRepoValue = ghRepoName.trim() + const ghRepoExists = ghRepos.some((r) => ghRepoValue === r.path || ghRepoValue === r.name) + const ghRepoPathForUrl = ghRepoValue.includes("/") ? ghRepoValue : ghLogin && ghRepoValue ? `${ghLogin}/${ghRepoValue}` : "" + const ghBaseUrl = (status?.gitHost?.baseUrl ?? (ghProvider === "github" ? "https://github.com" : "")).replace(/\/+$/, "") const ghRepoWebUrl = - ghLogin && ghRepoName.trim() - ? `${(status?.gitHost?.baseUrl ?? "https://github.com").replace(/\/+$/, "")}/${ghLogin}/${ghRepoName.trim()}` + ghBaseUrl && ghRepoPathForUrl + ? `${ghBaseUrl}/${ghRepoPathForUrl.replace(/\.git$/i, "")}` : null if (loading) { @@ -313,7 +328,7 @@ export function PersonalRepoPanel({ onDone, initialToken }: { onDone?: () => voi return } - if (!status.publicKey) { + if (!status.publicKey && !directRepo) { return (
No deploy key available — the server is probably missing{" "} @@ -329,7 +344,7 @@ export function PersonalRepoPanel({ onDone, initialToken }: { onDone?: () => voi return (
{/* wizard progress */} -
+ {!directRepo &&
{[ { k: "token", label: "Token" }, { k: "repo", label: "Repository" }, @@ -357,7 +372,7 @@ export function PersonalRepoPanel({ onDone, initialToken }: { onDone?: () => voi {i < 2 && →}
))} -
+
} {/* Device-flow hand-off: token prefilled, repos loading — skip the paste UI. */} {step === "token" && initialToken && ( @@ -438,7 +453,7 @@ export function PersonalRepoPanel({ onDone, initialToken }: { onDone?: () => voi