diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e6a23b61..87aaa066 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -46,6 +46,9 @@ jobs: - name: Compose bridge configuration run: node tests/compose-bridge-config.cjs + - name: OpenShell evaluation probe (no gateway) + run: node --test scripts/openshell/probe.test.mjs + - name: Release versioning run: tests/release-versioning.sh diff --git a/README.md b/README.md index 3856b254..b746e3c0 100644 --- a/README.md +++ b/README.md @@ -62,6 +62,12 @@ Omitting the expected-profile header remains supported for older clients, but provides no wrong-endpoint protection. There is deliberately no silent fallback between profiles and no automatic workspace or file migration. +## OpenShell evaluation + +An opt-in OpenShell backend is under [evaluation](docs/openshell/README.md). +The initial track provides a pinned lifecycle probe and security/compatibility +gates. It does not add a backend or change execution routing. + ## Sandbox Isolation Two modes are supported: diff --git a/docs/openshell/README.md b/docs/openshell/README.md new file mode 100644 index 00000000..a139ab7c --- /dev/null +++ b/docs/openshell/README.md @@ -0,0 +1,189 @@ +# OpenShell backend evaluation + +Status: Evaluation. No OpenShell backend is registered or enabled. + +Goal: add an opt-in, operator-hosted OpenShell backend behind Code API's existing +execution contract. Keep LibreChat clients, the default HTTP/libkrun backend, +Lambda MicroVMs, and outbound BYOM workers unchanged. + +## Baseline and upstream pin + +- Code API baseline: `836d001319015072f9493df5d7710fc371403962`. +- OpenShell evaluation release: `v0.1.2`, commit + `6648bd0c290efbc41ba131ee9831ee45cd431f94`. +- Pin the CLI, gateway, supervisor, and compute driver to that release. Record + image digests, effective policy, runtime/kernel, and resource limits with each + live result. The probe checks the CLI version, not the gateway's version. +- Revalidate this track against each proposed upgrade. Do not use rolling `dev` + tags for acceptance results. + +Upstream references at the evaluation commit: +[license](https://github.com/NVIDIA/OpenShell/blob/6648bd0c290efbc41ba131ee9831ee45cd431f94/LICENSE), +[third-party notices](https://github.com/NVIDIA/OpenShell/blob/6648bd0c290efbc41ba131ee9831ee45cd431f94/THIRD-PARTY-NOTICES), +[SDK](https://github.com/NVIDIA/OpenShell/blob/6648bd0c290efbc41ba131ee9831ee45cd431f94/sdk/typescript/README.md), +[policy schema](https://github.com/NVIDIA/OpenShell/blob/6648bd0c290efbc41ba131ee9831ee45cd431f94/docs/how-it-works/policies/schema.mdx). + +## Integration boundary + +Current flow: + +```text +LibreChat -> Code API auth/authorization -> Redis job + -> worker -> signed execute request -> SandboxBackend.execute + -> runner -> Code API egress gateway -> artifacts/tools + -> worker result finalization -> client +``` + +Proposed flow changes only the sandbox backend: + +```text +worker -> OpenShell adapter -> private OpenShell gateway + -> isolated workload + compatible Code API runner + -> unchanged signed execute contract and Code API egress gateway +``` + +The backend seam is +[`SandboxBackend`](../../service/src/sandbox-backend/types.ts). Its existing +consumers already own deadlines, queue/backend fencing, artifact restoration, +and result mapping. OpenShell's lifecycle API is not a substitute for the +runner's `/api/v2/execute` protocol. + +### Options + +| Approach | Trade-off | Decision | +| -------------------------------------------- | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | +| Existing backends only | No new control plane; no OpenShell policy/lifecycle integration | Keep as defaults and rollback targets | +| Adapter retaining the signed runner contract | Reuses Code API authorization, artifacts, tools, and finalization; runner compatibility is unproven | Evaluate first | +| Execute code directly with OpenShell exec | Avoids nested NsJail; requires rebuilding file/tool delivery, manifest validation, and result semantics | Defer unless the runner approach cannot retain its guarantees | + +Do not put the LibreChat application, the BYOM identity process, or Code API +control-plane credentials inside an agent workload. The gateway remains private; +Code API authenticates users and authorizes tenant/session access. + +## Invariants and owners + +| Invariant | Owner | +| ------------------------------------------------------------------------------ | ------------------------------------------------------------ | +| Tenant/user/session authorization and quotas | Code API | +| Signed body is immutable; `inputDelivery` storage IDs never reach the workload | Worker and adapter | +| Only scoped egress grants authorize artifact and tool access | Code API egress gateway and runner | +| Queue wait, provisioning, and execution share the original job deadline | Worker and adapter | +| Abort cannot settle as success; remote execution must terminate | Adapter and compute runtime | +| Retried jobs cannot create duplicate active execution or repeat mutations | Adapter, queue fencing, and runtime registry | +| Stateful workspace reuse waits for successful `sessionResultFinalizer` | Worker and adapter | +| Filesystem, process identity, network, and resource restrictions fail closed | OpenShell policy and selected runtime, plus runner hardening | +| Cleanup is confirmed, not merely requested; orphans remain observable/reapable | Adapter and lifecycle reconciler | + +OpenShell Docker/Podman policy rejects root workloads. The current runner +entrypoint performs mounts, ownership changes, cgroup setup, and an NsJail smoke +test. Compatibility is a blocker to resolve, not permission to run privileged, +disable manifests, relax Landlock, or bypass hardened startup. OpenShell policy +alone does not establish a separate-kernel tenant boundary. + +## Work sequence + +### 1. Lifecycle spike (this slice) + +- [x] Pin an upstream release and identify the existing backend seam. +- [x] Add an explicit, dependency-free CLI probe with strict filesystem policy, + no network allowances, no attached providers, resource limits, and cleanup. +- [x] Test validation, cancellation, partial failure, redaction, and nonterminal + cleanup without a live gateway. +- [ ] Run the probe against an isolated operator-owned gateway. Record results. + +This probe runs only a fixed shell/file round trip. It does **not** exercise Code +API, verify effective policy enforcement, or prove tenant isolation. A gateway +administrator's global policy can override the supplied sandbox policy. + +### 2. Hardened runner compatibility + +- [ ] Identify the supported compute/runtime boundary and prove startup without + weakening the existing security invariants. +- [ ] Forward the private runner listener through authenticated OpenShell + transport. Never expose its execute endpoint publicly. +- [ ] Execute a real worker-built signed request unchanged. Test Python/Bash, + input/output artifacts, tool-call replay, and error/result parity. +- [ ] Reject tampered/expired manifests, unauthorized storage handles, access to + supervisor/control-plane credentials, forbidden destinations, metadata + endpoints, and cross-tenant workspace reuse. +- [ ] Verify host/kernel filesystem protection and resource enforcement with + negative tests, not only effective-policy inspection. + +If nesting cannot work securely, stop and design the direct-exec replacement's +full authorization and delivery contract before implementation. + +### 3. Opt-in adapter + +- [ ] Use the same-release SDK behind lazy backend loading. Verify package + availability and locking; this release's TypeScript SDK uses GitHub Packages. + The CLI is a spike dependency, not a per-job production transport. +- [ ] Add an explicit backend config gate, isolated queues, and producer/consumer + backend checks. Old consumers must never pick up OpenShell jobs, including + legacy jobs without a backend marker. +- [ ] Start stateless. Reject affinity/strict mode until tenant-bound session + locks, fencing, finalization, and persistence are implemented and tested. +- [ ] Cover enqueue-anchored deadlines, cancellation while provisioning/running, + retry idempotency, gateway failure, ambiguous create responses, accepted/pending + deletion, worker crashes, orphan reconciliation, and shutdown. +- [ ] Keep auth tokens and provider credentials out of logs, argv, and workload + environment. Do not silently fall back to a different backend. + +### 4. Acceptance and rollout + +- [ ] Measure concurrency, queue/provision/execute/cleanup latency, resources, + artifact throughput, and failure rates against the existing backend using the + same workloads. Record measurements; define targets before promotion. +- [ ] Exercise gateway restart, worker restart, disconnects, upgrades, cleanup + backlogs, and quota pressure. Test database/runtime topology before claiming HA. +- [ ] Pin deployment artifacts, collect required notices/SBOMs, and document + private gateway authentication, backups, upgrades, and orphan operations. +- [ ] Canary a separate endpoint and worker pool only after compatibility and + security gates pass. Roll back routing, then drain the new queues. No automatic + file/workspace migration or production-default change is part of this track. + +## Run the lifecycle probe + +Requirements: Node 20+, an OpenShell `0.1.2` CLI, a registered named evaluation +gateway using the matching release, and a digest-pinned shell/coreutils image +pullable by that gateway. Install and configure these separately on dedicated +infrastructure. This repository does not install or start a gateway. + +```bash +node --test scripts/openshell/probe.test.mjs + +node scripts/openshell/probe.mjs \ + --gateway codeapi-evaluation \ + --workspace default \ + --image 'REGISTRY/IMAGE@sha256:REPLACE_WITH_64_HEX_DIGEST' +``` + +The live command creates and deletes one uniquely named `codeapi-probe-*` +sandbox. It uses the bundled policy, one CPU, 256 MiB, manual approvals, no +credential auto-discovery, and a fixed ten-second exec timeout. Each CLI phase +has a 120-second timeout (`--timeout-ms`, maximum 300000); this is not the +production job deadline. SIGINT/SIGTERM abort work but allow an independent +bounded cleanup attempt. SIGKILL or host/process failure cannot run cleanup. + +Only the pinned CLI's terminal `Deleted` or `already deleted` acknowledgement +confirms cleanup. Accepted/pending, unfamiliar, failed, and timed-out deletion +results fail the probe. CLI stdout/stderr are not emitted, except sanitized +phase/version/timing results. The live probe's overall success requires both +the file round trip and confirmed cleanup. + +On failure, use the emitted sandbox name and the **same gateway/workspace** to +inspect and reconcile it. A timed-out create can finish after cleanup; even an +`already deleted` acknowledgement does not resolve that race. Review the +gateway's resources after any ambiguous create or process failure. Do not retry +blindly or interpret `cleanup=confirmed` on a failed run as an orphan-free +fleet. Automated reconciliation is an adapter acceptance gate. + +## License implications + +OpenShell is Apache-2.0 at the pinned commit. That permits commercial hosted use +and modification without a network-use source-disclosure obligation. It does +not grant NVIDIA trademark rights or a support/warranty commitment. + +For distributed images or on-prem packages, include the license, preserve +applicable attribution/notices, mark modified upstream files, and audit bundled +runtime/SDK/image/model dependencies separately. This slice distributes no +OpenShell binaries or SDK and introduces no paid NVIDIA service dependency. diff --git a/scripts/openshell/probe-policy.yaml b/scripts/openshell/probe-policy.yaml new file mode 100644 index 00000000..bef0a892 --- /dev/null +++ b/scripts/openshell/probe-policy.yaml @@ -0,0 +1,11 @@ +version: 1 +filesystem_policy: + include_workdir: false + read_only: [/bin, /usr, /lib, /lib64, /etc] + read_write: [/tmp, /dev/null] +landlock: + compatibility: hard_requirement +process: + run_as_user: '1000' + run_as_group: '1000' +network_policies: {} diff --git a/scripts/openshell/probe.mjs b/scripts/openshell/probe.mjs new file mode 100644 index 00000000..116beb56 --- /dev/null +++ b/scripts/openshell/probe.mjs @@ -0,0 +1,217 @@ +import { execFile } from 'node:child_process'; +import { randomUUID } from 'node:crypto'; +import { parseArgs } from 'node:util'; +import { fileURLToPath, pathToFileURL } from 'node:url'; + +export const OPENSHELL_VERSION = '0.1.2'; +export const OPENSHELL_COMMIT = '6648bd0c290efbc41ba131ee9831ee45cd431f94'; +const POLICY = fileURLToPath(new URL('./probe-policy.yaml', import.meta.url)); +const MARKER = 'codeapi-openshell-probe-ok'; + +export class ProbeError extends Error { + constructor(stage) { + super(`OpenShell probe failed at ${stage}`); + this.stage = stage; + } +} + +export function commandRunner(binary = 'openshell') { + const env = { ...process.env, NO_COLOR: '1' }; + // Named gateway selection must not inherit an endpoint or insecure override. + delete env.OPENSHELL_GATEWAY_ENDPOINT; + delete env.OPENSHELL_GATEWAY_INSECURE; + return (args, { stage, timeoutMs, signal }) => + new Promise((resolve, reject) => { + const child = execFile( + binary, + args, + { + env, + timeout: timeoutMs, + killSignal: 'SIGKILL', + maxBuffer: 65536, + signal, + }, + (error, stdout) => { + // CLI errors can contain credentials, policy contents, or provider details. + if (error) reject(new ProbeError(stage)); + else resolve(stdout); + } + ); + child.stdin.end(); + }); +} + +export async function runProbe(options, run = commandRunner()) { + const { + gateway, + image, + workspace = 'default', + timeoutMs = 120000, + signal, + } = options; + if (!/^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,127}$/.test(gateway ?? '')) { + throw new ProbeError('gateway validation'); + } + if (!/^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,127}$/.test(workspace)) { + throw new ProbeError('workspace validation'); + } + if ( + !/^\S+@sha256:[a-f0-9]{64}$/.test(image ?? '') || + image.startsWith('-') + ) { + throw new ProbeError('image validation'); + } + if ( + !Number.isSafeInteger(timeoutMs) || + timeoutMs < 100 || + timeoutMs > 300000 + ) { + throw new ProbeError('timeout validation'); + } + const invoke = (args, stage, useSignal = true) => + run(args, { stage, timeoutMs, signal: useSignal ? signal : undefined }); + const version = await invoke(['--version'], 'version'); + if (version.trim() !== `openshell ${OPENSHELL_VERSION}`) { + throw new ProbeError('version mismatch'); + } + if (signal?.aborted) throw new ProbeError('cancelled'); + + const name = `codeapi-probe-${randomUUID()}`; + const scope = ['--gateway', gateway, '--workspace', workspace]; + const timings = {}; + const timed = async (stage, args, useSignal = true) => { + const start = performance.now(); + try { + return await invoke([...scope, ...args], stage, useSignal); + } finally { + timings[stage] = Math.round(performance.now() - start); + } + }; + let failure; + let cleanup = 'unconfirmed'; + try { + await timed('create', [ + 'sandbox', + 'create', + '--name', + name, + '--from', + image, + '--policy', + POLICY, + '--no-auto-providers', + '--approval-mode', + 'manual', + '--cpu', + '1', + '--memory', + '256Mi', + '--keep', + '--detach', + '--no-tty', + '--output', + 'json', + '--', + '/bin/sh', + '-c', + 'sleep 300', + ]); + const output = await timed('exec', [ + 'sandbox', + 'exec', + '--name', + name, + '--timeout', + '10', + '--no-tty', + '--no-login-shell', + '--', + '/bin/sh', + '-c', + `printf '%s' '${MARKER}' > /tmp/codeapi-probe && cat /tmp/codeapi-probe`, + ]); + if (output.trim() !== MARKER) throw new ProbeError('exec output'); + } catch (error) { + failure = error instanceof ProbeError ? error.stage : 'execution'; + } finally { + try { + const output = await timed( + 'delete', + ['sandbox', 'delete', name], + false + ); + const text = output.replace(/\x1b\[[0-9;]*m/g, '').trim(); + // Exit zero also covers accepted/pending deletion. Only terminal outcomes pass. + if ( + text === `✓ Deleted sandbox ${name}` || + text === `✓ Sandbox ${name} already deleted` + ) { + cleanup = 'confirmed'; + } + } catch { + // Keep the execution failure and report cleanup independently. + } + } + if (signal?.aborted && failure === undefined) failure = 'cancelled'; + return { + ok: failure === undefined && cleanup === 'confirmed', + version: OPENSHELL_VERSION, + upstreamCommit: OPENSHELL_COMMIT, + sandbox: name, + workspace, + cleanup, + ...(failure ? { failedStage: failure } : {}), + timingsMs: timings, + }; +} + +async function main() { + const { values } = parseArgs({ + options: { + gateway: { type: 'string' }, + workspace: { type: 'string', default: 'default' }, + image: { type: 'string' }, + 'timeout-ms': { type: 'string', default: '120000' }, + help: { type: 'boolean' }, + }, + }); + if (values.help) { + console.log( + 'node scripts/openshell/probe.mjs --gateway NAME --image REGISTRY/IMAGE@sha256:DIGEST [--workspace NAME] [--timeout-ms 120000]' + ); + return; + } + const controller = new AbortController(); + const abort = () => controller.abort(); + process.on('SIGINT', abort); + process.on('SIGTERM', abort); + try { + const result = await runProbe({ + gateway: values.gateway, + workspace: values.workspace, + image: values.image, + timeoutMs: Number(values['timeout-ms']), + signal: controller.signal, + }); + console.log(JSON.stringify(result, null, 2)); + process.exitCode = result.ok ? 0 : 1; + } finally { + process.off('SIGINT', abort); + process.off('SIGTERM', abort); + } +} + +if ( + process.argv[1] && + import.meta.url === pathToFileURL(process.argv[1]).href +) { + main().catch(error => { + console.error( + error instanceof ProbeError + ? error.message + : 'Invalid OpenShell probe invocation' + ); + process.exitCode = 1; + }); +} diff --git a/scripts/openshell/probe.test.mjs b/scripts/openshell/probe.test.mjs new file mode 100644 index 00000000..2ec95997 --- /dev/null +++ b/scripts/openshell/probe.test.mjs @@ -0,0 +1,273 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; +import { + commandRunner, + OPENSHELL_VERSION, + ProbeError, + runProbe, +} from './probe.mjs'; + +const options = { + gateway: 'evaluation', + workspace: 'codeapi-test', + image: `registry.test/shell@sha256:${'a'.repeat(64)}`, +}; +function fixture(overrides = {}) { + const calls = []; + const run = async (args, call) => { + calls.push({ args, ...call }); + if (overrides[call.stage]) return overrides[call.stage](args, call); + if (call.stage === 'version') return `openshell ${OPENSHELL_VERSION}\n`; + if (call.stage === 'exec') return 'codeapi-openshell-probe-ok'; + if (call.stage === 'delete') + return `✓ Deleted sandbox ${args.at(-1)}\n`; + return '{}'; + }; + return { run, calls }; +} + +test('scopes every operation and creates a bounded, credential-free sandbox', async () => { + const { run, calls } = fixture(); + const result = await runProbe(options, run); + assert.equal(result.ok, true); + assert.equal(result.cleanup, 'confirmed'); + assert.match(result.sandbox, /^codeapi-probe-[a-f0-9-]{36}$/); + assert.deepEqual( + calls.map(call => call.stage), + ['version', 'create', 'exec', 'delete'] + ); + for (const call of calls.slice(1)) { + assert.deepEqual(call.args.slice(0, 4), [ + '--gateway', + 'evaluation', + '--workspace', + 'codeapi-test', + ]); + assert.equal(call.timeoutMs, 120000); + assert.ok(call.args.includes(result.sandbox)); + } + const create = calls[1].args; + for (const flag of [ + '--policy', + '--no-auto-providers', + '--keep', + '--detach', + '--no-tty', + ]) + assert.ok(create.includes(flag)); + assert.equal(create.includes('--provider'), false); + assert.equal(create.includes('--env'), false); + assert.equal(create[create.indexOf('--from') + 1], options.image); + assert.equal(create[create.indexOf('--cpu') + 1], '1'); + assert.equal(create[create.indexOf('--memory') + 1], '256Mi'); + assert.equal(create[create.indexOf('--approval-mode') + 1], 'manual'); +}); + +test('rejects mutable images, ambiguous scope, and invalid timeouts before invoking the CLI', async () => { + for (const invalid of [ + { image: 'shell:latest' }, + { image: `-shell@sha256:${'a'.repeat(64)}` }, + { gateway: undefined }, + { workspace: '../default' }, + { timeoutMs: 0 }, + { timeoutMs: 300001 }, + ]) { + const { run, calls } = fixture(); + await assert.rejects( + runProbe({ ...options, ...invalid }, run), + ProbeError + ); + assert.equal(calls.length, 0); + } +}); + +test('rejects an unpinned CLI without creating or deleting a sandbox', async () => { + const { run, calls } = fixture({ version: () => 'openshell 0.1.3' }); + await assert.rejects(runProbe(options, run), /version mismatch/); + assert.equal(calls.length, 1); +}); + +for (const stage of ['create', 'exec']) { + test(`attempts cleanup after ${stage} failure without exposing upstream details`, async () => { + const { run, calls } = fixture({ + [stage]: () => { + throw new Error('secret provider credential'); + }, + }); + const result = await runProbe(options, run); + assert.equal(result.ok, false); + assert.equal(result.failedStage, 'execution'); + assert.equal(result.cleanup, 'confirmed'); + assert.equal(calls.at(-1).stage, 'delete'); + assert.equal(JSON.stringify(result).includes('secret'), false); + }); +} + +test('does not accept unexpected command output as success', async () => { + const { run } = fixture({ exec: () => 'unexpected' }); + const result = await runProbe(options, run); + assert.equal(result.ok, false); + assert.equal(result.failedStage, 'exec output'); + assert.equal(result.cleanup, 'confirmed'); +}); + +for (const output of [ + 'deletion accepted; cleanup is pending', + 'unsupported outcome', + '', + '✓ Deleted sandbox someone-else', +]) { + test(`does not report terminal cleanup for ${JSON.stringify( + output + )}`, async () => { + const { run } = fixture({ delete: () => output }); + const result = await runProbe(options, run); + assert.equal(result.ok, false); + assert.equal(result.cleanup, 'unconfirmed'); + }); +} + +test('accepts an already-absent sandbox as terminal cleanup', async () => { + const { run } = fixture({ + delete: args => `✓ Sandbox ${args.at(-1)} already deleted\n`, + }); + assert.equal((await runProbe(options, run)).cleanup, 'confirmed'); +}); + +test('preserves execution failure when cleanup also fails', async () => { + const { run } = fixture({ + exec: () => { + throw new ProbeError('exec'); + }, + delete: () => { + throw new Error('secret'); + }, + }); + const result = await runProbe(options, run); + assert.equal(result.failedStage, 'exec'); + assert.equal(result.cleanup, 'unconfirmed'); + assert.equal(result.ok, false); +}); + +test('cancellation before create does not launch work', async () => { + const controller = new AbortController(); + controller.abort(); + const { run, calls } = fixture(); + await assert.rejects( + runProbe({ ...options, signal: controller.signal }, run), + /cancelled/ + ); + assert.equal(calls.length, 1); +}); + +test('cancellation during create still uses an independent cleanup attempt', async () => { + const controller = new AbortController(); + const { run, calls } = fixture({ + create: () => { + controller.abort(); + throw new ProbeError('create'); + }, + }); + const result = await runProbe( + { ...options, signal: controller.signal }, + run + ); + assert.equal(result.ok, false); + assert.equal(calls[1].signal, controller.signal); + assert.equal(calls.at(-1).signal, undefined); + assert.equal(calls.at(-1).stage, 'delete'); +}); + +test('command runner redacts stdout and stderr on failure', async () => { + await assert.rejects( + commandRunner(process.execPath)( + [ + '-e', + 'console.log("secret"); console.error("private-key"); process.exit(2)', + ], + { stage: 'exec', timeoutMs: 1000 } + ), + error => { + assert.equal(error.message, 'OpenShell probe failed at exec'); + assert.equal(error.cause, undefined); + assert.equal(error.stdout, undefined); + assert.equal(error.stderr, undefined); + return true; + } + ); +}); + +test('command runner kills a stalled CLI within its timeout', async () => { + await assert.rejects( + commandRunner(process.execPath)(['-e', 'setInterval(() => {}, 1000)'], { + stage: 'create', + timeoutMs: 100, + }), + /failed at create/ + ); +}); + +test('command runner removes inherited endpoint and insecure overrides', async () => { + const savedEndpoint = process.env.OPENSHELL_GATEWAY_ENDPOINT; + const savedInsecure = process.env.OPENSHELL_GATEWAY_INSECURE; + try { + process.env.OPENSHELL_GATEWAY_ENDPOINT = 'http://wrong-gateway'; + process.env.OPENSHELL_GATEWAY_INSECURE = 'true'; + const output = await commandRunner(process.execPath)( + [ + '-e', + 'console.log(JSON.stringify([process.env.OPENSHELL_GATEWAY_ENDPOINT, process.env.OPENSHELL_GATEWAY_INSECURE]))', + ], + { stage: 'version', timeoutMs: 1000 } + ); + assert.deepEqual(JSON.parse(output), [null, null]); + } finally { + if (savedEndpoint === undefined) + delete process.env.OPENSHELL_GATEWAY_ENDPOINT; + else process.env.OPENSHELL_GATEWAY_ENDPOINT = savedEndpoint; + if (savedInsecure === undefined) + delete process.env.OPENSHELL_GATEWAY_INSECURE; + else process.env.OPENSHELL_GATEWAY_INSECURE = savedInsecure; + } +}); + +test('late cancellation cannot report success while cleanup completes', async () => { + const controller = new AbortController(); + const { run } = fixture({ + delete: args => { + controller.abort(); + return `✓ Deleted sandbox ${args.at(-1)}\n`; + }, + }); + const result = await runProbe( + { ...options, signal: controller.signal }, + run + ); + assert.equal(result.ok, false); + assert.equal(result.failedStage, 'cancelled'); + assert.equal(result.cleanup, 'confirmed'); +}); + +test('command runner rejects oversized output without exposing it', async () => { + await assert.rejects( + commandRunner(process.execPath)( + ['-e', 'process.stdout.write("x".repeat(100000))'], + { stage: 'exec', timeoutMs: 1000 } + ), + error => { + assert.equal(error.message, 'OpenShell probe failed at exec'); + assert.equal(error.stdout, undefined); + return true; + } + ); +}); + +test('command runner handles a missing CLI without exposing process details', async () => { + await assert.rejects( + commandRunner('codeapi-nonexistent-probe-binary')([], { + stage: 'version', + timeoutMs: 1000, + }), + /failed at version/ + ); +});