diff --git a/docs/remote-bridge/worker-runbook.md b/docs/remote-bridge/worker-runbook.md index 8cbeda98..51018940 100644 --- a/docs/remote-bridge/worker-runbook.md +++ b/docs/remote-bridge/worker-runbook.md @@ -533,7 +533,21 @@ command, cancellation, or settlement whose effects may be incomplete. - **Worker online but not ready:** check native sandbox preparation, definition validation, setup, quarantine, and readiness logs. - **Setup repeats on restart:** setup is intentionally per-start; make it - idempotent or remove it. + idempotent, remove it, or opt in to the bounded `setup.reuse` readiness + contract in the [worker package guide](../../packages/code/README.md#reusing-a-prepared-checkout). +- **Dependency copies fill the disk:** declare shared npm/uv/browser resources + instead of per-worktree downloads. On a verified clone-capable filesystem, + configure private copy-on-write snapshots and their lifecycle budget. Do not + symlink another branch's mutable `node_modules` or hardlink writable installs. +- **Managed preparation deferred for low space:** `storage.minFreeBytes` plus + `setupReserveBytes` is a soft pre-setup floor, not a hard quota. Expand the + volume or clean reproducible artifacts; do not clear quarantine as a disk fix. +- **Snapshot maintenance:** preview with `prune-environment-storage + --environment ` and use `--apply` only after reviewing its JSON. Active, + unknown and unmarked data stays intact. Include all environment definitions + for root-isolation checks. This does not archive source worktrees or prune + mutable tool caches. Keep control state on separate storage when hard + protection from arbitrary build writes is required. - **Git works on the host but not in tools:** verify the App installation, permissions, private-key mode/owner, and allowed GitHub domains. - **Repository label is present but files are absent:** `repo`/`ref` are diff --git a/packages/code/README.md b/packages/code/README.md index a861efd3..38a22d5d 100644 --- a/packages/code/README.md +++ b/packages/code/README.md @@ -1187,10 +1187,74 @@ Sharing is explicit within one worker's trust domain. Do not share mutable cache between unrelated principals, store credentials in them, or treat their contents as trusted worker code. Separate package caches from browsers and mutable browser profiles, Redis data, build output and test state. Stores currently have no automatic -eviction; the storage-lifecycle slice adds that separately. This shares downloads +eviction. The snapshot lifecycle below does not delete these mutable stores. This shares downloads and browser binaries, not installed `node_modules` trees. No setup output is sent to the model. +## Storage admission and snapshot lifecycle + +Use an explicit free-space floor before managed setup or restoration: + +```yaml +storage: + minFreeBytes: 5368709120 # 5 GiB left for the worker and other activity + setupReserveBytes: 2147483648 # estimated installation headroom +``` + +This works without dependency snapshots. Absent `storage`, preparation retains +today's behavior. A ready checkout can still pass its readiness check when disk +is low. When installation is needed, low space rejects it **before** starting the +setup command, with a specific remediation message. These are soft admission +checks, not allocated reservations: concurrent processes and arbitrary shell +writes can still consume space after admission. Put worker identity/quarantine +state on a separate small volume and use filesystem/project quotas for hard +containment. This option neither clears quarantine nor deletes source to recover. + +The private snapshot store can bound reproducible dependency versions: + +```yaml +setup: + # command and reuse inputs/readiness omitted here; retain the full contract above + reuse: + snapshot: + store: /srv/lia-state/dependency-snapshots + paths: [node_modules] + lifecycle: + maxStoreBytes: 21474836480 + maxEntries: 8 + retentionMs: 432000000 + scanLimit: 4096 +``` + +Cleanup runs during managed preparation and publication. Oldest inactive snapshots +are reclaimed by last-use age and logical payload/count budgets. Per-key kernel +locks protect installs, restores and publishers; maintenance never waits on or +deletes an active key. A short store-wide lock makes budget checks and publication +atomic, without serializing installations for different keys. Abandoned staging +directories require a worker ownership manifest and a free key lock before removal. +Unknown/unmarked entries are reported and preserved. Lock files are kept to avoid +splitting lock ownership. Scan limits fail closed instead of crawling unbounded data. + +The byte budget counts snapshot file lengths, not deduplicated physical blocks, +metadata overhead or working checkout copies. An otherwise valid prepared checkout +does not fail when the cache cannot fit another entry: publication is skipped with +a worker diagnostic, and later low-space admission still applies. + +Preview or explicitly apply maintenance without restarting the worker: + +```sh +librechat-code prune-environment-storage --environment /etc/librechat-code/app.yaml +librechat-code prune-environment-storage --environment /etc/librechat-code/app.yaml --apply +``` + +Pass all relevant definitions with repeated `--environment` flags so their roots +participate in isolation checks. Preview is the default and the JSON identifies +`dryRun`, proposed removals, active keys, unknown data and retained logical usage. +The command is host-operator-only, not an agent workspace action. It does not touch +source worktrees (including dirty or unpushed branches), `.verification`, build +outputs, arbitrary installations or mutable npm/uv caches. Source worktree archival +requires the separate worktree ownership/binding lifecycle, not an mtime heuristic. + Named actions are fixed commands without model-supplied substitution. The bridge advertises only their names and the definition fingerprint, never their shell source or host root. A command request can select `environmentAction: { name, fingerprint }`; diff --git a/packages/code/src/cli.ts b/packages/code/src/cli.ts index 4c64c0ae..cf05e263 100644 --- a/packages/code/src/cli.ts +++ b/packages/code/src/cli.ts @@ -14,6 +14,7 @@ import { } from './environment.js'; import { prepareCodeEnvironment } from './environment-preparation.js'; import { assertEnvironmentResourceIsolation } from './environment-resources.js'; +import { pruneDependencySnapshots } from './snapshot-lifecycle.js'; import { startFileRelay } from './relay.js'; import { DockerFileRelaySupervisor } from './relay-runtime.js'; import { @@ -1191,6 +1192,7 @@ async function run( }); await prepareCodeEnvironment({ snapshotStore: environment!.snapshotStore, + storage: environment!.definition.storage, snapshotScope: environment!.definition.repo ?? environment!.definition.name, root: instance.root, identity: instance.identity, setup, receiptPath: preparationReceipt(instance.root), @@ -1365,6 +1367,7 @@ async function run( let armed = false; const preparation = await prepareCodeEnvironment({ snapshotStore: environment.snapshotStore, + storage: environment.definition.storage, snapshotScope: environment.definition.repo ?? environment.definition.name, beforeMutation: async () => { if (!armed) { await guard.arm('Dependency restoration did not settle', 'setup'); armed = true; } @@ -1687,6 +1690,28 @@ async function clearMutationQuarantine(args: string[]): Promise { async function main(): Promise { const args = process.argv.slice(2); + if (args[0] === 'prune-environment-storage') { + const paths: string[] = []; + for (let i = 1; i < args.length; i++) { + if (args[i] === '--apply') continue; + if (args[i] !== '--environment' || !args[i + 1] || args[i + 1].startsWith('--')) + throw new Error('Usage: librechat-code prune-environment-storage --environment [--environment ...] [--apply]'); + paths.push(args[++i]); + } + if (!paths.length || paths.length > 32) throw new Error('Supply between one and 32 environment definitions'); + const loaded = await Promise.all(paths.map(loadCodeEnvironment)); + const roots = loaded.map(environment => ({ id: environment.definition.name, root: environment.definition.root })); + const stores = loaded.flatMap(environment => environment.snapshotStore ? [environment.snapshotStore] : []); + if (!stores.length) throw new Error('No dependency snapshot stores configured'); + await assertEnvironmentDefinitionsOutsideRoots(loaded, roots); + await assertEnvironmentDefinitionsOutsideRoots(stores.map(store => ({ path: store.store, sourceParents: store.controlPaths, + definition: { name: 'dependency-store', root: store.store }, fingerprint: '' })), roots); + await assertEnvironmentResourceIsolation(stores.map(store => ({ kind: 'npm-cache' as const, path: store.store, access: 'read-only' as const })), + roots.map(root => root.root), [...loaded.map(environment => environment.path), + ...loaded.flatMap(environment => (environment.resources ?? []).map(resource => resource.path))]); + for (const store of stores) process.stdout.write(`${JSON.stringify(await pruneDependencySnapshots(store, { dryRun: !args.includes('--apply') }))}\n`); + return; + } if (args[0] === 'projects') { const root = option(args, '--root'); if (!root || args.slice(1).some((arg, index, rest) => diff --git a/packages/code/src/dependency-snapshots.test.ts b/packages/code/src/dependency-snapshots.test.ts index b23ae70f..f7d59681 100644 --- a/packages/code/src/dependency-snapshots.test.ts +++ b/packages/code/src/dependency-snapshots.test.ts @@ -188,6 +188,12 @@ test( }, async t => { const { a, b, store, directory, ai, bi } = await fixture(t); + store.lifecycle = { + maxStoreBytes: 10000, + maxEntries: 1, + retentionMs: 5 * 24 * 60 * 60 * 1000, + scanLimit: 100, + }; for (const root of [a, b]) await writeFile(join(root, 'package-lock.json'), 'version-one'); const commands: string[] = []; @@ -204,6 +210,7 @@ test( paths: store.paths, maxBytes: store.maxBytes, maxFiles: store.maxFiles, + lifecycle: store.lifecycle, }, }, }; @@ -255,6 +262,16 @@ test( commands.filter(command => command === setup.command).length, 2, ); + assert.equal( + (await readdir(store.store)).filter(name => + /^[a-f0-9]{64}$/.test(name), + ).length, + 1, + ); + assert.equal( + await readFile(join(a, 'node_modules', 'package.js'), 'utf8'), + 'module.exports = 42', + ); const denied = await sandboxes[0].execute({ protocolVersion: 1, operation: 'execute_command', diff --git a/packages/code/src/dependency-snapshots.ts b/packages/code/src/dependency-snapshots.ts index 5c9ab173..89a76935 100644 --- a/packages/code/src/dependency-snapshots.ts +++ b/packages/code/src/dependency-snapshots.ts @@ -20,12 +20,21 @@ import { import { loadEnvironmentResource } from './environment-resources.js'; import { isSafePortableRelativePath } from './protocol.js'; import type { WorkspaceRootIdentity } from './root-identity.js'; +import { + parseSnapshotLifecycle, + pruneDependencySnapshots, + SNAPSHOT_MANIFEST, + STAGING_MANIFEST, + SnapshotBudgetFullError, +} from './snapshot-lifecycle.js'; +import type { SnapshotLifecyclePolicy } from './snapshot-lifecycle.js'; export interface DependencySnapshotConfig { store: string; paths: string[]; maxBytes: number; maxFiles: number; + lifecycle?: SnapshotLifecyclePolicy; } export interface DependencySnapshotStore extends DependencySnapshotConfig { identity: WorkspaceRootIdentity; @@ -41,7 +50,14 @@ export function parseDependencySnapshot( const maxFiles = v.maxFiles ?? 200_000; if ( Object.keys(v).some( - k => !['store', 'paths', 'maxBytes', 'maxFiles'].includes(k), + k => + ![ + 'store', + 'paths', + 'maxBytes', + 'maxFiles', + 'lifecycle', + ].includes(k), ) || typeof v.store !== 'string' || !isAbsolute(v.store) || @@ -74,6 +90,9 @@ export function parseDependencySnapshot( paths: v.paths as string[], maxBytes: maxBytes as number, maxFiles: maxFiles as number, + ...(v.lifecycle !== undefined + ? { lifecycle: parseSnapshotLifecycle(v.lifecycle) } + : {}), }; } export async function loadDependencySnapshot( @@ -172,7 +191,7 @@ async function cloneTree( config: DependencySnapshotConfig, checkout: string, signal?: AbortSignal, -): Promise { +): Promise<{ bytes: number; files: number }> { let bytes = 0, files = 0; const target = (operation: () => Promise) => @@ -285,6 +304,7 @@ async function cloneTree( join(checkout, config.paths[i]), ); }); + return { bytes, files }; } export async function withDependencySnapshot( @@ -306,6 +326,14 @@ export async function withDependencySnapshot( 'Dependency snapshot store changed while waiting', ); signal?.throwIfAborted(); + if (store.lifecycle) + await pruneDependencySnapshots(store, { + currentKey: key, + signal, + }).catch(error => { + if (!(error instanceof SnapshotBudgetFullError)) + throw error; + }); // Fail before installation, rather than discovering unsupported reflinks after npm ci. if (verifyCloneSupport) { const probe = await fs.mkdtemp(join(store.store, '.probe-')); @@ -331,7 +359,25 @@ export async function withDependencySnapshot( } } signal?.throwIfAborted(); - return operation(); + const result = await operation(); + const manifest = await fs + .open( + join(store.store, key, SNAPSHOT_MANIFEST), + constants.O_RDONLY | constants.O_NOFOLLOW, + ) + .catch((e: NodeJS.ErrnoException) => { + if (e.code === 'ENOENT') return undefined; + throw e; + }); + if (manifest) { + try { + const now = new Date(); + await manifest.utimes(now, now); + } finally { + await manifest.close(); + } + } + return result; }, signal, ); @@ -437,9 +483,20 @@ export async function publishDependencySnapshot( ) ) return; - const staging = await fs.mkdtemp(join(store.store, '.staging-')); + const staging = await fs.mkdtemp(join(store.store, `.staging-${key}-`)); try { - await cloneTree( + await fs.writeFile( + join(staging, STAGING_MANIFEST), + JSON.stringify({ + version: 1, + key, + bytes: 0, + files: 1, + createdAt: Date.now(), + }), + { mode: 0o600 }, + ); + const measured = await cloneTree( checkout, identity, staging, @@ -450,7 +507,31 @@ export async function publishDependencySnapshot( signal, ); signal?.throwIfAborted(); - await fs.rename(staging, destination); + await fs.writeFile( + join(staging, SNAPSHOT_MANIFEST), + JSON.stringify({ + version: 1, + key, + ...measured, + createdAt: Date.now(), + }), + { mode: 0o600 }, + ); + const publish = () => fs.rename(staging, destination); + if (store.lifecycle) + await pruneDependencySnapshots(store, { + currentKey: key, + incomingBytes: measured.bytes, + incomingEntries: 1, + signal, + publish, + }).catch(error => { + if (!(error instanceof SnapshotBudgetFullError)) throw error; + process.stderr.write( + 'librechat-code: dependency snapshot not cached because the store budget is full; prepared checkout remains valid\n', + ); + }); + else await publish(); } finally { await fs.rm(staging, { recursive: true, force: true }); } diff --git a/packages/code/src/environment-preparation.ts b/packages/code/src/environment-preparation.ts index 73df85f3..8dbb8751 100644 --- a/packages/code/src/environment-preparation.ts +++ b/packages/code/src/environment-preparation.ts @@ -14,6 +14,8 @@ import { publishDependencySnapshot, } from './dependency-snapshots.js'; import type { DependencySnapshotStore } from './dependency-snapshots.js'; +import { assertPreparationSpace } from './snapshot-lifecycle.js'; +import type { EnvironmentStoragePolicy } from './snapshot-lifecycle.js'; // Safety bounds on operator-declared hashing, not a dependency-store quota. const MAX_INPUT_BYTES = 8 * 1024 * 1024; @@ -34,6 +36,7 @@ export interface EnvironmentPreparationOptions { snapshotStore?: DependencySnapshotStore; snapshotScope?: string; beforeMutation?(): Promise; + storage?: EnvironmentStoragePolicy; } /** Checkout-local reuse. Never transfers mutable installations between worktrees. */ @@ -83,6 +86,8 @@ async function prepareInLock( return 'reused'; } if (options.snapshotStore && portable) { + if (options.storage) + await assertPreparationSpace(options.root, options.storage); await options.beforeMutation?.(); if ( await restoreDependencySnapshot( @@ -112,6 +117,8 @@ async function prepareInLock( } } } + if (options.storage) + await assertPreparationSpace(options.root, options.storage); const result = await options.execute( options.setup.command, options.setup.timeoutMs, diff --git a/packages/code/src/environment.ts b/packages/code/src/environment.ts index ca859a2e..576a149f 100644 --- a/packages/code/src/environment.ts +++ b/packages/code/src/environment.ts @@ -6,6 +6,8 @@ import { parseDocument } from 'yaml'; import { parseEnvironmentResources, loadEnvironmentResource } from './environment-resources.js'; import { parseDependencySnapshot, loadDependencySnapshot } from './dependency-snapshots.js'; import type { DependencySnapshotConfig, DependencySnapshotStore } from './dependency-snapshots.js'; +import { parseEnvironmentStorage } from './snapshot-lifecycle.js'; +import type { EnvironmentStoragePolicy } from './snapshot-lifecycle.js'; import type { EnvironmentResource, LoadedEnvironmentResource } from './environment-resources.js'; import { assertPrivateStorageAcl, @@ -38,6 +40,7 @@ export interface CodeEnvironmentDefinition { }; actions?: { name: string; command: string; timeoutMs: number }[]; resources?: EnvironmentResource[]; + storage?: EnvironmentStoragePolicy; } export interface LoadedCodeEnvironment { @@ -80,7 +83,7 @@ export function parseCodeEnvironment( !record(value) || Object.keys(value).some( key => - !['name', 'root', 'repo', 'ref', 'setup', 'actions', 'resources'].includes( + !['name', 'root', 'repo', 'ref', 'setup', 'actions', 'resources', 'storage'].includes( key, ), ) || @@ -182,6 +185,7 @@ export function parseCodeEnvironment( ...(setup ? { setup } : {}), ...(actions ? { actions } : {}), ...(value.resources !== undefined ? { resources: parseEnvironmentResources(value.resources) } : {}), + ...(value.storage !== undefined ? { storage: parseEnvironmentStorage(value.storage) } : {}), }; } diff --git a/packages/code/src/process-lock.ts b/packages/code/src/process-lock.ts index aeca561a..6e456d50 100644 --- a/packages/code/src/process-lock.ts +++ b/packages/code/src/process-lock.ts @@ -5,6 +5,7 @@ import { setTimeout as delay } from 'node:timers/promises'; const LOCK_EX = 2; const LOCK_NB = 4; const LOCK_UN = 8; +export class ProcessLockBusyError extends Error {} let binding: | Promise<{ flock: (fd: number, operation: number) => number; @@ -26,7 +27,8 @@ async function lockBinding() { export async function withProcessLock( path: string, operation: () => Promise, - signal?: AbortSignal + signal?: AbortSignal, + wait = true, ): Promise { signal?.throwIfAborted(); if (process.platform !== 'darwin' && process.platform !== 'linux') { @@ -48,6 +50,7 @@ export async function withProcessLock( `Conversation worktree lock failed with errno ${errno}` ); } + if (!wait) throw new ProcessLockBusyError('Process lock is active'); await delay(50, undefined, { signal }); } signal?.throwIfAborted(); @@ -57,3 +60,9 @@ export async function withProcessLock( await handle.close(); } } + +/** Non-blocking maintenance admission. Never masks operation errors as lock contention. */ +export async function tryWithProcessLock(path: string, operation: () => Promise, signal?: AbortSignal): Promise<{ acquired: true; value: T } | { acquired: false }> { + try { return { acquired: true, value: await withProcessLock(path, operation, signal, false) }; } + catch (error) { if (error instanceof ProcessLockBusyError) return { acquired: false }; throw error; } +} diff --git a/packages/code/src/snapshot-lifecycle.test.ts b/packages/code/src/snapshot-lifecycle.test.ts new file mode 100644 index 00000000..33e37c2d --- /dev/null +++ b/packages/code/src/snapshot-lifecycle.test.ts @@ -0,0 +1,226 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { + mkdtemp, + mkdir, + realpath, + writeFile, + readFile, + rm, + lstat, + utimes, + symlink, +} from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { execFile } from 'node:child_process'; +import { promisify } from 'node:util'; +import { + loadDependencySnapshot, + parseDependencySnapshot, +} from './dependency-snapshots.js'; +import { + assertPreparationSpace, + parseEnvironmentStorage, + parseSnapshotLifecycle, + pruneDependencySnapshots, + SNAPSHOT_MANIFEST, + STAGING_MANIFEST, +} from './snapshot-lifecycle.js'; +import { withProcessLock } from './process-lock.js'; +import { prepareCodeEnvironment } from './environment-preparation.js'; +import { captureWorkspaceRootIdentity } from './root-identity.js'; + +async function fixture(t: test.TestContext) { + const root = await realpath( + await mkdtemp(join(tmpdir(), 'snapshot-lifecycle-')), + ); + t.after(() => rm(root, { recursive: true, force: true })); + const path = join(root, 'store'); + await mkdir(path, { mode: 0o700 }); + const store = await loadDependencySnapshot( + parseDependencySnapshot({ + store: path, + paths: ['node_modules'], + lifecycle: { maxStoreBytes: 10, maxEntries: 2, retentionMs: 1000 }, + }), + ); + const add = async (key: string, bytes = 4, age = 0) => { + const directory = join(store.store, key); + await mkdir(directory, { mode: 0o700 }); + await writeFile( + join(directory, SNAPSHOT_MANIFEST), + JSON.stringify({ + version: 1, + key, + bytes, + files: 1, + createdAt: Date.now(), + }), + { mode: 0o600 }, + ); + const lastUsed = new Date(Date.now() - age); + await utimes(join(directory, SNAPSHOT_MANIFEST), lastUsed, lastUsed); + return directory; + }; + return { + root, + store, + add, + a: 'a'.repeat(64), + b: 'b'.repeat(64), + c: 'c'.repeat(64), + }; +} + +test('storage policy is opt-in, bounded and fails before a setup command', async t => { + assert.equal(parseEnvironmentStorage({}).minFreeBytes, 5 * 1024 ** 3); + assert.throws(() => parseEnvironmentStorage({ minFreeBytes: -1 })); + assert.throws(() => parseSnapshotLifecycle({ retentionMs: Infinity })); + await assertPreparationSpace( + '/', + { minFreeBytes: 5, setupReserveBytes: 2 }, + async () => 7n, + ); + await assert.rejects( + assertPreparationSpace( + '/', + { minFreeBytes: 5, setupReserveBytes: 2 }, + async () => 6n, + ), + /no setup command was started/, + ); + const { root } = await fixture(t); + const checkout = join(root, 'checkout'); + await mkdir(checkout); + const identity = await captureWorkspaceRootIdentity(checkout); + let executions = 0; + await assert.rejects( + prepareCodeEnvironment({ + root: checkout, + identity, + setup: { command: 'install', timeoutMs: 1000 }, + receiptPath: join(root, 'receipt'), + context: '', + storage: { + minFreeBytes: Number.MAX_SAFE_INTEGER, + setupReserveBytes: 0, + }, + execute: async () => { + executions++; + return { exitCode: 0, timedOut: false }; + }, + }), + /deferred/, + ); + assert.equal(executions, 0); +}); + +test('dry-run is non-destructive; cleanup removes only expired owned entries and retains unknown or linked data', async t => { + const { store, add, a, b, c, root } = await fixture(t); + const old = await add(a, 4, 10_000); + await add(b); + await mkdir(join(store.store, 'unrecognized')); + await writeFile(join(store.store, 'unrecognized', 'source'), 'keep'); + await symlink(root, join(store.store, c)); + const preview = await pruneDependencySnapshots(store, { dryRun: true }); + assert.deepEqual(preview.removed, [a]); + assert.equal((await lstat(old)).isDirectory(), true); + const actual = await pruneDependencySnapshots(store); + assert.deepEqual(actual.removed, [a]); + await assert.rejects(lstat(old), { code: 'ENOENT' }); + assert.equal( + await readFile(join(store.store, 'unrecognized', 'source'), 'utf8'), + 'keep', + ); + assert.equal((await lstat(join(store.store, c))).isSymbolicLink(), true); + assert.equal(actual.retainedBytes, 4); +}); + +test('cleanup preserves active keys and current preparation even when over budget', async t => { + const { store, add, a, b } = await fixture(t); + await add(a, 4, 10_000); + await add(b, 4, 10_000); + await withProcessLock(join(store.store, `${a}.lock`), async () => { + const result = await pruneDependencySnapshots(store, { currentKey: b }); + assert.deepEqual(result.removed, []); + assert.deepEqual(result.skippedActive, [a]); + await assert.rejects( + pruneDependencySnapshots(store, { + currentKey: b, + incomingBytes: 10, + }), + /budget is full/, + ); + }); + assert.equal((await lstat(join(store.store, a))).isDirectory(), true); +}); + +test('publication budgets evict LRU entries and serialize commits without deleting source', async t => { + const { store, add, a, b, c, root } = await fixture(t); + await add(a, 4, 500); + await add(b, 4, 100); + const source = join(root, 'dirty-worktree'); + await mkdir(source); + await writeFile(join(source, 'uncommitted'), 'keep'); + let published = false; + const result = await pruneDependencySnapshots(store, { + currentKey: c, + incomingBytes: 4, + incomingEntries: 1, + publish: async () => { + published = true; + }, + }); + assert.deepEqual(result.removed, [a]); + assert.equal(published, true); + assert.equal(await readFile(join(source, 'uncommitted'), 'utf8'), 'keep'); +}); + +test('operator CLI previews by default and applies only after explicit flag', async t => { + const { store, add, a, root } = await fixture(t); + await add(a, 4, 10_000); + const checkout = join(root, 'checkout'); + await mkdir(checkout); + const config = join(root, 'environment.yaml'); + await writeFile( + config, + `name: app\nroot: ${checkout}\nsetup:\n command: npm ci\n reuse:\n inputs: [package-lock.json]\n checkCommand: test -d node_modules\n snapshot:\n store: ${store.store}\n paths: [node_modules]\n lifecycle:\n retentionMs: 1000\n`, + { mode: 0o600 }, + ); + const run = promisify(execFile); + const cli = new URL('./cli.js', import.meta.url).pathname; + const args = [cli, 'prune-environment-storage', '--environment', config]; + const preview = await run(process.execPath, args); + assert.deepEqual(JSON.parse(preview.stdout).removed, [a]); + assert.equal((await lstat(join(store.store, a))).isDirectory(), true); + const applied = await run(process.execPath, [...args, '--apply']); + assert.deepEqual(JSON.parse(applied.stdout).removed, [a]); + await assert.rejects(lstat(join(store.store, a)), { code: 'ENOENT' }); +}); + +test('abandoned staging is reclaimed only with an ownership manifest and a free key lock', async t => { + const { store, a } = await fixture(t); + const name = `.staging-${a}-abcdef`; + const directory = join(store.store, name); + await mkdir(directory, { mode: 0o700 }); + await writeFile( + join(directory, STAGING_MANIFEST), + JSON.stringify({ + version: 1, + key: a, + bytes: 0, + files: 1, + createdAt: Date.now(), + }), + { mode: 0o600 }, + ); + await withProcessLock(join(store.store, `${a}.lock`), async () => { + const active = await pruneDependencySnapshots(store); + assert.deepEqual(active.skippedActive, [name]); + assert.deepEqual(active.removed, []); + }); + const result = await pruneDependencySnapshots(store); + assert.deepEqual(result.removed, [name]); + await assert.rejects(lstat(directory), { code: 'ENOENT' }); +}); diff --git a/packages/code/src/snapshot-lifecycle.ts b/packages/code/src/snapshot-lifecycle.ts new file mode 100644 index 00000000..dfa5926e --- /dev/null +++ b/packages/code/src/snapshot-lifecycle.ts @@ -0,0 +1,316 @@ +import { constants } from 'node:fs'; +import { lstat, open, rm, statfs } from 'node:fs/promises'; +import { join } from 'node:path'; +import { withProcessLock, tryWithProcessLock } from './process-lock.js'; +import { matchesWorkspaceRoot } from './root-identity.js'; +import { readdir, withWorkspaceRoot } from './root-access.js'; +import type { DependencySnapshotStore } from './dependency-snapshots.js'; + +export interface SnapshotLifecyclePolicy { + maxStoreBytes: number; + maxEntries: number; + retentionMs: number; + scanLimit: number; +} +export interface EnvironmentStoragePolicy { + minFreeBytes: number; + setupReserveBytes: number; +} +function positive(value: unknown, zero = false): value is number { + return ( + typeof value === 'number' && + Number.isSafeInteger(value) && + value >= (zero ? 0 : 1) + ); +} +export function parseSnapshotLifecycle( + value: unknown, +): SnapshotLifecyclePolicy { + if (!value || typeof value !== 'object' || Array.isArray(value)) + throw new Error('Invalid snapshot lifecycle'); + const v = value as Record; + const policy = { + maxStoreBytes: v.maxStoreBytes ?? 20 * 1024 ** 3, + maxEntries: v.maxEntries ?? 8, + retentionMs: v.retentionMs ?? 5 * 24 * 60 * 60 * 1000, + scanLimit: v.scanLimit ?? 4096, + }; + if ( + Object.keys(v).some(key => !Object.hasOwn(policy, key)) || + !Object.values(policy).every(value => positive(value)) + ) + throw new Error('Invalid snapshot lifecycle'); + return policy as SnapshotLifecyclePolicy; +} +export function parseEnvironmentStorage( + value: unknown, +): EnvironmentStoragePolicy { + if (!value || typeof value !== 'object' || Array.isArray(value)) + throw new Error('Invalid environment storage policy'); + const v = value as Record; + const policy = { + minFreeBytes: v.minFreeBytes ?? 5 * 1024 ** 3, + setupReserveBytes: v.setupReserveBytes ?? 2 * 1024 ** 3, + }; + if ( + Object.keys(v).some(key => !Object.hasOwn(policy, key)) || + !Object.values(policy).every(value => positive(value, true)) || + !Number.isSafeInteger( + (policy.minFreeBytes as number) + + (policy.setupReserveBytes as number), + ) + ) + throw new Error('Invalid environment storage policy'); + return policy as EnvironmentStoragePolicy; +} +/** Soft admission for managed preparation, not a reservation or arbitrary-write quota. */ +export async function assertPreparationSpace( + root: string, + policy: EnvironmentStoragePolicy, + available: (root: string) => Promise = async path => { + const status = await statfs(path, { bigint: true }); + return status.bavail * status.bsize; + }, +): Promise { + if ( + (await available(root)) < + BigInt(policy.minFreeBytes) + BigInt(policy.setupReserveBytes) + ) + throw new Error( + 'Managed environment preparation deferred: free disk is below the configured floor plus setup reserve. Clean reproducible artifacts or expand storage; no setup command was started.', + ); +} + +export const SNAPSHOT_MANIFEST = '.snapshot.json'; +export const STAGING_MANIFEST = '.staging.json'; +export class SnapshotBudgetFullError extends Error {} +export interface SnapshotManifest { + version: 1; + key: string; + bytes: number; + files: number; + createdAt: number; +} +async function manifest( + store: DependencySnapshotStore, + key: string, + name = key, + file = SNAPSHOT_MANIFEST, +): Promise { + const directory = join(store.store, name); + const status = await lstat(directory); + if (!status.isDirectory() || status.isSymbolicLink() || status.mode & 0o077) + return; + const handle = await open( + join(directory, file), + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK, + ).catch((e: NodeJS.ErrnoException) => { + if (e.code === 'ENOENT' || e.code === 'ELOOP') return undefined; + throw e; + }); + if (!handle) return; + try { + const metadata = await handle.stat(); + if ( + !metadata.isFile() || + metadata.size > 4096 || + metadata.mode & 0o077 || + (process.getuid && metadata.uid !== process.getuid()) + ) + return; + const buffer = Buffer.alloc(4097); + const { bytesRead } = await handle.read(buffer, 0, buffer.length, 0); + if (bytesRead > 4096) return; + let value: unknown; + try { + value = JSON.parse(buffer.subarray(0, bytesRead).toString()); + } catch { + return; + } + if (!value || typeof value !== 'object' || Array.isArray(value)) return; + const entry = value as Record; + if ( + entry.version !== 1 || + entry.key !== key || + !positive(entry.bytes, true) || + !positive(entry.files) || + !positive(entry.createdAt) + ) + return; + return value as SnapshotManifest; + } finally { + await handle.close(); + } +} +export interface SnapshotPruneResult { + dryRun: boolean; + removed: string[]; + skippedActive: string[]; + unknown: string[]; + retainedBytes: number; + retainedEntries: number; +} + +/** Only worker-published manifests authorize deletion. Source worktrees and unknown entries never qualify. */ +export async function pruneDependencySnapshots( + store: DependencySnapshotStore, + options: { + dryRun?: boolean; + currentKey?: string; + incomingBytes?: number; + incomingEntries?: number; + now?: number; + signal?: AbortSignal; + publish?: () => Promise; + } = {}, +): Promise { + const policy = store.lifecycle; + if (!policy) throw new Error('Snapshot lifecycle is not configured'); + return withProcessLock( + join(store.store, '.maintenance.lock'), + async () => { + if (!(await matchesWorkspaceRoot(store.store, store.identity))) + throw new Error( + 'Dependency snapshot store changed before cleanup', + ); + const result: SnapshotPruneResult = { + dryRun: options.dryRun === true, + removed: [], + skippedActive: [], + unknown: [], + retainedBytes: 0, + retainedEntries: 0, + }; + const candidates: { + key: string; + data: SnapshotManifest; + lastUsed: number; + }[] = []; + const names = await withWorkspaceRoot( + store.store, + store.identity, + () => readdir(store.store, policy.scanLimit), + ); + for (const key of names) { + options.signal?.throwIfAborted(); + const staging = /^\.staging-([a-f0-9]{64})-[A-Za-z0-9]+$/.exec( + key, + ); + if (staging) { + const owner = staging[1]; + const data = await manifest( + store, + owner, + key, + STAGING_MANIFEST, + ); + if (!data) { + result.unknown.push(key); + continue; + } + if (owner === options.currentKey) continue; + const lock = await tryWithProcessLock( + join(store.store, `${owner}.lock`), + async () => { + if ( + !(await manifest( + store, + owner, + key, + STAGING_MANIFEST, + )) + ) + return; + if (!options.dryRun) + await rm(join(store.store, key), { + recursive: true, + force: false, + }); + result.removed.push(key); + }, + options.signal, + ); + if (!lock.acquired) result.skippedActive.push(key); + continue; + } + if (!/^[a-f0-9]{64}$/.test(key)) { + if ( + !/^[a-f0-9]{64}\.lock$/.test(key) && + key !== '.maintenance.lock' + ) + result.unknown.push(key); + continue; + } + const data = await manifest(store, key); + if (!data) { + result.unknown.push(key); + continue; + } + candidates.push({ + key, + data, + lastUsed: ( + await lstat(join(store.store, key, SNAPSHOT_MANIFEST)) + ).mtimeMs, + }); + result.retainedBytes += data.bytes; + result.retainedEntries++; + } + const now = options.now ?? Date.now(); + candidates.sort( + (a, b) => a.lastUsed - b.lastUsed || a.key.localeCompare(b.key), + ); + for (const candidate of candidates) { + options.signal?.throwIfAborted(); + if (candidate.key === options.currentKey) continue; + if ( + now - candidate.lastUsed < policy.retentionMs && + result.retainedBytes + (options.incomingBytes ?? 0) <= + policy.maxStoreBytes && + result.retainedEntries + (options.incomingEntries ?? 0) <= + policy.maxEntries + ) + continue; + const admission = await tryWithProcessLock( + join(store.store, `${candidate.key}.lock`), + async () => { + const fresh = await manifest(store, candidate.key); + const updated = await lstat( + join(store.store, candidate.key, SNAPSHOT_MANIFEST), + ); + if ( + !fresh || + fresh.createdAt !== candidate.data.createdAt || + updated.mtimeMs !== candidate.lastUsed + ) + return; + if (!options.dryRun) + await rm(join(store.store, candidate.key), { + recursive: true, + force: false, + }); + result.removed.push(candidate.key); + result.retainedBytes -= candidate.data.bytes; + result.retainedEntries--; + }, + options.signal, + ); + if (!admission.acquired) + result.skippedActive.push(candidate.key); + } + if ( + !options.dryRun && + (result.retainedBytes + (options.incomingBytes ?? 0) > + policy.maxStoreBytes || + result.retainedEntries + (options.incomingEntries ?? 0) > + policy.maxEntries) + ) + throw new SnapshotBudgetFullError( + 'Dependency snapshot budget is full; active or unknown data was not deleted', + ); + if (!options.dryRun) await options.publish?.(); + return result; + }, + options.signal, + ); +}