Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion docs/remote-bridge/worker-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -533,7 +533,21 @@ command, cancellation, or settlement whose effects may be incomplete.
- **Worker online but not ready:** check native sandbox preparation, definition
validation, setup, quarantine, and readiness logs.
- **Setup repeats on restart:** setup is intentionally per-start; make it
idempotent or remove it.
idempotent, remove it, or opt in to the bounded `setup.reuse` readiness
contract in the [worker package guide](../../packages/code/README.md#reusing-a-prepared-checkout).
- **Dependency copies fill the disk:** declare shared npm/uv/browser resources
instead of per-worktree downloads. On a verified clone-capable filesystem,
configure private copy-on-write snapshots and their lifecycle budget. Do not
symlink another branch's mutable `node_modules` or hardlink writable installs.
- **Managed preparation deferred for low space:** `storage.minFreeBytes` plus
`setupReserveBytes` is a soft pre-setup floor, not a hard quota. Expand the
volume or clean reproducible artifacts; do not clear quarantine as a disk fix.
- **Snapshot maintenance:** preview with `prune-environment-storage
--environment <file>` and use `--apply` only after reviewing its JSON. Active,
unknown and unmarked data stays intact. Include all environment definitions
for root-isolation checks. This does not archive source worktrees or prune
mutable tool caches. Keep control state on separate storage when hard
protection from arbitrary build writes is required.
- **Git works on the host but not in tools:** verify the App installation,
permissions, private-key mode/owner, and allowed GitHub domains.
- **Repository label is present but files are absent:** `repo`/`ref` are
Expand Down
66 changes: 65 additions & 1 deletion packages/code/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1187,10 +1187,74 @@ Sharing is explicit within one worker's trust domain. Do not share mutable cache
between unrelated principals, store credentials in them, or treat their contents
as trusted worker code. Separate package caches from browsers and mutable browser
profiles, Redis data, build output and test state. Stores currently have no automatic
eviction; the storage-lifecycle slice adds that separately. This shares downloads
eviction. The snapshot lifecycle below does not delete these mutable stores. This shares downloads
and browser binaries, not installed `node_modules` trees.
No setup output is sent to the model.

## Storage admission and snapshot lifecycle

Use an explicit free-space floor before managed setup or restoration:

```yaml
storage:
minFreeBytes: 5368709120 # 5 GiB left for the worker and other activity
setupReserveBytes: 2147483648 # estimated installation headroom
```

This works without dependency snapshots. Absent `storage`, preparation retains
today's behavior. A ready checkout can still pass its readiness check when disk
is low. When installation is needed, low space rejects it **before** starting the
setup command, with a specific remediation message. These are soft admission
checks, not allocated reservations: concurrent processes and arbitrary shell
writes can still consume space after admission. Put worker identity/quarantine
state on a separate small volume and use filesystem/project quotas for hard
containment. This option neither clears quarantine nor deletes source to recover.

The private snapshot store can bound reproducible dependency versions:

```yaml
setup:
# command and reuse inputs/readiness omitted here; retain the full contract above
reuse:
snapshot:
store: /srv/lia-state/dependency-snapshots
paths: [node_modules]
lifecycle:
maxStoreBytes: 21474836480
maxEntries: 8
retentionMs: 432000000
scanLimit: 4096
```

Cleanup runs during managed preparation and publication. Oldest inactive snapshots
are reclaimed by last-use age and logical payload/count budgets. Per-key kernel
locks protect installs, restores and publishers; maintenance never waits on or
deletes an active key. A short store-wide lock makes budget checks and publication
atomic, without serializing installations for different keys. Abandoned staging
directories require a worker ownership manifest and a free key lock before removal.
Unknown/unmarked entries are reported and preserved. Lock files are kept to avoid
splitting lock ownership. Scan limits fail closed instead of crawling unbounded data.

The byte budget counts snapshot file lengths, not deduplicated physical blocks,
metadata overhead or working checkout copies. An otherwise valid prepared checkout
does not fail when the cache cannot fit another entry: publication is skipped with
a worker diagnostic, and later low-space admission still applies.

Preview or explicitly apply maintenance without restarting the worker:

```sh
librechat-code prune-environment-storage --environment /etc/librechat-code/app.yaml
librechat-code prune-environment-storage --environment /etc/librechat-code/app.yaml --apply
```

Pass all relevant definitions with repeated `--environment` flags so their roots
participate in isolation checks. Preview is the default and the JSON identifies
`dryRun`, proposed removals, active keys, unknown data and retained logical usage.
The command is host-operator-only, not an agent workspace action. It does not touch
source worktrees (including dirty or unpushed branches), `.verification`, build
outputs, arbitrary installations or mutable npm/uv caches. Source worktree archival
requires the separate worktree ownership/binding lifecycle, not an mtime heuristic.

Named actions are fixed commands without model-supplied substitution. The bridge
advertises only their names and the definition fingerprint, never their shell source
or host root. A command request can select `environmentAction: { name, fingerprint }`;
Expand Down
25 changes: 25 additions & 0 deletions packages/code/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import {
} from './environment.js';
import { prepareCodeEnvironment } from './environment-preparation.js';
import { assertEnvironmentResourceIsolation } from './environment-resources.js';
import { pruneDependencySnapshots } from './snapshot-lifecycle.js';
import { startFileRelay } from './relay.js';
import { DockerFileRelaySupervisor } from './relay-runtime.js';
import {
Expand Down Expand Up @@ -1191,6 +1192,7 @@ async function run(
});
await prepareCodeEnvironment({
snapshotStore: environment!.snapshotStore,
storage: environment!.definition.storage,
snapshotScope: environment!.definition.repo ?? environment!.definition.name,
root: instance.root, identity: instance.identity, setup,
receiptPath: preparationReceipt(instance.root),
Expand Down Expand Up @@ -1365,6 +1367,7 @@ async function run(
let armed = false;
const preparation = await prepareCodeEnvironment({
snapshotStore: environment.snapshotStore,
storage: environment.definition.storage,
snapshotScope: environment.definition.repo ?? environment.definition.name,
beforeMutation: async () => {
if (!armed) { await guard.arm('Dependency restoration did not settle', 'setup'); armed = true; }
Expand Down Expand Up @@ -1687,6 +1690,28 @@ async function clearMutationQuarantine(args: string[]): Promise<void> {

async function main(): Promise<void> {
const args = process.argv.slice(2);
if (args[0] === 'prune-environment-storage') {
const paths: string[] = [];
for (let i = 1; i < args.length; i++) {
if (args[i] === '--apply') continue;
if (args[i] !== '--environment' || !args[i + 1] || args[i + 1].startsWith('--'))
throw new Error('Usage: librechat-code prune-environment-storage --environment <file> [--environment <file> ...] [--apply]');
paths.push(args[++i]);
}
if (!paths.length || paths.length > 32) throw new Error('Supply between one and 32 environment definitions');
const loaded = await Promise.all(paths.map(loadCodeEnvironment));
const roots = loaded.map(environment => ({ id: environment.definition.name, root: environment.definition.root }));
const stores = loaded.flatMap(environment => environment.snapshotStore ? [environment.snapshotStore] : []);
if (!stores.length) throw new Error('No dependency snapshot stores configured');
await assertEnvironmentDefinitionsOutsideRoots(loaded, roots);
await assertEnvironmentDefinitionsOutsideRoots(stores.map(store => ({ path: store.store, sourceParents: store.controlPaths,
definition: { name: 'dependency-store', root: store.store }, fingerprint: '' })), roots);
await assertEnvironmentResourceIsolation(stores.map(store => ({ kind: 'npm-cache' as const, path: store.store, access: 'read-only' as const })),
roots.map(root => root.root), [...loaded.map(environment => environment.path),
...loaded.flatMap(environment => (environment.resources ?? []).map(resource => resource.path))]);
for (const store of stores) process.stdout.write(`${JSON.stringify(await pruneDependencySnapshots(store, { dryRun: !args.includes('--apply') }))}\n`);
return;
}
if (args[0] === 'projects') {
const root = option(args, '--root');
if (!root || args.slice(1).some((arg, index, rest) =>
Expand Down
17 changes: 17 additions & 0 deletions packages/code/src/dependency-snapshots.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,12 @@ test(
},
async t => {
const { a, b, store, directory, ai, bi } = await fixture(t);
store.lifecycle = {
maxStoreBytes: 10000,
maxEntries: 1,
retentionMs: 5 * 24 * 60 * 60 * 1000,
scanLimit: 100,
};
for (const root of [a, b])
await writeFile(join(root, 'package-lock.json'), 'version-one');
const commands: string[] = [];
Expand All @@ -204,6 +210,7 @@ test(
paths: store.paths,
maxBytes: store.maxBytes,
maxFiles: store.maxFiles,
lifecycle: store.lifecycle,
},
},
};
Expand Down Expand Up @@ -255,6 +262,16 @@ test(
commands.filter(command => command === setup.command).length,
2,
);
assert.equal(
(await readdir(store.store)).filter(name =>
/^[a-f0-9]{64}$/.test(name),
).length,
1,
);
assert.equal(
await readFile(join(a, 'node_modules', 'package.js'), 'utf8'),
'module.exports = 42',
);
const denied = await sandboxes[0].execute({
protocolVersion: 1,
operation: 'execute_command',
Expand Down
93 changes: 87 additions & 6 deletions packages/code/src/dependency-snapshots.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,21 @@ import {
import { loadEnvironmentResource } from './environment-resources.js';
import { isSafePortableRelativePath } from './protocol.js';
import type { WorkspaceRootIdentity } from './root-identity.js';
import {
parseSnapshotLifecycle,
pruneDependencySnapshots,
SNAPSHOT_MANIFEST,
STAGING_MANIFEST,
SnapshotBudgetFullError,
} from './snapshot-lifecycle.js';
import type { SnapshotLifecyclePolicy } from './snapshot-lifecycle.js';

export interface DependencySnapshotConfig {
store: string;
paths: string[];
maxBytes: number;
maxFiles: number;
lifecycle?: SnapshotLifecyclePolicy;
}
export interface DependencySnapshotStore extends DependencySnapshotConfig {
identity: WorkspaceRootIdentity;
Expand All @@ -41,7 +50,14 @@ export function parseDependencySnapshot(
const maxFiles = v.maxFiles ?? 200_000;
if (
Object.keys(v).some(
k => !['store', 'paths', 'maxBytes', 'maxFiles'].includes(k),
k =>
![
'store',
'paths',
'maxBytes',
'maxFiles',
'lifecycle',
].includes(k),
) ||
typeof v.store !== 'string' ||
!isAbsolute(v.store) ||
Expand Down Expand Up @@ -74,6 +90,9 @@ export function parseDependencySnapshot(
paths: v.paths as string[],
maxBytes: maxBytes as number,
maxFiles: maxFiles as number,
...(v.lifecycle !== undefined
? { lifecycle: parseSnapshotLifecycle(v.lifecycle) }
: {}),
};
}
export async function loadDependencySnapshot(
Expand Down Expand Up @@ -172,7 +191,7 @@ async function cloneTree(
config: DependencySnapshotConfig,
checkout: string,
signal?: AbortSignal,
): Promise<void> {
): Promise<{ bytes: number; files: number }> {
let bytes = 0,
files = 0;
const target = <T>(operation: () => Promise<T>) =>
Expand Down Expand Up @@ -285,6 +304,7 @@ async function cloneTree(
join(checkout, config.paths[i]),
);
});
return { bytes, files };
}

export async function withDependencySnapshot<T>(
Expand All @@ -306,6 +326,14 @@ export async function withDependencySnapshot<T>(
'Dependency snapshot store changed while waiting',
);
signal?.throwIfAborted();
if (store.lifecycle)
await pruneDependencySnapshots(store, {
currentKey: key,
signal,
}).catch(error => {
if (!(error instanceof SnapshotBudgetFullError))
throw error;
});
// Fail before installation, rather than discovering unsupported reflinks after npm ci.
if (verifyCloneSupport) {
const probe = await fs.mkdtemp(join(store.store, '.probe-'));
Expand All @@ -331,7 +359,25 @@ export async function withDependencySnapshot<T>(
}
}
signal?.throwIfAborted();
return operation();
const result = await operation();
const manifest = await fs
.open(
join(store.store, key, SNAPSHOT_MANIFEST),
constants.O_RDONLY | constants.O_NOFOLLOW,
)
.catch((e: NodeJS.ErrnoException) => {
if (e.code === 'ENOENT') return undefined;
throw e;
});
if (manifest) {
try {
const now = new Date();
await manifest.utimes(now, now);
} finally {
await manifest.close();
}
}
return result;
},
signal,
);
Expand Down Expand Up @@ -437,9 +483,20 @@ export async function publishDependencySnapshot(
)
)
return;
const staging = await fs.mkdtemp(join(store.store, '.staging-'));
const staging = await fs.mkdtemp(join(store.store, `.staging-${key}-`));
try {
await cloneTree(
await fs.writeFile(
join(staging, STAGING_MANIFEST),
JSON.stringify({
version: 1,
key,
bytes: 0,
files: 1,
createdAt: Date.now(),
}),
{ mode: 0o600 },
);
const measured = await cloneTree(
checkout,
identity,
staging,
Expand All @@ -450,7 +507,31 @@ export async function publishDependencySnapshot(
signal,
);
signal?.throwIfAborted();
await fs.rename(staging, destination);
await fs.writeFile(
join(staging, SNAPSHOT_MANIFEST),
JSON.stringify({
version: 1,
key,
...measured,
createdAt: Date.now(),
}),
{ mode: 0o600 },
);
const publish = () => fs.rename(staging, destination);
if (store.lifecycle)
await pruneDependencySnapshots(store, {
currentKey: key,
incomingBytes: measured.bytes,
incomingEntries: 1,
signal,
publish,
}).catch(error => {
if (!(error instanceof SnapshotBudgetFullError)) throw error;
process.stderr.write(
'librechat-code: dependency snapshot not cached because the store budget is full; prepared checkout remains valid\n',
);
});
else await publish();
} finally {
await fs.rm(staging, { recursive: true, force: true });
}
Expand Down
7 changes: 7 additions & 0 deletions packages/code/src/environment-preparation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ import {
publishDependencySnapshot,
} from './dependency-snapshots.js';
import type { DependencySnapshotStore } from './dependency-snapshots.js';
import { assertPreparationSpace } from './snapshot-lifecycle.js';
import type { EnvironmentStoragePolicy } from './snapshot-lifecycle.js';

// Safety bounds on operator-declared hashing, not a dependency-store quota.
const MAX_INPUT_BYTES = 8 * 1024 * 1024;
Expand All @@ -34,6 +36,7 @@ export interface EnvironmentPreparationOptions {
snapshotStore?: DependencySnapshotStore;
snapshotScope?: string;
beforeMutation?(): Promise<void>;
storage?: EnvironmentStoragePolicy;
}

/** Checkout-local reuse. Never transfers mutable installations between worktrees. */
Expand Down Expand Up @@ -83,6 +86,8 @@ async function prepareInLock(
return 'reused';
}
if (options.snapshotStore && portable) {
if (options.storage)
await assertPreparationSpace(options.root, options.storage);
await options.beforeMutation?.();
if (
await restoreDependencySnapshot(
Expand Down Expand Up @@ -112,6 +117,8 @@ async function prepareInLock(
}
}
}
if (options.storage)
await assertPreparationSpace(options.root, options.storage);
const result = await options.execute(
options.setup.command,
options.setup.timeoutMs,
Expand Down
Loading