feat(deploy): two-phase stage/activate for deploy_component (draft/RFC)

DESIGN.md

@@ -338,6 +338,143 @@ this fix doesn't attempt to solve. `deploy_component`/`package_component` still
 declared entry points (`jsResource`/`graphqlSchema`) survived extraction — a truncation from some
 other future cause would still report success silently; that's a deferred, separate fix.
 
+## Two-phase deploy: stage then activate (`components/Application.ts`, `components/operations.js`)
+
+`deploy_component` runs internally as two replicated phases so a cluster deploy is all-or-nothing at
+the point of go-live. **Phase 1 (stage)** builds the incoming version — download/`npm pack` (incl. a
+git clone), extract, `npm install` — into a hidden staging directory on every node. **Phase 2
+(activate)** atomically renames the staged copy into the live component path and restarts. The origin
+stages locally, **waits for every node to report a successful stage before any node activates**
+(`ignore_replication_errors` opts out of the barrier), then activates. If a node can't fetch the
+package or fails `npm install`, it fails during staging while the live component is still untouched _on
+every node_ — where the old one-shot path could leave a peer half-installed after other peers had
+already restarted onto the new code. The request/response contract is unchanged; only the SSE phase
+names differ (`stage`/`activate` vs the old `prepare`/`replicate`). `two_phase: false` forces the
+legacy one-shot path.
+
+**There is only one public operation — `deploy_component`.** The two phases are NOT separate public
+operations; the peer fan-out is `deploy_component` itself tagged with an internal `_phase: 'stage' |
+'activate'` marker (the same `_`-prefixed internal-field convention peers already branch on, alongside
+`_deploymentId`). `deployComponent` dispatches: a replicated execution with `_phase` runs the peer
+stage/activate work (`deployPhaseStage` / `deployPhaseActivate`) and never re-fans; a public call runs
+the origin orchestrator. Two public properties expose the phases when an operator wants them separated
+(e.g. pre-stage the cluster now, flip later — or a CI-stages / approver-activates split): `activate:
+false` stages cluster-wide and stops, returning the `deployment_id` in a `staged` state; passing that
+`deployment_id` back to `deploy_component` (with no new payload) activates the already-staged build.
+This was a deliberate API-surface choice (harper#1849 review): peer fan-out needs a wire format, not
+two extra public ops, and folding the phases into `deploy_component` keeps the surface at one op while
+the convergence properties cover the stage-now/activate-later use case. (`revert_component` stays a
+distinct public op — it is a rollback, not a deploy phase.)
+
+**Scope of the barrier's guarantee: fetch + install, not load.** The cluster-wide "nobody activates
+until everybody staged" guarantee covers the download/`npm pack` and `npm install` steps — the slow,
+failure-prone work. The pre-go-live component _load_ check (`loadValidateComponent`, which surfaces a
+component that installs cleanly but throws at load) runs during stage on the origin and on any node
+whose stage executes on a worker (e.g. the op-API worker for an `activate: false` stage), but it is a
+no-op on the main thread — and replicated peer stage executions run on the main thread
+(`replicateOperation` → `sendOperationToNode` execute there), where app code deliberately isn't
+loaded. So a load-time-only fault on a peer is not caught by the barrier; it surfaces at
+activate/restart like any other. Gating load-time faults cluster-wide would require dispatching the
+throwaway load to a worker on each peer during stage — a possible follow-up, not done here.
+
+The staging directory (`.deploy-staging/<deploymentId>/<name>`) lives **under the components root**,
+not in `os.tmpdir()`, even though its contents are transient. This is deliberate and load-bearing:
+the go-live step is `rename(stagingDir, liveDir)`, which is only atomic when both paths share a
+filesystem. `os.tmpdir()` is frequently a different mount (tmpfs, a separate volume); a cross-device
+rename throws `EXDEV` and Node has no atomic fallback — you'd be back to a slow recursive copy at the
+exact moment you want the swap to be instantaneous, reintroducing the downtime window the split
+exists to remove. The leading dot keeps `loadComponentDirectories` from loading it as a phantom
+component, and it is **not** the watched base of any component's file watcher (those are rooted at
+each live component dir, `EntryHandler`/`deriveCommonPatternBase`) — so building here fires no
+restart-on-change events and needs no `deploy:start` watcher suppression. That suppression is now
+scoped to `activateApplication`, the only phase that writes the live path. Staging is deterministic
+from the deployment id precisely so the activate phase (a separate replicated `deploy_component`
+invocation on peers, tagged `_phase: 'activate'`) can reconstruct the same path the stage built —
+peers build a fresh `Application` per phase invocation, so there is no shared in-memory handle to rely
+on. The deployment id sits ABOVE the component name (`…/<deploymentId>/<name>`, not
+`…/<name>/<deploymentId>`) for two reasons: the leaf directory's basename is then the real component
+name, which the pre-go-live validation load needs (`componentLoader` keys the `ApplicationScope` and
+status registry off `basename(componentDirectory)`, so a UUID leaf would register the throwaway load
+under a bogus name); and each deploy gets its own parent directory, so a parallel or queued deploy of
+the same component can never share a directory or have its staged build swept by another's cleanup.
+`extractApplication`/`installApplication` build into `application.buildDirPath`, which defaults
+to the live dir (`dirPath`) — this is what keeps the legacy one-shot path, boot-time
+`installApplications`, and the direct `extractApplication` callers unchanged — and is repointed at
+the staging dir only for the duration of a stage.
+
+Two-phase requires the `system` database to be replicated on the origin (`isSystemDatabaseReplicated`),
+since the `hdb_deployment` row's `payload_blob` is how peers fetch the tarball and correlate the two
+phases by deployment id. When `system` is excluded from a narrow `REPLICATION_DATABASES`, or the
+caller passes `two_phase: false`, or the invocation is a peer replaying a one-shot deploy,
+`deploy_component` falls back to `deployComponentOneShot` (the previous behavior, preserved verbatim).
+Cross-version skew is a non-issue by policy — a cluster stays in lockstep on its Harper version, so
+every node understands the `_phase`-tagged `deploy_component` fan-out — which is why there is no
+capability negotiation on it.
+
+**Replicator contract this rides on (`harper-pro/replication/replicator.ts`).**
+`server.replication.replicateOperation(op, {onPeerResult})` fans `op` to every node in `server.nodes`
+in parallel, setting `op.replicated = false` on the copy it sends so a peer never re-fans (the deploy
+handlers additionally detect a replicated execution by the presence of `_deploymentId` — always set on
+the sub-operations — and run the peer stage/activate work off the `_phase` marker without re-fanning). Per-peer failures never throw — `sendOperationToNode` rejections are caught and
+surface as `{status:'failed', reason, node}` entries in the returned `replicated[]` array and via
+`onPeerResult`, which is exactly the shape `DeploymentRecorder.normalizePeerResult` consumes. Peers
+authenticate node-to-node by TLS certificate, and the receive side runs the op via
+`server.operation(data, {user}, !isAuthorizedNode)` — for a trusted cluster node the authorize flag is
+`false`, so a replicated super-user op skips the permission gate. That is why the `_phase`-tagged
+`deploy_component` fan-out and `revert_component` (registered with the same `permission(true, [])`,
+dispatched by `operation` name) replicate without an `hdb_user`, identically to the long-proven
+one-shot `deploy_component` fan-out.
+
+**Reversibility: retained previous + `revert_component`.** `activateApplication` no longer discards the
+outgoing live version — it retains it as `.deploy-previous/<name>` (`retainAsPrevious`, evicting the
+older one so exactly one previous is kept per component). `revert_component` swaps the live directory
+with that retained previous via three same-filesystem renames through a hidden holding path, cluster-
+wide and replicated like activate. The swap is bidirectional, so reverting a revert rolls forward
+again. This backs two things: a customer can deploy, run their own health checks against the live
+version, and `revert` if unhappy even when the cluster looks healthy; and `deploy_component`'s opt-in
+`revert_on_failure` rolls the whole cluster back to the previous version when the activate phase leaves
+some nodes live and some not, so the cluster reconverges on one version. The previous copy is retained
+per-node (each node retains its own outgoing version during its own activate), so a replicated revert
+has a local rollback source on every node.
+
+**Staged-build retention.** A full deploy consumes its staged build immediately (activate renames it
+live), so the only builds that accumulate are `activate: false` stage-and-stops that are never
+activated — each leaves `.deploy-staging/<deploymentId>/<name>` in place so a later
+`deploy_component({deployment_id})` can activate it. `stageApplication` bounds this: after a successful
+stage it evicts the oldest not-yet-activated staged builds for that component beyond
+`deployment_stagingRetention_maxCount` (default 5, `pruneStagedBuilds`), always keeping the just-staged
+one and the newest N−1 by mtime. Eviction is best-effort (`allSettled`, trace-logged) but awaited so
+the count is settled when the stage returns. Retention is deliberately count-only and automatic:
+per the harper#1849 discussion, `hdb_deployment` rows stay as the audit trail (payload blobs already
+self-reclaim by size, `deployment_payloadRetention_maxSize`), and no `delete_deployment` op was added —
+eviction-on-stage keeps the surface at zero new operations. Consequence: activating a `deployment_id`
+that has already aged out of the window fails with "no staged build found" — expected once more than
+`maxCount` newer stages have landed for that component.
+
+**Payload retention.** Two independent bounds apply to the tarballs stored in `hdb_deployment`'s
+`payload_blob`, and they answer different questions:
+
+- `deployment_payloadRetention_maxSize` (default 10 MiB) — reclaims _this_ deploy's payload right after
+  it succeeds, if the tarball was large. Bounds the size of any single retained payload.
+- `deployment_payloadRetention_maxCount` (default **1**, `pruneProjectPayloads`) — keeps at most N
+  stored payloads _per project_, newest first, dropping the `payload_blob` of the rest after a
+  successful deploy. Bounds how many payloads accumulate over time, which is what actually caps disk.
+
+Only rows that still hold a payload count toward `maxCount`, so the cap reads literally as "at most N
+stored payloads per project." Rows are never deleted — pruning nulls the blob and appends a
+`payload_dropped` event, so the audit trail and `get_deployment` stay intact; only
+`get_deployment_payload` stops working for pruned deployments (`payload_blob_present: false`). A
+non-terminal deployment is counted but never dropped: its blob may still be the replication channel
+peers are installing from (the same guard `delete_deployment_payload` uses). Pruning is best-effort and
+off the deploy's critical path — a prune failure is logged, never fatal — and is skipped entirely when a
+peer failed, since the older payloads are the retry artifact in that case.
+
+The default of 1 is deliberately conservative rather than a redeploy-window convenience: instances on
+small quotas (free tier is 5 GB total) must not have N copies of a large app payload quietly competing
+with the customer's own data. Operators who want a wider redeploy-by-reference window raise it
+explicitly; 0 retains none. Note that an explicit `delete_deployment_payload` (harper#1893) forfeits
+redeployability for that deployment the same way an automatic prune does.
+
 ## Scheduler: cluster-once execution without a consensus primitive (`resources/scheduler/`)
 
 The built-in `scheduler` plugin (#951) runs config-declared jobs "exactly once per cluster." The

bin/cliOperations.ts

@@ -18,7 +18,34 @@ import { DeployRenderer } from './deployRenderer.ts';
 import { getHdbPid } from '../utility/processManagement/processManagement.js';
 import { initConfig, getConfigPath } from '../config/configUtils.ts';
 
-const OP_ALIASES = { deploy: 'deploy_component', package: 'package_component' };
+const OP_ALIASES = {
+	deploy: 'deploy_component',
+	package: 'package_component',
+	revert: 'revert_component',
+};
+
+// CLI verbs that are sugar over `deploy_component` with preset properties (the stage/activate phases
+// are folded into deploy_component; there are no separate stage/activate operations). `harper stage`
+// packages + uploads the incoming version to a hidden staging dir cluster-wide and stops before
+// go-live (`activate: false`), printing the staged deployment_id; `harper activate deployment_id=<id>`
+// takes that staged deployment live (no upload).
+const OP_VERB_PROPS: Record<string, Record<string, unknown>> = {
+	stage: { operation: 'deploy_component', activate: false },
+	// `_verb` is a CLI-internal marker (stripped before the request) so verbRequirementError can enforce
+	// that `harper activate` carries a deployment_id — without it, deploy_component's generic
+	// "no deployment_id → full deploy" fallback would silently build a brand-new deploy from the CWD.
+	activate: { operation: 'deploy_component', _verb: 'activate' },
+};
+
+// Guard CLI-verb requirements that the operation itself can't enforce (the op has no notion of which
+// verb invoked it). Returns an error message, or null when the request is fine. Pure + exported so it
+// is unit-testable without the network/process-exit machinery in cliOperations.
+function verbRequirementError(req: any): string | null {
+	if (req._verb === 'activate' && !req.deployment_id) {
+		return '`harper activate` requires a deployment_id from a prior `harper stage` — usage: harper activate project=<name> deployment_id=<id>';
+	}
+	return null;
+}
 
 // Shown for any local-instance connection failure (missing pid, missing/stale domain
 // socket, or a refused/ENOENT connect against it) — they're all the same user-facing
@@ -31,7 +58,7 @@ const LOCAL_NOT_RUNNING_MESSAGE = 'Harper is not running. Use `harperdb run` (or
 // deploy completes. Add an operation here only after wiring its server-side
 // SSE_PROGRESS_OPERATIONS entry — otherwise the server returns the buffered JSON path and
 // the SSE parser sees no events.
-const SSE_OPERATIONS = new Set(['deploy_component']);
+const SSE_OPERATIONS = new Set(['deploy_component', 'revert_component']);
 
 // Properties on `req` that the CLI itself uses for transport/UX, not the operations API.
 // They never get serialized into the request body. `username`/`password` are deliberately
@@ -249,39 +276,49 @@ function redactCredentials(req: any): any {
 	return redacted;
 }
 
-export { cliOperations, buildRequest, redactCredentials, refreshExpiredOperationToken };
-const PREPARE_OPERATION: any = {
-	deploy_component: async (req) => {
-		if (req.package) {
-			return;
-		}
+export { cliOperations, buildRequest, redactCredentials, refreshExpiredOperationToken, verbRequirementError };
 
-		const projectPath = process.cwd();
-		if (!req.project) req.project = path.basename(projectPath);
-		const packageOptions = {
-			skip_node_modules: req.skip_node_modules !== false,
-			skip_symlinks: req.skip_symlinks === true,
-		};
-		// Store path + options for deferred stream creation after the renderer is set up,
-		// so the pre-gzip onBytes callback can be wired directly to renderer.countUploadBytes.
-		req._projectPath = projectPath;
-		req._packageOptions = packageOptions;
-		// Pre-walk the directory once for both the uncompressed-size estimate (progress bar
-		// total) and the dangling-symlink list — a dangling symlink would otherwise silently
-		// truncate the tarball (tar-fs finalizes early on the broken target). Packaging skips
-		// them; the list is reused below (no second walk) and warns the user which links were
-		// skipped so the omission is visible.
-		const scan = await scanPackageDirectory(projectPath, packageOptions);
-		req._uploadSizeEstimate = scan.totalSize;
-		req._danglingSymlinks = scan.danglingSymlinks;
-		if (scan.danglingSymlinks.length) {
-			process.stderr.write(
-				`warning: skipping ${scan.danglingSymlinks.length} broken symlink(s) — their linked content will NOT be deployed:\n` +
-					scan.danglingSymlinks.map((p) => `  ${p}\n`).join('')
-			);
-		}
-		req._multipart = true;
-	},
+// Package the current working directory into a multipart tarball upload for deploy_component. Covers
+// `harper deploy` and `harper stage` (deploy_component with activate:false) — both upload the incoming
+// version. Nothing to package when a `package` identifier is given (the server fetches it) or when
+// activating a previously-staged deployment (`deployment_id`, i.e. `harper activate`).
+const packageCwdForUpload = async (req) => {
+	if (req.package || req.deployment_id) {
+		return;
+	}
+
+	const projectPath = process.cwd();
+	if (!req.project) req.project = path.basename(projectPath);
+	const packageOptions = {
+		skip_node_modules: req.skip_node_modules !== false,
+		skip_symlinks: req.skip_symlinks === true,
+	};
+	// Store path + options for deferred stream creation after the renderer is set up,
+	// so the pre-gzip onBytes callback can be wired directly to renderer.countUploadBytes.
+	req._projectPath = projectPath;
+	req._packageOptions = packageOptions;
+	// Pre-walk the directory once for both the uncompressed-size estimate (progress bar
+	// total) and the dangling-symlink list — a dangling symlink would otherwise silently
+	// truncate the tarball (tar-fs finalizes early on the broken target). Packaging skips
+	// them; the list is reused below (no second walk) and warns the user which links were
+	// skipped so the omission is visible.
+	const scan = await scanPackageDirectory(projectPath, packageOptions);
+	req._uploadSizeEstimate = scan.totalSize;
+	req._danglingSymlinks = scan.danglingSymlinks;
+	if (scan.danglingSymlinks.length) {
+		process.stderr.write(
+			`warning: skipping ${scan.danglingSymlinks.length} broken symlink(s) — their linked content will NOT be deployed:\n` +
+				scan.danglingSymlinks.map((p) => `  ${p}\n`).join('')
+		);
+	}
+	req._multipart = true;
+};
+
+const PREPARE_OPERATION: any = {
+	// deploy_component covers `harper deploy` and `harper stage` (activate:false); packageCwdForUpload
+	// itself skips the upload for a `package` identifier or a `deployment_id` activate. revert takes no
+	// payload, so it needs no prep step.
+	deploy_component: packageCwdForUpload,
 };
 
 /**
@@ -292,6 +329,9 @@ function buildRequest(): any {
 	for (const arg of process.argv.slice(2)) {
 		if (OP_ALIASES.hasOwnProperty(arg)) {
 			req.operation = OP_ALIASES[arg];
+		} else if (OP_VERB_PROPS.hasOwnProperty(arg)) {
+			// Sugar verb (stage/activate) → deploy_component + preset props (e.g. activate:false).
+			Object.assign(req, OP_VERB_PROPS[arg]);
 		} else if (arg.includes('=')) {
 			let [first, ...rest] = arg.split('=');
 			let restStr: any = rest.join('=');
@@ -420,6 +460,14 @@ async function cliOperations(req: any, skipResponseLog = false) {
 			process.exit(1);
 		}
 	}
+	// Enforce CLI-verb requirements (e.g. `harper activate` needs a deployment_id) BEFORE packaging so a
+	// mistake fails fast instead of building + uploading a fresh deploy from the CWD.
+	const verbError = verbRequirementError(req);
+	if (verbError) {
+		console.error(verbError);
+		process.exit(1);
+	}
+	delete req._verb; // CLI-internal marker; never send it in the request body
 	await PREPARE_OPERATION[req.operation]?.(req);
 	try {
 		let options = target ?? {