diff --git a/CHANGELOG.md b/CHANGELOG.md index 8bfb556..3619523 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,85 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 argv (`ps`, shell history), and a password field in a native window has neither. +- **The project env vault** ([`pb env`](docs/env-vault.md)) — the variables a + *project* needs, held the way the key vault holds credentials: names and + provenance in `~/.config/patchbay/projects.json` (`0600`), values in the macOS + Keychain, and no plaintext `.env` anywhere. A project is a portable **name**, + not a path: the manifest holds ids, environments and sync config and no + absolute path at all, so copying it to another machine is the supported way to + take your projects with you. Each of its environments has two layers: `synced`, + which `pb env pull` replaces wholesale from Infisical, and `local`, which you + set by hand and which wins on merge. Those are `.env.local` semantics, and + they only hold because **patchbay never pushes** — there is no code path that + writes a variable to a remote, so a local override is invisible to the cloud + by construction rather than by policy, and a pull can never carry your + container's `DATABASE_URL` into the team's shared set. Values are stored one + Keychain item per project × environment × layer (account + `env://`), holding the whole layer as one JSON + blob, so an export is one Keychain round trip and not one per variable. No + `last4` is recorded: four characters of `true` or `5432` is not a hint, it is + the value. `pb env pull` also pins the account a project belongs to and checks + it before spending a subprocess — the Infisical CLI's active login is + machine-global, and under the wrong one the API answers 403 with "project does + not belong to your selected organization", which reads like a problem with the + project rather than with the login; patchbay refuses first and names + `pb use infisical ` instead. +- **Two ways a directory resolves to a project**, in that order. An + **attachment** (`pb env attach ` / `pb env detach`) binds a directory on + this machine, in `~/.config/patchbay/attachments.json` — deepest attached + ancestor wins, several roots per project, so every worktree and second clone + shares one vault. A **marker** — a committed `.patchbay.toml` holding + `project = ""`, written by `pb env init` unless `--no-marker` — resolves + a checkout by its content, so a fresh `git clone` works on any machine whose + registry holds that project, with no attach step. An attachment always beats a + marker: a deliberate local act outranks whatever the repo ships, and nothing + in a repo can take that override back. A marker can only *name* a project the + machine already has, and one that names an unknown project is a loud error + pointing at the machine's `projects.json` rather than a silent miss. The + tradeoff, taken deliberately: repo content selects which registered project's + variables the tooling hands out, which assumes you run repos you trust. +- **Moving to a new machine** is therefore: bring `projects.json` over, clone the + repo (the marker travels with it), `pb env pull`. `pb export` carries that + manifest inside the bundle and `pb import` registers what is not here yet, so + the migration path is the normal route and copying the file by hand is the + fallback; a project id the destination already has is skipped with a note + rather than overwritten, because the machine in front of you may be the newer + one. What does **not** travel: `attachments.json`, since those are paths from + a machine that is not this one; every variable value in either layer; and the + local layer's variable *names* along with them, because a name with no value + behind it would make `pb env list` promise what `pb env run` could not + deliver. `SETUP.md`, the `pb plan` checklist and the `plan_setup` MCP tool + each carry one `pb env pull --project ` per linked project, marked + `auto: false` and naming the pinned account — a pull under the wrong + machine-global infisical login fails confusingly, so `pb use infisical + ` may have to come first. A project the old machine had *unlinked* but + with a synced layer is a gap instead: nothing on the new machine can rebuild + it. +- **`pb env`** — `init` (registers the project, attaches this directory, leaves + a marker to commit, picking up `.infisical.json`), `attach`, `detach`, + `link`, `projects`, `list`, `pull`, `set`, `unset`, `import`, `diff`, `run`, + `export`, `forget`. `init` in a worktree of a project this machine already + knows attaches it instead of failing on the duplicate id; `forget` takes the + project, its Keychain blobs and this machine's attachments, and leaves + committed markers alone (`rm .patchbay.toml`). `list` and `diff` answer from + the name lists alone and never touch the Keychain; `set` takes its value from + stdin or a hidden prompt, never argv; `run -- ` injects the merged + environment into one child process and is the blessed read path, with + `export` (dotenv or JSON, TTY warning) there for the cases where a file is + genuinely what you need. + `import ` bulk-loads an existing `.env` into the local layer, + all-or-nothing, reporting a bad line by number and never by content. +- **MCP tools** — `list_env_projects`, `list_env_vars`, `pull_env` and + `set_env_var`. `list_env_projects` reports each project's machine-local + `roots` alongside its environments, and says what an empty list means, so an + agent does not read a path there as where the user is working. The first two + are metadata only; `pull_env` executes the Infisical CLI but its outcome + carries counts and names, not values, so it is ungated; + `set_env_var` is open like `store_key`, so an agent that creates a + project credential registers it. Nothing reads a value back — not even behind + `PATCHBAY_ALLOW_SECRET_READ`. An environment is dozens of secrets at once, + and `pb env run` in your own terminal is the answer instead. + ## [0.2.0] - 2026-08-13 ### Added @@ -119,73 +198,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 values, since those fields routinely hold API keys. A `copy` does carry values (a server that cannot authenticate is useless) and names what travelled. -- **The project env vault** ([`pb env`](docs/env-vault.md)) — the variables a - *project* needs, held the way the key vault holds credentials: names and - provenance in `~/.config/patchbay/projects.json` (`0600`), values in the macOS - Keychain, and no plaintext `.env` anywhere. A project is a portable **name**, - not a path: the manifest holds ids, environments and sync config and no - absolute path at all, so copying it to another machine is the supported way to - take your projects with you. Each of its environments has two layers: `synced`, - which `pb env pull` replaces wholesale from Infisical, and `local`, which you - set by hand and which wins on merge. Those are `.env.local` semantics, and - they only hold because **patchbay never pushes** — there is no code path that - writes a variable to a remote, so a local override is invisible to the cloud - by construction rather than by policy, and a pull can never carry your - container's `DATABASE_URL` into the team's shared set. Values are stored one - Keychain item per project × environment × layer (account - `env://`), holding the whole layer as one JSON - blob, so an export is one Keychain round trip and not one per variable. No - `last4` is recorded: four characters of `true` or `5432` is not a hint, it is - the value. `pb env pull` also pins the account a project belongs to and checks - it before spending a subprocess — the Infisical CLI's active login is - machine-global, and under the wrong one the API answers 403 with "project does - not belong to your selected organization", which reads like a problem with the - project rather than with the login; patchbay refuses first and names - `pb use infisical ` instead. -- **Two ways a directory resolves to a project**, in that order. An - **attachment** (`pb env attach ` / `pb env detach`) binds a directory on - this machine, in `~/.config/patchbay/attachments.json` — deepest attached - ancestor wins, several roots per project, so every worktree and second clone - shares one vault. A **marker** — a committed `.patchbay.toml` holding - `project = ""`, written by `pb env init` unless `--no-marker` — resolves - a checkout by its content, so a fresh `git clone` works on any machine whose - registry holds that project, with no attach step. An attachment always beats a - marker: a deliberate local act outranks whatever the repo ships, and nothing - in a repo can take that override back. A marker can only *name* a project the - machine already has, and one that names an unknown project is a loud error - pointing at the machine's `projects.json` rather than a silent miss. The - tradeoff, taken deliberately: repo content selects which registered project's - variables the tooling hands out, which assumes you run repos you trust. -- **Moving to a new machine** is therefore: copy `projects.json`, clone the repo - (the marker travels with it), `pb env pull`. Attachments deliberately do not - travel — they are paths from a machine that is not this one — and neither does - the local layer, since a `DATABASE_URL` pointing at a container on the old - laptop is exactly what must not follow you. -- **`pb env`** — `init` (registers the project, attaches this directory, leaves - a marker to commit, picking up `.infisical.json`), `attach`, `detach`, - `link`, `projects`, `list`, `pull`, `set`, `unset`, `import`, `diff`, `run`, - `export`, `forget`. `init` in a worktree of a project this machine already - knows attaches it instead of failing on the duplicate id; `forget` takes the - project, its Keychain blobs and this machine's attachments, and leaves - committed markers alone (`rm .patchbay.toml`). `list` and `diff` answer from - the name lists alone and never touch the Keychain; `set` takes its value from - stdin or a hidden prompt, never argv; `run -- ` injects the merged - environment into one child process and is the blessed read path, with - `export` (dotenv or JSON, TTY warning) there for the cases where a file is - genuinely what you need. - `import ` bulk-loads an existing `.env` into the local layer, - all-or-nothing, reporting a bad line by number and never by content. -- **MCP tools** — `list_env_projects`, `list_env_vars`, `pull_env` and - `set_env_var`. `list_env_projects` reports each project's machine-local - `roots` alongside its environments, and says what an empty list means, so an - agent does not read a path there as where the user is working. The first two - are metadata only; `pull_env` executes the Infisical CLI but its outcome - carries counts and names, not values, so it is ungated; - `set_env_var` is open like `store_key`, so an agent that creates a - project credential registers it. Nothing reads a value back — not even behind - `PATCHBAY_ALLOW_SECRET_READ`. An environment is dozens of secrets at once, - and `pb env run` in your own terminal is the answer instead. - ### Changed - `patchbay_core::util` now owns the write-safety machinery MCP client diff --git a/README.md b/README.md index d9ee2f0..197f0de 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ - **Permissions** — see what your tokens can actually do (`gh` scopes today) and fix missing scopes with one hint. - **[MCP client management](docs/mcp-clients.md)** — every MCP server registered in Claude Code, Claude Desktop, Cursor, Codex, Windsurf and VS Code in one matrix; copy a server between clients without hand-editing four files in two formats. - **[Key vault](docs/key-vault.md)** — standalone API keys no CLI tracks: values in the macOS Keychain, metadata on disk, provider-aware `pb key verify`, and AI registration over MCP. -- **[Project env vault](docs/env-vault.md)** — a project's environment variables without a plaintext `.env`: pull from Infisical, keep hand-set local overrides that never sync back, run a command with the merged result. A project is a portable name, not a path — copy one file to a new machine, clone the repo, pull. +- **[Project env vault](docs/env-vault.md)** — a project's environment variables without a plaintext `.env`: pull from Infisical, keep hand-set local overrides that never sync back, run a command with the merged result. A project is a portable name, not a path — `pb export` carries the manifest to a new machine (or copy the one file), clone the repo, pull. - **[Keeping CLIs current](#keeping-clis-current)** — which tools are outdated, which were renamed out from under you, and the exact command to update each one. - **Migrate** — export to a new machine; whatever can't travel, your AI walks you through re-authing. - **[Migrate](docs/migration.md)** — export to a new machine; whatever can't travel, your AI walks you through re-authing. @@ -102,6 +102,11 @@ each becomes one line with the command that fixes it. Your AI can work that list over MCP (`plan_setup`, `mark_setup_done`), and patchbay re-probes after every step rather than believing it. +The [env vault](docs/env-vault.md)'s projects ride along as metadata — ids, +environments and sync pins, so the new machine knows what to pull. No variable +value travels, in either layer, and neither does this machine's list of which +directories belong to which project. + Encrypted with a passphrase, refuses to be written into a cloud-sync folder, and never copies a private key. **[Full details, and the per-tool portability table →](docs/migration.md)** diff --git a/crates/patchbay-cli/src/main.rs b/crates/patchbay-cli/src/main.rs index 9ab0ca0..7561daa 100644 --- a/crates/patchbay-cli/src/main.rs +++ b/crates/patchbay-cli/src/main.rs @@ -155,10 +155,12 @@ fn run() -> Result { if let Some(manifest) = diff { let vault = KeyRegistry::detect()?; let clients = McpClientRegistry::with_paths(registry.paths().clone()); + let envs = patchbay_core::EnvRegistry::detect()?; return migrate::print_status_diff( ®istry, &vault, &clients, + &envs, &manifest, &styles(), ); diff --git a/crates/patchbay-cli/src/migrate.rs b/crates/patchbay-cli/src/migrate.rs index d7f43fa..020fd8f 100644 --- a/crates/patchbay-cli/src/migrate.rs +++ b/crates/patchbay-cli/src/migrate.rs @@ -19,7 +19,7 @@ use patchbay_core::migrate::{ self, export, import, manifest::SetupStatus, Exporter, ImportOptions, Importer, KeySelection, Manifest, SetupItem, }; -use patchbay_core::{KeyRegistry, McpClientRegistry, Registry}; +use patchbay_core::{EnvRegistry, KeyRegistry, McpClientRegistry, Registry}; use crate::render::{self, Styles}; @@ -69,6 +69,7 @@ pub fn run(command: Command, styles: &Styles) -> Result { let paths = registry.paths().clone(); let vault = KeyRegistry::detect()?; let clients = McpClientRegistry::with_paths(paths.clone()); + let envs = EnvRegistry::detect()?; match command { Command::Export { @@ -88,6 +89,7 @@ pub fn run(command: Command, styles: &Styles) -> Result { registry: ®istry, vault: &vault, clients: &clients, + envs: &envs, } .payload(&selection, Utc::now())?; @@ -120,6 +122,7 @@ pub fn run(command: Command, styles: &Styles) -> Result { registry: ®istry, vault: &vault, clients: &clients, + envs: &envs, } .run(&payload, &ImportOptions { dry_run })?; @@ -137,7 +140,14 @@ pub fn run(command: Command, styles: &Styles) -> Result { json, } => { let manifest = manifest.as_deref().map(read_manifest).transpose()?; - let items = migrate::plan(&paths, ®istry, &vault, &clients, manifest.as_ref()); + let items = migrate::plan( + &paths, + ®istry, + &vault, + &clients, + &envs, + manifest.as_ref(), + ); let shown: Vec<&SetupItem> = items .iter() .filter(|i| all || i.status != SetupStatus::Done) @@ -160,11 +170,19 @@ pub fn print_status_diff( registry: &Registry, vault: &KeyRegistry, clients: &McpClientRegistry, + envs: &EnvRegistry, manifest: &std::path::Path, styles: &Styles, ) -> Result { let manifest = read_manifest(manifest)?; - let items = migrate::plan(registry.paths(), registry, vault, clients, Some(&manifest)); + let items = migrate::plan( + registry.paths(), + registry, + vault, + clients, + envs, + Some(&manifest), + ); let open: Vec<&SetupItem> = items.iter().filter(|i| i.is_open()).collect(); println!( "{} of {} things the other machine had are not true here", @@ -266,6 +284,15 @@ fn print_export(report: &export::ExportReport, styles: &Styles) { ); } } + if !report.env_projects.is_empty() { + // Metadata only, and saying so here is the point: an env project in a + // bundle is a name, not a set of variables. + println!( + " env: {} project(s), names only ({})", + report.env_projects.len(), + report.env_projects.join(", ") + ); + } println!( " {} item(s) will need doing on the new machine", report.gaps @@ -313,6 +340,16 @@ fn print_import(report: &import::ImportReport, styles: &Styles) { server.name ); } + for project in &report.env_projects { + println!( + " {:<10} env project {}", + project.outcome.label(), + project.id + ); + if let import::FileOutcome::Skipped { reason } = &project.outcome { + println!(" {reason}"); + } + } println!(); for note in &report.notes { println!("{}", styles.paint(warn_style(), &format!("! {note}"))); @@ -397,6 +434,7 @@ fn export_json(report: &export::ExportReport) -> serde_json::Value { "keys_listed_only": report.keys_listed, "mcp_servers": report.mcp_carried, "mcp_value_names_carried": report.mcp_values_carried, + "env_projects": report.env_projects, "gaps": report.gaps, "warnings": report.warnings, }) @@ -427,6 +465,9 @@ fn import_json(report: &import::ImportReport) -> serde_json::Value { "action": m.outcome.label(), "value_names_carried": m.values_carried, })).collect::>(), + "env_projects": report.env_projects.iter().map(|p| serde_json::json!({ + "id": p.id, "action": p.outcome.label(), + })).collect::>(), "notes": report.notes, "remaining": report.remaining, }) diff --git a/crates/patchbay-core/src/envs.rs b/crates/patchbay-core/src/envs.rs index 7981eb7..8569dd6 100644 --- a/crates/patchbay-core/src/envs.rs +++ b/crates/patchbay-core/src/envs.rs @@ -25,8 +25,9 @@ //! work. See [`EnvRegistry::find_by_dir`] for the precedence rule and the //! tradeoff that buys. //! -//! **Taking your environment to a new machine** is therefore: copy -//! `projects.json` over and `pb env pull` to rebuild every synced layer from the +//! **Taking your environment to a new machine** is therefore: get +//! `projects.json` over — `pb export` carries it inside the bundle, or copy the +//! file by hand — and `pb env pull` to rebuild every synced layer from the //! remote. Checkouts carrying a marker resolve on their own; anything else takes //! one `pb env attach `. The local layer deliberately does *not* travel. //! `.env.local` semantics are per-machine overrides, and a `DATABASE_URL` @@ -672,6 +673,58 @@ impl EnvRegistry { Ok(entry) } + /// Take a project entry exactly as another machine had it — the write path + /// [`crate::migrate::import`] uses, and the only one that sets + /// `environments` and `sync` wholesale. + /// + /// [`EnvRegistry::register`] is the interactive route and is deliberately + /// narrow: an id and a default environment, because everything else is + /// earned by a later command. A migration bundle carries a project as a + /// *record* instead, so restoring one is a single write rather than a + /// replay of the commands that built it. + /// + /// **An id that already exists is refused**, and the caller reports the + /// skip. The destination may be the newer machine, and quietly replacing + /// its sync pin or its environment list with a copy from a bundle is not + /// something an import gets to do behind the user's back. + /// + /// **Every environment's `local_names` is cleared on the way in**, whatever + /// the caller passed. No local *value* can travel — the local layer is + /// per-machine overrides by definition — and a name list with no values + /// behind it would make `pb env list` claim variables `pb env run` could + /// never inject. The exporter drops them too; this is the half that holds + /// even for a bundle patchbay did not write. + /// + /// No keychain item is created either. An environment's values arrive from + /// `pb env pull`, or they do not arrive. + pub fn adopt(&self, entry: &ProjectEntry) -> anyhow::Result { + validate_project_id(&entry.id)?; + validate_env_name(&entry.default_env)?; + for (name, meta) in &entry.environments { + validate_env_name(name)?; + for var in &meta.synced_names { + validate_var_name(var)?; + } + } + + let mut file = self.load()?; + if file.projects.iter().any(|p| p.id == entry.id) { + anyhow::bail!( + "a project is already registered as `{}` on this machine; it was left exactly as \ + it is", + entry.id + ); + } + + let mut entry = entry.clone(); + for meta in entry.environments.values_mut() { + meta.local_names.clear(); + } + file.projects.push(entry.clone()); + self.save(&file)?; + Ok(entry) + } + /// Attach a directory on this machine to a project. /// /// `root` is stored exactly as given — callers pass an absolute path, and @@ -1532,6 +1585,43 @@ mod tests { assert_eq!(v.registry.projects().unwrap().len(), 1); } + /// `adopt` is `pb import`'s way in. It takes the whole entry, and it takes + /// no values with it — including the local layer's *names*, which a bundle + /// should never have carried in the first place. + #[test] + fn test_adopt_takes_the_whole_entry_and_drops_the_local_layer() { + let source = seeded(); + let carried = source.registry.get("pathors").unwrap().unwrap(); + assert!(!carried.env("dev").unwrap().local_names.is_empty()); + + let dest = vault(); + let landed = dest.registry.adopt(&carried).unwrap(); + + let dev = landed.env("dev").unwrap(); + assert_eq!(dev.synced_names, vec!["API_KEY", "DATABASE_URL"]); + assert_eq!(dev.synced_at, carried.env("dev").unwrap().synced_at); + assert!(dev.local_names.is_empty(), "{dev:?}"); + assert_eq!(dest.registry.get("pathors").unwrap().unwrap(), landed); + // Metadata only: no value is invented, and no path is adopted either. + assert!(dest.store.is_empty()); + assert!(dest.registry.attachments().unwrap().is_empty()); + } + + #[test] + fn test_adopt_refuses_an_id_this_machine_already_has() { + let source = seeded(); + let carried = source.registry.get("pathors").unwrap().unwrap(); + + let dest = vault(); + dest.registry.register("pathors", "staging").unwrap(); + let before = dest.registry.get("pathors").unwrap().unwrap(); + + let err = dest.registry.adopt(&carried).unwrap_err().to_string(); + assert!(err.contains("already registered as `pathors`"), "{err}"); + // Untouched, not merged, not half-written: this machine may be newer. + assert_eq!(dest.registry.get("pathors").unwrap().unwrap(), before); + } + #[test] fn test_register_validates_the_id_and_the_default_env() { let v = vault(); diff --git a/crates/patchbay-core/src/migrate/bundle.rs b/crates/patchbay-core/src/migrate/bundle.rs index 61c2397..48f2f83 100644 --- a/crates/patchbay-core/src/migrate/bundle.rs +++ b/crates/patchbay-core/src/migrate/bundle.rs @@ -41,6 +41,7 @@ use serde::{Deserialize, Serialize}; use super::manifest::{Manifest, BUNDLE_VERSION}; use super::policy::Location; +use crate::envs::ProjectEntry; /// Cleartext header, terminated by a newline. const HEADER_PREFIX: &str = "patchbay-bundle/"; @@ -132,6 +133,23 @@ pub struct Payload { pub secrets: Vec, #[serde(default)] pub mcp: Vec, + /// The env vault's portable project manifest: ids, environments and sync + /// pins. [`ProjectEntry`] is portable by construction — no absolute path, + /// no value — and the entries carried here have had every `local_names` + /// list cleared as well, because the local layer's *values* cannot travel + /// and names without values would make `pb env list` lie on arrival. + /// + /// `#[serde(default)]` rather than a [`BUNDLE_VERSION`] bump: a bundle + /// written before this field existed imports with an empty section, and a + /// bundle written *after* it, opened by an older patchbay, loses the + /// section silently — serde ignores unknown fields. That asymmetry is + /// acceptable here and would not be for a credential file: nothing else in + /// the payload depends on this list, and everything it points at is + /// rebuildable by `pb env pull`. Bumping the version instead would make an + /// older build refuse the whole bundle, which trades a recoverable omission + /// for an unrecoverable one. + #[serde(default)] + pub env_projects: Vec, } impl Payload { @@ -184,6 +202,7 @@ impl std::fmt::Debug for Payload { .field("bytes_carried", &self.bytes_carried()) .field("secrets", &self.secrets.len()) .field("mcp", &self.mcp.len()) + .field("env_projects", &self.env_projects.len()) .finish_non_exhaustive() } } @@ -333,6 +352,7 @@ mod tests { tools: vec![], keys: vec![], mcp: vec![], + env_projects: vec![], gaps: vec![], }, setup_md: "# Setting up\n".into(), @@ -348,6 +368,7 @@ mod tests { secret: "cf-token-value-9876".into(), }], mcp: vec![], + env_projects: vec![], } } diff --git a/crates/patchbay-core/src/migrate/export.rs b/crates/patchbay-core/src/migrate/export.rs index a0e9026..e7372fe 100644 --- a/crates/patchbay-core/src/migrate/export.rs +++ b/crates/patchbay-core/src/migrate/export.rs @@ -13,6 +13,11 @@ //! 4. **`SETUP.md`** — written at export time so a machine with no patchbay on //! it yet still has instructions. //! +//! And one thing that travels as *metadata only, by construction*: the env +//! vault's project manifest ([`crate::envs`]). Ids, environments, sync pins — +//! never a value, and never this machine's `attachments.json`, whose paths mean +//! nothing on the next laptop. See [`Exporter::collect_env_projects`]. +//! //! # Where the bundle is allowed to land //! //! Not in a cloud-sync folder, unless the user insists. Copying credential @@ -27,10 +32,12 @@ use chrono::{DateTime, Utc}; use super::bundle::{self, BundleFile, BundleMcpServer, BundleSecret, Payload}; use super::manifest::{ - KeyRecord, Manifest, McpRecord, SetupItem, Source, ToolRecord, BUNDLE_VERSION, + EnvEnvironmentRecord, EnvProjectRecord, EnvSyncRecord, KeyRecord, Manifest, McpRecord, + SetupItem, Source, ToolRecord, BUNDLE_VERSION, }; use super::policy::{policy_for, Portability}; use super::setup; +use crate::envs::{EnvRegistry, ProjectEntry}; use crate::keys::KeyRegistry; use crate::mcp_clients::{McpClientRegistry, TransportSpec}; use crate::paths::Paths; @@ -80,6 +87,9 @@ pub struct ExportReport { /// Names — never values — of the MCP env vars and headers whose values are /// inside the bundle. The caller must say these out loud. pub mcp_values_carried: Vec, + /// Env vault projects whose metadata travelled. Their values did not, in + /// either layer. + pub env_projects: Vec, pub gaps: usize, pub warnings: Vec, } @@ -159,6 +169,7 @@ pub struct Exporter<'a> { pub registry: &'a Registry, pub vault: &'a KeyRegistry, pub clients: &'a McpClientRegistry, + pub envs: &'a EnvRegistry, } impl Exporter<'_> { @@ -246,6 +257,8 @@ impl Exporter<'_> { let (key_records, secrets, key_gaps) = self.collect_keys(keys)?; gaps.extend(key_gaps); let (mcp_records, mcp_servers) = self.collect_mcp(); + let (env_records, env_entries, env_gaps) = self.collect_env_projects(); + gaps.extend(env_gaps); let manifest = Manifest { version: BUNDLE_VERSION, @@ -257,6 +270,7 @@ impl Exporter<'_> { tools, keys: key_records, mcp: mcp_records, + env_projects: env_records, gaps, }; @@ -268,6 +282,7 @@ impl Exporter<'_> { files, secrets, mcp: mcp_servers, + env_projects: env_entries, }) } @@ -345,6 +360,98 @@ impl Exporter<'_> { Ok((records, secrets, gaps)) } + /// The env vault's portable project manifest, and nothing else it owns. + /// + /// Three exclusions, all deliberate and all load-bearing: + /// + /// * **`attachments.json` never travels.** It is a list of directories on + /// *this* machine, and a path from the old laptop is at best noise and at + /// worst a directory that exists on the new one and means something else. + /// Nothing here reads it. `pb env attach` is the new machine's own job. + /// * **No value travels, in either layer.** Not the synced one — a pull + /// rebuilds it from the remote, which is authoritative in a way a + /// week-old bundle is not — and emphatically not the local one, which is + /// `.env.local` semantics: the `DATABASE_URL` pointing at a container on + /// the old machine is the exact thing that must not follow you. + /// * **Every `local_names` list is cleared** from the entries that are + /// carried, because names without values would make `pb env list` on the + /// new machine promise variables `pb env run` could not produce. + /// `synced_names` and `synced_at` stay: those are an honest statement of + /// what a pull will restore and when it last happened. + /// + /// A registry that cannot be read is a note, not a failed export — the same + /// tolerance [`Exporter::collect_keys`] has, for the same reason: the + /// credential files are the half worth saving. + fn collect_env_projects(&self) -> (Vec, Vec, Vec) { + let Ok(projects) = self.envs.projects() else { + return (Vec::new(), Vec::new(), Vec::new()); + }; + + let mut records = Vec::new(); + let mut entries = Vec::new(); + let mut gaps = Vec::new(); + for project in projects { + records.push(EnvProjectRecord { + id: project.id.clone(), + default_env: project.default_env.clone(), + environments: project + .environments + .iter() + .map(|(name, meta)| EnvEnvironmentRecord { + name: name.clone(), + synced_vars: meta.synced_names.len(), + synced_at: meta.synced_at, + }) + .collect(), + sync: project.sync.as_ref().map(|sync| EnvSyncRecord { + provider: sync.provider.clone(), + project_id: sync.project_id.clone(), + account: sync.account.clone(), + }), + }); + + // A linked project is no gap at all: `pb env pull` rebuilds it, and + // the plan says so. An unlinked one with a synced layer is the case + // worth a line — something pulled those variables once, and nothing + // on the new machine knows from where. + if project.sync.is_none() { + let synced: usize = project + .environments + .values() + .map(|meta| meta.synced_names.len()) + .sum(); + if synced > 0 { + gaps.push( + SetupItem::new( + format!("env:{}", project.id), + "env vault", + format!( + "`{}` has {synced} synced variable(s) but no sync config, so its \ + synced layer cannot be rebuilt by `pb env pull` on the new \ + machine — values are not in the bundle", + project.id + ), + ) + .command( + format!( + "pb env link --project {} --project-id ", + project.id + ), + false, + ) + .detail( + "or set the values by hand there (`pb env set`) / load a `.env` with \ + `pb env import`", + ), + ); + } + } + + entries.push(carried_entry(&project)); + } + (records, entries, gaps) + } + /// Every user-scope MCP registration, by name in the manifest and with /// values in the payload. Project scopes are read but never carried: they /// belong to a repository, not to the machine. @@ -392,6 +499,18 @@ impl Exporter<'_> { } } +/// One project entry as it is allowed to leave the machine: the local layer's +/// name lists dropped, everything else verbatim. See +/// [`Exporter::collect_env_projects`] for why, and +/// [`crate::envs::EnvRegistry::adopt`] for the other half of the same rule. +fn carried_entry(project: &ProjectEntry) -> ProjectEntry { + let mut entry = project.clone(); + for meta in entry.environments.values_mut() { + meta.local_names.clear(); + } + entry +} + /// The gap a non-portable tool leaves behind. `None` when there is nothing to /// re-create — a tool that was never set up here is not a chore over there. fn tool_gap( @@ -481,13 +600,14 @@ pub fn write( .collect(), mcp_carried: payload.mcp.len(), mcp_values_carried: mcp_values, + env_projects: payload.env_projects.iter().map(|p| p.id.clone()).collect(), gaps: payload.manifest.gaps.len(), warnings, }) } #[cfg(test)] -mod tests { +pub(crate) mod tests { use super::*; use crate::keystore::MemoryKeystore; use crate::migrate::policy::Location; @@ -568,23 +688,31 @@ mod tests { ]) } - pub(crate) fn exporter_parts(home: &Path) -> (Paths, Registry, KeyRegistry, McpClientRegistry) { + pub(crate) fn exporter_parts( + home: &Path, + ) -> (Paths, Registry, KeyRegistry, McpClientRegistry, EnvRegistry) { let paths = Paths::for_test(home); let registry = Registry::all(paths.clone()); let vault = KeyRegistry::new(home.join("keys.json"), Box::new(MemoryKeystore::new())); let clients = McpClientRegistry::with_paths(paths.clone()); - (paths, registry, vault, clients) + let envs = EnvRegistry::new( + home.join("projects.json"), + home.join("attachments.json"), + Box::new(MemoryKeystore::new()), + ); + (paths, registry, vault, clients, envs) } #[test] fn test_the_payload_carries_the_portable_files_and_nothing_else() { let home = machine(); - let (paths, registry, vault, clients) = exporter_parts(home.path()); + let (paths, registry, vault, clients, envs) = exporter_parts(home.path()); let exporter = Exporter { paths: &paths, registry: ®istry, vault: &vault, clients: &clients, + envs: &envs, }; let payload = exporter.payload(&KeySelection::None, Utc::now()).unwrap(); @@ -628,12 +756,13 @@ mod tests { ".config/gh/hosts.yml", "github.com:\n user: octocat\n users:\n octocat:\n", )]); - let (paths, registry, vault, clients) = exporter_parts(home.path()); + let (paths, registry, vault, clients, envs) = exporter_parts(home.path()); let payload = Exporter { paths: &paths, registry: ®istry, vault: &vault, clients: &clients, + envs: &envs, } .payload(&KeySelection::None, Utc::now()) .unwrap(); @@ -658,7 +787,7 @@ mod tests { #[test] fn test_keys_are_metadata_only_until_asked_for() { let home = fake_home(&[]); - let (paths, registry, vault, clients) = exporter_parts(home.path()); + let (paths, registry, vault, clients, envs) = exporter_parts(home.path()); vault .add( crate::keys::NewKey::new("cf-api", "cli").provider("cloudflare"), @@ -678,6 +807,7 @@ mod tests { registry: ®istry, vault: &vault, clients: &clients, + envs: &envs, }; // Default: metadata travels, values do not, and each becomes a gap. @@ -719,12 +849,13 @@ mod tests { ".cursor/mcp.json", r#"{"mcpServers":{"grafana":{"command":"uvx","args":["mcp-grafana"],"env":{"GRAFANA_TOKEN":"glsa_secret"}}}}"#, )]); - let (paths, registry, vault, clients) = exporter_parts(home.path()); + let (paths, registry, vault, clients, envs) = exporter_parts(home.path()); let payload = Exporter { paths: &paths, registry: ®istry, vault: &vault, clients: &clients, + envs: &envs, } .payload(&KeySelection::None, Utc::now()) .unwrap(); @@ -739,16 +870,202 @@ mod tests { assert!(!manifest.contains("glsa_secret"), "{manifest}"); } + /// Two projects, one of each shape, with values in the keystore and an + /// attachment on this machine — everything the exclusions are about. + /// + /// `pathors` is linked and has both layers. `legacy` has a synced layer and + /// no link, which is the only case that becomes a gap. + pub(crate) fn seed_env_vault(envs: &EnvRegistry, attached: &Path) -> DateTime { + let pulled = DateTime::parse_from_rfc3339("2026-08-01T09:30:00Z") + .unwrap() + .with_timezone(&Utc); + + envs.register("pathors", "dev").unwrap(); + envs.replace_synced( + "pathors", + "dev", + [( + "DATABASE_URL".to_string(), + "postgres://remote/db".to_string(), + )] + .into_iter() + .collect(), + pulled, + ) + .unwrap(); + envs.set_local( + "pathors", + "dev", + "DATABASE_URL", + "postgres://localhost:5432/dev", + ) + .unwrap(); + envs.set_local("pathors", "dev", "LOCAL_ONLY_TOKEN", "local-value-1234") + .unwrap(); + envs.set_sync( + "pathors", + crate::envs::SyncConfig { + provider: "infisical".into(), + project_id: "9f2c-uuid".into(), + account: "me@work.com".into(), + domain: None, + env_map: Default::default(), + }, + ) + .unwrap(); + // A directory on THIS machine. Its path must not appear in a bundle. + envs.attach(attached, "pathors").unwrap(); + + envs.register("legacy", "dev").unwrap(); + envs.replace_synced( + "legacy", + "dev", + [("OLD_KEY".to_string(), "old-value-5678".to_string())] + .into_iter() + .collect(), + pulled, + ) + .unwrap(); + pulled + } + + #[test] + fn test_the_env_vault_travels_as_names_and_pins_only() { + let home = fake_home(&[]); + let (paths, registry, vault, clients, envs) = exporter_parts(home.path()); + let pulled = seed_env_vault(&envs, &home.path().join("repos/pathors")); + + let payload = Exporter { + paths: &paths, + registry: ®istry, + vault: &vault, + clients: &clients, + envs: &envs, + } + .payload(&KeySelection::None, Utc::now()) + .unwrap(); + + let carried = &payload.env_projects; + assert_eq!(carried.len(), 2, "{carried:?}"); + let pathors = carried.iter().find(|p| p.id == "pathors").unwrap(); + let dev = pathors.env("dev").unwrap(); + // What a pull will restore, and when it last ran: honest, and useful. + assert_eq!(dev.synced_names, vec!["DATABASE_URL"]); + assert_eq!(dev.synced_at, Some(pulled)); + // The local layer's NAMES are dropped with its values. A name list + // without values would make `pb env list` promise what it cannot give. + assert!(dev.local_names.is_empty(), "{dev:?}"); + assert_eq!(pathors.sync.as_ref().unwrap().account, "me@work.com"); + + // The manifest half: counts and the pin, no names, no local anything. + let record = payload + .manifest + .env_projects + .iter() + .find(|p| p.id == "pathors") + .unwrap(); + assert_eq!(record.environments[0].name, "dev"); + assert_eq!(record.environments[0].synced_vars, 1); + assert_eq!(record.sync.as_ref().unwrap().project_id, "9f2c-uuid"); + + // A linked project is no gap — `pb env pull` rebuilds it. An unlinked + // one with a synced layer is, because nothing here can. + assert!(!payload.manifest.gaps.iter().any(|g| g.id == "env:pathors")); + let gap = payload + .manifest + .gaps + .iter() + .find(|g| g.id == "env:legacy") + .expect("a synced layer with no sync config cannot be rebuilt"); + assert!(gap.what.contains("cannot be rebuilt"), "{gap:?}"); + assert!( + gap.command.contains("pb env link --project legacy"), + "{gap:?}" + ); + } + + #[test] + fn test_no_env_value_and_no_attachment_path_is_anywhere_in_a_bundle() { + let home = fake_home(&[]); + let out = tempfile::tempdir().unwrap(); + let (paths, registry, vault, clients, envs) = exporter_parts(home.path()); + let attached = home.path().join("repos/pathors"); + seed_env_vault(&envs, &attached); + + let payload = Exporter { + paths: &paths, + registry: ®istry, + vault: &vault, + clients: &clients, + envs: &envs, + } + .payload(&KeySelection::None, Utc::now()) + .unwrap(); + + // The decrypted payload is the real test: the encryption is not what + // keeps these out, the collection rules are. + let json = serde_json::to_string(&payload).unwrap(); + for forbidden in [ + // synced values: rebuilt by a pull, never carried + "postgres://remote/db", + "old-value-5678", + // local values: per-machine overrides, carried by nothing, ever + "postgres://localhost:5432/dev", + "local-value-1234", + "LOCAL_ONLY_TOKEN", + // this machine's attachment root + "repos/pathors", + ] { + assert!(!json.contains(forbidden), "`{forbidden}` is in the payload"); + } + assert!(!json.contains(&attached.display().to_string())); + // Names of the synced layer DO travel: that is the honest part. + assert!(json.contains("DATABASE_URL"), "{json}"); + + let path = out.path().join("b.pbx"); + let report = write(&path, &payload, "pass", false, Some(10)).unwrap(); + assert_eq!(report.env_projects, vec!["pathors", "legacy"]); + let raw = String::from_utf8_lossy(&std::fs::read(&path).unwrap()).into_owned(); + for forbidden in ["local-value-1234", "postgres://", "repos/pathors"] { + assert!(!raw.contains(forbidden), "`{forbidden}` is in the .pbx"); + } + } + + #[test] + fn test_an_unreadable_env_registry_does_not_take_the_export_down() { + let home = fake_home(&[]); + let (paths, registry, vault, clients, _) = exporter_parts(home.path()); + std::fs::write(home.path().join("projects.json"), "{not json").unwrap(); + let envs = EnvRegistry::new( + home.path().join("projects.json"), + home.path().join("attachments.json"), + Box::new(MemoryKeystore::new()), + ); + + let payload = Exporter { + paths: &paths, + registry: ®istry, + vault: &vault, + clients: &clients, + envs: &envs, + } + .payload(&KeySelection::None, Utc::now()) + .expect("the credential files are the half worth saving"); + assert!(payload.env_projects.is_empty()); + assert!(payload.manifest.env_projects.is_empty()); + } + #[test] fn test_write_reports_what_went_in() { let home = machine(); let out = tempfile::tempdir().unwrap(); - let (paths, registry, vault, clients) = exporter_parts(home.path()); + let (paths, registry, vault, clients, envs) = exporter_parts(home.path()); let payload = Exporter { paths: &paths, registry: ®istry, vault: &vault, clients: &clients, + envs: &envs, } .payload(&KeySelection::None, Utc::now()) .unwrap(); diff --git a/crates/patchbay-core/src/migrate/import.rs b/crates/patchbay-core/src/migrate/import.rs index 1573a7c..2f1d6de 100644 --- a/crates/patchbay-core/src/migrate/import.rs +++ b/crates/patchbay-core/src/migrate/import.rs @@ -22,6 +22,13 @@ //! no write, no backup, and [`FileOutcome::Unchanged`] in the report. So the //! second run of an import is a no-op that says so, which is what makes it safe //! to re-run after fixing one item in the plan. +//! +//! # The one thing that is never replaced +//! +//! An env vault project ([`crate::envs`]) that already exists here is skipped +//! rather than overwritten, with a note naming it — see +//! [`Importer::restore_env_projects`]. Files get a backup and can be put back; +//! a project entry that lost its sync pin to a stale copy cannot. use std::path::PathBuf; @@ -29,6 +36,7 @@ use super::bundle::{BundleMcpServer, Payload}; use super::manifest::SetupItem; use super::plan; use super::policy::Location; +use crate::envs::EnvRegistry; use crate::keys::{KeyRegistry, NewKey}; use crate::mcp_clients::{McpClientRegistry, ServerSpec, TransportSpec}; use crate::paths::Paths; @@ -90,6 +98,13 @@ pub struct McpResult { pub values_carried: Vec, } +/// One env vault project the bundle carried. +#[derive(Debug, Clone, PartialEq)] +pub struct EnvProjectResult { + pub id: String, + pub outcome: FileOutcome, +} + /// Everything an import did, or would do. #[derive(Debug, Clone, PartialEq)] pub struct ImportReport { @@ -97,6 +112,7 @@ pub struct ImportReport { pub files: Vec, pub keys: Vec, pub mcp: Vec, + pub env_projects: Vec, /// The gap list, re-evaluated against this machine after the restore. pub remaining: Vec, pub notes: Vec, @@ -126,6 +142,7 @@ pub struct Importer<'a> { pub registry: &'a Registry, pub vault: &'a KeyRegistry, pub clients: &'a McpClientRegistry, + pub envs: &'a EnvRegistry, } impl Importer<'_> { @@ -135,6 +152,7 @@ impl Importer<'_> { files: Vec::new(), keys: Vec::new(), mcp: Vec::new(), + env_projects: Vec::new(), remaining: Vec::new(), notes: Vec::new(), }; @@ -142,6 +160,7 @@ impl Importer<'_> { self.restore_files(payload, options, &mut report)?; self.restore_keys(payload, options, &mut report); self.restore_mcp(payload, options, &mut report); + self.restore_env_projects(payload, options, &mut report); // The gaps are recomputed here rather than copied out of the manifest: // the manifest's list is what the *source* predicted, and by now some @@ -151,6 +170,7 @@ impl Importer<'_> { self.registry, self.vault, self.clients, + self.envs, Some(&payload.manifest), ); Ok(report) @@ -354,6 +374,67 @@ impl Importer<'_> { ); } } + + /// Register the env vault projects the bundle carried, through + /// [`EnvRegistry::adopt`]. + /// + /// **A project id this machine already has is skipped, by name.** The + /// destination may well be the newer of the two machines — it may have + /// pulled since the bundle was written, or been re-linked to a different + /// remote — and an import that overwrote its entry would replace a live + /// sync pin with a stale one, silently. Skipping is recoverable (`pb env + /// forget` then re-import); overwriting is not. + /// + /// Neither `attachments.json` nor the keychain is touched. The values come + /// back from `pb env pull`, and which directories here belong to a project + /// is this machine's own business — see the plan items in [`super::plan`]. + fn restore_env_projects( + &self, + payload: &Payload, + options: &ImportOptions, + report: &mut ImportReport, + ) { + for project in &payload.env_projects { + let existing = self.envs.get(&project.id).ok().flatten(); + let outcome = match (existing, options.dry_run) { + (Some(_), _) => { + report.notes.push(format!( + "env project `{}` is already registered here and was left exactly as it \ + is; the bundle's copy was not applied (`pb env projects` shows what this \ + machine has)", + project.id + )); + FileOutcome::Skipped { + reason: "already registered on this machine".to_string(), + } + } + (None, true) => FileOutcome::Created, + (None, false) => match self.envs.adopt(project) { + Ok(_) => FileOutcome::Created, + // One unusable entry must not abort the rest of the import. + Err(e) => FileOutcome::Skipped { + reason: format!("{e:#}"), + }, + }, + }; + report.env_projects.push(EnvProjectResult { + id: project.id.clone(), + outcome, + }); + } + if report + .env_projects + .iter() + .any(|p| p.outcome == FileOutcome::Created) + { + report.notes.push( + "env projects arrived as metadata only — no variable value is ever in a bundle. \ + Run `pb env pull --project ` for each linked project to rebuild its synced \ + layer, and `pb env attach ` in the directories that belong to them" + .to_string(), + ); + } + } } fn to_spec(server: &BundleMcpServer) -> ServerSpec { @@ -391,6 +472,7 @@ mod tests { registry: Registry, vault: KeyRegistry, clients: McpClientRegistry, + envs: EnvRegistry, } impl Machine { @@ -407,6 +489,11 @@ mod tests { registry: Registry::all(paths.clone()), vault: KeyRegistry::new(home.join("keys.json"), Box::new(MemoryKeystore::new())), clients: McpClientRegistry::with_paths(paths.clone()), + envs: EnvRegistry::new( + home.join("projects.json"), + home.join("attachments.json"), + Box::new(MemoryKeystore::new()), + ), paths, home, _dir: dir, @@ -419,6 +506,7 @@ mod tests { registry: &self.registry, vault: &self.vault, clients: &self.clients, + envs: &self.envs, } .payload(&keys, Utc::now()) .unwrap() @@ -430,6 +518,7 @@ mod tests { registry: &self.registry, vault: &self.vault, clients: &self.clients, + envs: &self.envs, } .run(payload, &ImportOptions { dry_run }) .unwrap() @@ -652,6 +741,126 @@ mod tests { assert_eq!(again.mcp[0].outcome, FileOutcome::Unchanged); } + #[test] + fn test_env_projects_round_trip_without_their_values() { + let source = Machine::new(&[]); + let attached = source.home.join("repos/pathors"); + let pulled = crate::migrate::export::tests::seed_env_vault(&source.envs, &attached); + let payload = source.payload(KeySelection::None); + + let dest = Machine::new(&[]); + let report = dest.import(&payload, false); + assert_eq!(report.env_projects.len(), 2); + assert!(report + .env_projects + .iter() + .all(|p| p.outcome == FileOutcome::Created)); + + let landed = dest.envs.get("pathors").unwrap().unwrap(); + assert_eq!(landed.default_env, "dev"); + let dev = landed.env("dev").unwrap(); + assert_eq!(dev.synced_names, vec!["DATABASE_URL"]); + assert_eq!(dev.synced_at, Some(pulled)); + // The local layer did not travel, in either half: no names… + assert!(dev.local_names.is_empty(), "{dev:?}"); + // …and no values, in either layer. `merged` reads the keychain. + assert!(dest.envs.merged("pathors", "dev").unwrap().vars.is_empty()); + // The pin survived, so a pull here knows where and as whom. + let sync = landed.sync.as_ref().unwrap(); + assert_eq!(sync.project_id, "9f2c-uuid"); + assert_eq!(sync.account, "me@work.com"); + + // Nothing on this machine is attached to it: paths are machine-local. + assert!(dest.envs.attachments().unwrap().is_empty()); + assert!(!dest.exists("attachments.json")); + assert!(report.notes.iter().any(|n| n.contains("pb env pull"))); + } + + #[test] + fn test_a_project_that_already_exists_here_is_never_overwritten() { + let source = Machine::new(&[]); + crate::migrate::export::tests::seed_env_vault( + &source.envs, + &source.home.join("repos/pathors"), + ); + let payload = source.payload(KeySelection::None); + + // The destination has its own `pathors`, linked somewhere else — it may + // well be the newer machine. + let dest = Machine::new(&[]); + dest.envs.register("pathors", "staging").unwrap(); + dest.envs + .set_sync( + "pathors", + crate::envs::SyncConfig { + provider: "infisical".into(), + project_id: "newer-uuid".into(), + account: "me@home.com".into(), + domain: None, + env_map: Default::default(), + }, + ) + .unwrap(); + let before = dest.envs.get("pathors").unwrap().unwrap(); + + let report = dest.import(&payload, false); + let skipped = report + .env_projects + .iter() + .find(|p| p.id == "pathors") + .unwrap(); + assert!( + matches!(&skipped.outcome, FileOutcome::Skipped { reason } if reason.contains("already")), + "{skipped:?}" + ); + assert_eq!(dest.envs.get("pathors").unwrap().unwrap(), before); + assert!( + report.notes.iter().any(|n| n.contains("`pathors`")), + "the skip has to be named: {:?}", + report.notes + ); + // The other one still landed: one skip is not an aborted import. + assert!(dest.envs.get("legacy").unwrap().is_some()); + } + + #[test] + fn test_a_dry_run_registers_no_project() { + let source = Machine::new(&[]); + crate::migrate::export::tests::seed_env_vault( + &source.envs, + &source.home.join("repos/pathors"), + ); + let payload = source.payload(KeySelection::None); + + let dest = Machine::new(&[]); + let report = dest.import(&payload, true); + assert_eq!(report.env_projects.len(), 2); + assert!(dest.envs.projects().unwrap().is_empty()); + } + + /// A bundle written before the env vault existed has no `env_projects` key + /// at all. Serde's default fills it in, and the import is otherwise + /// unchanged — which is why this section did not cost a `BUNDLE_VERSION`. + #[test] + fn test_a_bundle_from_before_the_env_vault_still_imports() { + let source = Machine::new(&source_files()); + let payload = source.payload(KeySelection::None); + + let mut json = serde_json::to_value(&payload).unwrap(); + json.as_object_mut().unwrap().remove("env_projects"); + assert!(json.get("env_projects").is_none()); + let old: Payload = serde_json::from_value(json).unwrap(); + assert!(old.env_projects.is_empty()); + + let dest = Machine::new(&[]); + let report = dest.import(&old, false); + assert!(report.env_projects.is_empty()); + assert!(report.written() >= 6, "{report:?}"); + for (rel, body) in source_files() { + assert_eq!(dest.read(rel), body, "{rel}"); + } + } + #[cfg(unix)] #[test] fn test_the_source_mode_travels_with_the_file() { @@ -694,12 +903,18 @@ mod tests { Box::new(MemoryKeystore::new()), ); let clients = McpClientRegistry::with_paths(paths.clone()); + let envs = EnvRegistry::new( + dest_home.path().join("projects.json"), + dest_home.path().join("attachments.json"), + Box::new(MemoryKeystore::new()), + ); Importer { paths: &paths, registry: ®istry, vault: &vault, clients: &clients, + envs: &envs, } .run(&payload, &ImportOptions::default()) .unwrap(); @@ -730,11 +945,17 @@ mod tests { Box::new(MemoryKeystore::new()), ); let clients = McpClientRegistry::with_paths(paths.clone()); + let envs = EnvRegistry::new( + dir.path().join("projects.json"), + dir.path().join("attachments.json"), + Box::new(MemoryKeystore::new()), + ); let payload = Exporter { paths: &paths, registry: ®istry, vault: &vault, clients: &clients, + envs: &envs, } .payload(&KeySelection::None, Utc::now()) .unwrap(); diff --git a/crates/patchbay-core/src/migrate/manifest.rs b/crates/patchbay-core/src/migrate/manifest.rs index 9fd6a0a..0c39df6 100644 --- a/crates/patchbay-core/src/migrate/manifest.rs +++ b/crates/patchbay-core/src/migrate/manifest.rs @@ -159,6 +159,52 @@ pub struct KeyRecord { pub included: bool, } +/// One project from the env vault, readable without decrypting anything — +/// [`KeyRecord`]'s role for [`crate::envs`]. +/// +/// Names and counts only. A variable name is not a secret, but a *list* of +/// them is noise in a document meant to be read, so an environment reports how +/// many variables its synced layer holds rather than which; the payload's own +/// [`crate::envs::ProjectEntry`] has the names for the code that restores them. +/// +/// **No local-layer count.** The local layer does not travel at all, and a +/// number next to it in a manifest would imply that something of it did. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct EnvProjectRecord { + pub id: String, + pub default_env: String, + #[serde(default)] + pub environments: Vec, + /// Where the synced layer is pulled from, if the project is linked. Absent + /// means nothing on the new machine can rebuild it. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub sync: Option, +} + +/// One environment of one carried project. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct EnvEnvironmentRecord { + pub name: String, + /// How many variables a `pb env pull` is expected to restore here. + pub synced_vars: usize, + /// When the source machine last pulled. `null` if it never did. + #[serde(default)] + pub synced_at: Option>, +} + +/// A project's sync pin, as the manifest reports it: enough to see which +/// remote and which login a pull will need, and nothing that could authorize +/// one. The API base URL and the environment-name mapping stay in the payload's +/// own entry — they are configuration, not something a reader needs. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct EnvSyncRecord { + pub provider: String, + /// The remote's own project identifier. + pub project_id: String, + /// The account the pull has to run as. + pub account: String, +} + /// One MCP server as one client had it registered. Same value-free contract as /// [`crate::mcp_clients::McpServerEntry`]: names of environment variables and /// headers, never their values. @@ -196,6 +242,10 @@ pub struct Manifest { pub keys: Vec, #[serde(default)] pub mcp: Vec, + /// The env vault's projects, by id. Their *values* are not in the bundle at + /// all; `pb env pull` rebuilds each synced layer on the new machine. + #[serde(default)] + pub env_projects: Vec, /// What will not have travelled, with the command for each. #[serde(default)] pub gaps: Vec, @@ -272,6 +322,20 @@ mod tests { header_keys: vec!["Authorization".into()], carried: true, }], + env_projects: vec![EnvProjectRecord { + id: "pathors".into(), + default_env: "dev".into(), + environments: vec![EnvEnvironmentRecord { + name: "dev".into(), + synced_vars: 12, + synced_at: None, + }], + sync: Some(EnvSyncRecord { + provider: "infisical".into(), + project_id: "3f0b-uuid".into(), + account: "me@work.com".into(), + }), + }], gaps: vec![ SetupItem::new("tool:gh", "gh", "re-authenticate").command("gh auth login", true) ], @@ -304,6 +368,19 @@ mod tests { assert!(json.contains("\"last4\": \"9876\""), "{json}"); } + /// The env vault's half of the same contract: what a pull will restore, in + /// counts, and not one word about the layer that does not travel. + #[test] + fn test_an_env_project_record_counts_the_synced_layer_and_never_the_local_one() { + let json = manifest_with_secret_shaped_everything().to_json(); + assert!(json.contains("\"synced_vars\": 12"), "{json}"); + assert!(json.contains("me@work.com"), "{json}"); + // No `local_names`, no local count, no attachment root: a number beside + // the local layer would imply something of it had moved. + assert!(!json.contains("local"), "{json}"); + assert!(!json.contains("Users"), "{json}"); + } + #[test] fn test_manifest_round_trips() { let manifest = manifest_with_secret_shaped_everything(); diff --git a/crates/patchbay-core/src/migrate/mod.rs b/crates/patchbay-core/src/migrate/mod.rs index c4d2006..59db098 100644 --- a/crates/patchbay-core/src/migrate/mod.rs +++ b/crates/patchbay-core/src/migrate/mod.rs @@ -39,7 +39,10 @@ pub use bundle::{peek_version, Payload, BUNDLE_EXTENSION}; pub use export::{ check_destination, cloud_service, default_file_name, ExportReport, Exporter, KeySelection, }; -pub use import::{FileOutcome, ImportOptions, ImportReport, Importer}; -pub use manifest::{Manifest, SetupItem, SetupStatus, ToolRecord, BUNDLE_VERSION}; +pub use import::{EnvProjectResult, FileOutcome, ImportOptions, ImportReport, Importer}; +pub use manifest::{ + EnvEnvironmentRecord, EnvProjectRecord, EnvSyncRecord, Manifest, SetupItem, SetupStatus, + ToolRecord, BUNDLE_VERSION, +}; pub use plan::{plan, recheck}; pub use policy::{policy_for, Location, Portability, PortabilityKind, ToolPolicy, POLICIES}; diff --git a/crates/patchbay-core/src/migrate/plan.rs b/crates/patchbay-core/src/migrate/plan.rs index e0c0513..924f4ac 100644 --- a/crates/patchbay-core/src/migrate/plan.rs +++ b/crates/patchbay-core/src/migrate/plan.rs @@ -21,6 +21,7 @@ use super::manifest::{Manifest, SetupItem, SetupStatus, ToolRecord}; use super::policy::{policy_for, Portability}; +use crate::envs::EnvRegistry; use crate::keys::KeyRegistry; use crate::mcp_clients::McpClientRegistry; use crate::paths::Paths; @@ -36,6 +37,7 @@ pub fn plan( registry: &Registry, vault: &KeyRegistry, clients: &McpClientRegistry, + envs: &EnvRegistry, manifest: Option<&Manifest>, ) -> Vec { let _ = paths; @@ -48,6 +50,7 @@ pub fn plan( } items.extend(key_items(vault, manifest)); items.extend(mcp_items(clients, manifest)); + items.extend(env_items(envs, manifest)); items } @@ -58,10 +61,11 @@ pub fn recheck( registry: &Registry, vault: &KeyRegistry, clients: &McpClientRegistry, + envs: &EnvRegistry, manifest: Option<&Manifest>, id: &str, ) -> Option { - plan(paths, registry, vault, clients, manifest) + plan(paths, registry, vault, clients, envs, manifest) .into_iter() .find(|item| item.id == id) } @@ -330,6 +334,89 @@ fn mcp_items(clients: &McpClientRegistry, manifest: Option<&Manifest>) -> Vec` first. An +/// agent firing these blind would collect a row of confusing failures, so the +/// account goes in the detail and the human decides. +/// +/// Without a manifest there is nothing to say: a project registered here with +/// no pull yet is a normal state, not an outstanding move. +fn env_items(envs: &EnvRegistry, manifest: Option<&Manifest>) -> Vec { + let Some(manifest) = manifest else { + return Vec::new(); + }; + let here = envs.projects().unwrap_or_default(); + + let mut items = Vec::new(); + for record in &manifest.env_projects { + let Some(sync) = &record.sync else { + // Not linked: `collect_env_projects` already made a gap for the + // ones that had a synced layer, and there is no pull to suggest. + continue; + }; + let local = here.iter().find(|p| p.id == record.id); + // Done only when this machine has pulled SINCE the bundle was written. + // `synced_at` travels with the entry, so its mere presence proves + // nothing — a restored project looks pulled the moment it lands. + let pulled_here = local.is_some_and(|project| { + record.environments.iter().any(|env| { + let before = env.synced_at; + let now = project.env(&env.name).and_then(|meta| meta.synced_at); + match (now, before) { + (Some(now), Some(before)) => now > before, + (Some(_), None) => true, + _ => false, + } + }) + }); + + let mut item = SetupItem::new( + format!("env:{}", record.id), + "env vault", + if pulled_here { + format!("`{}`'s synced layer has been pulled here", record.id) + } else { + format!( + "`{}`'s variables are not on this machine; no value travels in a bundle, so \ + the synced layer is rebuilt by pulling it", + record.id + ) + }, + ) + .command(format!("pb env pull --project {}", record.id), false) + .auto(false) + .status(if pulled_here { + SetupStatus::Done + } else { + SetupStatus::Open + }) + .detail(format!( + "it is pinned to the {} account `{}`; the CLI's active login is machine-global, so \ + `pb use infisical {}` may have to come first", + sync.provider, sync.account, sync.account + )); + + if local.is_none() { + item = item.detail(format!( + "no project `{}` is registered here yet — `pb import` registers it, or `pb env \ + init --id {}` does", + record.id, record.id + )); + } + items.push(item); + } + items +} + #[cfg(test)] mod tests { use super::*; @@ -347,6 +434,7 @@ mod tests { registry: Registry, vault: KeyRegistry, clients: McpClientRegistry, + envs: EnvRegistry, } impl Machine { @@ -363,6 +451,11 @@ mod tests { registry: Registry::all(paths.clone()), vault: KeyRegistry::new(home.join("keys.json"), Box::new(MemoryKeystore::new())), clients: McpClientRegistry::with_paths(paths.clone()), + envs: EnvRegistry::new( + home.join("projects.json"), + home.join("attachments.json"), + Box::new(MemoryKeystore::new()), + ), paths, home, _dir: dir, @@ -375,6 +468,7 @@ mod tests { registry: &self.registry, vault: &self.vault, clients: &self.clients, + envs: &self.envs, } .payload(&KeySelection::None, Utc::now()) .unwrap() @@ -387,6 +481,7 @@ mod tests { &self.registry, &self.vault, &self.clients, + &self.envs, manifest, ) } @@ -540,6 +635,82 @@ mod tests { assert!(!json.contains("glsa_x"), "{json}"); } + /// The env vault's item is the one patchbay deliberately will NOT run + /// itself: a pull is only valid under the account the project is pinned to, + /// and that login is machine-global. + #[test] + fn test_a_carried_env_project_asks_for_a_pull_and_names_the_account() { + let source = Machine::new(&[]); + crate::migrate::export::tests::seed_env_vault( + &source.envs, + &source.home.join("repos/pathors"), + ); + let manifest = source.manifest(); + + let dest = Machine::new(&[]); + let items = dest.plan(Some(&manifest)); + let pull = item(&items, "env:pathors"); + assert_eq!(pull.command, "pb env pull --project pathors"); + assert!(!pull.auto, "a pull can fail on the wrong login: {pull:?}"); + assert!(!pull.needs_browser); + assert_eq!(pull.status, SetupStatus::Open); + assert!( + pull.detail.iter().any(|d| d.contains("me@work.com")), + "the pinned account has to be on the item: {pull:?}" + ); + assert!( + pull.detail.iter().any(|d| d.contains("pb use infisical")), + "{pull:?}" + ); + // `legacy` has no sync config, so there is no pull to suggest — the + // export already made that a gap of its own. + assert!( + !items.iter().any(|i| i.id == "env:legacy"), + "{:?}", + ids(&items) + ); + // Names, never values, here as everywhere. + let json = serde_json::to_string(&items).unwrap(); + assert!(!json.contains("postgres://"), "{json}"); + } + + #[test] + fn test_an_env_project_closes_once_it_has_actually_been_pulled_here() { + let source = Machine::new(&[]); + crate::migrate::export::tests::seed_env_vault( + &source.envs, + &source.home.join("repos/pathors"), + ); + let manifest = source.manifest(); + + // Import-shaped destination: the entry is here, `synced_at` and all, + // which must NOT read as done — it is the source's timestamp. + let dest = Machine::new(&[]); + for project in &source.envs.projects().unwrap() { + dest.envs.adopt(project).unwrap(); + } + assert_eq!( + item(&dest.plan(Some(&manifest)), "env:pathors").status, + SetupStatus::Open + ); + + // A pull on THIS machine moves the timestamp, and only then. + dest.envs + .replace_synced( + "pathors", + "dev", + [("DATABASE_URL".to_string(), "postgres://here/db".to_string())] + .into_iter() + .collect(), + Utc::now(), + ) + .unwrap(); + assert_eq!( + item(&dest.plan(Some(&manifest)), "env:pathors").status, + SetupStatus::Done + ); + } + /// docker, rclone, ssh and npm use every credential they have at once. /// Reporting them as "logged out" forever would make the plan unfinishable. #[test] @@ -661,6 +832,7 @@ mod tests { &dest.registry, &dest.vault, &dest.clients, + &dest.envs, Some(&manifest), "tool:gh", ) @@ -676,6 +848,7 @@ mod tests { &dest.registry, &dest.vault, &dest.clients, + &dest.envs, Some(&manifest), "tool:gh", ) @@ -688,6 +861,7 @@ mod tests { &dest.registry, &dest.vault, &dest.clients, + &dest.envs, Some(&manifest), "tool:invented", ) diff --git a/crates/patchbay-core/src/migrate/setup.rs b/crates/patchbay-core/src/migrate/setup.rs index f0925e7..f0396a0 100644 --- a/crates/patchbay-core/src/migrate/setup.rs +++ b/crates/patchbay-core/src/migrate/setup.rs @@ -138,6 +138,52 @@ pub fn render(manifest: &Manifest) -> String { out.push('\n'); } + // --- env vault --------------------------------------------------------- + if !manifest.env_projects.is_empty() { + out.push_str( + "## 7. Project env vault\n\n\ + The projects themselves travelled — ids, environments and sync pins. **No variable \ + value did**, in either layer: the synced layer comes back from the remote, and the \ + local layer is per-machine overrides that deliberately stay behind. Which \ + directories on this machine belong to a project is also local: attach them here.\n\n\ + | project | environments | synced vars | pull with |\n|---|---|---|---|\n", + ); + for project in &manifest.env_projects { + let synced: usize = project.environments.iter().map(|e| e.synced_vars).sum(); + let envs: Vec<&str> = project + .environments + .iter() + .map(|e| e.name.as_str()) + .collect(); + let pull = match &project.sync { + Some(sync) => format!( + "`pb env pull --project {}` (as {})", + project.id, sync.account + ), + None => "not linked — set the values by hand".to_string(), + }; + out.push_str(&format!( + "| `{}` | {} | {synced} | {pull} |\n", + project.id, + if envs.is_empty() { + "—".to_string() + } else { + envs.join(", ") + }, + )); + } + out.push_str( + "\n```sh\n\ + pb env projects # what arrived\n\ + pb env pull --project # rebuild a synced layer\n\ + pb env attach # bind a directory here (a committed\n\ + \x20 # .patchbay.toml does it on its own)\n\ + ```\n\n\ + A pull runs the infisical CLI, whose active login is machine-global: if the project \ + is pinned to another account, run `pb use infisical ` first.\n\n", + ); + } + out.push_str( "---\n\n\ patchbay never copies an SSH private key, and never moves a credential the OS keychain \ @@ -159,7 +205,8 @@ pub fn portability_label(kind: PortabilityKind) -> &'static str { mod tests { use super::*; use crate::migrate::manifest::{ - KeyRecord, McpRecord, SetupItem, Source, ToolRecord, BUNDLE_VERSION, + EnvEnvironmentRecord, EnvProjectRecord, EnvSyncRecord, KeyRecord, McpRecord, SetupItem, + Source, ToolRecord, BUNDLE_VERSION, }; use crate::migrate::policy::Location; use crate::types::{Profile, ToolCategory}; @@ -219,6 +266,20 @@ mod tests { header_keys: vec![], carried: true, }], + env_projects: vec![EnvProjectRecord { + id: "pathors".into(), + default_env: "dev".into(), + environments: vec![EnvEnvironmentRecord { + name: "dev".into(), + synced_vars: 12, + synced_at: None, + }], + sync: Some(EnvSyncRecord { + provider: "infisical".into(), + project_id: "3f0b-uuid".into(), + account: "me@work.com".into(), + }), + }], gaps: vec![SetupItem::new("tool:gh", "gh", "log in to gh as `octocat`") .command("gh auth login", true) .detail("the OAuth token lives in the OS keychain")], @@ -257,11 +318,23 @@ mod tests { } } + #[test] + fn test_setup_md_names_the_pull_each_carried_env_project_needs() { + let md = render(&manifest()); + assert!(md.contains("pb env pull --project pathors"), "{md}"); + assert!(md.contains("me@work.com"), "{md}"); + assert!(md.contains("pb use infisical"), "{md}"); + // The exclusions, stated where the person doing the move will read them. + assert!(md.contains("**No variable value did**"), "{md}"); + assert!(md.contains("pb env attach"), "{md}"); + } + #[test] fn test_an_empty_machine_still_renders() { let mut manifest = manifest(); manifest.tools.clear(); manifest.keys.clear(); + manifest.env_projects.clear(); manifest.gaps.clear(); let md = render(&manifest); assert!(md.contains("Nothing outstanding"), "{md}"); diff --git a/crates/patchbay-mcp/src/migrate.rs b/crates/patchbay-mcp/src/migrate.rs index 5f105a6..9e7daf9 100644 --- a/crates/patchbay-mcp/src/migrate.rs +++ b/crates/patchbay-mcp/src/migrate.rs @@ -40,7 +40,8 @@ pub struct PlanParams { #[derive(Debug, Deserialize, JsonSchema)] pub struct MarkDoneParams { /// The `id` of the item, exactly as `plan_setup` returned it — for example - /// "tool:gh", "install:kubectl", "switch:gcloud", "key:cf-api". + /// "tool:gh", "install:kubectl", "switch:gcloud", "key:cf-api", + /// "env:pathors". pub item_id: String, /// Path to the same `manifest.json` you passed to `plan_setup`, if any. /// Leaving it out when the plan used one will report the item as unknown. @@ -129,12 +130,14 @@ about the key vault.")] let registry = self.registry.clone(); let keys = self.keys.clone(); let clients = self.clients.clone(); + let envs = self.envs.clone(); let items = offload(move || { migrate::plan( registry.paths(), ®istry, &keys, &clients, + &envs, manifest.as_ref(), ) }) @@ -174,6 +177,7 @@ That is the tool working, not a bug: wait for them, then re-check.")] let registry = self.registry.clone(); let keys = self.keys.clone(); let clients = self.clients.clone(); + let envs = self.envs.clone(); let id = item_id.clone(); let found = offload(move || { migrate::recheck( @@ -181,6 +185,7 @@ That is the tool working, not a bug: wait for them, then re-check.")] ®istry, &keys, &clients, + &envs, manifest.as_ref(), &id, ) diff --git a/docs/env-vault.md b/docs/env-vault.md index a4b124c..40c96be 100644 --- a/docs/env-vault.md +++ b/docs/env-vault.md @@ -91,22 +91,37 @@ and attach by hand instead. This is the whole point of the split. A new laptop is three steps: ```sh -cp projects.json ~/.config/patchbay/ # from the old machine +pb import patchbay-*.pbx # projects.json rides inside the bundle git clone git@github.com:you/pathors # the marker comes with the checkout cd pathors && pb env pull # rebuild the synced layer from Infisical ``` +[`pb export`](migration.md#the-project-env-vault) carries the project manifest — +ids, environments and sync pins — so the migration bundle is the normal route. +Copying the file by hand still works and is the fallback when you are not moving +a whole machine: + +```sh +cp projects.json ~/.config/patchbay/ # from the old machine +``` + +Either way, an id that already exists here is left alone: an import skips it +with a note rather than overwriting what may be the newer entry. + A checkout carrying a marker resolves on its own; anything else takes one `pb env attach `. Attachments deliberately do not travel — they are paths from a machine that is not this one — so `attachments.json` is excluded from every migration, copy and export story patchbay has. A project that arrived in a -copied `projects.json` and has no attachment here shows `—` under ROOTS in -`pb env projects`, which is normal, not broken. +bundle or a copied `projects.json` and has no attachment here shows `—` under +ROOTS in `pb env projects`, which is normal, not broken. The **local layer deliberately does not travel either**. `.env.local` semantics are per-machine overrides, and a `DATABASE_URL` pointing at a container on the old laptop is exactly the thing that must not follow you. What the remote holds comes back with `pb env pull`; what you set by hand you set again, on purpose. +Not even its variable *names* are carried, in a bundle or in a copied +`projects.json`: a name with no value behind it would make `pb env list` on the +new machine promise something `pb env run` could not deliver. ### Environments and names diff --git a/docs/migration.md b/docs/migration.md index 781fb40..7680e0e 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -31,9 +31,47 @@ One encrypted file, four parts: key metadata, MCP registrations by name, and the gap list. 4. **`SETUP.md`** — generated at export time, including how to install patchbay on a machine that does not have it yet. +5. **The env vault's project manifest** — names and sync pins, never values. + See [below](#the-project-env-vault). Parts 3 and 4 live *inside* the encrypted payload and are written out on import. +## The project env vault + +[`pb env`](env-vault.md) rides along, as metadata. The bundle carries the +**portable project manifest** — ids, environments, `synced_at`, and each +project's sync pin — so the new machine knows your projects exist and where +their variables come from. + +Three things deliberately do not travel: + +| not carried | why | +|---|---| +| every variable **value**, synced or local | the synced layer is rebuilt from the remote, which is more current than a bundle from last Tuesday; the local layer is `.env.local` semantics, and a `DATABASE_URL` pointing at a container on the old laptop is exactly what must not follow you | +| the local layer's variable **names** | names without values would make `pb env list` on the new machine promise variables `pb env run` could not produce | +| `attachments.json` | which directories belong to a project is a list of paths on *this* machine, and meaningless on the next one | + +So on the new machine: + +```sh +pb env projects # the projects arrived, with no values +pb env pull --project # rebuild each linked project's synced layer +pb env attach # bind a directory — a committed .patchbay.toml + # does this on its own for a fresh clone +``` + +`pb plan` lists one `pb env pull --project ` per linked project, marked +`auto: false` on purpose: a pull only works under the account the project is +pinned to, and the infisical CLI's active login is machine-global, so the item +names that account and lets you (or `pb use infisical `) sort it out +first. A project the old machine had *unlinked* but with a synced layer becomes +a gap instead — nothing here can rebuild it. + +Importing never overwrites a project this machine already has, even if the +bundle's copy is different. It is skipped with a note naming it: the machine in +front of you may be the newer one, and a stale sync pin written over a live one +is not recoverable the way a backed-up file is. + ## The portability table Every tool on the board declares its own policy in