feat: add whole-group export/import for worker groups and edge fleets

CLAUDE.md

@@ -12,10 +12,13 @@
 
 Python CLI built with Click. Two kinds of commands:
 
-1. **Hand-written (25):** `cribl_cli/commands/*.py` — complex operations (workers, routes, search, edge, health report, ingest dashboard, billing, finops, etc.)
+1. **Hand-written (26):** `cribl_cli/commands/*.py` — complex operations
+   (workers, groups export/import, routes, search, edge, health report,
+   ingest dashboard, billing, finops, etc.)
 2. **Factory-generated (52):** Declared in `commands/registry.py`, built by `commands/command_factory.py` — standard CRUD
 
 Key modules:
+
 - `cli.py` — Click group (`CriblCLI`), registers all commands, skips auth for `config` subcommand
 - `api/client.py` — httpx client with `AuthTransport` and `DryRunTransport`
 - `api/endpoint_factory.py` — generic CRUD for four scope types (group/global/search/lake)
@@ -34,30 +37,70 @@ Key modules:
 
 ## CLI Commands
 
-Worker groups are managed via `workers`, not `groups`.
+Worker groups are listed and managed via `workers` (there is no `groups list`).
+The `groups` command is separate: it exports and imports an entire worker group's
+or edge fleet's config. Both are the same API object — a fleet is a group with
+`isFleet: true`.
 
-**Hand-written commands:** alerts, billing, config, destinations, edge, finops, health (check, report, cpu), ingest (dashboard, query), jobs, kms, license-usage, logger, metrics, notebooks, overview, packs, pipelines, preview, profiler, routes, search, sources, system, version, workers
+**Hand-written commands:** alerts, billing, config, destinations, edge, finops,
+groups, health (check, report, cpu), ingest (dashboard, query), jobs, kms,
+license-usage, logger, metrics, notebooks, overview, packs, pipelines, preview,
+profiler, routes, search, sources, system, version, workers
 
 **Key subcommands:**
+
 - `workers list` — list worker groups; `workers nodes` — list individual worker nodes (supports `-g` group filter)
 - `edge nodes` — list individual edge nodes (supports `-f` fleet filter)
-
-**Factory-generated commands (standard CRUD):** ai-settings, alert-monitors, appscope, auth-settings, banners, certificates, collectors, conditions, credentials, dashboard-categories, dashboards, dataset-providers, datasets, datatypes, db-connections, encryption-keys, event-breakers, executors, feature-flags, functions, git-settings, global-vars, grok, hmac-functions, lake-datasets, licenses, lookups, macros, messages, notification-targets, notifications, outposts, parquet-schemas, parsers, policies, protobuf-libs, regex, roles, samples, saved-searches, schemas, scripts, sds-rules, sds-rulesets, secrets, storage-locations, subscriptions, teams, trust-policies, usage-groups, users, workspaces
+- `groups export <group>` — pull all config for one worker group or edge fleet
+  (JSON to stdout, or `--out-dir` for one file per resource type); `groups import
+  <group>` — push it back (see Safety Rules)
+
+**Factory-generated commands (standard CRUD):** ai-settings, alert-monitors,
+appscope, auth-settings, banners, certificates, collectors, conditions,
+credentials, dashboard-categories, dashboards, dataset-providers, datasets,
+datatypes, db-connections, encryption-keys, event-breakers, executors,
+feature-flags, functions, git-settings, global-vars, grok, hmac-functions,
+lake-datasets, licenses, lookups, macros, messages, notification-targets,
+notifications, outposts, parquet-schemas, parsers, policies, protobuf-libs,
+regex, roles, samples, saved-searches, schemas, scripts, sds-rules,
+sds-rulesets, secrets, storage-locations, subscriptions, teams, trust-policies,
+usage-groups, users, workspaces
 
 ## Safety Rules
 
-- **Never replace the route table wholesale.** `routes create` fetches existing routes, inserts before the catch-all, then updates.
+- **Never replace the route table wholesale from `routes`.** `routes create`
+  fetches existing routes, inserts before the catch-all, then updates. The one
+  deliberate exception is `groups import --with-routes` — a high-risk, explicit
+  opt-in that replaces the whole table via `replace_route_table()`.
 - **Always confirm before deploying.** `version deploy` pushes config to live workers.
+- **`groups import` is staged, never deployed.** It upserts config but never
+  auto-commits or deploys; routes are skipped unless `--with-routes`, and
+  `--deploy` requires `--yes`. Review with `version status`/`diff` first.
+- **`groups export` excludes secrets by default.** Secrets, credentials, and
+  certificates are omitted unless `--include-sensitive`; what was skipped or
+  failed is always reported in the `_meta` block.
 - Cloud OAuth audience is always `https://api.cribl.cloud`, not the org-specific URL.
 
 ## Conventions
 
 - New CRUD resources go in `commands/registry.py` — only write a hand-written command file if the resource needs custom logic.
-- Merge-on-update pattern: fetch existing object, strip server-only fields (`status`, `notifications`), deep-merge user updates (nested dicts are merged recursively, not replaced).
+- Merge-on-update pattern: fetch existing object, strip server-only fields
+  (`status`, `notifications`), deep-merge user updates (nested dicts are merged
+  recursively, not replaced).
 - `--dry-run` logs request details to stderr and raises `DryRunAbort` (caught by error handler, exits 0).
 - Config tests mock the filesystem to avoid touching real `~/.criblrc`.
-- `health report` aggregates nodes, capacity alerts, versions, unhealthy IO, and error logs into a single command. Supports `--json`, `--skip-errors`, and `-g` group filter.
+- `health report` aggregates nodes, capacity alerts, versions, unhealthy IO, and
+  error logs into a single command. Supports `--json`, `--skip-errors`, and `-g`
+  group filter.
 - `ingest dashboard` shows daily ingest totals (events/bytes in/out) by source (Stream, Edge, Search). Supports `--json`, `--table`, and `--hours`.
 - `ingest query` runs a raw metric query — accepts a JSON payload as argument.
 - `dashboards/` contains Cribl Search dashboard definitions for Daily Ingest (overview, by source, by route). Deploy with `cribl dashboards create "$(cat dashboards/<file>.json)"`.
-- When running CLI commands to read data, use default JSON output (no `--table`). JSON is structured and easier to parse. Only use `--table` if the user explicitly asks for it.
+- `groups export`/`groups import` move a whole group's config; the resource list
+  is derived from `commands/registry.py` group-scoped entries plus the
+  hand-written sources/destinations/pipelines/packs/routes, so it never drifts.
+  Export drops Cribl-shipped built-ins (`lib == "cribl"` or `destroyable` false)
+  and pack-owned items (`id` prefixed `pack:`) — they list but 4xx/5xx on import,
+  so only user-authored config is kept; the count is reported in `_meta.skipped.builtin`.
+- When running CLI commands to read data, use default JSON output (no `--table`).
+  JSON is structured and easier to parse. Only use `--table` if the user
+  explicitly asks for it.

cribl_cli/api/endpoints/routes.py

@@ -87,6 +87,22 @@ def list_routes(client: httpx.Client, group: str) -> Any:
     return result
 
 
+def replace_route_table(
+    client: httpx.Client, group: str, items: list[dict[str, Any]]
+) -> Any:
+    """Replace the whole route table's items in one PATCH.
+
+    Routes are a single document, not individually addressable, so importing one
+    group's routes into another means swapping the entire items array at once
+    rather than upserting routes one by one. Fetches the live table first so the
+    edge/stream wrapper format is preserved on the way back. Used by
+    ``groups import --with-routes``.
+    """
+    table = _fetch_route_table(client, group)
+    table["items"] = list(items)
+    return _patch_route_table(client, group, table)
+
+
 def get_route(client: httpx.Client, group: str, route_id: str) -> Any:
     """Get a single route by ID from the route table."""
     table = _fetch_route_table(client, group)

cribl_cli/cli.py

@@ -66,6 +66,7 @@ def _register_commands() -> None:
     # Hand-written commands
     from cribl_cli.commands.config_cmd import config_group
     from cribl_cli.commands.workers import workers_group
+    from cribl_cli.commands.groups import groups_group
     from cribl_cli.commands.sources import sources_group
     from cribl_cli.commands.destinations import destinations_group
     from cribl_cli.commands.metrics import metrics_group
@@ -91,7 +92,7 @@ def _register_commands() -> None:
     from cribl_cli.commands.finops import finops_group
 
     for group in [
-        config_group, workers_group, sources_group, destinations_group,
+        config_group, workers_group, groups_group, sources_group, destinations_group,
         metrics_group, search_group, notebooks_group, pipelines_group,
         routes_group, jobs_group, version_group, system_group, edge_group,
         kms_group, preview_group, logger_group, profiler_group, health_group,

cribl_cli/commands/groups.py

@@ -0,0 +1,121 @@
+"""Whole-group config export/import for worker groups and edge fleets."""
+from __future__ import annotations
+
+import json
+import sys
+
+import click
+
+from cribl_cli.api.client import get_client
+from cribl_cli.api.endpoints.version import commit_version
+from cribl_cli.api.endpoints.workers import deploy_group
+from cribl_cli.output.formatter import format_output
+from cribl_cli.utils.errors import handle_error
+from cribl_cli.utils.group_resolver import resolve_group
+from cribl_cli.utils.group_transfer import (
+    apply,
+    collect,
+    format_caveat,
+    read_input,
+    write_dir,
+)
+
+
+@click.group("groups", help="Export and import all config for a worker group or edge fleet.")
+def groups_group():
+    pass
+
+
+@groups_group.command("export", help="Pull all config for one worker group or edge fleet.")
+@click.argument("group")
+@click.option("--out-dir", default=None, help="Write one file per resource type under <out-dir>/<group>/.")
+@click.option("--include-sensitive", is_flag=True, help="Include secrets, credentials, and certificates.")
+@click.option("--include-packs", is_flag=True, help="Include pack configurations (definitions only).")
+@click.option("--include-lookups", is_flag=True, help="Include lookup configurations (definitions only).")
+@click.option("--table", "use_table", is_flag=True, help="Table output (stdout mode only).")
+def groups_export(group, out_dir, include_sensitive, include_packs, include_lookups, use_table):
+    """Export every group-scoped resource for GROUP (a worker group or edge fleet).
+
+    Outputs one aggregated JSON object to stdout, or use --out-dir to write a
+    file per resource type. Sensitive resources are excluded unless
+    --include-sensitive is passed; a caveat of everything skipped or failed is
+    always printed to stderr.
+    """
+    try:
+        client = get_client()
+        g = resolve_group(client, group)
+        result = collect(
+            client, g, include_sensitive=include_sensitive,
+            include_packs=include_packs, include_lookups=include_lookups
+        )
+
+        if out_dir:
+            base = write_dir(result, out_dir)
+            click.echo(format_output({"written": str(base), "resource_types": len(result["resources"])}))
+        else:
+            click.echo(format_output(result, table=use_table))
+
+        click.echo(format_caveat(result), err=True)
+    except Exception as e:
+        handle_error(e)
+
+
+@groups_group.command("import", help="Push exported config into a worker group or edge fleet.")
+@click.argument("group")
+@click.option("--in", "in_path", default=None, help="Read payload from a JSON file or an --out-dir directory.")
+@click.option("--with-routes", is_flag=True, help="Also replace the route table (wholesale; off by default).")
+@click.option("--with-packs", is_flag=True, help="Also import packs (off by default).")
+@click.option("--with-lookups", is_flag=True, help="Also import lookups (off by default).")
+@click.option("--commit", "commit_message", default=None, help="Commit staged changes with this message.")
+@click.option("--deploy", is_flag=True, help="Commit and deploy to live workers (requires --yes).")
+@click.option("--yes", is_flag=True, help="Confirm deployment.")
+def groups_import(group, in_path, with_routes, with_packs, with_lookups, commit_message, deploy, yes):
+    """Import config into GROUP (a worker group or edge fleet).
+
+    Reads an export payload from --in (a file or an --out-dir directory) or from
+    stdin, then upserts each resource. The route table is left untouched unless
+    --with-routes is passed. Nothing is deployed: changes stay staged until you
+    review them and deploy explicitly, or pass --deploy --yes.
+    """
+    try:
+        if in_path:
+            payload = read_input(in_path)
+        else:
+            raw = sys.stdin.read()
+            if not raw.strip():
+                raise ValueError("No input. Pass --in FILE|DIR or pipe an export JSON on stdin.")
+            payload = json.loads(raw)
+
+        client = get_client()
+        target = resolve_group(client, group)
+
+        if deploy and not yes:
+            click.echo(
+                "WARNING: --deploy commits and pushes config to live workers. Pass --yes to confirm.",
+                err=True,
+            )
+            sys.exit(1)
+
+        report = apply(
+            client, target, payload,
+            with_routes=with_routes, with_packs=with_packs, with_lookups=with_lookups
+        )
+        click.echo(format_output(report))
+
+        if commit_message or deploy:
+            message = commit_message or "Import group config"
+            commit_version(client, target, message)
+            click.echo(f"Committed: {message}", err=True)
+
+        if deploy:
+            deploy_group(client, target)
+            click.echo(f"Deployed to {target}.", err=True)
+        else:
+            click.echo(
+                f"Config staged (not deployed). Review with "
+                f"`cribl version status -g {target}` / `cribl version diff -g {target}`, "
+                f"then `cribl version deploy -g {target} --yes`.",
+                err=True,
+            )
+    except Exception as e:
+        handle_error(e)

cribl_cli/commands/registry.py

@@ -20,30 +20,30 @@ class CommandRegistration:
 
 REGISTRY: list[CommandRegistration] = [
     # Group-scoped (full CRUD)
-    CommandRegistration("parsers", EndpointConfig("group", "system/parsers")),
-    CommandRegistration("schemas", EndpointConfig("group", "schemas")),
+    CommandRegistration("parsers", EndpointConfig("group", "lib/parsers")),
+    CommandRegistration("schemas", EndpointConfig("group", "lib/schemas")),
     CommandRegistration("regex", EndpointConfig("group", "lib/regex")),
     CommandRegistration("grok", EndpointConfig("group", "lib/grok")),
     CommandRegistration("event-breakers", EndpointConfig("group", "lib/breakers")),
     CommandRegistration("global-vars", EndpointConfig("group", "lib/vars")),
-    CommandRegistration("db-connections", EndpointConfig("group", "lib/db-connections")),
+    CommandRegistration("db-connections", EndpointConfig("group", "lib/database-connections")),
     CommandRegistration("secrets", EndpointConfig("group", "system/secrets")),
     CommandRegistration("credentials", EndpointConfig("group", "system/credentials")),
     CommandRegistration("collectors", EndpointConfig("group", "collectors")),
-    CommandRegistration("conditions", EndpointConfig("group", "lib/conditions")),
+    CommandRegistration("conditions", EndpointConfig("group", "conditions")),
     CommandRegistration("parquet-schemas", EndpointConfig("group", "lib/parquet-schemas")),
     CommandRegistration("protobuf-libs", EndpointConfig("group", "lib/protobuf-libs")),
-    CommandRegistration("sds-rules", EndpointConfig("group", "lib/sds/rules")),
-    CommandRegistration("sds-rulesets", EndpointConfig("group", "lib/sds/rulesets")),
-    CommandRegistration("appscope", EndpointConfig("group", "lib/appscope")),
+    CommandRegistration("sds-rules", EndpointConfig("group", "lib/sds-rules")),
+    CommandRegistration("sds-rulesets", EndpointConfig("group", "lib/sds-rulesets")),
+    CommandRegistration("appscope", EndpointConfig("group", "lib/appscope-configs")),
     # Group-scoped (limited)
     CommandRegistration("certificates", EndpointConfig("group", "system/certificates"), ["list", "get", "create", "delete"]),
     CommandRegistration("samples", EndpointConfig("group", "system/samples"), ["list", "get", "create", "delete"]),
     CommandRegistration("scripts", EndpointConfig("group", "system/scripts"), ["list", "get", "create", "delete"]),
     CommandRegistration("lookups", EndpointConfig("group", "system/lookups")),
     # packs: hand-written command in commands/packs.py (export, install, upgrade)
     CommandRegistration("executors", EndpointConfig("group", "executors"), ["list", "get"]),
-    CommandRegistration("hmac-functions", EndpointConfig("group", "lib/hmac"), ["list", "get"]),
+    CommandRegistration("hmac-functions", EndpointConfig("group", "lib/hmac-functions"), ["list", "get"]),
     CommandRegistration("functions", EndpointConfig("group", "system/functions"), ["list", "get"]),
     # Global-scoped (full CRUD)
     CommandRegistration("users", EndpointConfig("global", "system/users")),