feat: add whole-group export/import for worker groups and edge fleets by JacobPEvans-personal · Pull Request #1 · VisiCore/vct-cribl-cli

JacobPEvans-personal · 2026-06-07T22:17:33Z

Summary

Adds a groups command that moves an entire worker group's or edge fleet's
config between groups. A fleet is the same API object as a worker group
(isFleet: true), so both flow through one path. The resource list is derived
from commands/registry.py plus the hand-written sources/destinations/
pipelines/packs/routes, so it never drifts from the rest of the CLI.

Stacked on #2. This PR is based on fix/registry-group-paths (#2) because
export/import is what surfaced those 404 paths and its tests assume the
corrected registry. Merge #2 first; GitHub will then retarget this to main.

Changes

commands/groups.py — groups export / groups import CLI:
- export <group> → all group-scoped config to stdout, or --out-dir for one
  file per resource type (dir/files locked to 0700/0600).
- import <group> → staged upsert; never auto-deploys. Routes skipped unless
  --with-routes; --deploy requires --yes.
- Secrets/credentials/certificates excluded unless --include-sensitive;
  what was skipped or failed is reported in _meta.
utils/group_transfer.py — the export/import engine. Import honors each
resource type's registry operations: read-only types (executors,
hmac-functions, functions) are skipped, and create-only types without
update (certificates, samples, scripts) are created, not PATCHed.
api/endpoints/routes.py — new public replace_route_table() (wholesale
swap preserving edge/stream format), so group import no longer imports
private route helpers.
cli.py registers the command; CLAUDE.md documents it.

Test Plan

pytest -m "not integration" — 142 unit tests pass.
cribl groups --help / groups export --help / groups import --help wire up.
Integration tests (pytest -m integration) need CRIBL_INTEGRATION_TEST=true
- live credentials — not run here.

🤖 Generated with Claude Code

Copilot

Pull request overview

Adds a new groups CLI command for exporting/importing an entire worker group or edge fleet configuration, backed by a new group_transfer utility module and unit tests.

Changes:

Introduces cribl_cli.utils.group_transfer to collect per-group resources into an export payload and apply an import payload with safety flags (routes/packs/lookups/sensitive).
Adds a new cribl groups export|import Click command and registers it in the main CLI.
Adds unit tests and updates CLAUDE.md documentation for the new feature.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`cribl_cli/utils/group_transfer.py`	Implements whole-group collect/export, import/apply, and directory-based payload IO.
`cribl_cli/commands/groups.py`	Adds `groups export` and `groups import` CLI commands with deploy confirmation safeguards.
`cribl_cli/cli.py`	Registers the new `groups` command group in the root CLI.
`tests/unit/test_group_transfer.py`	Adds unit tests for resource planning, export aggregation, IO roundtrip, and import safety.
`CLAUDE.md`	Documents the new command and its safety rules/conventions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

write_dir() chmods the export directory to 0700 and each file to 0600, so --include-sensitive exports are not left world-readable (mirrors the ~/.criblrc 0600 protection). read_input() raises on an ambiguous directory (no _meta.json plus multiple group subdirectories) instead of silently returning an empty payload that would make import a no-op. Addresses Copilot review feedback on PR #1. Assisted-by: Claude:claude-opus-4-8[1m]

Eight group-scoped resources used endpoint paths that return HTTP 404 on Cribl 4.17.1 (verified live against a real Leader). The originals were inconsistent — some under system/, one with no library prefix, two with a nested lib/sds/... segment, and several with abbreviated names — so the commands built from them could never reach the API. parsers system/parsers -> lib/parsers schemas schemas -> lib/schemas db-connections lib/db-connections -> lib/database-connections conditions lib/conditions -> conditions sds-rules lib/sds/rules -> lib/sds-rules sds-rulesets lib/sds/rulesets -> lib/sds-rulesets appscope lib/appscope -> lib/appscope-configs hmac-functions lib/hmac -> lib/hmac-functions registry.py drives every factory-generated CRUD command, so this also repairs the existing `cribl parsers list`, `schemas`, `sds-rules`, `appscope`, etc. commands — not just the new group export/import that surfaced the problem.

Add a `groups` command that moves an entire worker group's or edge fleet's config between groups (a fleet is the same API object with `isFleet: true`, so both flow through one path): - `groups export <group>` pulls every group-scoped resource to stdout, or `--out-dir` writes one file per resource type (dir/files locked to 0700/0600). - `groups import <group>` upserts the config back. It is staged, never deployed: routes are skipped unless `--with-routes`, and `--deploy` requires `--yes`. - Secrets, credentials, and certificates are excluded unless `--include-sensitive`; everything skipped or failed is reported in `_meta`. - Import honors each resource type's registry operations: read-only types (executors, hmac-functions, functions) are skipped and create-only types without `update` (certificates, samples, scripts) are created, not PATCHed. The resource list is derived from `commands/registry.py` plus the hand-written sources/destinations/pipelines/packs/routes, so it never drifts from the rest of the CLI. Route import uses a new public `replace_route_table()` helper in `api/endpoints/routes.py` (wholesale swap, preserving edge/stream format). Depends on the registry path corrections in #2 (the export/import surfaced the 404s those fix).

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

- write_dir() clears stale *.json from a prior export before writing, so a secrets.json left by an earlier --include-sensitive run can't linger and be re-imported by read_input(). - Narrow the "never replace the route table wholesale" rule in CLAUDE.md to note that `groups import --with-routes` is the deliberate, gated exception. - Add unit tests for replace_route_table() (stream + edge wrapper) and for write_dir() stale-file cleanup.

Whole-group export captured Cribl-shipped library content (lib == "cribl"), built-in system objects (destroyable false), and pack-owned items (id prefixed "pack:") because they appear in list responses. None are writable as standalone group config, so importing them produced hundreds of 4xx/5xx errors. Filter them at export so the payload is only user-authored config. Verified live: a real group's import failures dropped from ~300 to ~14 (the remainder being pack archives, which need the .crbl, and read-only system conditions). The dropped count is reported in _meta.skipped.builtin. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

werd-drew · 2026-06-25T05:28:18Z

Pushed an additional commit (6d75f82) on top of this branch: export now filters out non-importable resources.

While live-testing the import path (round-trip into a throwaway group), the export was capturing Cribl-shipped content that can't be written back — built-in library items (lib == "cribl": 220 SDS rules, regex, parsers, breakers…), built-in system objects (destroyable == false), and pack-owned items (id prefixed pack:). Importing them produced ~300 4xx/5xx failures.

The commit drops these at export so the payload is only user-authored config. Verified live: import failures fell from ~300 → 14. The remaining 14 are inherent and reported gracefully:

packs ×3 — need the .crbl archive (already the documented binary caveat)
conditions ×10 — Cribl's built-in notification conditions (403/read-only, no marker field to filter on)
collectors/filesystem ×1 — pack-dependent

Dropped count is surfaced in _meta.skipped.builtin. Added a unit test; full suite 143 passing. The merge/overwrite + safety behavior of the original PR was validated live and is unchanged (routes only replaced with --with-routes, nothing deleted, nothing deployed).

werd-drew

Approved. Live-validated end to end against our Leader: export (read-only) pulls all user-authored group config; import merges by id (creates then updates), never deletes target-only resources, and only replaces the route table with --with-routes. Added commit 6d75f82 filters built-in/pack-owned content so real-world import is clean (failures ~300 → 14). 143 unit tests pass. 🚢

JacobPEvans-personal requested a review from Copilot June 7, 2026 22:32

Copilot started reviewing on behalf of JacobPEvans-personal June 7, 2026 22:32 View session

JacobPEvans-personal requested a review from werd-drew June 7, 2026 22:33

JacobPEvans-personal marked this pull request as draft June 7, 2026 22:35

Copilot AI reviewed Jun 7, 2026

View reviewed changes

Comment thread cribl_cli/utils/group_transfer.py

Comment thread cribl_cli/utils/group_transfer.py

Comment thread cribl_cli/utils/group_transfer.py

JacobPEvans-personal marked this pull request as ready for review June 21, 2026 17:36

JacobPEvans-personal mentioned this pull request Jun 21, 2026

fix: correct 8 group-scoped resource API paths #2

Merged

JacobPEvans-personal added 2 commits June 21, 2026 14:20

JacobPEvans-personal force-pushed the feat/groups-export-import branch from 31cfefc to 9d899e3 Compare June 21, 2026 18:22

JacobPEvans-personal changed the base branch from main to fix/registry-group-paths June 21, 2026 18:22

JacobPEvans-personal marked this pull request as draft June 21, 2026 18:32

JacobPEvans-personal requested a review from Copilot June 21, 2026 18:58

Copilot started reviewing on behalf of JacobPEvans-personal June 21, 2026 18:59 View session

Copilot AI reviewed Jun 21, 2026

View reviewed changes

Comment thread cribl_cli/utils/group_transfer.py

Comment thread CLAUDE.md Outdated

Comment thread cribl_cli/api/endpoints/routes.py

werd-drew deleted the branch main June 25, 2026 04:50

werd-drew closed this Jun 25, 2026

werd-drew reopened this Jun 25, 2026

werd-drew changed the base branch from fix/registry-group-paths to main June 25, 2026 04:59

werd-drew approved these changes Jun 25, 2026

View reviewed changes

werd-drew marked this pull request as ready for review June 25, 2026 05:28

werd-drew merged commit 514491a into main Jun 25, 2026

werd-drew deleted the feat/groups-export-import branch June 25, 2026 05:28

Navigation Menu

feat: add whole-group export/import for worker groups and edge fleets - #1

feat: add whole-group export/import for worker groups and edge fleets#1
werd-drew merged 4 commits into
mainfrom
feat/groups-export-import

JacobPEvans-personal commented Jun 7, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

werd-drew commented Jun 25, 2026

Uh oh!

werd-drew left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

JacobPEvans-personal commented Jun 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Test Plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

werd-drew commented Jun 25, 2026

Uh oh!

werd-drew left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JacobPEvans-personal commented Jun 7, 2026 •

edited

Loading