refactor: restructure package to mirror vct-cribl-cli's API-endpoint framework

.envrc

@@ -1 +1,3 @@
 use flake
+# Load local credentials (gitignored; see .env.example). No-op if .env is absent.
+dotenv_if_exists

AGENTS.md

@@ -22,29 +22,45 @@ supported variable, and [README.md](./README.md) for connecting to a server.
 
 ## Architecture
 
-The package separates a Click-free core from a thin CLI shell (the "functional
-core, imperative shell" pattern):
-
-- `src/vct_splunk/core/` — plain functions and typed errors. **Never imports
-  Click.** This is the reusable, unit-testable library: `client` (transport,
-  auth, retries, pagination, dry-run), `auth` (session login), `profiles`
-  (INI profile loading), `errors`, `audit`, `namespace`
-  (owner/app resolution), `resource` (the generic CRUD engine: `Spec`/`Field`/
-  `CrudResource`), `backends` + `acs/` (Splunk Cloud ACS support), and one
-  module per hand-written operation (`server`, `api`, `jobs`, `search`,
-  `saved_searches` for dispatch, `health`).
+The package follows the same API-endpoint framework as `vct-cribl-cli`, with a
+Click-free library under `api/` + `auth/` + `config/` + `utils/` and a thin CLI
+shell in `commands/` (the "functional core, imperative shell" pattern). See
+[docs/architecture.md](./docs/architecture.md) for the full map.
+
+- `src/vct_splunk/api/` — the reusable REST layer. **Never imports Click.**
+  `client.py` holds the layered httpx transport stack (`AuthTransport` injects
+  the Authorization header per request, `RetryTransport` retries 429/503) and
+  the envelope-aware `SplunkClient`; `endpoint_factory.py` is the generic CRUD
+  engine (`EndpointConfig`/`Field`/`Endpoints`); `endpoints/` holds one module
+  per hand-written operation (`server`, `search`, `jobs`, `saved_searches`,
+  `kvstore`, `hec`, `apps`, `cluster`, `license`, `deploy`, `lookups`,
+  `datamodel`, `health`, `raw`); `acs/` is the Splunk Cloud ACS
+  management-plane client.
+- `src/vct_splunk/auth/` — `session.py`: credential resolution
+  (token / session key / username+password login) with cribl-style caching,
+  called per request by the auth transport.
+- `src/vct_splunk/config/` — `types.py` (`SplunkConfig`) and `loader.py`
+  (INI profiles, env vars, flag merging: flag > env > profile > default).
+- `src/vct_splunk/utils/` — typed `errors`, `redact`, `namespace` (owner/app
+  resolution), `path`, `validation`, `audit`, `backends` (Cloud deduction).
+- `src/vct_splunk/output/` — `formatter.py`: JSON/table rendering and the
+  error envelope.
 - `src/vct_splunk/commands/` — Click adapters, one module per hand-written
   command group (`server`, `api`, `auth`, `search`, `health`, `inspect`, plus
   `saved_search`'s `run`), plus shared plumbing: `context` (the `command`
-  decorator and `Ctx`), `output` (rendering, error envelope), `write` (the
-  single gated write path), `dispatch` (routes a few reads to Cloud ACS), and
-  `registry` + `factory` (resource specs declared as data, turned into
-  generated CRUD groups — `index`, `saved-search`, `user`, `role`, `macro`,
-  the data inputs/outputs, and friends).
+  decorator and `Ctx`), `write` (the single gated write path), `dispatch`
+  (routes a few reads to Cloud ACS), and `registry` + `command_factory`
+  (resource configs declared as data, turned into generated CRUD groups —
+  `index`, `saved-search`, `user`, `role`, `macro`, the data inputs/outputs,
+  and friends).
 - `src/vct_splunk/cli.py` assembles the root group and the `splunk` entry point;
   `__main__.py` enables `python -m vct_splunk`.
 
-Dependencies flow one way: `commands` import `core`, never the reverse.
+Dependencies flow one way: `commands` import the library packages, never the
+reverse. A downstream application embeds the library surface directly:
+`config.loader.load_config()` → `api.client.create_client()` → the
+`api.endpoints.*` / `api.endpoint_factory.Endpoints` functions (plus the
+cribl-style `api.client.set_client()`/`get_client()` process-wide hook).
 
 Two cross-cutting ideas to know about:
 

docs/architecture.md

@@ -0,0 +1,180 @@
+# Architecture
+
+Developer guide to the vct-splunk-cli codebase. The layout mirrors
+`vct-cribl-cli` so downstream applications (e.g. the vizzy web interface) can
+embed both API layers the same way.
+
+## Project structure
+
+```text
+vct_splunk/
+  __init__.py              # Package marker + __version__
+  __main__.py              # Entry point for `python -m vct_splunk`
+  cli.py                   # Click command tree assembly, `splunk` entry point
+
+  api/
+    client.py              # httpx transport stack (AuthTransport, RetryTransport),
+                           # SplunkClient, create_client, get/set_client singletons
+    endpoint_factory.py    # Generic CRUD Endpoints engine, EndpointConfig/Field
+    endpoints/             # Hand-written endpoint modules (14 files)
+      server.py, search.py, jobs.py, saved_searches.py, kvstore.py,
+      hec.py, apps.py, cluster.py, license.py, deploy.py, lookups.py,
+      datamodel.py, health.py, raw.py
+    acs/                   # Splunk Cloud ACS management-plane client (read-only)
+      client.py, operations.py
+
+  auth/
+    session.py             # Credential resolution + session login, cached per process
+
+  commands/
+    command_factory.py     # Generates Click CRUD subcommands from the registry
+    registry.py            # Declarative list of factory-generated resources
+    context.py             # Shared `command` decorator + Ctx (builds clients)
+    write.py               # The single gated write path (confirm + audit)
+    dispatch.py            # Routes index/role/hec-token list to ACS on Cloud
+    server.py, api.py, auth.py, search.py, saved_search.py, health.py,
+    kvstore.py, hec.py, apps.py, cluster.py, shcluster.py, license.py,
+    deploy.py, lookup.py, datamodel.py, inspect.py
+
+  config/
+    loader.py              # INI profiles + env var + CLI flag merging
+    types.py               # SplunkConfig, AuthStatus dataclasses
+
+  output/
+    formatter.py           # JSON / table formatting, error envelope, prompts
+
+  utils/
+    errors.py              # Typed SplunkError hierarchy with exit codes
+    redact.py              # Secret redaction by field name; safe_target for URLs
+    namespace.py           # /servicesNS/<owner>/<app>/ path building + policy
+    path.py                # Path-segment validation/encoding (traversal-safe)
+    validation.py          # KEY=VALUE parsing
+    audit.py               # Append-only local audit log for writes
+    backends.py            # Enterprise-vs-Cloud deduction from SPLUNK_URL
+
+tests/
+  cli_catalog.py           # Single catalog of every command leaf
+  unit/                    # pytest + httpx.MockTransport (no mock package)
+  integration/             # Live suites, gated behind env vars
+```
+
+## Key patterns
+
+### HTTP client transport stack
+
+```text
+Request
+  -> AuthTransport       (injects Authorization header, per request)
+    -> RetryTransport    (retries 429/503, honors Retry-After)
+      -> httpx.HTTPTransport (sends request; owns TLS verify settings)
+```
+
+`AuthTransport` resolves the credential on every request via
+`auth.session.get_auth_header()`:
+
+- `SPLUNK_TOKEN` (a JWT) → `Authorization: Bearer <token>`
+- `SPLUNK_SESSION_KEY` → `Authorization: Splunk <key>`
+- `SPLUNK_USERNAME`/`SPLUNK_PASSWORD` → a session key minted via
+  `/services/auth/login`, cached for 55 minutes and re-minted transparently —
+  so a long-running embedding process survives session expiry.
+
+`SplunkClient` wraps the stack with Splunk's envelope concerns: the
+`output_mode=json` parameter, `entry[].content` handling hand-off, pagination
+(`get_collection`), typed error mapping (401/403 → `AuthError`, 404 →
+`NotFoundError`, else `APIError`), and the dry-run gate — `write()` /
+`write_json()` return a structured preview and send nothing when
+`config.dry_run` is set.
+
+### Embedding (vizzy-style)
+
+```python
+from vct_splunk.config.loader import load_config
+from vct_splunk.api.client import create_client, set_client, get_client
+from vct_splunk.api.endpoints import server, search
+
+cfg = load_config()  # env + profile merging, no network I/O
+client = create_client(cfg)  # auth resolved lazily, per request
+set_client(client)  # optional process-wide hook
+
+info = server.get_server_info(get_client())
+hits = search.run_search(get_client(), "index=_internal | head 5")
+```
+
+The CLI itself builds a client per command (via `commands/context.py`) instead
+of the singleton, so `--help` and config commands never touch the network.
+
+### Two-tier endpoint design
+
+**Factory endpoints** (`api/endpoint_factory.py`) — a generic `Endpoints` class
+with `list/get/create/update/delete` (+ `enable`/`disable`) driven by a
+declarative `EndpointConfig`. Two URL scopes:
+
+| Scope        | URL pattern                          |
+|--------------|--------------------------------------|
+| `global`     | `/services/{path}` (path absolute)   |
+| `namespaced` | `/servicesNS/{owner}/{app}/{path}`   |
+
+`Field` entries map friendly CLI options to Splunk form keys (with typing,
+scaling, and secret handling) and drive the generated Click options.
+
+**Hand-written endpoints** (`api/endpoints/*.py`) — for resources that need
+special logic: search (oneshot jobs, NDJSON export parsing), the KV Store
+document store (JSON bodies, no envelope), HEC token rotation (mints a secret),
+health checks (multi-dimension verdicts), data model acceleration
+(read-modify-write of a JSON sub-document), app install, deployment server,
+cluster/license reads, and the raw read-only escape hatch (`raw.py`).
+
+### Command registration
+
+`cli.py` adds the hand-written groups, then loops over
+`commands/registry.py:REGISTRY` — a flat list of `EndpointConfig` entries —
+and generates a Click group per resource via
+`commands/command_factory.py:build_group()`.
+
+To add a standard CRUD resource: add one `EndpointConfig` to `registry.py`.
+To add a hand-written command: create `api/endpoints/<thing>.py` +
+`commands/<thing>.py`, then register the group in `cli.py`.
+
+### Config priority chain
+
+```text
+CLI flags (--base-url, --profile, --app, ...)
+  > Environment variables (SPLUNK_URL, SPLUNK_TOKEN, ...)
+    > Active profile in $XDG_CONFIG_HOME/vct-splunk/config
+      > Built-in defaults
+```
+
+### Write safety
+
+Every mutation funnels through `commands/write.py:do_write()`: `--dry-run`
+previews the exact request and sends nothing; otherwise it confirms on a TTY or
+requires `--yes` when non-interactive, then appends a record to the local audit
+log. Reads redact secret-named fields by default; only commands whose purpose
+is to mint a credential reveal one. Splunk Cloud targets refuse writes and
+route supported reads through ACS.
+
+### Error handling
+
+The library raises typed errors (`utils/errors.py`); the `command` decorator in
+`commands/context.py` renders them once as a JSON envelope on stderr with the
+documented exit codes: 0 ok, 1 API/transport, 2 usage/config, 3 auth,
+4 not found, 5 health-check findings.
+
+## Development
+
+```bash
+python3 -m venv .venv                       # use a Python linked against OpenSSL
+.venv/bin/python -m pip install -e ".[dev]"
+
+.venv/bin/pytest                            # unit tests (httpx.MockTransport)
+.venv/bin/ruff check . && .venv/bin/ruff format .
+.venv/bin/pyright
+
+.venv/bin/splunk server info                # run the CLI
+```
+
+> Note: macOS's system Python 3.9 links LibreSSL 2.8, whose TLS handshake
+> stalls against Splunk 10's TLS configuration. Use a Homebrew/python.org
+> interpreter (OpenSSL 1.1.1+).
+
+Live suites are documented in [tests/TESTING.md](../tests/TESTING.md).

src/vct_splunk/api/__init__.py

@@ -0,0 +1,14 @@
+"""The Splunk REST API layer: client, endpoint factory, and endpoint modules.
+
+Mirrors the cribl-cli ``api/`` package. Nothing here imports Click — this is
+the reusable library surface a downstream application (e.g. a web backend)
+embeds directly:
+
+* :mod:`vct_splunk.api.client` — the httpx transport stack (auth, retries) and
+  the envelope-aware :class:`~vct_splunk.api.client.SplunkClient`.
+* :mod:`vct_splunk.api.endpoint_factory` — the generic CRUD engine driven by
+  declarative :class:`~vct_splunk.api.endpoint_factory.EndpointConfig` entries.
+* :mod:`vct_splunk.api.endpoints` — hand-written endpoint modules for
+  resources that do not fit the CRUD shape.
+* :mod:`vct_splunk.api.acs` — the Splunk Cloud ACS management-plane client.
+"""

src/vct_splunk/api/acs/__init__.py

@@ -0,0 +1 @@
+"""Read-only Splunk Cloud ACS (adminconfig/v2) client and operations."""

src/vct_splunk/core/acs/client.py → src/vct_splunk/api/acs/client.py

@@ -3,7 +3,7 @@
 ACS is a different surface from splunkd: a different base URL
 (``https://admin.splunk.com/<stack>/adminconfig/v2``), a stack auth token, and
 plain JSON responses (not the form-encoded ``entry[].content`` shape). So it gets
-its own small client rather than reusing :class:`~vct_splunk.core.client.SplunkClient`.
+its own small client rather than reusing :class:`~vct_splunk.api.client.SplunkClient`.
 Writes are intentionally absent this release.
 """
 
@@ -17,8 +17,8 @@
 
 import httpx
 
-from ..errors import APIError, AuthError, NotFoundError, TransportError, UsageError
-from ..redact import safe_target
+from ...utils.errors import APIError, AuthError, NotFoundError, TransportError, UsageError
+from ...utils.redact import safe_target
 
 ACS_BASE_URL = "https://admin.splunk.com"
 _STACK_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9-]*$")

src/vct_splunk/core/acs/operations.py → src/vct_splunk/api/acs/operations.py

@@ -4,8 +4,8 @@
 
 from typing import Any
 
-from ..errors import APIError
-from ..redact import redact_secrets
+from ...utils.errors import APIError
+from ...utils.redact import redact_secrets
 from .client import AcsClient
 
 # ACS read paths and their official success envelopes. This is the runtime