From a0e41f43cf2be3b551f83d9ada4707ece3850f0b Mon Sep 17 00:00:00 2001 From: JacobPEvans <20714140+JacobPEvans-personal@users.noreply.github.com> Date: Sun, 16 Aug 2026 21:10:01 -0400 Subject: [PATCH] docs: add human-operated Splunk Cloud validation runbook --- tests/TESTING.md | 173 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 161 insertions(+), 12 deletions(-) diff --git a/tests/TESTING.md b/tests/TESTING.md index 9622005..451fa64 100644 --- a/tests/TESTING.md +++ b/tests/TESTING.md @@ -109,27 +109,176 @@ export SPLUNK_TEST_SERVER_FIXTURE_DIR=/opt/splunk/var/run/splunk/lookup_tmp Clean up when you are finished: `docker rm -f splunk-test`. -## Group 4: Cloud reads +## Group 4: human-operated Splunk Cloud validation -Read-only. It needs a real Splunk Cloud stack and an ACS token. +This is the approval runbook for a real Splunk Cloud stack. It is read-only and +requires no AI or interpretation service: a person runs the commands, checks +the stated results, and completes the sign-off table at the end. + +Coverage means every endpoint and command this CLI supports, not every API +endpoint Splunk Cloud exposes. The guaranteed ACS-backed surface is `inspect`, +`index list`, `role list`, and `hec-token list`. The catalog-driven read suite +exercises every other supported read and records the documented unsupported or +credential boundary. Search-head REST is a separate, optional check because it +needs a second credential and a different network path. + +### 1. Prepare a clean local environment + +Start at the repository root on the revision you intend to approve. Python 3.9 +or newer is the only prerequisite. + +```bash +git status --short --branch +python3 --version +python3 -m venv .venv +.venv/bin/python -m pip install -e ".[dev]" +``` + +`git status` must name the expected revision or branch and show no unexpected +changes. The install must finish successfully. + +### 2. Prove the Cloud contract without credentials + +Run the local end-to-end Cloud routes, every Cloud write refusal, and the +current public ACS OpenAPI contract: + +```bash +.venv/bin/python -m pytest \ + tests/unit/test_acs_loopback.py \ + tests/unit/test_cloud_write_refusal.py \ + -q --tb=short + +SPLUNK_ACS_SPEC_TEST=true .venv/bin/python -m pytest \ + tests/integration/test_acs_public_spec.py \ + -q --tb=short +``` + +Both commands must end in `passed`, with no failures or errors. These checks +send no request to your stack; the second command downloads Splunk's public API +description from `admin.splunk.com`. + +### 3. Connect to ACS + +You need the stack URL and a short-lived ACS JWT whose role can list indexes, +roles, and HEC tokens. If ACS access is restricted by an IP allow list, the +computer running this runbook must already be allowed. + +Clear profile and Splunk REST credentials first so they cannot silently affect +the ACS-only result. The Python prompt hides the token and keeps it out of +shell history: ```bash +unset SPLUNK_PROFILE VCT_SPLUNK_CONFIG +unset SPLUNK_TOKEN SPLUNK_SESSION_KEY SPLUNK_USERNAME SPLUNK_PASSWORD +unset SPLUNK_ACS_STACK SPLUNK_ACS_BASE_URL + export SPLUNK_ACS_LIVE_TEST=true export SPLUNK_URL="https://your-stack.splunkcloud.com" -export SPLUNK_ACS_TOKEN="" -# export SPLUNK_ACS_BASE_URL="https://admin.splunkcloudgc.com" # only for FedRAMP +export SPLUNK_ACS_TOKEN="$( + .venv/bin/python -c 'import getpass; print(getpass.getpass("ACS token: "))' +)" +``` + +For FedRAMP IL2, set the alternate ACS origin after the block above: -.venv/bin/python -m pytest tests/integration/cloud/read -v +```bash +export SPLUNK_ACS_BASE_URL="https://admin.splunkcloudgc.com" ``` -This runs every read command your Cloud stack could receive. The reads Cloud -serves must succeed. Every other read must fail with a proper error message and -a documented exit code, which is how the tool proves it never guesses at an -endpoint your stack does not offer. +Do not add `:8089` yet. The ACS checks use the normal Cloud stack URL. + +### 4. Exercise every supported ACS command and Cloud read + +Run each public ACS-backed command directly so the human approver can see its +output: + +```bash +.venv/bin/splunk inspect --output json +.venv/bin/splunk index list --output json +.venv/bin/splunk role list --output json +.venv/bin/splunk hec-token list --output json +``` + +Every command must exit `0` and print a top-level `data` and `meta` object. +`inspect` must report the `cloud` backend and the intended stack. A secret field +in HEC output must contain ``; an actual token value must never +appear. + +Now exercise the complete catalog of supported Cloud reads through the public +CLI. The directory is intentional: it runs both catalog-driven CLI reads and +the direct ACS operations in the live suite. + +```bash +.venv/bin/python -m pytest \ + tests/integration/cloud/read \ + -q --tb=short +``` + +The command must end in `passed`, with no failure, error, traceback, or +undocumented exit code. With only the ACS credential configured, non-ACS reads +stop at their documented credential or unsupported-backend boundary. + +### 5. Optionally validate search-head REST + +Run this section only when the stack exposes its search-head API. It requires: + +- port `8089` open for the deployment; +- this computer on the `search-api` IP allow list; and +- a short-lived Splunk authentication token, separate from the ACS token. + +Free-trial Cloud stacks do not expose this API. Mark this section `N/A` when a +prerequisite is intentionally unavailable; do not weaken TLS verification. + +```bash +export SPLUNK_URL="https://your-stack.splunkcloud.com:8089" +export SPLUNK_TOKEN="$( + .venv/bin/python -c 'import getpass; print(getpass.getpass("Splunk REST token: "))' +)" + +.venv/bin/splunk search run \ + --query '| makeresults | stats count' \ + --earliest -5m \ + --latest now \ + --max-rows 5 \ + --timeout 60 \ + --export \ + --output json + +.venv/bin/python -m pytest \ + tests/integration/cloud/read \ + -q --tb=short +``` + +The bounded search must exit `0` and return one result. The complete Cloud +read suite must again end in `passed`; this run carries both credentials and +therefore reaches the full dispatch path available to the stack. + +### 6. Clean up credentials and approve the run + +```bash +unset SPLUNK_ACS_LIVE_TEST SPLUNK_ACS_TOKEN SPLUNK_ACS_STACK SPLUNK_ACS_BASE_URL +unset SPLUNK_URL SPLUNK_TOKEN SPLUNK_SESSION_KEY SPLUNK_USERNAME SPLUNK_PASSWORD +``` -Supplying `SPLUNK_TOKEN` as well exercises the reads Cloud does not serve -through the full dispatch path. Without it they stop earlier, at the -credential check. +Revoke both short-lived tokens in Splunk Cloud. These read-only commands create +no durable Cloud object and no local write-audit record. + +Record `PASS`, `FAIL`, or `N/A` beside each row. Overall approval is `PASS` only +when every required row passes and the cleanup is complete. + +| Check | Required result | Result | +| --- | --- | --- | +| Revision and clean checkout | Intended revision; no unexpected changes | | +| Credential-free Cloud routes and write refusals | Pytest passed | | +| Public ACS OpenAPI contract | Pytest passed | | +| Backend inspection | `cloud` and intended stack | | +| Index list | Exit 0; `data` + `meta` | | +| Role list | Exit 0; `data` + `meta` | | +| HEC token list | Exit 0; secrets absent or `` | | +| Complete Cloud read suite (ACS-only) | Pytest passed | | +| Search-head REST | Bounded search and complete read suite passed, or justified N/A | | +| Cleanup | Variables unset and short-lived tokens revoked | | +| **Overall approval** | **PASS** | | ## Group 5: ACS public contract