Cost context governance

docs/cost-context-governance-design.md

@@ -0,0 +1,118 @@
+# Cost & Context Governance — v0.7.0 Phase 1 Design
+
+## Version
+This document captures the MiniCISO-side design and documentation updates aligned to the Hermes cost/context governance v0.7.0 rollout.
+
+## Goal
+Implement a mandatory, provider-independent governance layer for MiniCISO that preserves evidence quality and independent QA while preventing runaway token/context/tool consumption.
+
+## Repositories inspected
+- MiniCISO overlay checkout: `<miniciso-repo-root>`
+- Hermes runtime checkout: `<hermes-agent-repo-root>`
+
+## Existing integration points
+
+### MiniCISO overlay
+- `profiles/chief-of-staff/SOUL.md`: core Chief-of-Staff operating instructions. Best place to make the governance skill mandatory at the procedural layer.
+- `config/chief-of-staff.public.yaml`: public example config that can expose governance defaults.
+- No existing MiniCISO-owned runtime package for deterministic governance.
+
+### Hermes runtime
+- `run_agent.py`: `AIAgent` wrappers, turn execution entrypoints, tool execution dispatch, `delegate_task` dispatch, context compression hook.
+- `agent/conversation_loop.py`: main model-call loop, iteration handling, context growth, turn completion path.
+- `agent/context_compressor.py`: existing compaction/externalization mechanism.
+- `agent/tool_executor.py`: deterministic tool execution path; best insertion point for tool-call accounting and pre-tool circuit breakers.
+- `agent/tool_guardrails.py`: existing repeated-tool/no-progress primitive that can be complemented by broader engagement governance.
+- `tools/delegate_tool.py`: child-agent creation, task context packaging, timeout/error/summary handling; best insertion point for child budget allocation, bounded child context manifests, and structured partial handoffs.
+- `model_tools.py`: tool-definition assembly; best insertion point for exact allowlist filtering and tool-schema overhead measurement.
+- `hermes_cli/config.py`: default config surface for observe/enforce/disabled modes and budget profiles.
+
+## Selected implementation strategy
+Use a dual-layer design:
+
+1. **Procedural layer (MiniCISO-owned)**
+   - Add mandatory skill `cost-context-governance`.
+   - Patch Chief-of-Staff SOUL to require loading/classification on every request.
+   - Keep conversational requests lightweight via explicit pass-through classification.
+
+2. **Runtime layer (Hermes-side generic enforcement, MiniCISO-enabled by config)**
+   - Add a new controller module that:
+     - classifies requests;
+     - creates an engagement workspace + JSON artifacts;
+     - allocates hierarchical envelopes;
+     - records telemetry/context manifests/tool-schema overhead;
+     - enforces model/tool/delegation/time/context thresholds;
+     - preserves partial handoffs on timeout/hard stop.
+   - Enable by default in the active Chief-of-Staff profile config.
+
+## Affected files
+
+### New files
+- `<hermes-agent-repo-root>/agent/cost_context_governance.py`
+- `<hermes-agent-repo-root>/tests/agent/test_cost_context_governance.py`
+- `<hermes-agent-repo-root>/tests/tools/test_delegate.py`
+- `skills/cost-context-governance/SKILL.md`
+- `docs/cost-context-governance-design.md`
+
+### Patched Hermes files
+- `run_agent.py`
+- `agent/conversation_loop.py`
+- `agent/tool_executor.py`
+- `tools/delegate_tool.py`
+- `model_tools.py`
+- `hermes_cli/config.py`
+
+### Patched MiniCISO/profile files
+- `profiles/chief-of-staff/SOUL.md`
+- `<hermes-profile-root>/SOUL.md`
+- `<hermes-profile-root>/config.yaml`
+- `<hermes-profile-root>/skills/cost-context-governance/SKILL.md`
+
+## Runtime data layout
+Under the active profile:
+- `engagements/<engagement_id>/brief.json`
+- `engagements/<engagement_id>/budget.json`
+- `engagements/<engagement_id>/telemetry.jsonl`
+- `engagements/<engagement_id>/evidence_ledger.jsonl`
+- `engagements/<engagement_id>/claim_ledger.jsonl`
+- `engagements/<engagement_id>/checkpoints/*.json`
+- `engagements/<engagement_id>/partial_handoffs/*.json`
+
+## Scope of first implementation
+Implemented now for immediate VPS effect:
+- observe/enforce/disabled modes;
+- root + child envelope accounting with file-lock persistence;
+- configurable budget profiles and QA reserve;
+- model/tool/delegation/time/context thresholds;
+- context manifests and tool-schema overhead metrics;
+- bounded child context package metadata;
+- structured partial handoffs on timeout / limit / hard stop;
+- role/toolset filtering via runtime allowlist intersection;
+- local JSONL telemetry and engagement summaries.
+
+## Compatibility risks
+- Hermes currently lacks a native pre-model-call governance abstraction, so the first implementation must patch runtime call sites directly.
+- Tool filtering must not break existing sessions that intentionally grant broader toolsets; fallback is intersection-only when governance is active.
+- Child timeout handling varies by provider/runtime path; partial handoff synthesis must work even when the provider returns no summary.
+- Existing MiniCISO repo has no skill-sync convention yet; immediate VPS install will patch the active profile directly and also stage the skill inside the overlay repo.
+
+## Test plan
+Automated tests will cover at least:
+- threshold/circuit-breaker evaluation before 50 unbounded child calls;
+- concurrency-safe shared root budget;
+- timeout → persisted partial handoff (not null);
+- QA reserve protection;
+- per-task tool filtering before schema assembly;
+- measurable unused tool-schema overhead;
+- compaction/externalization before threshold breach;
+- lightweight conversational pass-through;
+- usage estimation fallback when provider metadata is missing.
+
+## Why MiniCISO-level enforcement alone is insufficient
+MiniCISO prompt/skill instructions can require planning and checkpointing, but they cannot deterministically:
+- prevent an agent from sending oversized tool schemas;
+- stop a child before the next model call once budget is exhausted;
+- atomically coordinate concurrent child consumption from a shared root budget;
+- force structured partial handoffs when runtime timeouts occur.
+
+Therefore the smallest required upstream-compatible change is a generic Hermes runtime governance controller that MiniCISO enables by config. The fallback for unsupported runtimes is visible observation-only mode with explicit warning in child/task results.

docs/dependencies-and-configuration.md

@@ -69,6 +69,7 @@ env -u VIRTUAL_ENV uv run bigua-analyzer --help
 - KAG query builder
 - deterministic retrieval selector
 - manual wrapper with shadow-mode logging
+- RTK execution output optimizer experiment for narrow operational command classes
 
 **Repo-side location:**
 - `tools/headroom_phase1/`
@@ -79,6 +80,13 @@ env -u VIRTUAL_ENV uv run bigua-analyzer --help
 - no raw evidence artifacts in the repo
 - absence in retrieval pack must remain `not_verified_in_raw`
 - keep selection-first logs and code separable from confidential engagement data
+- RTK reduced output is never authoritative
+- RTK default mode is `shadow`
+- `MINICISO_EXECUTION_OUTPUT_OPTIMIZER=0` must preserve rollback to passthrough
+
+**RTK MVP scope:**
+- included: `git_status`, `git_diff_stat`, `ls`, `find`, `tree`, `git_fetch`
+- excluded: `read_file`, `search_files`, `grep`, reports/findings, SARIF, SBOM, PoCs, HTTP traces, SME/Security QA responses
 
 ### ProjectDiscovery Cloud / passive discovery layer
 **Purpose:** passive asset discovery and cloud-assisted recon support when the assessment model includes authorized external inventory work.

docs/github-pr-access.md

@@ -40,7 +40,7 @@ If you use a classic PAT instead of a fine-grained one:
 Store it outside the repo, in the active profile:
 
 ```bash
-/home/vpsadmin/.hermes/profiles/chief-of-staff/.env
+<hermes-profile-root>/.env
 ```
 
 Add:
@@ -74,8 +74,8 @@ gh auth status
 Always valid:
 
 ```bash
-git -C /home/vpsadmin/miniCISO remote -v
-git -C /home/vpsadmin/miniCISO ls-remote origin
+git -C <miniciso-repo-root> remote -v
+git -C <miniciso-repo-root> ls-remote origin
 ```
 
 To validate the API directly:
@@ -97,7 +97,7 @@ PY
 ### With `gh`
 
 ```bash
-cd /home/vpsadmin/miniCISO
+cd <miniciso-repo-root>
 git checkout -b chore/my-change
 # edit / export / validate
 git add -A

docs/headroom-kag-selective-retrieval.md

@@ -45,6 +45,49 @@ The initial recommendation is to run in **shadow mode**:
 4. continue running the raw/full flow in parallel;
 5. compare savings, recovered evidence, and `decision_delta`.
 
+## RTK execution output optimizer (experimental)
+
+A second, narrower experiment now lives beside the wrapper in `tools/headroom_phase1/`: the **RTK execution output optimizer**.
+
+Its contract is intentionally strict:
+
+- `raw` remains authoritative at all times;
+- the optimizer runs only in **shadow mode** by default;
+- the reduced view is derivative/log-only and never replaces the effective payload;
+- a single env var kill switch restores baseline behavior immediately.
+
+### Included MVP operation classes
+
+- `git_status`
+- `git_diff_stat`
+- `ls`
+- `find`
+- `tree`
+- `git_fetch`
+
+### Explicitly excluded from this MVP
+
+- `read_file`
+- `search_files`
+- `grep`
+- evidence artifacts
+- reports / findings
+- SARIF / SBOM / PoC material
+- HTTP traces
+- SME or Security QA responses
+
+### Runtime knobs
+
+```text
+MINICISO_EXECUTION_OUTPUT_OPTIMIZER=1
+MINICISO_EXECUTION_OUTPUT_OPTIMIZER_MODE=shadow
+MINICISO_EXECUTION_OUTPUT_OPTIMIZER_ALLOWLIST=git_status,git_diff_stat,ls,find,tree,git_fetch
+```
+
+### Rollback rule
+
+Set `MINICISO_EXECUTION_OUTPUT_OPTIMIZER=0` to force passthrough and preserve the pre-experiment baseline without changing the wrapper's real output path.
+
 ## Minimum pack provenance
 
 Each selected slice must preserve, at minimum:

docs/self-update-capability.md

@@ -55,8 +55,8 @@ You can also point to explicit sources:
 
 ```bash
 python3 scripts/export_safe_self_state.py \
-  --source-workspace /home/vpsadmin/miniciso-security \
-  --source-profile /home/vpsadmin/.hermes/profiles/chief-of-staff \
+  --source-workspace <local-workspace-root> \
+  --source-profile <hermes-profile-root> \
   --apply
 ```
 

scripts/bootstrap.ps1

@@ -91,7 +91,7 @@ if (-not $SkipProviderSetup) {
     Invoke-Hermes -Arguments @('setup')
 }
 
-$profileRoot = Join-Path $HOME '.hermes\profiles'
+$profileRoot = Join-Path $HermesHome 'profiles'
 $profiles = Get-ChildItem -LiteralPath (Join-Path $repoRoot 'profiles') -Directory | Sort-Object Name
 if ($profiles.Count -ne 9) { throw "Expected 9 profiles; found $($profiles.Count)." }
 

scripts/bootstrap.sh

@@ -72,7 +72,7 @@ if [[ "$SKIP_PROVIDER_SETUP" == false ]]; then
   hermes setup
 fi
 
-profile_root="$HOME/.hermes/profiles"
+profile_root="$HERMES_HOME/profiles"
 profiles=()
 for profile_dir in "$REPO_ROOT"/profiles/*; do
   [[ -d "$profile_dir" ]] && profiles+=("${profile_dir##*/}")

scripts/export_safe_self_state.py

@@ -37,8 +37,16 @@
 def parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser(description="Export public/safe MiniCISO state from the VPS into this repo.")
     parser.add_argument("--repo-root", default=str(Path(__file__).resolve().parents[1]))
-    parser.add_argument("--source-workspace", default="/home/vpsadmin/miniciso-security")
-    parser.add_argument("--source-profile", default="/home/vpsadmin/.hermes/profiles/chief-of-staff")
+    parser.add_argument(
+        "--source-workspace",
+        default=str(Path.home() / "miniciso-security"),
+        help="Path to the local MiniCISO workspace to export from.",
+    )
+    parser.add_argument(
+        "--source-profile",
+        default=str(Path.home() / ".hermes/profiles/chief-of-staff"),
+        help="Path to the local Hermes profile root to export from.",
+    )
     parser.add_argument("--apply", action="store_true", help="Actually write files. Default is dry-run.")
     return parser.parse_args()
 

scripts/smoke-test.ps1

@@ -3,6 +3,7 @@ param([switch]$Online)
 
 $ErrorActionPreference = 'Stop'
 $repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path
+$hermesHome = if ($env:HERMES_HOME) { $env:HERMES_HOME } else { Join-Path $HOME '.hermes' }
 $profiles = Get-ChildItem -LiteralPath (Join-Path $repoRoot 'profiles') -Directory | Sort-Object Name
 $hermes = Get-Command hermes -ErrorAction Stop
 $profileList = (& $hermes.Source profile list 2>&1 | Out-String)
@@ -12,7 +13,7 @@ foreach ($profile in $profiles) {
     if ($profileList -notmatch [regex]::Escape($profile.Name)) {
         throw "Profile not registered in Hermes: $($profile.Name)"
     }
-    $installedSoul = Join-Path $HOME ".hermes\profiles\$($profile.Name)\SOUL.md"
+    $installedSoul = Join-Path $hermesHome "profiles\$($profile.Name)\SOUL.md"
     if (-not (Test-Path -LiteralPath $installedSoul)) {
         throw "SOUL.md not installed: $($profile.Name)"
     }

scripts/smoke-test.sh

@@ -15,7 +15,7 @@ done
 
 for profile in "${profiles[@]}"; do
   grep -Fq "$profile" <<<"$profile_list" || { echo "Profile not registered: $profile" >&2; exit 1; }
-  [[ -f "$HOME/.hermes/profiles/$profile/SOUL.md" ]] || { echo "SOUL.md not installed: $profile" >&2; exit 1; }
+  [[ -f "${HERMES_HOME:-$HOME/.hermes}/profiles/$profile/SOUL.md" ]] || { echo "SOUL.md not installed: $profile" >&2; exit 1; }
   echo "OK: $profile"
   if [[ "$ONLINE" == true ]]; then
     hermes -p "$profile" chat -Q -q 'Answer in one line starting with OK and state your role in MiniCISO.'

tools/headroom_phase1/hr_manual_wrapper.py

@@ -98,7 +98,11 @@ def parse_args() -> argparse.Namespace:
         help="Human/QA verdict for this artifact run",
     )
     p.add_argument("--note", default="", help="Free-form operator note")
-    p.add_argument("--log-dir", default="/home/vpsadmin/miniciso-security/headroom_phase1/logs", help="Log directory")
+    p.add_argument(
+        "--log-dir",
+        default=str(Path.home() / "miniciso-security/headroom_phase1/logs"),
+        help="Log directory",
+    )
     p.add_argument("--selection-index", default="", help="Path to structural index JSON for selection-first shadow mode")
     p.add_argument("--selection-query", default="", help="Path to KAG query JSON for selection-first shadow mode")
     p.add_argument("--selection-pack", default="", help="Path to retrieval pack JSON for selection-first shadow mode")