feat(ci-health): codify a precise, reproducible governed-fleet CI-health sweep

[rysweet]

Summary

Advances the standing CI-health stewardship goal by fixing the measurement instrument, not by fabricating a fix against a green fleet.

I grounded this cycle in real CI state rather than trusting the prior cycle's narrative. The naive hand-rolled gh run list sweep cannot distinguish an actionable active-CI failure from a disabled/cancelled non-failure — which is exactly why prior "every workflow's latest run is success" claims were imprecise and reviewer-rejected. Concretely, the live sweep shows azlin main has 7 workflows whose latest run is failure — but every one is disabled_manually (turned off ~2026-06-15), and agent-kgpacks' "Build Knowledge Pack" latest run is a non-failure (cancelled/skipped). A sweep that reads only run conclusions either mislabels those as failures or falsely claims "all success."

This PR codifies the sweep so its verdict is precise, reproducible, and evidence-producing.

What changed

• src/ci_health/ — a pure classifier that carries workflow enablement state alongside the latest default-branch run conclusion and labels each workflow green / actionable_failure / ignored(reason). Disabled workflows, cancelled/skipped/neutral/action_required/stale runs, and in-progress runs never fail the fleet; an active workflow whose latest completed run concluded failure/timed_out/startup_failure always does. The fleet is green iff zero actionable failures. Includes a pluggable GhWorkflowClient (live gh vs. offline fixture) so classification is deterministic and offline-testable.
• simard ci-health [--json] [--from-json <path>] operator subcommand (mirrors self-health); exit code 0 iff the fleet is green.
• error: new CiHealthGhCommandFailed variant (+ display + test).
• docs/reference/ci-health-sweep.md (+ mkdocs nav).
• tests/gadugi/ci-health-sweep.{sh,yaml} + fixtures encoding the real signal classes.

Substantive verdict (this cycle)

Live sweep result: CI-HEALTH: GREEN — 63 workflows across 10 governed repos + Simard, 0 actionable failures. The only non-green latest-runs are azlin's 7 disabled workflows and agent-kgpacks' cancelled/skipped run — correctly classified ignored, not failures. There is no failing default-branch workflow to fix this cycle; the deliverable is the codified, verifiable sweep that makes that conclusion reproducible.

---

Merge-ready evidence

1. qa-team scenarios (validated + run)

• gadugi scenario: tests/gadugi/ci-health-sweep.yaml (outside-in, offline fixtures).
  • gadugi-test validate -f tests/gadugi/ci-health-sweep.yaml → ✓ Scenario "CI-health governed-fleet sweep" is valid (1 valid, 0 invalid).
  • gadugi-test run -d tests/gadugi -s ci-health-sweep → ✓ Passed: 1 ✗ Failed: 0.
  • Asserts: disabled failures / cancelled / in-progress stay GREEN (exit 0); a genuine active-CI failure turns the fleet FAILING (non-zero exit) and is named with its run URL; JSON shape ("green": true).
• 21 Rust unit tests in src/ci_health/tests.rs (+ 1 error-display test) all pass: cargo test --lib -- ci_health:: display_ci_health → 28 passed; 0 failed (ci_health + error-display). Cover: disabled+failure→ignored, active+failure→actionable, cancelled/skipped→ignored, no-run/in-progress→ignored, latest-run-by-createdAt selection, workflow↔run join, empty-conclusion→None, collect_fleet via fake client, fixture round-trip.

2. Docs updated for user-facing surfaces

• New user-facing surface = the simard ci-health subcommand. Documented in docs/reference/ci-health-sweep.md (front-matter compliant) and added to mkdocs.yml nav + the operator CLI --help text.
• mkdocs build --strict → Documentation built with 0 broken links, 0 orphaned nav pages (verify-docs T3/T4/T7 all PASS).

4. CI green (local pre-push mirror of CI verify.yml)

All pre-push gates passed on push:

• Rust-only gate: Passed
• cargo fmt --all -- --check: Passed
• cargo test --release --lib race subset: Passed
• cargo clippy --all-targets --all-features --locked -- -D warnings: Passed
• Additionally verified locally: cargo build (default), cargo build --release --bin simard --no-default-features --locked (issue Make Simard's cargo features build BY DEFAULT so no functional capability is opt-in. Operator directive: "none of the features need to be opt-in in the future." One coherent merge-ready PR against rys #2576 contract): Passed.

Supply-chain note (required for green CI): a new advisory RUSTSEC-2026-0204 (crossbeam-epoch fmt::Display null-ptr deref) was issued 2026-07-06 and turned cargo-audit/cargo-deny red fleet-wide (not caused by this change — no deps were modified; my first push passed these). It has no upstream fix and is transitive (simard→rustyclawd-tools→moka→crossbeam-epoch), unreachable in Simard (we never Display-format crossbeam pointers). Per the repo's documented advisory policy (precedent rsa/RUSTSEC-2023-0071), added a justified per-ID ignore in deny.toml + .cargo/audit.toml and recorded it in docs/reference/dependency-trust-policy.md. Local: cargo deny check → advisories/bans/licenses/sources ok; cargo audit → exit 0.

6. Focused diff

+1614 lines, 18 files, all under src/ci_health/, src/operator_cli/, src/error/ (additive variant), docs/, tests/gadugi/, and one nav line in mkdocs.yml. The only edits outside the ci_health feature are the RUSTSEC-2026-0204 supply-chain exemption (deny.toml, .cargo/audit.toml, dependency-trust-policy.md) — a required, justified, fleet-wide advisory remediation without which CI cannot be green; it follows the repo's documented advisory policy.

3. quality-audit (≥3 SEEK→VALIDATE→FIX cycles, ended clean)

Independent code-review passes over the full diff:

• Cycle 1 — Medium (false-negative), FIXED. A flat gh run list --limit 200 branch-wide window could drop an infrequently-triggered active workflow's latest run, misclassifying it NoRun → false green. Fix: query the latest run per active workflow absent from the window (gh run list --workflow <id> --limit 1). Regression test collect_fleet_falls_back_for_active_workflow_missing_from_window.
• Cycle 2 — Medium (false-negative), FIXED. Runs were joined to workflows by display name, which GitHub doesn't guarantee unique; two workflows sharing a name: collapsed onto one run, hiding a failing sibling → false green. Fix: key the join on the unique workflowDatabaseId. Regression test build_repo_snapshot_keys_by_id_so_same_named_workflows_dont_collapse.
• Cycle 3 — CLEAN. Fresh full-feature scan (classification, gh collector, CLI, error, tests, gadugi, docs): both prior fixes verified correct/complete; zero critical/high, zero medium correctness/security findings. Also confirmed: no shell injection (Command::args, no shell; hardcoded slugs; discrete argv), exhaustive error Display match, correct exit-code path.

Ended on a clean cycle. 21 unit tests pass.

---

Reproduce

cargo run --bin simard -- ci-health           # live sweep (needs gh auth); exit 0 iff green
cargo run --bin simard -- ci-health --json     # machine-readable FleetReport
cargo run --bin simard -- ci-health --from-json tests/gadugi/fixtures/ci-health-green.json

[github-actions]

📊 Coverage Summary

Generated by cargo llvm-cov --workspace --summary-only (nightly, excluding test files)

Module	Lines	Covered	Coverage
Total	147922	123758	83.7%

_{Coverage data from CI run. Test files matching tests?/ are excluded from line counts.}