From f305f9e6a4562e7e7f0d2393bece2f8b31bbb2b6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 8 Jul 2026 15:13:43 +0300 Subject: [PATCH 1/2] docs(36-02): append freshness footer to README --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index d8c09f2..e7b59d3 100644 --- a/README.md +++ b/README.md @@ -76,3 +76,5 @@ tests/ Determinism, safety, and CLI contract tests 3. Add repeated-run statistics, signed reports, and longitudinal trends. See [SECURITY.md](SECURITY.md) and [CONTRIBUTING.md](CONTRIBUTING.md). + + From 3ead4655ecf078b304661ce12da0a710d48f7543 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kim=20Harjam=C3=A4ki?= Date: Wed, 8 Jul 2026 15:24:14 +0300 Subject: [PATCH 2/2] docs(36-03): add docs/wiki tree (Home, Architecture, Operations, Decisions) - Architecture.md documents the evidence gate flow (offline + live reference-product modes) and the in-progress registry-fetch smoke check (PR #9, companion to cas-contracts PR #18) - Operations.md verified CLI/gate/CI commands - Decisions.md indexes phase history and open PRs --- docs/wiki/Architecture.md | 55 +++++++++++++++++++++++++++++++++++++++ docs/wiki/Decisions.md | 27 +++++++++++++++++++ docs/wiki/Home.md | 34 ++++++++++++++++++++++++ docs/wiki/Operations.md | 41 +++++++++++++++++++++++++++++ 4 files changed, 157 insertions(+) create mode 100644 docs/wiki/Architecture.md create mode 100644 docs/wiki/Decisions.md create mode 100644 docs/wiki/Home.md create mode 100644 docs/wiki/Operations.md diff --git a/docs/wiki/Architecture.md b/docs/wiki/Architecture.md new file mode 100644 index 0000000..937e0fa --- /dev/null +++ b/docs/wiki/Architecture.md @@ -0,0 +1,55 @@ +# Architecture + +## Evidence gate flow + +```mermaid +flowchart TD + CLI["cas_evals.cli"] + FixtureJSON[("Fixture JSON\ngolden.json / adversarial.json")] + CLI -->|"Parses"| FixtureJSON + FixtureJSON -->|"Yields Suite & Cases"| Router{"Live mode\nenabled?"} + Router -- "No (default)" --> OfflineEv["Offline Evaluator\nevaluator.py"] + Router -- "Yes (--reference-product-url)" --> LiveRef["Reference Product Adapter\nreference_product.py"] + subgraph LiveFlow["Live reference-product adapter"] + LiveRef --> BuildEnv["Build PromptEnvelope"] --> HTTPPost["HTTP POST /api/v1/workflows"] --> ValidateMeta{"Validate lifecycle\nevents/metadata"} + ValidateMeta -- Invalid --> RefError(("ReferenceProductError")) + ValidateMeta -- Valid --> LiveCase["Inject live output as response"] + end + LiveCase --> SharedEval["Shared core evaluator\n_evaluate_case_with_evidence"] + OfflineEv --> SharedEval + SharedEval --> Metrics["Quality / Safety / Cost / Latency"] + Metrics --> ValidateContract["Validate EvaluationResult\nagainst vendored cas-contracts schema"] + ValidateContract --> Gate{"release --check\nexit code"} + Gate -- "0" --> Pass(("CI green")) + Gate -- "non-zero" --> Fail(("CI red")) +``` + + + +## Modules + +- `cli.py` — parses arguments, loads the fixture suite, writes JSON results to stdout and/or + `--output`. +- `evaluator.py` — pure evaluation kernel (`_evaluate_case_with_evidence`): scores quality + (fraction of expected keywords present), safety (absence of prohibited content, mandatory + 100%), and validates cost/latency against configured thresholds. Produces the standard + `EvaluationResult`. +- `reference_product.py` — HTTP transport and lifecycle-metadata validation for the opt-in + live mode; wraps the core evaluator with a dynamic response instead of a fixture response. +- `contracts.py` — validates evidence offline against the pinned `cas-contracts` schemas + vendored under `vendor/cas-contracts/`. + +## Registry-fetch smoke check — in progress (PR #9) + +`main` today validates schema conformance **offline only**, against the vendored copy under +`vendor/cas-contracts/v0.1.0/` — it does not prove the live Pages registry actually resolves. +PR #9 (`feat(contracts): verify registry-fetch smoke check + update vendored $id to resolvable +Pages URL`) adds a `cas_evals.registry_check` module (stdlib `urllib`, no new dependencies) +that performs live HTTP GETs against `index.json`, `v0.1/manifest.json`, and two schema files +on the Pages registry and asserts HTTP 200 — proving resolution, which the offline check +cannot. It also re-vendors the local `$id` expectations to the new resolvable Pages URL +convention, as a companion to `cas-contracts` PR #18 (independent of whether #18 has merged, +per the PR's own verification: `python -m cas_evals.registry_check` was confirmed to exit 0 +against the live registry pre-merge). Until PR #9 merges, no live-resolution check runs in CI. + + diff --git a/docs/wiki/Decisions.md b/docs/wiki/Decisions.md new file mode 100644 index 0000000..0538039 --- /dev/null +++ b/docs/wiki/Decisions.md @@ -0,0 +1,27 @@ +# Decisions + +## ADR convention + +`docs/adr/README.md` establishes the convention (sequential numbering, Context/Decision/ +Consequences) but **no numbered ADR files exist in the repo yet**. Decisions to date live in +`.planning/phases/` plan/summary pairs instead. + +## Phase history (`.planning/phases/`, this repo's own GSD project) + +| Phase | Topic | +|---|---| +| 02 | Shared contracts and corpus | + +See `.planning/phases/02-shared-contracts-and-corpus/*-SUMMARY.md` for the detailed record. + +## Open decisions tracked in this Phase 36 refresh + +- **PR #9** (`feat/registry-fetch-smoke-check`) — adds a live registry-fetch smoke check + (`cas_evals.registry_check`) and re-vendors `$id` expectations to the resolvable Pages URL + convention, as a companion to `cas-contracts` PR #18; open, not yet merged. See + [Architecture](./Architecture.md#registry-fetch-smoke-check--in-progress-pr-9). +- **PR #10** (`ci/sha-pin-actions`) — pins third-party GitHub Actions to commit SHAs; open, + not yet merged. (Note: the current `ci.yml` already shows pinned `checkout`/`setup-python` + actions on `main` — verify PR #10's remaining scope before assuming no pinning exists.) + + diff --git a/docs/wiki/Home.md b/docs/wiki/Home.md new file mode 100644 index 0000000..94bb046 --- /dev/null +++ b/docs/wiki/Home.md @@ -0,0 +1,34 @@ +# CAS Evals Wiki + +## Role in the CAS portfolio + +`cas-evals` is part of the **Governance plane** of the Coding-Autopilot-System three-plane +model (Control / Execution / Governance). It is the evidence gate: it runs versioned golden +and adversarial tasks against deterministic reference responses (or, opt-in, against a live +`cas-reference-product` workflow) and produces machine-readable, schema-conformant +quality/safety/cost/latency evidence with a CLI that exits non-zero on any mandatory metric +failure — making "done" mean something checkable, not just claimed. + +| Plane | This repo's responsibility | +|---|---| +| Control | *(consumed indirectly — evidence gate for control-plane goal completion)* | +| Execution | *(consumed indirectly — evaluates `cas-reference-product` live output, opt-in)* | +| Governance | Deterministic evaluation kernel, evidence gate, `cas-contracts` schema validation | + +## Quickstart + +- [README.md](../../README.md) — Quickstart, Metrics, Evidence Contract +- [Architecture](./Architecture.md) — evidence gate flow (offline + live reference-product modes) +- [Operations](./Operations.md) — verified CLI/gate/CI commands +- [Decisions](./Decisions.md) — phase history and open PRs (including the registry-fetch + smoke check, PR #9) + +## Ecosystem links + +Part of the [Coding-Autopilot-System](https://github.com/Coding-Autopilot-System) org: +[gsd-orchestrator](https://github.com/Coding-Autopilot-System/gsd-orchestrator) (control plane) · +[autogen](https://github.com/Coding-Autopilot-System/autogen) (execution plane) · +[Promptimprover](https://github.com/Coding-Autopilot-System/Promptimprover) (prompt governance) · +[cas-contracts](https://github.com/Coding-Autopilot-System/cas-contracts) (shared schemas) + + diff --git a/docs/wiki/Operations.md b/docs/wiki/Operations.md new file mode 100644 index 0000000..e7ca142 --- /dev/null +++ b/docs/wiki/Operations.md @@ -0,0 +1,41 @@ +# Operations + +## Setup and run + +```powershell +python -m pip install -e . +python -m unittest discover -s tests -v +python -m cas_evals.cli benchmarks/v0.2/golden.json --output artifacts/golden.json +python -m cas_evals.cli benchmarks/v0.2/adversarial.json --output artifacts/adversarial.json +python -m cas_evals.release --check +``` + +The CLI exits non-zero when any mandatory metric fails, making each suite usable as a CI +regression gate. Windows users can run the complete verification path with +`.\scripts\verify.ps1`. + +## Live reference-product mode (opt-in) + +```powershell +python -m cas_evals.cli benchmarks/reference-product/v0.1/golden.json --reference-product-url +python -m cas_evals.cli benchmarks/reference-product/v0.1/adversarial.json --reference-product-url +``` + +## CI (`.github/workflows/ci.yml`, matrix: ubuntu-latest + windows-latest, Python 3.11 + 3.13, 10-minute timeout) + +1. `actions/checkout` (pinned to commit SHA) +2. `actions/setup-python` (pinned to commit SHA) +3. `python -m pip install -e .` +4. `python -m unittest discover -s tests -v` +5. `python -m unittest discover -s tests -p test_reference_product.py -v` +6. `python -m cas_evals.cli benchmarks/v0.2/golden.json` +7. `python -m cas_evals.cli benchmarks/v0.2/adversarial.json` +8. `python -m cas_evals.release --check` — the CLI-exit-code gate; **not** a coverage badge, + since no formal coverage-percentage gate exists in this repo. + +A separate `.github/workflows/codeql.yml` runs CodeQL analysis, and `.github/workflows/ +pages.yml` publishes documentation (badges in the root `README.md`). A registry-fetch smoke +check (live HTTP verification of the Pages registry) is in progress — see +[Architecture](./Architecture.md) and PR #9. + +