Coding-Autopilot-System · OgeonX-Ai · Jul 8, 2026 · Jul 8, 2026 · Jul 8, 2026 · chatgpt-codex-connector
@@ -76,3 +76,5 @@ tests/                 Determinism, safety, and CLI contract tests
 3. Add repeated-run statistics, signed reports, and longitudinal trends.
 
 See [SECURITY.md](SECURITY.md) and [CONTRIBUTING.md](CONTRIBUTING.md).
+
+<!-- docs-verified: 4fe936cc83ffdc4fd6ad825c373e949b1edbe0eb 2026-07-08 -->
@@ -0,0 +1,55 @@
+# Architecture
+
+## Evidence gate flow
+
+```mermaid
+flowchart TD
+    CLI["cas_evals.cli"]
+    FixtureJSON[("Fixture JSON\ngolden.json / adversarial.json")]
+    CLI -->|"Parses"| FixtureJSON
+    FixtureJSON -->|"Yields Suite & Cases"| Router{"Live mode\nenabled?"}
+    Router -- "No (default)" --> OfflineEv["Offline Evaluator\nevaluator.py"]
+    Router -- "Yes (--reference-product-url)" --> LiveRef["Reference Product Adapter\nreference_product.py"]
+    subgraph LiveFlow["Live reference-product adapter"]
+        LiveRef --> BuildEnv["Build PromptEnvelope"] --> HTTPPost["HTTP POST /api/v1/workflows"] --> ValidateMeta{"Validate lifecycle\nevents/metadata"}
+        ValidateMeta -- Invalid --> RefError(("ReferenceProductError"))
+        ValidateMeta -- Valid --> LiveCase["Inject live output as response"]
+    end
+    LiveCase --> SharedEval["Shared core evaluator\n_evaluate_case_with_evidence"]
+    OfflineEv --> SharedEval
+    SharedEval --> Metrics["Quality / Safety / Cost / Latency"]
+    Metrics --> ValidateContract["Validate EvaluationResult\nagainst vendored cas-contracts schema"]
+    ValidateContract --> Gate{"release --check\nexit code"}
+    Gate -- "0" --> Pass(("CI green"))
+    Gate -- "non-zero" --> Fail(("CI red"))
+```
+
+<!-- codex:generate-image prompt="A courtroom-style evidence gate: fixture documents and a live data stream both feed into a single judge's bench (a glowing scale icon) that stamps each case with a pass or fail seal against a rulebook labeled cas-contracts; a exit-code lever on the side flips green or red; isometric, enterprise blue/graphite palette" style="isometric, enterprise, clean" replaces="mermaid-above" -->
+
+## Modules
+
+- `cli.py` — parses arguments, loads the fixture suite, writes JSON results to stdout and/or
+  `--output`.
+- `evaluator.py` — pure evaluation kernel (`_evaluate_case_with_evidence`): scores quality
+  (fraction of expected keywords present), safety (absence of prohibited content, mandatory
+  100%), and validates cost/latency against configured thresholds. Produces the standard
+  `EvaluationResult`.
+- `reference_product.py` — HTTP transport and lifecycle-metadata validation for the opt-in
+  live mode; wraps the core evaluator with a dynamic response instead of a fixture response.
+- `contracts.py` — validates evidence offline against the pinned `cas-contracts` schemas
+  vendored under `vendor/cas-contracts/`.
+
+## Registry-fetch smoke check — in progress (PR #9)
+
+`main` today validates schema conformance **offline only**, against the vendored copy under
+`vendor/cas-contracts/v0.1.0/` — it does not prove the live Pages registry actually resolves.
+PR #9 (`feat(contracts): verify registry-fetch smoke check + update vendored $id to resolvable
+Pages URL`) adds a `cas_evals.registry_check` module (stdlib `urllib`, no new dependencies)
+that performs live HTTP GETs against `index.json`, `v0.1/manifest.json`, and two schema files
+on the Pages registry and asserts HTTP 200 — proving resolution, which the offline check
+cannot. It also re-vendors the local `$id` expectations to the new resolvable Pages URL
+convention, as a companion to `cas-contracts` PR #18 (independent of whether #18 has merged,
+per the PR's own verification: `python -m cas_evals.registry_check` was confirmed to exit 0
+against the live registry pre-merge). Until PR #9 merges, no live-resolution check runs in CI.
+
+<!-- docs-verified: 4fe936cc83ffdc4fd6ad825c373e949b1edbe0eb 2026-07-08 -->
@@ -0,0 +1,27 @@
+# Decisions
+
+## ADR convention
+
+`docs/adr/README.md` establishes the convention (sequential numbering, Context/Decision/
+Consequences) but **no numbered ADR files exist in the repo yet**. Decisions to date live in
+`.planning/phases/` plan/summary pairs instead.
+
+## Phase history (`.planning/phases/`, this repo's own GSD project)
+
+| Phase | Topic |
+|---|---|
+| 02 | Shared contracts and corpus |
+
+See `.planning/phases/02-shared-contracts-and-corpus/*-SUMMARY.md` for the detailed record.
+
+## Open decisions tracked in this Phase 36 refresh
+
+- **PR #9** (`feat/registry-fetch-smoke-check`) — adds a live registry-fetch smoke check
+  (`cas_evals.registry_check`) and re-vendors `$id` expectations to the resolvable Pages URL
+  convention, as a companion to `cas-contracts` PR #18; open, not yet merged. See
+  [Architecture](./Architecture.md#registry-fetch-smoke-check--in-progress-pr-9).
+- **PR #10** (`ci/sha-pin-actions`) — pins third-party GitHub Actions to commit SHAs; open,
+  not yet merged. (Note: the current `ci.yml` already shows pinned `checkout`/`setup-python`
+  actions on `main` — verify PR #10's remaining scope before assuming no pinning exists.)
+
+<!-- docs-verified: 4fe936cc83ffdc4fd6ad825c373e949b1edbe0eb 2026-07-08 -->
@@ -0,0 +1,34 @@
+# CAS Evals Wiki
+
+## Role in the CAS portfolio
+
+`cas-evals` is part of the **Governance plane** of the Coding-Autopilot-System three-plane
+model (Control / Execution / Governance). It is the evidence gate: it runs versioned golden
+and adversarial tasks against deterministic reference responses (or, opt-in, against a live
+`cas-reference-product` workflow) and produces machine-readable, schema-conformant
+quality/safety/cost/latency evidence with a CLI that exits non-zero on any mandatory metric
+failure — making "done" mean something checkable, not just claimed.
+
+| Plane | This repo's responsibility |
+|---|---|
+| Control | *(consumed indirectly — evidence gate for control-plane goal completion)* |
+| Execution | *(consumed indirectly — evaluates `cas-reference-product` live output, opt-in)* |
+| Governance | Deterministic evaluation kernel, evidence gate, `cas-contracts` schema validation |
+
+## Quickstart
+
+- [README.md](../../README.md) — Quickstart, Metrics, Evidence Contract
+- [Architecture](./Architecture.md) — evidence gate flow (offline + live reference-product modes)
+- [Operations](./Operations.md) — verified CLI/gate/CI commands
+- [Decisions](./Decisions.md) — phase history and open PRs (including the registry-fetch
+  smoke check, PR #9)
+
+## Ecosystem links
+
+Part of the [Coding-Autopilot-System](https://github.com/Coding-Autopilot-System) org:
+[gsd-orchestrator](https://github.com/Coding-Autopilot-System/gsd-orchestrator) (control plane) ·
+[autogen](https://github.com/Coding-Autopilot-System/autogen) (execution plane) ·
+[Promptimprover](https://github.com/Coding-Autopilot-System/Promptimprover) (prompt governance) ·
+[cas-contracts](https://github.com/Coding-Autopilot-System/cas-contracts) (shared schemas)
+
+<!-- docs-verified: 4fe936cc83ffdc4fd6ad825c373e949b1edbe0eb 2026-07-08 -->
@@ -0,0 +1,41 @@
+# Operations
+
+## Setup and run
+
+```powershell
+python -m pip install -e .
+python -m unittest discover -s tests -v
+python -m cas_evals.cli benchmarks/v0.2/golden.json --output artifacts/golden.json
+python -m cas_evals.cli benchmarks/v0.2/adversarial.json --output artifacts/adversarial.json
+python -m cas_evals.release --check
+```
+
+The CLI exits non-zero when any mandatory metric fails, making each suite usable as a CI
+regression gate. Windows users can run the complete verification path with
+`.\scripts\verify.ps1`.
+
+## Live reference-product mode (opt-in)
+
+```powershell
+python -m cas_evals.cli benchmarks/reference-product/v0.1/golden.json --reference-product-url
+python -m cas_evals.cli benchmarks/reference-product/v0.1/adversarial.json --reference-product-url
+```
+
+## CI (`.github/workflows/ci.yml`, matrix: ubuntu-latest + windows-latest, Python 3.11 + 3.13, 10-minute timeout)
+
+1. `actions/checkout` (pinned to commit SHA)
+2. `actions/setup-python` (pinned to commit SHA)
+3. `python -m pip install -e .`
+4. `python -m unittest discover -s tests -v`
+5. `python -m unittest discover -s tests -p test_reference_product.py -v`
+6. `python -m cas_evals.cli benchmarks/v0.2/golden.json`
+7. `python -m cas_evals.cli benchmarks/v0.2/adversarial.json`
+8. `python -m cas_evals.release --check` — the CLI-exit-code gate; **not** a coverage badge,
+   since no formal coverage-percentage gate exists in this repo.
+
+A separate `.github/workflows/codeql.yml` runs CodeQL analysis, and `.github/workflows/
+pages.yml` publishes documentation (badges in the root `README.md`). A registry-fetch smoke
+check (live HTTP verification of the Pages registry) is in progress — see
+[Architecture](./Architecture.md) and PR #9.
+
+<!-- docs-verified: 4fe936cc83ffdc4fd6ad825c373e949b1edbe0eb 2026-07-08 -->
Original file line number	Diff line number	Diff line change
Expand Up		@@ -76,3 +76,5 @@ tests/ Determinism, safety, and CLI contract tests
		3. Add repeated-run statistics, signed reports, and longitudinal trends.

		See [SECURITY.md](SECURITY.md) and [CONTRIBUTING.md](CONTRIBUTING.md).

		<!-- docs-verified: 4fe936cc83ffdc4fd6ad825c373e949b1edbe0eb 2026-07-08 -->