Skip to content

codex-evaluation-foundation: fix harness contracts and add isolated CLI evaluation - #90

Open
tarr1124 wants to merge 3 commits into
mainfrom
feature/codex-evaluation-foundation
Open

tarr1124 wants to merge 3 commits into
mainfrom
feature/codex-evaluation-foundation

Conversation

@tarr1124

@tarr1124 tarr1124 commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Rill's skill tests could report success after child assertions failed, Codex write hooks did not consistently parse actual tool inputs or return blocking exit codes, and legacy/plugin commands could be projected as skills without required metadata. This change propagates failures, normalizes hook targets and denials, and generates thin metadata-bearing skill wrappers while preserving user-owned files and source resources.

Adds an opt-in real-CLI comparison runner with fresh synthetic vaults, explicit models, structured run records, independent expectations, original-file and duplicate checks, timeouts, and contamination monitoring. Copies only allowlisted tracked deployment inputs, includes container schemas, and excludes local plugin configuration/state. Raw logs stay local; ordinary CI makes no model calls.

Validation:

  • All 12 model-free CLI suites passed; the expanded foundation suite has 11 regression tests, including injected failures through the exact CI shell block.
  • Bash syntax, ShellCheck, CJK/PII guards and diff checks passed.
  • Codex CLI 0.154.0 / gpt-6-astra: create-note PASS (35.769s), distill PASS (119.524s), repeated distill PASS (66.059s). Original journal preserved, one knowledge note retained, processing recorded exactly once, production-vault watch unchanged. Measured at 4d5e849; the subsequent change handles unsupported command names and is covered by the projection regression suite.
  • Claude Code 2.1.269 was invoked but returned an organization subscription-access HTTP 403 before completing a model task. This is an environment failure, not a quality comparison. Successful Claude live validation is explicitly deferred to follow-up work with user approval; it is not a completion gate for this PR. The failed run remains failed, and grading criteria are unchanged.
  • Independent Codex review found issues in schema distribution, timeout cleanup, private plugin state exclusion, directory-symlink monitoring and legacy-name compatibility; each has been fixed with regression coverage. Final read-only re-review at 46ac69c found no clear introduced defects; the reviewer did not execute write-requiring tests.

CI: all four jobs passed at 46ac69c (Ubuntu, macOS, lint, guard).

Scope: PR only; no merge or production-vault update. The small cases establish functional behavior, not statistical model quality. Live runs do not bypass hook trust; exact hook inputs, exit codes and post-write effects are covered separately by deterministic replay tests.

@tarr1124
tarr1124 marked this pull request as ready for review September 14, 2026 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant