From 9f28c7f44053f86a0ae516f447ab85efd745acfb Mon Sep 17 00:00:00 2001 From: Ahmad Ragab Date: Sun, 27 Sep 2026 17:23:07 -0700 Subject: [PATCH 1/8] docs: align release guidance and plugin distribution spec --- .gitignore | 5 +- EXAMPLES.md | 11 +- WALKTHROUGH.md | 11 +- docs/smoke-gates.md | 122 ++++++++++++++++++ .../subscription-backends-audit.md | 2 +- evaluator-cookbook.md | 12 +- examples/README.md | 10 +- .../cross-client-plugin-distribution/spec.md | 7 +- 8 files changed, 163 insertions(+), 17 deletions(-) create mode 100644 docs/smoke-gates.md diff --git a/.gitignore b/.gitignore index 55bcd75..10687e7 100644 --- a/.gitignore +++ b/.gitignore @@ -19,7 +19,8 @@ node_modules/ # miscellaneous files seed.txt docs/* -# Exception: allow verification evidence to be committed +# Exceptions: track active smoke instructions and verification evidence +!docs/smoke-gates.md !docs/verification/ smoke_outputs/ artifacts/ @@ -34,3 +35,5 @@ ROADMAP* runs/ .codegraph/ todos/ +.maestro +.cursor/ diff --git a/EXAMPLES.md b/EXAMPLES.md index d803835..be5df36 100644 --- a/EXAMPLES.md +++ b/EXAMPLES.md @@ -1,10 +1,14 @@ # EXAMPLES -Worked examples for optimize-anything v2. +Illustrative CLI examples for optimize-anything v2. Replace seed, evaluator, +and dataset paths with your files. These commands assume the global CLI installer +from [install.md](install.md); use `uv run optimize-anything` from a source +checkout. The shown API models can incur provider charges. `--budget` counts +evaluator calls, not dollars, and an iteration can exceed the requested count. ## Result JSON shape (current contract) -Optimization output examples should follow this structure: +An abbreviated optimization output has this structure: ```json { @@ -165,7 +169,8 @@ optimize-anything optimize strategy.md \ --evaluator-command bash eval_unbounded.sh \ --model openai/gpt-5.6-sol \ --objective "Maximize reward" \ - --score-range any + --score-range any \ + --budget 20 ``` Use when your evaluator emits finite scores outside `[0,1]`. diff --git a/WALKTHROUGH.md b/WALKTHROUGH.md index 395e01a..b3071fe 100644 --- a/WALKTHROUGH.md +++ b/WALKTHROUGH.md @@ -6,7 +6,12 @@ Step-by-step v2 workflow for optimize-anything. - Python 3.10+ - `uv` -- API key(s) for any LLM providers you plan to use +- API key(s) for the models in this walkthrough + +These commands use API models and can incur provider charges. `--budget` limits +evaluator calls approximately, not dollars; GEPA may finish an iteration after +the limit is reached. For local subscription backends and billed fallback +controls, see [install.md](install.md). ## Step 1: Install @@ -49,8 +54,8 @@ chmod +x evaluators/eval.sh ## Step 4: Test evaluator contract ```bash -echo '{"_protocol_version":2,"candidate":"test"}' | python evaluators/eval.py -# expected: JSON with required "score" +echo '{"_protocol_version":2,"candidate":"test"}' | uv run python evaluators/eval.py +# expected: JSON with required "score"; the default judge makes an API call ``` ## Step 5: Baseline score diff --git a/docs/smoke-gates.md b/docs/smoke-gates.md new file mode 100644 index 0000000..eeee2e8 --- /dev/null +++ b/docs/smoke-gates.md @@ -0,0 +1,122 @@ +# Smoke Gates + +Repeatable smoke checks for `RB-009` and `RB-010`. + +## Prerequisites +- Dependencies installed: `uv sync` +- `OPENAI_API_KEY` for the default `openai/gpt-5.6-sol` proposer, or set `OPTIMIZE_ANYTHING_MODEL` and provide that model's API credential + +## Run Smoke Harness (RB-009) + +One command: + +```bash +uv run python scripts/smoke_harness.py --budget 1 +``` + +What it does: +- Creates a temporary seed file and temporary evaluator command script. +- Runs CLI optimize smoke (`optimize_anything.cli` path) with a temporary command evaluator. +- Asserts that the CLI summary has a non-empty `best_artifact`, positive `total_metric_calls`, `top_diagnostics`, and `score_summary`. +- Runs `generate-evaluator` and checks that it emits a script with a shebang. +- Runs `intake` and checks that its JSON includes `execution_mode`. +- Saves logs/artifacts under `smoke_outputs/smoke-/` by default. +- Exits non-zero on any assertion failure. + +Use a custom output directory: + +```bash +uv run python scripts/smoke_harness.py --budget 1 --output-dir /tmp/opt-anything-smoke +``` + +## Run Consecutive Smoke Gate (RB-010) + +One command: + +```bash +uv run python scripts/consecutive_smoke_gate.py --budget 1 +``` + +What it does: +- Runs `scripts/smoke_harness.py` twice consecutively. +- Prints concise summary fields: + - `pass1` + - `pass2` + - `overall` +- Exits non-zero if either pass fails. +- Saves gate logs and per-pass outputs under `smoke_outputs/consecutive-/` by default. + +Use a custom output directory: + +```bash +uv run python scripts/consecutive_smoke_gate.py --budget 1 --output-dir /tmp/opt-anything-consecutive +``` + +## Subscription Live Gates (opt-in) + +These gates exercise the Codex and Claude subscription backends against real +local logins. They are marked `pytest.mark.integration` and skip unless their +environment variable is set to `1`, so default CI and `uv run pytest` never +spend subscription quota or bill an API account. Run them on macOS with a saved +Codex login (`uv sync --extra codex`) and Claude Code 2.1.278 or newer. + +| Gate | Opt-in variable | Spends | Test file | +|---|---|---|---| +| Subscription live (plan steps 8-10) | `OPTIMIZE_ANYTHING_RUN_SUBSCRIPTION_LIVE=1` | Local Codex / Claude subscription quota | `tests/test_subscription_live.py` | +| Paid API fallback (plan step 11) | `OPTIMIZE_ANYTHING_RUN_PAID_FALLBACK_LIVE=1` | OpenAI and Anthropic API accounts (billed) | `tests/test_api_fallback_live.py` | + +### Subscription live gate + +Invalid API-key sentinels prove the run cannot silently fall back to a paid +key. Run each provider separately: + +```bash +OPTIMIZE_ANYTHING_RUN_SUBSCRIPTION_LIVE=1 OPENAI_API_KEY=deliberately-invalid-live-gate ANTHROPIC_API_KEY=deliberately-invalid-live-gate uv run pytest tests/test_subscription_live.py -k codex -v -s +OPTIMIZE_ANYTHING_RUN_SUBSCRIPTION_LIVE=1 OPENAI_API_KEY=deliberately-invalid-live-gate ANTHROPIC_API_KEY=deliberately-invalid-live-gate uv run pytest tests/test_subscription_live.py -k claude -v -s +``` + +Each covers structured completion, a seedless budget-1 proposer optimize, and a +generated judge evaluator. After both pass, run one built-in judge canary per +provider: + +```bash +OPENAI_API_KEY=deliberately-invalid-live-gate ANTHROPIC_API_KEY=deliberately-invalid-live-gate uv run optimize-anything score examples/seeds/sample_seed.txt --judge-backend codex --no-api-fallback --objective "Score clarity" +OPENAI_API_KEY=deliberately-invalid-live-gate ANTHROPIC_API_KEY=deliberately-invalid-live-gate uv run optimize-anything score examples/seeds/sample_seed.txt --judge-backend claude --no-api-fallback --objective "Score clarity" +``` + +The score JSON must carry `llm_provenance` with `actual_backend` matching the +provider, `auth_class=subscription`, and no `fallback_source` / +`fallback_reason` keys (those appear only when API fallback ran). + +### Paid API fallback gate + +This gate bills real API calls. A fake subscription adapter forces an eligible +failure so the real `FallbackBackend` and `LiteLLMBackend` run. Keep +`ANTHROPIC_API_KEY` out of your shell and pass it through an env file (outside +the repo, mode 600). Unset `ANTHROPIC_BASE_URL` so LiteLLM reaches +`https://api.anthropic.com` directly instead of a local proxy: + +```bash +env -u ANTHROPIC_BASE_URL OPTIMIZE_ANYTHING_RUN_PAID_FALLBACK_LIVE=1 \ + uv run --env-file "$HOME/.config/optimize-anything/paid-fallback.env" \ + pytest tests/test_api_fallback_live.py -v -s +``` + +It asserts same-vendor fallback, the billing warning before dispatch, the sticky +per-role circuit, and zero API calls under `--no-api-fallback`. + +### Evidence to record + +Every live run records, without account identity, secrets, or artifact prompts: + +- Versions: uv, Python, `openai-codex`, Codex CLI, Claude Code CLI (and pytest / litellm for the paid gate) +- OS and architecture +- Auth class and auth source per provider +- Requested and actual backend and model +- Whether fallback ran (`fallback_source` / `fallback_reason`) and `retry_count`; for the paid gate, fallback model, billed call count, and token usage +- Isolation assertions: no secret or identity leakage in retained files, coordination state, or cache keys; Claude child with tools and MCP disabled and a scrubbed environment +- Key scan result over logs and the report +- Pass/fail per gate + +Latest evidence: [[Subscription-Live-Evidence-2026-09-26]] +(`docs/verification/subscription-live-evidence-2026-09-26.md`). diff --git a/docs/verification/subscription-backends-audit.md b/docs/verification/subscription-backends-audit.md index 8225a08..766b5d6 100644 --- a/docs/verification/subscription-backends-audit.md +++ b/docs/verification/subscription-backends-audit.md @@ -20,7 +20,7 @@ This report audits units U1-U7 of `docs/plans/2026-09-22-1841-feature-subscripti - **Initial audited commit:** `0b0a99562e657d173b32c94f1f27062669e7b794` on `feat/codex-claude-subscription`. - `src/` is identical to `origin/main` at `70e1fdf2905a1f44b5b447eaa027f5ecfb6dbcef`, because PR #6 merged the backends. - - PR #7 is still open: https://github.com/ASRagab/optimize-anything/pull/7 + - PR #7 was open during the initial audit and merged on 2026-09-27: https://github.com/ASRagab/optimize-anything/pull/7 - **Post-fix baseline:** `279f022`; the final offline gates below ran on this code before the audit report update. - **Live evidence:** Verification Contract steps 8-11 (the opt-in subscription and paid-fallback gates) are recorded in `docs/verification/subscription-live-evidence-2026-09-26.md`. - This audit covers code and offline tests only. diff --git a/evaluator-cookbook.md b/evaluator-cookbook.md index 7b1d50b..b70d7ee 100644 --- a/evaluator-cookbook.md +++ b/evaluator-cookbook.md @@ -2,6 +2,13 @@ An evaluator is a function that **scores a candidate artifact**. `optimize-anything` supports three types: **shell commands**, **HTTP endpoints**, and **LLM judges**. +Commands below run from a source checkout with `uv run`. Optimization examples +use the default API proposer unless `--proposer-backend` is set; this and API +judges can incur provider charges. `--budget` counts evaluator calls, not +dollars, and an iteration can exceed the requested count. See +[install.md](install.md) for local subscription backends and billed fallback +controls. + --- ## 1. Evaluator Contract @@ -159,7 +166,8 @@ if __name__ == "__main__": **Run it:** ```bash python3 evaluators/json_validator.py # Terminal 1 -uv run optimize-anything optimize seed.json \ # Terminal 2 +# Terminal 2 +uv run optimize-anything optimize seed.json \ --evaluator-url "http://localhost:3456" \ --objective "Generate a valid package.json" --budget 10 ``` @@ -536,7 +544,7 @@ Outcome: The optimizer always receives a valid score, even when your evaluator e 1. **Start simple.** Create a minimal evaluator that returns a constant score first to verify JSON I/O works, then add real logic incrementally. 2. **Always return rich diagnostics.** Include sub-scores (`length`, `violations`), booleans (`hasIntro`), and `improvementSuggestions`. This produces more targeted mutations because gepa's reflection LM uses these fields to guide the next proposal. 3. **Test evaluators independently** before plugging into optimize-anything. Run `echo '{"candidate":"test"}' | bash evaluators/your-evaluator.sh` to verify JSON parsing and catch logic errors early. -4. **Set `--budget`** to cap evaluator calls. You should never run without a budget — this prevents unexpectedly long or expensive runs. +4. **Set `--budget`** to limit evaluator calls. It is not a dollar cap, and GEPA may finish an iteration after the limit is reached. Choose a small initial budget and monitor provider usage. 5. **Favor determinism.** Avoid random seeds and time-based behavior — same input must produce same score for reproducible, debuggable runs. 6. **Use `--score-range any` intentionally.** The default `unit` range catches accidental out-of-bounds scores. Only switch to `any` when your metric genuinely lives outside `[0,1]`. 7. **Read `example` defensively.** Always guard `example = payload.get("example", {})` — the field is absent when not in dataset mode. Your evaluator must handle both cases. diff --git a/examples/README.md b/examples/README.md index 1801605..702fba3 100644 --- a/examples/README.md +++ b/examples/README.md @@ -2,15 +2,17 @@ ## Evaluators -- `evaluators/echo_score.sh` -- Simple bash evaluator (scores by length) -- `evaluators/http_evaluator.py` -- HTTP evaluator server example +- `examples/evaluators/echo_score.sh` -- Simple bash evaluator (scores by length) +- `examples/evaluators/http_evaluator.py` -- HTTP evaluator server example ## Seeds -- `seeds/sample_seed.txt` -- Example prompt seed for optimization +- `examples/seeds/sample_seed.txt` -- Example prompt seed for optimization ## Quick Test +From the repository root: + ```bash -echo '{"candidate":"test"}' | bash evaluators/echo_score.sh +echo '{"candidate":"test"}' | bash examples/evaluators/echo_score.sh ``` diff --git a/openspec/specs/cross-client-plugin-distribution/spec.md b/openspec/specs/cross-client-plugin-distribution/spec.md index 4cb7ea9..4b1e8c6 100644 --- a/openspec/specs/cross-client-plugin-distribution/spec.md +++ b/openspec/specs/cross-client-plugin-distribution/spec.md @@ -66,12 +66,13 @@ User-facing documentation SHALL provide verified Claude Code and Codex installat - **AND** does not imply that plugin metadata or skills are installed with the CLI ### Requirement: Release versions remain aligned -Release metadata SHALL keep the Python package, Claude plugin, Claude marketplace entry, Codex plugin, and Codex marketplace entry on the same release version. +Release metadata SHALL keep the Python package, Claude plugin manifest, active Claude marketplace version fields, and Codex plugin manifest on the same release version. The Codex repository marketplace entry identifies the plugin source and has no version field. #### Scenario: Release contract test runs - **WHEN** release metadata tests inspect all package and plugin manifests -- **THEN** every active version field has the same value -- **AND** a mismatch fails the test with the differing sources identified +- **THEN** every active version field in the Python package and plugin manifests has the same value +- **AND** the Codex marketplace source resolves to the repository plugin root +- **AND** a version mismatch fails the test with the differing sources identified ### Requirement: Distribution has offline and optional live verification The repository SHALL provide offline contract checks for both plugin packages and SHALL retain optional live host scenarios for confirming skill discovery, runtime launch, inline prompt return, and repository prompt application. From 546e382c0aa6eb7ed7ab3f169ed76a1dfd68dfa1 Mon Sep 17 00:00:00 2001 From: Ahmad Ragab Date: Sun, 27 Sep 2026 17:32:40 -0700 Subject: [PATCH 2/8] fix: bundle compatible Codex SDK for plugin installs --- install.md | 46 +++++++++---- install.sh | 20 +++++- pyproject.toml | 2 +- scripts/run-optimize-anything | 2 +- skills/generate-evaluator/SKILL.md | 26 ++++++-- skills/optimization-guide/SKILL.md | 97 +++++++++++++--------------- skills/optimize-prompt/SKILL.md | 5 +- tests/test_plugin_launcher.py | 50 ++++++++++++++ tests/test_prompt_plugin_contract.py | 18 ++++++ uv.lock | 2 +- 10 files changed, 194 insertions(+), 74 deletions(-) diff --git a/install.md b/install.md index cf1d0b2..a2cc9a8 100644 --- a/install.md +++ b/install.md @@ -2,16 +2,24 @@ ## Optional local subscription backends -The default install keeps LiteLLM/API behavior. To reuse a Codex login made -through ChatGPT, install the pinned optional SDK and log in with the provider's -CLI: +The plugin launcher selects the locked `codex` SDK extra automatically. To +reuse a Codex login made through ChatGPT, authenticate with the provider CLI: ```bash -uv sync --extra codex codex login codex login status ``` +For a source checkout, install the optional SDK in that project environment: + +```bash +uv sync --extra codex +``` + +For a global CLI install, pass `--codex` to `install.sh`; this installs SDK +version 0.156.0 required by the adapter in the isolated uv tool environment. +The default global install keeps LiteLLM/API behavior. + Claude subscription support uses the locally installed `claude` executable; install Claude Code and run `claude auth login`. The adapter requires Claude Code 2.1.278 or newer and verifies `claude.ai` first-party authentication. @@ -26,20 +34,21 @@ Choose the host integration or standalone runtime you need: | Method | Skills | Slash commands | Runtime | Prerequisites | |---|---|---|---|---| | **Claude Code plugin** | Yes | Yes | Bundled, locked project | uv, Python >= 3.10 | -| **Codex plugin** | Yes | No | Bundled, locked project | uv, Python >= 3.10 | +| **Codex plugin** | Yes | No | Bundled, locked project with `codex` extra | uv, Python >= 3.10; `codex login` for subscription use | | **CLI installer** | No | No | Global `optimize-anything` | None; installer adds uv | | **From source** | Local checkout | No | `uv run` | uv, Python >= 3.10 | The Claude Code plugin and Codex plugin discover the same `skills/` tree. Their -launcher runs the repository project directly, so neither plugin requires a -separately installed global CLI. The CLI installer does not install either -plugin. +launcher runs the repository project with the locked `codex` extra, so neither +plugin requires a separately installed global CLI. Generated evaluator child +processes use that same project environment. The CLI installer does not install +either plugin. ### Supported versions | Component | Requirement | Tested (2026-09-26) | |---|---|---| -| `openai-codex` Python SDK | `>=0.156.0,<0.157.0` (the `codex` extra); the adapter refuses any other SDK version | 0.156.0 | +| `openai-codex` Python SDK | Exactly 0.156.0 (the `codex` extra and adapter) | 0.156.0 | | Codex CLI | Any CLI that can `codex login` with ChatGPT | 0.155.1 | | Claude Code (`claude`) | 2.1.278 or newer, `claude.ai` first-party auth | 2.1.283 | | Platform | macOS, local machine only | macOS 26.6.2 arm64, Python 3.12.14 | @@ -130,7 +139,8 @@ subprocesses. Any other value prints a warning such as - Return to API defaults by omitting `--proposer-backend`, `--judge-backend`, and `--analysis-backend` (or passing `api`), and removing `backend` entries from `[model.proposer]` / `[model.judge]` tables in TOML spec files. -- Drop the Codex SDK with a plain `uv sync` (without `--extra codex`). +- In a source checkout, drop the Codex SDK with a plain `uv sync` (without + `--extra codex`). The plugin launcher reinstalls it on the next invocation. - Remove the plugins with the `claude plugin uninstall` and `codex plugin remove` commands below. Provider logins belong to the provider CLIs; use `codex logout` or `claude auth logout` if you also want to sign @@ -181,7 +191,10 @@ codex plugin add optimize-anything@optimize-anything Start a new Codex thread, then invoke `$optimize-anything:optimize-prompt`. Codex namespaces plugin skills and discovers the canonical `skills/` tree -through `.codex-plugin/plugin.json`. +through `.codex-plugin/plugin.json`. The launcher installs the locked Codex SDK +extra on first use. Run `codex login` before choosing a Codex subscription +backend. Use `--no-api-fallback` to make missing auth or SDK failures terminal +without a billed API request. Update the Git marketplace and reinstall the plugin, or remove it: @@ -206,6 +219,13 @@ optimize-anything --help The installer places the command in `~/.local/bin/`. If that directory is not on `PATH`, add it in the shell that will run the CLI. +To include Codex subscription support in the CLI tool environment: + +```bash +curl -fsSL https://raw.githubusercontent.com/ASRagab/optimize-anything/main/install.sh | bash -s -- --codex +codex login +``` + Remove it with: ```bash @@ -224,7 +244,7 @@ uv run optimize-anything --help The plugin-equivalent launcher is available at `scripts/run-optimize-anything`. It resolves this checkout, verifies the -prerequisites, and runs `uv run --project --locked`. +prerequisites, and runs `uv run --project --locked --no-dev --extra codex`. ## Verify Prompt Optimization @@ -251,6 +271,8 @@ controls; see the prompt workflow in [README.md](README.md). |---|---|---| | `uv: command not found` | Plugin runtime prerequisite missing | Install uv from https://docs.astral.sh/uv/ | | `Python 3.10 or newer is required` | No supported interpreter is available | Run `uv python install 3.10` | +| `Install the Codex backend` or `Codex SDK 0.156.0 is required` | Source or global CLI lacks the Codex extra, or has the wrong SDK version | Run `uv sync --extra codex` from source or reinstall the global CLI with `install.sh --codex` | +| Codex authentication failure | No usable ChatGPT login is saved | Run `codex login`, then `codex login status`; add `--no-api-fallback` when API billing is not acceptable | | Model credential error | Selected proposer, task, or judge model is not authenticated | Export that provider's credential before launching the host | | Skill is not visible | Host loaded the prior plugin snapshot | Reinstall/update, then start a new thread or session | | Evaluator command fails | Script path or working directory is wrong | Set `--evaluator-cwd` and run the evaluator preflight payload manually | diff --git a/install.sh b/install.sh index bc0dec2..5ec0ca9 100755 --- a/install.sh +++ b/install.sh @@ -3,6 +3,8 @@ # # Install: # curl -fsSL https://raw.githubusercontent.com/ASRagab/optimize-anything/main/install.sh | bash +# Codex subscription support: +# curl -fsSL https://raw.githubusercontent.com/ASRagab/optimize-anything/main/install.sh | bash -s -- --codex # # Uninstall: # curl -fsSL https://raw.githubusercontent.com/ASRagab/optimize-anything/main/install.sh | bash -s -- --uninstall @@ -15,12 +17,15 @@ # Environment variables: # OPTIMIZE_ANYTHING_REPO - Git URL or local path to install from (default: GitHub repo) # OPTIMIZE_ANYTHING_REF - Git ref to install (default: main) +# +# --codex installs the pinned Codex SDK in the isolated tool environment. set -euo pipefail # --- Configuration --- REPO="${OPTIMIZE_ANYTHING_REPO:-git+https://github.com/ASRagab/optimize-anything}" REF="${OPTIMIZE_ANYTHING_REF:-main}" +CODEX_EXTRA=0 # Local paths don't use @ref syntax if [[ "${REPO}" == /* ]] || [[ "${REPO}" == ./* ]]; then @@ -92,14 +97,18 @@ install_uv() { install_tool() { info "Installing optimize-anything from ${INSTALL_SOURCE}..." - if uv tool install "${INSTALL_SOURCE}" --force 2>&1; then + local install_args=() + if [ "$CODEX_EXTRA" -eq 1 ]; then + install_args+=(--with 'openai-codex==0.156.0') + fi + if uv tool install "${INSTALL_SOURCE}" "${install_args[@]}" --force 2>&1; then info "optimize-anything installed successfully" else error "Installation failed" echo "" echo "Troubleshooting:" echo " - Check the repo URL: ${REPO}" - echo " - Try installing manually: uv tool install '${INSTALL_SOURCE}'" + echo " - Try installing manually: uv tool install '${INSTALL_SOURCE}' ${install_args[*]}" echo " - For local installs: uv tool install /path/to/optimize-anything" exit 1 fi @@ -134,11 +143,16 @@ verify() { # --- Main --- main() { - # Check for --uninstall flag + # Check for --uninstall and optional Codex support. for arg in "$@"; do if [ "$arg" = "--uninstall" ] || [ "$arg" = "uninstall" ]; then uninstall exit 0 + elif [ "$arg" = "--codex" ]; then + CODEX_EXTRA=1 + else + error "Unknown option: $arg" + exit 2 fi done diff --git a/pyproject.toml b/pyproject.toml index c78d7b1..ebef964 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,7 +20,7 @@ dependencies = [ ] [project.optional-dependencies] -codex = ["openai-codex>=0.156.0,<0.157.0"] +codex = ["openai-codex==0.156.0"] [project.urls] Homepage = "https://github.com/ASRagab/optimize-anything" diff --git a/scripts/run-optimize-anything b/scripts/run-optimize-anything index 9573611..2d61a08 100755 --- a/scripts/run-optimize-anything +++ b/scripts/run-optimize-anything @@ -13,4 +13,4 @@ if ! uv python find '>=3.10' >/dev/null 2>&1; then exit 1 fi -exec uv run --project "$OPTIMIZE_ANYTHING_PLUGIN_ROOT" --locked optimize-anything "$@" +exec uv run --project "$OPTIMIZE_ANYTHING_PLUGIN_ROOT" --locked --no-dev --extra codex optimize-anything "$@" diff --git a/skills/generate-evaluator/SKILL.md b/skills/generate-evaluator/SKILL.md index 0ccb454..5bf8d9d 100644 --- a/skills/generate-evaluator/SKILL.md +++ b/skills/generate-evaluator/SKILL.md @@ -9,6 +9,22 @@ description: >- ## Objective Generate an evaluator that scores candidate artifacts for optimization with gepa. Include diagnostic feedback so reflections can improve weak dimensions. +## Resolve the bundled runtime + +Locate this `SKILL.md` in the installed plugin, then set: + +```bash +GENERATE_EVALUATOR_SKILL_DIR="/absolute/path/to/skills/generate-evaluator" +OPTIMIZE_ANYTHING_ROOT="$(cd "$GENERATE_EVALUATOR_SKILL_DIR/../.." && pwd)" +OPTIMIZE_ANYTHING_RUNNER="$OPTIMIZE_ANYTHING_ROOT/scripts/run-optimize-anything" +``` + +Use `$OPTIMIZE_ANYTHING_RUNNER` for every CLI call. The plugin does not require a +global `optimize-anything` command. For Codex subscription judging, run +`codex login` first; the launcher installs the locked `codex` SDK extra. In a +source checkout use `uv sync --extra codex`; a global CLI needs the SDK in its +tool environment. Use `--no-api-fallback` to prevent billed API fallback. + ## Evaluator Contract - Input JSON on stdin (`--evaluator-command`) or HTTP POST body (`--evaluator-url`) - Default payload: `{"candidate": ""}` @@ -63,12 +79,13 @@ Generate a judge evaluator and test it: ```bash # Generate -optimize-anything generate-evaluator seed.txt \ +"$OPTIMIZE_ANYTHING_RUNNER" generate-evaluator seed.txt \ --objective "Score clarity and specificity" \ --model openai/gpt-5.6-luna > eval_judge.py # Test it -echo '{"candidate":"Your artifact text here"}' | python3 eval_judge.py +echo '{"candidate":"Your artifact text here"}' | \ + uv run --project "$OPTIMIZE_ANYTHING_ROOT" --locked --no-dev --extra codex python eval_judge.py ``` This returns JSON like: @@ -80,11 +97,12 @@ This returns JSON like: For dataset-aware evaluators: ```bash -optimize-anything generate-evaluator seed.txt \ +"$OPTIMIZE_ANYTHING_RUNNER" generate-evaluator seed.txt \ --objective "Score correctness" \ --dataset examples.jsonl > eval_dataset.py -echo '{"candidate":"text","example":{"input":"q","expected":"a"}}' | python3 eval_dataset.py +echo '{"candidate":"text","example":{"input":"q","expected":"a"}}' | \ + uv run --project "$OPTIMIZE_ANYTHING_ROOT" --locked --no-dev --extra codex python eval_dataset.py ``` ## Workflow diff --git a/skills/optimization-guide/SKILL.md b/skills/optimization-guide/SKILL.md index c4e5edb..a6e42b6 100644 --- a/skills/optimization-guide/SKILL.md +++ b/skills/optimization-guide/SKILL.md @@ -5,17 +5,30 @@ description: >- optimization workflows. Use when asked how to optimize a prompt, artifact, config, or skill, or when troubleshooting evaluator feedback, budget, or score interpretation. --- -End-to-end guide for optimizing text artifacts with `optimize-anything` and `gepa`. +## Resolve the bundled runtime + +Locate this installed `SKILL.md`, then set: + +```bash +OPTIMIZATION_GUIDE_SKILL_DIR="/absolute/path/to/skills/optimization-guide" +OPTIMIZE_ANYTHING_ROOT="$(cd "$OPTIMIZATION_GUIDE_SKILL_DIR/../.." && pwd)" +OPTIMIZE_ANYTHING_RUNNER="$OPTIMIZE_ANYTHING_ROOT/scripts/run-optimize-anything" +``` + +Use the runner for every CLI call; no global command is needed. It selects the +locked Codex SDK. For Codex subscription use, run `codex login`; source installs +need `uv sync --extra codex`, and global installs need `install.sh --codex`. +Pass `--no-api-fallback` to prevent billed fallback. ## Workflow ### 1. Prepare the Seed -Start with your current best version of the artifact. `gepa` evolves from here. +Start with your best artifact. 1. Set `objective` if you have no seed, and let `gepa` bootstrap one from the description. -2. Use a dict like `{"system_prompt": "...", "examples": "..."}` for multi-component artifacts (e.g., `system_prompt` + few-shot examples). +2. Use a dict like `{"system_prompt": "...", "examples": "..."}` for multi-component artifacts. ### 2. Create an Evaluator -Use the **generate-evaluator** skill to create one matched to your objective. The evaluator is the most critical piece—`gepa`'s optimization quality is bounded by your evaluator's feedback quality. +Use the **generate-evaluator** skill to create one matched to your objective. ### 2b. Choose Your Evaluator Interface @@ -25,7 +38,7 @@ Use the **generate-evaluator** skill to create one matched to your objective. Th Prefer the Python API. For command templates, use the **generate-evaluator** and **evaluator-patterns** skills. -**Command preflight and timeouts:** Before optimization, the CLI sends `{"_protocol_version":2,"candidate":"__optimize_anything_preflight__"}` and waits 10 seconds. Detect that sentinel and immediately return `{"score":0.5}`. Normal command evaluations time out after 30 seconds; use the Python API for slower work. +**Command preflight:** The CLI sends `{"candidate":"__optimize_anything_preflight__"}` with protocol version 2. Return `{"score":0.5}` within 10 seconds. Normal calls time out after 30 seconds. ### 3. Choose Optimization Mode @@ -35,14 +48,9 @@ Prefer the Python API. For command templates, use the **generate-evaluator** and ### 4. Set Budget and Configuration -Use the `budget` subcommand for a starting point, then adjust: - -| Seed length | Recommended budget | Rationale | -|---|---|---| -| < 100 chars | 50 | Short artifact, fewer mutations needed | -| 100-499 | 100 | Moderate exploration | -| 500-1999 | 200 | More search space to cover | -| 2000+ | 300 | Extensive exploration recommended | +Run `"$OPTIMIZE_ANYTHING_RUNNER" budget seed.txt` for a starting budget. +Current defaults by seed length are 50 (<100 chars), 100 (<500), 200 (<2000), +and 300 otherwise. Adjust for cost and evaluator quality. Configure options via `GEPAConfig`: ```python @@ -58,10 +66,8 @@ config = GEPAConfig( ) ``` -For direct API fan-out, import `SameParentSampling` from -`gepa.strategies.proposal_sampling` and pass -`sampling_strategy=SameParentSampling(n=3)`. Omit the strategy for the default -single proposal. +For API fan-out, pass `SameParentSampling(n=3)` from +`gepa.strategies.proposal_sampling` as `sampling_strategy`. ### 5. Run Optimization @@ -70,20 +76,19 @@ logged-in subscription explicitly: ```bash # Codex host -optimize-anything optimize seed.txt --proposer-backend codex --judge-backend codex ... +"$OPTIMIZE_ANYTHING_RUNNER" optimize seed.txt --proposer-backend codex --judge-backend codex ... # Claude Code host -optimize-anything optimize seed.txt --proposer-backend claude --judge-backend claude ... +"$OPTIMIZE_ANYTHING_RUNNER" optimize seed.txt --proposer-backend claude --judge-backend claude ... ``` -Subscription calls default to one concurrent call per provider. Announce the -backend and possible billed API fallback before execution; add -`--no-api-fallback` to prohibit it. Unknown hosts omit backend flags and keep -the API defaults. +Subscription calls default to one per provider. Announce the backend and +possible billed API fallback; use `--no-api-fallback` to prohibit it. Unknown +hosts omit backend flags and keep API defaults. **Via CLI:** ```bash -optimize-anything optimize seed.txt --evaluator-command bash evaluators/eval.sh --budget 100 --objective "maximize clarity" -o result.txt +"$OPTIMIZE_ANYTHING_RUNNER" optimize seed.txt --evaluator-command bash evaluators/eval.sh --budget 100 --objective "maximize clarity" -o result.txt ``` To request multiple mutations from the selected parent, add @@ -94,11 +99,9 @@ To request multiple mutations from the selected parent, add proposals_per_iteration = 3 ``` -An explicit CLI value overrides the spec. Proposal fan-out is independent of -`--workers`, `--parallel`, and `--no-parallel`, which control evaluator-call -concurrency. Fan-out increases reflection and evaluation work, small datasets -may reuse a minibatch across proposals, and the final iteration can exceed -`--budget` because GEPA checks the limit between iterations. +The CLI value overrides the spec. Fan-out adds proposal and evaluation work; +`--workers` and `--parallel` govern evaluator concurrency. The final iteration +can exceed `--budget` because GEPA checks the limit between iterations. **Via Python API:** ```python @@ -121,7 +124,7 @@ print(result.best_candidate) Use plateau-based early stopping to avoid wasting budget after convergence: ```bash -optimize-anything optimize seed.txt \ +"$OPTIMIZE_ANYTHING_RUNNER" optimize seed.txt \ --evaluator-command bash evaluators/eval.sh \ --budget 120 \ --early-stop \ @@ -130,14 +133,14 @@ optimize-anything optimize seed.txt \ ``` Notes: -1. `--early-stop` is auto-enabled when `--budget > 30`. -2. Tune `--early-stop-window` and `--early-stop-threshold` for noisier evaluators. -3. CLI output includes `early_stopped` and `stopped_at_iteration` when a run exits early. +1. `--early-stop` activates automatically above budget 30. +2. Tune window and threshold for noisy evaluators. +3. Check `early_stopped` and `stopped_at_iteration` in the result. For cache reuse across runs, copy prior disk cache entries into a new run directory: ```bash -optimize-anything optimize seed.txt \ +"$OPTIMIZE_ANYTHING_RUNNER" optimize seed.txt \ --evaluator-command bash evaluators/eval.sh \ --run-dir runs \ --cache \ @@ -146,8 +149,8 @@ optimize-anything optimize seed.txt \ Notes: 1. `--cache-from` requires `--cache` and `--run-dir`. -2. `--cache-from` copies `fitness_cache/` from the previous run before optimization starts. -3. GEPA 0.1.4 can migrate older run state forward, but its state is not expected to load under GEPA 0.1.1 after a rollback. +2. It copies `fitness_cache/` from the prior run before optimization. +3. GEPA 0.1.4 can migrate older state forward; rollback to 0.1.1 may not load it. ### 7. Interpret Results @@ -156,21 +159,13 @@ Expected output: 2. Review `val_aggregate_scores` — score progression across iterations. 3. Check `total_metric_calls` — how many evaluator invocations were used. -**Signs of a good run:** -1. You should see scores trend upward over iterations. -2. Compare `total_metric_calls` with `budget`; the final iteration can overshoot the budget, especially with proposal fan-out. -3. Compare `best_candidate` against `seed.txt` or in-memory seed to see targeted differences. - -**Signs of problems:** -1. Detect flat scores from start — evaluator may not be discriminating enough. -2. Notice oscillating scores — evaluator may be noisy or non-deterministic. -3. Investigate runs where best score barely beats `seed` — add richer feedback, increase `budget`, or refine `objective`. +Compare the best candidate with the seed and check that scores rise. Flat scores +suggest weak discrimination; oscillation suggests noise. If improvement is +small, refine the objective or evaluator before increasing budget. ## Tips -1. Start small: Run with `budget` 20-50 first to validate your evaluator on `seed.txt` and confirm that scores change meaningfully. -2. Provide rich feedback: Include sub-scores, error messages, and specific improvement hints in evaluator output — this drives `gepa`'s reflection. -3. Clarify the objective: Set the `objective` string that is injected into `gepa`'s reflection prompt and specify constraints like token limits or format requirements. -4. Add background context: Use `background` for domain knowledge, constraints, or strategies such as "Target audience is non-technical users. Never use jargon." -5. Iterate on the evaluator: Improve the evaluator before increasing `budget` if optimization results on `seed.txt` are poor. -6. Set evaluator working directory: Pass `evaluator_cwd` as an absolute project path next to `seed.txt` and `evaluators/eval.sh` when `evaluators/eval.sh` or other evaluator commands use repo-relative files or scripts. +1. Start with budget 20-50 to check evaluator discrimination. +2. Return sub-scores, errors, and specific improvement hints to guide reflection. +3. Put constraints and audience in `objective` or `background`. +4. Pass `--evaluator-cwd` when evaluator scripts use relative paths. diff --git a/skills/optimize-prompt/SKILL.md b/skills/optimize-prompt/SKILL.md index 8fadeff..12a103a 100644 --- a/skills/optimize-prompt/SKILL.md +++ b/skills/optimize-prompt/SKILL.md @@ -20,7 +20,10 @@ OPTIMIZE_ANYTHING_RUNNER="$OPTIMIZE_ANYTHING_ROOT/scripts/run-optimize-anything" Invoke every CLI subcommand through `$OPTIMIZE_ANYTHING_RUNNER`. Do not assume a global `optimize-anything` executable exists. The launcher requires `uv` and -Python 3.10 or newer; model-backed modes also require the relevant credentials. +Python 3.10 or newer and installs the locked `codex` SDK extra. Codex +subscription mode also requires `codex login`; Claude Code subscription mode +requires `claude auth login`. Pass `--no-api-fallback` to keep subscription +failures from switching to a billed API call. ## 1. Capture the source and destination diff --git a/tests/test_plugin_launcher.py b/tests/test_plugin_launcher.py index 309c87b..52df4ba 100644 --- a/tests/test_plugin_launcher.py +++ b/tests/test_plugin_launcher.py @@ -7,9 +7,19 @@ import subprocess from pathlib import Path +from optimize_anything.llm_backends.codex_backend import _SDK_VERSION + REPO_ROOT = Path(__file__).resolve().parents[1] LAUNCHER = REPO_ROOT / "scripts" / "run-optimize-anything" +INSTALLER = REPO_ROOT / "install.sh" + + +def test_codex_extra_matches_runtime_sdk_version(): + project = (REPO_ROOT / "pyproject.toml").read_text(encoding="utf-8") + installer = INSTALLER.read_text(encoding="utf-8") + assert f'codex = ["openai-codex=={_SDK_VERSION}"]' in project + assert f"openai-codex=={_SDK_VERSION}" in installer def _write_fake_uv(path: Path, body: str) -> None: @@ -47,6 +57,9 @@ def test_launcher_resolves_relocated_root_and_forwards_arguments(tmp_path: Path) "--project", str(relocated), "--locked", + "--no-dev", + "--extra", + "codex", "optimize-anything", "score", "prompt.txt", @@ -84,3 +97,40 @@ def test_launcher_reports_missing_supported_python(tmp_path: Path): ) assert result.returncode == 1 assert "Python 3.10 or newer" in result.stderr + + +def test_global_installer_codex_flag_adds_sdk_to_tool_environment(tmp_path: Path): + bin_dir = tmp_path / "bin" + bin_dir.mkdir() + args_file = tmp_path / "uv-args.txt" + _write_fake_uv( + bin_dir / "uv", + 'if [ "$1" = "--version" ]; then echo "uv 0.0.0"; exit 0; fi\n' + 'printf "%s\\n" "$@" > "$UV_ARGS_FILE"\n', + ) + env = os.environ.copy() + env.update( + { + "HOME": str(tmp_path), + "PATH": f"{bin_dir}:/usr/bin:/bin", + "UV_ARGS_FILE": str(args_file), + "OPTIMIZE_ANYTHING_REPO": str(REPO_ROOT), + } + ) + + result = subprocess.run( + ["/bin/bash", str(INSTALLER), "--codex"], + capture_output=True, + text=True, + env=env, + ) + + assert result.returncode == 0, result.stderr + assert args_file.read_text(encoding="utf-8").splitlines() == [ + "tool", + "install", + str(REPO_ROOT), + "--with", + "openai-codex==0.156.0", + "--force", + ] diff --git a/tests/test_prompt_plugin_contract.py b/tests/test_prompt_plugin_contract.py index d19c711..92a48f8 100644 --- a/tests/test_prompt_plugin_contract.py +++ b/tests/test_prompt_plugin_contract.py @@ -95,6 +95,24 @@ def test_claude_commands_use_the_bundled_launcher(): assert not offenders, f"commands bypass bundled launcher: {offenders}" +def test_shared_skills_resolve_the_bundled_launcher_for_cli_examples(): + expected = { + "generate-evaluator", + "optimization-guide", + "optimize-prompt", + "evaluator-patterns", + } + skill_dirs = {path.name for path in (REPO_ROOT / "skills").iterdir() if path.is_dir()} + assert skill_dirs == expected + for name in expected: + skill = _read(f"skills/{name}/SKILL.md") + assert re.match(rf"^---\nname: {name}\ndescription: .+\n---\n", skill, re.DOTALL) + if re.search(r"(?=4.26,<5" }, { name = "litellm", specifier = "!=1.82.7,!=1.82.8,>=1.83.0,<1.92" }, - { name = "openai-codex", marker = "extra == 'codex'", specifier = ">=0.156.0,<0.157.0" }, + { name = "openai-codex", marker = "extra == 'codex'", specifier = "==0.156.0" }, ] provides-extras = ["codex"] From 003986dd14f34fe1222bbd0549f7759c34a1638d Mon Sep 17 00:00:00 2001 From: Ahmad Ragab Date: Sun, 27 Sep 2026 17:34:27 -0700 Subject: [PATCH 3/8] docs: preserve cookbook score and GEPA contract --- evaluator-cookbook.md | 31 +++++++++++++++--------------- skills/optimization-guide/SKILL.md | 2 +- 2 files changed, 17 insertions(+), 16 deletions(-) diff --git a/evaluator-cookbook.md b/evaluator-cookbook.md index b70d7ee..5469434 100644 --- a/evaluator-cookbook.md +++ b/evaluator-cookbook.md @@ -2,12 +2,9 @@ An evaluator is a function that **scores a candidate artifact**. `optimize-anything` supports three types: **shell commands**, **HTTP endpoints**, and **LLM judges**. -Commands below run from a source checkout with `uv run`. Optimization examples -use the default API proposer unless `--proposer-backend` is set; this and API -judges can incur provider charges. `--budget` counts evaluator calls, not -dollars, and an iteration can exceed the requested count. See -[install.md](install.md) for local subscription backends and billed fallback -controls. +From source, run commands with `uv run`. The default API proposer and API +judges may bill. `--budget` limits evaluator calls, not cost; a final iteration +can exceed it. See [install.md](install.md) for subscription backends and fallback controls. --- @@ -163,10 +160,15 @@ if __name__ == "__main__": server.serve_forever() ``` -**Run it:** +Start the server: + +```bash +python3 evaluators/json_validator.py +``` + +In another terminal, optimize: + ```bash -python3 evaluators/json_validator.py # Terminal 1 -# Terminal 2 uv run optimize-anything optimize seed.json \ --evaluator-url "http://localhost:3456" \ --objective "Generate a valid package.json" --budget 10 @@ -541,11 +543,10 @@ Outcome: The optimizer always receives a valid score, even when your evaluator e ## 12. Tips -1. **Start simple.** Create a minimal evaluator that returns a constant score first to verify JSON I/O works, then add real logic incrementally. -2. **Always return rich diagnostics.** Include sub-scores (`length`, `violations`), booleans (`hasIntro`), and `improvementSuggestions`. This produces more targeted mutations because gepa's reflection LM uses these fields to guide the next proposal. -3. **Test evaluators independently** before plugging into optimize-anything. Run `echo '{"candidate":"test"}' | bash evaluators/your-evaluator.sh` to verify JSON parsing and catch logic errors early. -4. **Set `--budget`** to limit evaluator calls. It is not a dollar cap, and GEPA may finish an iteration after the limit is reached. Choose a small initial budget and monitor provider usage. -5. **Favor determinism.** Avoid random seeds and time-based behavior — same input must produce same score for reproducible, debuggable runs. +1. **Start simple.** Check JSON I/O with a constant score, then add scoring logic. +2. **Return diagnostics.** Include sub-scores and improvement hints to guide GEPA's next proposal. +3. **Test independently.** Run `echo '{"candidate":"test"}' | bash evaluators/your-evaluator.sh` to catch parsing and logic errors. +4. **Set a small `--budget`** and monitor usage. It limits evaluator calls, not dollars, and GEPA may overshoot. +5. **Favor determinism.** The same input should produce the same score. 6. **Use `--score-range any` intentionally.** The default `unit` range catches accidental out-of-bounds scores. Only switch to `any` when your metric genuinely lives outside `[0,1]`. 7. **Read `example` defensively.** Always guard `example = payload.get("example", {})` — the field is absent when not in dataset mode. Your evaluator must handle both cases. -8. **Composite > single-strategy for production.** Heuristic checks are fast and free; LLM calls are slow and cost money. A composite evaluator with a short-circuit threshold gives you the best of both. diff --git a/skills/optimization-guide/SKILL.md b/skills/optimization-guide/SKILL.md index a6e42b6..eaf0bff 100644 --- a/skills/optimization-guide/SKILL.md +++ b/skills/optimization-guide/SKILL.md @@ -150,7 +150,7 @@ For cache reuse across runs, copy prior disk cache entries into a new run direct Notes: 1. `--cache-from` requires `--cache` and `--run-dir`. 2. It copies `fitness_cache/` from the prior run before optimization. -3. GEPA 0.1.4 can migrate older state forward; rollback to 0.1.1 may not load it. +3. GEPA 0.1.4 can migrate older state forward; rollback to GEPA 0.1.1 may not load it. ### 7. Interpret Results From 9c03531b613ac464630932ea2309489b968160ad Mon Sep 17 00:00:00 2001 From: Ahmad Ragab Date: Sun, 27 Sep 2026 17:41:50 -0700 Subject: [PATCH 4/8] docs: record Codex optimization rejection and score delta --- docs/verification/release-0.6.0.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 docs/verification/release-0.6.0.md diff --git a/docs/verification/release-0.6.0.md b/docs/verification/release-0.6.0.md new file mode 100644 index 0000000..27b3972 --- /dev/null +++ b/docs/verification/release-0.6.0.md @@ -0,0 +1,21 @@ +# 0.6.0 release verification + +## Live Codex skill optimization + +**Run date:** 2026-09-27 PDT (2026-09-28 UTC) +**Decision:** Reject all generated candidates. The skill and `scores.json` remain unchanged. + +The post-packaging source of `skills/generate-evaluator/SKILL.md` scored **0.8876** with `evaluators/skill_clarity.sh` before the live run. Its SHA-256 was `36176f2956c44802031495bae173725c6843ec8d3b14391dbabcbf98deefa108`. The older `scores.json` entry records 0.8503; that historical value was not used as this run's baseline. The acceptance target was 0.9 plus the evaluator contract and plugin workflow checks. + +Both runs used the direct CLI `optimize` path with `--proposer-backend codex`, `--no-api-fallback`, `--no-parallel`, one proposal per iteration, and the deterministic evaluator. Each requested six metric calls. `--evaluator-command bash evaluators/skill_clarity.sh` was the last flag. Raw prompts, candidate text, and run logs remain in ignored `integration_runs/release-0.6.0-u3/`. + +| Run ID | Actual metric calls | Codex proposal calls | Finite proposal scores | Retained `num_candidates` | Best score | +|---|---:|---:|---|---:|---:| +| `94a607a28ce9408597ec49b8694be3e8` | 7 | 3 | 0.8275, 0.8728, 0.8473 | 1 | 0.8876 | +| `5e22321923c64eb98ac53d0ba6d6646a` | 7 | 3 | 0.8374, 0.8477, 0.8360 | 1 | 0.8876 | + +The six proposals were distinct from the seed, with three distinct proposal texts per run. All six Codex calls reported `actual_backend=codex`, `auth_class=subscription`, and model `gpt-5.6-terra`; the backend plan disabled API fallback and reported no fallback cause or mixed backend. Total spend was **14 evaluator calls** and **6 Codex subscription calls** (57,914 reported tokens across the two runs). GEPA used one more evaluator call than each requested budget because it checks the budget between iterations. No billed API fallback was used. + +`num_candidates=1` in each summary counts retained candidates, so it does not mean the run failed to evaluate proposals. The run logs record three evaluated proposals in each run. The best proposal scored 0.8728, a **-0.0148** delta from the frozen baseline; the retained best stayed at 0.8876, a **0.0** delta. The release plan's stricter `num_candidates > 1` verification condition was **not met**, and no candidate reached the 0.9 target. No candidate was applied, so candidate-specific Protocol v2, preflight, numeric score, evaluator-pattern, host/backend, no-fallback, and bundled-launcher acceptance checks were not reached. This is an explicit rejection under the 15-call cap, not an accepted optimization improvement. + +After both runs, the skill still matched the frozen source byte for byte, so the applied skill diff is empty. `scores.json` still had SHA-256 `ddfade5810c4323b38daede71a2ff10e9afb0e91df65b1c5c7d40038d177377a`; no other score entries were changed. From 053d3c74adf9a811d981c57a6f5247e022d109b3 Mon Sep 17 00:00:00 2001 From: Ahmad Ragab Date: Sun, 27 Sep 2026 17:46:30 -0700 Subject: [PATCH 5/8] chore: prepare 0.6.0 metadata and release evidence --- .claude-plugin/marketplace.json | 4 ++-- .claude-plugin/plugin.json | 2 +- .codex-plugin/plugin.json | 2 +- CHANGELOG.md | 8 +++++++- docs/verification/release-0.6.0.md | 24 ++++++++++++++++++++++++ install.md | 5 +++++ pyproject.toml | 2 +- uv.lock | 2 +- 8 files changed, 42 insertions(+), 7 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 987754b..d27555e 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -5,14 +5,14 @@ }, "metadata": { "description": "Claude Code plugin for optimizing text artifacts using gepa", - "version": "0.5.1" + "version": "0.6.0" }, "plugins": [ { "name": "optimize-anything", "source": "./", "description": "Optimize any text artifact using gepa — prompts, code, configs, skills", - "version": "0.5.1", + "version": "0.6.0", "homepage": "https://github.com/ASRagab/optimize-anything", "repository": "https://github.com/ASRagab/optimize-anything", "keywords": ["optimization", "gepa", "prompts", "evaluator", "llm-judge"], diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 6d99b65..814e823 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "optimize-anything", - "version": "0.5.1", + "version": "0.6.0", "description": "Optimize any text artifact using gepa — prompts, code, configs, skills", "keywords": ["optimization", "gepa", "prompts", "evaluator"], "author": { diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json index 450dac7..e95975e 100644 --- a/.codex-plugin/plugin.json +++ b/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "optimize-anything", - "version": "0.5.1", + "version": "0.6.0", "description": "Optimize prompts and other text artifacts with measured evaluator feedback", "author": { "name": "optimize-anything contributors" diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c39f4c..a9d0530 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,13 @@ ## Unreleased +## v0.6.0 - 2026-09-27 + +### Plugin and documentation repairs +- Made shared skills use the bundled runtime in plugin installs, including the compatible Codex SDK extra +- Clarified subscription backend prerequisites, API fallback behavior, evaluator generation, and current smoke gates across active guides +- Aligned Python, Claude Code, and Codex plugin metadata at 0.6.0 + ### Subscription-backed LLM backends - Added support for Codex and Claude as subscription-backed proposer, judge, analysis, score, and validation roles - Integrated optional `codex` extras with secure auth isolation and no API key exposure on host @@ -14,7 +21,6 @@ - Run-scoped coordination with same-vendor conservative fallback (Codex falls back to OpenAI, Claude to Anthropic) - Provenance field (`llm_provenance`) in evaluation results capturing backend, auth class, and fallback decisions - Opt-in live gates for subscription backend verification with no impact on default CI or offline workflows -- No version bump to pyproject.toml, plugin metadata, or Codex plugin in this release ## v0.5.1 - 2026-07-28 diff --git a/docs/verification/release-0.6.0.md b/docs/verification/release-0.6.0.md index 27b3972..3861cac 100644 --- a/docs/verification/release-0.6.0.md +++ b/docs/verification/release-0.6.0.md @@ -19,3 +19,27 @@ The six proposals were distinct from the seed, with three distinct proposal text `num_candidates=1` in each summary counts retained candidates, so it does not mean the run failed to evaluate proposals. The run logs record three evaluated proposals in each run. The best proposal scored 0.8728, a **-0.0148** delta from the frozen baseline; the retained best stayed at 0.8876, a **0.0** delta. The release plan's stricter `num_candidates > 1` verification condition was **not met**, and no candidate reached the 0.9 target. No candidate was applied, so candidate-specific Protocol v2, preflight, numeric score, evaluator-pattern, host/backend, no-fallback, and bundled-launcher acceptance checks were not reached. This is an explicit rejection under the 15-call cap, not an accepted optimization improvement. After both runs, the skill still matched the frozen source byte for byte, so the applied skill diff is empty. `scores.json` still had SHA-256 `ddfade5810c4323b38daede71a2ff10e9afb0e91df65b1c5c7d40038d177377a`; no other score entries were changed. + +## Premerge release-candidate checks + +**Checked on:** 2026-09-27 PDT (2026-09-28 UTC). **Tested base commit:** `9c03531b613ac464630932ea2309489b968160ad`, with the 0.6.0 metadata, changelog, lockfile, and install-guide changes in the working tree. These local checks are not a claim that the eventual PR head has passed CI. The final candidate SHA and PR checks must be recorded after the release changes are committed. + +| Gate | Status | Observed result | +|---|---|---| +| Version and lockfile | PASS | `uv lock` resolved 77 packages and changed the local project entry from 0.5.1 to 0.6.0. Python, Claude manifest and marketplace, and Codex manifest fields now read 0.6.0; the Codex marketplace resolves its version from the manifest. | +| Focused release contracts | PASS | `uv run pytest tests/test_prompt_plugin_contract.py tests/test_doc_contract.py`: 32 passed. The version-parity contract reads all active version fields. | +| Offline project gate | PASS | `uv run python scripts/check.py --skip-smoke`: 545 passed, 18 skipped; score check passed for all three tracked baselines. The smoke harness and credentialed plugin regression were explicitly skipped by this command. | +| CLI smoke | PASS | `uv run python scripts/smoke_harness.py --budget 1`: CLI and overall passed. Output remained under ignored `smoke_outputs/`. | +| Package build and wheel install | PASS | `uv build` made `optimize_anything-0.6.0.tar.gz` and `optimize_anything-0.6.0-py3-none-any.whl`. The wheel lists the CLI module, entry point, and 0.6.0 metadata. An isolated `uv tool run --from dist/optimize_anything-0.6.0-py3-none-any.whl optimize-anything --help` succeeded. | +| Plugin payload inspection | PASS | The 0.6.0 source archive lists both host manifests, the Claude marketplace and commands, the shared launcher, all four canonical skills, and the prompt skill's agent, reference, and evaluator resources. The wheel is the Python CLI package; plugin distribution uses the repository/source tree. | +| Clean host install (U2 result, reused) | PASS | A fresh copied plugin snapshot with no global CLI exposed four shared skills and nine Claude commands. Its bundled launcher ran `budget` with Codex SDK 0.156.0, generated a Codex/no-fallback evaluator, and passed child preflight. A separate isolated global install and Python 3.10.21 check passed after pinning the SDK to 0.156.0; U2 reported 56 focused tests passed. This check was not rerun during U4 package inspection. | +| PR head, review, and post-merge gates | PENDING | PR candidate SHA, CI results, merge SHA, and tag/install results belong to the release operation. | + +The plan's `num_candidates > 1` proxy is a failed check, not a publication gate: GEPA counts retained candidates there. The live logs instead show six distinct evaluated proposals with finite scores, actual Codex subscription provenance, and no API fallback. This satisfies the real-candidate part of R4. All scores were below the frozen baseline and the 0.9 acceptance target, so R5 rejected them and left both source and score history intact. No candidate improvement or cross-provider corroboration is claimed. + +## Known audit findings and release disposition + +- **GEPA evaluator cache identity (R4): follow-up, with an operating constraint.** `test_cache_fingerprint_uses_actual_route_without_exposing_input` covers the backend's safe fingerprint, but GEPA's evaluator `fitness_cache` key does not include the actual completion route after a subscription-to-API fallback. This affects opt-in `--cache`/`--cache-from` runs when an evaluator or judge can switch routes; it does not affect the default uncached path or the uncached Codex release run above. `install.md` now tells users to omit those cache flags for such runs. A route-aware cache policy or GEPA integration remains follow-up work; the audit's R4 status remains partial. +- **TOML table-over-scalar precedence (R6): follow-up.** One TOML document cannot define `model.proposer` as both a scalar and a table; standard parsing rejects it before precedence can apply. `test_scalar_table_model_conflict_names_both_keys` verifies the specific diagnostic, while existing CLI-over-spec tests verify the supported override path. Use one representation per role and a CLI flag for an override. Multi-file layering or new syntax requires a separate product decision; the audit's R6 status remains partial. + +The premerge evidence and changelog contain no credentials, account identity, raw prompts, or private candidate text. Detailed live artifacts remain in ignored `integration_runs/`; built packages and smoke outputs remain outside the committed release evidence. diff --git a/install.md b/install.md index a2cc9a8..0923f92 100644 --- a/install.md +++ b/install.md @@ -96,6 +96,11 @@ The role stays on the API for the rest of the run, including generated evaluator child processes; other roles keep their own circuits. Pass `--no-api-fallback` to make every subscription failure terminal. +When an evaluator or judge can switch from a subscription backend to API +fallback, omit `--cache` and `--cache-from`. GEPA's evaluator cache key does +not include the route actually used for each completion, so a cached score +could be reused after that route changes. + ### Data handling - Each request runs in a new, empty temporary workspace that is deleted diff --git a/pyproject.toml b/pyproject.toml index ebef964..8a7c941 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "optimize-anything" -version = "0.5.1" +version = "0.6.0" description = "Optimize any text artifact using gepa — prompts, code, configs, skills" readme = "README.md" license = {text = "MIT"} diff --git a/uv.lock b/uv.lock index 0a3c581..6c0b3f5 100644 --- a/uv.lock +++ b/uv.lock @@ -1394,7 +1394,7 @@ wheels = [ [[package]] name = "optimize-anything" -version = "0.5.1" +version = "0.6.0" source = { editable = "." } dependencies = [ { name = "cloudpickle" }, From 0db70f587f52d05a5415af3ad3088952f61ee158 Mon Sep 17 00:00:00 2001 From: Ahmad Ragab Date: Sun, 27 Sep 2026 17:48:11 -0700 Subject: [PATCH 6/8] test: tighten plugin contract checks --- tests/test_plugin_launcher.py | 2 +- tests/test_prompt_plugin_contract.py | 12 +++++------- 2 files changed, 6 insertions(+), 8 deletions(-) diff --git a/tests/test_plugin_launcher.py b/tests/test_plugin_launcher.py index 52df4ba..d83a4be 100644 --- a/tests/test_plugin_launcher.py +++ b/tests/test_plugin_launcher.py @@ -131,6 +131,6 @@ def test_global_installer_codex_flag_adds_sdk_to_tool_environment(tmp_path: Path "install", str(REPO_ROOT), "--with", - "openai-codex==0.156.0", + f"openai-codex=={_SDK_VERSION}", "--force", ] diff --git a/tests/test_prompt_plugin_contract.py b/tests/test_prompt_plugin_contract.py index 92a48f8..62ecf80 100644 --- a/tests/test_prompt_plugin_contract.py +++ b/tests/test_prompt_plugin_contract.py @@ -96,17 +96,15 @@ def test_claude_commands_use_the_bundled_launcher(): def test_shared_skills_resolve_the_bundled_launcher_for_cli_examples(): - expected = { - "generate-evaluator", - "optimization-guide", - "optimize-prompt", - "evaluator-patterns", - } + expected = {path.parent.name for path in SKILL_FILES} skill_dirs = {path.name for path in (REPO_ROOT / "skills").iterdir() if path.is_dir()} assert skill_dirs == expected for name in expected: skill = _read(f"skills/{name}/SKILL.md") - assert re.match(rf"^---\nname: {name}\ndescription: .+\n---\n", skill, re.DOTALL) + assert re.match( + rf"^---\nname: {name}\ndescription: [^\n]+\n(?: [^\n]+\n)*---\n", + skill, + ) if re.search(r"(? Date: Sun, 27 Sep 2026 18:06:21 -0700 Subject: [PATCH 7/8] fix(review): repair installer and bundled skill examples (#1 #2 #4) --- install.sh | 3 ++- skills/evaluator-patterns/SKILL.md | 16 ++++++++++++- skills/generate-evaluator/SKILL.md | 4 +++- skills/optimize-prompt/SKILL.md | 2 +- tests/test_plugin_launcher.py | 35 ++++++++++++++++++++++++++++ tests/test_prompt_plugin_contract.py | 17 +++++++------- 6 files changed, 65 insertions(+), 12 deletions(-) diff --git a/install.sh b/install.sh index 5ec0ca9..ea86bdc 100755 --- a/install.sh +++ b/install.sh @@ -101,7 +101,8 @@ install_tool() { if [ "$CODEX_EXTRA" -eq 1 ]; then install_args+=(--with 'openai-codex==0.156.0') fi - if uv tool install "${INSTALL_SOURCE}" "${install_args[@]}" --force 2>&1; then + install_args+=(--force) + if uv tool install "${INSTALL_SOURCE}" "${install_args[@]}" 2>&1; then info "optimize-anything installed successfully" else error "Installation failed" diff --git a/skills/evaluator-patterns/SKILL.md b/skills/evaluator-patterns/SKILL.md index c1b0e58..e660674 100644 --- a/skills/evaluator-patterns/SKILL.md +++ b/skills/evaluator-patterns/SKILL.md @@ -7,6 +7,19 @@ description: Complete runnable evaluator templates for prompt, code, documentati Use this skill to generate evaluator scripts that follow the optimize-anything command evaluator contract. +## Resolve the bundled runtime + +Locate this installed `SKILL.md`, then set: + +```bash +EVALUATOR_PATTERNS_SKILL_DIR="/absolute/path/to/skills/evaluator-patterns" +OPTIMIZE_ANYTHING_ROOT="$(cd "$EVALUATOR_PATTERNS_SKILL_DIR/../.." && pwd)" +OPTIMIZE_ANYTHING_RUNNER="$OPTIMIZE_ANYTHING_ROOT/scripts/run-optimize-anything" +``` + +Use `$OPTIMIZE_ANYTHING_RUNNER` for CLI calls. Run the LiteLLM example with +the plugin's locked project dependencies as shown below. + ## Evaluator I/O contract (all patterns) - Read one JSON object from stdin: `{"candidate": "..."}` - Write one JSON object to stdout on a single line: `{"score": , ...diagnostics...}` @@ -124,7 +137,8 @@ if __name__ == "__main__": **Test it:** ```bash -echo '{"candidate":"You are a careful coding assistant. Ask one clarifying question if needed, then provide concise steps."}' | python3 eval_prompt.py +echo '{"candidate":"You are a careful coding assistant. Ask one clarifying question if needed, then provide concise steps."}' | \ + uv run --project "$OPTIMIZE_ANYTHING_ROOT" --locked --no-dev --extra codex python eval_prompt.py ``` --- diff --git a/skills/generate-evaluator/SKILL.md b/skills/generate-evaluator/SKILL.md index 5bf8d9d..aac58cb 100644 --- a/skills/generate-evaluator/SKILL.md +++ b/skills/generate-evaluator/SKILL.md @@ -99,12 +99,14 @@ For dataset-aware evaluators: ```bash "$OPTIMIZE_ANYTHING_RUNNER" generate-evaluator seed.txt \ --objective "Score correctness" \ - --dataset examples.jsonl > eval_dataset.py + --dataset > eval_dataset.py echo '{"candidate":"text","example":{"input":"q","expected":"a"}}' | \ uv run --project "$OPTIMIZE_ANYTHING_ROOT" --locked --no-dev --extra codex python eval_dataset.py ``` +`generate-evaluator --dataset` selects the dataset-aware template. Pass the JSONL file later to `optimize --dataset examples.jsonl`. + ## Workflow 1. Clarify artifact + objective + hard constraints. 2. Pick evaluator pattern (judge default, composite for safety gates). diff --git a/skills/optimize-prompt/SKILL.md b/skills/optimize-prompt/SKILL.md index 12a103a..6b8b9f6 100644 --- a/skills/optimize-prompt/SKILL.md +++ b/skills/optimize-prompt/SKILL.md @@ -134,7 +134,7 @@ JUDGE_MODEL="$JUDGE_MODEL" "$OPTIMIZE_ANYTHING_RUNNER" optimize "$SEED_FILE" \ --task-model "$TASK_MODEL" --model "$PROPOSER_MODEL" \ --budget "$BUDGET" --output "$CANDIDATE_FILE" --diff \ --run-dir "$RUNS_DIR" --early-stop --evaluator-cwd "$WORK_DIR" \ - --evaluator-command uv run --project "$OPTIMIZE_ANYTHING_ROOT" --locked \ + --evaluator-command uv run --project "$OPTIMIZE_ANYTHING_ROOT" --locked --no-dev --extra codex \ python "$OPTIMIZE_PROMPT_SKILL_DIR/scripts/prompt_execution_evaluator.py" ``` diff --git a/tests/test_plugin_launcher.py b/tests/test_plugin_launcher.py index d83a4be..e82adc5 100644 --- a/tests/test_plugin_launcher.py +++ b/tests/test_plugin_launcher.py @@ -134,3 +134,38 @@ def test_global_installer_codex_flag_adds_sdk_to_tool_environment(tmp_path: Path f"openai-codex=={_SDK_VERSION}", "--force", ] + + +def test_global_installer_without_flags_uses_plain_tool_install(tmp_path: Path): + bin_dir = tmp_path / "bin" + bin_dir.mkdir() + args_file = tmp_path / "uv-args.txt" + _write_fake_uv( + bin_dir / "uv", + 'if [ "$1" = "--version" ]; then echo "uv 0.0.0"; exit 0; fi\n' + 'printf "%s\\n" "$@" > "$UV_ARGS_FILE"\n', + ) + env = os.environ.copy() + env.update( + { + "HOME": str(tmp_path), + "PATH": f"{bin_dir}:/usr/bin:/bin", + "UV_ARGS_FILE": str(args_file), + "OPTIMIZE_ANYTHING_REPO": str(REPO_ROOT), + } + ) + + result = subprocess.run( + ["/bin/bash", str(INSTALLER)], + capture_output=True, + text=True, + env=env, + ) + + assert result.returncode == 0, result.stderr + assert args_file.read_text(encoding="utf-8").splitlines() == [ + "tool", + "install", + str(REPO_ROOT), + "--force", + ] diff --git a/tests/test_prompt_plugin_contract.py b/tests/test_prompt_plugin_contract.py index 62ecf80..d446a0e 100644 --- a/tests/test_prompt_plugin_contract.py +++ b/tests/test_prompt_plugin_contract.py @@ -17,6 +17,12 @@ Path("skills/optimize-prompt/scripts/prompt_execution_evaluator.py"), Path("scripts/run-optimize-anything"), ) +CLI_SUBCOMMANDS = ( + "optimize", "generate-evaluator", "intake", "explain", "budget", "score", "validate", "analyze" +) +GLOBAL_CLI_RE = re.compile( + r"(? str: @@ -85,11 +91,7 @@ def test_claude_commands_use_the_bundled_launcher(): offenders = [] for path in sorted((REPO_ROOT / "commands").glob("*.md")): text = path.read_text(encoding="utf-8") - global_invocation = re.search( - r"(? Date: Sun, 27 Sep 2026 18:07:25 -0700 Subject: [PATCH 8/8] docs: record reviewed release candidate and platform limit --- CHANGELOG.md | 1 + docs/verification/release-0.6.0.md | 18 ++++++++++-------- install.md | 4 ++++ 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a9d0530..d3330ef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ ### Plugin and documentation repairs - Made shared skills use the bundled runtime in plugin installs, including the compatible Codex SDK extra +- Fixed the default installer on macOS Bash 3.2 and repaired dataset and LiteLLM evaluator examples - Clarified subscription backend prerequisites, API fallback behavior, evaluator generation, and current smoke gates across active guides - Aligned Python, Claude Code, and Codex plugin metadata at 0.6.0 diff --git a/docs/verification/release-0.6.0.md b/docs/verification/release-0.6.0.md index 3861cac..76ca0df 100644 --- a/docs/verification/release-0.6.0.md +++ b/docs/verification/release-0.6.0.md @@ -3,7 +3,7 @@ ## Live Codex skill optimization **Run date:** 2026-09-27 PDT (2026-09-28 UTC) -**Decision:** Reject all generated candidates. The skill and `scores.json` remain unchanged. +**Decision:** Reject all generated candidates. No candidate changed the skill or `scores.json` during the run; a later release-review correction changed one skill example. The post-packaging source of `skills/generate-evaluator/SKILL.md` scored **0.8876** with `evaluators/skill_clarity.sh` before the live run. Its SHA-256 was `36176f2956c44802031495bae173725c6843ec8d3b14391dbabcbf98deefa108`. The older `scores.json` entry records 0.8503; that historical value was not used as this run's baseline. The acceptance target was 0.9 plus the evaluator contract and plugin workflow checks. @@ -18,28 +18,30 @@ The six proposals were distinct from the seed, with three distinct proposal text `num_candidates=1` in each summary counts retained candidates, so it does not mean the run failed to evaluate proposals. The run logs record three evaluated proposals in each run. The best proposal scored 0.8728, a **-0.0148** delta from the frozen baseline; the retained best stayed at 0.8876, a **0.0** delta. The release plan's stricter `num_candidates > 1` verification condition was **not met**, and no candidate reached the 0.9 target. No candidate was applied, so candidate-specific Protocol v2, preflight, numeric score, evaluator-pattern, host/backend, no-fallback, and bundled-launcher acceptance checks were not reached. This is an explicit rejection under the 15-call cap, not an accepted optimization improvement. -After both runs, the skill still matched the frozen source byte for byte, so the applied skill diff is empty. `scores.json` still had SHA-256 `ddfade5810c4323b38daede71a2ff10e9afb0e91df65b1c5c7d40038d177377a`; no other score entries were changed. +Immediately after both runs, the skill still matched the frozen source byte for byte, so the applied optimization diff was empty. Release review later corrected the dataset-aware CLI example; the current skill has SHA-256 `dfbc4465dc5de7ca515520b65422039b08ea518e0396bbc7384b32e182c6e4e7` and scores 0.8875 with the same evaluator, 0.0001 below the frozen run seed. `scores.json` still has SHA-256 `ddfade5810c4323b38daede71a2ff10e9afb0e91df65b1c5c7d40038d177377a`; no score entries were changed. ## Premerge release-candidate checks -**Checked on:** 2026-09-27 PDT (2026-09-28 UTC). **Tested base commit:** `9c03531b613ac464630932ea2309489b968160ad`, with the 0.6.0 metadata, changelog, lockfile, and install-guide changes in the working tree. These local checks are not a claim that the eventual PR head has passed CI. The final candidate SHA and PR checks must be recorded after the release changes are committed. +**Checked on:** 2026-09-27 PDT (2026-09-28 UTC). **Tested release-code commit:** `d9defd1ec5295223527b81fdf2c83ae5ef752a9c`. The full gate, focused contracts, Bash syntax, type check, and package build ran on that tree immediately before or after its commit. The earlier U2 clean-install check is identified separately below. Subsequent release-evidence edits need the final PR-head CI check; these local results do not claim it has passed. | Gate | Status | Observed result | |---|---|---| | Version and lockfile | PASS | `uv lock` resolved 77 packages and changed the local project entry from 0.5.1 to 0.6.0. Python, Claude manifest and marketplace, and Codex manifest fields now read 0.6.0; the Codex marketplace resolves its version from the manifest. | -| Focused release contracts | PASS | `uv run pytest tests/test_prompt_plugin_contract.py tests/test_doc_contract.py`: 32 passed. The version-parity contract reads all active version fields. | -| Offline project gate | PASS | `uv run python scripts/check.py --skip-smoke`: 545 passed, 18 skipped; score check passed for all three tracked baselines. The smoke harness and credentialed plugin regression were explicitly skipped by this command. | -| CLI smoke | PASS | `uv run python scripts/smoke_harness.py --budget 1`: CLI and overall passed. Output remained under ignored `smoke_outputs/`. | +| Focused release contracts | PASS | `uv run pytest tests/test_prompt_plugin_contract.py tests/test_doc_contract.py tests/test_plugin_launcher.py`: 38 passed. The version-parity contract reads all active version fields; the installer test covers default Bash 3.2 and Codex paths. | +| Full project gate | PASS | `uv run python scripts/check.py`: 546 passed, 18 skipped; CLI smoke and all three tracked score baselines passed. Credentialed plugin regression was skipped because `--with-plugin` was not set. The reviewed skill's current deterministic score is 0.8875. | +| Type and shell checks | PASS | `uv run mypy src/optimize_anything`: no issues in 27 source files. `/bin/bash -n install.sh scripts/run-optimize-anything` and `git diff --check` passed. | | Package build and wheel install | PASS | `uv build` made `optimize_anything-0.6.0.tar.gz` and `optimize_anything-0.6.0-py3-none-any.whl`. The wheel lists the CLI module, entry point, and 0.6.0 metadata. An isolated `uv tool run --from dist/optimize_anything-0.6.0-py3-none-any.whl optimize-anything --help` succeeded. | | Plugin payload inspection | PASS | The 0.6.0 source archive lists both host manifests, the Claude marketplace and commands, the shared launcher, all four canonical skills, and the prompt skill's agent, reference, and evaluator resources. The wheel is the Python CLI package; plugin distribution uses the repository/source tree. | | Clean host install (U2 result, reused) | PASS | A fresh copied plugin snapshot with no global CLI exposed four shared skills and nine Claude commands. Its bundled launcher ran `budget` with Codex SDK 0.156.0, generated a Codex/no-fallback evaluator, and passed child preflight. A separate isolated global install and Python 3.10.21 check passed after pinning the SDK to 0.156.0; U2 reported 56 focused tests passed. This check was not rerun during U4 package inspection. | -| PR head, review, and post-merge gates | PENDING | PR candidate SHA, CI results, merge SHA, and tag/install results belong to the release operation. | +| Review and fixes | PASS | Compound Engineering review run `20260927-174849-e5d41434` completed with an independent Claude adversarial pass. Three validated findings were fixed: default Bash installer, dataset quick start, and bundled LiteLLM example/guard. The evaluator child's dependency flags were aligned. A platform limitation remains documented below. | +| PR head and post-merge gates | PENDING | Final PR-head CI results, merge SHA, main workflow, and tag/install results belong to the release operation. | -The plan's `num_candidates > 1` proxy is a failed check, not a publication gate: GEPA counts retained candidates there. The live logs instead show six distinct evaluated proposals with finite scores, actual Codex subscription provenance, and no API fallback. This satisfies the real-candidate part of R4. All scores were below the frozen baseline and the 0.9 acceptance target, so R5 rejected them and left both source and score history intact. No candidate improvement or cross-provider corroboration is claimed. +The plan's `num_candidates > 1` proxy is a failed check, not a publication gate: GEPA counts retained candidates there. The live logs instead show six distinct evaluated proposals with finite scores, actual Codex subscription provenance, and no API fallback. This satisfies the real-candidate part of R4. All scores were below the frozen baseline and the 0.9 acceptance target, so R5 rejected them; no optimization candidate changed the source or score history. The later CLI example correction is separate from that decision. No candidate improvement or cross-provider corroboration is claimed. ## Known audit findings and release disposition - **GEPA evaluator cache identity (R4): follow-up, with an operating constraint.** `test_cache_fingerprint_uses_actual_route_without_exposing_input` covers the backend's safe fingerprint, but GEPA's evaluator `fitness_cache` key does not include the actual completion route after a subscription-to-API fallback. This affects opt-in `--cache`/`--cache-from` runs when an evaluator or judge can switch routes; it does not affect the default uncached path or the uncached Codex release run above. `install.md` now tells users to omit those cache flags for such runs. A route-aware cache policy or GEPA integration remains follow-up work; the audit's R4 status remains partial. - **TOML table-over-scalar precedence (R6): follow-up.** One TOML document cannot define `model.proposer` as both a scalar and a table; standard parsing rejects it before precedence can apply. `test_scalar_table_model_conflict_names_both_keys` verifies the specific diagnostic, while existing CLI-over-spec tests verify the supported override path. Use one representation per role and a CLI flag for an override. Multi-file layering or new syntax requires a separate product decision; the audit's R6 status remains partial. +- **Plugin platforms without a Codex binary wheel: follow-up.** The bundled launcher selects the Codex extra for every plugin invocation. On an architecture without an `openai-codex-cli-bin` wheel, even API-only plugin commands cannot start. The release install check covers macOS arm64; other platforms remain untested. `install.md` directs API-only users on those platforms to the global CLI or source install without the Codex extra. A broader plugin runtime policy needs a separate compatibility decision. The premerge evidence and changelog contain no credentials, account identity, raw prompts, or private candidate text. Detailed live artifacts remain in ignored `integration_runs/`; built packages and smoke outputs remain outside the committed release evidence. diff --git a/install.md b/install.md index 0923f92..de1c451 100644 --- a/install.md +++ b/install.md @@ -55,6 +55,10 @@ either plugin. Other platforms, hosted runners, and newer SDK releases are untested; the adapters fail closed rather than guessing when a required control is missing. +The plugin launcher always selects the Codex SDK extra. On a platform without +a compatible Codex binary wheel, plugin commands cannot start, including API-only +commands. For API-only use there, install the global CLI without `--codex` or +run from source without the `codex` extra. ### Experimental Claude scope