feat(arena): skills arena + genetic synthesis PoCs (#28) - #13
Draft
rotnov wants to merge 5 commits into
Draft
Conversation
Phase 1 of the skills arena roadmap (per .ievo/research/2026-05-24-skills-arena.md). Standalone script that runs ONE controlled-experiment arena round to validate the A/B eval methodology before investing in full implementation. ## Controlled experiment design - Candidate A: full production-style SKILL.md (verbose ievo:init orchestrator with 7 steps, pre-flight, stack detection, security audit, install, summary, journal) - Candidate B: deliberately stripped variant of the same skill (3-line body, no steps, no failure handling) - Same task: "you just ran /ievo:init in a Next.js+TS project, walk through every step" - Ground truth: A should win. If arena says B or tie, methodology has a bug. ## Pipeline 1. Run each candidate K times (default 3) with Haiku, passing SKILL.md as system prompt 2. Pick representative output per candidate (longest non-empty — verbosity bias is a known issue, Phase 5 of roadmap adds correction) 3. Opus judge sees both outputs side-by-side, picks winner + emits JSON verdict 4. Position-swap: re-run judge with outputs reversed 5. Consistent-winner only counts (mitigates first-position bias — GPT-4 30%, Claude-v1 75% per MT-Bench) ## Reuses - `cortex.claude.query_raw` — same machinery as cortex evolution A/B eval (task #31, shipped v0.5.2) - Methodology validated in `.ievo/research/2026-03-06-blind-ab-eval.md` ## Configuration via env - `ARENA_RUNS_PER` (default 3) — Haiku runs per candidate - `ARENA_RUNNER_MODEL` (default haiku) — model for candidate runs - `ARENA_JUDGE_MODEL` (default opus) — judge model ## Output - Stdout: human-readable report with both verdicts + consistency check - `dist/arena-poc/round-A-vs-B.json` — full transcripts + cost data - Exit codes: 0 = A won (methodology validated), 1 = B won (unexpected — investigate), 2 = tie, 3 = position-bias inconsistent ## Cost ~$0.50-1.00 per round (3 Haiku × 2 candidates + 2 Opus judges). Requires CLAUDE_CODE_OAUTH_TOKEN in env. ## Out of scope (deferred to Phase 2+ of roadmap) - Real sandbox installer (current PoC inlines SKILL.md content as system prompt; production version installs via `npx skills add`) - Token-length anti-verbosity correction - Author/owner anonymization at the metadata level (judge prompt already strips author info) - Real candidate pairs from skills.sh - Multi-candidate (N>2) round-robin or tournament logic - `/ievo:arena <name>` CLI integration - Periodic re-rank workflow in cortex CI ## Next step after PoC runs successfully If consistent_winner = production: methodology validated → Phase 2 (real sandbox installer). If unexpected: investigate Haiku output patterns or judge prompt before scaling. Co-Authored-By: iEVO <noreply@ievo.ai>
First PoC run (2026-05-24 10:51-11:00) validated methodology
mechanically but with degenerate inputs: tools=[] + max_turns=3
caused Haiku to produce empty text in 4/6 candidate runs. Judge was
comparing real-vs-empty, not real-vs-real.
Operator caught it: "у агента нет доступка к тулам?" — yes, that was
the bug. Skill orchestrators ARE agent harnesses; without tools the
agent has nothing to demonstrate.
## Changes
- Each iteration gets a fresh sandbox cwd at $TMPDIR/arena_poc_*/
with minimal Next.js+TS project markers (package.json with next/
react/typescript deps, tsconfig, README, .gitignore) so the skill
agent has something realistic to inspect
- Tool set: Read, Write, Glob, Grep, Bash — realistic for a skill
orchestrator doing project inspection + execution simulation
- max_turns: 3 → 15 (realistic budget for an orchestrator with
several steps)
- Each candidate iteration gets its OWN sandbox (no contamination
between runs of the same candidate or between candidates)
## Second run result (2026-05-24 11:16-11:19)
Both candidates produced real text outputs. Judges substantively
ranked them on quality dimensions:
- Forward: A wins on security audit, trust tiers, ecosystem
specificity, failure handling
- Swapped: same quality observations, swap-consistent
- Consistent winner: ievo-init-full ✓
- Cost: $0.5756
Methodology produces meaningful quality signals, not just
real-vs-empty distinctions. Phase 1 PoC fully validated.
Co-Authored-By: iEVO <noreply@ievo.ai>
rotnov
marked this pull request as ready for review
May 24, 2026 10:27
rotnov
marked this pull request as draft
May 24, 2026 10:30
Tests whether two parent skills crossed by Opus produce a child that beats both in arena. Multi-child population (N=3) + all-vs-all round-robin to validate the effect isn't lucky-draw variance. Result on ievo-init-full × ievo-init-weak: all 3 children beat both parents (2/4 wins each vs 1/4 strong, 0/4 weak). Cost: $3.81. Caveat: weak parent contributed nothing — child is compressed strong parent. Real idea-mixing requires parents with different valid strategies (next step). Co-Authored-By: iEVO <noreply@ievo.ai>
v1 limitation: strong+weak parents → child was compressed strong parent (weak contributed nothing). Crossover acted as compressor, not idea-mixer. v2: two valid, philosophically opposite parents for the same /ievo:init goal: - defensive-sequential: strict order, audit-first, halt on ambiguity - optimistic-parallel: parallel discovery, retry transients, rollback Result: all 3 children beat BOTH parents. Parents got 0/4 wins each (including head-to-head: inconsistent). Cost $4.75. Best child synthesizes: - Parallel read-only work (Phase 1) — explicit safety classification - Audit gate before write (Phase 4) — combines safety + parallel speedup - Cascading rollback (Phase 5) — per-candidate + halt-if-rollback-fails - "Fast where safe, strict where it matters" — synthesizing principle This is real idea-mixing, not compression. L5 genetic synthesis is viable for skill domain when parents represent distinct valid strategies. Co-Authored-By: iEVO <noreply@ievo.ai>
Tests: does gen-2 keep improving over gen-1, or plateau?
Pipeline: defensive × optimistic → 2 gen-1 children → cross them →
3 grandchildren → 5-way arena.
Result: gen-2 strictly dominates gen-1.
Head-to-head: gen-2 wins 4, gen-1 wins 0, ties/inconsistent 2.
Best grandchild: 3/4 wins. Both gen-1 parents: 0/4 each.
Total cost: $5.36.
Key finding: variance is higher at gen-2 — only 1/3 grandchildren
strongly improved (3 wins), the others stayed flat (1 win each).
This means multi-gen needs explicit best-of-N selection: a single
gen-2 sample is unreliable; pooling 3+ and keeping the winner
recovers the improvement signal.
Qualitative diff (v3 grandchild-2 vs v2 synth-1):
- 4-way decision tree (parallel/sequential/rollback/halt) vs
2-way "fast where safe" slogan
- Phase 5 reverted to sequential install — safer trade-off
- discover.mjs failure: explicit user choice instead of silent
fallback
- Exponential backoff 2s→4s→8s vs flat 2s
- Concrete manifest JSON schema with exact keys
- Named invariant: "no code lands on disk before its audit clears"
For #28 design: arena loop should "generate N → keep best →
recurse → stop when no improvement for K generations."
Co-Authored-By: iEVO <noreply@ievo.ai>
rotnov
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four PoC scripts validating the methodology and design choices for godfather task #28 (skills arena — L4+L5 of the iEvo evolution hierarchy).
skills_arena_poc.pyskills_genetic_poc.pyskills_genetic_v2_poc.pyskills_genetic_v3_multigen_poc.pyTotal: ~$14.50 to empirically answer four #28 design questions.
Key findings for #28 design
Files
scripts/skills_arena_poc.py— arena methodology PoCscripts/skills_genetic_poc.py— single-gen genetic synthesisscripts/skills_genetic_v2_poc.py— strategy-diverse parents (defensive × optimistic)scripts/skills_genetic_v3_multigen_poc.py— multi-generation crossoverNext steps (not in this PR)
Update task #28 spec with PoC findings; design the production arena loop with best-of-N selection and gen-stopping criterion.
🧬 Evolved with iEVO