Experiment runner extension

[ajonkisz]

Experiment Runner extension

Ships a generic experiment & research runner as an installable Bobbit market pack (market-packs/experiment-runner/), turning nested goals + the #822 goal-metadata layer + inlineRoles + gates/cost accounting into a data-driven experimentation + autonomous-optimization capability.

One engine, two front doors

• A/B comparison (default, bounded): fan out variant × repeat child goals (each carrying distinct per-arm metadata + inlineRoles), collect outcomes, aggregate (median + spread, same-completion-bar filtering), and render a comparison. Run-count/cost projected before launch. No self-modification.
• Autoresearch (opt-in, OFF by default, hard-capped): a deterministic generate → evaluate-under-fixed-budget → keep-iff-improves-AND-verifies → iterate-until-plateau/budget/target loop with a fed-forward ledger. Refuses to start uncapped; enforces per-run budget + maxCostUsd (incl. next-iteration projection) + plateau/target/wall-clock stops; correctness-gated; the LLM proposes, the framework decides keep/stop.

Minimal core/host change

The only core change is the sanctioned host.agents.spawnGoal({ title, spec, runKey, parentGoalId?, metadata?, inlineRoles?, workflowId?, workflow? }) → { goalId } seam (src/server/agent/experiment-spawn-goal.ts, wired through the extension host). Parent is server-derived (parentGoalId is an assertion only → PARENT_MISMATCH); runKey is idempotent with an in-flight reservation; capacity-blocked children are parked state:"blocked"; per-arm treatment propagates across each run's sub-tree via #822. The route-name allowlist was widened to accept camelCase route ids.

Extensibility (seams from day one)

• Shared reporting library src/shared/experiment-report/** (single source of truth for aggregation/series/widgets), bundled to the pack via build:packs.
• Metric-extractor registry (built-ins + user-metric channel + generic command metric) and widget-renderer registry + editable dashboard view-spec — add a metric/widget by registration, not refactor. Per-experiment metric selection + dashboard re-render from stored raw outcomes without re-running.

Tests

Pack unit (run-config/aggregate/accept-reject/stop/guardrails/extensibility), shared-report unit (+ no-fork single-source guards), API E2E (A/B fan-out via spawnGoal with distinct per-arm metadata; stubbed autoresearch keep-best + stop; live REST outcome parsing against a self-signed HTTPS server), and browser E2E (A/B default, autoresearch guardrails + successful launch, dashboard spec edit/re-render, install/uninstall). npm run check / test:unit / test:e2e green; test:manual run (flakes only).

🤖 Generated with Bobbit

[ajonkisz]

Review summary — Experiment Runner extension

Ships a generic experiment & research runner as an installable market pack (market-packs/experiment-runner/), built on nested goals + the #822 goal-metadata layer + inlineRoles + gates/cost accounting. 51 files, +11.9k/-13.

Two modes off one engine:

• A/B comparison (default): fan out variant × repeat child goals, each with per-arm metadata/inlineRoles, collect outcomes, aggregate (median + spread, same-completion-bar filtering), render a comparison. Cost projected pre-launch.
• Autoresearch (opt-in, OFF by default): deterministic generate → evaluate-under-fixed-budget → keep-iff-improves-and-verifies → iterate-until-plateau/budget/target loop. Refuses to start uncapped.

The only core change is the host.agents.spawnGoal seam (experiment-spawn-goal.ts), wired as a 7th agents verb through the extension host, plus a shared reporting lib (src/shared/experiment-report/**) and a boot seed that ships the pack disabled-by-default.

Security / bug review — nothing substantial found

Focused on the trust-sensitive seams:

• spawnGoal parent derivation — authoritative server-side (owner.goalId ?? teamGoalId); caller parentGoalId is an assertion only → PARENT_MISMATCH. No caller-supplied cwd/project/sandbox (all inherited). Child sessions blocked from spawning (assertCanSpawn). Idempotency via in-flight reservation + sibling scan collapses the TOCTOU race. Subgoal/nesting policy enforced. Solid.
• Route-name regex widening (/^[A-Za-z0-9][A-Za-z0-9_-]*$/) — only adds uppercase; still anchored, no .//, so no traversal. Safe.
• Cost caps — autoresearch validateDef rejects uncapped runs, requires a positive per-run budget + an effective stop condition; shouldStop enforces maxCostUsd pre-spawn (cumulative + projected next). Per-run budget enforced in framework space (applyBudget). Solid.
• execFileSync — only git rev-parse --git-common-dir, no shell, no user input. The runnable.command is embedded into goal spec markdown, never executed by the pack. Safe.
• rejectUnauthorized:false — the sanctioned pattern for the self-signed local gateway cert (matches watchdog.ts), localhost only.
• HTML rendering — report HTML is server-generated through the shared renderer; every user-controlled value passes through escapeHtml. The panel's innerHTML consumes only that escaped, trusted output.
• Boot seed — ships the pack disabled, idempotent via a durable marker, respects existing user/admin activation, never throws.

No malicious intent, no verifiable substantial bug.

Minor (non-blocking)

• buildLedger comment says "only accepted runs advance the best-so-far baseline" but pushes every run to prior; behaviour is correct because computeBestSoFar filters internally — the comment is just slightly misleading.
• markDoneIfFinished flips A/B state to done on all-settled (pre-collect); it's display-only and doesn't block the collect route, so harmless.
• The pack's server routes read the gateway bearer token from .bobbit/state/token to call the REST API as the user (documented design choice for reading goal outcomes) — flagging for reviewer awareness, since an installed pack's routes thereby gain full authenticated gateway API access.

🤖 Generated with Bobbit