Skip to content

V27.14: measure repeated-run model variability - #46

Merged
Aevumard merged 6 commits into
feature/v27-decision-separation-corpusfrom
feature/v27.14-replicate-runner
Oct 6, 2026
Merged

Aevumard merged 6 commits into
feature/v27-decision-separation-corpusfrom
feature/v27.14-replicate-runner

Conversation

@Aevumard

@Aevumard Aevumard commented Oct 6, 2026

Copy link
Copy Markdown
Owner

V27.14

Adds a provider-neutral repeated-run runner on top of the existing V27.6+ resumable execution path.

What changes

  • repeats the exact same 1,200-ticket condition across independent replicas
  • gives each replica its own checkpoint and artifacts
  • aggregates mean/sample std/min/max across replicas
  • measures pairwise raw/guarded action and full-decision disagreement by ticket
  • preserves the existing per-run experiment analysis and paired bootstrap
  • records provider usage coverage without interpreting missing usage as zero
  • stores a sanitized provider manifest and system-prompt SHA-256 fingerprint
  • rejects output-directory reuse when suite/condition/provider metadata changes

Scope

  • historical harness is untouched
  • live provider execution remains opt-in and is not run in CI
  • the repeated-run variability is explicitly not treated as causal evidence

Tests cover independent checkpoints, resume behavior, manifest binding, secret metadata filtering, and variability measurement.

@Aevumard
Aevumard merged commit 3342fb4 into feature/v27-decision-separation-corpus Oct 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant