Skip to content

Add one-use sealed synthetic CVRP check and honest trial coverage - #10

Merged
ucsandman merged 1 commit into
masterfrom
codex/discovery-holdout-20260915
Sep 15, 2026
Merged

ucsandman merged 1 commit into
masterfrom
codex/discovery-holdout-20260915

Conversation

@ucsandman

Copy link
Copy Markdown
Owner

The dashboard can prepare one fresh synthetic CVRP cohort and spend it exactly once. Candidate and seed-baseline bytes are frozen before generation, the six uniform and clustered EUC_2D cases at 50, 100 and 150 customers are written under the ignored sealed-release tree, and one OS-locked transition consumes the cohort before the first solver call. An interrupted or partial run stays consumed and the same candidate hash cannot be resealed.

Both solvers run through one immutable network-disabled Docker image on two matched private seeds per case. A trusted verifier reads an explicit sealed instance path, bounds route count and customer entries before distance work, and keeps only sanitized objectives, outcomes and failures. Dashboard reads require a regular non-symlink result whose SHA-256, cohort ID, schema and final state match the manifest, so edited or missing results display as invalid rather than completed.

The trial report is now a coverage report first. Its denominator is the configured 14-night two-track schedule, and clean rows require terminal research, a completed eligible retro, the scheduled requested arm, and a successful physical generation call routed as requested. Duplicates, operational extras, validation, zero-call and critique-only work, mismatch and fallback stay visible outside clean summaries.

Validation:

  • Full check on this branch: 291 tests passed, 5 skipped; Ruff passed; 77 Python files compiled.
  • Combined with the islands branch (Add governed solver islands for matrix multiplication #9): 298 tests passed, 5 skipped; Ruff passed; 78 Python files compiled. Merge conflicts were documentation sections only.
  • Recorded in docs/ERRORS.md: a disposable real-Docker proof completed all 24 planned solver runs and all 12 matched pairs with zero failures on one immutable image; identical frozen seed solvers tied in all 12 comparisons and a second evaluation request was rejected as consumed.

Every holdout result stays labeled synthetic and descriptive. It cannot publish, promote, feed prompts, or claim performance on unseen public benchmarks.

@ucsandman
ucsandman merged commit 921e138 into master Sep 15, 2026
4 checks passed
@ucsandman
ucsandman deleted the codex/discovery-holdout-20260915 branch September 15, 2026 02:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant