Skip to content

Commit 7eee5d0

Browse files
authored
Stabilize the objective coordinator routing test cohort (#307)
The objective coordinator routing test intermittently returns `deferred` because its positive fixture has only 68 holdouts. The real coordinator replenishes this pool below its floor of 80, introduces randomly ordered negative controls, and can select enough of them to make the eight-item experiment underpowered. Prepare a second genuine scenario pool and apply the fixture’s existing positive-only policy before this one test, keeping at least 88 holdouts available. The legacy-forensics prohibition, resume assertions, production statistics, replenishment behavior, and independent negative-control tests remain unchanged. Validation: autonomous/evolution suites passed 74 cases, with one existing sandbox probe skipped because user namespaces are disabled; the statistics test passed; six fresh-UUID repetitions of the coordinator test passed. Independent review found no issues and separately passed all 37 autonomous tests. The rebased branch contains only the nine-line test change.
1 parent 8ae7b8d commit 7eee5d0

1 file changed

Lines changed: 9 additions & 0 deletions

File tree

‎test/test_autonomous_evolution.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -403,6 +403,15 @@ def test_objective_coordinator_never_calls_legacy_dual_model_forensics(autonomou
403403
from infermatrix_copilot.improve import coordinator
404404
from infermatrix_copilot.engine.steps import improve
405405
st, store, model, worker = autonomous
406+
# Keep this large-effect routing test's positive cohort above the
407+
# coordinator's replenishment floor, including after its eight holdouts
408+
# are consumed. Negative controls are independently asserted above.
409+
objectives.prepare(st, store, workflow=objectives.ENGINE)
410+
path = objectives.data_root(st) / objectives.ENGINE / 'dataset.json'
411+
manifest = json.loads(path.read_text())
412+
manifest['items'] = [r for r in manifest['items'] if not r['objective'].get('negative')]
413+
artifacts.atomic_json(path, manifest)
414+
assert sum(r['split'] == 'holdout' for r in manifest['items']) >= 88
406415
def forbidden(*args, **kwargs): raise AssertionError('legacy model or human benchmark requested')
407416
monkeypatch.setattr(improve, '_forensics', forbidden)
408417
result = coordinator.run(st, store, workflow=objectives.ENGINE, llm=model, sandbox=worker)

0 commit comments

Comments
 (0)