Skip to content

test(gate): add unit tests for the evaluation gate decision core#122

Open
nankingjing wants to merge 1 commit into
microsoft:mainfrom
nankingjing:test-gate-module
Open

test(gate): add unit tests for the evaluation gate decision core#122
nankingjing wants to merge 1 commit into
microsoft:mainfrom
nankingjing:test-gate-module

Conversation

@nankingjing

Copy link
Copy Markdown

What

Adds tests/test_gate.py — 22 unit tests for skillopt/evaluation/gate.py, the optimizer's validation-gate decision core (select_gate_score, evaluate_gate, GateResult).

Why

skillopt/evaluation/gate.py is the accept/reject decision at the heart of the ReflACT training loop — the analog of validation-based model selection / early stopping. It decides whether a candidate skill becomes the new current skill and whether it is promoted to best-so-far. Despite being pure, side-effect-free logic (the trainer owns all I/O), it currently has no dedicated test coverage — it is only reached indirectly through engine/trainer.py, so a regression in the gate's branching or metric math would ship silently.

This is in the spirit of #94 (bring evaluate_gate under direct test), applied to the main optimizer's gate.

Coverage

select_gate_score:

  • hard / soft / default-metric projection
  • mixed weighting (default weight, custom weight, w=0→hard, w=1→soft)
  • out-of-range mixed_weight clamping to [0, 1] (both directions)
  • return type is float; unknown metric raises ValueError

evaluate_gate — all three actions and their exact resulting state:

  • accept_new_best (beats current and best; best_step advances to global_step)
  • accept (beats current, not best; best-so-far incl. best_step preserved) — the branch only reachable when current_score < best_score
  • reject (does not beat current; all state preserved)
  • strict-> semantics: a tie with current_score is rejected (no lateral moves); a tie with best_score above current is an accept, not a new best
  • metric routing via cand_soft (soft/mixed), and that the default hard metric ignores cand_soft

GateResult: field wiring and frozen-dataclass immutability.

Plus a behavioral invariant test documenting that when s_cur == s_best at the start (the trainer's initialization), the two stay locked together and the accept branch is never taken — capturing the design property discussed in #113.

Verification

Tests are pure-logic (no LLM, rollout, or network). Ran the full file against the unmodified gate.py: 22 passed. Style matches the existing suite (tests/test_scoring.py, tests/test_types.py): class-grouped, pytest.approx for float math, docstrings per group.

No production code changed — tests only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant