Skip to content

fix(eval): M3 selectivity gate is exactly <= 1/3 (#209 amendment) - #215

Merged
leo-aa88 merged 2 commits into
mainfrom
fix/m3-selectivity-one-third-209
Sep 27, 2026
Merged

leo-aa88 merged 2 commits into
mainfrom
fix/m3-selectivity-one-third-209

Conversation

@leo-aa88

Copy link
Copy Markdown
Member

This encodes the M3 protocol amendment recorded on #209 before any capture exists (amendment). The selectivity bar is now median candidate fraction ≤ 1/3. It was "≤ 0.33".

Why

  • The bound means "at least as selective as M1 on otel-fresh".
  • The evaluator (feat(eval): frozen M3 evaluator for the structural model (#209) #214) reproduces that reference value as exactly 1/3, and "0.33" was it rounded to two decimals.
  • Read literally, 0.33 fails M1's own reference value and draws an arbitrary line: 0.331 passes and 0.334 fails.

Change

  • CANDIDATE_FRACTION_MAX = Fraction(1, 3).
  • The gate takes the median over Fraction(|localization|, |services|), so a result of exactly 1/3 can never fail on float rounding. For example, the median of 1/4 and 5/12 is exactly 1/3.
  • The report keeps the float median and adds median_candidate_fraction_exact, e.g. "11/30". The criterion prints <= 1/3.

Nothing else in the protocol or the evaluator changes. After capture, no criterion changes.

Tests

The test that pinned "exact 1/3 fails 0.33" is replaced by four tests:

  • exact 1/3 passes;
  • 1/3 as an even-count median passes;
  • 0.334 fails and 0.331 passes;
  • the exact median is reported.

make lint ✅ · make test-unit ✅ (1587). A smoke run on trace-loc serializes and renders correctly.

Eval delta

None on the product. This touches src/eval/ only; src/core/ is unchanged. On the spent otel-fresh corpus the frozen arm's fraction is 0.139, so the gate verdict is unchanged there. The only behaviour change is a median landing in (0.33, 1/3].

Refs #209.

🤖 Generated with Claude Code

The M3 protocol's "median candidate fraction <= 0.33" was M1's otel-fresh
value rounded to two decimals. The evaluator reproduces that value as
exactly 1/3. Read literally, the bound fails its own reference and draws
an arbitrary line (0.331 passes, 0.334 fails). The protocol was amended
on #209 before any M3 capture exists: <= 1/3, in exact arithmetic.

The gate now computes the median over Fraction(|localization|,
|services|), so it is never decided by float rounding. For example, the
median of 1/4 and 5/12 is exactly 1/3. The float median is still
reported, and the exact ratio is added alongside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leo-aa88

Copy link
Copy Markdown
Member Author

/review

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Open in Web View Automation 

Sent by Cursor Automation: Code Reviewer

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST CHANGES

The comparison in decide is exact, and the new unit tests actually pin the float-rounding case. The decision record those tests never look at does not contain the value that was compared. CI is green. This is not a failing-test problem.

1. The criterion record is not the comparison the gate ran

BLOCKING.

decide computes Fraction median, compares that to Fraction(1, 3), then writes a different object into generalization_criteria: float(median) and the string "1/3". render_markdown — the as-is post for #209 — runs every criterion value through _num, i.e. two decimal places, and never reads median_candidate_fraction_exact.

So the three medians this PR exists to separate all print as the same line:

  • 1/3 passes, posts 0.33 (pre-registered <= 1/3)
  • 331/1000 passes, posts 0.33 (pre-registered <= 1/3)
  • 334/1000 (= 167/500) fails, posts 0.33 (pre-registered <= 1/3)

The table cell is the same 0.33. The exact ratio is on the arm dict inside the JSON and then dropped on the floor by the publisher. Before this patch the criteria triple was the pair of operands the branch used. It is not anymore: 0.334 <= "1/3" does not even type-check, and the Fraction that did get compared is not stored.

ಠ_ಠ You refused to gate on the float, then stored the float as the official criterion, so _num could keep rounding it back to the literal the amendment just retired.

Record the exact ratio in the criterion value and print that ratio in the verdict line and the selectivity row. The float can stay beside it. Add a render test for 1/3, 331/1000, and 334/1000 that asserts the posted lines are not identical.

VERDICT

The gate boolean matches the #209 amendment. The audit record does not. This is an abstraction failure: two representations of one threshold, and the one the protocol posts is the rounded one. Do not merge until the posted criterion is the ratio decide compared.

Open in Web View Automation 

Sent by Cursor Automation: Code Reviewer

Comment thread src/eval/structural_m3.py Outdated
…pared (#215 review)

decide() compared the exact Fraction median to Fraction(1, 3), but it
recorded float(median). render_markdown then printed that float to two
decimals. So 1/3 (pass), 331/1000 (pass) and 167/500 (fail) all posted
the same "0.33", which is the literal the amendment retired.

The criterion triple now records the operands the gate compared: the
exact ratio "n/d" against "1/3". The verdict line and the selectivity
row print that ratio, with its decimal beside it (e.g. "167/500
(≈0.3340)"). A render test asserts that the three posted lines differ.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leo-aa88

Copy link
Copy Markdown
Member Author

/review

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Open in Web View Automation 

Sent by Cursor Automation: Code Reviewer

@leo-aa88
leo-aa88 merged commit ce20897 into main Sep 27, 2026
7 checks passed
@leo-aa88
leo-aa88 deleted the fix/m3-selectivity-one-third-209 branch September 27, 2026 00:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant