Let a ledger say which shapes must never be explained - #366
Open
derek73 wants to merge 11 commits into
Open
Conversation
An exclusion is the absorption bug pointed the other way: a rule matching too widely turns a regression into a classified diff, and an exclusion matching too widely turns a legitimate classification into UNEXPLAINED, which reads as catastrophic regression rather than as a bad exclusion. The checks mirror validate_rules', plus one it has no equivalent for -- an entry whose examples do not match its own name_regex protects nothing while looking complete. The examples guards are ordered so a bare string reports the type error rather than 'no examples' -- the misleading half of a mistake a TOML author is likely to make.
Consulted before the rules and winning outright, which is what makes exclusions monotone: an entry only ever removes a name from classification, never moves it between rules, so its blast radius is exactly the names it captures and does not depend on rule order. That is the property the reordering ideas in #328 could not offer.
ASCII parens mark nicknames, maiden names, suffixes and credentials alike, and no name_regex tells them apart. The promise being encoded is about the NICKNAME reading, so a name-only exclusion would also silence 'Jenny (Johnson) Baker' and 'Lon (Jr.) Williams' -- measured, both in the corpus, both in areas under active development (#335, suffix in parens). Hiding a regression there is the exact failure this feature exists to prevent. So an exclusion narrows the same two ways a rule does, and the loops now have the same shape. Typographic delimiters need none of this, which is why feat(#273)'s own rule can be a bare character class.
Both promises were false. 'John Smith, Ph. D.' was claimed by
fix(comma-family) on file order and by fix(suffix-routing), which has
no name_regex and reaches every name; the ASCII nickname pairs were
claimed by the latter.
The ASCII entry is narrowed twice and both narrowings are measured. A
bare [("'] class reaches 47 corpus names, all 47 currently claimed --
every credential in parens. Even the narrow shape reaches 'Jenny
(Johnson) Baker' and 'Lon (Jr.) Williams', whose parens are a maiden
name and a suffix, so the entry names the nickname reading and leaves
those classifiable.
The Ph. D. entry is ASCII-anchored because an unrestricted trailing
anchor also caught '田中さん, Ph. D.', a real CJK classification --
measured, it took the gate to unexplained: 1. And 'John Smith, Jr.
Ph. D.' is deliberately not an example: its order diff is an intended
change (fix(credential-pair-order)), not the parity shape this
protects.
The prose now points at the entries rather than restating a guarantee
it could not give.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The harness runs by hand at release; this runs on every push, so it fires when a rule is widened to absorb a protected shape rather than when someone finally runs the comparison. That gap is why #328 went unseen for a year. Every non-empty subset of the seven roles, because the promise is 'if it ever starts diffing', not 'if it diffs the way I guessed'. An entry naming `fields` is asked only about the subsets it covers, so the ASCII-pairs entry is held to the nickname reading and a suffix diff on the same name stays somebody else's business.
test_default_baseline_has_a_ledger_and_nothing_else_in_it asserted the open cycle defines nothing but `change`, which would have rejected the new key outright -- and it still catches a mistyped one. The harness README documents the grammar beside [[change]]'s, including why `fields` exists: ASCII parens mark nicknames, maiden names, suffixes and credentials alike, so a name-only exclusion would silence shapes in areas under active development. Release step 8 mentions the section, and notes it needs no enrollment -- the guard discovers entries from the ledger itself.
The existing guard checks an exclusion is WIDE enough -- its examples stay unclassified. Nothing checked it was NARROW enough, and the two failures are not symmetric: an over-wide exclusion silences diffs a rule should explain, invisibly, because the guard only ever looks at names the entry lists as examples. Measured while verifying the branch: deleting fields = ["nickname", "middle"] from the ASCII-pairs entry passed every other check here. The entry then refuses ANY diff on the ten corpus names it captures -- including 'Jenny (Johnson) Baker' and 'Lon (Jr.) Williams', whose parens are a maiden name and a suffix, both under active development (#335, suffix in parens). Nothing failed, because none of those names diffs today and none of them is an example. So an entry that names `fields` must leave something behind: some corpus name it captures must still be classifiable on a reading outside them. Both failure directions verified -- deleting the key and widening it to cover everything the entry reaches each turn the suite red. This is the check the cut Task 5 was reaching for and would have missed. Recorded corpus REACH is name-based, and dropping `fields` changes no name.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #366 +/- ##
=======================================
Coverage 98.50% 98.51%
=======================================
Files 44 44
Lines 2883 2894 +11
=======================================
+ Hits 2840 2851 +11
Misses 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
test_no_rule_claims_a_shape_the_ledger_excludes could not fail. It asked whether a rule claims a protected shape WITH the exclusion active, and classify() consults exclusions first and returns None -- and it only asked about subsets the exclusion covers, so the answer was None by construction. Measured: prepending a catch-all rule, and deleting every rule in the ledger, both left it green across all 387 subsets. Its only reachable failure was an example that stops matching its own entry, which validate_exclusions already rejects at startup and test_validate_exclusions_accepts_the_shipped_entries already runs over every shipped ledger. So the 387-way loop was a weaker restatement of a check one file away. That is the tenth inert measurement recorded in this tree, and it shipped as this PR's headline. I verified it during implementation and was fooled the same way: the probe narrowed the EXCLUSION so its examples stopped matching, the guard went red with a message naming an absorbing rule, and it looked right. A plausible red is not proof -- it has to come from mutating the thing the guard claims to watch. So ask with exclusions OFF and record the answer. _EXCLUSION_EFFECT holds, per entry, which rules would claim each protected reading, plus the count and digest of the corpus names it captures. The first covers the #328 event -- a rule widened to reach a protected shape -- which is otherwise invisible everywhere, because the exclusion correctly hides it from the harness. The second covers the opposite drift, which nothing watched: dropping the Ph. D. entry's ASCII anchor kept the whole suite green while taking a bare run to unexplained: 1. All three mutations now fail, two of which the old guard missed entirely. The README and release step 8 said the old thing; both now describe what is actually checked, and step 8 no longer claims exclusions need no enrollment -- the record IS one.
Both [[never]] entries were narrower than their own promise. The trailing-'Ph. D.' anchor was ASCII-bounded, so it protected 'Jose Smith, Ph. D.' and not 'José Smith, Ph. D.'. The split-token healing is script-independent; the bound exists only to keep the entry off '田中さん, Ph. D.', whose diff is a real fix(cjk-comma-compound). Cut at U+0250 instead -- the threshold _is_latin_only already uses -- which covers Latin-1 and Latin Extended-A and still excludes CJK. Corpus reach is unchanged, so the recorded digest is too. The ASCII-pairs shape required a word character left of the delimited run, so 'Xyz. (Bud) Smith' and 'Cherice J. (Johnson) Williams' -- a middle initial before the parens, which is ordinary -- fell outside a comment describing "a run BETWEEN two name tokens". Allow a period there: 10 corpus names to 13, harness unchanged at 107/0. Widening further is the wrong direction and the comment now says why. Leading and trailing pairs take it to 34 names, and the 21 it gains are the trailing credentials -- 'Andrew Perkins (JD)' and its kin -- which is what the medial cut is for. So the feat(#273) promise is kept for medial pairs only, and both sites now say so rather than reading as complete. Three further claims corrected against measurement: the 47 names a bare [("'] class reaches are not "every credential in parens" (9 are; 11 match a bare apostrophe, "Brian O'connor" and kin); naming fields does not leave 'Lon (Jr.) Williams' classifiable, since it parses its suffix into `middle`, which the entry names; and reconciling 'John Smith, Jr. Ph. D.' cannot be done "by giving the entry `fields`" -- the two renderings differ in `suffix` alone, the one field the promise is about. Also take both pins' subset universe from compare._RULE_FIELDS rather than Role, so a subset containing '_ambiguities' is built at all. A rule with fields = ["_ambiguities"] and no name_regex claims the ambiguity-only reading of every protected shape; only the wider universe grows absorbed_by and fails. It does not close the matching hole on the exclusion side, and the docstring says which is which. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three sites said an exclusion that reaches too far would hide a regression in an area under active development. It is the opposite: classify() returns None for an excluded name, main() counts it unexplained, and the run exits non-zero. validate_exclusions' own docstring says so nine lines above the sentence that contradicted it. The real cost is worth stating on its own terms, so it now is: such a name becomes permanently unexplainable, so an intended change there can never be recorded and every release blocks on the same false alarm. Loud, but uselessly so. Two sites also said the promises had been prose "for a year". The harness is three weeks old: the trailing-'Ph. D.' promise shipped with it, the feat(#273) one three days later, and #328 found them both a fortnight after that. Say when they were written instead of guessing how long they stood. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A second review round over the corrections themselves. All five are
prose; no behavior changes, and the gate is unmoved at 107/0.
"Silencing all 47 would turn 47 legitimate classifications into
UNEXPLAINED" was the biggest, and the first review round already
raised it -- I rewrote the surrounding block and left the number.
Measured by adding exactly that exclusion and running the gate: three
names, all CJK, and the other 44 do not diff at all. The real cost is
those three plus 44 shapes pre-silenced, which counting names both
overstates and hides.
'Senator "Rick" Edmonds' was cited as a leading pair the entry fails
to protect. It is medial and IS captured -- 'Senator' supplies the
left flank. The corpus holds a separate '"Rick" Edmonds', which is the
genuinely unguarded one; name that instead, and note the trap.
"The 21 it gains are almost entirely that family" read as "credentials"
against its own antecedent. 20 of 21 put the delimiter last, but only 8
are credentials and 6 are trailing NICKNAMES -- the reading this entry
exists to protect. So the medial cut is about position, not about
credentials: at the trailing position a nickname and a credential wear
the same shape, and protection and over-reach arrive together. Whether
`fields` already makes the wider shape safe is now flagged as an open
question rather than answered by assertion.
"Only 9 of those 47 are parenthesised credentials" is 8. Nine is the
count a name_regex-carrying rule reaches -- two measurements conflated.
'Xyz. (Bud) Smith' was the example for a middle initial; it parses
'Xyz.' as a title. 'Cherice J. (Johnson) Williams' is the real one.
Also: the _EXCLUSION_EFFECT preamble claimed in the present tense that
dropping the Ph. D. anchor keeps the suite green -- true before this
pin existed, self-refuting in the tree that contains it. And the
fields-deletion note had the mechanism backwards: absorbed_by GROWS
from () to ('fix(suffix-routing)',), it does not see a reading go
unclaimed. Both now say which era they describe.
Two limits named rather than left implicit: absorbed_by records only
the first rule matching each subset, so a fully shadowed rule does not
move it; and an entry carrying `fields` is asked about only the
subsets those cover, which is where the 387 comes from.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #328.
expected_since_1.4.0.tomltwice promised in prose that a shape would stay unclassified, so a regression there would fail the harness. Both promises were false. Measured againstmasterwith the real ledger and_sorted_rules:John Smith, Ph. D.fix(comma-family),fix(suffix-routing)John Ph. D.fix(suffix-routing)John "Jack" Kennedyfix(suffix-routing)fix(suffix-routing)is the more general mechanism: afields-only rule with noname_regexat all, so it reaches every name in the corpus.Why #328's four directions don't work
The issue proposes a specificity order, reporting the runner-up, a non-absorbing flag, or asserting the promise. Three assume a narrower correct rule exists to promote. None does — the ledger deliberately has no rule for these shapes, which is the whole promise.
Measured rather than argued (corrected in review — an earlier version of this section claimed "exactly one" label changed, which is not reproducible under any reading of the sort). Patching
_sorted_rulesto sort by field width then regex length, and running the full harness at baseline 1.4.0 both ways, moves five rule populations:fix(suffix-routing)fix(cjk-comma-compound)fix(cjk-honorific-suffix)fix(#271/#272/#298)fix(cjk-delimited-nickname)The gate stays at
107 intentional / 0 unexplainedeither way, so nothing is caught or missed — the labels just move. And they move the wrong way: ranking by field width promotes the fields-only rules, so 17 more names fall to the genericfix(suffix-routing)and the CJK-specific rules are drained. That is the absorption #328 is about, made worse. Keeping the existing tier first and sorting within it changes no label at all.Reordering rearranges a set containing no right answer. The gap is not precedence but vocabulary: the ledger could say "this diff is intended, and here is the rule that explains it" and could not say "this shape must never be explained."
[[never]]classify()consults exclusions before rules; a match returnsNone. That order makes them monotone — an entry only ever removes a name from classification, never moves it between rules — so its blast radius is exactly the names it captures, independent of rule order. That is the property the reordering ideas could not offer.examplesis required and load-bearing: a protected shape need not be in any corpus (John "Jack" Kennedyisn't), so the entry carries its own test data.fields, and why it was neededThe second entry protects ASCII nickname pairs — and ASCII parens mark four different things:
A bare
[("']class reaches 47 corpus names, all 47 currently claimed. Even a narrow shape reaches the ten above. So an entry may narrow byfieldstoo, naming the reading it protects — the same subset test rules use. The nickname reading is refused; a suffix diff on the same name stays classifiable. Typographic delimiters carry no such ambiguity, which is whyfeat(#273)'s own rule can be a bare character class and its exclusion cannot.Guards
validate_rules, plus one it has no equivalent for: an entry whoseexamplesdon't match its ownname_regexprotects nothing while looking complete.test_no_rule_claims_a_shape_the_ledger_excludes— every example, every non-empty subset of the seven roles (387 subsets today). Fires when a rule is widened, on every push, rather than at the next hand-run of the harness. That gap is whyexpected_changes.tomlpromises the harness will fail on aPh. D.regression; it won't #328 went unseen for a year.test_a_fields_narrowing_actually_narrows_something— the other direction, found while verifying this branch. Deletingfieldsfrom the ASCII entry passed every other check: the entry then refuses any diff on ten corpus names including the maiden and suffix shapes, and nothing failed because none of them diffs today and none is an example. Both failure directions now verified.Two things worth knowing
田中さん, Ph. D., a real CJK classification — measured, it took the gate tounexplained: 1.John Smith, Jr. Ph. D.is deliberately not an example. Its order diff is an intended change (fix(credential-pair-order)incases.py), not the parity shape this protects. The regex still over-reaches it; no corpus contains it, and refusing is the safe direction, so the entry documents that as an accepted limitation.Full suite 3117 passed; mypy and ruff clean; all three baselines report
unexplained: 0(0 / 107 / 89 intentional).🤖 Generated with Claude Code