Redesign CombineFrontmatter name matching, merging and YAML rewriting (fixes #175 #176 #177 #178 #162 #167 #154 #150 #151 #153 #161)

[matt-edmondson]

Covers: #175, #176, #177, #178, #162, #167, #154, #150, #151, #153, #161, and the PropertyMerger half of #113. These are one bug cluster with shared root causes, so fix them through this design, not one at a time. Each covered issue is linked as a sub-issue here; close them from the PR(s) that implement the matching step.

Decision (maintainer, 2026-09-30)

The design below is approved as written, with these calls:

• The default merge behaviour changes: differing scalar values are kept rather than dropped, and no value is ever silently lost.
• The existing tests that pin the old behaviour (listed under API and breaking impact) are rewritten to the invariants below, with the reason stated in each commit. This release is [major].
• The earlier decisions stand: Duplicate keys in one block keep the last value, although YamlSerializer's code and comment say the first occurrence is preserved #154 first duplicate key wins (2026-09-28); Delegate property-name fuzzy matching to ktsu.FuzzySearch #113 Maximum uses ktsu.FuzzySearch per-word scoring (2026-09-28).

---

Scope

In scope (all open, all caused by the three root causes below):

• Aggressive and Maximum merging delete properties whose name merely contains another key: subtitle, tagline, username, author_url, canonical_url and summary_image lose their values #175 Aggressive/Maximum pair keys by raw substring and drop a value (subtitle, tagline, username, property12).
• Maximum merging can wipe out entire list properties: two merge groups write the same key and the second overwrites the first #176 Maximum: two merge groups write one key; the second overwrites the first's list.
• Default CombineFrontmatter renames short keys to longer, unrelated standard fields: id → video, key → keywords, list → unlisted, seo → description_seo #177 Default Standard naming renames a key to a standard name that contains it (id→video, key→keywords).
• Rewriting a document changes unquoted numbers: version: 1.10 becomes 1.1, zip: 01234 becomes 1234, lat: 51.507351 becomes 51.50735 #178 Re-serializing rewrites plain numbers (1.10→1.1, 01234→1234, 0x1F→31).
• Default Standard naming renames compound keys to a different field: author_email → author, previous_version → version #162 Default Standard naming renames a key to a standard name it contains (author_email→author).
• Maximum merging depends on key order: name_of_author + author_name merge in one order and stay separate in the other, because each key is scored against itself #167 Maximum scores a key against itself, so grouping depends on key order and is not transitive.
• Duplicate keys in one block keep the last value, although YamlSerializer's code and comment say the first occurrence is preserved #154 Duplicate keys in one block: last wins. Decision (2026-09-28): first wins, via YamlStream.
• Standard naming lowercases the canonical redirectFrom/redirectTo keys to redirectfrom/redirectto, and StandardOrder cannot place them #150 Standard naming lowercases canonical redirectFrom/redirectTo; StandardOrder.Compare can't find them.
• FrontmatterOrder.Sorted ignores standard keys that differ only in case (e.g. Title, Date) under AsIs naming, contradicting StandardOrder.Compare #151 Sorted places standard keys case-sensitively, disagreeing with StandardOrder.Compare.
• Delegate property-name fuzzy matching to ktsu.FuzzySearch #113 (PropertyMerger half) Decision (2026-09-28): adopt ktsu.FuzzySearch per-word scoring in Maximum; pin it with a test.
• Adjacent, taken because step 1 rewrites the code they touch: A | block scalar loses its trailing newline when it is the last key in the frontmatter (and CombineFrontmatter rewrites it as >-) #153 (| scalar loses its newline, then comes back as >-) and Frontmatter whose lines are uniformly indented is treated as unreadable, so AddFrontmatter silently drops the new properties #161 (block.Trim() breaks uniformly indented YAML).

Excluded, each fixable on its own after this lands: #170 (comments: the node model keeps flow style but not comments; needs a text-splice path),
#169, #160, #158, #157, #152 (splitter and newline issues), #148 (caches; step 1 deletes ParsedYamlCache anyway),
#159 (the new parser should reject AnchorAlias events, which is cheap to do in step 1 but belongs to its own issue and test).
Also out of scope: auditing the alias table (type→layout, group→categories and name→title are questionable Conservative renames that nobody has reported).

Root causes

1. Name matching by substring or fuzzy score instead of an explicit relation. NameStandardizer.FindStandardPropertyMatch
   and PropertyMerger.FindBasicCanonicalName both accept a.Contains(b) || b.Contains(a), and IsInSameCategory passes when only
   the target is a known name. Containment cannot tell canonical_url (decoration) from author_email (qualification) or
   id from video. Causes Default Standard naming renames compound keys to a different field: author_email → author, previous_version → version #162, Aggressive and Maximum merging delete properties whose name merely contains another key: subtitle, tagline, username, author_url, canonical_url and summary_image lose their values #175 and Default CombineFrontmatter renames short keys to longer, unrelated standard fields: id → video, key → keywords, list → unlisted, seo → description_seo #177. The length floor (A key normalizing to a single character is containment-matched against unrelated properties, reattributing and losing values #128) and the empty-string guard (Fold the two NormalizePropertyName helpers, and stop decoration-only keys being silently renamed #124) only patch this.
2. Merge groups are not a partition. Each key picks a canonical name on its own (GetCanonicalName per key, Maximum including the key itself),
   and the groups are then keyed by those picks. Nothing makes the relation symmetric or transitive, and nothing stops two groups writing one output key.
   MergePropertyGroup also keeps only the first scalar of a group, so even the default Conservative strategy drops creator: Bob beside author: Alice,
   and abstract beside desc. Causes Maximum merging depends on key order: name_of_author + author_name merge in one order and stay separate in the other, because each key is scored against itself #167, Maximum merging can wipe out entire list properties: two merge groups write the same key and the second overwrites the first #176 and the value loss in Aggressive and Maximum merging delete properties whose name merely contains another key: subtitle, tagline, username, author_url, canonical_url and summary_image lose their values #175.
3. YAML round-trips through CLR values. WithAttemptingUnquotedStringTypeDeserialization turns plain scalars into Byte/Int16/Single,
   and the serializer re-formats them. Deserialize<Dictionary> collapses duplicates before the code sees them, and scalar and collection style is discarded.
   Causes Rewriting a document changes unquoted numbers: version: 1.10 becomes 1.1, zip: 01234 becomes 1234, lat: 51.507351 becomes 51.50735 #178, Duplicate keys in one block keep the last value, although YamlSerializer's code and comment say the first occurrence is preserved #154 and A | block scalar loses its trailing newline when it is the last key in the frontmatter (and CombineFrontmatter rewrites it as >-) #153 (with the Trim() in ExtractFrontmatterObjects).
   The canonical-name spelling problem (Standard naming lowercases the canonical redirectFrom/redirectTo keys to redirectfrom/redirectto, and StandardOrder cannot place them #150, FrontmatterOrder.Sorted ignores standard keys that differ only in case (e.g. Title, Date) under AsIs naming, contradicting StandardOrder.Compare #151) is a small version of the same thing: names are compared lowercased and written back lowercased
   instead of being looked up.

Design

A. One name relation (PropertyNameMatcher, replacing the matching halves of both classes)

• CanonicalSpelling(name): case-insensitive lookup in StandardOrder.PropertyNames. It returns the list's own spelling, so redirectfrom gives redirectFrom (Standard naming lowercases the canonical redirectFrom/redirectTo keys to redirectfrom/redirectto, and StandardOrder cannot place them #150).
  StandardOrder.Compare and SortFrontmatterProperties use the same lookup (FrontmatterOrder.Sorted ignores standard keys that differ only in case (e.g. Title, Date) under AsIs naming, contradicting StandardOrder.Compare #151).
• Label(key, strategy) is a pure function of one key. Keys with equal labels are equivalent:
  • None: the key itself.
  • Conservative: PropertyMappings.All[key] if present, otherwise the key.
  • Aggressive and Maximum: the alias of the key, else the alias of Normalize(key) with one trailing s folded, else that normalized form.
    Normalize strips only the known prefix and suffix lists, so custom_title→title and page_related~related merge.
    subtitle, tagline, author_url, property12 and step10 never do. There is no containment anywhere.
• Standard naming (NameStandardizer) renames a key only when its Aggressive-level label equals a standard name, spelled by CanonicalSpelling.
  The existing "only when the target is free" rule stays. So id, key, author_email, image_credit, subtitles_track and meta_ag_field are all kept as written.
• Because matching is by label, a key's result no longer depends on the other keys, so PropertyNameCache is keyed correctly by construction.

B. Groups form a partition, and exactly one group owns each output key (MergePlanner)

1. The classes are the keys grouped by Label. That is an equivalence relation, so the classes are disjoint and independent of key order.
2. Maximum only: join classes whose representatives are mutual best matches under per-word FuzzyRanking scoring (Delegate property-name fuzzy matching to ktsu.FuzzySearch #113).
   A class is never scored against itself, a minimum score is required, and ties break by PreferredName.
   A class has at most one best match, so mutual-best edges pair classes without forming chains, and the result is still a partition (Maximum merging depends on key order: name_of_author + author_name merge in one order and stay separate in the other, because each key is scored against itself #167).
   CalculateWordMatchScore becomes the boolean SharesRelatedWord gate, as the Delegate property-name fuzzy matching to ktsu.FuzzySearch #113 decision says.
3. Group name: the canonical label if the group has one (spelled canonically), else PreferredName(members) (known name, then shortest, then ordinal).
   A key whose text equals another group's label is in that group by definition, so names cannot collide. Write still asserts uniqueness
   and throws InvalidOperationException on a clash. A bug then fails loudly instead of losing data (Maximum merging can wipe out entire list properties: two merge groups write the same key and the second overwrites the first #176).
4. Value rule, applied per group:
   • lists: append them, dropping only items an earlier list supplied (the CombineFrontmatter silently removes repeated items from every list, even when no keys are being merged #142 rule);
   • scalars whose text is identical: collapse to one;
   • anything else (scalars that differ, mixed kinds, a null, a mapping): no merge, every member keeps its own key and value.
     A lone key always keeps its own name. Renaming is the standardizer's job.

What each strategy may do: None changes nothing. Conservative merges the alias table only. Aggressive also merges keys that differ only by
case, separators, known decoration or a plural. Maximum also joins mutual-best fuzzy pairs. None of them may drop a value or change a value's text.

C. Scalars preserved verbatim (YamlDocumentModel)

• Parse each block with YamlStream (representation model), from the untrimmed block text, keeping the final line ending (A | block scalar loses its trailing newline when it is the last key in the frontmatter (and CombineFrontmatter rewrites it as >-) #153, Frontmatter whose lines are uniformly indented is treated as unreadable, so AddFrontmatter silently drops the new properties #161).
  Walk the YamlMappingNode and skip keys already seen, so the first occurrence wins (Duplicate keys in one block keep the last value, although YamlSerializer's code and comment say the first occurrence is preserved #154).
  The pipeline carries List<(string Key, YamlNode Value)> and never a CLR value.
• Emit through YamlStream.Save/Emitter. Each document-derived scalar is written with its original Value and Style,
  and each sequence and mapping with its original Style, so 1.10, 01234, 0x1F, | stay as written (Rewriting a document changes unquoted numbers: version: 1.10 becomes 1.1, zip: 01234 becomes 1234, lat: 51.507351 becomes 51.50735 #178, A | block scalar loses its trailing newline when it is the last key in the frontmatter (and CombineFrontmatter rewrites it as >-) #153).
• Caller-supplied CLR values (the AddFrontmatter/ReplaceFrontmatter dictionaries) become nodes through the existing Serializer
  (serialize, then parse to a node), so the Round-tripping strips quotes, so "true", "1.0", "007" and "null" come back as bool/number/null, and empty values become '' #137 quoting of "007" and "true" is kept.
• The public read path (ExtractFrontmatter, TryParseYamlObject) types a plain scalar only when the typed value reproduces its text exactly:
  null/~/empty → null; YAML 1.2 core true/false → bool; an integer text that long formats back identically → long; the same for a float and double;
  anything else stays a string. So count: 3 → 3L, draft: false → bool, version: 1.10 → "1.10", zip: 01234 → "01234". The Round-tripping strips quotes, so "true", "1.0", "007" and "null" come back as bool/number/null, and empty values become '' #137 tests keep passing.
• Delete ParsedYamlCache: nodes are mutable, and Process-wide caches keep every processed document forever, so long-running hosts leak memory #148 asks for its removal anyway.
  Keep DeepCloneValue/ConvertValue only if the public read path still needs them.

Invariants (each one is a property test)

1. No value is lost. For every strategy and naming mode, the multiset of scalar texts in the input header equals the multiset in the output header.
   The one exception is identical scalars collapsed within a merge group, and those collapses are counted and asserted. The same holds for list items, modulo the CombineFrontmatter silently removes repeated items from every list, even when no keys are being merged #142 rule.
2. A document with only standard keys, already sorted, is byte-identical after CombineFrontmatter under the default options.
3. CombineFrontmatter(AsIs, AsIs, None) is the identity on every value's text and style.
4. Order independence: permuting the keys of an input changes only the key order of the output, never its key set or its key→value pairs.
5. Idempotence: running Combine on its own output returns it unchanged.
6. One owner: no output key is written twice (the Write assertion).
7. Never a stranger's name: an output key is either an input key or a canonical name whose group contains an input key with that label.

API and breaking impact

• No signature changes. Behaviour changes that callers can see:
  • fewer renames under the default Standard naming;
  • fewer merges under Aggressive and Maximum;
  • under the default Conservative strategy, differing scalar aliases (author+creator, desc+abstract) are now both kept;
  • ExtractFrontmatter returns long/double/string where [minor] Keep scalar types and nulls through a frontmatter round trip #174 returned Byte/Int16/Single;
  • StandardOrder.Compare changes for mixed-case names.
• Commit tag: [major]. Public return types and default output both change, and Enums.cs docs must be rewritten to the "may do" list above.
• Tests that deliberately pin old behaviour, to be rewritten with the reason stated in each commit:
  • FuzzyRankingTests (subtitles_track, review_status_flag);
  • SingleCharacterKeyTests.StandardizePropertyNames_StillMatchesATwoCharacterFragment;
  • NameStandardizerTests.StandardizesSimilarProperties (it currently asserts abstract is lost);
  • MergeStrategyIsolationTests line 30 (differing scalars are no longer merged; use equal values or lists);
  • MergeSimilarProperties_WithContainmentSimilarity_MapsCorrectly.
    These reversals are signed off (see the decision above).

Test plan

• Data-driven regression rows copied verbatim from each issue: the Aggressive and Maximum merging delete properties whose name merely contains another key: subtitle, tagline, username, author_url, canonical_url and summary_image lose their values #175 table under Aggressive and Maximum, property0..199 keeping 200 keys,
  the Maximum merging can wipe out entire list properties: two merge groups write the same key and the second overwrites the first #176 repro keeping a1 b1 c1, every key in the Default CombineFrontmatter renames short keys to longer, unrelated standard fields: id → video, key → keywords, list → unlisted, seo → description_seo #177 table plus the nested seo: block, the Default Standard naming renames compound keys to a different field: author_email → author, previous_version → version #162 keys, both Maximum merging depends on key order: name_of_author + author_name merge in one order and stay separate in the other, because each key is scored against itself #167 orders,
  the Rewriting a document changes unquoted numbers: version: 1.10 becomes 1.1, zip: 01234 becomes 1234, lat: 51.507351 becomes 51.50735 #178 values plus .inf and +12, Duplicate keys in one block keep the last value, although YamlSerializer's code and comment say the first occurrence is preserved #154 First, the A | block scalar loses its trailing newline when it is the last key in the frontmatter (and CombineFrontmatter rewrites it as >-) #153 | and |+ cases, the Frontmatter whose lines are uniformly indented is treated as unreadable, so AddFrontmatter silently drops the new properties #161 indented block,
  and the Standard naming lowercases the canonical redirectFrom/redirectTo keys to redirectfrom/redirectto, and StandardOrder cannot place them #150/FrontmatterOrder.Sorted ignores standard keys that differ only in case (e.g. Title, Date) under AsIs naming, contradicting StandardOrder.Compare #151 sort and Compare rows.
• Property tests with a seeded generator, no new package, 500 cases each, over keys drawn from the alias table, StandardOrder.PropertyNames,
  decorated and qualified variants, and random words, with scalar, list and null values. Assert invariants 1 and 3–7 for all 4 strategies × 2 namings.
• Pin Delegate property-name fuzzy matching to ktsu.FuzzySearch #113: a Maximum case that reaches the fuzzy step and where per-word Fuzzy.Contains picks a different partner than raw word overlap would.
• Keep every Round-tripping strips quotes, so "true", "1.0", "007" and "null" come back as bool/number/null, and empty values become '' #137 test in ScalarTypeRoundTripTests unchanged.

Implementation plan (one PR, commits in this order, suite green after each)

1. YamlDocumentModel: parse with YamlStream (first wins), stop trimming blocks, emit with styles, add text-faithful typing for the public read path,
   route Add/Replace/Combine through nodes, delete ParsedYamlCache. Tests: Rewriting a document changes unquoted numbers: version: 1.10 becomes 1.1, zip: 01234 becomes 1234, lat: 51.507351 becomes 51.50735 #178, Duplicate keys in one block keep the last value, although YamlSerializer's code and comment say the first occurrence is preserved #154, A | block scalar loses its trailing newline when it is the last key in the frontmatter (and CombineFrontmatter rewrites it as >-) #153, Frontmatter whose lines are uniformly indented is treated as unreadable, so AddFrontmatter silently drops the new properties #161. [patch]-style commit message; the tag goes on the last commit.
2. PropertyNameMatcher with CanonicalSpelling and Label. Switch StandardOrder.Compare and SortFrontmatterProperties to it (Standard naming lowercases the canonical redirectFrom/redirectTo keys to redirectfrom/redirectto, and StandardOrder cannot place them #150, FrontmatterOrder.Sorted ignores standard keys that differ only in case (e.g. Title, Date) under AsIs naming, contradicting StandardOrder.Compare #151).
3. Rewrite NameStandardizer to rename by label only, and delete its containment and ranking code (Default Standard naming renames compound keys to a different field: author_email → author, previous_version → version #162, Default CombineFrontmatter renames short keys to longer, unrelated standard fields: id → video, key → keywords, list → unlisted, seo → description_seo #177). Update the pinned tests listed above.
4. MergePlanner: label classes, the partition, group naming, the value rule and the asserting Write. Delete GetCanonicalName, IsInSameCategory,
   FindBasicCanonicalName and TryFindEquivalenceClassName (Aggressive and Maximum merging delete properties whose name merely contains another key: subtitle, tagline, username, author_url, canonical_url and summary_image lose their values #175, Maximum merging can wipe out entire list properties: two merge groups write the same key and the second overwrites the first #176).
5. The Maximum mutual-best fuzzy join with the SharesRelatedWord gate, plus the Delegate property-name fuzzy matching to ktsu.FuzzySearch #113 pinning test (Maximum merging depends on key order: name_of_author + author_name merge in one order and stay separate in the other, because each key is scored against itself #167, Delegate property-name fuzzy matching to ktsu.FuzzySearch #113).
6. The property-test suite for invariants 1–7. Then update Enums.cs docs, the README merge-strategy section and CLAUDE.md (the pipeline description,
   and "FuzzySearch is used only by Maximum"). The final commit carries [major].
7. The PR body closes Standard naming lowercases the canonical redirectFrom/redirectTo keys to redirectfrom/redirectto, and StandardOrder cannot place them #150, FrontmatterOrder.Sorted ignores standard keys that differ only in case (e.g. Title, Date) under AsIs naming, contradicting StandardOrder.Compare #151, A | block scalar loses its trailing newline when it is the last key in the frontmatter (and CombineFrontmatter rewrites it as >-) #153, Duplicate keys in one block keep the last value, although YamlSerializer's code and comment say the first occurrence is preserved #154, Frontmatter whose lines are uniformly indented is treated as unreadable, so AddFrontmatter silently drops the new properties #161, Default Standard naming renames compound keys to a different field: author_email → author, previous_version → version #162, Maximum merging depends on key order: name_of_author + author_name merge in one order and stay separate in the other, because each key is scored against itself #167, Aggressive and Maximum merging delete properties whose name merely contains another key: subtitle, tagline, username, author_url, canonical_url and summary_image lose their values #175, Maximum merging can wipe out entire list properties: two merge groups write the same key and the second overwrites the first #176, Default CombineFrontmatter renames short keys to longer, unrelated standard fields: id → video, key → keywords, list → unlisted, seo → description_seo #177 and Rewriting a document changes unquoted numbers: version: 1.10 becomes 1.1, zip: 01234 becomes 1234, lat: 51.507351 becomes 51.50735 #178, and lists Delegate property-name fuzzy matching to ktsu.FuzzySearch #113 as fully resolved.