En.MC (longbook_choice_eng): Name-substitution artifacts — questions reference names absent from the substituted book text, and option strings/book text contain corrupted lexical items

[uditaspatel]

Summary

While evaluating models on longbook_choice_eng (En.MC), we found a systematic name-substitution inconsistency in the shipped xinrongzhang2022/InfiniteBench (v1, HuggingFace, downloaded 2026-07): ~5% of En.MC questions reference proper names that do not appear anywhere in the corresponding book text. The pattern suggests the substitution pass was applied to the input (question) and context (book) fields independently, so questions can reference a substituted name that was mapped to a different string in that book's context, making the question unanswerable from the shipped text alone.

We understand character-name substitution is intentional for contamination defense — this report is about the inconsistency between question and context, not the substitution policy itself.

We also observed isolated option-string and mid-word substitution artifacts in id=58; happy to share detailsif useful.

Environment

• Dataset: xinrongzhang2022/InfiniteBench on HuggingFace
• Split: longbook_choice_eng
• Schema: {id, context, input, answer, options}
• Rows examined: all 229 En.MC questions
• Verification: substrings checked with str.count(...) against the raw context field of each row's own book; no preprocessing on our side.

Artifact 1 — Questions reference proper names absent from the corresponding book

Programmatic sweep for personal names in the input field (proper nouns used with possessives, titles, or as clear subject/actor) that do not occur anywhere in the same row's context field surfaced 14 confirmed cases. Examples (question id: name absent from that book's context):

id	Book (inferred)	Name in question	Occurrences in context
43	(unknown)	Nils	0
58	The Grapes of Wrath	Ruben	0
60	The Grapes of Wrath	Jailmer	0
114	Looking Backward	Bellamy	0
115	Looking Backward	Bellamy	0
242	The Ambassadors	Chad	0
244	The Ambassadors	Gloriani	0
245	The Ambassadors	Strether	0
250	Brave New World	Honor	0
275	A Game of Thrones	Arryn	0
292	The Idiot	Rogozhin	0

These are not author-metadata questions; each requires knowing something that happens inside the book, so an evaluator faithfully reading only the shipped context cannot answer them without inferring the mapping from an unrelated substitute name.

For id=58 (Grapes of Wrath), the mapping is discoverable — the book substitutes Joad → Teigan and preserves Grampa, while the question was substituted to Teigans + Ruben. A model has to first infer "Ruben = Grampa" from context to make progress. That inference is doable but adds a layer of ambiguity the benchmark did not intend to test.

Thanks for the benchmark — happy to help debug or contribute a fix if useful.