ingest: MIME email extraction silently deletes all non-ASCII characters from plain .md/.txt files

[owrede]

Summary

contextfit ingest silently destroys all non-ASCII characters in plain .md/.txt files: German umlauts (ä ö ü ß), accented letters etc. are deleted from the indexed text, and characters above U+00FF (typographic quotes, dashes) are stored as literal \uXXXX escape strings. For non-English corpora this corrupts a large share of the vocabulary — BM25 matching and stored chunk text are both affected.

Verified in 0.1.0 and in the 0.1.1 wheel (same code at cli.py:485-486).

Reproduction

mkdir -p /tmp/cf-test/src
printf 'Jörg Meier wohnt in Jülich bei Köln. „Zitat“ über Überschriften.\n' > /tmp/cf-test/src/umlaut.md
contextfit --kb /tmp/cf-test/kb ingest /tmp/cf-test/src --rebuild-index-after-ingest
contextfit --kb /tmp/cf-test/kb query "Jülich" --method bm25 --top-k 1

Result:

--- Chunk 1 (id=0, score=0.5754) ---
Jrg Meier wohnt in Jlich bei Kln. „Zitat" ber berschriften.

Root cause

_ingest_file() routes every file — including plain markdown/text — through _extract_email_text():

raw = path.read_text(errors="ignore")          # cli.py:485
text, email_meta = _extract_email_text(raw, path)

For a non-MIME file this hits the raw-MIME branch:
email.message_from_string(raw, policy=compat32) → msg.get_payload(decode=True). With a str-sourced message and no Content-Transfer-Encoding, compat32 encodes the payload with raw-unicode-escape (chars ≤ U+00FF become single Latin-1 bytes; chars > U+00FF become literal \uXXXX sequences). The subsequent payload.decode(charset, errors="ignore") with charset utf-8 then drops every Latin-1 byte as invalid UTF-8 — so ö vanishes entirely and „ survives only as the 6-character string „.

Side effect of the same code path: the email parser treats everything before the first blank line as RFC-822 headers, so YAML frontmatter keys of markdown notes are (mis)parsed as email headers.

Suggested fix

Only route real emails through MIME extraction; treat everything else as plain text. This is what we patched locally:

raw = path.read_text(encoding="utf-8", errors="ignore")
looks_like_email = (
    path.suffix == ".eml"
    or bool(re.match(r"#\s+Email:", raw.lstrip()))
    or "**From:**" in raw[:500]
    or bool(re.match(r"(From|Return-Path|Received|Delivered-To):", raw))
)
if looks_like_email:
    text, file_meta = _extract_email_text(raw, path)
else:
    text, file_meta = raw, {"source": str(path)}

Also note path.read_text(errors="ignore") without an explicit encoding= is locale-dependent; under a C/POSIX locale it decodes as ASCII and drops all non-ASCII bytes even before the email path — encoding="utf-8" should be explicit.

With this patch, the repro above returns the text intact and the BM25 score for the umlaut query improves (0.58 → 0.86 in our test).

Environment: contextfit 0.1.0/0.1.1, Python 3.14, Linux; corpus = German Obsidian vault (~440k words) via @owrede/vault-memory (ADR-008 ContextFit backend).

[owrede]

Follow-up with a tested fix: PR #2.

Root cause (short version)

_preprocess_file routes every discovered file through _extract_email_text. Non-email files fall into the raw-MIME branch, where compat32 get_payload(decode=True) on a str payload round-trips the text through raw-unicode-escape + utf-8/ignore:

• chars U+0080–U+00FF are silently deleted (Köln → Kln, Straße → Strae)
• chars above U+00FF become literal \uXXXX sequences in the index

Repro against current master:

>>> _extract_email_text('# Notizen Köln\n\nGrüße aus der Straße — 日本語 ✅.\n', Path('note.md'))
('# Notizen Kln\n\nGre aus der Strae \\u2014 \\u65e5\\u672c\\u8a9e \\u2705.', {...})

Fix in PR #2

1. MIME extraction is gated behind _looks_like_email() — only .eml, markdown-exported emails, or files opening with RFC-5322 headers are parsed as email; everything else is ingested verbatim.
2. Files are read as explicit UTF-8 instead of the locale default.

Includes a regression test (.md with umlauts/CJK/emoji survives _preprocess_file end-to-end); existing email/calendar tests pass.

Suggested follow-up (not in the PR): parse real emails from bytes (email.message_from_bytes(..., policy=email.policy.default)) so the compat32 str-payload trap disappears entirely — the gate removes the trigger, that would remove the root.