Parse documents into a shared model, render Markdown once

[lemcoder]

Follows the anydoc architecture: every format parses into one document model, and a single serializer produces the Markdown.

bytes ──► MimeDetector ──► DocumentConverter.parse ──► Document ──► MarkdownRenderer ──► Markdown
                                (per format)          (blocks,        (one GFM
                                                       inlines,        serializer)
                                                       tables,
                                                       assets)

Before, nine converters × two source sets each built Markdown strings by hand, so pipe escaping, heading clamping, table shaping and blank-line handling were reimplemented per format — and drifted.

Changes

• model/Document.kt — Document(blocks, title, metadata, assets). Blocks: Heading, Paragraph, CodeBlock, BlockQuote, ListBlock/ListItem, Table (cells with colspan/rowspan), ThematicBreak, HtmlComment, RawBlock. Inlines: Text, Strong, Emphasis, Strikethrough, CodeSpan, Link, Image, LineBreak, RawInline.
• model/DocumentBuilder.kt — document { … } DSL so parsers stay free of Markdown syntax.
• render/MarkdownRenderer.kt — the only place Markdown syntax is produced. Context-aware escaping (intraword _ left alone), heading clamp, nested-list indentation, GFM tables (ragged rows padded, colspan expanded, pipes escaped, newlines → <br>), fence-length bump. MarkdownOptions covers padded tables, YAML front matter, images-as-text and URL length caps. MarkdownNormalizer's post-hoc regex pass is gone.
• API: DocumentConverter.convert → parse(bytes, info): Document; MikroMarkdown renders. MikroMarkdown.parse() and ConversionResult.document expose the model, so callers can re-render with their own options.
• HTML/EPUB go through a Jsoup DOM → model walker (HtmlToDocument) instead of flexmark, which is dropped as a dependency.
• All converters migrated on both JVM and Android: csv, json, xml, pdf, docx, xlsx, pptx, epub, html, plaintext, markdown passthrough.

Benchmark

scripts/benchmark.py converts the fixtures with MikroMarkdown, Python markitdown and Rust anydoc, then reports content recall, structure counts, table integrity and timings to build/benchmark/report.md. Recall is measured against the consensus vocabulary: tokens at least two engines agree on.

engine	files	mean recall	broken table rows	3+ blank runs
mikromarkdown	8	100.0%	0	0
markitdown	8	100.0%	8	0
anydoc	5 (binary formats only)	94.2%	0	0

Timings: anydoc ~25 ms, MikroMarkdown 140–450 ms warm (JVM startup measured separately at ~290 ms), markitdown 400–580 ms.

The benchmark found three defects, all fixed here:

1. The Log4j banner PDFBox triggers was written to stdout, corrupting the CLI's Markdown.
2. PDF text collapsed into a single paragraph; PDFBox paragraph formatting is now enabled.
3. De-hyphenation merged real compounds (conversation-centric → conversationcentric). It now keeps the hyphen when both fragments appear as standalone words elsewhere in the document, while still repairing flexibil-/ity. markitdown and anydoc leave both broken.

Tests

:library:jvmTest and :library:testAndroidHostTest pass. Two normalizations were added to PythonComparisonTest so the comparison stays fair: Java's \s ignores hair/thin spaces where Python's does not, and markitdown keeps <noscript> tracking pixels that we drop.

Note

DOCX output is 161 KB against their 4.6 KB, because embedded images are inlined as base64 data URIs. The bytes now also live in Document.assets, so MarkdownOptions(imagesAsText = true) or maxInlineImageUrl drops the data URI without losing them. Default left unchanged.

🤖 Generated with Claude Code