Cut CLI startup roughly in half

[lemcoder]

Goal was matching anydoc. We got about 2x faster; the remaining gap is process startup, not the pipeline. Details below, including what I could not close.

Measure first

The only numbers we had were cold CLI wall time, which cannot distinguish conversion from JVM startup. A new :benchmark module times the pipeline in-process:

./gradlew :benchmark:run

Conversion was never the problem — best of 50 after warmup:

fixture	size	parse	render	total
test.json	0.4 KB	0.03 ms	0.01 ms	0.04 ms
test.epub	2 KB	0.42 ms	0.00 ms	0.42 ms
test_blog.html	25 KB	0.85 ms	0.11 ms	0.96 ms
test.xlsx	11 KB	1.09 ms	0.00 ms	1.03 ms
test.docx	132 KB	3.28 ms	0.00 ms	2.81 ms
test.pdf	90 KB	3.69 ms	0.00 ms	3.47 ms
test.pptx	271 KB	4.79 ms	0.00 ms	4.53 ms
test_wikipedia.html	385 KB	12.98 ms	1.82 ms	14.80 ms

For everything but Wikipedia, our in-process conversion is already faster than anydoc's entire process.

What was actually slow

• Tika cost ~90 ms building its MIME registry on first use — more than converting most documents. SignatureMimeDetector reads the leading bytes and consults a fixed table, so a mislabelled .txt that is really a PDF or OOXML package is still identified correctly. TikaMimeDetector moves out of utils, becomes public, and stays available for callers who want content sniffing of text formats too.
• Converters initialised eagerly, so building the registry loaded POI and Jackson even to convert a text file. Those fields are lazy now.
• HtmlToDocument compiled a Regex per text node and rebuilt a parent list per table row — on a document with tens of thousands of both.
• JsonConverter allocated a second ObjectMapper per call.

Startup

• installDist now records a class-data-sharing archive into the distribution, which roughly halves startup. It uses a static two-step dump (DumpLoadedClassList then -Xshare:dump) rather than -XX:ArchiveClassesAtExit, because the dynamic form needs the JDK's own base archive and JetBrains Runtime does not ship one. The start script only passes the archive when it exists, so distZip and MIKROMARKDOWN_NO_CDS=1 both keep working.
• The CLI compiles with C1 only (-XX:TieredStopAtLevel=1). C2 never pays for itself in a 150 ms process. Library embedders are unaffected — this is a start-script flag.

Result

CLI wall time, best of eight:

fixture	before	after		fixture	before	after
json	298 ms	95 ms		docx	446 ms	190 ms
epub	329 ms	100 ms		pdf	437 ms	201 ms
blog	352 ms	109 ms		pptx	506 ms	240 ms
wikipedia	362 ms	159 ms		xlsx	446 ms	183 ms

Three-way, whole process: MikroMarkdown 100–240 ms, markitdown 410–540 ms, anydoc 22–27 ms.

The gap I did not close

We are still 4–8x anydoc on CLI wall time, and I don't think that closes with more tuning. java -version alone is 41 ms on this machine, and a JSON conversion — 0.04 ms of actual work — still takes 95 ms. What remains is class loading for Kotlin's stdlib, clikt, kotlinx-io and the format libraries.

Matching a native binary needs ahead-of-time compilation. GraalVM native-image is the real option, but POI and XMLBeans lean on reflection heavily enough that it is its own project, plausibly viable for a PDF/HTML/CSV/JSON-only build first. Happy to spike it if you want the number.

Verification

./gradlew check passes. Output is byte-identical across all eight fixtures — this changes when work happens, not what is produced.

The Konsist rules are now scoped to the library module, so the benchmark module's println and its Main.kt don't trip rules that describe library architecture.

🤖 Generated with Claude Code