From 9c2887d5e7a9a79992ce7af6273f7d153b6ed256 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Wed, 12 Aug 2026 20:58:28 +0200 Subject: [PATCH] Cut CLI startup roughly in half, keep output identical MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measurement first: the existing numbers were cold CLI wall time, which says nothing about where the time goes. A new :benchmark module times the pipeline in-process, and it turns out conversion was never the problem — every fixture but Wikipedia parses and renders in under 5 ms. What was slow: - Tika's MIME registry cost ~90 ms to build on first use, more than converting most documents. SignatureMimeDetector reads the leading bytes and consults a fixed table instead, so content still beats a wrong extension. TikaMimeDetector moves out of utils and becomes public for callers who want full sniffing. - converters built their heavy fields eagerly, so constructing the registry loaded POI and Jackson even to convert a text file - HtmlToDocument compiled a whitespace Regex per text node and rebuilt a parent list per table row, on a document with tens of thousands of both - JsonConverter allocated a second ObjectMapper on every call The CLI now also optimizes for short runs: installDist records a class-data-sharing archive into the distribution, and the JVM compiles with C1 only. The start script skips the archive when it is absent, so distZip and MIKROMARKDOWN_NO_CDS=1 both work. CLI wall time, best of eight: json 298 -> 95 ms docx 446 -> 190 ms epub 329 -> 100 ms pdf 437 -> 201 ms blog 352 -> 109 ms pptx 506 -> 240 ms wiki 362 -> 159 ms xlsx 446 -> 183 ms That is about 2x faster than Python markitdown and still 4-8x slower than Rust anydoc, which is process startup: java -version alone is 41 ms here. Closing that would need ahead-of-time compilation, not a faster pipeline. Converted output is byte-identical across all eight fixtures. Also scopes the Konsist rules to the library module, so the new benchmark module's println and Main.kt do not trip rules describing library architecture. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 37 ++++++ benchmark/build.gradle.kts | 13 ++ .../lemcoder/mikromarkdown/benchmark/Main.kt | 86 ++++++++++++++ cli/build.gradle.kts | 112 +++++++++++++++++- .../mikromarkdown/MikroMarkdownFactory.kt | 3 +- .../utils/AndroidMimeDetector.kt | 20 ---- .../mikromarkdown/SignatureMimeDetector.kt | 88 ++++++++++++++ .../mikromarkdown/render/MarkdownRenderer.kt | 4 +- .../mikromarkdown/utils/TextBlocks.kt | 4 +- .../mikromarkdown/MikroMarkdownFactory.kt | 3 +- .../{utils => }/TikaMimeDetector.kt | 15 ++- .../mikromarkdown/converters/JsonConverter.kt | 30 ++--- .../mikromarkdown/converters/XlsxConverter.kt | 3 +- .../mikromarkdown/utils/HtmlToDocument.kt | 23 +++- .../mikromarkdown/ArchitectureTest.kt | 4 +- settings.gradle.kts | 2 + 16 files changed, 395 insertions(+), 52 deletions(-) create mode 100644 benchmark/build.gradle.kts create mode 100644 benchmark/src/main/kotlin/io/github/lemcoder/mikromarkdown/benchmark/Main.kt delete mode 100644 library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/utils/AndroidMimeDetector.kt create mode 100644 library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/SignatureMimeDetector.kt rename library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/{utils => }/TikaMimeDetector.kt (51%) diff --git a/README.md b/README.md index c3df417..b233003 100644 --- a/README.md +++ b/README.md @@ -156,6 +156,43 @@ between source sets (the drift that the `jvmShared` set removed). ktfmt-gradle only derives tasks for the common and JVM source sets, so `library/build.gradle.kts` registers matching tasks for the Android ones. +## Performance + +Conversion itself is a few milliseconds; a CLI run is mostly JVM startup and class loading. +`:benchmark` measures the pipeline in-process, `scripts/benchmark.py` measures whole processes. + +```bash +./gradlew :benchmark:run # in-process, per stage +python3 scripts/benchmark.py # whole process, against markitdown and anydoc +``` + +In-process, best of 50 runs after warmup: + +| fixture | size | parse | render | total | +|---|---|---|---|---| +| test.json | 0.4 KB | 0.03 ms | 0.01 ms | 0.04 ms | +| test.epub | 2 KB | 0.42 ms | 0.00 ms | 0.42 ms | +| test_blog.html | 25 KB | 0.85 ms | 0.11 ms | 0.96 ms | +| test.xlsx | 11 KB | 1.09 ms | 0.00 ms | 1.03 ms | +| test.docx | 132 KB | 3.28 ms | 0.00 ms | 2.81 ms | +| test.pdf | 90 KB | 3.69 ms | 0.00 ms | 3.47 ms | +| test.pptx | 271 KB | 4.79 ms | 0.00 ms | 4.53 ms | +| test_wikipedia.html | 385 KB | 12.98 ms | 1.82 ms | 14.80 ms | + +End to end the CLI runs in 100–240 ms, against ~25 ms for the Rust +[anydoc](https://github.com/firecrawl/anydoc) and 410–540 ms for Python markitdown. Nearly all of +what is left is process startup: `java -version` alone costs 41 ms on the same machine, and the +conversion is under 5 ms for every fixture but Wikipedia. Matching a native binary would take +ahead-of-time compilation, not a faster pipeline. + +The CLI therefore optimizes startup rather than throughput: + +- `installDist` records a [class-data-sharing](https://docs.oracle.com/en/java/javase/21/vm/class-data-sharing.html) + archive into the distribution, which roughly halves startup. Set `MIKROMARKDOWN_NO_CDS=1` to skip + it; the start script also skips it when the archive is missing, so `distZip` still works. +- it compiles with C1 only (`-XX:TieredStopAtLevel=1`), since C2 never pays for itself in a run this + short. Embedders using the library get the normal JIT. + ## Benchmark `scripts/benchmark.py` converts the test fixtures with MikroMarkdown, Python diff --git a/benchmark/build.gradle.kts b/benchmark/build.gradle.kts new file mode 100644 index 0000000..38ad149 --- /dev/null +++ b/benchmark/build.gradle.kts @@ -0,0 +1,13 @@ +plugins { + alias(libs.plugins.kotlin.jvm) + application +} + +java { toolchain { languageVersion = JavaLanguageVersion.of(21) } } + +application { mainClass = "io.github.lemcoder.mikromarkdown.benchmark.MainKt" } + +// Fixtures are addressed from the repository root, not this module's directory. +tasks.named("run") { workingDir = rootProject.projectDir } + +dependencies { implementation(project(":library")) } diff --git a/benchmark/src/main/kotlin/io/github/lemcoder/mikromarkdown/benchmark/Main.kt b/benchmark/src/main/kotlin/io/github/lemcoder/mikromarkdown/benchmark/Main.kt new file mode 100644 index 0000000..a68c38c --- /dev/null +++ b/benchmark/src/main/kotlin/io/github/lemcoder/mikromarkdown/benchmark/Main.kt @@ -0,0 +1,86 @@ +package io.github.lemcoder.mikromarkdown.benchmark + +import io.github.lemcoder.mikromarkdown.MikroMarkdown +import io.github.lemcoder.mikromarkdown.StreamInfo +import java.io.File +import kotlin.system.measureNanoTime + +/** + * In-process timings for the conversion pipeline. + * + * The CLI's wall clock is dominated by JVM startup and class loading, which says nothing about the pipeline itself. + * This measures the stages separately on a warmed-up JVM, and separately reports the first conversion in a fresh JVM — + * the one that pays for loading POI, PDFBox and Tika. + * + * Usage: ./gradlew :benchmark:run --args="[fixtureDir] [warmup] [iterations]" + */ +fun main(args: Array) { + // Cold modes run exactly one conversion and exit, so the number includes class loading. + // Comparing them isolates what format detection costs on a cold JVM. + when (args.firstOrNull()) { + "cold-bytes" -> return coldBytes(File(args[1])) + "cold-path" -> return coldPath(File(args[1])) + } + + val fixtures = File(args.getOrElse(0) { "library/src/commonTest/resources/test_files" }) + val warmup = args.getOrElse(1) { "20" }.toInt() + val iterations = args.getOrElse(2) { "50" }.toInt() + + val files = fixtures.listFiles().orEmpty().filter { it.isFile && !it.name.startsWith(".") }.sortedBy { it.name } + require(files.isNotEmpty()) { "no fixtures in ${fixtures.absolutePath}" } + + val first = files.first() + val coldStart = measureNanoTime { MikroMarkdown().convert(first.absolutePath) } + report("first conversion in a fresh JVM, class loading included (${first.name})", coldStart) + println() + + val mikroMarkdown = MikroMarkdown() + + println("Best of $iterations runs after $warmup warmup runs, milliseconds.") + println() + println("| fixture | KB | parse | render | convert(bytes) | convert(path) |") + println("|---|---|---|---|---|---|") + + for (file in files) { + val bytes = file.readBytes() + val info = StreamInfo(extension = file.extension, filename = file.name, localPath = file.absolutePath) + + repeat(warmup) { mikroMarkdown.convert(bytes, info) } + + val parse = best(iterations) { mikroMarkdown.parse(bytes, info) } + val convertBytes = best(iterations) { mikroMarkdown.convert(bytes, info) } + val convertPath = best(iterations) { mikroMarkdown.convert(file.absolutePath) } + // Rendering is whatever convert adds on top of parse; timing it alone would re-parse. + val render = (convertBytes - parse).coerceAtLeast(0) + + println( + "| ${file.name} | ${file.length() / 1024} | ${parse.ms()} | ${render.ms()} | " + + "${convertBytes.ms()} | ${convertPath.ms()} |" + ) + } +} + +private fun coldBytes(file: File) { + val bytes = file.readBytes() + val info = StreamInfo(extension = file.extension, filename = file.name, localPath = file.absolutePath) + val elapsed = measureNanoTime { MikroMarkdown().convert(bytes, info) } + report("cold convert(bytes), no detection (${file.name})", elapsed) +} + +private fun coldPath(file: File) { + val elapsed = measureNanoTime { MikroMarkdown().convert(file.absolutePath) } + report("cold convert(path), with detection (${file.name})", elapsed) +} + +private fun report(label: String, nanos: Long) = println("$label: ${nanos.ms()} ms") + +private fun best(iterations: Int, block: () -> Any?): Long { + var best = Long.MAX_VALUE + repeat(iterations) { + val elapsed = measureNanoTime { block() } + if (elapsed < best) best = elapsed + } + return best +} + +private fun Long.ms(): String = "%.2f".format(this / 1_000_000.0) diff --git a/cli/build.gradle.kts b/cli/build.gradle.kts index 9601598..ca4c35a 100644 --- a/cli/build.gradle.kts +++ b/cli/build.gradle.kts @@ -5,7 +5,16 @@ plugins { java { toolchain { languageVersion = JavaLanguageVersion.of(21) } } -application { mainClass = "com.mikromarkdown.cli.MainKt" } +val cdsArchiveName = "mikromarkdown.jsa" + +application { + mainClass = "com.mikromarkdown.cli.MainKt" + // Class loading, not conversion, is what a short CLI run spends its time on. A class-data-sharing + // archive maps those classes in pre-parsed. -Xshare:auto keeps the CLI working when it is absent. + // Every CLI run is short, so C2 never pays for itself: compiling with C1 only is faster + // end to end. Long-running embedders use the library directly and are unaffected. + applicationDefaultJvmArgs = listOf("-Xshare:auto", "-XX:-UsePerfData", "-XX:TieredStopAtLevel=1") +} dependencies { implementation(project(":library")) @@ -13,3 +22,104 @@ dependencies { } tasks.test { useJUnitPlatform() } + +// The archive flag is added by the script itself, and only when the archive is there: naming a +// missing archive stops the JVM loading its base one, which is also what recording needs. +tasks.named("startScripts") { + doLast { + val unixGuard = + """ + CDS_ARCHIVE="${'$'}APP_HOME/lib/$cdsArchiveName" + if [ -f "${'$'}CDS_ARCHIVE" ] && [ -z "${'$'}MIKROMARKDOWN_NO_CDS" ] ; then + DEFAULT_JVM_OPTS="${'$'}DEFAULT_JVM_OPTS \"-XX:SharedArchiveFile=${'$'}CDS_ARCHIVE\"" + fi + """ + .trimIndent() + + // Lambda form: the guard contains $ sequences that must not be read as group references. + unixScript.writeText( + unixScript.readText().replace(Regex("(?m)^(DEFAULT_JVM_OPTS=.*)${'$'}")) { match -> + "${match.value}\n\n$unixGuard" + } + ) + + val windowsGuard = + """ + set CDS_ARCHIVE=%APP_HOME%\\lib\\$cdsArchiveName + if exist "%CDS_ARCHIVE%" if not defined MIKROMARKDOWN_NO_CDS set DEFAULT_JVM_OPTS=%DEFAULT_JVM_OPTS% "-XX:SharedArchiveFile=%CDS_ARCHIVE%" + """ + .trimIndent() + + windowsScript.writeText( + windowsScript.readText().replace(Regex("(?m)^(set DEFAULT_JVM_OPTS=.*)${'$'}")) { match -> + "${match.value}\r\n$windowsGuard" + } + ) + } +} + +/** + * Builds a class-data-sharing archive for the installed distribution, in two steps: + * + * 1. run the CLI once with `-XX:DumpLoadedClassList` to learn which classes a conversion touches; + * 2. `-Xshare:dump` that list into an archive. + * + * This static form is used rather than `-XX:ArchiveClassesAtExit` because the dynamic one needs the JDK's own base + * archive, which some distributions (JetBrains Runtime among them) do not ship. + * + * Step 1 goes through the start script so the classpath recorded is the one real runs use; step 2 reads that same + * classpath back out of the script, because a mismatch makes the JVM drop the archive without saying so. + */ +val cdsArchive by tasks.registering { + group = "distribution" + description = "Builds a class-data-sharing archive into the installed distribution." + + val installDir = layout.buildDirectory.dir("install/${application.applicationName}").get() + val appName = application.applicationName + val sample = rootProject.layout.projectDirectory.file("library/src/commonTest/resources/test_files/test.docx") + val javaHome = javaToolchains.launcherFor(java.toolchain).get().metadata.installationPath + + doLast { + val script = installDir.file("bin/$appName").asFile + val classList = installDir.file("lib/$cdsArchiveName.classlist").asFile + val archive = installDir.file("lib/$cdsArchiveName").asFile + archive.delete() + + providers + .exec { + commandLine(script.absolutePath, sample.asFile.absolutePath) + environment("JAVA_OPTS", "-XX:DumpLoadedClassList=${classList.absolutePath}") + environment("MIKROMARKDOWN_NO_CDS", "1") + } + .standardOutput + .asText + .get() + check(classList.exists()) { "the JVM recorded no class list" } + + val classpath = + script + .readLines() + .first { it.startsWith("CLASSPATH=") } + .removePrefix("CLASSPATH=") + .replace("\$APP_HOME", installDir.asFile.absolutePath) + + providers + .exec { + commandLine( + javaHome.file("bin/java").asFile.absolutePath, + "-Xshare:dump", + "-XX:SharedClassListFile=${classList.absolutePath}", + "-XX:SharedArchiveFile=${archive.absolutePath}", + "-cp", + classpath, + ) + } + .standardOutput + .asText + .get() + check(archive.exists()) { "no CDS archive was produced" } + logger.lifecycle("CDS archive: ${archive.length() / 1024} KB") + } +} + +tasks.named("installDist") { finalizedBy(cdsArchive) } diff --git a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt index 8bafa41..65b3f85 100644 --- a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt +++ b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt @@ -13,7 +13,6 @@ import io.github.lemcoder.mikromarkdown.converters.PlainTextConverter import io.github.lemcoder.mikromarkdown.converters.PptxConverter import io.github.lemcoder.mikromarkdown.converters.XlsxConverter import io.github.lemcoder.mikromarkdown.converters.XmlConverter -import io.github.lemcoder.mikromarkdown.utils.AndroidMimeDetector import java.io.File /** @@ -22,7 +21,7 @@ import java.io.File * PDF support needs a [Context]: pdfbox-android loads its resources from the app's assets. */ public fun MikroMarkdown(context: Context? = null): MikroMarkdown = - MikroMarkdown(AndroidMimeDetector).apply { + MikroMarkdown(SignatureMimeDetector).apply { register(MarkdownPassthroughConverter()) register(HtmlConverter()) register(CsvConverter()) diff --git a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/utils/AndroidMimeDetector.kt b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/utils/AndroidMimeDetector.kt deleted file mode 100644 index dddd0a2..0000000 --- a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/utils/AndroidMimeDetector.kt +++ /dev/null @@ -1,20 +0,0 @@ -package io.github.lemcoder.mikromarkdown.utils - -import android.webkit.MimeTypeMap -import io.github.lemcoder.mikromarkdown.MimeDetector -import io.github.lemcoder.mikromarkdown.StreamInfo -import java.io.File - -internal object AndroidMimeDetector : MimeDetector { - override fun detect(path: String): StreamInfo { - val file = File(path) - val extension = file.extension.lowercase().ifEmpty { null } - val mimetype = extension?.let { MimeTypeMap.getSingleton().getMimeTypeFromExtension(it) } - return StreamInfo( - mimetype = mimetype, - extension = extension, - filename = file.name, - localPath = path, - ) - } -} diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/SignatureMimeDetector.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/SignatureMimeDetector.kt new file mode 100644 index 0000000..b7168a8 --- /dev/null +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/SignatureMimeDetector.kt @@ -0,0 +1,88 @@ +package io.github.lemcoder.mikromarkdown + +import kotlinx.io.buffered +import kotlinx.io.files.Path +import kotlinx.io.files.SystemFileSystem +import kotlinx.io.readByteArray + +/** + * Detects a format from the file's leading bytes, falling back to its extension. + * + * This is the default because it costs microseconds: it reads a handful of bytes and consults a fixed table, where a + * full MIME registry (Tika) spends ~90 ms building itself on first use — more than the entire conversion for most + * documents. Content still wins over the extension, so a mislabelled `.txt` that is really a PDF or an OOXML package is + * identified correctly. + * + * Formats that are plain text with no signature (CSV, JSON, XML, HTML, Markdown) are recognised by extension. Pass + * [TikaMimeDetector] to `MikroMarkdown` if you need content sniffing for those too. + */ +public object SignatureMimeDetector : MimeDetector { + + private const val SIGNATURE_BYTES = 8 + + private val byExtension = + mapOf( + "csv" to "text/csv", + "docx" to "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + "epub" to "application/epub+zip", + "htm" to "text/html", + "html" to "text/html", + "json" to "application/json", + "md" to "text/markdown", + "markdown" to "text/markdown", + "pdf" to "application/pdf", + "pptx" to "application/vnd.openxmlformats-officedocument.presentationml.presentation", + "txt" to "text/plain", + "log" to "text/plain", + "xlsx" to "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + "xml" to "application/xml", + ) + + /** ZIP-based formats are told apart by extension; the signature only proves it is a package. */ + private val zipExtensions = setOf("docx", "xlsx", "pptx", "epub", "zip") + + override fun detect(path: String): StreamInfo { + val filename = path.substringAfterLast('/').substringAfterLast('\\') + val extension = filename.substringAfterLast('.', "").lowercase().ifEmpty { null } + val signature = readSignature(path) + + return StreamInfo( + mimetype = mimetypeOf(signature, extension), + extension = extension, + filename = filename, + localPath = path, + ) + } + + private fun mimetypeOf(signature: ByteArray, extension: String?): String? = + when { + signature.startsWith("%PDF") -> "application/pdf" + // Every OOXML container and EPUB is a ZIP; the extension says which one. + signature.startsWith("PK") -> + if (extension in zipExtensions) byExtension[extension] ?: "application/zip" else "application/zip" + // Legacy OLE compound files: .doc/.xls/.ppt, which no converter handles yet. + signature.startsWithBytes(0xD0, 0xCF, 0x11, 0xE0) -> "application/x-ole-storage" + else -> byExtension[extension] + } + + private fun readSignature(path: String): ByteArray = + try { + SystemFileSystem.source(Path(path)).buffered().use { source -> + // Files shorter than the signature are read whole rather than failing. + if (source.request(SIGNATURE_BYTES.toLong())) source.readByteArray(SIGNATURE_BYTES) + else source.readByteArray() + } + } catch (_: Exception) { + ByteArray(0) + } + + private fun ByteArray.startsWith(prefix: String): Boolean { + if (size < prefix.length) return false + return prefix.indices.all { this[it].toInt().toChar() == prefix[it] } + } + + private fun ByteArray.startsWithBytes(vararg prefix: Int): Boolean { + if (size < prefix.size) return false + return prefix.indices.all { this[it].toInt() and 0xFF == prefix[it] } + } +} diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt index 183f02d..bd7e2b6 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt @@ -102,7 +102,7 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt .replace("\r\n", "\n") .lines() .joinToString("\n") { it.trimEnd() } - .replace(Regex("\n{3,}"), "\n\n") + .replace(BLANK_LINES, "\n\n") .trim() } @@ -381,5 +381,7 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt public companion object { public val Default: MarkdownRenderer = MarkdownRenderer() + + private val BLANK_LINES = Regex("\n{3,}") } } diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TextBlocks.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TextBlocks.kt index 95a46e3..787f028 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TextBlocks.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TextBlocks.kt @@ -16,7 +16,7 @@ internal fun plainTextBlocks(text: String, reflow: Boolean = true): List val vocabulary = if (reflow) wordsIn(normalized) else emptySet() val paragraphs = mutableListOf() - for (chunk in normalized.split(Regex("\n[ \t]*\n"))) { + for (chunk in normalized.split(PARAGRAPH_BREAK)) { val lines = chunk.lines().map { it.trim() }.filter { it.isNotEmpty() } if (lines.isEmpty()) continue val joined = if (reflow) joinWrappedLines(lines, vocabulary) else lines.joinToString("\n") @@ -34,6 +34,8 @@ internal fun plainTextBlocks(text: String, reflow: Boolean = true): List private fun String.endsWithWordBreak(): Boolean = endsWith("-") && length > 1 && this[length - 2].isLetter() +private val PARAGRAPH_BREAK = Regex("\n[ \t]*\n") + private val WORD = Regex("[\\p{L}]{2,}") /** Words the document uses on their own; the de-hyphenation heuristic consults this. */ diff --git a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt index 8adc31c..c13dc2d 100644 --- a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt +++ b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt @@ -11,12 +11,11 @@ import io.github.lemcoder.mikromarkdown.converters.PlainTextConverter import io.github.lemcoder.mikromarkdown.converters.PptxConverter import io.github.lemcoder.mikromarkdown.converters.XlsxConverter import io.github.lemcoder.mikromarkdown.converters.XmlConverter -import io.github.lemcoder.mikromarkdown.utils.TikaMimeDetector import java.io.File /** A [MikroMarkdown] with every JVM converter registered and Tika-based format detection. */ public fun MikroMarkdown(): MikroMarkdown = - MikroMarkdown(TikaMimeDetector).apply { + MikroMarkdown(SignatureMimeDetector).apply { register(MarkdownPassthroughConverter()) register(HtmlConverter()) register(CsvConverter()) diff --git a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TikaMimeDetector.kt b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/TikaMimeDetector.kt similarity index 51% rename from library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TikaMimeDetector.kt rename to library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/TikaMimeDetector.kt index 80ab826..95823d4 100644 --- a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TikaMimeDetector.kt +++ b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/TikaMimeDetector.kt @@ -1,12 +1,17 @@ -package io.github.lemcoder.mikromarkdown.utils +package io.github.lemcoder.mikromarkdown -import io.github.lemcoder.mikromarkdown.MimeDetector -import io.github.lemcoder.mikromarkdown.StreamInfo import java.io.File import org.apache.tika.Tika -internal object TikaMimeDetector : MimeDetector { - private val tika = Tika() +/** + * Full content sniffing through Apache Tika's MIME registry. + * + * Slower to start than [SignatureMimeDetector] — building the registry costs about 90 ms, more than converting most + * documents — but it recognises text formats by content rather than by extension. + */ +public object TikaMimeDetector : MimeDetector { + // Building Tika's MIME registry is expensive; convert(bytes, info) never needs it. + private val tika by lazy { Tika() } override fun detect(path: String): StreamInfo { val file = File(path) diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt index d5348f2..c1ad124 100644 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt +++ b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt @@ -11,21 +11,23 @@ import io.github.lemcoder.mikromarkdown.model.CodeBlock import io.github.lemcoder.mikromarkdown.model.Document public class JsonConverter : DocumentConverter { - private val writer = - ObjectMapper() - .apply { registerKotlinModule() } - .writer( - object : DefaultPrettyPrinter() { - init { - indentArraysWith(DefaultIndenter(" ", "\n")) - indentObjectsWith(DefaultIndenter(" ", "\n")) - } + // Constructing a converter must not load Jackson: accepts() only looks at the extension. + private val mapper by lazy { ObjectMapper().apply { registerKotlinModule() } } - override fun createInstance() = this - - override fun writeObjectFieldValueSeparator(g: JsonGenerator) = g.writeRaw(": ") + private val writer by lazy { + mapper.writer( + object : DefaultPrettyPrinter() { + init { + indentArraysWith(DefaultIndenter(" ", "\n")) + indentObjectsWith(DefaultIndenter(" ", "\n")) } - ) + + override fun createInstance() = this + + override fun writeObjectFieldValueSeparator(g: JsonGenerator) = g.writeRaw(": ") + } + ) + } override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { return info.extension == "json" || info.mimetype in setOf("application/json", "text/json") @@ -35,7 +37,7 @@ public class JsonConverter : DocumentConverter { val json = bytes.toString(Charsets.UTF_8) val pretty = try { - writer.writeValueAsString(ObjectMapper().readTree(json)) + writer.writeValueAsString(mapper.readTree(json)) } catch (_: Exception) { json } diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XlsxConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XlsxConverter.kt index 9255b53..a6c0e76 100644 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XlsxConverter.kt +++ b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XlsxConverter.kt @@ -15,7 +15,8 @@ import org.apache.poi.ss.usermodel.DataFormatter import org.apache.poi.xssf.usermodel.XSSFWorkbook public class XlsxConverter : DocumentConverter { - private val formatter = DataFormatter() + // Constructing a converter must not load POI: accepts() only looks at the extension. + private val formatter by lazy { DataFormatter() } override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { return info.extension == "xlsx" || diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt index 1c2684d..a8a9260 100644 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt +++ b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt @@ -48,6 +48,11 @@ internal object HtmlToDocument { "select", ) + private val DROPPED_SELECTOR = DROPPED_TAGS.joinToString(", ") + + /** Compiled once: this runs for every text node in the document. */ + private val COLLAPSIBLE_WHITESPACE = Regex("\\s+") + private val HEADINGS = mapOf("h1" to 1, "h2" to 2, "h3" to 3, "h4" to 4, "h5" to 5, "h6" to 6) /** Tags that only group other content; their children are lifted into the parent block flow. */ @@ -75,7 +80,7 @@ internal object HtmlToDocument { internal fun parse(html: String, baseUri: String = ""): Document { val doc = Jsoup.parse(html, baseUri) - doc.select(DROPPED_TAGS.joinToString(", ")).remove() + doc.select(DROPPED_SELECTOR).remove() val title = doc.title().ifBlank { null } val root = doc.body() ?: doc val blocks = blocks(root) @@ -226,8 +231,18 @@ internal object HtmlToDocument { return listOf(Table(header = header, rows = body, caption = caption)) } - /** The nearest enclosing table, so nested tables do not steal each other's rows. */ - private fun Element.parentTable(): Element? = parents().firstOrNull { it.tagName() == "table" } + /** + * The nearest enclosing table, so nested tables do not steal each other's rows. Walks the parent chain directly: + * Jsoup's parents() allocates a list per call, and this runs once per row. + */ + private fun Element.parentTable(): Element? { + var parent = parent() + while (parent != null) { + if (parent.tagName() == "table") return parent + parent = parent.parent() + } + return null + } private fun inlines(element: Element): List { val out = mutableListOf() @@ -295,7 +310,7 @@ internal object HtmlToDocument { /** HTML collapses runs of whitespace; do the same before the text reaches the model. */ // Non-breaking spaces are not collapsible whitespace in HTML, so they survive verbatim. - private fun TextNode.normalizedText(): String = wholeText.replace(Regex("\\s+"), " ") + private fun TextNode.normalizedText(): String = wholeText.replace(COLLAPSIBLE_WHITESPACE, " ") private fun List.startsWithSpace(): Boolean = (firstOrNull() as? Text)?.value?.startsWith(" ") == true diff --git a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt index 4e2fe60..0bba043 100644 --- a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt +++ b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt @@ -15,7 +15,9 @@ import kotlin.test.assertEquals */ class ArchitectureTest { - private val scope = Konsist.scopeFromProject() + // These rules describe the library's architecture. The CLI and benchmark modules are + // applications: they print, and their entry points are allowed to be called Main.kt. + private val scope = Konsist.scopeFromModule("library") private val production = scope.files.filterNot { it.path.contains("Test") } // ---- layering ------------------------------------------------------------------------- diff --git a/settings.gradle.kts b/settings.gradle.kts index c3b33ad..d74dfc8 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -18,3 +18,5 @@ rootProject.name = "mikromarkdown" include(":library") include(":cli") + +include(":benchmark")