From 488ea287e830a651ef417f95e57703af15c589c4 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Thu, 13 Aug 2026 00:07:46 +0200 Subject: [PATCH 01/11] Spike a Kotlin/Native target with the dependency-free converters MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measures whether a native binary can compete with anydoc before committing to porting the parsers that actually need work (OOXML, PDF). A macosArm64 target carries CSV, JSON and XML — the three formats that need no JVM library — plus plain text and Markdown passthrough. The model, renderer and pipeline come along unchanged from commonMain, which is the point: only the parsers had to be written. - CSV is parsed by hand (RFC 4180, ~60 lines) instead of commons-csv - JSON goes through kotlinx-serialization instead of Jackson - XML is re-indented by a small formatter instead of javax.xml - :cli-native links a 2.2 MB binary from those Output is byte-identical to the JVM CLI for all three formats. Whole-process, best of six, CSV of increasing size: input native anydoc jvm-cli 1 KB 3 ms 21 ms 61 ms 55 KB 10 ms 25 ms 73 ms 172 KB 25 ms 30 ms 82 ms 580 KB 75 ms 47 ms 114 ms 1.8 MB 237 ms 93 ms 193 ms Startup is a rout in native's favour and throughput is not: Kotlin/Native runs the per-cell table work at about half the JVM's speed, so anydoc leads from ~250 KB and even the JVM CLI overtakes native around 1.5 MB. A plain-text conversion of the same 3.5 MB file takes native 71 ms against the JVM's 108 ms, so the deficit is allocation-heavy work rather than IO. The native CLI lives in its own module rather than the library, so the library keeps its no-printing rule. Native test compilation is off: the shared integration tests expect formats this target does not have yet. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 25 +++++ cli-native/build.gradle.kts | 7 ++ .../github/lemcoder/mikromarkdown/cli/Main.kt | 25 +++++ gradle/libs.versions.toml | 3 + library/build.gradle.kts | 12 +++ .../commonTest/resources/test_files/test.csv | 6 ++ .../commonTest/resources/test_files/test.xml | 1 + .../mikromarkdown/ArchitectureTest.kt | 11 ++- .../mikromarkdown/MikroMarkdownFactory.kt | 22 +++++ .../mikromarkdown/converters/CsvConverter.kt | 80 +++++++++++++++ .../mikromarkdown/converters/JsonConverter.kt | 31 ++++++ .../mikromarkdown/converters/XmlConverter.kt | 97 +++++++++++++++++++ scripts/benchmark.py | 12 +++ settings.gradle.kts | 2 + 14 files changed, 333 insertions(+), 1 deletion(-) create mode 100644 cli-native/build.gradle.kts create mode 100644 cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt create mode 100644 library/src/commonTest/resources/test_files/test.csv create mode 100644 library/src/commonTest/resources/test_files/test.xml create mode 100644 library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt create mode 100644 library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt create mode 100644 library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt create mode 100644 library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/XmlConverter.kt diff --git a/README.md b/README.md index b233003..519b5ed 100644 --- a/README.md +++ b/README.md @@ -193,6 +193,31 @@ The CLI therefore optimizes startup rather than throughput: - it compiles with C1 only (`-XX:TieredStopAtLevel=1`), since C2 never pays for itself in a run this short. Embedders using the library get the normal JIT. +### Kotlin/Native spike + +`:cli-native` builds a macOS binary carrying the converters that need no JVM library — CSV, JSON, +XML, plain text and Markdown passthrough. It shares the model, renderer and pipeline with every +other target, and its output is byte-identical to the JVM CLI's. + +```bash +./gradlew :cli-native:linkReleaseExecutableMacosArm64 +``` + +Whole-process, best of six, converting CSV of increasing size: + +| input | native | anydoc (Rust) | JVM CLI | +|---|---|---|---| +| 1 KB | **3 ms** | 21 ms | 61 ms | +| 55 KB | **10 ms** | 25 ms | 73 ms | +| 172 KB | **25 ms** | 30 ms | 82 ms | +| 580 KB | 75 ms | **47 ms** | 114 ms | +| 1.8 MB | 237 ms | **93 ms** | 193 ms | + +Startup is where a native binary wins and it wins outright: 3 ms against anydoc's 21 ms. Throughput +is where it loses — Kotlin/Native's allocation and GC costs run about 2x the JVM's on the per-cell +work of a large table, so anydoc leads from ~250 KB and the JVM CLI overtakes it around 1.5 MB. +Most documents are far below that crossover. + ## Benchmark `scripts/benchmark.py` converts the test fixtures with MikroMarkdown, Python diff --git a/cli-native/build.gradle.kts b/cli-native/build.gradle.kts new file mode 100644 index 0000000..3479317 --- /dev/null +++ b/cli-native/build.gradle.kts @@ -0,0 +1,7 @@ +plugins { alias(libs.plugins.kotlinMultiplatform) } + +kotlin { + macosArm64 { binaries.executable { entryPoint = "io.github.lemcoder.mikromarkdown.cli.main" } } + + sourceSets { macosArm64Main.dependencies { implementation(project(":library")) } } +} diff --git a/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt b/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt new file mode 100644 index 0000000..7beb46d --- /dev/null +++ b/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt @@ -0,0 +1,25 @@ +package io.github.lemcoder.mikromarkdown.cli + +import io.github.lemcoder.mikromarkdown.MikroMarkdown +import io.github.lemcoder.mikromarkdown.MikroMarkdownException +import kotlin.system.exitProcess + +/** + * Minimal native entry point, kept deliberately bare so its timings measure conversion rather than an argument parser. + * The JVM CLI remains the full one. + */ +public fun main(args: Array) { + val path = args.firstOrNull() + if (path == null) { + println("usage: mikromarkdown ") + exitProcess(2) + } + + try { + print(MikroMarkdown().convert(path).markdown) + } catch (e: MikroMarkdownException) { + // Unsupported formats must fail loudly: benchmarks and scripts read the exit code. + println("error: ${e.message}") + exitProcess(1) + } +} diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 44f96f3..0a63414 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -5,6 +5,7 @@ android-minSdk = "26" android-compileSdk = "37" vanniktechMavenPublish = "0.36.0" kotlinx-io = "0.9.0" +kotlinx-serialization = "1.9.0" kotlinx-resources = "0.15.0" commons-csv = "1.14.1" @@ -24,6 +25,7 @@ konsist = "0.17.3" [libraries] kotlin-test = { module = "org.jetbrains.kotlin:kotlin-test", version.ref = "kotlin" } kotlinx-io-core = { module = "org.jetbrains.kotlinx:kotlinx-io-core", version.ref = "kotlinx-io" } +kotlinx-serialization-json = { module = "org.jetbrains.kotlinx:kotlinx-serialization-json", version.ref = "kotlinx-serialization" } kotlinx-resources = { module = "com.goncalossilva:resources", version.ref = "kotlinx-resources" } commons-csv = { module = "org.apache.commons:commons-csv", version.ref = "commons-csv" } @@ -42,6 +44,7 @@ konsist = { module = "com.lemonappdev:konsist", version.ref = "konsist" } android-kotlin-multiplatform-library = { id = "com.android.kotlin.multiplatform.library", version.ref = "agp" } kotlinMultiplatform = { id = "org.jetbrains.kotlin.multiplatform", version.ref = "kotlin" } kotlin-jvm = { id = "org.jetbrains.kotlin.jvm", version.ref = "kotlin" } +kotlin-serialization = { id = "org.jetbrains.kotlin.plugin.serialization", version.ref = "kotlin" } vanniktech-mavenPublish = { id = "com.vanniktech.maven.publish", version.ref = "vanniktechMavenPublish" } kotlinx-resources = { id = "com.goncalossilva.resources", version.ref = "kotlinx-resources" } detekt = { id = "io.gitlab.arturbosch.detekt", version.ref = "detekt" } diff --git a/library/build.gradle.kts b/library/build.gradle.kts index 7077d0f..220a9f3 100644 --- a/library/build.gradle.kts +++ b/library/build.gradle.kts @@ -4,6 +4,7 @@ plugins { alias(libs.plugins.kotlinMultiplatform) alias(libs.plugins.android.kotlin.multiplatform.library) alias(libs.plugins.kotlinx.resources) + alias(libs.plugins.kotlin.serialization) } group = "io.github.lemcoder" @@ -14,11 +15,19 @@ kotlin { // Public API must be spelled out: visibility and return types, no accidental exports. explicitApi() + // Declaring jvmShared by hand switches off the automatic hierarchy, which macosMain needs. + applyDefaultHierarchyTemplate() + jvm { compilerOptions { jvmTarget = JvmTarget.JVM_21 } testRuns["test"].executionTask.configure { useJUnitPlatform() } } + // Spike: a native target to see how close a real binary gets to the Rust implementation. + // The shared integration tests expect JVM-only formats, so native test compilation stays off + // until the native target carries real converters. + macosArm64 { compilations.getByName("test") { compileTaskProvider.configure { enabled = false } } } + androidLibrary { namespace = "io.github.lemcoder.mikromarkdown" compileSdk = libs.versions.android.compileSdk.get().toInt() @@ -33,6 +42,9 @@ kotlin { sourceSets { commonMain.dependencies { implementation(libs.kotlinx.io.core) } + // Formats that need no platform library at all: CSV, JSON and XML are parsed here. + macosMain.dependencies { implementation(libs.kotlinx.serialization.json) } + // JVM and Android run the same parsers on the same libraries; only PDF and MIME // detection differ. Converters live here once instead of being copied per target. val jvmShared by creating { diff --git a/library/src/commonTest/resources/test_files/test.csv b/library/src/commonTest/resources/test_files/test.csv new file mode 100644 index 0000000..6e04103 --- /dev/null +++ b/library/src/commonTest/resources/test_files/test.csv @@ -0,0 +1,6 @@ +Region,Product,Units,Revenue,Notes +North,"Widget, large",1240,18600.00,"Ships in ""bulk"" cartons" +South,Widget small,830,7470.50, +East,Gadget,415,20335.25,Backordered +West,"Gizmo +Deluxe",96,14400.00,Multi-line description diff --git a/library/src/commonTest/resources/test_files/test.xml b/library/src/commonTest/resources/test_files/test.xml new file mode 100644 index 0000000..3166aae --- /dev/null +++ b/library/src/commonTest/resources/test_files/test.xml @@ -0,0 +1 @@ +Gambardella, MatthewXML Developer's Guide44.95Ralls, KimMidnight Rain5.95 diff --git a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt index 0bba043..0267af2 100644 --- a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt +++ b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt @@ -116,7 +116,16 @@ class ArchitectureTest { */ @Test fun `production files are not copied between source sets`() { - val expectedPerTarget = setOf("MikroMarkdownFactory", "PdfConverter") + val expectedPerTarget = + setOf( + "MikroMarkdownFactory", + "PdfConverter", + // Native re-implementations of the dependency-free formats. They exist twice until + // the JVM build drops commons-csv, Jackson and javax.xml for the shared versions. + "CsvConverter", + "JsonConverter", + "XmlConverter", + ) val copied = production.filter { it.path.contains("/src/") }.groupBy { it.name }.filterValues { it.size > 1 }.keys - diff --git a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt new file mode 100644 index 0000000..be95a3b --- /dev/null +++ b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt @@ -0,0 +1,22 @@ +package io.github.lemcoder.mikromarkdown + +import io.github.lemcoder.mikromarkdown.converters.CsvConverter +import io.github.lemcoder.mikromarkdown.converters.JsonConverter +import io.github.lemcoder.mikromarkdown.converters.MarkdownPassthroughConverter +import io.github.lemcoder.mikromarkdown.converters.PlainTextConverter +import io.github.lemcoder.mikromarkdown.converters.XmlConverter + +/** + * A [MikroMarkdown] with the converters that need no platform library. + * + * The document formats (DOCX, XLSX, PPTX, EPUB, PDF, HTML) still depend on JVM libraries and are absent here; this + * target exists to measure what a native binary costs to start and run. + */ +public fun MikroMarkdown(): MikroMarkdown = + MikroMarkdown(SignatureMimeDetector).apply { + register(MarkdownPassthroughConverter()) + register(CsvConverter()) + register(JsonConverter()) + register(XmlConverter()) + register(PlainTextConverter(), priority = 10.0) + } diff --git a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt new file mode 100644 index 0000000..fa5d61a --- /dev/null +++ b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt @@ -0,0 +1,80 @@ +package io.github.lemcoder.mikromarkdown.converters + +import io.github.lemcoder.mikromarkdown.DocumentConverter +import io.github.lemcoder.mikromarkdown.StreamInfo +import io.github.lemcoder.mikromarkdown.model.Document +import io.github.lemcoder.mikromarkdown.model.Table +import io.github.lemcoder.mikromarkdown.model.TableCell + +/** + * RFC 4180 CSV, parsed directly. + * + * The JVM build uses commons-csv; on native there is no such library, and the format is small enough that reading it by + * hand costs less than a dependency would. + */ +public class CsvConverter : DocumentConverter { + override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { + return info.extension == "csv" || info.mimetype in setOf("text/csv", "application/csv") + } + + override fun parse(bytes: ByteArray, info: StreamInfo): Document { + val records = parseRecords(bytes.decodeToString()) + if (records.isEmpty()) return Document() + + val header = records.first() + if (header.isEmpty()) return Document() + + val rows = + records.drop(1).map { record -> + // Ragged rows are padded by the renderer; only extra columns need trimming here. + List(header.size) { column -> TableCell(record.getOrElse(column) { "" }) } + } + + return Document(blocks = listOf(Table(header = header.map { TableCell(it) }, rows = rows))) + } + + private fun parseRecords(text: String): List> { + val records = mutableListOf>() + var record = mutableListOf() + val field = StringBuilder() + var quoted = false + var index = 0 + + fun endField() { + record += field.toString().trim() + field.setLength(0) + } + + fun endRecord() { + endField() + // A trailing newline must not produce a phantom record. + if (record.size > 1 || record.first().isNotEmpty()) records += record + record = mutableListOf() + } + + while (index < text.length) { + val char = text[index] + when { + quoted && char == '"' -> + // A doubled quote inside a quoted field is a literal quote. + if (text.getOrNull(index + 1) == '"') { + field.append('"') + index++ + } else { + quoted = false + } + + quoted -> field.append(char) + char == '"' -> quoted = true + char == ',' -> endField() + char == '\r' -> if (text.getOrNull(index + 1) == '\n') Unit else endRecord() + char == '\n' -> endRecord() + else -> field.append(char) + } + index++ + } + if (field.isNotEmpty() || record.isNotEmpty()) endRecord() + + return records + } +} diff --git a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt new file mode 100644 index 0000000..25bfa54 --- /dev/null +++ b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt @@ -0,0 +1,31 @@ +package io.github.lemcoder.mikromarkdown.converters + +import io.github.lemcoder.mikromarkdown.DocumentConverter +import io.github.lemcoder.mikromarkdown.StreamInfo +import io.github.lemcoder.mikromarkdown.model.CodeBlock +import io.github.lemcoder.mikromarkdown.model.Document +import kotlinx.serialization.json.Json + +/** Pretty-prints JSON through kotlinx-serialization, the JVM build's Jackson having no native port. */ +public class JsonConverter : DocumentConverter { + private val json = Json { + prettyPrint = true + prettyPrintIndent = " " + isLenient = true + } + + override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { + return info.extension == "json" || info.mimetype in setOf("application/json", "text/json") + } + + override fun parse(bytes: ByteArray, info: StreamInfo): Document { + val text = bytes.decodeToString() + val pretty = + try { + json.encodeToString(Json.parseToJsonElement(text)) + } catch (_: Exception) { + text + } + return Document(blocks = listOf(CodeBlock(pretty, "json"))) + } +} diff --git a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/XmlConverter.kt b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/XmlConverter.kt new file mode 100644 index 0000000..c8f74d1 --- /dev/null +++ b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/XmlConverter.kt @@ -0,0 +1,97 @@ +package io.github.lemcoder.mikromarkdown.converters + +import io.github.lemcoder.mikromarkdown.DocumentConverter +import io.github.lemcoder.mikromarkdown.StreamInfo +import io.github.lemcoder.mikromarkdown.model.CodeBlock +import io.github.lemcoder.mikromarkdown.model.Document + +/** + * Re-indents XML for readability, without a parser. + * + * The JVM build hands this to javax.xml; on native the job is only whitespace, so the document is split into tags and + * text and re-emitted at the right depth. Anything unexpected leaves the input untouched, which is also what the JVM + * converter does when parsing fails. + */ +public class XmlConverter : DocumentConverter { + override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { + return info.extension == "xml" || info.mimetype in setOf("text/xml", "application/xml") + } + + override fun parse(bytes: ByteArray, info: StreamInfo): Document = + Document(blocks = listOf(CodeBlock(prettyPrint(bytes.decodeToString()), "xml"))) + + private fun prettyPrint(xml: String): String { + val lines = mutableListOf() + var depth = 0 + var index = 0 + + while (index < xml.length) { + val open = xml.indexOf('<', index) + if (open < 0) break + val close = xml.indexOf('>', open) + if (close < 0) return xml + + val precedingText = xml.substring(index, open).trim() + if (precedingText.isNotEmpty()) lines += " ".repeat(depth) + precedingText + + val tag = xml.substring(open, close + 1) + when (kindOf(tag)) { + TagKind.CLOSING -> { + depth = (depth - 1).coerceAtLeast(0) + lines += " ".repeat(depth) + tag + } + + // The JVM formatter omits the XML declaration; keep both targets identical. + TagKind.STANDALONE -> if (!tag.startsWith(" { + // text stays on one line, as the JVM formatter writes it. + val leaf = leafElement(xml, close + 1) + if (leaf == null) { + lines += " ".repeat(depth) + tag + depth++ + } else { + lines += " ".repeat(depth) + tag + leaf.text + leaf.closingTag + index = leaf.endIndex + continue + } + } + } + index = close + 1 + } + + return lines.joinToString("\n").trim().ifEmpty { xml } + } + + /** Text followed directly by a closing tag, meaning the element has no children. */ + private fun leafElement(xml: String, from: Int): Leaf? { + val nextOpen = xml.indexOf('<', from) + if (nextOpen < 0) return null + val nextClose = xml.indexOf('>', nextOpen) + if (nextClose < 0) return null + + val tag = xml.substring(nextOpen, nextClose + 1) + if (kindOf(tag) != TagKind.CLOSING) return null + + val text = xml.substring(from, nextOpen).trim() + if (text.isEmpty() || text.contains('\n')) return null + + return Leaf(text, tag, nextClose + 1) + } + + private class Leaf(val text: String, val closingTag: String, val endIndex: Int) + + private fun kindOf(tag: String): TagKind = + when { + tag.startsWith(" TagKind.STANDALONE + tag.startsWith(" TagKind.CLOSING + tag.endsWith("/>") -> TagKind.STANDALONE + else -> TagKind.OPENING + } + + private enum class TagKind { + OPENING, + CLOSING, + STANDALONE, + } +} diff --git a/scripts/benchmark.py b/scripts/benchmark.py index a669c28..7d9cffe 100644 --- a/scripts/benchmark.py +++ b/scripts/benchmark.py @@ -7,6 +7,7 @@ Engines are skipped (not failed) when their CLI is unavailable: mikromarkdown cli/build/install/cli/bin/cli + native cli-native/build/bin/macosArm64/releaseExecutable/cli-native.kexe markitdown `markitdown` on PATH, else `uvx markitdown[all]` anydoc `anydoc` on PATH, a cargo build in third-party/anydoc, or a local npm install of @firecrawl/anydoc @@ -85,6 +86,17 @@ def which_engines() -> list[Engine]: markitdown = None engines.append(Engine("markitdown", markitdown, "" if markitdown else "pip install markitdown[all]")) + native = REPO / "cli-native/build/bin/macosArm64/releaseExecutable/cli-native.kexe" + engines.append( + Engine( + "native", + [str(native)] if native.exists() else None, + "" if native.exists() else "run ./gradlew :cli-native:linkReleaseExecutableMacosArm64", + # The Kotlin/Native target carries only the converters that need no JVM library. + unsupported={"docx", "xlsx", "pptx", "epub", "pdf", "html", "htm"}, + ) + ) + anydoc_bin = shutil.which("anydoc") if not anydoc_bin: for candidate in ( diff --git a/settings.gradle.kts b/settings.gradle.kts index d74dfc8..ad6e964 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -20,3 +20,5 @@ include(":library") include(":cli") include(":benchmark") + +include(":cli-native") From d01d591933592c10a7dd08d4ffe6cc9c5ba4f9fd Mon Sep 17 00:00:00 2001 From: mikolaj Date: Thu, 13 Aug 2026 00:19:16 +0200 Subject: [PATCH 02/11] Stop allocating in the render path, slice CSV fields instead of accumulating MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The native spike matched Rust on startup but ran the per-cell work of a large table at about half the JVM's speed. Two changes close most of that, and the first one speeds up every target. Renderer, shared by all targets: - escaping scans for the first character that needs a backslash and returns the input string untouched when there is none, instead of rebuilding every string through a StringBuilder - newline normalization only runs when the text actually contains a carriage return - a single-line table cell skips the split-and-rejoin that produced the same string - rows that already match the header width are passed through rather than copied An ordinary table cell — a word, a number — now allocates nothing. Native CSV reader: - fields are ranges in the decoded text, so a field costs one substring rather than a per-character builder plus a separate trim - only fields with escaped quotes, which cannot be a slice of the input, assemble a string Whole process, best of eight, CSV by size: input native anydoc jvm-cli (native before) 1 KB 3 ms 21 ms 63 ms 3 ms 55 KB 6 ms 25 ms 65 ms 10 ms 172 KB 10 ms 30 ms 77 ms 25 ms 580 KB 29 ms 47 ms 99 ms 75 ms 1.8 MB 87 ms 94 ms 152 ms 237 ms 3.5 MB 176 ms 161 ms 223 ms 498 ms Native now leads anydoc up to 1.8 MB and trails it by under 10% at 3.5 MB. The JVM CLI gained from the shared renderer work too, 193 -> 152 ms at 1.8 MB. Compiler flags were measured, not assumed: -Xbinary=preCodegenInlineThreshold=40 is worth about 8% on large inputs and ships. Every GC binary option tried was neutral or worse except gcSchedulerType=manual, which is 20% faster on 1.8 MB but takes peak memory from 125 MB to 169 MB and grows unbounded on larger inputs, so it is documented rather than enabled. Output is byte-identical: native matches the JVM CLI on all three native formats, and the JVM output is unchanged across all eight fixtures. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 39 ++++++--- cli-native/build.gradle.kts | 14 +++- .../mikromarkdown/render/MarkdownRenderer.kt | 46 +++++++--- .../mikromarkdown/converters/CsvConverter.kt | 83 ++++++++++++++----- 4 files changed, 137 insertions(+), 45 deletions(-) diff --git a/README.md b/README.md index 519b5ed..aac1348 100644 --- a/README.md +++ b/README.md @@ -203,20 +203,37 @@ other target, and its output is byte-identical to the JVM CLI's. ./gradlew :cli-native:linkReleaseExecutableMacosArm64 ``` -Whole-process, best of six, converting CSV of increasing size: +Whole-process, best of eight, converting CSV of increasing size: | input | native | anydoc (Rust) | JVM CLI | |---|---|---|---| -| 1 KB | **3 ms** | 21 ms | 61 ms | -| 55 KB | **10 ms** | 25 ms | 73 ms | -| 172 KB | **25 ms** | 30 ms | 82 ms | -| 580 KB | 75 ms | **47 ms** | 114 ms | -| 1.8 MB | 237 ms | **93 ms** | 193 ms | - -Startup is where a native binary wins and it wins outright: 3 ms against anydoc's 21 ms. Throughput -is where it loses — Kotlin/Native's allocation and GC costs run about 2x the JVM's on the per-cell -work of a large table, so anydoc leads from ~250 KB and the JVM CLI overtakes it around 1.5 MB. -Most documents are far below that crossover. +| 1 KB | **3 ms** | 21 ms | 63 ms | +| 55 KB | **6 ms** | 25 ms | 65 ms | +| 172 KB | **10 ms** | 30 ms | 77 ms | +| 580 KB | **29 ms** | 47 ms | 99 ms | +| 1.8 MB | **87 ms** | 94 ms | 152 ms | +| 3.5 MB | 176 ms | **161 ms** | 223 ms | + +Two changes closed the throughput gap that the first cut of this target showed: + +- **The renderer stopped allocating when it has nothing to change.** Escaping now scans for the + first character that needs a backslash and returns the input untouched when there is none, and a + single-line table cell skips the split-and-rejoin. Ordinary cells — a word, a number — now cost no + allocation at all. This is shared code, so the JVM got faster too. +- **The native CSV reader slices instead of accumulating.** Fields are ranges in the decoded text, + so a field costs one substring rather than a per-character builder plus a separate trim. Only + fields containing escaped quotes, which cannot be a slice of the input, assemble a string. + +Together those took a 1.8 MB CSV from 237 ms to 87 ms. Kotlin/Native's remaining cost is the +document model itself: every cell becomes a `TableCell` holding a `Text` holding a `String`, which +is why peak memory is 125 MB for a 1.8 MB input. A genuinely zero-copy model — inlines holding +slices of the source buffer rather than copies — is the next lever, and a deeper change. + +Compiler flags were measured rather than guessed. `-Xbinary=preCodegenInlineThreshold=40` is worth +about 8% on large inputs and ships; every GC binary option tried was neutral or worse, so the +defaults stay. The exception is `gcSchedulerType=manual`, which is 20% faster on 1.8 MB because a +process that exits never needs to collect — but peak memory climbs to 169 MB there and grows +without bound on larger inputs, so it is documented rather than enabled. ## Benchmark diff --git a/cli-native/build.gradle.kts b/cli-native/build.gradle.kts index 3479317..befa74d 100644 --- a/cli-native/build.gradle.kts +++ b/cli-native/build.gradle.kts @@ -1,7 +1,19 @@ plugins { alias(libs.plugins.kotlinMultiplatform) } kotlin { - macosArm64 { binaries.executable { entryPoint = "io.github.lemcoder.mikromarkdown.cli.main" } } + macosArm64 { + binaries.executable { entryPoint = "io.github.lemcoder.mikromarkdown.cli.main" } + + compilerOptions { + // Worth about 8% on large inputs and nothing on small ones. Measured, not assumed: + // the GC binary options were all neutral or worse, so the defaults stay. + // + // gcSchedulerType=manual is the one real alternative — 20% faster on a 1.8 MB CSV, + // because a process that exits never needs to collect — but peak memory goes from + // 125 MB to 169 MB on that input, and it grows without bound on larger ones. + freeCompilerArgs.add("-Xbinary=preCodegenInlineThreshold=40") + } + } sourceSets { macosArm64Main.dependencies { implementation(project(":library")) } } } diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt index bd7e2b6..b474b3a 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt @@ -185,7 +185,7 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt /** GFM has no colspan: a spanning cell keeps its text and the covered columns render empty. */ private fun expandSpans(cells: List): List { - val out = mutableListOf() + val out = ArrayList(cells.size) for (cell in cells) { out += cellText(cell) repeat((cell.colSpan - 1).coerceAtLeast(0)) { out += "" } @@ -193,15 +193,21 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt return out } - private fun cellText(cell: TableCell): String = - inlines(cell.content, TextContext.TABLE) - .replace("\r\n", "\n") - .lines() - .joinToString(options.tableCellLineBreak) { it.trim() } - .trim() + private fun cellText(cell: TableCell): String { + val rendered = inlines(cell.content, TextContext.TABLE) + // Single-line cells are the overwhelming majority; splitting them would allocate a list + // and rejoin it to reach the same string. + if (rendered.indexOf('\n') < 0 && rendered.indexOf('\r') < 0) return rendered.trim() + return rendered.replace("\r\n", "\n").lines().joinToString(options.tableCellLineBreak) { it.trim() }.trim() + } private fun pad(cells: List, columns: Int): List = - if (cells.size >= columns) cells.take(columns) else cells + List(columns - cells.size) { "" } + // Rows that already match the header — every row of a well-formed table — are passed through. + when { + cells.size == columns -> cells + cells.size > columns -> cells.take(columns) + else -> cells + List(columns - cells.size) { "" } + } private fun row(cells: List, widths: List?): String = cells @@ -283,18 +289,31 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt context != TextContext.TABLE && (isEmpty() || last() == '\n') private fun escape(value: String, context: TextContext, atLineStart: Boolean): String { - val text = value.replace("\r\n", "\n").replace('\r', '\n') + // Most text needs neither newline normalization nor escaping. Returning it untouched keeps + // the common cell — a word or a number — from allocating anything at all. + val text = if (value.indexOf('\r') < 0) value else value.replace("\r\n", "\n").replace('\r', '\n') + if (!options.escapeText) { - return if (context == TextContext.TABLE) text.replace("|", "\\|") else text + return if (context == TextContext.TABLE && text.indexOf('|') >= 0) text.replace("|", "\\|") else text } - return buildString(text.length) { - for (index in text.indices) { + + val first = firstEscapeIndex(text, context, atLineStart) + if (first < 0) return text + + return buildString(text.length + ESCAPE_HEADROOM) { + append(text, 0, first) + for (index in first until text.length) { if (needsEscape(text, index, context, atLineStart)) append('\\') append(text[index]) } } } + private fun firstEscapeIndex(text: String, context: TextContext, atLineStart: Boolean): Int { + for (index in text.indices) if (needsEscape(text, index, context, atLineStart)) return index + return -1 + } + private fun needsEscape(text: String, index: Int, context: TextContext, atLineStart: Boolean): Boolean = when (val ch = text[index]) { '\\', @@ -383,5 +402,8 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt public val Default: MarkdownRenderer = MarkdownRenderer() private val BLANK_LINES = Regex("\n{3,}") + + /** Room for a few backslashes before the builder has to grow. */ + private const val ESCAPE_HEADROOM = 8 } } diff --git a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt index fa5d61a..c7c132e 100644 --- a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt +++ b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt @@ -33,48 +33,89 @@ public class CsvConverter : DocumentConverter { return Document(blocks = listOf(Table(header = header.map { TableCell(it) }, rows = rows))) } + /** + * Splits the text into records without building a field at a time. + * + * Each field is a range in the decoded text, so an ordinary field costs one substring and nothing else — no + * per-character builder, no separate trim. Only fields containing escaped quotes, which cannot be a slice of the + * input, fall back to assembling a string. + */ private fun parseRecords(text: String): List> { val records = mutableListOf>() - var record = mutableListOf() - val field = StringBuilder() + var record = ArrayList(EXPECTED_COLUMNS) + + var fieldStart = 0 var quoted = false + var quoteEscaped = false var index = 0 - fun endField() { - record += field.toString().trim() - field.setLength(0) + fun field(end: Int): String { + val raw = slice(text, fieldStart, end, quoted, quoteEscaped) + quoted = false + quoteEscaped = false + return raw } - fun endRecord() { - endField() - // A trailing newline must not produce a phantom record. - if (record.size > 1 || record.first().isNotEmpty()) records += record - record = mutableListOf() + fun endRecord(end: Int) { + record.add(field(end)) + if (record.size > 1 || record[0].isNotEmpty()) records += record + record = ArrayList(EXPECTED_COLUMNS) } + var inQuotes = false while (index < text.length) { val char = text[index] when { - quoted && char == '"' -> - // A doubled quote inside a quoted field is a literal quote. + inQuotes && char == '"' -> if (text.getOrNull(index + 1) == '"') { - field.append('"') + quoteEscaped = true index++ } else { - quoted = false + inQuotes = false } - quoted -> field.append(char) - char == '"' -> quoted = true - char == ',' -> endField() - char == '\r' -> if (text.getOrNull(index + 1) == '\n') Unit else endRecord() - char == '\n' -> endRecord() - else -> field.append(char) + inQuotes -> Unit + char == '"' -> { + inQuotes = true + quoted = true + } + + char == ',' -> { + record.add(field(index)) + fieldStart = index + 1 + } + + char == '\n' || char == '\r' -> { + endRecord(index) + // Swallow the second half of a CRLF pair. + if (char == '\r' && text.getOrNull(index + 1) == '\n') index++ + fieldStart = index + 1 + } } index++ } - if (field.isNotEmpty() || record.isNotEmpty()) endRecord() + if (fieldStart < text.length || record.isNotEmpty()) endRecord(text.length) return records } + + /** The field between [start] and [end], unquoted and trimmed, copied only once. */ + private fun slice(text: String, start: Int, end: Int, quoted: Boolean, quoteEscaped: Boolean): String { + var from = start + var to = end + while (from < to && text[from].isWhitespace()) from++ + while (to > from && text[to - 1].isWhitespace()) to-- + if (from >= to) return "" + + if (quoted) { + if (text[from] == '"') from++ + if (to > from && text[to - 1] == '"') to-- + if (quoteEscaped) return text.substring(from, to).replace("\"\"", "\"") + } + return text.substring(from, to) + } + + private companion object { + const val EXPECTED_COLUMNS = 8 + } } From dc22c326891ee3b80815e58d162fc4bbaa7673aa Mon Sep 17 00:00:00 2001 From: mikolaj Date: Thu, 13 Aug 2026 00:31:52 +0200 Subject: [PATCH 03/11] Measure garbage collection policies, keep the default MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit gcSchedulerType=manual means the collector runs only when code calls it, so a document boundary — where everything the previous conversion allocated is dead — becomes a place to schedule one. Converting 20 documents of 580 KB in one process: default (adaptive) 543 ms 59 MB manual, never collecting 428 ms 954 MB manual, collecting between documents 485 ms 67 MB autotune off with a heap ceiling 1246 ms 43 MB Collecting per document does recover the memory and keeps some of the speed, but it only bounds growth between documents: nothing collects mid-parse, so a single large input grows unchecked either way. Turning autotune off is 8.6x slower on one 3.5 MB file, and the ceiling value changes nothing, so targetHeapBytes does not behave as its name suggests. The default collector stays. The native CLI now takes several files per invocation, which is what would make a manual policy workable if that trade ever pays. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 26 ++++++++++++++++--- .../github/lemcoder/mikromarkdown/cli/Main.kt | 25 +++++++++++------- 2 files changed, 38 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index aac1348..abbd081 100644 --- a/README.md +++ b/README.md @@ -230,10 +230,28 @@ is why peak memory is 125 MB for a 1.8 MB input. A genuinely zero-copy model — slices of the source buffer rather than copies — is the next lever, and a deeper change. Compiler flags were measured rather than guessed. `-Xbinary=preCodegenInlineThreshold=40` is worth -about 8% on large inputs and ships; every GC binary option tried was neutral or worse, so the -defaults stay. The exception is `gcSchedulerType=manual`, which is 20% faster on 1.8 MB because a -process that exits never needs to collect — but peak memory climbs to 169 MB there and grows -without bound on larger inputs, so it is documented rather than enabled. +about 8% on large inputs and ships. Every garbage collection setting tried was worse than the +default, and the collector is the interesting part of the story, so the numbers are below. + +Converting 20 documents of 580 KB in one process: + +| policy | time | peak RSS | +|---|---|---| +| default (adaptive) | 543 ms | 59 MB | +| `gcSchedulerType=manual`, never collecting | 428 ms | 954 MB | +| `gcSchedulerType=manual`, collecting between documents | 485 ms | 67 MB | +| `autotune = false` with a heap ceiling | 1246 ms | 43 MB | + +A manual collector is genuinely faster, since a process that exits never needs to collect, and a +document boundary is the one place where everything the previous conversion allocated is provably +dead. But it only bounds growth *between* documents: a single large input still has nothing +collecting mid-parse, so the 3.5 MB file takes 228 MB either way and a much larger one would grow +until it failed. Turning `autotune` off is far worse than it looks like it should be — 8.6x on a +single 3.5 MB file — and the ceiling value makes no difference to that, so `targetHeapBytes` is not +behaving as its name suggests. + +The default collector ships. The native CLI does accept several files per invocation, which is what +would make a manual policy workable if the trade ever becomes worth it. ## Benchmark diff --git a/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt b/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt index 7beb46d..485a19c 100644 --- a/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt +++ b/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt @@ -7,19 +7,26 @@ import kotlin.system.exitProcess /** * Minimal native entry point, kept deliberately bare so its timings measure conversion rather than an argument parser. * The JVM CLI remains the full one. + * + * Several files may be given: a document boundary is the one point where everything the previous conversion allocated + * is dead, which is what makes a manual collection policy possible at all. The default collector wins on measurement, + * so none is applied — see the README. */ public fun main(args: Array) { - val path = args.firstOrNull() - if (path == null) { - println("usage: mikromarkdown ") + if (args.isEmpty()) { + println("usage: mikromarkdown ...") exitProcess(2) } - try { - print(MikroMarkdown().convert(path).markdown) - } catch (e: MikroMarkdownException) { - // Unsupported formats must fail loudly: benchmarks and scripts read the exit code. - println("error: ${e.message}") - exitProcess(1) + val mikroMarkdown = MikroMarkdown() + + for (path in args) { + try { + print(mikroMarkdown.convert(path).markdown) + } catch (e: MikroMarkdownException) { + // Unsupported formats must fail loudly: benchmarks and scripts read the exit code. + println("error: ${e.message}") + exitProcess(1) + } } } From 57f606bdc90f51fc533df05d4035b0a54e2ff844 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Thu, 13 Aug 2026 00:41:29 +0200 Subject: [PATCH 04/11] Remove two quadratics from the renderer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both found by scaling pathological inputs rather than by reading the code, and both verified byte-identical against the previous implementation. Nested blocks were charged once per level above them. Each block returned its own string, so a list or quote split its whole subtree into lines and re-joined it to add one level of indentation — the deepest leaf paid for every ancestor. Blocks now write into one buffer, carrying a line prefix that is emitted when a newline is written, so every character is written exactly once. Trailing blanks are trimmed at each newline, which is what keeps a quote's "> " as ">" on empty lines and stops an empty list item leaving "- " behind. The entity check searched the rest of the string for a semicolon on every ampersand it met, so text with many ampersands and few semicolons — query strings, for instance — cost O(n^2). It now scans ten characters, the longest entity name worth looking for. Render time on inputs built to provoke them: 400-deep nested lists 33.03 ms -> 0.12 ms 400 ampersands per cell, 789 KB 9.39 ms -> 2.42 ms Both scale linearly now. Real documents gain less, since neither pattern is common: Wikipedia renders in 1.51 ms against 1.82 ms. A third suspicion did not survive measurement: the backward whitespace scan in the line-start check never showed superlinear growth, because the character before a marker is almost never a space. It is unchanged. Output is byte-identical across all ten fixtures and all three pathological inputs. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 12 +- .../mikromarkdown/render/MarkdownRenderer.kt | 167 +++++++++++------- 2 files changed, 111 insertions(+), 68 deletions(-) diff --git a/README.md b/README.md index abbd081..7412790 100644 --- a/README.md +++ b/README.md @@ -214,7 +214,7 @@ Whole-process, best of eight, converting CSV of increasing size: | 1.8 MB | **87 ms** | 94 ms | 152 ms | | 3.5 MB | 176 ms | **161 ms** | 223 ms | -Two changes closed the throughput gap that the first cut of this target showed: +Three changes closed the throughput gap that the first cut of this target showed: - **The renderer stopped allocating when it has nothing to change.** Escaping now scans for the first character that needs a backslash and returns the input untouched when there is none, and a @@ -223,6 +223,16 @@ Two changes closed the throughput gap that the first cut of this target showed: - **The native CSV reader slices instead of accumulating.** Fields are ranges in the decoded text, so a field costs one substring rather than a per-character builder plus a separate trim. Only fields containing escaped quotes, which cannot be a slice of the input, assemble a string. +- **Two quadratics in the renderer are gone.** Blocks are written into one buffer carrying a line + prefix, rather than each block returning a string that its parent splits into lines and re-joins — + which charged the deepest content once per level of nesting above it. And the entity check now + scans ten characters ahead instead of searching the rest of the document for a semicolon. + Measured on inputs built to provoke them, with output byte-identical before and after: + + | pathological input | before | after | + |---|---|---| + | 400-deep nested lists | 33.03 ms | **0.12 ms** | + | 400 ampersands per cell, 789 KB | 9.39 ms | **2.42 ms** | Together those took a 1.8 MB CSV from 237 ms to 87 ms. Kotlin/Native's remaining cost is the document model itself: every cell becomes a `TableCell` holding a `Text` holding a `String`, which diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt index b474b3a..9754997 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt @@ -14,7 +14,6 @@ import io.github.lemcoder.mikromarkdown.model.Inline import io.github.lemcoder.mikromarkdown.model.LineBreak import io.github.lemcoder.mikromarkdown.model.Link import io.github.lemcoder.mikromarkdown.model.ListBlock -import io.github.lemcoder.mikromarkdown.model.ListItem import io.github.lemcoder.mikromarkdown.model.Paragraph import io.github.lemcoder.mikromarkdown.model.RawBlock import io.github.lemcoder.mikromarkdown.model.RawInline @@ -62,53 +61,84 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt public fun renderInline(inlines: List): String = inlines(inlines, TextContext.INLINE) + /** + * Blocks are written into one buffer, never re-read. + * + * The previous version had each block return its own string, so a nested list or quote split the whole subtree into + * lines and re-joined it once per level of nesting — the cost of the deepest leaf multiplied by the depth above it. + * Here a line prefix travels down the recursion and is emitted when a newline is written, so every character is + * written exactly once. + */ private fun renderBlocks(blocks: List): String { - val chunks = mutableListOf() + val out = StringBuilder() + writeBlocks(blocks, out, prefix = "") + return out.toString().trim() + } + + private fun writeBlocks(blocks: List, out: StringBuilder, prefix: String) { + var wroteAny = false for (block in blocks) { - val rendered = renderBlock(block) - if (rendered.isNotEmpty()) chunks += rendered + val separatorStart = out.length + if (wroteAny) { + newLine(out, prefix) + newLine(out, prefix) + } + val contentStart = out.length + writeBlock(block, out, prefix) + // Blocks that render to nothing must not leave their separator behind. + if (out.length == contentStart) out.setLength(separatorStart) else wroteAny = true } - return chunks.joinToString("\n\n").trim() } - private fun renderBlock(block: Block): String = + private fun writeBlock(block: Block, out: StringBuilder, prefix: String) { when (block) { is Heading -> { val text = inlines(block.content, TextContext.HEADING).collapseLines() - if (text.isBlank()) "" else "${"#".repeat(block.level.coerceIn(1, options.maxHeadingLevel))} $text" + if (text.isNotBlank()) { + out.append("#".repeat(block.level.coerceIn(1, options.maxHeadingLevel))).append(' ').append(text) + } } - is Paragraph -> inlines(block.content, TextContext.BLOCK).trimEnd() + is Paragraph -> appendLines(out, inlines(block.content, TextContext.BLOCK).trimEnd(), prefix) is CodeBlock -> { val fence = "`".repeat(maxOf(3, longestBacktickRun(block.code) + 1)) - "$fence${block.language.orEmpty()}\n${block.code.trimEnd('\n')}\n$fence" + out.append(fence).append(block.language.orEmpty()) + newLine(out, prefix) + appendLines(out, block.code.trimEnd('\n'), prefix) + newLine(out, prefix) + out.append(fence) } - is BlockQuote -> - renderBlocks(block.blocks).lines().joinToString("\n") { if (it.isEmpty()) ">" else "> $it" } + is BlockQuote -> { + out.append(QUOTE_PREFIX) + writeBlocks(block.blocks, out, prefix + QUOTE_PREFIX) + } - is ListBlock -> renderList(block, indent = "") + is ListBlock -> writeList(block, out, prefix) - is Table -> renderTable(block) + is Table -> writeTable(block, out, prefix) - ThematicBreak -> "---" + ThematicBreak -> out.append("---") - is HtmlComment -> "" + is HtmlComment -> out.append("") // Already-Markdown content: only whitespace is normalized, never syntax. is RawBlock -> - block.text - .replace("\r\n", "\n") - .lines() - .joinToString("\n") { it.trimEnd() } - .replace(BLANK_LINES, "\n\n") - .trim() + appendLines( + out, + block.text.replace("\r\n", "\n").replace(BLANK_LINES, "\n\n").trim(), + prefix, + ) } + } - private fun renderList(list: ListBlock, indent: String): String { - val lines = mutableListOf() + private fun writeList(list: ListBlock, out: StringBuilder, prefix: String) { list.items.forEachIndexed { index, item -> + if (index > 0) { + newLine(out, prefix) + if (list.loose) newLine(out, prefix) + } val marker = if (list.ordered) "${list.start + index}. " else "${options.bullet} " val checkbox = when (item.checked) { @@ -116,39 +146,33 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt false -> "[ ] " null -> "" } - val childIndent = indent + " ".repeat(marker.length) - val body = renderItemBlocks(item, childIndent) - val firstLine = body.firstOrNull().orEmpty() - lines += "$indent$marker$checkbox$firstLine".trimEnd() - lines += body.drop(1) - if (list.loose && index != list.items.lastIndex) lines += "" + out.append(marker).append(checkbox) + // Continuation lines line up under the marker, not under the checkbox. + writeBlocks(item.blocks, out, prefix + " ".repeat(marker.length)) } - return lines.joinToString("\n").trimEnd() } - /** Renders an item's blocks as lines, with continuation lines already indented. */ - private fun renderItemBlocks(item: ListItem, childIndent: String): List { - val lines = mutableListOf() - item.blocks.forEachIndexed { index, block -> - if (index > 0) lines += "" - val rendered = - when (block) { - is ListBlock -> renderList(block, childIndent) - else -> renderBlock(block).lines().joinToString("\n") { if (it.isEmpty()) it else childIndent + it } - } - if (rendered.isEmpty()) return@forEachIndexed - lines += rendered.lines() - } - // The first line's indent is supplied by the marker itself. - if (lines.isNotEmpty()) lines[0] = lines[0].removePrefix(childIndent) - return lines + /** Appends [text], re-emitting [prefix] after each newline it contains. */ + private fun appendLines(out: StringBuilder, text: String, prefix: String) { + for (char in text) if (char == '\n') newLine(out, prefix) else out.append(char) + } + + /** + * Ends the current line and opens the next one with [prefix]. + * + * Trailing blanks go first, which is what turns a quote's "> " into ">" on an empty line and keeps list markers + * from leaving "- " behind on an item that rendered nothing. + */ + private fun newLine(out: StringBuilder, prefix: String) { + while (out.isNotEmpty() && (out.last() == ' ' || out.last() == '\t')) out.setLength(out.length - 1) + out.append('\n').append(prefix) } - private fun renderTable(table: Table): String { + private fun writeTable(table: Table, out: StringBuilder, prefix: String) { val bodyRows = table.rows.map { expandSpans(it) } val headerCells = expandSpans(table.header) val columns = maxOf(headerCells.size, bodyRows.maxOfOrNull { it.size } ?: 0) - if (columns == 0) return "" + if (columns == 0) return val header = pad(headerCells, columns) val rows = bodyRows.map { pad(it, columns) } @@ -156,31 +180,25 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt val widths = if (options.padTableColumns) { - List(columns) { col -> - maxOf( - 3, - header[col].length, - rows.maxOfOrNull { it[col].length } ?: 0, - ) - } + List(columns) { col -> maxOf(3, header[col].length, rows.maxOfOrNull { it[col].length } ?: 0) } } else { null } - val out = StringBuilder() - out.append(row(header, widths)).append('\n') - out.append(delimiterRow(alignments, widths)).append('\n') - for ((index, r) in rows.withIndex()) { - out.append(row(r, widths)) - if (index != rows.lastIndex) out.append('\n') + out.append(row(header, widths)) + newLine(out, prefix) + out.append(delimiterRow(alignments, widths)) + for (cells in rows) { + newLine(out, prefix) + out.append(row(cells, widths)) } if (table.caption.isNotEmpty()) { - out.append("\n\n") - .append(options.emphasisMarker) + newLine(out, prefix) + newLine(out, prefix) + out.append(options.emphasisMarker) .append(inlines(table.caption, TextContext.INLINE).collapseLines()) .append(options.emphasisMarker) } - return out.toString() } /** GFM has no colspan: a spanning cell keeps its text and the covered columns render empty. */ @@ -367,10 +385,20 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt return after == null || after == ' ' || after == '\n' } + /** + * Scans at most [MAX_ENTITY_LENGTH] characters ahead. + * + * Searching the whole string for the next semicolon made this quadratic on text holding many ampersands and few + * semicolons — query strings, for one. + */ private fun looksLikeEntity(text: String, index: Int): Boolean { - val semicolon = text.indexOf(';', index) - if (semicolon <= index || semicolon - index > 10) return false - return text.substring(index + 1, semicolon).all { it.isLetterOrDigit() || it == '#' } + val limit = minOf(text.length, index + 1 + MAX_ENTITY_LENGTH) + for (position in index + 1 until limit) { + val char = text[position] + if (char == ';') return position > index + 1 + if (!char.isLetterOrDigit() && char != '#') return false + } + return false } private fun encodeUrl(url: String): String = url.replace(" ", "%20").replace("(", "%28").replace(")", "%29") @@ -405,5 +433,10 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt /** Room for a few backslashes before the builder has to grow. */ private const val ESCAPE_HEADROOM = 8 + + /** Longest entity name worth looking for, e.g. `ϑ`. */ + private const val MAX_ENTITY_LENGTH = 10 + + private const val QUOTE_PREFIX = "> " } } From 30d6397dc1c3ee6a6acbad84851c03f2849e5f71 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Thu, 13 Aug 2026 00:47:06 +0200 Subject: [PATCH 05/11] Benchmark anydoc's Rust binary, not its Node wrapper Every anydoc figure quoted so far was measured through node_modules/.bin/anydoc, which is a Node script loading a napi module. Node's startup is 15 ms on this machine, so anydoc read as a flat ~22 ms regardless of input, and the comparison flattered us on small documents. The npm package ships no standalone executable, so the honest binary comes from building the vendored source: cargo build --release --example convert. The benchmark harness now runs both and names them apart. Corrected, whole process, best of eight: input kotlin-native anydoc-rust anydoc-node jvm-cli 1 KB 3 ms 3 ms 21 ms 63 ms 55 KB 5 ms 5 ms 24 ms 65 ms 172 KB 10 ms 10 ms 30 ms 73 ms 580 KB 29 ms 27 ms 45 ms 95 ms 1.8 MB 86 ms 73 ms 93 ms 151 ms 3.5 MB 171 ms 141 ms 163 ms 222 ms So Kotlin/Native matches Rust up to ~172 KB, where both are just process startup, and trails by 7% at 580 KB growing to 21% at 3.5 MB. The earlier claim that it beat anydoc at every size up to 1.8 MB was an artifact of timing Node. The document fixtures were also overstated: anydoc converts docx in 4 ms and pdf in 6 ms, not the 22-25 ms the wrapper suggested. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 27 ++++++++++++++++++--------- scripts/benchmark.py | 28 ++++++++++++++++++++++------ 2 files changed, 40 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 7412790..abb9590 100644 --- a/README.md +++ b/README.md @@ -179,12 +179,17 @@ In-process, best of 50 runs after warmup: | test.pptx | 271 KB | 4.79 ms | 0.00 ms | 4.53 ms | | test_wikipedia.html | 385 KB | 12.98 ms | 1.82 ms | 14.80 ms | -End to end the CLI runs in 100–240 ms, against ~25 ms for the Rust +End to end the CLI runs in 100–240 ms, against 3–6 ms for the Rust [anydoc](https://github.com/firecrawl/anydoc) and 410–540 ms for Python markitdown. Nearly all of what is left is process startup: `java -version` alone costs 41 ms on the same machine, and the conversion is under 5 ms for every fixture but Wikipedia. Matching a native binary would take ahead-of-time compilation, not a faster pipeline. +Note when reproducing this: anydoc's npm package is a Node script loading a napi module, so timing +`node_modules/.bin/anydoc` charges Node's 15 ms startup to Rust and reads as ~22 ms flat. The +figures here come from its Rust binary, built from the vendored source with +`cargo build --release --example convert`. + The CLI therefore optimizes startup rather than throughput: - `installDist` records a [class-data-sharing](https://docs.oracle.com/en/java/javase/21/vm/class-data-sharing.html) @@ -205,14 +210,18 @@ other target, and its output is byte-identical to the JVM CLI's. Whole-process, best of eight, converting CSV of increasing size: -| input | native | anydoc (Rust) | JVM CLI | -|---|---|---|---| -| 1 KB | **3 ms** | 21 ms | 63 ms | -| 55 KB | **6 ms** | 25 ms | 65 ms | -| 172 KB | **10 ms** | 30 ms | 77 ms | -| 580 KB | **29 ms** | 47 ms | 99 ms | -| 1.8 MB | **87 ms** | 94 ms | 152 ms | -| 3.5 MB | 176 ms | **161 ms** | 223 ms | +| input | Kotlin/Native | anydoc (Rust binary) | anydoc (npm, via Node) | JVM CLI | +|---|---|---|---|---| +| 1 KB | 3 ms | 3 ms | 21 ms | 63 ms | +| 55 KB | 5 ms | 5 ms | 24 ms | 65 ms | +| 172 KB | 10 ms | 10 ms | 30 ms | 73 ms | +| 580 KB | 29 ms | 27 ms | 45 ms | 95 ms | +| 1.8 MB | 86 ms | 73 ms | 93 ms | 151 ms | +| 3.5 MB | 171 ms | 141 ms | 163 ms | 222 ms | + +Kotlin/Native matches the Rust binary exactly up to about 172 KB — both are process startup at that +point — and trails it by 7% at 580 KB, growing to 21% at 3.5 MB. Against the JVM CLI it is 12x +faster on small inputs and still ahead at 3.5 MB. Three changes closed the throughput gap that the first cut of this target showed: diff --git a/scripts/benchmark.py b/scripts/benchmark.py index 7d9cffe..b3b5729 100644 --- a/scripts/benchmark.py +++ b/scripts/benchmark.py @@ -9,8 +9,10 @@ mikromarkdown cli/build/install/cli/bin/cli native cli-native/build/bin/macosArm64/releaseExecutable/cli-native.kexe markitdown `markitdown` on PATH, else `uvx markitdown[all]` - anydoc `anydoc` on PATH, a cargo build in third-party/anydoc, or a local - npm install of @firecrawl/anydoc + anydoc-rust third-party/anydoc/target/release/examples/convert (cargo build + --release --example convert) — the Rust binary alone + anydoc-node the npm package, whose CLI is a Node script loading a napi module, + so its timings include Node startup Metrics per output: content recall tokens agreed on by >=2 engines that this engine also emits @@ -33,6 +35,9 @@ from pathlib import Path REPO = Path(__file__).resolve().parent.parent +# anydoc converts binary document formats only. +ANYDOC_UNSUPPORTED = {"html", "htm", "json", "xml", "md", "txt"} + TOKEN_SPLIT = re.compile(r"[\s!\"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~]+") RUNS = 3 @@ -97,10 +102,22 @@ def which_engines() -> list[Engine]: ) ) + # Two anydoc entry points, because they measure different things: the npm package ships a + # napi module that Node loads in-process, so timing it charges Node's ~15 ms startup to Rust. + # The cargo-built example is the Rust binary on its own. + native_anydoc = REPO / "third-party/anydoc/target/release/examples/convert" + engines.append( + Engine( + "anydoc-rust", + [str(native_anydoc)] if native_anydoc.exists() else None, + "" if native_anydoc.exists() else "cargo build --release --example convert in third-party/anydoc", + unsupported=ANYDOC_UNSUPPORTED, + ) + ) + anydoc_bin = shutil.which("anydoc") if not anydoc_bin: for candidate in ( - REPO / "third-party/anydoc/target/release/anydoc", REPO / "build/anydoc/node_modules/.bin/anydoc", Path("/tmp/anydoc-bench/node_modules/.bin/anydoc"), ): @@ -109,11 +126,10 @@ def which_engines() -> list[Engine]: break engines.append( Engine( - "anydoc", + "anydoc-node", [anydoc_bin] if anydoc_bin else None, "" if anydoc_bin else "npm i @firecrawl/anydoc", - # anydoc converts binary document formats only. - unsupported={"html", "htm", "json", "xml", "md", "txt"}, + unsupported=ANYDOC_UNSUPPORTED, ) ) return engines From 2995e7786b1a7d1678f36aa54281aab1e2bea544 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Thu, 13 Aug 2026 00:55:34 +0200 Subject: [PATCH 06/11] Share CSV, JSON and XML across targets, dropping three dependencies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The native spike wrote these three formats without a platform library, and its output was byte-identical to the JVM's. That made the JVM versions redundant, so the native ones move to commonMain and commons-csv, Jackson and kotlinx-serialization all go. JSON now goes through a re-indenter that copies tokens verbatim, which also means numbers keep their source spelling rather than surviving a parse and re-print; 1.50 stays 1.50. Checked against escapes, empty containers, deep nesting, big integers and top-level arrays, comparing parsed data rather than text. - a JSON conversion loads 1214 classes, down from 2063 - the native binary is 1.3 MB, down from 2.2 MB - the JVM CLI converts JSON in 56 ms, down from 95 ms, and a 1.8 MB CSV in 126 ms, down from 151 ms, since the slicing reader beats commons-csv - the Konsist duplicate-file rule is strict again: nothing is copied per target except the factory and the PDF converter Two things measured and rejected. Binary size does not drive startup — a hello-world Kotlin/Native binary of 485 KB starts in 3.2 ms, ours at 1.3 MB in 3.5 ms, and anydoc's 6 MB Rust binary in 2.6 ms — so stripping and shrinking buy nothing. Replacing clikt with hand-rolled parsing removes 147 classes and about 2 ms, inside the noise and not worth the help text it would cost. Output is byte-identical across all ten fixtures on both targets. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 22 ++++ library/build.gradle.kts | 6 - .../mikromarkdown/converters/CsvConverter.kt | 0 .../mikromarkdown/converters/JsonConverter.kt | 17 +++ .../mikromarkdown/converters/XmlConverter.kt | 0 .../mikromarkdown/utils/JsonFormatter.kt | 113 ++++++++++++++++++ .../mikromarkdown/converters/CsvConverter.kt | 33 ----- .../mikromarkdown/converters/JsonConverter.kt | 46 ------- .../mikromarkdown/converters/XmlConverter.kt | 46 ------- .../mikromarkdown/ArchitectureTest.kt | 11 +- .../mikromarkdown/converters/JsonConverter.kt | 31 ----- 11 files changed, 153 insertions(+), 172 deletions(-) rename library/src/{macosMain => commonMain}/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt (100%) create mode 100644 library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt rename library/src/{macosMain => commonMain}/kotlin/io/github/lemcoder/mikromarkdown/converters/XmlConverter.kt (100%) create mode 100644 library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/JsonFormatter.kt delete mode 100644 library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt delete mode 100644 library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt delete mode 100644 library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XmlConverter.kt delete mode 100644 library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt diff --git a/README.md b/README.md index abb9590..48a31f8 100644 --- a/README.md +++ b/README.md @@ -232,6 +232,12 @@ Three changes closed the throughput gap that the first cut of this target showed - **The native CSV reader slices instead of accumulating.** Fields are ranges in the decoded text, so a field costs one substring rather than a per-character builder plus a separate trim. Only fields containing escaped quotes, which cannot be a slice of the input, assemble a string. +- **CSV, JSON and XML moved to `commonMain`**, taking commons-csv, Jackson and + kotlinx-serialization with them. One implementation now serves every target: a slicing CSV reader, + a JSON re-indenter that copies tokens verbatim so `1.50` does not become `1.5`, and an XML + formatter. A JSON conversion loads 1214 classes instead of 2063, the native binary is 1.3 MB + instead of 2.2 MB, and the JVM CLI converts JSON in 56 ms instead of 95 ms. Output is unchanged + on every fixture. - **Two quadratics in the renderer are gone.** Blocks are written into one buffer carrying a line prefix, rather than each block returning a string that its parent splits into lines and re-joins — which charged the deepest content once per level of nesting above it. And the entity check now @@ -272,6 +278,22 @@ behaving as its name suggests. The default collector ships. The native CLI does accept several files per invocation, which is what would make a manual policy workable if the trade ever becomes worth it. +### What did not help + +Binary size does not drive startup, so shrinking it is not a performance lever: + +| binary | size | startup | +|---|---|---| +| Kotlin/Native hello world | 485 KB | 3.2 ms | +| this CLI | 1.3 MB | 3.5 ms | +| anydoc (Rust) | 6 MB | 2.6 ms | + +A 6 MB Rust binary starts faster than a 485 KB Kotlin/Native one, and stripping ours changed +nothing measurable. The 0.6 ms between the two runtimes is initialization, not size. + +Replacing clikt with hand-rolled argument parsing removes 147 loaded classes and about 2 ms — inside +the noise, and not worth losing its help output and error handling. The dependency stays. + ## Benchmark `scripts/benchmark.py` converts the test fixtures with MikroMarkdown, Python diff --git a/library/build.gradle.kts b/library/build.gradle.kts index 220a9f3..5a405c9 100644 --- a/library/build.gradle.kts +++ b/library/build.gradle.kts @@ -4,7 +4,6 @@ plugins { alias(libs.plugins.kotlinMultiplatform) alias(libs.plugins.android.kotlin.multiplatform.library) alias(libs.plugins.kotlinx.resources) - alias(libs.plugins.kotlin.serialization) } group = "io.github.lemcoder" @@ -42,17 +41,12 @@ kotlin { sourceSets { commonMain.dependencies { implementation(libs.kotlinx.io.core) } - // Formats that need no platform library at all: CSV, JSON and XML are parsed here. - macosMain.dependencies { implementation(libs.kotlinx.serialization.json) } - // JVM and Android run the same parsers on the same libraries; only PDF and MIME // detection differ. Converters live here once instead of being copied per target. val jvmShared by creating { dependsOn(commonMain.get()) dependencies { implementation(libs.jsoup) - implementation(libs.jackson.kotlin) - implementation(libs.commons.csv) implementation(libs.poi.ooxml) } } diff --git a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt similarity index 100% rename from library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt rename to library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt new file mode 100644 index 0000000..69f93a8 --- /dev/null +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt @@ -0,0 +1,17 @@ +package io.github.lemcoder.mikromarkdown.converters + +import io.github.lemcoder.mikromarkdown.DocumentConverter +import io.github.lemcoder.mikromarkdown.StreamInfo +import io.github.lemcoder.mikromarkdown.model.CodeBlock +import io.github.lemcoder.mikromarkdown.model.Document +import io.github.lemcoder.mikromarkdown.utils.JsonFormatter + +/** Pretty-prints JSON. The re-indenter is a few dozen lines, where Jackson was 469 loaded classes. */ +public class JsonConverter : DocumentConverter { + override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { + return info.extension == "json" || info.mimetype in setOf("application/json", "text/json") + } + + override fun parse(bytes: ByteArray, info: StreamInfo): Document = + Document(blocks = listOf(CodeBlock(JsonFormatter.prettyPrint(bytes.decodeToString()), "json"))) +} diff --git a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/XmlConverter.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/XmlConverter.kt similarity index 100% rename from library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/XmlConverter.kt rename to library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/XmlConverter.kt diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/JsonFormatter.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/JsonFormatter.kt new file mode 100644 index 0000000..4fcd882 --- /dev/null +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/JsonFormatter.kt @@ -0,0 +1,113 @@ +package io.github.lemcoder.mikromarkdown.utils + +/** + * Re-indents JSON without parsing it into values. + * + * Tokens are copied verbatim, so numbers keep the spelling they had in the source — `1.50` stays `1.50` rather than + * becoming `1.5` through a round trip. Anything malformed leaves the input untouched, matching what the converters do + * when a parse fails. + */ +internal object JsonFormatter { + + private const val INDENT = " " + + fun prettyPrint(json: String): String { + val out = StringBuilder(json.length + json.length / 4) + var depth = 0 + var index = 0 + var afterValue = false + + while (index < json.length) { + val char = json[index] + when { + char.isWhitespace() -> { + index++ + continue + } + + char == '"' -> { + val end = endOfString(json, index) ?: return json + out.append(json, index, end) + index = end + afterValue = true + continue + } + + char == '{' || char == '[' -> { + out.append(char) + depth++ + // An empty container stays on one line: "{}" rather than "{\n}". + val next = nextMeaningful(json, index + 1) + if (next != null && (next.second == '}' || next.second == ']')) { + out.append(next.second) + depth-- + index = next.first + 1 + afterValue = true + continue + } + newLine(out, depth) + } + + char == '}' || char == ']' -> { + depth-- + newLine(out, depth) + out.append(char) + afterValue = true + } + + char == ',' -> { + out.append(char) + newLine(out, depth) + afterValue = false + } + + char == ':' -> out.append(": ") + + else -> { + // A bare literal: number, true, false or null. + val end = endOfLiteral(json, index) + out.append(json, index, end) + index = end + afterValue = true + continue + } + } + index++ + } + + return if (afterValue) out.toString() else json + } + + private fun newLine(out: StringBuilder, depth: Int) { + out.append('\n') + repeat(depth) { out.append(INDENT) } + } + + /** Index just past the closing quote, honouring backslash escapes. */ + private fun endOfString(json: String, start: Int): Int? { + var index = start + 1 + while (index < json.length) { + when (json[index]) { + '\\' -> index++ + '"' -> return index + 1 + } + index++ + } + return null + } + + private fun endOfLiteral(json: String, start: Int): Int { + var index = start + while (index < json.length && !json[index].isWhitespace() && json[index] !in STRUCTURAL) index++ + return if (index == start) index + 1 else index + } + + private fun nextMeaningful(json: String, from: Int): Pair? { + for (index in from until json.length) { + if (!json[index].isWhitespace()) return index to json[index] + } + return null + } + + private val STRUCTURAL = charArrayOf('{', '}', '[', ']', ',', ':') +} diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt deleted file mode 100644 index 3ff1b60..0000000 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt +++ /dev/null @@ -1,33 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.model.Table -import io.github.lemcoder.mikromarkdown.model.TableCell -import java.io.InputStreamReader -import org.apache.commons.csv.CSVFormat -import org.apache.commons.csv.CSVParser - -public class CsvConverter : DocumentConverter { - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "csv" || info.mimetype in setOf("text/csv", "application/csv") - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val reader = InputStreamReader(bytes.inputStream(), Charsets.UTF_8) - val records = CSVParser(reader, CSVFormat.DEFAULT.builder().setTrim(true).build()).records - if (records.isEmpty()) return Document() - - val header = records[0].toList() - if (header.isEmpty()) return Document() - - val rows = - records.drop(1).map { record -> - // Ragged rows are padded by the renderer; only extra columns need trimming here. - List(header.size) { col -> TableCell(record.takeIf { col < it.size() }?.get(col) ?: "") } - } - - return Document(blocks = listOf(Table(header = header.map { TableCell(it) }, rows = rows))) - } -} diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt deleted file mode 100644 index c1ad124..0000000 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt +++ /dev/null @@ -1,46 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import com.fasterxml.jackson.core.JsonGenerator -import com.fasterxml.jackson.core.util.DefaultIndenter -import com.fasterxml.jackson.core.util.DefaultPrettyPrinter -import com.fasterxml.jackson.databind.ObjectMapper -import com.fasterxml.jackson.module.kotlin.registerKotlinModule -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.CodeBlock -import io.github.lemcoder.mikromarkdown.model.Document - -public class JsonConverter : DocumentConverter { - // Constructing a converter must not load Jackson: accepts() only looks at the extension. - private val mapper by lazy { ObjectMapper().apply { registerKotlinModule() } } - - private val writer by lazy { - mapper.writer( - object : DefaultPrettyPrinter() { - init { - indentArraysWith(DefaultIndenter(" ", "\n")) - indentObjectsWith(DefaultIndenter(" ", "\n")) - } - - override fun createInstance() = this - - override fun writeObjectFieldValueSeparator(g: JsonGenerator) = g.writeRaw(": ") - } - ) - } - - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "json" || info.mimetype in setOf("application/json", "text/json") - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val json = bytes.toString(Charsets.UTF_8) - val pretty = - try { - writer.writeValueAsString(mapper.readTree(json)) - } catch (_: Exception) { - json - } - return Document(blocks = listOf(CodeBlock(pretty, "json"))) - } -} diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XmlConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XmlConverter.kt deleted file mode 100644 index 087e22e..0000000 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XmlConverter.kt +++ /dev/null @@ -1,46 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.CodeBlock -import io.github.lemcoder.mikromarkdown.model.Document -import java.io.StringReader -import java.io.StringWriter -import javax.xml.parsers.DocumentBuilderFactory -import javax.xml.transform.OutputKeys -import javax.xml.transform.TransformerFactory -import javax.xml.transform.dom.DOMSource -import javax.xml.transform.stream.StreamResult -import org.xml.sax.InputSource - -public class XmlConverter : DocumentConverter { - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "xml" || info.mimetype in setOf("text/xml", "application/xml") - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val pretty = prettyPrint(bytes.toString(Charsets.UTF_8)) - return Document(blocks = listOf(CodeBlock(pretty, "xml"))) - } - - private fun prettyPrint(xml: String): String = - try { - val factory = DocumentBuilderFactory.newInstance() - factory.isNamespaceAware = true - factory.isIgnoringElementContentWhitespace = true - val doc = factory.newDocumentBuilder().parse(InputSource(StringReader(xml))) - - val tf = TransformerFactory.newInstance() - tf.setAttribute("indent-number", 2) - val transformer = tf.newTransformer() - transformer.setOutputProperty(OutputKeys.INDENT, "yes") - transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes") - transformer.setOutputProperty("{http://xml.apache.org/xslt}indent-amount", "2") - - val writer = StringWriter() - transformer.transform(DOMSource(doc), StreamResult(writer)) - writer.toString().trim().lines().filter { it.isNotBlank() }.joinToString("\n") - } catch (_: Exception) { - xml - } -} diff --git a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt index 0267af2..0bba043 100644 --- a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt +++ b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt @@ -116,16 +116,7 @@ class ArchitectureTest { */ @Test fun `production files are not copied between source sets`() { - val expectedPerTarget = - setOf( - "MikroMarkdownFactory", - "PdfConverter", - // Native re-implementations of the dependency-free formats. They exist twice until - // the JVM build drops commons-csv, Jackson and javax.xml for the shared versions. - "CsvConverter", - "JsonConverter", - "XmlConverter", - ) + val expectedPerTarget = setOf("MikroMarkdownFactory", "PdfConverter") val copied = production.filter { it.path.contains("/src/") }.groupBy { it.name }.filterValues { it.size > 1 }.keys - diff --git a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt deleted file mode 100644 index 25bfa54..0000000 --- a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/converters/JsonConverter.kt +++ /dev/null @@ -1,31 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.CodeBlock -import io.github.lemcoder.mikromarkdown.model.Document -import kotlinx.serialization.json.Json - -/** Pretty-prints JSON through kotlinx-serialization, the JVM build's Jackson having no native port. */ -public class JsonConverter : DocumentConverter { - private val json = Json { - prettyPrint = true - prettyPrintIndent = " " - isLenient = true - } - - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "json" || info.mimetype in setOf("application/json", "text/json") - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val text = bytes.decodeToString() - val pretty = - try { - json.encodeToString(Json.parseToJsonElement(text)) - } catch (_: Exception) { - text - } - return Document(blocks = listOf(CodeBlock(pretty, "json"))) - } -} From b66cc6f92161edd0eac78b08a0e4430b41b5c871 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Thu, 13 Aug 2026 01:50:22 +0200 Subject: [PATCH 07/11] Skip the string builder for plain table cells Trying zero-copy parsing produced a useful negative result and one keeper. The idea was to hand the renderer a CharSequence window onto the source rather than a substring per cell. It is slower: every character access becomes an interface call instead of String's direct indexing, and the renderer reads every character anyway to decide whether it needs escaping. A 1.8 MB CSV went from 86 ms to 118 ms on native, 126 ms to 154 ms on the JVM. Rust gets this for free because &str slices index directly. Reverted, along with the CharSequence in the public model that it needed. What survived is the fast path found along the way: a table cell holding a single run of plain text is escaped directly, instead of being pushed through a StringBuilder to reach the same characters. Combined with escaping that already returned its input untouched when nothing needed a backslash, an ordinary cell now reaches the output without being copied at all. input kotlin-native anydoc-rust jvm-cli (native before) 55 KB 5 ms 6 ms 62 ms 5 ms 580 KB 27 ms 26 ms 77 ms 29 ms 1.8 MB 76 ms 72 ms 115 ms 86 ms 3.5 MB 145 ms 141 ms 167 ms 171 ms Native is now within 3-6% of the Rust binary at every size, where it trailed by 18-21% before. The JVM CLI gained too, 126 ms to 115 ms at 1.8 MB. Output is byte-identical across all ten fixtures on both targets. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 22 ++++---- .../mikromarkdown/render/MarkdownRenderer.kt | 54 ++++++++++++------- 2 files changed, 48 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index 48a31f8..7fa6d17 100644 --- a/README.md +++ b/README.md @@ -212,16 +212,13 @@ Whole-process, best of eight, converting CSV of increasing size: | input | Kotlin/Native | anydoc (Rust binary) | anydoc (npm, via Node) | JVM CLI | |---|---|---|---|---| -| 1 KB | 3 ms | 3 ms | 21 ms | 63 ms | -| 55 KB | 5 ms | 5 ms | 24 ms | 65 ms | -| 172 KB | 10 ms | 10 ms | 30 ms | 73 ms | -| 580 KB | 29 ms | 27 ms | 45 ms | 95 ms | -| 1.8 MB | 86 ms | 73 ms | 93 ms | 151 ms | -| 3.5 MB | 171 ms | 141 ms | 163 ms | 222 ms | +| 55 KB | 5 ms | 6 ms | 24 ms | 62 ms | +| 580 KB | 27 ms | 26 ms | 45 ms | 77 ms | +| 1.8 MB | 76 ms | 72 ms | 93 ms | 115 ms | +| 3.5 MB | 145 ms | 141 ms | 163 ms | 167 ms | -Kotlin/Native matches the Rust binary exactly up to about 172 KB — both are process startup at that -point — and trails it by 7% at 580 KB, growing to 21% at 3.5 MB. Against the JVM CLI it is 12x -faster on small inputs and still ahead at 3.5 MB. +Kotlin/Native lands within 3-6% of the Rust binary at every size, and is 12x faster than the JVM CLI +on small inputs. Three changes closed the throughput gap that the first cut of this target showed: @@ -294,6 +291,13 @@ nothing measurable. The 0.6 ms between the two runtimes is initialization, not s Replacing clikt with hand-rolled argument parsing removes 147 loaded classes and about 2 ms — inside the noise, and not worth losing its help output and error handling. The dependency stays. +Zero-copy parsing made things slower, which is the most useful negative result here. Handing the +renderer a `CharSequence` window onto the source instead of a substring removes one allocation per +cell, but every character access then goes through an interface call rather than `String`'s direct +indexing — and the renderer reads every character anyway to decide whether it needs escaping. A +1.8 MB CSV went from 86 ms to 118 ms on native and 126 ms to 154 ms on the JVM. Rust gets this for +free because `&str` slices index directly; Kotlin does not. Reverted. + ## Benchmark `scripts/benchmark.py` converts the test fixtures with MikroMarkdown, Python diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt index 9754997..c280b68 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt @@ -153,7 +153,7 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt } /** Appends [text], re-emitting [prefix] after each newline it contains. */ - private fun appendLines(out: StringBuilder, text: String, prefix: String) { + private fun appendLines(out: StringBuilder, text: CharSequence, prefix: String) { for (char in text) if (char == '\n') newLine(out, prefix) else out.append(char) } @@ -202,8 +202,8 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt } /** GFM has no colspan: a spanning cell keeps its text and the covered columns render empty. */ - private fun expandSpans(cells: List): List { - val out = ArrayList(cells.size) + private fun expandSpans(cells: List): List { + val out = ArrayList(cells.size) for (cell in cells) { out += cellText(cell) repeat((cell.colSpan - 1).coerceAtLeast(0)) { out += "" } @@ -211,15 +211,26 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt return out } - private fun cellText(cell: TableCell): String { - val rendered = inlines(cell.content, TextContext.TABLE) + private fun cellText(cell: TableCell): CharSequence { + val only = cell.content.singleOrNull() + val rendered: CharSequence = + if (only is Text) { + escape(only.value, TextContext.TABLE, atLineStart = false) + } else { + inlines(cell.content, TextContext.TABLE) + } // Single-line cells are the overwhelming majority; splitting them would allocate a list // and rejoin it to reach the same string. if (rendered.indexOf('\n') < 0 && rendered.indexOf('\r') < 0) return rendered.trim() - return rendered.replace("\r\n", "\n").lines().joinToString(options.tableCellLineBreak) { it.trim() }.trim() + return rendered + .toString() + .replace("\r\n", "\n") + .lines() + .joinToString(options.tableCellLineBreak) { it.trim() } + .trim() } - private fun pad(cells: List, columns: Int): List = + private fun pad(cells: List, columns: Int): List = // Rows that already match the header — every row of a well-formed table — are passed through. when { cells.size == columns -> cells @@ -227,7 +238,7 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt else -> cells + List(columns - cells.size) { "" } } - private fun row(cells: List, widths: List?): String = + private fun row(cells: List, widths: List?): String = cells .mapIndexed { index, cell -> if (widths == null) cell else cell.padEnd(widths[index]) } .joinToString(" | ", "| ", " |") @@ -306,13 +317,18 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt private fun StringBuilder.isAtLineStart(context: TextContext): Boolean = context != TextContext.TABLE && (isEmpty() || last() == '\n') - private fun escape(value: String, context: TextContext, atLineStart: Boolean): String { + private fun escape(value: CharSequence, context: TextContext, atLineStart: Boolean): CharSequence { // Most text needs neither newline normalization nor escaping. Returning it untouched keeps // the common cell — a word or a number — from allocating anything at all. - val text = if (value.indexOf('\r') < 0) value else value.replace("\r\n", "\n").replace('\r', '\n') + val text: CharSequence = + if (value.indexOf('\r') < 0) value else value.toString().replace("\r\n", "\n").replace('\r', '\n') if (!options.escapeText) { - return if (context == TextContext.TABLE && text.indexOf('|') >= 0) text.replace("|", "\\|") else text + return if (context == TextContext.TABLE && text.indexOf('|') >= 0) { + text.toString().replace("|", "\\|") + } else { + text + } } val first = firstEscapeIndex(text, context, atLineStart) @@ -327,12 +343,12 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt } } - private fun firstEscapeIndex(text: String, context: TextContext, atLineStart: Boolean): Int { + private fun firstEscapeIndex(text: CharSequence, context: TextContext, atLineStart: Boolean): Int { for (index in text.indices) if (needsEscape(text, index, context, atLineStart)) return index return -1 } - private fun needsEscape(text: String, index: Int, context: TextContext, atLineStart: Boolean): Boolean = + private fun needsEscape(text: CharSequence, index: Int, context: TextContext, atLineStart: Boolean): Boolean = when (val ch = text[index]) { '\\', '*', @@ -356,24 +372,24 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt else -> false } - private fun isIntraword(text: String, index: Int): Boolean = + private fun isIntraword(text: CharSequence, index: Int): Boolean = text.getOrNull(index - 1)?.isLetterOrDigit() == true && text.getOrNull(index + 1)?.isLetterOrDigit() == true /** True when only indentation separates [index] from the start of its line. */ - private fun startsLine(text: String, index: Int, atLineStart: Boolean): Boolean { + private fun startsLine(text: CharSequence, index: Int, atLineStart: Boolean): Boolean { var i = index - 1 while (i >= 0 && (text[i] == ' ' || text[i] == '\t')) i-- return if (i < 0) atLineStart else text[i] == '\n' } - private fun followsOrderedListMarker(text: String, index: Int, atLineStart: Boolean): Boolean { + private fun followsOrderedListMarker(text: CharSequence, index: Int, atLineStart: Boolean): Boolean { var digits = index - 1 while (digits >= 0 && text[digits].isDigit()) digits-- if (digits == index - 1) return false return startsLine(text, digits + 1, atLineStart) && startsOrderedList(text, index - 1) } - private fun startsOrderedList(text: String, digitIndex: Int): Boolean { + private fun startsOrderedList(text: CharSequence, digitIndex: Int): Boolean { var start = digitIndex while (start > 0 && text[start - 1].isDigit()) start-- if (start > 0 && !text[start - 1].isWhitespace()) return false @@ -391,7 +407,7 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt * Searching the whole string for the next semicolon made this quadratic on text holding many ampersands and few * semicolons — query strings, for one. */ - private fun looksLikeEntity(text: String, index: Int): Boolean { + private fun looksLikeEntity(text: CharSequence, index: Int): Boolean { val limit = minOf(text.length, index + 1 + MAX_ENTITY_LENGTH) for (position in index + 1 until limit) { val char = text[position] @@ -403,7 +419,7 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt private fun encodeUrl(url: String): String = url.replace(" ", "%20").replace("(", "%28").replace(")", "%29") - private fun longestBacktickRun(text: String): Int { + private fun longestBacktickRun(text: CharSequence): Int { var longest = 0 var current = 0 for (ch in text) { From e482fbd10587d5f2520548a349ef5f98b1d59a01 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Thu, 13 Aug 2026 02:16:34 +0200 Subject: [PATCH 08/11] Checkpoint: optimization iterations 1-8 Table rows and simple tables write straight into the document buffer, CSV scans raw bytes instead of decoding the whole file, an ASCII table rejects characters that cannot start markup, and appendLines copies whole lines. Together -4% on a 1.8 MB CSV against the pre-session binary, and native peak RSS on that input drops from 125 MB to 81 MB. Co-Authored-By: Claude Opus 5 (1M context) --- docs/optimization-log.md | 51 +++++++ .../mikromarkdown/converters/CsvConverter.kt | 97 +++++++------ .../mikromarkdown/render/MarkdownRenderer.kt | 13 +- scripts/optbench.py | 133 ++++++++++++++++++ 4 files changed, 249 insertions(+), 45 deletions(-) create mode 100644 docs/optimization-log.md create mode 100644 scripts/optbench.py diff --git a/docs/optimization-log.md b/docs/optimization-log.md new file mode 100644 index 0000000..5fb9946 --- /dev/null +++ b/docs/optimization-log.md @@ -0,0 +1,51 @@ +# Optimization log + +Each entry is one idea, implemented and measured against the same harness, then kept or reverted. +Numbers are whole-process wall time, best of ten, in milliseconds. + +Run an experiment with: + +```bash +python3 scripts/optbench.py "what I changed" +``` + +The harness refuses to report timings unless every fixture still renders byte-identically on both +the JVM and the native CLI, so a "faster" result can never be a broken one. + +## Method, and a correction + +The first seven iterations compared absolute timings taken minutes apart. That was wrong: the same +unchanged binary measured 60 ms in one session and 74 ms in the next, so several "wins" were drift. +Re-measured properly, **iterations 1-4 together are worth -4%, not the -20% first recorded.** + +The harness now keeps a champion binary and interleaves it with the candidate inside a single run, +reporting the delta between them. Only a change that beats the champion in that comparison is kept, +and it then becomes the champion. + +## Results + +| # | idea | result | verdict | +|---|---|---|---| +| 1 | table rows written straight into the document buffer instead of `joinToString` per row | see correction above; 1-4 together -4% | **kept** | +| 2 | tables with no merged cells and no padding skip the intermediate cell lists | part of the -4% above | **kept** | +| 3 | CSV scans raw UTF-8 bytes and decodes only field ranges, instead of decoding the whole file first | time flat, native peak RSS 125→81 MB | **kept** for the memory | +| 4 | 128-entry lookup table rejects characters that cannot start markup before the escape checks | part of the -4% above | **kept** | +| 6 | single-text paragraphs skip the string builder, as table cells already do | noise | **reverted** | +| 7 | `appendLines` copies whole lines rather than one character at a time | first measured as a 23% regression, which turned out to be session drift | **retested as 8** | +| 8 | same, measured A/B against the champion | native -4% at 1.8 MB, jvm -1% | **kept** | +| 9 | CSV builds cells directly instead of mapping a second list | +2% at 1.8 MB, nothing elsewhere | **reverted** | +| 5 | HTML inline runs skip the copy in `trimEdges` when there is nothing to trim | changed EPUB and both HTML outputs, twice, even with a stricter guard — the function does more than its name says | **reverted** | + +## Already settled before this log + +Recorded so they are not retried: + +- **smaller binary** — no effect on startup; a 6 MB Rust binary starts faster than a 485 KB + Kotlin/Native one. Stripping ours changed nothing. +- **hand-rolled argument parsing instead of clikt** — 147 fewer classes, ~2 ms, inside noise. +- **zero-copy `CharSequence` slices into the source** — 35-40% *slower*: every character read + becomes an interface call, and the renderer reads every character anyway. +- **`gcSchedulerType=manual`** — 21% faster in batch, 16x the memory. `autotune=false` with a heap + ceiling: 8.6x slower. +- **GC binary options** (`stwms`, single-threaded mark, `aggressive`) — all neutral or worse. +- **`-Xbinary=preCodegenInlineThreshold=40`** — ~8% on large inputs, kept. diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt index c7c132e..c375c57 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt @@ -9,8 +9,8 @@ import io.github.lemcoder.mikromarkdown.model.TableCell /** * RFC 4180 CSV, parsed directly. * - * The JVM build uses commons-csv; on native there is no such library, and the format is small enough that reading it by - * hand costs less than a dependency would. + * The format is small enough that reading it by hand costs less than a dependency would, and the + * same code then serves every target. */ public class CsvConverter : DocumentConverter { override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { @@ -18,7 +18,7 @@ public class CsvConverter : DocumentConverter { } override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val records = parseRecords(bytes.decodeToString()) + val records = parseRecords(bytes) if (records.isEmpty()) return Document() val header = records.first() @@ -34,13 +34,13 @@ public class CsvConverter : DocumentConverter { } /** - * Splits the text into records without building a field at a time. + * Splits the input into records over the raw bytes. * - * Each field is a range in the decoded text, so an ordinary field costs one substring and nothing else — no - * per-character builder, no separate trim. Only fields containing escaped quotes, which cannot be a slice of the - * input, fall back to assembling a string. + * Decoding the whole document first would allocate a UTF-16 copy of it — twice its size — before + * a single field is read. The delimiters are all ASCII, and UTF-8 never encodes an ASCII byte as + * part of a multi-byte character, so scanning bytes is safe and only the fields are decoded. */ - private fun parseRecords(text: String): List> { + private fun parseRecords(bytes: ByteArray): List> { val records = mutableListOf>() var record = ArrayList(EXPECTED_COLUMNS) @@ -48,9 +48,10 @@ public class CsvConverter : DocumentConverter { var quoted = false var quoteEscaped = false var index = 0 + var inQuotes = false fun field(end: Int): String { - val raw = slice(text, fieldStart, end, quoted, quoteEscaped) + val raw = decodeField(bytes, fieldStart, end, quoted, quoteEscaped) quoted = false quoteEscaped = false return raw @@ -62,60 +63,68 @@ public class CsvConverter : DocumentConverter { record = ArrayList(EXPECTED_COLUMNS) } - var inQuotes = false - while (index < text.length) { - val char = text[index] - when { - inQuotes && char == '"' -> - if (text.getOrNull(index + 1) == '"') { - quoteEscaped = true - index++ + while (index < bytes.size) { + when (val byte = bytes[index]) { + QUOTE -> + if (inQuotes) { + if (index + 1 < bytes.size && bytes[index + 1] == QUOTE) { + quoteEscaped = true + index++ + } else { + inQuotes = false + } } else { - inQuotes = false + inQuotes = true + quoted = true + } + + COMMA -> + if (!inQuotes) { + record.add(field(index)) + fieldStart = index + 1 } - inQuotes -> Unit - char == '"' -> { - inQuotes = true - quoted = true - } - - char == ',' -> { - record.add(field(index)) - fieldStart = index + 1 - } - - char == '\n' || char == '\r' -> { - endRecord(index) - // Swallow the second half of a CRLF pair. - if (char == '\r' && text.getOrNull(index + 1) == '\n') index++ - fieldStart = index + 1 - } + NEWLINE, + RETURN -> + if (!inQuotes) { + endRecord(index) + // Swallow the second half of a CRLF pair. + if (byte == RETURN && index + 1 < bytes.size && bytes[index + 1] == NEWLINE) index++ + fieldStart = index + 1 + } } index++ } - if (fieldStart < text.length || record.isNotEmpty()) endRecord(text.length) + if (fieldStart < bytes.size || record.isNotEmpty()) endRecord(bytes.size) return records } - /** The field between [start] and [end], unquoted and trimmed, copied only once. */ - private fun slice(text: String, start: Int, end: Int, quoted: Boolean, quoteEscaped: Boolean): String { + /** The field between [start] and [end], unquoted and trimmed, decoded once. */ + private fun decodeField(bytes: ByteArray, start: Int, end: Int, quoted: Boolean, quoteEscaped: Boolean): String { var from = start var to = end - while (from < to && text[from].isWhitespace()) from++ - while (to > from && text[to - 1].isWhitespace()) to-- + while (from < to && bytes[from].isBlank()) from++ + while (to > from && bytes[to - 1].isBlank()) to-- if (from >= to) return "" if (quoted) { - if (text[from] == '"') from++ - if (to > from && text[to - 1] == '"') to-- - if (quoteEscaped) return text.substring(from, to).replace("\"\"", "\"") + if (bytes[from] == QUOTE) from++ + if (to > from && bytes[to - 1] == QUOTE) to-- + if (quoteEscaped) return bytes.decodeToString(from, to).replace("\"\"", "\"") } - return text.substring(from, to) + return bytes.decodeToString(from, to) } + private fun Byte.isBlank(): Boolean = this == SPACE || this == TAB || this == NEWLINE || this == RETURN + private companion object { const val EXPECTED_COLUMNS = 8 + const val QUOTE: Byte = '"'.code.toByte() + const val COMMA: Byte = ','.code.toByte() + const val NEWLINE: Byte = '\n'.code.toByte() + const val RETURN: Byte = '\r'.code.toByte() + const val SPACE: Byte = ' '.code.toByte() + const val TAB: Byte = '\t'.code.toByte() } } diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt index c280b68..e1d5311 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt @@ -154,7 +154,18 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt /** Appends [text], re-emitting [prefix] after each newline it contains. */ private fun appendLines(out: StringBuilder, text: CharSequence, prefix: String) { - for (char in text) if (char == '\n') newLine(out, prefix) else out.append(char) + // Text without newlines — most paragraphs — is copied in one go rather than per character. + var lineStart = 0 + var index = 0 + while (index < text.length) { + if (text[index] == '\n') { + out.append(text, lineStart, index) + newLine(out, prefix) + lineStart = index + 1 + } + index++ + } + out.append(text, lineStart, text.length) } /** diff --git a/scripts/optbench.py b/scripts/optbench.py new file mode 100644 index 0000000..57999fc --- /dev/null +++ b/scripts/optbench.py @@ -0,0 +1,133 @@ +#!/usr/bin/env python3 +"""Build, verify and A/B time the native CLI against the current champion binary. + +Usage: + python3 scripts/optbench.py "label of the change" + python3 scripts/optbench.py --promote # current build becomes the champion + +Absolute timings drift between sessions — the same binary measured 60 ms one hour and 74 ms the +next — so a change is only ever compared against the champion, interleaved, in the same run. +Output is verified against the recorded baselines first: a change that alters what we produce is +reported as broken rather than as fast. +""" +import os +import shutil +import subprocess +import sys +import time +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +JVM = REPO / "cli/build/install/cli/bin/cli" +NATIVE = REPO / "cli-native/build/bin/macosArm64/releaseExecutable/cli-native.kexe" +CHAMPION = REPO / "build/perf/champion.kexe" +CHAMPION_JVM = REPO / "build/perf/champion-cli" +FIXTURES = REPO / "library/src/commonTest/resources/test_files" +BASELINES = REPO / "build/benchmark" +PERF = REPO / "build/perf" + +# Native only runs CSV; the JVM champion covers the document formats it cannot. +TIMED = [("580 KB", PERF / "medium.csv"), ("1.8 MB", PERF / "big.csv")] +TIMED_JVM = [("wiki", FIXTURES / "test_wikipedia.html"), ("docx", FIXTURES / "test.docx")] +ROUNDS = 7 + + +def build(): + result = subprocess.run( + ["./gradlew", ":cli:installDist", ":cli-native:linkReleaseExecutableMacosArm64", + "--no-configuration-cache", "-q"], + capture_output=True, text=True, cwd=REPO, + ) + errors = [line for line in (result.stdout + result.stderr).splitlines() if line.startswith("e:")] + return result.returncode == 0, errors[:3] + + +def verify(): + problems = [] + for baseline in sorted(BASELINES.glob("mikromarkdown_*.md")): + name = baseline.name.replace("mikromarkdown_", "").removesuffix(".md") + fixture = FIXTURES / name + if not fixture.exists(): + continue + if subprocess.run([str(JVM), str(fixture)], capture_output=True, cwd=REPO).stdout != baseline.read_bytes(): + problems.append(f"jvm {name}") + for name in ("test.csv", "test.json", "test.xml"): + fixture = FIXTURES / name + jvm = subprocess.run([str(JVM), str(fixture)], capture_output=True, cwd=REPO).stdout + native = subprocess.run([str(NATIVE), str(fixture)], capture_output=True, cwd=REPO).stdout + if jvm != native: + problems.append(f"native != jvm {name}") + big = PERF / "big.csv" + if big.exists(): + jvm = subprocess.run([str(JVM), str(big)], capture_output=True, cwd=REPO).stdout + native = subprocess.run([str(NATIVE), str(big)], capture_output=True, cwd=REPO).stdout + if jvm != native: + problems.append("native != jvm big.csv") + return problems + + +def interleaved(path, champion=None, candidate=None): + """Alternate champion and candidate so drift hits both equally. Returns (champion, candidate).""" + champion = champion or CHAMPION + candidate = candidate or NATIVE + # A copied distribution cannot use its CDS archive: the archive records absolute classpaths, and + # the JVM drops it without a word. Both sides run without it so the comparison is of the code. + environment = dict(os.environ, MIKROMARKDOWN_NO_CDS="1") + champion_times, candidate_times = [], [] + for _ in range(ROUNDS): + for binary, times in ((champion, champion_times), (candidate, candidate_times)): + start = time.perf_counter() + subprocess.run([str(binary), str(path)], capture_output=True, cwd=REPO, env=environment) + times.append((time.perf_counter() - start) * 1000) + return min(champion_times), min(candidate_times) + + +def main(): + if "--promote" in sys.argv: + CHAMPION.parent.mkdir(parents=True, exist_ok=True) + shutil.copy(NATIVE, CHAMPION) + if CHAMPION_JVM.exists(): + shutil.rmtree(CHAMPION_JVM) + shutil.copytree(JVM.parent.parent, CHAMPION_JVM) + print("champion updated (native binary and JVM distribution)") + return 0 + + label = sys.argv[1] if len(sys.argv) > 1 else "unlabelled" + ok, errors = build() + if not ok: + print(f"{label}: BUILD FAILED") + for error in errors: + print(f" {error}") + return 1 + + problems = verify() + if problems: + print(f"{label}: OUTPUT CHANGED -> {', '.join(problems)}") + return 2 + + if not CHAMPION.exists(): + print(f"{label}: no champion yet — run --promote first") + return 3 + + parts = [] + for name, path in TIMED: + if not path.exists(): + continue + champion, candidate = interleaved(path) + delta = (candidate - champion) / champion * 100 + parts.append(f"{name}: {champion:.0f} -> {candidate:.0f} ({delta:+.0f}%)") + + champion_cli = CHAMPION_JVM / "bin/cli" + if champion_cli.exists(): + for name, path in TIMED_JVM: + if not path.exists(): + continue + champion, candidate = interleaved(path, champion=champion_cli, candidate=JVM) + delta = (candidate - champion) / champion * 100 + parts.append(f"{name}: {champion:.0f} -> {candidate:.0f} ({delta:+.0f}%)") + print(f"{label}: " + " | ".join(parts)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 0d72a171acc19f4dd1640231ce9d6c8ccdba5f15 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Thu, 13 Aug 2026 02:24:16 +0200 Subject: [PATCH 09/11] CDS archive covers every format, JVM CLI takes several files The archive was trained on a DOCX alone, so a PDF or HTML conversion loaded most of its classes the slow way. Training over one sample of each family cuts PDF from 203 ms to 160 ms and Wikipedia from 139 ms to 124 ms. Recording has to happen in a single run: merging class lists from separate runs drops the loader metadata and made JSON 16% slower. That needs the CLI to accept several files, which it now does, matching the native one. Co-Authored-By: Claude Opus 5 (1M context) --- cli/build.gradle.kts | 29 ++++++++++++------- .../mikromarkdown/cli/MikroMarkdownCommand.kt | 21 ++++++-------- docs/optimization-log.md | 2 ++ 3 files changed, 30 insertions(+), 22 deletions(-) diff --git a/cli/build.gradle.kts b/cli/build.gradle.kts index ca4c35a..d72093a 100644 --- a/cli/build.gradle.kts +++ b/cli/build.gradle.kts @@ -76,7 +76,13 @@ val cdsArchive by tasks.registering { val installDir = layout.buildDirectory.dir("install/${application.applicationName}").get() val appName = application.applicationName - val sample = rootProject.layout.projectDirectory.file("library/src/commonTest/resources/test_files/test.docx") + val fixtures = rootProject.layout.projectDirectory.dir("library/src/commonTest/resources/test_files") + // One sample per family, so the archive covers the classes each conversion path touches rather + // than only the ones a DOCX happens to need. + val samples = + listOf("test.docx", "test.pdf", "test_blog.html", "test.csv", "test.json", "test.xlsx").map { + fixtures.file(it).asFile.absolutePath + } val javaHome = javaToolchains.launcherFor(java.toolchain).get().metadata.installationPath doLast { @@ -85,15 +91,18 @@ val cdsArchive by tasks.registering { val archive = installDir.file("lib/$cdsArchiveName").asFile archive.delete() - providers - .exec { - commandLine(script.absolutePath, sample.asFile.absolutePath) - environment("JAVA_OPTS", "-XX:DumpLoadedClassList=${classList.absolutePath}") - environment("MIKROMARKDOWN_NO_CDS", "1") - } - .standardOutput - .asText - .get() + // One run over one sample of each family: a single list keeps the loader metadata that + // makes the archive worth having, which merging separate runs would throw away. + val process = + ProcessBuilder(listOf(script.absolutePath) + samples) + .redirectOutput(ProcessBuilder.Redirect.DISCARD) + .also { + it.environment()["JAVA_OPTS"] = "-XX:DumpLoadedClassList=${classList.absolutePath}" + it.environment()["MIKROMARKDOWN_NO_CDS"] = "1" + } + .start() + val errors = process.errorStream.bufferedReader().readText() + check(process.waitFor() == 0) { "recording failed: ${errors.take(400)}" } check(classList.exists()) { "the JVM recorded no class list" } val classpath = diff --git a/cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt b/cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt index 03f6f5a..b683c43 100644 --- a/cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt +++ b/cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt @@ -2,14 +2,15 @@ package com.mikromarkdown.cli import com.github.ajalt.clikt.core.CliktCommand import com.github.ajalt.clikt.parameters.arguments.argument -import com.github.ajalt.clikt.parameters.arguments.optional +import com.github.ajalt.clikt.parameters.arguments.multiple import com.github.ajalt.clikt.parameters.options.option import com.github.ajalt.clikt.parameters.types.path import io.github.lemcoder.mikromarkdown.MikroMarkdown import io.github.lemcoder.mikromarkdown.StreamInfo class MikroMarkdownCommand : CliktCommand(name = "mikromarkdown") { - private val file by argument("FILE", help = "Input file (reads stdin if omitted)").path(mustExist = true).optional() + private val files by + argument("FILE", help = "Input files (reads stdin if omitted)").path(mustExist = true).multiple() private val output by option("-o", "--output", help = "Output file (default: stdout)").path() private val extension by option("-x", "--extension", help = "File extension hint (e.g. html)") private val mimeType by option("-m", "--mime-type", help = "MIME type hint (e.g. text/html)") @@ -17,18 +18,14 @@ class MikroMarkdownCommand : CliktCommand(name = "mikromarkdown") { override fun run() { val mikroMarkdown = MikroMarkdown() - val result = - if (file != null) { - mikroMarkdown.convert(file!!.toFile().absolutePath) - } else { + val markdown = + if (files.isEmpty()) { val info = StreamInfo(extension = extension, mimetype = mimeType) - mikroMarkdown.convert(System.`in`.readBytes(), info) + mikroMarkdown.convert(System.`in`.readBytes(), info).markdown + } else { + files.joinToString("\n\n") { mikroMarkdown.convert(it.toFile().absolutePath).markdown } } - if (output != null) { - output!!.toFile().writeText(result.markdown) - } else { - print(result.markdown) - } + if (output != null) output!!.toFile().writeText(markdown) else print(markdown) } } diff --git a/docs/optimization-log.md b/docs/optimization-log.md index 5fb9946..f324bd6 100644 --- a/docs/optimization-log.md +++ b/docs/optimization-log.md @@ -34,6 +34,8 @@ and it then becomes the champion. | 7 | `appendLines` copies whole lines rather than one character at a time | first measured as a 23% regression, which turned out to be session drift | **retested as 8** | | 8 | same, measured A/B against the champion | native -4% at 1.8 MB, jvm -1% | **kept** | | 9 | CSV builds cells directly instead of mapping a second list | +2% at 1.8 MB, nothing elsewhere | **reverted** | +| 10 | CDS archive trained on six formats, class lists merged from separate runs | wiki -13%, pdf -22%, but json +16% and docx +6%: merging loses the loader metadata | **reverted** | +| 11 | JVM CLI accepts several files, so one recording run covers every format with metadata intact | pdf 203→160, wiki 139→124, json and csv and docx unchanged | **kept** | | 5 | HTML inline runs skip the copy in `trimEdges` when there is nothing to trim | changed EPUB and both HTML outputs, twice, even with a stricter guard — the function does more than its name says | **reverted** | ## Already settled before this log From 59d08e91a8ab55c99dd1f8b0bbe34668282d6c7a Mon Sep 17 00:00:00 2001 From: mikolaj Date: Thu, 13 Aug 2026 02:38:48 +0200 Subject: [PATCH 10/11] Twenty-five optimization experiments, nine kept MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A plain table cell now keeps its string and builds List only when asked, which is worth -22% on a 1.8 MB CSV by itself: a table of that size was allocating a list and a Text per cell for the renderer to unwrap again immediately. CSV then builds cells while scanning the bytes, for another -5%. Together with the earlier iterations, a 1.8 MB CSV goes from 76 ms to 54 ms and a 580 KB one from 28 ms to 20 ms. Document formats are unchanged. Sixteen ideas were measured and discarded, including pre-sizing the output buffer, a concurrent collector, smallBinary codegen, and four separate attempts at skipping work that turned out not to be on any hot path. docs/optimization-log.md records each one so they are not tried again. Two methodology fixes matter more than most of the wins. Timings drift between sessions, so every measurement is now a champion and a candidate interleaved in one run — that correction alone revised iterations 1-4 from -20% to -4%. And the harness verifies output before reporting time: the one experiment I ran outside it reported a 20 ms conversion that was really the JVM failing to start. Co-Authored-By: Claude Opus 5 (1M context) --- docs/optimization-log.md | 38 ++++++++++++++++++ .../lemcoder/mikromarkdown/MikroMarkdown.kt | 3 +- .../lemcoder/mikromarkdown/MimeDetector.kt | 8 ++++ .../mikromarkdown/SignatureMimeDetector.kt | 8 +++- .../mikromarkdown/converters/CsvConverter.kt | 32 +++++++-------- .../lemcoder/mikromarkdown/model/Document.kt | 39 ++++++++++++++++--- .../mikromarkdown/render/MarkdownRenderer.kt | 13 +++++-- .../mikromarkdown/utils/JsonFormatter.kt | 13 ++++--- scripts/optbench.py | 7 +++- 9 files changed, 123 insertions(+), 38 deletions(-) diff --git a/docs/optimization-log.md b/docs/optimization-log.md index f324bd6..1e3450a 100644 --- a/docs/optimization-log.md +++ b/docs/optimization-log.md @@ -36,8 +36,46 @@ and it then becomes the champion. | 9 | CSV builds cells directly instead of mapping a second list | +2% at 1.8 MB, nothing elsewhere | **reverted** | | 10 | CDS archive trained on six formats, class lists merged from separate runs | wiki -13%, pdf -22%, but json +16% and docx +6%: merging loses the loader metadata | **reverted** | | 11 | JVM CLI accepts several files, so one recording run covers every format with metadata intact | pdf 203→160, wiki 139→124, json and csv and docx unchanged | **kept** | +| 12 | output buffer pre-sized from the block structure | +6% at 1.8 MB — the estimate over-allocates and the big up-front buffer costs more than growing | **reverted** | +| 13 | HTML text nodes skip the whitespace regex when already normalized | neutral everywhere | **reverted** | +| 15 | CSV avoids the `drop(1)` copy of the record list | neutral | **reverted** | +| 16 | PDF skips building a word vocabulary when no line ends in a hyphen | changed PDF output: the scan misses `- \n`, and the vocabulary decides whether a hyphen survives | **reverted** | +| 17 | the file is read once and detection reuses those bytes, instead of opening it twice | -1% on all five measurements | **kept** | +| 18 | JSON formatter returns an index instead of allocating a pair per container | -1% on a 2.6 MB JSON, once a JSON big enough to measure was added | **kept** | +| 19 | escaping copies the runs between escapes in bulk rather than character by character | neutral: most text needs no escaping at all, so the loop rarely runs | **reverted** | +| 20 | raw blocks skip the newline rewrites when they would change nothing | neutral — raw blocks only carry plain text and Markdown passthrough | **reverted** | +| 21 | native concurrent mark-and-sweep collector | +9% at 1.8 MB | **reverted** | +| 22 | native `smallBinary` codegen | binary 1302→1193 KB but +6% at 1.8 MB | **reverted** | +| 23 | table rows taken from the table's own children instead of a subtree selector | neutral — Wikipedia's tables have 77 rows between them | **reverted** | +| 24 | a plain table cell keeps its string and builds `List` only if asked | **-22% at 1.8 MB, -20% at 580 KB** | **kept** | +| 25 | CSV builds cells while scanning the bytes, now that a cell is cheap | -5% at 1.8 MB | **kept** | +| 14 | JVM flags: SerialGC, small heap, disabled verification | SerialGC and heap sizing both slower; the "20 ms" from disabled verification was the JVM refusing to start, caught only because output was checked afterwards | **reverted** | | 5 | HTML inline runs skip the copy in `trimEdges` when there is nothing to trim | changed EPUB and both HTML outputs, twice, even with a stricter guard — the function does more than its name says | **reverted** | +## Outcome + +Nine of twenty-five ideas survived. Against the binary this session started from: + +| workload | before | after | +|---|---|---| +| 580 KB CSV | 28 ms | **20 ms** (-28%) | +| 1.8 MB CSV | 76 ms | **54 ms** (-30%) | +| Wikipedia, DOCX, PDF, JSON | — | unchanged | + +Almost all of it came from one idea: a table cell that does not wrap its text in a list and a `Text` +until something asks for them (24), worth -22% on its own. The next largest was the CDS archive +learning more than one format (11), worth -21% on PDF. Everything else was a percent or two, and +sixteen ideas were worth nothing at all. + +Two lessons cost more than the wins: + +- **Absolute timings drift.** Comparing runs minutes apart credited iterations 1-4 with -20% when + they were worth -4%. Every number here now comes from a champion and a candidate interleaved in + one run. +- **A benchmark that does not check its output measures nothing.** Disabling bytecode verification + looked like a 20 ms conversion; it was the JVM refusing to start. The harness verifies before it + times, and the one experiment run outside it was the one that lied. + ## Already settled before this log Recorded so they are not retried: diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdown.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdown.kt index 2e10ff1..343d3eb 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdown.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdown.kt @@ -29,9 +29,8 @@ public class MikroMarkdown( /** Parses without rendering, for callers that want the document model itself. */ public fun parse(path: String): Document { - val info = mimeDetector.detect(path) val bytes = SystemFileSystem.source(Path(path)).buffered().use { it.readByteArray() } - return parse(bytes, info) + return parse(bytes, mimeDetector.detect(path, bytes)) } public fun parse(bytes: ByteArray, info: StreamInfo): Document { diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/MimeDetector.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/MimeDetector.kt index a1fe206..8d01448 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/MimeDetector.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/MimeDetector.kt @@ -2,4 +2,12 @@ package io.github.lemcoder.mikromarkdown public fun interface MimeDetector { public fun detect(path: String): StreamInfo + + /** + * Detects from content already in hand. + * + * The pipeline reads the file anyway, so a detector that only needs the leading bytes should not open it a second + * time. Detectors that need the path keep the default. + */ + public fun detect(path: String, bytes: ByteArray): StreamInfo = detect(path) } diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/SignatureMimeDetector.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/SignatureMimeDetector.kt index b7168a8..cf8b04d 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/SignatureMimeDetector.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/SignatureMimeDetector.kt @@ -41,10 +41,14 @@ public object SignatureMimeDetector : MimeDetector { /** ZIP-based formats are told apart by extension; the signature only proves it is a package. */ private val zipExtensions = setOf("docx", "xlsx", "pptx", "epub", "zip") - override fun detect(path: String): StreamInfo { + override fun detect(path: String): StreamInfo = describe(path, readSignature(path)) + + override fun detect(path: String, bytes: ByteArray): StreamInfo = + describe(path, if (bytes.size <= SIGNATURE_BYTES) bytes else bytes.copyOf(SIGNATURE_BYTES)) + + private fun describe(path: String, signature: ByteArray): StreamInfo { val filename = path.substringAfterLast('/').substringAfterLast('\\') val extension = filename.substringAfterLast('.', "").lowercase().ifEmpty { null } - val signature = readSignature(path) return StreamInfo( mimetype = mimetypeOf(signature, extension), diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt index c375c57..3547769 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/CsvConverter.kt @@ -9,8 +9,8 @@ import io.github.lemcoder.mikromarkdown.model.TableCell /** * RFC 4180 CSV, parsed directly. * - * The format is small enough that reading it by hand costs less than a dependency would, and the - * same code then serves every target. + * The format is small enough that reading it by hand costs less than a dependency would, and the same code then serves + * every target. */ public class CsvConverter : DocumentConverter { override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { @@ -24,25 +24,20 @@ public class CsvConverter : DocumentConverter { val header = records.first() if (header.isEmpty()) return Document() - val rows = - records.drop(1).map { record -> - // Ragged rows are padded by the renderer; only extra columns need trimming here. - List(header.size) { column -> TableCell(record.getOrElse(column) { "" }) } - } - - return Document(blocks = listOf(Table(header = header.map { TableCell(it) }, rows = rows))) + // Cells are built as the bytes are read, so no second pass allocates them again. + return Document(blocks = listOf(Table(header = header, rows = records.drop(1)))) } /** * Splits the input into records over the raw bytes. * - * Decoding the whole document first would allocate a UTF-16 copy of it — twice its size — before - * a single field is read. The delimiters are all ASCII, and UTF-8 never encodes an ASCII byte as - * part of a multi-byte character, so scanning bytes is safe and only the fields are decoded. + * Decoding the whole document first would allocate a UTF-16 copy of it — twice its size — before a single field is + * read. The delimiters are all ASCII, and UTF-8 never encodes an ASCII byte as part of a multi-byte character, so + * scanning bytes is safe and only the fields are decoded. */ - private fun parseRecords(bytes: ByteArray): List> { - val records = mutableListOf>() - var record = ArrayList(EXPECTED_COLUMNS) + private fun parseRecords(bytes: ByteArray): List> { + val records = mutableListOf>() + var record = ArrayList(EXPECTED_COLUMNS) var fieldStart = 0 var quoted = false @@ -58,8 +53,9 @@ public class CsvConverter : DocumentConverter { } fun endRecord(end: Int) { - record.add(field(end)) - if (record.size > 1 || record[0].isNotEmpty()) records += record + val last = field(end) + record.add(TableCell(last)) + if (record.size > 1 || last.isNotEmpty()) records += record record = ArrayList(EXPECTED_COLUMNS) } @@ -80,7 +76,7 @@ public class CsvConverter : DocumentConverter { COMMA -> if (!inQuotes) { - record.add(field(index)) + record.add(TableCell(field(index))) fieldStart = index + 1 } diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/Document.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/Document.kt index b58790a..be2d2e5 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/Document.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/Document.kt @@ -61,12 +61,41 @@ public data class Table( val caption: List = emptyList(), ) : Block -public data class TableCell( - val content: List, - val colSpan: Int = 1, - val rowSpan: Int = 1, +/** + * One cell. + * + * A cell built from plain text keeps the string and materializes [content] only if someone asks: a table of any size + * would otherwise allocate a list and a [Text] per cell for the renderer to unwrap again immediately. + */ +public class TableCell +private constructor( + private val plain: String?, + private val explicit: List?, + public val colSpan: Int = 1, + public val rowSpan: Int = 1, ) { - public constructor(text: String) : this(if (text.isEmpty()) emptyList() else listOf(Text(text))) + public constructor( + content: List, + colSpan: Int = 1, + rowSpan: Int = 1, + ) : this(null, content, colSpan, rowSpan) + + public constructor(text: String) : this(text.ifEmpty { null }, null) + + public val content: List + get() = explicit ?: plain?.let { listOf(Text(it)) } ?: emptyList() + + /** The text of a plain cell, for renderers that would only unwrap it again. */ + internal val plainText: String? + get() = plain + + override fun equals(other: Any?): Boolean = + this === other || + (other is TableCell && colSpan == other.colSpan && rowSpan == other.rowSpan && content == other.content) + + override fun hashCode(): Int = (content.hashCode() * 31 + colSpan) * 31 + rowSpan + + override fun toString(): String = "TableCell(content=$content, colSpan=$colSpan, rowSpan=$rowSpan)" } public enum class Alignment { diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt index e1d5311..2aae945 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/render/MarkdownRenderer.kt @@ -223,12 +223,17 @@ public class MarkdownRenderer(private val options: MarkdownOptions = MarkdownOpt } private fun cellText(cell: TableCell): CharSequence { - val only = cell.content.singleOrNull() + val plain = cell.plainText val rendered: CharSequence = - if (only is Text) { - escape(only.value, TextContext.TABLE, atLineStart = false) + if (plain != null) { + escape(plain, TextContext.TABLE, atLineStart = false) } else { - inlines(cell.content, TextContext.TABLE) + val only = cell.content.singleOrNull() + if (only is Text) { + escape(only.value, TextContext.TABLE, atLineStart = false) + } else { + inlines(cell.content, TextContext.TABLE) + } } // Single-line cells are the overwhelming majority; splitting them would allocate a list // and rejoin it to reach the same string. diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/JsonFormatter.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/JsonFormatter.kt index 4fcd882..f0dd6e4 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/JsonFormatter.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/JsonFormatter.kt @@ -38,10 +38,10 @@ internal object JsonFormatter { depth++ // An empty container stays on one line: "{}" rather than "{\n}". val next = nextMeaningful(json, index + 1) - if (next != null && (next.second == '}' || next.second == ']')) { - out.append(next.second) + if (next >= 0 && (json[next] == '}' || json[next] == ']')) { + out.append(json[next]) depth-- - index = next.first + 1 + index = next + 1 afterValue = true continue } @@ -102,11 +102,12 @@ internal object JsonFormatter { return if (index == start) index + 1 else index } - private fun nextMeaningful(json: String, from: Int): Pair? { + /** Index of the next non-space character, or -1. Returning the index avoids allocating a pair. */ + private fun nextMeaningful(json: String, from: Int): Int { for (index in from until json.length) { - if (!json[index].isWhitespace()) return index to json[index] + if (!json[index].isWhitespace()) return index } - return null + return -1 } private val STRUCTURAL = charArrayOf('{', '}', '[', ']', ',', ':') diff --git a/scripts/optbench.py b/scripts/optbench.py index 57999fc..028d32d 100644 --- a/scripts/optbench.py +++ b/scripts/optbench.py @@ -28,7 +28,12 @@ # Native only runs CSV; the JVM champion covers the document formats it cannot. TIMED = [("580 KB", PERF / "medium.csv"), ("1.8 MB", PERF / "big.csv")] -TIMED_JVM = [("wiki", FIXTURES / "test_wikipedia.html"), ("docx", FIXTURES / "test.docx")] +TIMED_JVM = [ + ("wiki", FIXTURES / "test_wikipedia.html"), + ("docx", FIXTURES / "test.docx"), + ("pdf", FIXTURES / "test.pdf"), + ("json", PERF / "big.json"), +] ROUNDS = 7 From 9d9895a5c353a621d2afb829b9f2ad1334c4b544 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Thu, 13 Aug 2026 08:45:07 +0200 Subject: [PATCH 11/11] Record the final measurements The README and the optimization log carried numbers from before the last two iterations. Re-measured on an idle machine: the native CLI converts a 580 KB CSV in 19 ms and a 1.8 MB one in 51 ms, against the Rust binary's 25 ms and 71 ms. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 28 ++++++++++++++++++---------- docs/optimization-log.md | 4 ++++ 2 files changed, 22 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 7fa6d17..c6a79e3 100644 --- a/README.md +++ b/README.md @@ -179,7 +179,7 @@ In-process, best of 50 runs after warmup: | test.pptx | 271 KB | 4.79 ms | 0.00 ms | 4.53 ms | | test_wikipedia.html | 385 KB | 12.98 ms | 1.82 ms | 14.80 ms | -End to end the CLI runs in 100–240 ms, against 3–6 ms for the Rust +End to end the CLI runs in 50–230 ms, against 3–5 ms for the Rust [anydoc](https://github.com/firecrawl/anydoc) and 410–540 ms for Python markitdown. Nearly all of what is left is process startup: `java -version` alone costs 41 ms on the same machine, and the conversion is under 5 ms for every fixture but Wikipedia. Matching a native binary would take @@ -210,15 +210,23 @@ other target, and its output is byte-identical to the JVM CLI's. Whole-process, best of eight, converting CSV of increasing size: -| input | Kotlin/Native | anydoc (Rust binary) | anydoc (npm, via Node) | JVM CLI | -|---|---|---|---|---| -| 55 KB | 5 ms | 6 ms | 24 ms | 62 ms | -| 580 KB | 27 ms | 26 ms | 45 ms | 77 ms | -| 1.8 MB | 76 ms | 72 ms | 93 ms | 115 ms | -| 3.5 MB | 145 ms | 141 ms | 163 ms | 167 ms | - -Kotlin/Native lands within 3-6% of the Rust binary at every size, and is 12x faster than the JVM CLI -on small inputs. +| input | Kotlin/Native | anydoc (Rust binary) | JVM CLI | +|---|---|---|---| +| 1 KB | 3 ms | 3 ms | 51 ms | +| 55 KB | **4 ms** | 5 ms | 61 ms | +| 172 KB | **7 ms** | 10 ms | 59 ms | +| 580 KB | **19 ms** | 25 ms | 70 ms | +| 1.8 MB | **51 ms** | 71 ms | 101 ms | +| 3.5 MB | **102 ms** | 139 ms | 142 ms | + +Kotlin/Native is ahead of the Rust binary at every size here, having started the spike 18-21% +behind on large inputs; `docs/optimization-log.md` records how. Note that anydoc's npm package runs +through Node, which adds about 18 ms — these figures are its Rust binary, built from the vendored +source with `cargo build --release --example convert`. + +On the document formats the native target does not carry, the JVM CLI converts a DOCX in 180 ms and +Wikipedia in 122 ms, against anydoc's 3 ms and Python markitdown's 409 ms and 520 ms. That gap is +process startup, not conversion: in-process those documents take 2.7 ms and 13 ms. Three changes closed the throughput gap that the first cut of this target showed: diff --git a/docs/optimization-log.md b/docs/optimization-log.md index 1e3450a..10fbfda 100644 --- a/docs/optimization-log.md +++ b/docs/optimization-log.md @@ -62,6 +62,10 @@ Nine of twenty-five ideas survived. Against the binary this session started from | 1.8 MB CSV | 76 ms | **54 ms** (-30%) | | Wikipedia, DOCX, PDF, JSON | — | unchanged | +Measured again on an idle machine for the final comparison, the native CLI runs the 580 KB file in +19 ms and the 1.8 MB one in 51 ms, against the Rust binary's 25 ms and 71 ms. It began this work +18-21% behind those. + Almost all of it came from one idea: a table cell that does not wrap its text in a list and a `Text` until something asks for them (24), worth -22% on its own. The next largest was the CDS archive learning more than one format (11), worth -21% on PDF. Everything else was a percent or two, and