diff --git a/README.md b/README.md
index c6a79e3..9c7061a 100644
--- a/README.md
+++ b/README.md
@@ -6,19 +6,22 @@ Kotlin Multiplatform (JVM + Android) library that converts documents to Markdown
## Supported formats
-| Format | Extension |
-|--------|-----------|
-| Word | `.docx` |
-| Excel | `.xlsx` |
-| PowerPoint | `.pptx` |
-| EPUB | `.epub` |
-| HTML | `.html`, `.htm` |
-| PDF | `.pdf` |
-| CSV | `.csv` |
-| JSON | `.json` |
-| XML | `.xml` |
-| Plain text | `.txt` and others |
-| Markdown | `.md` (passthrough) |
+Office formats were removed deliberately: DOCX, XLSX and PPTX are editing formats, while a reader
+meets PDF and EPUB. Dropping them took Apache POI with them — the distribution went from 66 MB to
+36 MB. If they are wanted back, they return as an opt-in module the way PDF is heading, rather than
+as a dependency everyone carries.
+
+
+| Format | Extension | Notes |
+|--------|-----------|-------|
+| EPUB | `.epub` | |
+| HTML | `.html`, `.htm` | |
+| PDF | `.pdf` | opt-in: `:pdfium` module, `register(PdfiumConverter())` |
+| CSV | `.csv` | |
+| JSON | `.json` | |
+| XML | `.xml` | |
+| Plain text | `.txt` and others | |
+| Markdown | `.md` (passthrough) | |
## Architecture
@@ -34,12 +37,13 @@ bytes ──► MimeDetector ──► DocumentConverter.parse ──► Documen
```
Converters contain no Markdown syntax, so escaping, table shaping, list indentation and spacing are
-fixed once for all formats. JVM and Android share one `jvmShared` source set, so a converter exists
-once rather than per target; only PDF extraction and MIME detection are platform-specific. The model is public: `mid.parse(path)` returns the `Document`, and
-`ConversionResult.document` exposes it alongside the rendered Markdown.
+fixed once for all formats. Every converter lives in `commonMain` and runs on every target; only PDF
+is platform-specific, and it lives in its own module because it needs a native library. The model is
+public: `mid.parse(path)` returns the `Document`, and `ConversionResult.document` exposes it
+alongside the rendered Markdown.
```kotlin
-val document = mid.parse("/path/to/report.docx")
+val document = mid.parse("/path/to/book.epub")
document.blocks.filterIsInstance
().forEach { println(it.rows.size) }
// Render with different options
@@ -66,7 +70,7 @@ import io.github.lemcoder.mikromarkdown.MikroMarkdown
val mid = MikroMarkdown()
// from file path
-val result = mid.convert("/path/to/document.docx")
+val result = mid.convert("/path/to/book.epub")
// from bytes with explicit format hint
val bytes = File("document.html").readBytes()
@@ -151,170 +155,52 @@ import the renderer or each other; Markdown syntax appears only under `render/`.
every `DocumentConverter` is named `*Converter` and lives in `converters`.
*Hygiene* — no wildcard imports, no printing from library code, and no source file duplicated
-between source sets (the drift that the `jvmShared` set removed).
+between source sets.
ktfmt-gradle only derives tasks for the common and JVM source sets, so `library/build.gradle.kts`
registers matching tasks for the Android ones.
## Performance
-Conversion itself is a few milliseconds; a CLI run is mostly JVM startup and class loading.
-`:benchmark` measures the pipeline in-process, `scripts/benchmark.py` measures whole processes.
-
-```bash
-./gradlew :benchmark:run # in-process, per stage
-python3 scripts/benchmark.py # whole process, against markitdown and anydoc
-```
-
-In-process, best of 50 runs after warmup:
-
-| fixture | size | parse | render | total |
-|---|---|---|---|---|
-| test.json | 0.4 KB | 0.03 ms | 0.01 ms | 0.04 ms |
-| test.epub | 2 KB | 0.42 ms | 0.00 ms | 0.42 ms |
-| test_blog.html | 25 KB | 0.85 ms | 0.11 ms | 0.96 ms |
-| test.xlsx | 11 KB | 1.09 ms | 0.00 ms | 1.03 ms |
-| test.docx | 132 KB | 3.28 ms | 0.00 ms | 2.81 ms |
-| test.pdf | 90 KB | 3.69 ms | 0.00 ms | 3.47 ms |
-| test.pptx | 271 KB | 4.79 ms | 0.00 ms | 4.53 ms |
-| test_wikipedia.html | 385 KB | 12.98 ms | 1.82 ms | 14.80 ms |
-
-End to end the CLI runs in 50–230 ms, against 3–5 ms for the Rust
-[anydoc](https://github.com/firecrawl/anydoc) and 410–540 ms for Python markitdown. Nearly all of
-what is left is process startup: `java -version` alone costs 41 ms on the same machine, and the
-conversion is under 5 ms for every fixture but Wikipedia. Matching a native binary would take
-ahead-of-time compilation, not a faster pipeline.
-
-Note when reproducing this: anydoc's npm package is a Node script loading a napi module, so timing
-`node_modules/.bin/anydoc` charges Node's 15 ms startup to Rust and reads as ~22 ms flat. The
-figures here come from its Rust binary, built from the vendored source with
-`cargo build --release --example convert`.
-
-The CLI therefore optimizes startup rather than throughput:
-
-- `installDist` records a [class-data-sharing](https://docs.oracle.com/en/java/javase/21/vm/class-data-sharing.html)
- archive into the distribution, which roughly halves startup. Set `MIKROMARKDOWN_NO_CDS=1` to skip
- it; the start script also skips it when the archive is missing, so `distZip` still works.
-- it compiles with C1 only (`-XX:TieredStopAtLevel=1`), since C2 never pays for itself in a run this
- short. Embedders using the library get the normal JIT.
-
-### Kotlin/Native spike
-
-`:cli-native` builds a macOS binary carrying the converters that need no JVM library — CSV, JSON,
-XML, plain text and Markdown passthrough. It shares the model, renderer and pipeline with every
-other target, and its output is byte-identical to the JVM CLI's.
+The command line tool is the Kotlin/Native binary; there is no JVM CLI. `:benchmark` measures the
+library in-process on the JVM, and `scripts/optbench.py` A/B times the native binary against a saved
+champion so that session-to-session drift cannot be mistaken for a change.
```bash
./gradlew :cli-native:linkReleaseExecutableMacosArm64
+./gradlew :benchmark:run # library, in-process, per stage
+python3 scripts/optbench.py "..." # native binary, against the champion
+python3 scripts/benchmark.py # whole process, against markitdown and anydoc
```
-Whole-process, best of eight, converting CSV of increasing size:
-
-| input | Kotlin/Native | anydoc (Rust binary) | JVM CLI |
-|---|---|---|---|
-| 1 KB | 3 ms | 3 ms | 51 ms |
-| 55 KB | **4 ms** | 5 ms | 61 ms |
-| 172 KB | **7 ms** | 10 ms | 59 ms |
-| 580 KB | **19 ms** | 25 ms | 70 ms |
-| 1.8 MB | **51 ms** | 71 ms | 101 ms |
-| 3.5 MB | **102 ms** | 139 ms | 142 ms |
-
-Kotlin/Native is ahead of the Rust binary at every size here, having started the spike 18-21%
-behind on large inputs; `docs/optimization-log.md` records how. Note that anydoc's npm package runs
-through Node, which adds about 18 ms — these figures are its Rust binary, built from the vendored
-source with `cargo build --release --example convert`.
-
-On the document formats the native target does not carry, the JVM CLI converts a DOCX in 180 ms and
-Wikipedia in 122 ms, against anydoc's 3 ms and Python markitdown's 409 ms and 520 ms. That gap is
-process startup, not conversion: in-process those documents take 2.7 ms and 13 ms.
-
-Three changes closed the throughput gap that the first cut of this target showed:
-
-- **The renderer stopped allocating when it has nothing to change.** Escaping now scans for the
- first character that needs a backslash and returns the input untouched when there is none, and a
- single-line table cell skips the split-and-rejoin. Ordinary cells — a word, a number — now cost no
- allocation at all. This is shared code, so the JVM got faster too.
-- **The native CSV reader slices instead of accumulating.** Fields are ranges in the decoded text,
- so a field costs one substring rather than a per-character builder plus a separate trim. Only
- fields containing escaped quotes, which cannot be a slice of the input, assemble a string.
-- **CSV, JSON and XML moved to `commonMain`**, taking commons-csv, Jackson and
- kotlinx-serialization with them. One implementation now serves every target: a slicing CSV reader,
- a JSON re-indenter that copies tokens verbatim so `1.50` does not become `1.5`, and an XML
- formatter. A JSON conversion loads 1214 classes instead of 2063, the native binary is 1.3 MB
- instead of 2.2 MB, and the JVM CLI converts JSON in 56 ms instead of 95 ms. Output is unchanged
- on every fixture.
-- **Two quadratics in the renderer are gone.** Blocks are written into one buffer carrying a line
- prefix, rather than each block returning a string that its parent splits into lines and re-joins —
- which charged the deepest content once per level of nesting above it. And the entity check now
- scans ten characters ahead instead of searching the rest of the document for a semicolon.
- Measured on inputs built to provoke them, with output byte-identical before and after:
-
- | pathological input | before | after |
- |---|---|---|
- | 400-deep nested lists | 33.03 ms | **0.12 ms** |
- | 400 ampersands per cell, 789 KB | 9.39 ms | **2.42 ms** |
-
-Together those took a 1.8 MB CSV from 237 ms to 87 ms. Kotlin/Native's remaining cost is the
-document model itself: every cell becomes a `TableCell` holding a `Text` holding a `String`, which
-is why peak memory is 125 MB for a 1.8 MB input. A genuinely zero-copy model — inlines holding
-slices of the source buffer rather than copies — is the next lever, and a deeper change.
-
-Compiler flags were measured rather than guessed. `-Xbinary=preCodegenInlineThreshold=40` is worth
-about 8% on large inputs and ships. Every garbage collection setting tried was worse than the
-default, and the collector is the interesting part of the story, so the numbers are below.
-
-Converting 20 documents of 580 KB in one process:
-
-| policy | time | peak RSS |
-|---|---|---|
-| default (adaptive) | 543 ms | 59 MB |
-| `gcSchedulerType=manual`, never collecting | 428 ms | 954 MB |
-| `gcSchedulerType=manual`, collecting between documents | 485 ms | 67 MB |
-| `autotune = false` with a heap ceiling | 1246 ms | 43 MB |
-
-A manual collector is genuinely faster, since a process that exits never needs to collect, and a
-document boundary is the one place where everything the previous conversion allocated is provably
-dead. But it only bounds growth *between* documents: a single large input still has nothing
-collecting mid-parse, so the 3.5 MB file takes 228 MB either way and a much larger one would grow
-until it failed. Turning `autotune` off is far worse than it looks like it should be — 8.6x on a
-single 3.5 MB file — and the ceiling value makes no difference to that, so `targetHeapBytes` is not
-behaving as its name suggests.
-
-The default collector ships. The native CLI does accept several files per invocation, which is what
-would make a manual policy workable if the trade ever becomes worth it.
-
-### What did not help
+Whole process, best of ten, against the Rust [anydoc](https://github.com/firecrawl/anydoc):
-Binary size does not drive startup, so shrinking it is not a performance lever:
-
-| binary | size | startup |
+| input | Kotlin/Native | anydoc (Rust) |
|---|---|---|
-| Kotlin/Native hello world | 485 KB | 3.2 ms |
-| this CLI | 1.3 MB | 3.5 ms |
-| anydoc (Rust) | 6 MB | 2.6 ms |
-
-A 6 MB Rust binary starts faster than a 485 KB Kotlin/Native one, and stripping ours changed
-nothing measurable. The 0.6 ms between the two runtimes is initialization, not size.
+| 1 KB CSV | 3 ms | 3 ms |
+| 172 KB CSV | 7 ms | 10 ms |
+| 1.8 MB CSV | 51 ms | 71 ms |
+| 3.5 MB CSV | 102 ms | 139 ms |
+| EPUB | 5 ms | 3 ms |
+| Wikipedia HTML | 65 ms | — |
-Replacing clikt with hand-rolled argument parsing removes 147 loaded classes and about 2 ms — inside
-the noise, and not worth losing its help output and error handling. The dependency stays.
+Note when reproducing this: anydoc's npm package is a Node script loading a napi module, so timing
+`node_modules/.bin/anydoc` charges Node's 15 ms startup to Rust and reads as ~22 ms flat. The figures
+here come from its Rust binary, built from the vendored source with
+`cargo build --release --example convert`.
-Zero-copy parsing made things slower, which is the most useful negative result here. Handing the
-renderer a `CharSequence` window onto the source instead of a substring removes one allocation per
-cell, but every character access then goes through an interface call rather than `String`'s direct
-indexing — and the renderer reads every character anyway to decide whether it needs escaping. A
-1.8 MB CSV went from 86 ms to 118 ms on native and 126 ms to 154 ms on the JVM. Rust gets this for
-free because `&str` slices index directly; Kotlin does not. Reverted.
+`docs/optimization-log.md` records twenty-five measured experiments, nine of which survived, and the
+two methodology mistakes that cost more than most of the wins.
## Benchmark
-`scripts/benchmark.py` converts the test fixtures with MikroMarkdown, Python
+`scripts/benchmark.py` converts the test fixtures with the native binary, Python
[markitdown](https://github.com/microsoft/markitdown) and Rust
[anydoc](https://github.com/firecrawl/anydoc), then reports content recall, structure counts, table
integrity and timings to `build/benchmark/report.md`:
```bash
-./gradlew :cli:installDist
+./gradlew :cli-native:linkReleaseExecutableMacosArm64
python3 scripts/benchmark.py
```
diff --git a/cli-native/build.gradle.kts b/cli-native/build.gradle.kts
index befa74d..f554494 100644
--- a/cli-native/build.gradle.kts
+++ b/cli-native/build.gradle.kts
@@ -2,7 +2,15 @@ plugins { alias(libs.plugins.kotlinMultiplatform) }
kotlin {
macosArm64 {
- binaries.executable { entryPoint = "io.github.lemcoder.mikromarkdown.cli.main" }
+ // Kotlin/Native does not carry a klib's linker options to the binary that uses it, so the
+ // consumer names pdfium itself. Worth turning into a shared convention if a second
+ // consumer appears.
+ val pdfiumLib = rootProject.layout.projectDirectory.dir("pdfium/build/pdfium/mac-arm64/lib").asFile
+
+ binaries.executable {
+ entryPoint = "io.github.lemcoder.mikromarkdown.cli.main"
+ linkerOpts("-L${pdfiumLib.absolutePath}", "-lpdfium", "-rpath", pdfiumLib.absolutePath)
+ }
compilerOptions {
// Worth about 8% on large inputs and nothing on small ones. Measured, not assumed:
@@ -15,5 +23,11 @@ kotlin {
}
}
- sourceSets { macosArm64Main.dependencies { implementation(project(":library")) } }
+ sourceSets {
+ macosArm64Main.dependencies {
+ implementation(project(":library"))
+ // PDF is a separate module by design; the CLI is the thing that opts in.
+ implementation(project(":pdfium"))
+ }
+ }
}
diff --git a/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt b/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt
index 485a19c..665be14 100644
--- a/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt
+++ b/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt
@@ -2,6 +2,7 @@ package io.github.lemcoder.mikromarkdown.cli
import io.github.lemcoder.mikromarkdown.MikroMarkdown
import io.github.lemcoder.mikromarkdown.MikroMarkdownException
+import io.github.lemcoder.mikromarkdown.pdf.PdfiumConverter
import kotlin.system.exitProcess
/**
@@ -18,7 +19,8 @@ public fun main(args: Array) {
exitProcess(2)
}
- val mikroMarkdown = MikroMarkdown()
+ // PDF lives in its own module; the CLI opts in, the library does not depend on it.
+ val mikroMarkdown = MikroMarkdown().apply { register(PdfiumConverter()) }
for (path in args) {
try {
diff --git a/cli/build.gradle.kts b/cli/build.gradle.kts
deleted file mode 100644
index d72093a..0000000
--- a/cli/build.gradle.kts
+++ /dev/null
@@ -1,134 +0,0 @@
-plugins {
- alias(libs.plugins.kotlin.jvm)
- application
-}
-
-java { toolchain { languageVersion = JavaLanguageVersion.of(21) } }
-
-val cdsArchiveName = "mikromarkdown.jsa"
-
-application {
- mainClass = "com.mikromarkdown.cli.MainKt"
- // Class loading, not conversion, is what a short CLI run spends its time on. A class-data-sharing
- // archive maps those classes in pre-parsed. -Xshare:auto keeps the CLI working when it is absent.
- // Every CLI run is short, so C2 never pays for itself: compiling with C1 only is faster
- // end to end. Long-running embedders use the library directly and are unaffected.
- applicationDefaultJvmArgs = listOf("-Xshare:auto", "-XX:-UsePerfData", "-XX:TieredStopAtLevel=1")
-}
-
-dependencies {
- implementation(project(":library"))
- implementation(libs.clikt)
-}
-
-tasks.test { useJUnitPlatform() }
-
-// The archive flag is added by the script itself, and only when the archive is there: naming a
-// missing archive stops the JVM loading its base one, which is also what recording needs.
-tasks.named("startScripts") {
- doLast {
- val unixGuard =
- """
- CDS_ARCHIVE="${'$'}APP_HOME/lib/$cdsArchiveName"
- if [ -f "${'$'}CDS_ARCHIVE" ] && [ -z "${'$'}MIKROMARKDOWN_NO_CDS" ] ; then
- DEFAULT_JVM_OPTS="${'$'}DEFAULT_JVM_OPTS \"-XX:SharedArchiveFile=${'$'}CDS_ARCHIVE\""
- fi
- """
- .trimIndent()
-
- // Lambda form: the guard contains $ sequences that must not be read as group references.
- unixScript.writeText(
- unixScript.readText().replace(Regex("(?m)^(DEFAULT_JVM_OPTS=.*)${'$'}")) { match ->
- "${match.value}\n\n$unixGuard"
- }
- )
-
- val windowsGuard =
- """
- set CDS_ARCHIVE=%APP_HOME%\\lib\\$cdsArchiveName
- if exist "%CDS_ARCHIVE%" if not defined MIKROMARKDOWN_NO_CDS set DEFAULT_JVM_OPTS=%DEFAULT_JVM_OPTS% "-XX:SharedArchiveFile=%CDS_ARCHIVE%"
- """
- .trimIndent()
-
- windowsScript.writeText(
- windowsScript.readText().replace(Regex("(?m)^(set DEFAULT_JVM_OPTS=.*)${'$'}")) { match ->
- "${match.value}\r\n$windowsGuard"
- }
- )
- }
-}
-
-/**
- * Builds a class-data-sharing archive for the installed distribution, in two steps:
- *
- * 1. run the CLI once with `-XX:DumpLoadedClassList` to learn which classes a conversion touches;
- * 2. `-Xshare:dump` that list into an archive.
- *
- * This static form is used rather than `-XX:ArchiveClassesAtExit` because the dynamic one needs the JDK's own base
- * archive, which some distributions (JetBrains Runtime among them) do not ship.
- *
- * Step 1 goes through the start script so the classpath recorded is the one real runs use; step 2 reads that same
- * classpath back out of the script, because a mismatch makes the JVM drop the archive without saying so.
- */
-val cdsArchive by tasks.registering {
- group = "distribution"
- description = "Builds a class-data-sharing archive into the installed distribution."
-
- val installDir = layout.buildDirectory.dir("install/${application.applicationName}").get()
- val appName = application.applicationName
- val fixtures = rootProject.layout.projectDirectory.dir("library/src/commonTest/resources/test_files")
- // One sample per family, so the archive covers the classes each conversion path touches rather
- // than only the ones a DOCX happens to need.
- val samples =
- listOf("test.docx", "test.pdf", "test_blog.html", "test.csv", "test.json", "test.xlsx").map {
- fixtures.file(it).asFile.absolutePath
- }
- val javaHome = javaToolchains.launcherFor(java.toolchain).get().metadata.installationPath
-
- doLast {
- val script = installDir.file("bin/$appName").asFile
- val classList = installDir.file("lib/$cdsArchiveName.classlist").asFile
- val archive = installDir.file("lib/$cdsArchiveName").asFile
- archive.delete()
-
- // One run over one sample of each family: a single list keeps the loader metadata that
- // makes the archive worth having, which merging separate runs would throw away.
- val process =
- ProcessBuilder(listOf(script.absolutePath) + samples)
- .redirectOutput(ProcessBuilder.Redirect.DISCARD)
- .also {
- it.environment()["JAVA_OPTS"] = "-XX:DumpLoadedClassList=${classList.absolutePath}"
- it.environment()["MIKROMARKDOWN_NO_CDS"] = "1"
- }
- .start()
- val errors = process.errorStream.bufferedReader().readText()
- check(process.waitFor() == 0) { "recording failed: ${errors.take(400)}" }
- check(classList.exists()) { "the JVM recorded no class list" }
-
- val classpath =
- script
- .readLines()
- .first { it.startsWith("CLASSPATH=") }
- .removePrefix("CLASSPATH=")
- .replace("\$APP_HOME", installDir.asFile.absolutePath)
-
- providers
- .exec {
- commandLine(
- javaHome.file("bin/java").asFile.absolutePath,
- "-Xshare:dump",
- "-XX:SharedClassListFile=${classList.absolutePath}",
- "-XX:SharedArchiveFile=${archive.absolutePath}",
- "-cp",
- classpath,
- )
- }
- .standardOutput
- .asText
- .get()
- check(archive.exists()) { "no CDS archive was produced" }
- logger.lifecycle("CDS archive: ${archive.length() / 1024} KB")
- }
-}
-
-tasks.named("installDist") { finalizedBy(cdsArchive) }
diff --git a/cli/src/main/kotlin/com/mikromarkdown/cli/Main.kt b/cli/src/main/kotlin/com/mikromarkdown/cli/Main.kt
deleted file mode 100644
index 11974c1..0000000
--- a/cli/src/main/kotlin/com/mikromarkdown/cli/Main.kt
+++ /dev/null
@@ -1,12 +0,0 @@
-package com.mikromarkdown.cli
-
-import com.github.ajalt.clikt.core.main
-
-fun main(args: Array) {
- // PDFBox pulls in the Log4j API; without a provider it writes a banner to stdout,
- // which would corrupt the Markdown we print there.
- System.setProperty("log4j2.loggerContextFactory", "org.apache.logging.log4j.simple.SimpleLoggerContextFactory")
- System.setProperty("log4j2.simplelogLevel", "OFF")
- System.setProperty("log4j2.statusLoggerLevel", "OFF")
- MikroMarkdownCommand().main(args)
-}
diff --git a/cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt b/cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt
deleted file mode 100644
index b683c43..0000000
--- a/cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt
+++ /dev/null
@@ -1,31 +0,0 @@
-package com.mikromarkdown.cli
-
-import com.github.ajalt.clikt.core.CliktCommand
-import com.github.ajalt.clikt.parameters.arguments.argument
-import com.github.ajalt.clikt.parameters.arguments.multiple
-import com.github.ajalt.clikt.parameters.options.option
-import com.github.ajalt.clikt.parameters.types.path
-import io.github.lemcoder.mikromarkdown.MikroMarkdown
-import io.github.lemcoder.mikromarkdown.StreamInfo
-
-class MikroMarkdownCommand : CliktCommand(name = "mikromarkdown") {
- private val files by
- argument("FILE", help = "Input files (reads stdin if omitted)").path(mustExist = true).multiple()
- private val output by option("-o", "--output", help = "Output file (default: stdout)").path()
- private val extension by option("-x", "--extension", help = "File extension hint (e.g. html)")
- private val mimeType by option("-m", "--mime-type", help = "MIME type hint (e.g. text/html)")
-
- override fun run() {
- val mikroMarkdown = MikroMarkdown()
-
- val markdown =
- if (files.isEmpty()) {
- val info = StreamInfo(extension = extension, mimetype = mimeType)
- mikroMarkdown.convert(System.`in`.readBytes(), info).markdown
- } else {
- files.joinToString("\n\n") { mikroMarkdown.convert(it.toFile().absolutePath).markdown }
- }
-
- if (output != null) output!!.toFile().writeText(markdown) else print(markdown)
- }
-}
diff --git a/docs/common-converters-plan.md b/docs/common-converters-plan.md
new file mode 100644
index 0000000..e43eda0
--- /dev/null
+++ b/docs/common-converters-plan.md
@@ -0,0 +1,249 @@
+# Plan: the remaining converters in commonMain
+
+Status: proposal. Branch `common-converters`. Nothing implemented yet.
+
+## Where things stand
+
+**The office formats are gone.** DOCX, XLSX and PPTX were deleted rather than ported: they are
+editing formats, and a reader meets PDF and EPUB. That took Apache POI with them — 66 MB of
+distribution down to 36 MB, 35 jars to 23 — and removed the phase most likely to change output
+silently. They can return as an opt-in module in the shape `:pdfium` is taking; the fixtures stay,
+and a test pins that they currently raise `UnsupportedFormatException`.
+
+What is left JVM-only is PDF, and nothing else:
+
+| file | lines | depends on | plan |
+|---|---|---|---|
+| `PdfConverter.kt` ×2 | 32 | PDFBox / pdfbox-android | → a separate `:pdfium` module |
+
+Moving them is not a file move: `commonMain` cannot use POI, Jsoup, `java.util.zip` or `javax.xml`,
+so each is a rewrite against the raw format. CSV, JSON and XML made this trip already and their
+output stayed byte-identical, which is the bar for everything except PDF.
+
+## Why bother
+
+**iOS.** The document formats are the reason a SwiftUI reader cannot exist today. This is the blocker,
+and it is worth doing even if nothing gets faster.
+
+**A smaller JVM build** — already banked by the deletion, and worth being exact about what it bought:
+36 MB instead of 66 MB, but **no faster**. POI was loaded lazily, so a CSV or EPUB conversion never
+paid for it; the startup numbers did not move. Size and dependency surface improved, speed did not.
+
+What remains to remove is Jsoup, and it goes when HTML moves to Ksoup.
+
+Against all of it: POI, Jsoup and PDFBox absorb an enormous amount of real-world malformation.
+Hand-written parsers will be less forgiving, and the fixture corpus is ten files. **Expanding it is
+part of the work, not an afterthought.**
+
+## Building blocks
+
+| need | choice | notes |
+|---|---|---|
+| inflate | `com.soywiz:korlibs-compression:6.0.0` | EPUB is a ZIP; nothing in kotlinx-io or okio inflates on native. It ships **no ZIP reader**, so the central directory is ours to parse — about 150 lines |
+| HTML parsing | `com.fleeksoft.ksoup:ksoup:0.2.6` | KMP port of Jsoup, near-identical API, `macosarm64` published |
+| PDF | pdfium via cinterop + JNI | see below |
+
+Verified on `macosArm64`, running rather than merely linking: Ksoup parses HTML and, in XML mode,
+reads an OPF well enough to pull `dc:title`, manifest hrefs, spine idrefs and the cover meta —
+**so no separate XML library is needed**. korlibs deflate round-trips (1000 bytes → 31 → 1000).
+The korlibs API lives under `korlibs.io.compression.*`, not `korlibs.compression.*`.
+
+## PDF: a `:pdfium` module
+
+PDFBox has no KMP equivalent and text extraction is its own project, so PDF goes native through
+[pdfium](https://pdfium.googlesource.com/pdfium/), bound with
+[KonanPlugin](https://github.com/lemcoder/KonanPlugin) (`io.github.lemcoder.konanplugin:1.2.0-alpha05`,
+on the plugin portal). The plugin generates JNI bindings from the *same* `.def` file cinterop binds,
+so one declaration serves Kotlin/Native, JVM and Android.
+
+```
+pdfium/
+ src/nativeInterop/cinterop/pdfium.def headers = fpdfview.h fpdf_text.h fpdf_doc.h
+ src/commonMain/…/PdfiumConverter.kt DocumentConverter over the shim
+ src/commonMain/…/Pdfium.kt expect: open, pageCount, pageText, close
+ src/macosArm64Main/…/Pdfium.kt actual over the cinterop klib
+ src/jvmMain/…/Pdfium.kt actual over the generated JNI bridges
+ build.gradle.kts konanplugin: cinterop + jvmInterops from one def
+```
+
+Binaries come from [`bblanchon/pdfium-binaries`](https://github.com/bblanchon/pdfium-binaries), which
+publishes per-platform archives (`pdfium-mac-arm64.tgz`, `pdfium-linux-x64.tgz`,
+`pdfium-android-arm64.tgz`, …) containing headers and a library. A Gradle task downloads and unpacks
+a **pinned release with a checksum** into `build/pdfium//`; nothing binary is committed.
+
+Text needs `FPDF_InitLibrary`, `FPDF_LoadMemDocument`, `FPDF_GetPageCount`, `FPDF_LoadPage`,
+`FPDFText_LoadPage`, `FPDFText_CountChars`, `FPDFText_GetText` (UTF-16), and the matching closes.
+The existing `plainTextBlocks` reflow and de-hyphenation sit on top unchanged.
+
+**Images matter as much as text here**, because the Compose and SwiftUI readers need them, so the
+module walks page objects rather than only the text layer:
+
+| step | call |
+|---|---|
+| iterate objects on a page | `FPDFPage_CountObjects`, `FPDFPage_GetObject` |
+| keep the images | `FPDFPageObj_GetType` == `FPDF_PAGEOBJ_IMAGE` |
+| where it sits on the page | `FPDFPageObj_GetBounds` |
+| size and colour depth | `FPDFImageObj_GetImageMetadata` |
+| how it is stored | `FPDFImageObj_GetImageFilterCount`, `FPDFImageObj_GetImageFilter` |
+| the bytes | `FPDFImageObj_GetImageDataDecoded`, or `FPDFImageObj_GetRenderedBitmap` |
+
+Two cases, and the second is the awkward one:
+
+- **`DCTDecode` or `JPXDecode`** — the decoded stream *is* a JPEG or JPEG 2000 file, so the bytes go
+ straight into an `Asset` with the matching media type. Free.
+- **Anything else** (Flate-compressed raw pixels, which is very common) — there is no image file in
+ the PDF, only pixels. `FPDFImageObj_GetRenderedBitmap` returns BGRA, and something has to encode
+ it. `commonMain` has no PNG encoder, so we write one: PNG is a header, an IDAT of deflated
+ scanlines and a CRC, and korlibs-compression is already there for the deflate. Perhaps 150 lines,
+ and worth having anyway — no other target has an encoder either.
+
+**Placement.** `FPDFPageObj_GetBounds` gives each image a rectangle and `FPDFText_GetCharBox` gives
+the text the same, so images can be emitted in reading order by vertical position rather than dumped
+at the end of the page. Without it a figure lands after the prose that discusses it. This is the same
+problem the PDFBox route had and never solved.
+
+**Scanned pages** — no text and one full-page image — are detectable (an image covering most of the
+mediabox, almost no characters) and worth flagging rather than emitting as a wall of nothing.
+
+**The module does not register itself.** `:library` keeps no PDF dependency, and a caller opts in:
+
+```kotlin
+val mikroMarkdown = MikroMarkdown().apply { register(PdfiumConverter()) }
+```
+
+That keeps `:pdfium` genuinely extractable — delete the module and the rest still builds.
+
+### What this costs
+
+- **PDF output will change.** pdfium and PDFBox extract text differently, so the `test.pdf` baseline
+ has to be re-recorded. Byte-identity cannot be the gate here; `PythonComparisonTest`'s token recall
+ against Python markitdown becomes the check, plus a read of the diff.
+- **Distribution gets heavier and platform-shaped.** The JVM artifact needs the pdfium library and the
+ generated stub per platform, 4-8 MB each. Building stubs for every JVM host means a CI matrix —
+ from one machine we can only produce the host's.
+- **Licensing**: pdfium is BSD-3-Clause with Apache-2.0 parts; the notices ship with the artifact.
+
+## Sequence
+
+Each phase is shippable on its own, risky ones last.
+
+### Phase 0 — infrastructure ✅
+Ksoup and korlibs-compression are in `commonMain` and proven to run on the native binary. xmlutil
+turned out to be unnecessary — Ksoup's XML mode covers EPUB's container and OPF. Still to do when a
+phase needs it: extend `scripts/optbench.py` to verify native output for the fixtures it unlocks.
+
+### Phase 1 — HTML, via Ksoup ✅
+Done, and the feared risk did not materialise: `test_blog.html` and `test_wikipedia.html` are
+byte-identical through Ksoup, and the native binary matches the JVM on both. Two API differences
+only — `wholeText` is a method rather than a property, and `Charsets` does not exist in commonMain.
+
+Jsoup is gone from the build. HTML on native converts a blog in 7 ms against the JVM's 80, and
+Wikipedia in 66 ms against 119.
+
+### Phase 2 — EPUB ✅
+Done, byte-identical, and the JVM-only source set is gone with it. `ZipArchive` in commonMain reads
+the central directory and inflates through korlibs; the container and package documents go through
+Ksoup's XML mode; chapters reuse Phase 1. EPUB now converts on native.
+
+### Phase 3 — PDF, the `:pdfium` module — text done, geometry outstanding
+The binding works: the release is pinned by checksum, unpacked at build time, and the native CLI
+converts a PDF through `PdfiumConverter`. Two notes on getting there — the published dylib names
+itself `./libpdfium.dylib`, which the loader resolves against the working directory, so the unpack
+step rewrites it to `@rpath`; and Kotlin/Native does not carry a klib's linker options to the binary
+that links it, so the consumer names pdfium itself.
+
+Against PDFBox on the same file, pdfium keeps 96.2% of its tokens and the differences are pdfium
+reading *better*: PDFBox leaves `flexibil`, `firming`, `gramming` as fragments where pdfium plus
+de-hyphenation produces whole words.
+
+**Two defects remain, and geometry fixes both.**
+
+pdfium emits U+FFFE where a glyph has no Unicode mapping, which in a typeset paper is the hyphen at
+a line break. Dropping it fuses real compounds (`chat-optimized` → `chatoptimized`); keeping it
+splits real words. The document's own vocabulary decides today, which gets four joins right and two
+compounds wrong — and cannot do better, because the halves of a broken word are only in the text at
+all because the break put them there. **The real discriminator is that a hyphenation hyphen sits at
+the end of a line and a compound hyphen does not**, which `FPDFText_GetCharBox` answers directly.
+
+The same call answers paragraphs. pdfium returns a page as one run of text, so a PDF currently
+renders as a single paragraph where PDFBox produced seven. Character boxes give the line breaks, and
+the vertical gaps between them give the paragraphs.
+
+### Phase 3b — PDF images
+The **PNG encoder is done**: `PngEncoder` in `commonMain` writes 8-bit RGBA through korlibs' ZLib,
+checked on the JVM by decoding what it writes with ImageIO and on native by validating the chunks
+and CRCs outside the process. Filtering stays at "none" — a larger file for a much smaller encoder.
+
+What is left is the pdfium side: walking page objects, telling a JPEG stream (hand it on) from raw
+pixels (encode), and placing images in reading order by their bounds.
+
+### Phase 4 — FB2, if wanted
+Book's Story parses FB2 with the same walker it uses for HTML, through Jsoup's XML mode. Ksoup has
+that mode too, so once Phase 1 lands FB2 is a tag-mapping layer over `HtmlToDocument` — a format we
+do not support today for about half a day's work. Worth it for a reader; skip it otherwise.
+
+### Removed — DOCX, XLSX, PPTX
+Deleted, not deferred. If they return it is as an `:office` module mirroring `:pdfium`: its own
+Gradle module, its own dependency, registered by the caller rather than by the factory. The deleted
+implementations are in the history, and the fixtures are still in the repository.
+
+## Assets, across every phase
+
+The model already carries them — `Asset(id, mediaType, bytes, name)`, `Document.assets` and
+`Image.assetId` — but only DOCX populates them today. The readers need images from everything, so
+each phase extracts what its format holds:
+
+| format | where the images are | notes |
+|---|---|---|
+| docx | done | ids are the file name and can collide; needs fixing |
+| pdf | page objects, as above | needs a PNG encoder for raw-pixel images |
+| pptx | `XSLFPictureShape` today emits a fabricated `shapeName.jpg` and no bytes | real assets when the raw-XML rewrite lands |
+| epub | `
` resolved against the chapter directory into a zip entry we already hold | needs an asset resolver in the HTML walker |
+| html | remote URLs stay URLs; `data:` URIs decode into assets | cheap |
+| xlsx | deferred with the rest of XLSX | |
+
+Three things this needs that do not exist yet:
+
+1. **An asset policy.** DOCX inlines base64 today, which is why its output is 161 KB against
+ markitdown's 4.6 KB. `MarkdownOptions` should carry `Inline` / `Reference` (write files, emit
+ relative links) / `AltTextOnly`; the current `imagesAsText` and `maxInlineImageUrl` are a crude
+ stand-in. A PDF full of figures makes this urgent rather than tidy.
+2. **Stable asset ids.** The DOCX id is the embedded file name and repeats across parts.
+3. **Intrinsic size on `Image`.** Width and height from the source, so a Compose or SwiftUI layout
+ does not jump while loading. `FPDFImageObj_GetImageMetadata` provides it for PDF.
+
+## Ground rules per phase
+
+1. The new implementation lands in `commonMain` and the platform one is deleted in the same commit.
+ The Konsist duplicate-file rule catches anything copied per target by accident; where a split is
+ deliberate, as the PDF module's JVM and Android legs are, it is written down as such.
+2. `scripts/optbench.py` must report every fixture byte-identical on both targets before timings are
+ read. Where a difference is deliberate — PDF, and possibly HTML — the baseline is updated in the
+ same commit with the diff quoted in the message.
+3. `PythonComparisonTest` stays at 100% token recall.
+4. Timings are A/B against the champion binary, never absolute.
+5. Each phase adds fixtures for what it implements: a DOCX with numbered and nested lists, a PPTX with
+ a chart, a PDF with columns and hyphenation. Ten fixtures is too few to rewrite parsers against.
+
+## Estimate
+
+| phase | effort | risk |
+|---|---|---|
+| 0 infrastructure | half a day | low — or the plan changes, if a library will not build |
+| 1 HTML | 1 day | medium — parser differences on messy input |
+| 2 EPUB | half a day | low |
+| 3 PDF via pdfium, text | 2 days | medium — binding is routine, packaging and output changes are not |
+| 3b PDF images, PNG encoder, placement | 2-3 days | medium — the PNG writer is small but the reading-order interleave is fiddly |
+| assets for epub, html, pptx | 1 day, spread across their phases | low |
+| asset policy, ids, intrinsic size | half a day | low, but blocks the readers |
+| 4 FB2, optional | half a day | low |
+
+Roughly a week, images included, now that the office formats are out.
+
+## Worth deciding before starting
+
+- **Which targets beyond macOS?** Adding `iosArm64` and `linuxX64` early keeps the code honest;
+ adding them late risks finding a dependency — or a pdfium archive — that does not fit.
+- **How are pdfium binaries shipped to JVM users** — bundled per platform in the artifact, or
+ downloaded at build time by the consumer? The first is convenient and large; the second is small
+ and one more thing to go wrong.
diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml
index 0a63414..3f394ec 100644
--- a/gradle/libs.versions.toml
+++ b/gradle/libs.versions.toml
@@ -6,17 +6,15 @@ android-compileSdk = "37"
vanniktechMavenPublish = "0.36.0"
kotlinx-io = "0.9.0"
kotlinx-serialization = "1.9.0"
+ksoup = "0.2.6"
+korlibs = "6.0.0"
+konanplugin = "1.2.0-alpha06"
kotlinx-resources = "0.15.0"
commons-csv = "1.14.1"
jackson = "2.21.3"
-jsoup = "1.22.2"
junit = "6.1.0"
-pdfbox = "3.0.7"
-pdfbox-android = "2.0.27.0"
-poi = "5.5.1"
tika = "3.3.0"
-clikt = "5.1.0"
coreKtx = "1.7.0"
detekt = "1.23.8"
ktfmt = "0.27.0"
@@ -25,18 +23,15 @@ konsist = "0.17.3"
[libraries]
kotlin-test = { module = "org.jetbrains.kotlin:kotlin-test", version.ref = "kotlin" }
kotlinx-io-core = { module = "org.jetbrains.kotlinx:kotlinx-io-core", version.ref = "kotlinx-io" }
+ksoup = { module = "com.fleeksoft.ksoup:ksoup", version.ref = "ksoup" }
+korlibs-compression = { module = "com.soywiz:korlibs-compression", version.ref = "korlibs" }
kotlinx-serialization-json = { module = "org.jetbrains.kotlinx:kotlinx-serialization-json", version.ref = "kotlinx-serialization" }
kotlinx-resources = { module = "com.goncalossilva:resources", version.ref = "kotlinx-resources" }
commons-csv = { module = "org.apache.commons:commons-csv", version.ref = "commons-csv" }
jackson-kotlin = { module = "com.fasterxml.jackson.module:jackson-module-kotlin", version.ref = "jackson" }
-jsoup = { module = "org.jsoup:jsoup", version.ref = "jsoup" }
junit-jupiter = { module = "org.junit.jupiter:junit-jupiter", version.ref = "junit" }
-pdfbox = { module = "org.apache.pdfbox:pdfbox", version.ref = "pdfbox" }
-pdfbox-android = { module = "com.tom-roush:pdfbox-android", version.ref = "pdfbox-android" }
-poi-ooxml = { module = "org.apache.poi:poi-ooxml", version.ref = "poi" }
tika-core = { module = "org.apache.tika:tika-core", version.ref = "tika" }
-clikt = { module = "com.github.ajalt.clikt:clikt", version.ref = "clikt" }
core-ktx = { group = "androidx.test", name = "core-ktx", version.ref = "coreKtx" }
konsist = { module = "com.lemonappdev:konsist", version.ref = "konsist" }
@@ -48,4 +43,5 @@ kotlin-serialization = { id = "org.jetbrains.kotlin.plugin.serialization", versi
vanniktech-mavenPublish = { id = "com.vanniktech.maven.publish", version.ref = "vanniktechMavenPublish" }
kotlinx-resources = { id = "com.goncalossilva.resources", version.ref = "kotlinx-resources" }
detekt = { id = "io.gitlab.arturbosch.detekt", version.ref = "detekt" }
+konanplugin = { id = "io.github.lemcoder.konanplugin", version.ref = "konanplugin" }
ktfmt = { id = "com.ncorti.ktfmt.gradle", version.ref = "ktfmt" }
diff --git a/library/:memory:.ses b/library/:memory:.ses
new file mode 100644
index 0000000..95ddb84
--- /dev/null
+++ b/library/:memory:.ses
@@ -0,0 +1,2 @@
+1787007442705
+2b657774-881a-47ed-9141-c15c8c8f1ab7
diff --git a/library/build.gradle.kts b/library/build.gradle.kts
index 5a405c9..f750228 100644
--- a/library/build.gradle.kts
+++ b/library/build.gradle.kts
@@ -14,7 +14,6 @@ kotlin {
// Public API must be spelled out: visibility and return types, no accidental exports.
explicitApi()
- // Declaring jvmShared by hand switches off the automatic hierarchy, which macosMain needs.
applyDefaultHierarchyTemplate()
jvm {
@@ -39,30 +38,14 @@ kotlin {
}
sourceSets {
- commonMain.dependencies { implementation(libs.kotlinx.io.core) }
-
- // JVM and Android run the same parsers on the same libraries; only PDF and MIME
- // detection differ. Converters live here once instead of being copied per target.
- val jvmShared by creating {
- dependsOn(commonMain.get())
- dependencies {
- implementation(libs.jsoup)
- implementation(libs.poi.ooxml)
- }
+ commonMain.dependencies {
+ implementation(libs.kotlinx.io.core)
+ // Phase 0 of the commonMain migration: HTML parsing and the inflate that ZIP needs.
+ implementation(libs.ksoup)
+ implementation(libs.korlibs.compression)
}
- jvmMain {
- dependsOn(jvmShared)
- dependencies {
- implementation(libs.tika.core)
- implementation(libs.pdfbox)
- }
- }
-
- androidMain {
- dependsOn(jvmShared)
- dependencies { implementation(libs.pdfbox.android) }
- }
+ jvmMain { dependencies { implementation(libs.tika.core) } }
commonTest.dependencies {
implementation(libs.kotlin.test)
diff --git a/library/src/androidDeviceTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.android.kt b/library/src/androidDeviceTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.android.kt
index 97d5560..15c3cc4 100644
--- a/library/src/androidDeviceTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.android.kt
+++ b/library/src/androidDeviceTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.android.kt
@@ -1,5 +1,5 @@
package io.github.lemcoder.mikromarkdown
actual fun testMikroMarkdown(): MikroMarkdown {
- return MikroMarkdown(context = null)
+ return MikroMarkdown()
}
diff --git a/library/src/androidHostTest/kotlin/io/github/lemcoder/mikromarkdown/TestMikroMarkdown.kt b/library/src/androidHostTest/kotlin/io/github/lemcoder/mikromarkdown/TestMikroMarkdown.kt
index 1b9af9d..7c1314e 100644
--- a/library/src/androidHostTest/kotlin/io/github/lemcoder/mikromarkdown/TestMikroMarkdown.kt
+++ b/library/src/androidHostTest/kotlin/io/github/lemcoder/mikromarkdown/TestMikroMarkdown.kt
@@ -1,3 +1,3 @@
package io.github.lemcoder.mikromarkdown
-actual fun testMikroMarkdown(): MikroMarkdown = MikroMarkdown(context = null)
+actual fun testMikroMarkdown(): MikroMarkdown = MikroMarkdown()
diff --git a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt
index 65b3f85..8463eae 100644
--- a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt
+++ b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt
@@ -1,40 +1,28 @@
package io.github.lemcoder.mikromarkdown
-import android.content.Context
-import com.tom_roush.pdfbox.android.PDFBoxResourceLoader
import io.github.lemcoder.mikromarkdown.converters.CsvConverter
-import io.github.lemcoder.mikromarkdown.converters.DocxConverter
import io.github.lemcoder.mikromarkdown.converters.EpubConverter
import io.github.lemcoder.mikromarkdown.converters.HtmlConverter
import io.github.lemcoder.mikromarkdown.converters.JsonConverter
import io.github.lemcoder.mikromarkdown.converters.MarkdownPassthroughConverter
-import io.github.lemcoder.mikromarkdown.converters.PdfConverter
import io.github.lemcoder.mikromarkdown.converters.PlainTextConverter
-import io.github.lemcoder.mikromarkdown.converters.PptxConverter
-import io.github.lemcoder.mikromarkdown.converters.XlsxConverter
import io.github.lemcoder.mikromarkdown.converters.XmlConverter
import java.io.File
/**
* A [MikroMarkdown] with every Android converter registered.
*
- * PDF support needs a [Context]: pdfbox-android loads its resources from the app's assets.
+ * PDF is not among them: it needs a native library, so the `:pdfium` module provides it and the caller opts in with
+ * `register(PdfiumConverter())`.
*/
-public fun MikroMarkdown(context: Context? = null): MikroMarkdown =
+public fun MikroMarkdown(): MikroMarkdown =
MikroMarkdown(SignatureMimeDetector).apply {
register(MarkdownPassthroughConverter())
register(HtmlConverter())
register(CsvConverter())
register(JsonConverter())
register(XmlConverter())
- register(DocxConverter())
- register(XlsxConverter())
- register(PptxConverter())
register(EpubConverter())
- if (context != null) {
- PDFBoxResourceLoader.init(context)
- register(PdfConverter())
- }
register(PlainTextConverter(), priority = 10.0)
}
diff --git a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt
deleted file mode 100644
index 4b13531..0000000
--- a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt
+++ /dev/null
@@ -1,32 +0,0 @@
-package io.github.lemcoder.mikromarkdown.converters
-
-import com.tom_roush.pdfbox.pdmodel.PDDocument
-import com.tom_roush.pdfbox.text.PDFTextStripper
-import io.github.lemcoder.mikromarkdown.DocumentConverter
-import io.github.lemcoder.mikromarkdown.StreamInfo
-import io.github.lemcoder.mikromarkdown.model.Document
-import io.github.lemcoder.mikromarkdown.utils.plainTextBlocks
-
-public class PdfConverter : DocumentConverter {
- override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean {
- return info.extension == "pdf" || info.mimetype == "application/pdf"
- }
-
- override fun parse(bytes: ByteArray, info: StreamInfo): Document {
- val doc = PDDocument.load(bytes)
- try {
- val title = doc.documentInformation?.title?.trim()?.ifBlank { null }
- // Paragraph markers let the text blocks split on real paragraph breaks
- // instead of collapsing a page into one block.
- val stripper =
- PDFTextStripper().apply {
- sortByPosition = true
- setAddMoreFormatting(true)
- paragraphStart = "\n"
- }
- return Document(blocks = plainTextBlocks(stripper.getText(doc)), title = title)
- } finally {
- doc.close()
- }
- }
-}
diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt
new file mode 100644
index 0000000..16f8323
--- /dev/null
+++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt
@@ -0,0 +1,99 @@
+package io.github.lemcoder.mikromarkdown.converters
+
+import com.fleeksoft.ksoup.Ksoup
+import com.fleeksoft.ksoup.parser.Parser
+import io.github.lemcoder.mikromarkdown.DocumentConverter
+import io.github.lemcoder.mikromarkdown.StreamInfo
+import io.github.lemcoder.mikromarkdown.model.Block
+import io.github.lemcoder.mikromarkdown.model.Document
+import io.github.lemcoder.mikromarkdown.model.Paragraph
+import io.github.lemcoder.mikromarkdown.model.Strong
+import io.github.lemcoder.mikromarkdown.model.Text
+import io.github.lemcoder.mikromarkdown.utils.HtmlToDocument
+import io.github.lemcoder.mikromarkdown.utils.ZipArchive
+
+/**
+ * An EPUB is a ZIP of XHTML.
+ *
+ * `META-INF/container.xml` names the package document, which lists the manifest and the reading order; the chapters
+ * themselves go through the same walker as any other HTML. The XML is read with Ksoup's XML parser rather than a second
+ * library, which is why there is no XML dependency.
+ */
+public class EpubConverter : DocumentConverter {
+ private val metaFields =
+ listOf(
+ "title" to "Title",
+ "creator" to "Authors",
+ "language" to "Language",
+ "description" to "Description",
+ "identifier" to "Identifier",
+ )
+
+ override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean {
+ return info.extension == "epub" || info.mimetype == "application/epub+zip"
+ }
+
+ override fun parse(bytes: ByteArray, info: StreamInfo): Document {
+ val archive = ZipArchive.open(bytes) ?: return Document()
+
+ val container = archive.readText("META-INF/container.xml") ?: return Document()
+ val opfPath = opfPath(container) ?: return Document()
+ val opf = archive.readText(opfPath) ?: archive.readText(opfPath.removePrefix("/")) ?: return Document()
+ val opfDirectory = opfPath.substringBeforeLast("/", "")
+
+ val packageDocument = Ksoup.parse(html = opf, parser = Parser.xmlParser())
+ val metadata = metadata(packageDocument)
+ val manifest = manifest(packageDocument)
+
+ val blocks = mutableListOf()
+ var title: String? = metadata["title"]
+
+ for ((key, label) in metaFields) {
+ val value = metadata[key] ?: continue
+ blocks += Paragraph(listOf(Strong(listOf(Text("$label:"))), Text(" $value")))
+ }
+
+ for (idref in spine(packageDocument)) {
+ val href = manifest[idref] ?: continue
+ val path = if (opfDirectory.isEmpty()) href else "$opfDirectory/$href"
+ val html = archive.readText(path) ?: archive.readText(path.removePrefix("/")) ?: continue
+ val chapter = HtmlToDocument.parse(html)
+ if (title == null) title = chapter.title
+ blocks += chapter.blocks
+ }
+
+ return Document(blocks = blocks, title = title, metadata = metadata)
+ }
+
+ private fun opfPath(container: String): String? =
+ Ksoup.parse(html = container, parser = Parser.xmlParser()).selectFirst("rootfile")?.attr("full-path")?.ifEmpty {
+ null
+ }
+
+ private fun metadata(packageDocument: com.fleeksoft.ksoup.nodes.Document): Map {
+ val metadata = LinkedHashMap()
+ for (field in listOf("title", "creator", "language", "description", "identifier")) {
+ // Namespaced in the source as dc:title; Ksoup escapes the prefix with a pipe.
+ val text = packageDocument.selectFirst("dc|$field")?.text()?.trim()
+ if (!text.isNullOrEmpty()) metadata[field] = text
+ }
+ return metadata
+ }
+
+ /** Manifest ids to hrefs, keeping only the documents that carry text. */
+ private fun manifest(packageDocument: com.fleeksoft.ksoup.nodes.Document): Map {
+ val manifest = LinkedHashMap()
+ for (item in packageDocument.select("manifest > item")) {
+ val id = item.attr("id")
+ val href = item.attr("href")
+ val mediaType = item.attr("media-type")
+ if (id.isNotEmpty() && href.isNotEmpty() && mediaType.contains("html")) {
+ manifest[id] = href
+ }
+ }
+ return manifest
+ }
+
+ private fun spine(packageDocument: com.fleeksoft.ksoup.nodes.Document): List =
+ packageDocument.select("spine > itemref").map { it.attr("idref") }.filter { it.isNotEmpty() }
+}
diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt
similarity index 87%
rename from library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt
rename to library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt
index fee70f9..38f1c3b 100644
--- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt
+++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt
@@ -11,5 +11,5 @@ public class HtmlConverter : DocumentConverter {
}
override fun parse(bytes: ByteArray, info: StreamInfo): Document =
- HtmlToDocument.parse(bytes.toString(Charsets.UTF_8), info.localPath.orEmpty())
+ HtmlToDocument.parse(bytes.decodeToString(), info.localPath.orEmpty())
}
diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoder.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoder.kt
new file mode 100644
index 0000000..9d4d3cf
--- /dev/null
+++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoder.kt
@@ -0,0 +1,102 @@
+package io.github.lemcoder.mikromarkdown.model
+
+import korlibs.io.compression.compress
+import korlibs.io.compression.deflate.ZLib
+
+/**
+ * Writes 8-bit RGBA pixels as a PNG.
+ *
+ * Needed because most images inside a PDF are not image files: a JPEG stream can be handed on as it stands, but a
+ * Flate-compressed bitmap is raw pixels with no container, and every target needs one to produce. Nothing in the Kotlin
+ * ecosystem encodes PNG on Kotlin/Native, and the format's essentials are small — a signature, three chunks, a CRC
+ * apiece, and the deflate that korlibs-compression already provides.
+ *
+ * Filtering is left at "none", which trades a larger file for a much simpler encoder. A screenshot inside a PDF
+ * compresses well enough on deflate alone.
+ */
+public object PngEncoder {
+
+ private val SIGNATURE = byteArrayOf(-119, 80, 78, 71, 13, 10, 26, 10)
+
+ private const val BIT_DEPTH = 8
+ private const val COLOR_TYPE_RGBA = 6
+ private const val CHANNELS = 4
+
+ /**
+ * @param pixels RGBA, four bytes per pixel, row-major, [width] * [height] * 4 bytes long.
+ * @return the PNG file, or null if the dimensions and the buffer disagree.
+ */
+ public fun encode(width: Int, height: Int, pixels: ByteArray): ByteArray? {
+ if (width <= 0 || height <= 0) return null
+ if (pixels.size != width * height * CHANNELS) return null
+
+ val header = ByteArray(13)
+ header.writeInt(0, width)
+ header.writeInt(4, height)
+ header[8] = BIT_DEPTH.toByte()
+ header[9] = COLOR_TYPE_RGBA.toByte()
+ // Compression 0 (deflate), filter 0 (adaptive), interlace 0 (none) — the only values PNG has.
+
+ val chunks =
+ listOf(
+ SIGNATURE,
+ chunk("IHDR", header),
+ chunk("IDAT", scanlines(width, height, pixels).compress(ZLib)),
+ chunk("IEND", ByteArray(0)),
+ )
+
+ // Concatenated by hand: a list of boxed bytes would cost more than the encoding does.
+ val out = ByteArray(chunks.sumOf { it.size })
+ var at = 0
+ for (part in chunks) {
+ part.copyInto(out, at)
+ at += part.size
+ }
+ return out
+ }
+
+ /** Each row is preceded by its filter byte; 0 means the row is stored as it is. */
+ private fun scanlines(width: Int, height: Int, pixels: ByteArray): ByteArray {
+ val stride = width * CHANNELS
+ val out = ByteArray(height * (stride + 1))
+ for (row in 0 until height) {
+ val target = row * (stride + 1)
+ out[target] = 0
+ pixels.copyInto(out, target + 1, row * stride, (row + 1) * stride)
+ }
+ return out
+ }
+
+ /** length, type, payload, CRC of type and payload — the shape every PNG chunk shares. */
+ private fun chunk(type: String, payload: ByteArray): ByteArray {
+ val typeBytes = type.encodeToByteArray()
+ val out = ByteArray(payload.size + 12)
+ out.writeInt(0, payload.size)
+ typeBytes.copyInto(out, 4)
+ payload.copyInto(out, 8)
+ out.writeInt(payload.size + 8, crc32(typeBytes, payload))
+ return out
+ }
+
+ private fun ByteArray.writeInt(at: Int, value: Int) {
+ this[at] = (value ushr 24).toByte()
+ this[at + 1] = (value ushr 16).toByte()
+ this[at + 2] = (value ushr 8).toByte()
+ this[at + 3] = value.toByte()
+ }
+
+ private val CRC_TABLE =
+ IntArray(256) { index ->
+ var value = index
+ repeat(8) { value = if (value and 1 != 0) 0xEDB88320.toInt() xor (value ushr 1) else value ushr 1 }
+ value
+ }
+
+ private fun crc32(vararg parts: ByteArray): Int {
+ var crc = -1
+ for (part in parts) {
+ for (byte in part) crc = CRC_TABLE[(crc xor byte.toInt()) and 0xFF] xor (crc ushr 8)
+ }
+ return crc.inv()
+ }
+}
diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TextBlocks.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/TextBlocks.kt
similarity index 88%
rename from library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TextBlocks.kt
rename to library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/TextBlocks.kt
index 787f028..e80aa3f 100644
--- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TextBlocks.kt
+++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/TextBlocks.kt
@@ -1,16 +1,15 @@
-package io.github.lemcoder.mikromarkdown.utils
-
-import io.github.lemcoder.mikromarkdown.model.Block
-import io.github.lemcoder.mikromarkdown.model.Paragraph
-import io.github.lemcoder.mikromarkdown.model.Text
+package io.github.lemcoder.mikromarkdown.model
/**
- * Turns extracted plain text (PDF pages, speaker notes, …) into paragraph blocks.
+ * Turns extracted plain text into paragraph blocks.
+ *
+ * Public because it is what any text-extracting converter needs, including ones outside this module: the `:pdfium`
+ * module builds its documents with it.
*
* Blank lines separate paragraphs; soft-wrapped lines inside a paragraph are rejoined, so the Markdown does not inherit
* the source layout's line breaks.
*/
-internal fun plainTextBlocks(text: String, reflow: Boolean = true): List {
+public fun plainTextBlocks(text: String, reflow: Boolean = true): List {
// Form feeds mark PDF page breaks; treat them as paragraph boundaries.
val normalized = text.replace("\r\n", "\n").replace('\r', '\n').replace('\u000C', '\n')
val vocabulary = if (reflow) wordsIn(normalized) else emptySet()
diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt
similarity index 97%
rename from library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt
rename to library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt
index a8a9260..9220084 100644
--- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt
+++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt
@@ -1,5 +1,9 @@
package io.github.lemcoder.mikromarkdown.utils
+import com.fleeksoft.ksoup.Ksoup
+import com.fleeksoft.ksoup.nodes.Element
+import com.fleeksoft.ksoup.nodes.Node
+import com.fleeksoft.ksoup.nodes.TextNode
import io.github.lemcoder.mikromarkdown.model.Block
import io.github.lemcoder.mikromarkdown.model.BlockQuote
import io.github.lemcoder.mikromarkdown.model.CodeBlock
@@ -21,10 +25,6 @@ import io.github.lemcoder.mikromarkdown.model.TableCell
import io.github.lemcoder.mikromarkdown.model.Text
import io.github.lemcoder.mikromarkdown.model.ThematicBreak
import io.github.lemcoder.mikromarkdown.model.plainText
-import org.jsoup.Jsoup
-import org.jsoup.nodes.Element
-import org.jsoup.nodes.Node
-import org.jsoup.nodes.TextNode
/**
* Walks an HTML DOM into the shared document model.
@@ -79,7 +79,7 @@ internal object HtmlToDocument {
)
internal fun parse(html: String, baseUri: String = ""): Document {
- val doc = Jsoup.parse(html, baseUri)
+ val doc = Ksoup.parse(html = html, baseUri = baseUri)
doc.select(DROPPED_SELECTOR).remove()
val title = doc.title().ifBlank { null }
val root = doc.body() ?: doc
@@ -310,7 +310,7 @@ internal object HtmlToDocument {
/** HTML collapses runs of whitespace; do the same before the text reaches the model. */
// Non-breaking spaces are not collapsible whitespace in HTML, so they survive verbatim.
- private fun TextNode.normalizedText(): String = wholeText.replace(COLLAPSIBLE_WHITESPACE, " ")
+ private fun TextNode.normalizedText(): String = getWholeText().replace(COLLAPSIBLE_WHITESPACE, " ")
private fun List.startsWithSpace(): Boolean = (firstOrNull() as? Text)?.value?.startsWith(" ") == true
diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/ZipArchive.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/ZipArchive.kt
new file mode 100644
index 0000000..b031e56
--- /dev/null
+++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/ZipArchive.kt
@@ -0,0 +1,120 @@
+package io.github.lemcoder.mikromarkdown.utils
+
+import korlibs.io.compression.deflate.Deflate
+import korlibs.io.compression.uncompress
+
+/**
+ * Reads a ZIP container held in memory.
+ *
+ * korlibs supplies the inflate and nothing else, and neither kotlinx-io nor okio reads archives, so the container
+ * itself is parsed here: find the end-of-central-directory record, walk the directory, and follow each entry to its
+ * local header. Enough of the format for EPUB and, later, OOXML.
+ */
+internal class ZipArchive private constructor(private val source: ByteArray, private val entries: Map) {
+
+ val names: Set
+ get() = entries.keys
+
+ /** The entry's bytes, inflated if it was deflated, or null if it is absent or unreadable. */
+ fun read(name: String): ByteArray? {
+ val entry = entries[name] ?: return null
+ val start = dataStart(entry) ?: return null
+ if (start + entry.compressedSize > source.size) return null
+
+ val raw = source.copyOfRange(start, start + entry.compressedSize)
+ return when (entry.method) {
+ STORED -> raw
+ DEFLATED ->
+ try {
+ raw.uncompress(Deflate)
+ } catch (_: Exception) {
+ null
+ }
+ // Any other method — bzip2, lzma, encrypted — is not something we can read.
+ else -> null
+ }
+ }
+
+ /** The entry decoded as UTF-8, which every XML and XHTML part of an EPUB is. */
+ fun readText(name: String): String? = read(name)?.decodeToString()
+
+ /**
+ * Where an entry's bytes begin.
+ *
+ * The central directory records where the local header is, but the local header repeats the name and extra field
+ * with lengths of its own, so the data offset can only be computed from there.
+ */
+ private fun dataStart(entry: Entry): Int? {
+ val header = entry.localHeaderOffset
+ if (header + LOCAL_HEADER_MINIMUM > source.size) return null
+ if (source.readInt(header) != LOCAL_HEADER_SIGNATURE) return null
+ val nameLength = source.readShort(header + 26)
+ val extraLength = source.readShort(header + 28)
+ return header + LOCAL_HEADER_MINIMUM + nameLength + extraLength
+ }
+
+ private class Entry(val localHeaderOffset: Int, val compressedSize: Int, val method: Int)
+
+ companion object {
+ private const val STORED = 0
+ private const val DEFLATED = 8
+ private const val END_OF_DIRECTORY_SIGNATURE = 0x06054b50
+ private const val DIRECTORY_ENTRY_SIGNATURE = 0x02014b50
+ private const val LOCAL_HEADER_SIGNATURE = 0x04034b50
+ private const val LOCAL_HEADER_MINIMUM = 30
+ private const val DIRECTORY_ENTRY_MINIMUM = 46
+ private const val END_OF_DIRECTORY_MINIMUM = 22
+
+ /** The comment that may follow the end record, and so how far back it can hide. */
+ private const val MAX_COMMENT = 0xFFFF
+
+ fun open(bytes: ByteArray): ZipArchive? {
+ val end = findEndOfDirectory(bytes) ?: return null
+ val count = bytes.readShort(end + 10)
+ var offset = bytes.readInt(end + 16)
+
+ val entries = LinkedHashMap(count)
+ repeat(count) {
+ if (offset + DIRECTORY_ENTRY_MINIMUM > bytes.size) return@repeat
+ if (bytes.readInt(offset) != DIRECTORY_ENTRY_SIGNATURE) return@repeat
+
+ val method = bytes.readShort(offset + 10)
+ val compressedSize = bytes.readInt(offset + 20)
+ val nameLength = bytes.readShort(offset + 28)
+ val extraLength = bytes.readShort(offset + 30)
+ val commentLength = bytes.readShort(offset + 32)
+ val localHeaderOffset = bytes.readInt(offset + 42)
+
+ val nameStart = offset + DIRECTORY_ENTRY_MINIMUM
+ if (nameStart + nameLength > bytes.size) return@repeat
+ val name = bytes.decodeToString(nameStart, nameStart + nameLength)
+ // Directories carry no data and end in a separator.
+ if (!name.endsWith("/")) {
+ entries[name] = Entry(localHeaderOffset, compressedSize, method)
+ }
+ offset = nameStart + nameLength + extraLength + commentLength
+ }
+
+ return ZipArchive(bytes, entries)
+ }
+
+ /** The end record sits last, unless a comment follows it, so the search runs backwards. */
+ private fun findEndOfDirectory(bytes: ByteArray): Int? {
+ if (bytes.size < END_OF_DIRECTORY_MINIMUM) return null
+ val earliest = maxOf(0, bytes.size - END_OF_DIRECTORY_MINIMUM - MAX_COMMENT)
+ for (offset in bytes.size - END_OF_DIRECTORY_MINIMUM downTo earliest) {
+ if (bytes.readInt(offset) == END_OF_DIRECTORY_SIGNATURE) return offset
+ }
+ return null
+ }
+
+ private fun ByteArray.readShort(at: Int): Int =
+ (this[at].toInt() and 0xFF) or ((this[at + 1].toInt() and 0xFF) shl 8)
+
+ private fun ByteArray.readInt(at: Int): Int =
+ (this[at].toInt() and 0xFF) or
+ ((this[at + 1].toInt() and 0xFF) shl 8) or
+ ((this[at + 2].toInt() and 0xFF) shl 16) or
+ ((this[at + 3].toInt() and 0xFF) shl 24)
+ }
+}
diff --git a/library/src/commonTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.kt b/library/src/commonTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.kt
index 076e5aa..5473077 100644
--- a/library/src/commonTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.kt
+++ b/library/src/commonTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.kt
@@ -2,6 +2,7 @@ package io.github.lemcoder.mikromarkdown
import com.goncalossilva.resources.Resource
import kotlin.test.Test
+import kotlin.test.assertFailsWith
import kotlin.test.assertFalse
import kotlin.test.assertTrue
@@ -27,45 +28,18 @@ class FileIntegrationTest {
}
}
+ /**
+ * The office formats were removed rather than ported: they are editing formats, and a reader meets PDF and EPUB.
+ * The fixtures stay for whenever an office plugin arrives, and this test pins the behaviour callers see until then.
+ */
@Test
- fun testDocx() =
- assertConversion(
- filename = "test.docx",
- mustInclude =
- listOf(
- "314b0a30-5b04-470b-b9f7-eed2c2bec74a",
- "49e168b7-d2ae-407f-a055-2167576f39a1",
- "# Abstract",
- "# Introduction",
- "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation",
- ),
- )
-
- @Test
- fun testXlsx() =
- assertConversion(
- filename = "test.xlsx",
- mustInclude =
- listOf(
- "09060124-b5e7-4717-9d07-3c046eb",
- "6ff4173b-42a5-4784-9b19-f49caff4d93d",
- "affc7dad-52dc-4b98-9b5d-51e65d8a8ad0",
- ),
- )
-
- @Test
- fun testPptx() =
- assertConversion(
- filename = "test.pptx",
- mustInclude =
- listOf(
- "2cdda5c8-e50e-4db4-b5f0-9722a649f455",
- "04191ea8-5c73-4215-a1d3-1cfb43aaaf12",
- "44bf7d06-5e7a-4a40-a2e1-a2e42ef28c8a",
- "1b92870d-e3b5-4e65-8153-919f4ff45592",
- "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation",
- ),
- )
+ fun officeFormatsAreNotSupported() {
+ for (filename in listOf("test.docx", "test.xlsx", "test.pptx")) {
+ val bytes = Resource("test_files/$filename").readBytes()
+ val info = StreamInfo(extension = filename.substringAfterLast("."))
+ assertFailsWith(filename) { mid.convert(bytes, info) }
+ }
+ }
@Test
fun testBlogHtml() =
diff --git a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt
index c13dc2d..9695eb7 100644
--- a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt
+++ b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt
@@ -1,19 +1,20 @@
package io.github.lemcoder.mikromarkdown
import io.github.lemcoder.mikromarkdown.converters.CsvConverter
-import io.github.lemcoder.mikromarkdown.converters.DocxConverter
import io.github.lemcoder.mikromarkdown.converters.EpubConverter
import io.github.lemcoder.mikromarkdown.converters.HtmlConverter
import io.github.lemcoder.mikromarkdown.converters.JsonConverter
import io.github.lemcoder.mikromarkdown.converters.MarkdownPassthroughConverter
-import io.github.lemcoder.mikromarkdown.converters.PdfConverter
import io.github.lemcoder.mikromarkdown.converters.PlainTextConverter
-import io.github.lemcoder.mikromarkdown.converters.PptxConverter
-import io.github.lemcoder.mikromarkdown.converters.XlsxConverter
import io.github.lemcoder.mikromarkdown.converters.XmlConverter
import java.io.File
-/** A [MikroMarkdown] with every JVM converter registered and Tika-based format detection. */
+/**
+ * A [MikroMarkdown] with every JVM converter registered.
+ *
+ * PDF is not among them: it needs a native library, so the `:pdfium` module provides it and the caller opts in with
+ * `register(PdfiumConverter())`.
+ */
public fun MikroMarkdown(): MikroMarkdown =
MikroMarkdown(SignatureMimeDetector).apply {
register(MarkdownPassthroughConverter())
@@ -21,11 +22,7 @@ public fun MikroMarkdown(): MikroMarkdown =
register(CsvConverter())
register(JsonConverter())
register(XmlConverter())
- register(DocxConverter())
- register(XlsxConverter())
- register(PptxConverter())
register(EpubConverter())
- register(PdfConverter())
register(PlainTextConverter(), priority = 10.0)
}
diff --git a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt
deleted file mode 100644
index 58a536d..0000000
--- a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt
+++ /dev/null
@@ -1,32 +0,0 @@
-package io.github.lemcoder.mikromarkdown.converters
-
-import io.github.lemcoder.mikromarkdown.DocumentConverter
-import io.github.lemcoder.mikromarkdown.StreamInfo
-import io.github.lemcoder.mikromarkdown.model.Document
-import io.github.lemcoder.mikromarkdown.utils.plainTextBlocks
-import org.apache.pdfbox.Loader
-import org.apache.pdfbox.text.PDFTextStripper
-
-public class PdfConverter : DocumentConverter {
- override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean {
- return info.extension == "pdf" || info.mimetype == "application/pdf"
- }
-
- override fun parse(bytes: ByteArray, info: StreamInfo): Document {
- val doc = Loader.loadPDF(bytes)
- try {
- val title = doc.documentInformation?.title?.trim()?.ifBlank { null }
- // Paragraph markers let the text blocks split on real paragraph breaks
- // instead of collapsing a page into one block.
- val stripper =
- PDFTextStripper().apply {
- sortByPosition = true
- setAddMoreFormatting(true)
- paragraphStart = "\n"
- }
- return Document(blocks = plainTextBlocks(stripper.getText(doc)), title = title)
- } finally {
- doc.close()
- }
- }
-}
diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/DocxConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/DocxConverter.kt
deleted file mode 100644
index 596863f..0000000
--- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/DocxConverter.kt
+++ /dev/null
@@ -1,164 +0,0 @@
-package io.github.lemcoder.mikromarkdown.converters
-
-import io.github.lemcoder.mikromarkdown.DocumentConverter
-import io.github.lemcoder.mikromarkdown.StreamInfo
-import io.github.lemcoder.mikromarkdown.model.Asset
-import io.github.lemcoder.mikromarkdown.model.Block
-import io.github.lemcoder.mikromarkdown.model.Document
-import io.github.lemcoder.mikromarkdown.model.Heading
-import io.github.lemcoder.mikromarkdown.model.Image
-import io.github.lemcoder.mikromarkdown.model.Inline
-import io.github.lemcoder.mikromarkdown.model.ListBlock
-import io.github.lemcoder.mikromarkdown.model.ListItem
-import io.github.lemcoder.mikromarkdown.model.Paragraph
-import io.github.lemcoder.mikromarkdown.model.Table
-import io.github.lemcoder.mikromarkdown.model.TableCell
-import io.github.lemcoder.mikromarkdown.model.Text
-import io.github.lemcoder.mikromarkdown.model.plainText
-import io.github.lemcoder.mikromarkdown.model.styled
-import java.util.Base64
-import org.apache.poi.xwpf.usermodel.XWPFDocument
-import org.apache.poi.xwpf.usermodel.XWPFParagraph
-import org.apache.poi.xwpf.usermodel.XWPFTable
-
-public class DocxConverter : DocumentConverter {
- override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean {
- return info.extension == "docx" ||
- info.mimetype == "application/vnd.openxmlformats-officedocument.wordprocessingml.document"
- }
-
- override fun parse(bytes: ByteArray, info: StreamInfo): Document {
- val docx = XWPFDocument(bytes.inputStream())
- try {
- val blocks = mutableListOf()
- val assets = mutableListOf()
- var title: String? = docx.properties?.coreProperties?.title?.trim()?.ifBlank { null }
- val pendingListItems = mutableListOf>>()
-
- fun flushList() {
- if (pendingListItems.isEmpty()) return
- blocks += buildNestedList(pendingListItems)
- pendingListItems.clear()
- }
-
- for (element in docx.bodyElements) {
- when (element) {
- is XWPFParagraph -> {
- val content = paragraphInlines(element, assets)
- if (content.plainText().isBlank() && content.none { it is Image }) continue
-
- val level = headingLevel(element.styleID)
- when {
- level > 0 -> {
- flushList()
- if (title == null) title = content.plainText().trim()
- blocks += Heading(level, content)
- }
-
- element.numID != null ->
- pendingListItems += (element.numIlvl?.toInt() ?: 0).coerceAtLeast(0) to content
-
- else -> {
- flushList()
- blocks += Paragraph(content)
- }
- }
- }
-
- is XWPFTable -> {
- flushList()
- table(element)?.let { blocks += it }
- }
- }
- }
- flushList()
-
- return Document(blocks = blocks, title = title, assets = assets)
- } finally {
- docx.close()
- }
- }
-
- private fun paragraphInlines(para: XWPFParagraph, assets: MutableList): List {
- val out = mutableListOf()
- for (run in para.runs) {
- val pictures = run.embeddedPictures
- if (pictures.isNotEmpty()) {
- for (picture in pictures) {
- val data = picture.pictureData
- val alt = picture.description.orEmpty()
- if (data == null) {
- if (alt.isNotBlank()) out += Text(alt)
- continue
- }
- val mime = data.pictureTypeEnum.contentType
- val id = data.fileName ?: "image-${assets.size + 1}"
- assets += Asset(id = id, mediaType = mime, bytes = data.data, name = data.fileName)
- out +=
- Image(
- alt = alt,
- url = "data:$mime;base64,${Base64.getEncoder().encodeToString(data.data)}",
- assetId = id,
- )
- }
- continue
- }
-
- val text = run.text() ?: continue
- if (text.isEmpty()) continue
- if (text.isBlank()) {
- out += Text(text)
- continue
- }
- out += styled(listOf(Text(text)), bold = run.isBold, italic = run.isItalic, strike = run.isStrikeThrough)
- }
- return out
- }
-
- /** Rebuilds Word's flat numbering levels into nested list blocks. */
- private fun buildNestedList(items: List>>): ListBlock {
- var index = 0
-
- fun build(level: Int): List {
- val result = mutableListOf()
- while (index < items.size) {
- val (itemLevel, content) = items[index]
- when {
- itemLevel < level -> break
- itemLevel == level -> {
- index++
- val children =
- if (index < items.size && items[index].first > level) {
- listOf(ListBlock(ordered = false, items = build(items[index].first)))
- } else {
- emptyList()
- }
- result += ListItem(listOf(Paragraph(content)) + children)
- }
- // A deeper first item without a parent: promote it to this level.
- else -> result += ListItem(listOf(ListBlock(ordered = false, items = build(itemLevel))))
- }
- }
- return result
- }
-
- return ListBlock(ordered = false, items = build(items.minOf { it.first }))
- }
-
- private fun table(table: XWPFTable): Table? {
- val rows = table.rows
- if (rows.isEmpty()) return null
- val header = rows[0].tableCells.map { TableCell(it.text.trim()) }
- val body = rows.drop(1).map { row -> row.tableCells.map { TableCell(it.text.trim()) } }
- return Table(header = header, rows = body)
- }
-
- private fun headingLevel(style: String?): Int {
- val s = style?.replace("\\s+".toRegex(), "") ?: return 0
- if (s.startsWith("Heading", ignoreCase = true)) {
- return s.drop(7).toIntOrNull()?.coerceIn(1, 6) ?: 0
- }
- // OOXML numeric style IDs 1-6 map directly to heading levels
- return s.toIntOrNull()?.takeIf { it in 1..6 } ?: 0
- }
-}
diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt
deleted file mode 100644
index b700606..0000000
--- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt
+++ /dev/null
@@ -1,132 +0,0 @@
-package io.github.lemcoder.mikromarkdown.converters
-
-import io.github.lemcoder.mikromarkdown.DocumentConverter
-import io.github.lemcoder.mikromarkdown.StreamInfo
-import io.github.lemcoder.mikromarkdown.model.Block
-import io.github.lemcoder.mikromarkdown.model.Document
-import io.github.lemcoder.mikromarkdown.model.Paragraph
-import io.github.lemcoder.mikromarkdown.model.Strong
-import io.github.lemcoder.mikromarkdown.model.Text
-import io.github.lemcoder.mikromarkdown.utils.HtmlToDocument
-import java.io.StringReader
-import java.util.zip.ZipInputStream
-import javax.xml.parsers.DocumentBuilderFactory
-import org.w3c.dom.Element
-import org.xml.sax.InputSource
-
-public class EpubConverter : DocumentConverter {
- private val metaFields =
- listOf(
- "title" to "Title",
- "creator" to "Authors",
- "language" to "Language",
- "description" to "Description",
- "identifier" to "Identifier",
- )
-
- override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean {
- return info.extension == "epub" || info.mimetype == "application/epub+zip"
- }
-
- override fun parse(bytes: ByteArray, info: StreamInfo): Document {
- val entries = readZip(bytes)
-
- val containerXml = entries["META-INF/container.xml"] ?: return Document()
- val opfPath = parseOpfPath(containerXml) ?: return Document()
- val opfBytes = entries[opfPath] ?: entries[opfPath.removePrefix("/")] ?: return Document()
- val opfDir = opfPath.substringBeforeLast("/", "")
-
- val (manifest, spine, metadata) = parseOpf(opfBytes)
-
- val blocks = mutableListOf()
- var title: String? = metadata["title"]
-
- for ((key, label) in metaFields) {
- val value = metadata[key] ?: continue
- blocks += Paragraph(listOf(Strong(listOf(Text("$label:"))), Text(" $value")))
- }
-
- for (idref in spine) {
- val href = manifest[idref] ?: continue
- val fullPath = if (opfDir.isEmpty()) href else "$opfDir/$href"
- val htmlBytes = entries[fullPath] ?: entries[fullPath.removePrefix("/")] ?: continue
- val chapter = HtmlToDocument.parse(htmlBytes.toString(Charsets.UTF_8))
- if (title == null) title = chapter.title
- blocks += chapter.blocks
- }
-
- return Document(blocks = blocks, title = title, metadata = metadata)
- }
-
- private fun readZip(bytes: ByteArray): Map {
- val entries = mutableMapOf()
- ZipInputStream(bytes.inputStream()).use { zip ->
- var entry = zip.nextEntry
- while (entry != null) {
- if (!entry.isDirectory) {
- entries[entry.name] = zip.readBytes()
- }
- zip.closeEntry()
- entry = zip.nextEntry
- }
- }
- return entries
- }
-
- private fun parseOpfPath(containerXml: ByteArray): String? {
- val doc = parseXml(containerXml) ?: return null
- val rootfiles = doc.getElementsByTagName("rootfile")
- if (rootfiles.length == 0) return null
- return (rootfiles.item(0) as? Element)?.getAttribute("full-path")
- }
-
- private fun parseOpf(opfBytes: ByteArray): Triple