diff --git a/README.md b/README.md index c6a79e3..9c7061a 100644 --- a/README.md +++ b/README.md @@ -6,19 +6,22 @@ Kotlin Multiplatform (JVM + Android) library that converts documents to Markdown ## Supported formats -| Format | Extension | -|--------|-----------| -| Word | `.docx` | -| Excel | `.xlsx` | -| PowerPoint | `.pptx` | -| EPUB | `.epub` | -| HTML | `.html`, `.htm` | -| PDF | `.pdf` | -| CSV | `.csv` | -| JSON | `.json` | -| XML | `.xml` | -| Plain text | `.txt` and others | -| Markdown | `.md` (passthrough) | +Office formats were removed deliberately: DOCX, XLSX and PPTX are editing formats, while a reader +meets PDF and EPUB. Dropping them took Apache POI with them — the distribution went from 66 MB to +36 MB. If they are wanted back, they return as an opt-in module the way PDF is heading, rather than +as a dependency everyone carries. + + +| Format | Extension | Notes | +|--------|-----------|-------| +| EPUB | `.epub` | | +| HTML | `.html`, `.htm` | | +| PDF | `.pdf` | opt-in: `:pdfium` module, `register(PdfiumConverter())` | +| CSV | `.csv` | | +| JSON | `.json` | | +| XML | `.xml` | | +| Plain text | `.txt` and others | | +| Markdown | `.md` (passthrough) | | ## Architecture @@ -34,12 +37,13 @@ bytes ──► MimeDetector ──► DocumentConverter.parse ──► Documen ``` Converters contain no Markdown syntax, so escaping, table shaping, list indentation and spacing are -fixed once for all formats. JVM and Android share one `jvmShared` source set, so a converter exists -once rather than per target; only PDF extraction and MIME detection are platform-specific. The model is public: `mid.parse(path)` returns the `Document`, and -`ConversionResult.document` exposes it alongside the rendered Markdown. +fixed once for all formats. Every converter lives in `commonMain` and runs on every target; only PDF +is platform-specific, and it lives in its own module because it needs a native library. The model is +public: `mid.parse(path)` returns the `Document`, and `ConversionResult.document` exposes it +alongside the rendered Markdown. ```kotlin -val document = mid.parse("/path/to/report.docx") +val document = mid.parse("/path/to/book.epub") document.blocks.filterIsInstance().forEach { println(it.rows.size) } // Render with different options @@ -66,7 +70,7 @@ import io.github.lemcoder.mikromarkdown.MikroMarkdown val mid = MikroMarkdown() // from file path -val result = mid.convert("/path/to/document.docx") +val result = mid.convert("/path/to/book.epub") // from bytes with explicit format hint val bytes = File("document.html").readBytes() @@ -151,170 +155,52 @@ import the renderer or each other; Markdown syntax appears only under `render/`. every `DocumentConverter` is named `*Converter` and lives in `converters`. *Hygiene* — no wildcard imports, no printing from library code, and no source file duplicated -between source sets (the drift that the `jvmShared` set removed). +between source sets. ktfmt-gradle only derives tasks for the common and JVM source sets, so `library/build.gradle.kts` registers matching tasks for the Android ones. ## Performance -Conversion itself is a few milliseconds; a CLI run is mostly JVM startup and class loading. -`:benchmark` measures the pipeline in-process, `scripts/benchmark.py` measures whole processes. - -```bash -./gradlew :benchmark:run # in-process, per stage -python3 scripts/benchmark.py # whole process, against markitdown and anydoc -``` - -In-process, best of 50 runs after warmup: - -| fixture | size | parse | render | total | -|---|---|---|---|---| -| test.json | 0.4 KB | 0.03 ms | 0.01 ms | 0.04 ms | -| test.epub | 2 KB | 0.42 ms | 0.00 ms | 0.42 ms | -| test_blog.html | 25 KB | 0.85 ms | 0.11 ms | 0.96 ms | -| test.xlsx | 11 KB | 1.09 ms | 0.00 ms | 1.03 ms | -| test.docx | 132 KB | 3.28 ms | 0.00 ms | 2.81 ms | -| test.pdf | 90 KB | 3.69 ms | 0.00 ms | 3.47 ms | -| test.pptx | 271 KB | 4.79 ms | 0.00 ms | 4.53 ms | -| test_wikipedia.html | 385 KB | 12.98 ms | 1.82 ms | 14.80 ms | - -End to end the CLI runs in 50–230 ms, against 3–5 ms for the Rust -[anydoc](https://github.com/firecrawl/anydoc) and 410–540 ms for Python markitdown. Nearly all of -what is left is process startup: `java -version` alone costs 41 ms on the same machine, and the -conversion is under 5 ms for every fixture but Wikipedia. Matching a native binary would take -ahead-of-time compilation, not a faster pipeline. - -Note when reproducing this: anydoc's npm package is a Node script loading a napi module, so timing -`node_modules/.bin/anydoc` charges Node's 15 ms startup to Rust and reads as ~22 ms flat. The -figures here come from its Rust binary, built from the vendored source with -`cargo build --release --example convert`. - -The CLI therefore optimizes startup rather than throughput: - -- `installDist` records a [class-data-sharing](https://docs.oracle.com/en/java/javase/21/vm/class-data-sharing.html) - archive into the distribution, which roughly halves startup. Set `MIKROMARKDOWN_NO_CDS=1` to skip - it; the start script also skips it when the archive is missing, so `distZip` still works. -- it compiles with C1 only (`-XX:TieredStopAtLevel=1`), since C2 never pays for itself in a run this - short. Embedders using the library get the normal JIT. - -### Kotlin/Native spike - -`:cli-native` builds a macOS binary carrying the converters that need no JVM library — CSV, JSON, -XML, plain text and Markdown passthrough. It shares the model, renderer and pipeline with every -other target, and its output is byte-identical to the JVM CLI's. +The command line tool is the Kotlin/Native binary; there is no JVM CLI. `:benchmark` measures the +library in-process on the JVM, and `scripts/optbench.py` A/B times the native binary against a saved +champion so that session-to-session drift cannot be mistaken for a change. ```bash ./gradlew :cli-native:linkReleaseExecutableMacosArm64 +./gradlew :benchmark:run # library, in-process, per stage +python3 scripts/optbench.py "..." # native binary, against the champion +python3 scripts/benchmark.py # whole process, against markitdown and anydoc ``` -Whole-process, best of eight, converting CSV of increasing size: - -| input | Kotlin/Native | anydoc (Rust binary) | JVM CLI | -|---|---|---|---| -| 1 KB | 3 ms | 3 ms | 51 ms | -| 55 KB | **4 ms** | 5 ms | 61 ms | -| 172 KB | **7 ms** | 10 ms | 59 ms | -| 580 KB | **19 ms** | 25 ms | 70 ms | -| 1.8 MB | **51 ms** | 71 ms | 101 ms | -| 3.5 MB | **102 ms** | 139 ms | 142 ms | - -Kotlin/Native is ahead of the Rust binary at every size here, having started the spike 18-21% -behind on large inputs; `docs/optimization-log.md` records how. Note that anydoc's npm package runs -through Node, which adds about 18 ms — these figures are its Rust binary, built from the vendored -source with `cargo build --release --example convert`. - -On the document formats the native target does not carry, the JVM CLI converts a DOCX in 180 ms and -Wikipedia in 122 ms, against anydoc's 3 ms and Python markitdown's 409 ms and 520 ms. That gap is -process startup, not conversion: in-process those documents take 2.7 ms and 13 ms. - -Three changes closed the throughput gap that the first cut of this target showed: - -- **The renderer stopped allocating when it has nothing to change.** Escaping now scans for the - first character that needs a backslash and returns the input untouched when there is none, and a - single-line table cell skips the split-and-rejoin. Ordinary cells — a word, a number — now cost no - allocation at all. This is shared code, so the JVM got faster too. -- **The native CSV reader slices instead of accumulating.** Fields are ranges in the decoded text, - so a field costs one substring rather than a per-character builder plus a separate trim. Only - fields containing escaped quotes, which cannot be a slice of the input, assemble a string. -- **CSV, JSON and XML moved to `commonMain`**, taking commons-csv, Jackson and - kotlinx-serialization with them. One implementation now serves every target: a slicing CSV reader, - a JSON re-indenter that copies tokens verbatim so `1.50` does not become `1.5`, and an XML - formatter. A JSON conversion loads 1214 classes instead of 2063, the native binary is 1.3 MB - instead of 2.2 MB, and the JVM CLI converts JSON in 56 ms instead of 95 ms. Output is unchanged - on every fixture. -- **Two quadratics in the renderer are gone.** Blocks are written into one buffer carrying a line - prefix, rather than each block returning a string that its parent splits into lines and re-joins — - which charged the deepest content once per level of nesting above it. And the entity check now - scans ten characters ahead instead of searching the rest of the document for a semicolon. - Measured on inputs built to provoke them, with output byte-identical before and after: - - | pathological input | before | after | - |---|---|---| - | 400-deep nested lists | 33.03 ms | **0.12 ms** | - | 400 ampersands per cell, 789 KB | 9.39 ms | **2.42 ms** | - -Together those took a 1.8 MB CSV from 237 ms to 87 ms. Kotlin/Native's remaining cost is the -document model itself: every cell becomes a `TableCell` holding a `Text` holding a `String`, which -is why peak memory is 125 MB for a 1.8 MB input. A genuinely zero-copy model — inlines holding -slices of the source buffer rather than copies — is the next lever, and a deeper change. - -Compiler flags were measured rather than guessed. `-Xbinary=preCodegenInlineThreshold=40` is worth -about 8% on large inputs and ships. Every garbage collection setting tried was worse than the -default, and the collector is the interesting part of the story, so the numbers are below. - -Converting 20 documents of 580 KB in one process: - -| policy | time | peak RSS | -|---|---|---| -| default (adaptive) | 543 ms | 59 MB | -| `gcSchedulerType=manual`, never collecting | 428 ms | 954 MB | -| `gcSchedulerType=manual`, collecting between documents | 485 ms | 67 MB | -| `autotune = false` with a heap ceiling | 1246 ms | 43 MB | - -A manual collector is genuinely faster, since a process that exits never needs to collect, and a -document boundary is the one place where everything the previous conversion allocated is provably -dead. But it only bounds growth *between* documents: a single large input still has nothing -collecting mid-parse, so the 3.5 MB file takes 228 MB either way and a much larger one would grow -until it failed. Turning `autotune` off is far worse than it looks like it should be — 8.6x on a -single 3.5 MB file — and the ceiling value makes no difference to that, so `targetHeapBytes` is not -behaving as its name suggests. - -The default collector ships. The native CLI does accept several files per invocation, which is what -would make a manual policy workable if the trade ever becomes worth it. - -### What did not help +Whole process, best of ten, against the Rust [anydoc](https://github.com/firecrawl/anydoc): -Binary size does not drive startup, so shrinking it is not a performance lever: - -| binary | size | startup | +| input | Kotlin/Native | anydoc (Rust) | |---|---|---| -| Kotlin/Native hello world | 485 KB | 3.2 ms | -| this CLI | 1.3 MB | 3.5 ms | -| anydoc (Rust) | 6 MB | 2.6 ms | - -A 6 MB Rust binary starts faster than a 485 KB Kotlin/Native one, and stripping ours changed -nothing measurable. The 0.6 ms between the two runtimes is initialization, not size. +| 1 KB CSV | 3 ms | 3 ms | +| 172 KB CSV | 7 ms | 10 ms | +| 1.8 MB CSV | 51 ms | 71 ms | +| 3.5 MB CSV | 102 ms | 139 ms | +| EPUB | 5 ms | 3 ms | +| Wikipedia HTML | 65 ms | — | -Replacing clikt with hand-rolled argument parsing removes 147 loaded classes and about 2 ms — inside -the noise, and not worth losing its help output and error handling. The dependency stays. +Note when reproducing this: anydoc's npm package is a Node script loading a napi module, so timing +`node_modules/.bin/anydoc` charges Node's 15 ms startup to Rust and reads as ~22 ms flat. The figures +here come from its Rust binary, built from the vendored source with +`cargo build --release --example convert`. -Zero-copy parsing made things slower, which is the most useful negative result here. Handing the -renderer a `CharSequence` window onto the source instead of a substring removes one allocation per -cell, but every character access then goes through an interface call rather than `String`'s direct -indexing — and the renderer reads every character anyway to decide whether it needs escaping. A -1.8 MB CSV went from 86 ms to 118 ms on native and 126 ms to 154 ms on the JVM. Rust gets this for -free because `&str` slices index directly; Kotlin does not. Reverted. +`docs/optimization-log.md` records twenty-five measured experiments, nine of which survived, and the +two methodology mistakes that cost more than most of the wins. ## Benchmark -`scripts/benchmark.py` converts the test fixtures with MikroMarkdown, Python +`scripts/benchmark.py` converts the test fixtures with the native binary, Python [markitdown](https://github.com/microsoft/markitdown) and Rust [anydoc](https://github.com/firecrawl/anydoc), then reports content recall, structure counts, table integrity and timings to `build/benchmark/report.md`: ```bash -./gradlew :cli:installDist +./gradlew :cli-native:linkReleaseExecutableMacosArm64 python3 scripts/benchmark.py ``` diff --git a/cli-native/build.gradle.kts b/cli-native/build.gradle.kts index befa74d..f554494 100644 --- a/cli-native/build.gradle.kts +++ b/cli-native/build.gradle.kts @@ -2,7 +2,15 @@ plugins { alias(libs.plugins.kotlinMultiplatform) } kotlin { macosArm64 { - binaries.executable { entryPoint = "io.github.lemcoder.mikromarkdown.cli.main" } + // Kotlin/Native does not carry a klib's linker options to the binary that uses it, so the + // consumer names pdfium itself. Worth turning into a shared convention if a second + // consumer appears. + val pdfiumLib = rootProject.layout.projectDirectory.dir("pdfium/build/pdfium/mac-arm64/lib").asFile + + binaries.executable { + entryPoint = "io.github.lemcoder.mikromarkdown.cli.main" + linkerOpts("-L${pdfiumLib.absolutePath}", "-lpdfium", "-rpath", pdfiumLib.absolutePath) + } compilerOptions { // Worth about 8% on large inputs and nothing on small ones. Measured, not assumed: @@ -15,5 +23,11 @@ kotlin { } } - sourceSets { macosArm64Main.dependencies { implementation(project(":library")) } } + sourceSets { + macosArm64Main.dependencies { + implementation(project(":library")) + // PDF is a separate module by design; the CLI is the thing that opts in. + implementation(project(":pdfium")) + } + } } diff --git a/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt b/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt index 485a19c..665be14 100644 --- a/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt +++ b/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt @@ -2,6 +2,7 @@ package io.github.lemcoder.mikromarkdown.cli import io.github.lemcoder.mikromarkdown.MikroMarkdown import io.github.lemcoder.mikromarkdown.MikroMarkdownException +import io.github.lemcoder.mikromarkdown.pdf.PdfiumConverter import kotlin.system.exitProcess /** @@ -18,7 +19,8 @@ public fun main(args: Array) { exitProcess(2) } - val mikroMarkdown = MikroMarkdown() + // PDF lives in its own module; the CLI opts in, the library does not depend on it. + val mikroMarkdown = MikroMarkdown().apply { register(PdfiumConverter()) } for (path in args) { try { diff --git a/cli/build.gradle.kts b/cli/build.gradle.kts deleted file mode 100644 index d72093a..0000000 --- a/cli/build.gradle.kts +++ /dev/null @@ -1,134 +0,0 @@ -plugins { - alias(libs.plugins.kotlin.jvm) - application -} - -java { toolchain { languageVersion = JavaLanguageVersion.of(21) } } - -val cdsArchiveName = "mikromarkdown.jsa" - -application { - mainClass = "com.mikromarkdown.cli.MainKt" - // Class loading, not conversion, is what a short CLI run spends its time on. A class-data-sharing - // archive maps those classes in pre-parsed. -Xshare:auto keeps the CLI working when it is absent. - // Every CLI run is short, so C2 never pays for itself: compiling with C1 only is faster - // end to end. Long-running embedders use the library directly and are unaffected. - applicationDefaultJvmArgs = listOf("-Xshare:auto", "-XX:-UsePerfData", "-XX:TieredStopAtLevel=1") -} - -dependencies { - implementation(project(":library")) - implementation(libs.clikt) -} - -tasks.test { useJUnitPlatform() } - -// The archive flag is added by the script itself, and only when the archive is there: naming a -// missing archive stops the JVM loading its base one, which is also what recording needs. -tasks.named("startScripts") { - doLast { - val unixGuard = - """ - CDS_ARCHIVE="${'$'}APP_HOME/lib/$cdsArchiveName" - if [ -f "${'$'}CDS_ARCHIVE" ] && [ -z "${'$'}MIKROMARKDOWN_NO_CDS" ] ; then - DEFAULT_JVM_OPTS="${'$'}DEFAULT_JVM_OPTS \"-XX:SharedArchiveFile=${'$'}CDS_ARCHIVE\"" - fi - """ - .trimIndent() - - // Lambda form: the guard contains $ sequences that must not be read as group references. - unixScript.writeText( - unixScript.readText().replace(Regex("(?m)^(DEFAULT_JVM_OPTS=.*)${'$'}")) { match -> - "${match.value}\n\n$unixGuard" - } - ) - - val windowsGuard = - """ - set CDS_ARCHIVE=%APP_HOME%\\lib\\$cdsArchiveName - if exist "%CDS_ARCHIVE%" if not defined MIKROMARKDOWN_NO_CDS set DEFAULT_JVM_OPTS=%DEFAULT_JVM_OPTS% "-XX:SharedArchiveFile=%CDS_ARCHIVE%" - """ - .trimIndent() - - windowsScript.writeText( - windowsScript.readText().replace(Regex("(?m)^(set DEFAULT_JVM_OPTS=.*)${'$'}")) { match -> - "${match.value}\r\n$windowsGuard" - } - ) - } -} - -/** - * Builds a class-data-sharing archive for the installed distribution, in two steps: - * - * 1. run the CLI once with `-XX:DumpLoadedClassList` to learn which classes a conversion touches; - * 2. `-Xshare:dump` that list into an archive. - * - * This static form is used rather than `-XX:ArchiveClassesAtExit` because the dynamic one needs the JDK's own base - * archive, which some distributions (JetBrains Runtime among them) do not ship. - * - * Step 1 goes through the start script so the classpath recorded is the one real runs use; step 2 reads that same - * classpath back out of the script, because a mismatch makes the JVM drop the archive without saying so. - */ -val cdsArchive by tasks.registering { - group = "distribution" - description = "Builds a class-data-sharing archive into the installed distribution." - - val installDir = layout.buildDirectory.dir("install/${application.applicationName}").get() - val appName = application.applicationName - val fixtures = rootProject.layout.projectDirectory.dir("library/src/commonTest/resources/test_files") - // One sample per family, so the archive covers the classes each conversion path touches rather - // than only the ones a DOCX happens to need. - val samples = - listOf("test.docx", "test.pdf", "test_blog.html", "test.csv", "test.json", "test.xlsx").map { - fixtures.file(it).asFile.absolutePath - } - val javaHome = javaToolchains.launcherFor(java.toolchain).get().metadata.installationPath - - doLast { - val script = installDir.file("bin/$appName").asFile - val classList = installDir.file("lib/$cdsArchiveName.classlist").asFile - val archive = installDir.file("lib/$cdsArchiveName").asFile - archive.delete() - - // One run over one sample of each family: a single list keeps the loader metadata that - // makes the archive worth having, which merging separate runs would throw away. - val process = - ProcessBuilder(listOf(script.absolutePath) + samples) - .redirectOutput(ProcessBuilder.Redirect.DISCARD) - .also { - it.environment()["JAVA_OPTS"] = "-XX:DumpLoadedClassList=${classList.absolutePath}" - it.environment()["MIKROMARKDOWN_NO_CDS"] = "1" - } - .start() - val errors = process.errorStream.bufferedReader().readText() - check(process.waitFor() == 0) { "recording failed: ${errors.take(400)}" } - check(classList.exists()) { "the JVM recorded no class list" } - - val classpath = - script - .readLines() - .first { it.startsWith("CLASSPATH=") } - .removePrefix("CLASSPATH=") - .replace("\$APP_HOME", installDir.asFile.absolutePath) - - providers - .exec { - commandLine( - javaHome.file("bin/java").asFile.absolutePath, - "-Xshare:dump", - "-XX:SharedClassListFile=${classList.absolutePath}", - "-XX:SharedArchiveFile=${archive.absolutePath}", - "-cp", - classpath, - ) - } - .standardOutput - .asText - .get() - check(archive.exists()) { "no CDS archive was produced" } - logger.lifecycle("CDS archive: ${archive.length() / 1024} KB") - } -} - -tasks.named("installDist") { finalizedBy(cdsArchive) } diff --git a/cli/src/main/kotlin/com/mikromarkdown/cli/Main.kt b/cli/src/main/kotlin/com/mikromarkdown/cli/Main.kt deleted file mode 100644 index 11974c1..0000000 --- a/cli/src/main/kotlin/com/mikromarkdown/cli/Main.kt +++ /dev/null @@ -1,12 +0,0 @@ -package com.mikromarkdown.cli - -import com.github.ajalt.clikt.core.main - -fun main(args: Array) { - // PDFBox pulls in the Log4j API; without a provider it writes a banner to stdout, - // which would corrupt the Markdown we print there. - System.setProperty("log4j2.loggerContextFactory", "org.apache.logging.log4j.simple.SimpleLoggerContextFactory") - System.setProperty("log4j2.simplelogLevel", "OFF") - System.setProperty("log4j2.statusLoggerLevel", "OFF") - MikroMarkdownCommand().main(args) -} diff --git a/cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt b/cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt deleted file mode 100644 index b683c43..0000000 --- a/cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt +++ /dev/null @@ -1,31 +0,0 @@ -package com.mikromarkdown.cli - -import com.github.ajalt.clikt.core.CliktCommand -import com.github.ajalt.clikt.parameters.arguments.argument -import com.github.ajalt.clikt.parameters.arguments.multiple -import com.github.ajalt.clikt.parameters.options.option -import com.github.ajalt.clikt.parameters.types.path -import io.github.lemcoder.mikromarkdown.MikroMarkdown -import io.github.lemcoder.mikromarkdown.StreamInfo - -class MikroMarkdownCommand : CliktCommand(name = "mikromarkdown") { - private val files by - argument("FILE", help = "Input files (reads stdin if omitted)").path(mustExist = true).multiple() - private val output by option("-o", "--output", help = "Output file (default: stdout)").path() - private val extension by option("-x", "--extension", help = "File extension hint (e.g. html)") - private val mimeType by option("-m", "--mime-type", help = "MIME type hint (e.g. text/html)") - - override fun run() { - val mikroMarkdown = MikroMarkdown() - - val markdown = - if (files.isEmpty()) { - val info = StreamInfo(extension = extension, mimetype = mimeType) - mikroMarkdown.convert(System.`in`.readBytes(), info).markdown - } else { - files.joinToString("\n\n") { mikroMarkdown.convert(it.toFile().absolutePath).markdown } - } - - if (output != null) output!!.toFile().writeText(markdown) else print(markdown) - } -} diff --git a/docs/common-converters-plan.md b/docs/common-converters-plan.md new file mode 100644 index 0000000..e43eda0 --- /dev/null +++ b/docs/common-converters-plan.md @@ -0,0 +1,249 @@ +# Plan: the remaining converters in commonMain + +Status: proposal. Branch `common-converters`. Nothing implemented yet. + +## Where things stand + +**The office formats are gone.** DOCX, XLSX and PPTX were deleted rather than ported: they are +editing formats, and a reader meets PDF and EPUB. That took Apache POI with them — 66 MB of +distribution down to 36 MB, 35 jars to 23 — and removed the phase most likely to change output +silently. They can return as an opt-in module in the shape `:pdfium` is taking; the fixtures stay, +and a test pins that they currently raise `UnsupportedFormatException`. + +What is left JVM-only is PDF, and nothing else: + +| file | lines | depends on | plan | +|---|---|---|---| +| `PdfConverter.kt` ×2 | 32 | PDFBox / pdfbox-android | → a separate `:pdfium` module | + +Moving them is not a file move: `commonMain` cannot use POI, Jsoup, `java.util.zip` or `javax.xml`, +so each is a rewrite against the raw format. CSV, JSON and XML made this trip already and their +output stayed byte-identical, which is the bar for everything except PDF. + +## Why bother + +**iOS.** The document formats are the reason a SwiftUI reader cannot exist today. This is the blocker, +and it is worth doing even if nothing gets faster. + +**A smaller JVM build** — already banked by the deletion, and worth being exact about what it bought: +36 MB instead of 66 MB, but **no faster**. POI was loaded lazily, so a CSV or EPUB conversion never +paid for it; the startup numbers did not move. Size and dependency surface improved, speed did not. + +What remains to remove is Jsoup, and it goes when HTML moves to Ksoup. + +Against all of it: POI, Jsoup and PDFBox absorb an enormous amount of real-world malformation. +Hand-written parsers will be less forgiving, and the fixture corpus is ten files. **Expanding it is +part of the work, not an afterthought.** + +## Building blocks + +| need | choice | notes | +|---|---|---| +| inflate | `com.soywiz:korlibs-compression:6.0.0` | EPUB is a ZIP; nothing in kotlinx-io or okio inflates on native. It ships **no ZIP reader**, so the central directory is ours to parse — about 150 lines | +| HTML parsing | `com.fleeksoft.ksoup:ksoup:0.2.6` | KMP port of Jsoup, near-identical API, `macosarm64` published | +| PDF | pdfium via cinterop + JNI | see below | + +Verified on `macosArm64`, running rather than merely linking: Ksoup parses HTML and, in XML mode, +reads an OPF well enough to pull `dc:title`, manifest hrefs, spine idrefs and the cover meta — +**so no separate XML library is needed**. korlibs deflate round-trips (1000 bytes → 31 → 1000). +The korlibs API lives under `korlibs.io.compression.*`, not `korlibs.compression.*`. + +## PDF: a `:pdfium` module + +PDFBox has no KMP equivalent and text extraction is its own project, so PDF goes native through +[pdfium](https://pdfium.googlesource.com/pdfium/), bound with +[KonanPlugin](https://github.com/lemcoder/KonanPlugin) (`io.github.lemcoder.konanplugin:1.2.0-alpha05`, +on the plugin portal). The plugin generates JNI bindings from the *same* `.def` file cinterop binds, +so one declaration serves Kotlin/Native, JVM and Android. + +``` +pdfium/ + src/nativeInterop/cinterop/pdfium.def headers = fpdfview.h fpdf_text.h fpdf_doc.h + src/commonMain/…/PdfiumConverter.kt DocumentConverter over the shim + src/commonMain/…/Pdfium.kt expect: open, pageCount, pageText, close + src/macosArm64Main/…/Pdfium.kt actual over the cinterop klib + src/jvmMain/…/Pdfium.kt actual over the generated JNI bridges + build.gradle.kts konanplugin: cinterop + jvmInterops from one def +``` + +Binaries come from [`bblanchon/pdfium-binaries`](https://github.com/bblanchon/pdfium-binaries), which +publishes per-platform archives (`pdfium-mac-arm64.tgz`, `pdfium-linux-x64.tgz`, +`pdfium-android-arm64.tgz`, …) containing headers and a library. A Gradle task downloads and unpacks +a **pinned release with a checksum** into `build/pdfium//`; nothing binary is committed. + +Text needs `FPDF_InitLibrary`, `FPDF_LoadMemDocument`, `FPDF_GetPageCount`, `FPDF_LoadPage`, +`FPDFText_LoadPage`, `FPDFText_CountChars`, `FPDFText_GetText` (UTF-16), and the matching closes. +The existing `plainTextBlocks` reflow and de-hyphenation sit on top unchanged. + +**Images matter as much as text here**, because the Compose and SwiftUI readers need them, so the +module walks page objects rather than only the text layer: + +| step | call | +|---|---| +| iterate objects on a page | `FPDFPage_CountObjects`, `FPDFPage_GetObject` | +| keep the images | `FPDFPageObj_GetType` == `FPDF_PAGEOBJ_IMAGE` | +| where it sits on the page | `FPDFPageObj_GetBounds` | +| size and colour depth | `FPDFImageObj_GetImageMetadata` | +| how it is stored | `FPDFImageObj_GetImageFilterCount`, `FPDFImageObj_GetImageFilter` | +| the bytes | `FPDFImageObj_GetImageDataDecoded`, or `FPDFImageObj_GetRenderedBitmap` | + +Two cases, and the second is the awkward one: + +- **`DCTDecode` or `JPXDecode`** — the decoded stream *is* a JPEG or JPEG 2000 file, so the bytes go + straight into an `Asset` with the matching media type. Free. +- **Anything else** (Flate-compressed raw pixels, which is very common) — there is no image file in + the PDF, only pixels. `FPDFImageObj_GetRenderedBitmap` returns BGRA, and something has to encode + it. `commonMain` has no PNG encoder, so we write one: PNG is a header, an IDAT of deflated + scanlines and a CRC, and korlibs-compression is already there for the deflate. Perhaps 150 lines, + and worth having anyway — no other target has an encoder either. + +**Placement.** `FPDFPageObj_GetBounds` gives each image a rectangle and `FPDFText_GetCharBox` gives +the text the same, so images can be emitted in reading order by vertical position rather than dumped +at the end of the page. Without it a figure lands after the prose that discusses it. This is the same +problem the PDFBox route had and never solved. + +**Scanned pages** — no text and one full-page image — are detectable (an image covering most of the +mediabox, almost no characters) and worth flagging rather than emitting as a wall of nothing. + +**The module does not register itself.** `:library` keeps no PDF dependency, and a caller opts in: + +```kotlin +val mikroMarkdown = MikroMarkdown().apply { register(PdfiumConverter()) } +``` + +That keeps `:pdfium` genuinely extractable — delete the module and the rest still builds. + +### What this costs + +- **PDF output will change.** pdfium and PDFBox extract text differently, so the `test.pdf` baseline + has to be re-recorded. Byte-identity cannot be the gate here; `PythonComparisonTest`'s token recall + against Python markitdown becomes the check, plus a read of the diff. +- **Distribution gets heavier and platform-shaped.** The JVM artifact needs the pdfium library and the + generated stub per platform, 4-8 MB each. Building stubs for every JVM host means a CI matrix — + from one machine we can only produce the host's. +- **Licensing**: pdfium is BSD-3-Clause with Apache-2.0 parts; the notices ship with the artifact. + +## Sequence + +Each phase is shippable on its own, risky ones last. + +### Phase 0 — infrastructure ✅ +Ksoup and korlibs-compression are in `commonMain` and proven to run on the native binary. xmlutil +turned out to be unnecessary — Ksoup's XML mode covers EPUB's container and OPF. Still to do when a +phase needs it: extend `scripts/optbench.py` to verify native output for the fixtures it unlocks. + +### Phase 1 — HTML, via Ksoup ✅ +Done, and the feared risk did not materialise: `test_blog.html` and `test_wikipedia.html` are +byte-identical through Ksoup, and the native binary matches the JVM on both. Two API differences +only — `wholeText` is a method rather than a property, and `Charsets` does not exist in commonMain. + +Jsoup is gone from the build. HTML on native converts a blog in 7 ms against the JVM's 80, and +Wikipedia in 66 ms against 119. + +### Phase 2 — EPUB ✅ +Done, byte-identical, and the JVM-only source set is gone with it. `ZipArchive` in commonMain reads +the central directory and inflates through korlibs; the container and package documents go through +Ksoup's XML mode; chapters reuse Phase 1. EPUB now converts on native. + +### Phase 3 — PDF, the `:pdfium` module — text done, geometry outstanding +The binding works: the release is pinned by checksum, unpacked at build time, and the native CLI +converts a PDF through `PdfiumConverter`. Two notes on getting there — the published dylib names +itself `./libpdfium.dylib`, which the loader resolves against the working directory, so the unpack +step rewrites it to `@rpath`; and Kotlin/Native does not carry a klib's linker options to the binary +that links it, so the consumer names pdfium itself. + +Against PDFBox on the same file, pdfium keeps 96.2% of its tokens and the differences are pdfium +reading *better*: PDFBox leaves `flexibil`, `firming`, `gramming` as fragments where pdfium plus +de-hyphenation produces whole words. + +**Two defects remain, and geometry fixes both.** + +pdfium emits U+FFFE where a glyph has no Unicode mapping, which in a typeset paper is the hyphen at +a line break. Dropping it fuses real compounds (`chat-optimized` → `chatoptimized`); keeping it +splits real words. The document's own vocabulary decides today, which gets four joins right and two +compounds wrong — and cannot do better, because the halves of a broken word are only in the text at +all because the break put them there. **The real discriminator is that a hyphenation hyphen sits at +the end of a line and a compound hyphen does not**, which `FPDFText_GetCharBox` answers directly. + +The same call answers paragraphs. pdfium returns a page as one run of text, so a PDF currently +renders as a single paragraph where PDFBox produced seven. Character boxes give the line breaks, and +the vertical gaps between them give the paragraphs. + +### Phase 3b — PDF images +The **PNG encoder is done**: `PngEncoder` in `commonMain` writes 8-bit RGBA through korlibs' ZLib, +checked on the JVM by decoding what it writes with ImageIO and on native by validating the chunks +and CRCs outside the process. Filtering stays at "none" — a larger file for a much smaller encoder. + +What is left is the pdfium side: walking page objects, telling a JPEG stream (hand it on) from raw +pixels (encode), and placing images in reading order by their bounds. + +### Phase 4 — FB2, if wanted +Book's Story parses FB2 with the same walker it uses for HTML, through Jsoup's XML mode. Ksoup has +that mode too, so once Phase 1 lands FB2 is a tag-mapping layer over `HtmlToDocument` — a format we +do not support today for about half a day's work. Worth it for a reader; skip it otherwise. + +### Removed — DOCX, XLSX, PPTX +Deleted, not deferred. If they return it is as an `:office` module mirroring `:pdfium`: its own +Gradle module, its own dependency, registered by the caller rather than by the factory. The deleted +implementations are in the history, and the fixtures are still in the repository. + +## Assets, across every phase + +The model already carries them — `Asset(id, mediaType, bytes, name)`, `Document.assets` and +`Image.assetId` — but only DOCX populates them today. The readers need images from everything, so +each phase extracts what its format holds: + +| format | where the images are | notes | +|---|---|---| +| docx | done | ids are the file name and can collide; needs fixing | +| pdf | page objects, as above | needs a PNG encoder for raw-pixel images | +| pptx | `XSLFPictureShape` today emits a fabricated `shapeName.jpg` and no bytes | real assets when the raw-XML rewrite lands | +| epub | `` resolved against the chapter directory into a zip entry we already hold | needs an asset resolver in the HTML walker | +| html | remote URLs stay URLs; `data:` URIs decode into assets | cheap | +| xlsx | deferred with the rest of XLSX | | + +Three things this needs that do not exist yet: + +1. **An asset policy.** DOCX inlines base64 today, which is why its output is 161 KB against + markitdown's 4.6 KB. `MarkdownOptions` should carry `Inline` / `Reference` (write files, emit + relative links) / `AltTextOnly`; the current `imagesAsText` and `maxInlineImageUrl` are a crude + stand-in. A PDF full of figures makes this urgent rather than tidy. +2. **Stable asset ids.** The DOCX id is the embedded file name and repeats across parts. +3. **Intrinsic size on `Image`.** Width and height from the source, so a Compose or SwiftUI layout + does not jump while loading. `FPDFImageObj_GetImageMetadata` provides it for PDF. + +## Ground rules per phase + +1. The new implementation lands in `commonMain` and the platform one is deleted in the same commit. + The Konsist duplicate-file rule catches anything copied per target by accident; where a split is + deliberate, as the PDF module's JVM and Android legs are, it is written down as such. +2. `scripts/optbench.py` must report every fixture byte-identical on both targets before timings are + read. Where a difference is deliberate — PDF, and possibly HTML — the baseline is updated in the + same commit with the diff quoted in the message. +3. `PythonComparisonTest` stays at 100% token recall. +4. Timings are A/B against the champion binary, never absolute. +5. Each phase adds fixtures for what it implements: a DOCX with numbered and nested lists, a PPTX with + a chart, a PDF with columns and hyphenation. Ten fixtures is too few to rewrite parsers against. + +## Estimate + +| phase | effort | risk | +|---|---|---| +| 0 infrastructure | half a day | low — or the plan changes, if a library will not build | +| 1 HTML | 1 day | medium — parser differences on messy input | +| 2 EPUB | half a day | low | +| 3 PDF via pdfium, text | 2 days | medium — binding is routine, packaging and output changes are not | +| 3b PDF images, PNG encoder, placement | 2-3 days | medium — the PNG writer is small but the reading-order interleave is fiddly | +| assets for epub, html, pptx | 1 day, spread across their phases | low | +| asset policy, ids, intrinsic size | half a day | low, but blocks the readers | +| 4 FB2, optional | half a day | low | + +Roughly a week, images included, now that the office formats are out. + +## Worth deciding before starting + +- **Which targets beyond macOS?** Adding `iosArm64` and `linuxX64` early keeps the code honest; + adding them late risks finding a dependency — or a pdfium archive — that does not fit. +- **How are pdfium binaries shipped to JVM users** — bundled per platform in the artifact, or + downloaded at build time by the consumer? The first is convenient and large; the second is small + and one more thing to go wrong. diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 0a63414..3f394ec 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -6,17 +6,15 @@ android-compileSdk = "37" vanniktechMavenPublish = "0.36.0" kotlinx-io = "0.9.0" kotlinx-serialization = "1.9.0" +ksoup = "0.2.6" +korlibs = "6.0.0" +konanplugin = "1.2.0-alpha06" kotlinx-resources = "0.15.0" commons-csv = "1.14.1" jackson = "2.21.3" -jsoup = "1.22.2" junit = "6.1.0" -pdfbox = "3.0.7" -pdfbox-android = "2.0.27.0" -poi = "5.5.1" tika = "3.3.0" -clikt = "5.1.0" coreKtx = "1.7.0" detekt = "1.23.8" ktfmt = "0.27.0" @@ -25,18 +23,15 @@ konsist = "0.17.3" [libraries] kotlin-test = { module = "org.jetbrains.kotlin:kotlin-test", version.ref = "kotlin" } kotlinx-io-core = { module = "org.jetbrains.kotlinx:kotlinx-io-core", version.ref = "kotlinx-io" } +ksoup = { module = "com.fleeksoft.ksoup:ksoup", version.ref = "ksoup" } +korlibs-compression = { module = "com.soywiz:korlibs-compression", version.ref = "korlibs" } kotlinx-serialization-json = { module = "org.jetbrains.kotlinx:kotlinx-serialization-json", version.ref = "kotlinx-serialization" } kotlinx-resources = { module = "com.goncalossilva:resources", version.ref = "kotlinx-resources" } commons-csv = { module = "org.apache.commons:commons-csv", version.ref = "commons-csv" } jackson-kotlin = { module = "com.fasterxml.jackson.module:jackson-module-kotlin", version.ref = "jackson" } -jsoup = { module = "org.jsoup:jsoup", version.ref = "jsoup" } junit-jupiter = { module = "org.junit.jupiter:junit-jupiter", version.ref = "junit" } -pdfbox = { module = "org.apache.pdfbox:pdfbox", version.ref = "pdfbox" } -pdfbox-android = { module = "com.tom-roush:pdfbox-android", version.ref = "pdfbox-android" } -poi-ooxml = { module = "org.apache.poi:poi-ooxml", version.ref = "poi" } tika-core = { module = "org.apache.tika:tika-core", version.ref = "tika" } -clikt = { module = "com.github.ajalt.clikt:clikt", version.ref = "clikt" } core-ktx = { group = "androidx.test", name = "core-ktx", version.ref = "coreKtx" } konsist = { module = "com.lemonappdev:konsist", version.ref = "konsist" } @@ -48,4 +43,5 @@ kotlin-serialization = { id = "org.jetbrains.kotlin.plugin.serialization", versi vanniktech-mavenPublish = { id = "com.vanniktech.maven.publish", version.ref = "vanniktechMavenPublish" } kotlinx-resources = { id = "com.goncalossilva.resources", version.ref = "kotlinx-resources" } detekt = { id = "io.gitlab.arturbosch.detekt", version.ref = "detekt" } +konanplugin = { id = "io.github.lemcoder.konanplugin", version.ref = "konanplugin" } ktfmt = { id = "com.ncorti.ktfmt.gradle", version.ref = "ktfmt" } diff --git a/library/:memory:.ses b/library/:memory:.ses new file mode 100644 index 0000000..95ddb84 --- /dev/null +++ b/library/:memory:.ses @@ -0,0 +1,2 @@ +1787007442705 +2b657774-881a-47ed-9141-c15c8c8f1ab7 diff --git a/library/build.gradle.kts b/library/build.gradle.kts index 5a405c9..f750228 100644 --- a/library/build.gradle.kts +++ b/library/build.gradle.kts @@ -14,7 +14,6 @@ kotlin { // Public API must be spelled out: visibility and return types, no accidental exports. explicitApi() - // Declaring jvmShared by hand switches off the automatic hierarchy, which macosMain needs. applyDefaultHierarchyTemplate() jvm { @@ -39,30 +38,14 @@ kotlin { } sourceSets { - commonMain.dependencies { implementation(libs.kotlinx.io.core) } - - // JVM and Android run the same parsers on the same libraries; only PDF and MIME - // detection differ. Converters live here once instead of being copied per target. - val jvmShared by creating { - dependsOn(commonMain.get()) - dependencies { - implementation(libs.jsoup) - implementation(libs.poi.ooxml) - } + commonMain.dependencies { + implementation(libs.kotlinx.io.core) + // Phase 0 of the commonMain migration: HTML parsing and the inflate that ZIP needs. + implementation(libs.ksoup) + implementation(libs.korlibs.compression) } - jvmMain { - dependsOn(jvmShared) - dependencies { - implementation(libs.tika.core) - implementation(libs.pdfbox) - } - } - - androidMain { - dependsOn(jvmShared) - dependencies { implementation(libs.pdfbox.android) } - } + jvmMain { dependencies { implementation(libs.tika.core) } } commonTest.dependencies { implementation(libs.kotlin.test) diff --git a/library/src/androidDeviceTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.android.kt b/library/src/androidDeviceTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.android.kt index 97d5560..15c3cc4 100644 --- a/library/src/androidDeviceTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.android.kt +++ b/library/src/androidDeviceTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.android.kt @@ -1,5 +1,5 @@ package io.github.lemcoder.mikromarkdown actual fun testMikroMarkdown(): MikroMarkdown { - return MikroMarkdown(context = null) + return MikroMarkdown() } diff --git a/library/src/androidHostTest/kotlin/io/github/lemcoder/mikromarkdown/TestMikroMarkdown.kt b/library/src/androidHostTest/kotlin/io/github/lemcoder/mikromarkdown/TestMikroMarkdown.kt index 1b9af9d..7c1314e 100644 --- a/library/src/androidHostTest/kotlin/io/github/lemcoder/mikromarkdown/TestMikroMarkdown.kt +++ b/library/src/androidHostTest/kotlin/io/github/lemcoder/mikromarkdown/TestMikroMarkdown.kt @@ -1,3 +1,3 @@ package io.github.lemcoder.mikromarkdown -actual fun testMikroMarkdown(): MikroMarkdown = MikroMarkdown(context = null) +actual fun testMikroMarkdown(): MikroMarkdown = MikroMarkdown() diff --git a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt index 65b3f85..8463eae 100644 --- a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt +++ b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt @@ -1,40 +1,28 @@ package io.github.lemcoder.mikromarkdown -import android.content.Context -import com.tom_roush.pdfbox.android.PDFBoxResourceLoader import io.github.lemcoder.mikromarkdown.converters.CsvConverter -import io.github.lemcoder.mikromarkdown.converters.DocxConverter import io.github.lemcoder.mikromarkdown.converters.EpubConverter import io.github.lemcoder.mikromarkdown.converters.HtmlConverter import io.github.lemcoder.mikromarkdown.converters.JsonConverter import io.github.lemcoder.mikromarkdown.converters.MarkdownPassthroughConverter -import io.github.lemcoder.mikromarkdown.converters.PdfConverter import io.github.lemcoder.mikromarkdown.converters.PlainTextConverter -import io.github.lemcoder.mikromarkdown.converters.PptxConverter -import io.github.lemcoder.mikromarkdown.converters.XlsxConverter import io.github.lemcoder.mikromarkdown.converters.XmlConverter import java.io.File /** * A [MikroMarkdown] with every Android converter registered. * - * PDF support needs a [Context]: pdfbox-android loads its resources from the app's assets. + * PDF is not among them: it needs a native library, so the `:pdfium` module provides it and the caller opts in with + * `register(PdfiumConverter())`. */ -public fun MikroMarkdown(context: Context? = null): MikroMarkdown = +public fun MikroMarkdown(): MikroMarkdown = MikroMarkdown(SignatureMimeDetector).apply { register(MarkdownPassthroughConverter()) register(HtmlConverter()) register(CsvConverter()) register(JsonConverter()) register(XmlConverter()) - register(DocxConverter()) - register(XlsxConverter()) - register(PptxConverter()) register(EpubConverter()) - if (context != null) { - PDFBoxResourceLoader.init(context) - register(PdfConverter()) - } register(PlainTextConverter(), priority = 10.0) } diff --git a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt deleted file mode 100644 index 4b13531..0000000 --- a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt +++ /dev/null @@ -1,32 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import com.tom_roush.pdfbox.pdmodel.PDDocument -import com.tom_roush.pdfbox.text.PDFTextStripper -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.utils.plainTextBlocks - -public class PdfConverter : DocumentConverter { - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "pdf" || info.mimetype == "application/pdf" - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val doc = PDDocument.load(bytes) - try { - val title = doc.documentInformation?.title?.trim()?.ifBlank { null } - // Paragraph markers let the text blocks split on real paragraph breaks - // instead of collapsing a page into one block. - val stripper = - PDFTextStripper().apply { - sortByPosition = true - setAddMoreFormatting(true) - paragraphStart = "\n" - } - return Document(blocks = plainTextBlocks(stripper.getText(doc)), title = title) - } finally { - doc.close() - } - } -} diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt new file mode 100644 index 0000000..16f8323 --- /dev/null +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt @@ -0,0 +1,99 @@ +package io.github.lemcoder.mikromarkdown.converters + +import com.fleeksoft.ksoup.Ksoup +import com.fleeksoft.ksoup.parser.Parser +import io.github.lemcoder.mikromarkdown.DocumentConverter +import io.github.lemcoder.mikromarkdown.StreamInfo +import io.github.lemcoder.mikromarkdown.model.Block +import io.github.lemcoder.mikromarkdown.model.Document +import io.github.lemcoder.mikromarkdown.model.Paragraph +import io.github.lemcoder.mikromarkdown.model.Strong +import io.github.lemcoder.mikromarkdown.model.Text +import io.github.lemcoder.mikromarkdown.utils.HtmlToDocument +import io.github.lemcoder.mikromarkdown.utils.ZipArchive + +/** + * An EPUB is a ZIP of XHTML. + * + * `META-INF/container.xml` names the package document, which lists the manifest and the reading order; the chapters + * themselves go through the same walker as any other HTML. The XML is read with Ksoup's XML parser rather than a second + * library, which is why there is no XML dependency. + */ +public class EpubConverter : DocumentConverter { + private val metaFields = + listOf( + "title" to "Title", + "creator" to "Authors", + "language" to "Language", + "description" to "Description", + "identifier" to "Identifier", + ) + + override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { + return info.extension == "epub" || info.mimetype == "application/epub+zip" + } + + override fun parse(bytes: ByteArray, info: StreamInfo): Document { + val archive = ZipArchive.open(bytes) ?: return Document() + + val container = archive.readText("META-INF/container.xml") ?: return Document() + val opfPath = opfPath(container) ?: return Document() + val opf = archive.readText(opfPath) ?: archive.readText(opfPath.removePrefix("/")) ?: return Document() + val opfDirectory = opfPath.substringBeforeLast("/", "") + + val packageDocument = Ksoup.parse(html = opf, parser = Parser.xmlParser()) + val metadata = metadata(packageDocument) + val manifest = manifest(packageDocument) + + val blocks = mutableListOf() + var title: String? = metadata["title"] + + for ((key, label) in metaFields) { + val value = metadata[key] ?: continue + blocks += Paragraph(listOf(Strong(listOf(Text("$label:"))), Text(" $value"))) + } + + for (idref in spine(packageDocument)) { + val href = manifest[idref] ?: continue + val path = if (opfDirectory.isEmpty()) href else "$opfDirectory/$href" + val html = archive.readText(path) ?: archive.readText(path.removePrefix("/")) ?: continue + val chapter = HtmlToDocument.parse(html) + if (title == null) title = chapter.title + blocks += chapter.blocks + } + + return Document(blocks = blocks, title = title, metadata = metadata) + } + + private fun opfPath(container: String): String? = + Ksoup.parse(html = container, parser = Parser.xmlParser()).selectFirst("rootfile")?.attr("full-path")?.ifEmpty { + null + } + + private fun metadata(packageDocument: com.fleeksoft.ksoup.nodes.Document): Map { + val metadata = LinkedHashMap() + for (field in listOf("title", "creator", "language", "description", "identifier")) { + // Namespaced in the source as dc:title; Ksoup escapes the prefix with a pipe. + val text = packageDocument.selectFirst("dc|$field")?.text()?.trim() + if (!text.isNullOrEmpty()) metadata[field] = text + } + return metadata + } + + /** Manifest ids to hrefs, keeping only the documents that carry text. */ + private fun manifest(packageDocument: com.fleeksoft.ksoup.nodes.Document): Map { + val manifest = LinkedHashMap() + for (item in packageDocument.select("manifest > item")) { + val id = item.attr("id") + val href = item.attr("href") + val mediaType = item.attr("media-type") + if (id.isNotEmpty() && href.isNotEmpty() && mediaType.contains("html")) { + manifest[id] = href + } + } + return manifest + } + + private fun spine(packageDocument: com.fleeksoft.ksoup.nodes.Document): List = + packageDocument.select("spine > itemref").map { it.attr("idref") }.filter { it.isNotEmpty() } +} diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt similarity index 87% rename from library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt rename to library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt index fee70f9..38f1c3b 100644 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt @@ -11,5 +11,5 @@ public class HtmlConverter : DocumentConverter { } override fun parse(bytes: ByteArray, info: StreamInfo): Document = - HtmlToDocument.parse(bytes.toString(Charsets.UTF_8), info.localPath.orEmpty()) + HtmlToDocument.parse(bytes.decodeToString(), info.localPath.orEmpty()) } diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoder.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoder.kt new file mode 100644 index 0000000..9d4d3cf --- /dev/null +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoder.kt @@ -0,0 +1,102 @@ +package io.github.lemcoder.mikromarkdown.model + +import korlibs.io.compression.compress +import korlibs.io.compression.deflate.ZLib + +/** + * Writes 8-bit RGBA pixels as a PNG. + * + * Needed because most images inside a PDF are not image files: a JPEG stream can be handed on as it stands, but a + * Flate-compressed bitmap is raw pixels with no container, and every target needs one to produce. Nothing in the Kotlin + * ecosystem encodes PNG on Kotlin/Native, and the format's essentials are small — a signature, three chunks, a CRC + * apiece, and the deflate that korlibs-compression already provides. + * + * Filtering is left at "none", which trades a larger file for a much simpler encoder. A screenshot inside a PDF + * compresses well enough on deflate alone. + */ +public object PngEncoder { + + private val SIGNATURE = byteArrayOf(-119, 80, 78, 71, 13, 10, 26, 10) + + private const val BIT_DEPTH = 8 + private const val COLOR_TYPE_RGBA = 6 + private const val CHANNELS = 4 + + /** + * @param pixels RGBA, four bytes per pixel, row-major, [width] * [height] * 4 bytes long. + * @return the PNG file, or null if the dimensions and the buffer disagree. + */ + public fun encode(width: Int, height: Int, pixels: ByteArray): ByteArray? { + if (width <= 0 || height <= 0) return null + if (pixels.size != width * height * CHANNELS) return null + + val header = ByteArray(13) + header.writeInt(0, width) + header.writeInt(4, height) + header[8] = BIT_DEPTH.toByte() + header[9] = COLOR_TYPE_RGBA.toByte() + // Compression 0 (deflate), filter 0 (adaptive), interlace 0 (none) — the only values PNG has. + + val chunks = + listOf( + SIGNATURE, + chunk("IHDR", header), + chunk("IDAT", scanlines(width, height, pixels).compress(ZLib)), + chunk("IEND", ByteArray(0)), + ) + + // Concatenated by hand: a list of boxed bytes would cost more than the encoding does. + val out = ByteArray(chunks.sumOf { it.size }) + var at = 0 + for (part in chunks) { + part.copyInto(out, at) + at += part.size + } + return out + } + + /** Each row is preceded by its filter byte; 0 means the row is stored as it is. */ + private fun scanlines(width: Int, height: Int, pixels: ByteArray): ByteArray { + val stride = width * CHANNELS + val out = ByteArray(height * (stride + 1)) + for (row in 0 until height) { + val target = row * (stride + 1) + out[target] = 0 + pixels.copyInto(out, target + 1, row * stride, (row + 1) * stride) + } + return out + } + + /** length, type, payload, CRC of type and payload — the shape every PNG chunk shares. */ + private fun chunk(type: String, payload: ByteArray): ByteArray { + val typeBytes = type.encodeToByteArray() + val out = ByteArray(payload.size + 12) + out.writeInt(0, payload.size) + typeBytes.copyInto(out, 4) + payload.copyInto(out, 8) + out.writeInt(payload.size + 8, crc32(typeBytes, payload)) + return out + } + + private fun ByteArray.writeInt(at: Int, value: Int) { + this[at] = (value ushr 24).toByte() + this[at + 1] = (value ushr 16).toByte() + this[at + 2] = (value ushr 8).toByte() + this[at + 3] = value.toByte() + } + + private val CRC_TABLE = + IntArray(256) { index -> + var value = index + repeat(8) { value = if (value and 1 != 0) 0xEDB88320.toInt() xor (value ushr 1) else value ushr 1 } + value + } + + private fun crc32(vararg parts: ByteArray): Int { + var crc = -1 + for (part in parts) { + for (byte in part) crc = CRC_TABLE[(crc xor byte.toInt()) and 0xFF] xor (crc ushr 8) + } + return crc.inv() + } +} diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TextBlocks.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/TextBlocks.kt similarity index 88% rename from library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TextBlocks.kt rename to library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/TextBlocks.kt index 787f028..e80aa3f 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TextBlocks.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/TextBlocks.kt @@ -1,16 +1,15 @@ -package io.github.lemcoder.mikromarkdown.utils - -import io.github.lemcoder.mikromarkdown.model.Block -import io.github.lemcoder.mikromarkdown.model.Paragraph -import io.github.lemcoder.mikromarkdown.model.Text +package io.github.lemcoder.mikromarkdown.model /** - * Turns extracted plain text (PDF pages, speaker notes, …) into paragraph blocks. + * Turns extracted plain text into paragraph blocks. + * + * Public because it is what any text-extracting converter needs, including ones outside this module: the `:pdfium` + * module builds its documents with it. * * Blank lines separate paragraphs; soft-wrapped lines inside a paragraph are rejoined, so the Markdown does not inherit * the source layout's line breaks. */ -internal fun plainTextBlocks(text: String, reflow: Boolean = true): List { +public fun plainTextBlocks(text: String, reflow: Boolean = true): List { // Form feeds mark PDF page breaks; treat them as paragraph boundaries. val normalized = text.replace("\r\n", "\n").replace('\r', '\n').replace('\u000C', '\n') val vocabulary = if (reflow) wordsIn(normalized) else emptySet() diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt similarity index 97% rename from library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt rename to library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt index a8a9260..9220084 100644 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt @@ -1,5 +1,9 @@ package io.github.lemcoder.mikromarkdown.utils +import com.fleeksoft.ksoup.Ksoup +import com.fleeksoft.ksoup.nodes.Element +import com.fleeksoft.ksoup.nodes.Node +import com.fleeksoft.ksoup.nodes.TextNode import io.github.lemcoder.mikromarkdown.model.Block import io.github.lemcoder.mikromarkdown.model.BlockQuote import io.github.lemcoder.mikromarkdown.model.CodeBlock @@ -21,10 +25,6 @@ import io.github.lemcoder.mikromarkdown.model.TableCell import io.github.lemcoder.mikromarkdown.model.Text import io.github.lemcoder.mikromarkdown.model.ThematicBreak import io.github.lemcoder.mikromarkdown.model.plainText -import org.jsoup.Jsoup -import org.jsoup.nodes.Element -import org.jsoup.nodes.Node -import org.jsoup.nodes.TextNode /** * Walks an HTML DOM into the shared document model. @@ -79,7 +79,7 @@ internal object HtmlToDocument { ) internal fun parse(html: String, baseUri: String = ""): Document { - val doc = Jsoup.parse(html, baseUri) + val doc = Ksoup.parse(html = html, baseUri = baseUri) doc.select(DROPPED_SELECTOR).remove() val title = doc.title().ifBlank { null } val root = doc.body() ?: doc @@ -310,7 +310,7 @@ internal object HtmlToDocument { /** HTML collapses runs of whitespace; do the same before the text reaches the model. */ // Non-breaking spaces are not collapsible whitespace in HTML, so they survive verbatim. - private fun TextNode.normalizedText(): String = wholeText.replace(COLLAPSIBLE_WHITESPACE, " ") + private fun TextNode.normalizedText(): String = getWholeText().replace(COLLAPSIBLE_WHITESPACE, " ") private fun List.startsWithSpace(): Boolean = (firstOrNull() as? Text)?.value?.startsWith(" ") == true diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/ZipArchive.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/ZipArchive.kt new file mode 100644 index 0000000..b031e56 --- /dev/null +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/ZipArchive.kt @@ -0,0 +1,120 @@ +package io.github.lemcoder.mikromarkdown.utils + +import korlibs.io.compression.deflate.Deflate +import korlibs.io.compression.uncompress + +/** + * Reads a ZIP container held in memory. + * + * korlibs supplies the inflate and nothing else, and neither kotlinx-io nor okio reads archives, so the container + * itself is parsed here: find the end-of-central-directory record, walk the directory, and follow each entry to its + * local header. Enough of the format for EPUB and, later, OOXML. + */ +internal class ZipArchive private constructor(private val source: ByteArray, private val entries: Map) { + + val names: Set + get() = entries.keys + + /** The entry's bytes, inflated if it was deflated, or null if it is absent or unreadable. */ + fun read(name: String): ByteArray? { + val entry = entries[name] ?: return null + val start = dataStart(entry) ?: return null + if (start + entry.compressedSize > source.size) return null + + val raw = source.copyOfRange(start, start + entry.compressedSize) + return when (entry.method) { + STORED -> raw + DEFLATED -> + try { + raw.uncompress(Deflate) + } catch (_: Exception) { + null + } + // Any other method — bzip2, lzma, encrypted — is not something we can read. + else -> null + } + } + + /** The entry decoded as UTF-8, which every XML and XHTML part of an EPUB is. */ + fun readText(name: String): String? = read(name)?.decodeToString() + + /** + * Where an entry's bytes begin. + * + * The central directory records where the local header is, but the local header repeats the name and extra field + * with lengths of its own, so the data offset can only be computed from there. + */ + private fun dataStart(entry: Entry): Int? { + val header = entry.localHeaderOffset + if (header + LOCAL_HEADER_MINIMUM > source.size) return null + if (source.readInt(header) != LOCAL_HEADER_SIGNATURE) return null + val nameLength = source.readShort(header + 26) + val extraLength = source.readShort(header + 28) + return header + LOCAL_HEADER_MINIMUM + nameLength + extraLength + } + + private class Entry(val localHeaderOffset: Int, val compressedSize: Int, val method: Int) + + companion object { + private const val STORED = 0 + private const val DEFLATED = 8 + private const val END_OF_DIRECTORY_SIGNATURE = 0x06054b50 + private const val DIRECTORY_ENTRY_SIGNATURE = 0x02014b50 + private const val LOCAL_HEADER_SIGNATURE = 0x04034b50 + private const val LOCAL_HEADER_MINIMUM = 30 + private const val DIRECTORY_ENTRY_MINIMUM = 46 + private const val END_OF_DIRECTORY_MINIMUM = 22 + + /** The comment that may follow the end record, and so how far back it can hide. */ + private const val MAX_COMMENT = 0xFFFF + + fun open(bytes: ByteArray): ZipArchive? { + val end = findEndOfDirectory(bytes) ?: return null + val count = bytes.readShort(end + 10) + var offset = bytes.readInt(end + 16) + + val entries = LinkedHashMap(count) + repeat(count) { + if (offset + DIRECTORY_ENTRY_MINIMUM > bytes.size) return@repeat + if (bytes.readInt(offset) != DIRECTORY_ENTRY_SIGNATURE) return@repeat + + val method = bytes.readShort(offset + 10) + val compressedSize = bytes.readInt(offset + 20) + val nameLength = bytes.readShort(offset + 28) + val extraLength = bytes.readShort(offset + 30) + val commentLength = bytes.readShort(offset + 32) + val localHeaderOffset = bytes.readInt(offset + 42) + + val nameStart = offset + DIRECTORY_ENTRY_MINIMUM + if (nameStart + nameLength > bytes.size) return@repeat + val name = bytes.decodeToString(nameStart, nameStart + nameLength) + // Directories carry no data and end in a separator. + if (!name.endsWith("/")) { + entries[name] = Entry(localHeaderOffset, compressedSize, method) + } + offset = nameStart + nameLength + extraLength + commentLength + } + + return ZipArchive(bytes, entries) + } + + /** The end record sits last, unless a comment follows it, so the search runs backwards. */ + private fun findEndOfDirectory(bytes: ByteArray): Int? { + if (bytes.size < END_OF_DIRECTORY_MINIMUM) return null + val earliest = maxOf(0, bytes.size - END_OF_DIRECTORY_MINIMUM - MAX_COMMENT) + for (offset in bytes.size - END_OF_DIRECTORY_MINIMUM downTo earliest) { + if (bytes.readInt(offset) == END_OF_DIRECTORY_SIGNATURE) return offset + } + return null + } + + private fun ByteArray.readShort(at: Int): Int = + (this[at].toInt() and 0xFF) or ((this[at + 1].toInt() and 0xFF) shl 8) + + private fun ByteArray.readInt(at: Int): Int = + (this[at].toInt() and 0xFF) or + ((this[at + 1].toInt() and 0xFF) shl 8) or + ((this[at + 2].toInt() and 0xFF) shl 16) or + ((this[at + 3].toInt() and 0xFF) shl 24) + } +} diff --git a/library/src/commonTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.kt b/library/src/commonTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.kt index 076e5aa..5473077 100644 --- a/library/src/commonTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.kt +++ b/library/src/commonTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.kt @@ -2,6 +2,7 @@ package io.github.lemcoder.mikromarkdown import com.goncalossilva.resources.Resource import kotlin.test.Test +import kotlin.test.assertFailsWith import kotlin.test.assertFalse import kotlin.test.assertTrue @@ -27,45 +28,18 @@ class FileIntegrationTest { } } + /** + * The office formats were removed rather than ported: they are editing formats, and a reader meets PDF and EPUB. + * The fixtures stay for whenever an office plugin arrives, and this test pins the behaviour callers see until then. + */ @Test - fun testDocx() = - assertConversion( - filename = "test.docx", - mustInclude = - listOf( - "314b0a30-5b04-470b-b9f7-eed2c2bec74a", - "49e168b7-d2ae-407f-a055-2167576f39a1", - "# Abstract", - "# Introduction", - "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation", - ), - ) - - @Test - fun testXlsx() = - assertConversion( - filename = "test.xlsx", - mustInclude = - listOf( - "09060124-b5e7-4717-9d07-3c046eb", - "6ff4173b-42a5-4784-9b19-f49caff4d93d", - "affc7dad-52dc-4b98-9b5d-51e65d8a8ad0", - ), - ) - - @Test - fun testPptx() = - assertConversion( - filename = "test.pptx", - mustInclude = - listOf( - "2cdda5c8-e50e-4db4-b5f0-9722a649f455", - "04191ea8-5c73-4215-a1d3-1cfb43aaaf12", - "44bf7d06-5e7a-4a40-a2e1-a2e42ef28c8a", - "1b92870d-e3b5-4e65-8153-919f4ff45592", - "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation", - ), - ) + fun officeFormatsAreNotSupported() { + for (filename in listOf("test.docx", "test.xlsx", "test.pptx")) { + val bytes = Resource("test_files/$filename").readBytes() + val info = StreamInfo(extension = filename.substringAfterLast(".")) + assertFailsWith(filename) { mid.convert(bytes, info) } + } + } @Test fun testBlogHtml() = diff --git a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt index c13dc2d..9695eb7 100644 --- a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt +++ b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt @@ -1,19 +1,20 @@ package io.github.lemcoder.mikromarkdown import io.github.lemcoder.mikromarkdown.converters.CsvConverter -import io.github.lemcoder.mikromarkdown.converters.DocxConverter import io.github.lemcoder.mikromarkdown.converters.EpubConverter import io.github.lemcoder.mikromarkdown.converters.HtmlConverter import io.github.lemcoder.mikromarkdown.converters.JsonConverter import io.github.lemcoder.mikromarkdown.converters.MarkdownPassthroughConverter -import io.github.lemcoder.mikromarkdown.converters.PdfConverter import io.github.lemcoder.mikromarkdown.converters.PlainTextConverter -import io.github.lemcoder.mikromarkdown.converters.PptxConverter -import io.github.lemcoder.mikromarkdown.converters.XlsxConverter import io.github.lemcoder.mikromarkdown.converters.XmlConverter import java.io.File -/** A [MikroMarkdown] with every JVM converter registered and Tika-based format detection. */ +/** + * A [MikroMarkdown] with every JVM converter registered. + * + * PDF is not among them: it needs a native library, so the `:pdfium` module provides it and the caller opts in with + * `register(PdfiumConverter())`. + */ public fun MikroMarkdown(): MikroMarkdown = MikroMarkdown(SignatureMimeDetector).apply { register(MarkdownPassthroughConverter()) @@ -21,11 +22,7 @@ public fun MikroMarkdown(): MikroMarkdown = register(CsvConverter()) register(JsonConverter()) register(XmlConverter()) - register(DocxConverter()) - register(XlsxConverter()) - register(PptxConverter()) register(EpubConverter()) - register(PdfConverter()) register(PlainTextConverter(), priority = 10.0) } diff --git a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt deleted file mode 100644 index 58a536d..0000000 --- a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt +++ /dev/null @@ -1,32 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.utils.plainTextBlocks -import org.apache.pdfbox.Loader -import org.apache.pdfbox.text.PDFTextStripper - -public class PdfConverter : DocumentConverter { - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "pdf" || info.mimetype == "application/pdf" - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val doc = Loader.loadPDF(bytes) - try { - val title = doc.documentInformation?.title?.trim()?.ifBlank { null } - // Paragraph markers let the text blocks split on real paragraph breaks - // instead of collapsing a page into one block. - val stripper = - PDFTextStripper().apply { - sortByPosition = true - setAddMoreFormatting(true) - paragraphStart = "\n" - } - return Document(blocks = plainTextBlocks(stripper.getText(doc)), title = title) - } finally { - doc.close() - } - } -} diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/DocxConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/DocxConverter.kt deleted file mode 100644 index 596863f..0000000 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/DocxConverter.kt +++ /dev/null @@ -1,164 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Asset -import io.github.lemcoder.mikromarkdown.model.Block -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.model.Heading -import io.github.lemcoder.mikromarkdown.model.Image -import io.github.lemcoder.mikromarkdown.model.Inline -import io.github.lemcoder.mikromarkdown.model.ListBlock -import io.github.lemcoder.mikromarkdown.model.ListItem -import io.github.lemcoder.mikromarkdown.model.Paragraph -import io.github.lemcoder.mikromarkdown.model.Table -import io.github.lemcoder.mikromarkdown.model.TableCell -import io.github.lemcoder.mikromarkdown.model.Text -import io.github.lemcoder.mikromarkdown.model.plainText -import io.github.lemcoder.mikromarkdown.model.styled -import java.util.Base64 -import org.apache.poi.xwpf.usermodel.XWPFDocument -import org.apache.poi.xwpf.usermodel.XWPFParagraph -import org.apache.poi.xwpf.usermodel.XWPFTable - -public class DocxConverter : DocumentConverter { - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "docx" || - info.mimetype == "application/vnd.openxmlformats-officedocument.wordprocessingml.document" - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val docx = XWPFDocument(bytes.inputStream()) - try { - val blocks = mutableListOf() - val assets = mutableListOf() - var title: String? = docx.properties?.coreProperties?.title?.trim()?.ifBlank { null } - val pendingListItems = mutableListOf>>() - - fun flushList() { - if (pendingListItems.isEmpty()) return - blocks += buildNestedList(pendingListItems) - pendingListItems.clear() - } - - for (element in docx.bodyElements) { - when (element) { - is XWPFParagraph -> { - val content = paragraphInlines(element, assets) - if (content.plainText().isBlank() && content.none { it is Image }) continue - - val level = headingLevel(element.styleID) - when { - level > 0 -> { - flushList() - if (title == null) title = content.plainText().trim() - blocks += Heading(level, content) - } - - element.numID != null -> - pendingListItems += (element.numIlvl?.toInt() ?: 0).coerceAtLeast(0) to content - - else -> { - flushList() - blocks += Paragraph(content) - } - } - } - - is XWPFTable -> { - flushList() - table(element)?.let { blocks += it } - } - } - } - flushList() - - return Document(blocks = blocks, title = title, assets = assets) - } finally { - docx.close() - } - } - - private fun paragraphInlines(para: XWPFParagraph, assets: MutableList): List { - val out = mutableListOf() - for (run in para.runs) { - val pictures = run.embeddedPictures - if (pictures.isNotEmpty()) { - for (picture in pictures) { - val data = picture.pictureData - val alt = picture.description.orEmpty() - if (data == null) { - if (alt.isNotBlank()) out += Text(alt) - continue - } - val mime = data.pictureTypeEnum.contentType - val id = data.fileName ?: "image-${assets.size + 1}" - assets += Asset(id = id, mediaType = mime, bytes = data.data, name = data.fileName) - out += - Image( - alt = alt, - url = "data:$mime;base64,${Base64.getEncoder().encodeToString(data.data)}", - assetId = id, - ) - } - continue - } - - val text = run.text() ?: continue - if (text.isEmpty()) continue - if (text.isBlank()) { - out += Text(text) - continue - } - out += styled(listOf(Text(text)), bold = run.isBold, italic = run.isItalic, strike = run.isStrikeThrough) - } - return out - } - - /** Rebuilds Word's flat numbering levels into nested list blocks. */ - private fun buildNestedList(items: List>>): ListBlock { - var index = 0 - - fun build(level: Int): List { - val result = mutableListOf() - while (index < items.size) { - val (itemLevel, content) = items[index] - when { - itemLevel < level -> break - itemLevel == level -> { - index++ - val children = - if (index < items.size && items[index].first > level) { - listOf(ListBlock(ordered = false, items = build(items[index].first))) - } else { - emptyList() - } - result += ListItem(listOf(Paragraph(content)) + children) - } - // A deeper first item without a parent: promote it to this level. - else -> result += ListItem(listOf(ListBlock(ordered = false, items = build(itemLevel)))) - } - } - return result - } - - return ListBlock(ordered = false, items = build(items.minOf { it.first })) - } - - private fun table(table: XWPFTable): Table? { - val rows = table.rows - if (rows.isEmpty()) return null - val header = rows[0].tableCells.map { TableCell(it.text.trim()) } - val body = rows.drop(1).map { row -> row.tableCells.map { TableCell(it.text.trim()) } } - return Table(header = header, rows = body) - } - - private fun headingLevel(style: String?): Int { - val s = style?.replace("\\s+".toRegex(), "") ?: return 0 - if (s.startsWith("Heading", ignoreCase = true)) { - return s.drop(7).toIntOrNull()?.coerceIn(1, 6) ?: 0 - } - // OOXML numeric style IDs 1-6 map directly to heading levels - return s.toIntOrNull()?.takeIf { it in 1..6 } ?: 0 - } -} diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt deleted file mode 100644 index b700606..0000000 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt +++ /dev/null @@ -1,132 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Block -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.model.Paragraph -import io.github.lemcoder.mikromarkdown.model.Strong -import io.github.lemcoder.mikromarkdown.model.Text -import io.github.lemcoder.mikromarkdown.utils.HtmlToDocument -import java.io.StringReader -import java.util.zip.ZipInputStream -import javax.xml.parsers.DocumentBuilderFactory -import org.w3c.dom.Element -import org.xml.sax.InputSource - -public class EpubConverter : DocumentConverter { - private val metaFields = - listOf( - "title" to "Title", - "creator" to "Authors", - "language" to "Language", - "description" to "Description", - "identifier" to "Identifier", - ) - - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "epub" || info.mimetype == "application/epub+zip" - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val entries = readZip(bytes) - - val containerXml = entries["META-INF/container.xml"] ?: return Document() - val opfPath = parseOpfPath(containerXml) ?: return Document() - val opfBytes = entries[opfPath] ?: entries[opfPath.removePrefix("/")] ?: return Document() - val opfDir = opfPath.substringBeforeLast("/", "") - - val (manifest, spine, metadata) = parseOpf(opfBytes) - - val blocks = mutableListOf() - var title: String? = metadata["title"] - - for ((key, label) in metaFields) { - val value = metadata[key] ?: continue - blocks += Paragraph(listOf(Strong(listOf(Text("$label:"))), Text(" $value"))) - } - - for (idref in spine) { - val href = manifest[idref] ?: continue - val fullPath = if (opfDir.isEmpty()) href else "$opfDir/$href" - val htmlBytes = entries[fullPath] ?: entries[fullPath.removePrefix("/")] ?: continue - val chapter = HtmlToDocument.parse(htmlBytes.toString(Charsets.UTF_8)) - if (title == null) title = chapter.title - blocks += chapter.blocks - } - - return Document(blocks = blocks, title = title, metadata = metadata) - } - - private fun readZip(bytes: ByteArray): Map { - val entries = mutableMapOf() - ZipInputStream(bytes.inputStream()).use { zip -> - var entry = zip.nextEntry - while (entry != null) { - if (!entry.isDirectory) { - entries[entry.name] = zip.readBytes() - } - zip.closeEntry() - entry = zip.nextEntry - } - } - return entries - } - - private fun parseOpfPath(containerXml: ByteArray): String? { - val doc = parseXml(containerXml) ?: return null - val rootfiles = doc.getElementsByTagName("rootfile") - if (rootfiles.length == 0) return null - return (rootfiles.item(0) as? Element)?.getAttribute("full-path") - } - - private fun parseOpf(opfBytes: ByteArray): Triple, List, Map> { - val doc = parseXml(opfBytes) ?: return Triple(emptyMap(), emptyList(), emptyMap()) - - val metadata = mutableMapOf() - for (tag in listOf("dc:title", "dc:creator", "dc:language", "dc:description", "dc:identifier")) { - val nodes = doc.getElementsByTagName(tag) - if (nodes.length > 0) { - val text = nodes.item(0).textContent?.trim() - if (!text.isNullOrEmpty()) { - metadata[tag.removePrefix("dc:")] = text - } - } - } - - val manifest = mutableMapOf() - val manifestItems = doc.getElementsByTagName("item") - for (i in 0 until manifestItems.length) { - val item = manifestItems.item(i) as? Element ?: continue - val id = item.getAttribute("id") - val href = item.getAttribute("href") - val mediaType = item.getAttribute("media-type") - if (id.isNotEmpty() && href.isNotEmpty() && isReadableChapter(mediaType)) { - manifest[id] = href - } - } - - val spine = mutableListOf() - val itemrefs = doc.getElementsByTagName("itemref") - for (i in 0 until itemrefs.length) { - val itemref = itemrefs.item(i) as? Element ?: continue - val idref = itemref.getAttribute("idref") - if (idref.isNotEmpty()) spine.add(idref) - } - - return Triple(manifest, spine, metadata) - } - - /** Only the spine's (X)HTML documents carry text; images and styles are skipped. */ - private fun isReadableChapter(mediaType: String): Boolean = mediaType.contains("html") - - private fun parseXml(bytes: ByteArray) = - try { - val factory = DocumentBuilderFactory.newInstance() - factory.isNamespaceAware = false - factory.isExpandEntityReferences = false - factory.newDocumentBuilder().parse(InputSource(StringReader(bytes.toString(Charsets.UTF_8)))) - } catch (_: Exception) { - null - } -} diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/PptxConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/PptxConverter.kt deleted file mode 100644 index 6fa57dc..0000000 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/PptxConverter.kt +++ /dev/null @@ -1,237 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Block -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.model.Heading -import io.github.lemcoder.mikromarkdown.model.HtmlComment -import io.github.lemcoder.mikromarkdown.model.Image -import io.github.lemcoder.mikromarkdown.model.ListBlock -import io.github.lemcoder.mikromarkdown.model.ListItem -import io.github.lemcoder.mikromarkdown.model.Paragraph -import io.github.lemcoder.mikromarkdown.model.Table -import io.github.lemcoder.mikromarkdown.model.TableCell -import io.github.lemcoder.mikromarkdown.model.Text -import org.apache.poi.sl.usermodel.Placeholder -import org.apache.poi.sl.usermodel.Shape -import org.apache.poi.xslf.usermodel.XMLSlideShow -import org.apache.poi.xslf.usermodel.XSLFChart -import org.apache.poi.xslf.usermodel.XSLFGraphicFrame -import org.apache.poi.xslf.usermodel.XSLFGroupShape -import org.apache.poi.xslf.usermodel.XSLFPictureShape -import org.apache.poi.xslf.usermodel.XSLFSimpleShape -import org.apache.poi.xslf.usermodel.XSLFTable -import org.apache.poi.xslf.usermodel.XSLFTextShape -import org.openxmlformats.schemas.drawingml.x2006.chart.CTAreaSer -import org.openxmlformats.schemas.drawingml.x2006.chart.CTAxDataSource -import org.openxmlformats.schemas.drawingml.x2006.chart.CTBarSer -import org.openxmlformats.schemas.drawingml.x2006.chart.CTLineSer -import org.openxmlformats.schemas.drawingml.x2006.chart.CTNumDataSource -import org.openxmlformats.schemas.drawingml.x2006.chart.CTPieSer -import org.openxmlformats.schemas.drawingml.x2006.chart.CTScatterSer -import org.openxmlformats.schemas.drawingml.x2006.chart.CTSerTx -import org.openxmlformats.schemas.presentationml.x2006.main.CTPicture - -public class PptxConverter : DocumentConverter { - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "pptx" || - info.mimetype == "application/vnd.openxmlformats-officedocument.presentationml.presentation" - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val slideShow = XMLSlideShow(bytes.inputStream()) - try { - val blocks = mutableListOf() - var title: String? = null - - for ((index, slide) in slideShow.slides.withIndex()) { - blocks += HtmlComment("Slide number: ${index + 1}") - blocks += shapeBlocks(slide.shapes) { if (index == 0 && title == null) title = it } - - val notes = - slide.notes - ?.shapes - ?.filterIsInstance() - ?.filter { (it as? XSLFSimpleShape)?.placeholder != Placeholder.SLIDE_IMAGE } - ?.joinToString("\n") { it.text } - ?.trim() - .orEmpty() - if (notes.isNotBlank()) { - blocks += Heading(3, listOf(Text("Notes:"))) - blocks += notes.lines().filter { it.isNotBlank() }.map { Paragraph(listOf(Text(it.trim()))) } - } - } - - return Document(blocks = blocks, title = title) - } finally { - slideShow.close() - } - } - - private fun shapeBlocks(shapes: Iterable>, onTitle: (String) -> Unit): List { - val blocks = mutableListOf() - for (shape in shapes) { - when { - shape is XSLFGroupShape -> blocks += shapeBlocks(shape.shapes, onTitle) - - shape is XSLFTextShape -> { - val text = shape.text.trim() - if (text.isBlank()) continue - - val placeholder = (shape as? XSLFSimpleShape)?.placeholder - if (placeholder == Placeholder.TITLE || placeholder == Placeholder.CENTERED_TITLE) { - blocks += Heading(1, listOf(Text(text))) - onTitle(text) - continue - } - - // Consecutive bullet paragraphs become one list; plain ones stay paragraphs. - val bullets = mutableListOf() - fun flushBullets() { - if (bullets.isEmpty()) return - blocks += ListBlock(ordered = false, items = bullets.toList()) - bullets.clear() - } - for (para in shape.textParagraphs) { - val paraText = para.text.trim() - if (paraText.isBlank()) continue - if (para.isBullet) { - bullets += ListItem(listOf(Paragraph(listOf(Text(paraText))))) - } else { - flushBullets() - blocks += Paragraph(listOf(Text(paraText))) - } - } - flushBullets() - } - - shape is XSLFPictureShape -> { - val description = (shape.xmlObject as? CTPicture)?.nvPicPr?.cNvPr?.descr.orEmpty() - val alt = description.ifBlank { shape.shapeName } - val filename = shape.shapeName.replace(Regex("\\W"), "") + ".jpg" - blocks += Paragraph(listOf(Image(alt, filename))) - } - - shape is XSLFGraphicFrame && shape.hasChart() -> blocks += chartBlocks(shape.chart) - - shape is XSLFTable -> table(shape)?.let { blocks += it } - } - } - return blocks - } - - private fun chartBlocks(chart: XSLFChart): List { - val blocks = mutableListOf() - blocks += Heading(3, listOf(Text(listOfNotNull("Chart", chartTitle(chart)).joinToString(": ")))) - - val series = - try { - seriesOf(chart) - } catch (_: Exception) { - blocks += Paragraph(listOf(Text("[unsupported chart]"))) - return blocks - } - if (series.isEmpty()) return blocks - - val rowCount = series.maxOf { it.categories.size } - blocks += - Table( - header = (listOf("Category") + series.map { it.name }).map { TableCell(it) }, - rows = - (0 until rowCount).map { row -> - val category = series.first().categories.getOrElse(row) { "" } - (listOf(category) + series.map { it.values.getOrElse(row) { "" } }).map { TableCell(it) } - }, - ) - return blocks - } - - private fun chartTitle(chart: XSLFChart): String? = - try { - val ctChart = chart.ctChart - if (!ctChart.isSetTitle) { - null - } else { - val tx = ctChart.title?.tx - when { - tx?.isSetRich == true -> - tx.rich.pList.flatMap { p -> p.rList.map { r -> r.t.orEmpty() } }.joinToString("") - tx?.isSetStrRef == true -> tx.strRef?.strCache?.ptList?.firstOrNull()?.v - else -> null - }?.ifBlank { null } - } - } catch (_: Exception) { - null - } - - private data class Series(val name: String, val categories: List, val values: List) - - private fun seriesOf(chart: XSLFChart): List { - val plot = chart.ctChart.plotArea - // bar/bar3D, line/line3D and area/area3D each share one generated series type. - val all = - plot.barChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.bar3DChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.lineChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.line3DChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.areaChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.area3DChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.scatterChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.pieChartList.flatMap { it.serList.map { s -> s.toSeries() } } - return all.filter { it.categories.isNotEmpty() || it.values.isNotEmpty() } - } - - private fun CTBarSer.toSeries() = - series(if (isSetTx) tx else null, if (isSetCat) cat else null, if (isSetVal) `val` else null) - - private fun CTLineSer.toSeries() = - series(if (isSetTx) tx else null, if (isSetCat) cat else null, if (isSetVal) `val` else null) - - private fun CTAreaSer.toSeries() = - series(if (isSetTx) tx else null, if (isSetCat) cat else null, if (isSetVal) `val` else null) - - private fun CTPieSer.toSeries() = - series(if (isSetTx) tx else null, if (isSetCat) cat else null, if (isSetVal) `val` else null) - - private fun CTScatterSer.toSeries() = - series(if (isSetTx) tx else null, if (isSetXVal) xVal else null, if (isSetYVal) yVal else null) - - private fun series(tx: CTSerTx?, categories: CTAxDataSource?, values: CTNumDataSource?) = - Series(seriesName(tx), categoryValues(categories), numericValues(values)) - - private fun categoryValues(cat: CTAxDataSource?): List = - when { - cat == null -> emptyList() - cat.isSetStrRef -> cat.strRef?.strCache?.ptList?.sortedBy { it.idx }?.map { it.v } ?: emptyList() - cat.isSetNumRef -> cat.numRef?.numCache?.ptList?.sortedBy { it.idx }?.map { it.v.orEmpty() } ?: emptyList() - cat.isSetNumLit -> cat.numLit?.ptList?.sortedBy { it.idx }?.map { it.v.orEmpty() } ?: emptyList() - cat.isSetStrLit -> cat.strLit?.ptList?.sortedBy { it.idx }?.map { it.v } ?: emptyList() - else -> emptyList() - } - - private fun numericValues(v: CTNumDataSource?): List = - when { - v == null -> emptyList() - v.isSetNumRef -> v.numRef?.numCache?.ptList?.sortedBy { it.idx }?.map { it.v.orEmpty() } ?: emptyList() - v.isSetNumLit -> v.numLit?.ptList?.sortedBy { it.idx }?.map { it.v.orEmpty() } ?: emptyList() - else -> emptyList() - } - - private fun seriesName(tx: CTSerTx?): String = - when { - tx == null -> "" - tx.isSetV -> tx.v - tx.isSetStrRef -> tx.strRef?.strCache?.ptList?.firstOrNull()?.v.orEmpty() - else -> "" - } - - private fun table(table: XSLFTable): Table? { - val rows = table.rows - if (rows.isEmpty()) return null - return Table( - header = rows[0].cells.map { TableCell(it.text.trim()) }, - rows = rows.drop(1).map { row -> row.cells.map { TableCell(it.text.trim()) } }, - ) - } -} diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XlsxConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XlsxConverter.kt deleted file mode 100644 index a6c0e76..0000000 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XlsxConverter.kt +++ /dev/null @@ -1,64 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Block -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.model.Heading -import io.github.lemcoder.mikromarkdown.model.Table -import io.github.lemcoder.mikromarkdown.model.TableCell -import io.github.lemcoder.mikromarkdown.model.Text -import kotlin.math.floor -import org.apache.poi.ss.usermodel.Cell -import org.apache.poi.ss.usermodel.CellType -import org.apache.poi.ss.usermodel.DataFormatter -import org.apache.poi.xssf.usermodel.XSSFWorkbook - -public class XlsxConverter : DocumentConverter { - // Constructing a converter must not load POI: accepts() only looks at the extension. - private val formatter by lazy { DataFormatter() } - - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "xlsx" || - info.mimetype == "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val workbook = XSSFWorkbook(bytes.inputStream()) - try { - val blocks = mutableListOf() - - for (sheet in workbook) { - val rows = sheet.toList() - if (rows.isEmpty()) continue - - val columns = rows.maxOf { it.lastCellNum.toInt().coerceAtLeast(0) } - if (columns == 0) continue - - blocks += Heading(2, listOf(Text(sheet.sheetName))) - blocks += - Table( - header = (0 until columns).map { TableCell(cellValue(rows[0].getCell(it))) }, - rows = - rows.drop(1).map { row -> (0 until columns).map { TableCell(cellValue(row.getCell(it))) } }, - ) - } - - return Document(blocks = blocks) - } finally { - workbook.close() - } - } - - private fun cellValue(cell: Cell?): String { - if (cell == null) return "" - return when (cell.cellType) { - CellType.NUMERIC -> { - val v = cell.numericCellValue - if (v == floor(v) && !v.isInfinite()) v.toLong().toString() else formatter.formatCellValue(cell) - } - CellType.BLANK -> "" - else -> formatter.formatCellValue(cell).trim() - } - } -} diff --git a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt index 0bba043..2f4093a 100644 --- a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt +++ b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt @@ -70,9 +70,17 @@ class ArchitectureTest { fun `helpers under utils stay internal`() { val utils = { name: String? -> name?.contains(".utils") == true } - // internal or private: anything but part of the published API. - scope.classes().filter { utils(it.packagee?.name) }.assertFalse { it.hasPublicOrDefaultModifier } - scope.objects().filter { utils(it.packagee?.name) }.assertFalse { it.hasPublicOrDefaultModifier } + // internal or private: anything but part of the published API. Only top-level declarations + // are checked — a companion inside an internal class is already unreachable, and demanding a + // modifier on it would be noise. + scope + .classes() + .filter { utils(it.packagee?.name) && it.isTopLevel } + .assertFalse { it.hasPublicOrDefaultModifier } + scope + .objects() + .filter { utils(it.packagee?.name) && it.isTopLevel } + .assertFalse { it.hasPublicOrDefaultModifier } scope .functions() .filter { utils(it.packagee?.name) && it.isTopLevel } diff --git a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/DumpOutputTest.kt b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/DumpOutputTest.kt deleted file mode 100644 index 84c9627..0000000 --- a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/DumpOutputTest.kt +++ /dev/null @@ -1,26 +0,0 @@ -package io.github.lemcoder.mikromarkdown - -import java.io.File -import org.junit.jupiter.api.Test - -class DumpOutputTest { - private val mid = MikroMarkdown() - - @Test - fun dumpAll() { - for (name in - listOf( - "test.docx", - "test.xlsx", - "test.pptx", - "test.epub", - "test.json", - "test_blog.html", - "test_wikipedia.html", - )) { - val url = javaClass.classLoader.getResource("test_files/$name") ?: continue - val output = mid.convert(File(url.toURI()).absolutePath).markdown - File("/tmp/kt_$name.md").writeText(output) - } - } -} diff --git a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/PythonComparisonTest.kt b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/PythonComparisonTest.kt index b744e55..e38d826 100644 --- a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/PythonComparisonTest.kt +++ b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/PythonComparisonTest.kt @@ -9,12 +9,6 @@ import org.junit.jupiter.api.Test class PythonComparisonTest { - @Test fun testDocx() = compare("test.docx") - - @Test fun testXlsx() = compare("test.xlsx") - - @Test fun testPptx() = compare("test.pptx") - @Test fun testEpub() = compare("test.epub") @Test fun testJson() = compare("test.json") diff --git a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoderTest.kt b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoderTest.kt new file mode 100644 index 0000000..de9b4a7 --- /dev/null +++ b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoderTest.kt @@ -0,0 +1,63 @@ +package io.github.lemcoder.mikromarkdown.model + +import java.io.ByteArrayInputStream +import javax.imageio.ImageIO +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertNotNull +import kotlin.test.assertNull + +/** + * The encoder is checked by decoding what it writes with an independent decoder — ImageIO — rather than by comparing + * bytes against a recording. A PNG we wrote and only we can read would pass a golden-file test and still be useless to + * a Compose or SwiftUI reader. + */ +class PngEncoderTest { + + @Test + fun `ImageIO reads back every pixel, alpha included`() { + val width = 7 + val height = 5 + val pixels = ByteArray(width * height * 4) + for (y in 0 until height) { + for (x in 0 until width) { + val at = (y * width + x) * 4 + pixels[at] = (x * 30).toByte() + pixels[at + 1] = (y * 50).toByte() + pixels[at + 2] = ((x + y) * 20).toByte() + pixels[at + 3] = (255 - x * 10).toByte() + } + } + + val png = assertNotNull(PngEncoder.encode(width, height, pixels)) + val image = assertNotNull(ImageIO.read(ByteArrayInputStream(png)), "ImageIO could not read it") + + assertEquals(width, image.width) + assertEquals(height, image.height) + for (y in 0 until height) { + for (x in 0 until width) { + val at = (y * width + x) * 4 + val argb = image.getRGB(x, y) + assertEquals(pixels[at].toInt() and 0xFF, (argb shr 16) and 0xFF, "red at $x,$y") + assertEquals(pixels[at + 1].toInt() and 0xFF, (argb shr 8) and 0xFF, "green at $x,$y") + assertEquals(pixels[at + 2].toInt() and 0xFF, argb and 0xFF, "blue at $x,$y") + assertEquals(pixels[at + 3].toInt() and 0xFF, (argb ushr 24) and 0xFF, "alpha at $x,$y") + } + } + } + + @Test + fun `the signature and chunk order are what a decoder expects`() { + val png = assertNotNull(PngEncoder.encode(1, 1, ByteArray(4))) + + assertEquals(listOf(137, 80, 78, 71, 13, 10, 26, 10), png.take(8).map { it.toInt() and 0xFF }) + val text = png.decodeToString() + assertEquals(listOf("IHDR", "IDAT", "IEND"), listOf("IHDR", "IDAT", "IEND").sortedBy { text.indexOf(it) }) + } + + @Test + fun `dimensions that disagree with the buffer are refused`() { + assertNull(PngEncoder.encode(2, 2, ByteArray(4)), "a short buffer must not produce a truncated image") + assertNull(PngEncoder.encode(0, 4, ByteArray(0))) + } +} diff --git a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt index be95a3b..0d9432b 100644 --- a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt +++ b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt @@ -1,6 +1,8 @@ package io.github.lemcoder.mikromarkdown import io.github.lemcoder.mikromarkdown.converters.CsvConverter +import io.github.lemcoder.mikromarkdown.converters.EpubConverter +import io.github.lemcoder.mikromarkdown.converters.HtmlConverter import io.github.lemcoder.mikromarkdown.converters.JsonConverter import io.github.lemcoder.mikromarkdown.converters.MarkdownPassthroughConverter import io.github.lemcoder.mikromarkdown.converters.PlainTextConverter @@ -9,12 +11,13 @@ import io.github.lemcoder.mikromarkdown.converters.XmlConverter /** * A [MikroMarkdown] with the converters that need no platform library. * - * The document formats (DOCX, XLSX, PPTX, EPUB, PDF, HTML) still depend on JVM libraries and are absent here; this - * target exists to measure what a native binary costs to start and run. + * PDF is the only format still missing here; everything else the library converts is shared. */ public fun MikroMarkdown(): MikroMarkdown = MikroMarkdown(SignatureMimeDetector).apply { register(MarkdownPassthroughConverter()) + register(HtmlConverter()) + register(EpubConverter()) register(CsvConverter()) register(JsonConverter()) register(XmlConverter()) diff --git a/pdfium/build.gradle.kts b/pdfium/build.gradle.kts new file mode 100644 index 0000000..25ad398 --- /dev/null +++ b/pdfium/build.gradle.kts @@ -0,0 +1,104 @@ +import io.github.lemcoder.interop.jvmInterops + +plugins { + alias(libs.plugins.kotlinMultiplatform) + alias(libs.plugins.android.kotlin.multiplatform.library) + alias(libs.plugins.konanplugin) +} + +/** + * PDF support, kept in its own module so it can be dropped or extracted whole. + * + * pdfium is a prebuilt binary rather than a source dependency; fetching and verifying it lives in + * pdfium-binaries.gradle.kts. + */ +apply(from = "pdfium-binaries.gradle.kts") + +@Suppress("UNCHECKED_CAST") val pdfiumRoot = extra["pdfiumRoot"] as Directory + +@Suppress("UNCHECKED_CAST") val pdfiumAbis = extra["pdfiumAbis"] as Map + +kotlin { + macosArm64 { + val platform = pdfiumRoot.dir("mac-arm64") + compilations.getByName("main").cinterops.create("pdfium") { + defFile("src/nativeInterop/cinterop/pdfium.def") + includeDirs(platform.dir("include")) + // The archive ships a dylib, so the binary carries an rpath to find it at run time. + extraOpts("-libraryPath", platform.dir("lib").asFile.absolutePath) + } + } + + androidLibrary { + namespace = "io.github.lemcoder.mikromarkdown.pdfium" + compileSdk = libs.versions.android.compileSdk.get().toInt() + minSdk = libs.versions.android.minSdk.get().toInt() + withHostTestBuilder {}.configure {} + } + + jvm() + + sourceSets { + commonMain.dependencies { implementation(project(":library")) } + jvmTest.dependencies { implementation(libs.kotlin.test) } + } +} + +// The bindings load the stub by name, and the stub finds pdfium through the rpath CMake gave it. +tasks.named("jvmTest") { + dependsOn("linkJvmInteropPdfium") + useJUnitPlatform() + systemProperty( + "java.library.path", + layout.buildDirectory.dir("jvmInterop/pdfium/lib").get().asFile.absolutePath, + ) +} + +/** + * The JVM and Android legs bind the same .def separately, one declaration each. + * + * They could share one, and deliberately do not: the two runtimes diverge over time — how a library is loaded, what a + * file path means — and a shared declaration turns the first difference into a restructure rather than an edit. + */ +kotlin.jvm().compilations["main"].jvmInterops { + create("pdfium") { + defFile(project.file("src/nativeInterop/cinterop/pdfium.def")) + includeDirs.from(pdfiumRoot.dir("mac-arm64/include")) + + externalNativeBuild { + cmake { + path.set(project.file("native/CMakeLists.txt")) + targets.add("pdfium-jni") + arguments.add("-DPDFium_DIR=${pdfiumRoot.dir("mac-arm64").asFile.absolutePath}") + } + } + } +} + +kotlin.targets.getByName("android").compilations.getByName("main").jvmInterops { + create("pdfiumAndroid") { + defFile(project.file("src/nativeInterop/cinterop/pdfium.def")) + includeDirs.from(pdfiumRoot.dir("android-arm64/include")) + + externalNativeBuild { + cmake { + path.set(project.file("native/CMakeLists.txt")) + targets.add("pdfium-jni") + + for ((abiName, platformName) in pdfiumAbis) { + abi(abiName) { + platform.set(libs.versions.android.minSdk.get().toInt()) + arguments.add("-DPDFium_DIR=${pdfiumRoot.dir(platformName).asFile.absolutePath}") + } + } + } + } + } +} + +// Every binding path needs the headers and the library unpacked first. +tasks.matching { it.name.startsWith("cinteropPdfium") }.configureEach { dependsOn("downloadPdfium") } + +tasks + .matching { it.name.startsWith("generateJvmInterop") || it.name.startsWith("cmakeConfigure") } + .configureEach { dependsOn("downloadPdfium") } diff --git a/pdfium/native/CMakeLists.txt b/pdfium/native/CMakeLists.txt new file mode 100644 index 0000000..1259d5f --- /dev/null +++ b/pdfium/native/CMakeLists.txt @@ -0,0 +1,29 @@ +cmake_minimum_required(VERSION 3.20) +project(pdfium_jni C) + +# Supplied by the Konan plugin: where it wrote the JNI stub, and what the bindings will load. +set(KONAN_JNI_STUB_DIR "" CACHE PATH "") +set(KONAN_JNI_LIB_NAME "" CACHE STRING "") +set(KONAN_JNI_INCLUDE_DIRS "" CACHE STRING "") + +# pdfium ships PDFiumConfig.cmake; PDFium_DIR points at the unpacked release. +# +# Cross-compiling for Android, the NDK toolchain confines find_library and find_path to the sysroot, +# so a prebuilt library unpacked elsewhere is invisible and the error names the package rather than +# the cause. pdfium is exactly that, so the search is widened for it. +if(ANDROID) + list(APPEND CMAKE_FIND_ROOT_PATH "${PDFium_DIR}") + set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY BOTH) + set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE BOTH) +endif() + +find_package(PDFium REQUIRED) + +file(GLOB JNI_SOURCES "${KONAN_JNI_STUB_DIR}/*.c") +add_library(pdfium-jni SHARED ${JNI_SOURCES}) +set_target_properties(pdfium-jni PROPERTIES OUTPUT_NAME "${KONAN_JNI_LIB_NAME}") +target_include_directories(pdfium-jni PRIVATE ${KONAN_JNI_INCLUDE_DIRS}) + +# Linking through CMake rather than konan's linker is the point: pdfium is C++, and its runtime +# comes along correctly this way. +target_link_libraries(pdfium-jni PRIVATE pdfium) diff --git a/pdfium/pdfium-binaries.gradle.kts b/pdfium/pdfium-binaries.gradle.kts new file mode 100644 index 0000000..47eaf55 --- /dev/null +++ b/pdfium/pdfium-binaries.gradle.kts @@ -0,0 +1,92 @@ +import java.net.URI +import java.security.MessageDigest +import javax.inject.Inject +import org.gradle.process.ExecOperations + +/** + * Fetches the prebuilt pdfium binaries. + * + * Applied by the module's build file rather than living in it: pinning a release, verifying it and unpacking it per + * platform is its own concern, and it is the part most likely to grow — one entry per platform we support. + * + * Exposes through `extra`: + * - `pdfiumRoot` the directory holding `/include` and `/lib` + * - `pdfiumAbis` Android ABI name to the platform name pdfium publishes + */ +val pdfiumRelease = "chromium/8009" + +val pdfiumArchives = + mapOf( + "mac-arm64" to "b1f2f17c7432a9942514dda5094ee9822c743bdfd07e7187725efbd34fde941f", + "android-arm64" to "eebf9df88c68a080efd379058651596c96043117dfdbe71ec0d03c953ae7e805", + "android-x64" to "d1068eca5710d77653d453fa7d922dbba8e97b9d1d5dc06ce02aa0d8d599c20a", + ) + +val pdfiumRoot: Directory = layout.buildDirectory.dir("pdfium").get() + +extra["pdfiumRoot"] = pdfiumRoot + +extra["pdfiumAbis"] = mapOf("arm64-v8a" to "android-arm64", "x86_64" to "android-x64") + +/** + * A task class rather than a `doLast` block, because the configuration cache cannot serialize the script object that a + * block in a script plugin captures the moment it calls `uri()`, `providers` or `logger`. Everything the action needs + * arrives as an input or an injected service. + */ +abstract class DownloadPdfium : DefaultTask() { + + @get:Input abstract val release: Property + + /** platform name to the SHA-256 of its archive. */ + @get:Input abstract val archives: MapProperty + + @get:OutputDirectory abstract val root: DirectoryProperty + + @get:Inject abstract val exec: ExecOperations + + @TaskAction + fun download() { + val releaseTag = release.get() + for ((platform, sha256) in archives.get()) { + val target = root.get().dir(platform).asFile + if (target.resolve("lib").exists()) continue + + val archive = root.get().file("pdfium-$platform.tgz").asFile + archive.parentFile.mkdirs() + if (!archive.exists()) { + val url = + "https://github.com/bblanchon/pdfium-binaries/releases/download/" + + "$releaseTag/pdfium-$platform.tgz" + logger.lifecycle("downloading pdfium $releaseTag for $platform") + URI(url).toURL().openStream().use { input -> + archive.outputStream().use { output -> input.copyTo(output) } + } + } + + val digest = + MessageDigest.getInstance("SHA-256").digest(archive.readBytes()).joinToString("") { + (it.toInt() and 0xFF).toString(16).padStart(2, '0') + } + check(digest == sha256) { "pdfium-$platform.tgz checksum $digest, expected $sha256" } + + target.mkdirs() + exec.exec { commandLine("tar", "xzf", archive.absolutePath, "-C", target.absolutePath) } + + // The macOS dylib calls itself ./libpdfium.dylib, which the loader resolves against the + // working directory rather than the binary. Rewrite it to @rpath so anything linking it + // can find it. Mach-O only; the Android .so needs nothing. + val dylib = target.resolve("lib/libpdfium.dylib") + if (dylib.exists()) { + exec.exec { commandLine("install_name_tool", "-id", "@rpath/libpdfium.dylib", dylib.absolutePath) } + } + } + } +} + +tasks.register("downloadPdfium") { + description = "Downloads and unpacks the pinned pdfium binaries." + group = "build setup" + release.set(pdfiumRelease) + archives.set(pdfiumArchives) + root.set(pdfiumRoot) +} diff --git a/pdfium/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.android.kt b/pdfium/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.android.kt new file mode 100644 index 0000000..243f89a --- /dev/null +++ b/pdfium/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.android.kt @@ -0,0 +1,80 @@ +package io.github.lemcoder.mikromarkdown.pdf + +import pdfium.kniBridge1 +import pdfium.kniBridge15 +import pdfium.kniBridge16 +import pdfium.kniBridge2 +import pdfium.kniBridge26 +import pdfium.kniBridge27 +import pdfium.kniBridge5 +import pdfium.kniBridge52 +import pdfium.kniBridge53 +import pdfium.kniBridge54 +import pdfium.kniBridge71 + +/** + * The JVM half, over the JNI bridges the Konan plugin generates from the same `.def` cinterop binds. + * + * The bridges are numbered rather than named — that is what a runtime-free binding looks like — so each is wrapped here + * with the name from its doc comment, and nothing else in the module sees them. + */ +internal actual fun extractText(bytes: ByteArray): String { + val text = StringBuilder() + + initLibrary() + try { + val document = loadDocument(bytes, null) + if (document == 0L) return "" + try { + for (index in 0 until pageCount(document)) { + val page = loadPage(document, index) + if (page == 0L) continue + val textPage = loadTextPage(page) + if (textPage != 0L) { + text.append(pageText(textPage)) + text.append('\n') + closeTextPage(textPage) + } + closePage(page) + } + } finally { + closeDocument(document) + } + } finally { + destroyLibrary() + } + + return text.toString() +} + +/** pdfium writes UTF-16 into a caller-supplied buffer and counts the terminating NUL. */ +private fun pageText(textPage: Long): String { + val count = charCount(textPage) + if (count <= 0) return "" + val buffer = ShortArray(count + 1) + val written = readText(textPage, 0, count, buffer) + return if (written <= 1) "" else CharArray(written - 1) { Char(buffer[it].toInt() and 0xFFFF) }.concatToString() +} + +private fun initLibrary() = kniBridge1() + +private fun destroyLibrary() = kniBridge2() + +private fun loadDocument(bytes: ByteArray, password: String?): Long = kniBridge5(bytes, bytes.size, password) + +private fun pageCount(document: Long): Int = kniBridge15(document) + +private fun loadPage(document: Long, index: Int): Long = kniBridge16(document, index) + +private fun closePage(page: Long) = kniBridge26(page) + +private fun closeDocument(document: Long) = kniBridge27(document) + +private fun loadTextPage(page: Long): Long = kniBridge52(page) + +private fun closeTextPage(textPage: Long) = kniBridge53(textPage) + +private fun charCount(textPage: Long): Int = kniBridge54(textPage) + +private fun readText(textPage: Long, start: Int, count: Int, buffer: ShortArray): Int = + kniBridge71(textPage, start, count, buffer) diff --git a/pdfium/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt b/pdfium/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt new file mode 100644 index 0000000..273d4e5 --- /dev/null +++ b/pdfium/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt @@ -0,0 +1,73 @@ +package io.github.lemcoder.mikromarkdown.pdf + +import io.github.lemcoder.mikromarkdown.DocumentConverter +import io.github.lemcoder.mikromarkdown.StreamInfo +import io.github.lemcoder.mikromarkdown.model.Document +import io.github.lemcoder.mikromarkdown.model.plainTextBlocks + +/** + * PDF text through pdfium. + * + * Not registered by the library's factory: PDF costs a native library, so a caller asks for it. + * + * ``` + * val mikroMarkdown = MikroMarkdown().apply { register(PdfiumConverter()) } + * ``` + * + * The extraction itself is per-platform — cinterop on native, generated JNI bridges on the JVM — but both reach the + * same pdfium, and everything above [extractText] is shared. + */ +public class PdfiumConverter : DocumentConverter { + + override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { + return info.extension == "pdf" || info.mimetype == "application/pdf" + } + + override fun parse(bytes: ByteArray, info: StreamInfo): Document = + Document(blocks = plainTextBlocks(extractText(bytes).joinHyphenatedWords())) + + /** + * pdfium emits U+FFFE where a glyph has no Unicode mapping, which in a typeset document is nearly always the hyphen + * at a line break. + * + * Dropping it always would fuse real compounds — "chat-optimized" became "chatoptimized" — so the document decides: + * if both halves appear elsewhere as words in their own right the hyphen was the author's and is restored, + * otherwise the halves are one broken word and are joined. The halves themselves are cut from the vocabulary first, + * since they are only in the text because the break put them there. + * + * This is a heuristic standing in for geometry. A hyphenation hyphen ends a line and a compound hyphen does not, + * which `FPDFText_GetCharBox` would answer outright. + */ + private fun String.joinHyphenatedWords(): String { + if (indexOf(UNMAPPED_GLYPH) < 0) return this + + val vocabulary = WORD.findAll(replace(HYPHEN_BREAK, " ")).map { it.value.lowercase() }.toSet() + val out = StringBuilder(length) + for (index in indices) { + val char = this[index] + if (char != UNMAPPED_GLYPH) { + out.append(char) + continue + } + var wordStart = out.length + while (wordStart > 0 && out[wordStart - 1].isLetter()) wordStart-- + val left = out.subSequence(wordStart, out.length).toString().lowercase() + val right = substring(index + 1).takeWhile { it.isLetter() }.lowercase() + if (isRealCompound(left, right, vocabulary)) out.append('-') + } + return out.toString() + } + + /** A hyphen the author wrote, rather than one the typesetter added at a line break. */ + private fun isRealCompound(left: String, right: String, vocabulary: Set): Boolean = + left.isNotEmpty() && right.isNotEmpty() && left in vocabulary && right in vocabulary + + private companion object { + const val UNMAPPED_GLYPH = '\uFFFE' + val WORD = Regex("[\\p{L}]{2,}") + val HYPHEN_BREAK = Regex("\\p{L}+\uFFFE\\p{L}+") + } +} + +/** Every page's text, concatenated, one page per line. */ +internal expect fun extractText(bytes: ByteArray): String diff --git a/pdfium/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt b/pdfium/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt new file mode 100644 index 0000000..243f89a --- /dev/null +++ b/pdfium/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt @@ -0,0 +1,80 @@ +package io.github.lemcoder.mikromarkdown.pdf + +import pdfium.kniBridge1 +import pdfium.kniBridge15 +import pdfium.kniBridge16 +import pdfium.kniBridge2 +import pdfium.kniBridge26 +import pdfium.kniBridge27 +import pdfium.kniBridge5 +import pdfium.kniBridge52 +import pdfium.kniBridge53 +import pdfium.kniBridge54 +import pdfium.kniBridge71 + +/** + * The JVM half, over the JNI bridges the Konan plugin generates from the same `.def` cinterop binds. + * + * The bridges are numbered rather than named — that is what a runtime-free binding looks like — so each is wrapped here + * with the name from its doc comment, and nothing else in the module sees them. + */ +internal actual fun extractText(bytes: ByteArray): String { + val text = StringBuilder() + + initLibrary() + try { + val document = loadDocument(bytes, null) + if (document == 0L) return "" + try { + for (index in 0 until pageCount(document)) { + val page = loadPage(document, index) + if (page == 0L) continue + val textPage = loadTextPage(page) + if (textPage != 0L) { + text.append(pageText(textPage)) + text.append('\n') + closeTextPage(textPage) + } + closePage(page) + } + } finally { + closeDocument(document) + } + } finally { + destroyLibrary() + } + + return text.toString() +} + +/** pdfium writes UTF-16 into a caller-supplied buffer and counts the terminating NUL. */ +private fun pageText(textPage: Long): String { + val count = charCount(textPage) + if (count <= 0) return "" + val buffer = ShortArray(count + 1) + val written = readText(textPage, 0, count, buffer) + return if (written <= 1) "" else CharArray(written - 1) { Char(buffer[it].toInt() and 0xFFFF) }.concatToString() +} + +private fun initLibrary() = kniBridge1() + +private fun destroyLibrary() = kniBridge2() + +private fun loadDocument(bytes: ByteArray, password: String?): Long = kniBridge5(bytes, bytes.size, password) + +private fun pageCount(document: Long): Int = kniBridge15(document) + +private fun loadPage(document: Long, index: Int): Long = kniBridge16(document, index) + +private fun closePage(page: Long) = kniBridge26(page) + +private fun closeDocument(document: Long) = kniBridge27(document) + +private fun loadTextPage(page: Long): Long = kniBridge52(page) + +private fun closeTextPage(textPage: Long) = kniBridge53(textPage) + +private fun charCount(textPage: Long): Int = kniBridge54(textPage) + +private fun readText(textPage: Long, start: Int, count: Int, buffer: ShortArray): Int = + kniBridge71(textPage, start, count, buffer) diff --git a/pdfium/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverterTest.kt b/pdfium/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverterTest.kt new file mode 100644 index 0000000..fa85c96 --- /dev/null +++ b/pdfium/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverterTest.kt @@ -0,0 +1,24 @@ +package io.github.lemcoder.mikromarkdown.pdf + +import io.github.lemcoder.mikromarkdown.StreamInfo +import java.io.File +import kotlin.test.Test +import kotlin.test.assertContains +import kotlin.test.assertTrue + +class PdfiumConverterTest { + + private val fixture = File("../library/src/commonTest/resources/test_files/test.pdf") + + @Test + fun `extracts text through the generated JNI bridges`() { + val document = PdfiumConverter().parse(fixture.readBytes(), StreamInfo(extension = "pdf")) + val text = document.blocks.joinToString("\n") { it.toString() } + + assertTrue(text.length > 1000, "expected a page of text, got ${text.length} characters") + assertContains(text, "Introduction") + // The de-hyphenation ran: pdfium reports a broken word with U+FFFE between the halves. + assertContains(text, "confirming") + assertTrue('￾' !in text, "unmapped glyphs should not reach the model") + } +} diff --git a/pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.macos.kt b/pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.macos.kt new file mode 100644 index 0000000..655e0f1 --- /dev/null +++ b/pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.macos.kt @@ -0,0 +1,63 @@ +package io.github.lemcoder.mikromarkdown.pdf + +import kotlinx.cinterop.ExperimentalForeignApi +import kotlinx.cinterop.UShortVar +import kotlinx.cinterop.addressOf +import kotlinx.cinterop.allocArray +import kotlinx.cinterop.get +import kotlinx.cinterop.memScoped +import kotlinx.cinterop.usePinned +import pdfium.FPDFText_ClosePage +import pdfium.FPDFText_CountChars +import pdfium.FPDFText_GetText +import pdfium.FPDFText_LoadPage +import pdfium.FPDF_CloseDocument +import pdfium.FPDF_ClosePage +import pdfium.FPDF_DestroyLibrary +import pdfium.FPDF_GetPageCount +import pdfium.FPDF_InitLibrary +import pdfium.FPDF_LoadMemDocument +import pdfium.FPDF_LoadPage + +/** The native half, over the cinterop bindings. */ +@OptIn(ExperimentalForeignApi::class) +internal actual fun extractText(bytes: ByteArray): String { + val text = StringBuilder() + + FPDF_InitLibrary() + try { + bytes.usePinned { pinned -> + val document = FPDF_LoadMemDocument(pinned.addressOf(0), bytes.size, null) ?: return@usePinned + try { + for (index in 0 until FPDF_GetPageCount(document)) { + val page = FPDF_LoadPage(document, index) ?: continue + val textPage = FPDFText_LoadPage(page) + if (textPage != null) { + text.append(pageText(textPage)) + text.append('\n') + FPDFText_ClosePage(textPage) + } + FPDF_ClosePage(page) + } + } finally { + FPDF_CloseDocument(document) + } + } + } finally { + FPDF_DestroyLibrary() + } + + return text.toString() +} + +/** pdfium writes UTF-16 into a caller-supplied buffer and counts the terminating NUL. */ +@OptIn(ExperimentalForeignApi::class) +private fun pageText(textPage: pdfium.FPDF_TEXTPAGE): String { + val count = FPDFText_CountChars(textPage) + if (count <= 0) return "" + return memScoped { + val buffer = allocArray(count + 1) + val written = FPDFText_GetText(textPage, 0, count, buffer) + if (written <= 1) "" else CharArray(written - 1) { Char(buffer[it].toInt()) }.concatToString() + } +} diff --git a/pdfium/src/nativeInterop/cinterop/pdfium.def b/pdfium/src/nativeInterop/cinterop/pdfium.def new file mode 100644 index 0000000..89d16f8 --- /dev/null +++ b/pdfium/src/nativeInterop/cinterop/pdfium.def @@ -0,0 +1,4 @@ +# Bound by cinterop for native targets and, later, by the Konan plugin's JNI generator for the JVM. +headers = fpdfview.h fpdf_text.h +headerFilter = fpdf*.h +package = pdfium diff --git a/scripts/__pycache__/benchmark.cpython-314.pyc b/scripts/__pycache__/benchmark.cpython-314.pyc new file mode 100644 index 0000000..9eebc1b Binary files /dev/null and b/scripts/__pycache__/benchmark.cpython-314.pyc differ diff --git a/scripts/benchmark.py b/scripts/benchmark.py index b3b5729..35b6406 100644 --- a/scripts/benchmark.py +++ b/scripts/benchmark.py @@ -2,11 +2,10 @@ """Compare MikroMarkdown's Markdown against markitdown (Python) and anydoc (Rust). Usage: - ./gradlew :cli:installDist + ./gradlew :cli-native:linkReleaseExecutableMacosArm64 python3 scripts/benchmark.py [--fixtures DIR] [--out DIR] Engines are skipped (not failed) when their CLI is unavailable: - mikromarkdown cli/build/install/cli/bin/cli native cli-native/build/bin/macosArm64/releaseExecutable/cli-native.kexe markitdown `markitdown` on PATH, else `uvx markitdown[all]` anydoc-rust third-party/anydoc/target/release/examples/convert (cargo build @@ -77,12 +76,6 @@ def run(self, path: Path) -> tuple[str | None, float]: def which_engines() -> list[Engine]: engines: list[Engine] = [] - cli = REPO / "cli/build/install/cli/bin/cli" - engines.append( - Engine("mikromarkdown", [str(cli)] if cli.exists() else None, - "" if cli.exists() else "run ./gradlew :cli:installDist") - ) - if shutil.which("markitdown"): markitdown = ["markitdown"] elif shutil.which("uvx"): @@ -97,8 +90,8 @@ def which_engines() -> list[Engine]: "native", [str(native)] if native.exists() else None, "" if native.exists() else "run ./gradlew :cli-native:linkReleaseExecutableMacosArm64", - # The Kotlin/Native target carries only the converters that need no JVM library. - unsupported={"docx", "xlsx", "pptx", "epub", "pdf", "html", "htm"}, + # Office formats were removed from the project entirely. + unsupported={"docx", "xlsx", "pptx"}, ) ) diff --git a/scripts/optbench.py b/scripts/optbench.py index 028d32d..6030d33 100644 --- a/scripts/optbench.py +++ b/scripts/optbench.py @@ -7,10 +7,11 @@ Absolute timings drift between sessions — the same binary measured 60 ms one hour and 74 ms the next — so a change is only ever compared against the champion, interleaved, in the same run. + Output is verified against the recorded baselines first: a change that alters what we produce is -reported as broken rather than as fast. +reported as broken rather than as fast. Those baselines were recorded when a JVM CLI still existed +and matched it byte for byte, so they remain the reference for what each format should produce. """ -import os import shutil import subprocess import sys @@ -18,29 +19,30 @@ from pathlib import Path REPO = Path(__file__).resolve().parent.parent -JVM = REPO / "cli/build/install/cli/bin/cli" NATIVE = REPO / "cli-native/build/bin/macosArm64/releaseExecutable/cli-native.kexe" CHAMPION = REPO / "build/perf/champion.kexe" -CHAMPION_JVM = REPO / "build/perf/champion-cli" FIXTURES = REPO / "library/src/commonTest/resources/test_files" BASELINES = REPO / "build/benchmark" PERF = REPO / "build/perf" -# Native only runs CSV; the JVM champion covers the document formats it cannot. -TIMED = [("580 KB", PERF / "medium.csv"), ("1.8 MB", PERF / "big.csv")] -TIMED_JVM = [ +TIMED = [ + ("580 KB", PERF / "medium.csv"), + ("1.8 MB", PERF / "big.csv"), ("wiki", FIXTURES / "test_wikipedia.html"), - ("docx", FIXTURES / "test.docx"), - ("pdf", FIXTURES / "test.pdf"), + ("epub", FIXTURES / "test.epub"), ("json", PERF / "big.json"), ] + +# PDF is deliberately not pinned: pdfium reads a document differently from the PDFBox that recorded +# the baselines, and that difference is documented rather than frozen. +UNVERIFIED = {"test.pdf"} + ROUNDS = 7 def build(): result = subprocess.run( - ["./gradlew", ":cli:installDist", ":cli-native:linkReleaseExecutableMacosArm64", - "--no-configuration-cache", "-q"], + ["./gradlew", ":cli-native:linkReleaseExecutableMacosArm64", "--no-configuration-cache", "-q"], capture_output=True, text=True, cwd=REPO, ) errors = [line for line in (result.stdout + result.stderr).splitlines() if line.startswith("e:")] @@ -52,37 +54,23 @@ def verify(): for baseline in sorted(BASELINES.glob("mikromarkdown_*.md")): name = baseline.name.replace("mikromarkdown_", "").removesuffix(".md") fixture = FIXTURES / name - if not fixture.exists(): + if not fixture.exists() or name in UNVERIFIED: continue - if subprocess.run([str(JVM), str(fixture)], capture_output=True, cwd=REPO).stdout != baseline.read_bytes(): - problems.append(f"jvm {name}") - for name in ("test.csv", "test.json", "test.xml"): - fixture = FIXTURES / name - jvm = subprocess.run([str(JVM), str(fixture)], capture_output=True, cwd=REPO).stdout - native = subprocess.run([str(NATIVE), str(fixture)], capture_output=True, cwd=REPO).stdout - if jvm != native: - problems.append(f"native != jvm {name}") - big = PERF / "big.csv" - if big.exists(): - jvm = subprocess.run([str(JVM), str(big)], capture_output=True, cwd=REPO).stdout - native = subprocess.run([str(NATIVE), str(big)], capture_output=True, cwd=REPO).stdout - if jvm != native: - problems.append("native != jvm big.csv") + produced = subprocess.run([str(NATIVE), str(fixture)], capture_output=True, cwd=REPO) + if produced.returncode != 0: + problems.append(f"{name} failed to convert") + elif produced.stdout != baseline.read_bytes(): + problems.append(name) return problems -def interleaved(path, champion=None, candidate=None): +def interleaved(path): """Alternate champion and candidate so drift hits both equally. Returns (champion, candidate).""" - champion = champion or CHAMPION - candidate = candidate or NATIVE - # A copied distribution cannot use its CDS archive: the archive records absolute classpaths, and - # the JVM drops it without a word. Both sides run without it so the comparison is of the code. - environment = dict(os.environ, MIKROMARKDOWN_NO_CDS="1") champion_times, candidate_times = [], [] for _ in range(ROUNDS): - for binary, times in ((champion, champion_times), (candidate, candidate_times)): + for binary, times in ((CHAMPION, champion_times), (NATIVE, candidate_times)): start = time.perf_counter() - subprocess.run([str(binary), str(path)], capture_output=True, cwd=REPO, env=environment) + subprocess.run([str(binary), str(path)], capture_output=True, cwd=REPO) times.append((time.perf_counter() - start) * 1000) return min(champion_times), min(candidate_times) @@ -91,10 +79,7 @@ def main(): if "--promote" in sys.argv: CHAMPION.parent.mkdir(parents=True, exist_ok=True) shutil.copy(NATIVE, CHAMPION) - if CHAMPION_JVM.exists(): - shutil.rmtree(CHAMPION_JVM) - shutil.copytree(JVM.parent.parent, CHAMPION_JVM) - print("champion updated (native binary and JVM distribution)") + print("champion updated") return 0 label = sys.argv[1] if len(sys.argv) > 1 else "unlabelled" @@ -121,15 +106,6 @@ def main(): champion, candidate = interleaved(path) delta = (candidate - champion) / champion * 100 parts.append(f"{name}: {champion:.0f} -> {candidate:.0f} ({delta:+.0f}%)") - - champion_cli = CHAMPION_JVM / "bin/cli" - if champion_cli.exists(): - for name, path in TIMED_JVM: - if not path.exists(): - continue - champion, candidate = interleaved(path, champion=champion_cli, candidate=JVM) - delta = (candidate - champion) / champion * 100 - parts.append(f"{name}: {champion:.0f} -> {candidate:.0f} ({delta:+.0f}%)") print(f"{label}: " + " | ".join(parts)) return 0 diff --git a/settings.gradle.kts b/settings.gradle.kts index ad6e964..e678536 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -17,8 +17,8 @@ rootProject.name = "mikromarkdown" include(":library") -include(":cli") - include(":benchmark") include(":cli-native") + +include(":pdfium")