From 5a7a3c647533725bae1d16afab42442a1f1a601b Mon Sep 17 00:00:00 2001 From: mikolaj Date: Fri, 14 Aug 2026 00:02:18 +0200 Subject: [PATCH 01/15] Plan moving the remaining converters into commonMain Sequenced so each phase ships on its own: infrastructure, HTML, EPUB, DOCX, XLSX, PPTX, with PDF deliberately excluded. Records the library choices (korlibs-compression for ZIP, xmlutil for XML, Ksoup for HTML, all published for macosArm64), the POI surface each converter actually uses, and the two places the plan is likely to hurt: Excel number formatting, and losing POI's tolerance of malformed files. Co-Authored-By: Claude Opus 5 (1M context) --- docs/common-converters-plan.md | 141 +++++++++++++++++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100644 docs/common-converters-plan.md diff --git a/docs/common-converters-plan.md b/docs/common-converters-plan.md new file mode 100644 index 0000000..8d42642 --- /dev/null +++ b/docs/common-converters-plan.md @@ -0,0 +1,141 @@ +# Plan: the remaining converters in commonMain + +Status: proposal. Branch `common-converters`. Nothing implemented yet. + +## Where things stand + +Five converters and one helper are still JVM-only, plus PDF on each platform: + +| file | lines | depends on | +|---|---|---| +| `HtmlToDocument.kt` | 334 | Jsoup | +| `PptxConverter.kt` | 237 | POI `XSLF*`, XMLBeans `CT*` | +| `DocxConverter.kt` | 164 | POI `XWPF*` | +| `EpubConverter.kt` | 132 | `java.util.zip`, `javax.xml`, Jsoup | +| `XlsxConverter.kt` | 64 | POI `XSSFWorkbook`, `DataFormatter` | +| `HtmlConverter.kt` | 15 | Jsoup, via the helper | +| `PdfConverter.kt` ×2 | 32 | PDFBox / pdfbox-android | + +Moving them is not a file move: `commonMain` cannot use POI, Jsoup, `java.util.zip` or `javax.xml`, +so each is a rewrite against the raw format. CSV, JSON and XML made this trip already and their +output stayed byte-identical, which is the bar here too. + +## Why bother + +Two payoffs, and they are worth separating because they carry different risk appetites. + +**iOS.** The document formats are the reason a SwiftUI reader cannot exist today. This work is the +blocker, and it is worth doing even if nothing gets faster. + +**A smaller, faster JVM build.** POI dominates the 66 MB distribution, and a DOCX takes 183 ms end +to end against 2.7 ms of actual conversion — most of the rest is loading POI. Dropping commons-csv, +Jackson and kotlinx-serialization already cut a JSON conversion from 2063 classes to 1214; POI is a +much larger slice. + +Against that: POI, Jsoup and PDFBox absorb an enormous amount of real-world malformation. Hand-written +parsers will be less forgiving, and the fixture corpus is ten files. **Expanding that corpus is part +of the work, not an afterthought.** + +## Building blocks needed first + +| need | choice | why | +|---|---|---| +| ZIP + inflate | `com.soywiz:korlibs-compression:6.0.0` | OOXML and EPUB are ZIP containers; no inflate in kotlinx-io or okio on native | +| XML parsing | `io.github.pdvrieze.xmlutil:core:0.91.1` | KMP pull parser; hand-rolling one is possible but entity and namespace handling is where such things go wrong | +| HTML parsing | `com.fleeksoft.ksoup:ksoup:0.2.6` | KMP port of Jsoup with a near-identical API, `macosarm64` published | + +All three are dependencies we would carry in `commonMain`, replacing heavier JVM-only ones. Worth +confirming each builds for `macosArm64` in a throwaway module before committing to the sequence. + +## Sequence + +Ordered so each phase produces something shippable and the risky ones come last. + +### Phase 0 — infrastructure + +Add the three dependencies to `commonMain`, confirm the native target still links, and extend the +verification harness so `scripts/optbench.py` checks native output for every fixture a phase makes +available, not just CSV/JSON/XML. Nothing user-visible. + +### Phase 1 — HTML, via Ksoup + +`HtmlToDocument` is written against a Jsoup-shaped API, so the port is mostly mechanical: `Jsoup.parse`, +`childNodes()`, `attr`, `selectFirst`. The risk is not the API, it is the parser: Wikipedia is messy +HTML and Ksoup may recover from it differently. Acceptance is byte-identical output on `test_blog.html` +and `test_wikipedia.html`, and if it is not, the diff decides whether the difference is defensible. + +Unlocks: HTML on native, and the EPUB rewrite. + +### Phase 2 — EPUB + +Needs ZIP (Phase 0) plus the XML parser for `container.xml` and the OPF, then reuses Phase 1 for the +chapters. The smallest of the container formats and a good first exercise of the ZIP reader. + +### Phase 3 — DOCX + +`word/document.xml`: `w:p`, `w:r`, `w:t`, `w:rPr` for bold/italic/strike, `w:pStyle` for heading +level, `w:numPr` for list level. Images resolve through `word/_rels/document.xml.rels` into +`word/media/`. We use three POI types today and a narrow slice of each, so the rewrite is bounded. + +### Phase 4 — XLSX + +`xl/workbook.xml` for sheet names, `xl/worksheets/sheetN.xml` for cells, `xl/sharedStrings.xml` for +text. **The catch is `DataFormatter`**: POI's implementation of Excel number formats runs to +thousands of lines, and we call it for every non-integer cell. Replicating it in general is out of +scope; the plan is to implement the common format codes and treat anything else as the raw value, +then check what the fixtures and a wider corpus actually exercise. This phase is the most likely to +change output, and the point where "byte-identical" may have to give way to "defensibly different". + +### Phase 5 — PPTX + +The largest: slides, shapes, group shapes, placeholders, pictures, tables, and charts across eight +chart types currently read through XMLBeans `CT*` classes. Chart XML is verbose but regular. Leave +it last because it is the most code for the least reach. + +### Phase 6 — PDF + +**Not a `commonMain` candidate.** There is no KMP PDF library, and text extraction with layout +analysis is a project in itself — anydoc wrote their own. Two options, to decide when the rest lands: + +1. keep `expect`/`actual` with PDFBox on JVM and Android, and no PDF on other targets; +2. cinterop to pdfium or mupdf for native targets. + +Option 1 is the honest default. Option 2 only pays if iOS PDF support is required. + +## Ground rules per phase + +1. The new implementation lands in `commonMain`; the `jvmShared` version is deleted in the same + commit. The Konsist duplicate-file rule enforces that nothing is left copied per target. +2. `scripts/optbench.py` must report every fixture byte-identical on both targets before timings are + read. Where a difference is deliberate, the fixture baseline is updated in the same commit, with + the diff quoted in the message. +3. `PythonComparisonTest` stays at 100% token recall against Python markitdown. +4. Timings are A/B against the champion binary, never absolute — the log records why. +5. Each phase adds fixtures that exercise what it implements: a DOCX with numbered and nested lists, + an XLSX with dates and currency, a PPTX with a chart. Ten fixtures is too few to rewrite parsers + against. + +## Estimate + +| phase | effort | risk | +|---|---|---| +| 0 infrastructure | half a day | low — or the whole plan changes if a library will not build | +| 1 HTML | 1 day | medium — parser differences on messy input | +| 2 EPUB | half a day | low | +| 3 DOCX | 2-3 days | medium | +| 4 XLSX | 2-4 days | **high** — number formatting | +| 5 PPTX | 3-5 days | medium, mostly volume | +| 6 PDF | — | decide later | + +Call it two weeks to get everything but PDF into `commonMain`, with XLSX the phase most likely to +need a scope conversation. + +## Worth deciding before starting + +- **Is byte-identical output a hard requirement, or a default that XLSX may negotiate?** The answer + changes how Phase 4 is approached. +- **Should the JVM keep POI as an option?** A `mikromarkdown-poi` artifact could keep POI-backed + converters for callers who value its tolerance of broken files over startup time. It costs an + artifact and a registration path, and it would make the switch reversible per format. +- **Which targets beyond macOS?** Adding `iosArm64` and `linuxX64` early keeps the code honest; + adding them late risks discovering a dependency that does not publish for them. From 00845a6d688b6235f8927c5acb4d7e416cdba880 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Mon, 17 Aug 2026 23:32:36 +0200 Subject: [PATCH 02/15] Revise the plan: PDF through pdfium, XLSX deferred MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit XLSX stays on POI. Excel's number-format language is thousands of lines in DataFormatter and we call it for every non-integer cell, so it is the phase most likely to change output silently. Consequence worth naming: POI stays on the JVM classpath until it moves, so the smaller-JVM payoff is deferred while the portability one still lands. PDF becomes a separate :pdfium module, bound with KonanPlugin so one .def serves cinterop and JNI. Binaries come from bblanchon/pdfium-binaries, downloaded from a pinned release rather than committed. The module registers nothing on its own — callers opt in with register(PdfiumConverter()) — so deleting it leaves the rest building. Two costs recorded: pdfium extracts text differently from PDFBox, so that fixture baseline changes and token recall replaces byte-identity as the gate; and JVM distribution needs the library plus a generated stub per platform, which is a CI matrix rather than something one machine can produce. Co-Authored-By: Claude Opus 5 (1M context) --- docs/common-converters-plan.md | 201 ++++++++++++++++++--------------- 1 file changed, 112 insertions(+), 89 deletions(-) diff --git a/docs/common-converters-plan.md b/docs/common-converters-plan.md index 8d42642..6679b42 100644 --- a/docs/common-converters-plan.md +++ b/docs/common-converters-plan.md @@ -6,136 +6,159 @@ Status: proposal. Branch `common-converters`. Nothing implemented yet. Five converters and one helper are still JVM-only, plus PDF on each platform: -| file | lines | depends on | -|---|---|---| -| `HtmlToDocument.kt` | 334 | Jsoup | -| `PptxConverter.kt` | 237 | POI `XSLF*`, XMLBeans `CT*` | -| `DocxConverter.kt` | 164 | POI `XWPF*` | -| `EpubConverter.kt` | 132 | `java.util.zip`, `javax.xml`, Jsoup | -| `XlsxConverter.kt` | 64 | POI `XSSFWorkbook`, `DataFormatter` | -| `HtmlConverter.kt` | 15 | Jsoup, via the helper | -| `PdfConverter.kt` ×2 | 32 | PDFBox / pdfbox-android | +| file | lines | depends on | plan | +|---|---|---|---| +| `HtmlToDocument.kt` | 334 | Jsoup | → commonMain (Ksoup) | +| `PptxConverter.kt` | 237 | POI `XSLF*`, XMLBeans `CT*` | → commonMain (raw XML) | +| `DocxConverter.kt` | 164 | POI `XWPF*` | → commonMain (raw XML) | +| `EpubConverter.kt` | 132 | `java.util.zip`, `javax.xml`, Jsoup | → commonMain | +| `HtmlConverter.kt` | 15 | Jsoup, via the helper | → commonMain | +| `XlsxConverter.kt` | 64 | POI `XSSFWorkbook`, `DataFormatter` | **stays on POI for now** | +| `PdfConverter.kt` ×2 | 32 | PDFBox / pdfbox-android | → a separate `:pdfium` module | Moving them is not a file move: `commonMain` cannot use POI, Jsoup, `java.util.zip` or `javax.xml`, so each is a rewrite against the raw format. CSV, JSON and XML made this trip already and their -output stayed byte-identical, which is the bar here too. +output stayed byte-identical, which is the bar for everything except PDF. ## Why bother -Two payoffs, and they are worth separating because they carry different risk appetites. - -**iOS.** The document formats are the reason a SwiftUI reader cannot exist today. This work is the -blocker, and it is worth doing even if nothing gets faster. +**iOS.** The document formats are the reason a SwiftUI reader cannot exist today. This is the blocker, +and it is worth doing even if nothing gets faster. -**A smaller, faster JVM build.** POI dominates the 66 MB distribution, and a DOCX takes 183 ms end -to end against 2.7 ms of actual conversion — most of the rest is loading POI. Dropping commons-csv, -Jackson and kotlinx-serialization already cut a JSON conversion from 2063 classes to 1214; POI is a -much larger slice. +**A smaller, faster JVM build** — but only partly, now. POI dominates the 66 MB distribution and most +of the 183 ms a DOCX takes end to end against 2.7 ms of actual conversion. Keeping XLSX on POI means +**POI stays on the JVM classpath**, so that payoff is deferred until XLSX moves too. The portability +payoff lands in full: native gets everything but XLSX. -Against that: POI, Jsoup and PDFBox absorb an enormous amount of real-world malformation. Hand-written -parsers will be less forgiving, and the fixture corpus is ten files. **Expanding that corpus is part -of the work, not an afterthought.** +Against all of it: POI, Jsoup and PDFBox absorb an enormous amount of real-world malformation. +Hand-written parsers will be less forgiving, and the fixture corpus is ten files. **Expanding it is +part of the work, not an afterthought.** -## Building blocks needed first +## Building blocks -| need | choice | why | +| need | choice | notes | |---|---|---| -| ZIP + inflate | `com.soywiz:korlibs-compression:6.0.0` | OOXML and EPUB are ZIP containers; no inflate in kotlinx-io or okio on native | -| XML parsing | `io.github.pdvrieze.xmlutil:core:0.91.1` | KMP pull parser; hand-rolling one is possible but entity and namespace handling is where such things go wrong | -| HTML parsing | `com.fleeksoft.ksoup:ksoup:0.2.6` | KMP port of Jsoup with a near-identical API, `macosarm64` published | - -All three are dependencies we would carry in `commonMain`, replacing heavier JVM-only ones. Worth -confirming each builds for `macosArm64` in a throwaway module before committing to the sequence. +| ZIP + inflate | `com.soywiz:korlibs-compression:6.0.0` | OOXML and EPUB are ZIP; no inflate in kotlinx-io or okio on native | +| XML parsing | `io.github.pdvrieze.xmlutil:core:0.91.1` | KMP pull parser; hand-rolling one is where entities and namespaces go wrong | +| HTML parsing | `com.fleeksoft.ksoup:ksoup:0.2.6` | KMP port of Jsoup, near-identical API, `macosarm64` published | +| PDF | pdfium via cinterop + JNI | see below | + +Confirm each links for `macosArm64` in a throwaway module before committing to the sequence. + +## PDF: a `:pdfium` module + +PDFBox has no KMP equivalent and text extraction is its own project, so PDF goes native through +[pdfium](https://pdfium.googlesource.com/pdfium/), bound with +[KonanPlugin](https://github.com/lemcoder/KonanPlugin) (`io.github.lemcoder.konanplugin:1.2.0-alpha05`, +on the plugin portal). The plugin generates JNI bindings from the *same* `.def` file cinterop binds, +so one declaration serves Kotlin/Native, JVM and Android. + +``` +pdfium/ + src/nativeInterop/cinterop/pdfium.def headers = fpdfview.h fpdf_text.h fpdf_doc.h + src/commonMain/…/PdfiumConverter.kt DocumentConverter over the shim + src/commonMain/…/Pdfium.kt expect: open, pageCount, pageText, close + src/macosArm64Main/…/Pdfium.kt actual over the cinterop klib + src/jvmMain/…/Pdfium.kt actual over the generated JNI bridges + build.gradle.kts konanplugin: cinterop + jvmInterops from one def +``` + +Binaries come from [`bblanchon/pdfium-binaries`](https://github.com/bblanchon/pdfium-binaries), which +publishes per-platform archives (`pdfium-mac-arm64.tgz`, `pdfium-linux-x64.tgz`, +`pdfium-android-arm64.tgz`, …) containing headers and a library. A Gradle task downloads and unpacks +a **pinned release with a checksum** into `build/pdfium//`; nothing binary is committed. + +The API needed is small: `FPDF_InitLibrary`, `FPDF_LoadMemDocument`, `FPDF_GetPageCount`, +`FPDF_LoadPage`, `FPDFText_LoadPage`, `FPDFText_CountChars`, `FPDFText_GetText` (UTF-16), +and the matching closes. The existing `plainTextBlocks` reflow and de-hyphenation sit on top unchanged. + +**The module does not register itself.** `:library` keeps no PDF dependency, and a caller opts in: + +```kotlin +val mikroMarkdown = MikroMarkdown().apply { register(PdfiumConverter()) } +``` + +That keeps `:pdfium` genuinely extractable — delete the module and the rest still builds. + +### What this costs + +- **PDF output will change.** pdfium and PDFBox extract text differently, so the `test.pdf` baseline + has to be re-recorded. Byte-identity cannot be the gate here; `PythonComparisonTest`'s token recall + against Python markitdown becomes the check, plus a read of the diff. +- **Distribution gets heavier and platform-shaped.** The JVM artifact needs the pdfium library and the + generated stub per platform, 4-8 MB each. Building stubs for every JVM host means a CI matrix — + from one machine we can only produce the host's. +- **Licensing**: pdfium is BSD-3-Clause with Apache-2.0 parts; the notices ship with the artifact. ## Sequence -Ordered so each phase produces something shippable and the risky ones come last. +Each phase is shippable on its own, risky ones last. ### Phase 0 — infrastructure - -Add the three dependencies to `commonMain`, confirm the native target still links, and extend the -verification harness so `scripts/optbench.py` checks native output for every fixture a phase makes -available, not just CSV/JSON/XML. Nothing user-visible. +Add the three commonMain dependencies, confirm the native target still links, and extend +`scripts/optbench.py` so it verifies native output for every fixture a phase unlocks. ### Phase 1 — HTML, via Ksoup - -`HtmlToDocument` is written against a Jsoup-shaped API, so the port is mostly mechanical: `Jsoup.parse`, -`childNodes()`, `attr`, `selectFirst`. The risk is not the API, it is the parser: Wikipedia is messy -HTML and Ksoup may recover from it differently. Acceptance is byte-identical output on `test_blog.html` -and `test_wikipedia.html`, and if it is not, the diff decides whether the difference is defensible. - -Unlocks: HTML on native, and the EPUB rewrite. +`HtmlToDocument` is written against a Jsoup-shaped API, so the port is mechanical. The risk is the +parser, not the API: Wikipedia is messy and Ksoup may recover differently. Acceptance is +byte-identical `test_blog.html` and `test_wikipedia.html`; if it is not, the diff decides whether the +difference is defensible. ### Phase 2 — EPUB +ZIP plus the XML parser for `container.xml` and the OPF, then Phase 1 for the chapters. Smallest +container format, and a good first exercise of the ZIP reader. -Needs ZIP (Phase 0) plus the XML parser for `container.xml` and the OPF, then reuses Phase 1 for the -chapters. The smallest of the container formats and a good first exercise of the ZIP reader. - -### Phase 3 — DOCX +### Phase 3 — PDF, the `:pdfium` module +Independent of the OOXML work, so it can run in parallel or first if iOS PDF matters more. -`word/document.xml`: `w:p`, `w:r`, `w:t`, `w:rPr` for bold/italic/strike, `w:pStyle` for heading -level, `w:numPr` for list level. Images resolve through `word/_rels/document.xml.rels` into -`word/media/`. We use three POI types today and a narrow slice of each, so the rewrite is bounded. - -### Phase 4 — XLSX - -`xl/workbook.xml` for sheet names, `xl/worksheets/sheetN.xml` for cells, `xl/sharedStrings.xml` for -text. **The catch is `DataFormatter`**: POI's implementation of Excel number formats runs to -thousands of lines, and we call it for every non-integer cell. Replicating it in general is out of -scope; the plan is to implement the common format codes and treat anything else as the raw value, -then check what the fixtures and a wider corpus actually exercise. This phase is the most likely to -change output, and the point where "byte-identical" may have to give way to "defensibly different". +### Phase 4 — DOCX +`word/document.xml`: `w:p`, `w:r`, `w:t`, `w:rPr` for bold/italic/strike, `w:pStyle` for heading level, +`w:numPr` for list level; images through `word/_rels/document.xml.rels` into `word/media/`. We use +three POI types and a narrow slice of each, so the rewrite is bounded. ### Phase 5 — PPTX - The largest: slides, shapes, group shapes, placeholders, pictures, tables, and charts across eight -chart types currently read through XMLBeans `CT*` classes. Chart XML is verbose but regular. Leave -it last because it is the most code for the least reach. - -### Phase 6 — PDF - -**Not a `commonMain` candidate.** There is no KMP PDF library, and text extraction with layout -analysis is a project in itself — anydoc wrote their own. Two options, to decide when the rest lands: - -1. keep `expect`/`actual` with PDFBox on JVM and Android, and no PDF on other targets; -2. cinterop to pdfium or mupdf for native targets. +chart types currently read through XMLBeans. Chart XML is verbose but regular. Last because it is the +most code for the least reach. -Option 1 is the honest default. Option 2 only pays if iOS PDF support is required. +### Not now — XLSX +Stays on POI. `DataFormatter` implements Excel's number-format language in thousands of lines and we +call it for every non-integer cell; reimplementing it is a project of its own and the phase most +likely to change output silently. Revisit once the rest has landed and the corpus is wider — and note +that until then the JVM build still carries POI. ## Ground rules per phase 1. The new implementation lands in `commonMain`; the `jvmShared` version is deleted in the same - commit. The Konsist duplicate-file rule enforces that nothing is left copied per target. + commit. The Konsist duplicate-file rule keeps anything from being copied per target. 2. `scripts/optbench.py` must report every fixture byte-identical on both targets before timings are - read. Where a difference is deliberate, the fixture baseline is updated in the same commit, with - the diff quoted in the message. -3. `PythonComparisonTest` stays at 100% token recall against Python markitdown. -4. Timings are A/B against the champion binary, never absolute — the log records why. -5. Each phase adds fixtures that exercise what it implements: a DOCX with numbered and nested lists, - an XLSX with dates and currency, a PPTX with a chart. Ten fixtures is too few to rewrite parsers - against. + read. Where a difference is deliberate — PDF, and possibly HTML — the baseline is updated in the + same commit with the diff quoted in the message. +3. `PythonComparisonTest` stays at 100% token recall. +4. Timings are A/B against the champion binary, never absolute. +5. Each phase adds fixtures for what it implements: a DOCX with numbered and nested lists, a PPTX with + a chart, a PDF with columns and hyphenation. Ten fixtures is too few to rewrite parsers against. ## Estimate | phase | effort | risk | |---|---|---| -| 0 infrastructure | half a day | low — or the whole plan changes if a library will not build | +| 0 infrastructure | half a day | low — or the plan changes, if a library will not build | | 1 HTML | 1 day | medium — parser differences on messy input | | 2 EPUB | half a day | low | -| 3 DOCX | 2-3 days | medium | -| 4 XLSX | 2-4 days | **high** — number formatting | +| 3 PDF via pdfium | 2-3 days | medium — binding is routine, packaging and output changes are not | +| 4 DOCX | 2-3 days | medium | | 5 PPTX | 3-5 days | medium, mostly volume | -| 6 PDF | — | decide later | -Call it two weeks to get everything but PDF into `commonMain`, with XLSX the phase most likely to -need a scope conversation. +Roughly a week and a half without XLSX. ## Worth deciding before starting -- **Is byte-identical output a hard requirement, or a default that XLSX may negotiate?** The answer - changes how Phase 4 is approached. -- **Should the JVM keep POI as an option?** A `mikromarkdown-poi` artifact could keep POI-backed - converters for callers who value its tolerance of broken files over startup time. It costs an - artifact and a registration path, and it would make the switch reversible per format. - **Which targets beyond macOS?** Adding `iosArm64` and `linuxX64` early keeps the code honest; - adding them late risks discovering a dependency that does not publish for them. + adding them late risks finding a dependency — or a pdfium archive — that does not fit. +- **Does the JVM keep POI-backed converters as an option?** A `mikromarkdown-poi` artifact would let + callers choose POI's tolerance of broken files over startup time, and would make each format's + switch reversible. It costs an artifact and a registration path. +- **How are pdfium binaries shipped to JVM users** — bundled per platform in the artifact, or + downloaded at build time by the consumer? The first is convenient and large; the second is small + and one more thing to go wrong. From 95600632fe0d86fa5f817e88d050dbb5d6e116aa Mon Sep 17 00:00:00 2001 From: mikolaj Date: Mon, 17 Aug 2026 23:43:00 +0200 Subject: [PATCH 03/15] Fold asset extraction into the plan, PDF images included MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Compose and SwiftUI readers need images, not just text, so the pdfium module walks page objects rather than only the text layer. Two cases: DCTDecode and JPXDecode streams are already JPEG files and pass straight through, while Flate-compressed raw pixels have no image file to extract and need encoding — so commonMain gains a small PNG writer over the deflate that korlibs-compression already provides. Placement matters too: object bounds and character boxes let images be emitted in reading order instead of dumped at the end of a page, which the PDFBox route never did. Also records what every other format owes: EPUB and HTML assets, real PPTX picture bytes instead of today's fabricated filename, and the three gaps the readers hit — no asset policy (DOCX inlines base64, which is why its output is 161 KB), asset ids that can collide, and no intrinsic size on Image. Co-Authored-By: Claude Opus 5 (1M context) --- docs/common-converters-plan.md | 71 +++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 6 deletions(-) diff --git a/docs/common-converters-plan.md b/docs/common-converters-plan.md index 6679b42..f6913a3 100644 --- a/docs/common-converters-plan.md +++ b/docs/common-converters-plan.md @@ -68,9 +68,39 @@ publishes per-platform archives (`pdfium-mac-arm64.tgz`, `pdfium-linux-x64.tgz`, `pdfium-android-arm64.tgz`, …) containing headers and a library. A Gradle task downloads and unpacks a **pinned release with a checksum** into `build/pdfium//`; nothing binary is committed. -The API needed is small: `FPDF_InitLibrary`, `FPDF_LoadMemDocument`, `FPDF_GetPageCount`, -`FPDF_LoadPage`, `FPDFText_LoadPage`, `FPDFText_CountChars`, `FPDFText_GetText` (UTF-16), -and the matching closes. The existing `plainTextBlocks` reflow and de-hyphenation sit on top unchanged. +Text needs `FPDF_InitLibrary`, `FPDF_LoadMemDocument`, `FPDF_GetPageCount`, `FPDF_LoadPage`, +`FPDFText_LoadPage`, `FPDFText_CountChars`, `FPDFText_GetText` (UTF-16), and the matching closes. +The existing `plainTextBlocks` reflow and de-hyphenation sit on top unchanged. + +**Images matter as much as text here**, because the Compose and SwiftUI readers need them, so the +module walks page objects rather than only the text layer: + +| step | call | +|---|---| +| iterate objects on a page | `FPDFPage_CountObjects`, `FPDFPage_GetObject` | +| keep the images | `FPDFPageObj_GetType` == `FPDF_PAGEOBJ_IMAGE` | +| where it sits on the page | `FPDFPageObj_GetBounds` | +| size and colour depth | `FPDFImageObj_GetImageMetadata` | +| how it is stored | `FPDFImageObj_GetImageFilterCount`, `FPDFImageObj_GetImageFilter` | +| the bytes | `FPDFImageObj_GetImageDataDecoded`, or `FPDFImageObj_GetRenderedBitmap` | + +Two cases, and the second is the awkward one: + +- **`DCTDecode` or `JPXDecode`** — the decoded stream *is* a JPEG or JPEG 2000 file, so the bytes go + straight into an `Asset` with the matching media type. Free. +- **Anything else** (Flate-compressed raw pixels, which is very common) — there is no image file in + the PDF, only pixels. `FPDFImageObj_GetRenderedBitmap` returns BGRA, and something has to encode + it. `commonMain` has no PNG encoder, so we write one: PNG is a header, an IDAT of deflated + scanlines and a CRC, and korlibs-compression is already there for the deflate. Perhaps 150 lines, + and worth having anyway — no other target has an encoder either. + +**Placement.** `FPDFPageObj_GetBounds` gives each image a rectangle and `FPDFText_GetCharBox` gives +the text the same, so images can be emitted in reading order by vertical position rather than dumped +at the end of the page. Without it a figure lands after the prose that discusses it. This is the same +problem the PDFBox route had and never solved. + +**Scanned pages** — no text and one full-page image — are detectable (an image covering most of the +mediabox, almost no characters) and worth flagging rather than emitting as a wall of nothing. **The module does not register itself.** `:library` keeps no PDF dependency, and a caller opts in: @@ -109,7 +139,8 @@ ZIP plus the XML parser for `container.xml` and the OPF, then Phase 1 for the ch container format, and a good first exercise of the ZIP reader. ### Phase 3 — PDF, the `:pdfium` module -Independent of the OOXML work, so it can run in parallel or first if iOS PDF matters more. +Text first, then images and placement. Independent of the OOXML work, so it can run in parallel or +first if iOS PDF matters more. ### Phase 4 — DOCX `word/document.xml`: `w:p`, `w:r`, `w:t`, `w:rPr` for bold/italic/strike, `w:pStyle` for heading level, @@ -127,6 +158,31 @@ call it for every non-integer cell; reimplementing it is a project of its own an likely to change output silently. Revisit once the rest has landed and the corpus is wider — and note that until then the JVM build still carries POI. +## Assets, across every phase + +The model already carries them — `Asset(id, mediaType, bytes, name)`, `Document.assets` and +`Image.assetId` — but only DOCX populates them today. The readers need images from everything, so +each phase extracts what its format holds: + +| format | where the images are | notes | +|---|---|---| +| docx | done | ids are the file name and can collide; needs fixing | +| pdf | page objects, as above | needs a PNG encoder for raw-pixel images | +| pptx | `XSLFPictureShape` today emits a fabricated `shapeName.jpg` and no bytes | real assets when the raw-XML rewrite lands | +| epub | `` resolved against the chapter directory into a zip entry we already hold | needs an asset resolver in the HTML walker | +| html | remote URLs stay URLs; `data:` URIs decode into assets | cheap | +| xlsx | deferred with the rest of XLSX | | + +Three things this needs that do not exist yet: + +1. **An asset policy.** DOCX inlines base64 today, which is why its output is 161 KB against + markitdown's 4.6 KB. `MarkdownOptions` should carry `Inline` / `Reference` (write files, emit + relative links) / `AltTextOnly`; the current `imagesAsText` and `maxInlineImageUrl` are a crude + stand-in. A PDF full of figures makes this urgent rather than tidy. +2. **Stable asset ids.** The DOCX id is the embedded file name and repeats across parts. +3. **Intrinsic size on `Image`.** Width and height from the source, so a Compose or SwiftUI layout + does not jump while loading. `FPDFImageObj_GetImageMetadata` provides it for PDF. + ## Ground rules per phase 1. The new implementation lands in `commonMain`; the `jvmShared` version is deleted in the same @@ -146,11 +202,14 @@ that until then the JVM build still carries POI. | 0 infrastructure | half a day | low — or the plan changes, if a library will not build | | 1 HTML | 1 day | medium — parser differences on messy input | | 2 EPUB | half a day | low | -| 3 PDF via pdfium | 2-3 days | medium — binding is routine, packaging and output changes are not | +| 3 PDF via pdfium, text | 2 days | medium — binding is routine, packaging and output changes are not | +| 3b PDF images, PNG encoder, placement | 2-3 days | medium — the PNG writer is small but the reading-order interleave is fiddly | +| assets for epub, html, pptx | 1 day, spread across their phases | low | +| asset policy, ids, intrinsic size | half a day | low, but blocks the readers | | 4 DOCX | 2-3 days | medium | | 5 PPTX | 3-5 days | medium, mostly volume | -Roughly a week and a half without XLSX. +Roughly two weeks without XLSX, images included. ## Worth deciding before starting From 94cb3c8d8aa2735b05bee1eb3fc1e61eb7c16d62 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Tue, 18 Aug 2026 00:08:00 +0200 Subject: [PATCH 04/15] Remove the office formats, and Apache POI with them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DOCX, XLSX and PPTX are editing formats; a reader meets PDF and EPUB. Deleting the three converters drops POI from the build: the distribution goes from 66 MB to 36 MB across 23 jars instead of 35, and the CDS archive from 41 MB to 23 MB. It buys no speed, which is worth stating plainly. POI was loaded lazily, so a CSV or EPUB conversion never paid for it — the startup numbers are unchanged. What improved is size and dependency surface. The fixtures stay and a test pins the behaviour callers now see, an UnsupportedFormatException rather than a wrong answer. If the formats return it is as an :office module mirroring the :pdfium design: separate module, separate dependency, registered by the caller. Also removes XLSX from the commonMain plan — it was the phase most likely to change output silently, since POI's DataFormatter implements Excel's whole number-format language — and records FB2 as a cheap addition once HTML moves to Ksoup. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 16 +- cli/build.gradle.kts | 2 +- docs/common-converters-plan.md | 50 ++-- gradle/libs.versions.toml | 2 - library/build.gradle.kts | 5 +- .../mikromarkdown/MikroMarkdownFactory.kt | 6 - .../mikromarkdown/FileIntegrationTest.kt | 50 +--- .../mikromarkdown/MikroMarkdownFactory.kt | 6 - .../mikromarkdown/converters/DocxConverter.kt | 164 ------------ .../mikromarkdown/converters/PptxConverter.kt | 237 ------------------ .../mikromarkdown/converters/XlsxConverter.kt | 64 ----- .../lemcoder/mikromarkdown/DumpOutputTest.kt | 6 +- .../mikromarkdown/PythonComparisonTest.kt | 6 - scripts/optbench.py | 2 +- 14 files changed, 48 insertions(+), 568 deletions(-) delete mode 100644 library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/DocxConverter.kt delete mode 100644 library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/PptxConverter.kt delete mode 100644 library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XlsxConverter.kt diff --git a/README.md b/README.md index c6a79e3..9aad4c0 100644 --- a/README.md +++ b/README.md @@ -6,11 +6,14 @@ Kotlin Multiplatform (JVM + Android) library that converts documents to Markdown ## Supported formats +Office formats were removed deliberately: DOCX, XLSX and PPTX are editing formats, while a reader +meets PDF and EPUB. Dropping them took Apache POI with them — the distribution went from 66 MB to +36 MB. If they are wanted back, they return as an opt-in module the way PDF is heading, rather than +as a dependency everyone carries. + + | Format | Extension | |--------|-----------| -| Word | `.docx` | -| Excel | `.xlsx` | -| PowerPoint | `.pptx` | | EPUB | `.epub` | | HTML | `.html`, `.htm` | | PDF | `.pdf` | @@ -39,7 +42,7 @@ once rather than per target; only PDF extraction and MIME detection are platform `ConversionResult.document` exposes it alongside the rendered Markdown. ```kotlin -val document = mid.parse("/path/to/report.docx") +val document = mid.parse("/path/to/book.epub") document.blocks.filterIsInstance().forEach { println(it.rows.size) } // Render with different options @@ -66,7 +69,7 @@ import io.github.lemcoder.mikromarkdown.MikroMarkdown val mid = MikroMarkdown() // from file path -val result = mid.convert("/path/to/document.docx") +val result = mid.convert("/path/to/book.epub") // from bytes with explicit format hint val bytes = File("document.html").readBytes() @@ -173,10 +176,7 @@ In-process, best of 50 runs after warmup: | test.json | 0.4 KB | 0.03 ms | 0.01 ms | 0.04 ms | | test.epub | 2 KB | 0.42 ms | 0.00 ms | 0.42 ms | | test_blog.html | 25 KB | 0.85 ms | 0.11 ms | 0.96 ms | -| test.xlsx | 11 KB | 1.09 ms | 0.00 ms | 1.03 ms | -| test.docx | 132 KB | 3.28 ms | 0.00 ms | 2.81 ms | | test.pdf | 90 KB | 3.69 ms | 0.00 ms | 3.47 ms | -| test.pptx | 271 KB | 4.79 ms | 0.00 ms | 4.53 ms | | test_wikipedia.html | 385 KB | 12.98 ms | 1.82 ms | 14.80 ms | End to end the CLI runs in 50–230 ms, against 3–5 ms for the Rust diff --git a/cli/build.gradle.kts b/cli/build.gradle.kts index d72093a..89f64b0 100644 --- a/cli/build.gradle.kts +++ b/cli/build.gradle.kts @@ -80,7 +80,7 @@ val cdsArchive by tasks.registering { // One sample per family, so the archive covers the classes each conversion path touches rather // than only the ones a DOCX happens to need. val samples = - listOf("test.docx", "test.pdf", "test_blog.html", "test.csv", "test.json", "test.xlsx").map { + listOf("test.pdf", "test.epub", "test_blog.html", "test.csv", "test.json", "test.xml").map { fixtures.file(it).asFile.absolutePath } val javaHome = javaToolchains.launcherFor(java.toolchain).get().metadata.installationPath diff --git a/docs/common-converters-plan.md b/docs/common-converters-plan.md index f6913a3..8ac11ae 100644 --- a/docs/common-converters-plan.md +++ b/docs/common-converters-plan.md @@ -4,16 +4,19 @@ Status: proposal. Branch `common-converters`. Nothing implemented yet. ## Where things stand -Five converters and one helper are still JVM-only, plus PDF on each platform: +**The office formats are gone.** DOCX, XLSX and PPTX were deleted rather than ported: they are +editing formats, and a reader meets PDF and EPUB. That took Apache POI with them — 66 MB of +distribution down to 36 MB, 35 jars to 23 — and removed the phase most likely to change output +silently. They can return as an opt-in module in the shape `:pdfium` is taking; the fixtures stay, +and a test pins that they currently raise `UnsupportedFormatException`. + +What is left JVM-only is two converters, one helper, and PDF on each platform: | file | lines | depends on | plan | |---|---|---|---| | `HtmlToDocument.kt` | 334 | Jsoup | → commonMain (Ksoup) | -| `PptxConverter.kt` | 237 | POI `XSLF*`, XMLBeans `CT*` | → commonMain (raw XML) | -| `DocxConverter.kt` | 164 | POI `XWPF*` | → commonMain (raw XML) | | `EpubConverter.kt` | 132 | `java.util.zip`, `javax.xml`, Jsoup | → commonMain | | `HtmlConverter.kt` | 15 | Jsoup, via the helper | → commonMain | -| `XlsxConverter.kt` | 64 | POI `XSSFWorkbook`, `DataFormatter` | **stays on POI for now** | | `PdfConverter.kt` ×2 | 32 | PDFBox / pdfbox-android | → a separate `:pdfium` module | Moving them is not a file move: `commonMain` cannot use POI, Jsoup, `java.util.zip` or `javax.xml`, @@ -25,10 +28,11 @@ output stayed byte-identical, which is the bar for everything except PDF. **iOS.** The document formats are the reason a SwiftUI reader cannot exist today. This is the blocker, and it is worth doing even if nothing gets faster. -**A smaller, faster JVM build** — but only partly, now. POI dominates the 66 MB distribution and most -of the 183 ms a DOCX takes end to end against 2.7 ms of actual conversion. Keeping XLSX on POI means -**POI stays on the JVM classpath**, so that payoff is deferred until XLSX moves too. The portability -payoff lands in full: native gets everything but XLSX. +**A smaller JVM build** — already banked by the deletion, and worth being exact about what it bought: +36 MB instead of 66 MB, but **no faster**. POI was loaded lazily, so a CSV or EPUB conversion never +paid for it; the startup numbers did not move. Size and dependency surface improved, speed did not. + +What remains to remove is Jsoup, and it goes when HTML moves to Ksoup. Against all of it: POI, Jsoup and PDFBox absorb an enormous amount of real-world malformation. Hand-written parsers will be less forgiving, and the fixture corpus is ten files. **Expanding it is @@ -142,21 +146,15 @@ container format, and a good first exercise of the ZIP reader. Text first, then images and placement. Independent of the OOXML work, so it can run in parallel or first if iOS PDF matters more. -### Phase 4 — DOCX -`word/document.xml`: `w:p`, `w:r`, `w:t`, `w:rPr` for bold/italic/strike, `w:pStyle` for heading level, -`w:numPr` for list level; images through `word/_rels/document.xml.rels` into `word/media/`. We use -three POI types and a narrow slice of each, so the rewrite is bounded. - -### Phase 5 — PPTX -The largest: slides, shapes, group shapes, placeholders, pictures, tables, and charts across eight -chart types currently read through XMLBeans. Chart XML is verbose but regular. Last because it is the -most code for the least reach. +### Phase 4 — FB2, if wanted +Book's Story parses FB2 with the same walker it uses for HTML, through Jsoup's XML mode. Ksoup has +that mode too, so once Phase 1 lands FB2 is a tag-mapping layer over `HtmlToDocument` — a format we +do not support today for about half a day's work. Worth it for a reader; skip it otherwise. -### Not now — XLSX -Stays on POI. `DataFormatter` implements Excel's number-format language in thousands of lines and we -call it for every non-integer cell; reimplementing it is a project of its own and the phase most -likely to change output silently. Revisit once the rest has landed and the corpus is wider — and note -that until then the JVM build still carries POI. +### Removed — DOCX, XLSX, PPTX +Deleted, not deferred. If they return it is as an `:office` module mirroring `:pdfium`: its own +Gradle module, its own dependency, registered by the caller rather than by the factory. The deleted +implementations are in the history, and the fixtures are still in the repository. ## Assets, across every phase @@ -206,18 +204,14 @@ Three things this needs that do not exist yet: | 3b PDF images, PNG encoder, placement | 2-3 days | medium — the PNG writer is small but the reading-order interleave is fiddly | | assets for epub, html, pptx | 1 day, spread across their phases | low | | asset policy, ids, intrinsic size | half a day | low, but blocks the readers | -| 4 DOCX | 2-3 days | medium | -| 5 PPTX | 3-5 days | medium, mostly volume | +| 4 FB2, optional | half a day | low | -Roughly two weeks without XLSX, images included. +Roughly a week, images included, now that the office formats are out. ## Worth deciding before starting - **Which targets beyond macOS?** Adding `iosArm64` and `linuxX64` early keeps the code honest; adding them late risks finding a dependency — or a pdfium archive — that does not fit. -- **Does the JVM keep POI-backed converters as an option?** A `mikromarkdown-poi` artifact would let - callers choose POI's tolerance of broken files over startup time, and would make each format's - switch reversible. It costs an artifact and a registration path. - **How are pdfium binaries shipped to JVM users** — bundled per platform in the artifact, or downloaded at build time by the consumer? The first is convenient and large; the second is small and one more thing to go wrong. diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 0a63414..0dfe11d 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -14,7 +14,6 @@ jsoup = "1.22.2" junit = "6.1.0" pdfbox = "3.0.7" pdfbox-android = "2.0.27.0" -poi = "5.5.1" tika = "3.3.0" clikt = "5.1.0" coreKtx = "1.7.0" @@ -34,7 +33,6 @@ jsoup = { module = "org.jsoup:jsoup", version.ref = "jsoup" } junit-jupiter = { module = "org.junit.jupiter:junit-jupiter", version.ref = "junit" } pdfbox = { module = "org.apache.pdfbox:pdfbox", version.ref = "pdfbox" } pdfbox-android = { module = "com.tom-roush:pdfbox-android", version.ref = "pdfbox-android" } -poi-ooxml = { module = "org.apache.poi:poi-ooxml", version.ref = "poi" } tika-core = { module = "org.apache.tika:tika-core", version.ref = "tika" } clikt = { module = "com.github.ajalt.clikt:clikt", version.ref = "clikt" } core-ktx = { group = "androidx.test", name = "core-ktx", version.ref = "coreKtx" } diff --git a/library/build.gradle.kts b/library/build.gradle.kts index 5a405c9..9b7899d 100644 --- a/library/build.gradle.kts +++ b/library/build.gradle.kts @@ -45,10 +45,7 @@ kotlin { // detection differ. Converters live here once instead of being copied per target. val jvmShared by creating { dependsOn(commonMain.get()) - dependencies { - implementation(libs.jsoup) - implementation(libs.poi.ooxml) - } + dependencies { implementation(libs.jsoup) } } jvmMain { diff --git a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt index 65b3f85..292c9f9 100644 --- a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt +++ b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt @@ -3,15 +3,12 @@ package io.github.lemcoder.mikromarkdown import android.content.Context import com.tom_roush.pdfbox.android.PDFBoxResourceLoader import io.github.lemcoder.mikromarkdown.converters.CsvConverter -import io.github.lemcoder.mikromarkdown.converters.DocxConverter import io.github.lemcoder.mikromarkdown.converters.EpubConverter import io.github.lemcoder.mikromarkdown.converters.HtmlConverter import io.github.lemcoder.mikromarkdown.converters.JsonConverter import io.github.lemcoder.mikromarkdown.converters.MarkdownPassthroughConverter import io.github.lemcoder.mikromarkdown.converters.PdfConverter import io.github.lemcoder.mikromarkdown.converters.PlainTextConverter -import io.github.lemcoder.mikromarkdown.converters.PptxConverter -import io.github.lemcoder.mikromarkdown.converters.XlsxConverter import io.github.lemcoder.mikromarkdown.converters.XmlConverter import java.io.File @@ -27,9 +24,6 @@ public fun MikroMarkdown(context: Context? = null): MikroMarkdown = register(CsvConverter()) register(JsonConverter()) register(XmlConverter()) - register(DocxConverter()) - register(XlsxConverter()) - register(PptxConverter()) register(EpubConverter()) if (context != null) { PDFBoxResourceLoader.init(context) diff --git a/library/src/commonTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.kt b/library/src/commonTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.kt index 076e5aa..5473077 100644 --- a/library/src/commonTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.kt +++ b/library/src/commonTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.kt @@ -2,6 +2,7 @@ package io.github.lemcoder.mikromarkdown import com.goncalossilva.resources.Resource import kotlin.test.Test +import kotlin.test.assertFailsWith import kotlin.test.assertFalse import kotlin.test.assertTrue @@ -27,45 +28,18 @@ class FileIntegrationTest { } } + /** + * The office formats were removed rather than ported: they are editing formats, and a reader meets PDF and EPUB. + * The fixtures stay for whenever an office plugin arrives, and this test pins the behaviour callers see until then. + */ @Test - fun testDocx() = - assertConversion( - filename = "test.docx", - mustInclude = - listOf( - "314b0a30-5b04-470b-b9f7-eed2c2bec74a", - "49e168b7-d2ae-407f-a055-2167576f39a1", - "# Abstract", - "# Introduction", - "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation", - ), - ) - - @Test - fun testXlsx() = - assertConversion( - filename = "test.xlsx", - mustInclude = - listOf( - "09060124-b5e7-4717-9d07-3c046eb", - "6ff4173b-42a5-4784-9b19-f49caff4d93d", - "affc7dad-52dc-4b98-9b5d-51e65d8a8ad0", - ), - ) - - @Test - fun testPptx() = - assertConversion( - filename = "test.pptx", - mustInclude = - listOf( - "2cdda5c8-e50e-4db4-b5f0-9722a649f455", - "04191ea8-5c73-4215-a1d3-1cfb43aaaf12", - "44bf7d06-5e7a-4a40-a2e1-a2e42ef28c8a", - "1b92870d-e3b5-4e65-8153-919f4ff45592", - "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation", - ), - ) + fun officeFormatsAreNotSupported() { + for (filename in listOf("test.docx", "test.xlsx", "test.pptx")) { + val bytes = Resource("test_files/$filename").readBytes() + val info = StreamInfo(extension = filename.substringAfterLast(".")) + assertFailsWith(filename) { mid.convert(bytes, info) } + } + } @Test fun testBlogHtml() = diff --git a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt index c13dc2d..48a534b 100644 --- a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt +++ b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt @@ -1,15 +1,12 @@ package io.github.lemcoder.mikromarkdown import io.github.lemcoder.mikromarkdown.converters.CsvConverter -import io.github.lemcoder.mikromarkdown.converters.DocxConverter import io.github.lemcoder.mikromarkdown.converters.EpubConverter import io.github.lemcoder.mikromarkdown.converters.HtmlConverter import io.github.lemcoder.mikromarkdown.converters.JsonConverter import io.github.lemcoder.mikromarkdown.converters.MarkdownPassthroughConverter import io.github.lemcoder.mikromarkdown.converters.PdfConverter import io.github.lemcoder.mikromarkdown.converters.PlainTextConverter -import io.github.lemcoder.mikromarkdown.converters.PptxConverter -import io.github.lemcoder.mikromarkdown.converters.XlsxConverter import io.github.lemcoder.mikromarkdown.converters.XmlConverter import java.io.File @@ -21,9 +18,6 @@ public fun MikroMarkdown(): MikroMarkdown = register(CsvConverter()) register(JsonConverter()) register(XmlConverter()) - register(DocxConverter()) - register(XlsxConverter()) - register(PptxConverter()) register(EpubConverter()) register(PdfConverter()) register(PlainTextConverter(), priority = 10.0) diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/DocxConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/DocxConverter.kt deleted file mode 100644 index 596863f..0000000 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/DocxConverter.kt +++ /dev/null @@ -1,164 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Asset -import io.github.lemcoder.mikromarkdown.model.Block -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.model.Heading -import io.github.lemcoder.mikromarkdown.model.Image -import io.github.lemcoder.mikromarkdown.model.Inline -import io.github.lemcoder.mikromarkdown.model.ListBlock -import io.github.lemcoder.mikromarkdown.model.ListItem -import io.github.lemcoder.mikromarkdown.model.Paragraph -import io.github.lemcoder.mikromarkdown.model.Table -import io.github.lemcoder.mikromarkdown.model.TableCell -import io.github.lemcoder.mikromarkdown.model.Text -import io.github.lemcoder.mikromarkdown.model.plainText -import io.github.lemcoder.mikromarkdown.model.styled -import java.util.Base64 -import org.apache.poi.xwpf.usermodel.XWPFDocument -import org.apache.poi.xwpf.usermodel.XWPFParagraph -import org.apache.poi.xwpf.usermodel.XWPFTable - -public class DocxConverter : DocumentConverter { - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "docx" || - info.mimetype == "application/vnd.openxmlformats-officedocument.wordprocessingml.document" - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val docx = XWPFDocument(bytes.inputStream()) - try { - val blocks = mutableListOf() - val assets = mutableListOf() - var title: String? = docx.properties?.coreProperties?.title?.trim()?.ifBlank { null } - val pendingListItems = mutableListOf>>() - - fun flushList() { - if (pendingListItems.isEmpty()) return - blocks += buildNestedList(pendingListItems) - pendingListItems.clear() - } - - for (element in docx.bodyElements) { - when (element) { - is XWPFParagraph -> { - val content = paragraphInlines(element, assets) - if (content.plainText().isBlank() && content.none { it is Image }) continue - - val level = headingLevel(element.styleID) - when { - level > 0 -> { - flushList() - if (title == null) title = content.plainText().trim() - blocks += Heading(level, content) - } - - element.numID != null -> - pendingListItems += (element.numIlvl?.toInt() ?: 0).coerceAtLeast(0) to content - - else -> { - flushList() - blocks += Paragraph(content) - } - } - } - - is XWPFTable -> { - flushList() - table(element)?.let { blocks += it } - } - } - } - flushList() - - return Document(blocks = blocks, title = title, assets = assets) - } finally { - docx.close() - } - } - - private fun paragraphInlines(para: XWPFParagraph, assets: MutableList): List { - val out = mutableListOf() - for (run in para.runs) { - val pictures = run.embeddedPictures - if (pictures.isNotEmpty()) { - for (picture in pictures) { - val data = picture.pictureData - val alt = picture.description.orEmpty() - if (data == null) { - if (alt.isNotBlank()) out += Text(alt) - continue - } - val mime = data.pictureTypeEnum.contentType - val id = data.fileName ?: "image-${assets.size + 1}" - assets += Asset(id = id, mediaType = mime, bytes = data.data, name = data.fileName) - out += - Image( - alt = alt, - url = "data:$mime;base64,${Base64.getEncoder().encodeToString(data.data)}", - assetId = id, - ) - } - continue - } - - val text = run.text() ?: continue - if (text.isEmpty()) continue - if (text.isBlank()) { - out += Text(text) - continue - } - out += styled(listOf(Text(text)), bold = run.isBold, italic = run.isItalic, strike = run.isStrikeThrough) - } - return out - } - - /** Rebuilds Word's flat numbering levels into nested list blocks. */ - private fun buildNestedList(items: List>>): ListBlock { - var index = 0 - - fun build(level: Int): List { - val result = mutableListOf() - while (index < items.size) { - val (itemLevel, content) = items[index] - when { - itemLevel < level -> break - itemLevel == level -> { - index++ - val children = - if (index < items.size && items[index].first > level) { - listOf(ListBlock(ordered = false, items = build(items[index].first))) - } else { - emptyList() - } - result += ListItem(listOf(Paragraph(content)) + children) - } - // A deeper first item without a parent: promote it to this level. - else -> result += ListItem(listOf(ListBlock(ordered = false, items = build(itemLevel)))) - } - } - return result - } - - return ListBlock(ordered = false, items = build(items.minOf { it.first })) - } - - private fun table(table: XWPFTable): Table? { - val rows = table.rows - if (rows.isEmpty()) return null - val header = rows[0].tableCells.map { TableCell(it.text.trim()) } - val body = rows.drop(1).map { row -> row.tableCells.map { TableCell(it.text.trim()) } } - return Table(header = header, rows = body) - } - - private fun headingLevel(style: String?): Int { - val s = style?.replace("\\s+".toRegex(), "") ?: return 0 - if (s.startsWith("Heading", ignoreCase = true)) { - return s.drop(7).toIntOrNull()?.coerceIn(1, 6) ?: 0 - } - // OOXML numeric style IDs 1-6 map directly to heading levels - return s.toIntOrNull()?.takeIf { it in 1..6 } ?: 0 - } -} diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/PptxConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/PptxConverter.kt deleted file mode 100644 index 6fa57dc..0000000 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/PptxConverter.kt +++ /dev/null @@ -1,237 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Block -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.model.Heading -import io.github.lemcoder.mikromarkdown.model.HtmlComment -import io.github.lemcoder.mikromarkdown.model.Image -import io.github.lemcoder.mikromarkdown.model.ListBlock -import io.github.lemcoder.mikromarkdown.model.ListItem -import io.github.lemcoder.mikromarkdown.model.Paragraph -import io.github.lemcoder.mikromarkdown.model.Table -import io.github.lemcoder.mikromarkdown.model.TableCell -import io.github.lemcoder.mikromarkdown.model.Text -import org.apache.poi.sl.usermodel.Placeholder -import org.apache.poi.sl.usermodel.Shape -import org.apache.poi.xslf.usermodel.XMLSlideShow -import org.apache.poi.xslf.usermodel.XSLFChart -import org.apache.poi.xslf.usermodel.XSLFGraphicFrame -import org.apache.poi.xslf.usermodel.XSLFGroupShape -import org.apache.poi.xslf.usermodel.XSLFPictureShape -import org.apache.poi.xslf.usermodel.XSLFSimpleShape -import org.apache.poi.xslf.usermodel.XSLFTable -import org.apache.poi.xslf.usermodel.XSLFTextShape -import org.openxmlformats.schemas.drawingml.x2006.chart.CTAreaSer -import org.openxmlformats.schemas.drawingml.x2006.chart.CTAxDataSource -import org.openxmlformats.schemas.drawingml.x2006.chart.CTBarSer -import org.openxmlformats.schemas.drawingml.x2006.chart.CTLineSer -import org.openxmlformats.schemas.drawingml.x2006.chart.CTNumDataSource -import org.openxmlformats.schemas.drawingml.x2006.chart.CTPieSer -import org.openxmlformats.schemas.drawingml.x2006.chart.CTScatterSer -import org.openxmlformats.schemas.drawingml.x2006.chart.CTSerTx -import org.openxmlformats.schemas.presentationml.x2006.main.CTPicture - -public class PptxConverter : DocumentConverter { - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "pptx" || - info.mimetype == "application/vnd.openxmlformats-officedocument.presentationml.presentation" - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val slideShow = XMLSlideShow(bytes.inputStream()) - try { - val blocks = mutableListOf() - var title: String? = null - - for ((index, slide) in slideShow.slides.withIndex()) { - blocks += HtmlComment("Slide number: ${index + 1}") - blocks += shapeBlocks(slide.shapes) { if (index == 0 && title == null) title = it } - - val notes = - slide.notes - ?.shapes - ?.filterIsInstance() - ?.filter { (it as? XSLFSimpleShape)?.placeholder != Placeholder.SLIDE_IMAGE } - ?.joinToString("\n") { it.text } - ?.trim() - .orEmpty() - if (notes.isNotBlank()) { - blocks += Heading(3, listOf(Text("Notes:"))) - blocks += notes.lines().filter { it.isNotBlank() }.map { Paragraph(listOf(Text(it.trim()))) } - } - } - - return Document(blocks = blocks, title = title) - } finally { - slideShow.close() - } - } - - private fun shapeBlocks(shapes: Iterable>, onTitle: (String) -> Unit): List { - val blocks = mutableListOf() - for (shape in shapes) { - when { - shape is XSLFGroupShape -> blocks += shapeBlocks(shape.shapes, onTitle) - - shape is XSLFTextShape -> { - val text = shape.text.trim() - if (text.isBlank()) continue - - val placeholder = (shape as? XSLFSimpleShape)?.placeholder - if (placeholder == Placeholder.TITLE || placeholder == Placeholder.CENTERED_TITLE) { - blocks += Heading(1, listOf(Text(text))) - onTitle(text) - continue - } - - // Consecutive bullet paragraphs become one list; plain ones stay paragraphs. - val bullets = mutableListOf() - fun flushBullets() { - if (bullets.isEmpty()) return - blocks += ListBlock(ordered = false, items = bullets.toList()) - bullets.clear() - } - for (para in shape.textParagraphs) { - val paraText = para.text.trim() - if (paraText.isBlank()) continue - if (para.isBullet) { - bullets += ListItem(listOf(Paragraph(listOf(Text(paraText))))) - } else { - flushBullets() - blocks += Paragraph(listOf(Text(paraText))) - } - } - flushBullets() - } - - shape is XSLFPictureShape -> { - val description = (shape.xmlObject as? CTPicture)?.nvPicPr?.cNvPr?.descr.orEmpty() - val alt = description.ifBlank { shape.shapeName } - val filename = shape.shapeName.replace(Regex("\\W"), "") + ".jpg" - blocks += Paragraph(listOf(Image(alt, filename))) - } - - shape is XSLFGraphicFrame && shape.hasChart() -> blocks += chartBlocks(shape.chart) - - shape is XSLFTable -> table(shape)?.let { blocks += it } - } - } - return blocks - } - - private fun chartBlocks(chart: XSLFChart): List { - val blocks = mutableListOf() - blocks += Heading(3, listOf(Text(listOfNotNull("Chart", chartTitle(chart)).joinToString(": ")))) - - val series = - try { - seriesOf(chart) - } catch (_: Exception) { - blocks += Paragraph(listOf(Text("[unsupported chart]"))) - return blocks - } - if (series.isEmpty()) return blocks - - val rowCount = series.maxOf { it.categories.size } - blocks += - Table( - header = (listOf("Category") + series.map { it.name }).map { TableCell(it) }, - rows = - (0 until rowCount).map { row -> - val category = series.first().categories.getOrElse(row) { "" } - (listOf(category) + series.map { it.values.getOrElse(row) { "" } }).map { TableCell(it) } - }, - ) - return blocks - } - - private fun chartTitle(chart: XSLFChart): String? = - try { - val ctChart = chart.ctChart - if (!ctChart.isSetTitle) { - null - } else { - val tx = ctChart.title?.tx - when { - tx?.isSetRich == true -> - tx.rich.pList.flatMap { p -> p.rList.map { r -> r.t.orEmpty() } }.joinToString("") - tx?.isSetStrRef == true -> tx.strRef?.strCache?.ptList?.firstOrNull()?.v - else -> null - }?.ifBlank { null } - } - } catch (_: Exception) { - null - } - - private data class Series(val name: String, val categories: List, val values: List) - - private fun seriesOf(chart: XSLFChart): List { - val plot = chart.ctChart.plotArea - // bar/bar3D, line/line3D and area/area3D each share one generated series type. - val all = - plot.barChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.bar3DChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.lineChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.line3DChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.areaChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.area3DChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.scatterChartList.flatMap { it.serList.map { s -> s.toSeries() } } + - plot.pieChartList.flatMap { it.serList.map { s -> s.toSeries() } } - return all.filter { it.categories.isNotEmpty() || it.values.isNotEmpty() } - } - - private fun CTBarSer.toSeries() = - series(if (isSetTx) tx else null, if (isSetCat) cat else null, if (isSetVal) `val` else null) - - private fun CTLineSer.toSeries() = - series(if (isSetTx) tx else null, if (isSetCat) cat else null, if (isSetVal) `val` else null) - - private fun CTAreaSer.toSeries() = - series(if (isSetTx) tx else null, if (isSetCat) cat else null, if (isSetVal) `val` else null) - - private fun CTPieSer.toSeries() = - series(if (isSetTx) tx else null, if (isSetCat) cat else null, if (isSetVal) `val` else null) - - private fun CTScatterSer.toSeries() = - series(if (isSetTx) tx else null, if (isSetXVal) xVal else null, if (isSetYVal) yVal else null) - - private fun series(tx: CTSerTx?, categories: CTAxDataSource?, values: CTNumDataSource?) = - Series(seriesName(tx), categoryValues(categories), numericValues(values)) - - private fun categoryValues(cat: CTAxDataSource?): List = - when { - cat == null -> emptyList() - cat.isSetStrRef -> cat.strRef?.strCache?.ptList?.sortedBy { it.idx }?.map { it.v } ?: emptyList() - cat.isSetNumRef -> cat.numRef?.numCache?.ptList?.sortedBy { it.idx }?.map { it.v.orEmpty() } ?: emptyList() - cat.isSetNumLit -> cat.numLit?.ptList?.sortedBy { it.idx }?.map { it.v.orEmpty() } ?: emptyList() - cat.isSetStrLit -> cat.strLit?.ptList?.sortedBy { it.idx }?.map { it.v } ?: emptyList() - else -> emptyList() - } - - private fun numericValues(v: CTNumDataSource?): List = - when { - v == null -> emptyList() - v.isSetNumRef -> v.numRef?.numCache?.ptList?.sortedBy { it.idx }?.map { it.v.orEmpty() } ?: emptyList() - v.isSetNumLit -> v.numLit?.ptList?.sortedBy { it.idx }?.map { it.v.orEmpty() } ?: emptyList() - else -> emptyList() - } - - private fun seriesName(tx: CTSerTx?): String = - when { - tx == null -> "" - tx.isSetV -> tx.v - tx.isSetStrRef -> tx.strRef?.strCache?.ptList?.firstOrNull()?.v.orEmpty() - else -> "" - } - - private fun table(table: XSLFTable): Table? { - val rows = table.rows - if (rows.isEmpty()) return null - return Table( - header = rows[0].cells.map { TableCell(it.text.trim()) }, - rows = rows.drop(1).map { row -> row.cells.map { TableCell(it.text.trim()) } }, - ) - } -} diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XlsxConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XlsxConverter.kt deleted file mode 100644 index a6c0e76..0000000 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/XlsxConverter.kt +++ /dev/null @@ -1,64 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Block -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.model.Heading -import io.github.lemcoder.mikromarkdown.model.Table -import io.github.lemcoder.mikromarkdown.model.TableCell -import io.github.lemcoder.mikromarkdown.model.Text -import kotlin.math.floor -import org.apache.poi.ss.usermodel.Cell -import org.apache.poi.ss.usermodel.CellType -import org.apache.poi.ss.usermodel.DataFormatter -import org.apache.poi.xssf.usermodel.XSSFWorkbook - -public class XlsxConverter : DocumentConverter { - // Constructing a converter must not load POI: accepts() only looks at the extension. - private val formatter by lazy { DataFormatter() } - - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "xlsx" || - info.mimetype == "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val workbook = XSSFWorkbook(bytes.inputStream()) - try { - val blocks = mutableListOf() - - for (sheet in workbook) { - val rows = sheet.toList() - if (rows.isEmpty()) continue - - val columns = rows.maxOf { it.lastCellNum.toInt().coerceAtLeast(0) } - if (columns == 0) continue - - blocks += Heading(2, listOf(Text(sheet.sheetName))) - blocks += - Table( - header = (0 until columns).map { TableCell(cellValue(rows[0].getCell(it))) }, - rows = - rows.drop(1).map { row -> (0 until columns).map { TableCell(cellValue(row.getCell(it))) } }, - ) - } - - return Document(blocks = blocks) - } finally { - workbook.close() - } - } - - private fun cellValue(cell: Cell?): String { - if (cell == null) return "" - return when (cell.cellType) { - CellType.NUMERIC -> { - val v = cell.numericCellValue - if (v == floor(v) && !v.isInfinite()) v.toLong().toString() else formatter.formatCellValue(cell) - } - CellType.BLANK -> "" - else -> formatter.formatCellValue(cell).trim() - } - } -} diff --git a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/DumpOutputTest.kt b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/DumpOutputTest.kt index 84c9627..1a123c5 100644 --- a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/DumpOutputTest.kt +++ b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/DumpOutputTest.kt @@ -10,11 +10,11 @@ class DumpOutputTest { fun dumpAll() { for (name in listOf( - "test.docx", - "test.xlsx", - "test.pptx", "test.epub", + "test.pdf", + "test.csv", "test.json", + "test.xml", "test_blog.html", "test_wikipedia.html", )) { diff --git a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/PythonComparisonTest.kt b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/PythonComparisonTest.kt index b744e55..e38d826 100644 --- a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/PythonComparisonTest.kt +++ b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/PythonComparisonTest.kt @@ -9,12 +9,6 @@ import org.junit.jupiter.api.Test class PythonComparisonTest { - @Test fun testDocx() = compare("test.docx") - - @Test fun testXlsx() = compare("test.xlsx") - - @Test fun testPptx() = compare("test.pptx") - @Test fun testEpub() = compare("test.epub") @Test fun testJson() = compare("test.json") diff --git a/scripts/optbench.py b/scripts/optbench.py index 028d32d..3db83df 100644 --- a/scripts/optbench.py +++ b/scripts/optbench.py @@ -30,7 +30,7 @@ TIMED = [("580 KB", PERF / "medium.csv"), ("1.8 MB", PERF / "big.csv")] TIMED_JVM = [ ("wiki", FIXTURES / "test_wikipedia.html"), - ("docx", FIXTURES / "test.docx"), + ("epub", FIXTURES / "test.epub"), ("pdf", FIXTURES / "test.pdf"), ("json", PERF / "big.json"), ] From d6f20fdfbfd74902a2a2a8d34b9f9e7d89930053 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Tue, 18 Aug 2026 00:17:24 +0200 Subject: [PATCH 05/15] Phase 0: Ksoup and korlibs run on native, xmlutil is not needed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both dependencies are in commonMain and were checked by running them on the macosArm64 binary rather than by watching them resolve: Ksoup parses HTML, and korlibs deflate round-trips a kilobyte down to 31 bytes and back. The useful discovery is that Ksoup's XML mode reads an EPUB OPF — dc:title, manifest hrefs, spine idrefs, the cover meta — so the planned XML library drops out of the plan entirely. korlibs ships no ZIP reader, so the central directory stays ours to parse, but that was always the smaller half. Its API is under korlibs.io.compression, not korlibs.compression, which the package listing in the klib does not make obvious. Co-Authored-By: Claude Opus 5 (1M context) --- docs/common-converters-plan.md | 15 +++++++++------ gradle/libs.versions.toml | 4 ++++ library/build.gradle.kts | 7 ++++++- 3 files changed, 19 insertions(+), 7 deletions(-) diff --git a/docs/common-converters-plan.md b/docs/common-converters-plan.md index 8ac11ae..3718549 100644 --- a/docs/common-converters-plan.md +++ b/docs/common-converters-plan.md @@ -42,12 +42,14 @@ part of the work, not an afterthought.** | need | choice | notes | |---|---|---| -| ZIP + inflate | `com.soywiz:korlibs-compression:6.0.0` | OOXML and EPUB are ZIP; no inflate in kotlinx-io or okio on native | -| XML parsing | `io.github.pdvrieze.xmlutil:core:0.91.1` | KMP pull parser; hand-rolling one is where entities and namespaces go wrong | +| inflate | `com.soywiz:korlibs-compression:6.0.0` | EPUB is a ZIP; nothing in kotlinx-io or okio inflates on native. It ships **no ZIP reader**, so the central directory is ours to parse — about 150 lines | | HTML parsing | `com.fleeksoft.ksoup:ksoup:0.2.6` | KMP port of Jsoup, near-identical API, `macosarm64` published | | PDF | pdfium via cinterop + JNI | see below | -Confirm each links for `macosArm64` in a throwaway module before committing to the sequence. +Verified on `macosArm64`, running rather than merely linking: Ksoup parses HTML and, in XML mode, +reads an OPF well enough to pull `dc:title`, manifest hrefs, spine idrefs and the cover meta — +**so no separate XML library is needed**. korlibs deflate round-trips (1000 bytes → 31 → 1000). +The korlibs API lives under `korlibs.io.compression.*`, not `korlibs.compression.*`. ## PDF: a `:pdfium` module @@ -128,9 +130,10 @@ That keeps `:pdfium` genuinely extractable — delete the module and the rest st Each phase is shippable on its own, risky ones last. -### Phase 0 — infrastructure -Add the three commonMain dependencies, confirm the native target still links, and extend -`scripts/optbench.py` so it verifies native output for every fixture a phase unlocks. +### Phase 0 — infrastructure ✅ +Ksoup and korlibs-compression are in `commonMain` and proven to run on the native binary. xmlutil +turned out to be unnecessary — Ksoup's XML mode covers EPUB's container and OPF. Still to do when a +phase needs it: extend `scripts/optbench.py` to verify native output for the fixtures it unlocks. ### Phase 1 — HTML, via Ksoup `HtmlToDocument` is written against a Jsoup-shaped API, so the port is mechanical. The risk is the diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 0dfe11d..7b1713c 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -6,6 +6,8 @@ android-compileSdk = "37" vanniktechMavenPublish = "0.36.0" kotlinx-io = "0.9.0" kotlinx-serialization = "1.9.0" +ksoup = "0.2.6" +korlibs = "6.0.0" kotlinx-resources = "0.15.0" commons-csv = "1.14.1" @@ -24,6 +26,8 @@ konsist = "0.17.3" [libraries] kotlin-test = { module = "org.jetbrains.kotlin:kotlin-test", version.ref = "kotlin" } kotlinx-io-core = { module = "org.jetbrains.kotlinx:kotlinx-io-core", version.ref = "kotlinx-io" } +ksoup = { module = "com.fleeksoft.ksoup:ksoup", version.ref = "ksoup" } +korlibs-compression = { module = "com.soywiz:korlibs-compression", version.ref = "korlibs" } kotlinx-serialization-json = { module = "org.jetbrains.kotlinx:kotlinx-serialization-json", version.ref = "kotlinx-serialization" } kotlinx-resources = { module = "com.goncalossilva:resources", version.ref = "kotlinx-resources" } diff --git a/library/build.gradle.kts b/library/build.gradle.kts index 9b7899d..a859134 100644 --- a/library/build.gradle.kts +++ b/library/build.gradle.kts @@ -39,7 +39,12 @@ kotlin { } sourceSets { - commonMain.dependencies { implementation(libs.kotlinx.io.core) } + commonMain.dependencies { + implementation(libs.kotlinx.io.core) + // Phase 0 of the commonMain migration: HTML parsing and the inflate that ZIP needs. + implementation(libs.ksoup) + implementation(libs.korlibs.compression) + } // JVM and Android run the same parsers on the same libraries; only PDF and MIME // detection differ. Converters live here once instead of being copied per target. From 16fb03457331e407f2352b58414d0486be4b06df Mon Sep 17 00:00:00 2001 From: mikolaj Date: Tue, 18 Aug 2026 00:29:31 +0200 Subject: [PATCH 06/15] Phase 1: HTML moves to commonMain on Ksoup, Jsoup is gone MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The walker was written against a Jsoup-shaped API and Ksoup is a port of it, so the change is two lines of difference: wholeText is a method rather than a property, and Charsets does not exist in commonMain. The risk was never the API — it was whether Ksoup recovers from messy HTML the same way. It does: test_blog.html and test_wikipedia.html are byte-identical through the new parser, and the native binary matches the JVM on both. HTML now converts on native, at 7 ms for the blog against the JVM's 80, and 66 ms for Wikipedia against 119. The harness verifies both HTML fixtures across targets from now on, so a future change cannot quietly diverge them. jvmShared is down to EpubConverter alone, kept only until its zip and XML move. Co-Authored-By: Claude Opus 5 (1M context) --- docs/common-converters-plan.md | 14 +++++++------- gradle/libs.versions.toml | 2 -- library/build.gradle.kts | 9 +++------ .../mikromarkdown/converters/HtmlConverter.kt | 2 +- .../lemcoder/mikromarkdown/utils/HtmlToDocument.kt | 12 ++++++------ .../lemcoder/mikromarkdown/MikroMarkdownFactory.kt | 5 +++-- scripts/optbench.py | 2 +- 7 files changed, 21 insertions(+), 25 deletions(-) rename library/src/{jvmShared => commonMain}/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt (87%) rename library/src/{jvmShared => commonMain}/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt (97%) diff --git a/docs/common-converters-plan.md b/docs/common-converters-plan.md index 3718549..b747130 100644 --- a/docs/common-converters-plan.md +++ b/docs/common-converters-plan.md @@ -14,9 +14,7 @@ What is left JVM-only is two converters, one helper, and PDF on each platform: | file | lines | depends on | plan | |---|---|---|---| -| `HtmlToDocument.kt` | 334 | Jsoup | → commonMain (Ksoup) | | `EpubConverter.kt` | 132 | `java.util.zip`, `javax.xml`, Jsoup | → commonMain | -| `HtmlConverter.kt` | 15 | Jsoup, via the helper | → commonMain | | `PdfConverter.kt` ×2 | 32 | PDFBox / pdfbox-android | → a separate `:pdfium` module | Moving them is not a file move: `commonMain` cannot use POI, Jsoup, `java.util.zip` or `javax.xml`, @@ -135,11 +133,13 @@ Ksoup and korlibs-compression are in `commonMain` and proven to run on the nativ turned out to be unnecessary — Ksoup's XML mode covers EPUB's container and OPF. Still to do when a phase needs it: extend `scripts/optbench.py` to verify native output for the fixtures it unlocks. -### Phase 1 — HTML, via Ksoup -`HtmlToDocument` is written against a Jsoup-shaped API, so the port is mechanical. The risk is the -parser, not the API: Wikipedia is messy and Ksoup may recover differently. Acceptance is -byte-identical `test_blog.html` and `test_wikipedia.html`; if it is not, the diff decides whether the -difference is defensible. +### Phase 1 — HTML, via Ksoup ✅ +Done, and the feared risk did not materialise: `test_blog.html` and `test_wikipedia.html` are +byte-identical through Ksoup, and the native binary matches the JVM on both. Two API differences +only — `wholeText` is a method rather than a property, and `Charsets` does not exist in commonMain. + +Jsoup is gone from the build. HTML on native converts a blog in 7 ms against the JVM's 80, and +Wikipedia in 66 ms against 119. ### Phase 2 — EPUB ZIP plus the XML parser for `container.xml` and the OPF, then Phase 1 for the chapters. Smallest diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 7b1713c..7bca1a6 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -12,7 +12,6 @@ kotlinx-resources = "0.15.0" commons-csv = "1.14.1" jackson = "2.21.3" -jsoup = "1.22.2" junit = "6.1.0" pdfbox = "3.0.7" pdfbox-android = "2.0.27.0" @@ -33,7 +32,6 @@ kotlinx-resources = { module = "com.goncalossilva:resources", version.ref = "kot commons-csv = { module = "org.apache.commons:commons-csv", version.ref = "commons-csv" } jackson-kotlin = { module = "com.fasterxml.jackson.module:jackson-module-kotlin", version.ref = "jackson" } -jsoup = { module = "org.jsoup:jsoup", version.ref = "jsoup" } junit-jupiter = { module = "org.junit.jupiter:junit-jupiter", version.ref = "junit" } pdfbox = { module = "org.apache.pdfbox:pdfbox", version.ref = "pdfbox" } pdfbox-android = { module = "com.tom-roush:pdfbox-android", version.ref = "pdfbox-android" } diff --git a/library/build.gradle.kts b/library/build.gradle.kts index a859134..f1ce703 100644 --- a/library/build.gradle.kts +++ b/library/build.gradle.kts @@ -46,12 +46,9 @@ kotlin { implementation(libs.korlibs.compression) } - // JVM and Android run the same parsers on the same libraries; only PDF and MIME - // detection differ. Converters live here once instead of being copied per target. - val jvmShared by creating { - dependsOn(commonMain.get()) - dependencies { implementation(libs.jsoup) } - } + // What is left for JVM and Android to share: EPUB, until its zip and XML move to + // commonMain. Everything else already lives there. + val jvmShared by creating { dependsOn(commonMain.get()) } jvmMain { dependsOn(jvmShared) diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt similarity index 87% rename from library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt rename to library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt index fee70f9..38f1c3b 100644 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/HtmlConverter.kt @@ -11,5 +11,5 @@ public class HtmlConverter : DocumentConverter { } override fun parse(bytes: ByteArray, info: StreamInfo): Document = - HtmlToDocument.parse(bytes.toString(Charsets.UTF_8), info.localPath.orEmpty()) + HtmlToDocument.parse(bytes.decodeToString(), info.localPath.orEmpty()) } diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt similarity index 97% rename from library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt rename to library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt index a8a9260..9220084 100644 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/HtmlToDocument.kt @@ -1,5 +1,9 @@ package io.github.lemcoder.mikromarkdown.utils +import com.fleeksoft.ksoup.Ksoup +import com.fleeksoft.ksoup.nodes.Element +import com.fleeksoft.ksoup.nodes.Node +import com.fleeksoft.ksoup.nodes.TextNode import io.github.lemcoder.mikromarkdown.model.Block import io.github.lemcoder.mikromarkdown.model.BlockQuote import io.github.lemcoder.mikromarkdown.model.CodeBlock @@ -21,10 +25,6 @@ import io.github.lemcoder.mikromarkdown.model.TableCell import io.github.lemcoder.mikromarkdown.model.Text import io.github.lemcoder.mikromarkdown.model.ThematicBreak import io.github.lemcoder.mikromarkdown.model.plainText -import org.jsoup.Jsoup -import org.jsoup.nodes.Element -import org.jsoup.nodes.Node -import org.jsoup.nodes.TextNode /** * Walks an HTML DOM into the shared document model. @@ -79,7 +79,7 @@ internal object HtmlToDocument { ) internal fun parse(html: String, baseUri: String = ""): Document { - val doc = Jsoup.parse(html, baseUri) + val doc = Ksoup.parse(html = html, baseUri = baseUri) doc.select(DROPPED_SELECTOR).remove() val title = doc.title().ifBlank { null } val root = doc.body() ?: doc @@ -310,7 +310,7 @@ internal object HtmlToDocument { /** HTML collapses runs of whitespace; do the same before the text reaches the model. */ // Non-breaking spaces are not collapsible whitespace in HTML, so they survive verbatim. - private fun TextNode.normalizedText(): String = wholeText.replace(COLLAPSIBLE_WHITESPACE, " ") + private fun TextNode.normalizedText(): String = getWholeText().replace(COLLAPSIBLE_WHITESPACE, " ") private fun List.startsWithSpace(): Boolean = (firstOrNull() as? Text)?.value?.startsWith(" ") == true diff --git a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt index be95a3b..8541451 100644 --- a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt +++ b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt @@ -1,6 +1,7 @@ package io.github.lemcoder.mikromarkdown import io.github.lemcoder.mikromarkdown.converters.CsvConverter +import io.github.lemcoder.mikromarkdown.converters.HtmlConverter import io.github.lemcoder.mikromarkdown.converters.JsonConverter import io.github.lemcoder.mikromarkdown.converters.MarkdownPassthroughConverter import io.github.lemcoder.mikromarkdown.converters.PlainTextConverter @@ -9,12 +10,12 @@ import io.github.lemcoder.mikromarkdown.converters.XmlConverter /** * A [MikroMarkdown] with the converters that need no platform library. * - * The document formats (DOCX, XLSX, PPTX, EPUB, PDF, HTML) still depend on JVM libraries and are absent here; this - * target exists to measure what a native binary costs to start and run. + * EPUB and PDF are still JVM-only; everything else the library converts is here. */ public fun MikroMarkdown(): MikroMarkdown = MikroMarkdown(SignatureMimeDetector).apply { register(MarkdownPassthroughConverter()) + register(HtmlConverter()) register(CsvConverter()) register(JsonConverter()) register(XmlConverter()) diff --git a/scripts/optbench.py b/scripts/optbench.py index 3db83df..9cbda99 100644 --- a/scripts/optbench.py +++ b/scripts/optbench.py @@ -56,7 +56,7 @@ def verify(): continue if subprocess.run([str(JVM), str(fixture)], capture_output=True, cwd=REPO).stdout != baseline.read_bytes(): problems.append(f"jvm {name}") - for name in ("test.csv", "test.json", "test.xml"): + for name in ("test.csv", "test.json", "test.xml", "test_blog.html", "test_wikipedia.html"): fixture = FIXTURES / name jvm = subprocess.run([str(JVM), str(fixture)], capture_output=True, cwd=REPO).stdout native = subprocess.run([str(NATIVE), str(fixture)], capture_output=True, cwd=REPO).stdout From 9e57beff17ed6feb42f5cdd30edf1c21701209e1 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Tue, 18 Aug 2026 00:36:52 +0200 Subject: [PATCH 07/15] Phase 2: EPUB moves to commonMain, the JVM-only source set is gone MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A ZipArchive in commonMain reads the central directory, follows each entry to its local header and inflates through korlibs — korlibs supplies no reader of its own, and neither kotlinx-io nor okio reads archives. The container and package documents go through Ksoup's XML mode, so there is still no XML dependency, and chapters reuse the HTML walker from Phase 1. Output is byte-identical to the java.util.zip and javax.xml implementation it replaces, and the native binary matches the JVM. With EPUB moved, jvmShared held nothing and is deleted: every converter but PDF now lives in commonMain, and the JVM keeps only Tika and PDFBox. One architecture rule needed narrowing rather than obeying. It flagged ZipArchive's companion as a public helper under utils, but a companion inside an internal class is already unreachable; the rule now looks at top-level declarations only, and was re-checked against a planted public helper to confirm it still bites. Co-Authored-By: Claude Opus 5 (1M context) --- docs/common-converters-plan.md | 10 +- library/build.gradle.kts | 11 +- .../mikromarkdown/converters/EpubConverter.kt | 99 +++++++++++++ .../mikromarkdown/utils/ZipArchive.kt | 120 ++++++++++++++++ .../mikromarkdown/converters/EpubConverter.kt | 132 ------------------ .../mikromarkdown/ArchitectureTest.kt | 14 +- .../mikromarkdown/MikroMarkdownFactory.kt | 4 +- scripts/optbench.py | 2 +- 8 files changed, 240 insertions(+), 152 deletions(-) create mode 100644 library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt create mode 100644 library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/ZipArchive.kt delete mode 100644 library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt diff --git a/docs/common-converters-plan.md b/docs/common-converters-plan.md index b747130..37874e6 100644 --- a/docs/common-converters-plan.md +++ b/docs/common-converters-plan.md @@ -10,11 +10,10 @@ distribution down to 36 MB, 35 jars to 23 — and removed the phase most likely silently. They can return as an opt-in module in the shape `:pdfium` is taking; the fixtures stay, and a test pins that they currently raise `UnsupportedFormatException`. -What is left JVM-only is two converters, one helper, and PDF on each platform: +What is left JVM-only is PDF, and nothing else: | file | lines | depends on | plan | |---|---|---|---| -| `EpubConverter.kt` | 132 | `java.util.zip`, `javax.xml`, Jsoup | → commonMain | | `PdfConverter.kt` ×2 | 32 | PDFBox / pdfbox-android | → a separate `:pdfium` module | Moving them is not a file move: `commonMain` cannot use POI, Jsoup, `java.util.zip` or `javax.xml`, @@ -141,9 +140,10 @@ only — `wholeText` is a method rather than a property, and `Charsets` does not Jsoup is gone from the build. HTML on native converts a blog in 7 ms against the JVM's 80, and Wikipedia in 66 ms against 119. -### Phase 2 — EPUB -ZIP plus the XML parser for `container.xml` and the OPF, then Phase 1 for the chapters. Smallest -container format, and a good first exercise of the ZIP reader. +### Phase 2 — EPUB ✅ +Done, byte-identical, and the JVM-only source set is gone with it. `ZipArchive` in commonMain reads +the central directory and inflates through korlibs; the container and package documents go through +Ksoup's XML mode; chapters reuse Phase 1. EPUB now converts on native. ### Phase 3 — PDF, the `:pdfium` module Text first, then images and placement. Independent of the OOXML work, so it can run in parallel or diff --git a/library/build.gradle.kts b/library/build.gradle.kts index f1ce703..fce51b7 100644 --- a/library/build.gradle.kts +++ b/library/build.gradle.kts @@ -14,7 +14,6 @@ kotlin { // Public API must be spelled out: visibility and return types, no accidental exports. explicitApi() - // Declaring jvmShared by hand switches off the automatic hierarchy, which macosMain needs. applyDefaultHierarchyTemplate() jvm { @@ -46,22 +45,14 @@ kotlin { implementation(libs.korlibs.compression) } - // What is left for JVM and Android to share: EPUB, until its zip and XML move to - // commonMain. Everything else already lives there. - val jvmShared by creating { dependsOn(commonMain.get()) } - jvmMain { - dependsOn(jvmShared) dependencies { implementation(libs.tika.core) implementation(libs.pdfbox) } } - androidMain { - dependsOn(jvmShared) - dependencies { implementation(libs.pdfbox.android) } - } + androidMain { dependencies { implementation(libs.pdfbox.android) } } commonTest.dependencies { implementation(libs.kotlin.test) diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt new file mode 100644 index 0000000..16f8323 --- /dev/null +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt @@ -0,0 +1,99 @@ +package io.github.lemcoder.mikromarkdown.converters + +import com.fleeksoft.ksoup.Ksoup +import com.fleeksoft.ksoup.parser.Parser +import io.github.lemcoder.mikromarkdown.DocumentConverter +import io.github.lemcoder.mikromarkdown.StreamInfo +import io.github.lemcoder.mikromarkdown.model.Block +import io.github.lemcoder.mikromarkdown.model.Document +import io.github.lemcoder.mikromarkdown.model.Paragraph +import io.github.lemcoder.mikromarkdown.model.Strong +import io.github.lemcoder.mikromarkdown.model.Text +import io.github.lemcoder.mikromarkdown.utils.HtmlToDocument +import io.github.lemcoder.mikromarkdown.utils.ZipArchive + +/** + * An EPUB is a ZIP of XHTML. + * + * `META-INF/container.xml` names the package document, which lists the manifest and the reading order; the chapters + * themselves go through the same walker as any other HTML. The XML is read with Ksoup's XML parser rather than a second + * library, which is why there is no XML dependency. + */ +public class EpubConverter : DocumentConverter { + private val metaFields = + listOf( + "title" to "Title", + "creator" to "Authors", + "language" to "Language", + "description" to "Description", + "identifier" to "Identifier", + ) + + override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { + return info.extension == "epub" || info.mimetype == "application/epub+zip" + } + + override fun parse(bytes: ByteArray, info: StreamInfo): Document { + val archive = ZipArchive.open(bytes) ?: return Document() + + val container = archive.readText("META-INF/container.xml") ?: return Document() + val opfPath = opfPath(container) ?: return Document() + val opf = archive.readText(opfPath) ?: archive.readText(opfPath.removePrefix("/")) ?: return Document() + val opfDirectory = opfPath.substringBeforeLast("/", "") + + val packageDocument = Ksoup.parse(html = opf, parser = Parser.xmlParser()) + val metadata = metadata(packageDocument) + val manifest = manifest(packageDocument) + + val blocks = mutableListOf() + var title: String? = metadata["title"] + + for ((key, label) in metaFields) { + val value = metadata[key] ?: continue + blocks += Paragraph(listOf(Strong(listOf(Text("$label:"))), Text(" $value"))) + } + + for (idref in spine(packageDocument)) { + val href = manifest[idref] ?: continue + val path = if (opfDirectory.isEmpty()) href else "$opfDirectory/$href" + val html = archive.readText(path) ?: archive.readText(path.removePrefix("/")) ?: continue + val chapter = HtmlToDocument.parse(html) + if (title == null) title = chapter.title + blocks += chapter.blocks + } + + return Document(blocks = blocks, title = title, metadata = metadata) + } + + private fun opfPath(container: String): String? = + Ksoup.parse(html = container, parser = Parser.xmlParser()).selectFirst("rootfile")?.attr("full-path")?.ifEmpty { + null + } + + private fun metadata(packageDocument: com.fleeksoft.ksoup.nodes.Document): Map { + val metadata = LinkedHashMap() + for (field in listOf("title", "creator", "language", "description", "identifier")) { + // Namespaced in the source as dc:title; Ksoup escapes the prefix with a pipe. + val text = packageDocument.selectFirst("dc|$field")?.text()?.trim() + if (!text.isNullOrEmpty()) metadata[field] = text + } + return metadata + } + + /** Manifest ids to hrefs, keeping only the documents that carry text. */ + private fun manifest(packageDocument: com.fleeksoft.ksoup.nodes.Document): Map { + val manifest = LinkedHashMap() + for (item in packageDocument.select("manifest > item")) { + val id = item.attr("id") + val href = item.attr("href") + val mediaType = item.attr("media-type") + if (id.isNotEmpty() && href.isNotEmpty() && mediaType.contains("html")) { + manifest[id] = href + } + } + return manifest + } + + private fun spine(packageDocument: com.fleeksoft.ksoup.nodes.Document): List = + packageDocument.select("spine > itemref").map { it.attr("idref") }.filter { it.isNotEmpty() } +} diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/ZipArchive.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/ZipArchive.kt new file mode 100644 index 0000000..b031e56 --- /dev/null +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/ZipArchive.kt @@ -0,0 +1,120 @@ +package io.github.lemcoder.mikromarkdown.utils + +import korlibs.io.compression.deflate.Deflate +import korlibs.io.compression.uncompress + +/** + * Reads a ZIP container held in memory. + * + * korlibs supplies the inflate and nothing else, and neither kotlinx-io nor okio reads archives, so the container + * itself is parsed here: find the end-of-central-directory record, walk the directory, and follow each entry to its + * local header. Enough of the format for EPUB and, later, OOXML. + */ +internal class ZipArchive private constructor(private val source: ByteArray, private val entries: Map) { + + val names: Set + get() = entries.keys + + /** The entry's bytes, inflated if it was deflated, or null if it is absent or unreadable. */ + fun read(name: String): ByteArray? { + val entry = entries[name] ?: return null + val start = dataStart(entry) ?: return null + if (start + entry.compressedSize > source.size) return null + + val raw = source.copyOfRange(start, start + entry.compressedSize) + return when (entry.method) { + STORED -> raw + DEFLATED -> + try { + raw.uncompress(Deflate) + } catch (_: Exception) { + null + } + // Any other method — bzip2, lzma, encrypted — is not something we can read. + else -> null + } + } + + /** The entry decoded as UTF-8, which every XML and XHTML part of an EPUB is. */ + fun readText(name: String): String? = read(name)?.decodeToString() + + /** + * Where an entry's bytes begin. + * + * The central directory records where the local header is, but the local header repeats the name and extra field + * with lengths of its own, so the data offset can only be computed from there. + */ + private fun dataStart(entry: Entry): Int? { + val header = entry.localHeaderOffset + if (header + LOCAL_HEADER_MINIMUM > source.size) return null + if (source.readInt(header) != LOCAL_HEADER_SIGNATURE) return null + val nameLength = source.readShort(header + 26) + val extraLength = source.readShort(header + 28) + return header + LOCAL_HEADER_MINIMUM + nameLength + extraLength + } + + private class Entry(val localHeaderOffset: Int, val compressedSize: Int, val method: Int) + + companion object { + private const val STORED = 0 + private const val DEFLATED = 8 + private const val END_OF_DIRECTORY_SIGNATURE = 0x06054b50 + private const val DIRECTORY_ENTRY_SIGNATURE = 0x02014b50 + private const val LOCAL_HEADER_SIGNATURE = 0x04034b50 + private const val LOCAL_HEADER_MINIMUM = 30 + private const val DIRECTORY_ENTRY_MINIMUM = 46 + private const val END_OF_DIRECTORY_MINIMUM = 22 + + /** The comment that may follow the end record, and so how far back it can hide. */ + private const val MAX_COMMENT = 0xFFFF + + fun open(bytes: ByteArray): ZipArchive? { + val end = findEndOfDirectory(bytes) ?: return null + val count = bytes.readShort(end + 10) + var offset = bytes.readInt(end + 16) + + val entries = LinkedHashMap(count) + repeat(count) { + if (offset + DIRECTORY_ENTRY_MINIMUM > bytes.size) return@repeat + if (bytes.readInt(offset) != DIRECTORY_ENTRY_SIGNATURE) return@repeat + + val method = bytes.readShort(offset + 10) + val compressedSize = bytes.readInt(offset + 20) + val nameLength = bytes.readShort(offset + 28) + val extraLength = bytes.readShort(offset + 30) + val commentLength = bytes.readShort(offset + 32) + val localHeaderOffset = bytes.readInt(offset + 42) + + val nameStart = offset + DIRECTORY_ENTRY_MINIMUM + if (nameStart + nameLength > bytes.size) return@repeat + val name = bytes.decodeToString(nameStart, nameStart + nameLength) + // Directories carry no data and end in a separator. + if (!name.endsWith("/")) { + entries[name] = Entry(localHeaderOffset, compressedSize, method) + } + offset = nameStart + nameLength + extraLength + commentLength + } + + return ZipArchive(bytes, entries) + } + + /** The end record sits last, unless a comment follows it, so the search runs backwards. */ + private fun findEndOfDirectory(bytes: ByteArray): Int? { + if (bytes.size < END_OF_DIRECTORY_MINIMUM) return null + val earliest = maxOf(0, bytes.size - END_OF_DIRECTORY_MINIMUM - MAX_COMMENT) + for (offset in bytes.size - END_OF_DIRECTORY_MINIMUM downTo earliest) { + if (bytes.readInt(offset) == END_OF_DIRECTORY_SIGNATURE) return offset + } + return null + } + + private fun ByteArray.readShort(at: Int): Int = + (this[at].toInt() and 0xFF) or ((this[at + 1].toInt() and 0xFF) shl 8) + + private fun ByteArray.readInt(at: Int): Int = + (this[at].toInt() and 0xFF) or + ((this[at + 1].toInt() and 0xFF) shl 8) or + ((this[at + 2].toInt() and 0xFF) shl 16) or + ((this[at + 3].toInt() and 0xFF) shl 24) + } +} diff --git a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt b/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt deleted file mode 100644 index b700606..0000000 --- a/library/src/jvmShared/kotlin/io/github/lemcoder/mikromarkdown/converters/EpubConverter.kt +++ /dev/null @@ -1,132 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Block -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.model.Paragraph -import io.github.lemcoder.mikromarkdown.model.Strong -import io.github.lemcoder.mikromarkdown.model.Text -import io.github.lemcoder.mikromarkdown.utils.HtmlToDocument -import java.io.StringReader -import java.util.zip.ZipInputStream -import javax.xml.parsers.DocumentBuilderFactory -import org.w3c.dom.Element -import org.xml.sax.InputSource - -public class EpubConverter : DocumentConverter { - private val metaFields = - listOf( - "title" to "Title", - "creator" to "Authors", - "language" to "Language", - "description" to "Description", - "identifier" to "Identifier", - ) - - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "epub" || info.mimetype == "application/epub+zip" - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val entries = readZip(bytes) - - val containerXml = entries["META-INF/container.xml"] ?: return Document() - val opfPath = parseOpfPath(containerXml) ?: return Document() - val opfBytes = entries[opfPath] ?: entries[opfPath.removePrefix("/")] ?: return Document() - val opfDir = opfPath.substringBeforeLast("/", "") - - val (manifest, spine, metadata) = parseOpf(opfBytes) - - val blocks = mutableListOf() - var title: String? = metadata["title"] - - for ((key, label) in metaFields) { - val value = metadata[key] ?: continue - blocks += Paragraph(listOf(Strong(listOf(Text("$label:"))), Text(" $value"))) - } - - for (idref in spine) { - val href = manifest[idref] ?: continue - val fullPath = if (opfDir.isEmpty()) href else "$opfDir/$href" - val htmlBytes = entries[fullPath] ?: entries[fullPath.removePrefix("/")] ?: continue - val chapter = HtmlToDocument.parse(htmlBytes.toString(Charsets.UTF_8)) - if (title == null) title = chapter.title - blocks += chapter.blocks - } - - return Document(blocks = blocks, title = title, metadata = metadata) - } - - private fun readZip(bytes: ByteArray): Map { - val entries = mutableMapOf() - ZipInputStream(bytes.inputStream()).use { zip -> - var entry = zip.nextEntry - while (entry != null) { - if (!entry.isDirectory) { - entries[entry.name] = zip.readBytes() - } - zip.closeEntry() - entry = zip.nextEntry - } - } - return entries - } - - private fun parseOpfPath(containerXml: ByteArray): String? { - val doc = parseXml(containerXml) ?: return null - val rootfiles = doc.getElementsByTagName("rootfile") - if (rootfiles.length == 0) return null - return (rootfiles.item(0) as? Element)?.getAttribute("full-path") - } - - private fun parseOpf(opfBytes: ByteArray): Triple, List, Map> { - val doc = parseXml(opfBytes) ?: return Triple(emptyMap(), emptyList(), emptyMap()) - - val metadata = mutableMapOf() - for (tag in listOf("dc:title", "dc:creator", "dc:language", "dc:description", "dc:identifier")) { - val nodes = doc.getElementsByTagName(tag) - if (nodes.length > 0) { - val text = nodes.item(0).textContent?.trim() - if (!text.isNullOrEmpty()) { - metadata[tag.removePrefix("dc:")] = text - } - } - } - - val manifest = mutableMapOf() - val manifestItems = doc.getElementsByTagName("item") - for (i in 0 until manifestItems.length) { - val item = manifestItems.item(i) as? Element ?: continue - val id = item.getAttribute("id") - val href = item.getAttribute("href") - val mediaType = item.getAttribute("media-type") - if (id.isNotEmpty() && href.isNotEmpty() && isReadableChapter(mediaType)) { - manifest[id] = href - } - } - - val spine = mutableListOf() - val itemrefs = doc.getElementsByTagName("itemref") - for (i in 0 until itemrefs.length) { - val itemref = itemrefs.item(i) as? Element ?: continue - val idref = itemref.getAttribute("idref") - if (idref.isNotEmpty()) spine.add(idref) - } - - return Triple(manifest, spine, metadata) - } - - /** Only the spine's (X)HTML documents carry text; images and styles are skipped. */ - private fun isReadableChapter(mediaType: String): Boolean = mediaType.contains("html") - - private fun parseXml(bytes: ByteArray) = - try { - val factory = DocumentBuilderFactory.newInstance() - factory.isNamespaceAware = false - factory.isExpandEntityReferences = false - factory.newDocumentBuilder().parse(InputSource(StringReader(bytes.toString(Charsets.UTF_8)))) - } catch (_: Exception) { - null - } -} diff --git a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt index 0bba043..2f4093a 100644 --- a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt +++ b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/ArchitectureTest.kt @@ -70,9 +70,17 @@ class ArchitectureTest { fun `helpers under utils stay internal`() { val utils = { name: String? -> name?.contains(".utils") == true } - // internal or private: anything but part of the published API. - scope.classes().filter { utils(it.packagee?.name) }.assertFalse { it.hasPublicOrDefaultModifier } - scope.objects().filter { utils(it.packagee?.name) }.assertFalse { it.hasPublicOrDefaultModifier } + // internal or private: anything but part of the published API. Only top-level declarations + // are checked — a companion inside an internal class is already unreachable, and demanding a + // modifier on it would be noise. + scope + .classes() + .filter { utils(it.packagee?.name) && it.isTopLevel } + .assertFalse { it.hasPublicOrDefaultModifier } + scope + .objects() + .filter { utils(it.packagee?.name) && it.isTopLevel } + .assertFalse { it.hasPublicOrDefaultModifier } scope .functions() .filter { utils(it.packagee?.name) && it.isTopLevel } diff --git a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt index 8541451..0d9432b 100644 --- a/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt +++ b/library/src/macosMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt @@ -1,6 +1,7 @@ package io.github.lemcoder.mikromarkdown import io.github.lemcoder.mikromarkdown.converters.CsvConverter +import io.github.lemcoder.mikromarkdown.converters.EpubConverter import io.github.lemcoder.mikromarkdown.converters.HtmlConverter import io.github.lemcoder.mikromarkdown.converters.JsonConverter import io.github.lemcoder.mikromarkdown.converters.MarkdownPassthroughConverter @@ -10,12 +11,13 @@ import io.github.lemcoder.mikromarkdown.converters.XmlConverter /** * A [MikroMarkdown] with the converters that need no platform library. * - * EPUB and PDF are still JVM-only; everything else the library converts is here. + * PDF is the only format still missing here; everything else the library converts is shared. */ public fun MikroMarkdown(): MikroMarkdown = MikroMarkdown(SignatureMimeDetector).apply { register(MarkdownPassthroughConverter()) register(HtmlConverter()) + register(EpubConverter()) register(CsvConverter()) register(JsonConverter()) register(XmlConverter()) diff --git a/scripts/optbench.py b/scripts/optbench.py index 9cbda99..a745025 100644 --- a/scripts/optbench.py +++ b/scripts/optbench.py @@ -56,7 +56,7 @@ def verify(): continue if subprocess.run([str(JVM), str(fixture)], capture_output=True, cwd=REPO).stdout != baseline.read_bytes(): problems.append(f"jvm {name}") - for name in ("test.csv", "test.json", "test.xml", "test_blog.html", "test_wikipedia.html"): + for name in ("test.csv", "test.json", "test.xml", "test_blog.html", "test_wikipedia.html", "test.epub"): fixture = FIXTURES / name jvm = subprocess.run([str(JVM), str(fixture)], capture_output=True, cwd=REPO).stdout native = subprocess.run([str(NATIVE), str(fixture)], capture_output=True, cwd=REPO).stdout From 14cb388a9281769084e8e1359aa922d6e0a4333f Mon Sep 17 00:00:00 2001 From: mikolaj Date: Tue, 18 Aug 2026 00:58:19 +0200 Subject: [PATCH 08/15] PDF through pdfium on native, in its own module MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A :pdfium module binds pdfium with cinterop and converts a PDF on the native CLI. The release is pinned by checksum and unpacked at build time; nothing binary is committed. The library does not depend on it — the CLI registers PdfiumConverter itself, so the module can be deleted without touching anything else. Two things cost time and are worth recording. The published dylib calls itself ./libpdfium.dylib, which the loader resolves against the working directory rather than the binary, so the unpack step rewrites the id to @rpath. And Kotlin/Native does not carry a klib's linker options through to the binary that links it, so the consumer has to name pdfium itself. Against PDFBox on the same document pdfium keeps 96.2% of its tokens, and the difference is in pdfium's favour: PDFBox leaves flexibil, firming and gramming as fragments where pdfium plus de-hyphenation produces whole words. Two defects remain and one fix covers both. pdfium emits U+FFFE for a glyph with no Unicode mapping, which here is always a hyphen at a line break; the document's vocabulary decides whether to join or keep it, which gets four words right and two compounds wrong, and cannot do better because a broken word's halves are only in the text because the break put them there. Separately a page arrives as one run of text, so a PDF renders as a single paragraph where PDFBox produced seven. FPDFText_GetCharBox answers both: a hyphenation hyphen ends a line, and the gaps between lines are the paragraphs. plainTextBlocks moves from utils to model and becomes public, since a converter in another module needs it and the architecture rules keep utils unpublished. Co-Authored-By: Claude Opus 5 (1M context) --- cli-native/build.gradle.kts | 18 ++- .../github/lemcoder/mikromarkdown/cli/Main.kt | 4 +- docs/common-converters-plan.md | 30 ++++- library/:memory:.ses | 2 + .../mikromarkdown/converters/PdfConverter.kt | 2 +- .../{utils => model}/TextBlocks.kt | 13 +- .../mikromarkdown/converters/PdfConverter.kt | 2 +- pdfium/build.gradle.kts | 89 +++++++++++++ .../mikromarkdown/pdf/PdfiumConverter.kt | 124 ++++++++++++++++++ pdfium/src/nativeInterop/cinterop/pdfium.def | 4 + scripts/__pycache__/benchmark.cpython-314.pyc | Bin 0 -> 18469 bytes settings.gradle.kts | 2 + 12 files changed, 275 insertions(+), 15 deletions(-) create mode 100644 library/:memory:.ses rename library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/{utils => model}/TextBlocks.kt (88%) create mode 100644 pdfium/build.gradle.kts create mode 100644 pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt create mode 100644 pdfium/src/nativeInterop/cinterop/pdfium.def create mode 100644 scripts/__pycache__/benchmark.cpython-314.pyc diff --git a/cli-native/build.gradle.kts b/cli-native/build.gradle.kts index befa74d..f554494 100644 --- a/cli-native/build.gradle.kts +++ b/cli-native/build.gradle.kts @@ -2,7 +2,15 @@ plugins { alias(libs.plugins.kotlinMultiplatform) } kotlin { macosArm64 { - binaries.executable { entryPoint = "io.github.lemcoder.mikromarkdown.cli.main" } + // Kotlin/Native does not carry a klib's linker options to the binary that uses it, so the + // consumer names pdfium itself. Worth turning into a shared convention if a second + // consumer appears. + val pdfiumLib = rootProject.layout.projectDirectory.dir("pdfium/build/pdfium/mac-arm64/lib").asFile + + binaries.executable { + entryPoint = "io.github.lemcoder.mikromarkdown.cli.main" + linkerOpts("-L${pdfiumLib.absolutePath}", "-lpdfium", "-rpath", pdfiumLib.absolutePath) + } compilerOptions { // Worth about 8% on large inputs and nothing on small ones. Measured, not assumed: @@ -15,5 +23,11 @@ kotlin { } } - sourceSets { macosArm64Main.dependencies { implementation(project(":library")) } } + sourceSets { + macosArm64Main.dependencies { + implementation(project(":library")) + // PDF is a separate module by design; the CLI is the thing that opts in. + implementation(project(":pdfium")) + } + } } diff --git a/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt b/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt index 485a19c..665be14 100644 --- a/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt +++ b/cli-native/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/cli/Main.kt @@ -2,6 +2,7 @@ package io.github.lemcoder.mikromarkdown.cli import io.github.lemcoder.mikromarkdown.MikroMarkdown import io.github.lemcoder.mikromarkdown.MikroMarkdownException +import io.github.lemcoder.mikromarkdown.pdf.PdfiumConverter import kotlin.system.exitProcess /** @@ -18,7 +19,8 @@ public fun main(args: Array) { exitProcess(2) } - val mikroMarkdown = MikroMarkdown() + // PDF lives in its own module; the CLI opts in, the library does not depend on it. + val mikroMarkdown = MikroMarkdown().apply { register(PdfiumConverter()) } for (path in args) { try { diff --git a/docs/common-converters-plan.md b/docs/common-converters-plan.md index 37874e6..ac1e500 100644 --- a/docs/common-converters-plan.md +++ b/docs/common-converters-plan.md @@ -145,9 +145,33 @@ Done, byte-identical, and the JVM-only source set is gone with it. `ZipArchive` the central directory and inflates through korlibs; the container and package documents go through Ksoup's XML mode; chapters reuse Phase 1. EPUB now converts on native. -### Phase 3 — PDF, the `:pdfium` module -Text first, then images and placement. Independent of the OOXML work, so it can run in parallel or -first if iOS PDF matters more. +### Phase 3 — PDF, the `:pdfium` module — text done, geometry outstanding +The binding works: the release is pinned by checksum, unpacked at build time, and the native CLI +converts a PDF through `PdfiumConverter`. Two notes on getting there — the published dylib names +itself `./libpdfium.dylib`, which the loader resolves against the working directory, so the unpack +step rewrites it to `@rpath`; and Kotlin/Native does not carry a klib's linker options to the binary +that links it, so the consumer names pdfium itself. + +Against PDFBox on the same file, pdfium keeps 96.2% of its tokens and the differences are pdfium +reading *better*: PDFBox leaves `flexibil`, `firming`, `gramming` as fragments where pdfium plus +de-hyphenation produces whole words. + +**Two defects remain, and geometry fixes both.** + +pdfium emits U+FFFE where a glyph has no Unicode mapping, which in a typeset paper is the hyphen at +a line break. Dropping it fuses real compounds (`chat-optimized` → `chatoptimized`); keeping it +splits real words. The document's own vocabulary decides today, which gets four joins right and two +compounds wrong — and cannot do better, because the halves of a broken word are only in the text at +all because the break put them there. **The real discriminator is that a hyphenation hyphen sits at +the end of a line and a compound hyphen does not**, which `FPDFText_GetCharBox` answers directly. + +The same call answers paragraphs. pdfium returns a page as one run of text, so a PDF currently +renders as a single paragraph where PDFBox produced seven. Character boxes give the line breaks, and +the vertical gaps between them give the paragraphs. + +### Phase 3b — PDF images +Unstarted. Page objects, the two storage cases, the PNG encoder for raw pixels, and placement by +bounds — all as described above. ### Phase 4 — FB2, if wanted Book's Story parses FB2 with the same walker it uses for HTML, through Jsoup's XML mode. Ksoup has diff --git a/library/:memory:.ses b/library/:memory:.ses new file mode 100644 index 0000000..95ddb84 --- /dev/null +++ b/library/:memory:.ses @@ -0,0 +1,2 @@ +1787007442705 +2b657774-881a-47ed-9141-c15c8c8f1ab7 diff --git a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt index 4b13531..3e77b98 100644 --- a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt +++ b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt @@ -5,7 +5,7 @@ import com.tom_roush.pdfbox.text.PDFTextStripper import io.github.lemcoder.mikromarkdown.DocumentConverter import io.github.lemcoder.mikromarkdown.StreamInfo import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.utils.plainTextBlocks +import io.github.lemcoder.mikromarkdown.model.plainTextBlocks public class PdfConverter : DocumentConverter { override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TextBlocks.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/TextBlocks.kt similarity index 88% rename from library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TextBlocks.kt rename to library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/TextBlocks.kt index 787f028..e80aa3f 100644 --- a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/utils/TextBlocks.kt +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/TextBlocks.kt @@ -1,16 +1,15 @@ -package io.github.lemcoder.mikromarkdown.utils - -import io.github.lemcoder.mikromarkdown.model.Block -import io.github.lemcoder.mikromarkdown.model.Paragraph -import io.github.lemcoder.mikromarkdown.model.Text +package io.github.lemcoder.mikromarkdown.model /** - * Turns extracted plain text (PDF pages, speaker notes, …) into paragraph blocks. + * Turns extracted plain text into paragraph blocks. + * + * Public because it is what any text-extracting converter needs, including ones outside this module: the `:pdfium` + * module builds its documents with it. * * Blank lines separate paragraphs; soft-wrapped lines inside a paragraph are rejoined, so the Markdown does not inherit * the source layout's line breaks. */ -internal fun plainTextBlocks(text: String, reflow: Boolean = true): List { +public fun plainTextBlocks(text: String, reflow: Boolean = true): List { // Form feeds mark PDF page breaks; treat them as paragraph boundaries. val normalized = text.replace("\r\n", "\n").replace('\r', '\n').replace('\u000C', '\n') val vocabulary = if (reflow) wordsIn(normalized) else emptySet() diff --git a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt index 58a536d..55a1b27 100644 --- a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt +++ b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt @@ -3,7 +3,7 @@ package io.github.lemcoder.mikromarkdown.converters import io.github.lemcoder.mikromarkdown.DocumentConverter import io.github.lemcoder.mikromarkdown.StreamInfo import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.utils.plainTextBlocks +import io.github.lemcoder.mikromarkdown.model.plainTextBlocks import org.apache.pdfbox.Loader import org.apache.pdfbox.text.PDFTextStripper diff --git a/pdfium/build.gradle.kts b/pdfium/build.gradle.kts new file mode 100644 index 0000000..21d6331 --- /dev/null +++ b/pdfium/build.gradle.kts @@ -0,0 +1,89 @@ +import java.security.MessageDigest + +plugins { alias(libs.plugins.kotlinMultiplatform) } + +/** + * PDF support, kept in its own module so it can be dropped or extracted whole. + * + * pdfium is a prebuilt binary rather than a source dependency: the release is pinned and verified by checksum, unpacked + * into the build directory, and never committed. + */ +val pdfiumRelease = "chromium/8009" + +val pdfiumArchives = mapOf("mac-arm64" to "b1f2f17c7432a9942514dda5094ee9822c743bdfd07e7187725efbd34fde941f") + +val pdfiumRoot: Provider = layout.buildDirectory.dir("pdfium") + +val downloadPdfium by tasks.registering { + description = "Downloads and unpacks the pinned pdfium binaries." + outputs.dir(pdfiumRoot) + + doLast { + for ((platform, sha256) in pdfiumArchives) { + val target = pdfiumRoot.get().dir(platform).asFile + if (target.resolve("lib").exists()) continue + + val archive = pdfiumRoot.get().file("pdfium-$platform.tgz").asFile + archive.parentFile.mkdirs() + if (!archive.exists()) { + val url = + "https://github.com/bblanchon/pdfium-binaries/releases/download/" + + "$pdfiumRelease/pdfium-$platform.tgz" + logger.lifecycle("downloading pdfium $pdfiumRelease for $platform") + uri(url).toURL().openStream().use { input -> + archive.outputStream().use { output -> input.copyTo(output) } + } + } + + val digest = + MessageDigest.getInstance("SHA-256").digest(archive.readBytes()).joinToString("") { + (it.toInt() and 0xFF).toString(16).padStart(2, '0') + } + check(digest == sha256) { "pdfium-$platform.tgz checksum $digest, expected $sha256" } + + target.mkdirs() + providers + .exec { commandLine("tar", "xzf", archive.absolutePath, "-C", target.absolutePath) } + .standardOutput + .asText + .get() + + // The published dylib calls itself ./libpdfium.dylib, which the loader resolves + // against the working directory rather than the binary. Rewrite it to @rpath so a + // linked executable can find it wherever it runs from. + val dylib = target.resolve("lib/libpdfium.dylib") + if (dylib.exists()) { + providers + .exec { commandLine("install_name_tool", "-id", "@rpath/libpdfium.dylib", dylib.absolutePath) } + .standardOutput + .asText + .get() + } + } + } +} + +kotlin { + macosArm64 { + val platform = pdfiumRoot.get().dir("mac-arm64") + compilations.getByName("main").cinterops.create("pdfium") { + defFile("src/nativeInterop/cinterop/pdfium.def") + includeDirs(platform.dir("include")) + // The archive ships a dylib, so the binary carries an rpath to find it at run time. + extraOpts("-libraryPath", platform.dir("lib").asFile.absolutePath) + } + binaries.executable { + entryPoint = "io.github.lemcoder.mikromarkdown.pdf.main" + linkerOpts( + "-L${platform.dir("lib").asFile.absolutePath}", + "-lpdfium", + "-rpath", + platform.dir("lib").asFile.absolutePath, + ) + } + } + + sourceSets { macosArm64Main.dependencies { implementation(project(":library")) } } +} + +tasks.matching { it.name.startsWith("cinteropPdfium") }.configureEach { dependsOn(downloadPdfium) } diff --git a/pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt b/pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt new file mode 100644 index 0000000..63001c0 --- /dev/null +++ b/pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt @@ -0,0 +1,124 @@ +package io.github.lemcoder.mikromarkdown.pdf + +import io.github.lemcoder.mikromarkdown.DocumentConverter +import io.github.lemcoder.mikromarkdown.StreamInfo +import io.github.lemcoder.mikromarkdown.model.Document +import io.github.lemcoder.mikromarkdown.model.plainTextBlocks +import kotlinx.cinterop.ExperimentalForeignApi +import kotlinx.cinterop.UShortVar +import kotlinx.cinterop.addressOf +import kotlinx.cinterop.allocArray +import kotlinx.cinterop.get +import kotlinx.cinterop.memScoped +import kotlinx.cinterop.usePinned +import pdfium.FPDFText_ClosePage +import pdfium.FPDFText_CountChars +import pdfium.FPDFText_GetText +import pdfium.FPDFText_LoadPage +import pdfium.FPDF_CloseDocument +import pdfium.FPDF_ClosePage +import pdfium.FPDF_DestroyLibrary +import pdfium.FPDF_GetPageCount +import pdfium.FPDF_InitLibrary +import pdfium.FPDF_LoadMemDocument +import pdfium.FPDF_LoadPage + +/** + * PDF text through pdfium. + * + * Not registered by the library's factory: PDF costs a native library, so a caller asks for it. + * + * ``` + * val mikroMarkdown = MikroMarkdown().apply { register(PdfiumConverter()) } + * ``` + */ +public class PdfiumConverter : DocumentConverter { + + private companion object { + const val UNMAPPED_GLYPH = '\uFFFE' + val WORD = Regex("[\\p{L}]{2,}") + val HYPHEN_BREAK = Regex("\\p{L}+\uFFFE\\p{L}+") + } + + override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { + return info.extension == "pdf" || info.mimetype == "application/pdf" + } + + @OptIn(ExperimentalForeignApi::class) + override fun parse(bytes: ByteArray, info: StreamInfo): Document { + val text = StringBuilder() + + FPDF_InitLibrary() + try { + bytes.usePinned { pinned -> + val document = FPDF_LoadMemDocument(pinned.addressOf(0), bytes.size, null) ?: return@usePinned + try { + for (index in 0 until FPDF_GetPageCount(document)) { + val page = FPDF_LoadPage(document, index) ?: continue + val textPage = FPDFText_LoadPage(page) + if (textPage != null) { + text.append(pageText(textPage)) + // A page break reads as a paragraph break to the text blocks. + text.append('\n') + FPDFText_ClosePage(textPage) + } + FPDF_ClosePage(page) + } + } finally { + FPDF_CloseDocument(document) + } + } + } finally { + FPDF_DestroyLibrary() + } + + return Document(blocks = plainTextBlocks(text.toString().joinHyphenatedWords())) + } + + /** + * pdfium emits U+FFFE — a permanent non-character — where a glyph has no Unicode mapping, which for a typeset + * document is nearly always the hyphen at a line break. + * + * Dropping it always would fuse real compounds: "chat-optimized" became "chatoptimized". So the document decides. + * If both halves appear elsewhere as words in their own right the hyphen was real and is restored; otherwise the + * halves are two pieces of one broken word and are joined. + */ + private fun String.joinHyphenatedWords(): String { + if (indexOf(UNMAPPED_GLYPH) < 0) return this + + // The halves of a broken word must not vouch for themselves: "con" and "firming" are only + // in the text because the break put them there, so every break is cut before counting. + val unbroken = replace(HYPHEN_BREAK, " ") + val vocabulary = WORD.findAll(unbroken).map { it.value.lowercase() }.toSet() + val out = StringBuilder(length) + for (index in indices) { + val char = this[index] + if (char != UNMAPPED_GLYPH) { + out.append(char) + continue + } + var wordStart = out.length + while (wordStart > 0 && out[wordStart - 1].isLetter()) wordStart-- + val left = out.subSequence(wordStart, out.length).toString().lowercase() + val right = substring(index + 1).takeWhile { it.isLetter() }.lowercase() + if (isRealCompound(left, right, vocabulary)) out.append('-') + } + return out.toString() + } + + /** A hyphen the author wrote, rather than one the typesetter added at a line break. */ + private fun isRealCompound(left: String, right: String, vocabulary: Set): Boolean = + left.isNotEmpty() && right.isNotEmpty() && left in vocabulary && right in vocabulary + + /** pdfium writes UTF-16 into a caller-supplied buffer and counts the terminating NUL. */ + @OptIn(ExperimentalForeignApi::class) + private fun pageText(textPage: pdfium.FPDF_TEXTPAGE): String { + val count = FPDFText_CountChars(textPage) + if (count <= 0) return "" + return memScoped { + val buffer = allocArray(count + 1) + val written = FPDFText_GetText(textPage, 0, count, buffer) + if (written <= 1) "" else CharArray(written - 1) { Char(buffer[it].toInt()) }.concatToString() + } + } +} diff --git a/pdfium/src/nativeInterop/cinterop/pdfium.def b/pdfium/src/nativeInterop/cinterop/pdfium.def new file mode 100644 index 0000000..89d16f8 --- /dev/null +++ b/pdfium/src/nativeInterop/cinterop/pdfium.def @@ -0,0 +1,4 @@ +# Bound by cinterop for native targets and, later, by the Konan plugin's JNI generator for the JVM. +headers = fpdfview.h fpdf_text.h +headerFilter = fpdf*.h +package = pdfium diff --git a/scripts/__pycache__/benchmark.cpython-314.pyc b/scripts/__pycache__/benchmark.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..9eebc1b75b4e8a26590df9bb0ebcde4b08f5202e GIT binary patch literal 18469 zcmbt+ZB!dqmSB}s`j+?*LVTM7VL+IV*v8la8^^(5V+_UxmXlZ@3rQH+LX@jY{!mD_ z`s1A7q|aD(J8hg~*5q{ei8Gx&q0i}=CbP3Wp6;35?Ah5f!h%MrFemPG&hE}R`vbR~ z&B^}S`(9N^GJ)9XdA6!vy^s6eyYJrn?z`_@w&hri1cdEdzjJn_i6H)nK9r!5BILsa zNf0rDAp~NSV8~wbuu4!>qPVJ8EvWIE5-9xE2pas>3R?Wu2|E1N3wr!E2nKkodyR)p zf(gT*dd-I|g5_|Ikb}Q9y}5_;guF^(B(I7X$!~?WQ5f7Ry|bUK^2cCpcB&NLC{3_E zLomgYq)<4b8EpJ|ohpLSD(wkHqiUg;(G?MlzK96r2_++zLIO${AjR0Kl4>hG5K0*n zq?>WcvQ42|($LE{Et8WjQ^Di{%sdQp3zH9VD~?w(1rWF4IL#D7+>YZ_Bh^gNo)-wh zpF z8E-Jm3-l!91qGB#*AHG6CZ@tp+8g%6@3Mc&N7pkEUT`)WjVE~TnLxXdhS1z{hV%ME zfs1szFBEJ?V7#Hw!5}Z-Y!*Y=N%KA~$O?SRSRm}1K+QC>m+4bYP2<5!LWB$O^ubqKL7t9; zy%!+UI~J0AH;G1!n!{tFp@%JFkzmN*BKNZerv$_3jr$w+3c(8jG%y?>xk*Zr>y1I# zNw05;Kfq14wYG49P{7Lvx-SKM5dk%e0Le9)&jl_8FwiWtBB-Y`->2!RFgbmYRNtXVh(s2n+sVMtTZ#K8<87{Wx5^Ebh)3YS}?tQNt`oe2m^ms$dsypwDw zz_<9O!WROZK-c@Az!Z(gk3k|hS)r*(9u3{pBv+>8x=#AD*bRhg8c_fZNbqu(X>VvM zEDccZTzJZlQHKhK*-4u9`pyBNG{Q_x!QkXc^3nsSoJ0|HXv*skhR@JmI_zbG^yHL3 z5(+eK)IWZT4hlRi1Sg?9PY1)kPz2yez+mt~gf$xb0|FQH@iZIYXdopvB1mKk0}%pY zf#w1}z$xHw>RccUv~`9H09nB#j9sQX_wAqqGSLYWUI8JQms4P{(0*VNdVu@J3tYs9 zC|5?{L;w$ir(0-D2PhH(CV(ynCZ&(?IrI)r0!C28hmbH1>=2M3;%YR&2?-MjC_jM+ ze`;KUM{`pbc^QwssZeAx3;-f9ph>Ubn*e|a=cp3_4)<^3@|j>Dtn`kicW$G{Lf-JX zrV#KH%|*g|BQ0>iD+u=&C!m)+%oa>l1X#}JMZCraS-ERGi;$EFV*wZ#^lSi6sRRs( z3DgJPI~ky1h9W#Y39JDa^3z9L{WORLJVQ=WGcDqML zJ*Quue&stO+k9v>kyjJ~#2=xC&xbTbVuXtrRqgp9NKZ&BCytXKCqYs&YKCe-@>26} zNal-_5V$0WYM5kE%LQPX!~aju2ZU3QF!CM&ebSCkae#RqwNfqaYB|9NIKBm>%~Z&H zwuR?>Esx3CmW{-pX!dwyOawe0E)PJ#;!N;c;1MIfB-CUN`2auT1#9>{TP%G)gc1ZE zC5oV+g&!W}e}PDhaHBLF2IVzO`9^P8-(ZDz5v@=I1(tAAQVdpLof6BuUBsXv^QFYr zuQ++c2{NL3hKQ(oiU20*HYp`tq_QfFs`uO=h%*@XeT4+!BG0M%N(jP0Y=L?#>40Y? zjs*#)Dq;f&r+z#4SuED{S?smXVjq0=z1x3`#YRMuc4|cx7l0{>jESbC`{D!G|16#K-BrBCMSWFMO0);2mq<)`UB(MNJ#LEdws$b zcbPLI049(i5tg0egn<9w!*84(cZUTQ=d=*z5YnQ>Aj?Am)HAtbjr2Gy6v7B+fovzq zf$l>{%!elvGmlsxjj%bJtvXOaG{uOcay2Cn&|{WbL$5^Dv-p$ChkRc)qo7U!ks2PV zAQcIXK}c910?#55;vzR$F-uok0ISc_FL z1gmx<{NUh5D8fN0kw^5r0gOqxk)*+4(5l2T?~E$RYTSqcaTv5KvCO+_6TG;Ap{0;v z{AMy+69;jt9{m+V?UuQK)jn;XT$)C_m671HI+u2^RB0{q9wqi%1bo39loN~wi4+F` zCCf7{N?6us))t-y+g&avDS=nA~3h6PvRVxzP+Ax#3>0KGi|ix#ZCPQ6I+fzUXIwo@X7 zmTOUsGD#1}N3;-&6oMw|0wFKU2mCxDx!PK3g_k&L7pf}j;LXq?W~6_H#2Aq>McFp^ej`kJDbBm7Wl8)!^`d4~fi#@I-hbvW3o-miM=Gx*X zUZ0+!QaRQc^=iKThVYZJnFFh~;uYK0d$z4fTis0eYC*-#&g-42Jljq4HS^EZa|eHF zn0IUmcE0jEjSdpt-i zdOV`hQOc%!C5x(Nl2!FPo>)I^zev}IrL!C`$k>h{2r5R!s-B23&5PCP`kC?Q zHW*KO1dRT%Sy4Z+4TDSlXU&RQ!B8`aXXWXjWJX~G8&KgS$(WQSMu7*jlNpWqvC&XZ z9F4_A{aK@-l-A|ZJROwGXmT=(`GzBU620Xr?Xwz}Mw#<7U6>Ox*vQMqh8yWbI527J z+)fp!5TIeQT3AeVN`B_8s4(dk@Cf=q`^w_CH0`pQN}^wdH2){!-J0pOA7lk6T^bpa z>Cs60In0p--;6C6C<9J=v;tdxHnRnZmd7kd0~`&*jBbH-@EZ(XqA_FG5!GPHiI%}0 z8VnWKB(~nH6FrbI75$mUB~dHcrlQY&y<@TwsvQ=pbIf`xRe#!o1-i5Y4QqG=E zswad=QO)|tMG8iKNu&ZSY(ObU4S`C<3XpOs#9wk6K$q(tJO;x8vt`(iS9MVvxCHwx zUeevP{6s_uhC~e*Uwjjy^}xW32ak1mKmj~4ICzX1?mme3LrxtiX_8G!R0F|qeb5aK zEwrLVntV_{&@6~X>Foh5i3T6+;eudG3Gk@fvgJnF_gZj+9vLX-g?zkj|1E4KLANX4 z5!+^t&)1}^#W$O-HQj8v-g4#nMBUhD#RHE_T7&VC zS`VEnB??OtWyh9n$7222uu4)j+gED#EY|EE-Q>ue9&S zC7jz({$mOv2v$iQqam`Hf|xOsNlGHpqcA;7>Y`2Bw+F*QBk;NOc(U5~7XT!^N4Jyo z0|Fj#-x_))w(P*4lKLoHJD_~p6avS2-drH&ZPE{`h_>4Oqj z%KjEDW$yI8f1^FSE4^GPi<YZv>h&9o}3^uUxbiHIaZ^Wi<>A8`? z+aDYs2V*Ne4hK6~R&QtXK#xdH4}-uoViO?Rq4A<&_`vf=yFJXYlcENj$~k16K)YbH z5eGvh$kBt{M|=Bwhr1b6%&C|5^IEAlB83d^qIQf!2L{MKhHUS=s1JrcxP(ZdhD41o z5DM|qKKe0nYr>`Q&X4ELLwlIo??EI++^^V@pti(%;4w1{`{EdEVr{X*X#1OcE%*0% zRWsD8$#!-BmHo4=D<=A$iC(o8%~H3Gw~TL@SBe`JiyQ9jOcuAS6z^IrMjP8$sS zzUw^?)P${OO-tk#&XR8!;?ySD#T7G$Q})u=4n3$M@~hXX3A1(9o-kD=sLBkJDvbB7 zgvxuG4TVaMKj_0&N-R^JEKc49-15jOgiL-~Qu_cs;*K@+nr+5knT|p+91tggx=KFO zVrjwJ92s-=aQ9y05jb{TB6gU$5Un0dML> zBR(_ofOg48weIlCt&OjUCgs@3bCHjl>i17NPtk5+WLu4B3QV%XWe?IgQSnpm5qG_) z2PMmc^rL74D!aFSf7KY*IbGG%G~MDn`22Yx^o1yF~p`EvOkKt z@nF~whjyT;+MUu{Z*;w_x@~;Zm~adx7+13RWYYG+ znn7dMt`QoIHbXxOm3%8g=XmJlH<_}rHDw+RR-t1PqGNc(yVlTab^w26`T=EeK(<{> z$J3aOGgJU`Ecf*y z=gxtLi&t|Vy;KbsN zqzyPR&9jNE&)+?mEbUF&kEHauHwtbrH($E`QiASS7)};GpUm${>GN(lZj9YLd;M&p zYWIRIY41#0_h<9+x%t*)*`8$4vkS+Ui#nlvycQaW`=Lzy!dkuoF|uJJMn+Q#WcBUr zr08fscp^jp20-(6jFB&i36iutz|R?<^p=qeMzLQx;SjZrb16XB|U|ScI zz78Uj^SYqGQ_8R^7g(=Os3WRobV9wKAkOa`gcIoXL8(_KXUJ#OAcnz`iRNxy7)0yT zy0C$L7iO`X=-Y>&45<1JNJ=oPg5#&`F(P5VE~NWUf$b3)_16VP3;%SPxyl@8DI-BP zKiKRrvfYke2LL{mvC1|yeSa=vfNf3n*Grl~%v356W$T6~<(NdvY$Fnb@Ir zgT+t^gY`-*^JXlWzV{7Fcwur_y?dKd2HlTCBT&YZ#)!dX7^Ia_nRj|Lop>ulF^}0b zS)Ih6t_XZ8y{sf!2-b{fuIC-Vyd{G% z#_FgjO>&~xy-o|flzqaLSkBBCC%xon;*cU+=Gprn&M8~1`K*sOnyZO3EW zy%8Q=X6!AzEa}Kp0IhD>p;KlT=6_@O#ZN3z`NR_0Fo32{ zSR<+yB#+=kn6FHgZU(2BfO>G?kL%f>lz9-H z>8=Y(8+k;xx~GpIFy#~it>q|uB}+s&A8_S-GlUcN6Gag<9v?_a49U(Gx$JI`gHmu6 zF@=p;QcuAj^b}4Fa8HX_sFT3~LxVIZ9t8QlK4f~X#1v?3Hf5w|V3b66k^I0nVDKn; z!bnO_ltt9|ZWD|fkEE0>bCtneGB9Y!7y(J{Duw$@uo|JBN7QT?TiS1MgS3}|(l54r z6AG(x6(T91^!wjIiPcYp;m9iSgn8^%^*kewx`N%}+Vc3EvAOOpWrP{-XDi*^8>G^% zVpm1>jI&j)a)q+|4m1~8Tnu@Q@}>pBQhJAMC?xmtIxG8!hwRkL0aKGr_07%1R+bvr0_Yd__} zP}r@TP;c=d$cyy>&ZO%HNLPiK7$6xZ#HvUAk4fJK3(DCBIS!>v)`-)qiIV54WE!)_ zot56SNqX}p=`EY2KeI{t_D#}vY?w|nJ2y#h-6Vb2hUr`J2-^nIm&X(u0O??RoR>CQ z6C$&`IEv`tDp`$QP+StXEm{ZCj6E7({Y&c*;+>MCV+XQwFAPnHhw3G=$4 zR-Ijot##EZHD$SZOqohotNoBaXjIPiBf@EQ3~)BH z4f59UD=xo|$8g#@KEi41cnGJh;~%DNNBV|BCDHd^zmYo9D*PYu}d8!;hH!C2o4Dxx`7W}Ry98(l7YuAjT`o!0k4WnEY~R(O@hx<^WM-D zIF#_6&FPE;RK};7{{|gPe={I?E$h0lq;Z*`Ejj1Dqt z^uT4zu|TtMNrgqd1Ns&yNP^7!OW05x#-G09YA|gn*9P6<3@3&YiRmd|G%XoZ5kN4g5#G zQ{ZKD5gcn|w;SYOLxXQc#+3y1wi!IvcpjlccH4n|hG{=Hg?+-HS>)Q&yfKj=r8`X| za3}hFK8R~AysKq007DpBdir4Ngx(A=6($FAKr-m5xj29jNuEO+6&_I^*gUbxle-Qd z+>hb$CzOTq@uU}bE!OOwJ^$PMg#&Nr|3m(*{AA7UuducB`}Sfos&+W&=wD1Lo<-A& zQ_?gz-vrM`0&wSj>@si}ycG*FM3KoDC1H~=yk$qCX~o|Nz#tbPsEdy((_hHzN1A*n ziadU3Tndh5Xml(C6~I4fI=XMV365m`+}{}(I)t_Tn1`Y&x<0C#DA~NnI!alNZa>&rdA<7pu_|O77&4}6y-cTgKb7-rCP5$^8 zPSgMuPle$kx)*Lkot=W~NZJUC?jj*?ok=GgTHvul^Jui(e?VZ#aZv{s)IF%JTM=>Ud8D-ws+pWncb|8F{c_F^eqvDDnN)Fdsn==PLxg(_O4ie}C4`+ilE zcy@ew$9S^&OoA#}q9!uAy7vp-A5OG(FE@23op3(^au20U`75UKMN@gwv?X@%^W4H4 z{@K>s`)=)nJNPU1?Thy9N&C)ZZforD$7POWE^}*7-23k3cP_vC@(prf``@>H*tXJf zbg|>;M?)(e$8VBx^46Z%;Y2R8nq!T(-rRF-&+B2h00H;S`oGsdTXV-2>tCXpKhG(9 zeJXb76KlzgHdR)2yYqJEZ09xIe91!F-*nSu+z=+u}OpGm+F+fRBT%e9cajY@e@A6zziX zQHEEKe(z|!{cYb|{acf>o|5|9oX_8Ntf@)sA@UJH zY7Ue4<%-rN$|eJWI@T9(_KS~Y|6$+3WL5WvCl|=Ms&{MOseQlh7tVXtJ3oXQdx@&< zk2?_=eRyUe|4!8hweQu=SN~PRhi76(63(9c<&N9^xB71%17y!oGiN`|E%_KO_U%5f zvis2D?n6tvdzT82Ebi`oPq=e_zAEm0+YXak`*TCGq~V^u0WxNci-kv4^NSaYyXKCp z)a+WU*|k)&J6YTX=Y%ub)#tkI9{F8vf*D$RZfLIU-M#PZ{aNRdZSOmsw_0bn&wA&N zFI4@~b@O=q_&c3&2)Xp4JNe@1jN!gC!*PDLb+$I{UeZ^~a~@}w^u=-8 za)m_U@RI&Ss%qOx)%L}z?JHF~7tIwjeY3VyLD8)CjlDAmSMy3{Ykr&yo-aUb)Q>GQ z2U6DZ*;C6_=S#c_q4WK&dBZOTeqY%B(a=oK=itH;{qg>}<4Xm# zkJN;{WVYaK{;kOEm*0GOp?#&JZ?U6qsiS|Xacr@p|Cg=v+vmNB<4nTkT|Dmn;P{>6 zka`dDDt*f(zNNyzYGcd1`WJ`S)P$wretuy*_y)N0Ea%tFJfF%fot;|BZA#JAb311* z&2jHL?)c_+EST>K%X|8kclIS4fPoXv`ghilho`arYbh6ILK*)L6<I#ME-ofTaNS``bhsz=k6-$%n3Q&bu#gSf7umCoEVo=#)DGiKD0fuT;NDp zYt}TV@%uG(a-`mQr}01LChGRc-+Mab$liSmlgoRLCOZ1%g#JM}0#i6NvV43rG3b#~ zJY#aiH<=g<%keP#2)Fe}O<8h4$XSZwFE5)NE9QnpbHkmsCG(Dy*&6RyHdn5gYZuM6 zbLW@L&TmMX)2*26?;Kb*H~rpR|6m(Y-MZFD7;=!zi!&e>A9RyMMdR9GlF%5jWGPz$ z&!n#li$wn95*3y+68Wu5)GkSoLvPw{c3eX}0%@9?7caUR{ZSP0J@A6-_NiN#A9SC# zcKnEs?|yw61XtZ#q2=6$J43(EedcZfc#%mAn##$O#D3f|J2aO+xBKU%bLKl;^VIy% zo#=vX!F$*EpC=bQA9W=Lk0+RymIqIL^h(0*K}}XH=T;_i=sES=fw|Dnk0l(f^IZ$% zLeuh|!-?I!A6Y>VTKdWVt5zF~4=iZ$uqmTO;_$x31~|4<%S&qEB2^go-wa+0&S{eN z+9j%Pl`_oKTyQWPVL*!R~ zI!Y4NJ>*xSv*)XaM#A1le)UJ&-me~(5=}mmNBh9v))bvKQNJU$pr=C%&;Q=I<#ZnP z&w2LKh1CBjuRL9<{}(Jd3*glVsv;=Q z9MX}wQ@9G)Bv>fL*-;W!OyQ_V_M7vf*f@Fw(c?UNMA74i=z%mN4xPq`#-YnR{9T6b zOF`+@2EKYBDN*{plILG1_Xj9R!jE5t2T%h^KA;Lo>WfB#bo`OX{e&>!|CUdP{7(qu zCxq!=iONrj`hO*g|DZF?9R9&~R&*7Mx{9Q(az$6WsH}#LW_LPnw-E1Gog Date: Tue, 18 Aug 2026 01:20:14 +0200 Subject: [PATCH 09/15] PDF on the JVM through the Konan plugin's JNI bindings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The :pdfium module now binds pdfium twice from one .def: cinterop for the native target, and the Konan plugin's generated JNI bridges for the JVM. CMake links the stub, which is the mode the plugin documents for a vendor library — pdfium is C++, and letting konan's linker near a foreign toolchain's archive is what produces mismatched runtimes. PdfiumConverter moves to commonMain over an expect/actual extractText, so the de-hyphenation and the document building are shared and only the extraction is per-platform. Both legs produce the same 488 words from the same PDF. Needs two plugin changes, published locally until they are released: const void* now crosses as a ByteArray rather than an address, without which the JVM cannot call FPDF_LoadMemDocument at all; and the CMake build receives the JNI include roots, without which the generated stub cannot find jni.h. Co-Authored-By: Claude Opus 5 (1M context) --- gradle/libs.versions.toml | 2 + pdfium/build.gradle.kts | 45 ++++++- pdfium/native/CMakeLists.txt | 19 +++ .../mikromarkdown/pdf/PdfiumConverter.kt | 73 +++++++++++ .../lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt | 80 +++++++++++ .../mikromarkdown/pdf/PdfiumConverterTest.kt | 24 ++++ .../mikromarkdown/pdf/Pdfium.macos.kt | 63 +++++++++ .../mikromarkdown/pdf/PdfiumConverter.kt | 124 ------------------ settings.gradle.kts | 3 + 9 files changed, 307 insertions(+), 126 deletions(-) create mode 100644 pdfium/native/CMakeLists.txt create mode 100644 pdfium/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt create mode 100644 pdfium/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt create mode 100644 pdfium/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverterTest.kt create mode 100644 pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.macos.kt delete mode 100644 pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 7bca1a6..1b7a7d0 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -8,6 +8,7 @@ kotlinx-io = "0.9.0" kotlinx-serialization = "1.9.0" ksoup = "0.2.6" korlibs = "6.0.0" +konanplugin = "1.2.0-alpha06" kotlinx-resources = "0.15.0" commons-csv = "1.14.1" @@ -48,4 +49,5 @@ kotlin-serialization = { id = "org.jetbrains.kotlin.plugin.serialization", versi vanniktech-mavenPublish = { id = "com.vanniktech.maven.publish", version.ref = "vanniktechMavenPublish" } kotlinx-resources = { id = "com.goncalossilva.resources", version.ref = "kotlinx-resources" } detekt = { id = "io.gitlab.arturbosch.detekt", version.ref = "detekt" } +konanplugin = { id = "io.github.lemcoder.konanplugin", version.ref = "konanplugin" } ktfmt = { id = "com.ncorti.ktfmt.gradle", version.ref = "ktfmt" } diff --git a/pdfium/build.gradle.kts b/pdfium/build.gradle.kts index 21d6331..eaa7da3 100644 --- a/pdfium/build.gradle.kts +++ b/pdfium/build.gradle.kts @@ -1,6 +1,10 @@ +import io.github.lemcoder.interop.jvmInterops import java.security.MessageDigest -plugins { alias(libs.plugins.kotlinMultiplatform) } +plugins { + alias(libs.plugins.kotlinMultiplatform) + alias(libs.plugins.konanplugin) +} /** * PDF support, kept in its own module so it can be dropped or extracted whole. @@ -83,7 +87,44 @@ kotlin { } } - sourceSets { macosArm64Main.dependencies { implementation(project(":library")) } } + jvm { + // The JNI leg binds the same .def the native target does, so one declaration covers both. + compilations["main"].jvmInterops { + create("pdfium") { + defFile(project.file("src/nativeInterop/cinterop/pdfium.def")) + includeDirs.from(pdfiumRoot.get().dir("mac-arm64/include")) + + externalNativeBuild { + cmake { + path.set(project.file("native/CMakeLists.txt")) + targets.add("pdfium-jni") + arguments.add("-DPDFium_DIR=${pdfiumRoot.get().dir("mac-arm64").asFile.absolutePath}") + } + } + } + } + } + + sourceSets { + macosArm64Main.dependencies { implementation(project(":library")) } + jvmMain.dependencies { implementation(project(":library")) } + jvmTest.dependencies { implementation(libs.kotlin.test) } + } } +// The bindings load the stub by name, and the stub finds pdfium through the rpath CMake gave it. +tasks.named("jvmTest") { + dependsOn("linkJvmInteropPdfium") + useJUnitPlatform() + systemProperty( + "java.library.path", + layout.buildDirectory.dir("jvmInterop/pdfium/lib").get().asFile.absolutePath, + ) +} + +// Every binding path needs the headers and the library unpacked first. tasks.matching { it.name.startsWith("cinteropPdfium") }.configureEach { dependsOn(downloadPdfium) } + +tasks + .matching { it.name.startsWith("generateJvmInterop") || it.name.startsWith("cmakeConfigure") } + .configureEach { dependsOn(downloadPdfium) } diff --git a/pdfium/native/CMakeLists.txt b/pdfium/native/CMakeLists.txt new file mode 100644 index 0000000..eef696d --- /dev/null +++ b/pdfium/native/CMakeLists.txt @@ -0,0 +1,19 @@ +cmake_minimum_required(VERSION 3.20) +project(pdfium_jni C) + +# Supplied by the Konan plugin: where it wrote the JNI stub, and what the bindings will load. +set(KONAN_JNI_STUB_DIR "" CACHE PATH "") +set(KONAN_JNI_LIB_NAME "" CACHE STRING "") +set(KONAN_JNI_INCLUDE_DIRS "" CACHE STRING "") + +# pdfium ships PDFiumConfig.cmake; PDFium_DIR points at the unpacked release. +find_package(PDFium REQUIRED) + +file(GLOB JNI_SOURCES "${KONAN_JNI_STUB_DIR}/*.c") +add_library(pdfium-jni SHARED ${JNI_SOURCES}) +set_target_properties(pdfium-jni PROPERTIES OUTPUT_NAME "${KONAN_JNI_LIB_NAME}") +target_include_directories(pdfium-jni PRIVATE ${KONAN_JNI_INCLUDE_DIRS}) + +# Linking through CMake rather than konan's linker is the point: pdfium is C++, and its runtime +# comes along correctly this way. +target_link_libraries(pdfium-jni PRIVATE pdfium) diff --git a/pdfium/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt b/pdfium/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt new file mode 100644 index 0000000..273d4e5 --- /dev/null +++ b/pdfium/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt @@ -0,0 +1,73 @@ +package io.github.lemcoder.mikromarkdown.pdf + +import io.github.lemcoder.mikromarkdown.DocumentConverter +import io.github.lemcoder.mikromarkdown.StreamInfo +import io.github.lemcoder.mikromarkdown.model.Document +import io.github.lemcoder.mikromarkdown.model.plainTextBlocks + +/** + * PDF text through pdfium. + * + * Not registered by the library's factory: PDF costs a native library, so a caller asks for it. + * + * ``` + * val mikroMarkdown = MikroMarkdown().apply { register(PdfiumConverter()) } + * ``` + * + * The extraction itself is per-platform — cinterop on native, generated JNI bridges on the JVM — but both reach the + * same pdfium, and everything above [extractText] is shared. + */ +public class PdfiumConverter : DocumentConverter { + + override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { + return info.extension == "pdf" || info.mimetype == "application/pdf" + } + + override fun parse(bytes: ByteArray, info: StreamInfo): Document = + Document(blocks = plainTextBlocks(extractText(bytes).joinHyphenatedWords())) + + /** + * pdfium emits U+FFFE where a glyph has no Unicode mapping, which in a typeset document is nearly always the hyphen + * at a line break. + * + * Dropping it always would fuse real compounds — "chat-optimized" became "chatoptimized" — so the document decides: + * if both halves appear elsewhere as words in their own right the hyphen was the author's and is restored, + * otherwise the halves are one broken word and are joined. The halves themselves are cut from the vocabulary first, + * since they are only in the text because the break put them there. + * + * This is a heuristic standing in for geometry. A hyphenation hyphen ends a line and a compound hyphen does not, + * which `FPDFText_GetCharBox` would answer outright. + */ + private fun String.joinHyphenatedWords(): String { + if (indexOf(UNMAPPED_GLYPH) < 0) return this + + val vocabulary = WORD.findAll(replace(HYPHEN_BREAK, " ")).map { it.value.lowercase() }.toSet() + val out = StringBuilder(length) + for (index in indices) { + val char = this[index] + if (char != UNMAPPED_GLYPH) { + out.append(char) + continue + } + var wordStart = out.length + while (wordStart > 0 && out[wordStart - 1].isLetter()) wordStart-- + val left = out.subSequence(wordStart, out.length).toString().lowercase() + val right = substring(index + 1).takeWhile { it.isLetter() }.lowercase() + if (isRealCompound(left, right, vocabulary)) out.append('-') + } + return out.toString() + } + + /** A hyphen the author wrote, rather than one the typesetter added at a line break. */ + private fun isRealCompound(left: String, right: String, vocabulary: Set): Boolean = + left.isNotEmpty() && right.isNotEmpty() && left in vocabulary && right in vocabulary + + private companion object { + const val UNMAPPED_GLYPH = '\uFFFE' + val WORD = Regex("[\\p{L}]{2,}") + val HYPHEN_BREAK = Regex("\\p{L}+\uFFFE\\p{L}+") + } +} + +/** Every page's text, concatenated, one page per line. */ +internal expect fun extractText(bytes: ByteArray): String diff --git a/pdfium/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt b/pdfium/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt new file mode 100644 index 0000000..243f89a --- /dev/null +++ b/pdfium/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt @@ -0,0 +1,80 @@ +package io.github.lemcoder.mikromarkdown.pdf + +import pdfium.kniBridge1 +import pdfium.kniBridge15 +import pdfium.kniBridge16 +import pdfium.kniBridge2 +import pdfium.kniBridge26 +import pdfium.kniBridge27 +import pdfium.kniBridge5 +import pdfium.kniBridge52 +import pdfium.kniBridge53 +import pdfium.kniBridge54 +import pdfium.kniBridge71 + +/** + * The JVM half, over the JNI bridges the Konan plugin generates from the same `.def` cinterop binds. + * + * The bridges are numbered rather than named — that is what a runtime-free binding looks like — so each is wrapped here + * with the name from its doc comment, and nothing else in the module sees them. + */ +internal actual fun extractText(bytes: ByteArray): String { + val text = StringBuilder() + + initLibrary() + try { + val document = loadDocument(bytes, null) + if (document == 0L) return "" + try { + for (index in 0 until pageCount(document)) { + val page = loadPage(document, index) + if (page == 0L) continue + val textPage = loadTextPage(page) + if (textPage != 0L) { + text.append(pageText(textPage)) + text.append('\n') + closeTextPage(textPage) + } + closePage(page) + } + } finally { + closeDocument(document) + } + } finally { + destroyLibrary() + } + + return text.toString() +} + +/** pdfium writes UTF-16 into a caller-supplied buffer and counts the terminating NUL. */ +private fun pageText(textPage: Long): String { + val count = charCount(textPage) + if (count <= 0) return "" + val buffer = ShortArray(count + 1) + val written = readText(textPage, 0, count, buffer) + return if (written <= 1) "" else CharArray(written - 1) { Char(buffer[it].toInt() and 0xFFFF) }.concatToString() +} + +private fun initLibrary() = kniBridge1() + +private fun destroyLibrary() = kniBridge2() + +private fun loadDocument(bytes: ByteArray, password: String?): Long = kniBridge5(bytes, bytes.size, password) + +private fun pageCount(document: Long): Int = kniBridge15(document) + +private fun loadPage(document: Long, index: Int): Long = kniBridge16(document, index) + +private fun closePage(page: Long) = kniBridge26(page) + +private fun closeDocument(document: Long) = kniBridge27(document) + +private fun loadTextPage(page: Long): Long = kniBridge52(page) + +private fun closeTextPage(textPage: Long) = kniBridge53(textPage) + +private fun charCount(textPage: Long): Int = kniBridge54(textPage) + +private fun readText(textPage: Long, start: Int, count: Int, buffer: ShortArray): Int = + kniBridge71(textPage, start, count, buffer) diff --git a/pdfium/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverterTest.kt b/pdfium/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverterTest.kt new file mode 100644 index 0000000..fa85c96 --- /dev/null +++ b/pdfium/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverterTest.kt @@ -0,0 +1,24 @@ +package io.github.lemcoder.mikromarkdown.pdf + +import io.github.lemcoder.mikromarkdown.StreamInfo +import java.io.File +import kotlin.test.Test +import kotlin.test.assertContains +import kotlin.test.assertTrue + +class PdfiumConverterTest { + + private val fixture = File("../library/src/commonTest/resources/test_files/test.pdf") + + @Test + fun `extracts text through the generated JNI bridges`() { + val document = PdfiumConverter().parse(fixture.readBytes(), StreamInfo(extension = "pdf")) + val text = document.blocks.joinToString("\n") { it.toString() } + + assertTrue(text.length > 1000, "expected a page of text, got ${text.length} characters") + assertContains(text, "Introduction") + // The de-hyphenation ran: pdfium reports a broken word with U+FFFE between the halves. + assertContains(text, "confirming") + assertTrue('￾' !in text, "unmapped glyphs should not reach the model") + } +} diff --git a/pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.macos.kt b/pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.macos.kt new file mode 100644 index 0000000..655e0f1 --- /dev/null +++ b/pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.macos.kt @@ -0,0 +1,63 @@ +package io.github.lemcoder.mikromarkdown.pdf + +import kotlinx.cinterop.ExperimentalForeignApi +import kotlinx.cinterop.UShortVar +import kotlinx.cinterop.addressOf +import kotlinx.cinterop.allocArray +import kotlinx.cinterop.get +import kotlinx.cinterop.memScoped +import kotlinx.cinterop.usePinned +import pdfium.FPDFText_ClosePage +import pdfium.FPDFText_CountChars +import pdfium.FPDFText_GetText +import pdfium.FPDFText_LoadPage +import pdfium.FPDF_CloseDocument +import pdfium.FPDF_ClosePage +import pdfium.FPDF_DestroyLibrary +import pdfium.FPDF_GetPageCount +import pdfium.FPDF_InitLibrary +import pdfium.FPDF_LoadMemDocument +import pdfium.FPDF_LoadPage + +/** The native half, over the cinterop bindings. */ +@OptIn(ExperimentalForeignApi::class) +internal actual fun extractText(bytes: ByteArray): String { + val text = StringBuilder() + + FPDF_InitLibrary() + try { + bytes.usePinned { pinned -> + val document = FPDF_LoadMemDocument(pinned.addressOf(0), bytes.size, null) ?: return@usePinned + try { + for (index in 0 until FPDF_GetPageCount(document)) { + val page = FPDF_LoadPage(document, index) ?: continue + val textPage = FPDFText_LoadPage(page) + if (textPage != null) { + text.append(pageText(textPage)) + text.append('\n') + FPDFText_ClosePage(textPage) + } + FPDF_ClosePage(page) + } + } finally { + FPDF_CloseDocument(document) + } + } + } finally { + FPDF_DestroyLibrary() + } + + return text.toString() +} + +/** pdfium writes UTF-16 into a caller-supplied buffer and counts the terminating NUL. */ +@OptIn(ExperimentalForeignApi::class) +private fun pageText(textPage: pdfium.FPDF_TEXTPAGE): String { + val count = FPDFText_CountChars(textPage) + if (count <= 0) return "" + return memScoped { + val buffer = allocArray(count + 1) + val written = FPDFText_GetText(textPage, 0, count, buffer) + if (written <= 1) "" else CharArray(written - 1) { Char(buffer[it].toInt()) }.concatToString() + } +} diff --git a/pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt b/pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt deleted file mode 100644 index 63001c0..0000000 --- a/pdfium/src/macosArm64Main/kotlin/io/github/lemcoder/mikromarkdown/pdf/PdfiumConverter.kt +++ /dev/null @@ -1,124 +0,0 @@ -package io.github.lemcoder.mikromarkdown.pdf - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.model.plainTextBlocks -import kotlinx.cinterop.ExperimentalForeignApi -import kotlinx.cinterop.UShortVar -import kotlinx.cinterop.addressOf -import kotlinx.cinterop.allocArray -import kotlinx.cinterop.get -import kotlinx.cinterop.memScoped -import kotlinx.cinterop.usePinned -import pdfium.FPDFText_ClosePage -import pdfium.FPDFText_CountChars -import pdfium.FPDFText_GetText -import pdfium.FPDFText_LoadPage -import pdfium.FPDF_CloseDocument -import pdfium.FPDF_ClosePage -import pdfium.FPDF_DestroyLibrary -import pdfium.FPDF_GetPageCount -import pdfium.FPDF_InitLibrary -import pdfium.FPDF_LoadMemDocument -import pdfium.FPDF_LoadPage - -/** - * PDF text through pdfium. - * - * Not registered by the library's factory: PDF costs a native library, so a caller asks for it. - * - * ``` - * val mikroMarkdown = MikroMarkdown().apply { register(PdfiumConverter()) } - * ``` - */ -public class PdfiumConverter : DocumentConverter { - - private companion object { - const val UNMAPPED_GLYPH = '\uFFFE' - val WORD = Regex("[\\p{L}]{2,}") - val HYPHEN_BREAK = Regex("\\p{L}+\uFFFE\\p{L}+") - } - - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "pdf" || info.mimetype == "application/pdf" - } - - @OptIn(ExperimentalForeignApi::class) - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val text = StringBuilder() - - FPDF_InitLibrary() - try { - bytes.usePinned { pinned -> - val document = FPDF_LoadMemDocument(pinned.addressOf(0), bytes.size, null) ?: return@usePinned - try { - for (index in 0 until FPDF_GetPageCount(document)) { - val page = FPDF_LoadPage(document, index) ?: continue - val textPage = FPDFText_LoadPage(page) - if (textPage != null) { - text.append(pageText(textPage)) - // A page break reads as a paragraph break to the text blocks. - text.append('\n') - FPDFText_ClosePage(textPage) - } - FPDF_ClosePage(page) - } - } finally { - FPDF_CloseDocument(document) - } - } - } finally { - FPDF_DestroyLibrary() - } - - return Document(blocks = plainTextBlocks(text.toString().joinHyphenatedWords())) - } - - /** - * pdfium emits U+FFFE — a permanent non-character — where a glyph has no Unicode mapping, which for a typeset - * document is nearly always the hyphen at a line break. - * - * Dropping it always would fuse real compounds: "chat-optimized" became "chatoptimized". So the document decides. - * If both halves appear elsewhere as words in their own right the hyphen was real and is restored; otherwise the - * halves are two pieces of one broken word and are joined. - */ - private fun String.joinHyphenatedWords(): String { - if (indexOf(UNMAPPED_GLYPH) < 0) return this - - // The halves of a broken word must not vouch for themselves: "con" and "firming" are only - // in the text because the break put them there, so every break is cut before counting. - val unbroken = replace(HYPHEN_BREAK, " ") - val vocabulary = WORD.findAll(unbroken).map { it.value.lowercase() }.toSet() - val out = StringBuilder(length) - for (index in indices) { - val char = this[index] - if (char != UNMAPPED_GLYPH) { - out.append(char) - continue - } - var wordStart = out.length - while (wordStart > 0 && out[wordStart - 1].isLetter()) wordStart-- - val left = out.subSequence(wordStart, out.length).toString().lowercase() - val right = substring(index + 1).takeWhile { it.isLetter() }.lowercase() - if (isRealCompound(left, right, vocabulary)) out.append('-') - } - return out.toString() - } - - /** A hyphen the author wrote, rather than one the typesetter added at a line break. */ - private fun isRealCompound(left: String, right: String, vocabulary: Set): Boolean = - left.isNotEmpty() && right.isNotEmpty() && left in vocabulary && right in vocabulary - - /** pdfium writes UTF-16 into a caller-supplied buffer and counts the terminating NUL. */ - @OptIn(ExperimentalForeignApi::class) - private fun pageText(textPage: pdfium.FPDF_TEXTPAGE): String { - val count = FPDFText_CountChars(textPage) - if (count <= 0) return "" - return memScoped { - val buffer = allocArray(count + 1) - val written = FPDFText_GetText(textPage, 0, count, buffer) - if (written <= 1) "" else CharArray(written - 1) { Char(buffer[it].toInt()) }.concatToString() - } - } -} diff --git a/settings.gradle.kts b/settings.gradle.kts index 4326077..771b36c 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -1,5 +1,8 @@ pluginManagement { repositories { + // KonanPlugin is consumed from a local publish while the void-buffer marshalling it needs + // for pdfium is unreleased; drop this once 1.2.0-alpha06 is on the portal. + mavenLocal() google() mavenCentral() gradlePluginPortal() From 759b100619ce55faa0771e581b4d3815599867a8 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Tue, 18 Aug 2026 01:31:49 +0200 Subject: [PATCH 10/15] Drop PDFBox and the JVM CLI; pdfium is the only PDF path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PDFBox goes with the JVM CLI that used it. The library's JVM target no longer converts PDF at all — the :pdfium module does, through cinterop on native and the generated JNI bridges on the JVM, and a caller registers it. Deleting the JVM CLI takes a surprising amount of machinery with it: the class-data-sharing archive and its two-step dump, the start-script guards that made it optional, clikt, and the distribution packaging that would have had to carry two dylibs. The native binary was already faster on every fixture, so the JVM one was paying that complexity to lose. Android keeps pdfbox-android for now: the :pdfium module has no Android ABIs yet, and removing its converter would leave Android with no PDF at all. The harness follows: optbench compares the native binary against a champion and verifies its output against the recorded baselines, which the JVM CLI produced back when the two matched byte for byte. PDF is exempt from that check, since pdfium reads documents differently from PDFBox by design. DumpOutputTest is deleted rather than fixed — it wrote conversions to /tmp for eyeballing, which the baselines and the harness now do properly. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 188 ++++-------------- cli/build.gradle.kts | 134 ------------- .../main/kotlin/com/mikromarkdown/cli/Main.kt | 12 -- .../mikromarkdown/cli/MikroMarkdownCommand.kt | 31 --- gradle/libs.versions.toml | 4 - library/build.gradle.kts | 7 +- .../mikromarkdown/MikroMarkdownFactory.kt | 9 +- .../mikromarkdown/converters/PdfConverter.kt | 32 --- .../lemcoder/mikromarkdown/DumpOutputTest.kt | 26 --- scripts/benchmark.py | 13 +- scripts/optbench.py | 68 ++----- settings.gradle.kts | 2 - 12 files changed, 69 insertions(+), 457 deletions(-) delete mode 100644 cli/build.gradle.kts delete mode 100644 cli/src/main/kotlin/com/mikromarkdown/cli/Main.kt delete mode 100644 cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt delete mode 100644 library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt delete mode 100644 library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/DumpOutputTest.kt diff --git a/README.md b/README.md index 9aad4c0..9c7061a 100644 --- a/README.md +++ b/README.md @@ -12,16 +12,16 @@ meets PDF and EPUB. Dropping them took Apache POI with them — the distribution as a dependency everyone carries. -| Format | Extension | -|--------|-----------| -| EPUB | `.epub` | -| HTML | `.html`, `.htm` | -| PDF | `.pdf` | -| CSV | `.csv` | -| JSON | `.json` | -| XML | `.xml` | -| Plain text | `.txt` and others | -| Markdown | `.md` (passthrough) | +| Format | Extension | Notes | +|--------|-----------|-------| +| EPUB | `.epub` | | +| HTML | `.html`, `.htm` | | +| PDF | `.pdf` | opt-in: `:pdfium` module, `register(PdfiumConverter())` | +| CSV | `.csv` | | +| JSON | `.json` | | +| XML | `.xml` | | +| Plain text | `.txt` and others | | +| Markdown | `.md` (passthrough) | | ## Architecture @@ -37,9 +37,10 @@ bytes ──► MimeDetector ──► DocumentConverter.parse ──► Documen ``` Converters contain no Markdown syntax, so escaping, table shaping, list indentation and spacing are -fixed once for all formats. JVM and Android share one `jvmShared` source set, so a converter exists -once rather than per target; only PDF extraction and MIME detection are platform-specific. The model is public: `mid.parse(path)` returns the `Document`, and -`ConversionResult.document` exposes it alongside the rendered Markdown. +fixed once for all formats. Every converter lives in `commonMain` and runs on every target; only PDF +is platform-specific, and it lives in its own module because it needs a native library. The model is +public: `mid.parse(path)` returns the `Document`, and `ConversionResult.document` exposes it +alongside the rendered Markdown. ```kotlin val document = mid.parse("/path/to/book.epub") @@ -154,167 +155,52 @@ import the renderer or each other; Markdown syntax appears only under `render/`. every `DocumentConverter` is named `*Converter` and lives in `converters`. *Hygiene* — no wildcard imports, no printing from library code, and no source file duplicated -between source sets (the drift that the `jvmShared` set removed). +between source sets. ktfmt-gradle only derives tasks for the common and JVM source sets, so `library/build.gradle.kts` registers matching tasks for the Android ones. ## Performance -Conversion itself is a few milliseconds; a CLI run is mostly JVM startup and class loading. -`:benchmark` measures the pipeline in-process, `scripts/benchmark.py` measures whole processes. - -```bash -./gradlew :benchmark:run # in-process, per stage -python3 scripts/benchmark.py # whole process, against markitdown and anydoc -``` - -In-process, best of 50 runs after warmup: - -| fixture | size | parse | render | total | -|---|---|---|---|---| -| test.json | 0.4 KB | 0.03 ms | 0.01 ms | 0.04 ms | -| test.epub | 2 KB | 0.42 ms | 0.00 ms | 0.42 ms | -| test_blog.html | 25 KB | 0.85 ms | 0.11 ms | 0.96 ms | -| test.pdf | 90 KB | 3.69 ms | 0.00 ms | 3.47 ms | -| test_wikipedia.html | 385 KB | 12.98 ms | 1.82 ms | 14.80 ms | - -End to end the CLI runs in 50–230 ms, against 3–5 ms for the Rust -[anydoc](https://github.com/firecrawl/anydoc) and 410–540 ms for Python markitdown. Nearly all of -what is left is process startup: `java -version` alone costs 41 ms on the same machine, and the -conversion is under 5 ms for every fixture but Wikipedia. Matching a native binary would take -ahead-of-time compilation, not a faster pipeline. - -Note when reproducing this: anydoc's npm package is a Node script loading a napi module, so timing -`node_modules/.bin/anydoc` charges Node's 15 ms startup to Rust and reads as ~22 ms flat. The -figures here come from its Rust binary, built from the vendored source with -`cargo build --release --example convert`. - -The CLI therefore optimizes startup rather than throughput: - -- `installDist` records a [class-data-sharing](https://docs.oracle.com/en/java/javase/21/vm/class-data-sharing.html) - archive into the distribution, which roughly halves startup. Set `MIKROMARKDOWN_NO_CDS=1` to skip - it; the start script also skips it when the archive is missing, so `distZip` still works. -- it compiles with C1 only (`-XX:TieredStopAtLevel=1`), since C2 never pays for itself in a run this - short. Embedders using the library get the normal JIT. - -### Kotlin/Native spike - -`:cli-native` builds a macOS binary carrying the converters that need no JVM library — CSV, JSON, -XML, plain text and Markdown passthrough. It shares the model, renderer and pipeline with every -other target, and its output is byte-identical to the JVM CLI's. +The command line tool is the Kotlin/Native binary; there is no JVM CLI. `:benchmark` measures the +library in-process on the JVM, and `scripts/optbench.py` A/B times the native binary against a saved +champion so that session-to-session drift cannot be mistaken for a change. ```bash ./gradlew :cli-native:linkReleaseExecutableMacosArm64 +./gradlew :benchmark:run # library, in-process, per stage +python3 scripts/optbench.py "..." # native binary, against the champion +python3 scripts/benchmark.py # whole process, against markitdown and anydoc ``` -Whole-process, best of eight, converting CSV of increasing size: - -| input | Kotlin/Native | anydoc (Rust binary) | JVM CLI | -|---|---|---|---| -| 1 KB | 3 ms | 3 ms | 51 ms | -| 55 KB | **4 ms** | 5 ms | 61 ms | -| 172 KB | **7 ms** | 10 ms | 59 ms | -| 580 KB | **19 ms** | 25 ms | 70 ms | -| 1.8 MB | **51 ms** | 71 ms | 101 ms | -| 3.5 MB | **102 ms** | 139 ms | 142 ms | - -Kotlin/Native is ahead of the Rust binary at every size here, having started the spike 18-21% -behind on large inputs; `docs/optimization-log.md` records how. Note that anydoc's npm package runs -through Node, which adds about 18 ms — these figures are its Rust binary, built from the vendored -source with `cargo build --release --example convert`. - -On the document formats the native target does not carry, the JVM CLI converts a DOCX in 180 ms and -Wikipedia in 122 ms, against anydoc's 3 ms and Python markitdown's 409 ms and 520 ms. That gap is -process startup, not conversion: in-process those documents take 2.7 ms and 13 ms. - -Three changes closed the throughput gap that the first cut of this target showed: - -- **The renderer stopped allocating when it has nothing to change.** Escaping now scans for the - first character that needs a backslash and returns the input untouched when there is none, and a - single-line table cell skips the split-and-rejoin. Ordinary cells — a word, a number — now cost no - allocation at all. This is shared code, so the JVM got faster too. -- **The native CSV reader slices instead of accumulating.** Fields are ranges in the decoded text, - so a field costs one substring rather than a per-character builder plus a separate trim. Only - fields containing escaped quotes, which cannot be a slice of the input, assemble a string. -- **CSV, JSON and XML moved to `commonMain`**, taking commons-csv, Jackson and - kotlinx-serialization with them. One implementation now serves every target: a slicing CSV reader, - a JSON re-indenter that copies tokens verbatim so `1.50` does not become `1.5`, and an XML - formatter. A JSON conversion loads 1214 classes instead of 2063, the native binary is 1.3 MB - instead of 2.2 MB, and the JVM CLI converts JSON in 56 ms instead of 95 ms. Output is unchanged - on every fixture. -- **Two quadratics in the renderer are gone.** Blocks are written into one buffer carrying a line - prefix, rather than each block returning a string that its parent splits into lines and re-joins — - which charged the deepest content once per level of nesting above it. And the entity check now - scans ten characters ahead instead of searching the rest of the document for a semicolon. - Measured on inputs built to provoke them, with output byte-identical before and after: - - | pathological input | before | after | - |---|---|---| - | 400-deep nested lists | 33.03 ms | **0.12 ms** | - | 400 ampersands per cell, 789 KB | 9.39 ms | **2.42 ms** | - -Together those took a 1.8 MB CSV from 237 ms to 87 ms. Kotlin/Native's remaining cost is the -document model itself: every cell becomes a `TableCell` holding a `Text` holding a `String`, which -is why peak memory is 125 MB for a 1.8 MB input. A genuinely zero-copy model — inlines holding -slices of the source buffer rather than copies — is the next lever, and a deeper change. - -Compiler flags were measured rather than guessed. `-Xbinary=preCodegenInlineThreshold=40` is worth -about 8% on large inputs and ships. Every garbage collection setting tried was worse than the -default, and the collector is the interesting part of the story, so the numbers are below. - -Converting 20 documents of 580 KB in one process: - -| policy | time | peak RSS | -|---|---|---| -| default (adaptive) | 543 ms | 59 MB | -| `gcSchedulerType=manual`, never collecting | 428 ms | 954 MB | -| `gcSchedulerType=manual`, collecting between documents | 485 ms | 67 MB | -| `autotune = false` with a heap ceiling | 1246 ms | 43 MB | - -A manual collector is genuinely faster, since a process that exits never needs to collect, and a -document boundary is the one place where everything the previous conversion allocated is provably -dead. But it only bounds growth *between* documents: a single large input still has nothing -collecting mid-parse, so the 3.5 MB file takes 228 MB either way and a much larger one would grow -until it failed. Turning `autotune` off is far worse than it looks like it should be — 8.6x on a -single 3.5 MB file — and the ceiling value makes no difference to that, so `targetHeapBytes` is not -behaving as its name suggests. - -The default collector ships. The native CLI does accept several files per invocation, which is what -would make a manual policy workable if the trade ever becomes worth it. - -### What did not help +Whole process, best of ten, against the Rust [anydoc](https://github.com/firecrawl/anydoc): -Binary size does not drive startup, so shrinking it is not a performance lever: - -| binary | size | startup | +| input | Kotlin/Native | anydoc (Rust) | |---|---|---| -| Kotlin/Native hello world | 485 KB | 3.2 ms | -| this CLI | 1.3 MB | 3.5 ms | -| anydoc (Rust) | 6 MB | 2.6 ms | - -A 6 MB Rust binary starts faster than a 485 KB Kotlin/Native one, and stripping ours changed -nothing measurable. The 0.6 ms between the two runtimes is initialization, not size. +| 1 KB CSV | 3 ms | 3 ms | +| 172 KB CSV | 7 ms | 10 ms | +| 1.8 MB CSV | 51 ms | 71 ms | +| 3.5 MB CSV | 102 ms | 139 ms | +| EPUB | 5 ms | 3 ms | +| Wikipedia HTML | 65 ms | — | -Replacing clikt with hand-rolled argument parsing removes 147 loaded classes and about 2 ms — inside -the noise, and not worth losing its help output and error handling. The dependency stays. +Note when reproducing this: anydoc's npm package is a Node script loading a napi module, so timing +`node_modules/.bin/anydoc` charges Node's 15 ms startup to Rust and reads as ~22 ms flat. The figures +here come from its Rust binary, built from the vendored source with +`cargo build --release --example convert`. -Zero-copy parsing made things slower, which is the most useful negative result here. Handing the -renderer a `CharSequence` window onto the source instead of a substring removes one allocation per -cell, but every character access then goes through an interface call rather than `String`'s direct -indexing — and the renderer reads every character anyway to decide whether it needs escaping. A -1.8 MB CSV went from 86 ms to 118 ms on native and 126 ms to 154 ms on the JVM. Rust gets this for -free because `&str` slices index directly; Kotlin does not. Reverted. +`docs/optimization-log.md` records twenty-five measured experiments, nine of which survived, and the +two methodology mistakes that cost more than most of the wins. ## Benchmark -`scripts/benchmark.py` converts the test fixtures with MikroMarkdown, Python +`scripts/benchmark.py` converts the test fixtures with the native binary, Python [markitdown](https://github.com/microsoft/markitdown) and Rust [anydoc](https://github.com/firecrawl/anydoc), then reports content recall, structure counts, table integrity and timings to `build/benchmark/report.md`: ```bash -./gradlew :cli:installDist +./gradlew :cli-native:linkReleaseExecutableMacosArm64 python3 scripts/benchmark.py ``` diff --git a/cli/build.gradle.kts b/cli/build.gradle.kts deleted file mode 100644 index 89f64b0..0000000 --- a/cli/build.gradle.kts +++ /dev/null @@ -1,134 +0,0 @@ -plugins { - alias(libs.plugins.kotlin.jvm) - application -} - -java { toolchain { languageVersion = JavaLanguageVersion.of(21) } } - -val cdsArchiveName = "mikromarkdown.jsa" - -application { - mainClass = "com.mikromarkdown.cli.MainKt" - // Class loading, not conversion, is what a short CLI run spends its time on. A class-data-sharing - // archive maps those classes in pre-parsed. -Xshare:auto keeps the CLI working when it is absent. - // Every CLI run is short, so C2 never pays for itself: compiling with C1 only is faster - // end to end. Long-running embedders use the library directly and are unaffected. - applicationDefaultJvmArgs = listOf("-Xshare:auto", "-XX:-UsePerfData", "-XX:TieredStopAtLevel=1") -} - -dependencies { - implementation(project(":library")) - implementation(libs.clikt) -} - -tasks.test { useJUnitPlatform() } - -// The archive flag is added by the script itself, and only when the archive is there: naming a -// missing archive stops the JVM loading its base one, which is also what recording needs. -tasks.named("startScripts") { - doLast { - val unixGuard = - """ - CDS_ARCHIVE="${'$'}APP_HOME/lib/$cdsArchiveName" - if [ -f "${'$'}CDS_ARCHIVE" ] && [ -z "${'$'}MIKROMARKDOWN_NO_CDS" ] ; then - DEFAULT_JVM_OPTS="${'$'}DEFAULT_JVM_OPTS \"-XX:SharedArchiveFile=${'$'}CDS_ARCHIVE\"" - fi - """ - .trimIndent() - - // Lambda form: the guard contains $ sequences that must not be read as group references. - unixScript.writeText( - unixScript.readText().replace(Regex("(?m)^(DEFAULT_JVM_OPTS=.*)${'$'}")) { match -> - "${match.value}\n\n$unixGuard" - } - ) - - val windowsGuard = - """ - set CDS_ARCHIVE=%APP_HOME%\\lib\\$cdsArchiveName - if exist "%CDS_ARCHIVE%" if not defined MIKROMARKDOWN_NO_CDS set DEFAULT_JVM_OPTS=%DEFAULT_JVM_OPTS% "-XX:SharedArchiveFile=%CDS_ARCHIVE%" - """ - .trimIndent() - - windowsScript.writeText( - windowsScript.readText().replace(Regex("(?m)^(set DEFAULT_JVM_OPTS=.*)${'$'}")) { match -> - "${match.value}\r\n$windowsGuard" - } - ) - } -} - -/** - * Builds a class-data-sharing archive for the installed distribution, in two steps: - * - * 1. run the CLI once with `-XX:DumpLoadedClassList` to learn which classes a conversion touches; - * 2. `-Xshare:dump` that list into an archive. - * - * This static form is used rather than `-XX:ArchiveClassesAtExit` because the dynamic one needs the JDK's own base - * archive, which some distributions (JetBrains Runtime among them) do not ship. - * - * Step 1 goes through the start script so the classpath recorded is the one real runs use; step 2 reads that same - * classpath back out of the script, because a mismatch makes the JVM drop the archive without saying so. - */ -val cdsArchive by tasks.registering { - group = "distribution" - description = "Builds a class-data-sharing archive into the installed distribution." - - val installDir = layout.buildDirectory.dir("install/${application.applicationName}").get() - val appName = application.applicationName - val fixtures = rootProject.layout.projectDirectory.dir("library/src/commonTest/resources/test_files") - // One sample per family, so the archive covers the classes each conversion path touches rather - // than only the ones a DOCX happens to need. - val samples = - listOf("test.pdf", "test.epub", "test_blog.html", "test.csv", "test.json", "test.xml").map { - fixtures.file(it).asFile.absolutePath - } - val javaHome = javaToolchains.launcherFor(java.toolchain).get().metadata.installationPath - - doLast { - val script = installDir.file("bin/$appName").asFile - val classList = installDir.file("lib/$cdsArchiveName.classlist").asFile - val archive = installDir.file("lib/$cdsArchiveName").asFile - archive.delete() - - // One run over one sample of each family: a single list keeps the loader metadata that - // makes the archive worth having, which merging separate runs would throw away. - val process = - ProcessBuilder(listOf(script.absolutePath) + samples) - .redirectOutput(ProcessBuilder.Redirect.DISCARD) - .also { - it.environment()["JAVA_OPTS"] = "-XX:DumpLoadedClassList=${classList.absolutePath}" - it.environment()["MIKROMARKDOWN_NO_CDS"] = "1" - } - .start() - val errors = process.errorStream.bufferedReader().readText() - check(process.waitFor() == 0) { "recording failed: ${errors.take(400)}" } - check(classList.exists()) { "the JVM recorded no class list" } - - val classpath = - script - .readLines() - .first { it.startsWith("CLASSPATH=") } - .removePrefix("CLASSPATH=") - .replace("\$APP_HOME", installDir.asFile.absolutePath) - - providers - .exec { - commandLine( - javaHome.file("bin/java").asFile.absolutePath, - "-Xshare:dump", - "-XX:SharedClassListFile=${classList.absolutePath}", - "-XX:SharedArchiveFile=${archive.absolutePath}", - "-cp", - classpath, - ) - } - .standardOutput - .asText - .get() - check(archive.exists()) { "no CDS archive was produced" } - logger.lifecycle("CDS archive: ${archive.length() / 1024} KB") - } -} - -tasks.named("installDist") { finalizedBy(cdsArchive) } diff --git a/cli/src/main/kotlin/com/mikromarkdown/cli/Main.kt b/cli/src/main/kotlin/com/mikromarkdown/cli/Main.kt deleted file mode 100644 index 11974c1..0000000 --- a/cli/src/main/kotlin/com/mikromarkdown/cli/Main.kt +++ /dev/null @@ -1,12 +0,0 @@ -package com.mikromarkdown.cli - -import com.github.ajalt.clikt.core.main - -fun main(args: Array) { - // PDFBox pulls in the Log4j API; without a provider it writes a banner to stdout, - // which would corrupt the Markdown we print there. - System.setProperty("log4j2.loggerContextFactory", "org.apache.logging.log4j.simple.SimpleLoggerContextFactory") - System.setProperty("log4j2.simplelogLevel", "OFF") - System.setProperty("log4j2.statusLoggerLevel", "OFF") - MikroMarkdownCommand().main(args) -} diff --git a/cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt b/cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt deleted file mode 100644 index b683c43..0000000 --- a/cli/src/main/kotlin/com/mikromarkdown/cli/MikroMarkdownCommand.kt +++ /dev/null @@ -1,31 +0,0 @@ -package com.mikromarkdown.cli - -import com.github.ajalt.clikt.core.CliktCommand -import com.github.ajalt.clikt.parameters.arguments.argument -import com.github.ajalt.clikt.parameters.arguments.multiple -import com.github.ajalt.clikt.parameters.options.option -import com.github.ajalt.clikt.parameters.types.path -import io.github.lemcoder.mikromarkdown.MikroMarkdown -import io.github.lemcoder.mikromarkdown.StreamInfo - -class MikroMarkdownCommand : CliktCommand(name = "mikromarkdown") { - private val files by - argument("FILE", help = "Input files (reads stdin if omitted)").path(mustExist = true).multiple() - private val output by option("-o", "--output", help = "Output file (default: stdout)").path() - private val extension by option("-x", "--extension", help = "File extension hint (e.g. html)") - private val mimeType by option("-m", "--mime-type", help = "MIME type hint (e.g. text/html)") - - override fun run() { - val mikroMarkdown = MikroMarkdown() - - val markdown = - if (files.isEmpty()) { - val info = StreamInfo(extension = extension, mimetype = mimeType) - mikroMarkdown.convert(System.`in`.readBytes(), info).markdown - } else { - files.joinToString("\n\n") { mikroMarkdown.convert(it.toFile().absolutePath).markdown } - } - - if (output != null) output!!.toFile().writeText(markdown) else print(markdown) - } -} diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 1b7a7d0..9a43c76 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -14,10 +14,8 @@ kotlinx-resources = "0.15.0" commons-csv = "1.14.1" jackson = "2.21.3" junit = "6.1.0" -pdfbox = "3.0.7" pdfbox-android = "2.0.27.0" tika = "3.3.0" -clikt = "5.1.0" coreKtx = "1.7.0" detekt = "1.23.8" ktfmt = "0.27.0" @@ -34,10 +32,8 @@ kotlinx-resources = { module = "com.goncalossilva:resources", version.ref = "kot commons-csv = { module = "org.apache.commons:commons-csv", version.ref = "commons-csv" } jackson-kotlin = { module = "com.fasterxml.jackson.module:jackson-module-kotlin", version.ref = "jackson" } junit-jupiter = { module = "org.junit.jupiter:junit-jupiter", version.ref = "junit" } -pdfbox = { module = "org.apache.pdfbox:pdfbox", version.ref = "pdfbox" } pdfbox-android = { module = "com.tom-roush:pdfbox-android", version.ref = "pdfbox-android" } tika-core = { module = "org.apache.tika:tika-core", version.ref = "tika" } -clikt = { module = "com.github.ajalt.clikt:clikt", version.ref = "clikt" } core-ktx = { group = "androidx.test", name = "core-ktx", version.ref = "coreKtx" } konsist = { module = "com.lemonappdev:konsist", version.ref = "konsist" } diff --git a/library/build.gradle.kts b/library/build.gradle.kts index fce51b7..21cb42b 100644 --- a/library/build.gradle.kts +++ b/library/build.gradle.kts @@ -45,12 +45,7 @@ kotlin { implementation(libs.korlibs.compression) } - jvmMain { - dependencies { - implementation(libs.tika.core) - implementation(libs.pdfbox) - } - } + jvmMain { dependencies { implementation(libs.tika.core) } } androidMain { dependencies { implementation(libs.pdfbox.android) } } diff --git a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt index 48a534b..9695eb7 100644 --- a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt +++ b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt @@ -5,12 +5,16 @@ import io.github.lemcoder.mikromarkdown.converters.EpubConverter import io.github.lemcoder.mikromarkdown.converters.HtmlConverter import io.github.lemcoder.mikromarkdown.converters.JsonConverter import io.github.lemcoder.mikromarkdown.converters.MarkdownPassthroughConverter -import io.github.lemcoder.mikromarkdown.converters.PdfConverter import io.github.lemcoder.mikromarkdown.converters.PlainTextConverter import io.github.lemcoder.mikromarkdown.converters.XmlConverter import java.io.File -/** A [MikroMarkdown] with every JVM converter registered and Tika-based format detection. */ +/** + * A [MikroMarkdown] with every JVM converter registered. + * + * PDF is not among them: it needs a native library, so the `:pdfium` module provides it and the caller opts in with + * `register(PdfiumConverter())`. + */ public fun MikroMarkdown(): MikroMarkdown = MikroMarkdown(SignatureMimeDetector).apply { register(MarkdownPassthroughConverter()) @@ -19,7 +23,6 @@ public fun MikroMarkdown(): MikroMarkdown = register(JsonConverter()) register(XmlConverter()) register(EpubConverter()) - register(PdfConverter()) register(PlainTextConverter(), priority = 10.0) } diff --git a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt b/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt deleted file mode 100644 index 55a1b27..0000000 --- a/library/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt +++ /dev/null @@ -1,32 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.model.plainTextBlocks -import org.apache.pdfbox.Loader -import org.apache.pdfbox.text.PDFTextStripper - -public class PdfConverter : DocumentConverter { - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "pdf" || info.mimetype == "application/pdf" - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val doc = Loader.loadPDF(bytes) - try { - val title = doc.documentInformation?.title?.trim()?.ifBlank { null } - // Paragraph markers let the text blocks split on real paragraph breaks - // instead of collapsing a page into one block. - val stripper = - PDFTextStripper().apply { - sortByPosition = true - setAddMoreFormatting(true) - paragraphStart = "\n" - } - return Document(blocks = plainTextBlocks(stripper.getText(doc)), title = title) - } finally { - doc.close() - } - } -} diff --git a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/DumpOutputTest.kt b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/DumpOutputTest.kt deleted file mode 100644 index 1a123c5..0000000 --- a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/DumpOutputTest.kt +++ /dev/null @@ -1,26 +0,0 @@ -package io.github.lemcoder.mikromarkdown - -import java.io.File -import org.junit.jupiter.api.Test - -class DumpOutputTest { - private val mid = MikroMarkdown() - - @Test - fun dumpAll() { - for (name in - listOf( - "test.epub", - "test.pdf", - "test.csv", - "test.json", - "test.xml", - "test_blog.html", - "test_wikipedia.html", - )) { - val url = javaClass.classLoader.getResource("test_files/$name") ?: continue - val output = mid.convert(File(url.toURI()).absolutePath).markdown - File("/tmp/kt_$name.md").writeText(output) - } - } -} diff --git a/scripts/benchmark.py b/scripts/benchmark.py index b3b5729..35b6406 100644 --- a/scripts/benchmark.py +++ b/scripts/benchmark.py @@ -2,11 +2,10 @@ """Compare MikroMarkdown's Markdown against markitdown (Python) and anydoc (Rust). Usage: - ./gradlew :cli:installDist + ./gradlew :cli-native:linkReleaseExecutableMacosArm64 python3 scripts/benchmark.py [--fixtures DIR] [--out DIR] Engines are skipped (not failed) when their CLI is unavailable: - mikromarkdown cli/build/install/cli/bin/cli native cli-native/build/bin/macosArm64/releaseExecutable/cli-native.kexe markitdown `markitdown` on PATH, else `uvx markitdown[all]` anydoc-rust third-party/anydoc/target/release/examples/convert (cargo build @@ -77,12 +76,6 @@ def run(self, path: Path) -> tuple[str | None, float]: def which_engines() -> list[Engine]: engines: list[Engine] = [] - cli = REPO / "cli/build/install/cli/bin/cli" - engines.append( - Engine("mikromarkdown", [str(cli)] if cli.exists() else None, - "" if cli.exists() else "run ./gradlew :cli:installDist") - ) - if shutil.which("markitdown"): markitdown = ["markitdown"] elif shutil.which("uvx"): @@ -97,8 +90,8 @@ def which_engines() -> list[Engine]: "native", [str(native)] if native.exists() else None, "" if native.exists() else "run ./gradlew :cli-native:linkReleaseExecutableMacosArm64", - # The Kotlin/Native target carries only the converters that need no JVM library. - unsupported={"docx", "xlsx", "pptx", "epub", "pdf", "html", "htm"}, + # Office formats were removed from the project entirely. + unsupported={"docx", "xlsx", "pptx"}, ) ) diff --git a/scripts/optbench.py b/scripts/optbench.py index a745025..6030d33 100644 --- a/scripts/optbench.py +++ b/scripts/optbench.py @@ -7,10 +7,11 @@ Absolute timings drift between sessions — the same binary measured 60 ms one hour and 74 ms the next — so a change is only ever compared against the champion, interleaved, in the same run. + Output is verified against the recorded baselines first: a change that alters what we produce is -reported as broken rather than as fast. +reported as broken rather than as fast. Those baselines were recorded when a JVM CLI still existed +and matched it byte for byte, so they remain the reference for what each format should produce. """ -import os import shutil import subprocess import sys @@ -18,29 +19,30 @@ from pathlib import Path REPO = Path(__file__).resolve().parent.parent -JVM = REPO / "cli/build/install/cli/bin/cli" NATIVE = REPO / "cli-native/build/bin/macosArm64/releaseExecutable/cli-native.kexe" CHAMPION = REPO / "build/perf/champion.kexe" -CHAMPION_JVM = REPO / "build/perf/champion-cli" FIXTURES = REPO / "library/src/commonTest/resources/test_files" BASELINES = REPO / "build/benchmark" PERF = REPO / "build/perf" -# Native only runs CSV; the JVM champion covers the document formats it cannot. -TIMED = [("580 KB", PERF / "medium.csv"), ("1.8 MB", PERF / "big.csv")] -TIMED_JVM = [ +TIMED = [ + ("580 KB", PERF / "medium.csv"), + ("1.8 MB", PERF / "big.csv"), ("wiki", FIXTURES / "test_wikipedia.html"), ("epub", FIXTURES / "test.epub"), - ("pdf", FIXTURES / "test.pdf"), ("json", PERF / "big.json"), ] + +# PDF is deliberately not pinned: pdfium reads a document differently from the PDFBox that recorded +# the baselines, and that difference is documented rather than frozen. +UNVERIFIED = {"test.pdf"} + ROUNDS = 7 def build(): result = subprocess.run( - ["./gradlew", ":cli:installDist", ":cli-native:linkReleaseExecutableMacosArm64", - "--no-configuration-cache", "-q"], + ["./gradlew", ":cli-native:linkReleaseExecutableMacosArm64", "--no-configuration-cache", "-q"], capture_output=True, text=True, cwd=REPO, ) errors = [line for line in (result.stdout + result.stderr).splitlines() if line.startswith("e:")] @@ -52,37 +54,23 @@ def verify(): for baseline in sorted(BASELINES.glob("mikromarkdown_*.md")): name = baseline.name.replace("mikromarkdown_", "").removesuffix(".md") fixture = FIXTURES / name - if not fixture.exists(): + if not fixture.exists() or name in UNVERIFIED: continue - if subprocess.run([str(JVM), str(fixture)], capture_output=True, cwd=REPO).stdout != baseline.read_bytes(): - problems.append(f"jvm {name}") - for name in ("test.csv", "test.json", "test.xml", "test_blog.html", "test_wikipedia.html", "test.epub"): - fixture = FIXTURES / name - jvm = subprocess.run([str(JVM), str(fixture)], capture_output=True, cwd=REPO).stdout - native = subprocess.run([str(NATIVE), str(fixture)], capture_output=True, cwd=REPO).stdout - if jvm != native: - problems.append(f"native != jvm {name}") - big = PERF / "big.csv" - if big.exists(): - jvm = subprocess.run([str(JVM), str(big)], capture_output=True, cwd=REPO).stdout - native = subprocess.run([str(NATIVE), str(big)], capture_output=True, cwd=REPO).stdout - if jvm != native: - problems.append("native != jvm big.csv") + produced = subprocess.run([str(NATIVE), str(fixture)], capture_output=True, cwd=REPO) + if produced.returncode != 0: + problems.append(f"{name} failed to convert") + elif produced.stdout != baseline.read_bytes(): + problems.append(name) return problems -def interleaved(path, champion=None, candidate=None): +def interleaved(path): """Alternate champion and candidate so drift hits both equally. Returns (champion, candidate).""" - champion = champion or CHAMPION - candidate = candidate or NATIVE - # A copied distribution cannot use its CDS archive: the archive records absolute classpaths, and - # the JVM drops it without a word. Both sides run without it so the comparison is of the code. - environment = dict(os.environ, MIKROMARKDOWN_NO_CDS="1") champion_times, candidate_times = [], [] for _ in range(ROUNDS): - for binary, times in ((champion, champion_times), (candidate, candidate_times)): + for binary, times in ((CHAMPION, champion_times), (NATIVE, candidate_times)): start = time.perf_counter() - subprocess.run([str(binary), str(path)], capture_output=True, cwd=REPO, env=environment) + subprocess.run([str(binary), str(path)], capture_output=True, cwd=REPO) times.append((time.perf_counter() - start) * 1000) return min(champion_times), min(candidate_times) @@ -91,10 +79,7 @@ def main(): if "--promote" in sys.argv: CHAMPION.parent.mkdir(parents=True, exist_ok=True) shutil.copy(NATIVE, CHAMPION) - if CHAMPION_JVM.exists(): - shutil.rmtree(CHAMPION_JVM) - shutil.copytree(JVM.parent.parent, CHAMPION_JVM) - print("champion updated (native binary and JVM distribution)") + print("champion updated") return 0 label = sys.argv[1] if len(sys.argv) > 1 else "unlabelled" @@ -121,15 +106,6 @@ def main(): champion, candidate = interleaved(path) delta = (candidate - champion) / champion * 100 parts.append(f"{name}: {champion:.0f} -> {candidate:.0f} ({delta:+.0f}%)") - - champion_cli = CHAMPION_JVM / "bin/cli" - if champion_cli.exists(): - for name, path in TIMED_JVM: - if not path.exists(): - continue - champion, candidate = interleaved(path, champion=champion_cli, candidate=JVM) - delta = (candidate - champion) / champion * 100 - parts.append(f"{name}: {champion:.0f} -> {candidate:.0f} ({delta:+.0f}%)") print(f"{label}: " + " | ".join(parts)) return 0 diff --git a/settings.gradle.kts b/settings.gradle.kts index 771b36c..c3a41e7 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -20,8 +20,6 @@ rootProject.name = "mikromarkdown" include(":library") -include(":cli") - include(":benchmark") include(":cli-native") From 2104e545abd355ffe770ecf720d2a73d576449ad Mon Sep 17 00:00:00 2001 From: mikolaj Date: Tue, 18 Aug 2026 01:34:45 +0200 Subject: [PATCH 11/15] Add a PNG encoder for PDF images stored as raw pixels MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Most images inside a PDF are not image files. A DCTDecode stream is a JPEG and can be handed straight to an Asset, but a Flate-compressed bitmap is pixels with no container, and nothing in the Kotlin ecosystem encodes PNG on Kotlin/Native. PngEncoder writes 8-bit RGBA: signature, IHDR, a single IDAT of deflated scanlines through korlibs' ZLib, IEND, each chunk with its CRC. Row filtering stays at 'none', which trades a larger file for a far smaller encoder — deflate alone handles a screenshot well enough. Verified by decoding rather than by recording bytes. On the JVM ImageIO reads it back and every pixel matches, alpha included; a golden file would have passed just as happily for a PNG only we could read. On native the same code ran through the binary and its chunks, CRCs and pixels were checked from outside the process, since korlibs' ZLib wrapper had not been exercised there — the deflate had. Co-Authored-By: Claude Opus 5 (1M context) --- docs/common-converters-plan.md | 8 +- .../mikromarkdown/model/PngEncoder.kt | 102 ++++++++++++++++++ .../mikromarkdown/model/PngEncoderTest.kt | 63 +++++++++++ 3 files changed, 171 insertions(+), 2 deletions(-) create mode 100644 library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoder.kt create mode 100644 library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoderTest.kt diff --git a/docs/common-converters-plan.md b/docs/common-converters-plan.md index ac1e500..4814b7f 100644 --- a/docs/common-converters-plan.md +++ b/docs/common-converters-plan.md @@ -170,8 +170,12 @@ renders as a single paragraph where PDFBox produced seven. Character boxes give the vertical gaps between them give the paragraphs. ### Phase 3b — PDF images -Unstarted. Page objects, the two storage cases, the PNG encoder for raw pixels, and placement by -bounds — all as described above. +The **PNG encoder is done**: `PngEncoder` in `commonMain` writes 8-bit RGBA through korlibs' ZLib, +checked on the JVM by decoding what it writes with ImageIO and on native by validating the chunks +and CRCs outside the process. Filtering stays at "none" — a larger file for a much smaller encoder. + +What is left is the pdfium side: walking page objects, telling a JPEG stream (hand it on) from raw +pixels (encode), and placing images in reading order by their bounds. ### Phase 4 — FB2, if wanted Book's Story parses FB2 with the same walker it uses for HTML, through Jsoup's XML mode. Ksoup has diff --git a/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoder.kt b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoder.kt new file mode 100644 index 0000000..9d4d3cf --- /dev/null +++ b/library/src/commonMain/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoder.kt @@ -0,0 +1,102 @@ +package io.github.lemcoder.mikromarkdown.model + +import korlibs.io.compression.compress +import korlibs.io.compression.deflate.ZLib + +/** + * Writes 8-bit RGBA pixels as a PNG. + * + * Needed because most images inside a PDF are not image files: a JPEG stream can be handed on as it stands, but a + * Flate-compressed bitmap is raw pixels with no container, and every target needs one to produce. Nothing in the Kotlin + * ecosystem encodes PNG on Kotlin/Native, and the format's essentials are small — a signature, three chunks, a CRC + * apiece, and the deflate that korlibs-compression already provides. + * + * Filtering is left at "none", which trades a larger file for a much simpler encoder. A screenshot inside a PDF + * compresses well enough on deflate alone. + */ +public object PngEncoder { + + private val SIGNATURE = byteArrayOf(-119, 80, 78, 71, 13, 10, 26, 10) + + private const val BIT_DEPTH = 8 + private const val COLOR_TYPE_RGBA = 6 + private const val CHANNELS = 4 + + /** + * @param pixels RGBA, four bytes per pixel, row-major, [width] * [height] * 4 bytes long. + * @return the PNG file, or null if the dimensions and the buffer disagree. + */ + public fun encode(width: Int, height: Int, pixels: ByteArray): ByteArray? { + if (width <= 0 || height <= 0) return null + if (pixels.size != width * height * CHANNELS) return null + + val header = ByteArray(13) + header.writeInt(0, width) + header.writeInt(4, height) + header[8] = BIT_DEPTH.toByte() + header[9] = COLOR_TYPE_RGBA.toByte() + // Compression 0 (deflate), filter 0 (adaptive), interlace 0 (none) — the only values PNG has. + + val chunks = + listOf( + SIGNATURE, + chunk("IHDR", header), + chunk("IDAT", scanlines(width, height, pixels).compress(ZLib)), + chunk("IEND", ByteArray(0)), + ) + + // Concatenated by hand: a list of boxed bytes would cost more than the encoding does. + val out = ByteArray(chunks.sumOf { it.size }) + var at = 0 + for (part in chunks) { + part.copyInto(out, at) + at += part.size + } + return out + } + + /** Each row is preceded by its filter byte; 0 means the row is stored as it is. */ + private fun scanlines(width: Int, height: Int, pixels: ByteArray): ByteArray { + val stride = width * CHANNELS + val out = ByteArray(height * (stride + 1)) + for (row in 0 until height) { + val target = row * (stride + 1) + out[target] = 0 + pixels.copyInto(out, target + 1, row * stride, (row + 1) * stride) + } + return out + } + + /** length, type, payload, CRC of type and payload — the shape every PNG chunk shares. */ + private fun chunk(type: String, payload: ByteArray): ByteArray { + val typeBytes = type.encodeToByteArray() + val out = ByteArray(payload.size + 12) + out.writeInt(0, payload.size) + typeBytes.copyInto(out, 4) + payload.copyInto(out, 8) + out.writeInt(payload.size + 8, crc32(typeBytes, payload)) + return out + } + + private fun ByteArray.writeInt(at: Int, value: Int) { + this[at] = (value ushr 24).toByte() + this[at + 1] = (value ushr 16).toByte() + this[at + 2] = (value ushr 8).toByte() + this[at + 3] = value.toByte() + } + + private val CRC_TABLE = + IntArray(256) { index -> + var value = index + repeat(8) { value = if (value and 1 != 0) 0xEDB88320.toInt() xor (value ushr 1) else value ushr 1 } + value + } + + private fun crc32(vararg parts: ByteArray): Int { + var crc = -1 + for (part in parts) { + for (byte in part) crc = CRC_TABLE[(crc xor byte.toInt()) and 0xFF] xor (crc ushr 8) + } + return crc.inv() + } +} diff --git a/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoderTest.kt b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoderTest.kt new file mode 100644 index 0000000..de9b4a7 --- /dev/null +++ b/library/src/jvmTest/kotlin/io/github/lemcoder/mikromarkdown/model/PngEncoderTest.kt @@ -0,0 +1,63 @@ +package io.github.lemcoder.mikromarkdown.model + +import java.io.ByteArrayInputStream +import javax.imageio.ImageIO +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertNotNull +import kotlin.test.assertNull + +/** + * The encoder is checked by decoding what it writes with an independent decoder — ImageIO — rather than by comparing + * bytes against a recording. A PNG we wrote and only we can read would pass a golden-file test and still be useless to + * a Compose or SwiftUI reader. + */ +class PngEncoderTest { + + @Test + fun `ImageIO reads back every pixel, alpha included`() { + val width = 7 + val height = 5 + val pixels = ByteArray(width * height * 4) + for (y in 0 until height) { + for (x in 0 until width) { + val at = (y * width + x) * 4 + pixels[at] = (x * 30).toByte() + pixels[at + 1] = (y * 50).toByte() + pixels[at + 2] = ((x + y) * 20).toByte() + pixels[at + 3] = (255 - x * 10).toByte() + } + } + + val png = assertNotNull(PngEncoder.encode(width, height, pixels)) + val image = assertNotNull(ImageIO.read(ByteArrayInputStream(png)), "ImageIO could not read it") + + assertEquals(width, image.width) + assertEquals(height, image.height) + for (y in 0 until height) { + for (x in 0 until width) { + val at = (y * width + x) * 4 + val argb = image.getRGB(x, y) + assertEquals(pixels[at].toInt() and 0xFF, (argb shr 16) and 0xFF, "red at $x,$y") + assertEquals(pixels[at + 1].toInt() and 0xFF, (argb shr 8) and 0xFF, "green at $x,$y") + assertEquals(pixels[at + 2].toInt() and 0xFF, argb and 0xFF, "blue at $x,$y") + assertEquals(pixels[at + 3].toInt() and 0xFF, (argb ushr 24) and 0xFF, "alpha at $x,$y") + } + } + } + + @Test + fun `the signature and chunk order are what a decoder expects`() { + val png = assertNotNull(PngEncoder.encode(1, 1, ByteArray(4))) + + assertEquals(listOf(137, 80, 78, 71, 13, 10, 26, 10), png.take(8).map { it.toInt() and 0xFF }) + val text = png.decodeToString() + assertEquals(listOf("IHDR", "IDAT", "IEND"), listOf("IHDR", "IDAT", "IEND").sortedBy { text.indexOf(it) }) + } + + @Test + fun `dimensions that disagree with the buffer are refused`() { + assertNull(PngEncoder.encode(2, 2, ByteArray(4)), "a short buffer must not produce a truncated image") + assertNull(PngEncoder.encode(0, 4, ByteArray(0))) + } +} From 9c0a00fbd07aad9f05ec91a2edab26daa3008279 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Tue, 18 Aug 2026 10:20:32 +0200 Subject: [PATCH 12/15] PDF on Android through pdfium, and pdfbox-android is gone The :pdfium module now builds a JNI stub per Android ABI as well as for the host: arm64-v8a and x86_64, each linked against the pdfium published for it, landing in jniLibs where AGP packages them. Both are real ELF objects naming libpdfium.so. JVM and Android share one source set for the bindings and the actual written over them, so the two legs cannot drift. Fetching the binaries moves into pdfium-binaries.gradle.kts. It was already the longest thing in the module's build file and it grows by a line per platform. The NDK toolchain exposed one thing our CMakeLists has to handle rather than the plugin: cross-compiling confines find_library to the sysroot, so pdfium unpacked elsewhere is invisible and the error names the package, not the cause. The search is widened for Android only. Needs three more plugin changes, published locally until released: interops declared on a source set, the NDK toolchain for ABI builds, and a host build alongside the ABIs. Co-Authored-By: Claude Opus 5 (1M context) --- gradle/libs.versions.toml | 2 - library/build.gradle.kts | 2 - .../FileIntegrationTest.android.kt | 2 +- .../mikromarkdown/TestMikroMarkdown.kt | 2 +- .../mikromarkdown/MikroMarkdownFactory.kt | 12 +- .../mikromarkdown/converters/PdfConverter.kt | 32 ---- pdfium/build.gradle.kts | 139 +++++++----------- pdfium/native/CMakeLists.txt | 10 ++ pdfium/pdfium-binaries.gradle.kts | 76 ++++++++++ .../lemcoder/mikromarkdown/pdf/Pdfium.jni.kt} | 0 10 files changed, 144 insertions(+), 133 deletions(-) delete mode 100644 library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt create mode 100644 pdfium/pdfium-binaries.gradle.kts rename pdfium/src/{jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt => jniMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jni.kt} (100%) diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 9a43c76..3f394ec 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -14,7 +14,6 @@ kotlinx-resources = "0.15.0" commons-csv = "1.14.1" jackson = "2.21.3" junit = "6.1.0" -pdfbox-android = "2.0.27.0" tika = "3.3.0" coreKtx = "1.7.0" detekt = "1.23.8" @@ -32,7 +31,6 @@ kotlinx-resources = { module = "com.goncalossilva:resources", version.ref = "kot commons-csv = { module = "org.apache.commons:commons-csv", version.ref = "commons-csv" } jackson-kotlin = { module = "com.fasterxml.jackson.module:jackson-module-kotlin", version.ref = "jackson" } junit-jupiter = { module = "org.junit.jupiter:junit-jupiter", version.ref = "junit" } -pdfbox-android = { module = "com.tom-roush:pdfbox-android", version.ref = "pdfbox-android" } tika-core = { module = "org.apache.tika:tika-core", version.ref = "tika" } core-ktx = { group = "androidx.test", name = "core-ktx", version.ref = "coreKtx" } konsist = { module = "com.lemonappdev:konsist", version.ref = "konsist" } diff --git a/library/build.gradle.kts b/library/build.gradle.kts index 21cb42b..f750228 100644 --- a/library/build.gradle.kts +++ b/library/build.gradle.kts @@ -47,8 +47,6 @@ kotlin { jvmMain { dependencies { implementation(libs.tika.core) } } - androidMain { dependencies { implementation(libs.pdfbox.android) } } - commonTest.dependencies { implementation(libs.kotlin.test) implementation(libs.kotlinx.resources) diff --git a/library/src/androidDeviceTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.android.kt b/library/src/androidDeviceTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.android.kt index 97d5560..15c3cc4 100644 --- a/library/src/androidDeviceTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.android.kt +++ b/library/src/androidDeviceTest/kotlin/io/github/lemcoder/mikromarkdown/FileIntegrationTest.android.kt @@ -1,5 +1,5 @@ package io.github.lemcoder.mikromarkdown actual fun testMikroMarkdown(): MikroMarkdown { - return MikroMarkdown(context = null) + return MikroMarkdown() } diff --git a/library/src/androidHostTest/kotlin/io/github/lemcoder/mikromarkdown/TestMikroMarkdown.kt b/library/src/androidHostTest/kotlin/io/github/lemcoder/mikromarkdown/TestMikroMarkdown.kt index 1b9af9d..7c1314e 100644 --- a/library/src/androidHostTest/kotlin/io/github/lemcoder/mikromarkdown/TestMikroMarkdown.kt +++ b/library/src/androidHostTest/kotlin/io/github/lemcoder/mikromarkdown/TestMikroMarkdown.kt @@ -1,3 +1,3 @@ package io.github.lemcoder.mikromarkdown -actual fun testMikroMarkdown(): MikroMarkdown = MikroMarkdown(context = null) +actual fun testMikroMarkdown(): MikroMarkdown = MikroMarkdown() diff --git a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt index 292c9f9..8463eae 100644 --- a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt +++ b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/MikroMarkdownFactory.kt @@ -1,13 +1,10 @@ package io.github.lemcoder.mikromarkdown -import android.content.Context -import com.tom_roush.pdfbox.android.PDFBoxResourceLoader import io.github.lemcoder.mikromarkdown.converters.CsvConverter import io.github.lemcoder.mikromarkdown.converters.EpubConverter import io.github.lemcoder.mikromarkdown.converters.HtmlConverter import io.github.lemcoder.mikromarkdown.converters.JsonConverter import io.github.lemcoder.mikromarkdown.converters.MarkdownPassthroughConverter -import io.github.lemcoder.mikromarkdown.converters.PdfConverter import io.github.lemcoder.mikromarkdown.converters.PlainTextConverter import io.github.lemcoder.mikromarkdown.converters.XmlConverter import java.io.File @@ -15,9 +12,10 @@ import java.io.File /** * A [MikroMarkdown] with every Android converter registered. * - * PDF support needs a [Context]: pdfbox-android loads its resources from the app's assets. + * PDF is not among them: it needs a native library, so the `:pdfium` module provides it and the caller opts in with + * `register(PdfiumConverter())`. */ -public fun MikroMarkdown(context: Context? = null): MikroMarkdown = +public fun MikroMarkdown(): MikroMarkdown = MikroMarkdown(SignatureMimeDetector).apply { register(MarkdownPassthroughConverter()) register(HtmlConverter()) @@ -25,10 +23,6 @@ public fun MikroMarkdown(context: Context? = null): MikroMarkdown = register(JsonConverter()) register(XmlConverter()) register(EpubConverter()) - if (context != null) { - PDFBoxResourceLoader.init(context) - register(PdfConverter()) - } register(PlainTextConverter(), priority = 10.0) } diff --git a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt b/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt deleted file mode 100644 index 3e77b98..0000000 --- a/library/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/converters/PdfConverter.kt +++ /dev/null @@ -1,32 +0,0 @@ -package io.github.lemcoder.mikromarkdown.converters - -import com.tom_roush.pdfbox.pdmodel.PDDocument -import com.tom_roush.pdfbox.text.PDFTextStripper -import io.github.lemcoder.mikromarkdown.DocumentConverter -import io.github.lemcoder.mikromarkdown.StreamInfo -import io.github.lemcoder.mikromarkdown.model.Document -import io.github.lemcoder.mikromarkdown.model.plainTextBlocks - -public class PdfConverter : DocumentConverter { - override fun accepts(bytes: ByteArray, info: StreamInfo): Boolean { - return info.extension == "pdf" || info.mimetype == "application/pdf" - } - - override fun parse(bytes: ByteArray, info: StreamInfo): Document { - val doc = PDDocument.load(bytes) - try { - val title = doc.documentInformation?.title?.trim()?.ifBlank { null } - // Paragraph markers let the text blocks split on real paragraph breaks - // instead of collapsing a page into one block. - val stripper = - PDFTextStripper().apply { - sortByPosition = true - setAddMoreFormatting(true) - paragraphStart = "\n" - } - return Document(blocks = plainTextBlocks(stripper.getText(doc)), title = title) - } finally { - doc.close() - } - } -} diff --git a/pdfium/build.gradle.kts b/pdfium/build.gradle.kts index eaa7da3..eef5ff2 100644 --- a/pdfium/build.gradle.kts +++ b/pdfium/build.gradle.kts @@ -1,113 +1,52 @@ import io.github.lemcoder.interop.jvmInterops -import java.security.MessageDigest plugins { alias(libs.plugins.kotlinMultiplatform) + alias(libs.plugins.android.kotlin.multiplatform.library) alias(libs.plugins.konanplugin) } /** * PDF support, kept in its own module so it can be dropped or extracted whole. * - * pdfium is a prebuilt binary rather than a source dependency: the release is pinned and verified by checksum, unpacked - * into the build directory, and never committed. + * pdfium is a prebuilt binary rather than a source dependency; fetching and verifying it lives in + * pdfium-binaries.gradle.kts. */ -val pdfiumRelease = "chromium/8009" - -val pdfiumArchives = mapOf("mac-arm64" to "b1f2f17c7432a9942514dda5094ee9822c743bdfd07e7187725efbd34fde941f") - -val pdfiumRoot: Provider = layout.buildDirectory.dir("pdfium") - -val downloadPdfium by tasks.registering { - description = "Downloads and unpacks the pinned pdfium binaries." - outputs.dir(pdfiumRoot) - - doLast { - for ((platform, sha256) in pdfiumArchives) { - val target = pdfiumRoot.get().dir(platform).asFile - if (target.resolve("lib").exists()) continue - - val archive = pdfiumRoot.get().file("pdfium-$platform.tgz").asFile - archive.parentFile.mkdirs() - if (!archive.exists()) { - val url = - "https://github.com/bblanchon/pdfium-binaries/releases/download/" + - "$pdfiumRelease/pdfium-$platform.tgz" - logger.lifecycle("downloading pdfium $pdfiumRelease for $platform") - uri(url).toURL().openStream().use { input -> - archive.outputStream().use { output -> input.copyTo(output) } - } - } +apply(from = "pdfium-binaries.gradle.kts") - val digest = - MessageDigest.getInstance("SHA-256").digest(archive.readBytes()).joinToString("") { - (it.toInt() and 0xFF).toString(16).padStart(2, '0') - } - check(digest == sha256) { "pdfium-$platform.tgz checksum $digest, expected $sha256" } - - target.mkdirs() - providers - .exec { commandLine("tar", "xzf", archive.absolutePath, "-C", target.absolutePath) } - .standardOutput - .asText - .get() - - // The published dylib calls itself ./libpdfium.dylib, which the loader resolves - // against the working directory rather than the binary. Rewrite it to @rpath so a - // linked executable can find it wherever it runs from. - val dylib = target.resolve("lib/libpdfium.dylib") - if (dylib.exists()) { - providers - .exec { commandLine("install_name_tool", "-id", "@rpath/libpdfium.dylib", dylib.absolutePath) } - .standardOutput - .asText - .get() - } - } - } -} +@Suppress("UNCHECKED_CAST") val pdfiumRoot = extra["pdfiumRoot"] as Directory + +@Suppress("UNCHECKED_CAST") val pdfiumAbis = extra["pdfiumAbis"] as Map kotlin { macosArm64 { - val platform = pdfiumRoot.get().dir("mac-arm64") + val platform = pdfiumRoot.dir("mac-arm64") compilations.getByName("main").cinterops.create("pdfium") { defFile("src/nativeInterop/cinterop/pdfium.def") includeDirs(platform.dir("include")) // The archive ships a dylib, so the binary carries an rpath to find it at run time. extraOpts("-libraryPath", platform.dir("lib").asFile.absolutePath) } - binaries.executable { - entryPoint = "io.github.lemcoder.mikromarkdown.pdf.main" - linkerOpts( - "-L${platform.dir("lib").asFile.absolutePath}", - "-lpdfium", - "-rpath", - platform.dir("lib").asFile.absolutePath, - ) - } } - jvm { - // The JNI leg binds the same .def the native target does, so one declaration covers both. - compilations["main"].jvmInterops { - create("pdfium") { - defFile(project.file("src/nativeInterop/cinterop/pdfium.def")) - includeDirs.from(pdfiumRoot.get().dir("mac-arm64/include")) - - externalNativeBuild { - cmake { - path.set(project.file("native/CMakeLists.txt")) - targets.add("pdfium-jni") - arguments.add("-DPDFium_DIR=${pdfiumRoot.get().dir("mac-arm64").asFile.absolutePath}") - } - } - } - } + androidLibrary { + namespace = "io.github.lemcoder.mikromarkdown.pdfium" + compileSdk = libs.versions.android.compileSdk.get().toInt() + minSdk = libs.versions.android.minSdk.get().toInt() + withHostTestBuilder {}.configure {} } + jvm() + sourceSets { - macosArm64Main.dependencies { implementation(project(":library")) } - jvmMain.dependencies { implementation(project(":library")) } + // JVM and Android call the same generated bridges, so both the bindings and the actual + // written over them live once, in a source set the two share. + val jniMain by creating { dependsOn(commonMain.get()) } + + jvmMain { dependsOn(jniMain) } + androidMain { dependsOn(jniMain) } + + commonMain.dependencies { implementation(project(":library")) } jvmTest.dependencies { implementation(libs.kotlin.test) } } } @@ -122,9 +61,37 @@ tasks.named("jvmTest") { ) } +/** + * One declaration serves the JVM and every Android ABI: the same .def cinterop binds, generated into the source set + * both share, and CMake linking a stub per platform against the pdfium built for it. + */ +jvmInterops(kotlin.sourceSets.getByName("jniMain")) { + create("pdfium") { + defFile(project.file("src/nativeInterop/cinterop/pdfium.def")) + includeDirs.from(pdfiumRoot.dir("mac-arm64/include")) + + externalNativeBuild { + cmake { + path.set(project.file("native/CMakeLists.txt")) + targets.add("pdfium-jni") + arguments.add("-DPDFium_DIR=${pdfiumRoot.dir("mac-arm64").asFile.absolutePath}") + // The desktop JVM needs the host stub too, not only the Android ABIs. + hostBuild.set(true) + + for ((abi, platformName) in pdfiumAbis) { + abi(abi) { + platform.set(libs.versions.android.minSdk.get().toInt()) + arguments.add("-DPDFium_DIR=${pdfiumRoot.dir(platformName).asFile.absolutePath}") + } + } + } + } + } +} + // Every binding path needs the headers and the library unpacked first. -tasks.matching { it.name.startsWith("cinteropPdfium") }.configureEach { dependsOn(downloadPdfium) } +tasks.matching { it.name.startsWith("cinteropPdfium") }.configureEach { dependsOn("downloadPdfium") } tasks .matching { it.name.startsWith("generateJvmInterop") || it.name.startsWith("cmakeConfigure") } - .configureEach { dependsOn(downloadPdfium) } + .configureEach { dependsOn("downloadPdfium") } diff --git a/pdfium/native/CMakeLists.txt b/pdfium/native/CMakeLists.txt index eef696d..1259d5f 100644 --- a/pdfium/native/CMakeLists.txt +++ b/pdfium/native/CMakeLists.txt @@ -7,6 +7,16 @@ set(KONAN_JNI_LIB_NAME "" CACHE STRING "") set(KONAN_JNI_INCLUDE_DIRS "" CACHE STRING "") # pdfium ships PDFiumConfig.cmake; PDFium_DIR points at the unpacked release. +# +# Cross-compiling for Android, the NDK toolchain confines find_library and find_path to the sysroot, +# so a prebuilt library unpacked elsewhere is invisible and the error names the package rather than +# the cause. pdfium is exactly that, so the search is widened for it. +if(ANDROID) + list(APPEND CMAKE_FIND_ROOT_PATH "${PDFium_DIR}") + set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY BOTH) + set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE BOTH) +endif() + find_package(PDFium REQUIRED) file(GLOB JNI_SOURCES "${KONAN_JNI_STUB_DIR}/*.c") diff --git a/pdfium/pdfium-binaries.gradle.kts b/pdfium/pdfium-binaries.gradle.kts new file mode 100644 index 0000000..46e57e7 --- /dev/null +++ b/pdfium/pdfium-binaries.gradle.kts @@ -0,0 +1,76 @@ +import java.security.MessageDigest + +/** + * Fetches the prebuilt pdfium binaries. + * + * Applied by the module's build file rather than living in it: pinning a release, verifying it and unpacking it per + * platform is its own concern, and it is the part most likely to grow — one entry per platform we support. + * + * Exposes through `extra`: + * - `pdfiumRoot` the directory holding `/include` and `/lib` + * - `pdfiumAbis` Android ABI name to the platform name pdfium publishes + */ +val pdfiumRelease = "chromium/8009" + +val pdfiumArchives = + mapOf( + "mac-arm64" to "b1f2f17c7432a9942514dda5094ee9822c743bdfd07e7187725efbd34fde941f", + "android-arm64" to "eebf9df88c68a080efd379058651596c96043117dfdbe71ec0d03c953ae7e805", + "android-x64" to "d1068eca5710d77653d453fa7d922dbba8e97b9d1d5dc06ce02aa0d8d599c20a", + ) + +val pdfiumRoot: Directory = layout.buildDirectory.dir("pdfium").get() + +extra["pdfiumRoot"] = pdfiumRoot + +extra["pdfiumAbis"] = mapOf("arm64-v8a" to "android-arm64", "x86_64" to "android-x64") + +tasks.register("downloadPdfium") { + description = "Downloads and unpacks the pinned pdfium binaries." + group = "build setup" + outputs.dir(pdfiumRoot) + + doLast { + for ((platform, sha256) in pdfiumArchives) { + val target = pdfiumRoot.dir(platform).asFile + if (target.resolve("lib").exists()) continue + + val archive = pdfiumRoot.file("pdfium-$platform.tgz").asFile + archive.parentFile.mkdirs() + if (!archive.exists()) { + val url = + "https://github.com/bblanchon/pdfium-binaries/releases/download/" + + "$pdfiumRelease/pdfium-$platform.tgz" + logger.lifecycle("downloading pdfium $pdfiumRelease for $platform") + uri(url).toURL().openStream().use { input -> + archive.outputStream().use { output -> input.copyTo(output) } + } + } + + val digest = + MessageDigest.getInstance("SHA-256").digest(archive.readBytes()).joinToString("") { + (it.toInt() and 0xFF).toString(16).padStart(2, '0') + } + check(digest == sha256) { "pdfium-$platform.tgz checksum $digest, expected $sha256" } + + target.mkdirs() + providers + .exec { commandLine("tar", "xzf", archive.absolutePath, "-C", target.absolutePath) } + .standardOutput + .asText + .get() + + // The macOS dylib calls itself ./libpdfium.dylib, which the loader resolves against the + // working directory rather than the binary. Rewrite it to @rpath so anything linking it + // can find it. Mach-O only; the Android .so needs nothing. + val dylib = target.resolve("lib/libpdfium.dylib") + if (dylib.exists()) { + providers + .exec { commandLine("install_name_tool", "-id", "@rpath/libpdfium.dylib", dylib.absolutePath) } + .standardOutput + .asText + .get() + } + } + } +} diff --git a/pdfium/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt b/pdfium/src/jniMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jni.kt similarity index 100% rename from pdfium/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt rename to pdfium/src/jniMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jni.kt From c36f15884decac65c55581401fac6a36b806ad95 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Tue, 18 Aug 2026 11:15:25 +0200 Subject: [PATCH 13/15] Keep the JVM and Android PDF legs apart MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reverts the shared jniMain source set. The two legs read alike today, but JVM and Android diverge — library loading, lifecycle, what a file path means — and a shared source set makes the first divergence a restructure instead of an edit. Each compilation declares its own interop against the same .def: the JVM one builds a host stub, the Android one builds arm64-v8a and x86_64. Bindings generate per compilation and the actual is written per source set, which is the duplication the split is worth. Co-Authored-By: Claude Opus 5 (1M context) --- pdfium/build.gradle.kts | 35 ++++---- .../mikromarkdown/pdf/Pdfium.android.kt} | 0 .../lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt | 80 +++++++++++++++++++ 3 files changed, 101 insertions(+), 14 deletions(-) rename pdfium/src/{jniMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jni.kt => androidMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.android.kt} (100%) create mode 100644 pdfium/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt diff --git a/pdfium/build.gradle.kts b/pdfium/build.gradle.kts index eef5ff2..25ad398 100644 --- a/pdfium/build.gradle.kts +++ b/pdfium/build.gradle.kts @@ -39,13 +39,6 @@ kotlin { jvm() sourceSets { - // JVM and Android call the same generated bridges, so both the bindings and the actual - // written over them live once, in a source set the two share. - val jniMain by creating { dependsOn(commonMain.get()) } - - jvmMain { dependsOn(jniMain) } - androidMain { dependsOn(jniMain) } - commonMain.dependencies { implementation(project(":library")) } jvmTest.dependencies { implementation(libs.kotlin.test) } } @@ -62,10 +55,12 @@ tasks.named("jvmTest") { } /** - * One declaration serves the JVM and every Android ABI: the same .def cinterop binds, generated into the source set - * both share, and CMake linking a stub per platform against the pdfium built for it. + * The JVM and Android legs bind the same .def separately, one declaration each. + * + * They could share one, and deliberately do not: the two runtimes diverge over time — how a library is loaded, what a + * file path means — and a shared declaration turns the first difference into a restructure rather than an edit. */ -jvmInterops(kotlin.sourceSets.getByName("jniMain")) { +kotlin.jvm().compilations["main"].jvmInterops { create("pdfium") { defFile(project.file("src/nativeInterop/cinterop/pdfium.def")) includeDirs.from(pdfiumRoot.dir("mac-arm64/include")) @@ -75,11 +70,23 @@ jvmInterops(kotlin.sourceSets.getByName("jniMain")) { path.set(project.file("native/CMakeLists.txt")) targets.add("pdfium-jni") arguments.add("-DPDFium_DIR=${pdfiumRoot.dir("mac-arm64").asFile.absolutePath}") - // The desktop JVM needs the host stub too, not only the Android ABIs. - hostBuild.set(true) + } + } + } +} + +kotlin.targets.getByName("android").compilations.getByName("main").jvmInterops { + create("pdfiumAndroid") { + defFile(project.file("src/nativeInterop/cinterop/pdfium.def")) + includeDirs.from(pdfiumRoot.dir("android-arm64/include")) + + externalNativeBuild { + cmake { + path.set(project.file("native/CMakeLists.txt")) + targets.add("pdfium-jni") - for ((abi, platformName) in pdfiumAbis) { - abi(abi) { + for ((abiName, platformName) in pdfiumAbis) { + abi(abiName) { platform.set(libs.versions.android.minSdk.get().toInt()) arguments.add("-DPDFium_DIR=${pdfiumRoot.dir(platformName).asFile.absolutePath}") } diff --git a/pdfium/src/jniMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jni.kt b/pdfium/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.android.kt similarity index 100% rename from pdfium/src/jniMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jni.kt rename to pdfium/src/androidMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.android.kt diff --git a/pdfium/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt b/pdfium/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt new file mode 100644 index 0000000..243f89a --- /dev/null +++ b/pdfium/src/jvmMain/kotlin/io/github/lemcoder/mikromarkdown/pdf/Pdfium.jvm.kt @@ -0,0 +1,80 @@ +package io.github.lemcoder.mikromarkdown.pdf + +import pdfium.kniBridge1 +import pdfium.kniBridge15 +import pdfium.kniBridge16 +import pdfium.kniBridge2 +import pdfium.kniBridge26 +import pdfium.kniBridge27 +import pdfium.kniBridge5 +import pdfium.kniBridge52 +import pdfium.kniBridge53 +import pdfium.kniBridge54 +import pdfium.kniBridge71 + +/** + * The JVM half, over the JNI bridges the Konan plugin generates from the same `.def` cinterop binds. + * + * The bridges are numbered rather than named — that is what a runtime-free binding looks like — so each is wrapped here + * with the name from its doc comment, and nothing else in the module sees them. + */ +internal actual fun extractText(bytes: ByteArray): String { + val text = StringBuilder() + + initLibrary() + try { + val document = loadDocument(bytes, null) + if (document == 0L) return "" + try { + for (index in 0 until pageCount(document)) { + val page = loadPage(document, index) + if (page == 0L) continue + val textPage = loadTextPage(page) + if (textPage != 0L) { + text.append(pageText(textPage)) + text.append('\n') + closeTextPage(textPage) + } + closePage(page) + } + } finally { + closeDocument(document) + } + } finally { + destroyLibrary() + } + + return text.toString() +} + +/** pdfium writes UTF-16 into a caller-supplied buffer and counts the terminating NUL. */ +private fun pageText(textPage: Long): String { + val count = charCount(textPage) + if (count <= 0) return "" + val buffer = ShortArray(count + 1) + val written = readText(textPage, 0, count, buffer) + return if (written <= 1) "" else CharArray(written - 1) { Char(buffer[it].toInt() and 0xFFFF) }.concatToString() +} + +private fun initLibrary() = kniBridge1() + +private fun destroyLibrary() = kniBridge2() + +private fun loadDocument(bytes: ByteArray, password: String?): Long = kniBridge5(bytes, bytes.size, password) + +private fun pageCount(document: Long): Int = kniBridge15(document) + +private fun loadPage(document: Long, index: Int): Long = kniBridge16(document, index) + +private fun closePage(page: Long) = kniBridge26(page) + +private fun closeDocument(document: Long) = kniBridge27(document) + +private fun loadTextPage(page: Long): Long = kniBridge52(page) + +private fun closeTextPage(textPage: Long) = kniBridge53(textPage) + +private fun charCount(textPage: Long): Int = kniBridge54(textPage) + +private fun readText(textPage: Long, start: Int, count: Int, buffer: ShortArray): Int = + kniBridge71(textPage, start, count, buffer) From c40be20963e21da0e574078af393abce068e14f8 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Tue, 18 Aug 2026 15:47:24 +0200 Subject: [PATCH 14/15] Drop the last mention of jvmShared from the plan The source set went when EPUB moved to commonMain; the migration rule naming it outlived it. The rule now says what it means: accidental duplication is caught, deliberate splits are written down. Co-Authored-By: Claude Opus 5 (1M context) --- docs/common-converters-plan.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/common-converters-plan.md b/docs/common-converters-plan.md index 4814b7f..e43eda0 100644 --- a/docs/common-converters-plan.md +++ b/docs/common-converters-plan.md @@ -214,8 +214,9 @@ Three things this needs that do not exist yet: ## Ground rules per phase -1. The new implementation lands in `commonMain`; the `jvmShared` version is deleted in the same - commit. The Konsist duplicate-file rule keeps anything from being copied per target. +1. The new implementation lands in `commonMain` and the platform one is deleted in the same commit. + The Konsist duplicate-file rule catches anything copied per target by accident; where a split is + deliberate, as the PDF module's JVM and Android legs are, it is written down as such. 2. `scripts/optbench.py` must report every fixture byte-identical on both targets before timings are read. Where a difference is deliberate — PDF, and possibly HTML — the baseline is updated in the same commit with the diff quoted in the message. From 63a9e6958696b2a944d787f94c10910ace5a2038 Mon Sep 17 00:00:00 2001 From: mikolaj Date: Tue, 18 Aug 2026 16:48:13 +0200 Subject: [PATCH 15/15] Resolve KonanPlugin from the plugin portal, and fix the configuration cache MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1.2.0-alpha06 is published, so mavenLocal() comes back out of settings.gradle.kts. Verified by hiding the local artifact and resolving with --refresh-dependencies. downloadPdfium was a doLast block in an applied script, and calling uri(), providers or logger there captures the script object, which the configuration cache cannot serialize. It is now a task class taking the release, the checksums and the output directory as inputs and an injected ExecOperations, so nothing of the script crosses into the action. This was failing before, not newly broken by the repository change: an earlier check reused a stale cache entry and reported success. Re-verified from a wiped build/pdfium — all three archives download, verify and unpack, and the dylib id is rewritten to @rpath. Co-Authored-By: Claude Opus 5 --- pdfium/pdfium-binaries.gradle.kts | 58 ++++++++++++++++++++----------- settings.gradle.kts | 3 -- 2 files changed, 37 insertions(+), 24 deletions(-) diff --git a/pdfium/pdfium-binaries.gradle.kts b/pdfium/pdfium-binaries.gradle.kts index 46e57e7..47eaf55 100644 --- a/pdfium/pdfium-binaries.gradle.kts +++ b/pdfium/pdfium-binaries.gradle.kts @@ -1,4 +1,7 @@ +import java.net.URI import java.security.MessageDigest +import javax.inject.Inject +import org.gradle.process.ExecOperations /** * Fetches the prebuilt pdfium binaries. @@ -25,24 +28,37 @@ extra["pdfiumRoot"] = pdfiumRoot extra["pdfiumAbis"] = mapOf("arm64-v8a" to "android-arm64", "x86_64" to "android-x64") -tasks.register("downloadPdfium") { - description = "Downloads and unpacks the pinned pdfium binaries." - group = "build setup" - outputs.dir(pdfiumRoot) +/** + * A task class rather than a `doLast` block, because the configuration cache cannot serialize the script object that a + * block in a script plugin captures the moment it calls `uri()`, `providers` or `logger`. Everything the action needs + * arrives as an input or an injected service. + */ +abstract class DownloadPdfium : DefaultTask() { + + @get:Input abstract val release: Property + + /** platform name to the SHA-256 of its archive. */ + @get:Input abstract val archives: MapProperty - doLast { - for ((platform, sha256) in pdfiumArchives) { - val target = pdfiumRoot.dir(platform).asFile + @get:OutputDirectory abstract val root: DirectoryProperty + + @get:Inject abstract val exec: ExecOperations + + @TaskAction + fun download() { + val releaseTag = release.get() + for ((platform, sha256) in archives.get()) { + val target = root.get().dir(platform).asFile if (target.resolve("lib").exists()) continue - val archive = pdfiumRoot.file("pdfium-$platform.tgz").asFile + val archive = root.get().file("pdfium-$platform.tgz").asFile archive.parentFile.mkdirs() if (!archive.exists()) { val url = "https://github.com/bblanchon/pdfium-binaries/releases/download/" + - "$pdfiumRelease/pdfium-$platform.tgz" - logger.lifecycle("downloading pdfium $pdfiumRelease for $platform") - uri(url).toURL().openStream().use { input -> + "$releaseTag/pdfium-$platform.tgz" + logger.lifecycle("downloading pdfium $releaseTag for $platform") + URI(url).toURL().openStream().use { input -> archive.outputStream().use { output -> input.copyTo(output) } } } @@ -54,23 +70,23 @@ tasks.register("downloadPdfium") { check(digest == sha256) { "pdfium-$platform.tgz checksum $digest, expected $sha256" } target.mkdirs() - providers - .exec { commandLine("tar", "xzf", archive.absolutePath, "-C", target.absolutePath) } - .standardOutput - .asText - .get() + exec.exec { commandLine("tar", "xzf", archive.absolutePath, "-C", target.absolutePath) } // The macOS dylib calls itself ./libpdfium.dylib, which the loader resolves against the // working directory rather than the binary. Rewrite it to @rpath so anything linking it // can find it. Mach-O only; the Android .so needs nothing. val dylib = target.resolve("lib/libpdfium.dylib") if (dylib.exists()) { - providers - .exec { commandLine("install_name_tool", "-id", "@rpath/libpdfium.dylib", dylib.absolutePath) } - .standardOutput - .asText - .get() + exec.exec { commandLine("install_name_tool", "-id", "@rpath/libpdfium.dylib", dylib.absolutePath) } } } } } + +tasks.register("downloadPdfium") { + description = "Downloads and unpacks the pinned pdfium binaries." + group = "build setup" + release.set(pdfiumRelease) + archives.set(pdfiumArchives) + root.set(pdfiumRoot) +} diff --git a/settings.gradle.kts b/settings.gradle.kts index c3a41e7..e678536 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -1,8 +1,5 @@ pluginManagement { repositories { - // KonanPlugin is consumed from a local publish while the void-buffer marshalling it needs - // for pdfium is unreleased; drop this once 1.2.0-alpha06 is on the portal. - mavenLocal() google() mavenCentral() gradlePluginPortal()