commonMain converters, office formats out, PDF through pdfium - #8
Merged
Conversation
Sequenced so each phase ships on its own: infrastructure, HTML, EPUB, DOCX, XLSX, PPTX, with PDF deliberately excluded. Records the library choices (korlibs-compression for ZIP, xmlutil for XML, Ksoup for HTML, all published for macosArm64), the POI surface each converter actually uses, and the two places the plan is likely to hurt: Excel number formatting, and losing POI's tolerance of malformed files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
XLSX stays on POI. Excel's number-format language is thousands of lines in DataFormatter and we call it for every non-integer cell, so it is the phase most likely to change output silently. Consequence worth naming: POI stays on the JVM classpath until it moves, so the smaller-JVM payoff is deferred while the portability one still lands. PDF becomes a separate :pdfium module, bound with KonanPlugin so one .def serves cinterop and JNI. Binaries come from bblanchon/pdfium-binaries, downloaded from a pinned release rather than committed. The module registers nothing on its own — callers opt in with register(PdfiumConverter()) — so deleting it leaves the rest building. Two costs recorded: pdfium extracts text differently from PDFBox, so that fixture baseline changes and token recall replaces byte-identity as the gate; and JVM distribution needs the library plus a generated stub per platform, which is a CI matrix rather than something one machine can produce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Compose and SwiftUI readers need images, not just text, so the pdfium module walks page objects rather than only the text layer. Two cases: DCTDecode and JPXDecode streams are already JPEG files and pass straight through, while Flate-compressed raw pixels have no image file to extract and need encoding — so commonMain gains a small PNG writer over the deflate that korlibs-compression already provides. Placement matters too: object bounds and character boxes let images be emitted in reading order instead of dumped at the end of a page, which the PDFBox route never did. Also records what every other format owes: EPUB and HTML assets, real PPTX picture bytes instead of today's fabricated filename, and the three gaps the readers hit — no asset policy (DOCX inlines base64, which is why its output is 161 KB), asset ids that can collide, and no intrinsic size on Image. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DOCX, XLSX and PPTX are editing formats; a reader meets PDF and EPUB. Deleting the three converters drops POI from the build: the distribution goes from 66 MB to 36 MB across 23 jars instead of 35, and the CDS archive from 41 MB to 23 MB. It buys no speed, which is worth stating plainly. POI was loaded lazily, so a CSV or EPUB conversion never paid for it — the startup numbers are unchanged. What improved is size and dependency surface. The fixtures stay and a test pins the behaviour callers now see, an UnsupportedFormatException rather than a wrong answer. If the formats return it is as an :office module mirroring the :pdfium design: separate module, separate dependency, registered by the caller. Also removes XLSX from the commonMain plan — it was the phase most likely to change output silently, since POI's DataFormatter implements Excel's whole number-format language — and records FB2 as a cheap addition once HTML moves to Ksoup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both dependencies are in commonMain and were checked by running them on the macosArm64 binary rather than by watching them resolve: Ksoup parses HTML, and korlibs deflate round-trips a kilobyte down to 31 bytes and back. The useful discovery is that Ksoup's XML mode reads an EPUB OPF — dc:title, manifest hrefs, spine idrefs, the cover meta — so the planned XML library drops out of the plan entirely. korlibs ships no ZIP reader, so the central directory stays ours to parse, but that was always the smaller half. Its API is under korlibs.io.compression, not korlibs.compression, which the package listing in the klib does not make obvious. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The walker was written against a Jsoup-shaped API and Ksoup is a port of it, so the change is two lines of difference: wholeText is a method rather than a property, and Charsets does not exist in commonMain. The risk was never the API — it was whether Ksoup recovers from messy HTML the same way. It does: test_blog.html and test_wikipedia.html are byte-identical through the new parser, and the native binary matches the JVM on both. HTML now converts on native, at 7 ms for the blog against the JVM's 80, and 66 ms for Wikipedia against 119. The harness verifies both HTML fixtures across targets from now on, so a future change cannot quietly diverge them. jvmShared is down to EpubConverter alone, kept only until its zip and XML move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A ZipArchive in commonMain reads the central directory, follows each entry to its local header and inflates through korlibs — korlibs supplies no reader of its own, and neither kotlinx-io nor okio reads archives. The container and package documents go through Ksoup's XML mode, so there is still no XML dependency, and chapters reuse the HTML walker from Phase 1. Output is byte-identical to the java.util.zip and javax.xml implementation it replaces, and the native binary matches the JVM. With EPUB moved, jvmShared held nothing and is deleted: every converter but PDF now lives in commonMain, and the JVM keeps only Tika and PDFBox. One architecture rule needed narrowing rather than obeying. It flagged ZipArchive's companion as a public helper under utils, but a companion inside an internal class is already unreachable; the rule now looks at top-level declarations only, and was re-checked against a planted public helper to confirm it still bites. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A :pdfium module binds pdfium with cinterop and converts a PDF on the native CLI. The release is pinned by checksum and unpacked at build time; nothing binary is committed. The library does not depend on it — the CLI registers PdfiumConverter itself, so the module can be deleted without touching anything else. Two things cost time and are worth recording. The published dylib calls itself ./libpdfium.dylib, which the loader resolves against the working directory rather than the binary, so the unpack step rewrites the id to @rpath. And Kotlin/Native does not carry a klib's linker options through to the binary that links it, so the consumer has to name pdfium itself. Against PDFBox on the same document pdfium keeps 96.2% of its tokens, and the difference is in pdfium's favour: PDFBox leaves flexibil, firming and gramming as fragments where pdfium plus de-hyphenation produces whole words. Two defects remain and one fix covers both. pdfium emits U+FFFE for a glyph with no Unicode mapping, which here is always a hyphen at a line break; the document's vocabulary decides whether to join or keep it, which gets four words right and two compounds wrong, and cannot do better because a broken word's halves are only in the text because the break put them there. Separately a page arrives as one run of text, so a PDF renders as a single paragraph where PDFBox produced seven. FPDFText_GetCharBox answers both: a hyphenation hyphen ends a line, and the gaps between lines are the paragraphs. plainTextBlocks moves from utils to model and becomes public, since a converter in another module needs it and the architecture rules keep utils unpublished. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The :pdfium module now binds pdfium twice from one .def: cinterop for the native target, and the Konan plugin's generated JNI bridges for the JVM. CMake links the stub, which is the mode the plugin documents for a vendor library — pdfium is C++, and letting konan's linker near a foreign toolchain's archive is what produces mismatched runtimes. PdfiumConverter moves to commonMain over an expect/actual extractText, so the de-hyphenation and the document building are shared and only the extraction is per-platform. Both legs produce the same 488 words from the same PDF. Needs two plugin changes, published locally until they are released: const void* now crosses as a ByteArray rather than an address, without which the JVM cannot call FPDF_LoadMemDocument at all; and the CMake build receives the JNI include roots, without which the generated stub cannot find jni.h. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PDFBox goes with the JVM CLI that used it. The library's JVM target no longer converts PDF at all — the :pdfium module does, through cinterop on native and the generated JNI bridges on the JVM, and a caller registers it. Deleting the JVM CLI takes a surprising amount of machinery with it: the class-data-sharing archive and its two-step dump, the start-script guards that made it optional, clikt, and the distribution packaging that would have had to carry two dylibs. The native binary was already faster on every fixture, so the JVM one was paying that complexity to lose. Android keeps pdfbox-android for now: the :pdfium module has no Android ABIs yet, and removing its converter would leave Android with no PDF at all. The harness follows: optbench compares the native binary against a champion and verifies its output against the recorded baselines, which the JVM CLI produced back when the two matched byte for byte. PDF is exempt from that check, since pdfium reads documents differently from PDFBox by design. DumpOutputTest is deleted rather than fixed — it wrote conversions to /tmp for eyeballing, which the baselines and the harness now do properly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Most images inside a PDF are not image files. A DCTDecode stream is a JPEG and can be handed straight to an Asset, but a Flate-compressed bitmap is pixels with no container, and nothing in the Kotlin ecosystem encodes PNG on Kotlin/Native. PngEncoder writes 8-bit RGBA: signature, IHDR, a single IDAT of deflated scanlines through korlibs' ZLib, IEND, each chunk with its CRC. Row filtering stays at 'none', which trades a larger file for a far smaller encoder — deflate alone handles a screenshot well enough. Verified by decoding rather than by recording bytes. On the JVM ImageIO reads it back and every pixel matches, alpha included; a golden file would have passed just as happily for a PNG only we could read. On native the same code ran through the binary and its chunks, CRCs and pixels were checked from outside the process, since korlibs' ZLib wrapper had not been exercised there — the deflate had. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The :pdfium module now builds a JNI stub per Android ABI as well as for the host: arm64-v8a and x86_64, each linked against the pdfium published for it, landing in jniLibs where AGP packages them. Both are real ELF objects naming libpdfium.so. JVM and Android share one source set for the bindings and the actual written over them, so the two legs cannot drift. Fetching the binaries moves into pdfium-binaries.gradle.kts. It was already the longest thing in the module's build file and it grows by a line per platform. The NDK toolchain exposed one thing our CMakeLists has to handle rather than the plugin: cross-compiling confines find_library to the sysroot, so pdfium unpacked elsewhere is invisible and the error names the package, not the cause. The search is widened for Android only. Needs three more plugin changes, published locally until released: interops declared on a source set, the NDK toolchain for ABI builds, and a host build alongside the ABIs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reverts the shared jniMain source set. The two legs read alike today, but JVM and Android diverge — library loading, lifecycle, what a file path means — and a shared source set makes the first divergence a restructure instead of an edit. Each compilation declares its own interop against the same .def: the JVM one builds a host stub, the Android one builds arm64-v8a and x86_64. Bindings generate per compilation and the actual is written per source set, which is the duplication the split is worth. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The source set went when EPUB moved to commonMain; the migration rule naming it outlived it. The rule now says what it means: accidental duplication is caught, deliberate splits are written down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lemcoder
changed the title
… cache 1.2.0-alpha06 is published, so mavenLocal() comes back out of settings.gradle.kts. Verified by hiding the local artifact and resolving with --refresh-dependencies. downloadPdfium was a doLast block in an applied script, and calling uri(), providers or logger there captures the script object, which the configuration cache cannot serialize. It is now a task class taking the release, the checksums and the output directory as inputs and an injected ExecOperations, so nothing of the script crosses into the action. This was failing before, not newly broken by the repository change: an earlier check reused a stale cache entry and reported success. Re-verified from a wiped build/pdfium — all three archives download, verify and unpack, and the dylib id is rewritten to @rpath. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
:pdfiummodule needs KonanPlugin#3, now published to the Gradle Plugin Portal as1.2.0-alpha06and resolved from there —mavenLocal()is out ofsettings.gradle.ktsagain, checked by hiding the local artifact and resolving with--refresh-dependencies.What changed
Office formats deleted, POI with them. DOCX, XLSX and PPTX are editing formats; a reader meets PDF and EPUB. The fixtures stay and a test pins that they raise
UnsupportedFormatExceptionrather than returning something wrong. If they return it is as an opt-in module, the shape:pdfiumhas.HTML and EPUB moved to
commonMain. HTML runs on Ksoup, EPUB on aZipArchivethat reads the central directory and inflates through korlibs, with Ksoup's XML mode for the container and package documents. Both are byte-identical to the Jsoup,java.util.zipandjavax.xmlimplementations they replace. Jsoup is gone; xmlutil was planned and turned out unnecessary.PDF is a separate
:pdfiummodule binding pdfium three ways from one.def: cinterop for Kotlin/Native, and the Konan plugin's generated JNI bridges for the JVM and for each Android ABI, with CMake linking the stubs. Binaries come from a pinnedbblanchon/pdfium-binariesrelease verified by checksum; nothing binary is committed.:librarydoes not depend on it — a caller writesregister(PdfiumConverter())— so the module deletes cleanly.PDFBox and pdfbox-android are gone, and so is the JVM CLI. Deleting it removed the class-data-sharing archive and its two-step dump, the start-script guards, clikt, and the packaging that would have had to carry two dylibs. The native binary was faster on every fixture, so the JVM one was paying that complexity to lose.
A PNG encoder in
commonMain, for the PDF images that are raw pixels rather than JPEG streams. Verified by decoding rather than by recording bytes: ImageIO reads it back pixel-exact on the JVM, and on native its chunks, CRCs and pixels were checked from outside the process.Where the library lands
Verification
./gradlew checkpasses. Every fixture except PDF is byte-identical tomainon both targets, and the harness refuses to report a timing otherwise. PDF is deliberately exempt: pdfium reads a document differently from PDFBox, keeping 96.2% of its tokens, and the difference favours pdfium — PDFBox leavesflexibil,firming,grammingas fragments where pdfium plus de-hyphenation produces whole words.Two things left undone, and why
PDF renders as one paragraph. pdfium returns a page as a single run of text where PDFBox produced seven paragraphs.
FPDFText_GetCharBoxgives the line boxes that recover them.Hyphen versus compound is a heuristic. pdfium emits U+FFFE where a glyph has no Unicode mapping, which in a typeset paper is the hyphen at a line break. The document's own vocabulary decides whether to join or keep it, which gets four words right and two compounds wrong — and cannot do better in principle, because a broken word's halves are only in the text because the break put them there. The real discriminator is that a hyphenation hyphen ends a line, which is the same geometry call.
PDF images need that geometry too, for placement, so all three land together in the next pass.
🤖 Generated with Claude Code