Cleanup, a use-after-free in every PDF conversion, and named pdfium bindings

[lemcoder]

Important

Four commits added after review. Two claims in the original description below did not survive them, and are corrected here.

Review fixes — 21c353f

Fourteen findings. The detector's ZIP branch had collapsed to a constant ("zip" was in the extension set but absent from the table, so only "epub" ever changed the answer) and the OLE signature went with the office cleanup, leaving legacy .doc/.xls/.ppt as No converter found for: unknown. Nothing exercised SignatureMimeDetector at all — the integration tests build their own StreamInfo and never reach detection — so 7 new tests cover it.

CI compiled neither the native target nor the pdfium bindings; there is now a macOS leg running :library:macosArm64Test :pdfium:jvmTest. ktfmt named the Android source sets explicitly and would have missed any target added later. The benchmark died on the second fixture, because the office files it cannot convert are still in the directory. The four testMikroMarkdown() actuals went, along with the expect — the factory is common now, so commonTest calls it directly.

Correction: the description below says the library has no third-party JVM dependency. commonMain declares three — kotlinx-io, Ksoup, korlibs-compression — and all three reach JVM consumers. What went was the JVM-only dependency, Tika. README now says that.

publish.yml is deleted: the publishing plugin has not been applied since 5113f53, so every release run could only fail, and README pointed at a Maven Central coordinate nothing publishes.

PDF wraps, from geometry — 7fb8260

Correction: the description below says no output changes, every fixture byte for byte. PDF output did change, and for the better. It could not be verified as stated, for the reason in the next section.

U+FFFE is not an unmapped glyph. FPDFText_HasUnicodeMapError is 0 at all eleven occurrences in the fixture and FPDFText_IsHyphen is 1: it marks a hyphen pdfium removed while joining a wrapped line. The halves arrive adjacent with nothing to say a line ended between them, which is why replace(UNMAPPED_GLYPH, "-\n") had to assume one — and why it fabricated breaks where there were none, leaked a stray - for any other undecodable glyph, and swallowed the following paragraph when a marker landed at a paragraph end.

FPDFText_GetCharBox settles the one open question: every marker sits at the right margin with the next character on a lower line, so the discarded break is put back on evidence. A marker the boxes do not confirm now leaves nothing behind.

The geometry cannot say which hyphen it was — equally true of con-|firming and chat-|optimized — so nothing guesses. The vocabulary heuristic that used to is gone: it fused chat-optimized into chatoptimized, a word in no document anywhere. Keeping the hyphen writes con-firming where the page meant confirming, which reads slightly worse and destroys nothing. TextBlocks goes 113 → 71 lines, and the pdfium sentinel leaves model/, which ArchitectureTest pins as depending on nothing.

A use-after-free in every PDF conversion — a3b45e7

The reason "byte for byte" could not be verified. FPDF_LoadMemDocument reads the caller's buffer for as long as the document is open; the generated JNI bridge releases it before returning:

jbyte* p0 = (*jniEnv)->GetByteArrayElements(jniEnv, j0, 0);
jlong kniResult = (jlong)FPDF_LoadMemDocument((void*)p0, p1, ...);
(*jniEnv)->ReleaseByteArrayElements(jniEnv, j0, p0, 0);
return kniResult;

Every page load afterwards read memory the JVM had taken back. Zeroing the array under an open document drops extraction to 0 characters, so pdfium was reading our array directly, not a copy.

It mostly worked, which is the worst way to fail: the first conversion in a process was correct and later ones lost four generated spaces, turning AutoGen uses into AutoGenuses. One fixture, repeatedly, in a single process: 5220 / 5216 / 5220 / 5216, and a different sequence in the next process. Through FPDF_LoadDocument, which owns what it reads: 5220 every time, everywhere.

Costs 0.12 ms against 2.66 ms of parse for a 90 KB PDF. The native leg was never affected — usePinned holds the array for the document's lifetime — and is untouched. A regression test now converts the same bytes twice and asserts the documents match.

anydoc pinned to v0.1.8 — b3dc095

The comparison figures were measured against whatever the submodule happened to be at. v0.1.8 does not change the hyphen handling: eleven of the twelve word- word sites in its output of our fixture are still broken words, from an unconditional space in pdf-inspector's line join. Unreported upstream.

---

Everything below describes the original commit, d0843c7.

Cleanup only. No output changes — every fixture, PDF included, renders byte for byte what it rendered on main, verified through the native CLI before and after.

517 lines deleted, 190 added.

Dead code

The builder DSL, 160 lines, zero users. DocumentBuilder/InlineBuilder/styled were written for the office converters, which built documents statement by statement. What remains constructs Document directly or comes out of HtmlToDocument; the only mention left in the repo was the README example. Two secondary constructors — Heading(level, text) and Paragraph(text) — cover what the DSL was actually pleasant for:

Document(blocks = listOf(Heading(1, "Custom"), Paragraph(bytes.decodeToString())))

Three copies of the factory. JVM, Android and macOS each registered the same seven converters, which is what a registration list looks like once every converter is common. Now one, in commonMain. The convert(file: File) overloads two of them carried had no callers at all — the JVM tests and the benchmark have always used convert(file.absolutePath).

TikaMimeDetector. Unreferenced, untested, and it put tika-core on every JVM consumer's classpath to provide an implementation of a fun interface whose entire purpose is that a caller supplies their own. The library now has no third-party JVM dependency and no jvmMain source set — production code is entirely common.

Also: HtmlComment (nothing produced it once the builder went, and RawBlock covers it); docx/xlsx/pptx and the OLE signature in the mime detector, for formats deliberately dropped; four libraries and two plugins left in the version catalog by POI, Jackson, commons-csv and kotlinx-serialization; .codegraph/.

One heuristic instead of two

PdfiumConverter decided whether a U+FFFE was the author's hyphen or the typesetter's. plainTextBlocks decided the same thing for a hyphen at a line break. Two vocabularies, two copies of Regex("[\\p{L}]{2,}"), one question. Since the unmapped glyph is a hyphen at a line break, spelling it as one leaves a single rule:

.replace(UNMAPPED_GLYPH, "-\n")

and PdfiumConverter drops from 76 lines to 29.

That also fixed the copy that was wrong. plainTextBlocks built its vocabulary from the whole text, so flexibil- broken across a line entered the vocabulary as a word and then vouched for itself, keeping a hyphen it should have dropped. pdfium's copy cut the halves out first, which is why nothing was visibly broken — and why no output moves now that the correct rule is the only one.

Two things it turned up

Konsist's duplicate-file allowlist named a file that no longer exists. The rule against one production file living in two source sets was excepting MikroMarkdownFactory and PdfConverter; the second went with the office removal, the first goes here. The allowlist is empty and the rule is now absolute.

The native target's tests were switched off. The reason recorded in library/build.gradle.kts was that the shared integration tests expected JVM-only formats — untrue since the converters moved to commonMain. Re-enabled:

> Task :library:macosArm64Test
  officeFormatsAreNotSupported[macosArm64]  testBlogHtml[macosArm64]
  testWikipediaHtml[macosArm64]  testJson[macosArm64]  testEpub[macosArm64]

API removals

Pre-1.0, and all of them things nothing used: the builder DSL, convert(File), TikaMimeDetector, HtmlComment. A caller who wants Tika-grade sniffing passes a MimeDetector — the README now says so, and shows it.

./gradlew check passes.

🤖 Generated with Claude Code