feat(reader): simple-font decoding for text extraction - #407
Open
Tim81 wants to merge 2 commits into
Open
Conversation
Reads a Type1, MMType1 or TrueType simple font's character-code to glyph-name table from /Encoding and /Differences (ISO 32000-2 §9.6.5), maps names to Unicode through the Adobe Glyph List and a bundled ZapfDingbats list, and fills /Widths or the standard 14 AFM metrics for advance widths. Five diagnostic codes (400-404) cover an unreadable font, a malformed encoding, malformed widths, a font with no route to Unicode, and an unmapped glyph. Annex D.2's WinAnsi and MacRoman tables were re-transcribed from rendered page images rather than reusing Conformance's copy, since that copy carries fifteen MacRoman codes from Table 113 (the TrueType (1,0) cmap fallback table, not part of MacRomanEncoding itself) plus two renamed cells, 0xCA and 0xDB; this reader's copy fixes both. The Symbol and ZapfDingbats built-in encodings and AFM widths come from a generator, eng/generate-symbol-font-metrics.py, run against the Adobe Core 14 AFM files (not committed; see NOTICE) and pinned by a normalised SHA-256 manifest: src/VellumPdf.Reader/Fonts/SymbolFontMetrics.cs is up to date. Bound table (§5.9), measured: - /Differences array: walks the array, assigns at most 256 cells, stops past code 255. - /Differences name length: 128 chars, 401 and the cell left undefined. - /Widths array: reads at most LastChar - FirstChar + 1 (<=256). - /BaseFont name: TryResolve rejects over 128 chars without scanning; Report quotes 32 chars via DiagnosticExcerpt. - Resolution hops: one, via PdfDocumentReader.ResolveValue. - AGL component count: <=64, bounded by the 128-char name limit. - Per-font tables: three fixed 256-slot arrays. - FontCache: <=10,000 entries, insert-only, no eviction. - AdobeGlyphList / ZapfDingbatsGlyphList: parsed once per process, 4282 and 201 entries. - Report calls per font: <=5 distinct codes, each once. - TryDecodeNext: O(1) per byte, no allocation. - Create on a 100,000-element shared-instance /Widths array: measured 31,752 bytes allocated (bound asserted at 64 KiB). Departures from the plan: - TryGetCodepoints is named TryMapToUnicode and returns a string: some AGL entries, and uniXXXXYYYY... names, are multi-codepoint. - FontCache is insert-only with a 10,000-entry cap, not an LRU. - ZapfDingbatsGlyphList bundles the Adobe AGL repository's own zapfdingbats.txt rather than a hand-transcribed table. - Codes 401 (FontEncodingMalformed) and 402 (FontWidthsMalformed) are new; the plan's FontUnreadable, FontNoUnicodeRoute and UnmappedGlyphs are unchanged. - Symbol and ZapfDingbats tables are generated, with a committed generator and a hash manifest, not hand-transcribed.
Three corrections to the simple-font reader before review, each found by checking a shipped sentence against ISO 32000-2 or the reader's own API. Symbol and ZapfDingbats took their built-in encoding unconditionally, so a /Differences array or a named /Encoding on them was ignored. 9.6.5.2 says an /Encoding entry "shall override a Type 1 font's mapping", so the two built-in tables are now the Table 112 default base table and /Encoding applies to them as to any other font. HasToUnicode was always false on a parsed file: it tested Resolve(...) is PdfStream, and PdfDocumentReader.Resolve returns a stream object's dictionary, never a stream. It now follows the reference with ResolveStream (7.3.8.1 makes a stream indirect) and keeps a direct PdfStream arm for dictionaries built in memory. A true value also withholds UnmappedGlyphs, since the unparsed stream may map the code. The class remarks claimed 9.6.5.4's closing step, "any undefined entries in the table shall be filled using StandardEncoding", could change no cell. It changes twelve: the StandardEncoding codes MacRomanEncoding leaves undefined (0xAD, 0xB2, 0xB3, 0xB6 to 0xBA, 0xBD, 0xC3, 0xC5, 0xC6). The fill is applied with the clause's own scope: TrueType, Nonsymbolic (read from the Symbolic bit, which Table 121 makes exclusive with it), a dictionary /Encoding, after /Differences, and not over the all-null MacExpert table. A KAT pins the twelve-code set and WinAnsi's empty one; reader tests cover the fill and its four non-cases. Also: encoding-table remarks corrected for the three Annex D.2 footnotes, Standard14Names doc, brief-section references removed from shipped comments, NOTICE trademark line for ITC Zapf Dingbats.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #98 (PR 4 of the v2.4 plan: simple fonts). Does not close it; text extraction itself is the next change.
What
Reads a Type1, MMType1 or TrueType font dictionary into a per-font decoder (
SimpleFontReader, internal) that turns each one-byte code into a glyph name, an advance width and, where the glyph-name route allows, a Unicode string:SimpleFontEncodings(Reader's own copy; Conformance's table is untouched), the Symbol and ZapfDingbats built-in encodings from the Adobe Core 14 AFM files (Annex D.5, D.6),/BaseEncodingand/Differences, and §9.6.5.4's closing StandardEncoding fill with the clause's own scope (TrueType, Nonsymbolic, dictionary/Encoding).AdobeGlyphList.TryMapToUnicode:uniXXXX,uXXXX[XX], multi-codepoint entries, the.-suffix rule) and, for ZapfDingbats, Adobe's ownzapfdingbats.txt, both embedded as resources./Widths,/FirstCharand/MissingWidth(§9.6.2), and from the standard 14 metrics for a PDF 1.x font that omits them (§9.6.2.2): the twelve text fonts through Kernel's publicStandard14Metrics, Symbol and ZapfDingbats through a generatedSymbolFontMetricstable.Standard14Namesresolves the base-font aliases.4xxrange: 400FontUnreadable(Warning), 401FontEncodingMalformed, 402FontWidthsMalformed, 403FontNoUnicodeRoute(Info), 404UnmappedGlyphs(Info). These five enum members are the only public-surface change.FontCache(per reader, keyed on object number and generation, 10 000 entries, insert-only) behindPdfDocumentReader.GetFontReader, which dispatches on/Subtype; Type0 and Type3 return null until their readers exist./ToUnicodeis recorded (HasToUnicode) and not parsed yet. A font that names one does not report 404, since the unparsed stream has priority over the glyph-name route (§9.10.2) and may map the code.Annex D.2 versus the Conformance table
The Reader tables were transcribed from the rendered Annex D.2 pages, not copied from
src/VellumPdf.Conformance/Rules/Fonts/SimpleFontEncoding.cs. Measured differences, each pinned bySimpleFontEncodingsTestsand byReaderEncodingParityTestsinConformance.Tests(which reads both tables throughInternalsVisibleTo):0x7F, 0x81, 0x8D, 0x8F, 0x90, 0x9D) arebullet. Footnote 6:0xA0isspace. Footnote 5:0xADishyphen. Conformance leaves all eight null. The result is 224 defined cells, every code from 0x20 to 0xFF.0xCA(space) and0xDB(currency; Conformance hasEuro). 208 defined cells.The §9.6.5.4 fill changes exactly the twelve StandardEncoding codes MacRoman leaves undefined (
0xAD guilsinglright, 0xB2 dagger, 0xB3 daggerdbl, 0xB6 paragraph, 0xB7 bullet, 0xB8 quotesinglbase, 0xB9 quotedblbase, 0xBA quotedblright, 0xBD perthousand, 0xC3 circumflex, 0xC5 macron, 0xC6 breve), and none over a WinAnsi or Standard base;MacRoman_leavesExactlyTwelveStandardCodes_undefinedpins the set.Data provenance
Resources/AdobeGlyphList.txt: byte-identical to Conformance's copy (asserted in this PR's checks, and the NOTICE entry says so).Resources/ZapfDingbatsGlyphList.txt:zapfdingbats.txtfrom the Adobe AGL repository at commit4036a9ca80a62f64f9de4f7321a9a045ad0ecfd6, normalised to LF; SHA-256f6394e3cb8a447e84a1dad75d4baaf2aa7f45dc104faf369f4720e1a774ef2dc, recorded in NOTICE. 201 names (everyZapfDingbats.afmglyph exceptspace), including the fourteen names for codes 0x80 to 0x8D that the AFM codes and Annex D.6 does not print.Fonts/SymbolFontMetrics.cs: generated byeng/generate-symbol-font-metrics.pyfromSymbol.afmandZapfDingbats.afm, which are not committed. The script pins each AFM with a normalised SHA-256 (a336805b…363d7,b56fbcae…f699) and aCrecord count (190, 202), and--checkreproduces the committed file. NOTICE carries the two Adobe copyright lines, the ITC trademark line and the licence paragraph the AFM bundle requires to travel with derived data.Departures from the plan, and errors in the brief
Departures from plan §D, all recorded in
a7a4c36's body:TryGetCodepointsisTryMapToUnicodereturning a string: 81 AGL entries and everyuniXXXXYYYYname are multi-codepoint.FontCacheis insert-only with a 10 000 cap, not an LRU.ZapfDingbatsToUnicodeis Adobe's own file, embedded, not a transcription.Errors in the implementation brief, found by the orchestrator's verification pass and corrected in
01ed1a7(the brief itself is not shipped):3f951aa1…,66e3a6b7…) were raw hashes of the veraPDF-jar copies, mislabelled as normalised hashes of the Adobe files. The generator's manifest carries the hashes it computes with its documented normalisation, and--checkpasses against the Adobe files./Differencesand a named/Encodingon those fonts were ignored. §9.6.5.2 says an/Encodingentry "shall override a Type 1 font's mapping from character codes to character names"; the two built-in tables are now the Table 112 default base table and/Encodingapplies as for any other font. Two tests.Two defects in
a7a4c36that its own tests did not discriminate, also fixed in01ed1a7:HasToUnicodewas always false on a parsed file: it testedResolve(...) is PdfStream, andPdfDocumentReader.Resolvereturns a stream object's dictionary, never aPdfStream. No fixture had a/ToUnicodeentry, so nothing failed. It now follows the reference withResolveStream(§7.3.8.1 makes every stream indirect) and keeps a direct-PdfStreamarm for in-memory dictionaries. The 403 gate depends on it, and 404 now also consults it./BaseEncoding /MacRomanEncodingdecodes 0xB2 as U+2020, and the four non-cases (named/MacRomanEncoding, Type1, symbolic, MacExpert base) do not.Not constructed: brief §5.7 item 15 (a
/Widthsarray whose elements are indirect references to shared objects at the 100 000-element scale is measured for allocation, 31 752 bytes, but a document where/FirstCharexceeds/LastCharcombined with a shared/Widthswas not built).Bounds
Every input-sized quantity is bounded in this class (
PdfObjectParserhas no name-length, string-length or element-count limits of its own):/Differences/WidthsLastChar - FirstChar + 1(≤ 256) elements/BaseFontStandard14Names.TryResolverejects a name over 128 characters without scanning; messages quote throughDiagnosticExcerpt(32 characters)ResolveValue;Resolvethrows pastMaxResolveDepth, caught inCreateas 400uniXXXXgroups, or 64 list entries)FontCacheReportper fontTryDecodeNextTests
tests/VellumPdf.Reader.Tests/Fonts/:SimpleFontEncodingsTests(36: every footnote cell, the twelve-code KAT, immutability),AdobeGlyphListTests(ffi,uni00660066,u1F600,.scsuffix, malformed forms),ZapfDingbatsGlyphListTests,Standard14NamesTests,SymbolFontMetricsTests(widths against the AFM text: Symbol alpha 631 and space 250, ZapfDingbats a1 974, a89 390, a191 918; the Kernel route for the twelve text fonts),SimpleFontReaderTests(43, hand-built and writer-built documents throughFontTestSupport),FontFuzzTests(CsCheck, 3 000 iterations,CreateandTryDecodeNextnever throw).ReaderEncodingParityTestsinConformance.Testspins the eight, seventeen and zero differing codes against the Conformance table.PdfReaderDiagnosticCodeTestscovers the five new codes' ranges.Reader suite: 1499 cases (1349 at
52c403e), 0 failed, 12 skipped (the same 12 asmain, oracle skips underREQUIRE_ORACLESthat areAssert.Skipby design); Conformance 1284/0/0 with oracles.Gates at
01ed1a7Build 0 warnings / 0 errors;
dotnet format --verify-no-changesclean; clean-room script clean (plusgit grep -ifor the banned fragment, since the script's directory filter skips worktree paths);git diff --stat 52c403e -- src/VellumPdf.Conformance src/VellumPdf.Kernelempty;PublicAPI.Unshipped.txt+5 lines, ASCII, LF, no BOM,Shipped.txtuntouched;eng/generate-symbol-font-metrics.py --checkup to date; branch adds no em dashes; comment lines within 100 characters.Follow-up issue to file at merge: consolidating Conformance's
SimpleFontEncoding, its two ToUnicode string scans andEmbeddedCMap.csonto the Reader implementations (plan §D; Shipped surface, separate review).