Skip to content

feat(reader): simple-font decoding for text extraction - #407

Open
Tim81 wants to merge 2 commits into
mainfrom
feat/98-simple-fonts
Open

feat(reader): simple-font decoding for text extraction#407
Tim81 wants to merge 2 commits into
mainfrom
feat/98-simple-fonts

Conversation

@Tim81

@Tim81 Tim81 commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Part of #98 (PR 4 of the v2.4 plan: simple fonts). Does not close it; text extraction itself is the next change.

What

Reads a Type1, MMType1 or TrueType font dictionary into a per-font decoder (SimpleFontReader, internal) that turns each one-byte code into a glyph name, an advance width and, where the glyph-name route allows, a Unicode string:

  • Character codes to glyph names per ISO 32000-2 §9.6.5: StandardEncoding, WinAnsiEncoding and MacRomanEncoding transcribed from Annex D.2 into SimpleFontEncodings (Reader's own copy; Conformance's table is untouched), the Symbol and ZapfDingbats built-in encodings from the Adobe Core 14 AFM files (Annex D.5, D.6), /BaseEncoding and /Differences, and §9.6.5.4's closing StandardEncoding fill with the clause's own scope (TrueType, Nonsymbolic, dictionary /Encoding).
  • Glyph names to Unicode through the Adobe Glyph List (AdobeGlyphList.TryMapToUnicode: uniXXXX, uXXXX[XX], multi-codepoint entries, the .-suffix rule) and, for ZapfDingbats, Adobe's own zapfdingbats.txt, both embedded as resources.
  • Widths from /Widths, /FirstChar and /MissingWidth (§9.6.2), and from the standard 14 metrics for a PDF 1.x font that omits them (§9.6.2.2): the twelve text fonts through Kernel's public Standard14Metrics, Symbol and ZapfDingbats through a generated SymbolFontMetrics table. Standard14Names resolves the base-font aliases.
  • Five diagnostic codes in the 4xx range: 400 FontUnreadable (Warning), 401 FontEncodingMalformed, 402 FontWidthsMalformed, 403 FontNoUnicodeRoute (Info), 404 UnmappedGlyphs (Info). These five enum members are the only public-surface change.
  • FontCache (per reader, keyed on object number and generation, 10 000 entries, insert-only) behind PdfDocumentReader.GetFontReader, which dispatches on /Subtype; Type0 and Type3 return null until their readers exist.

/ToUnicode is recorded (HasToUnicode) and not parsed yet. A font that names one does not report 404, since the unparsed stream has priority over the glyph-name route (§9.10.2) and may map the code.

Annex D.2 versus the Conformance table

The Reader tables were transcribed from the rendered Annex D.2 pages, not copied from src/VellumPdf.Conformance/Rules/Fonts/SimpleFontEncoding.cs. Measured differences, each pinned by SimpleFontEncodingsTests and by ReaderEncodingParityTests in Conformance.Tests (which reads both tables through InternalsVisibleTo):

  • WinAnsi differs at eight codes. Footnote 3: the six unused codes above 040 (0x7F, 0x81, 0x8D, 0x8F, 0x90, 0x9D) are bullet. Footnote 6: 0xA0 is space. Footnote 5: 0xAD is hyphen. Conformance leaves all eight null. The result is 224 defined cells, every code from 0x20 to 0xFF.
  • MacRoman differs at seventeen codes: the fifteen Table 113 names Conformance folds into its table (Table 113 is the TrueType (1,0) cmap fallback table, not MacRomanEncoding), 0xCA (space) and 0xDB (currency; Conformance has Euro). 208 defined cells.
  • StandardEncoding differs at none. 149 defined cells.
  • MacExpertEncoding is recognised as a name but its table is all-null (Annex D.4 is not transcribed); a font using it decodes to no glyphs, with no diagnostic, the same outcome as a symbolic font with no encoding.

The §9.6.5.4 fill changes exactly the twelve StandardEncoding codes MacRoman leaves undefined (0xAD guilsinglright, 0xB2 dagger, 0xB3 daggerdbl, 0xB6 paragraph, 0xB7 bullet, 0xB8 quotesinglbase, 0xB9 quotedblbase, 0xBA quotedblright, 0xBD perthousand, 0xC3 circumflex, 0xC5 macron, 0xC6 breve), and none over a WinAnsi or Standard base; MacRoman_leavesExactlyTwelveStandardCodes_undefined pins the set.

Data provenance

  • Resources/AdobeGlyphList.txt: byte-identical to Conformance's copy (asserted in this PR's checks, and the NOTICE entry says so).
  • Resources/ZapfDingbatsGlyphList.txt: zapfdingbats.txt from the Adobe AGL repository at commit 4036a9ca80a62f64f9de4f7321a9a045ad0ecfd6, normalised to LF; SHA-256 f6394e3cb8a447e84a1dad75d4baaf2aa7f45dc104faf369f4720e1a774ef2dc, recorded in NOTICE. 201 names (every ZapfDingbats.afm glyph except space), including the fourteen names for codes 0x80 to 0x8D that the AFM codes and Annex D.6 does not print.
  • Fonts/SymbolFontMetrics.cs: generated by eng/generate-symbol-font-metrics.py from Symbol.afm and ZapfDingbats.afm, which are not committed. The script pins each AFM with a normalised SHA-256 (a336805b…363d7, b56fbcae…f699) and a C record count (190, 202), and --check reproduces the committed file. NOTICE carries the two Adobe copyright lines, the ITC trademark line and the licence paragraph the AFM bundle requires to travel with derived data.

Departures from the plan, and errors in the brief

Departures from plan §D, all recorded in a7a4c36's body:

  • TryGetCodepoints is TryMapToUnicode returning a string: 81 AGL entries and every uniXXXXYYYY name are multi-codepoint.
  • FontCache is insert-only with a 10 000 cap, not an LRU.
  • ZapfDingbatsToUnicode is Adobe's own file, embedded, not a transcription.
  • Codes 401 and 402 were not in the plan's list.
  • Symbol and ZapfDingbats data are generated with a committed generator and hash manifest.

Errors in the implementation brief, found by the orchestrator's verification pass and corrected in 01ed1a7 (the brief itself is not shipped):

  • The brief's AFM hashes (3f951aa1…, 66e3a6b7…) were raw hashes of the veraPDF-jar copies, mislabelled as normalised hashes of the Adobe files. The generator's manifest carries the hashes it computes with its documented normalisation, and --check passes against the Adobe files.
  • The brief's step 4 told the implementer to take the Symbol and ZapfDingbats built-in encodings unconditionally, so /Differences and a named /Encoding on those fonts were ignored. §9.6.5.2 says an /Encoding entry "shall override a Type 1 font's mapping from character codes to character names"; the two built-in tables are now the Table 112 default base table and /Encoding applies as for any other font. Two tests.
  • Comments referred to brief sections ("§3.9 step 9"); rewritten to cite the standard.

Two defects in a7a4c36 that its own tests did not discriminate, also fixed in 01ed1a7:

  • HasToUnicode was always false on a parsed file: it tested Resolve(...) is PdfStream, and PdfDocumentReader.Resolve returns a stream object's dictionary, never a PdfStream. No fixture had a /ToUnicode entry, so nothing failed. It now follows the reference with ResolveStream (§7.3.8.1 makes every stream indirect) and keeps a direct-PdfStream arm for in-memory dictionaries. The 403 gate depends on it, and 404 now also consults it.
  • The class remarks claimed the §9.6.5.4 fill "would change no cell", and a test asserted that every Standard code is defined in WinAnsi and MacRoman. Twelve MacRoman codes are not (list above). The fill is implemented with the clause's scope and the test replaced by the exact-set KAT; a TrueType nonsymbolic font with /BaseEncoding /MacRomanEncoding decodes 0xB2 as U+2020, and the four non-cases (named /MacRomanEncoding, Type1, symbolic, MacExpert base) do not.

Not constructed: brief §5.7 item 15 (a /Widths array whose elements are indirect references to shared objects at the 100 000-element scale is measured for allocation, 31 752 bytes, but a document where /FirstChar exceeds /LastChar combined with a shared /Widths was not built).

Bounds

Every input-sized quantity is bounded in this class (PdfObjectParser has no name-length, string-length or element-count limits of its own):

Input Bound
/Differences walks the array once; assigns at most 256 cells; stops past code 255; a name over 128 characters reports 401 and leaves the cell undefined
/Widths reads at most LastChar - FirstChar + 1 (≤ 256) elements
/BaseFont Standard14Names.TryResolve rejects a name over 128 characters without scanning; messages quote through DiagnosticExcerpt (32 characters)
resolution one hop per entry through ResolveValue; Resolve throws past MaxResolveDepth, caught in Create as 400
AGL components ≤ 64, from the 128-character name bound
per-font tables three fixed 256-slot arrays; each Unicode string is bounded by the 128-character glyph name that produced it (at most 31 uniXXXX groups, or 64 list entries)
FontCache ≤ 10 000 entries; a font past the cap is built and not retained
Report per font ≤ 5 distinct codes, each at most once
TryDecodeNext O(1) per byte, no allocation

Tests

tests/VellumPdf.Reader.Tests/Fonts/: SimpleFontEncodingsTests (36: every footnote cell, the twelve-code KAT, immutability), AdobeGlyphListTests (ffi, uni00660066, u1F600, .sc suffix, malformed forms), ZapfDingbatsGlyphListTests, Standard14NamesTests, SymbolFontMetricsTests (widths against the AFM text: Symbol alpha 631 and space 250, ZapfDingbats a1 974, a89 390, a191 918; the Kernel route for the twelve text fonts), SimpleFontReaderTests (43, hand-built and writer-built documents through FontTestSupport), FontFuzzTests (CsCheck, 3 000 iterations, Create and TryDecodeNext never throw). ReaderEncodingParityTests in Conformance.Tests pins the eight, seventeen and zero differing codes against the Conformance table. PdfReaderDiagnosticCodeTests covers the five new codes' ranges.

Reader suite: 1499 cases (1349 at 52c403e), 0 failed, 12 skipped (the same 12 as main, oracle skips under REQUIRE_ORACLES that are Assert.Skip by design); Conformance 1284/0/0 with oracles.

Gates at 01ed1a7

Build 0 warnings / 0 errors; dotnet format --verify-no-changes clean; clean-room script clean (plus git grep -i for the banned fragment, since the script's directory filter skips worktree paths); git diff --stat 52c403e -- src/VellumPdf.Conformance src/VellumPdf.Kernel empty; PublicAPI.Unshipped.txt +5 lines, ASCII, LF, no BOM, Shipped.txt untouched; eng/generate-symbol-font-metrics.py --check up to date; branch adds no em dashes; comment lines within 100 characters.

Follow-up issue to file at merge: consolidating Conformance's SimpleFontEncoding, its two ToUnicode string scans and EmbeddedCMap.cs onto the Reader implementations (plan §D; Shipped surface, separate review).

Reads a Type1, MMType1 or TrueType simple font's character-code to
glyph-name table from /Encoding and /Differences (ISO 32000-2
§9.6.5), maps names to Unicode through the Adobe Glyph List and a
bundled ZapfDingbats list, and fills /Widths or the standard 14 AFM
metrics for advance widths. Five diagnostic codes (400-404) cover an
unreadable font, a malformed encoding, malformed widths, a font with
no route to Unicode, and an unmapped glyph.

Annex D.2's WinAnsi and MacRoman tables were re-transcribed from
rendered page images rather than reusing Conformance's copy, since
that copy carries fifteen MacRoman codes from Table 113 (the
TrueType (1,0) cmap fallback table, not part of MacRomanEncoding
itself) plus two renamed cells, 0xCA and 0xDB; this reader's copy
fixes both. The Symbol and ZapfDingbats built-in encodings and AFM
widths come from a generator, eng/generate-symbol-font-metrics.py,
run against the Adobe Core 14 AFM files (not committed; see NOTICE)
and pinned by a normalised SHA-256 manifest:

  src/VellumPdf.Reader/Fonts/SymbolFontMetrics.cs is up to date.

Bound table (§5.9), measured:
- /Differences array: walks the array, assigns at most 256 cells,
  stops past code 255.
- /Differences name length: 128 chars, 401 and the cell left
  undefined.
- /Widths array: reads at most LastChar - FirstChar + 1 (<=256).
- /BaseFont name: TryResolve rejects over 128 chars without
  scanning; Report quotes 32 chars via DiagnosticExcerpt.
- Resolution hops: one, via PdfDocumentReader.ResolveValue.
- AGL component count: <=64, bounded by the 128-char name limit.
- Per-font tables: three fixed 256-slot arrays.
- FontCache: <=10,000 entries, insert-only, no eviction.
- AdobeGlyphList / ZapfDingbatsGlyphList: parsed once per process,
  4282 and 201 entries.
- Report calls per font: <=5 distinct codes, each once.
- TryDecodeNext: O(1) per byte, no allocation.
- Create on a 100,000-element shared-instance /Widths array:
  measured 31,752 bytes allocated (bound asserted at 64 KiB).

Departures from the plan:
- TryGetCodepoints is named TryMapToUnicode and returns a string:
  some AGL entries, and uniXXXXYYYY... names, are multi-codepoint.
- FontCache is insert-only with a 10,000-entry cap, not an LRU.
- ZapfDingbatsGlyphList bundles the Adobe AGL repository's own
  zapfdingbats.txt rather than a hand-transcribed table.
- Codes 401 (FontEncodingMalformed) and 402 (FontWidthsMalformed)
  are new; the plan's FontUnreadable, FontNoUnicodeRoute and
  UnmappedGlyphs are unchanged.
- Symbol and ZapfDingbats tables are generated, with a committed
  generator and a hash manifest, not hand-transcribed.
Three corrections to the simple-font reader before review, each found by
checking a shipped sentence against ISO 32000-2 or the reader's own API.

Symbol and ZapfDingbats took their built-in encoding unconditionally, so
a /Differences array or a named /Encoding on them was ignored. 9.6.5.2
says an /Encoding entry "shall override a Type 1 font's mapping", so the
two built-in tables are now the Table 112 default base table and
/Encoding applies to them as to any other font.

HasToUnicode was always false on a parsed file: it tested Resolve(...)
is PdfStream, and PdfDocumentReader.Resolve returns a stream object's
dictionary, never a stream. It now follows the reference with
ResolveStream (7.3.8.1 makes a stream indirect) and keeps a direct
PdfStream arm for dictionaries built in memory. A true value also
withholds UnmappedGlyphs, since the unparsed stream may map the code.

The class remarks claimed 9.6.5.4's closing step, "any undefined entries
in the table shall be filled using StandardEncoding", could change no
cell. It changes twelve: the StandardEncoding codes MacRomanEncoding
leaves undefined (0xAD, 0xB2, 0xB3, 0xB6 to 0xBA, 0xBD, 0xC3, 0xC5,
0xC6). The fill is applied with the clause's own scope: TrueType,
Nonsymbolic (read from the Symbolic bit, which Table 121 makes exclusive
with it), a dictionary /Encoding, after /Differences, and not over the
all-null MacExpert table. A KAT pins the twelve-code set and WinAnsi's
empty one; reader tests cover the fill and its four non-cases.

Also: encoding-table remarks corrected for the three Annex D.2
footnotes, Standard14Names doc, brief-section references removed from
shipped comments, NOTICE trademark line for ITC Zapf Dingbats.
@Tim81 Tim81 added this to the v2.4 — PDF content extraction milestone Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant