program: read a symbolic TrueType font through its own character map

[tannevaled]

A simple TrueType font flagged symbolic, carrying an embedded FontFile2, with neither /Encoding nor /ToUnicode, yielded no text at all.

/Users/Shared/pdfforms/us-dol/1999-schedule-i.pdf is one. pdftotext reads 1 105 words off page 1; this package read 10 — although the page's codes are plain ASCII and the content stream literally says (Department of Labor) Tj.

Before (page 1, entire):

2 0 9 9 0 0 0 1 1 M

After (first lines):

SCHEDULE I Financial Information -- Small Plan Official Use Only
(Form 5500) OMB No. 1210-0110
This schedule is required to be filed under Section 104 of the Employee
Department of the Treasury
Internal Revenue Service Retirement Income Security Act of 1974 (ERISA) and section 6058(a) of the

Why it happened

The namedByTheDocument guard is right and stays. pdffont does fall back to StandardEncoding when a font has no /Encoding, so names are assigned; Font.Text then refuses them for a symbolic font, because a mathematical symbolic font puts a capital gamma where the standard encoding puts an inverted exclamation mark, and reading the name would give the wrong letter with nothing to say so.

So the answer has to come from the program — which is what the fallback installed here is for. It only knew one way to ask: the program's own code-to-name encoding. Only a PostScript program carries one; Font.GlyphIndexByCode is documented to report nothing for every TrueType font, so the last resort returned nothing at the first step for all six embedded fonts of that file. (The post table is beside the point: this package reads glyph names from CFF and Type 1 programs only.)

What this does

Asks the other way as well, through the font's character map, using the new subtable API in go-opentype/opentype v0.12.0.

The way in is the one poppler uses for a font the document gave no encoding — GfxFont.cc, Gfx8BitFont::getCodeToGIDMap, case 2: the Microsoft Symbol (3,0) subtable if there is one, else the Macintosh Roman (1,0) one, addressed by the raw code and then, failing that, by 0xF000 + code, which is where such subtables are conventionally written.

The way out is the Unicode subtable, inverted — the font's own statement of which character that glyph is for.

Order is unchanged: /ToUnicode, then a name the document chose in /Differences, then the program. The program stays the last resort.

What it refuses to say

• No Unicode subtable, no answer. A glyph on its own says nothing about which character it is, and a guess would be exactly the wrong letter the guard exists to refuse. A genuinely symbolic font — the NelcoSymbols of that same file carries only (1,0) and (3,0) — stays silent, which is right.
• No private-use or control character. A page of those searches no better than a page of nothing while looking, wrongly, like a page that had been read.

Note on the references

Neither poppler nor pdf.js consults the cmap for text at all: for a TrueType font with no /Encoding poppler defaults to WinAnsiEncoding names and pdf.js to MacRomanEncoding when symbolic, then maps those names through the glyph list. That is the guess this package deliberately refuses. Going through the cmap is stricter — it reports the character of the glyph that actually gets drawn — and it is why the four forms land two words short of poppler rather than level with it.

Measured

Four US Department of Labor forms, page 1, against pdftotext on the same page:

file	before	after	poppler
1999-schedule-i.pdf	10	1103	1105
1999-5500-schedule-a.pdf	10	244	248
1999-schedule-d.pdf	10	312	314
1999-schedule-g.pdf	10	227	229

Corpus, first three pages of every file, text digests compared before and after:

• 1 633 real forms: 4 files gain text, 0 lose any.
• 118 843 arXiv documents: not one byte of text changes. 48 675 of their simple fonts are embedded, symbolic, and have neither /Encoding nor /ToUnicode — but 98% are PostScript programs the name route already answered for, and the rest carry no usable subtable pair. Nothing regressed.

Checks

go vet, gofmt, -race clean; 100% statement coverage; the nine cross-compile targets build.