Skip to content

Write out the scan, instead of a box saying the picture was unreadable - #3

Merged
tannevaled merged 1 commit into
mainfrom
bump-fax-and-structure
Aug 27, 2026
Merged

Write out the scan, instead of a box saying the picture was unreadable#3
tannevaled merged 1 commit into
mainfrom
bump-fax-and-structure

Conversation

@tannevaled

@tannevaled tannevaled commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

reader v0.4.2 → v0.5.0. No code change.

What was wrong

picture() hands anything that is not a JPEG or a JPEG 2000 to samples(), and
samples() refuses data that cannot fill the page before it reads any of it:

func stencil(data []byte, w, h int) (image.Image, bool) {
	stride := (w + 7) / 8
	if len(data) < stride*h {
		return nil, false          // <- a fax always landed here
	}

A /CCITTFaxDecode image arrived still compressed, so there were never enough
bytes, so picture returned false and figure emitted a placeholder. Every
scanned picture in a document came out as a box saying it could not be read.

That guard is the reason this was a missing figure rather than a wrong one — the
same data painted without a length check is noise, which is what render v0.10.0
had to stop doing.

Measured

Two binaries calling latex.Source, each verified with go version -m:

old  github.com/go-pdfkit/reader v0.4.2   pdffont v0.3.0   extract v0.1.0
new  github.com/go-pdfkit/reader v0.5.0   pdffont v0.3.0   extract v0.1.0

Corpus: all 2 268 forms in /Users/Shared/pdfforms. Prevalence is over the
1 633 real documents.

files
compared 2 268
source differs 59
… of the 1 633 real documents 57 (3.5%)
… of the 635 vendor fixtures 2
longer after 57
shorter after 0
status changed 0

Across those 57 real documents:

before after
\framebox{...unreadable image} 291 50
\includegraphics 12 253

241 placeholders became figures, one for one. Verbatim, from
fr-cerfa/cerfa_10983.pdf:

before:  \framebox[41.0pt]{\rule{0pt}{21.0pt}unreadable image}
after :  \includegraphics[width=41.0pt,height=21.0pt]{image000.png}

and from fr-cerfa/cerfa_10701.pdf, two on one page:

before:  \framebox[31.0pt]{\rule{0pt}{16.0pt}unreadable image}
after :  \includegraphics[width=31.0pt,height=16.0pt]{image001.png}
before:  \framebox[31.0pt]{\rule{0pt}{16.0pt}unreadable image}
after :  \includegraphics[width=31.0pt,height=16.0pt]{image003.png}

The 50 that are still unreadable are not faxes. They are colour spaces
samples() does not claim to read without a full renderer behind it, and they
were not part of this bump.

Not measured

Wall clock and peak memory: three other corpus jobs were on the machine, so a
timing number would be fiction. The output diff does not depend on load.

Gates

go vet, gofmt, exact 100% statement coverage, nine cross-compile targets, all
pass locally with GOWORK=off CGO_ENABLED=0.

reader v0.4.2 -> v0.5.0, which decodes /CCITTFaxDecode.

picture() hands anything that is not a JPEG or a JPEG 2000 to samples(),
and samples() checks that there are enough bytes to fill the page before it
reads any: `len(data) < stride*h`. A fax arrived still compressed, so there
never were, so every scanned picture in a document came out as

    \framebox[41.0pt]{\rule{0pt}{21.0pt}unreadable image}

Now the bytes are samples and the check passes, so the picture is written
out beside the source as a PNG and referred to properly.

Measured over all 2 268 forms in /Users/Shared/pdfforms, both binaries
verified with `go version -m`:

  59 of 2 268 files produce different source -- 57 of the 1 633 real
  documents. 57 longer, 0 shorter.
  Across those 57: \framebox{unreadable image} 291 -> 50, and
  \includegraphics 12 -> 253. Two hundred and forty-one placeholders
  became figures, one for one.

  before: \framebox[41.0pt]{\rule{0pt}{21.0pt}unreadable image}
  after : \includegraphics[width=41.0pt,height=21.0pt]{image000.png}

The fifty that are still unreadable are not faxes; they are colour spaces
samples() does not claim to read without a renderer behind it.
@tannevaled
tannevaled merged commit 53107c6 into main Aug 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant