Decode a fax, so a scanned form is not a blank page - #15
Merged
Conversation
A scanned form is a fax. 67 of the 1 633 real forms in the corpus — the eleven
issuing bodies, not the vendor test suites — carry a CCITT-encoded image, 263
images between them, and 6 of their pages have nothing else on them at all.
Those pages drew entirely blank, and a blank page is the failure a reader
notices before any other.
WHY IT BELONGS IN THE READER
The other filters this package stops at carry an image with its own idea of how
many components it has and how deep they are. A fax does not: it produces
bilevel samples, one bit a pixel, rows padded to a byte, and the stream
dictionary says what they mean. That is a byte stream, so it is a filter — and
decoding it here means every caller gets it rather than each writing its own.
/CCITTFaxDecode and /CCF therefore leave ImageFilter's list, which is a change
to a documented contract and is written up as one.
Group 3 one-dimensional, Group 3 mixed and Group 4 are all decoded, with /K,
/Columns, /Rows, /BlackIs1, /EncodedByteAlign and /EndOfBlock read.
THE REFERENCE READ BEFORE WRITING IT
ITU-T T.4 and T.6, by way of the tables and the changing-element algorithm in
golang.org/x/image/ccitt — read, not imported, because this package has no
dependencies outside the standard library and gains none. The 218 code table
entries were extracted mechanically from that package's gen.go rather than
retyped, because a table of 218 variable-length codes transcribed by hand is a
table with a mistake in it.
MEASURED AGAINST THAT REFERENCE, IMAGE BY IMAGE
Every CCITT image in the corpus, decoded by both and compared byte for byte,
with both sides working the height out for themselves where /Rows is absent:
1 633 real forms 263 images 260 byte-identical 0 differ 3 the
reference
refuses
arXiv sample 47 images 47 byte-identical 0 differ
The three the reference gives up on ("invalid code", "invalid offset") are
pages 2, 3 and 4 of fr-cerfa/cerfa_10455.pdf, where it stops and we decode all
3 504 rows — exactly the /Height the image dictionary names. Being more
forgiving than the reference is deliberate: a damaged scan comes back as far as
it got, because refusing it turns a form into a blank page.
TWO THINGS THE MEASUREMENT FOUND THAT READING THE SPECIFICATION DID NOT
Zero bits decode as perfectly good two-dimensional modes — pass mode is 0001,
vertical-left-3 is 0000010 — so a decoder that does not stop at the padding
goes on inventing rows that look like the last real one. fr-cerfa/cerfa_10701
gave 1 636 rows for a 208-row image that way, every surplus row with plausible
ink on it. Both /EndOfBlock's marker and a plain "the rest is fill" check are
now honoured.
The fuzz target found the second: on a truncated fax the early return skipped
the padding that makes the answer as long as /Rows promised, so two bytes of
input gave one row where the caller had asked for four, with no error to say
so. 6.7 million fuzz executions since, with none failing.
WHAT IT CHANGES WHEN A PAGE IS DRAWN
Through render, on the first page of all 1 633 real forms: 69 pages change.
One goes from blank to drawn — fr-cerfa/cerfa_11818, 0 inked pixels to 28 989.
Fourteen gain ink. **Fifty-one lose ink**, and that is the interesting number:
those pages were not blank before, they were noise. 273 of the corpus's image
masks carry an encoded filter — 236 CCITT, 9 JBIG2 — and render hands a mask
to its stencil path before asking whether the bytes are still compressed, so a
fax was being drawn as a stencil of its own compressed bytes. Decoding it
replaces that noise with the picture, which has less ink in it because most of
a form is white. The remaining 9 are a defect in render, filed separately.
100% statement coverage, go vet and -race clean, nine cross-compile targets,
nothing outside the standard library.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled
added a commit
to go-pdfkit/render
that referenced
this pull request
Aug 27, 2026
decodeImage asks whether an image is a mask before it asks whether the filter chain finished, so a mask whose filter stopped at an image format nothing here decodes was handed to the stencil path still compressed — and drawn. A stencil is one bit a pixel; compressed bytes painted through it are noise in the shape of nothing at all. This is not a rare shape. 273 of the image masks in the 1 633 real forms carry an encoded filter: 236 CCITT and 9 JBIG2. Fifty-one first pages of real forms were showing that noise, and it took decoding faxes in the reader (go-pdfkit/reader#15) to make it visible: those pages came out with *less* ink afterwards, because most of a form is white and the noise was not. The remaining nine are JBIG2, which nothing here decodes. Until something does, the honest answer is the one the rest of decodeImage already gives: "the image is not drawn rather than drawn wrong." The test draws a mask of eight zero bytes — as samples, a solid black stencil, so anything drawn at all is visible — once with /Filter /JBIG2Decode and once with no filter, and requires the first to draw nothing and the second to draw. Against the parent commit the first fails with "drawn = true, want false". 100% statement coverage, go vet clean, nine cross-compile targets. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A scanned form is a fax. 67 of the 1 633 real forms in the corpus — the eleven
issuing bodies, not the vendor test suites — carry a CCITT-encoded image, 263
images between them, and 6 of their pages have nothing else on them at all.
Those pages drew entirely blank, and a blank page is the failure a reader
notices before any other.
WHY IT BELONGS IN THE READER
The other filters this package stops at carry an image with its own idea of how
many components it has and how deep they are. A fax does not: it produces
bilevel samples, one bit a pixel, rows padded to a byte, and the stream
dictionary says what they mean. That is a byte stream, so it is a filter — and
decoding it here means every caller gets it rather than each writing its own.
/CCITTFaxDecode and /CCF therefore leave ImageFilter's list, which is a change
to a documented contract and is written up as one.
Group 3 one-dimensional, Group 3 mixed and Group 4 are all decoded, with /K,
/Columns, /Rows, /BlackIs1, /EncodedByteAlign and /EndOfBlock read.
THE REFERENCE READ BEFORE WRITING IT
ITU-T T.4 and T.6, by way of the tables and the changing-element algorithm in
golang.org/x/image/ccitt — read, not imported, because this package has no
dependencies outside the standard library and gains none. The 218 code table
entries were extracted mechanically from that package's gen.go rather than
retyped, because a table of 218 variable-length codes transcribed by hand is a
table with a mistake in it.
MEASURED AGAINST THAT REFERENCE, IMAGE BY IMAGE
Every CCITT image in the corpus, decoded by both and compared byte for byte,
with both sides working the height out for themselves where /Rows is absent:
1 633 real forms 263 images 260 byte-identical 0 differ 3 the
reference
refuses
arXiv sample 47 images 47 byte-identical 0 differ
The three the reference gives up on ("invalid code", "invalid offset") are
pages 2, 3 and 4 of fr-cerfa/cerfa_10455.pdf, where it stops and we decode all
3 504 rows — exactly the /Height the image dictionary names. Being more
forgiving than the reference is deliberate: a damaged scan comes back as far as
it got, because refusing it turns a form into a blank page.
TWO THINGS THE MEASUREMENT FOUND THAT READING THE SPECIFICATION DID NOT
Zero bits decode as perfectly good two-dimensional modes — pass mode is 0001,
vertical-left-3 is 0000010 — so a decoder that does not stop at the padding
goes on inventing rows that look like the last real one. fr-cerfa/cerfa_10701
gave 1 636 rows for a 208-row image that way, every surplus row with plausible
ink on it. Both /EndOfBlock's marker and a plain "the rest is fill" check are
now honoured.
The fuzz target found the second: on a truncated fax the early return skipped
the padding that makes the answer as long as /Rows promised, so two bytes of
input gave one row where the caller had asked for four, with no error to say
so. 6.7 million fuzz executions since, with none failing.
WHAT IT CHANGES WHEN A PAGE IS DRAWN
Through render, on the first page of all 1 633 real forms: 69 pages change.
One goes from blank to drawn — fr-cerfa/cerfa_11818, 0 inked pixels to 28 989.
Fourteen gain ink. Fifty-one lose ink, and that is the interesting number:
those pages were not blank before, they were noise. 273 of the corpus's image
masks carry an encoded filter — 236 CCITT, 9 JBIG2 — and render hands a mask
to its stencil path before asking whether the bytes are still compressed, so a
fax was being drawn as a stencil of its own compressed bytes. Decoding it
replaces that noise with the picture, which has less ink in it because most of
a form is white. The remaining 9 are a defect in render, filed separately.
100% statement coverage, go vet and -race clean, nine cross-compile targets,
nothing outside the standard library.