Skip to content

Decode a fax, so a scanned form is not a blank page - #15

Merged
tannevaled merged 1 commit into
mainfrom
feat/ccitt-fax
Aug 27, 2026
Merged

Decode a fax, so a scanned form is not a blank page#15
tannevaled merged 1 commit into
mainfrom
feat/ccitt-fax

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

A scanned form is a fax. 67 of the 1 633 real forms in the corpus — the eleven
issuing bodies, not the vendor test suites — carry a CCITT-encoded image, 263
images between them, and 6 of their pages have nothing else on them at all.
Those pages drew entirely blank, and a blank page is the failure a reader
notices before any other.

WHY IT BELONGS IN THE READER

The other filters this package stops at carry an image with its own idea of how
many components it has and how deep they are. A fax does not: it produces
bilevel samples, one bit a pixel, rows padded to a byte, and the stream
dictionary says what they mean. That is a byte stream, so it is a filter — and
decoding it here means every caller gets it rather than each writing its own.
/CCITTFaxDecode and /CCF therefore leave ImageFilter's list, which is a change
to a documented contract and is written up as one.

Group 3 one-dimensional, Group 3 mixed and Group 4 are all decoded, with /K,
/Columns, /Rows, /BlackIs1, /EncodedByteAlign and /EndOfBlock read.

THE REFERENCE READ BEFORE WRITING IT

ITU-T T.4 and T.6, by way of the tables and the changing-element algorithm in
golang.org/x/image/ccitt — read, not imported, because this package has no
dependencies outside the standard library and gains none. The 218 code table
entries were extracted mechanically from that package's gen.go rather than
retyped, because a table of 218 variable-length codes transcribed by hand is a
table with a mistake in it.

MEASURED AGAINST THAT REFERENCE, IMAGE BY IMAGE

Every CCITT image in the corpus, decoded by both and compared byte for byte,
with both sides working the height out for themselves where /Rows is absent:

1 633 real forms 263 images 260 byte-identical 0 differ 3 the
reference
refuses
arXiv sample 47 images 47 byte-identical 0 differ

The three the reference gives up on ("invalid code", "invalid offset") are
pages 2, 3 and 4 of fr-cerfa/cerfa_10455.pdf, where it stops and we decode all
3 504 rows — exactly the /Height the image dictionary names. Being more
forgiving than the reference is deliberate: a damaged scan comes back as far as
it got, because refusing it turns a form into a blank page.

TWO THINGS THE MEASUREMENT FOUND THAT READING THE SPECIFICATION DID NOT

Zero bits decode as perfectly good two-dimensional modes — pass mode is 0001,
vertical-left-3 is 0000010 — so a decoder that does not stop at the padding
goes on inventing rows that look like the last real one. fr-cerfa/cerfa_10701
gave 1 636 rows for a 208-row image that way, every surplus row with plausible
ink on it. Both /EndOfBlock's marker and a plain "the rest is fill" check are
now honoured.

The fuzz target found the second: on a truncated fax the early return skipped
the padding that makes the answer as long as /Rows promised, so two bytes of
input gave one row where the caller had asked for four, with no error to say
so. 6.7 million fuzz executions since, with none failing.

WHAT IT CHANGES WHEN A PAGE IS DRAWN

Through render, on the first page of all 1 633 real forms: 69 pages change.
One goes from blank to drawn — fr-cerfa/cerfa_11818, 0 inked pixels to 28 989.
Fourteen gain ink. Fifty-one lose ink, and that is the interesting number:
those pages were not blank before, they were noise. 273 of the corpus's image
masks carry an encoded filter — 236 CCITT, 9 JBIG2 — and render hands a mask
to its stencil path before asking whether the bytes are still compressed, so a
fax was being drawn as a stencil of its own compressed bytes. Decoding it
replaces that noise with the picture, which has less ink in it because most of
a form is white. The remaining 9 are a defect in render, filed separately.

100% statement coverage, go vet and -race clean, nine cross-compile targets,
nothing outside the standard library.

A scanned form is a fax. 67 of the 1 633 real forms in the corpus — the eleven
issuing bodies, not the vendor test suites — carry a CCITT-encoded image, 263
images between them, and 6 of their pages have nothing else on them at all.
Those pages drew entirely blank, and a blank page is the failure a reader
notices before any other.

WHY IT BELONGS IN THE READER

The other filters this package stops at carry an image with its own idea of how
many components it has and how deep they are. A fax does not: it produces
bilevel samples, one bit a pixel, rows padded to a byte, and the stream
dictionary says what they mean. That is a byte stream, so it is a filter — and
decoding it here means every caller gets it rather than each writing its own.
/CCITTFaxDecode and /CCF therefore leave ImageFilter's list, which is a change
to a documented contract and is written up as one.

Group 3 one-dimensional, Group 3 mixed and Group 4 are all decoded, with /K,
/Columns, /Rows, /BlackIs1, /EncodedByteAlign and /EndOfBlock read.

THE REFERENCE READ BEFORE WRITING IT

ITU-T T.4 and T.6, by way of the tables and the changing-element algorithm in
golang.org/x/image/ccitt — read, not imported, because this package has no
dependencies outside the standard library and gains none. The 218 code table
entries were extracted mechanically from that package's gen.go rather than
retyped, because a table of 218 variable-length codes transcribed by hand is a
table with a mistake in it.

MEASURED AGAINST THAT REFERENCE, IMAGE BY IMAGE

Every CCITT image in the corpus, decoded by both and compared byte for byte,
with both sides working the height out for themselves where /Rows is absent:

  1 633 real forms   263 images   260 byte-identical   0 differ   3 the
                                                                 reference
                                                                 refuses
  arXiv sample        47 images    47 byte-identical   0 differ

The three the reference gives up on ("invalid code", "invalid offset") are
pages 2, 3 and 4 of fr-cerfa/cerfa_10455.pdf, where it stops and we decode all
3 504 rows — exactly the /Height the image dictionary names. Being more
forgiving than the reference is deliberate: a damaged scan comes back as far as
it got, because refusing it turns a form into a blank page.

TWO THINGS THE MEASUREMENT FOUND THAT READING THE SPECIFICATION DID NOT

Zero bits decode as perfectly good two-dimensional modes — pass mode is 0001,
vertical-left-3 is 0000010 — so a decoder that does not stop at the padding
goes on inventing rows that look like the last real one. fr-cerfa/cerfa_10701
gave 1 636 rows for a 208-row image that way, every surplus row with plausible
ink on it. Both /EndOfBlock's marker and a plain "the rest is fill" check are
now honoured.

The fuzz target found the second: on a truncated fax the early return skipped
the padding that makes the answer as long as /Rows promised, so two bytes of
input gave one row where the caller had asked for four, with no error to say
so. 6.7 million fuzz executions since, with none failing.

WHAT IT CHANGES WHEN A PAGE IS DRAWN

Through render, on the first page of all 1 633 real forms: 69 pages change.
One goes from blank to drawn — fr-cerfa/cerfa_11818, 0 inked pixels to 28 989.
Fourteen gain ink. **Fifty-one lose ink**, and that is the interesting number:
those pages were not blank before, they were noise. 273 of the corpus's image
masks carry an encoded filter — 236 CCITT, 9 JBIG2 — and render hands a mask
to its stencil path before asking whether the bytes are still compressed, so a
fax was being drawn as a stencil of its own compressed bytes. Decoding it
replaces that noise with the picture, which has less ink in it because most of
a form is white. The remaining 9 are a defect in render, filed separately.

100% statement coverage, go vet and -race clean, nine cross-compile targets,
nothing outside the standard library.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 2355850 into main Aug 27, 2026
1 check passed
tannevaled added a commit to go-pdfkit/render that referenced this pull request Aug 27, 2026
decodeImage asks whether an image is a mask before it asks whether the filter
chain finished, so a mask whose filter stopped at an image format nothing here
decodes was handed to the stencil path still compressed — and drawn. A stencil
is one bit a pixel; compressed bytes painted through it are noise in the shape
of nothing at all.

This is not a rare shape. 273 of the image masks in the 1 633 real forms carry
an encoded filter: 236 CCITT and 9 JBIG2. Fifty-one first pages of real forms
were showing that noise, and it took decoding faxes in the reader
(go-pdfkit/reader#15) to make it visible: those pages came out with *less* ink
afterwards, because most of a form is white and the noise was not.

The remaining nine are JBIG2, which nothing here decodes. Until something does,
the honest answer is the one the rest of decodeImage already gives: "the image
is not drawn rather than drawn wrong."

The test draws a mask of eight zero bytes — as samples, a solid black stencil,
so anything drawn at all is visible — once with /Filter /JBIG2Decode and once
with no filter, and requires the first to draw nothing and the second to draw.
Against the parent commit the first fails with "drawn = true, want false".

100% statement coverage, go vet clean, nine cross-compile targets.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant