A mask we cannot read takes its image with it - #26
Merged
Conversation
This reverses what this package did, and the corpus is why.
An image whose mask could not be decoded was drawn as it stood, on the
reasoning that an unreadable mask should not cost you the picture. That is
right for a photograph with a soft edge and catastrophic for a scanned page.
A scanned page is not one image. It is a low-resolution colour background with
a HIGH-RESOLUTION BITONAL INK LAYER over it, and that ink layer is a dark
rectangle whose shape comes entirely from a JBIG2 stencil in /Mask. Drawn
without the stencil it is a dark rectangle over the whole page — so v0.12.0,
which stopped those pages being blank, painted a great many of them dark
instead. That is worse than blank, and it shipped.
WHAT IT WAS, AND WHAT IT IS
Judged against poppler, page by page, both given the same box:
v0.12.0 with this
ia-medical median 0.9749 median 0.0005
ia-biodiversity 0.1073 0.0259
ia-texts 0.1324 0.0066
ia-americana 0.0278 0.0249
ia-uscourts 0.0009 0.0009
Medical scans go from 97% of the median page's pixels wrong to 0.05%, and the
count of pages agreeing to within one per cent goes from 10 of 119 to 100.
WHAT IS STILL MISSING, AND WHY THIS IS THE RIGHT ANSWER ANYWAY
The ink layer is not drawn at all, because JBIG2 is not decoded. So the page
comes out as its background: legible, missing its sharpest text. poppler draws
both. The remaining difference is that missing layer, which is what the numbers
above still show on biodiversity and americana.
Not drawing it is the same rule the rest of this file already follows — the
image is not drawn rather than drawn wrong — and the measurement says which of
the two mistakes is the smaller one by a factor of two thousand.
AND A CORRECTION TO A DECISION MADE EARLIER TODAY
JBIG2 was measured at 20 documents in 3 217 and three pages blank without it,
and set aside on that. The survey was counting pages whose only content is an
image in a given filter — and JBIG2 here is not the content, it is the MASK. Its
absence does not blank a page; it ruins one. The metric could not see that, and
the comparison against another renderer is what did.
100% statement coverage, go vet and -race clean, nine cross-compile targets.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Aug 28, 2026
) Two things, because the first exposed the second and neither is right alone. JBIG2 is decoded. A modern scanned page is two images: a low-resolution colour background stored as JPEG 2000, and a high-resolution bitonal ink layer over it stored as JBIG2. We could read the background and not the ink. Counting filters by what they encode as page CONTENT put JBIG2 in 20 documents of 3 217, which is why it had been set aside; counting the images it SHAPES put it in 4 089, because it is almost never the content — it is the /Mask. A stencil mask paints where its sample is 0, and applyStencilMask was clearing the alpha where the stencil painted rather than where it did not. Every explicit mask in the corpus was therefore showing the exact complement of itself: on a scanned page, the ink is the only part that was hidden. The test that covered it asserted the same inversion, so the suite was green. Asked which half of a two-colour page a mask of eight 0 bits and eight 1 bits paints, poppler answers the 0 half; so does this now, for a raw mask and a JBIG2 one alike. The inversion had been invisible because the two halves cancelled: masks that could not be decoded took their image with them (#26), and the ones that could were drawn inside out. Decoding JBIG2 without fixing the mask made 5 of 25 medical scans worse and 2 of them catastrophic — 98% of pixels wrong on the worst — which is how the second defect was found. Measured against poppler, first page of 25 medical scans: improved 11 unchanged 14 worse 0 mean differing pixels 0.0083 -> 0.0004 worst page 0.1087 -> 0.0004 No decoder is written here. The reference is github.com/dkrisman/gobig2, chosen by decoding 403 JBIG2 masks taken from real scans and comparing every pixel against poppler's own decoder: it was exact on all 396 it accepted, where the candidate that decoded the most was exact on 9%. Its seven refusals are a resource budget, not a defect, and this takes it at its default: a stream it declines is a stream not drawn, which is the rule the rest of the file follows. gobig2's limits are process-global, so they are not raised from inside a library. 100% statement coverage, go vet and -race clean, nine cross-compile targets, CGO_ENABLED=0. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This reverses what this package did, and the corpus is why.
An image whose mask could not be decoded was drawn as it stood, on the
reasoning that an unreadable mask should not cost you the picture. That is
right for a photograph with a soft edge and catastrophic for a scanned page.
A scanned page is not one image. It is a low-resolution colour background with
a HIGH-RESOLUTION BITONAL INK LAYER over it, and that ink layer is a dark
rectangle whose shape comes entirely from a JBIG2 stencil in /Mask. Drawn
without the stencil it is a dark rectangle over the whole page — so v0.12.0,
which stopped those pages being blank, painted a great many of them dark
instead. That is worse than blank, and it shipped.
WHAT IT WAS, AND WHAT IT IS
Judged against poppler, page by page, both given the same box:
ia-medical median 0.9749 median 0.0005
ia-biodiversity 0.1073 0.0259
ia-texts 0.1324 0.0066
ia-americana 0.0278 0.0249
ia-uscourts 0.0009 0.0009
Medical scans go from 97% of the median page's pixels wrong to 0.05%, and the
count of pages agreeing to within one per cent goes from 10 of 119 to 100.
WHAT IS STILL MISSING, AND WHY THIS IS THE RIGHT ANSWER ANYWAY
The ink layer is not drawn at all, because JBIG2 is not decoded. So the page
comes out as its background: legible, missing its sharpest text. poppler draws
both. The remaining difference is that missing layer, which is what the numbers
above still show on biodiversity and americana.
Not drawing it is the same rule the rest of this file already follows — the
image is not drawn rather than drawn wrong — and the measurement says which of
the two mistakes is the smaller one by a factor of two thousand.
AND A CORRECTION TO A DECISION MADE EARLIER TODAY
JBIG2 was measured at 20 documents in 3 217 and three pages blank without it,
and set aside on that. The survey was counting pages whose only content is an
image in a given filter — and JBIG2 here is not the content, it is the MASK. Its
absence does not blank a page; it ruins one. The metric could not see that, and
the comparison against another renderer is what did.
100% statement coverage, go vet and -race clean, nine cross-compile targets.