Draw a scanned page - #25
Merged
Merged
Conversation
A page whose only content is a JPEG 2000 image came out blank, and that is what
a scanned page is. Over the 250 biodiversity scans, rendering every page of
every document:
730 of 747 pages go from blank to drawn
0 pages lose anything
17 unchanged, being the pages that had something else on them
The 17 are the true control: nothing that was drawing before draws differently.
WHY THIS WAS NEARLY NOT DONE
Measured over the corpora this project had — government forms and arXiv figures
— JPEG 2000 is eleven files in sixteen hundred, and the decision taken on that
number was that it was not worth a decoder. Extending the corpus to where a
scanned page actually lives says otherwise: all 250 biodiversity scans carry
one, 248 of the 250 medical ones, and 144 of the 222 readable scanned books.
The old figure measured the corpus, not the world.
THE DECODER IS NOT WRITTEN HERE, AND NOT WRITTEN AT ALL
github.com/ajroetker/go-jpeg2000 is 20 050 lines of pure Go with no cgo, and it
decodes 1 610 of 1 610 real images taken out of eighty scanned books. It is
also registered in go-gfx/gfx/codec, which is where the fleet's codec choices
are made once.
THIS TAKES THE DECODER, NOT THE REGISTRY, AND THE REASON IS A NUMBER
go-gfx/gfx/codec brings TIFF, WEBP, BMP and ICO with it, and a PDF renderer has
no use for any of them. One of this package's consumers is a wasm binary a
browser downloads:
today 4.17 MB
the decoder on its own 5.35 MB
the whole registry 6.25 MB
Nine hundred kilobytes for four decoders nothing here calls. The codec choice
is still mutualised — the same reference library, chosen once in go-gfx — but
what is imported is the one format this package can be asked for.
WHAT IT COSTS
Decoding is real work, and the figure that looked good before was measuring a
blank page: those 747 pages took 1.1 seconds when nothing was drawn and 2
minutes 30 seconds when they were, which is about a fifth of a second for a
full-page scan.
TWO SMALLER THINGS
The size comes from the codestream rather than from the dictionary, as it does
for JPEG: a codestream carries its own, and where they disagree the one the
pixels are in is the one that can be drawn.
The "a format nothing here decodes" branch is now covered by JBIG2 rather than
by JPEG 2000, which is the honest representative of what is left: 12 of 250
court filings and 1 of 222 scanned books carry it, and it draws nothing rather
than drawing noise.
100% statement coverage, go vet and -race clean, nine cross-compile targets.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A page whose only content is a JPEG 2000 image came out blank, and that is what
a scanned page is. Over the 250 biodiversity scans, rendering every page of
every document:
730 of 747 pages go from blank to drawn
0 pages lose anything
17 unchanged, being the pages that had something else on them
The 17 are the true control: nothing that was drawing before draws differently.
WHY THIS WAS NEARLY NOT DONE
Measured over the corpora this project had — government forms and arXiv figures
— JPEG 2000 is eleven files in sixteen hundred, and the decision taken on that
number was that it was not worth a decoder. Extending the corpus to where a
scanned page actually lives says otherwise: all 250 biodiversity scans carry
one, 248 of the 250 medical ones, and 144 of the 222 readable scanned books.
The old figure measured the corpus, not the world.
THE DECODER IS NOT WRITTEN HERE, AND NOT WRITTEN AT ALL
github.com/ajroetker/go-jpeg2000 is 20 050 lines of pure Go with no cgo, and it
decodes 1 610 of 1 610 real images taken out of eighty scanned books. It is
also registered in go-gfx/gfx/codec, which is where the fleet's codec choices
are made once.
THIS TAKES THE DECODER, NOT THE REGISTRY, AND THE REASON IS A NUMBER
go-gfx/gfx/codec brings TIFF, WEBP, BMP and ICO with it, and a PDF renderer has
no use for any of them. One of this package's consumers is a wasm binary a
browser downloads:
today 4.17 MB
the decoder on its own 5.35 MB
the whole registry 6.25 MB
Nine hundred kilobytes for four decoders nothing here calls. The codec choice
is still mutualised — the same reference library, chosen once in go-gfx — but
what is imported is the one format this package can be asked for.
WHAT IT COSTS
Decoding is real work, and the figure that looked good before was measuring a
blank page: those 747 pages took 1.1 seconds when nothing was drawn and 2
minutes 30 seconds when they were, which is about a fifth of a second for a
full-page scan.
TWO SMALLER THINGS
The size comes from the codestream rather than from the dictionary, as it does
for JPEG: a codestream carries its own, and where they disagree the one the
pixels are in is the one that can be drawn.
The "a format nothing here decodes" branch is now covered by JBIG2 rather than
by JPEG 2000, which is the honest representative of what is left: 12 of 250
court filings and 1 of 222 scanned books carry it, and it draws nothing rather
than drawing noise.
100% statement coverage, go vet and -race clean, nine cross-compile targets.