Leave the unwritten spans of a formatted image as holes - #869
Open
MayCXC wants to merge 1 commit into
Open
Conversation
An image is formatted at the capacity the filesystem it will hold is given, and two of its structures are sized from that capacity and written out empty: the journal, which e2fsprogs convention caps at a gigabyte for a filesystem this large, and the inodes of the table past the ones the image holds. Writing them cost the host a gigabyte of disk for every image unpacked, before any of them held anything. An integration run unpacking twenty images spent twenty gigabytes on the emptiness alone and took the disk out from under the machine. Both spans read as zero, and so does a hole, so a reader of the image sees the same bytes either way and the host keeps the blocks. The bitmaps of the groups past the ones holding content are already placed this way. The last block of the journal is written rather than sought over. A journal that does not fit in the image failed at the I/O layer when the whole span was written, and the journal inode's extent is recorded on the strength of that, so the block that would be the first to fail is still written. An empty image at that capacity held 1,107,685,376 bytes of the host's disk and now holds a thirtieth of that, which is what the same image formatted without a journal has always held.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An image is formatted at the capacity the filesystem it will hold is given, and two of its structures are sized from that capacity and written out empty: the journal, which e2fsprogs convention caps at a gigabyte for a filesystem this large, and the inodes of the table past the ones the image holds. Writing them cost the host a gigabyte of disk for every image unpacked, before any of them held anything. An integration run unpacking twenty images spent twenty gigabytes on the emptiness alone and took the disk out from under the machine.
Both spans read as zero, and so does a hole, so a reader of the image sees the same bytes either way and the host keeps the blocks. The bitmaps of the groups past the ones holding content are already placed this way, so this follows a decision the formatter had already made elsewhere.
The last block of the journal is written rather than sought over, so a journal that does not fit in the image still fails at the point it is created rather than later at the I/O layer.
Motivation and Context
This is the difference between an image costing what it holds and costing what it was sized for. The visible symptom is disk exhaustion during an integration run, but it applies to every unpacked image: a 512 GiB sparse image should not cost a gigabyte before anything is written to it.
No issue filed, since this is an internal formatter change with no API or on-disk format implications: the resulting image is byte-identical to a reader. Happy to open one if the project prefers it recorded.
Testing
swift buildandmake checkclean.swift test: 595 tests in 82 suites passed, rebased onto currentmain(which now includes fix: efsck searching for external journal #856'sefsckchange to the same file; no conflict).TestEXT4Sparseasserts the formatted image's allocated size, which is what regresses if the spans are written again.Type of Change