Skip to content

Leave the unwritten spans of a formatted image as holes - #869

Open
MayCXC wants to merge 1 commit into
apple:mainfrom
MayCXC:sparse-inode-table
Open

Leave the unwritten spans of a formatted image as holes#869
MayCXC wants to merge 1 commit into
apple:mainfrom
MayCXC:sparse-inode-table

Conversation

@MayCXC

@MayCXC MayCXC commented Aug 27, 2026

Copy link
Copy Markdown

Summary

An image is formatted at the capacity the filesystem it will hold is given, and two of its structures are sized from that capacity and written out empty: the journal, which e2fsprogs convention caps at a gigabyte for a filesystem this large, and the inodes of the table past the ones the image holds. Writing them cost the host a gigabyte of disk for every image unpacked, before any of them held anything. An integration run unpacking twenty images spent twenty gigabytes on the emptiness alone and took the disk out from under the machine.

Both spans read as zero, and so does a hole, so a reader of the image sees the same bytes either way and the host keeps the blocks. The bitmaps of the groups past the ones holding content are already placed this way, so this follows a decision the formatter had already made elsewhere.

The last block of the journal is written rather than sought over, so a journal that does not fit in the image still fails at the point it is created rather than later at the I/O layer.

Motivation and Context

This is the difference between an image costing what it holds and costing what it was sized for. The visible symptom is disk exhaustion during an integration run, but it applies to every unpacked image: a 512 GiB sparse image should not cost a gigabyte before anything is written to it.

No issue filed, since this is an internal formatter change with no API or on-disk format implications: the resulting image is byte-identical to a reader. Happy to open one if the project prefers it recorded.

Testing

  • swift build and make check clean.
  • swift test: 595 tests in 82 suites passed, rebased onto current main (which now includes fix: efsck searching for external journal #856's efsck change to the same file; no conflict).
  • TestEXT4Sparse asserts the formatted image's allocated size, which is what regresses if the spans are written again.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update

An image is formatted at the capacity the filesystem it will hold is
given, and two of its structures are sized from that capacity and
written out empty: the journal, which e2fsprogs convention caps at a
gigabyte for a filesystem this large, and the inodes of the table past
the ones the image holds. Writing them cost the host a gigabyte of disk
for every image unpacked, before any of them held anything. An
integration run unpacking twenty images spent twenty gigabytes on the
emptiness alone and took the disk out from under the machine.

Both spans read as zero, and so does a hole, so a reader of the image
sees the same bytes either way and the host keeps the blocks. The
bitmaps of the groups past the ones holding content are already placed
this way.

The last block of the journal is written rather than sought over. A
journal that does not fit in the image failed at the I/O layer when the
whole span was written, and the journal inode's extent is recorded on
the strength of that, so the block that would be the first to fail is
still written.

An empty image at that capacity held 1,107,685,376 bytes of the host's
disk and now holds a thirtieth of that, which is what the same image
formatted without a journal has always held.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant