Identify and extract SPIFFS filesystems (#17) - #39
Merged
Merged
Conversation
SPIFFS is the classic SPI-NOR filesystem on ESP8266 / ESP32-classic and other
small MCUs, and neither binwalk nor unblob extract it. Add identification and
byte-exact extraction.
SPIFFS has no superblock magic and its page/block geometry is build-time config
that is not stored in the image, so both are handled specially:
- Identify: anchor on a committed object-index header (at a page boundary it is
span_ix 0, flags 0xF8, then 3 align zero-bytes -> the 6-byte pattern
00 00 F8 00 00 00, once per file). The validator infers the geometry over the
image and confirms a coherent object graph, so false anchors are rejected.
- Geometry inference: try candidate (page, block) sizes and score by how many
files reassemble completely, then bytes, then file count. The correct geometry
recovers full files; a wrong one finds index headers at aligned offsets but
truncates the data, so completeness disambiguates.
- Extract: reassemble each object from its FINAL index header (name + size) and
its data pages ordered by span index, into the SafeRoot. Scoped to a standalone
SPIFFS image (a dumped partition, or one moria extracted and re-scanned); a
64 MiB guard bounds inference cost against a stray anchor.
The page/object layout is a clean reimplementation of the SPIFFS on-disk format
(MIT), verified byte-exact against real mkspiffs images (noted in README).
- New: signatures/spiffs.toml, src/spiffs_parse.{hpp,cpp} (shared core),
src/validators/spiffs.{hpp,cpp}, src/extract/spiffs.{hpp,cpp}. Wired into the
validator registry, extractor registry, MIME map and build.
- Tests: a minimal-image fixture in gen_samples, and tests/test_spiffs.py
(synthetic identify/extract, a false-positive guard on a bare anchor, and a
real mkspiffs round-trip that extracts every file byte-exact for two geometries,
self-skipping when the tool is absent). Wired into run.sh and CTest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SPIFFS is the classic SPI-NOR filesystem on ESP8266 / ESP32-classic and other small MCUs, and neither binwalk nor unblob extract it — web assets, config, and credentials commonly live in a SPIFFS partition. This adds identification and byte-exact extraction.
SPIFFS has no superblock magic, and its page/block geometry is build-time config that is not stored in the image, so both are handled specially.
What it does
span_ix0,flags0xF8 (USED|FINAL|INDEX cleared), then 3 align zero-bytes, giving the 6-byte pattern00 00 F8 00 00 00(once per file). The validator then infers the geometry over the image and confirms a coherent object graph, so false anchors are rejected. It reports the inferredpage/blocksizes.SafeRoot. A 64 MiB guard bounds inference cost against a stray anchor.Scoped to a standalone SPIFFS image — a dumped partition, or one moria extracts and re-scans (the dominant workflow). An embedded-at-offset SPIFFS is reached via partition extraction.
Provenance
The page/object layout is a clean reimplementation of the SPIFFS on-disk format (MIT), written against moria's
Readerand verified byte-exact against realmkspiffsimages. Noted in the README's license section.Tests
gen_samples(identifies at the consistent tier).tests/test_spiffs.py: synthetic identify + extract, a false-positive guard (a bare anchor with no object graph is rejected), and a realmkspiffsround-trip that extracts every file byte-exact across two geometries (page 256/block 4096 and page 512/block 8192), self-skipping when the tool is absent.Closes #17