Skip to content

fix(filetype): emit each identity tag once, under the File group - #650

Merged
swackhamer merged 3 commits into
mainfrom
claude/great-sanderson-25f44a
Aug 10, 2026
Merged

fix(filetype): emit each identity tag once, under the File group#650
swackhamer merged 3 commits into
mainfrom
claude/great-sanderson-25f44a

Conversation

@swackhamer

Copy link
Copy Markdown
Collaborator

oxidex -json emitted FileType and MIMEType twice on any file that reached a parser — once under File:, once ungrouped — and on 21 of the 194 files in ExifTool's own t/images the two disagreed.

$ ./target/release/oxidex -json $ET/t/images/Geotag.log | rg -i 'FileType|MIMEType'
    "File:FileType": "TXT",
    "File:FileTypeExtension": "txt",
    "File:MIMEType": "text/plain",
    "FileType": "TXT",            <- ungrouped duplicate
    "MIMEType": "text/plain",     <- ungrouped duplicate

Font.dfont was the sharp version: File:FileType "DFONT" beside a bare FileType "ICO", left there by the ICO parser, which the file reaches because ExifTool's Font magic number matches anything starting \0\x01. Nothing downstream could say which of the two was meant.

Roughly forty parsers insert these tags ungrouped while the File:-grouped ones come from extract_file_metadata and add_identity_tags. No step reconciled them, and MetadataMap::merge does no group normalization.

The fix

normalize_identity_tags runs after add_identity_tags and leaves one answer per tag, under File:.

The File: value is the one kept. Both resolvers take it from crate::filetype, generated from ExifTool's %fileTypeLookup, %fileTypeExt and %mimeType; a parser's own string is its private spelling of the same fact at best (WebP where ExifTool says WEBP, Plist where it says PLIST) and a loose magic match at worst. Across the 21 disagreements the File: value is ExifTool's answer 20 times.

The one thing a parser can still contribute is a name where the tables produced none, so a bare FileType fills an absent or Unknown one before being dropped — EXE.elf keeps ELF rather than losing it to Unknown.

MIMEType is deliberately not treated the same way. application/octet-stream reads like a placeholder but is ExifTool's real answer for DR4, VRD, LNK, MOI and the Mach-O family, so overwriting it with a parser's value would replace a correct answer rather than fill a gap.

Verification

Identity sweep over ExifTool's t/images (194 files) against the pinned 13.59 oracle, before and after:

before after
File:-group matches 179 179
per-tag matches 563 563
ungrouped duplicates 50 0
contradictory pairs 21 0

The match counts are deliberately unchanged: #644, #648 and #641 already fixed the resolution side, so this is purely the de-duplication that was left over. Zero per-tag regressions.

cargo test --workspace green across 47 targets; clippy and fmt clean.

Test changes

19 assertions in production_wiring_tests.rs read the now-removed ungrouped key and were pointed at File:FileType. Three changed value, each toward ExifTool:

  • XML plist: PlistPLIST
  • two .eml fixtures: EMLTXT. ExifTool 13.59 has no EML file type — .eml is absent from %fileTypeLookup and an email reports TXT/txt/text/plain. EML was only OxiDex's name for the parser; the EML: tags those tests already assert are what show the routing worked.

Not in scope

  • FileSize has the same duplicate shape on the same files ("785 bytes" vs a bare "785" vs TEXT:FileSize), as do LineCount/WordCount/Newlines against their TEXT: copies.
  • XMP.xml reports TXT where ExifTool says XMP. fix(filetype): identify unlisted text files as XML or TXT by content #648's identify_text declines XMP by design; closing that gap means routing <rdf:RDF> content to the XMP parser, which is a coverage change rather than an identity one.
  • The 15 remaining sweep differences are the known separate gaps — ELF/Mach-O subtype descriptions, Win32 EXE, JXL Codestream, Extended WEBP, and several MIME-value mismatches.

🤖 Generated with Claude Code

swackhamer and others added 3 commits August 9, 2026 18:23
oxidex emitted FileType/MIMEType twice with contradictory values on any
file handled by the plain-text fallback: Geotag.log reported File:FileType
"Unknown" and File:MIMEType "application/octet-stream" beside a bare
FileType "TXT" and MIMEType "text/plain", and nothing downstream could say
which was meant. Roughly forty parsers insert those tags ungrouped while
the File:-grouped ones come from extract_file_metadata and
add_identity_tags, and no step reconciled them.

normalize_identity_tags folds the parser's answer into the File group and
drops the ungrouped keys. A real File: value is kept -- it came from
ExifTool's own tables, which outrank a parser's private spelling (WebP
where ExifTool says WEBP) -- so this never invents a value, it decides
which of two the output already contained survives.

Two supporting corrections the corpus forced:

FileTypeExtension follows the file type, not the filename.
extract_file_metadata echoes the on-disk extension, so a .log file
reported "log" where ExifTool reports "txt". It now comes from ExifTool's
table via filetype::canonical_extension, which declines on names the
tables do not carry rather than lowercasing a guess.

The text fallback could not tell XML from plain text, so .gpx/.kml/.xml
were named TXT. xml_file_type implements XMP.pm:4344-4426 -- <?xml is XML
unless <rdf:RDF or x:xmpmeta makes it XMP. A DOCTYPE or <?aid returns None
and the file stays TXT rather than being guessed at.

Also drops "gz" => "TAR" from the hand-written extension table. A gzip
stream is not a tar archive; the error was invisible while the GZIP parser
published a second, ungrouped FileType, because the output carried both
answers.

Identity sweep over ExifTool's t/images corpus (194 files, pinned 13.59):
File:-group matches 165 -> 176, per-tag matches 533 -> 562, ungrouped
duplicates 50 -> 0, contradictory pairs 28 -> 0, regressions 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n-25f44a

# Conflicts:
#	src/core/file_metadata.rs
#	src/filetype/mod.rs
@swackhamer
swackhamer merged commit e70bf1e into main Aug 10, 2026
10 checks passed
@swackhamer
swackhamer deleted the claude/great-sanderson-25f44a branch August 10, 2026 01:47
swackhamer added a commit that referenced this pull request Aug 10, 2026
* fix(tags): report a file's size under one key, not three

`oxidex -json` reported the size of a plain-text file three times over:

    "File:FileSize": "785 bytes",   <- what ExifTool 13.59 reports
    "FileSize": "785",              <- raw byte count, ungrouped
    "TEXT:FileSize": "785",

`extract_file_metadata` owns `File:FileSize` and formats it as ExifTool
prints it. 41 parsers additionally recorded `reader.size()` under a bare
`FileSize`, and the text parser mirrored that into `TEXT:FileSize`, so one
fact arrived under three keys with two spellings, two of which disagree
with the oracle. Under a group-aware sweep of ExifTool's `t/images` (194
files) 51 files carried the duplicate, 20 of them all three keys.

The dedup goes in `operations::drop_redundant_file_size`, after the one
merge where parser tags enter, because it is 41 parsers and not one. It
removes only the exactly-ungrouped key and only once `File:FileSize`
exists -- a badly-formatted answer still beats no answer.

Grouping alone does not make a distinct fact, so every grouped lookalike
was checked against the pinned oracle rather than inferred from its name.
`XML:FileSize`, `File:DPXFileSize` (12812288 against a 2.1 kB file),
`Prefetch:FileSize` and `LNK:TargetFileSize` are all real and untouched.
`EXE:FileSize` was the opposite -- `reader.size()` again, wearing a group
prefix that hid it from the `insert("FileSize"` search, and a tag ExifTool
emits for no Mach-O -- so it is removed at its source.

Also drops the `TEXT:` alias scheme. ExifTool 13.59 has no `TEXT` group:
the family-0 group of `Image::ExifTool::Text::Main` is `File`, so the
oracle reports `File:MIMEEncoding`, `File:Newlines`, `File:LineCount`,
`File:WordCount` and `File:ByteOrderMark`. The aliases emitted 101 keys
the oracle never produces under any group, each a second copy of a fact
already present, under names it does not even use (`LineEnding` for
`Newlines`, `HasBOM` for `ByteOrderMark`).

`FileType`/`MIMEType` are deliberately left duplicated: for a `.log` the
parser's ungrouped `FileType: TXT` is *correct* while `File:FileType`
still reads `Unknown`, so folding it away would delete the only right
copy. That is a detection gap, not this one.

Verified with a group-aware sweep (`scripts/compare_file.py` compares by
bare name, so it cannot see a group-placement defect and reported
`MISSING 0` throughout). Over all 194 files, per file, not just in total:
File:MATCH held at 2223, and no file's File:WRONG (40) or File:MISSING
(352) set moved. Every file now reports exactly one size key, down from
{1: 143, 2: 31, 3: 20}. The only keys that disappeared anywhere were 49
bare `FileSize` and 101 `TEXT:*`; none were added.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(tags): point the size dedup at its now-merged siblings

#648 and #650 landed while this was in review and moved the ground under
two comments here.

`drop_redundant_file_size` claimed `FileType`/`MIMEType` were "not folded
away" and called it a detection gap. #650 folds them, in
`normalize_identity_tags`, and #648 fixed the detection -- `Geotag.log`
now reports `File:FileType "TXT"`, not `Unknown`. Says instead why the two
stay separate functions: identity has rival *answers* to arbitrate and a
parser can name a type the tables left `Unknown`, whereas both size keys
report one byte count and the parser's is merely the unformatted spelling,
so it can never fill a gap in `File:FileSize`.

The text-parser note said relocating those facts into `File:` "needs the
FileType gate first". The gate now exists: `File:FileType` is the oracle's
answer on 12 of the 13 text-family files (XMP.xml still reads TXT). Records
that, and ExifTool's actual rule from `Text.pm` `ProcessTXT`, for whoever
picks it up. The nine-file figure still holds -- format dispatch continues
to route AFM, XML, JSON, RTF, URL and INX files through this parser.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
swackhamer added a commit that referenced this pull request Aug 10, 2026
…iscard

#650 and #654 left one answer per identity fact in the output, both by
discarding what the parsers wrote: `normalize_identity_tags` removes an
ungrouped `FileType`/`FileTypeExtension`/`MIMEType`, and
`drop_redundant_file_size` removes an ungrouped `FileSize`. Neither touched
the source. 45 inserts across 40 parsers therefore still computed values that
could not reach a caller, and the parsers still read as though `FileSize` and
`MIMEType` were theirs to report -- the trap that produced the duplicates in
the first place.

Removes them. `FileType` deliberately stays: `normalize_identity_tags`
promotes it into an absent or `Unknown` `File:FileType`, and that fallback is
the only reason SQLite, EVTX, Registry Hive, HDF5, X.509 and ELF are named at
all -- ExifTool's tables have no row for any of them, and it refuses to open
four outright. Both normalizers stay too, as the backstop for anything that
reintroduces one.

fits.rs is the one behavioural loose end. It wrote `File:FileType`,
`File:FileTypeExtension` and `File:MIMEType` directly -- hardcoded literals,
not table lookups -- which put it out of both normalizers' reach, since they
only ever removed the ungrouped spellings. `merge` gives format metadata
precedence over the `File:` group, so it was the last place a parser could
still outrank ExifTool's own tables, silently, had the two ever drifted. The
generated tables carry FITS in full (the `SIMPLE  = {20}T` magic number, the
`fits` extension row, `("FITS", "image/fits")`) and answer exactly as those
literals did.

Verified (instruments named):
- Full-JSON diff of both binaries over 207 combined-samples, this branch
  against main @9270f94a: no key lost, none gained, no value changed. The
  cleanup is provably invisible, fits.rs included -- which is the evidence
  that removing its literals cost nothing.
- `scripts/compare_file.py` over the same corpus: WRONG 134 -> 134,
  MISSING 1726 -> 1726, graded output byte-identical.
- `cargo test --workspace`: 4840 passed, 0 failed. tests/integration.rs
  574 -> 575, the new guard below; no test removed.
- clippy and fmt clean; no new warning in any changed file.

Guarded by `parsers_do_not_answer_identity_tags_the_file_group_owns`, which
runs eight formats through the real `read_metadata` path and asserts a parser
answers none of the three, while `File:FileType` still reads the expected
value and `File:FileSize` keeps ExifTool's formatting rather than the raw byte
count the parsers used to publish beside it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
swackhamer added a commit that referenced this pull request Aug 10, 2026
…iscard (#651)

#650 and #654 left one answer per identity fact in the output, both by
discarding what the parsers wrote: `normalize_identity_tags` removes an
ungrouped `FileType`/`FileTypeExtension`/`MIMEType`, and
`drop_redundant_file_size` removes an ungrouped `FileSize`. Neither touched
the source. 45 inserts across 40 parsers therefore still computed values that
could not reach a caller, and the parsers still read as though `FileSize` and
`MIMEType` were theirs to report -- the trap that produced the duplicates in
the first place.

Removes them. `FileType` deliberately stays: `normalize_identity_tags`
promotes it into an absent or `Unknown` `File:FileType`, and that fallback is
the only reason SQLite, EVTX, Registry Hive, HDF5, X.509 and ELF are named at
all -- ExifTool's tables have no row for any of them, and it refuses to open
four outright. Both normalizers stay too, as the backstop for anything that
reintroduces one.

fits.rs is the one behavioural loose end. It wrote `File:FileType`,
`File:FileTypeExtension` and `File:MIMEType` directly -- hardcoded literals,
not table lookups -- which put it out of both normalizers' reach, since they
only ever removed the ungrouped spellings. `merge` gives format metadata
precedence over the `File:` group, so it was the last place a parser could
still outrank ExifTool's own tables, silently, had the two ever drifted. The
generated tables carry FITS in full (the `SIMPLE  = {20}T` magic number, the
`fits` extension row, `("FITS", "image/fits")`) and answer exactly as those
literals did.

Verified (instruments named):
- Full-JSON diff of both binaries over 207 combined-samples, this branch
  against main @9270f94a: no key lost, none gained, no value changed. The
  cleanup is provably invisible, fits.rs included -- which is the evidence
  that removing its literals cost nothing.
- `scripts/compare_file.py` over the same corpus: WRONG 134 -> 134,
  MISSING 1726 -> 1726, graded output byte-identical.
- `cargo test --workspace`: 4840 passed, 0 failed. tests/integration.rs
  574 -> 575, the new guard below; no test removed.
- clippy and fmt clean; no new warning in any changed file.

Guarded by `parsers_do_not_answer_identity_tags_the_file_group_owns`, which
runs eight formats through the real `read_metadata` path and asserts a parser
answers none of the three, while `File:FileType` still reads the expected
value and `File:FileSize` keeps ExifTool's formatting rather than the raw byte
count the parsers used to publish beside it.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant