Fix embedart: retry pHash compare without colorspaces on empty IM 7 output - #6861
Open
e4779 wants to merge 5 commits into
Open
Fix embedart: retry pHash compare without colorspaces on empty IM 7 output#6861e4779 wants to merge 5 commits into
e4779 wants to merge 5 commits into
Conversation
`Model.store(fields=...)` used to walk every field name passed in and build an `UPDATE <table> SET <field>=?` clause for each dirty one, without checking whether the field was a fixed column or a flexible attribute. Callers such as `beet update` pass `library.Item._media_fields` when no explicit `fields=` is given, and a plugin that registers the same field via both `item_types` (typed flex attr) and `add_media_field()` lands in that set. For such fields there is no column, so the UPDATE crashed with `sqlite3.OperationalError: no such column: <field>`. Filter the requested field set to fixed columns before building the UPDATE; flexible attributes are already persisted just below via the `_flex_table` INSERT path, so they keep round-tripping correctly. Adds a regression test that stores a typed flex attribute through the `fields=` argument and asserts the value survives a fresh load. Fixes beetbox#5580.
…ash (beetbox#5580) Merges the topic-branch sent upstream as PR beetbox#6859 into fork master so prod can deploy from fork. Topic-branch kept intact for the upstream PR; this merge only advances fork master.
CI's Check docs job failed on the original entry because docstrfmt enforces a 72-column wrap. Reformat with .
…utput ImageMagick ≥7.1.1-44 sometimes emits empty stdout+stderr when invoked with `magick compare -define phash:colorspaces=sRGB,HCLp -metric PHASH`. The colorspaces define was added in a873a19 to make PHASH scores deterministic; on newer IM7 builds it silently produces no output (see ImageMagick/ImageMagick#5191). beets' IMBackend.compare() treated this as a parse error and returned None, which surfaced to users as "Error while checking art similarity; skipping" on every track during `beet import` (beetbox#6348). Fix: when the first compare pipeline exits cleanly (0 or 1) but produces no bytes on either stdout or stderr, retry the same pipeline WITHOUT the colorspaces define. The score is then less deterministic across IM builds but still usable for the compare_threshold gate. If the retry also yields no parseable output, compare() returns None (same as a genuine parse error). Implementation: - Extract the convert | compare subprocess orchestration into a `_run_compare_pipeline` helper returning a `CompareResult` dataclass that distinguishes the empty-output case from a real parse/failure. - Extract PHASH-output parsing (including the existing 7.1.1-44 "(diff)" paren extraction from beetbox#5650) into a static `_parse_compare_output` helper. - compare() calls the helper twice when the first run's `empty_output` flag is set, with a retry cmd that omits the define. Tests: two regression tests in ArtSimilarityTest (both fully mocked, no IM7 required): - test_compare_empty_output_triggers_retry: first call empty, retry parseable → similarity result reflects the retry score. - test_compare_empty_output_retry_also_fails: both empty → None (graceful). The existing 8 ArtSimilarityTest tests are unchanged and still pass — the empty-output path is only entered when both streams are empty AND the subprocess exited 0/1, so genuine parse errors and convert/compare failures short-circuit before the retry. Fixes beetbox#6348.
e4779
added a commit
to e4779/beets
that referenced
this pull request
Jul 19, 2026
…eetbox#6348) Merges the topic-branch sent upstream as PR beetbox#6861 into fork master so prod can pick up the fix. Topic-branch kept intact for the upstream PR.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6861 +/- ##
=======================================
Coverage 75.63% 75.63%
=======================================
Files 163 163
Lines 21312 21333 +21
Branches 3360 3363 +3
=======================================
+ Hits 16119 16136 +17
- Misses 4401 4403 +2
- Partials 792 794 +2
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When
magick compare -metric PHASHwith the-define phash:colorspaces=sRGB,HCLpdefine (added in a873a19 for score determinism) emits empty stdout AND stderr on ImageMagick ≥7.1.1-44,IMBackend.compare()now retries the pipeline without the define instead of returningNone. This restores a workingcompare_thresholdgate for affected users instead of logging "Error while checking art similarity; skipping" on every track.Fixes #6348.
Root cause
The
phash:colorspaces=sRGB,HCLpdefine was added in a873a19 to make PHASH scores deterministic across IM builds for the same input images. Starting with ImageMagick 7.1.1-44,magick comparesometimes emits no output at all on either stdout or stderr when this define is passed (see ImageMagick/ImageMagick#5191). beets then attemptsfloat(b""), which raisesValueError; the existing handler catches it, logs at debug level, and returnsNone. The caller inbeetsplug/_utils/art.py:52turns that into the user-facing warning"Error while checking art similarity; skipping", which fires on every track duringbeet import.The existing workaround in PR #5650 (
5c1817c78) handles only the parenthesized"0.84 (diff)"form — it does not handle the no-output-at-all case.Fix
In
IMBackend.compare():convert | comparesubprocess orchestration and the PHASH output parsing into two helpers (_run_compare_pipeline,_parse_compare_output) so the retry logic can be expressed cleanly._run_compare_pipelinereturns aCompareResult(score, empty_output). Theempty_outputflag distinguishes the plugin embedart compare_threshold throws error #6348 case (compare exited cleanly with code 0 or 1, but both streams are empty) from a genuine parse error (non-empty but unparseable bytes) or a subprocess failure (non-zero return other than 1).empty_output=True,compare()re-runs the same pipeline with acompare_cmdthat omits thephash:colorspaces=sRGB,HCLpdefine. The score loses cross-build determinism but remains usable for thecompare_thresholdgate.None,compare()returnsNoneas before (graceful-failure scenario).The existing return-code-1 path still works: IM writes the score to stderr when the images differ (returncode = "images differ"); the parser reads stderr for non-zero exits.
Why not just drop the define?
The define gives deterministic PHASH scores for the same images across IM7 builds. Without it, scores vary depending on the default colorspaces / delegate paths of the specific build. Users relying on
compare_thresholdfor consistent behavior between runs would prefer the deterministic path when it works. The retry-with-fallback preserves determinism for the vast majority of IM builds and only falls back to the non-deterministic path for the ones (7.1.1-44 and newer) that otherwise produce no score at all.Reproduction
I first tried to reproduce this locally on Debian trixie (apt
imagemagick=7.1.1.43+dfsg1-1+deb13u11). On .43 thephash:colorspacesdefine still works:The empty-output case happens only on ≥7.1.1-44 builds. beetbox CI runs on ubuntu-latest (=noble), whose apt ships ImageMagick 6.9.12 — the bug is not reproducible there either. The regression tests in this PR are fully mocked through the existing
@patch("beets.util.artresizer.subprocess")seam inArtSimilarityTest, so they exercise the path regardless of which IM version (if any) is installed.Tests
Two new tests added to
ArtSimilarityTest(test/plugins/test_embedart.py):test_compare_empty_output_triggers_retry— firstPopenpair yields empty streams, retryPopenpair yields a parseable score; asserts the similarity result reflects the retry score.test_compare_empty_output_retry_also_fails— both calls empty; assertsNone(graceful failure).Extended the existing
_mock_popenshelper with optionalretry_status/retry_stdout/retry_stderr/retry_convert_statusso tests can model the 4-Popen path (convert → compare → convert-retry → compare-retry). The existing 8 tests are unchanged.Locally: 2588 passed, 142 skipped, 0 unexpected failures.
CI proposal (separate concern)
While working on this, I noticed the
require_artresizer_comparedecorator (test/plugins/test_embedart.py:33-64) has carried a maintainer TODO for years:beetbox CI runs on IM 6.9.12 (apt on ubuntu-latest), so
test_reject_different_artandtest_accept_similar_artare currently skipped via that decorator. Adding an IM7-based job matrix (via a third-party image likedpokidov/imagemagick:7.1.2-12, since Ubuntu/Debian stable have no packaged IM7) would help catch this class of regression in the future. Happy to send that as a separate PR against.github/workflows/ci.yamlif useful — not doing it here to keep this PR focused on the fix.Checklist
ArtSimilarityTest)poe test,poe lint,poe format,poe check-typesall pass