Skip to content

Match scraped artwork against the local libraries to speed up big scrapes#56

Open
Thoroslives wants to merge 1 commit into
mscodemonkey:mainfrom
Thoroslives:speedup-upstream
Open

Match scraped artwork against the local libraries to speed up big scrapes#56
Thoroslives wants to merge 1 commit into
mscodemonkey:mainfrom
Thoroslives:speedup-upstream

Conversation

@Thoroslives

Copy link
Copy Markdown
Contributor

Scraping a large user page currently fetches every poster's page just to read its TMDb ID, before we even know whether the item is in the library. On a user with thousands of uploads that's hours of requests, most of them for titles that then get skipped as "not available on Plex". This adds an in-memory index of the configured libraries (one listing request per library per run, using the guid lookup pattern the plexapi docs recommend) and matches each scraped title and year against it locally first. Items that aren't in the library are skipped without any web request, and matched items take their TMDb ID from Plex's own guids. The poster page is still fetched to double-check the match right before artwork is actually uploaded or saved (once per title, cached), so nothing less accurate than before ever gets written. An ambiguous local match (two library items with the same title and year) still falls back to fetching the poster page. Full-catalog user scrapes drop from hours to roughly the page-crawl time, and re-runs with track_artwork_ids skip everything without hitting ThePosterDB at all, which is also a lot kinder to their servers.

  • local_library_matching in config.json (default true; set false to restore the previous per-poster lookups) plus a toggle in the web UI
  • Applies to ThePosterDB scrapes only, since MediUX sets already carry their TMDb IDs
  • Kometa asset mode gets the same speedup; the existing skip/exists checks now also avoid the poster-page request entirely
  • The label bookkeeping for replaced artwork now happens just before the replacement upload, so a skipped or failed upload no longer removes the old tracking label
  • README updated

@Thoroslives

Copy link
Copy Markdown
Contributor Author

For speed increase metrics I ran a bulk scrape and it completed around 33k assets in roughly 16 minutes, this is an example of real world usage speed increase.

@Thoroslives
Thoroslives force-pushed the speedup-upstream branch 2 times, most recently from b7f805b to 93cf858 Compare July 16, 2026 07:30
@Thoroslives

Copy link
Copy Markdown
Contributor Author

Rebased onto v0.8.8 for clean mergeability.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant