Add a persistent index of user uploads for incremental ThePosterDB scrapes#57
Open
Thoroslives wants to merge 3 commits into
Open
Add a persistent index of user uploads for incremental ThePosterDB scrapes#57Thoroslives wants to merge 3 commits into
Thoroslives wants to merge 3 commits into
Conversation
1 task
Contributor
Author
|
Pushed a couple of follow-up commits to this branch:
Note: the "stop at an empty page" part relies on |
Thoroslives
force-pushed
the
persistent-user-cache
branch
3 times, most recently
from
July 16, 2026 07:30
af104e7 to
a23be84
Compare
When ThePosterDB's upload counter overshoots the pages it lists, the crawl used to run on into empty pages; the parse error on those marked the crawl unclean, which blocked both the crawl-state record and the tombstone pass, so an affected user did a full crawl every run and never reconciled. Stop the crawl at the first page that fetches cleanly but lists nothing, and don't treat that empty page as a failure. Guard the tombstone pass so a full crawl that was cut short (a markup change parsing to zero posters) never deletes what it didn't reach: only reconcile when the crawl covered most of the user's reported uploads, and otherwise leave the full-crawl timestamp unset so the next run retries. Also surface how many new assets a run added to the cache.
Thoroslives
force-pushed
the
persistent-user-cache
branch
from
July 20, 2026 22:06
a23be84 to
a3bfcca
Compare
Contributor
Author
|
Rebased onto v0.8.8 for clean mergeability. |
… label _hydrate_from_cache built the artwork dicts that feed the upload processor with the key "type", but get_posters and the processor use "file_type". ARTWORK_ID_MAP.get(None) then returns None and PlexUploader.set_artwork does None + md5(url), raising "unsupported operand type(s) for +: NoneType and str". Latent until a cached crawl meets an unlocked in-library item. Regression test drives _hydrate_from_cache for real (the existing cache tests stub it out) and asserts every produced dict carries a file_type that ARTWORK_ID_MAP resolves. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scraping a ThePosterDB user currently re-fetches their entire uploads listing every run. For a user with thousands of uploads that is hundreds of page requests just to rediscover posters we have already seen. This adds an opt-in
cache_user_scrapesoption that keeps a small SQLite index (standard librarysqlite3, one file in the config directory) of each scraped user's uploads. User pages are newest-first, so a repeat scrape only fetches pages until it reaches a page consisting entirely of already-indexed uploads, records anything new, then processes the user's full catalogue from the index. The results (filters, excludes, labels, log lines and counts) are the same as a full crawl, it just skips re-downloading pages it has already seen. Steady-state re-runs of a big user drop from hundreds of page requests to two, which is also much kinder to ThePosterDB.To stay honest against edits, deletions and ordering surprises, every
user_cache_refresh_daysdays (default 7) the next scrape of that user quietly re-crawls every page and reconciles the index (uploads that have disappeared are tombstoned and no longer applied). The scrape also cross-checks ThePosterDB's own upload counter against the index each run and falls back to a full re-crawl on any unexplained difference.--no-cacheforces a full crawl for one run (CLI, bulk lines and the web UI), and the index is stored with a normalized title key so future features can look cached artwork up by title.cache_user_scrapesin config.json (defaultfalse, so behaviour is unchanged unless enabled) plus a toggle in the web UI;user_cache_refresh_days(default 7)tests/(using the markers already defined in pytest.ini)