Skip to content

Add a persistent index of user uploads for incremental ThePosterDB scrapes#57

Open
Thoroslives wants to merge 3 commits into
mscodemonkey:mainfrom
Thoroslives:persistent-user-cache
Open

Add a persistent index of user uploads for incremental ThePosterDB scrapes#57
Thoroslives wants to merge 3 commits into
mscodemonkey:mainfrom
Thoroslives:persistent-user-cache

Conversation

@Thoroslives

Copy link
Copy Markdown
Contributor

Scraping a ThePosterDB user currently re-fetches their entire uploads listing every run. For a user with thousands of uploads that is hundreds of page requests just to rediscover posters we have already seen. This adds an opt-in cache_user_scrapes option that keeps a small SQLite index (standard library sqlite3, one file in the config directory) of each scraped user's uploads. User pages are newest-first, so a repeat scrape only fetches pages until it reaches a page consisting entirely of already-indexed uploads, records anything new, then processes the user's full catalogue from the index. The results (filters, excludes, labels, log lines and counts) are the same as a full crawl, it just skips re-downloading pages it has already seen. Steady-state re-runs of a big user drop from hundreds of page requests to two, which is also much kinder to ThePosterDB.

To stay honest against edits, deletions and ordering surprises, every user_cache_refresh_days days (default 7) the next scrape of that user quietly re-crawls every page and reconciles the index (uploads that have disappeared are tombstoned and no longer applied). The scrape also cross-checks ThePosterDB's own upload counter against the index each run and falls back to a full re-crawl on any unexplained difference. --no-cache forces a full crawl for one run (CLI, bulk lines and the web UI), and the index is stored with a normalized title key so future features can look cached artwork up by title.

  • cache_user_scrapes in config.json (default false, so behaviour is unchanged unless enabled) plus a toggle in the web UI; user_cache_refresh_days (default 7)
  • Applies to ThePosterDB user scrapes only; set/poster URLs and MediUX are untouched
  • No new dependencies; a few pytest unit tests for the new index logic under tests/ (using the markers already defined in pytest.ini)
  • README updated

@Thoroslives

Copy link
Copy Markdown
Contributor Author

Pushed a couple of follow-up commits to this branch:

  • Stop the cached crawl at the end of the uploads. With the persistent index, a user whose TPDB upload counter overshoots the pages it actually lists was crawling on into empty pages every run; the parse error there marked the crawl "unclean", which blocked both the crawl-state record and the tombstone pass, so the affected user did a full crawl every time and never reconciled. Now the crawl stops at the first page that fetches cleanly but lists nothing.
  • Guard the tombstone pass. A full crawl only reconciles (tombstones assets it didn't see) when it actually covered most of the user's reported uploads; a crawl cut short by a bad page records a non-full crawl and retries next run instead of deleting what it didn't reach.
  • Report new cache entries in the run summary.

Note: the "stop at an empty page" part relies on get_posters() no longer raising on a page with no poster grid, which I've submitted separately as #61 (it's a general crawl bug independent of the index). If you're happy with both, merging #61 first keeps this one a clean rebase.

@Thoroslives
Thoroslives force-pushed the persistent-user-cache branch 3 times, most recently from af104e7 to a23be84 Compare July 16, 2026 07:30
When ThePosterDB's upload counter overshoots the pages it lists, the crawl used
to run on into empty pages; the parse error on those marked the crawl unclean,
which blocked both the crawl-state record and the tombstone pass, so an affected
user did a full crawl every run and never reconciled.

Stop the crawl at the first page that fetches cleanly but lists nothing, and
don't treat that empty page as a failure. Guard the tombstone pass so a full
crawl that was cut short (a markup change parsing to zero posters) never deletes
what it didn't reach: only reconcile when the crawl covered most of the user's
reported uploads, and otherwise leave the full-crawl timestamp unset so the next
run retries. Also surface how many new assets a run added to the cache.
@Thoroslives
Thoroslives force-pushed the persistent-user-cache branch from a23be84 to a3bfcca Compare July 20, 2026 22:06
@Thoroslives

Copy link
Copy Markdown
Contributor Author

Rebased onto v0.8.8 for clean mergeability.

… label

_hydrate_from_cache built the artwork dicts that feed the upload processor
with the key "type", but get_posters and the processor use "file_type".
ARTWORK_ID_MAP.get(None) then returns None and PlexUploader.set_artwork does
None + md5(url), raising "unsupported operand type(s) for +: NoneType and str".
Latent until a cached crawl meets an unlocked in-library item.

Regression test drives _hydrate_from_cache for real (the existing cache tests
stub it out) and asserts every produced dict carries a file_type that
ARTWORK_ID_MAP resolves.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant