Add a persistent index of user uploads for incremental ThePosterDB scrapes

README.md

@@ -16,7 +16,7 @@ We (optionally) store an artwork ID in a Plex label against each movie, show, ep
 If you really want to upload artwork again, use the ```--force``` option at the command line, in the bulk file, or when entering the URL in the Web UI.
 
 ### ThePosterDB scraping
-There are also a couple of new options for thePosterDb, which will allow you to also grab additional sets and additional posters from the same page.  This is sometimes useful for big sets like the Marvel or Disney movies, where you'll otherwise need to specify multiple sets.  This is against the terms of service of theposterdb.com so we encourage you to login, download the files you want, and upload them using this tool, rather than scraping.  Once an API is available we'll switch over ASAP.
+There are also a couple of new options for thePosterDb, which will allow you to also grab additional sets and additional posters from the same page.  This is sometimes useful for big sets like the Marvel or Disney movies, where you'll otherwise need to specify multiple sets.  This is against the terms of service of theposterdb.com so we encourage you to login, download the files you want, and upload them using this tool, rather than scraping.  Once an API is available we'll switch over ASAP.  If you regularly scrape the same ThePosterDB users, enable ```cache_user_scrapes``` so repeat scrapes only fetch new uploads instead of re-crawling the whole catalogue every time.
 
 ### Per-URL filtering and artwork excludes 
 And there are other options such as per-URL filtering, fixing missing things that I found while I was using the tool (where I wanted to apply episode title cards but didn't like the season artwork for example).  And if you don't like a particular piece of artwork or poster from a set, you can now exclude it. You can also exclude entire seasons or individual episodes, that way the app doesn't have to provess all the previous seasons for which you already have artwork applied.
@@ -159,7 +159,7 @@ Basic scheduler, so that you can leave this running and update all your artwork
 
 The basic version is available now on the bulk imports page, click on the clock to enable or disable per file.  
 
-It's there for when we have API access (and works for scrapers in the meantime) but is limited to running once a day which should be fine.
+It's there for when we have API access (and works for scrapers in the meantime) but is limited to running once a day which should be fine.  If you enable ```cache_user_scrapes```, a daily scheduled user scrape only fetches uploads that are new since the last run, so scheduled full-catalogue runs stay fast.
 
 If you enable ```skip_locked_artwork```, a scheduled user or bulk scrape will only fill items that are still on their default artwork and leave anything you've set by hand untouched, so it's safe to leave running against a curated library.
 
@@ -232,6 +232,12 @@ This is optional - if you don't do this, a new config.json will be created when
 ```"skip_locked_artwork"```
 - Setting this to ```true``` will skip any artwork whose target field (poster, background or square art) is locked in Plex, unless ```--force``` is used.  Plex locks a field whenever artwork is deliberately set - manually or by an upload - so this makes scheduled bulk imports and user scrapes fill items still on default artwork while leaving anything you've already set alone.
 - Setting to ```false``` (the default) keeps the existing behaviour where artwork is applied regardless of locks.
+```"cache_user_scrapes"```
+- Setting this to ```true``` keeps a local index of each ThePosterDB user's uploads (a small SQLite file in your config directory), so scraping a user page again only fetches pages until it reaches uploads it has already seen - full-catalogue re-runs drop from hundreds of page requests to a couple, which is also much kinder to ThePosterDB.  Every ```user_cache_refresh_days``` days the next scrape of that user re-crawls every page to pick up edited or deleted uploads.
+- Setting to ```false``` (the default) crawls every page on every user scrape, exactly as before.
+
+```"user_cache_refresh_days"```
+- The number of days between full re-crawls of a cached user's uploads (default ```7```).  Only used when ```cache_user_scrapes``` is ```true```.
 
 ```"auto_manage_bulk_files"```
 - Setting this to ```true``` will automatically add, label and sort URLs from the scrape tab into the currently loaded bulk import file.  At the moment it won't auto-save, but I might add that later.
@@ -387,6 +393,8 @@ The script supports various command-line arguments for flexible use.
 
 ```--stage```, in conjuction with --kometa (or if ```save_to_kometa```is true in ```config.json```), will download assets for TV show seasons and episodes not yet available in Plex. This can be useful if you run the script before a particular season or episode is downloaded by your automation. If ```stage_assets``` is set to ```true``` in ```config.json``` then this argument is not necessary. This option does not apply to the Specials season (Season 0).
 
+```--no-cache``` will crawl every page of a ThePosterDB user this run, ignoring the cached index (the index is still refreshed).  Handy to force a full refresh of a user when you have ```cache_user_scrapes``` enabled.  Works on the command line, in bulk files and in the Web UI.
+
 ### Using these options in files and GUI
 
    These options can also be used in the URL scraper GUI, and in your bulk file, just add them straight after the URL in each line, for example 

artwork_uploader.py

@@ -248,6 +248,7 @@ def process_bulk_import_from_ui(instance: Instance, parsed_urls: list, filename:
     # Track successful poster uploads (those with ✅ or ♻️)
     success_counter = [0]
     assets_processed = [0]
+    cached_counter = [0]
     errors = 0
 
     try:
@@ -275,7 +276,8 @@ def process_bulk_import_from_ui(instance: Instance, parsed_urls: list, filename:
             notify_web(instance, "element_disable", {"element": ["bulk_button"], "mode": True})
 
             try:
-                scrape_and_upload(instance, parsed_line.url, parsed_line.options, True, success_counter, assets_processed)
+                scrape_and_upload(instance, parsed_line.url, parsed_line.options, True, success_counter, assets_processed, cached_counter=cached_counter)
+                #time.sleep(1)
             except ScraperException as e:
                 update_log(instance, f"❌ Error processing line: '{parsed_line.url}'")
                 debug_me(f"ScraperException: Failed to scrape URL: {parsed_line.url} | {str(e)}")
@@ -295,6 +297,7 @@ def process_bulk_import_from_ui(instance: Instance, parsed_urls: list, filename:
                 + ("Scheduled b" if scheduled else "B")
                 + f"ulk import of '{display_filename}' stopped by user • "
                 + f"{assets_processed[0]} asset(s) processed • "
+                + (f"{cached_counter[0]} new in cache • " if cached_counter[0] else "")
                 + f"{success_counter[0]} asset(s) updated"
             )
             update_status(instance, message[2:], color=StatusColor.WARNING.value, sticky=False, spinner=False)
@@ -306,6 +309,7 @@ def process_bulk_import_from_ui(instance: Instance, parsed_urls: list, filename:
                 + f"ulk import of '{display_filename}' completed "
                 + (f"successfully in {elapsed} • " if errors == 0 else f"with {errors} error(s) in {elapsed}, check logs for details • ")
                 + f"{assets_processed[0]} asset(s) processed • "
+                + (f"{cached_counter[0]} new in cache • " if cached_counter[0] else "")
                 + f"{success_counter[0]} asset(s) updated"
             )
             update_status(instance, message[2:], color=StatusColor.SUCCESS.value if errors == 0 else StatusColor.WARNING.value, sticky=False, spinner=False)
@@ -327,7 +331,7 @@ def process_bulk_import_from_ui(instance: Instance, parsed_urls: list, filename:
         notify_web(instance, "add_spinner", { "element": "bulk_button", "mode": False })
 
 # Scraped the URL then uploads what it's scraped to Plex or download to Kometa asset directory
-def scrape_and_upload(instance: Instance, url, options, bulk=False, success_counter=None, assets_processed=None):
+def scrape_and_upload(instance: Instance, url, options, bulk=False, success_counter=None, assets_processed=None, cached_counter=None):
     """
     Scrape artwork from a URL and upload to Plex.
 
@@ -354,7 +358,8 @@ def progress_callback(current: int, total: int, title: str, bar_type:str = "main
         on_debug=debug_callback,
         on_progress_update=progress_callback,
         success_counter=success_counter,
-        assets_processed=assets_processed
+        assets_processed=assets_processed,
+        cached_counter=cached_counter
     )
 
     # Use the service to do the actual work
@@ -670,7 +675,8 @@ def update_scheduled_jobs():
                           year=args.year,
                           kometa=args.kometa,
                           stage=args.stage,
-                          temp=args.temp)  # Arguments per url to process
+                          temp=args.temp,
+                          no_cache=args.no_cache)  # Arguments per url to process
 
     # Create config as a global object
     config = Config()

core/config.py

@@ -31,6 +31,8 @@ class Config:
         stage_assets: Whether to download assets for seasons and episodes that are not in Plex yet (except Specials)
         track_artwork_ids: Whether to track artwork IDs using Plex labels
         skip_locked_artwork: Whether to skip artwork whose target field is locked in Plex (already set)
+        cache_user_scrapes: Whether to keep a persistent index of ThePosterDB users' uploads so repeat scrapes only fetch new ones
+        user_cache_refresh_days: Days between full re-crawls of a cached user's uploads (catches edits and deletions)
         auto_manage_bulk_files: Whether to auto-organize bulk files
         reset_overlay: Whether to reset Kometa overlay labels on upload
         schedules: List of scheduled bulk import jobs
@@ -55,6 +57,8 @@ def __init__(self, config_path: str = "config/config.json") -> None:
         self.stage_assets: bool = False
         self.track_artwork_ids: bool = True
         self.skip_locked_artwork: bool = False
+        self.cache_user_scrapes: bool = False
+        self.user_cache_refresh_days: int = 7
         self.auto_manage_bulk_files: bool = True
         self.reset_overlay: bool = False
         self.schedules: List[Dict[str, Any]] = []
@@ -93,6 +97,8 @@ def load(self) -> None:
             self.bulk_txt = config.get("bulk_txt", "bulk_import.txt")
             self.track_artwork_ids = config.get("track_artwork_ids", True)
             self.skip_locked_artwork = config.get("skip_locked_artwork", False)
+            self.cache_user_scrapes = config.get("cache_user_scrapes", False)
+            self.user_cache_refresh_days = config.get("user_cache_refresh_days", 7)
             self.auto_manage_bulk_files = config.get("auto_manage_bulk_files", True)
             self.reset_overlay = config.get("reset_overlay", False)
             self.schedules = config.get("schedules", [])
@@ -120,6 +126,8 @@ def create(self) -> None:
             "stage_assets": False,
             "track_artwork_ids": True,
             "skip_locked_artwork": False,
+            "cache_user_scrapes": False,
+            "user_cache_refresh_days": 7,
             "auto_manage_bulk_files": True,
             "reset_overlay": True,
             "schedules": [],
@@ -161,6 +169,8 @@ def save(self) -> None:
             "bulk_txt": self.bulk_txt,
             "track_artwork_ids": self.track_artwork_ids,
             "skip_locked_artwork": self.skip_locked_artwork,
+            "cache_user_scrapes": self.cache_user_scrapes,
+            "user_cache_refresh_days": self.user_cache_refresh_days,
             "auto_manage_bulk_files": self.auto_manage_bulk_files,
             "reset_overlay": self.reset_overlay,
             "schedules": self.schedules,

core/constants.py

@@ -20,6 +20,7 @@
 
 # File paths
 DEFAULT_CONFIG_PATH = "config.json"
+ASSET_INDEX_PATH = "config/asset_index.db"
 DEFAULT_BULK_IMPORTS_DIR = "bulk_imports"
 DEFAULT_BULK_IMPORT_FILE = "bulk_import.txt"
 
@@ -101,6 +102,10 @@
 TPDB_API_ASSETS_URL = "https://theposterdb.com/api/assets"
 TPDB_RATE_LIMIT_DELAY = 6  # seconds between requests
 TPDB_USER_UPLOADS_PER_PAGE = 24
+# A full crawl only tombstones assets it didn't see if it reached at least this share of the
+# user's reported uploads. ThePosterDB's counter runs a little high, so this is below 1.0, but a
+# crawl cut short by a bad page must never be mistaken for a catalogue that shrank.
+RECONCILE_MIN_COVERAGE = 0.9
 
 # MediUX configuration
 MEDIUX_BASE_URL = "https://mediux.pro"

models/arguments.py

@@ -15,6 +15,7 @@
 # --kometa          Saves artwork to Kometa asset directory (specified in config file) instead of uploading to Plex.
 # --stage           Downloads artwork for seasons and episodes that are not in Plex yet (except Specials).
 # --temp            Uses a temporary directory (specified in config file) instead of the Kometa asset directory.
+# --no-cache        Ignore the cached ThePosterDB user uploads index for this run and crawl every page.
 # ---------------------------------------------------------
 
 def parse_arguments():
@@ -35,5 +36,6 @@ def parse_arguments():
     parser.add_argument("--kometa", action='store_true', help="Saves artwork to Kometa asset directory (specified in config file) instead of uploading to Plex.")
     parser.add_argument("--stage", action='store_true', help="Downloads artwork for seasons and episodes that are not in Plex yet (except Specials).")
     parser.add_argument("--temp", action='store_true', help="Uses a temporary directory (specified in config file) instead of the Kometa asset directory.")
+    parser.add_argument("--no-cache", action='store_true', help="Ignore the cached ThePosterDB user uploads index for this run and crawl every page (the run still refreshes the index).")
 
     return parser.parse_args()

models/callbacks.py

@@ -15,6 +15,7 @@ class ProcessingCallbacks:
     on_debug: Optional[Callable[[str, Optional[str]], None]] = None  # (message, context) - for debug messages
     success_counter: Optional[list] = None  # Mutable list to track successful uploads (contains count as single element)
     assets_processed: Optional[list] = None  # Mutable list to track total assets processed (contains count as single element)
+    cached_counter: Optional[list] = None  # Mutable list to track assets newly added to the user cache (contains count as single element)
 
     def status(self, message: str, color: str = "info", spinner: bool = False, sticky: bool = False):
         if self.on_status_update:
@@ -38,4 +39,8 @@ def success(self, count: int):
 
     def assets(self, count: int):
         if self.assets_processed:
-            self.assets_processed[0] += count
+            self.assets_processed[0] += count
+
+    def cached(self, count: int):
+        if self.cached_counter:
+            self.cached_counter[0] += count

models/options.py

@@ -21,6 +21,7 @@ class Options:
         temp: Use temporary directory instead of Kometa asset directory
         force: Force re-upload even if artwork hasn't changed
         skip_locked: Skip artwork when the target Plex field is locked (already set)
+        no_cache: Ignore the cached user uploads index for this run and crawl every page
         filters: List of artwork types to include (e.g., ['show_cover', 'title_card'])
         exclude: List of artwork IDs to skip
         year: Override year for Plex matching
@@ -34,6 +35,7 @@ class Options:
     temp: bool = False
     force: bool = False
     skip_locked: bool = False
+    no_cache: bool = False
     filters: List[str] = field(default_factory=list)
     exclude: Optional[List[str]] = None
     year: Optional[int] = None