diff --git a/README.md b/README.md index c2b0912..451383c 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ We (optionally) store an artwork ID in a Plex label against each movie, show, ep If you really want to upload artwork again, use the ```--force``` option at the command line, in the bulk file, or when entering the URL in the Web UI. ### ThePosterDB scraping -There are also a couple of new options for thePosterDb, which will allow you to also grab additional sets and additional posters from the same page. This is sometimes useful for big sets like the Marvel or Disney movies, where you'll otherwise need to specify multiple sets. This is against the terms of service of theposterdb.com so we encourage you to login, download the files you want, and upload them using this tool, rather than scraping. Once an API is available we'll switch over ASAP. +There are also a couple of new options for thePosterDb, which will allow you to also grab additional sets and additional posters from the same page. This is sometimes useful for big sets like the Marvel or Disney movies, where you'll otherwise need to specify multiple sets. This is against the terms of service of theposterdb.com so we encourage you to login, download the files you want, and upload them using this tool, rather than scraping. Once an API is available we'll switch over ASAP. If you regularly scrape the same ThePosterDB users, enable ```cache_user_scrapes``` so repeat scrapes only fetch new uploads instead of re-crawling the whole catalogue every time. ### Per-URL filtering and artwork excludes And there are other options such as per-URL filtering, fixing missing things that I found while I was using the tool (where I wanted to apply episode title cards but didn't like the season artwork for example). And if you don't like a particular piece of artwork or poster from a set, you can now exclude it. You can also exclude entire seasons or individual episodes, that way the app doesn't have to provess all the previous seasons for which you already have artwork applied. @@ -159,7 +159,7 @@ Basic scheduler, so that you can leave this running and update all your artwork The basic version is available now on the bulk imports page, click on the clock to enable or disable per file. -It's there for when we have API access (and works for scrapers in the meantime) but is limited to running once a day which should be fine. +It's there for when we have API access (and works for scrapers in the meantime) but is limited to running once a day which should be fine. If you enable ```cache_user_scrapes```, a daily scheduled user scrape only fetches uploads that are new since the last run, so scheduled full-catalogue runs stay fast. If you enable ```skip_locked_artwork```, a scheduled user or bulk scrape will only fill items that are still on their default artwork and leave anything you've set by hand untouched, so it's safe to leave running against a curated library. @@ -232,6 +232,12 @@ This is optional - if you don't do this, a new config.json will be created when ```"skip_locked_artwork"``` - Setting this to ```true``` will skip any artwork whose target field (poster, background or square art) is locked in Plex, unless ```--force``` is used. Plex locks a field whenever artwork is deliberately set - manually or by an upload - so this makes scheduled bulk imports and user scrapes fill items still on default artwork while leaving anything you've already set alone. - Setting to ```false``` (the default) keeps the existing behaviour where artwork is applied regardless of locks. +```"cache_user_scrapes"``` +- Setting this to ```true``` keeps a local index of each ThePosterDB user's uploads (a small SQLite file in your config directory), so scraping a user page again only fetches pages until it reaches uploads it has already seen - full-catalogue re-runs drop from hundreds of page requests to a couple, which is also much kinder to ThePosterDB. Every ```user_cache_refresh_days``` days the next scrape of that user re-crawls every page to pick up edited or deleted uploads. +- Setting to ```false``` (the default) crawls every page on every user scrape, exactly as before. + +```"user_cache_refresh_days"``` +- The number of days between full re-crawls of a cached user's uploads (default ```7```). Only used when ```cache_user_scrapes``` is ```true```. ```"auto_manage_bulk_files"``` - Setting this to ```true``` will automatically add, label and sort URLs from the scrape tab into the currently loaded bulk import file. At the moment it won't auto-save, but I might add that later. @@ -387,6 +393,8 @@ The script supports various command-line arguments for flexible use. ```--stage```, in conjuction with --kometa (or if ```save_to_kometa```is true in ```config.json```), will download assets for TV show seasons and episodes not yet available in Plex. This can be useful if you run the script before a particular season or episode is downloaded by your automation. If ```stage_assets``` is set to ```true``` in ```config.json``` then this argument is not necessary. This option does not apply to the Specials season (Season 0). +```--no-cache``` will crawl every page of a ThePosterDB user this run, ignoring the cached index (the index is still refreshed). Handy to force a full refresh of a user when you have ```cache_user_scrapes``` enabled. Works on the command line, in bulk files and in the Web UI. + ### Using these options in files and GUI These options can also be used in the URL scraper GUI, and in your bulk file, just add them straight after the URL in each line, for example diff --git a/artwork_uploader.py b/artwork_uploader.py index 47b91ed..148d2dc 100644 --- a/artwork_uploader.py +++ b/artwork_uploader.py @@ -248,6 +248,7 @@ def process_bulk_import_from_ui(instance: Instance, parsed_urls: list, filename: # Track successful poster uploads (those with ✅ or ♻️) success_counter = [0] assets_processed = [0] + cached_counter = [0] errors = 0 try: @@ -275,7 +276,8 @@ def process_bulk_import_from_ui(instance: Instance, parsed_urls: list, filename: notify_web(instance, "element_disable", {"element": ["bulk_button"], "mode": True}) try: - scrape_and_upload(instance, parsed_line.url, parsed_line.options, True, success_counter, assets_processed) + scrape_and_upload(instance, parsed_line.url, parsed_line.options, True, success_counter, assets_processed, cached_counter=cached_counter) + #time.sleep(1) except ScraperException as e: update_log(instance, f"❌ Error processing line: '{parsed_line.url}'") debug_me(f"ScraperException: Failed to scrape URL: {parsed_line.url} | {str(e)}") @@ -295,6 +297,7 @@ def process_bulk_import_from_ui(instance: Instance, parsed_urls: list, filename: + ("Scheduled b" if scheduled else "B") + f"ulk import of '{display_filename}' stopped by user • " + f"{assets_processed[0]} asset(s) processed • " + + (f"{cached_counter[0]} new in cache • " if cached_counter[0] else "") + f"{success_counter[0]} asset(s) updated" ) update_status(instance, message[2:], color=StatusColor.WARNING.value, sticky=False, spinner=False) @@ -306,6 +309,7 @@ def process_bulk_import_from_ui(instance: Instance, parsed_urls: list, filename: + f"ulk import of '{display_filename}' completed " + (f"successfully in {elapsed} • " if errors == 0 else f"with {errors} error(s) in {elapsed}, check logs for details • ") + f"{assets_processed[0]} asset(s) processed • " + + (f"{cached_counter[0]} new in cache • " if cached_counter[0] else "") + f"{success_counter[0]} asset(s) updated" ) update_status(instance, message[2:], color=StatusColor.SUCCESS.value if errors == 0 else StatusColor.WARNING.value, sticky=False, spinner=False) @@ -327,7 +331,7 @@ def process_bulk_import_from_ui(instance: Instance, parsed_urls: list, filename: notify_web(instance, "add_spinner", { "element": "bulk_button", "mode": False }) # Scraped the URL then uploads what it's scraped to Plex or download to Kometa asset directory -def scrape_and_upload(instance: Instance, url, options, bulk=False, success_counter=None, assets_processed=None): +def scrape_and_upload(instance: Instance, url, options, bulk=False, success_counter=None, assets_processed=None, cached_counter=None): """ Scrape artwork from a URL and upload to Plex. @@ -354,7 +358,8 @@ def progress_callback(current: int, total: int, title: str, bar_type:str = "main on_debug=debug_callback, on_progress_update=progress_callback, success_counter=success_counter, - assets_processed=assets_processed + assets_processed=assets_processed, + cached_counter=cached_counter ) # Use the service to do the actual work @@ -670,7 +675,8 @@ def update_scheduled_jobs(): year=args.year, kometa=args.kometa, stage=args.stage, - temp=args.temp) # Arguments per url to process + temp=args.temp, + no_cache=args.no_cache) # Arguments per url to process # Create config as a global object config = Config() diff --git a/core/config.py b/core/config.py index f593424..ec0e669 100644 --- a/core/config.py +++ b/core/config.py @@ -31,6 +31,8 @@ class Config: stage_assets: Whether to download assets for seasons and episodes that are not in Plex yet (except Specials) track_artwork_ids: Whether to track artwork IDs using Plex labels skip_locked_artwork: Whether to skip artwork whose target field is locked in Plex (already set) + cache_user_scrapes: Whether to keep a persistent index of ThePosterDB users' uploads so repeat scrapes only fetch new ones + user_cache_refresh_days: Days between full re-crawls of a cached user's uploads (catches edits and deletions) auto_manage_bulk_files: Whether to auto-organize bulk files reset_overlay: Whether to reset Kometa overlay labels on upload schedules: List of scheduled bulk import jobs @@ -55,6 +57,8 @@ def __init__(self, config_path: str = "config/config.json") -> None: self.stage_assets: bool = False self.track_artwork_ids: bool = True self.skip_locked_artwork: bool = False + self.cache_user_scrapes: bool = False + self.user_cache_refresh_days: int = 7 self.auto_manage_bulk_files: bool = True self.reset_overlay: bool = False self.schedules: List[Dict[str, Any]] = [] @@ -93,6 +97,8 @@ def load(self) -> None: self.bulk_txt = config.get("bulk_txt", "bulk_import.txt") self.track_artwork_ids = config.get("track_artwork_ids", True) self.skip_locked_artwork = config.get("skip_locked_artwork", False) + self.cache_user_scrapes = config.get("cache_user_scrapes", False) + self.user_cache_refresh_days = config.get("user_cache_refresh_days", 7) self.auto_manage_bulk_files = config.get("auto_manage_bulk_files", True) self.reset_overlay = config.get("reset_overlay", False) self.schedules = config.get("schedules", []) @@ -120,6 +126,8 @@ def create(self) -> None: "stage_assets": False, "track_artwork_ids": True, "skip_locked_artwork": False, + "cache_user_scrapes": False, + "user_cache_refresh_days": 7, "auto_manage_bulk_files": True, "reset_overlay": True, "schedules": [], @@ -161,6 +169,8 @@ def save(self) -> None: "bulk_txt": self.bulk_txt, "track_artwork_ids": self.track_artwork_ids, "skip_locked_artwork": self.skip_locked_artwork, + "cache_user_scrapes": self.cache_user_scrapes, + "user_cache_refresh_days": self.user_cache_refresh_days, "auto_manage_bulk_files": self.auto_manage_bulk_files, "reset_overlay": self.reset_overlay, "schedules": self.schedules, diff --git a/core/constants.py b/core/constants.py index 26fdee2..35880ca 100644 --- a/core/constants.py +++ b/core/constants.py @@ -20,6 +20,7 @@ # File paths DEFAULT_CONFIG_PATH = "config.json" +ASSET_INDEX_PATH = "config/asset_index.db" DEFAULT_BULK_IMPORTS_DIR = "bulk_imports" DEFAULT_BULK_IMPORT_FILE = "bulk_import.txt" @@ -101,6 +102,10 @@ TPDB_API_ASSETS_URL = "https://theposterdb.com/api/assets" TPDB_RATE_LIMIT_DELAY = 6 # seconds between requests TPDB_USER_UPLOADS_PER_PAGE = 24 +# A full crawl only tombstones assets it didn't see if it reached at least this share of the +# user's reported uploads. ThePosterDB's counter runs a little high, so this is below 1.0, but a +# crawl cut short by a bad page must never be mistaken for a catalogue that shrank. +RECONCILE_MIN_COVERAGE = 0.9 # MediUX configuration MEDIUX_BASE_URL = "https://mediux.pro" diff --git a/models/arguments.py b/models/arguments.py index 880d078..6f268fc 100644 --- a/models/arguments.py +++ b/models/arguments.py @@ -15,6 +15,7 @@ # --kometa Saves artwork to Kometa asset directory (specified in config file) instead of uploading to Plex. # --stage Downloads artwork for seasons and episodes that are not in Plex yet (except Specials). # --temp Uses a temporary directory (specified in config file) instead of the Kometa asset directory. +# --no-cache Ignore the cached ThePosterDB user uploads index for this run and crawl every page. # --------------------------------------------------------- def parse_arguments(): @@ -35,5 +36,6 @@ def parse_arguments(): parser.add_argument("--kometa", action='store_true', help="Saves artwork to Kometa asset directory (specified in config file) instead of uploading to Plex.") parser.add_argument("--stage", action='store_true', help="Downloads artwork for seasons and episodes that are not in Plex yet (except Specials).") parser.add_argument("--temp", action='store_true', help="Uses a temporary directory (specified in config file) instead of the Kometa asset directory.") + parser.add_argument("--no-cache", action='store_true', help="Ignore the cached ThePosterDB user uploads index for this run and crawl every page (the run still refreshes the index).") return parser.parse_args() diff --git a/models/callbacks.py b/models/callbacks.py index 2b5a104..3096f89 100644 --- a/models/callbacks.py +++ b/models/callbacks.py @@ -15,6 +15,7 @@ class ProcessingCallbacks: on_debug: Optional[Callable[[str, Optional[str]], None]] = None # (message, context) - for debug messages success_counter: Optional[list] = None # Mutable list to track successful uploads (contains count as single element) assets_processed: Optional[list] = None # Mutable list to track total assets processed (contains count as single element) + cached_counter: Optional[list] = None # Mutable list to track assets newly added to the user cache (contains count as single element) def status(self, message: str, color: str = "info", spinner: bool = False, sticky: bool = False): if self.on_status_update: @@ -38,4 +39,8 @@ def success(self, count: int): def assets(self, count: int): if self.assets_processed: - self.assets_processed[0] += count \ No newline at end of file + self.assets_processed[0] += count + + def cached(self, count: int): + if self.cached_counter: + self.cached_counter[0] += count \ No newline at end of file diff --git a/models/options.py b/models/options.py index 439f649..da15e4a 100644 --- a/models/options.py +++ b/models/options.py @@ -21,6 +21,7 @@ class Options: temp: Use temporary directory instead of Kometa asset directory force: Force re-upload even if artwork hasn't changed skip_locked: Skip artwork when the target Plex field is locked (already set) + no_cache: Ignore the cached user uploads index for this run and crawl every page filters: List of artwork types to include (e.g., ['show_cover', 'title_card']) exclude: List of artwork IDs to skip year: Override year for Plex matching @@ -34,6 +35,7 @@ class Options: temp: bool = False force: bool = False skip_locked: bool = False + no_cache: bool = False filters: List[str] = field(default_factory=list) exclude: Optional[List[str]] = None year: Optional[int] = None diff --git a/scrapers/theposterdb_scraper.py b/scrapers/theposterdb_scraper.py index bd32f9a..37eec1c 100644 --- a/scrapers/theposterdb_scraper.py +++ b/scrapers/theposterdb_scraper.py @@ -10,9 +10,14 @@ from core.config import Config from core import globals from core.enums import MediaType, ScraperSource -from core.constants import TPDB_API_ASSETS_URL, TPDB_USER_UPLOADS_PER_PAGE, BOOTSTRAP_COLORS, ANSI_RESET, ANSI_BOLD +from core.constants import TPDB_API_ASSETS_URL, TPDB_USER_UPLOADS_PER_PAGE, RECONCILE_MIN_COVERAGE, BOOTSTRAP_COLORS, ANSI_RESET, ANSI_BOLD from models.artwork_types import MovieArtworkList, TVArtworkList, CollectionArtworkList +import sqlite3 +from datetime import datetime, timezone +from urllib.parse import urlparse +from services.asset_index import AssetIndex, page_is_fully_known, full_crawl_due + class ThePosterDBScraper: @@ -40,6 +45,10 @@ def __init__(self, url: str, callbacks: Optional[ProcessingCallbacks]) -> None: self.user_uploads: int = 0 self.user_pages: int = 0 + # When set to a list, get_posters records every parsed asset (pre-filter) into it so a + # user crawl can populate the persistent index. None (default) disables recording. + self.catalog: Optional[list] = None + # Set options - otherwise will use defaults of False def set_options(self, options: Options) -> None: @@ -73,6 +82,16 @@ def scrape(self) -> None: self.callbacks.debug(f"There are {self.user_uploads} assets and {self.user_pages} pages for user {self.author}") self.callbacks.progress(0, 0, f"Collecting assets from TPDb user {self.author}") + # With the persistent index enabled, only fetch pages until we reach uploads we + # have already seen, then apply the user's full catalogue from the index. + if self.config.cache_user_scrapes: + try: + self._scrape_user_cached() + return + except sqlite3.Error as cache_error: + self.callbacks.debug(f"Asset index unavailable ({cache_error}); crawling every page", "ThePosterDBScraper/scrape") + self._reset_user_collections() + collected = 0 for user_page in range(self.user_pages): if globals.cancel_scrape: @@ -152,13 +171,15 @@ def scrape_user_info(self) -> None: except (AttributeError, KeyError, ValueError, TypeError) as e: raise ScraperException(f"Can't get user information, please check the URL you're using") from e - def scrape_user_page (self, page) -> None: + def scrape_user_page(self, page, catalog=None) -> bool: try: page_url = f"{self.url}?section=uploads&page={page + 1}" child_scraper = ThePosterDBScraper(page_url, self.callbacks) child_scraper.set_options(self.options) child_scraper.is_child = True + if catalog is not None: + child_scraper.catalog = catalog child_scraper.scrape() for artwork in child_scraper.collection_artwork: @@ -173,9 +194,11 @@ def scrape_user_page (self, page) -> None: self.filtered += child_scraper.filtered self.errored += child_scraper.errored self.total += child_scraper.total + return True except Exception as e: - self.callbacks.debug(f"Failed to scrape user asset page {page}: {str(e)}") + self.callbacks.debug(f"Failed to scrape user asset page {page}: {str(e)}", "ThePosterDBScraper/scrape_user_page") + return False def get_set_title(self, soup: Any) -> None: try: @@ -236,6 +259,14 @@ def get_posters(self, poster_div: Any) -> None: file_type = "show_cover" else: file_type = "season_cover" + + if self.catalog is not None: + self.catalog.append({ + "id": poster_id, "title": title, "year": year, + "season": season if isinstance(season, int) else None, + "media_type": file_type, "author": self.author, + "url": f"{TPDB_API_ASSETS_URL}/{poster_id}", + }) if (self.options.has_no_filters() and file_type in self.config.tpdb_filters) or self.options.has_filter(file_type): if not self.options.is_excluded(poster_id, season if isinstance(season, int) else None, None): @@ -272,6 +303,12 @@ def get_posters(self, poster_div: Any) -> None: ) elif media_type == MediaType.MOVIE.value: title, year = media_metadata.parse_movie(title_p) + if self.catalog is not None: + self.catalog.append({ + "id": poster_id, "title": title, "year": year, "season": None, + "media_type": "movie_poster", "author": self.author, + "url": f"{TPDB_API_ASSETS_URL}/{poster_id}", + }) if (self.options.has_no_filters() and "movie_poster" in self.config.tpdb_filters) or self.options.has_filter("movie_poster"): if not self.options.is_excluded(poster_id): self.callbacks.debug(f"{i+1}. ✅ Including movie poster for '{title} ({year})'.") @@ -293,6 +330,12 @@ def get_posters(self, poster_div: Any) -> None: self.filtered += 1 self.callbacks.debug(f"{i+1}. ⏩ Skipping movie poster for '{title} ({year})' based on filters.") elif media_type == MediaType.COLLECTION.value: + if self.catalog is not None: + self.catalog.append({ + "id": poster_id, "title": title_p, "year": None, "season": None, + "media_type": "collection_poster", "author": self.author, + "url": f"{TPDB_API_ASSETS_URL}/{poster_id}", + }) if (self.options.has_no_filters() and "collection_poster" in self.config.tpdb_filters) or self.options.has_filter("collection_poster"): if not self.options.is_excluded(poster_id): self.callbacks.debug(f"{i+1}. ✅ Including collection poster for '{title_p}'.") @@ -312,6 +355,12 @@ def get_posters(self, poster_div: Any) -> None: self.filtered += 1 self.callbacks.debug(f"{i+1}. ⏩ Skipping collection poster for '{title_p}' based on filters.") else: + if self.catalog is not None: + self.catalog.append({ + "id": poster_id, "title": title_p, "year": None, "season": None, + "media_type": "unknown", "author": self.author, + "url": f"{TPDB_API_ASSETS_URL}/{poster_id}", + }) self.errored += 1 self.callbacks.debug(f"⏩ Skipping artwork item - unknown media type: {title_p} | {poster_url}") self.callbacks.log(f"{f'⚠️ {self.title} • ' if self.title is not None else '⚠️ '}{self.author} | Skipping asset (unknown media type): {title_p}") @@ -353,3 +402,196 @@ def scrape_additional_sets(self) -> None: def scrape_posters(self, soup: Any) -> None: poster_div = soup.find('div', class_='row d-flex flex-wrap m-0 w-100 mx-n1 mt-n1') return self.get_posters(poster_div) + + def _reset_user_collections(self) -> None: + """Clear the collected artwork lists and skip counters for a user scrape.""" + self.movie_artwork = [] + self.tv_artwork = [] + self.collection_artwork = [] + self.skipped = 0 + self.exclusions = 0 + self.filtered = 0 + self.errored = 0 + self.total = 0 + + def _user_key(self) -> str: + """Stable key for a user URL: the handle after /user/, lower-cased so a web-entered + (lower-cased) URL and a CLI-cased one share a single index entry.""" + parts = [segment for segment in urlparse(self.url).path.split("/") if segment] + if "user" in parts: + index = parts.index("user") + if index + 1 < len(parts): + return parts[index + 1].casefold() + return "" + + def _scrape_user_cached(self) -> None: + """Incrementally crawl a user's uploads using the persistent index, then apply the + full catalogue from the index. Raises sqlite3.Error if the index is unusable, so the + caller can fall back to a full crawl.""" + index = AssetIndex() + user_key = self._user_key() + crawl_started_at = datetime.now(timezone.utc).isoformat() + state = index.crawl_state(user_key) + full = self.options.no_cache or full_crawl_due( + state["last_full_crawl"] if state else None, self.config.user_cache_refresh_days) + self.callbacks.debug( + f"Cache {'full crawl' if full else 'incremental crawl'} for user {self.author}", + "ThePosterDBScraper/scrape") + + new_rows, seen_ids, clean = self._crawl_user_pages(index, user_key, full) + + # Cross-check ThePosterDB's own upload counter: if an incremental crawl's arithmetic + # doesn't add up, something moved below the stop point (an edit-bumped re-order or a + # deletion) - promote this run to a full crawl once so nothing is missed. + if not full and clean and state is not None: + expected = (state["last_seen_count"] or 0) + new_rows + if self.user_uploads != expected: + self.callbacks.debug( + f"Upload count for {self.author} changed unexpectedly " + f"(ThePosterDB reports {self.user_uploads}, expected {expected}); re-crawling every page", + "ThePosterDBScraper/scrape") + full = True + new_rows, seen_ids, clean = self._crawl_user_pages(index, user_key, full=True) + + # Only reconcile/record over a crawl with no failed pages, so an invisible page error + # never mass-tombstones live assets. A full crawl now stops at the first empty page, which + # is normally the end of the uploads - but a TPDB markup change could make a mid-catalogue + # page parse to zero and cut the crawl short, and reconciling then would tombstone + # everything it didn't reach. So only count it as a full crawl (reconcile + advance the + # full-crawl timestamp) when it actually covered most of the user's reported uploads; + # otherwise leave the index alone so the next run retries the full crawl. + if clean: + covered = len(seen_ids) >= self.user_uploads * RECONCILE_MIN_COVERAGE + if full and covered: + index.reconcile(user_key, seen_ids, crawl_started_at) + elif full: + self.callbacks.debug( + f"Full crawl for {self.author} saw only {len(seen_ids)} of {self.user_uploads} " + f"reported uploads; not tombstoning and retrying a full crawl next run", + "ThePosterDBScraper/scrape") + index.record_crawl(user_key, full and covered, self.user_uploads) + + if new_rows: + self.callbacks.cached(new_rows) + self.callbacks.log(f"🆕 TPDb user • {self.author} | {new_rows} new asset(s) added to the cache") + + self._hydrate_from_cache(index, user_key) + + def _crawl_user_pages(self, index: AssetIndex, user_key: str, full: bool): + """Crawl the user's upload pages, recording each into the index. An incremental crawl + stops after the first page whose assets are all already indexed. Returns + (new_rows_recorded, ids_seen_this_crawl, all_pages_succeeded).""" + known = index.known_ids(user_key) + seen_ids = set() + new_rows = 0 + clean = True + collected = 0 + for user_page in range(self.user_pages): + self.callbacks.progress(user_page + 1, self.user_pages, f"Collecting assets from TPDb user {self.author} • {user_page + 1} of {self.user_pages} pages • {collected} assets collected of {self.user_uploads}") + page_catalog = [] + ok = self.scrape_user_page(user_page, catalog=page_catalog) + page_ids = {int(asset["id"]) for asset in page_catalog if str(asset.get("id", "")).isdigit()} + if ok: + new_rows += index.record(user_key, page_catalog) + seen_ids |= page_ids + collected += len(page_catalog) + else: + clean = False + self.callbacks.debug(f"Processed {user_page + 1} out of {self.user_pages} user pages. Collected {collected} assets so far", "ThePosterDBScraper/scrape") + # The uploads counter can be higher than the number of assets actually listed, so the + # page count can overshoot. A page that fetched cleanly but held nothing is the end of + # the user's uploads - stop here rather than fetching the phantom pages after it. + if ok and not page_catalog: + self.callbacks.debug(f"No assets on page {user_page + 1}, the user's uploads end here", "ThePosterDBScraper/scrape") + break + if not full and ok and page_is_fully_known(page_ids, known): + self.callbacks.debug(f"Reached already-indexed uploads at page {user_page + 1}, stopping incremental crawl", "ThePosterDBScraper/scrape") + break + return new_rows, seen_ids, clean + + def _hydrate_from_cache(self, index: AssetIndex, user_key: str) -> None: + """Rebuild the artwork lists from the index, applying this run's filters/exclusions + exactly as get_posters does, so a cached run produces the same result as a full crawl.""" + self._reset_user_collections() + cache_buster = f"&_cb={int(time.time())}" + for i, row in enumerate(index.assets_for_user(user_key)): + media_type = row["media_type"] + poster_id = str(row["asset_id"]) + title = row["title"] + year = row["year"] + poster_url = f"{row['url']}{cache_buster}" + + if media_type in ("show_cover", "season_cover"): + file_type = media_type + season = "Cover" if media_type == "show_cover" else row["season"] + if (self.options.has_no_filters() and file_type in self.config.tpdb_filters) or self.options.has_filter(file_type): + if not self.options.is_excluded(poster_id, season if isinstance(season, int) else None, None): + self.callbacks.debug( + f"{i+1}. ✅ Including {file_type.replace('_', ' ')} for '{title} ({year})'" + + (f", Season {season}." if isinstance(season, int) else "."), "ThePosterDBScraper/get_posters") + self.tv_artwork.append({ + "title": title, + "author": self.author, + "tmdb_id": self.tmdb_id, + "url": poster_url, + "season": season, + "episode": None, + "year": year, + "source": ScraperSource.THEPOSTERDB.value, + "id": poster_id, + "file_type": file_type, + }) + else: + self.exclusions += 1 + self.callbacks.debug( + f"{i+1}. ⏩ Skipping {file_type.replace('_', ' ')} for '{title} ({year})'" + + (f", Season {season}." if isinstance(season, int) else "") + + f" based on exclusions.", "ThePosterDBScraper/get_posters") + else: + self.filtered += 1 + self.callbacks.debug( + f"{i+1}. ⏩ Skipping {file_type.replace('_', ' ')} for '{title} ({year})'" + + (f", Season {season}." if isinstance(season, int) else "") + + f" based on filters.", "ThePosterDBScraper/get_posters") + elif media_type == "movie_poster": + if (self.options.has_no_filters() and "movie_poster" in self.config.tpdb_filters) or self.options.has_filter("movie_poster"): + if not self.options.is_excluded(poster_id): + self.callbacks.debug(f"{i+1}. ✅ Including movie poster for '{title} ({year})'.", "ThePosterDBScraper/get_posters") + self.movie_artwork.append({ + "title": title, + "author": self.author, + "tmdb_id": self.tmdb_id, + "url": poster_url, + "year": year, + "source": ScraperSource.THEPOSTERDB.value, + "id": poster_id, + "file_type": "movie_poster", + }) + else: + self.exclusions += 1 + self.callbacks.debug(f"{i+1}. ⏩ Skipping movie poster for '{title} ({year})' based on exclusions.", "ThePosterDBScraper/get_posters") + else: + self.filtered += 1 + self.callbacks.debug(f"{i+1}. ⏩ Skipping movie poster for '{title} ({year})' based on filters.", "ThePosterDBScraper/get_posters") + elif media_type == "collection_poster": + if (self.options.has_no_filters() and "collection_poster" in self.config.tpdb_filters) or self.options.has_filter("collection_poster"): + if not self.options.is_excluded(poster_id): + self.callbacks.debug(f"{i+1}. ✅ Including collection poster for '{title}'.", "ThePosterDBScraper/get_posters") + self.collection_artwork.append({ + "title": title, + "author": self.author, + "url": poster_url, + "source": ScraperSource.THEPOSTERDB.value, + "id": poster_id, + "file_type": "collection_poster", + }) + else: + self.exclusions += 1 + self.callbacks.debug(f"{i+1}. ⏩ Skipping collection poster for '{title}' based on exclusions.", "ThePosterDBScraper/get_posters") + else: + self.filtered += 1 + self.callbacks.debug(f"{i+1}. ⏩ Skipping collection poster for '{title}' based on filters.", "ThePosterDBScraper/get_posters") + + self.skipped = self.exclusions + self.filtered + self.errored + self.total = len(self.movie_artwork) + len(self.tv_artwork) + len(self.collection_artwork) + self.skipped + self.callbacks.debug(f"---------> Total assets applied from cache: {len(self.movie_artwork) + len(self.tv_artwork) + len(self.collection_artwork)} of {self.user_uploads}", "ThePosterDBScraper/scrape") diff --git a/services/__init__.py b/services/__init__.py index 025c7b9..bf58f04 100644 --- a/services/__init__.py +++ b/services/__init__.py @@ -13,6 +13,7 @@ from .utility_service import UtilityService from .authentication_service import AuthenticationService from .notify_service import NotifyService +from .asset_index import AssetIndex __all__ = [ 'BulkFileService', @@ -23,5 +24,6 @@ #'UpdateService', 'UtilityService', 'AuthenticationService', - 'NotifyService' + 'NotifyService', + 'AssetIndex' ] diff --git a/services/asset_index.py b/services/asset_index.py new file mode 100644 index 0000000..0fbe2e1 --- /dev/null +++ b/services/asset_index.py @@ -0,0 +1,266 @@ +""" +Persistent index of ThePosterDB users' uploads. + +A small SQLite database (one file in the config directory) that mirrors the assets a +ThePosterDB user has uploaded. Repeat scrapes of that user then only need to fetch pages +until they reach uploads that are already indexed, and other features can look cached +artwork up by title. +""" + +import os +import re +import sqlite3 +import time +import unicodedata +from datetime import datetime, timezone +from typing import List, Optional, Set + +from core.constants import ASSET_INDEX_PATH + + +def normalize_title(title: str) -> str: + """Lowercase, strip accents and punctuation so titles compare equal regardless of styling, + e.g. 'Mission: Impossible' vs 'Mission - Impossible', 'Léon' vs 'Leon'.""" + title = unicodedata.normalize("NFKD", title).encode("ascii", "ignore").decode() + title = title.casefold().replace("&", " and ") + title = re.sub(r"[^\w\s]", " ", title) + return re.sub(r"\s+", " ", title).strip() + + +def _now() -> str: + """Current time as an ISO-8601 UTC string - sortable and comparable as plain text.""" + return datetime.now(timezone.utc).isoformat() + + +_CREATE_ASSETS = """ + CREATE TABLE IF NOT EXISTS user_assets ( + user_key TEXT NOT NULL, + asset_id INTEGER NOT NULL, + title TEXT, + title_key TEXT, + year INTEGER, + season INTEGER, + media_type TEXT, + author TEXT, + url TEXT, + first_seen TEXT, + last_seen TEXT, + missing_since TEXT, + PRIMARY KEY (user_key, asset_id) + ) +""" + +_CREATE_ASSETS_INDEX = """ + CREATE INDEX IF NOT EXISTS idx_user_assets_title + ON user_assets (title_key, media_type) +""" + +_CREATE_CRAWLS = """ + CREATE TABLE IF NOT EXISTS user_crawls ( + user_key TEXT PRIMARY KEY, + last_full_crawl TEXT, + last_crawl TEXT, + last_seen_count INTEGER + ) +""" + + +class AssetIndex: + """ + Persistent index of ThePosterDB users' uploads (SQLite, one file in the config + directory). Writers are the scrape thread; readers may run on other threads, so the + connection is short-lived per call and the database runs in WAL mode. + """ + + def __init__(self, path: str = ASSET_INDEX_PATH) -> None: + self.path = path + try: + self._ensure_schema() + except sqlite3.DatabaseError as e: + # debug_me lives in utils.notifications, which imports the services package - import + # it lazily here to avoid a start-up import cycle when the scraper pulls in the index. + from utils.notifications import debug_me + # The file exists but is not a valid SQLite database (corruption, a truncated + # write on a flaky filesystem, ...). Preserve it for inspection and start clean: + # the next crawl repopulates it, nothing is lost that a scrape can't rebuild. + corrupt = f"{self.path}.corrupt-{int(time.time())}" + try: + os.rename(self.path, corrupt) + except OSError: + debug_me(f"Asset index at '{self.path}' is unreadable ({e}) and could not be " + f"moved aside; giving up on the cache.", "AssetIndex") + raise + debug_me(f"Asset index at '{self.path}' was unreadable ({e}); moved it to " + f"'{corrupt}' and started a fresh index.", "AssetIndex") + self._ensure_schema() + + def _connect(self) -> sqlite3.Connection: + conn = sqlite3.connect(self.path, timeout=30) + conn.execute("PRAGMA journal_mode=WAL") + conn.execute("PRAGMA busy_timeout=30000") + conn.row_factory = sqlite3.Row + return conn + + def _ensure_schema(self) -> None: + conn = self._connect() + try: + conn.execute("PRAGMA user_version = 1") + conn.execute(_CREATE_ASSETS) + conn.execute(_CREATE_ASSETS_INDEX) + conn.execute(_CREATE_CRAWLS) + conn.commit() + finally: + conn.close() + + def known_ids(self, user_key: str) -> Set[int]: + """Every asset id on record for the user, tombstoned ones included, so the stop rule + stays conservative (a re-listed deleted asset still counts as already known).""" + conn = self._connect() + try: + rows = conn.execute( + "SELECT asset_id FROM user_assets WHERE user_key = ?", (user_key,) + ).fetchall() + finally: + conn.close() + return {row["asset_id"] for row in rows} + + def record(self, user_key: str, assets: List[dict]) -> int: + """Upsert a batch of raw (pre-filter) assets for a user, returning how many were not + already indexed - the count the upload-count ledger checks against ThePosterDB.""" + if not assets: + return 0 + now = _now() + known = self.known_ids(user_key) + rows = [] + new_count = 0 + for asset in assets: + try: + asset_id = int(asset["id"]) + except (KeyError, TypeError, ValueError): + continue + if asset_id not in known: + new_count += 1 + known.add(asset_id) + title = asset.get("title") + season = asset.get("season") + rows.append(( + user_key, asset_id, title, normalize_title(title) if title else "", + asset.get("year"), season if isinstance(season, int) else None, + asset.get("media_type"), asset.get("author"), asset.get("url"), now, now, + )) + if not rows: + return 0 + conn = self._connect() + try: + conn.executemany( + """ + INSERT INTO user_assets + (user_key, asset_id, title, title_key, year, season, media_type, + author, url, first_seen, last_seen) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(user_key, asset_id) DO UPDATE SET + title=excluded.title, title_key=excluded.title_key, year=excluded.year, + season=excluded.season, media_type=excluded.media_type, + author=excluded.author, url=excluded.url, last_seen=excluded.last_seen, + missing_since=NULL + """, + rows, + ) + conn.commit() + finally: + conn.close() + return new_count + + def assets_for_user(self, user_key: str) -> List[sqlite3.Row]: + """Live (non-tombstoned, known-media-type) assets for the user, newest first - the + order a full crawl produces.""" + conn = self._connect() + try: + return conn.execute( + """ + SELECT * FROM user_assets + WHERE user_key = ? AND missing_since IS NULL AND media_type != 'unknown' + ORDER BY asset_id DESC + """, + (user_key,), + ).fetchall() + finally: + conn.close() + + def reconcile(self, user_key: str, seen_ids: Set[int], crawl_started_at: str) -> int: + """After a clean full crawl, tombstone assets that weren't seen (deleted from the + user's uploads), sparing any row inserted since the crawl began so an overlapping + scrape's fresh uploads are never tombstoned. Returns the number tombstoned.""" + conn = self._connect() + try: + conn.execute("CREATE TEMP TABLE IF NOT EXISTS seen (asset_id INTEGER PRIMARY KEY)") + conn.execute("DELETE FROM seen") + conn.executemany("INSERT OR IGNORE INTO seen (asset_id) VALUES (?)", + [(asset_id,) for asset_id in seen_ids]) + cursor = conn.execute( + """ + UPDATE user_assets SET missing_since = ? + WHERE user_key = ? AND missing_since IS NULL + AND asset_id NOT IN (SELECT asset_id FROM seen) + AND last_seen < ? + """, + (_now(), user_key, crawl_started_at), + ) + conn.commit() + return cursor.rowcount + finally: + conn.close() + + def crawl_state(self, user_key: str) -> Optional[sqlite3.Row]: + """The user's crawl bookkeeping row (last full/any crawl, last upload count), or None.""" + conn = self._connect() + try: + return conn.execute( + "SELECT * FROM user_crawls WHERE user_key = ?", (user_key,) + ).fetchone() + finally: + conn.close() + + def record_crawl(self, user_key: str, full: bool, seen_count: int) -> None: + """Update the crawl ledger. last_full_crawl only advances on a full crawl.""" + now = _now() + conn = self._connect() + try: + existing = conn.execute( + "SELECT last_full_crawl FROM user_crawls WHERE user_key = ?", (user_key,) + ).fetchone() + last_full = now if full else (existing["last_full_crawl"] if existing else None) + conn.execute( + """ + INSERT INTO user_crawls (user_key, last_full_crawl, last_crawl, last_seen_count) + VALUES (?, ?, ?, ?) + ON CONFLICT(user_key) DO UPDATE SET + last_full_crawl=excluded.last_full_crawl, + last_crawl=excluded.last_crawl, + last_seen_count=excluded.last_seen_count + """, + (user_key, last_full, now, seen_count), + ) + conn.commit() + finally: + conn.close() + + +def page_is_fully_known(page_ids: Set[int], known_ids: Set[int]) -> bool: + """Incremental stop rule: a page halts the crawl only if it had assets and every one was + already in the index. A page with any new asset (even mixed with known ones) keeps the + crawl going, which is why the rule is page- rather than item-granular - ThePosterDB + pages are newest-first by upload batch but not strictly by id within a batch.""" + return bool(page_ids) and page_ids <= known_ids + + +def full_crawl_due(last_full_crawl: Optional[str], refresh_days: int) -> bool: + """True when the user has never had a full crawl or the last one is at least refresh_days + old - the next scrape then re-crawls every page to catch edits and deletions.""" + if not last_full_crawl: + return True + try: + last = datetime.fromisoformat(last_full_crawl) + except (TypeError, ValueError): + return True + return (datetime.now(timezone.utc) - last).total_seconds() >= refresh_days * 86400 diff --git a/static/web_interface.js b/static/web_interface.js index 7ad2188..9d75186 100644 --- a/static/web_interface.js +++ b/static/web_interface.js @@ -631,6 +631,8 @@ function saveConfig() { // Checkbox for skipping artwork with locked fields in Plex save_config.skip_locked_artwork = document.getElementById("skip_locked_artwork").checked; toggleSkipLockedCheckbox(); + // Checkbox for caching ThePosterDB user scrapes + save_config.cache_user_scrapes = document.getElementById("cache_user_scrapes").checked; // Get selected mediux filters save_config.mediux_filters = Array.from(document.querySelectorAll('[id^="m_filter-"]:checked')) @@ -709,6 +711,7 @@ function loadConfig() { document.getElementById("auto_manage_bulk_files").checked = data.config.auto_manage_bulk_files; document.getElementById("reset_overlay").checked = data.config.reset_overlay; document.getElementById("skip_locked_artwork").checked = data.config.skip_locked_artwork; + document.getElementById("cache_user_scrapes").checked = data.config.cache_user_scrapes; document.getElementById("option-add-to-bulk").checked = data.config.auto_manage_bulk_files; document.getElementById("apprise_urls").value = data.config.apprise_urls.join(", "); diff --git a/templates/web_interface.html b/templates/web_interface.html index b8ca8ac..04c49ae 100644 --- a/templates/web_interface.html +++ b/templates/web_interface.html @@ -396,6 +396,10 @@
 Additional settings
+
+ + +
diff --git a/tests/test_asset_index.py b/tests/test_asset_index.py new file mode 100644 index 0000000..b9dba20 --- /dev/null +++ b/tests/test_asset_index.py @@ -0,0 +1,139 @@ +"""Unit tests for the persistent ThePosterDB user-uploads index (services/asset_index.py).""" + +import glob +from datetime import datetime, timedelta, timezone + +import pytest + +from services.asset_index import ( + AssetIndex, + normalize_title, + page_is_fully_known, + full_crawl_due, +) + +FAR_FUTURE = "2999-01-01T00:00:00+00:00" +FAR_PAST = "1999-01-01T00:00:00+00:00" + + +def _asset(asset_id, title="A Movie", year=2020, season=None, + media_type="movie_poster", author="artist", url=None): + return { + "id": asset_id, + "title": title, + "year": year, + "season": season, + "media_type": media_type, + "author": author, + "url": url or f"https://theposterdb.com/api/assets/{asset_id}", + } + + +@pytest.fixture +def index(tmp_path): + return AssetIndex(str(tmp_path / "asset_index.db")) + + +@pytest.mark.unit +def test_schema_creation_idempotent(tmp_path): + path = str(tmp_path / "asset_index.db") + AssetIndex(path) + AssetIndex(path).record("user", [_asset(1)]) # second init keeps the data, no error + assert AssetIndex(path).known_ids("user") == {1} + + +@pytest.mark.unit +def test_record_upsert_and_dedupe(index): + assert index.record("user", [_asset(1, title="First")]) == 1 + assert index.record("user", [_asset(1, title="First")]) == 0 # duplicate id collapses + assert index.known_ids("user") == {1} + # Tombstone then re-record: mutable fields update and the tombstone clears + index.reconcile("user", seen_ids=set(), crawl_started_at=FAR_FUTURE) + index.record("user", [_asset(1, title="Renamed", url="https://x/9")]) + rows = index.assets_for_user("user") + assert len(rows) == 1 + assert rows[0]["title"] == "Renamed" + assert rows[0]["url"] == "https://x/9" + assert rows[0]["missing_since"] is None + + +@pytest.mark.unit +def test_known_ids_includes_tombstoned(index): + index.record("user", [_asset(1), _asset(2)]) + index.reconcile("user", seen_ids={1}, crawl_started_at=FAR_FUTURE) + assert index.known_ids("user") == {1, 2} # 2 tombstoned but still "known" + + +@pytest.mark.unit +def test_assets_for_user_filters_and_order(index): + index.record("user", [ + _asset(1, media_type="movie_poster"), + _asset(3, media_type="movie_poster"), + _asset(2, media_type="unknown"), + ]) + index.record("user", [_asset(9, media_type="movie_poster")]) + index.reconcile("user", seen_ids={1, 3, 2}, crawl_started_at=FAR_FUTURE) + ids = [r["asset_id"] for r in index.assets_for_user("user")] + assert ids == [3, 1] # 9 tombstoned, 2 unknown-type; remainder newest-first + + +@pytest.mark.unit +def test_reconcile_tombstones_and_spares(index): + index.record("user", [_asset(1), _asset(2), _asset(3)]) + assert index.reconcile("user", seen_ids={1}, crawl_started_at=FAR_FUTURE) == 2 + assert {r["asset_id"] for r in index.assets_for_user("user")} == {1} + # A row inserted after the crawl started is spared even when unseen + index.record("user", [_asset(4)]) + assert index.reconcile("user", seen_ids=set(), crawl_started_at=FAR_PAST) == 0 + assert 4 in {r["asset_id"] for r in index.assets_for_user("user")} + + +@pytest.mark.unit +def test_crawl_state_roundtrip(index): + assert index.crawl_state("user") is None + index.record_crawl("user", full=True, seen_count=42) + state = index.crawl_state("user") + assert state["last_seen_count"] == 42 + full_ts = state["last_full_crawl"] + assert full_ts is not None + index.record_crawl("user", full=False, seen_count=50) # incremental keeps last_full_crawl + state = index.crawl_state("user") + assert state["last_seen_count"] == 50 + assert state["last_full_crawl"] == full_ts + + +@pytest.mark.unit +def test_page_is_fully_known(): + assert page_is_fully_known(set(), {1, 2}) is False # empty page never stops + assert page_is_fully_known({1, 9}, {1, 2}) is False # mixed page keeps crawling + assert page_is_fully_known({1, 2}, {1, 2, 3}) is True # fully known -> stop + + +@pytest.mark.unit +def test_full_crawl_due(): + now = datetime.now(timezone.utc) + assert full_crawl_due(None, 7) is True + assert full_crawl_due(now.isoformat(), 7) is False + stale = (now - timedelta(days=8)).isoformat() + assert full_crawl_due(stale, 7) is True + assert full_crawl_due(stale, 30) is False # 8 days old, 30-day window -> not due yet + assert full_crawl_due("not-a-date", 7) is True + + +@pytest.mark.unit +def test_normalize_title(): + assert normalize_title("Mission: Impossible") == normalize_title("Mission - Impossible") + assert normalize_title("Léon") == "leon" + assert normalize_title("Tom & Jerry") == "tom and jerry" + assert normalize_title(" Mad Max 2! ") == "mad max 2" + + +@pytest.mark.unit +def test_corrupt_file_self_heal(tmp_path): + path = str(tmp_path / "asset_index.db") + with open(path, "wb") as f: + f.write(b"this is not a sqlite database, just garbage bytes") + index = AssetIndex(path) # must self-heal, not raise + assert glob.glob(path + ".corrupt-*") + index.record("user", [_asset(1)]) + assert index.known_ids("user") == {1} diff --git a/tests/test_user_cache_crawl.py b/tests/test_user_cache_crawl.py new file mode 100644 index 0000000..cb29ee6 --- /dev/null +++ b/tests/test_user_cache_crawl.py @@ -0,0 +1,185 @@ +"""Tests for the cached user-crawl: stopping at the end of the uploads, not tombstoning a +catalogue from a crawl that was cut short, and reporting new assets added to the cache.""" + +import os + +import pytest + +from models.callbacks import ProcessingCallbacks +from models.options import Options +from services.asset_index import AssetIndex +import scrapers.theposterdb_scraper as tpdb +from scrapers.theposterdb_scraper import ThePosterDBScraper + + +@pytest.fixture(autouse=True) +def _isolate_cwd(tmp_path, monkeypatch): + os.makedirs(tmp_path / "config", exist_ok=True) + monkeypatch.chdir(tmp_path) + + +def _asset(asset_id): + return { + "id": asset_id, + "title": f"Film {asset_id}", + "year": 2020, + "season": None, + "media_type": "movie_poster", + "author": "someone", + "url": f"https://theposterdb.com/api/assets/{asset_id}", + } + + +def _scraper(counters=None): + scraper = ThePosterDBScraper("https://theposterdb.com/user/someone", + ProcessingCallbacks(**(counters or {}))) + scraper.set_options(Options()) + scraper.author = "someone" + return scraper + + +def _fill(pages): + """Return a fake scrape_user_page that fills the catalog from `pages` (dict page->id list).""" + fetched = [] + + def fake(page, catalog=None): + fetched.append(page) + for asset_id in pages.get(page, []): + catalog.append(_asset(asset_id)) + return True + + fake.fetched = fetched + return fake + + +# --- A2: stop at the end of the uploads, don't poison a clean crawl ---------------------------- + +def test_crawl_stops_at_empty_page_without_marking_it_unclean(tmp_path, monkeypatch): + scraper = _scraper() + scraper.user_uploads = 120 + scraper.user_pages = 5 # counter overshoots: only 2 pages hold assets + index = AssetIndex(str(tmp_path / "idx.db")) + fake = _fill({0: range(1000, 1024), 1: range(2000, 2024)}) + monkeypatch.setattr(scraper, "scrape_user_page", fake) + + new_rows, seen_ids, clean = scraper._crawl_user_pages(index, "someone", full=True) + + assert fake.fetched == [0, 1, 2] # stopped AT the first empty page + assert clean is True # a trailing empty page is not a failure + assert len(seen_ids) == 48 + + +def test_a_failed_page_still_marks_the_crawl_unclean(tmp_path, monkeypatch): + scraper = _scraper() + scraper.user_uploads = 72 + scraper.user_pages = 3 + index = AssetIndex(str(tmp_path / "idx.db")) + + def fake(page, catalog=None): + if page == 1: + return False # a genuine fetch failure, not an empty page + for asset_id in range(page * 1000, page * 1000 + 24): + catalog.append(_asset(asset_id)) + return True + + monkeypatch.setattr(scraper, "scrape_user_page", fake) + new_rows, seen_ids, clean = scraper._crawl_user_pages(index, "someone", full=True) + + assert clean is False # a real failure must still block reconcile + + +# --- reconcile guard: never tombstone from a crawl that was cut short -------------------------- + +def _seed(tmp_path, n): + index = AssetIndex(str(tmp_path / "idx.db")) + index.record("someone", [_asset(i) for i in range(n)]) + index.record_crawl("someone", True, n) + return index + + +def _run_full_crawl(scraper, tmp_path, monkeypatch, page_zero_ids): + scraper.options.no_cache = True # force a full crawl + monkeypatch.setattr(tpdb, "AssetIndex", lambda: AssetIndex(str(tmp_path / "idx.db"))) + monkeypatch.setattr(scraper, "_hydrate_from_cache", lambda *a, **k: None) + fake = _fill({0: page_zero_ids}) # page 1 is empty -> crawl stops after it + monkeypatch.setattr(scraper, "scrape_user_page", fake) + scraper._scrape_user_cached() + + +def test_short_full_crawl_does_not_tombstone(tmp_path, monkeypatch): + # Index has 100 assets. A full crawl that reaches only 10 of the reported 100 (a broken page, + # not a real shrink) must NOT tombstone the other 90. + index = _seed(tmp_path, 100) + scraper = _scraper() + scraper.user_uploads = 100 + scraper.user_pages = 50 + + _run_full_crawl(scraper, tmp_path, monkeypatch, page_zero_ids=range(0, 10)) + + live = index.assets_for_user("someone") + assert len(live) == 100 # coverage 10% < 90% -> nothing tombstoned + + +def test_full_crawl_with_good_coverage_tombstones_deleted(tmp_path, monkeypatch): + # Index has 100. The user really deleted 2, so the crawl sees 98 of a reported 98. Coverage is + # high, so the 2 that vanished are tombstoned as normal. + index = _seed(tmp_path, 100) + scraper = _scraper() + scraper.user_uploads = 98 + scraper.user_pages = 50 + + _run_full_crawl(scraper, tmp_path, monkeypatch, page_zero_ids=range(0, 98)) + + live = index.assets_for_user("someone") + assert len(live) == 98 # coverage 100% >= 90% -> 98,99 tombstoned + + +# --- the "new in cache" counter ---------------------------------------------------------------- + +def test_new_assets_are_reported_through_the_cached_counter(tmp_path, monkeypatch): + cached = [0] + scraper = _scraper({"cached_counter": cached}) + scraper.user_uploads = 48 + scraper.user_pages = 5 + monkeypatch.setattr(tpdb, "AssetIndex", lambda: AssetIndex(str(tmp_path / "idx.db"))) + monkeypatch.setattr(scraper, "_hydrate_from_cache", lambda *a, **k: None) + fake = _fill({0: range(1000, 1024), 1: range(2000, 2024)}) + monkeypatch.setattr(scraper, "scrape_user_page", fake) + + scraper._scrape_user_cached() + + assert cached[0] == 48 # all 48 were new to the cache + + +# --- regression: hydrate must key artwork as file_type, exactly as get_posters does ------------ + +def test_hydrate_from_cache_uses_the_processor_file_type_key(tmp_path): + """_hydrate_from_cache rebuilds the artwork lists that feed the upload processor, so its dicts + must carry 'file_type' (what get_posters writes and the processor reads), not 'type'. With + 'type', ARTWORK_ID_MAP.get(...) is None and the uploader crashes on None + md5 - the same + fault that broke the import webhook, latent here until a cached crawl meets an unlocked + in-library item.""" + from core.constants import ARTWORK_ID_MAP + + index = AssetIndex(str(tmp_path / "idx.db")) + index.record("someone", [ + {"id": 1, "title": "Dune", "year": 2021, "season": None, + "media_type": "movie_poster", "author": "someone", + "url": "https://theposterdb.com/api/assets/1"}, + {"id": 2, "title": "Severance", "year": 2022, "season": None, + "media_type": "show_cover", "author": "someone", + "url": "https://theposterdb.com/api/assets/2"}, + {"id": 3, "title": "Severance", "year": 2022, "season": 1, + "media_type": "season_cover", "author": "someone", + "url": "https://theposterdb.com/api/assets/3"}, + ]) + + scraper = _scraper() + scraper.config.tpdb_filters = ["movie_poster", "show_cover", "season_cover", "collection_poster"] + scraper._hydrate_from_cache(index, "someone") + + produced = scraper.movie_artwork + scraper.tv_artwork + assert produced, "hydrate produced no artwork - fixture/filter setup is wrong" + for artwork in produced: + assert "file_type" in artwork, f"hydrated artwork missing file_type: {artwork}" + assert ARTWORK_ID_MAP.get(artwork["file_type"]) is not None diff --git a/utils/utils.py b/utils/utils.py index 81407bd..a4a266d 100644 --- a/utils/utils.py +++ b/utils/utils.py @@ -215,7 +215,7 @@ def parse_url_and_options(line): year == None options.year = year # If it's one of these flags it shouldn't have any arguments - elif flag in ["add-posters", "add-sets", "add-to-bulk", "force", "kometa", "stage", "temp", "skip-locked"]: + elif flag in ["add-posters", "add-sets", "add-to-bulk", "force", "kometa", "stage", "temp", "skip-locked", "no-cache"]: inv_flags.append(f"--{flag} has too many argumens") # If we ge to this point it's not a valid flag else: @@ -224,7 +224,7 @@ def parse_url_and_options(line): else: if flag in ["filters", "exclude", "year"]: inv_flags.append(f"--{flag} needs at least one argument") - elif flag not in ["add-posters", "add-sets" ,"add-to-bulk" ,"force", "kometa", "stage", "temp", "skip-locked"]: + elif flag not in ["add-posters", "add-sets" ,"add-to-bulk" ,"force", "kometa", "stage", "temp", "skip-locked", "no-cache"]: inv_flags.append(f"--{flag} is not a valid flag") else: setattr(options, flag.replace("-","_"), True) # Convert add-posters into add_posters as the Options class expects