Back off retrying cover URLs that a CDN just rejected with 403 - #153
Open
jordanfelle wants to merge 1 commit into
Open
jordanfelle wants to merge 1 commit into
jordanfelle wants to merge 1 commit into
Conversation
Every book/author cover download that fails with 403 Forbidden was retried from scratch on the next rename, refresh, import, or RSS sync that touched the same book, with no backoff. When several books share a cover URL a CDN is currently blocking, this turns into an unthrottled retry storm against that host, which starves ManagedHttpDispatcher's 12-connections-per-host pool and can stall unrelated bulk commands (observed: RenameAuthorCommand hanging with zero progress for 15+ minutes on a ~1600-author library, cleared instantly by cancelling it). Adds a 6-hour negative cache keyed by URL in GetImageResponse, the single chokepoint both the author-cover and book-cover paths already share, so a known-403 URL is skipped without a network call until the backoff expires. Fixes Chaptarr#152 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ED3rga2EPdAvo3ChtnJCtH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MediaCoverServicehad no negative-caching/backoff for cover URLs a CDN has already rejected with403 Forbidden. Every rename, refresh, import, or RSS sync that touches a book/author re-attempts the exact same known-bad URL from scratch.ManagedHttpDispatchercaps outbound connections atmaxConnectionsPerServer: 12, so the retry rate can exceed the pool and starve it.RenameAuthorCommandcovering ~1600 authors hung with zero progress for 15+ minutes (messagefield frozen on one author, no new log lines) while the app itself stayed fully responsive. Diagnosing via/proc/<pid>/net/tcpon the host showed several leakedCLOSE_WAITsockets to another local service at the moment of the hang. Cancelling the stuck command viaDELETE /api/v1/command/{id}cleared theCLOSE_WAITsockets instantly and let every other queued command resume.Fix
Adds a 6-hour in-memory negative cache (
ConcurrentDictionary<string, DateTime>) keyed by URL insideGetImageResponse, the single chokepoint both the author-cover (DownloadCover) and book-cover (DownloadBookCover) paths already funnel through. A URL that just 403'd is skipped with a syntheticHttpException(fast-fail, no network call) for the backoff window instead of being retried by every subsequent book/author referencing it — existingcatch (HttpException e) when (... Forbidden)call sites are untouched and keep working exactly as before.This doesn't fix the CDN-side blocking itself (out of Chaptarr's control), but it stops one blocked URL from being able to stall unrelated bulk operations across the whole library.
Test plan
dotnet buildsucceeds with no new warnings/errors./proc/<pid>/net/tcp(leakedCLOSE_WAITto a same-host service) and log correlation (repeatedBook/Author cover download blocked (403 Forbidden)for the same handful of URLs across different book IDs), and confirmed cancelling the stuck command immediately cleared the connection pool and resumed the queue — consistent with the fix's target.Fixes #152
🤖 Generated with Claude Code
https://claude.ai/code/session_01ED3rga2EPdAvo3ChtnJCtH