diff --git a/docs/plans/2026-07-28 - sonarr-radarr-integration.md b/docs/plans/2026-07-28 - sonarr-radarr-integration.md new file mode 100644 index 0000000..c4286f2 --- /dev/null +++ b/docs/plans/2026-07-28 - sonarr-radarr-integration.md @@ -0,0 +1,173 @@ +# 2026-07-28 — Making the Sonarr/Radarr integration actually work (#6 + matching) + +**Status:** planned · **Milestone:** Foundation / ARD and KiKA indexer +**Builds on:** #4 (instances + outbound client), [DR-011](../architecture/DR-011-search-driven-indexing.md) + +"Get the Sonarr/Radarr integration done" is more than #6. #6 gives us a crawl work-list; it does **not** +make Sonarr able to *match* what we return. Reviewing the current surface, matching is the part that would +leave the integration looking broken even with #6 finished. + +## What is actually missing + +### 1. We never tell Sonarr which series a release belongs to + +`NewznabXml.Feed` emits only two attributes per item: + +```xml + + +``` + +No `tvdbid`, no `season`, no `episode`. So Sonarr has to identify the series **purely by parsing our release +title**, against a Mediathek catalog whose titles are famously inconsistent — the problem both MediathekArr +and RundfunkArr spend most of their effort on. A `tvdbid` attribute short-circuits all of it: Sonarr trusts +the id over the string. + +`Show.TvdbId` already exists on the entity and is **never set by anything**. + +### 2. We do not accept `tvdbid` as a search parameter + +The endpoint binds `q`, `season`, `ep` only, and `caps` advertises `supportedParams="q,season,ep"`. Sonarr +reads caps and adapts, so today it falls back to title searches — which works, but throws away the reliable +path. Advertising and honouring `tvdbid` lets Sonarr ask us the unambiguous question. + +### 3. Nothing links an `*arr` series to a broadcaster show + +`CrawlTarget(ProviderKey, ShowQuery)` is a broadcaster + a title substring. A Sonarr series is a title, a +`tvdbId`, a year, and a set of `alternateTitles`. Bridging the two **is** the integration, and it is the hard +part — not the HTTP call. + +## Research: why the names differ, and whether there is a trick (2026-07-28) + +**Why they differ at all.** TVDB is community-maintained and its *primary* title is often the English or +canonical form — but TVDB v4 stores per-language **translations and aliases**, and its **search endpoint +indexes them**. The Mediathek uses the broadcaster's own on-air branding. So the mismatch is real, yet TVDB +usually *does* hold the German name; it just is not the primary title. That reframes the problem: do not +compare primary titles, query TVDB's search with the German title and let it match its own aliases. + +**The trick that exists: Wikidata.** `P4835` is "TheTVDB series ID", and German shows carry it. A keyless +SPARQL query maps a German title (label *or* alias) to a TVDB id, with the broadcaster attached via `P449` +for disambiguation. Measured live: + +| Broadcaster | Series with a TVDB id | Coverage | +|---|---|---| +| ZDF | 359 / 2467 | **14.6%** | +| Das Erste | 270 / 1770 | **15.3%** | +| KiKA | 49 / 101 | **48.5%** | +| ZDFneo | 42 / 90 | **46.7%** | +| **Total** | **720 / 4428** | **16.3%** | + +So it is a genuine trick but a *partial* one — free, no API key, no scraping, and notably good for KiKA +(cartoons are well covered, which is exactly the geo-restricted content we already handle). It cannot be the +whole answer at 16%. + +Two concrete gotchas the data surfaced: + +- **Disambiguation is required, and Wikidata provides it.** *Die Biene Maja* returns **two** TVDB ids — + 73518 (1975, TV Asahi) and 266275 (2013, ZDF). Without the broadcaster we would pick wrong half the time. +- **ARD is a federation, so `P449` is often the regional member, not "Das Erste".** *extra 3* resolves to + tvdb 255986 with network **Norddeutscher Rundfunk**. Filtering on "Das Erste" alone would miss it. Accept + the ARD member set (NDR, WDR, BR, SWR, MDR, HR, RBB, SR). +- Case matters: `extra 3` missed, `Extra 3` matched — normalise before querying. +- Genuinely ambiguous titles (*Panorama*) match nothing useful and belong in the manual tail. + +**Conclusion: we do not have to mimic MediathekArr/RundfunkArr wholesale, but we cannot avoid a manual tail +either.** The layered design below reduces the manual work rather than eliminating it — which is the honest +version of "is there a neat trick". + +### Layered matching, cheapest first + +1. **TVDB search by the German Mediathek title** — primary. Their search indexes translations and aliases, + which is precisely the "names differ" problem. Uses **`TvdbClient`** (below). +2. **Wikidata corroboration** — keyless, and the way to disambiguate multiple TVDB candidates by broadcaster. + Also usable to pre-seed a mapping table in bulk, offline. +3. **Manual override in the UI** — for the tail. RundfunkArr's curated `shows.json` is the evidence that this + step is unavoidable; the goal is only to make it small. + +### TVDB client + +Use **`TvdbClient`** (`Chrison-dev/TvdbApi`, NuGet `TvdbClient` 4.7.12) rather than hand-rolling — it is +maintained in-house. Packages: `TvdbClient`, `TvdbClient.Models` (`Tvdb.Models`), `TvdbClient.Abstractions`. + +```csharp +builder.Configuration.AddTvdbClient(); +builder.Services.AddTvdbClient(builder.Configuration); + +var series = sp.GetRequiredService(); +var record = (await series.SeriesGetAsync(121361)).Data; +``` + +Configured via `TvdbConfiguration:BaseUrl / ApiKey / Pin`; login and auth headers are handled for us. +**Needs an API key** (subscriber PIN optional) — the one external dependency this work adds, and the reason +step 2 matters: Wikidata keeps basic mapping working for operators without a TVDB key. + +## Shape (three reviewable PRs) + +### PR 1 — Emit and accept TVDB ids · `feat/arr-tvdb-matching` + +Independent of #6 and the highest value per line changed. + +- `Release` gains `TvdbId`; `NewznabXml` emits `newznab:attr name="tvdbid"`, plus `season` and `episode` + where known. +- The endpoint binds `tvdbid` (and `rid`/`imdbid` are explicitly ignored for now); `caps` advertises + `supportedParams="q,tvdbid,season,ep"`. +- `SearchReleasesQuery` gains `TvdbId`; a search by id resolves to the mapped show and filters on it, + falling back to `q` when the id is unknown to us. +- `Show.TvdbId` is finally populated — by PR 2, until then it stays null and behaviour is unchanged. + +### PR 2 — Pull monitored series (#6) · `feat/arr-monitored-series` — **DEFERRED** + +On the back burner by decision 2026-07-28. Not needed for matching to work, and per DR-011 nothing may +depend on a configured instance anyway. + + +- `IArrClient` gains `GetMonitoredSeriesAsync` — Sonarr `GET /api/v3/series` filtered to `monitored: true`; + Radarr `GET /api/v3/movie`. Returns title, `tvdbId`/`tmdbId`, year, alternate titles. +- A background service polls each **enabled** instance on an interval and merges results into the standing + crawl list, **never replacing** it: a hand-configured `Crawl:Targets` entry must not vanish because an + instance went offline mid-poll. +- Failures degrade quietly per instance — log, keep going, never empty the list. +- Opt-in (`Crawl:PreWarmFromArrInstances`, default false) per DR-011: nothing may *depend* on an instance + being configured. + +### PR 3 — Show mapping · `feat/arr-show-mapping` + +The bridge, and where the effort really goes. + +- A `ShowMapping` record: `TvdbId` ↔ (`ProviderKey`, broadcaster show id/title), with provenance + (auto-matched vs. operator-confirmed). +- Auto-match attempt: normalise both sides (lower-case, strip punctuation and years, fold umlauts — + `ä→ae` etc., since Mediathek titles and TVDB disagree constantly), then compare title and alternate + titles. +- **Auto-matching will not be enough.** RundfunkArr resolves this with human-curated `shows.json` + + per-show `rulesets.json`, and that is a fair signal. So: surface unmatched monitored series in the UI and + let the operator pick the broadcaster show manually. Manual mappings win over auto ones. +- Once mapped, crawled episodes inherit `Show.TvdbId`, which is what makes PR 1's attribute meaningful. + +## Decisions + +**PR 1 goes first, and is useful alone.** It improves matching for the shows we already crawl without +needing any `*arr` instance configured — consistent with DR-011's rule that nothing may depend on one. + +**Radarr is deliberately second-class for now.** It uses `tmdbId`, not `tvdbId`, and the movie catalog on +ARD/ZDF is thin (MediathekArr's README says the same: *"You can find a few movies via interactive search, +but not a lot"*). Model the field, wire the fetch, do not over-invest in movie matching. + +**Do not scope-creep into a fuzzy-matching engine.** Exact-after-normalisation plus a manual override +covers the realistic case. Anything cleverer earns its place only once we can see real failures against a +real library. + +## Testing + +- Unit: TVDB attribute emission; `tvdbid` search resolving and falling back; title normalisation + (umlauts, punctuation, years); merge-not-replace semantics for the crawl list; an offline instance not + emptying the list. +- `ArrHttpClient` against a stubbed handler for the series/movie endpoints, including a monitored/unmonitored + mix and pagination if Sonarr paginates. +- **Live, against a real instance** (Christian has Sonarr and Radarr): `[Trait("Category","Live")]` reading + `KRAUTWATCH_TEST_SONARR_URL` / `KRAUTWATCH_TEST_SONARR_APIKEY`, skipping when unset — the same convention + `KRAUTWATCH_TEST_PROXY` already uses for the geo-proxy live test. Credentials stay in the operator's + environment and never enter the repo. +- End-to-end worth doing by hand once: point real Sonarr at Krautwatch as a Newznab indexer, trigger an + interactive search for a monitored German public-TV show, and confirm Sonarr both finds and *matches* the + release. diff --git a/src/Application/Indexing/Release.cs b/src/Application/Indexing/Release.cs index 72bf090..da58256 100644 --- a/src/Application/Indexing/Release.cs +++ b/src/Application/Indexing/Release.cs @@ -21,6 +21,13 @@ public record Release public required string DownloadToken { get; init; } public required string ShowTitle { get; init; } public required SeriesType SeriesType { get; init; } + + /// + /// The show's TVDB id, when we have mapped one. Emitted as a newznab:attr so Sonarr can identify + /// the series by id instead of parsing our title string — which is the difference between reliable + /// matching and hoping a Mediathek title parses. + /// + public int? TvdbId { get; init; } public int? Season { get; init; } public int? Episode { get; init; } public DateTimeOffset PublishDate { get; init; } diff --git a/src/Application/Indexing/ReleaseMapper.cs b/src/Application/Indexing/ReleaseMapper.cs index 50a9728..099eb32 100644 --- a/src/Application/Indexing/ReleaseMapper.cs +++ b/src/Application/Indexing/ReleaseMapper.cs @@ -16,6 +16,7 @@ public static partial class ReleaseMapper [MapProperty(nameof(Episode.Id), nameof(Release.DownloadToken))] [MapProperty([nameof(Episode.Show), nameof(Show.Title)], [nameof(Release.ShowTitle)])] [MapProperty([nameof(Episode.Show), nameof(Show.SeriesType)], [nameof(Release.SeriesType)])] + [MapProperty([nameof(Episode.Show), nameof(Show.TvdbId)], [nameof(Release.TvdbId)])] [MapProperty(nameof(Episode.SeasonNumber), nameof(Release.Season))] [MapProperty(nameof(Episode.EpisodeNumber), nameof(Release.Episode))] [MapProperty(nameof(Episode.BroadcastDate), nameof(Release.PublishDate))] diff --git a/src/Application/Indexing/SearchReleases.cs b/src/Application/Indexing/SearchReleases.cs index 552da22..ca8db6a 100644 --- a/src/Application/Indexing/SearchReleases.cs +++ b/src/Application/Indexing/SearchReleases.cs @@ -7,7 +7,12 @@ namespace Krautwatch.Application.Indexing; /// A Newznab search (t=search / t=tvsearch) or, when is empty, the RSS feed of the /// most recent releases. Optional season/episode narrow a Standard-series query. /// -public record SearchReleasesQuery(string? Q = null, int? Season = null, int? Episode = null, int Limit = 100); +public record SearchReleasesQuery( + string? Q = null, + int? Season = null, + int? Episode = null, + int Limit = 100, + int? TvdbId = null); /// /// Serves Newznab results from the catalog, resolving against the broadcasters on demand when a search term @@ -23,6 +28,19 @@ public async Task> HandleAsync(SearchReleasesQuery query, { var limit = Math.Clamp(query.Limit, 1, 500); + // A TVDB id is the unambiguous question, so answer it directly when Sonarr asks it. + if (query.TvdbId is not null) + { + var byId = Project(await episodes.GetByTvdbIdAsync(query.TvdbId.Value, ct), query, limit); + if (byId.Count > 0) + return byId; + + // The id is one we have not mapped yet. Fall through to the title search when Sonarr also sent + // one, rather than reporting "nothing exists" — an unmapped show is our gap, not an absent show. + if (string.IsNullOrWhiteSpace(query.Q)) + return byId; + } + // RSS (no query) is never resolved: RSS-Sync polls constantly with no particular target, so // resolving here would mean crawling on a timer for nothing specific. Per DR-011 it serves the // standing crawl list. diff --git a/src/Domain/Interfaces/IRepositories.cs b/src/Domain/Interfaces/IRepositories.cs index 0566553..7b0d9ac 100644 --- a/src/Domain/Interfaces/IRepositories.cs +++ b/src/Domain/Interfaces/IRepositories.cs @@ -39,6 +39,12 @@ public interface IEpisodeRepository Task> SearchAsync(string query, CancellationToken ct = default); // Newest episodes first — the Newznab RSS feed (no query) reads from here. Task> GetRecentAsync(int limit, CancellationToken ct = default); + + /// + /// Episodes of shows mapped to a TVDB id. This is the reliable lookup: Sonarr asks by id, so we answer + /// by id rather than re-parsing titles. + /// + Task> GetByTvdbIdAsync(int tvdbId, CancellationToken ct = default); Task> GetByChannelAsync(string channelId, ContentType? contentType = null, CancellationToken ct = default); Task> GetByShowAsync(string showId, CancellationToken ct = default); Task> GetByContentTypeAsync(ContentType contentType, string? channelId = null, CancellationToken ct = default); diff --git a/src/Infrastructure/Catalog/EpisodeRepository.cs b/src/Infrastructure/Catalog/EpisodeRepository.cs index 94fc15b..4574f20 100644 --- a/src/Infrastructure/Catalog/EpisodeRepository.cs +++ b/src/Infrastructure/Catalog/EpisodeRepository.cs @@ -14,6 +14,15 @@ await db.Episodes .Include(e => e.Streams) .FirstOrDefaultAsync(e => e.Id == id, ct); + public async Task> GetByTvdbIdAsync(int tvdbId, CancellationToken ct = default) => + await db.Episodes + .Include(e => e.Show).ThenInclude(s => s.Channel) + .Include(e => e.Streams) + .Where(e => e.Show.TvdbId == tvdbId) + .OrderByDescending(e => e.BroadcastDate) + .Take(200) + .ToListAsync(ct); + public async Task> SearchAsync(string query, CancellationToken ct = default) { var lower = query.ToLower(); diff --git a/src/Presentation/Api/NewznabIndexerApi/Endpoints/NewznabEndpoints.cs b/src/Presentation/Api/NewznabIndexerApi/Endpoints/NewznabEndpoints.cs index 78e0065..6a6c9fb 100644 --- a/src/Presentation/Api/NewznabIndexerApi/Endpoints/NewznabEndpoints.cs +++ b/src/Presentation/Api/NewznabIndexerApi/Endpoints/NewznabEndpoints.cs @@ -26,6 +26,7 @@ private static async Task HandleApiAsync( string? q, int? season, int? ep, + int? tvdbid, string? apikey, int? limit, CancellationToken ct) @@ -39,7 +40,8 @@ private static async Task HandleApiAsync( case "tvsearch": if (!ApiKeyGuard.IsAuthorized(config, apikey)) return Denied(); - var releases = await search.HandleAsync(new SearchReleasesQuery(q, season, ep, limit ?? 100), ct); + var releases = await search.HandleAsync( + new SearchReleasesQuery(q, season, ep, limit ?? 100, tvdbid), ct); var baseUrl = $"{http.Request.Scheme}://{http.Request.Host}"; var key = ApiKeyGuard.Configured(config); diff --git a/src/Presentation/Api/NewznabIndexerApi/Newznab/NewznabXml.cs b/src/Presentation/Api/NewznabIndexerApi/Newznab/NewznabXml.cs index d68d222..240a70d 100644 --- a/src/Presentation/Api/NewznabIndexerApi/Newznab/NewznabXml.cs +++ b/src/Presentation/Api/NewznabIndexerApi/Newznab/NewznabXml.cs @@ -24,7 +24,10 @@ public static string Capabilities() => new XElement("limits", new XAttribute("max", "500"), new XAttribute("default", "100")), new XElement("searching", new XElement("search", new XAttribute("available", "yes"), new XAttribute("supportedParams", "q")), - new XElement("tv-search", new XAttribute("available", "yes"), new XAttribute("supportedParams", "q,season,ep")), + new XElement("tv-search", new XAttribute("available", "yes"), + // Sonarr reads caps and only sends what we advertise, so tvdbid has to be listed + // here or it will keep falling back to title-only searches. + new XAttribute("supportedParams", "q,tvdbid,season,ep")), new XElement("movie-search", new XAttribute("available", "no"), new XAttribute("supportedParams", "q"))), new XElement("categories", new XElement("category", @@ -56,6 +59,8 @@ private static XElement Item(Release r, string url) Attr("category", r.Category), Attr("size", r.Size)); + // The one attribute that lets Sonarr skip title parsing entirely. + if (r.TvdbId is not null) item.Add(Attr("tvdbid", r.TvdbId.Value)); if (r.Season is not null) item.Add(Attr("season", r.Season.Value)); if (r.Episode is not null) item.Add(Attr("episode", r.Episode.Value)); return item; diff --git a/tests/Application.Tests/TvdbMatchingTests.cs b/tests/Application.Tests/TvdbMatchingTests.cs new file mode 100644 index 0000000..9c7c098 --- /dev/null +++ b/tests/Application.Tests/TvdbMatchingTests.cs @@ -0,0 +1,121 @@ +using Krautwatch.Application.Indexing; +using Krautwatch.Domain.Entities; +using Krautwatch.Domain.Enums; +using Krautwatch.Domain.Interfaces; +using NSubstitute; +using Shouldly; +using Xunit; + +namespace Krautwatch.Application.Tests; + +/// +/// Covers TVDB-id matching (#4 follow-up). The point of the id is that Sonarr no longer has to identify a +/// series by parsing our release title against notoriously inconsistent Mediathek naming. +/// +public class TvdbMatchingTests +{ + private static Episode Ep(string id, int? tvdbId, int? season = null, int? episode = null) => new() + { + Id = id, + Title = "an episode", + ShowId = "ard:show", + BroadcastDate = DateTimeOffset.UtcNow, + Duration = TimeSpan.FromMinutes(30), + SeasonNumber = season, + EpisodeNumber = episode, + Show = new Show + { + Id = "ard:show", Title = "extra 3", ChannelId = "ard", + SeriesType = SeriesType.Daily, TvdbId = tvdbId, + }, + }; + + // ── projection ──────────────────────────────────────────── + + [Fact] + public void A_release_carries_the_shows_tvdb_id() + { + ReleaseMapper.ToRelease(Ep("ard:1", tvdbId: 255986)).TvdbId.ShouldBe(255986); + } + + [Fact] + public void An_unmapped_show_yields_no_tvdb_id_rather_than_a_wrong_one() + { + // Most shows are unmapped today. Emitting a bogus id would be far worse than emitting none: Sonarr + // trusts the id over the title, so a wrong one silently attaches releases to the wrong series. + ReleaseMapper.ToRelease(Ep("ard:1", tvdbId: null)).TvdbId.ShouldBeNull(); + } + + // ── search by id ────────────────────────────────────────── + + [Fact] + public async Task A_search_by_tvdb_id_answers_from_the_id_not_the_title() + { + var episodes = Substitute.For(); + episodes.GetByTvdbIdAsync(255986, Arg.Any()).Returns([Ep("ard:1", 255986)]); + + var result = await new SearchReleasesHandler(episodes).HandleAsync( + new SearchReleasesQuery(TvdbId: 255986), TestContext.Current.CancellationToken); + + result.ShouldHaveSingleItem().TvdbId.ShouldBe(255986); + await episodes.DidNotReceive().SearchAsync(Arg.Any(), Arg.Any()); + } + + [Fact] + public async Task An_unmapped_id_falls_back_to_the_title_search_when_one_was_sent() + { + // An id we have not mapped is our gap, not proof the show does not exist — so if Sonarr also sent a + // title, use it rather than reporting nothing. + var episodes = Substitute.For(); + episodes.GetByTvdbIdAsync(999999, Arg.Any()).Returns([]); + episodes.SearchAsync("extra 3", Arg.Any()).Returns([Ep("ard:1", null)]); + + var result = await new SearchReleasesHandler(episodes).HandleAsync( + new SearchReleasesQuery(Q: "extra 3", TvdbId: 999999), TestContext.Current.CancellationToken); + + result.ShouldHaveSingleItem(); + await episodes.Received(1).SearchAsync("extra 3", Arg.Any()); + } + + [Fact] + public async Task An_unmapped_id_with_no_title_returns_empty_without_a_title_search() + { + var episodes = Substitute.For(); + episodes.GetByTvdbIdAsync(999999, Arg.Any()).Returns([]); + + var result = await new SearchReleasesHandler(episodes).HandleAsync( + new SearchReleasesQuery(TvdbId: 999999), TestContext.Current.CancellationToken); + + result.ShouldBeEmpty(); + await episodes.DidNotReceive().SearchAsync(Arg.Any(), Arg.Any()); + } + + [Fact] + public async Task Season_and_episode_still_narrow_a_search_by_id() + { + var episodes = Substitute.For(); + episodes.GetByTvdbIdAsync(266275, Arg.Any()).Returns( + [Ep("kika:1", 266275, season: 2, episode: 51), + Ep("kika:2", 266275, season: 2, episode: 52)]); + + var result = await new SearchReleasesHandler(episodes).HandleAsync( + new SearchReleasesQuery(Season: 2, Episode: 52, TvdbId: 266275), + TestContext.Current.CancellationToken); + + result.ShouldHaveSingleItem().Episode.ShouldBe(52); + } + + [Fact] + public async Task The_rss_feed_is_unaffected_by_the_id_path() + { + // No query and no id: still the recent-releases feed that RSS-Sync polls. + var episodes = Substitute.For(); + episodes.GetRecentAsync(Arg.Any(), Arg.Any()).Returns([Ep("ard:1", null)]); + + var result = await new SearchReleasesHandler(episodes).HandleAsync( + new SearchReleasesQuery(), TestContext.Current.CancellationToken); + + result.ShouldHaveSingleItem(); + await episodes.DidNotReceive().GetByTvdbIdAsync(Arg.Any(), Arg.Any()); + } +}