From 3897c34f795ce117517dffd83d3e20a2fe94b528 Mon Sep 17 00:00:00 2001 From: Chrison Simtian Date: Tue, 28 Jul 2026 09:47:17 +1200 Subject: [PATCH] =?UTF-8?q?docs(arch):=20DR-011=20=E2=80=94=20retract=20DR?= =?UTF-8?q?-010's=20reach-back=20clause;=20search=20is=20query-driven?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DR-010 stated that polling each Sonarr/Radarr instance for its monitored series "is the crawl work-list". That made the monitored list load-bearing: #6 was scoped as the work-list, #12 was blocked behind it, and #4/#48 became prerequisites for basic indexing. Before building it, checked how the comparable projects solve the same problem. Neither does it this way: - MediathekArr keeps no catalog at all. It is a live search proxy — "pretending to be a usenet indexer, but actually just fetching and parsing search results from MediathekViewWeb", with TVDB for metadata and the real effort in matching. - RundfunkArr uses curated configuration: data/shows.json plus data/rulesets.json (Mediathek topic, filters, season/episode regex), extended by community PRs. Neither reads Sonarr's monitored series via the Sonarr API, because the Newznab contract is pull-by-query: Sonarr already knows what it monitors and asks for exactly what it wants. An indexer is not supposed to know the monitored list. The reason the question arose for us is our own choice, not a property of being an *arr indexer: DR-010 de-emphasised MediathekView in favour of the direct ARD/ZDF APIs. MediathekViewWeb is effectively a complete dump, so MediathekArr gets breadth free; our ARD crawler must walk A-Z catalog -> show page -> episodes -> item page and therefore has to be told which shows to walk. The work-list problem is self-inflicted by the direct-API decision. DR-011 accordingly: - Search becomes query-driven (resolve on demand, cache), so any show works without pre-registration. - The standing crawl list survives only as RSS-feed input, since RSS-Sync genuinely needs a recent-releases list. - Reach-back is demoted to an optional pre-warm. Nothing may depend on a configured *arr instance to function. - Matching (EpisodeNumbering, SeriesType, the unset Show.TvdbId) is where the effort belongs, since that is where both comparables concentrate. Reach-back was also insufficient on its own: an interactive search for a show that is unmonitored, or monitored but not yet crawled, returns nothing — so a query-driven path was needed regardless, which demotes reach-back by definition. DR-010's clause is struck through in place rather than deleted, so the history stays readable, and its header records the retraction. CLAUDE.md stated the retracted claim as fact and is corrected. Recorded honestly in the DR: MediathekArr's README does not explicitly say live-versus-cached (inferred), and neither project documents RSS support — which is exactly the half that still needs a standing crawl, so our RSS design cannot be copied from either. Backlog: #6 re-scoped to an optional pre-warm and unblocked from #12; #58 opened for query-driven search; #49 (delete MediathekView) put on hold — its SharpCompress advisory was already fixed by a version bump in #53, and the filmliste is the only full-catalog source available, which is precisely what MediathekArr depends on. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 20 ++-- .../DR-010-arr-indexer-direction.md | 11 +- .../DR-011-search-driven-indexing.md | 106 ++++++++++++++++++ docs/architecture/README.md | 3 +- 4 files changed, 129 insertions(+), 11 deletions(-) create mode 100644 docs/architecture/DR-011-search-driven-indexing.md diff --git a/CLAUDE.md b/CLAUDE.md index d6f09f4..82c2db8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,13 +26,17 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co > Aspire-generated compose in the distribution milestone. > > **Known gaps (don't assume these exist):** -> - **The crawl work-list is hardcoded**, not Sonarr-driven. `CrawlOptions.Targets` binds from each -> agent's `Crawl` config section and falls back to seed shows in `Agents/{Ard,Zdf}/Program.cs` -> (`extra 3` / `Die Biene Maja` / `heute-show`). DR-010's reach-back — poll each Sonarr/Radarr -> instance for its `monitored` series — is **not implemented**, and neither is any -> Sonarr/Radarr-instance entity or config UI. `Settings` is download settings only. +> - **Search is not query-driven yet, and the crawl list is hardcoded.** `CrawlOptions.Targets` binds +> from each agent's `Crawl` config section and falls back to seed shows in `Agents/{Ard,Zdf}/Program.cs` +> (`extra 3` / `Die Biene Maja` / `heute-show`). So `t=tvsearch` only finds what has already been +> crawled — an unseeded show returns nothing. **DR-011** makes on-demand resolution the target and +> demotes the standing list to RSS-feed input; reach-back to Sonarr is an optional pre-warm, not a +> requirement (DR-010's work-list clause is retracted). No Sonarr/Radarr-instance config UI yet either +> — `Settings` is download settings only. > - `AppSettings.CatalogProviderKey` still defaults to `"mediathekview"`, and -> `Infrastructure/Catalog/MediathekView` survives from the DR-001 era — de-emphasised by DR-010. +> `Infrastructure/Catalog/MediathekView` survives from the DR-001 era. **Do not delete it yet** — +> per DR-011 the filmliste is the only full-catalog source available, and it is the fallback if +> per-show crawling proves too narrow (#49 is on hold). > - Auth is a single optional API key (see below); no user/identity model. > - `README.md` is **pre-DR-009 and wholly wrong** (describes the dropped role system, SQLite, > `Krautwatch.Worker`, the retired `/api/catalog` surface) — issue #25. @@ -236,7 +240,9 @@ DE proxy and `./build.cmd TestLive` does the real KiKA download (else it just pr ## Architecture decisions -`docs/architecture/` — **DR-009 (architecture reset) and DR-010 (arr-indexer direction) are current.** +`docs/architecture/` — **DR-009 (architecture reset), DR-010 (arr-indexer direction) and DR-011 +(search-driven indexing) are current.** DR-011 retracts DR-010's reach-back clause: the Sonarr monitored +list is *not* the crawl work-list. DR-002/004/008 are superseded; DR-001/003/005/006/007 still apply (some refined). Read the current DRs before structural changes. diff --git a/docs/architecture/DR-010-arr-indexer-direction.md b/docs/architecture/DR-010-arr-indexer-direction.md index e6aa8a0..c149ac4 100644 --- a/docs/architecture/DR-010-arr-indexer-direction.md +++ b/docs/architecture/DR-010-arr-indexer-direction.md @@ -2,10 +2,11 @@ | | | |---|---| -| **Status** | Accepted | +| **Status** | Accepted · **reach-back clause retracted by DR-011** | | **Date** | 2026-07-25 | | **Deciders** | Christian | | **Refines** | DR-001 (catalog provider abstraction) | +| **Refined by** | [DR-011](DR-011-search-driven-indexing.md) — the monitored list is *not* the crawl work-list | ## Context @@ -25,8 +26,12 @@ Krautwatch integrates with the `*arr` stack via the interfaces those apps alread catalog. - **Download client (SABnzbd-compatible):** queue / history / add-by-id, so Sonarr/Radarr treat Krautwatch as a usenet download client. Actual pulls are direct HTTP/HLS + ffmpeg, with subtitles. -- **Reach-back:** poll each configured Sonarr/Radarr instance for its `monitored` series list; that - is the crawl work-list. +- ~~**Reach-back:** poll each configured Sonarr/Radarr instance for its `monitored` series list; that + is the crawl work-list.~~ + > **Retracted by [DR-011](DR-011-search-driven-indexing.md).** Neither MediathekArr nor RundfunkArr + > reads Sonarr's monitored list — the Newznab contract is pull-by-query, so an indexer does not need + > it. Search becomes query-driven; the standing crawl list feeds only the RSS feed, and reach-back is + > an optional pre-warm. The **UI shrinks to configuration** (Sonarr/Radarr instances, connection test) — not a browse/search surface. diff --git a/docs/architecture/DR-011-search-driven-indexing.md b/docs/architecture/DR-011-search-driven-indexing.md new file mode 100644 index 0000000..af5557f --- /dev/null +++ b/docs/architecture/DR-011-search-driven-indexing.md @@ -0,0 +1,106 @@ +# DR-011 — Search-Driven Indexing (the monitored list is not the work-list) + +| | | +|---|---| +| **Status** | Accepted | +| **Date** | 2026-07-28 | +| **Deciders** | Christian | +| **Refines** | DR-010 (arr-indexer direction) — specifically retracts its **Reach-back** clause | + +## Context + +DR-010 established that Krautwatch is a Sonarr/Radarr **indexer + download client**, and that stands. +But one clause went further than the integration requires: + +> **Reach-back:** poll each configured Sonarr/Radarr instance for its `monitored` series list; that is +> the crawl work-list. + +That made the monitored list load-bearing: #6 was scoped as "the crawl work-list", #12 (RSS for +monitored shows) was blocked behind it, and #4/#5 existed largely to make it possible. Before building +it, we checked how the two comparable projects solve the same problem. + +### How the comparables actually do it + +**MediathekArr** keeps no catalog. It is a **live search proxy** — "pretending to be a usenet indexer, +but actually just fetching and parsing search results from **MediathekViewWeb**". TVDB supplies +metadata, and the effort goes into an "advanced filter and matching system for TV shows, seasons and +episodes" to survive "the horrendous lack of consistency and metadata in ARD/ZDF Mediatheken". + +**RundfunkArr** uses **curated configuration**: `data/shows.json` (show metadata incl. `tvdbId`, german +name, aliases, episodes) plus `data/rulesets.json` (per-show Mediathek topic, duration filters, regex +for season/episode extraction, matching strategy). Shows are added by hand or community pull request. +SQLite is cache and download history only. + +**Neither reads Sonarr's monitored series via the Sonarr API.** Both are pure Newznab providers. + +### Why they don't need to + +The Newznab contract is **pull-by-query, not push-by-work-list**. Sonarr already knows what it monitors +and issues `t=tvsearch&q=…&tvdbid=…&season=&ep=` for precisely what it wants. An indexer is not +supposed to know the monitored list — that is the client's job. + +The reason the question arose for us is a decision we made, not a property of being an `*arr` indexer: +DR-010 **de-emphasised MediathekView in favour of direct ARD/ZDF APIs**. MediathekViewWeb is +effectively a complete dump, so MediathekArr gets breadth for free. Our ARD crawler must walk A-Z +catalog → show page → episode list → item page, so it has to be told *which* shows to walk. +RundfunkArr has the identical constraint and answers it with human curation. + +**The work-list problem is self-inflicted by the direct-API choice.** + +## Decision + +**Retract the reach-back clause.** The monitored list is not the crawl work-list. Instead: + +1. **Search is query-driven.** `t=search`/`t=tvsearch` resolves against the broadcaster on demand and + caches the result, so any show Sonarr asks for works without having been pre-registered. This is + what makes Krautwatch behave like an indexer rather than a catalog with a search box. +2. **A standing crawl list exists only to feed the RSS feed.** RSS-Sync needs a recent-releases list, so + *something* must be crawled proactively. That list is configuration (today `CrawlOptions.Targets`), + optionally **pre-warmed** from a Sonarr/Radarr monitored list where one is configured. +3. **Reach-back becomes an optional convenience, never a prerequisite.** No feature may depend on a + configured `*arr` instance to function. +4. **Matching is where the effort goes.** Both comparables concentrate there, and it decides whether + Sonarr can match anything at all. Our equivalents are `EpisodeNumbering.Parse`, `SeriesType`, and the + currently-unset `Show.TvdbId`. + +### Why not reach-back as the primary mechanism + +- **It inverts the dependency.** Holding Sonarr credentials and calling back into it is a bidirectional + coupling neither comparable has, and it is the reason #4 and #48 became prerequisites for basic + indexing. +- **It does not satisfy search.** An interactive search for a show that is unmonitored, or monitored but + not yet crawled, returns nothing. A query-driven path is needed regardless — which demotes reach-back + to an optimisation by definition. + +## Consequences + +- ✅ Indexing works with **zero** `*arr` configuration, which is how a Newznab indexer should behave. +- ✅ Unblocks #12: the RSS feed can serve the standing crawl list without waiting on #6. +- ✅ #4/#5 stop being prerequisites for core function; they remain worthwhile for pre-warming and for + the setup experience (#54). +- ⚠️ On-demand search puts broadcaster latency in Sonarr's request path. The ARD flow is multi-hop + (A-Z widget → show page → episodes → item page), so caching and a sane timeout are mandatory, not + optional polish. +- ⚠️ Rate-limiting/politeness toward ARD/ZDF becomes a real concern once searches hit them live. +- ⚠️ **#49 (delete the MediathekView subsystem) is now on hold.** That filmliste parser is precisely the + full-catalog source MediathekArr depends on; it is the mechanism for breadth without per-show + curation. Decide the search model before deleting it. (The `SharpCompress` advisory that motivated #49 + was resolved by a version bump, so there is no security pressure to remove it.) +- ⚠️ DR-010 remains correct on everything else — product direction, the Newznab/SABnzbd surfaces, the + provider/port abstraction, and the UI shrinking to configuration. + +## Evidence and its limits + +Both READMEs were read directly (see Sources). Two gaps worth recording honestly: + +- MediathekArr's README does not *explicitly* state live-versus-cached; "fetching and parsing search + results" strongly implies live, but it is an inference. +- **Neither** project documents RSS feed support, so how they serve RSS-Sync is unknown — which is + precisely the part that still requires a standing crawl. Our RSS design cannot be copied from either + and has to be reasoned out. + +## Sources + +- and its README +- +- diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 9ca25ca..975670a 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -11,4 +11,5 @@ | [DR-007](DR-007-target-framework-net10.md) | Target Framework: .NET 10 LTS | Accepted | | [DR-008](DR-008-single-binary-role-architecture.md) | Single-Binary Role-Switchable Worker | ~~Accepted~~ · superseded by DR-009 | | [DR-009](DR-009-architecture-reset.md) | Architecture Reset: Hexagonal + Vertical-Slice CQRS/A Microservices | **Accepted** | -| [DR-010](DR-010-arr-indexer-direction.md) | Product Direction: Sonarr/Radarr Indexer | **Accepted** | +| [DR-010](DR-010-arr-indexer-direction.md) | Product Direction: Sonarr/Radarr Indexer | **Accepted** · reach-back clause retracted by DR-011 | +| [DR-011](DR-011-search-driven-indexing.md) | Search-Driven Indexing (the monitored list is not the work-list) | **Accepted** |