Skip to content

feat(indexing): query-driven search — resolve t=tvsearch on demand (#58, DR-011) - #62

Merged
ChrisonSimtian merged 3 commits into
mainfrom
feat/query-driven-search
Jul 27, 2026
Merged

feat(indexing): query-driven search — resolve t=tvsearch on demand (#58, DR-011)#62
ChrisonSimtian merged 3 commits into
mainfrom
feat/query-driven-search

Conversation

@ChrisonSimtian

Copy link
Copy Markdown
Contributor

Implements DR-011. Search previously served only what a crawler had already written, and the crawl list is three hardcoded seed shows — so Sonarr searching for anything else got an empty feed, indistinguishable from "not in the Mediathek". Now an uncrawled term resolves against the broadcasters on demand, which is what lets Krautwatch work with zero *arr configuration.

The wait is bounded; the crawl is not

Per your steer. A search waits RequestDeadline (default 8s, configurable) then answers with whatever landed — but the crawl keeps running to completion in the background, so the next call gets the full set. Abandoning a half-finished crawl would discard ARD round-trips already paid for and leave the cache permanently partial.

The consequence is the subtle bit: the crawl must not observe the request's CancellationToken. It's queued and drained by a hosted service under the host lifetime. Threading the request token through would cancel every crawl the instant the response was written — presenting as "resolution mysteriously never works".

There's a test for exactly that, and I mutation-checked it: reintroducing the bug fails that one test and nothing else, so the guarantee is genuinely enforced rather than incidentally true.

Negative caching matters more than positive

ResolvedQuery records normalised query, timestamp, result count, providers tried. Sonarr re-issues the same failing query every RSS-Sync cycle, so without a "we looked and found nothing" marker, each cycle triggers a fresh multi-hop ARD crawl. Hits trusted 6h, misses 45m.

Configuration

Indexing:OnDemandResolution:Enabled                   # true — kill switch
Indexing:OnDemandResolution:RequestDeadline           # 00:00:08
Indexing:OnDemandResolution:CrawlTimeout              # 00:02:00
Indexing:OnDemandResolution:PositiveTtl               # 06:00:00
Indexing:OnDemandResolution:NegativeTtl               # 00:45:00
Indexing:OnDemandResolution:MaxConcurrentResolutions  # 2 — politeness cap

Other decisions

RSS (no query) is never resolved — RSS-Sync polls constantly with no target, so resolving there means crawling on a timer for nothing specific. It keeps serving the standing crawl list per DR-011.

A miss returns 200 with an empty feed, never an error. Sonarr treats indexer errors as an availability problem and disables an indexer that keeps failing, so "no results yet" must stay distinguishable from "broken".

Concurrent identical queries coalesce to one crawl. Per-process only — several API replicas would each crawl once, not worth distributed locking.

The API host now registers the broadcaster crawlers, which until now only the agents had. That makes it run an IO-driven Action — the same narrow DR-009 deviation already recorded for TestArrConnection, because a synchronous request cannot wait on the durable bus.

Indexing gets its own crawl-and-upsert rather than calling Crawling's handler: the slice-isolation architecture test forbids the cross-slice call, and that small duplication is the intended cost of slice isolation, not an accident to refactor away.

Three bugs caught while writing this

None would have failed a build, so worth naming:

  1. Captive dependency — the singleton resolver was capturing a scoped repository (and its DbContext) for the process lifetime. Now takes IServiceScopeFactory.
  2. Data race — the provider list was being built inside concurrent Task.WhenAll lambdas, i.e. concurrent List.Add. Now captured before the fan-out.
  3. Name collision — Domain has its own Channel entity (a broadcaster channel), ambiguous with System.Threading.Channels.Channel.

Verified end to end against live ARD, from an empty database

Step Result
Cold search q=Panorama (never seeded) 8.56s, 0 items — deadline bounding the wait
Background crawl afterwards 16 episodes persisted; panorama|16|ard,kika,zdf
Warm search, same term 16 items in 98ms from Postgres
Nonexistent show, first call recorded zzznotarealshow|0
Nonexistent show, repeat 28ms, no re-crawl — Sonarr-retry protection
RSS (no query) ResolvedQueries unchanged at 2 rows
t=caps still served

That second row is the one the whole design turns on: the crawl outliving the request that triggered it.

Tests: App 85 / Arch 11 / Infra 102 green, 0 warnings.

Known trade-off

A first interactive search for an uncrawled show may return few or no results and take ~8s. That's the trade DR-011 accepts consciously — and it self-heals, since the same search moments later is instant and complete. Tunable via RequestDeadline if you'd rather wait longer for a complete first answer.

Not in this PR

Adding resolved shows to the standing crawl list, so RSS picks up their future episodes automatically. Tempting, but it grows the list without bound and needs its own retention policy — deliberately left out rather than smuggled in.

Plan: docs/plans/2026-07-28 - query-driven-search.md

🤖 Generated with Claude Code

ChrisonSimtian and others added 3 commits July 28, 2026 10:51
Implements DR-011. Search previously served only what a crawler had already written,
and the crawl list is three hardcoded seed shows — so Sonarr searching for anything else
got an empty feed, indistinguishable from "not in the Mediathek". Now an uncrawled term
is resolved against the broadcasters on demand, which is what lets Krautwatch work with
zero *arr configuration.

## The wait is bounded; the crawl is not

A search waits RequestDeadline (default 8s, configurable) and then answers with whatever
landed — but the crawl keeps running in the background to completion, so the next call
gets the full set. Abandoning a half-finished crawl would discard ARD round-trips already
paid for and leave the cache permanently partial.

The consequence is the subtle part: the crawl must NOT observe the request's
CancellationToken. It is queued and drained by a hosted service under the host lifetime.
Threading the request token through would cancel every crawl the instant the response was
written, presenting as "resolution mysteriously never works". There is a test for exactly
that, and it was mutation-checked: reintroducing the bug fails that one test and nothing
else.

## Negative caching matters more than positive

ResolvedQuery records normalised query, timestamp, result count and providers tried.
Sonarr re-issues the same failing query every RSS-Sync cycle, so without a marker for "we
looked and found nothing", each cycle would trigger a fresh multi-hop crawl of ARD.
Positive results are trusted 6h, misses 45m.

## Other decisions

RSS (no query) is never resolved — RSS-Sync polls constantly with no particular target, so
resolving there would mean crawling on a timer for nothing specific. It keeps serving the
standing crawl list per DR-011.

A miss returns 200 with an empty feed, never an error: Sonarr treats indexer errors as an
availability problem and disables an indexer that keeps failing, so "no results yet" has
to stay distinguishable from "broken".

Identical concurrent queries coalesce to one crawl; a second caller waits on the first's
signal. Per-process only — several API replicas would each crawl once, which is not worth
distributed locking.

The API host now registers the broadcaster crawlers, which until now only the agents had.
That makes it run an IO-driven Action — the same narrow DR-009 deviation already recorded
for TestArrConnection, because a synchronous request cannot wait on the durable bus.
Indexing gets its own crawl-and-upsert rather than calling Crawling's handler: the
slice-isolation architecture test forbids the cross-slice call, and that duplication is the
intended cost, not an accident.

Three bugs caught while writing this, worth noting since none would have failed a build:
a captive dependency (the singleton resolver capturing a scoped repository — now takes
IServiceScopeFactory), a data race building the provider list inside concurrent lambdas,
and Domain's own Channel entity colliding with System.Threading.Channels.Channel.

## Verified end to end against live ARD, from an empty database

- Cold search for "Panorama" (never seeded): returned in 8.56s with 0 items — the deadline
  bounding the wait.
- The background crawl then finished and persisted 16 episodes; ResolvedQueries recorded
  panorama|16|ard,kika,zdf. This is the behaviour the whole design turns on.
- Warm search for the same term: 16 items in 98ms, straight from Postgres.
- A nonexistent show recorded zzznotarealshow|0, and the repeat call took 28ms with no
  re-crawl — the Sonarr-retry protection working.
- RSS (no query) left ResolvedQueries untouched at 2 rows, and t=caps still serves.

Tests: App 85 / Arch 11 / Infra 102 green, 0 warnings.

Plan: docs/plans/2026-07-28 - query-driven-search.md

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…constant

Follow-up on review feedback: how long a search waits for an on-demand resolution is a
genuine trade-off between a complete first answer and a responsive indexer, so it is not
ours to decide. Replaces the compiled-in 8s RequestDeadline with a user setting.

  AppSettings.SearchWaitMode     ReturnFast (default) | WaitForComplete
  AppSettings.SearchWaitSeconds  advanced, 1-300, default 8, ReturnFast only

- ReturnFast: answer after SearchWaitSeconds with whatever resolved, crawl continues in
  the background. First search may under-report; the next is complete and instant.
- WaitForComplete: wait for the resolution so the first search is already complete.

It lives on AppSettings (database) rather than in config precisely so the settings page
(#4 PR 3) can own it — config keeps only the operational knobs (Enabled, CrawlTimeout,
TTLs, MaxConcurrentResolutions). RequestDeadline is removed rather than left alongside, so
there is exactly one source of truth.

WaitForComplete is still capped by CrawlTimeout. An unbounded wait would hang the request
forever on a stuck crawl, so "wait for complete" means "wait up to the crawl's own
ceiling". The XML docs and README both warn that a long wait can exceed Sonarr's indexer
timeout and make a working indexer look like it is failing — the operator should know that
before choosing it.

Reading the preference cannot break search: a failed settings read logs and falls back to
8s rather than throwing. Tested.

Exposed through SettingsResponse/SaveSettingsRequest with validation (1-300s), so PR 3 only
has to render it. The seconds value is validated even in WaitForComplete mode, so a stale
value cannot silently become active by flipping the mode back.

EF generated the migration with defaultValue "" for the enum column, which Enum.Parse would
throw on for any row inserted without an explicit value. Corrected to "ReturnFast" (and 8
rather than 0 for the seconds). Verified the real upgrade path, not just a fresh create: on
a database migrated to the previous migration, with the columns dropped and the history row
rewound, applying the migration yields Id=1 -> ReturnFast|8 and column defaults of
'ReturnFast'::character varying and 8.

Tests: App 88 / Arch 11 / Infra 102 green, 0 warnings. New coverage for WaitForComplete
waiting out a slow crawl, WaitForComplete still being bounded, and an unreadable preference
falling back instead of failing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI caught this; it passed locally. I set CrawlTimeout and the wait ceiling to the same
300ms, so two timers raced. Either can end the wait — the crawl hitting its timeout, which
completes the resolution (emptily, returning true), or the ceiling releasing the caller
(returning false) — and the winner varied by machine.

The `true` is correct behaviour, not a bug: the resolution genuinely finished, just with no
results, and re-reading is harmless. The test was asserting the mechanism rather than the
guarantee.

Now asserts what actually matters — that a stuck crawl does not hang the request — by
bounding elapsed time rather than pinning the bool. Ran 5x locally to confirm it is stable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ChrisonSimtian
ChrisonSimtian merged commit 5192173 into main Jul 27, 2026
1 check passed
@ChrisonSimtian
ChrisonSimtian deleted the feat/query-driven-search branch July 27, 2026 23:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant