Daily delta refresh: watermark deltas, revision sweep, daily schedule

[mr-j90]

Parent

Closes #10 — SPEC §4

What this delivers

Delta mode: watermark from the store's newest submission (2-day overlap, idempotent upserts), new submissions per pinned query + a lastUpdatedDate revision sweep per topic; withdrawals detected from the arXiv comment field (where notices actually live) + the abstract heuristic, flagged in DuckDB and removed from every retrieval layer. Full-text for newest papers, capped per run, skipping already-chunked papers. ingest/serve.py registers the daily-delta deployment (cron 0 6 * * *); compose gains a scheduler service and a shared data-runtime volume for api+scheduler.

Verified live: 410 papers swept, store 13,124 → 13,306, newest submission Jul 31 → Aug 6, 1,057 new chunks, snapshot artifacts untouched, deployment visible in Prefect.

Review notes (combined review; four hard findings, all addressed)

• Topic clobbering fixed: upserts now union stored topic labels (a paper resurfacing via one topic's sweep no longer loses its other labels) — tested.
• Compose wiring fixed: the api service's shared-volume mount had silently not applied; verified in rendered config now. First-boot seeding documented (empty store → delta runs fail with a clear message; scheduler stays alive).
• Withdrawal detection made real: comment-field parsing added (abstract-notice heuristic alone was structurally near-vacuous for swept papers). Documented limit: the sweep only sees papers still matching topic phrases — full id_list rotation over held ids is future work.
• Daily re-fetch waste fixed: already-chunked papers skip the polite 6s HTML round-trip.

🤖 Generated with Claude Code