Fix pipeline bugs, consolidate duplicated code, harden CI, and streamline README

[Ndevu12]

Summary

This PR fixes the concrete bugs found during a full project review, consolidates duplicated code into single implementations, closes quality gaps, and upgrades CI and the README. No features were removed — every change is a correctness fix, a consolidation, or an upgrade. The full test suite passes (353 passed, 1 skipped) and the new lint gate is clean.

Bug fixes

• Synthesis extraction now actually runs concurrently. The extraction loop awaited each paper sequentially inside the semaphore, so the synthesis.concurrency setting had no effect. Extractions now run through asyncio.gather under the semaphore, with the circuit breaker still cutting over to heuristics after repeated failures.
• Recency scoring no longer rots. signal_recency hardcoded current_year = 2026; it now derives the year from the current UTC date.
• Expanded queries are persisted. Session memory always saved an empty expanded_queries list; the pipeline now exposes the expansion artifact and the orchestrator stores the real variants and sub-questions.
• Failed providers are reported structurally. The retrieval stage rebuilt failed-provider names by string-splitting warning text while discarding the list it had already built. The provider fan-out now returns failed names directly, and results from providers sharing a name are aggregated instead of overwritten.
• LLMProvider.complete works on pydantic-ai 1.x. The structured-output branch used the legacy result_type / result.data API, which raises on the pinned pydantic-ai 1.101.0. It now uses output_type / result.output.
• Partial-recovery fallbacks no longer crash. Two fallback paths imported _attempt_partial_recovery from the orchestrator, where it never existed; they now call the real enhanced_partial_recovery helper.

Consolidation

• New src/research/text_utils.py holds the stop-word sets, generic query terms, and plural-aware term matching that were copy-pasted across ranking, relevance scoring, clustering, and query expansion.
• Cosine similarity is implemented once on the embeddings base module; the sentence-transformers provider and embedding_context delegate to it.
• Embedding deduplication computes all pairwise similarities with a single matrix product instead of per-pair Python loops.
• The retrieval stage reuses the registry's provider fan-out, removing a second copy of the gather-and-collect logic.
• Removed dead code: the deprecated search_openalex / search_semantic_scholar shims, src/utils/json_processing.py (unused in production — the handler uses helpers_modules/json_extraction), alias functions, and an unused system-prompt rebinding.

Quality upgrades

• Retrieval providers log retry and rate-limit events through the logging system instead of printing to stdout from the data layer.
• The shared analysis agent is created lazily on first use, so importing src.analysis no longer resolves settings or constructs a model as a side effect (analysis_agent remains importable).
• Fixed undefined type names, unused variables, and a placeholder-less f-string surfaced by the new lint gate.

CI and tests

• CI auto-discovers tests. The workflow previously hardcoded every test file path, so a new test file would silently never run. It now runs pytest tests/ and adds a ruff lint job (syntax errors, undefined names, unused code) configured in pyproject.toml.
• The suite is hermetic. A new tests/conftest.py stubs the Ollama setup path (no test exercised it intentionally), and subprocess-level tests use the new RA_SKIP_SETUP_CHECK env var. Locally this cut the suite from ~3.5 minutes with network-dependent failures to ~35 seconds, all green.
• Tests updated for the new structured return values, with stronger assertions on failed-provider reporting.

Docs

• README rewritten at a standard level: condensed configuration reference, one architecture diagram, trimmed troubleshooting, license section. The full detail remains in the mkdocs site.

Verification

• pipenv run pytest tests/ — 353 passed, 1 skipped
• ruff check src tests setups — clean
• pipenv run python scripts/check_docs_policy.py — docs policy OK