feat(search): tiered BM25 search (briefs/summaries/sources) + taxonomy accessors

[sebastianbraun25]

Note

This PR was created in collaboration between a human and AI: implementation, tests, and
PR text were created by an AI assistant under the guidance and review of the human author.

Problem

search_wiki (#233/#234) is a single BM25 index over concepts/, entities/, and
summaries/ combined. Concepts/entities are meant to be found by semantic browsing (an
LLM scanning one-line briefs and picking a slug by meaning), not keyword search — but they
share one BM25 corpus with summaries today, so a keyword-dense concept page can outrank the
summary a caller actually needed. Separately, one combined BM25 tier conflates two very
different recall needs (a generic term like "case" wants precision over which document is
about that topic; a specific fact like an exact field name wants recall into full
document bodies), and long PageIndex documents (sources/*.json) aren't searchable at all
today, even though details like authorship/creation dates only ever live there.

Solution / Changes

• openkb/fulltext_index.py:
  • Extracted _BM25Scorer from WikiFullTextIndex (pure refactor, same BM25 math, no
    behavior change — WikiFullTextIndex/search() keep their exact existing signature
    and output for backward compatibility with the current search_wiki tool).
  • New Locator (kind: "line"|"page", value: int) on SearchHit, computed at query
    time (first-match line) or fixed at indexing time (PageIndex page number).
  • New TieredWikiSearch with three independent BM25 tiers, deliberately excluding
    concepts/entities (see list_taxonomy_items below):
    1. briefs — one-line description/legacy brief frontmatter per summaries/*.md.
    2. summaries — full body of summaries/*.md (frontmatter stripped).
    3. sources — sources/*.md (whole file, line locator) + sources/*.json PageIndex
       docs, indexed per page (never per document), each hit carrying
       Locator(kind="page", value=N) directly usable with
       get_page_content(doc_name, pages=str(N)). Malformed/foreign JSON under sources/
       is skipped, not raised.
  • search(query, scope=None, top_k=5) returns {tier: [SearchHit, ...]}; scope
    restricts to a subset of TIERED_SCOPES = ("briefs", "summaries", "sources").
• openkb/frontmatter.py: new resolve_description(fm) / body_only(text) shared
  helpers. Kept as a separate copy from agent.compiler._resolve_description (same logic)
  rather than importing from compiler.py, which has several other open, unrelated PRs
  actively touching it.
• openkb/agent/tools.py: new list_taxonomy_items(wiki_root, kind=None) /
  get_taxonomy_item(slug, wiki_root, kind=None) for semantic browsing of persisted
  concepts//entities/ pages. Only ever surfaces committed .md pages —
  PendingTopicsStore's not-yet-paged candidates are a structurally separate buffer and
  are never returned.
• No CLI, MCP, or query/chat-agent wiring in this PR — pure, additive core logic. Follow-up
  PRs (tracked separately) wire this into openkb list-taxonomy/openkb search, the
  query/chat agent's tool set, and a new MCP server.

Testing

• tests/test_fulltext_index.py: 8 existing tests unchanged/still passing (backward
  compatibility of WikiFullTextIndex); +12 new tests for TieredWikiSearch (briefs vs.
  full-body matching, frontmatter exclusion from body search, line/page locators,
  malformed-JSON tolerance, scope validation, concepts/entities never searched).
• tests/test_agent_tools.py: +13 new tests for list_taxonomy_items/get_taxonomy_item
  (kind filtering, legacy brief: fallback, missing directories, path-traversal
  rejection, disambiguation by kind).
• ruff check, ruff format --check, mypy openkb, pytest all green — full suite: 1267
  passed (18 pre-existing, environment-specific failures on Windows unrelated to this
  change, e.g. POSIX-only symlink/chmod-mode tests, are unaffected and unchanged in count).

Dependencies

• Depends on #234 (feat/issue-233-hybrid-search) — this PR is built on top of that
  branch (WikiFullTextIndex/agent.tools.search_wiki, which it extends without changing
  their behavior). The diff below includes feat(agent): hybrid BM25 search_wiki tool for query/chat agent #234's commits; once feat(agent): hybrid BM25 search_wiki tool for query/chat agent #234 merges first, this
  PR's diff will shrink to just this PR's own commit.

Issues

Resolves #258