Problem
search_wiki (#233/#234) is a single BM25 index over concepts/, entities/, and
summaries/ combined. Two related issues surface as a wiki grows:
- Concepts/entities shouldn't be keyword-searched at all. They are meant to be
discovered by semantic browsing (an LLM scanning index.md's one-line briefs and
picking the relevant slug by meaning), not by keyword frequency — but they currently
share one BM25 corpus with summaries, so an unrelated but keyword-dense concept page
can outrank the summary the caller actually needed.
- One BM25 tier conflates two very different recall needs. A query for a generic
term (e.g. "case") mostly needs a precision filter (which document's one-line brief
is actually about that topic), while a query for a specific fact (e.g. an exact field
name) needs recall into full document bodies — a single combined index tunes for
neither well, and long PageIndex documents (sources/*.json) aren't searchable at all
today, even though details like authorship/creation dates only ever live there, never
in a generated summary.
Vorschlag
- Add
list_taxonomy_items()/get_taxonomy_item() (openkb/agent/tools.py) for semantic
browsing of persisted concepts//entities/ pages (kind-filterable), separate from any
BM25 search.
- Add
TieredWikiSearch (openkb/fulltext_index.py) with three independent BM25 tiers,
each over a different granularity so a query only pays the recall/precision trade-off
it actually needs:
briefs — one-line description/legacy brief frontmatter per summaries/*.md.
summaries — full body of summaries/*.md.
sources — sources/*.md (whole file) + sources/*.json PageIndex docs, indexed
per page (not per document), each hit carrying a Locator(kind="page", value=N)
directly usable with get_page_content(doc_name, pages=str(N)).
- Add
Locator (kind: "line"|"page") on SearchHit so a hit can point at an exact line
(short docs) or page (PageIndex docs), not just "this file".
This issue covers the core engine only — no CLI/MCP/agent wiring (tracked separately as
follow-ups building on this one).
Kontext
This issue was drafted with the assistance of an AI assistant.
Problem
search_wiki(#233/#234) is a single BM25 index overconcepts/,entities/, andsummaries/combined. Two related issues surface as a wiki grows:discovered by semantic browsing (an LLM scanning
index.md's one-line briefs andpicking the relevant slug by meaning), not by keyword frequency — but they currently
share one BM25 corpus with summaries, so an unrelated but keyword-dense concept page
can outrank the summary the caller actually needed.
term (e.g. "case") mostly needs a precision filter (which document's one-line brief
is actually about that topic), while a query for a specific fact (e.g. an exact field
name) needs recall into full document bodies — a single combined index tunes for
neither well, and long PageIndex documents (
sources/*.json) aren't searchable at alltoday, even though details like authorship/creation dates only ever live there, never
in a generated summary.
Vorschlag
list_taxonomy_items()/get_taxonomy_item()(openkb/agent/tools.py) for semanticbrowsing of persisted
concepts//entities/pages (kind-filterable), separate from anyBM25 search.
TieredWikiSearch(openkb/fulltext_index.py) with three independent BM25 tiers,each over a different granularity so a query only pays the recall/precision trade-off
it actually needs:
briefs— one-linedescription/legacybrieffrontmatter persummaries/*.md.summaries— full body ofsummaries/*.md.sources—sources/*.md(whole file) +sources/*.jsonPageIndex docs, indexedper page (not per document), each hit carrying a
Locator(kind="page", value=N)directly usable with
get_page_content(doc_name, pages=str(N)).Locator(kind: "line"|"page") onSearchHitso a hit can point at an exact line(short docs) or page (PageIndex docs), not just "this file".
This issue covers the core engine only — no CLI/MCP/agent wiring (tracked separately as
follow-ups building on this one).
Kontext
WikiFullTextIndex/search_wiki(kept unchanged, byte-for-bytebackward compatible, for existing callers) —
TieredWikiSearchis additive, sharing thesame BM25 math via an extracted
_BM25Scorer.This issue was drafted with the assistance of an AI assistant.