feat(cli,agent): expose tiered search + taxonomy via CLI and wire into query/chat agent

README.md

@@ -185,6 +185,8 @@ OpenKB commands fall into two layers: the **wiki foundation** (compile + manage
 | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
 | <code>openkb&nbsp;remove&nbsp;&lt;doc&gt;</code>                   | Remove a document and clean up its wiki pages, images, registry, and PageIndex state (`--dry-run` to preview, `--keep-raw` / `--keep-empty` to retain artifacts)                                                                     |
 | <code>openkb&nbsp;recompile&nbsp;[&lt;doc&gt;]&nbsp;[--all]</code> | Re-run the compile pipeline on already-indexed docs without re-indexing. Regenerates summaries and rewrites concept pages; manual edits are overwritten (`--dry-run` to preview, `--refresh-schema` to also update `wiki/AGENTS.md`) |
+| <code>openkb&nbsp;list-taxonomy&nbsp;[--kind&nbsp;concept&#124;entity]</code> | List persisted concept/entity pages with their one-line briefs — semantic browsing, not keyword search (`--json` for scripting)                                                                                       |
+| <code>openkb&nbsp;search&nbsp;"query"&nbsp;[--scope&nbsp;briefs,summaries,sources]</code> | Tiered BM25 full-text search over `summaries/`/`sources/` (never `concepts/`/`entities/` — use `list-taxonomy` for those); `--json` for scripting                                                             |
 | <code>openkb&nbsp;feedback&nbsp;["msg"]</code>                     | File feedback by opening a prefilled GitHub issue (`--type bug/feature/question` to tag it)                                                                                                                                          |
 
 </details>
@@ -207,6 +209,8 @@ A "generator" reads from the compiled wiki and produces something usable: an ans
 
 `openkb query "..."` answers a single question with a grounded, cited answer from your wiki. `openkb chat` is interactive, an ongoing multi-turn session over the same wiki (`--resume`, `--list`, `--delete` to manage sessions). → Walked through with real saved output in **[`examples/commands/`](examples/commands/)** (query) and **[`examples/chat/`](examples/chat/)** (chat).
 
+Retrieval is hybrid: the agent primarily navigates via `index.md`'s one-line summaries, and additionally has `list_taxonomy`/`get_taxonomy_item` (semantic browsing of `concepts/`/`entities/` pages by their one-line briefs — not a keyword search) and a tiered `search_wiki` tool — a dependency-free BM25 full-text search, in three independent tiers over `summaries/` briefs, full `summaries/` bodies, and `sources/` (including per-page indexing of long PageIndex documents) — for surfacing details a summary omits (an exact term, an author, a creation date). It's additive, not a replacement, so recall can only improve over index-only navigation. The same search/browse capability is available outside the agent via `openkb list-taxonomy` and `openkb search` (see `openkb --help`).
+
 Inside a chat, type `/` to access slash commands (Tab to complete).
 
 <details>

openkb/agent/query.py

@@ -14,6 +14,12 @@
     read_wiki_image,
     write_kb_file,
 )
+from openkb.agent.tools import (
+    list_taxonomy as list_taxonomy_impl,
+)
+from openkb.agent.tools import (
+    search_wiki as search_wiki_impl,
+)
 from openkb.config import LlmCredentialBundle, resolve_model_settings
 from openkb.schema import get_agents_md
 
@@ -25,20 +31,34 @@
 {schema_md}
 
 ## Search strategy
-1. Read index.md to see all documents and concepts with brief summaries.
-   Each document is marked (short) or (pageindex) to indicate its type.
+1. Read index.md to see all documents with brief summaries. Each document is
+   marked (short) or (pageindex) to indicate its type.
 2. Read relevant summary pages (summaries/) for document overviews.
    Summaries may omit details — if you need more, follow the summary's
-   `full_text` frontmatter field to the source (see step 4).
-3. Read concept pages (concepts/) for cross-document synthesis.
-4. For "who/what is X" questions about a specific named person, organization,
-   place, or product, read the matching page in entities/ first.
+   `full_text` frontmatter field to the source (see step 5).
+3. For concepts (cross-document synthesis) and entities ("who/what is X"
+   questions about a specific named person, organization, place, or
+   product), call list_taxonomy first — it's a compact, one-line-per-item
+   browse list, not a keyword search. Pick the slug(s) that match the
+   question's meaning by their brief, then read_file the matching
+   concepts/<slug>.md or entities/<slug>.md.
+4. If index.md's one-line summaries and list_taxonomy don't surface a
+   specific detail you need (a niche term, an exact figure, an
+   author/creation-date only present in a raw source), use
+   search_wiki(query, scope) — a tiered, keyword-level full-text search
+   over summaries/sources only (concepts/entities are step 3's job, never
+   search_wiki's). This is a hybrid fallback: use it in addition to, not
+   instead of, index.md/list_taxonomy navigation. Narrow scope to
+   ["sources"] when you specifically need a source-only detail (an exact
+   field name, an author, a date) that a generated summary would likely
+   omit; leave scope unset to search all tiers.
 5. When you need detailed source document content, each summary page has a
    `full_text` frontmatter field with the path to the original document content:
    - Short documents (doc_type: short): read_file with that path.
    - PageIndex documents (doc_type: pageindex): use get_page_content(doc_name, pages)
      with tight page ranges. The summary shows document tree structure with page
-     ranges to help you target. Never fetch the whole document.
+     ranges to help you target. Never fetch the whole document. A search_wiki
+     hit with a "page" locator names the exact page to fetch.
 6. Source content may reference images. Short-doc .md pages link them
    note-relative (e.g. ![image](images/doc/file.png), resolved from
    wiki/sources/); long-doc JSON page metadata lists them wiki-root-relative
@@ -83,6 +103,37 @@ def get_page_content(doc_name: str, pages: str) -> str:
         """
         return get_wiki_page_content(doc_name, pages, wiki_root)
 
+    @function_tool
+    def list_taxonomy(kind: str | None = None) -> str:
+        """List persisted concept/entity pages with one-line briefs (semantic browsing).
+
+        Call this first for concept/entity questions and pick a slug by
+        meaning — this is a browse list, not a keyword search. Follow up
+        with read_file on the matching concepts/<slug>.md or
+        entities/<slug>.md to get the full page.
+
+        Args:
+            kind: "concept" or "entity" to restrict the list; omit for both.
+        """
+        return list_taxonomy_impl(wiki_root, kind=kind)
+
+    @function_tool
+    def search_wiki(query: str, scope: list[str] | None = None) -> str:
+        """Tiered full-text (BM25) keyword search over summaries/sources.
+
+        Hybrid fallback for when index.md's one-line summaries and
+        list_taxonomy don't surface a specific buried detail (a niche term,
+        an exact figure, a fact only present in a raw source). Never
+        searches concepts/entities — use list_taxonomy for those. Use in
+        addition to, not instead of, index.md/list_taxonomy navigation.
+
+        Args:
+            query: Free-text search query (keywords or a natural-language question).
+            scope: Restrict to a subset of "briefs", "summaries", "sources";
+                omit to search all three tiers.
+        """
+        return search_wiki_impl(query, wiki_root, scope=scope)
+
     @function_tool
     def get_image(image_path: str) -> ToolOutputImage | ToolOutputText:
         """View an image from the wiki.
@@ -117,7 +168,7 @@ def get_image(image_path: str) -> ToolOutputImage | ToolOutputText:
     return Agent(
         name="wiki-query",
         instructions=instructions,
-        tools=[read_file, get_page_content, get_image],
+        tools=[read_file, get_page_content, list_taxonomy, search_wiki, get_image],
         model=f"litellm/{model}",
         model_settings=ModelSettings(**model_settings),
     )

openkb/agent/tools.py

@@ -9,10 +9,17 @@
 
 import contextlib
 import json as _json
+from dataclasses import dataclass
 from pathlib import Path, PurePosixPath
+from typing import Literal
 
+from openkb import frontmatter
 from openkb.locks import atomic_write_text
 
+# Maps a taxonomy "kind" to its wiki subdirectory. Single source of truth for
+# list_taxonomy_items/get_taxonomy_item below.
+_TAXONOMY_DIRS: dict[str, str] = {"concept": "concepts", "entity": "entities"}
+
 
 def list_wiki_files(directory: str, wiki_root: str) -> str:
     """List all Markdown files in a wiki subdirectory.
@@ -135,6 +142,180 @@ def get_wiki_page_content(doc_name: str, pages: str, wiki_root: str) -> str:
     return "\n\n".join(parts) + "\n\n"
 
 
+@dataclass(frozen=True)
+class TaxonomyItem:
+    """One persisted concept or entity page (never a pending candidate).
+
+    ``PendingTopicsStore`` (see ``openkb.pending``) buffers not-yet-paged
+    concept/entity candidates separately from the compiled ``.md`` pages
+    under ``concepts/``/``entities/`` — this dataclass, and
+    :func:`list_taxonomy_items`, only ever surface the latter, so a caller
+    never sees an in-progress candidate as if it were a real page.
+    """
+
+    kind: Literal["concept", "entity"]
+    slug: str
+    path: str  # wiki-root-relative, e.g. "concepts/attention.md"
+    brief: str
+    # Entity type (e.g. "person", "organization"); always None for concepts.
+    type: str | None = None
+
+
+def list_taxonomy_items(wiki_root: str, kind: str | None = None) -> list[TaxonomyItem]:
+    """List persisted concept and/or entity pages with their one-line briefs.
+
+    Intended as the first step of the search strategy: browse this compact,
+    semantically-scannable list and let the caller (an LLM) pick the
+    relevant slug(s) by meaning — this is deliberately not a keyword search
+    (see ``search_wiki`` for that, over summaries/sources only).
+
+    Args:
+        wiki_root: Absolute path to the wiki root directory.
+        kind: Restrict to ``"concept"`` or ``"entity"``; ``None`` returns both.
+
+    Returns:
+        Items sorted by kind, then slug. Empty list if the KB has neither
+        directory yet or both are empty.
+
+    Raises:
+        ValueError: *kind* is neither ``None``, ``"concept"``, nor ``"entity"``.
+    """
+    root = Path(wiki_root).resolve()
+    kinds = [kind] if kind else ["concept", "entity"]
+    for k in kinds:
+        if k not in _TAXONOMY_DIRS:
+            raise ValueError(f"Unknown kind {k!r}; expected 'concept' or 'entity'.")
+
+    items: list[TaxonomyItem] = []
+    for k in kinds:
+        directory = root / _TAXONOMY_DIRS[k]
+        if not directory.is_dir():
+            continue
+        for md_file in sorted(directory.glob("*.md")):
+            text = md_file.read_text(encoding="utf-8")
+            fm = frontmatter.parse(text)
+            brief = frontmatter.resolve_description(fm)
+            etype = None
+            if k == "entity":
+                etype = str(fm.get("type") or "").strip().lower() or "other"
+            items.append(
+                TaxonomyItem(
+                    kind=k,  # type: ignore[arg-type]  # validated against _TAXONOMY_DIRS above
+                    slug=md_file.stem,
+                    path=f"{_TAXONOMY_DIRS[k]}/{md_file.name}",
+                    brief=brief,
+                    type=etype,
+                )
+            )
+    return items
+
+
+def get_taxonomy_item(slug: str, wiki_root: str, kind: str | None = None) -> str:
+    """Read a persisted concept or entity page's full Markdown content.
+
+    Args:
+        slug: Page slug (filename without ``.md``), e.g. ``"attention"``.
+        wiki_root: Absolute path to the wiki root directory.
+        kind: ``"concept"`` or ``"entity"`` to disambiguate a same-named
+            slug; ``None`` checks ``concepts/`` first, then ``entities/``.
+
+    Returns:
+        Full file content, or a "not found" message if no match exists in
+        the requested (or either) directory.
+
+    Raises:
+        ValueError: *kind* is neither ``None``, ``"concept"``, nor ``"entity"``.
+    """
+    root = Path(wiki_root).resolve()
+    kinds = [kind] if kind else ["concept", "entity"]
+    for k in kinds:
+        if k not in _TAXONOMY_DIRS:
+            raise ValueError(f"Unknown kind {k!r}; expected 'concept' or 'entity'.")
+
+    for k in kinds:
+        path = (root / _TAXONOMY_DIRS[k] / f"{slug}.md").resolve()
+        if path.is_relative_to(root) and path.exists():
+            return path.read_text(encoding="utf-8")
+    return f"Taxonomy item not found: {slug}"
+
+
+def list_taxonomy(wiki_root: str, kind: str | None = None) -> str:
+    """Agent-facing text listing of persisted concept/entity pages.
+
+    Thin formatting wrapper around :func:`list_taxonomy_items` for use as an
+    LLM tool (see ``agent.query.build_query_agent``): one line per item with
+    its wikilink, entity type (if any), and one-line brief, so an LLM can
+    scan the whole taxonomy cheaply and pick a slug by meaning before calling
+    ``read_file`` on the matching page.
+
+    Args:
+        wiki_root: Absolute path to the wiki root directory.
+        kind: Restrict to ``"concept"`` or ``"entity"``; ``None`` returns both.
+
+    Returns:
+        One ``- [[path]] (type) — brief`` line per item, or a message if
+        none exist.
+    """
+    items = list_taxonomy_items(wiki_root, kind=kind)
+    if not items:
+        return "No concepts or entities found."
+
+    lines = []
+    for item in items:
+        wikilink = item.path[:-3] if item.path.endswith(".md") else item.path
+        type_suffix = f" ({item.type})" if item.type else ""
+        brief_suffix = f" — {item.brief}" if item.brief else ""
+        lines.append(f"- [[{wikilink}]]{type_suffix}{brief_suffix}")
+    return "\n".join(lines)
+
+
+def search_wiki(query: str, wiki_root: str, scope: list[str] | None = None, top_k: int = 5) -> str:
+    """Tiered full-text (BM25) search over summaries/sources wiki pages.
+
+    Hybrid retrieval helper: complements index.md/``list_taxonomy`` navigation
+    by surfacing pages whose one-line brief doesn't mention a specific buried
+    detail the query is looking for (a niche term, a figure, an exact fact).
+    Additive — use alongside, not instead of, index.md/``list_taxonomy``
+    navigation. Concepts/entities are never covered here — see
+    ``list_taxonomy``/``get_taxonomy_item`` for those (semantic browsing, not
+    keyword search).
+
+    Args:
+        query: Free-text search query (keywords or a natural-language question).
+        wiki_root: Absolute path to the wiki root directory.
+        scope: Restrict to a subset of ``fulltext_index.TIERED_SCOPES``
+            (``"briefs"``, ``"summaries"``, ``"sources"``); ``None`` searches
+            all three.
+        top_k: Maximum number of ranked results to return per tier.
+
+    Returns:
+        Ranked hits grouped by tier (wikilink, locator if any, title,
+        snippet), or a message indicating no matches were found, or an error
+        message if *scope* contains an invalid tier name.
+    """
+    from openkb.fulltext_index import TIERED_SCOPES, TieredWikiSearch
+
+    try:
+        results = TieredWikiSearch(wiki_root).search(query, scope=scope, top_k=top_k)
+    except ValueError as exc:
+        return str(exc)
+
+    sections = []
+    for tier in TIERED_SCOPES:
+        hits = results.get(tier)
+        if not hits:
+            continue
+        lines = [f"## {tier}"]
+        for i, hit in enumerate(hits, start=1):
+            wikilink = hit.path[:-3] if hit.path.endswith(".md") else hit.path
+            locator = f" [{hit.locator.kind} {hit.locator.value}]" if hit.locator else ""
+            lines.append(
+                f"{i}. [[{wikilink}]]{locator} — {hit.title} (score: {hit.score})\n   {hit.snippet}"
+            )
+        sections.append("\n".join(lines))
+    return "\n\n".join(sections) if sections else "No matching pages found."
+
+
 _MIME_TYPES = {
     ".png": "image/png",
     ".jpg": "image/jpeg",