feat(cli,agent): expose tiered search + taxonomy via CLI and wire into query/chat agent - #261
Open
sebastianbraun25 wants to merge 5 commits into
Open
Conversation
added 5 commits
August 28, 2026 16:25
Adds a dependency-free BM25 full-text index (openkb/fulltext_index.py) over concepts/entities/summaries pages, exposed as a new search_wiki tool alongside index.md-driven navigation in build_query_agent. Additive hybrid retrieval: surfaces pages whose one-line index summary omits a buried detail, without replacing existing navigation. Resolves VectifyAI#233.
…axonomy accessors - fulltext_index.py: extract shared _BM25Scorer from WikiFullTextIndex (no behavior change), add Locator (line/page) on SearchHit, add TieredWikiSearch with three independent tiers over summaries/ (briefs + full body) and sources/ (whole-file .md + per-page PageIndex .json, never the whole long doc as one BM25 unit). - frontmatter.py: add resolve_description()/body_only() shared helpers (kept separate from agent.compiler._resolve_description, which is under active unrelated development). - agent/tools.py: add list_taxonomy_items()/get_taxonomy_item() for semantic browsing of persisted concepts/entities (pending candidates in PendingTopicsStore are structurally excluded). - No wiring into CLI/MCP/query-agent yet (follow-up PRs); WikiFullTextIndex and agent.tools.search_wiki keep their existing signature/behavior.
…o query/chat agent - cli.py: new 'openkb list-taxonomy [--kind concept|entity] [--json]' and 'openkb search <query> [--scope briefs,summaries,sources] [--top-k N] [--json]' commands. - agent/tools.py: search_wiki now searches the new tiered briefs/summaries/sources index instead of the old combined concepts+entities+summaries index (concepts/entities move to the new list_taxonomy tool - semantic browsing, not keyword search); new list_taxonomy() text-formatting wrapper over list_taxonomy_items(). - agent/query.py (+ chat.py via tool inheritance): wires list_taxonomy and the retiered search_wiki in as agent tools; search strategy instructions updated to browse taxonomy first, then use scope-restricted search_wiki as a keyword fallback. - README.md: updated hybrid-retrieval paragraph and command table. - Intentional behavior change to agent.tools.search_wiki (scope param, concepts/entities out of scope, output grouped by tier) - safe since VectifyAI#234/VectifyAI#259, which introduced it, are not yet merged upstream; existing tests updated to the new contract.
sebastianbraun25
pushed a commit
to sebastianbraun25/OpenKB
that referenced
this pull request
Sep 11, 2026
- skills/openkb/SKILL.md: 'See what's available' now leads with list_taxonomy (MCP) / 'openkb list-taxonomy' (CLI) before falling back to reading the full index.md. - 'Read content' table adds search_wiki (MCP) / 'openkb search' (CLI) rows ahead of the existing grep fallback, with a note on why BM25 ranking beats raw grep occurrence count. - 'When the KB doesn't have the answer' and the openkb-query guidance updated to reference the new search options alongside grep. - Documentation-only change; no behavior change to the underlying tools/CLI/MCP server (VectifyAI#259/VectifyAI#261/VectifyAI#263).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This PR was created in collaboration between a human and AI: implementation, tests, and
PR text were created by an AI assistant under the guidance and review of the human author.
Problem
search_wiki/list_taxonomy_items/get_taxonomy_item(#258/#259) are only reachable frominside the
openkb query/openkb chatagent's Python process. There is no way for ascript, a CI job, or an external AI assistant (GitHub Copilot, Claude Code, etc.) that isn't
running the OpenKB agent itself to browse the taxonomy or run the tiered search — the only
options today are re-implementing keyword search with
grep, or paying for a fullopenkb queryLLM round-trip just to get a list of candidate pages. Separately, thequery/chat agent itself still navigated concepts/entities via ad-hoc
index.mdreadsrather than the new, purpose-built taxonomy tools.
Solution / Changes
openkb/cli.py:openkb list-taxonomy [--kind concept|entity] [--json]— lists persistedconcept/entity pages with their one-line briefs (semantic browsing, not keyword
search).
openkb search <query> [--scope briefs,summaries,sources] [--top-k N] [--json]— tieredBM25 search over summaries/sources, grouped by tier, printing each hit's
line/page locator when present.
openkb/agent/tools.py:search_wikinow searches the tieredbriefs/summaries/sourcesindex (newscopeparam) instead of the old combined
concepts+entities+summariesindex — conceptsand entities are no longer keyword-searched at all, only browsed via
list_taxonomy.This is an intentional behavior/contract change, safe because feat(agent): hybrid BM25 search_wiki tool for query/chat agent #234/feat(search): tiered BM25 search (briefs/summaries/sources) + taxonomy accessors #258/feat(search): tiered BM25 search (briefs/summaries/sources) + taxonomy accessors #259 (which
introduced/extended
search_wiki) are not yet merged upstream, so there is no releasedconsumer to break. Existing
TestSearchWikitests updated to the new contract(concepts/entities excluded, output grouped by tier,
scopevalidation).list_taxonomy(wiki_root, kind=None)— text-formatting wrapper overlist_taxonomy_itemsfor LLM tool consumption.openkb/agent/query.py(andchat.py, which extends the query agent's tools):build_query_agentnow exposes 5 tools:read_file,get_page_content,list_taxonomy,search_wiki,get_image.list_taxonomyfor concept/entityquestions (pick a slug by meaning, then
read_filethe page), fall back tosearch_wiki(query, scope)for buried details in summaries/sources, narrowingscopeto
["sources"]for source-only facts (author, date, exact field name) a generatedsummary would likely omit.
README.md: updated hybrid-retrieval paragraph and the wiki-commands table with the twonew CLI commands.
Testing
tests/test_agent_tools.py:TestSearchWikiupdated (concepts/entities no longermatched, tier-grouped output,
scopeparam, locator in output, invalid-scope errormessage) + new
TestListTaxonomy(3 tests: wikilink formatting,kindfilter, emptycase).
tests/test_query.py: tool count/name assertions updated for the new 5-tool set(
list_taxonomyadded).list-taxonomy(plain +--json+--kind),search(plain +--json+--scope+ invalid-scope error path) all verified against a throwaway KB.ruff check,ruff format --check,mypy openkb,pytestall green — full suite: 1274passed (18 pre-existing, environment-specific Windows failures unrelated to this change
are unaffected and unchanged in count).
Dependencies
Depends on #259(feat/issue-233-tiered-search) — this PR is built on top of thatbranch (
TieredWikiSearch,list_taxonomy_items/get_taxonomy_item), which it wiresinto the CLI and the query/chat agent without changing their internal behavior. The diff
below includes feat(search): tiered BM25 search (briefs/summaries/sources) + taxonomy accessors #259's (and thus feat(agent): hybrid BM25 search_wiki tool for query/chat agent #234's) commits; once feat(search): tiered BM25 search (briefs/summaries/sources) + taxonomy accessors #259 merges first, this PR's diff
will shrink to just this PR's own commit.
Issues
Resolves #260