feat(rag): RAG v0 — docs corpus retrieval, agon rag CLI, ProjectConte…#210
Merged
Conversation
…xt MCP tool First shippable RAG slice (6-engine brainstorm design, label rag-first-slice): docs-only corpus (root *.md + docs/) → markdown-aware chunking (heading boundaries, fenced code atomic, ~512tok + 64 overlap) → offline MiniLM embeddings via the new @kernlang/agon-dedup embedder.py sidecar (L2-normalized, persistent model cache under ~/.agon/cache/fastembed) → per-repo persistent index .agon/rag/<corpus-hash>/ (manifest + chunks.jsonl + Float32 embeddings.bin, corpus-hash reuse short-circuit) → cosine top-k → fail-closed grounding with [n] source Lx-y citations. Surfaces: - agon rag index|query|stats (citty action dispatch; --json, --force) - ProjectContext MCP tool (cited blocks + grounded flag; ungrounded queries return NO context text — never dressed-up weak hits) All new logic in KERN (core/src/kern/rag/, mcp project-context.kern, cli commands/rag.kern); thin TS facades only. 14 unit tests (chunker fence-atomicity, retriever floor, grounding fail-closed, store round-trip, mocked sidecar boundary). Review-pass fixes (agon review: codex blocking + important; agy clean): - ProjectContext no longer accepts an arbitrary root — an MCP client could point the retriever (reads docs, WRITES .agon/rag/) outside the repo boundary; tool now serves the server cwd only - queryRag reloads with the corpus hash the build actually produced, closing the hash-drift TOCTOU between hashCorpus and buildRagIndex KERN-GAP filed: doc text containing */ breaks the emitted JSDoc block (corpus.kern). Deferred (AGON.md ## RAG Roadmap): --ground Cesar wire-up, session corpus, review grounding, network exposure, live reindex, hybrid search, federation, citation UX. Verified live: index 6 files → 118 chunks 1.6s cold / 21ms warm; grounded query cites AGON.md L16-27 (0.63); off-topic fail-closed; MCP tool over real JSONRPC stdio. Gate: kern check clean ×6, build green, 2938/2938 tests. ⚔️ Forged by [Agon](https://github.com/KERNlang/agon) Co-Authored-By: agon (KERN) <292465531+KERN-Agon@users.noreply.github.com>
…hFile) sidecar.py / classifier.py / history-search.py still used fastembed's default tmpdir cache; macOS pruned the model snapshot and every bid-dedup pass crashed with onnxruntime NO_SUCHFILE (seen live in brainstorm cesar-mode-rag). Apply the same ~/.agon/cache/fastembed pinning that embedder.py shipped with. ⚔️ Forged by [Agon](https://github.com/KERNlang/agon) Co-Authored-By: agon (KERN) <292465531+KERN-Agon@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…xt MCP tool
First shippable RAG slice (6-engine brainstorm design, label rag-first-slice): docs-only corpus (root *.md + docs/) → markdown-aware chunking (heading boundaries, fenced code atomic, ~512tok + 64 overlap) → offline MiniLM embeddings via the new @kernlang/agon-dedup embedder.py sidecar (L2-normalized, persistent model cache under ~/.agon/cache/fastembed) → per-repo persistent index .agon/rag// (manifest + chunks.jsonl + Float32 embeddings.bin, corpus-hash reuse short-circuit) → cosine top-k → fail-closed grounding with [n] source Lx-y citations.
Surfaces:
All new logic in KERN (core/src/kern/rag/, mcp project-context.kern, cli commands/rag.kern); thin TS facades only. 14 unit tests (chunker fence-atomicity, retriever floor, grounding fail-closed, store round-trip, mocked sidecar boundary).
Review-pass fixes (agon review: codex blocking + important; agy clean):
KERN-GAP filed: doc text containing */ breaks the emitted JSDoc block (corpus.kern).
Deferred (AGON.md ## RAG Roadmap): --ground Cesar wire-up, session corpus, review grounding, network exposure, live reindex, hybrid search, federation, citation UX.
Verified live: index 6 files → 118 chunks 1.6s cold / 21ms warm; grounded query cites AGON.md L16-27 (0.63); off-topic fail-closed; MCP tool over real JSONRPC stdio. Gate: kern check clean ×6, build green, 2938/2938 tests.
⚔️ Forged by Agon