knowledge-vault-rs

📚 RAG in 100 lines | ⚡ 384-dim BGE-Micro embeddings | 🔍 SQLite-VSS vector search

Quick Start

use knowledge_vault::{KnowledgeVault, DocumentIndexer, LocalEmbedder};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Open vault with BGE-Micro (384 dimensions)
    let vault = KnowledgeVault::open("knowledge.db", 384)?;
    let embedder = LocalEmbedder::new(384)?;

    // Index documents
    let indexer = DocumentIndexer::new(vault.clone(), embedder);
    indexer.index_file("README.md").await?;

    // Search semantically
    let query_emb = embedder.embed("How do I search documents?").await?;
    let results = vault.search(&query_emb, 5)?;

    for result in results {
        println!("Score: {:.2} | {}", result.score, result.content);
    }

    Ok(())
}

Why knowledge-vault-rs?

Building RAG (Retrieval-Augmented Generation) systems is hard. You need to:

Split documents into chunks (while preserving context)
Generate embeddings (with the right model)
Store vectors (with efficient similarity search)
Watch for file changes (and auto-reindex)

knowledge-vault-rs handles all of this in ~100 lines of code.

The Problem

// ❌ Building RAG from scratch: 2000+ lines
let chunks = split_by_tokens(text, 512); // Lost context
let embeddings = openai::embed(&chunks).await?; // $0.10 per 1M tokens
let vector_db = Pinecone::new(api_key)?; // External dependency
vector_db.upsert(chunks).await?; // Network latency
// ... 500 lines later

The Solution

// ✅ knowledge-vault-rs: RAG in 100 lines
let vault = KnowledgeVault::open("db.sqlite", 384)?;
let embedder = LocalEmbedder::new(384)?; // Local BGE-Micro (free)
let indexer = DocumentIndexer::new(vault, embedder);

indexer.index_file("README.md").await?; // Auto-chunked
let results = vault.search(&query_emb, 5)?; // SQLite-VSS (local)

Features

Feature	Description	Status
Semantic Chunking	Split documents by paragraphs/sentences (not just tokens)	✅ Stable
BGE-Micro Embeddings	384-dimensional vectors (local, free, fast)	✅ Stable
SQLite-VSS	Approximate nearest neighbor search (no external DB)	✅ Stable
File Watcher	Auto-index on file changes (debounced)	✅ Stable
Thread-Safe Vault	`Arc<Mutex<Vault>>` for concurrent access	✅ Stable
Zero-Copy Search	No allocations during vector similarity	✅ Stable
Document Type Detection	Auto-detect code, markdown, text	✅ Stable
Deduplication	SHA256 content hashing	✅ Stable
Async Indexer	Channel-based (no lock contention)	✅ Stable
Hybrid Search	Vector + keyword filtering (planned)	🔄 Roadmap

Installation

[dependencies]
knowledge-vault = "0.1"
tokio = { version = "1.0", features = ["full"] }

Architecture

┌─────────────────────────────────────────────────────────────┐
│                      USER APPLICATION                        │
│                  (LLM, CLI, Desktop App)                    │
└────────────────────────┬────────────────────────────────────┘
                         │
        ┌────────────────┼────────────────┐
        ▼                ▼                ▼
┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│ File Watcher │  │  Indexer    │  │   Search     │
│ (notify)     │  │  (channel)  │  │  (VSS/cosine)│
└──────┬───────┘  └──────┬───────┘  └──────┬───────┘
       │                 │                 │
       └─────────────────┼─────────────────┘
                         ▼
              ┌──────────────────────┐
              │   KnowledgeVault     │
              │   (SQLite + VSS)     │
              └──────────────────────┘
                         │
        ┌────────────────┼────────────────┐
        ▼                ▼                ▼
┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│  Documents   │  │   Chunks     │  │  Embeddings  │
│  (metadata)  │  │ (text pieces)│  │ (384-dim)    │
└──────────────┘  └──────────────┘  └──────────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │  LocalEmbedder       │
              │  (BGE-Micro-v2)      │
              └──────────────────────┘

Component Overview

KnowledgeVault - SQLite-backed storage with VSS extension

Stores documents, chunks, and embeddings
Vector similarity search (cosine or VSS)
Thread-safe for concurrent access

DocumentIndexer - Channel-based async indexer

Background task processes indexing commands
Auto-chunking with semantic boundaries
Duplicate detection via SHA256

FileWatcher - Auto-index on file changes

Uses notify crate for file system events
Debounced to avoid redundant indexing
Extension filtering (code, docs, configs)

LocalEmbedder - BGE-Micro-v2 embeddings

384-dimensional vectors (compact, fast)
Runs locally (no API calls)
Falls back to SHA256 placeholder if model unavailable

VectorSearch - Semantic similarity search

Cosine similarity (pure Rust)
Optional SQLite-VSS for large datasets
Hybrid filtering (by type, doc ID, threshold)

Chunking Strategies

knowledge-vault-rs uses semantic chunking to preserve context:

1. Paragraph-Aware Chunking (Default)

Splits on paragraph boundaries first, then by size:

let options = ChunkOptions {
    chunk_size: 512,
    chunk_overlap: 50,
    respect_paragraphs: true,
    respect_sentences: true,
    ..Default::default()
};

let chunker = Chunker::with_options(options);
let chunks = chunker.chunk(document)?;

Result: Chunks rarely break sentences or paragraphs.

2. Sentence-Aware Chunking

Splits on sentence boundaries (good for narrative text):

let options = ChunkOptions {
    chunk_size: 512,
    chunk_overlap: 50,
    respect_sentences: true,
    respect_paragraphs: false, // Override paragraphs
    ..Default::default()
};

3. Size-Based Chunking (Fastest)

Simple character-based splitting (no semantic awareness):

let options = ChunkOptions {
    chunk_size: 512,
    chunk_overlap: 0,
    respect_sentences: false,
    respect_paragraphs: false,
    ..Default::default()
};

Overlap for Context

Chunk overlap ensures context isn't lost at boundaries:

ChunkOptions {
    chunk_size: 512,
    chunk_overlap: 50, // Last 50 tokens overlap
    ..Default::default()
}

Why overlap matters:

Without overlap: "The function takes a [break] [start] parameter..."
With overlap: "...takes a [start] parameter..." (context preserved)

Embeddings

BGE-Micro-v2 (Recommended)

384-dimensional embeddings, optimized for speed:

let embedder = LocalEmbedder::new(384)?;
let embedding = embedder.embed("Your text here").await?;

// Dimensions: 384
// Speed: ~1000 texts/sec (CPU)
// Size: ~1.5KB per embedding (384 * 4 bytes)

Why 384 dimensions?

Small enough for SQLite-VSS (fast indexing)
Large enough for semantic accuracy
4x smaller than OpenAI's ada-002 (1536 dims)

Placeholder Embeddings (Fallback)

SHA256-based embeddings (for testing):

let embedder = PlaceholderEmbedder::new(384);
let embedding = embedder.embed("Your text here").await?;

// Deterministic but not semantically meaningful
// Use only for development/testing

Vector Search

Basic Similarity Search

let embedder = LocalEmbedder::new(384)?;
let query_emb = embedder.embed("How do I index files?").await?;

let results = vault.search(&query_emb, 5)?;

for result in results {
    println!("{:.2} | {}", result.score, result.content);
}

Output:

0.89 | You can index files using the DocumentIndexer...
0.76 | The indexer supports channel-based async indexing...
0.65 | FileWatcher provides automatic reindexing...

Advanced Search Options

use knowledge_vault::{SearchOptions, VectorSearch};

let options = SearchOptions {
    limit: 10,
    threshold: 0.7, // Minimum similarity score
    doc_types: Some(vec!["markdown".to_string(), "code".to_string()]),
    doc_ids: Some(vec!["doc_abc123".to_string()]),
    include_content: true,
};

let search = VectorSearch::new(&vault);
let results = search.search(&query_emb, &options).await?;

Search Performance

Mode	Dataset Size	Query Time	Accuracy
Cosine	< 1K chunks	< 10ms	100% (exact)
Cosine	1K-10K	10-50ms	100% (exact)
SQLite-VSS	10K-100K	< 20ms	~95% (approx)
SQLite-VSS	100K-1M	20-100ms	~95% (approx)

Recommendation: Use cosine for < 10K chunks, SQLite-VSS for larger.

File Watching

Auto-index documents when they change:

use knowledge_vault::{FileWatcher, WatchConfig};
use tokio::sync::mpsc;

let config = WatchConfig {
    directories: vec!["./docs".into(), "./src".into()],
    extensions: Some(vec!["md".to_string(), "rs".to_string()]),
    debounce: Duration::from_secs(2),
    recursive: true,
    ..Default::default()
};

// Get channel from indexer
let (indexer, _handle) = DocumentIndexer::new(vault.clone(), embedder);
let command_tx = indexer.command_sender();

// Create watcher with auto-indexing
let watcher = FileWatcher::with_auto_index(config, command_tx)?;
watcher.start()?;

Events handled:

File created → Indexed
File modified → Re-indexed (SHA256 diff)
File deleted → Removed from vault
File renamed → Path updated

Performance

Benchmarks (M1 Pro, 512-token chunks)

Operation	Throughput	Latency
Chunking	10MB/sec	~1ms per chunk
Embedding (BGE-Micro)	1000 texts/sec	~1ms per text
Insertion	500 chunks/sec	~2ms per chunk
Search (1K chunks)	100 queries/sec	~10ms
Search (10K chunks)	20 queries/sec	~50ms
Search (100K chunks, VSS)	50 queries/sec	~20ms

Memory Usage

Component	Memory
Vault overhead	~5MB
Per document	~1KB (metadata)
Per chunk	~512 bytes (text) + 1.5KB (embedding)
10K chunks	~20MB
100K chunks	~200MB

Storage Size

Data	Size per 1K chunks
Chunks (512 tokens)	~500KB
Embeddings (384-dim)	~1.5MB
Metadata	~100KB
Total	~2.1MB

Thread Safety

The vault is designed for concurrent access:

use std::sync::Arc;
use tokio::sync::Mutex;

// Share vault across tasks
let vault = Arc::new(Mutex::new(KnowledgeVault::open("db.sqlite", 384)?));

// Spawn concurrent indexers
for file in files {
    let vault_clone = vault.clone();
    tokio::spawn(async move {
        let indexer = DocumentIndexer::new(vault_clone, embedder);
        indexer.index_file(file).await?;
    });
}

Thread safety guarantees:

SQLite connection is protected by Mutex
Multiple readers can access concurrently
Writers are serialized (one at a time)
Channel-based indexer prevents lock contention

Examples

See the examples/ directory for complete, runnable examples:

Example	Description
basic.rs	Hello world: Index a file and search
chunking.rs	Chunking strategies and options
embeddings.rs	Generate and compare embeddings
search.rs	Advanced search with filtering
watcher.rs	Auto-indexing file watcher

Run examples:

cargo run --example basic
cargo run --example chunking
cargo run --example embeddings
cargo run --example search
cargo run --example watcher

API Documentation

KnowledgeVault

// Open or create vault
let vault = KnowledgeVault::open(path: &Path, dimensions: u32)?;

// In-memory vault (for testing)
let vault = KnowledgeVault::in_memory()?;

// Search by embedding
let results = vault.search(query_embedding: &[f32], limit: usize)?;

// Get documents
let doc = vault.get_document(id: &str)?;
let docs = vault.list_documents(limit: usize)?;

// Delete
vault.delete_document(id: &str)?;

// Statistics
let stats = vault.stats()?;

DocumentIndexer

// Create indexer (spawns background task)
let (indexer, handle) = DocumentIndexer::new(
    vault: Arc<Mutex<KnowledgeVault>>,
    embedder: Arc<Mutex<LocalEmbedder>>,
    config: IndexerConfig,
);

// Index a file
indexer.index_file(path: PathBuf).await?;

// Index content directly
indexer.index_content(
    content: String,
    title: String,
    doc_type: String,
    path: Option<PathBuf>,
).await?;

// Index directory
indexer.index_directory(
    path: PathBuf,
    extensions: Option<Vec<String>>,
).await?;

// Get channel sender (for FileWatcher)
let tx = indexer.command_sender();

// Shutdown gracefully
handle.shutdown().await;

LocalEmbedder

// Create embedder
let embedder = LocalEmbedder::new(dimensions: u32)?;

// Single embedding
let embedding = embedder.embed(text: &str).await?;

// Batch embeddings (faster)
let embeddings = embedder.embed_batch(&["text1", "text2"]).await?;

// Get dimensions
let dims = embedder.dimensions();

FileWatcher

// Create config
let config = WatchConfig {
    directories: vec![PathBuf::from("./docs")],
    extensions: Some(vec!["md".to_string()]),
    debounce: Duration::from_secs(2),
    recursive: true,
    ..Default::default()
};

// Create watcher with auto-indexing
let watcher = FileWatcher::with_auto_index(
    config,
    indexer.command_sender(),
)?;

// Start watching (blocks)
watcher.start()?;

ChunkOptions

let options = ChunkOptions {
    chunk_size: 512,           // Target size in tokens
    chunk_overlap: 50,         // Overlap between chunks
    min_chunk_size: 100,       // Don't create smaller chunks
    respect_sentences: true,   // Don't break sentences
    respect_paragraphs: true,  // Don't break paragraphs (overrides sentences)
};

let chunker = Chunker::with_options(options);
let chunks = chunker.chunk(text)?;

SearchOptions

let options = SearchOptions {
    limit: 10,                              // Max results
    threshold: 0.7,                         // Min similarity (0.0-1.0)
    doc_types: Some(vec!["markdown".into()]), // Filter by type
    doc_ids: Some(vec!["doc_123".into()]),  // Filter by ID
    include_content: true,                  // Include chunk text
};

Error Handling

All operations return KnowledgeResult<T>:

use knowledge_vault::KnowledgeError;

match indexer.index_file("README.md").await {
    Ok(result) => {
        println!("Indexed {} chunks", result.chunk_count);
    },
    Err(KnowledgeError::NotFound(msg)) => {
        eprintln!("File not found: {}", msg);
    },
    Err(KnowledgeError::InvalidFormat(msg)) => {
        eprintln!("Invalid format: {}", msg);
    },
    Err(e) => {
        eprintln!("Error: {}", e);
    }
}

Best Practices

1. Use Channel-Based Indexer

❌ Avoid (blocks while embedding):

for file in files {
    indexer.index_file(file).await?; // Sequential, slow
}

✅ Prefer (concurrent):

let (indexer, _handle) = DocumentIndexer::new(vault, embedder);

// All files processed concurrently
for file in files {
    indexer.index_file(file).await?; // Non-blocking
}

2. Respect Semantic Boundaries

❌ Avoid (breaks sentences):

ChunkOptions {
    respect_sentences: false,
    respect_paragraphs: false,
    ..Default::default()
}

✅ Prefer (preserves context):

ChunkOptions {
    respect_paragraphs: true, // Try this first
    respect_sentences: true,  // Fallback
    ..Default::default()
}

3. Use Overlap for Context

❌ Avoid (loses boundary context):

ChunkOptions {
    chunk_overlap: 0,
    ..Default::default()
}

✅ Prefer (50-100 tokens overlap):

ChunkOptions {
    chunk_overlap: 50, // 10% of chunk_size
    ..Default::default()
}

4. Filter Search Results

❌ Avoid (too many low-quality results):

let results = vault.search(&query_emb, 100)?;

✅ Prefer (threshold filtering):

let options = SearchOptions {
    limit: 10,
    threshold: 0.7, // Only relevant results
    ..Default::default()
};

let results = search.search(&query_emb, &options).await?;

5. Deduplicate Content

❌ Avoid (indexes duplicates):

let config = IndexerConfig {
    skip_duplicates: false, // Default is true
    ..Default::default()
};

✅ Prefer (SHA256 deduplication):

let config = IndexerConfig {
    skip_duplicates: true, // Automatic deduplication
    ..Default::default()
};

Integration Examples

With LLM (RAG Pipeline)

async fn rag_question(
    vault: &KnowledgeVault,
    embedder: &LocalEmbedder,
    question: &str,
) -> Result<String, Box<dyn std::error::Error>> {
    // Embed question
    let query_emb = embedder.embed(question).await?;

    // Retrieve relevant chunks
    let results = vault.search(&query_emb, 3)?;

    // Build context
    let context = results
        .iter()
        .map(|r| r.content.as_str())
        .collect::<Vec<_>>()
        .join("\n\n");

    // Query LLM with context
    let prompt = format!(
        "Context:\n{}\n\nQuestion: {}\n\nAnswer:",
        context, question
    );

    let response = llm_query(&prompt).await?;
    Ok(response)
}

With CLI Tool

use clap::{Parser, Subcommand};

#[derive(Parser)]
struct Cli {
    #[command(subcommand)]
    command: Command,
}

#[derive(Subcommand)]
enum Command {
    /// Index a file
    Index { path: String },
    /// Search vault
    Search { query: String },
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cli = Cli::parse();
    let vault = KnowledgeVault::open("vault.db", 384)?;
    let embedder = LocalEmbedder::new(384)?;

    match cli.command {
        Command::Index { path } => {
            let indexer = DocumentIndexer::new(vault, embedder);
            indexer.index_file(path.into()).await?;
            println!("Indexed: {}", path);
        },
        Command::Search { query } => {
            let emb = embedder.embed(&query).await?;
            let results = vault.search(&emb, 5)?;

            for result in results {
                println!("{:.2} | {}", result.score, result.content);
            }
        },
    }

    Ok(())
}

With Web Server (Axum)

use axum::{extract::Json, routing::post, Router};
use serde::{Deserialize, Serialize};

#[derive(Deserialize)]
struct SearchRequest {
    query: String,
}

#[derive(Serialize)]
struct SearchResponse {
    results: Vec<ChunkResult>,
}

async fn search_handler(
    Json(req): Json<SearchRequest>,
) -> Result<Json<SearchResponse>, StatusCode> {
    let vault = vault_from_state();
    let embedder = embedder_from_state();

    let emb = embedder.embed(&req.query).await
        .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;

    let results = vault.search(&emb, 5)
        .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;

    Ok(Json(SearchResponse { results }))
}

#[tokio::main]
async fn main() {
    let app = Router::new()
        .route("/search", post(search_handler));

    axum::Server::bind(&"0.0.0.0:3000".parse()?)
        .serve(app.into_make_service())
        .await?;
}

Roadmap

v0.2 (Q1 2026)

Hybrid search (vector + keyword BM25)
Reranking (cross-encoder)
Query expansion
Metadata filtering

v0.3 (Q2 2026)

Distributed indexing (rayon)
Compression (quantization)
Incremental embeddings
Graph-based chunking

v0.4 (Q3 2026)

Multi-lingual support
Image embeddings (CLIP)
Audio embeddings (wav2vec)
Video indexing

Used By

SuperInstance - Tripartite AI system using knowledge-vault for RAG
synesis-knowledge - Original extraction source
Your project here!

Migration from synesis-knowledge

See MIGRATION_GUIDE.md for details.

# Old
[dependencies]
synesis-knowledge = "0.2"

# New
[dependencies]
knowledge-vault = "0.1"

API is 95% compatible, mainly renamed imports:

// Old
use synesis_knowledge::{KnowledgeVault, DocumentIndexer};

// New
use knowledge_vault::{KnowledgeVault, DocumentIndexer};

License

MIT OR Apache-2.0

Contributing

Contributions welcome! Please read CONTRIBUTING.md for details.

Acknowledgments

Built with:

rusqlite - SQLite bindings
sqlite-vss - Vector search extension
notify - File system watching
tokio - Async runtime
unicode-segmentation - Text segmentation

Special thanks to the authors of BGE-Micro-v2 for the compact embeddings model.

Index everything. Search anything. Retrieve context.

Name		Name	Last commit message	Last commit date
.github		.github
benches		benches
examples		examples
src		src
.gitignore		.gitignore
CI_CDR_SETUP_REPORT.md		CI_CDR_SETUP_REPORT.md
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
STATUS.md		STATUS.md
clippy.toml		clippy.toml
rustfmt.toml		rustfmt.toml

Latest commit

History

Folders and files

Repository files navigation

knowledge-vault-rs

Quick Start

Why knowledge-vault-rs?

The Problem

The Solution

Features

Installation

Architecture

Component Overview

Chunking Strategies

1. Paragraph-Aware Chunking (Default)

2. Sentence-Aware Chunking

3. Size-Based Chunking (Fastest)

Overlap for Context

Embeddings

BGE-Micro-v2 (Recommended)

Placeholder Embeddings (Fallback)

Vector Search

Basic Similarity Search

Advanced Search Options

Search Performance

File Watching

Performance

Benchmarks (M1 Pro, 512-token chunks)

Memory Usage

Storage Size

Thread Safety

Examples

API Documentation

KnowledgeVault

DocumentIndexer

LocalEmbedder

FileWatcher

ChunkOptions

SearchOptions

Error Handling

Best Practices

1. Use Channel-Based Indexer

2. Respect Semantic Boundaries

3. Use Overlap for Context

4. Filter Search Results

5. Deduplicate Content

Integration Examples

With LLM (RAG Pipeline)

With CLI Tool

With Web Server (Axum)

Roadmap

v0.2 (Q1 2026)

v0.3 (Q2 2026)

v0.4 (Q3 2026)

Used By

Migration from synesis-knowledge

License

Contributing

Acknowledgments

About

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages