Phase 2 Slice 1: File Tools and Real Embeddings

[SarkarShubhdeep]

Summary

Implements Phase 2 Slice 1 — file tools and real Ollama embeddings to complete deferred Phase 1 functionality:

• File tools: read_file and list_watched tools for reading bucket content and listing folder-watcher captured files
• Index tool: index_document for embedding content into the vector store
• Real embeddings: Replace random-vector placeholder in search_memory with actual Ollama nomic-embed-text embeddings
• Ingest pipeline: Collector → bucket → vector index pipeline for text file indexing

Changes

• packages/tools/src/runtime/embeddings.ts — shared Ollama embedding module
• packages/tools/src/runtime/file-tools.ts — read_file and list_watched tools
• packages/tools/src/runtime/index-document.ts — index_document tool
• packages/collector/src/ingest.ts — collector ingest pipeline
• packages/collector/src/folder-watcher.ts — wire ingestFile() for text files
• packages/tools/src/runtime/index.ts — register new tools in createJerryTools()
• packages/jerry-app/src/agent.ts — update agent instructions with new tools
• Unit tests for all new modules with mocked dependencies

Acceptance Criteria

• read_file and list_watched tools implemented and tested
• index_document tool working with real embeddings
• search_memory uses Ollama embeddings (no random vectors)
• Collector → footnote pipeline functional
• Unit tests pass; Ollama integration test documented
• Acceptance scenario verified manually for text files (drop .txt/.md → collector detects → index → search)
• Acceptance scenario verified manually (drop screenshot → index → search)

Dependencies

• Depends on: None (first slice)
• Blocks: Slice 2 (MCP Consume), Slice 3 (MCP Expose)

Testing

• Unit tests: All new tools have unit tests with mocked bucket/vectors/fetch — all pass
• Integration tests: Opt-in Ollama integration test available via JERRY_OLLAMA_TEST=1
• Manual verification: Pending (requires local stack with Ollama + nomic-embed-text model)

Notes

• Uses nomic-embed-text model (768 dimensions) — must be pulled locally: ollama pull nomic-embed-text
• Graceful fallback when Ollama unavailable — tools return informative error messages
• Acceptance scenario requires /v1/files and /v1/index HTTP endpoints on the worker (will be added as follow-up or documented as worker extension)

Made with Cursor

[SarkarShubhdeep]

Note: Screenshot / visual vector indexing is deferred

The folder watcher already captures image files (.png, .jpg, .gif, .webp) and pushes them as screenshot events, but embedding and indexing image content into the vector store is not implemented in this slice and is explicitly deferred.

Indexing images meaningfully requires a multimodal embedding model (e.g. llava, clip, or a vision-capable model), which is significantly larger than nomic-embed-text and not suitable for the current local-only runtime target.

This feature will be revisited when the BYO-cloud infrastructure option lands, where larger models can be hosted remotely and accessed without requiring them to run on the developer's machine.

For now, only text files (.txt, .md, .json) flow through the full collector → bucket → embed → vector index pipeline.