Memory

Cargo.toml

@@ -36,6 +36,9 @@ hex = "0.4"
 directories = "5"
 log = "0.4"
 env_logger = "0.11"
+hnsw_rs = "0.3.4"
+anndists = "0.1.4"
+sled = "0.34"
 
 [build-dependencies]
 sha2 = "0.10"

README.md

@@ -165,6 +165,8 @@ You can also initialize ai-mate at specific agent:
 ai-mate --agent planner
 ```
 
+If you want to use OpenTTS, start the docker service first: `docker run --rm --platform=linux/amd64 -p 5500:5500 synesthesiam/opentts:all` (it will pull the image the first time). Adjust the platform as needed depending on your hardware. 
+
 If you need help:
 
 ```

spec/acceleration.txt

@@ -21,4 +21,4 @@
 -----------------------------------------------------------------
   ARM64:
          Metal support + Accelerate + eigen
------------------------------------------------------------------
+-----------------------------------------------------------------

spec/memory.md

@@ -0,0 +1,95 @@
+# Memory System Overview
+
+The **Memory** module is a lightweight knowledge‑base that stores *knowledge units* together with their embeddings.  It is used by the AI‑assistant to recall past events and facts.
+
+## 1. Data Model
+
+- **KnowledgeUnit** – a semantic triple consisting of `subject`, `predicate`, `object`, optional `location` and a timestamp.
+- **VecKnowledgeUnit** – the unit plus a 128‑dimensional embedding vector.
+- **Memory** – holds the HNSW graph (`hnsw`), a map of IDs to `VecKnowledgeUnit` (`index_map`), and three *inverted indexes* (subject, predicate, object) that map a value to the set of IDs that contain it.
+
+## 2. Persistence
+
+The entire state is stored in a single **sled** database file named `memory.json` (the name is a bit of a misnomer – it is a sled DB, not a plain JSON file).  The DB contains one key:
+
+```
+"memory" => <bytes>
+```
+
+The bytes are a JSON object with the following fields:
+
+|-------------------------------------------------------------------------------------|
+| Field               | Type                       | Meaning                          |
+|---------------------|----------------------------|----------------------------------|
+| `next_id`           | u64                        | Next available ID for a new unit |
+| `units`             | map<u64, VecKnowledgeUnit> | All stored units, keyed by ID    |
+| `subject_index`     | map<string, Vec<u64>>      | Inverted index for subjects      |
+| `predicate_index`   | map<string, Vec<u64>>      | Inverted index for predicates    |
+| `object_index`      | map<string, Vec<u64>>      | Inverted index for objects       |
+---------------------------------------------------------------------------------------
+
+When `Memory::save_to_file` is called it serialises this structure with `serde_json::to_vec` and writes it to sled.  Loading reverses the process: the bytes are read, deserialised, and the HNSW graph is rebuilt from the embeddings.
+
+## 3. Storing a Unit
+
+```
+let id = memory.next_id;
+memory.index_map.insert(id, VecKnowledgeUnit{embedding, knowledge: unit.clone()});
+memory.hnsw.insert((&embedding, id));
+memory.next_id += 1;
+// update inverted indexes
+subject_index.entry(unit.subject).or_default().push(id);
+predicate_index.entry(unit.predicate.name).or_default().push(id);
+object_index.entry(unit.object).or_default().push(id);
+```
+
+After the in‑memory state is updated, `save_to_file` persists the data.
+
+## 4. Querying
+
+### a. Full‑text / Embedding query
+
+`Memory::query(query, k, ef_search)` performs a vector similarity search:
+1. The query string is turned into a 128‑dimensional vector via `embed_text` (hash‑based word frequency).
+2. HNSW returns the `k` nearest neighbour IDs.
+3. The corresponding `KnowledgeUnit`s are returned.
+4. A final filter keeps only results that contain any word from the query (case‑insensitive, alphanumeric only).
+
+### b. Inverted‑index lookup
+
+Functions `get_by_subject`, `get_by_predicate`, `get_by_object`, and `get_by_location` simply look up the relevant ID list in the inverted index and fetch the units.  This is O(1) to get the list plus O(1) per result.
+
+## 5. CLI Commands
+
+The binary exposes four new flags:
+
+```
+--get-memories-by-subject <subject>
+--get-memories-by-predicate <predicate>
+--get-memories-by-object <object>
+--get-memories-by-location <location>
+```
+
+Each flag loads `memory.json`, calls the appropriate `get_by_*` method, and prints each unit’s context via `Memory::build_context_from_units`.
+
+## 6. Summary
+
+* Units are stored with an embedding and an ID.
+* All data is persisted as a single sled database entry containing JSON.
+* Retrieval can be done either by embedding similarity or by fast inverted‑index lookups.
+* The CLI provides convenient commands for inspecting and querying the memory.
+
+This design keeps the runtime lightweight while enabling efficient queries over thousands of units.
+
+## 7. LLM integration
+
+Store memories:
+
+1. when sending llm prompt, provide "story_memory" tool with available predicates
+2. execute tool calls (new memories to store) from LLM response
+
+Retrieve memories:
+
+1. Before submitting prompt, query memories
+2. Collect results (knowledge units) and produce text readable sentences using build_context_from_units(&retrieved_units);
+3. Provide context to the llm

spec/messaging.txt

@@ -0,0 +1,55 @@
+Threads and the channels they use
+
+------------------------------------------------------------------------------------
+ Main   (src/main.rs)
+------------------------------------------------------------------------------------
+- Sends the global stop signal through stop_all_tx.
+- Sends raw microphone audio to the recording thread via tx_rec.
+- Sends recognized utterances to the conversation thread via tx_utt.
+- Sends raw microphone audio to the conversation thread via conv_tx (for synchronous processing).
+- Sends playback audio to the playback thread through tx_play.
+- Sends playback‑stop requests to the recording and playback threads via stop_play_tx.
+- Sends the playback‑stop receiver to the playback thread via stop_play_rx.
+
+------------------------------------------------------------------------------------
+ UI thread   (src/ui.rs)
+------------------------------------------------------------------------------------
+- Receives the global stop signal on stop_all_rx.
+- Reads the shared status_line, print_lock, and peak to update the terminal.
+
+------------------------------------------------------------------------------------
+ Playback thread   (src/playback.rs)
+------------------------------------------------------------------------------------
+- Receives the global stop signal on stop_all_rx.
+- Receives playback audio chunks on rx_play.
+- Receives a playback‑stop signal on stop_play_rx.
+- Reads control flags: playback_active, gate_until_ms, paused.
+- Reads UI state (ui) and updates it.
+- Reads and updates the shared volume.
+
+------------------------------------------------------------------------------------
+ Record thread   (src/record.rs)
+------------------------------------------------------------------------------------
+- Receives the global stop signal on stop_all_rx.
+- Reads the interrupt counter via interrupt_counter.
+- Reads and updates shared state: volume, recording_paused, and peak.
+- Sends microphone audio to the conversation thread via conv_tx.
+- Sends recognized utterance audio to the conversation thread via tx_utt.
+- Sends a playback‑stop request on stop_play_tx.
+
+------------------------------------------------------------------------------------
+ Conversation thread   (src/conversation.rs)
+------------------------------------------------------------------------------------
+- Receives the global stop signal on stop_all_rx.
+- Receives utterance chunks from the record thread on rx_utt.
+- Sends processed speech audio to the playback thread via tx_play.
+- Sends a global stop signal on stop_all_tx when needed.
+
+------------------------------------------------------------------------------------
+ Keyboard thread    (src/keyword.rs)
+------------------------------------------------------------------------------------
+- Receives the global stop signal on stop_all_rx.
+- Sends the global stop signal via stop_all_tx.
+- Reads control flags: paused, playback_active.
+- Reads the recording‑paused flag via recording_paused.
+- Reads configuration: voice, args.tts, args.language.

src/config.rs

@@ -100,6 +100,29 @@ pub struct Args {
 
   #[arg(long)]
   pub ptt: Option<bool>,
+
+  #[arg(long, action=clap::ArgAction::SetTrue)]
+  pub get_memories: bool,
+
+  /// Get memories by subject
+  #[arg(long, value_name="SUBJECT", action=clap::ArgAction::Set)]
+  pub get_memories_by_subject: Option<String>,
+
+  /// Get memories by predicate
+  #[arg(long, value_name="PREDICATE", action=clap::ArgAction::Set)]
+  pub get_memories_by_predicate: Option<String>,
+
+  /// Get memories by object
+  #[arg(long, value_name="OBJECT", action=clap::ArgAction::Set)]
+  pub get_memories_by_object: Option<String>,
+
+  /// Get memories by location
+  #[arg(long, value_name="LOCATION", action=clap::ArgAction::Set)]
+  pub get_memories_by_location: Option<String>,
+
+  /// Query memory using embeddings and print results
+  #[arg(long, value_name="QUERY", action=clap::ArgAction::Set)]
+  pub query_memory: Option<String>,
 }
 
 // internal static values