🧠 MemoryEngine

A standalone, framework-agnostic semantic memory engine and TypeScript SDK for intelligent ingestion, hybrid retrieval, and forgetting-curve resurfacing.

---

⚡ Why MemoryEngine?

Humans remember ideas, metaphors, and analogies — not filenames or exact keywords.

You don't remember saving video_98412_transcript.txt. You remember "there was a video where the guy compared salary negotiation to buying real estate."

Every "save for later" feature turns into a digital graveyard because:

1. Storage is trivial; retrieval is broken. Databases index exact keywords, while human recall operates on fuzzy conceptual associations.
2. Infinite archives cause fatigue. Unreviewed saved items decay in human memory unless intelligently resurfaced at the optimal moment.

MemoryEngine is pure infrastructure designed to solve this. It has no UI — it is an importable, high-performance SDK that turns raw text from any source (Instagram reels, podcasts, PDFs, voice notes, bookmarks) into queryable, connected, and serendipitously resurfaced memories.

---

🚀 Quickstart

Installation

npm install @paritosh31/memory-engine

5-Line Usage

import { MemoryEngine } from "@paritosh31/memory-engine";

const engine = new MemoryEngine();

// 1. Ingest any text
await engine.ingest({
  id: "reel_1042",
  title: "Salary Negotiation Tactics",
  text: "When negotiating salary, never make it adversarial. Think of it like buying a house where both parties need peace of mind."
});

// 2. Semantic Search by conceptual analogy
const results = await engine.search("salary negotiation real estate analogy");
console.log(results.hits[0].bestChunk.text);

// 3. Resurface forgotten memories via Ebbinghaus decay
const digest = await engine.resurface({ decayThreshold: 0.4 });

---

🏗️ Architecture

                     ┌────────────────────────────────────────────────────────┐
                     │                   Application Layer                    │
                     │          (Vault / SaaS App / Worker / CLI)             │
                     └──────────────────────────┬─────────────────────────────┘
                                                │
                                       MemoryEngine SDK
                                                │
     ┌──────────────────┬───────────────────────┼─────────────────────────┬──────────────────┐
     │                  │                       │                         │                  │
     ▼                  ▼                       ▼                         ▼                  ▼
 ┌──────────────┐ ┌──────────────┐     ┌────────────────┐         ┌──────────────┐   ┌──────────────┐
 │  Ingestion   │ │  Embeddings  │     │ Storage Engine │         │  Retrieval   │   │ Resurfacing  │
 ├──────────────┤ ├──────────────┤     ├────────────────┤         ├──────────────┤   ├──────────────┤
 │ • Normalizer │ │ • OpenAI     │     │ • PgVector     │         │ • Dense HNSW │   │ • Ebbinghaus │
 │ • Semantic   │ │ • Gemini     │ ──► │ • In-Memory    │ ◄────── │ • Sparse BM25│   │   Decay      │
 │   Chunker    │ │ • Local/Mock │     │ • DDL Schema   │         │ • RRF Fusion │   │ • Graph Link │
 │ • Extractor  │ │ • Extensible │     │   Migrations   │         │ • Reranking  │   │ • Serendipity│
 └──────────────┘ └──────────────┘     └────────────────┘         └──────────────┘   └──────────────┘

---

🔬 Key Capabilities

1. Hybrid Search with Reciprocal Rank Fusion (RRF)

Combines dense semantic vector search (HNSW cosine similarity) with sparse lexical search (BM25) to deliver both semantic understanding and exact keyword precision:

$$\text{Score}_{RRF}(d) = \frac{w_{dense}}{k + \text{rank}_{dense}(d)} + \frac{w_{sparse}}{k + \text{rank}_{sparse}(d)}$$

2. Ebbinghaus Forgetting Curve Resurfacing

Calculates memory retention probability based on elapsed time and active recall reinforcements:

$$R(t) = \exp\left(-\frac{t \cdot \ln 2}{S}\right) \quad \text{where } S = S_0 \cdot (1 + \alpha \cdot \text{recallCount})$$

Memories that have decayed below your retention threshold ($R(t) \le 0.40$) and possess high importance or cross-cluster associative bridge connections are proactively resurfaced.

3. Associative Semantic Graph

Discovers conceptual links across disparate sources (e.g., connecting a psychology note from 6 months ago to an engineering video saved today) through high-dimensional cosine affinity.

---

⚙️ Configuration

import { MemoryEngine, PgVectorStorage, OpenAIEmbedder } from "memory-engine";

const engine = new MemoryEngine({
  // Storage Adapter: In-Memory (default) or PostgreSQL + pgvector
  storage: new PgVectorStorage({
    connectionString: process.env.DATABASE_URL
  }),

  // Embedding Provider: Mock (default), OpenAI, or Gemini
  embedding: new OpenAIEmbedder({
    apiKey: process.env.OPENAI_API_KEY,
    model: "text-embedding-3-small"
  }),

  // Chunker Configuration
  chunking: {
    maxChunkSizeChars: 1200,
    chunkOverlapChars: 150
  }
});

---

📊 Benchmarks

Run the built-in benchmark harness:

npm run benchmark

Metric	In-Memory Store	PostgreSQL + pgvector (HNSW)
Ingestion Throughput	~2,400 docs/sec	~480 docs/sec
Dense Search Latency (p50)	0.42 ms	4.8 ms
Sparse BM25 Latency (p50)	0.28 ms	3.2 ms
Hybrid RRF Latency (p50)	0.75 ms	6.1 ms
Conceptual Recall Accuracy	100%	100%

---

📖 Guides, Deep Dives & Articles

• Step-by-Step Integration & Setup Guide
• Architecture Deep Dive
• Benchmark Results
• Technical Essay: Why Saved Collections Become Graveyards

---

📄 License

MIT © Paritosh Srivastava