Hi, thanks for publishing the memory benchmark results in BENCHMARKS.md (v8) — the uniform harness + blind-validated judge setup is really nice. I'd love to reproduce graphify's runs on LOCOMO and LongMemEval, but I couldn't quite piece together the details from the repo. Apologies in advance if I'm missing something obvious — a few things I was hoping you could shed some light on:
Where the reproduction command lives. BENCHMARKS.md mentions python memory/runner.py --phase 3 --split locomo --n 300 --adapters graphify_v1_surreal ..., but I couldn't find memory/runner.py (or the adapters) in this repo — am I looking in the wrong place, or is the memory harness not released yet? If it's the latter, are there plans to open it up? I'd also love to understand how recall@10 is computed — i.e. how retrieved graph nodes get mapped back to LOCOMO's gold evidence dia_ids.
Whether the conversation → graph build is LLM-driven or deterministic. The ingest costs seem to hint at the latter (graphify_v1_surreal ingests LOCOMO for $0, graph-expand for ~$1.40, vs mem0 $3.48 / supermemory $15.67) — am I right in reading this as the graph being built without LLM extraction (local BGE-m3 embeddings + deterministic structure)? If so, I'm curious what graph-expand's ~$1.40 goes toward — LLM calls at retrieval time, perhaps?
How conversations are processed into the graph. For LOCOMO / LongMemEval, what do the sessions/turns become — entity/fact nodes, episode nodes, or something else? And what edge types does the conversation graph have?
Thanks so much for any pointers, and congrats on the strong results!
Hi, thanks for publishing the memory benchmark results in
BENCHMARKS.md(v8) — the uniform harness + blind-validated judge setup is really nice. I'd love to reproduce graphify's runs on LOCOMO and LongMemEval, but I couldn't quite piece together the details from the repo. Apologies in advance if I'm missing something obvious — a few things I was hoping you could shed some light on:Where the reproduction command lives.
BENCHMARKS.mdmentionspython memory/runner.py --phase 3 --split locomo --n 300 --adapters graphify_v1_surreal ..., but I couldn't findmemory/runner.py(or the adapters) in this repo — am I looking in the wrong place, or is the memory harness not released yet? If it's the latter, are there plans to open it up? I'd also love to understand how recall@10 is computed — i.e. how retrieved graph nodes get mapped back to LOCOMO's gold evidencedia_ids.Whether the conversation → graph build is LLM-driven or deterministic. The ingest costs seem to hint at the latter (
graphify_v1_surrealingests LOCOMO for $0,graph-expandfor ~$1.40, vs mem0 $3.48 / supermemory $15.67) — am I right in reading this as the graph being built without LLM extraction (local BGE-m3 embeddings + deterministic structure)? If so, I'm curious what graph-expand's ~$1.40 goes toward — LLM calls at retrieval time, perhaps?How conversations are processed into the graph. For LOCOMO / LongMemEval, what do the sessions/turns become — entity/fact nodes, episode nodes, or something else? And what edge types does the conversation graph have?
Thanks so much for any pointers, and congrats on the strong results!