Add knowl and agentmemory as memory methods - #23
Merged
HUST-AI-HYZ merged 1 commit intoAug 20, 2026
Conversation
Two agentic-memory systems that the paper does not currently evaluate, wired in
the same way as the existing memory agents so their rows sit beside the
published ones on equal terms.
WHY THESE TWO. Both resolve write-time conflicts, which is the behaviour
FactConsolidation isolates, and neither has an FC score anywhere. agentmemory
publishes LongMemEval-S retrieval recall, a different task on a different
metric; knowl publishes nothing outside this harness.
CONFIGS ARE COPIED FROM Simple_rag_bm25, NOT CHOSEN. retrieve_num 10,
temperature 0.7, input_length_limit 10000000, buffer_length 200 -- the same
values BM25, Zep, Cognee, HippoRAG-v2, RAPTOR, GraphRAG, Self-RAG and every
embedding baseline already use.
The reader path is shared between both methods and reproduces _handle_bm25_rag
exactly: each retrieved item gets a trailing newline, items are labelled
"Memory i:" and joined, and the instruction trails the facts under the generic
system template. Retrieval goes through _extract_retrieval_query, as the RAG
handlers do -- skipping it is not a small error, since the ~200 tokens of task
boilerplate are byte-identical across all 100 questions and retrieving on them
retrieves on noise.
Neither method installs anything into this venv. knowl runs as a Node
subprocess over a sentinel-framed stdio protocol (the embedding runtime and
SQLite bindings both write to stdout freely, and an unframed protocol
eventually swallows a log line and desynchronises into a plausible score rather
than an error). agentmemory runs as its own service and is reached over REST.
Both ingest an identical parsed fact list, one record per write, in context
order. Feeding raw chunks was rejected: agentmemory stores one memory per
remember call, so a 4096-char chunk becomes a single record holding ~70 facts,
which measures chunking rather than memory.
Measured on this harness, FC-SH SubEM, gpt-4o-mini reader:
6k 262k
knowl 95.0 89.0
agentmemory 83.0 79.0
Single runs at temperature 0.7; treat as directional.
One asymmetry noticed while matching the reader: _handle_mem0_agent places its
memories in a system message and appends a "Current Time:" line, where the RAG
family does neither. Both methods here follow the RAG family. Flagging it in
case cross-family comparisons are meant to hold that constant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two agentic-memory systems the paper does not currently evaluate, wired the same way as the existing memory agents.
Why these two
Both resolve conflicts at write time, which is the behaviour FactConsolidation isolates, and neither has an FC score anywhere today. agentmemory publishes LongMemEval-S retrieval recall — a different task on a different metric. knowl publishes nothing outside this harness.
Configs are copied, not chosen
Every value comes from
Simple_rag_bm25:retrieve_num: 10,temperature: 0.7,input_length_limit: 10000000,buffer_length: 200— the same values BM25, Zep, Cognee, HippoRAG-v2, RAPTOR, GraphRAG, Self-RAG and all four embedding baselines already use.The reader path is shared between both new methods and reproduces
_handle_bm25_ragexactly: trailing newline per item,Memory i:labels, instruction trailing the facts under the generic system template. Retrieval goes through_extract_retrieval_queryas the RAG handlers do — skipping it is not a small error, since the ~200 tokens of task boilerplate are byte-identical across all 100 questions.Nothing is installed into this venv
:3111.Both need one env var / one running process, documented in the module docstrings.
Input normalisation
Both ingest an identical parsed fact list, one record per write, in context order. Feeding raw 4096-char chunks was considered and rejected: agentmemory stores one memory per
remembercall, so a chunk lands as a single record holding ~70 facts, which measures chunking rather than memory.Results on this harness
FC-SH SubEM, gpt-4o-mini reader:
Single runs at
temperature: 0.7— directional, not publication-grade. Raw per-question results and full method notes: https://github.com/dat999zx/knowl/blob/main/benchmarks/memoryagentbench/mab/FINDINGS.mdDisclosure: I maintain knowl. agentmemory is not mine; I ran it because no FC number existed and the comparison seemed worth having. Happy to drop the knowl half of this PR if a maintainer-run method is preferred.
One asymmetry noticed while matching the reader
_handle_mem0_agentplaces its memories in a system message and appends aCurrent Time:line to the user turn; the RAG family does neither. Both methods here follow the RAG family. Flagging it in case cross-family comparisons are meant to hold reader layout constant.Builds on #22 (UTF-8 result writes,
--force), which is still open.