Skip to content

Add knowl and agentmemory as memory methods - #23

Merged
HUST-AI-HYZ merged 1 commit into
HUST-AI-HYZ:mainfrom
dat999zx:feat/knowl-agentmemory-methods
Aug 20, 2026
Merged

Add knowl and agentmemory as memory methods#23
HUST-AI-HYZ merged 1 commit into
HUST-AI-HYZ:mainfrom
dat999zx:feat/knowl-agentmemory-methods

Conversation

@dat999zx

Copy link
Copy Markdown
Contributor

Adds two agentic-memory systems the paper does not currently evaluate, wired the same way as the existing memory agents.

Why these two

Both resolve conflicts at write time, which is the behaviour FactConsolidation isolates, and neither has an FC score anywhere today. agentmemory publishes LongMemEval-S retrieval recall — a different task on a different metric. knowl publishes nothing outside this harness.

Configs are copied, not chosen

Every value comes from Simple_rag_bm25: retrieve_num: 10, temperature: 0.7, input_length_limit: 10000000, buffer_length: 200 — the same values BM25, Zep, Cognee, HippoRAG-v2, RAPTOR, GraphRAG, Self-RAG and all four embedding baselines already use.

The reader path is shared between both new methods and reproduces _handle_bm25_rag exactly: trailing newline per item, Memory i: labels, instruction trailing the facts under the generic system template. Retrieval goes through _extract_retrieval_query as the RAG handlers do — skipping it is not a small error, since the ~200 tokens of task boilerplate are byte-identical across all 100 questions.

Nothing is installed into this venv

  • knowl runs as a Node subprocess over a sentinel-framed stdio protocol. The framing matters: its embedding runtime and SQLite bindings both write to stdout freely, and an unframed protocol eventually swallows a stray log line and desynchronises into a plausible score rather than an error.
  • agentmemory runs as its own service and is reached over REST on :3111.

Both need one env var / one running process, documented in the module docstrings.

Input normalisation

Both ingest an identical parsed fact list, one record per write, in context order. Feeding raw 4096-char chunks was considered and rejected: agentmemory stores one memory per remember call, so a chunk lands as a single record holding ~70 facts, which measures chunking rather than memory.

Results on this harness

FC-SH SubEM, gpt-4o-mini reader:

6k 262k
knowl 95.0 89.0
agentmemory 83.0 79.0

Single runs at temperature: 0.7 — directional, not publication-grade. Raw per-question results and full method notes: https://github.com/dat999zx/knowl/blob/main/benchmarks/memoryagentbench/mab/FINDINGS.md

Disclosure: I maintain knowl. agentmemory is not mine; I ran it because no FC number existed and the comparison seemed worth having. Happy to drop the knowl half of this PR if a maintainer-run method is preferred.

One asymmetry noticed while matching the reader

_handle_mem0_agent places its memories in a system message and appends a Current Time: line to the user turn; the RAG family does neither. Both methods here follow the RAG family. Flagging it in case cross-family comparisons are meant to hold reader layout constant.

Builds on #22 (UTF-8 result writes, --force), which is still open.

Two agentic-memory systems that the paper does not currently evaluate, wired in
the same way as the existing memory agents so their rows sit beside the
published ones on equal terms.

WHY THESE TWO. Both resolve write-time conflicts, which is the behaviour
FactConsolidation isolates, and neither has an FC score anywhere. agentmemory
publishes LongMemEval-S retrieval recall, a different task on a different
metric; knowl publishes nothing outside this harness.

CONFIGS ARE COPIED FROM Simple_rag_bm25, NOT CHOSEN. retrieve_num 10,
temperature 0.7, input_length_limit 10000000, buffer_length 200 -- the same
values BM25, Zep, Cognee, HippoRAG-v2, RAPTOR, GraphRAG, Self-RAG and every
embedding baseline already use.

The reader path is shared between both methods and reproduces _handle_bm25_rag
exactly: each retrieved item gets a trailing newline, items are labelled
"Memory i:" and joined, and the instruction trails the facts under the generic
system template. Retrieval goes through _extract_retrieval_query, as the RAG
handlers do -- skipping it is not a small error, since the ~200 tokens of task
boilerplate are byte-identical across all 100 questions and retrieving on them
retrieves on noise.

Neither method installs anything into this venv. knowl runs as a Node
subprocess over a sentinel-framed stdio protocol (the embedding runtime and
SQLite bindings both write to stdout freely, and an unframed protocol
eventually swallows a log line and desynchronises into a plausible score rather
than an error). agentmemory runs as its own service and is reached over REST.

Both ingest an identical parsed fact list, one record per write, in context
order. Feeding raw chunks was rejected: agentmemory stores one memory per
remember call, so a 4096-char chunk becomes a single record holding ~70 facts,
which measures chunking rather than memory.

Measured on this harness, FC-SH SubEM, gpt-4o-mini reader:

              6k    262k
  knowl      95.0   89.0
  agentmemory 83.0  79.0

Single runs at temperature 0.7; treat as directional.

One asymmetry noticed while matching the reader: _handle_mem0_agent places its
memories in a system message and appends a "Current Time:" line, where the RAG
family does neither. Both methods here follow the RAG family. Flagging it in
case cross-family comparisons are meant to hold that constant.
@HUST-AI-HYZ
HUST-AI-HYZ merged commit fe1735d into HUST-AI-HYZ:main Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants