Graph revision - #130
Open
wangyu-ustc wants to merge 12 commits into
Open
Conversation
…ph, cold-fact recovery, auto_dream Consolidated record of the pre-ablation research arc (per-version history is preserved at the graph_revision_pre_squash tag; the narrative is in docs/graph_memory_v7/development_history.md): - configurable graph-memory runtime, LongMemEval/RULER eval harness, v8 singleton pruning - v7.1 rerank (SH-Doc +4), v7.2 coverage, v7.3 propositions (rejected), upstream auto_dream merge - extractor line v7.4 GLiNER -> v7.6 LLM triples with relation edges -> v7.7 PPR -> v7.8 registry-guided entity resolution - v7.10 hypergraph (facts reified as citable V7Fact nodes) + the enumerate-then-count answerer - cold-fact recovery (+4.4, the first real QA gain) and hypergraph redundancy control at ingest - auto_dream at scale: graph maintenance, semantic reconsolidation, and the graph_only mode (flat store byte-identical, graph refined)
…CoMo-validated
The engineering round that made auto_dream safe and the graph retrieval honest:
- role-noise anchors blocked at the anchor funnel (a "Users" hub had reached
degree 1016) and the v7.1 reranker revived — a stale `== "v7.1"` guard had
disabled it for every later version, and the dispatcher was stranding
v7.4-v7.10 on the dead v5 pipeline; MIRIX_GRAPH_RERANK=0 keeps the unranked
baseline measurable (paired result: +2~3 on LoCoMo, second benchmark after
SH-Doc's +4).
- live ingest hooks now pass role provenance (organic graphs had silently lost
all user/assistant attribution; only rebuilt graphs had it).
- interleaved auto_dream during ingest (MIRIX_DREAM_EVERY_N_CHUNKS): one cycle
every N chunks plus a final one, the online periodic design.
- consolidation no longer corrupts the graph: zombie facts (citations all dead)
are swept and the temporal chain is rebuilt every maintenance cycle — the
dreamed graph now converges to the size of a graph built from the post-merge
store (1245 vs 1249 measured).
- the union-coverage gate: a merge that would drop any deterministic specific
of its sources (numbers, dates, month/weekday words, word-numbers, proper
names) is rejected before anything is deleted. Measured on conv-26: 29 lossy
merges rejected and rewritten, temporal accuracy back to the exact no-dream
level, 3-run QA {134,137,135} statistically identical to the no-dream band
{134,135,138} while the store still consolidates 227->214.
…archive sweep - locomo_dream_ablation.md: the six-arm LoCoMo table (graph/reranker/dream x gated) with store and graph sizes, mechanisms, and repro. - development_history.md: the research arc in nine phases with the meta-lessons; superseded design-era docs move to docs/graph_memory_v7/archive/. - archive/: every module the current design (v7.x/v8 hypergraph + rerank + gated dream) no longer reaches — the v5 dual-graph pipeline, the v6 entity index, v7.3 proposition and v7.4 GLiNER extraction, and the rejected eval experiments — moved out of the live tree; the dispatcher becomes a thin v7/v8 router and the ingest hooks warn-and-skip for unknown versions. - a second, adversarially verified audit (30 agents; every candidate checked against python/sh/yaml/docs references, FastAPI routing and tool auto-registration) then removed the leftovers nothing reaches: the never-measured MIRIX_GRAPH_ROUTED_SEARCH gate and its helper, the v5/v6 Neo4j DDL and neo4j_healthcheck, the LightRAG relation output surface (the live v7/v8 path reads entities only), four orphaned lightrag prompt symbols, token_tracker.set_phase, the dispatcher's vestigial budget parameters, EpisodicEventUpdate.source_refs, an unused json import, and the answerer's placeholder prompt vars; draw_hypergraph.py joins archive/evals/. Audit verdicts that would have over-cut were overruled and kept: check_raw_item (the answerer's live LLM tool), add_chunk(raw_input=...) (harness API), process_memory(entities=...) (public affordance), the conflict-resolution update-path mirror, get_last_dream_time, compact_graph, build_coldfacts and the non-mab longmem harness (used by run_eval.sh and the chunking study). Verified after each sweep: live imports (incl. full rest_api), gate behavior tests, V7Retriever rows on the live graph, server boot.
… variants of it
AutoDream merges near-duplicate anchors during ingest rather than in a separate offline
pass. This squashes the v2 work and the six policy variants that followed it, plus the
defects found along the way — several of which were only visible because a fix in one
direction exposed damage in the other:
* merge before sweeping. Pruning first destroys the very links a merge would have used.
* then sweep AGAIN after merging, because merges manufacture exactly the debris the
maintenance pass had just cleaned.
* anchor merge kept only the survivor's values, silently discarding the other node's.
* the coverage gate was rejecting clusters over their own titles.
* checkpoints were being filed under the admin user, so a second user's dream state was
read from the first user's row.
* consolidation belongs on the graph — graph_only is now the default.
The six variants (v7.14-v7.19) all answer one question: which anchors should a consolidation
pass consider, and when. v7.19 is what v7.19 through v7.24 use — bounded dirty-frontier, 80
pairs per intermediate cycle, 120 on the final plus a 60-pair repair round, and online dream
never cleans, merges or prunes Facts at all.
Measured on conv-26 (152 questions) the six sit at 137, 131, 139, 136, 136, 137 against
v7.12's 140, on a sample whose same-config re-run spread is about 4. Only v7.15 is
distinguishable, and it is distinguishably worse. That is not an argument for deleting the
others; it is an argument for never again choosing between policies on 152 questions.
Two constraints to know before extending this line. _MERGE_SAFE_TYPES admits only person,
organization, location, object and concept — event, date, content and method are excluded
deliberately, because merging them collapses distinct occurrences whose surface names are
near-identical. And the binding constraint is not the pair budget: 4989 anchors were
processed as sources with only 364 pairs rejected. It is _PAIR_COS = 0.93, which keeps 71%
of lexically redundant pairs from ever becoming candidates.
…n them
v7.12 makes the hypergraph actually n-ary: a fact is one frame carrying role-typed
arguments, instead of a fan of binary relations. It drops the ref-node layer (anchors carry
PG ids directly), stops writing V7_RELATION since the frame already holds it, and teaches
/memory/search to consult the graph at all — it was flat-only before.
Squashed in are the defects that made the early v7.2x numbers uninterpretable:
* vector anchor search returned OTHER USERS' anchors and then dropped them, so a
single-user store silently under-fetched
* node merging was dead on v7.12, and dedup deleted citations
* frame roles were not canonicalised, so the same fact landed on two nodes
* three defects in the ref-free retrieval replacements
* the version test was an equality, so a new version matched nothing and fell back to
pre-policy behaviour while looking like a regression. It is a floor now, and the policy
module resolves downwards instead of through 25 equality checks.
The policy line, all of it accumulated rather than exclusive:
v7.13 role canonicalisation — gated to v7.13 ONLY, off in every later version
v7.20 exact-match boost, low-degree anchors over hubs, person hubs penalised hardest,
a lexical pass over PG rows after vector recall
v7.21 a second lane: exact anchor, predicate-filtered fact, cited memory
v7.22 one normaliser shared by query and stored predicates; ranking by relation,
entities, object constraints and temporal qualifiers; separate quotas so the
precision lane cannot erase the recall lane
v7.23 adaptive evidence policy AND a second extraction pass in frame_extractor
v7.24 role, time, state and citation policy
v7.25 one fused ordering key — measured null
v7.26 wider candidate admission — measured -18, off by default
Three things a reader should not have to rediscover.
**Use v7.23.** It produced the best clean full run (1367/1540) and both same-store ablation
arms (1361/1370). v7.24 scores 1362 on the same graph, which reads as a 5-question
regression but is really multi-hop .883 -> .830 — 53 questions lost, 48 regained elsewhere —
and multi-hop is exactly where the gap to HyperMem is largest.
**v7.23 is two changes wearing one version string.** Its docstring says retrieval policy;
it also enables a second LLM extraction pass, and costs 36% more ingest wall-clock. If
either half is evaluated again they need separate versions.
**Every retrieval version keeps candidate generation inside Neo4j on purpose.** When the
graph owns episodic and semantic, their flat searches are not scheduled at all. That looks
like the cause of the 38 errors whose answer sits in the store unretrieved — but running
both lanes in union was measured at -2 questions, so it is not.
…failures
**The prompts that build and read memory contained LoCoMo's gold answers.** Every number
produced before this is contaminated and not comparable to anything after it. prompts/0201a
is the clean set; 0201b holds an experimental variant (predicate-shaped semantic facts,
speaker attribution in summaries) that was measured at +1 against a same-code control and is
therefore not the default.
Three failures that each returned a plausible number while not doing the work:
* a Neo4j outage made search_memory return an EMPTY memory system rather than failing.
1540 questions were answered from nothing, judged, and reported rc=0 at 0.3461. There is
a flat fallback now, and it says so in the log.
* one bad question could kill a six-hour eval, and often did.
* the evidence-budget knob was a FLOOR, so the experiment that "cut the budget and changed
nothing" had never cut anything.
Also here: MIRIX_EPISODE_DEDUP and MIRIX_WRITE_DEDUP, both off by default. A counting
question is answered by counting rows, and one event written as several rows is an extra
count — conv-44 holds two rows both resolved to 2023-08-01 describing the same planned hike.
occurred_at is already the RESOLVED event date, so the identity needed to merge them was
present and unused. Both merges are append-only and never regenerate text: every LLM rewrite
in this pipeline has been shown to lose or invent (71% of quoted work-titles in the store
appear nowhere in the source), and a dedup that rewrites would add such a pass exactly where
the surviving row is chosen. Neither will merge across a differing quantity or month.
MIRIX_GRAPH_UNION is here too, off, and the comment records why the obvious idea behind it
does not work: it runs flat search alongside the graph so a row the graph misses still has a
path in, it demonstrably fires, and it is worth -2 questions. The 15 questions it was meant
to recover were lost to the two arms having been built by two separate ingests, not to the
graph displacing flat rows.
…nt flags
The judge prompt lived in four files. One used U+2019 apostrophes where the others used
ASCII, worth 3 questions on the error side — two comparisons graded by different text are not
comparable, and nothing in the output said so. evals/llm_judge.py is now the only copy,
pinned with a sha256 and assert_canonical_prompt().
Worth adopting next: HyperMem grades every question THREE times and scores a 2-of-3 majority
(stage6_eval.py:344 num_runs, :287 three concurrent graders, :481 true_count >= num_runs/2).
We grade once and carry ~25 questions of judge noise on 1540 as a result. About $1 per full
run to remove, against 5.5h of ingest.
assert_store_sane.py is the pre-flight none of the failed runs had: rows exist under the
expected prefix, nothing sits outside it, the server answers, the graph has anchors. Three
runs produced a plausible number without running the experiment — an empty memory after a
Neo4j outage, a run that exported MIRIX_USER_PREFIX instead of MIRIX_EVAL_USER_PREFIX and
wrote onto an older run's anchors, and a hand-rolled uvicorn line that never loaded the app.
Two flags, both off:
MIRIX_WINDOW_TURNS ingest a session in overlapping N-turn windows instead of one call.
At 6 turns on conv-30 it recovered the traced case — rows containing
"Lean Startup" went 0 to 3 — but distinctive-token recall was 73.7%
before and after, QA moved 72 to 73 of 81, and the store grew 76%.
One conversation yields ~19 measurable tokens, too few to separate a
real effect from none. Retry on three or more, and carry the
store-size cost: more rows in front of the answerer is a headwind.
answerer flags the answerer could not run a reasoning model at all, and its own
instructions contradicted each other ("give every item" against "be
VERY CONCISE, only output the answer"). Resolved behind a flag.
Ten analysis scripts, and the results are the point of the commit — most of them are
negative, and each one is a week someone else does not have to spend.
WHERE THE ERRORS ARE. 1540 questions, 169 wrong, every stage of the triage carrying a
positive control on questions we answer correctly:
INGEST never written to the store 60 36%
GOLD not derivable from the source 41 24%
RETRIEVAL in the store, never retrieved 38 22%
ANSWERING evidence present, answer wrong 30 18%
Two ceilings. An ORACLE retriever — answer from the 15 store rows that best match each
question's own gold — rescues 67 of 169, so 102 errors are beyond any ranking work. And gold
defects cap the benchmark near 97.3%.
WHY FACTS ARE NOT WRITTEN. The extraction prompt asks for a summary that is "concise and
informative", and the extractor keeps the POINT of a turn rather than its CONTENT. "I'm
reading 'The Lean Startup' hoping it'll give me tips for my biz" becomes "Jon is wrapping up
a business plan" and no row in the store contains the title. "Hey Jo, guess what I did?"
becomes "Nate dyed his hair purple" and no row contains the nickname.
WHAT THE GRAPH IS WORTH. Four QA-only arms over ONE store, byte-identical rows, differing
only in whether retrieval enters through Neo4j: no-graph 1310/1295, graph 1361/1370, so
+63 questions. But single-hop gains 43 of that and multi-hop only 11.5 — it is a good index,
not a reasoning structure.
WHAT FAILED, so it is not tried again. Eleven answering-side interventions, all net-negative
or null: reworded format rules, naming the failure mode, forced verbatim citation,
select-then-answer, self-consistency over three samples, verify-then-retry, a stronger model
(-11 on the full run), WHO/WHEN/WHAT tables. The shape is always the same — 110 target
questions gain 4 to 11, the 1335 already-right lose more, because the control set is twelve
times larger. The only intervention that ever survived, countfirst (0/20 to 11/20), fired
only on questions starting "how many": it had a detectable trigger.
Verify-then-retry moved the target by exactly zero. Asked whether its own wrong answer was
supported by the evidence, the model says yes.
Anchor entity resolution: simulating a merge of 1829 near-duplicate anchors connects the
evidence for 1 of 31 failing multi-hop questions.
A verbatim cold-fact lane (build_coldfacts_locomo.py, 5832 turns indexed): the ranking works
— the gold-bearing turn is rank 1 for 24% of targets and top-3 for 37% — but there is no way
to FIRE it selectively. Threshold 0.86 reaches 24% of targets and fires on 78% of controls;
z-score and top1-minus-top10 gates do no better. Every LoCoMo question is about the
conversation, so every question has a semantically close turn. The offline gate cost twenty
minutes and saved three hours of QA arms.
TWO NOISE FLOORS, which change how everything above must be read. Re-ingesting with identical
code moves the score ~+5 on 411 questions. And two identical QA-only runs over one store
differ by 9-15 on 1540 — but pairing them shows 77% of questions receive byte-identical
retrieval and flip at 1.5%, while the rest flip at 5.4%. The drift is retrieval
non-determinism, not the answerer (temperature 0, fixed seed) and not the judge. Use McNemar
on discordant pairs, and for retrieval changes restrict to questions whose context actually
differed. Twelve graph versions "moved" the benchmark 16 questions in total; they were not
unmeasurable, they were measured by subtracting two accuracies.
Written for someone else to continue this. Part I is operational — environment and the env vars that are silently ignored when misspelled, the LoCoMo full-ingest and QA-only templates, how to run a no-graph/with-graph comparison, the four MemoryAgentBench tracks with the commands that produced the results already on disk, and the judging protocol. Part II is where the errors are, what each graph version changed, what to build next. Three things in it are worth more than the findings. **Run v7.23.** It produced the best clean full run (1367/1540) and both same-store ablation arms. v7.24 scores 1362 on the same graph, which reads as a 5-question regression but is multi-hop .883 -> .830 — 53 questions lost, 48 regained elsewhere — and multi-hop is exactly where the gap to HyperMem is largest. **Adopt HyperMem's judging.** They grade every question three times and score a 2-of-3 majority (stage6_eval.py:344 num_runs, :287 three concurrent graders, :481 true_count >= num_runs/2). We grade once and carry ~25 questions of judge noise on 1540. About $1 per full run to remove. Their accuracy prompt is byte-identical to ours, so judging is not where the 4-point gap lives. **Two questions, two experiments.** "Is MIRIX better with the graph" is a system claim and needs two independent ingests, because a real no-graph deployment never builds the graph and never runs AutoDream. "Why, and by how much on the read side" is answered by holding one store fixed and flipping the flag — four hours instead of fifteen. LoCoMo currently has no clean V1 number at all: clean_v723_full is V2, ab_nograph_r1/r2 are the read-side ablation over a graph-built store, and locomo_nograph_all is a real V1 whose prompts contained the gold answers. Filling that gap is the cheapest item on the page. The file map now separates results that can be quoted from results produced before the prompts were decontaminated on 2026-08-04. The 0.8279 no-graph figure is the one most likely to be cited by mistake and is wrong in three ways that do not cancel — contaminated prompts that hurt a no-graph arm more than a graph arm, its own store, and a comparison run with its own store too. Re-measured properly the no-graph arm scores 1302.5, twenty-four questions higher, and the graph is worth +63 rather than +80 or +96. Also here: export_store.py and store_exports/clean_r1. A pg_dump of the clean store is 177 MB and 119 MB of that is embedding vectors; the text is under one megabyte. Exporting rows and graph structure without the vectors brings the whole store — 2908 memory rows, 23,820 Neo4j nodes, 32,390 relationships — to 1.9 MB, small enough to commit. This matters because ingest is not reproducible: identical code re-ingested moves the score by ~5 questions per 411, so a fresh 5.5-hour ingest makes a DIFFERENT store, and nothing measured against it is comparable to the numbers on disk. The Neo4j half had never been exported at all, so the archived store could not have run a single graph arm.
HANDOVER.md referenced /home/lj/MIRIX_eval/... by absolute path for the scripts that produced every number in it, and none of them were in this repository. A clone had a runbook citing files it did not contain. evals/harness/ now holds the subset that produced a result someone is expected to build on: the run templates (locomo_clean.sh, graph_ablation.sh — the QA-only four-arm pattern that removes ingest drift), the measurement tools (hypermem_judge.py with its 2-of-3 majority, window_recall.py as the model for a judge-free deterministic metric), the error analysis (final_triage.py and its verdicts, oracle_ceiling.py, coldfact_gate.py), and the three replay harnesses holding the eleven answering-side interventions that all failed. Its README says what each one measured rather than what it does, so the next person can decide what to re-run without reading the code first. About eighty further one-off probes stay out — they answered a question for one afternoon and carrying them would obscure the dozen that matter. Paths inside the scripts still point at the original machine. Left as-is and flagged in the README rather than half-parameterised.
…hich half is unquotable HANDOVER.md says how to run things; this says what came out. It opens with the part that matters most: three cutoffs — prompt decontamination on 2026-08-04, the per-runner storage prefix, and same-store ablation discipline from 2026-08-08 — decide whether a stored result means anything, and roughly half of what is on disk fails one of them. Contents: the full settings (env vars, models, chunking, and the fact that every store is ada-002 despite the configs claiming 3-small); every quotable LoCoMo run with per-category breakdown and wall-clock; the four-arm graph ablation (+63 questions, two thirds of it single-hop); three noise floors with their causes; the four-bucket error triage and two ceilings; every intervention tried with its number, including the eleven answering-side failures and the four independent measurements saying more evidence makes the answerer worse; MemoryAgentBench including the one clean number (SHDocQA 79/100 on v7.1) and why the historical batch is not usable; the HyperMem reconciliation and two competitor claims that do not reconcile; and a table of silent failures that each exited 0 with a plausible score. Two things worth pulling out of it. Our number is the mean of same-config runs, ~89.0-89.6%, so the gap to HyperMem is 48-57 questions rather than the 4.7 points a single draw implies. And v7.24 reads as a 5-question regression while actually being multi-hop .883 -> .830 — 53 lost, 48 regained elsewhere — in the category where that gap is largest. Ends with what is still unmeasured, ranked, and an honest ceiling: 90.2-91.4% for everything currently on the list, which does not close the gap.
… run each Re-running SHDocQA's QA against the same store gave 79/100 twice with an identical per-question distribution. Adds §3.4 with the comparison: substring-judged RULER tracks re-run at 0 questions of spread, LoCoMo at 9-15 on 1540, LongMemEval-S at 7 on 60. The cause is a deterministic judge plus short exact-needle answers, where a shift in retrieval order rarely changes whether the needle appears. That halves the cost of four of the eight V1/V2 cells: RULER can be compared from single runs, LoCoMo and LongMemEval-S cannot. Also records the SHDocQA error shape — 21 single-entity lookups clustered on proper nouns, the same failure as LoCoMo's largest bucket on encyclopedic rather than conversational text — and corrects the char arm's stop to spent credits rather than a rate ceiling.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.