Skip to content

docs: pin Wave 1 local LoCoMo 14/30 (honest remasure) - #106

Merged
cursor[bot] merged 1 commit into
devfrom
pr/wave1-remasure-a6c7
Aug 13, 2026
Merged

docs: pin Wave 1 local LoCoMo 14/30 (honest remasure)#106
cursor[bot] merged 1 commit into
devfrom
pr/wave1-remasure-a6c7

Conversation

@sdntsng

@sdntsng sdntsng commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Wave 1 (#101#105: context vs proof, temporal features, retrieval budgets, assistant-extract policy) was merge-committed onto dev (a7a5184). This PR records the local remasure. It does not change product code.

Landed on dev via merge-commit bd987fa (same local-merge path as #100#105; GitHub merge API is 403). main untouched.

LoCoMo 1×30 on rebuilt local API+worker: 14/30 (MH 3/10, OD 2/4, temporal 9/16). Failure ledger 16/16 READER_MISS with evidence/semantic/retrieval/coverage oracles all supported. OpMem 13/13. Marketing 17/17.

This is not an improvement vs Gate 0 staging 18/30. Multi-hop did not improve (4/10 → 3/10). Do not read 14/30 as beats-Mem0: same-day Mem0 was 12/30 overall but 7/10 MH.

Confound vs local PR2 6/30: that run used harness BRAINY_USE_RECALL=1 but the API process did not have BRAINY_RECALL_LLM=1. This remasure enabled the hybrid reader so PR5 is actually exercised. Do not attribute 6→14 to Wave 1 code alone.

Changes

  • Pin: docs/benchmarks/artifacts/locomo-wave1-dev-1x30-20260813.md plus JSONL ledger
  • Non-reg pins: OpMem 13/13, marketing 17/17
  • Status / gap matrix / assessment pack / research README updated
  • PR6–PR8 remain deferred (coverage oracles still supported)

Test plan

  • go test ./internal/memory/ ./internal/jobs/ ./internal/api/ after Wave 1 merge (on dev)
  • Local ingest + /healthz after rebuild
  • python3 evals/run_opmem.py --systems brainy → 13/13
  • python3 evals/run_vertical_eval.py --fixture-dir fixtures/vertical/marketing → 17/17
  • python3 -m public.locomo.run_smoke --conversations 1 --questions 30 --run-id locomo-wave1-dev-1x30-20260813 → 14/30
  • Optional after this pin lands: staging 1×30 vs post-cutover 15/30 (local vs Render confound)

Linear / tracking

Competitive program Wave 1 remasure. No new Linear ID.

Risk / follow-ups

  • Remaining driver is still reader/enumerate emitting chat continuation, not missing edges. Do not start PR6–PR8 from this pin.
  • Next measured loop: /recall answer/enumerate quality (fact, not dialogue dump).
  • main / production was not touched. Merge this pin to dev only unless explicitly asked.
  • LME-20 quality remains 0/20; do not start LME-100 as a quality run.
Open in Web Open in Cursor 

Honest remasure on dest a7a5184 after merging #101-#105. Overall 14/30
(MH 3/10, temporal 9/16). Ledger 16/16 READER_MISS with coverage oracles
supported. OpMem 13/13 and marketing 17/17 passed. Record the hybrid-reader
confound vs local PR2 6/30. No SOTA / beats-Mem0 / Gate 0 improvement claim.

Co-authored-by: Siddhant Singh <sdntsng@users.noreply.github.com>
@sdntsng
sdntsng marked this pull request as ready for review August 13, 2026 13:11
@cursor
cursor Bot merged commit bd987fa into dev Aug 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant