Skip to content

feat(mem0): eval sets by Bob subagents + local reranker run - #22

Merged
ana-lan merged 6 commits into
mainfrom
feat/mem0-evals
Sep 27, 2026
Merged

ana-lan merged 6 commits into
mainfrom
feat/mem0-evals

Conversation

@ana-lan

@ana-lan ana-lan commented Sep 27, 2026

Copy link
Copy Markdown
Owner

Bob B12 wrote 51 eval cases for 3 mem0 features with 3 parallel subagents. Local reranker run on Qwen tiers: 7b 71%, 3b 62%, 1.5b 50%, 0.5b 50%, so no safe downgrade (3b keeps 88% < 95%); failures cluster in the partial-relevance band. Analysis cost $0.29. Report fix: clear sentence when there is no payback. Bob B13 updated the case-study doc and web (one invented claim corrected by hand); a README edit was undone with Bob's rollback.

@vercel

vercel Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
downshift Ready Ready Preview Sep 27, 2026 1:52am UTC

@github-actions

Copy link
Copy Markdown

Downshift cost diff

No LLM call site cost changes in working tree vs origin/main (0 added, 0 changed, 0 removed, 7 unchanged). Projected monthly cost stays at $7,339.50.

Projection, not a bill: tokens estimated from source (words in resolved prompt text; output = max_tokens, or 256 if unset) x illustrative prices x configured calls/day x 30 days.

@ana-lan

ana-lan commented Sep 27, 2026

Copy link
Copy Markdown
Owner Author

Bob built-in code review (/review, 34 tools, 1.98 coins) found 3 issues, all fixed by hand: rr-11 and rr-19 had wrong expected bands (documents that directly answer the query), and the report printed 'No payback' when payback was never computed. Only the 2 relabeled cases were re-run. Final: 7b 17/24, 3b 13/24, 1.5b 13/24, 0.5b 11/24; best candidates keep 76% of baseline, no safe downgrade.

@ana-lan
ana-lan merged commit 31912e6 into main Sep 27, 2026
7 checks passed
@ana-lan
ana-lan deleted the feat/mem0-evals branch September 27, 2026 01:54

This branch was successfully deployed

1 active deployment
Preview — 13176848 Deployed Sep 27, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant