feat(mem0): eval sets by Bob subagents + local reranker run

docs/case-study-mem0.md

@@ -112,11 +112,50 @@ Full output: [estimate.md](case-study/mem0/estimate.md).
 - **Output is priced at `max_tokens`**, a ceiling, so feature costs are an upper bound.
 - **Tokens are estimated from words**; real token counts for long prompts are usually
   higher. Runtime text (conversation, memories, documents) is not counted.
-- **No evals were run** (paid API calls, $0 budget), so no downgrade recommendations.
+- Evals were run locally (see below); no downgrade was safe for the reranker.
 - **Default setup only.** The other provider adapters, the integration and the example
   were copied from the scan unchanged and are not priced (0 calls a day, or no configured
   price for the example's `grok-3-beta`).
 
+## Evals and a local run
+
+Three Bob subagents wrote eval sets in parallel (task B12): 15 cases for fact extraction,
+12 for procedural memory, and 24 for the reranker, for a total of 51 cases. Image
+description has no evals: it needs a vision model and real image URLs.
+
+Grading is judge-based for the reranker (it returns a decimal score, so exact match is
+meaningless; the rubric checks the score band) and for extraction (free-text memories).
+For procedural memory the message role was a template placeholder; it was set to
+`assistant` before running.
+
+The run used `downshift.local.yaml` on Ollama with Qwen 2.5 tiers and SupportDesk's
+illustrative prices, not mem0's real models. Reranker results across all four candidates:
+
+| Model | Pass rate |
+|---|---|
+| `qwen2.5:7b` (baseline) | 17/24 (71%) |
+| `qwen2.5:3b` | 13/24 (54%) |
+| `qwen2.5:1.5b` | 13/24 (54%) |
+| `qwen2.5:0.5b` | 11/24 (46%) |
+
+No safe downgrade: the best candidates (3b and 1.5b, 13/24) keep 76% of baseline quality, below the 95% bar. The
+failures cluster in the "related but does not answer" band (scores 0.3 to 0.7); all
+models handle clear yes/no cases. The baseline itself only passes 71% of cases, below the
+80% floor, so the prompt or model needs improvement before a downgrade makes sense.
+
+Bob's built-in code review of this branch caught two mislabeled reranker cases (rr-11 and
+rr-19: documents that directly answer the query, labeled mid or low). Their bands were fixed
+to 0.8 to 1.0 and only those two cases were re-run; the table above is after the fix.
+
+Fact extraction and procedural memory have eval sets but were not run in this pass: their
+prompts are long (the extraction system prompt alone is several thousand tokens), so the
+local run covered only the reranker. They can be run the same way with `--site`.
+
+The total analysis cost was $0.29 one-time at illustrative prices (96 eval calls at $0.02
+plus an estimated 96 judge calls at $0.28). Local Ollama runs cost $0 in practice.
+
+Full results: [eval-report.md](case-study/mem0/eval-report.md).
+
 ## Reproduce
 
 ```bash

docs/case-study/mem0/downshift.audit.json

@@ -1880,7 +1880,7 @@
       "purpose": "Extract and store new facts from a conversation turn into the mem0 vector store using additive (ADD-only) extraction with memory linking.",
       "output_contract": "JSON object with key 'memory' whose value is a list of objects; each object has 'id' (sequential string), 'text' (string, 15\u2013100 words), and optionally 'linked_memory_ids' (array of UUID strings). No UPDATE or DELETE operations. Parsed via json.loads(...).get('memory', []).",
       "difficulty": "medium",
-      "grading": "json_fields",
+      "grading": "judge",
       "found_by": "bob"
     },
     {
@@ -1905,7 +1905,7 @@
           "resolved": true
         },
         {
-          "role": "{role}",
+          "role": "assistant",
           "content": "{content}",
           "resolved": true
         },
@@ -1926,7 +1926,8 @@
         "Bob: first message uses the PROCEDURAL_MEMORY_SYSTEM_PROMPT constant from mem0/configs/prompts.py unless the caller passes a custom 'prompt' argument, in which case that string replaces it. Placeholder {procedural_memory_system_prompt} represents the full constant value.",
         "Bob: the middle messages are the raw conversation messages passed in by the caller (*messages splice), represented here as a single template row with {role}/{content} placeholders.",
         "Bob: the async twin of this call (asyncio.to_thread wrapper at main.py line 3745) sends identical messages; no separate entry added for it.",
-        "Filled after the audit by a script: Bob wrote a placeholder for the module constant PROCEDURAL_MEMORY_SYSTEM_PROMPT; its text was copied from mem0/configs/prompts.py (skill step 5)."
+        "Filled after the audit by a script: Bob wrote a placeholder for the module constant PROCEDURAL_MEMORY_SYSTEM_PROMPT; its text was copied from mem0/configs/prompts.py (skill step 5).",
+        "Eval prep: the agent-history message role was the template {role}; set to 'assistant' (agent output) so eval cases only fill {content}."
       ],
       "purpose": "Summarise an agent's full execution history into a structured procedural memory record that captures every action, result, and key finding verbatim.",
       "output_contract": "Free-text markdown summary following the PROCEDURAL_MEMORY_SYSTEM_PROMPT template: starts with Task Objective and Progress Status, followed by numbered agent-action steps each containing Action, Action Result (verbatim), Key Findings, Navigation History, Errors, and Current Context. Must be non-empty; ValueError raised otherwise.",
@@ -2016,7 +2017,7 @@
       "purpose": "Score the relevance of a single retrieved memory document against a search query so that mem0 can rerank search results before returning them to the caller.",
       "output_contract": "A single decimal number between 0.0 and 1.0 with no surrounding text. Parsed by regex (decimal preferred, integer fallback) and clamped to [0.0, 1.0]. Any non-parseable response falls back to 0.5.",
       "difficulty": "easy",
-      "grading": "exact",
+      "grading": "judge",
       "found_by": "bob"
     }
   ]

docs/case-study/mem0/downshift.local.yaml

@@ -0,0 +1,38 @@
+# Local run of mem0's reranker evals on Ollama. Same volumes as downshift.yaml;
+# models and illustrative prices from the SupportDesk config. Not mem0's real models.
+version: 1
+provider:
+  base_url: http://localhost:11434/v1
+models:
+  baseline: qwen2.5:7b
+  candidates:
+  - qwen2.5:3b
+  - qwen2.5:1.5b
+  - qwen2.5:0.5b
+quality_threshold: 0.95
+min_pass_rate: 0.8
+pricing:
+  qwen2.5:7b:
+    input: 2.5
+    output: 10.0
+    tier: premium
+  qwen2.5:3b:
+    input: 0.4
+    output: 1.6
+    tier: standard
+  qwen2.5:1.5b:
+    input: 0.15
+    output: 0.6
+    tier: budget
+  qwen2.5:0.5b:
+    input: 0.1
+    output: 0.4
+    tier: nano
+volume:
+  default_per_day: 0
+  per_call_site:
+    mem0/llms/openai.py::OpenAILLM.generate_response: 20000
+    mem0/memory/main.py::Memory._add_to_vector_store: 10000
+    mem0/reranker/llm_reranker.py::LLMReranker.rerank: 100000
+    mem0/memory/main.py::Memory._create_procedural_memory: 100
+    mem0/memory/utils.py::get_image_description: 500

docs/case-study/mem0/eval-report.md

@@ -0,0 +1,97 @@
+# Downshift report
+
+> Costs are projections: measured tokens per call x illustrative prices x assumed volume
+> from the config. They are not a bill.
+
+## Summary
+
+| | Monthly cost |
+|---|---:|
+| Before (all on `qwen2.5:7b`) | $1,564.38 |
+| After | $1,564.38 |
+| Savings | $0.00 (0.0%) |
+
+Downgraded **0 of 3** call sites.
+
+Rule: a cheaper model must keep at least 95% of the baseline pass rate and pass at least 80% of cases on its own. Decisions use pass rate, not mean score.
+
+Totals exclude 2 call sites with unknown cost (see Needs attention).
+
+## What this analysis cost
+
+| | One-time cost |
+|---|---:|
+| Eval model calls (96) | $0.02 |
+| Judge calls, estimated (96) | $0.28 |
+| **Total** | **$0.29** |
+
+No payback: nothing was downgraded, so there are no projected savings.
+
+> Priced at the same illustrative prices as the rest of the report. Judge tokens are not recorded, so each judge call is estimated as (case prompt + output + 150) tokens in and 200 out, priced as `qwen2.5:7b`. Retries and warm-up calls are not counted. Local Ollama runs cost $0 in practice.
+
+## Decisions
+
+| Call site | Grading | Decision | Model | Pass rate | Before / month | After / month |
+|---|---|---|---|---|---:|---:|
+| `mem0/memory/main.py::Memory._add_to_vector_store` | judge | keep | `qwen2.5:7b` | n/a | n/a | n/a |
+| `mem0/memory/main.py::Memory._create_procedural_memory` | judge | keep | `qwen2.5:7b` | n/a | n/a | n/a |
+| `mem0/reranker/llm_reranker.py::LLMReranker.rerank` | judge | keep | `qwen2.5:7b` | 71% | $1,564.38 | $1,564.38 |
+
+## Quality per model
+
+| Call site | `qwen2.5:7b` | `qwen2.5:3b` | `qwen2.5:1.5b` | `qwen2.5:0.5b` |
+|---|--- | --- | --- | ---|
+| `mem0/memory/main.py::Memory._add_to_vector_store` | **n/a** | n/a | n/a | n/a |
+| `mem0/memory/main.py::Memory._create_procedural_memory` | **n/a** | n/a | n/a | n/a |
+| `mem0/reranker/llm_reranker.py::LLMReranker.rerank` | **17/24 (71%), judge 3.8/5** | 13/24 (54%), judge 3.2/5 | 13/24 (54%), judge 3.2/5 | 11/24 (46%), judge 2.8/5 |
+
+Judge-graded cases pass at 4/5 or higher; judge scores are averages on a 1 to 5 scale.
+
+## Needs attention
+
+### Baseline below the floor
+
+- `mem0/reranker/llm_reranker.py::LLMReranker.rerank`: baseline passes 71% of cases, below the 80% floor.  Improve the prompt or model before downgrading.
+
+### Missing data
+
+- `examples/misc/movie_recommendation_grok3.py::recommend_movie_with_memory`: no eval set, not decided.
+- `integrations/hermes-plugin-mem0/_openai_llm.py::DirectOpenAILLM.generate_response`: no eval set, not decided.
+- `mem0/llms/anthropic.py::AnthropicLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/azure_openai.py::AzureOpenAILLM.generate_response`: no eval set, not decided.
+- `mem0/llms/azure_openai_structured.py::AzureOpenAIStructuredLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/deepseek.py::DeepSeekLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/groq.py::GroqLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/lmstudio.py::LMStudioLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/minimax.py::MiniMaxLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/openai_structured.py::OpenAIStructuredLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/together.py::TogetherLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/vllm.py::VllmLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/xai.py::XAILLM.generate_response`: no eval set, not decided.
+- `mem0/memory/utils.py::get_image_description`: no eval set, not decided.
+- `mem0/memory/main.py::Memory._add_to_vector_store`: no baseline results.
+- `mem0/memory/main.py::Memory._create_procedural_memory`: no baseline results.
+
+## Details
+
+<details>
+<summary><code>mem0/memory/main.py::Memory._add_to_vector_store</code>: keep <code>qwen2.5:7b</code></summary>
+
+- Decision: no baseline results
+
+</details>
+<details>
+<summary><code>mem0/memory/main.py::Memory._create_procedural_memory</code>: keep <code>qwen2.5:7b</code></summary>
+
+- Decision: no baseline results
+
+</details>
+<details>
+<summary><code>mem0/reranker/llm_reranker.py::LLMReranker.rerank</code>: keep <code>qwen2.5:7b</code></summary>
+
+- Decision: no cheaper model passed the checks
+- `qwen2.5:3b`: keeps 76% of baseline quality, needs 95%
+- `qwen2.5:1.5b`: keeps 76% of baseline quality, needs 95%
+- `qwen2.5:0.5b`: keeps 65% of baseline quality, needs 95%
+
+</details>

docs/case-study/mem0/evals-brief.md

@@ -0,0 +1,47 @@
+# mem0 eval brief (for Bob subagents)
+
+Write eval sets for 3 mem0 features. Format: `.bob/skills/downshift-evals/format-reference.md`
+(read it first). Ignore the SupportDesk-specific rules in that skill (refund policy, tickets).
+Output folder: `docs/case-study/mem0/evals/`. File name = slug of the call site id
+(drop `.py`, `/` -> `.`, `::` -> `__`). Every case has `grading: "judge"`, and `expected` is a
+rubric a grader can check point by point.
+
+Data rules: synthetic only. Invented people, no real names, emails, phone numbers or
+addresses. No copied text from any website or dataset.
+
+Every case: `id` (prefix + number), `inputs` with exactly the prompt's placeholders,
+`expected` rubric, `grading: "judge"`, one-line `notes`. Only cases a careful human
+would grade the same way. Do not run commands.
+
+## 1. mem0/memory/main.py::Memory._add_to_vector_store (prefix `mem-`, 15 cases)
+Placeholders: summary, last_k_messages, recently_extracted_memories_json,
+existing_memories_json, new_messages_json, observation_date, current_date.
+JSON placeholders are JSON strings (lists of `{"role","content"}` or of `{"id","text"}`).
+Keep conversations short (2 to 6 messages). Dates in 2026.
+Rubric: output is a JSON object `{"memory": [...]}`; list the facts that must be captured;
+each text is self-contained (names the person, no "he/she" without a referent); no fact
+already in existing or recently extracted memories is repeated; nothing invented.
+Cover: personal facts, preferences, plans with dates, assistant recommendations the user
+accepted, a conversation with nothing memorable (expected: empty list), duplicates of
+existing memories (must not repeat), relative dates ("next Friday") resolved against
+observation_date.
+
+## 2. mem0/memory/main.py::Memory._create_procedural_memory (prefix `proc-`, 12 cases)
+Placeholders: role, content (one agent-history message; role is "assistant" or "user").
+`content` is a short agent execution log, 3 to 8 steps: browsing, coding, data or file tasks.
+Rubric: has Task Objective and Progress Status; one numbered step per agent action in
+order; each step's Action Result is copied verbatim (quote the exact strings to check);
+errors in the log are recorded; nothing invented.
+Cover: a finished task, a half-done task, a task with an error and a retry, a log with
+exact numbers or URLs that must survive verbatim, a one-step log.
+
+## 3. mem0/reranker/llm_reranker.py::LLMReranker.rerank (prefix `rr-`, 24 cases)
+Placeholders: query, doc_text. Documents are short memory lines (1 to 2 sentences) like
+mem0 stores, e.g. "Priya prefers aisle seats on long flights."
+Expected rubric: "Output is only a single number (no words) between A and B." Use the
+prompt's own bands and keep them wide enough that careful humans agree:
+- 8 clearly relevant, directly answers the query: between 0.8 and 1.0
+- 8 related topic but does not answer: between 0.3 and 0.7
+- 8 unrelated: between 0.0 and 0.3
+Drop any case where the band is debatable. Include a few tricky ones: same keyword but
+different meaning, a paraphrase with no shared words, a negation ("dislikes" vs "likes").