diff --git a/bob_sessions/downshift_task14_mem0_evals_a.png b/bob_sessions/downshift_task14_mem0_evals_a.png
new file mode 100644
index 0000000..e10441f
Binary files /dev/null and b/bob_sessions/downshift_task14_mem0_evals_a.png differ
diff --git a/bob_sessions/downshift_task14_mem0_evals_b.png b/bob_sessions/downshift_task14_mem0_evals_b.png
new file mode 100644
index 0000000..0b4fbb7
Binary files /dev/null and b/bob_sessions/downshift_task14_mem0_evals_b.png differ
diff --git a/bob_sessions/downshift_task14_mem0_evals_skill.png b/bob_sessions/downshift_task14_mem0_evals_skill.png
new file mode 100644
index 0000000..65397ea
Binary files /dev/null and b/bob_sessions/downshift_task14_mem0_evals_skill.png differ
diff --git a/bob_sessions/downshift_task15_docs_a01.png b/bob_sessions/downshift_task15_docs_a01.png
new file mode 100644
index 0000000..7f8bb75
Binary files /dev/null and b/bob_sessions/downshift_task15_docs_a01.png differ
diff --git a/bob_sessions/downshift_task15_docs_a02.png b/bob_sessions/downshift_task15_docs_a02.png
new file mode 100644
index 0000000..0ea8560
Binary files /dev/null and b/bob_sessions/downshift_task15_docs_a02.png differ
diff --git a/bob_sessions/downshift_task15_docs_b.png b/bob_sessions/downshift_task15_docs_b.png
new file mode 100644
index 0000000..1de448d
Binary files /dev/null and b/bob_sessions/downshift_task15_docs_b.png differ
diff --git a/bob_sessions/downshift_task15_docs_c.png b/bob_sessions/downshift_task15_docs_c.png
new file mode 100644
index 0000000..b79c9d1
Binary files /dev/null and b/bob_sessions/downshift_task15_docs_c.png differ
diff --git a/bob_sessions/downshift_task16_review_a.png b/bob_sessions/downshift_task16_review_a.png
new file mode 100644
index 0000000..44f662d
Binary files /dev/null and b/bob_sessions/downshift_task16_review_a.png differ
diff --git a/bob_sessions/downshift_task16_review_b.png b/bob_sessions/downshift_task16_review_b.png
new file mode 100644
index 0000000..57b7552
Binary files /dev/null and b/bob_sessions/downshift_task16_review_b.png differ
diff --git a/bob_sessions/downshift_task16_review_c.png b/bob_sessions/downshift_task16_review_c.png
new file mode 100644
index 0000000..7fb329d
Binary files /dev/null and b/bob_sessions/downshift_task16_review_c.png differ
diff --git a/bob_sessions/downshift_task16_review_exclusions.png b/bob_sessions/downshift_task16_review_exclusions.png
new file mode 100644
index 0000000..31eb662
Binary files /dev/null and b/bob_sessions/downshift_task16_review_exclusions.png differ
diff --git a/docs/case-study-mem0.md b/docs/case-study-mem0.md
index 35105d0..4b02775 100644
--- a/docs/case-study-mem0.md
+++ b/docs/case-study-mem0.md
@@ -112,11 +112,50 @@ Full output: [estimate.md](case-study/mem0/estimate.md).
- **Output is priced at `max_tokens`**, a ceiling, so feature costs are an upper bound.
- **Tokens are estimated from words**; real token counts for long prompts are usually
higher. Runtime text (conversation, memories, documents) is not counted.
-- **No evals were run** (paid API calls, $0 budget), so no downgrade recommendations.
+- Evals were run locally (see below); no downgrade was safe for the reranker.
- **Default setup only.** The other provider adapters, the integration and the example
were copied from the scan unchanged and are not priced (0 calls a day, or no configured
price for the example's `grok-3-beta`).
+## Evals and a local run
+
+Three Bob subagents wrote eval sets in parallel (task B12): 15 cases for fact extraction,
+12 for procedural memory, and 24 for the reranker, for a total of 51 cases. Image
+description has no evals: it needs a vision model and real image URLs.
+
+Grading is judge-based for the reranker (it returns a decimal score, so exact match is
+meaningless; the rubric checks the score band) and for extraction (free-text memories).
+For procedural memory the message role was a template placeholder; it was set to
+`assistant` before running.
+
+The run used `downshift.local.yaml` on Ollama with Qwen 2.5 tiers and SupportDesk's
+illustrative prices, not mem0's real models. Reranker results across all four candidates:
+
+| Model | Pass rate |
+|---|---|
+| `qwen2.5:7b` (baseline) | 17/24 (71%) |
+| `qwen2.5:3b` | 13/24 (54%) |
+| `qwen2.5:1.5b` | 13/24 (54%) |
+| `qwen2.5:0.5b` | 11/24 (46%) |
+
+No safe downgrade: the best candidates (3b and 1.5b, 13/24) keep 76% of baseline quality, below the 95% bar. The
+failures cluster in the "related but does not answer" band (scores 0.3 to 0.7); all
+models handle clear yes/no cases. The baseline itself only passes 71% of cases, below the
+80% floor, so the prompt or model needs improvement before a downgrade makes sense.
+
+Bob's built-in code review of this branch caught two mislabeled reranker cases (rr-11 and
+rr-19: documents that directly answer the query, labeled mid or low). Their bands were fixed
+to 0.8 to 1.0 and only those two cases were re-run; the table above is after the fix.
+
+Fact extraction and procedural memory have eval sets but were not run in this pass: their
+prompts are long (the extraction system prompt alone is several thousand tokens), so the
+local run covered only the reranker. They can be run the same way with `--site`.
+
+The total analysis cost was $0.29 one-time at illustrative prices (96 eval calls at $0.02
+plus an estimated 96 judge calls at $0.28). Local Ollama runs cost $0 in practice.
+
+Full results: [eval-report.md](case-study/mem0/eval-report.md).
+
## Reproduce
```bash
diff --git a/docs/case-study/mem0/downshift.audit.json b/docs/case-study/mem0/downshift.audit.json
index 4ea3f42..9364380 100644
--- a/docs/case-study/mem0/downshift.audit.json
+++ b/docs/case-study/mem0/downshift.audit.json
@@ -1880,7 +1880,7 @@
"purpose": "Extract and store new facts from a conversation turn into the mem0 vector store using additive (ADD-only) extraction with memory linking.",
"output_contract": "JSON object with key 'memory' whose value is a list of objects; each object has 'id' (sequential string), 'text' (string, 15\u2013100 words), and optionally 'linked_memory_ids' (array of UUID strings). No UPDATE or DELETE operations. Parsed via json.loads(...).get('memory', []).",
"difficulty": "medium",
- "grading": "json_fields",
+ "grading": "judge",
"found_by": "bob"
},
{
@@ -1905,7 +1905,7 @@
"resolved": true
},
{
- "role": "{role}",
+ "role": "assistant",
"content": "{content}",
"resolved": true
},
@@ -1926,7 +1926,8 @@
"Bob: first message uses the PROCEDURAL_MEMORY_SYSTEM_PROMPT constant from mem0/configs/prompts.py unless the caller passes a custom 'prompt' argument, in which case that string replaces it. Placeholder {procedural_memory_system_prompt} represents the full constant value.",
"Bob: the middle messages are the raw conversation messages passed in by the caller (*messages splice), represented here as a single template row with {role}/{content} placeholders.",
"Bob: the async twin of this call (asyncio.to_thread wrapper at main.py line 3745) sends identical messages; no separate entry added for it.",
- "Filled after the audit by a script: Bob wrote a placeholder for the module constant PROCEDURAL_MEMORY_SYSTEM_PROMPT; its text was copied from mem0/configs/prompts.py (skill step 5)."
+ "Filled after the audit by a script: Bob wrote a placeholder for the module constant PROCEDURAL_MEMORY_SYSTEM_PROMPT; its text was copied from mem0/configs/prompts.py (skill step 5).",
+ "Eval prep: the agent-history message role was the template {role}; set to 'assistant' (agent output) so eval cases only fill {content}."
],
"purpose": "Summarise an agent's full execution history into a structured procedural memory record that captures every action, result, and key finding verbatim.",
"output_contract": "Free-text markdown summary following the PROCEDURAL_MEMORY_SYSTEM_PROMPT template: starts with Task Objective and Progress Status, followed by numbered agent-action steps each containing Action, Action Result (verbatim), Key Findings, Navigation History, Errors, and Current Context. Must be non-empty; ValueError raised otherwise.",
@@ -2016,7 +2017,7 @@
"purpose": "Score the relevance of a single retrieved memory document against a search query so that mem0 can rerank search results before returning them to the caller.",
"output_contract": "A single decimal number between 0.0 and 1.0 with no surrounding text. Parsed by regex (decimal preferred, integer fallback) and clamped to [0.0, 1.0]. Any non-parseable response falls back to 0.5.",
"difficulty": "easy",
- "grading": "exact",
+ "grading": "judge",
"found_by": "bob"
}
]
diff --git a/docs/case-study/mem0/downshift.local.yaml b/docs/case-study/mem0/downshift.local.yaml
new file mode 100644
index 0000000..38e1809
--- /dev/null
+++ b/docs/case-study/mem0/downshift.local.yaml
@@ -0,0 +1,38 @@
+# Local run of mem0's reranker evals on Ollama. Same volumes as downshift.yaml;
+# models and illustrative prices from the SupportDesk config. Not mem0's real models.
+version: 1
+provider:
+ base_url: http://localhost:11434/v1
+models:
+ baseline: qwen2.5:7b
+ candidates:
+ - qwen2.5:3b
+ - qwen2.5:1.5b
+ - qwen2.5:0.5b
+quality_threshold: 0.95
+min_pass_rate: 0.8
+pricing:
+ qwen2.5:7b:
+ input: 2.5
+ output: 10.0
+ tier: premium
+ qwen2.5:3b:
+ input: 0.4
+ output: 1.6
+ tier: standard
+ qwen2.5:1.5b:
+ input: 0.15
+ output: 0.6
+ tier: budget
+ qwen2.5:0.5b:
+ input: 0.1
+ output: 0.4
+ tier: nano
+volume:
+ default_per_day: 0
+ per_call_site:
+ mem0/llms/openai.py::OpenAILLM.generate_response: 20000
+ mem0/memory/main.py::Memory._add_to_vector_store: 10000
+ mem0/reranker/llm_reranker.py::LLMReranker.rerank: 100000
+ mem0/memory/main.py::Memory._create_procedural_memory: 100
+ mem0/memory/utils.py::get_image_description: 500
diff --git a/docs/case-study/mem0/eval-report.md b/docs/case-study/mem0/eval-report.md
new file mode 100644
index 0000000..c95a53d
--- /dev/null
+++ b/docs/case-study/mem0/eval-report.md
@@ -0,0 +1,97 @@
+# Downshift report
+
+> Costs are projections: measured tokens per call x illustrative prices x assumed volume
+> from the config. They are not a bill.
+
+## Summary
+
+| | Monthly cost |
+|---|---:|
+| Before (all on `qwen2.5:7b`) | $1,564.38 |
+| After | $1,564.38 |
+| Savings | $0.00 (0.0%) |
+
+Downgraded **0 of 3** call sites.
+
+Rule: a cheaper model must keep at least 95% of the baseline pass rate and pass at least 80% of cases on its own. Decisions use pass rate, not mean score.
+
+Totals exclude 2 call sites with unknown cost (see Needs attention).
+
+## What this analysis cost
+
+| | One-time cost |
+|---|---:|
+| Eval model calls (96) | $0.02 |
+| Judge calls, estimated (96) | $0.28 |
+| **Total** | **$0.29** |
+
+No payback: nothing was downgraded, so there are no projected savings.
+
+> Priced at the same illustrative prices as the rest of the report. Judge tokens are not recorded, so each judge call is estimated as (case prompt + output + 150) tokens in and 200 out, priced as `qwen2.5:7b`. Retries and warm-up calls are not counted. Local Ollama runs cost $0 in practice.
+
+## Decisions
+
+| Call site | Grading | Decision | Model | Pass rate | Before / month | After / month |
+|---|---|---|---|---|---:|---:|
+| `mem0/memory/main.py::Memory._add_to_vector_store` | judge | keep | `qwen2.5:7b` | n/a | n/a | n/a |
+| `mem0/memory/main.py::Memory._create_procedural_memory` | judge | keep | `qwen2.5:7b` | n/a | n/a | n/a |
+| `mem0/reranker/llm_reranker.py::LLMReranker.rerank` | judge | keep | `qwen2.5:7b` | 71% | $1,564.38 | $1,564.38 |
+
+## Quality per model
+
+| Call site | `qwen2.5:7b` | `qwen2.5:3b` | `qwen2.5:1.5b` | `qwen2.5:0.5b` |
+|---|--- | --- | --- | ---|
+| `mem0/memory/main.py::Memory._add_to_vector_store` | **n/a** | n/a | n/a | n/a |
+| `mem0/memory/main.py::Memory._create_procedural_memory` | **n/a** | n/a | n/a | n/a |
+| `mem0/reranker/llm_reranker.py::LLMReranker.rerank` | **17/24 (71%), judge 3.8/5** | 13/24 (54%), judge 3.2/5 | 13/24 (54%), judge 3.2/5 | 11/24 (46%), judge 2.8/5 |
+
+Judge-graded cases pass at 4/5 or higher; judge scores are averages on a 1 to 5 scale.
+
+## Needs attention
+
+### Baseline below the floor
+
+- `mem0/reranker/llm_reranker.py::LLMReranker.rerank`: baseline passes 71% of cases, below the 80% floor. Improve the prompt or model before downgrading.
+
+### Missing data
+
+- `examples/misc/movie_recommendation_grok3.py::recommend_movie_with_memory`: no eval set, not decided.
+- `integrations/hermes-plugin-mem0/_openai_llm.py::DirectOpenAILLM.generate_response`: no eval set, not decided.
+- `mem0/llms/anthropic.py::AnthropicLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/azure_openai.py::AzureOpenAILLM.generate_response`: no eval set, not decided.
+- `mem0/llms/azure_openai_structured.py::AzureOpenAIStructuredLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/deepseek.py::DeepSeekLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/groq.py::GroqLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/lmstudio.py::LMStudioLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/minimax.py::MiniMaxLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/openai_structured.py::OpenAIStructuredLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/together.py::TogetherLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/vllm.py::VllmLLM.generate_response`: no eval set, not decided.
+- `mem0/llms/xai.py::XAILLM.generate_response`: no eval set, not decided.
+- `mem0/memory/utils.py::get_image_description`: no eval set, not decided.
+- `mem0/memory/main.py::Memory._add_to_vector_store`: no baseline results.
+- `mem0/memory/main.py::Memory._create_procedural_memory`: no baseline results.
+
+## Details
+
+
+mem0/memory/main.py::Memory._add_to_vector_store: keep qwen2.5:7b
+
+- Decision: no baseline results
+
+
+
+mem0/memory/main.py::Memory._create_procedural_memory: keep qwen2.5:7b
+
+- Decision: no baseline results
+
+
+
+mem0/reranker/llm_reranker.py::LLMReranker.rerank: keep qwen2.5:7b
+
+- Decision: no cheaper model passed the checks
+- `qwen2.5:3b`: keeps 76% of baseline quality, needs 95%
+- `qwen2.5:1.5b`: keeps 76% of baseline quality, needs 95%
+- `qwen2.5:0.5b`: keeps 65% of baseline quality, needs 95%
+
+
diff --git a/docs/case-study/mem0/evals-brief.md b/docs/case-study/mem0/evals-brief.md
new file mode 100644
index 0000000..b218be4
--- /dev/null
+++ b/docs/case-study/mem0/evals-brief.md
@@ -0,0 +1,47 @@
+# mem0 eval brief (for Bob subagents)
+
+Write eval sets for 3 mem0 features. Format: `.bob/skills/downshift-evals/format-reference.md`
+(read it first). Ignore the SupportDesk-specific rules in that skill (refund policy, tickets).
+Output folder: `docs/case-study/mem0/evals/`. File name = slug of the call site id
+(drop `.py`, `/` -> `.`, `::` -> `__`). Every case has `grading: "judge"`, and `expected` is a
+rubric a grader can check point by point.
+
+Data rules: synthetic only. Invented people, no real names, emails, phone numbers or
+addresses. No copied text from any website or dataset.
+
+Every case: `id` (prefix + number), `inputs` with exactly the prompt's placeholders,
+`expected` rubric, `grading: "judge"`, one-line `notes`. Only cases a careful human
+would grade the same way. Do not run commands.
+
+## 1. mem0/memory/main.py::Memory._add_to_vector_store (prefix `mem-`, 15 cases)
+Placeholders: summary, last_k_messages, recently_extracted_memories_json,
+existing_memories_json, new_messages_json, observation_date, current_date.
+JSON placeholders are JSON strings (lists of `{"role","content"}` or of `{"id","text"}`).
+Keep conversations short (2 to 6 messages). Dates in 2026.
+Rubric: output is a JSON object `{"memory": [...]}`; list the facts that must be captured;
+each text is self-contained (names the person, no "he/she" without a referent); no fact
+already in existing or recently extracted memories is repeated; nothing invented.
+Cover: personal facts, preferences, plans with dates, assistant recommendations the user
+accepted, a conversation with nothing memorable (expected: empty list), duplicates of
+existing memories (must not repeat), relative dates ("next Friday") resolved against
+observation_date.
+
+## 2. mem0/memory/main.py::Memory._create_procedural_memory (prefix `proc-`, 12 cases)
+Placeholders: role, content (one agent-history message; role is "assistant" or "user").
+`content` is a short agent execution log, 3 to 8 steps: browsing, coding, data or file tasks.
+Rubric: has Task Objective and Progress Status; one numbered step per agent action in
+order; each step's Action Result is copied verbatim (quote the exact strings to check);
+errors in the log are recorded; nothing invented.
+Cover: a finished task, a half-done task, a task with an error and a retry, a log with
+exact numbers or URLs that must survive verbatim, a one-step log.
+
+## 3. mem0/reranker/llm_reranker.py::LLMReranker.rerank (prefix `rr-`, 24 cases)
+Placeholders: query, doc_text. Documents are short memory lines (1 to 2 sentences) like
+mem0 stores, e.g. "Priya prefers aisle seats on long flights."
+Expected rubric: "Output is only a single number (no words) between A and B." Use the
+prompt's own bands and keep them wide enough that careful humans agree:
+- 8 clearly relevant, directly answers the query: between 0.8 and 1.0
+- 8 related topic but does not answer: between 0.3 and 0.7
+- 8 unrelated: between 0.0 and 0.3
+Drop any case where the band is debatable. Include a few tricky ones: same keyword but
+different meaning, a paraphrase with no shared words, a negation ("dislikes" vs "likes").
diff --git a/docs/case-study/mem0/evals/mem0.memory.main__Memory._add_to_vector_store.jsonl b/docs/case-study/mem0/evals/mem0.memory.main__Memory._add_to_vector_store.jsonl
new file mode 100644
index 0000000..f41ef46
--- /dev/null
+++ b/docs/case-study/mem0/evals/mem0.memory.main__Memory._add_to_vector_store.jsonl
@@ -0,0 +1,29 @@
+{"id": "mem-01", "inputs": {"summary": "", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"Hi! I'm Tariq. I was born on March 14, 1990 and I work as a structural engineer in Brampton.\"}, {\"role\": \"assistant\", \"content\": \"Nice to meet you, Tariq! Happy to help.\"}]", "observation_date": "2026-04-02", "current_date": "2026-04-02"}, "expected": "Output is a JSON object {\"memory\": [...]}. Must capture: (1) Tariq's birthday is March 14, 1990. (2) Tariq works as a structural engineer. (3) Tariq lives in Brampton. Each text is self-contained and names Tariq explicitly. No facts invented beyond what appears in new_messages_json. List may be in any order.", "grading": "judge", "notes": "first-time user: empty summary and empty existing memories; personal facts — birthday, job, city"}
+
+{"id": "mem-02", "inputs": {"summary": "Nadia is a vegetarian who enjoys hiking.", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[{\"id\": \"e1\", \"text\": \"Nadia follows a vegetarian diet.\"}, {\"id\": \"e2\", \"text\": \"Nadia enjoys hiking.\"}]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"I'm also gluten-free now, and I hate flying — I always take trains when I travel.\"}, {\"role\": \"assistant\", \"content\": \"Got it! I'll keep those preferences in mind.\"}]", "observation_date": "2026-05-10", "current_date": "2026-05-10"}, "expected": "Output is a JSON object {\"memory\": [...]}. Must capture: (1) Nadia follows a gluten-free diet. (2) Nadia dislikes flying and prefers trains when travelling. Must NOT repeat that Nadia is vegetarian or that she enjoys hiking (already in existing_memories_json). No facts invented. Each text names Nadia explicitly.", "grading": "judge", "notes": "duplicates of existing memories must not be repeated; new dietary preference and travel preference added"}
+
+{"id": "mem-03", "inputs": {"summary": "Kofi is a software developer based in Accra.", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[{\"id\": \"e1\", \"text\": \"Kofi works as a software developer in Accra.\"}]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"I'm planning a trip to Lisbon next Friday. Can you suggest a good neighbourhood to stay in?\"}, {\"role\": \"assistant\", \"content\": \"I'd recommend Alfama for a historic feel or Príncipe Real for a quieter, upscale vibe.\"}, {\"role\": \"user\", \"content\": \"I like the sound of Príncipe Real. I'll book there.\"}]", "observation_date": "2026-06-09", "current_date": "2026-06-09"}, "expected": "Output is a JSON object {\"memory\": [...]}. Must capture: (1) Kofi is planning a trip to Lisbon on 2026-06-12 (next Friday relative to observation_date 2026-06-09, which is a Tuesday, so next Friday = 2026-06-12) — or the memory text must reference the resolved date, not just 'next Friday'. (2) Kofi was recommended Príncipe Real neighbourhood in Lisbon and accepted it. Must NOT repeat that Kofi is a developer in Accra (already in existing_memories). No facts invented. Each text names Kofi.", "grading": "judge", "notes": "relative date 'next Friday' resolved against observation_date 2026-06-09 → 2026-06-12; assistant recommendation accepted by user"}
+
+{"id": "mem-04", "inputs": {"summary": "Priya is a data analyst who loves jazz music.", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[{\"id\": \"e1\", \"text\": \"Priya is a data analyst.\"}, {\"id\": \"e2\", \"text\": \"Priya loves jazz music.\"}]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"What's the weather like today?\"}, {\"role\": \"assistant\", \"content\": \"I don't have real-time weather data, but I can help you find a weather app.\"}]", "observation_date": "2026-03-15", "current_date": "2026-03-15"}, "expected": "Output is a JSON object {\"memory\": [...]}. The memory list must be empty: []. No memorable personal facts, preferences, plans, or recommendations were exchanged. A weather inquiry with no personal disclosure contains nothing extractable.", "grading": "judge", "notes": "nothing memorable in the conversation — expected empty list"}
+
+{"id": "mem-05", "inputs": {"summary": "Amara is a nurse in Nairobi who prefers window seats on flights.", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[{\"id\": \"e1\", \"text\": \"Amara works as a nurse in Nairobi.\"}, {\"id\": \"e2\", \"text\": \"Amara prefers window seats on flights.\"}]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"Actually, I've switched to aisle seats — I need to get up a lot during long-haul flights.\"}, {\"role\": \"assistant\", \"content\": \"Noted! Aisle seats make a lot of sense for long trips.\"}]", "observation_date": "2026-07-20", "current_date": "2026-07-20"}, "expected": "Output is a JSON object {\"memory\": [...]}. Must capture: (1) Amara now prefers aisle seats on flights, especially on long-haul routes, because she needs to get up frequently. Must NOT repeat the old window-seat preference (already in existing_memories — even though it is now corrected, the instruction is ADD-only; the correction itself is a new fact). No facts invented. Text names Amara.", "grading": "judge", "notes": "user corrects a prior fact — old preference already in existing memories; new corrected preference must be captured"}
+
+{"id": "mem-06", "inputs": {"summary": "", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"I'm Linh, 28 years old, and I'm about to start a new job at a robotics company on September 1st, 2026. I'm also lactose intolerant and love rock climbing.\"}, {\"role\": \"assistant\", \"content\": \"Exciting news about the new job! Feel free to ask me anything.\"}]", "observation_date": "2026-08-05", "current_date": "2026-08-05"}, "expected": "Output is a JSON object {\"memory\": [...]}. Must capture: (1) Linh is 28 years old. (2) Linh is starting a new job at a robotics company on 2026-09-01. (3) Linh is lactose intolerant. (4) Linh enjoys rock climbing. Each text is self-contained and names Linh. No facts invented. Order may vary.", "grading": "judge", "notes": "mixed-topic conversation: age, job start date, dietary restriction, hobby — all must be captured separately"}
+
+{"id": "mem-07", "inputs": {"summary": "Bayo is a teacher who lives in Lagos.", "last_k_messages": "[]", "recently_extracted_memories_json": "[{\"id\": \"r1\", \"text\": \"Bayo is planning to attend a teacher's conference in Abuja on 2026-11-03.\"}]", "existing_memories_json": "[{\"id\": \"e1\", \"text\": \"Bayo is a teacher living in Lagos.\"}]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"I'm going to that Abuja conference on November 3rd — I already told you. But now I also signed up for a weekend workshop on creative writing in Lagos on October 11th, 2026.\"}, {\"role\": \"assistant\", \"content\": \"Great, I have both events noted.\"}]", "observation_date": "2026-09-25", "current_date": "2026-09-25"}, "expected": "Output is a JSON object {\"memory\": [...]}. Must capture: (1) Bayo signed up for a creative writing workshop in Lagos on 2026-10-11. Must NOT repeat the Abuja conference (already in recently_extracted_memories_json) or that Bayo is a teacher in Lagos (already in existing_memories_json). No facts invented. Text names Bayo.", "grading": "judge", "notes": "duplicate of recently_extracted memory must not be repeated; only the genuinely new plan is captured"}
+
+{"id": "mem-08", "inputs": {"summary": "Selin is a freelance graphic designer.", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[{\"id\": \"e1\", \"text\": \"Selin works as a freelance graphic designer.\"}]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"I need a good project management tool for freelancers. Any suggestions?\"}, {\"role\": \"assistant\", \"content\": \"I'd recommend Notion for its flexibility or Toggl Track if time-tracking is the priority. Many freelancers combine both.\"}, {\"role\": \"user\", \"content\": \"Notion sounds perfect, I'll try it.\"}]", "observation_date": "2026-02-14", "current_date": "2026-02-14"}, "expected": "Output is a JSON object {\"memory\": [...]}. Must capture: (1) Selin was recommended Notion as a project management tool for freelancers and decided to try it. Must NOT repeat that Selin is a freelance graphic designer (already in existing_memories). No facts invented. Text names Selin.", "grading": "judge", "notes": "assistant recommendation accepted by user; existing memory about profession must not be duplicated"}
+
+{"id": "mem-09", "inputs": {"summary": "Dmitri enjoys cooking Italian food and runs marathons.", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[{\"id\": \"e1\", \"text\": \"Dmitri enjoys cooking Italian food.\"}, {\"id\": \"e2\", \"text\": \"Dmitri runs marathons.\"}]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"I love Italian food — pasta especially.\"}, {\"role\": \"assistant\", \"content\": \"Italian cuisine is wonderful. Homemade pasta is especially rewarding.\"}]", "observation_date": "2026-01-20", "current_date": "2026-01-20"}, "expected": "Output is a JSON object {\"memory\": [...]}. The memory list should be empty or contain at most a minor refinement (e.g., Dmitri specifically loves pasta within Italian cuisine). Must NOT add a new entry that duplicates the existing fact that Dmitri enjoys Italian food. Nothing invented.", "grading": "judge", "notes": "conversation only restates an existing memory about Italian food — no new extractable facts; must not repeat existing entry"}
+
+{"id": "mem-10", "inputs": {"summary": "Yuna is a university student studying environmental science.", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[{\"id\": \"e1\", \"text\": \"Yuna is a university student studying environmental science.\"}]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"My thesis defence is scheduled for June 30th, 2026. I also just adopted a rescue cat named Miso.\"}, {\"role\": \"assistant\", \"content\": \"Congratulations on both milestones! Good luck with the defence.\"}]", "observation_date": "2026-04-18", "current_date": "2026-04-18"}, "expected": "Output is a JSON object {\"memory\": [...]}. Must capture: (1) Yuna's thesis defence is scheduled for 2026-06-30. (2) Yuna adopted a rescue cat named Miso. Must NOT repeat that Yuna is a student studying environmental science (already in existing_memories). No facts invented. Each text names Yuna.", "grading": "judge", "notes": "plan with explicit date and a new personal fact (pet); existing fact about studies must not be repeated"}
+
+{"id": "mem-11", "inputs": {"summary": "", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"Can you explain how photosynthesis works?\"}, {\"role\": \"assistant\", \"content\": \"Photosynthesis is the process by which plants convert sunlight, water, and carbon dioxide into glucose and oxygen.\"}]", "observation_date": "2026-03-01", "current_date": "2026-03-01"}, "expected": "Output is a JSON object {\"memory\": [...]}. The memory list must be empty: []. The conversation is a generic factual question with no personal disclosure. No names, preferences, plans, or facts about the user were revealed. Nothing extractable.", "grading": "judge", "notes": "nothing memorable — generic knowledge question with no personal content; expected empty list"}
+
+{"id": "mem-12", "inputs": {"summary": "Fatou is a journalist who travels frequently for work.", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[{\"id\": \"e1\", \"text\": \"Fatou is a journalist who travels frequently for work.\"}]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"I'm heading to Dakar for an assignment next Monday. I prefer boutique hotels over large chains, and I always request a room away from the elevator because of the noise.\"}, {\"role\": \"assistant\", \"content\": \"Noted! I'll keep those hotel preferences in mind for future trip planning.\"}]", "observation_date": "2026-10-07", "current_date": "2026-10-07"}, "expected": "Output is a JSON object {\"memory\": [...]}. Must capture: (1) Fatou is travelling to Dakar for a work assignment on 2026-10-12 (next Monday relative to observation_date 2026-10-07, a Wednesday, so next Monday = 2026-10-12). (2) Fatou prefers boutique hotels over large chain hotels when travelling. (3) Fatou requests hotel rooms away from the elevator due to noise. Must NOT repeat that Fatou is a journalist who travels frequently (already in existing_memories). No facts invented. Each text names Fatou.", "grading": "judge", "notes": "relative date 'next Monday' resolved against observation_date 2026-10-07 (Wednesday) → 2026-10-12; two travel preferences captured"}
+
+{"id": "mem-13", "inputs": {"summary": "Riku is a retired architect living in Sapporo.", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[{\"id\": \"e1\", \"text\": \"Riku is a retired architect living in Sapporo.\"}]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"I was born on July 7th, 1958. I'm also diabetic, so I follow a low-sugar diet. And I've been learning watercolour painting since last year — I love it.\"}, {\"role\": \"assistant\", \"content\": \"Those are wonderful things to share! Watercolour is a beautiful hobby.\"}]", "observation_date": "2026-05-22", "current_date": "2026-05-22"}, "expected": "Output is a JSON object {\"memory\": [...]}. Must capture: (1) Riku was born on July 7th, 1958. (2) Riku is diabetic and follows a low-sugar diet. (3) Riku has been learning watercolour painting since 2025 and enjoys it. Must NOT repeat that Riku is a retired architect in Sapporo (already in existing_memories). No facts invented. Each text names Riku.", "grading": "judge", "notes": "personal facts: birthday, health condition with dietary implication, hobby — all distinct from existing memory"}
+
+{"id": "mem-14", "inputs": {"summary": "Celeste is a product manager at a tech startup.", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[{\"id\": \"e1\", \"text\": \"Celeste works as a product manager at a tech startup.\"}]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"I need a good book for improving negotiation skills at work. Any ideas?\"}, {\"role\": \"assistant\", \"content\": \"I'd recommend 'Never Split the Difference' by Chris Voss — it's practical and widely praised by professionals.\"}, {\"role\": \"user\", \"content\": \"That sounds exactly right. I'll order it today.\"}]", "observation_date": "2026-09-03", "current_date": "2026-09-03"}, "expected": "Output is a JSON object {\"memory\": [...]}. Must capture: (1) Celeste was recommended the book 'Never Split the Difference' by Chris Voss for improving negotiation skills and agreed to order it on 2026-09-03. Must NOT repeat that Celeste is a product manager at a tech startup (already in existing_memories). No facts invented. Text names Celeste.", "grading": "judge", "notes": "assistant recommendation accepted by user; captures recommendation and user's acceptance with date context"}
+
+{"id": "mem-15", "inputs": {"summary": "Haruto is a chef who specialises in French cuisine.", "last_k_messages": "[]", "recently_extracted_memories_json": "[]", "existing_memories_json": "[{\"id\": \"e1\", \"text\": \"Haruto is a chef who specialises in French cuisine.\"}]", "new_messages_json": "[{\"role\": \"user\", \"content\": \"Actually I've shifted focus — I'm now specialising in Japanese fusion, not French. I also just opened my own restaurant called Umi in Osaka on March 5th, 2026.\"}, {\"role\": \"assistant\", \"content\": \"That's a big change! Japanese fusion is a very exciting space.\"}]", "observation_date": "2026-03-10", "current_date": "2026-03-10"}, "expected": "Output is a JSON object {\"memory\": [...]}. Must capture: (1) Haruto now specialises in Japanese fusion cuisine, not French (correction of existing fact). (2) Haruto opened his own restaurant called Umi in Osaka on 2026-03-05. Must NOT introduce a new entry that simply repeats the old fact that Haruto specialises in French cuisine. No facts invented. Each text names Haruto.", "grading": "judge", "notes": "user corrects a prior fact about cuisine specialisation; new restaurant opening with date also captured"}
diff --git a/docs/case-study/mem0/evals/mem0.memory.main__Memory._create_procedural_memory.jsonl b/docs/case-study/mem0/evals/mem0.memory.main__Memory._create_procedural_memory.jsonl
new file mode 100644
index 0000000..7355d2b
--- /dev/null
+++ b/docs/case-study/mem0/evals/mem0.memory.main__Memory._create_procedural_memory.jsonl
@@ -0,0 +1,12 @@
+{"id":"proc-01","inputs":{"content":"Step 1.\nAction: Navigate to https://fakestore.example.com/products\nResult: Page loaded successfully. Found 42 product listings.\n\nStep 2.\nAction: Extract product names and prices from the listing grid.\nResult: Extracted 42 rows. Sample: [{\"name\": \"Widget A\", \"price\": 9.99}, {\"name\": \"Gadget B\", \"price\": 24.50}]\n\nStep 3.\nAction: Save extracted data to /tmp/products.json\nResult: File written successfully. Size: 4.2 KB.\n\nStep 4.\nAction: Confirm file integrity by re-reading /tmp/products.json\nResult: 42 records verified. Task complete."},"expected":"Output starts with Task Objective identifying a web scraping task targeting https://fakestore.example.com/products. Progress Status states 100% complete (or equivalent finished status). Four numbered steps appear in order. Step 1 Action describes navigating to https://fakestore.example.com/products; Action Result contains verbatim \"Page loaded successfully. Found 42 product listings.\" Step 2 Action describes extracting product names and prices; Action Result contains verbatim \"Extracted 42 rows. Sample: [{\\\"name\\\": \\\"Widget A\\\", \\\"price\\\": 9.99}, {\\\"name\\\": \\\"Gadget B\\\", \\\"price\\\": 24.50}]\". Step 3 Action describes saving to /tmp/products.json; Action Result contains verbatim \"File written successfully. Size: 4.2 KB.\" Step 4 Action describes confirming integrity; Action Result contains verbatim \"42 records verified. Task complete.\" No information is invented beyond what appears in the input.","grading":"judge","notes":"Fully completed web scraping task; all 4 steps and verbatim results must be recorded"}
+{"id":"proc-02","inputs":{"content":"Step 1.\nAction: Connect to database at db.internal.example:5432\nResult: Connection established. Database version: PostgreSQL 15.2.\n\nStep 2.\nAction: Run query SELECT COUNT(*) FROM orders WHERE status='pending'\nResult: count = 1847\n\nStep 3.\nAction: Export pending orders to /data/pending_orders.csv\nResult: Export started. Writing rows...\n\nStep 4.\nAction: Wait for export to complete.\nResult: Export interrupted. Wrote 903 of 1847 rows before timeout after 30 seconds."},"expected":"Output starts with Task Objective describing exporting pending database orders. Progress Status reflects partial completion (approximately 49% or equivalent, or notes that 903 of 1847 rows were written). Four numbered steps appear in order. Step 1 Action Result contains verbatim \"Connection established. Database version: PostgreSQL 15.2.\" Step 2 Action Result contains verbatim \"count = 1847\". Step 3 Action Result contains verbatim \"Export started. Writing rows...\" Step 4 Action Result contains verbatim \"Export interrupted. Wrote 903 of 1847 rows before timeout after 30 seconds.\" No information is invented.","grading":"judge","notes":"Half-done database export task interrupted mid-execution; partial progress and exact row counts must be preserved"}
+{"id":"proc-03","inputs":{"content":"Step 1.\nAction: Download file from https://cdn.example.org/datasets/q3_report.zip\nResult: ERROR: ConnectionTimeoutError - Failed to connect to cdn.example.org after 10 seconds.\n\nStep 2.\nAction: Retry download from https://cdn.example.org/datasets/q3_report.zip with 3 retries.\nResult: ERROR: ConnectionTimeoutError - Failed to connect to cdn.example.org after 10 seconds.\n\nStep 3.\nAction: Attempt download from mirror https://mirror.example.org/datasets/q3_report.zip\nResult: Download successful. File size: 18.7 MB. Saved to /tmp/q3_report.zip.\n\nStep 4.\nAction: Extract /tmp/q3_report.zip to /data/q3_report/\nResult: Extraction complete. 14 files extracted."},"expected":"Output starts with Task Objective describing downloading and extracting q3_report.zip. Progress Status notes that the task completed after errors and a retry. Four numbered steps appear in order. Step 1 Action Result contains verbatim \"ERROR: ConnectionTimeoutError - Failed to connect to cdn.example.org after 10 seconds.\" Step 2 Action Result contains verbatim \"ERROR: ConnectionTimeoutError - Failed to connect to cdn.example.org after 10 seconds.\" Step 3 Action Result contains verbatim \"Download successful. File size: 18.7 MB. Saved to /tmp/q3_report.zip.\" Step 4 Action Result contains verbatim \"Extraction complete. 14 files extracted.\" Both error messages recorded exactly; no details invented.","grading":"judge","notes":"Task with connection error and retry; both error messages must appear verbatim before the successful retry"}
+{"id":"proc-04","inputs":{"content":"Step 1.\nAction: Fetch API endpoint https://api.example.io/v2/metrics?token=xK9mP3&from=2024-06-01&to=2024-06-30\nResult: 200 OK. Response: {\"total_requests\": 482019, \"error_rate\": 0.0023, \"p99_latency_ms\": 347}\n\nStep 2.\nAction: Write metric summary to /reports/june_2024_metrics.md\nResult: File created at /reports/june_2024_metrics.md. Wrote 512 bytes.\n\nStep 3.\nAction: Upload /reports/june_2024_metrics.md to s3://company-reports.example/2024/june/\nResult: Upload complete. ETag: \"d41d8cd98f00b204e9800998ecf8427e\""},"expected":"Output starts with Task Objective describing fetching and storing June 2024 API metrics. Three numbered steps in order. Step 1 Action references the URL https://api.example.io/v2/metrics?token=xK9mP3&from=2024-06-01&to=2024-06-30; Action Result contains verbatim \"{\\\"total_requests\\\": 482019, \\\"error_rate\\\": 0.0023, \\\"p99_latency_ms\\\": 347}\". Step 2 Action Result contains verbatim \"File created at /reports/june_2024_metrics.md. Wrote 512 bytes.\" Step 3 Action Result contains verbatim \"Upload complete. ETag: \\\"d41d8cd98f00b204e9800998ecf8427e\\\"\". All numbers and the ETag hash must appear exactly as given; nothing invented.","grading":"judge","notes":"Log with exact URLs, numeric metrics, and a hash that must survive verbatim"}
+{"id":"proc-05","inputs":{"content":"Step 1.\nAction: Ping healthcheck endpoint https://health.example.net/status\nResult: HTTP 200. Response body: {\"status\": \"ok\", \"uptime_seconds\": 93042}"},"expected":"Output starts with Task Objective identifying a health check against https://health.example.net/status. Progress Status notes the task is complete (single step). Exactly one numbered step appears. Step 1 Action describes pinging the health endpoint; Action Result contains verbatim \"{\\\"status\\\": \\\"ok\\\", \\\"uptime_seconds\\\": 93042}\". No additional steps are invented.","grading":"judge","notes":"One-step log; output must not invent additional steps and must record the single result verbatim"}
+{"id":"proc-06","inputs":{"content":"Step 1.\nAction: Create file /workspace/src/utils/string_utils.py with helper functions strip_whitespace and truncate.\nResult: File /workspace/src/utils/string_utils.py created. 47 lines written.\n\nStep 2.\nAction: Run unit tests: pytest /workspace/tests/test_string_utils.py -v\nResult: ============================= test session starts ==============================\ncollected 8 items\ntest_string_utils.py::test_strip_basic PASSED\ntest_string_utils.py::test_strip_empty PASSED\ntest_string_utils.py::test_truncate_short PASSED\ntest_string_utils.py::test_truncate_long PASSED\ntest_string_utils.py::test_truncate_exact PASSED\ntest_string_utils.py::test_truncate_unicode PASSED\ntest_string_utils.py::test_strip_tabs PASSED\ntest_string_utils.py::test_strip_newlines PASSED\n8 passed in 0.43s\n\nStep 3.\nAction: Commit changes with message \"Add string_utils helpers\"\nResult: [main a3f9c21] Add string_utils helpers\n 1 file changed, 47 insertions(+)"},"expected":"Output starts with Task Objective describing creating string utility helpers and running tests. Progress Status shows 100% complete. Three numbered steps in order. Step 1 Action Result contains verbatim \"File /workspace/src/utils/string_utils.py created. 47 lines written.\" Step 2 Action describes running pytest; Action Result contains the full pytest output verbatim including \"8 passed in 0.43s\". Step 3 Action Result contains verbatim \"[main a3f9c21] Add string_utils helpers\\n 1 file changed, 47 insertions(+)\". No details invented.","grading":"judge","notes":"Coding task with file creation, test run output, and a git commit; verbatim pytest output must be preserved"}
+{"id":"proc-07","inputs":{"content":"Step 1.\nAction: Read CSV file /data/sales_2024_q2.csv\nResult: Loaded 15230 rows and 9 columns. Columns: order_id, date, product_id, quantity, unit_price, discount, region, customer_id, channel.\n\nStep 2.\nAction: Filter rows where region == 'EMEA'\nResult: 4871 rows retained after filter.\n\nStep 3.\nAction: Compute total revenue per product_id (revenue = quantity * unit_price * (1 - discount))\nResult: Aggregation complete. Top result: product_id=PRD-8821, total_revenue=128450.75\n\nStep 4.\nAction: Write aggregated results to /data/emea_revenue_q2.csv\nResult: File /data/emea_revenue_q2.csv written. 312 unique product IDs, 312 rows."},"expected":"Output starts with Task Objective describing a CSV sales data aggregation for the EMEA region. Four numbered steps in order. Step 1 Action Result contains verbatim \"Loaded 15230 rows and 9 columns. Columns: order_id, date, product_id, quantity, unit_price, discount, region, customer_id, channel.\" Step 2 Action Result contains verbatim \"4871 rows retained after filter.\" Step 3 Action Result contains verbatim \"Aggregation complete. Top result: product_id=PRD-8821, total_revenue=128450.75\" Step 4 Action Result contains verbatim \"File /data/emea_revenue_q2.csv written. 312 unique product IDs, 312 rows.\" All numeric values exact; nothing invented.","grading":"judge","notes":"Data task with CSV read, filter, aggregation, and write; numeric row counts and revenue values must be exact"}
+{"id":"proc-08","inputs":{"content":"Step 1.\nAction: Copy /archive/2023/logs/app.log to /backup/2023/logs/app.log.bak\nResult: Copy successful. Destination size: 2.1 GB.\n\nStep 2.\nAction: Rename /archive/2023/logs/app.log to /archive/2023/logs/app.log.processed\nResult: Rename successful.\n\nStep 3.\nAction: Delete temporary file /tmp/scratch_parse_20230912.tmp\nResult: ERROR: FileNotFoundError - /tmp/scratch_parse_20230912.tmp does not exist.\n\nStep 4.\nAction: Verify /backup/2023/logs/app.log.bak exists and is readable.\nResult: File exists. Size: 2.1 GB. MD5: 9e107d9d372bb6826bd81d3542a419d8"},"expected":"Output starts with Task Objective describing a file backup and cleanup operation on 2023 logs. Four numbered steps in order. Step 1 Action Result contains verbatim \"Copy successful. Destination size: 2.1 GB.\" Step 2 Action Result contains verbatim \"Rename successful.\" Step 3 Action Result contains verbatim \"ERROR: FileNotFoundError - /tmp/scratch_parse_20230912.tmp does not exist.\" noting it as an error. Step 4 Action Result contains verbatim \"File exists. Size: 2.1 GB. MD5: 9e107d9d372bb6826bd81d3542a419d8\". The error in step 3 must be captured; nothing invented.","grading":"judge","notes":"File operation task (copy, rename, delete) with a FileNotFoundError on the delete step; error must be verbatim"}
+{"id":"proc-09","inputs":{"content":"Step 1.\nAction: Open browser and navigate to https://news.example.org/\nResult: Page loaded. Title: \"Example News - Home\". 20 article links visible.\n\nStep 2.\nAction: Click on article \"Q3 Earnings Beat Forecasts\" (link: /articles/q3-earnings-2024)\nResult: Navigated to https://news.example.org/articles/q3-earnings-2024. Article loaded. Word count: 1240.\n\nStep 3.\nAction: Extract article body text.\nResult: Extracted 1240 words. First sentence: \"Markets rallied on Thursday after FakeCorp reported Q3 earnings of $2.47 per share, beating analyst consensus of $2.10.\"\n\nStep 4.\nAction: Navigate back to https://news.example.org/ to check remaining articles.\nResult: Returned to home page. 19 unvisited article links remain.\n\nStep 5.\nAction: Click on article \"Central Bank Holds Rates\" (link: /articles/central-bank-oct-2024)\nResult: Navigated to https://news.example.org/articles/central-bank-oct-2024. Article loaded. Word count: 875."},"expected":"Output starts with Task Objective describing browsing and extracting news articles from https://news.example.org/. Navigation History section (or equivalent) reflects: https://news.example.org/ → https://news.example.org/articles/q3-earnings-2024 → https://news.example.org/ → https://news.example.org/articles/central-bank-oct-2024. Five numbered steps in order. Step 2 Action Result contains verbatim \"Navigated to https://news.example.org/articles/q3-earnings-2024. Article loaded. Word count: 1240.\" Step 3 Action Result contains verbatim \"First sentence: \\\"Markets rallied on Thursday after FakeCorp reported Q3 earnings of $2.47 per share, beating analyst consensus of $2.10.\\\"\" Step 4 Action Result contains verbatim \"Returned to home page. 19 unvisited article links remain.\" Step 5 Action Result contains verbatim \"Navigated to https://news.example.org/articles/central-bank-oct-2024. Article loaded. Word count: 875.\" Navigation history must match the visited URLs in order.","grading":"judge","notes":"Browsing task with explicit navigation history across 4 URLs; navigation history section must list all visited URLs in order"}
+{"id":"proc-10","inputs":{"content":"Step 1.\nAction: Authenticate to API https://api.internal.example/auth with credentials from env.\nResult: ERROR: AuthenticationError - Invalid API key. Status 401.\n\nStep 2.\nAction: Reload credentials from /etc/app/secrets.env and retry authentication.\nResult: ERROR: AuthenticationError - Invalid API key. Status 401.\n\nStep 3.\nAction: Check /etc/app/secrets.env for key format issues.\nResult: ERROR: ValueError - API key 'XZ--badkey' does not match expected format '^[A-Z0-9]{32}$'.\n\nStep 4.\nAction: Request new API key from administrator (simulated: read from /etc/app/secrets_v2.env).\nResult: Loaded new key. Format validated successfully.\n\nStep 5.\nAction: Retry authentication with new key.\nResult: Authenticated successfully. Token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9 (truncated). Expires in 3600 seconds."},"expected":"Output starts with Task Objective describing authenticating to https://api.internal.example/auth. Progress Status notes multiple authentication failures before eventual success. Five numbered steps in order. Step 1 Action Result contains verbatim \"ERROR: AuthenticationError - Invalid API key. Status 401.\" Step 2 Action Result contains verbatim \"ERROR: AuthenticationError - Invalid API key. Status 401.\" Step 3 Action Result contains verbatim \"ERROR: ValueError - API key 'XZ--badkey' does not match expected format '^[A-Z0-9]{32}$'.\" All three errors must be recorded in their respective steps. Step 4 Action Result contains verbatim \"Loaded new key. Format validated successfully.\" Step 5 Action Result contains verbatim \"Authenticated successfully. Token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9 (truncated). Expires in 3600 seconds.\" No invented steps or details.","grading":"judge","notes":"Multiple errors in sequence (two 401s then a format error) before success; all three error messages must appear verbatim"}
+{"id":"proc-11","inputs":{"content":"Step 1.\nAction: Send GET request to https://api.weather.example/v1/forecast?city=Fakestad&days=7\nResult: 200 OK. Returned 7-day forecast for Fakestad. Temperatures range from 12°C to 24°C.\n\nStep 2.\nAction: Parse forecast JSON and extract daily high temperatures.\nResult: Highs: [18, 21, 24, 22, 19, 17, 12] (°C)\n\nStep 3.\nAction: Compute average high temperature.\nResult: Average high: 19.0°C\n\nStep 4.\nAction: Store result in memory key 'fakestad_forecast_avg_high'.\nResult: Stored. Value: 19.0"},"expected":"Output starts with Task Objective describing fetching and summarising a 7-day weather forecast for Fakestad. Progress Status shows task complete. Four numbered steps in order. Step 1 Action references URL https://api.weather.example/v1/forecast?city=Fakestad&days=7; Action Result contains verbatim \"200 OK. Returned 7-day forecast for Fakestad. Temperatures range from 12°C to 24°C.\" Step 2 Action Result contains verbatim \"Highs: [18, 21, 24, 22, 19, 17, 12] (°C)\". Step 3 Action Result contains verbatim \"Average high: 19.0°C\". Step 4 Action Result contains verbatim \"Stored. Value: 19.0\". All temperature values exact; nothing invented.","grading":"judge","notes":"assistant-role input; API fetch plus computation; numeric temperature array and average must survive verbatim"}
+{"id":"proc-12","inputs":{"content":"Step 1.\nAction: Upload /uploads/raw_transactions_nov.csv to processing pipeline.\nResult: File accepted. Job ID: JOB-77412. Queued for processing.\n\nStep 2.\nAction: Poll job status for JOB-77412.\nResult: Status: RUNNING. Progress: 34%\n\nStep 3.\nAction: Poll job status for JOB-77412 again (after 30 seconds).\nResult: Status: COMPLETED. Processed 88320 rows. Output at /results/transactions_nov_clean.csv.\n\nStep 4.\nAction: Download /results/transactions_nov_clean.csv and validate row count.\nResult: Downloaded. Local row count: 88320. Checksum: SHA256=c4ca4238a0b923820dcc509a6f75849b.\n\nStep 5.\nAction: Archive original file: move /uploads/raw_transactions_nov.csv to /archive/nov/\nResult: Moved successfully. Archive path: /archive/nov/raw_transactions_nov.csv."},"expected":"Output starts with Task Objective describing a CSV transaction file upload and processing pipeline for November data. Five numbered steps in order. Step 1 Action Result contains verbatim \"File accepted. Job ID: JOB-77412. Queued for processing.\" Step 2 Action Result contains verbatim \"Status: RUNNING. Progress: 34%\". Step 3 Action Result contains verbatim \"Status: COMPLETED. Processed 88320 rows. Output at /results/transactions_nov_clean.csv.\" Step 4 Action Result contains verbatim \"Downloaded. Local row count: 88320. Checksum: SHA256=c4ca4238a0b923820dcc509a6f75849b.\" Step 5 Action Result contains verbatim \"Moved successfully. Archive path: /archive/nov/raw_transactions_nov.csv.\" The job ID JOB-77412 and checksum must appear exactly; nothing invented.","grading":"judge","notes":"user-role input; multi-step data pipeline with job polling, row counts, a SHA256 checksum, and a final archive move"}
diff --git a/docs/case-study/mem0/evals/mem0.reranker.llm_reranker__LLMReranker.rerank.jsonl b/docs/case-study/mem0/evals/mem0.reranker.llm_reranker__LLMReranker.rerank.jsonl
new file mode 100644
index 0000000..1dc6213
--- /dev/null
+++ b/docs/case-study/mem0/evals/mem0.reranker.llm_reranker__LLMReranker.rerank.jsonl
@@ -0,0 +1,24 @@
+{"id":"rr-01","inputs":{"query":"What does Priya prefer for seating on long flights?","doc_text":"Priya prefers aisle seats on long flights."},"expected":"Output is only a single number (no words) between 0.8 and 1.0.","grading":"judge","notes":"Exact match — query asks directly about flight seating preference stored in memory"}
+{"id":"rr-02","inputs":{"query":"What kind of coffee does Tariq drink in the morning?","doc_text":"Tariq drinks a black coffee with no sugar every morning."},"expected":"Output is only a single number (no words) between 0.8 and 1.0.","grading":"judge","notes":"Direct answer — morning coffee habit stored as a memory fact"}
+{"id":"rr-03","inputs":{"query":"Does Lena have any dietary restrictions?","doc_text":"Lena is lactose intolerant and avoids all dairy products."},"expected":"Output is only a single number (no words) between 0.8 and 1.0.","grading":"judge","notes":"Direct answer — dietary restriction perfectly matches the query intent"}
+{"id":"rr-04","inputs":{"query":"What programming language does Omar use most at work?","doc_text":"Omar primarily uses Python for data engineering tasks at his job."},"expected":"Output is only a single number (no words) between 0.8 and 1.0.","grading":"judge","notes":"Direct answer — work programming language stored as a memory fact"}
+{"id":"rr-05","inputs":{"query":"Where does Sana like to vacation?","doc_text":"Sana loves visiting coastal towns in southern Portugal every summer."},"expected":"Output is only a single number (no words) between 0.8 and 1.0.","grading":"judge","notes":"Direct answer — preferred vacation destination stored in memory"}
+{"id":"rr-06","inputs":{"query":"What time does Kenji usually wake up?","doc_text":"Kenji sets his alarm for 5:45 AM on weekdays."},"expected":"Output is only a single number (no words) between 0.8 and 1.0.","grading":"judge","notes":"Direct answer — wake-up routine directly answers the query"}
+{"id":"rr-07","inputs":{"query":"What is Fatima's preferred way to exercise?","doc_text":"Fatima goes for a 30-minute run outdoors three times a week."},"expected":"Output is only a single number (no words) between 0.8 and 1.0.","grading":"judge","notes":"Direct answer — exercise preference fully answers the query"}
+{"id":"rr-08","inputs":{"query":"Does Rohan own a car?","doc_text":"Rohan owns a 2019 blue Honda Civic."},"expected":"Output is only a single number (no words) between 0.8 and 1.0.","grading":"judge","notes":"Direct answer — vehicle ownership directly addresses the query"}
+{"id":"rr-09","inputs":{"query":"What does Priya like to eat for breakfast?","doc_text":"Priya prefers aisle seats on long flights."},"expected":"Output is only a single number (no words) between 0.3 and 0.7.","grading":"judge","notes":"Same person, unrelated fact — flight preference does not answer food question"}
+{"id":"rr-10","inputs":{"query":"What languages does Omar speak?","doc_text":"Omar primarily uses Python for data engineering tasks at his job."},"expected":"Output is only a single number (no words) between 0.3 and 0.7.","grading":"judge","notes":"Related domain (Omar, languages) but programming language ≠ spoken language — partial relevance"}
+{"id":"rr-11","inputs":{"query":"Does Tariq have a caffeine habit?","doc_text":"Tariq drinks a black coffee with no sugar every morning."},"expected":"Output is only a single number (no words) between 0.8 and 1.0.","grading":"judge","notes":"Paraphrase — 'caffeine habit' and 'drinks coffee every morning' overlap but doc is more specific than the query implies (band fixed after Bob review: the document directly answers the query)"}
+{"id":"rr-12","inputs":{"query":"What does Lena avoid eating?","doc_text":"Lena is a certified yoga instructor who teaches on weekends."},"expected":"Output is only a single number (no words) between 0.3 and 0.7.","grading":"judge","notes":"Same person, different topic — yoga instruction is loosely health-related but doesn't address dietary avoidance"}
+{"id":"rr-13","inputs":{"query":"Does Kenji exercise regularly?","doc_text":"Kenji sets his alarm for 5:45 AM on weekdays."},"expected":"Output is only a single number (no words) between 0.3 and 0.7.","grading":"judge","notes":"Related lifestyle topic — early wake-up may imply active routine but doesn't confirm exercise"}
+{"id":"rr-14","inputs":{"query":"What hobbies does Sana have?","doc_text":"Sana loves visiting coastal towns in southern Portugal every summer."},"expected":"Output is only a single number (no words) between 0.3 and 0.7.","grading":"judge","notes":"Vacation travel is hobby-adjacent but does not enumerate hobbies; moderate relevance"}
+{"id":"rr-15","inputs":{"query":"How does Fatima get to work?","doc_text":"Fatima goes for a 30-minute run outdoors three times a week."},"expected":"Output is only a single number (no words) between 0.3 and 0.7.","grading":"judge","notes":"Running is mentioned but as exercise, not commuting — related activity, wrong context"}
+{"id":"rr-16","inputs":{"query":"What kind of music does Rohan listen to?","doc_text":"Rohan owns a 2019 blue Honda Civic and often drives on weekends."},"expected":"Output is only a single number (no words) between 0.3 and 0.7.","grading":"judge","notes":"Driving could imply listening to music but doc says nothing about music — weak indirect link"}
+{"id":"rr-17","inputs":{"query":"What is Tariq's favorite Java framework?","doc_text":"Tariq enjoys brewing single-origin Java coffee beans from Indonesia."},"expected":"Output is only a single number (no words) between 0.0 and 0.3.","grading":"judge","notes":"Tricky: 'Java' keyword matches but query means the programming language, doc means the coffee origin — unrelated meaning"}
+{"id":"rr-18","inputs":{"query":"What does Omar like to do on weekends?","doc_text":"Lena is lactose intolerant and avoids all dairy products."},"expected":"Output is only a single number (no words) between 0.0 and 0.3.","grading":"judge","notes":"Different person, different topic — completely irrelevant"}
+{"id":"rr-19","inputs":{"query":"Does Priya like window seats on planes?","doc_text":"Priya always requests aisle seats and dislikes window seats on any flight."},"expected":"Output is only a single number (no words) between 0.8 and 1.0.","grading":"judge","notes":"Tricky negation — query asks if she likes window seats; doc says she dislikes them, so it answers 'no' but the preference is opposite to relevance (band fixed after Bob review: the document directly answers the query)"}
+{"id":"rr-20","inputs":{"query":"What programming language does Kenji prefer?","doc_text":"Kenji is allergic to peanuts and carries an epinephrine pen."},"expected":"Output is only a single number (no words) between 0.0 and 0.3.","grading":"judge","notes":"Same person, completely unrelated fact — allergy vs programming language"}
+{"id":"rr-21","inputs":{"query":"What city does Fatima live in?","doc_text":"Rohan owns a 2019 blue Honda Civic."},"expected":"Output is only a single number (no words) between 0.0 and 0.3.","grading":"judge","notes":"Different person, different topic — no relevance whatsoever"}
+{"id":"rr-22","inputs":{"query":"What time does Sana go to bed?","doc_text":"Tariq drinks a black coffee with no sugar every morning."},"expected":"Output is only a single number (no words) between 0.0 and 0.3.","grading":"judge","notes":"Different person, different routine — no overlap in query or topic"}
+{"id":"rr-23","inputs":{"query":"Is Lena a morning person?","doc_text":"Fatima goes for a 30-minute run outdoors three times a week."},"expected":"Output is only a single number (no words) between 0.0 and 0.3.","grading":"judge","notes":"Different person, different topic — exercise routine of Fatima is irrelevant to Lena's sleep habits"}
+{"id":"rr-24","inputs":{"query":"Does Omar enjoy spicy food?","doc_text":"Omar primarily uses Python for data engineering tasks at his job."},"expected":"Output is only a single number (no words) between 0.0 and 0.3.","grading":"judge","notes":"Same person but completely different domain — programming fact doesn't relate to food preferences"}
diff --git a/docs/case-study/mem0/results/mem0.reranker.llm_reranker__LLMReranker.rerank/qwen2.5-0.5b.jsonl b/docs/case-study/mem0/results/mem0.reranker.llm_reranker__LLMReranker.rerank/qwen2.5-0.5b.jsonl
new file mode 100644
index 0000000..97b067a
--- /dev/null
+++ b/docs/case-study/mem0/results/mem0.reranker.llm_reranker__LLMReranker.rerank/qwen2.5-0.5b.jsonl
@@ -0,0 +1,24 @@
+{"case_id": "rr-01", "model": "qwen2.5:0.5b", "output": "1.0", "prompt_tokens": 191, "completion_tokens": 4, "latency_s": 0.101, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output exactly a single numeric score (1.0) within the required 0.8‑1.0 range, with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-02", "model": "qwen2.5:0.5b", "output": "1.0", "prompt_tokens": 197, "completion_tokens": 4, "latency_s": 0.089, "score": 1.0, "passed": true, "detail": "judge 5/5: Output is exactly a single numeric score within the required range", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-03", "model": "qwen2.5:0.5b", "output": "1.0", "prompt_tokens": 190, "completion_tokens": 4, "latency_s": 0.087, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric score of 1.0, which satisfies the requirement of being only a number within the 0.8‑1.0 range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-04", "model": "qwen2.5:0.5b", "output": "1.0", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.121, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric score of 1.0, satisfying the requirement of being only a number within the 0.8‑1.0 range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-05", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 191, "completion_tokens": 4, "latency_s": 0.097, "score": 0.0, "passed": false, "detail": "judge 1/5: Output number 0.0 is not within the required 0.8–1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-06", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 195, "completion_tokens": 4, "latency_s": 0.1, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.8-1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-07", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 197, "completion_tokens": 4, "latency_s": 0.087, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.8-1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-08", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 191, "completion_tokens": 4, "latency_s": 0.105, "score": 0.0, "passed": false, "detail": "judge 1/5: Output number is not within the required 0.8‑1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-09", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 190, "completion_tokens": 4, "latency_s": 0.089, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.3–0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-10", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 189, "completion_tokens": 4, "latency_s": 0.173, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but 0.0 is outside the required 0.3–0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-12", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 188, "completion_tokens": 4, "latency_s": 0.097, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but it's 0.0, which is outside the required 0.3–0.7 range.", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-13", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 192, "completion_tokens": 4, "latency_s": 0.086, "score": 0.0, "passed": false, "detail": "judge 1/5: The assistant gave a single number but it was 0.0, outside the required 0.3–0.7 range.", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-14", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 190, "completion_tokens": 4, "latency_s": 0.094, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but 0.0 is outside the required 0.3–0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-15", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 195, "completion_tokens": 4, "latency_s": 0.096, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.3–0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-16", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 199, "completion_tokens": 4, "latency_s": 0.092, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.3-0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-17", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 194, "completion_tokens": 4, "latency_s": 0.091, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value within the required 0.0‑0.3 range, meeting the rubric exactly.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-18", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 192, "completion_tokens": 4, "latency_s": 0.104, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, meeting the requirement of a number between 0.0 and 0.3 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-20", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 194, "completion_tokens": 4, "latency_s": 0.087, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, meeting the requirement of a number between 0.0 and 0.3 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-21", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 192, "completion_tokens": 4, "latency_s": 0.086, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output exactly a single numeric score within the required 0.0‑0.3 range with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-22", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.099, "score": 1.0, "passed": true, "detail": "judge 5/5: The output is exactly a single numeric value within the required 0.0‑0.3 range with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-23", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.094, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, satisfying the requirement of a lone number within the 0.0‑0.3 range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-24", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 189, "completion_tokens": 4, "latency_s": 0.102, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, satisfying the requirement of a number between 0.0 and 0.3 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-11", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.119, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.8-1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-19", "model": "qwen2.5:0.5b", "output": "0.0", "prompt_tokens": 194, "completion_tokens": 4, "latency_s": 0.093, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.8–1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
diff --git a/docs/case-study/mem0/results/mem0.reranker.llm_reranker__LLMReranker.rerank/qwen2.5-1.5b.jsonl b/docs/case-study/mem0/results/mem0.reranker.llm_reranker__LLMReranker.rerank/qwen2.5-1.5b.jsonl
new file mode 100644
index 0000000..b3ded11
--- /dev/null
+++ b/docs/case-study/mem0/results/mem0.reranker.llm_reranker__LLMReranker.rerank/qwen2.5-1.5b.jsonl
@@ -0,0 +1,24 @@
+{"case_id": "rr-01", "model": "qwen2.5:1.5b", "output": "1.0", "prompt_tokens": 191, "completion_tokens": 4, "latency_s": 0.278, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output exactly a single numeric score (1.0) within the required 0.8‑1.0 range, with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-02", "model": "qwen2.5:1.5b", "output": "0.4", "prompt_tokens": 197, "completion_tokens": 4, "latency_s": 0.15, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.8–1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-03", "model": "qwen2.5:1.5b", "output": "0.8", "prompt_tokens": 190, "completion_tokens": 4, "latency_s": 0.141, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric score of 0.8, satisfying the requirement of a number between 0.8 and 1.0 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-04", "model": "qwen2.5:1.5b", "output": "0.8", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.135, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric score of 0.8, satisfying the requirement of a number between 0.8 and 1.0 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-05", "model": "qwen2.5:1.5b", "output": "0.8", "prompt_tokens": 191, "completion_tokens": 4, "latency_s": 0.139, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.8, meeting the requirement of a number between 0.8 and 1.0 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-06", "model": "qwen2.5:1.5b", "output": "0.4", "prompt_tokens": 195, "completion_tokens": 4, "latency_s": 0.145, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.8–1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-07", "model": "qwen2.5:1.5b", "output": "0.4", "prompt_tokens": 197, "completion_tokens": 4, "latency_s": 0.165, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.8–1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-08", "model": "qwen2.5:1.5b", "output": "0.8", "prompt_tokens": 191, "completion_tokens": 4, "latency_s": 0.149, "score": 1.0, "passed": true, "detail": "judge 5/5: The output is exactly a single numeric value within the required range, with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-09", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 190, "completion_tokens": 4, "latency_s": 0.147, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.3–0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-10", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 189, "completion_tokens": 4, "latency_s": 0.232, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but it's 0.0, which is outside the required 0.3–0.7 range.", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-12", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 188, "completion_tokens": 4, "latency_s": 0.155, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but it's 0.0, which is outside the required 0.3–0.7 range.", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-13", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 192, "completion_tokens": 4, "latency_s": 0.161, "score": 0.0, "passed": false, "detail": "judge 1/5: The assistant gave a single number but it was 0.0, which is outside the required 0.3–0.7 range.", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-14", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 190, "completion_tokens": 4, "latency_s": 0.149, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.3-0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-15", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 195, "completion_tokens": 4, "latency_s": 0.157, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.3–0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-16", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 199, "completion_tokens": 4, "latency_s": 0.176, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but it is 0.0, which is outside the required 0.3–0.7 range.", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-17", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 194, "completion_tokens": 4, "latency_s": 0.142, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, satisfying the requirement of a number between 0.0 and 0.3 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-18", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 192, "completion_tokens": 4, "latency_s": 0.159, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, satisfying the requirement of a lone number within the 0.0‑0.3 range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-20", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 194, "completion_tokens": 4, "latency_s": 0.152, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output exactly a single numeric score within the required 0.0‑0.3 range with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-21", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 192, "completion_tokens": 4, "latency_s": 0.145, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, satisfying the requirement of being only a number within the 0.0‑0.3 range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-22", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.149, "score": 1.0, "passed": true, "detail": "judge 5/5: The response is exactly a single numeric value within the required 0.0‑0.3 range with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-23", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.147, "score": 1.0, "passed": true, "detail": "judge 5/5: The output is exactly a single numeric value within the required 0.0‑0.3 range with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-24", "model": "qwen2.5:1.5b", "output": "0.0", "prompt_tokens": 189, "completion_tokens": 4, "latency_s": 0.159, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, satisfying the requirement of a number between 0.0 and 0.3 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-11", "model": "qwen2.5:1.5b", "output": "0.8", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.269, "score": 1.0, "passed": true, "detail": "judge 5/5: The output is exactly a single numeric value (0.8) with no extra text, satisfying the required range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-19", "model": "qwen2.5:1.5b", "output": "0.4", "prompt_tokens": 194, "completion_tokens": 4, "latency_s": 0.211, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.8–1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
diff --git a/docs/case-study/mem0/results/mem0.reranker.llm_reranker__LLMReranker.rerank/qwen2.5-3b.jsonl b/docs/case-study/mem0/results/mem0.reranker.llm_reranker__LLMReranker.rerank/qwen2.5-3b.jsonl
new file mode 100644
index 0000000..663fe38
--- /dev/null
+++ b/docs/case-study/mem0/results/mem0.reranker.llm_reranker__LLMReranker.rerank/qwen2.5-3b.jsonl
@@ -0,0 +1,24 @@
+{"case_id": "rr-01", "model": "qwen2.5:3b", "output": "1.0", "prompt_tokens": 191, "completion_tokens": 4, "latency_s": 0.522, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output exactly a single numeric score within the required range, with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-02", "model": "qwen2.5:3b", "output": "0.9", "prompt_tokens": 197, "completion_tokens": 4, "latency_s": 0.224, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output is exactly a single numeric value (0.9) within the required 0.8–1.0 range, with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-03", "model": "qwen2.5:3b", "output": "1.0", "prompt_tokens": 190, "completion_tokens": 4, "latency_s": 0.219, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output exactly a single numeric score (1.0) within the required 0.8‑1.0 range, with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-04", "model": "qwen2.5:3b", "output": "0.8", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.234, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric score of 0.8, satisfying the requirement of a number between 0.8 and 1.0 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-05", "model": "qwen2.5:3b", "output": "0.8", "prompt_tokens": 191, "completion_tokens": 4, "latency_s": 0.225, "score": 1.0, "passed": true, "detail": "judge 5/5: The output is exactly a single numeric value within the required range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-06", "model": "qwen2.5:3b", "output": "0.4", "prompt_tokens": 195, "completion_tokens": 4, "latency_s": 0.242, "score": 0.0, "passed": false, "detail": "judge 1/5: Output number 0.4 is not within the required 0.8‑1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-07", "model": "qwen2.5:3b", "output": "0.6", "prompt_tokens": 197, "completion_tokens": 4, "latency_s": 0.23, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but falls outside the required 0.8–1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-08", "model": "qwen2.5:3b", "output": "1.0", "prompt_tokens": 191, "completion_tokens": 4, "latency_s": 0.239, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output exactly a single numeric score (1.0) within the required 0.8‑1.0 range, meeting the rubric.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-09", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 190, "completion_tokens": 4, "latency_s": 0.235, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.3–0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-10", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 189, "completion_tokens": 4, "latency_s": 0.217, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but it's 0.0, which is outside the required 0.3–0.7 range.", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-12", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 188, "completion_tokens": 4, "latency_s": 0.234, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but it's 0.0, which is outside the required 0.3–0.7 range.", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-13", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 192, "completion_tokens": 4, "latency_s": 0.237, "score": 0.0, "passed": false, "detail": "judge 1/5: The assistant gave a single number but it was 0.0, which is outside the required 0.3–0.7 range.", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-14", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 190, "completion_tokens": 4, "latency_s": 0.219, "score": 0.0, "passed": false, "detail": "judge 1/5: Output number 0.0 is not within the required 0.3–0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-15", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 195, "completion_tokens": 4, "latency_s": 0.224, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.3–0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-16", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 199, "completion_tokens": 4, "latency_s": 0.269, "score": 0.0, "passed": false, "detail": "judge 1/5: The output is a single number with no words, but it is 0.0, which is outside the required 0.3–0.7 range.", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-17", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 194, "completion_tokens": 4, "latency_s": 0.216, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value within the required 0.0‑0.3 range, meeting the rubric exactly.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-18", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 192, "completion_tokens": 4, "latency_s": 0.217, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, satisfying the requirement of a lone number within the 0.0‑0.3 range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-20", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 194, "completion_tokens": 4, "latency_s": 0.225, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, meeting the requirement of a number between 0.0 and 0.3 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-21", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 192, "completion_tokens": 4, "latency_s": 0.249, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, meeting the requirement of a number between 0.0 and 0.3 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-22", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.232, "score": 1.0, "passed": true, "detail": "judge 5/5: The output is exactly a single numeric value within the required 0.0‑0.3 range with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-23", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.228, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, satisfying the requirement of a lone number within the 0.0‑0.3 range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-24", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 189, "completion_tokens": 4, "latency_s": 0.233, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, satisfying the requirement of a number between 0.0 and 0.3 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-11", "model": "qwen2.5:3b", "output": "0.4", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.532, "score": 0.0, "passed": false, "detail": "judge 1/5: Output number 0.4 is not within the required 0.8–1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-19", "model": "qwen2.5:3b", "output": "0.0", "prompt_tokens": 194, "completion_tokens": 4, "latency_s": 0.225, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.8–1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
diff --git a/docs/case-study/mem0/results/mem0.reranker.llm_reranker__LLMReranker.rerank/qwen2.5-7b.jsonl b/docs/case-study/mem0/results/mem0.reranker.llm_reranker__LLMReranker.rerank/qwen2.5-7b.jsonl
new file mode 100644
index 0000000..1b91b90
--- /dev/null
+++ b/docs/case-study/mem0/results/mem0.reranker.llm_reranker__LLMReranker.rerank/qwen2.5-7b.jsonl
@@ -0,0 +1,24 @@
+{"case_id": "rr-01", "model": "qwen2.5:7b", "output": "1.0", "prompt_tokens": 191, "completion_tokens": 4, "latency_s": 1.15, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output exactly a single numeric score (1.0) within the required 0.8‑1.0 range, with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-02", "model": "qwen2.5:7b", "output": "0.8", "prompt_tokens": 197, "completion_tokens": 4, "latency_s": 0.377, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric score (0.8) with no extra text, satisfying the required range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-03", "model": "qwen2.5:7b", "output": "1.0", "prompt_tokens": 190, "completion_tokens": 4, "latency_s": 0.383, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output exactly a single numeric score (1.0) within the required 0.8‑1.0 range, with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-04", "model": "qwen2.5:7b", "output": "0.8", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.373, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric score of 0.8, meeting the requirement of a number between 0.8 and 1.0 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-05", "model": "qwen2.5:7b", "output": "0.8", "prompt_tokens": 191, "completion_tokens": 4, "latency_s": 0.402, "score": 1.0, "passed": true, "detail": "judge 5/5: The output is exactly a single numeric value within the required range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-06", "model": "qwen2.5:7b", "output": "0.8", "prompt_tokens": 195, "completion_tokens": 4, "latency_s": 0.385, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.8, satisfying the requirement of being a number between 0.8 and 1.0 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-07", "model": "qwen2.5:7b", "output": "0.8", "prompt_tokens": 197, "completion_tokens": 4, "latency_s": 0.423, "score": 1.0, "passed": true, "detail": "judge 5/5: The output is exactly a single numeric value within the required range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-08", "model": "qwen2.5:7b", "output": "1.0", "prompt_tokens": 191, "completion_tokens": 4, "latency_s": 0.389, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric score (1.0) within the required 0.8‑1.0 range, with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-09", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 190, "completion_tokens": 4, "latency_s": 0.374, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.3–0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-10", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 189, "completion_tokens": 4, "latency_s": 0.491, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.3–0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-12", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 188, "completion_tokens": 4, "latency_s": 0.376, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.3-0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-13", "model": "qwen2.5:7b", "output": "0.3", "prompt_tokens": 192, "completion_tokens": 4, "latency_s": 0.375, "score": 1.0, "passed": true, "detail": "judge 5/5: The response is a single numeric value within the required 0.3‑0.7 range and contains no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-14", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 190, "completion_tokens": 4, "latency_s": 0.373, "score": 0.0, "passed": false, "detail": "judge 1/5: Output number 0.0 is not within the required 0.3–0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-15", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 195, "completion_tokens": 4, "latency_s": 0.383, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.3–0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-16", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 199, "completion_tokens": 4, "latency_s": 0.528, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.3-0.7 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-17", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 194, "completion_tokens": 4, "latency_s": 0.394, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, satisfying the requirement of a number between 0.0 and 0.3 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-18", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 192, "completion_tokens": 4, "latency_s": 0.384, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, meeting the requirement of a number between 0.0 and 0.3 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-20", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 194, "completion_tokens": 4, "latency_s": 0.382, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, meeting the requirement of a number between 0.0 and 0.3 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-21", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 192, "completion_tokens": 4, "latency_s": 0.385, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output exactly a single numeric score within the required 0.0‑0.3 range with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-22", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.382, "score": 1.0, "passed": true, "detail": "judge 5/5: The response is exactly a single numeric value within the required 0.0‑0.3 range with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-23", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 0.402, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, satisfying the requirement of a lone number within the 0.0‑0.3 range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-24", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 189, "completion_tokens": 4, "latency_s": 0.403, "score": 1.0, "passed": true, "detail": "judge 5/5: The assistant output a single numeric value 0.0, meeting the requirement of a number between 0.0 and 0.3 with no extra text.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-11", "model": "qwen2.5:7b", "output": "0.8", "prompt_tokens": 193, "completion_tokens": 4, "latency_s": 1.185, "score": 1.0, "passed": true, "detail": "judge 5/5: The output is exactly a single numeric value within the required range.", "judge_score": 5, "error": null, "judge_model": "openai/gpt-oss-120b"}
+{"case_id": "rr-19", "model": "qwen2.5:7b", "output": "0.0", "prompt_tokens": 194, "completion_tokens": 4, "latency_s": 0.397, "score": 0.0, "passed": false, "detail": "judge 1/5: Output is a single number but not within the required 0.8‑1.0 range", "judge_score": 1, "error": null, "judge_model": "openai/gpt-oss-120b"}
diff --git a/src/downshift/report.py b/src/downshift/report.py
index 34cf2fd..7267c7f 100644
--- a/src/downshift/report.py
+++ b/src/downshift/report.py
@@ -258,8 +258,11 @@ def render_markdown(report: Report) -> str:
lines.append(f"| Assistant audit | {_fmt_money(ac.audit_cost)} |")
lines.append(f"| **Total** | **{_fmt_money(ac.total)}** |")
lines.append("")
- pb_str = format_payback(report.payback) if report.payback is not None else "n/a"
- lines.append(f"Pays back in **{pb_str}** of projected savings.")
+ if report.payback is not None and report.payback.hours is not None:
+ pb_str = format_payback(report.payback)
+ lines.append(f"Pays back in **{pb_str}** of projected savings.")
+ elif report.payback is not None:
+ lines.append("No payback: nothing was downgraded, so there are no projected savings.")
lines.append("")
blockquote = "> Priced at the same illustrative prices as the rest of the report."
if ac.judge_calls > 0:
diff --git a/tests/unit/test_report.py b/tests/unit/test_report.py
index cd76f0a..f5d392d 100644
--- a/tests/unit/test_report.py
+++ b/tests/unit/test_report.py
@@ -647,6 +647,8 @@ def test_judge_row_absent_when_no_judge_calls() -> None:
assert "## What this analysis cost" in md
assert "Judge calls" not in md
assert "Judge tokens are not recorded" not in md
+ assert "No payback: nothing was downgraded" in md
+ assert "Pays back in" not in md
def test_audit_row_absent_when_audit_cost_zero() -> None:
diff --git a/web/lib/content.ts b/web/lib/content.ts
index b1abcd3..d070d3f 100644
--- a/web/lib/content.ts
+++ b/web/lib/content.ts
@@ -304,11 +304,12 @@ export const CASE_STUDIES: CaseStudy[] = [
"Static-only pricing underestimates this bill about 7.6x.",
"The reranker calls the LLM once per candidate document. At 10 candidates, 10,000 searches become 100,000 LLM calls: $675 a month, 29% of the bill, from one for loop.",
"Fact extraction re-sends a 33,653-character prompt on every add(). Input alone is about $382 of that feature's $1,582, a strong candidate for prompt caching or a cheaper model.",
+ "51 eval cases (extraction 15, procedural memory 12, reranker 24) were written by 3 Bob subagents in parallel and run locally on Qwen 2.5 tiers; Bob's code review caught 2 mislabeled cases, fixed before the final run. No safe downgrade was found: the best candidates (3b, 1.5b) keep only 76% of reranker quality, below the 95% bar, with failures clustering in the ambiguous mid-band. The reranker baseline itself passes 71% of cases, below the 80% floor, so the prompt needs improvement before a downgrade can be made.",
],
limitations: [
"Projection, not a bill: OpenAI list prices (checked Sep 26, 2026) x assumed volume of 10,000 adds and 10,000 reranked searches a day.",
"Output is priced at max_tokens, a ceiling, so feature costs are an upper bound. Tokens are estimated from words; runtime text is not counted.",
- "No evals were run on this repo (paid API calls, $0 budget), so there are no downgrade recommendations here.",
+ "Evals run locally on Qwen 2.5 with illustrative prices, not mem0's real models. Only the reranker was run; fact extraction and procedural memory have eval sets but were not run (long prompts, left for later).",
"Default setup only: the other provider adapters, the integration and the example were not audited or priced.",
],
},