feat(device-eval): on-device open-ended + RAG eval, host-proxy fidelity validated by nmrenyi · Pull Request #6 · nmrenyi/mamai-eval

nmrenyi · 2026-06-11T10:20:55Z

What

Adds an on-device open-ended eval capability and uses it to validate that the cluster host proxy faithfully approximates the deployed LiteRT device, across both eval tracks and ±RAG.

Harness changes (`end_to_end_eval/run_eval_device.py`)

Support open_ended (SAQ) device runs.
Support open_ended_rubric (HealthBench, multi-turn history split) + --rag (sets use_retrieval so the device does its own Gecko + vector-store retrieval).
Fresh-process batching (--batch-size 8) with retry, to survive aggressive on-device process killing of long sustained LLM runs. Batch intermediates write to a temp dir, not the result folder.

Drives the app-side runEval service in nmrenyi/mamai#66.

Results committed

Device (LiteRT) generations + cluster gpt-oss-120b @ medium judge scores for four arms — SAQ ±RAG (369/arm) and HealthBench ±RAG (stratified 150/arm sample, manifest configs/config-v0.2.0/manifests/healthbench_device_sample.json, seed 42).

Finding (`configs/config-v0.2.0/reports/device-vs-host-fidelity-20260611.html`)

Host proxy validated. Device tracks host closely on both tracks and arms. On the clean no-RAG comparison (identical inputs) device is consistently a hair better on every metric → the Phase B host numbers are a mildly pessimistic lower bound. +RAG ≈ tied (±1–2 pp; those arms also exercise device-side retrieval). Every Phase B conclusion holds on-device, and the host-only leaked chain-of-thought was absent across all 669 device generations.

Caveats: same judge both sides (bias cancels in the Δ), no bootstrap CIs, small per-subset n — read single-subset deltas as indicative.

🤖 Generated with Claude Code

Extend the on-device runner beyond MCQ to open_ended (SAQ) datasets. Builds eval_input with the app open-ended system prompt + the bare question (matching the host no-RAG arm; the device branch already sends user_message verbatim via bypassPromptFormatting), and shapes the pulled responses into the host open_ended result format ({id, question, reference, key_facts, model_response, inference_time_s}) so they can be scored later by rescore_open_v2 with the pinned cluster judge. Judge scoring is deferred (needs the vLLM endpoint); the runner records response coverage. Smoke-validated on device (whb, 2 rows): clean open-ended responses, no leaked chain-of-thought. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…port Make the device runner fully cover the host tracks: add --rag (sets use_retrieval in eval_input so the device runs on-device retrieval + deployed context-injection), and open_ended_rubric support (split each healthbench conversation into final-user-turn `user_message` + prior-turn `history`, shape results with `rubrics` for rescore_rubric). no-RAG SAQ path unchanged. Requires the eval-branch APK build that honors use_retrieval + history (drafted in the mamai app repo). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Gemma 4 E4B run on the LiteRT-LM device stack for the SAQ no-RAG arm (whb 20 + afrimedqa_saq 37 + kenya 312 = 369), captured via run_eval_device.py in fresh-process batches of 8 (works around aggressive on-device process killing). 369/369 responses, 0 empty, 0 leaked chain-of-thought — confirming the leaked-CoT is a host-eval artifact, not device behavior. Unscored (judge rescore to follow). Pairs with host no-RAG 20260520T104611-cluster-norag-openended. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…AQ no-RAG raw) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Device no-RAG SAQ rescored by the pinned judge (RUNAI_LOCAL submit, H200). Device recall runs slightly ABOVE host on all 3 datasets (kenya 0.194 vs 0.178, afrimedqa 0.211 vs 0.164, whb 0.079 vs 0.039) — the host GGUF proxy slightly understates recall but is close (<5pp). Corrects the raw-regex 'device deflects more' signal: judge recall shows device conveys MORE content, not less. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…er, retract raw deflection claim Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…trieval, 369 rows) Gemma 4 E4B on the LiteRT device stack, +RAG arm — genuine on-device retrieval (Gecko embeddings + vector store, ~3 docs/query) + deployed context-injection, via run_eval_device.py --rag in batches of 8. 369/369 responses, 0 empty, 0 leaked-CoT. Unscored (judge rescore to follow). Pairs with host +RAG 20260520T104610-cluster-rag-openended. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…set) Stratified device sample (oss_eval/consensus/hard, 50 each, seed 42) on the LiteRT stack, no-RAG, batched. 150/150, 0 empty, 0 leaked-CoT. Unscored (rescore_rubric to follow). Manifest: manifests/healthbench_device_sample.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> EOF )

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…lt folder Per-batch eval_input/output now go to a tempfile scratch dir so the result dir only holds the final {dataset}.json — no rm cleanup (and no approval prompts). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Device +RAG SAQ rescored. Recall again slightly ABOVE host (kenya 0.171 vs 0.128, afrimedqa 0.214 vs 0.162, whb 0.070 vs 0.026), but harm rate HIGHER than host here (kenya 23.7 vs 18.9, afrimedqa 18.9 vs 8.1) — opposite of the no-RAG arm. Caveat: the +RAG arms conflate retrieval differences (device on-device Gecko retrieval vs host precomputed contexts) with generation, so not an isolated stack comparison. (Judge job failed once on a transient pypi timeout; resubmit succeeded.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…l, 150 rows) Stratified device sample, +RAG (on-device Gecko retrieval + injection), batched. 150/150, 0 leaked-CoT. Unscored (rescore_rubric to follow). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…cluster) Both HB device sample arms (150 rows each) rescored by rescore_rubric. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… both tracks SAQ +RAG + Healthbench ±RAG folded in. Bottom line: LiteRT device tracks the host GGUF proxy closely; no-RAG device consistently slightly better (proxy is a mildly pessimistic lower bound), +RAG ~tied; all Phase B conclusions hold on-device; leaked-CoT is host-only (0 on device across 669 generations). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Restyle the device-vs-host (cluster GGUF) fidelity report as HTML to match the SAQ/rubric Phase B reports (color-coded device-vs-host tables, TL;DR, caveats). Delete the markdown version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Adds on-device open-ended evaluation artifacts (HealthBench tracks) and a summarized HTML report comparing LiteRT device outputs vs the host GGUF proxy, to validate fidelity across ±RAG arms.

Changes:

Commit on-device HealthBench eval inputs/outputs (oss_eval, consensus, hard) in batched chunks.
Add a standalone HTML report summarizing device-vs-host fidelity findings for v0.2 open-ended evals.

Reviewed changes

Copilot reviewed 45 out of 57 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b6.eval_output.json	Device HealthBench oss_eval batch 6 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b6.eval_input.json	Device HealthBench oss_eval batch 6 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b5.eval_output.json	Device HealthBench oss_eval batch 5 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b5.eval_input.json	Device HealthBench oss_eval batch 5 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b4.eval_output.json	Device HealthBench oss_eval batch 4 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b4.eval_input.json	Device HealthBench oss_eval batch 4 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b3.eval_output.json	Device HealthBench oss_eval batch 3 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b3.eval_input.json	Device HealthBench oss_eval batch 3 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b2.eval_output.json	Device HealthBench oss_eval batch 2 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b2.eval_input.json	Device HealthBench oss_eval batch 2 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b1.eval_output.json	Device HealthBench oss_eval batch 1 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b1.eval_input.json	Device HealthBench oss_eval batch 1 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b0.eval_output.json	Device HealthBench oss_eval batch 0 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_oss_eval.b0.eval_input.json	Device HealthBench oss_eval batch 0 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_hard.b6.eval_output.json	Device HealthBench hard batch 6 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_hard.b6.eval_input.json	Device HealthBench hard batch 6 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_hard.b5.eval_output.json	Device HealthBench hard batch 5 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_hard.b5.eval_input.json	Device HealthBench hard batch 5 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_hard.b4.eval_output.json	Device HealthBench hard batch 4 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_hard.b4.eval_input.json	Device HealthBench hard batch 4 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_hard.b3.eval_output.json	Device HealthBench hard batch 3 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_hard.b3.eval_input.json	Device HealthBench hard batch 3 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_hard.b2.eval_output.json	Device HealthBench hard batch 2 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_hard.b2.eval_input.json	Device HealthBench hard batch 2 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_hard.b1.eval_output.json	Device HealthBench hard batch 1 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_hard.b1.eval_input.json	Device HealthBench hard batch 1 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_hard.b0.eval_output.json	Device HealthBench hard batch 0 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b6.eval_output.json	Device HealthBench consensus batch 6 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b6.eval_input.json	Device HealthBench consensus batch 6 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b5.eval_output.json	Device HealthBench consensus batch 5 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b5.eval_input.json	Device HealthBench consensus batch 5 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b4.eval_output.json	Device HealthBench consensus batch 4 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b4.eval_input.json	Device HealthBench consensus batch 4 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b3.eval_output.json	Device HealthBench consensus batch 3 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b3.eval_input.json	Device HealthBench consensus batch 3 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b2.eval_output.json	Device HealthBench consensus batch 2 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b2.eval_input.json	Device HealthBench consensus batch 2 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b1.eval_output.json	Device HealthBench consensus batch 1 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b1.eval_input.json	Device HealthBench consensus batch 1 inputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b0.eval_output.json	Device HealthBench consensus batch 0 outputs
configs/config-v0.2.0/results/end_to_end_eval/gemma4-e4b-device/20260611T080836/healthbench_consensus.b0.eval_input.json	Device HealthBench consensus batch 0 inputs
configs/config-v0.2.0/reports/device-vs-host-fidelity-20260611.html	Summarized device-vs-host fidelity report

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 45 out of 57 changed files in this pull request and generated 3 comments.

nmrenyi · 2026-06-11T11:05:05Z

Accepted as known/intentional. device_serial is the USB transport ID of our own test phone, kept deliberately as run provenance (which physical device produced these results). It's a low-sensitivity hardware identifier — not a credential, network address, or third-party data — so we're retaining it.

nmrenyi · 2026-06-11T11:05:07Z

Accepted as known/intentional. device_serial is the USB transport ID of our own test phone, kept deliberately as run provenance (which physical device produced these results). It's a low-sensitivity hardware identifier — not a credential, network address, or third-party data — so we're retaining it.

nmrenyi · 2026-06-11T11:05:08Z

+  "metadata": {
+    "model": "gemma4-e4b-device",
+    "inference_venue": "device",
+    "device_serial": "9da65e62",


Accepted as known/intentional. device_serial is the USB transport ID of our own test phone, kept deliberately as run provenance (which physical device produced these results). It's a low-sensitivity hardware identifier — not a credential, network address, or third-party data — so we're retaining it.

nmrenyi and others added 15 commits June 10, 2026 20:12

docs(device-fidelity): preliminary device-vs-host comparison (MCQ + S…

e70cead

…AQ no-RAG raw) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs(device-fidelity): SAQ no-RAG judge-scored — device slightly bett…

2b6ddb1

…er, retract raw deflection claim Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chore: healthbench device fidelity sample manifest (150 ids, seed 42)

f6e8338

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

results(device): Healthbench +RAG sample (LiteRT + on-device retrieva…

edcb32d

…l, 150 rows) Stratified device sample, +RAG (on-device Gecko retrieval + injection), batched. 150/150, 0 leaked-CoT. Unscored (rescore_rubric to follow). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

results(device): judge-scored Healthbench ±RAG sample (gpt-oss-120b, …

757cb0b

…cluster) Both HB device sample arms (150 rows each) rescored by rescore_rubric. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings June 11, 2026 10:20

Copilot started reviewing on behalf of nmrenyi June 11, 2026 10:21 View session

nmrenyi mentioned this pull request Jun 11, 2026

feat(benchmark): generalize on-device eval branch for open-ended + RAG nmrenyi/mamai#66

Merged

Copilot AI reviewed Jun 11, 2026

View reviewed changes

Comment thread configs/config-v0.2.0/reports/device-vs-host-fidelity-20260611.html

fix(report): add viewport meta tag for mobile rendering (Copilot review)

912ccbb

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

nmrenyi requested a review from Copilot June 11, 2026 10:26

Copilot started reviewing on behalf of nmrenyi June 11, 2026 10:26 View session

Copilot AI reviewed Jun 11, 2026

View reviewed changes

nmrenyi merged commit 4f8f074 into main Jun 11, 2026
1 check passed

nmrenyi deleted the feat/device-eval-openended-20260610 branch June 11, 2026 11:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(device-eval): on-device open-ended + RAG eval, host-proxy fidelity validated - #6

feat(device-eval): on-device open-ended + RAG eval, host-proxy fidelity validated#6
nmrenyi merged 16 commits into
mainfrom
feat/device-eval-openended-20260610

nmrenyi commented Jun 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

nmrenyi Jun 11, 2026

Uh oh!

nmrenyi Jun 11, 2026

Uh oh!

nmrenyi Jun 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nmrenyi commented Jun 11, 2026

What

Harness changes (end_to_end_eval/run_eval_device.py)

Results committed

Finding (configs/config-v0.2.0/reports/device-vs-host-fidelity-20260611.html)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

nmrenyi Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

nmrenyi Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

nmrenyi Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Harness changes (`end_to_end_eval/run_eval_device.py`)

Finding (`configs/config-v0.2.0/reports/device-vs-host-fidelity-20260611.html`)