feat(benchmark): generalize on-device eval branch for open-ended + RAG - #66
Merged
Conversation
…n-device RAG
Generalize the eval-mode branch (runMcqEval -> runEval) so the on-device eval
covers all host tracks, not just MCQ:
- use_retrieval flag (top-level in eval_input): when true, run on-device
retrieval + the deployed context-injection (the +RAG arm); when false, send
user_message verbatim (MCQ / SAQ no-RAG), as before.
- per-row optional `history` ([{role, text}]): replayed as conversation
history for multi-turn healthbench (rubric) prompts.
- bypassPromptFormatting = !use_retrieval (verbatim no-RAG; production
injection +RAG).
Backward-compatible with existing MCQ eval_input (no use_retrieval/history ->
identical behaviour).
NOTE: not yet build-validated — needs a Flutter/Gradle APK rebuild + install to
compile and run on device. Pairs with run_eval_device.py --rag / rubric support
in the mamai-eval repo.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR generalizes the Android BenchmarkForegroundService eval-mode path so the on-device harness can run non-MCQ (open-ended) evals and optionally exercise the full on-device RAG + context-injection formatting, including multi-turn conversation replay.
Changes:
- Renames
runMcqEval→runEvaland broadens the eval runner to handle open-ended eval rows. - Adds top-level
use_retrievalhandling to switch between verbatim/no-RAG vs production-style RAG + prompt formatting. - Adds per-row optional
historyparsing to support multi-turn HealthBench-style eval rows.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+539
to
+548
| val history = mutableListOf<Map<String, String>>() | ||
| row.optJSONArray("history")?.let { h -> | ||
| for (j in 0 until h.length()) { | ||
| val turn = h.getJSONObject(j) | ||
| history.add(mapOf( | ||
| "role" to turn.optString("role", "user"), | ||
| "text" to turn.optString("text", ""), | ||
| )) | ||
| } | ||
| } |
Owner
Author
There was a problem hiding this comment.
Fixed in 75b6b25: roles are normalized to user/model (with common aliases human/assistant/model/bot/ai), blank-text turns are skipped, and an unrecognized role is logged and skipped rather than silently defaulting to user. Verified compileDebugKotlin passes.
…view) RagPipeline maps role=="user" to a user turn and everything else to a model turn, while the eval parser defaulted a missing role to "user" and accepted any string — so an 'assistant'/'system' typo would silently mis-attribute a turn. Normalize to user/model with common aliases, skip blank-text turns, and log + skip on an unrecognized role instead of defaulting silently. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment on lines
473
to
477
| * On-device evaluation runner. Reads a pushed `eval_input.json` | ||
| * ({system_prompt, use_retrieval?, rows: [{id, user_message, history?}]}), | ||
| * iterates rows through the same LiteRT pipeline production uses, and | ||
| * writes `eval_output.json` ({rows: [{id, response_text, error?}]}) for | ||
| * the harness (run_eval_device.py) to pull and score. |
Comment on lines
+537
to
+541
| // Optional multi-turn history (healthbench): [{role, text}, …] replayed | ||
| // before user_message. Absent for single-turn MCQ/SAQ rows. | ||
| // RagPipeline maps role=="user" → user turn and everything else → | ||
| // model turn, so normalize to "user"/"model" here and fail loudly on | ||
| // anything unexpected rather than silently mis-attributing a turn. |
| val turn = h.getJSONObject(j) | ||
| val text = turn.optString("text", "").trim() | ||
| if (text.isEmpty()) continue // skip blank turns | ||
| val role = when (turn.optString("role", "").trim().lowercase()) { |
Owner
Author
There was a problem hiding this comment.
No change needed here: Kotlin's no-arg String.lowercase() performs a locale-invariant (Locale.ROOT) Unicode mapping — it's precisely the replacement for the deprecated locale-sensitive toLowerCase(). The Turkish 'AI'→'aı' issue applies to toLowerCase()/lowercase(Locale.getDefault()), not to the no-arg form used here, so role normalization is already locale-stable.
…Copilot) - KDoc eval_output rows now list inference_time_ms (was omitted). - History-role comment now describes the actual behavior (skip blank turns; log + skip unrecognized role) instead of 'fail loudly'. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Generalizes the Android
BenchmarkForegroundServiceeval branch so the on-device harness can drive open-ended evals (SAQ + HealthBench rubric) and on-device RAG, not just MCQ.Changes
runMcqEval→runEval— handles any eval set type, not only multiple-choice.use_retrievalfrom the eval input (bypassPromptFormatting = !useRetrieval), so a single arm can run with or without on-device retrieval (Gecko + vector store).historyarray → conversation history, for HealthBench-style multi-turn rows.Build-validated via a rebuilt APK and exercised end-to-end by the eval-side harness.
Why
This is the app-side half of the device-fidelity work. The eval harness (
run_eval_device.pyin nmrenyi/mamai-eval#6) pushes aneval_input.json, triggers this service in eval mode, and pullseval_output.json. We used it to confirm the cluster host proxy is a faithful (slightly pessimistic) stand-in for the deployed LiteRT device across SAQ and HealthBench, ±RAG.🤖 Generated with Claude Code