Repository navigation
bug: eval metrics module silently imports all evaluators regardless of dependencies, causing import errors for missing optional deps to surface as zero scores #6262
Description
Description
PR #6136 fixed ASCII-only tokenization in ROUGE-1 evaluation for non-English languages. However, the eval module structure has a broader issue: all evaluators are imported eagerly regardless of whether their optional dependencies are installed. Missing dependencies cause import-time failures that are caught and silently converted to zero scores.
Where
google/adk-python eval/ directory — metric registration and import system.
Impact
- Users get zero evaluation scores without realizing an optional dependency is missing
- No warning that
rouge-score,bert-score, or other eval deps weren't installed - Makes evaluation results silently unreliable
Expected Behavior
- Missing optional dependencies should raise clear errors at eval time, not silently return zero
- Or: evaluators should be lazily imported with user-friendly error messages
Related
- PR fix: ROUGE-1 eval returns 0 for non-English languages (ASCII-only tokenizer) #6136: fix: ROUGE-1 eval returns 0 for non-English languages (ASCII-only tokenizer)
Activity
llalitkumarrr commentedon Jul 3, 2026
Hello @tcconnally ,
Thank you for your contribution. Your PR is currently under review and we will reach out if any additional information is required. In the meantime, we appreciate your patience.
llalitkumarrr commentedon Jul 27, 2026
Hello @tcconnally ,
We noticed that you closed your pull request. Could you please let us know if you plan to update your contribution to resolve this issue?
Also, if possible, could you share any reproduction details (such as sample code or test files) so we can analyze the issue on our end as well?
tcconnally commentedon Aug 3, 2026
Thanks for the follow-up. To answer directly: I am not planning to update/reopen my eval-related contributions at this time (the earlier ROUGE-1 PRs were closed and the scope here is broader than those). I will close this report as withdrawn on my side so it does not linger as an open item — the underlying observation (eager evaluator imports masking missing optional deps as zero scores) is still valid if the maintainers want to pick it up independently.