Skip to content

fix: return ExactMatch mismatches for non-JSON primitives - #220

Open
Andrew Barnes (Bortlesboat) wants to merge 1 commit into
braintrustdata:mainfrom
Bortlesboat:fix/exact-match-non-json-values
Open

fix: return ExactMatch mismatches for non-JSON primitives#220
Andrew Barnes (Bortlesboat) wants to merge 1 commit into
braintrustdata:mainfrom
Bortlesboat:fix/exact-match-non-json-values

Conversation

@Bortlesboat

Copy link
Copy Markdown

Summary

ExactMatch()(output=None, expected={"amount": 100}) currently raises TypeError instead of returning a mismatch score. The same happens for booleans and numbers compared with a dict or list, including when the arguments are reversed. These comparisons now return 0, using the existing normalization fallback.

Validation

  • All 8 new parameterized cases failed before the fix.
  • pytest py/autoevals/test_values.py py/autoevals/test_json.py -q: 16 passed after the fix.
  • Black, Flake8, and git diff --check passed for the changed files.
  • Manually checked async evaluation, partial scorers, and existing bytes/bytearray JSON normalization.

The broader Python suite was attempted but is not green in this Windows environment: optional LiteLLM is missing, live API tests lack valid credentials, and the checkout's template symlink is not resolved.

Post-deploy validation

No service deployment is involved. After a package release, the example above should return a score of 0; I can help investigate any reported normalization regression.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant