Skip to content

fix(python): honor ValidJSON schemas across scorer call paths - #221

Open
JianYan (JianYan11) wants to merge 1 commit into
braintrustdata:mainfrom
JianYan11:fix/python-validjson-schema
Open

JianYan (JianYan11) wants to merge 1 commit into
braintrustdata:mainfrom
JianYan11:fix/python-validjson-schema

Conversation

@JianYan11

Copy link
Copy Markdown

Summary

Fix Python ValidJSON schema handling across constructor, keyword, positional, and partial scorer calls.

Two model-free repros on current main:

schema = {"type": "object", "required": ["answer"]}
ValidJSON(schema=schema).eval({}).score  # 1, but should be 0
ValidJSON().eval({}, schema=schema)     # TypeError: multiple values for 'schema'

The constructor saves self.schema but validation never reads it. Separately, Scorer.eval() forwards expected positionally, which collides with the schema parameter in ValidJSON._run_eval_sync(). The same collision affects async partial scorers.

Changes

  • Match the base scorer's expected argument slot, preserving the existing second-positional-argument schema convention as a fallback.
  • Use the constructor schema when no call-time schema is supplied.
  • Keep explicit call-time schemas authoritative, including {}, True, and False; do not mutate the configured schema.
  • Add sync, async, callable, partial, parsed-object and JSON-string regression coverage.

This is Python-specific argument plumbing. The TypeScript implementation already receives its schema in the scorer argument object and is unchanged. No scoring algorithm, dependency, or default validation policy changes.

Validation

  • Initial regression run before the fix: 8 failed, 7 passed (constructor schema ignored and duplicate-argument failures).
  • Final offline run: 26 passed, 1 deselected:
PYTHONPATH=py uv run --no-project --python 3.13 \
  --with polyleven --with chevron --with pyyaml --with jsonschema \
  --with pytest --with pytest-asyncio --with respx -- \
  python -m pytest py/autoevals/test_json.py py/autoevals/test_values.py \
  -k 'not test_list_contains' -q
  • Changed files pass Black 24.10.0, Ruff, and git diff --check.
  • test_list_contains was excluded after an initial run showed missing optional NumPy/SciPy dependencies. No live-model tests or full suite were run.

AI assistance: implemented and tested with OpenAI Codex.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant