Score nested object fields as individual extract cells - #35
Conversation
Object-valued fields now get per-child credit the same way arrays of objects already do, including objects on array rows. Schema combinators are flattened so nullable object/array wrappers still expose their properties. Co-authored-by: Cursor <cursoragent@cursor.com>
boyang-zhang1
left a comment
There was a problem hiding this comment.
Thanks @xujustinj
A small blocker:
When an object-array is empty (or null) in the ground truth but the prediction fills it with objects, it's scored as a single opaque cell instead of being expanded per child. That's the all-or-nothing behavior this PR is meant to remove, and it's still here for the empty-GT case.
The cause is is_object_array_subfield: it only recognizes a nested object-array when the GT rows already hold a non-empty list of objects. Empty GT + populated prediction slips through to the opaque path.
Concrete example from the set: comparable_vehicles[].equipment_adjustments is empty in the automotive valuation golds. A prediction that invents two adjustment objects (four child values) gets charged one miss instead of four. On one of those docs precision comes out 0.9971 where the expansion semantics should give ~0.988. The gap is small because those docs are ~350 cells, but it's wrong in exactly the direction this metric is supposed to catch, and it's live on ~10 docs.
The fix is to detect the object-array from the resolved item schema (and/or the predicted value), not just from non-empty GT rows, so an empty GT array still expands per child. Happy to put up a patch + test.
|
None of these change any numbers on the current set. Purely FYI for if we bring in more data or different structures down the line. Roughly in likelihood order:
|
An empty or null nested object-array still has item properties in the schema, so invented predicted rows should be scored per child instead of as one opaque miss. Co-authored-by: Cursor <cursoragent@cursor.com>
Array vs object vs scalar (and object-array columns) come from JSON Schema, including combinators. A list or dict in the extraction no longer changes the eval function. Co-authored-by: Cursor <cursoragent@cursor.com>
Without resolve_refs, schema-only columns are empty for items: {$ref: #/$defs/...} and the array is skipped.
Co-authored-by: Cursor <cursoragent@cursor.com>
Summary
anyOf/oneOf/allOf) after$refinlining, so nullable object and array wrappers still expose their fields.array_recordinlines$refbefore readingitems.properties, same as the unified scorer, so$defsitem schemas still produce columns.Sequence/Mapping) and a couple of bbox ingest/IoU numeric guards. Headline grounded/page metrics are unchanged in shape.Test plan
uv run pytest tests/extract_bench/evaluation/metrics/extract tests/extract_bench/inference/providers/extract/test_schema_utils.py tests/extract_bench/evaluation/evaluators/test_extract_field_metrics.pyuv run ruff check && uv run ruff format --check