Skip to content

Score nested object fields as individual extract cells - #35

Merged
xujustinj merged 4 commits into
mainfrom
justin/score-nested-object-fields
Sep 1, 2026
Merged

xujustinj merged 4 commits into
mainfrom
justin/score-nested-object-fields

Conversation

@xujustinj

@xujustinj xujustinj commented Aug 31, 2026

Copy link
Copy Markdown

Summary

  • Walk object-valued fields the same way ExtractBench already walks arrays of objects: each child is its own value cell, including objects nested on array rows.
  • Read item/object properties through JSON Schema combinators (anyOf / oneOf / allOf) after $ref inlining, so nullable object and array wrappers still expose their fields.
  • Empty or null gold object-arrays still expand from the item schema, so invented predicted children are per-field precision misses rather than one opaque cell.
  • Array vs object vs scalar (and which nested columns recurse) is decided from the schema only, not from whether gold or pred happens to be a list or dict.
  • array_record inlines $ref before reading items.properties, same as the unified scorer, so $defs item schemas still produce columns.
  • Small typing cleanups on the array-record helpers (Sequence / Mapping) and a couple of bbox ingest/IoU numeric guards. Headline grounded/page metrics are unchanged in shape.

Test plan

  • uv run pytest tests/extract_bench/evaluation/metrics/extract tests/extract_bench/inference/providers/extract/test_schema_utils.py tests/extract_bench/evaluation/evaluators/test_extract_field_metrics.py
  • uv run ruff check && uv run ruff format --check

Object-valued fields now get per-child credit the same way arrays of objects already do, including objects on array rows. Schema combinators are flattened so nullable object/array wrappers still expose their properties.

Co-authored-by: Cursor <cursoragent@cursor.com>

@boyang-zhang1 boyang-zhang1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @xujustinj
A small blocker:
When an object-array is empty (or null) in the ground truth but the prediction fills it with objects, it's scored as a single opaque cell instead of being expanded per child. That's the all-or-nothing behavior this PR is meant to remove, and it's still here for the empty-GT case.

The cause is is_object_array_subfield: it only recognizes a nested object-array when the GT rows already hold a non-empty list of objects. Empty GT + populated prediction slips through to the opaque path.

Concrete example from the set: comparable_vehicles[].equipment_adjustments is empty in the automotive valuation golds. A prediction that invents two adjustment objects (four child values) gets charged one miss instead of four. On one of those docs precision comes out 0.9971 where the expansion semantics should give ~0.988. The gap is small because those docs are ~350 cells, but it's wrong in exactly the direction this metric is supposed to catch, and it's live on ~10 docs.

The fix is to detect the object-array from the resolved item schema (and/or the predicted value), not just from non-empty GT rows, so an empty GT array still expands per child. Happy to put up a patch + test.

@boyang-zhang1

Copy link
Copy Markdown
Member

None of these change any numbers on the current set. Purely FYI for if we bring in more data or different structures down the line. Roughly in likelihood order:

  1. Scalar list on a plain object gets dropped, not scored. _score_node sends lists to _score_array, which bails when there are no object subfields. Fine today since every scalar list sits on an array row (still scored opaquely); a {obj: {tags: [...]}} shape would slip through.

  2. Row matching can go order-dependent when rows carry an object subfield. The cost treats the object as one opaque 0/1 but scores it per child, so ambiguous rows could pair by order. The 13F docs have voting_authority but rows are uniquely keyed by cusip, so it's stable now.

  3. _provenance isn't filtered inside nested objects (only at the root). The provider only puts it on array rows/root today, but the gate tolerates it at any level, so worth matching that in the scorer.

  4. Malformed bbox → perfect IoU: min(1.0, nan) is 1.0, and the bbox check allows non-finite/non-positive sizes. No degenerate boxes in the golds, but a pipeline emitting one would get free grounding.

  5. Evidence on an object path gets dropped (recursion only scores leaves). Nothing cites an object path today; if annotations ever land on one, they'd silently vanish.

  6. oneOf/anyOf branches are unioned into one property set, which would give a real discriminated union phantom-null credit. Every anyOf today is just a nullable wrapper, so it's inert.

  7. Legacy array_record doesn't resolve $ref the way unified does, so with Pydantic $defs the two can disagree on a hallucinated column. Low priority, just nice to keep them in sync.

xujustinj and others added 3 commits August 31, 2026 16:12
An empty or null nested object-array still has item properties in the schema, so invented predicted rows should be scored per child instead of as one opaque miss.

Co-authored-by: Cursor <cursoragent@cursor.com>
Array vs object vs scalar (and object-array columns) come from JSON Schema, including combinators. A list or dict in the extraction no longer changes the eval function.

Co-authored-by: Cursor <cursoragent@cursor.com>
Without resolve_refs, schema-only columns are empty for items: {$ref: #/$defs/...} and the array is skipped.

Co-authored-by: Cursor <cursoragent@cursor.com>
@boyang-zhang1
boyang-zhang1 self-requested a review August 31, 2026 21:33

@boyang-zhang1 boyang-zhang1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@xujustinj
xujustinj merged commit 3bb8999 into main Sep 1, 2026
1 check passed
@xujustinj
xujustinj deleted the justin/score-nested-object-fields branch September 1, 2026 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants