feat(extensions): add VariantGet for path extraction from variant arrays - #1206
Conversation
zeroshade
left a comment
There was a problem hiding this comment.
Some edge case tests that are missing which should probably get added:
- Mixed shredded/residual rows at root and nested levels
- Supported
AsTypeconversion matrix - Corrupted object metadata versus genuinely missing keys
- Field-on-scalar versus index-on-scalar semantics
- Index greater than
math.MaxUint32 - A missing path with a nested
AsType - Empty-string object key in the path
- Sliced and zero-length inputs
zeroshade
left a comment
There was a problem hiding this comment.
Found three blocking data-correctness issues: residual-backed rows are lost for non-empty paths, heterogeneous numeric leaves are silently dropped based on row order, and empty-string object keys are encoded as array index zero.
This review was drafted by an AI-assisted tool and confirmed by an Apache Arrow Go maintainer. After you've addressed the points above and pushed an update, an Apache Arrow Go maintainer — a real person — will take the next look at the PR. If you think one of the findings is misapplied, please reply on the PR and a maintainer will weigh in.
More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.
zeroshade
left a comment
There was a problem hiding this comment.
The three unresolved correctness findings remain on the current head: residual-backed rows are lost for non-empty paths, heterogeneous Variant widths are silently nulled based on row order, and empty object keys are indistinguishable from index zero.
I also found one additional blocking case inline: requesting an unsupported nested AsType silently produces an all-null array instead of returning an error.
CI is green, but these extraction paths can return incorrect successful results and need to be addressed before merging.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.
|
Thanks for the detailed review. I fixed all the pending issues. One intentional divergence to flag: field-access-on-a-scalar errors (matching arrow-rs, per your earlier note) rather than Spark's variant_get, which returns null there; similarly Strict errors on in-range truncation. |
zeroshade
left a comment
There was a problem hiding this comment.
The previous review findings are fixed at the current head, including residual-backed rows, heterogeneous scatter, field-path errors, oversized indexes, and unsupported list/struct targets.
Two current-head issues remain: UUID targets are rejected before reaching the existing UUID conversion support, and non-strict conversion drops every row in a natural-type group when only one row fails. Details and fix directions are inline.
Focused tests, race tests, vet, and direct reproducers otherwise pass. No CI checks are currently reported for this head.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.
zeroshade
left a comment
There was a problem hiding this comment.
Looks good! Just waiting for CI to finish for confirmation before merging
Rationale for this change
Reading one field from a variant column today reassembles the whole value per row (
VariantArray.Value) and then navigates to the field in Go. When the variant is shredded, that field is often already a typed column, so the full reassembly is wasted work.What changes are included in this PR?
compute.VariantGet(ctx, *extensions.VariantArray, VariantGetOptions{Path, AsType, Strict}), mirroring the arrow-rs variant_get. It follows the shreddedtyped_valuecolumns as far as the path allows (stepping into struct fields directly, gathering list elements with thetakekernel) and reassembles the residualvalueper-row only for what remains.AsType == nilreturns aVariantArraypointing at the path; set it to get a typed array. Typed output is produced by grouping leaves by their natural Arrow type and casting each group with the existingcomputecast kernels, so per-element values convert to the requested target independent of row order.Strictcontrols cast strictness: the default allows overflow/truncation via the cast kernels,Strictmakes a lossy or impossible cast error (arrow-go has no per-value null-on-failure mode).variant.VariantPath(an opaqueField/Index/Join/Len/StepAtbuilder) andvariant.Value.GetByPath, plus aVariantArray.VariantType()accessor.Follow-ups (out of scope here):
AsTypeoutput: struct/list target types returnarrow.ErrNotImplemented; materializing them would need a recursive per-field builder.Notes on cross-implementation behavior
Path navigation follows arrow-rs rather than Spark's
variant_getin two places: a field access on a non-object errors (Spark returns null), andStricttreats in-range fractional truncation as an error (Spark truncates). Absent paths, out-of-range indices, and object/array-to-primitive casts match Spark (null, or error underStrict).Are these changes tested?
arrow/compute/variant_get_test.gocovers typed and variant output, nested field and list-index paths, residual/mixed-shredded rows, missing field, null rows, dictionary-encoded metadata, mixed-width/type leaves, order independence, the perfect-shredding fast path,Strictvs default cast outcomes, nested-type rejection, field-on-scalar errors, andCheckedAllocatorleak checks.parquet/variant/path_test.gocovers theVariantPathbuilder andGetByPathnavigation, including the empty-string object key.Are there any user-facing changes?
Yes - new exported API:
compute.VariantGet,compute.VariantGetOptions,variant.VariantPath,variant.Value.GetByPath, andextensions.VariantArray.VariantType(). Additive only.