Skip to content

feat(extensions): add VariantGet for path extraction from variant arrays - #1206

Merged
zeroshade merged 5 commits into
apache:mainfrom
nssalian:add-variant-get
Sep 1, 2026
Merged

feat(extensions): add VariantGet for path extraction from variant arrays#1206
zeroshade merged 5 commits into
apache:mainfrom
nssalian:add-variant-get

Conversation

@nssalian

@nssalian nssalian commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Reading one field from a variant column today reassembles the whole value per row (VariantArray.Value) and then navigates to the field in Go. When the variant is shredded, that field is often already a typed column, so the full reassembly is wasted work.

What changes are included in this PR?

  • Adds compute.VariantGet(ctx, *extensions.VariantArray, VariantGetOptions{Path, AsType, Strict}), mirroring the arrow-rs variant_get. It follows the shredded typed_value columns as far as the path allows (stepping into struct fields directly, gathering list elements with the take kernel) and reassembles the residual value per-row only for what remains.
  • AsType == nil returns a VariantArray pointing at the path; set it to get a typed array. Typed output is produced by grouping leaves by their natural Arrow type and casting each group with the existing compute cast kernels, so per-element values convert to the requested target independent of row order.
  • Strict controls cast strictness: the default allows overflow/truncation via the cast kernels, Strict makes a lossy or impossible cast error (arrow-go has no per-value null-on-failure mode).
  • Adds variant.VariantPath (an opaque Field/Index/Join/Len/StepAt builder) and variant.Value.GetByPath, plus a VariantArray.VariantType() accessor.

Follow-ups (out of scope here):

  • Nested AsType output: struct/list target types return arrow.ErrNotImplemented; materializing them would need a recursive per-field builder.

Notes on cross-implementation behavior

Path navigation follows arrow-rs rather than Spark's variant_get in two places: a field access on a non-object errors (Spark returns null), and Strict treats in-range fractional truncation as an error (Spark truncates). Absent paths, out-of-range indices, and object/array-to-primitive casts match Spark (null, or error under Strict).

Are these changes tested?

  • arrow/compute/variant_get_test.go covers typed and variant output, nested field and list-index paths, residual/mixed-shredded rows, missing field, null rows, dictionary-encoded metadata, mixed-width/type leaves, order independence, the perfect-shredding fast path, Strict vs default cast outcomes, nested-type rejection, field-on-scalar errors, and CheckedAllocator leak checks.
  • parquet/variant/path_test.go covers the VariantPath builder and GetByPath navigation, including the empty-string object key.

Are there any user-facing changes?

Yes - new exported API: compute.VariantGet, compute.VariantGetOptions, variant.VariantPath, variant.Value.GetByPath, and extensions.VariantArray.VariantType(). Additive only.

@nssalian
nssalian marked this pull request as ready for review August 16, 2026 01:37
@nssalian
nssalian requested a review from zeroshade as a code owner August 16, 2026 01:37
@nssalian
nssalian marked this pull request as draft August 16, 2026 02:07
@nssalian
nssalian marked this pull request as ready for review August 16, 2026 03:06
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some edge case tests that are missing which should probably get added:

  • Mixed shredded/residual rows at root and nested levels
  • Supported AsType conversion matrix
  • Corrupted object metadata versus genuinely missing keys
  • Field-on-scalar versus index-on-scalar semantics
  • Index greater than math.MaxUint32
  • A missing path with a nested AsType
  • Empty-string object key in the path
  • Sliced and zero-length inputs

Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated
Comment thread arrow/extensions/variant_get.go Outdated

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found three blocking data-correctness issues: residual-backed rows are lost for non-empty paths, heterogeneous numeric leaves are silently dropped based on row order, and empty-string object keys are encoded as array index zero.


This review was drafted by an AI-assisted tool and confirmed by an Apache Arrow Go maintainer. After you've addressed the points above and pushed an update, an Apache Arrow Go maintainer — a real person — will take the next look at the PR. If you think one of the findings is misapplied, please reply on the PR and a maintainer will weigh in.

More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.

Comment thread arrow/compute/variant_get.go
Comment thread arrow/compute/variant_get.go Outdated
Comment thread parquet/variant/path.go Outdated

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three unresolved correctness findings remain on the current head: residual-backed rows are lost for non-empty paths, heterogeneous Variant widths are silently nulled based on row order, and empty object keys are indistinguishable from index zero.

I also found one additional blocking case inline: requesting an unsupported nested AsType silently produces an all-null array instead of returning an error.

CI is green, but these extraction paths can return incorrect successful results and need to be addressed before merging.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.

Comment thread arrow/compute/variant_get.go Outdated
@nssalian

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review. I fixed all the pending issues. One intentional divergence to flag: field-access-on-a-scalar errors (matching arrow-rs, per your earlier note) rather than Spark's variant_get, which returns null there; similarly Strict errors on in-range truncation.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous review findings are fixed at the current head, including residual-backed rows, heterogeneous scatter, field-path errors, oversized indexes, and unsupported list/struct targets.

Two current-head issues remain: UUID targets are rejected before reaching the existing UUID conversion support, and non-strict conversion drops every row in a natural-type group when only one row fails. Details and fix directions are inline.

Focused tests, race tests, vet, and direct reproducers otherwise pass. No CI checks are currently reported for this head.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.

Comment thread arrow/compute/variant_get.go Outdated
Comment thread arrow/compute/variant_get.go
@nssalian
nssalian requested a review from zeroshade August 31, 2026 22:02

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just waiting for CI to finish for confirmation before merging

@zeroshade
zeroshade merged commit 2e264f0 into apache:main Sep 1, 2026
43 of 45 checks passed
@nssalian
nssalian deleted the add-variant-get branch September 1, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants