feat!: id-based outer-reference resolution (Substrait v0.89.0)

[nielspardon]

Summary

Adds Substrait v0.89.0's id-based outer-reference resolution — RelCommon.rel_anchor + OuterReference.rel_reference — alongside the existing offset-based steps_out, and moves the native DataFrame to emit the id-based form. This completes the DAG / ReferenceRel scope of #188 (the offset-based steps_out is ambiguous once a relation is shared).

Mirrors the split substrait-java landed in #982: the builders stay agnostic (they still emit steps_out, and inference reads both forms), while the higher-level DataFrame layer resolves natively by id.

What changed

Consumer — type_inference.py resolves both forms.
infer_expression_type dispatches an OuterReference on its outer_reference_type oneof: steps_out resolves as before (against the correlated-subquery stack); rel_reference resolves against a plan-wide anchor index (_AnchorScope, mirroring the existing _SubtreeScope), built once in infer_plan_schema by walking every RelCommon.rel_anchor in the plan — subtrees, root, and subquery-embedded relations — and lazily inferring the anchored relation's output schema (memoized, cycle-guarded). This resolves correlated subqueries and, uniformly, an anchor set on a shared subtree reached via ReferenceRel. The index keys on field presence, so a legitimate rel_anchor of 0 is a distinct anchor (not confused with "absent").

Producer — utils.to_id_based_outer_references(plan).
A pure copy-and-rewrite pass that stamps a plan-wide-unique rel_anchor (>= 1, per the spec) on the relation whose output row a correlation resolves against, and rewrites steps_out → rel_reference. A steps_out reference names an enclosing query's row, which is the output of a specific relation, so the relation carrying that row is anchored:

• a single-input host (Filter / Project / …) exposes its input's row. If that input is a ReferenceRel (a cache()-shared subtree, which has no RelCommon), the shared subtree it points at is anchored instead — the reference has no emit, so its output is exactly the subtree's. This is the DAG case steps_out cannot address unambiguously.
• a post_join_filter, or a leaf host's own filter, exposes the host's output row, so the host is anchored.
• a join condition / residual_expression exposes the combined left+right row; the join's own output equals that row for a non-reducing join, so the join is anchored. For a reducing join (semi/anti) the two differ and no relation carries that row — such a reference is left offset-based (steps_out, still spec-valid and read by inference) rather than mis-anchored.

Dedupes by binding identity (references to the same scope share one anchor), reuses an anchor already present, pre-scans existing anchors, and is idempotent. A plan with nothing to rewrite is returned unchanged (no copy), via a cheap _plan_has_steps_out pre-scan. Raises only if a resolvable binding carries no RelCommon at all (e.g. an UpdateRel), which no correlated-subquery shape produces. Backed by descriptor-driven walkers (_iter_rel_expressions, _iter_subquery_rels, and _iter_named_direct_expressions, which tags each of a join's expressions with its owning field so output-scoped post_join_filter is told apart from the combined-inputs-scoped condition/residual) since nothing previously traversed expressions inside relations.

DataFrame — emits id-based only.
to_plan / to_substrait run the converter via a shared _finalize. The public outer(name, steps_out=N) API is unchanged — steps_out stays the ergonomic level selector; only the emitted wire form changes.

Breaking change

DataFrame-produced plans now encode correlated references as rel_reference rather than steps_out (except a reducing join's condition scope, which cannot be expressed id-based and stays steps_out). The builders and inference are backward-compatible (both forms are read; builders still emit steps_out).

Relationship to #228

#228 (lateral-join inference) first built the id-based primitive scoped to LateralJoinRel. This PR is the general mechanism. Once both land, #228 can drop its lateral-join-scoped outer_anchors contextvar and reuse _AnchorScope (with one special case: a lateral-join anchor resolves to its left-input schema). The builders don't conflict — #228 emits rel_reference directly, and this converter is idempotent on already-id-based references. Note #228 may emit rel_anchor values that the builder-side converter here never produces (e.g. 0); the presence-based anchor index reads them correctly.

Testing

• Converter unit tests (tests/test_utils.py): anchor stamping/rewrite, same-scope dedup, distinct-scope anchors, pre-existing-anchor reuse, non-correlated no-op, idempotency; ReferenceRel binding → shared-subtree anchor, multi-input join condition → join anchor, post_join_filter → join anchor, reducing-join condition → kept steps_out, and a no-RelCommon (UpdateRel) raise.
• Inference tests (tests/test_type_inference.py): rel_reference resolving against an anchored shared subtree (the DAG case), a legitimate rel_anchor=0, unknown-anchor error, missing-plan-context error.
• DataFrame tests (tests/dataframe/test_frame.py): correlated-subquery tests assert the id-based form; end-to-end regression tests for a correlated exists over a cache()d outer, in a join post_filter, and in a reducing-join condition (kept steps_out).
• Full suite: 606 passed, ruff check + ruff format --check clean.

Closes #188

🤖 Generated with AI