Skip to content

OuterReference.steps_out uses a 0-based convention, but Substrait requires steps_out >= 1 #231

Description

@nielspardon

Summary

substrait-python builds and resolves OuterReference.steps_out with a 0-based convention (0 = the immediately enclosing query), but Substrait specifies steps_out must be >= 1 (1-based). See the v0.89.0 release note ("... this is not a real breaking change as Substrait always required non-zero value", #188) and proto/substrait/algebra.proto (steps_out ... Must be >= 1).

Details

  • Builder default is 0: outer_reference(field, steps_out=0) (src/substrait/builders/extended_expression.py) and sub.outer(name, steps_out=0) (src/substrait/dataframe/expr.py), documented as "0 = the immediately enclosing query".
  • Inference treats 0 as immediate: infer_expression_type resolves stack[len(stack) - 1 - steps], so steps_out=0 selects the immediately enclosing schema (src/substrait/type_inference.py, the steps_out branch).
  • Tests encode the 0-based convention: tests/dataframe/test_frame.py::test_correlated_exists asserts steps_out == 0 for an immediate correlation; test_nested_correlation_steps_out uses steps_out=1 for "two levels out".

Impact

Plans emitted by the builder / DataFrame APIs carry steps_out=0, which the spec disallows, and are off-by-one for deeper nesting levels. Correlated subqueries may therefore not round-trip correctly with spec-conformant producers/consumers.

Suggested fix

Make the public API 1-based: default steps_out=1, emit and resolve accordingly, and update the outer_schemas indexing and the associated tests. This is a breaking change to the correlated-subquery API, so it warrants its own PR.

Context

Noticed while implementing lateral-join id-based references in #228. The lateral path uses OuterReference.rel_reference (id-based), not steps_out, so it is unaffected by this bug.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions