[C++][Substrait] Modern extension URNs fall back to function-name matching

[alexandrefimov]

Acero accepts a function from an unregistered modern Substrait extension URN and executes an Arrow function with the same name. A populated extension identifier reaches the legacy name-only fallback instead of identifying the requested extension or producing an unsupported error.

To reproduce

Install pyarrow==25.0.1 and substrait==0.31.0 — not substrait-protobuf, which makes import pyarrow.substrait fail on a missing substrait.proto — then run:

Reproducer — one file, pyarrow and the Substrait bindings

import pyarrow as pa
import pyarrow.substrait as ps
from substrait import plan_pb2, type_pb2

plan = plan_pb2.Plan()
plan.version.minor_number = 102
plan.extension_urns.add(
    extension_urn_anchor=1,
    urn="extension:example.com:unregistered_arithmetic",
)
extension = plan.extensions.add().extension_function
extension.extension_urn_reference = 1
extension.function_anchor = 1
extension.name = "add:i64_i64"

root = plan.relations.add().root
root.names.append("r")
project = root.input.project
project.common.emit.output_mapping.append(2)
read = project.input.read
read.named_table.names.append("t")
read.base_schema.names.extend(["a", "b"])
read.base_schema.struct.nullability = type_pb2.Type.NULLABILITY_REQUIRED
for _ in range(2):
    read.base_schema.struct.types.add().i64.nullability = type_pb2.Type.NULLABILITY_NULLABLE

function = project.expressions.add().scalar_function
function.function_reference = 1
function.output_type.i64.nullability = type_pb2.Type.NULLABILITY_NULLABLE
for index in range(2):
    selection = function.arguments.add().value.selection
    selection.direct_reference.struct_field.field = index
    selection.root_reference.SetInParent()

def provider(names, schema):
    assert names == ["t"]
    return pa.table({"a": [2], "b": [3]}, schema=schema)

result = ps.run_query(
    plan.SerializeToString(), table_provider=provider, use_threads=False
).read_all()
print(result.to_pydict())

It prints {'r': [5]} on PyArrow 25.0.1.

I expected rejection because no implementation was registered for this extension. The function name alone does not establish which extension it belongs to. This example serializes the URN fields with modern protobuf bindings; it does not use Arrow's older JSON-to-protobuf helper, which can drop those fields during encoding.

Decimal consequence and controls

With the standard extension:io.substrait:functions_arithmetic_decimal URN, divide:dec_dec over nullable decimal(10,2) and decimal(5,1) returns decimal(16,7). The v0.102.0 extension formula specifies decimal(21,8): scale is max(6, 2 + 5 + 1) = 8, and precision is 10 - 2 + 5 + 8 = 21.

This changes an exactly representable value: 1.00 / 256.0 returns 0.0039062 instead of 0.00390625. Native Arrow division produces the same result. The legacy URI form naming functions_arithmetic_decimal.yaml is rejected because no conversion is registered for that extension. An unregistered legacy URI is also rejected, while integer addition with the registered legacy arithmetic URI succeeds.

Unsupported extensions can be rejected. The problem is that the modern identifier is lost and a different function is selected by name.

Relevant code

GetExtensionSetFromMessage reads extension_uris and extension_uri_reference. Modern plans use extension_urns and extension_urn_reference, with different field numbers. The legacy reference defaults to zero, and the map lookup creates an empty URI. Scalar function conversion then uses the name-only fallback.

The fallback added in #14143 was intended for empty or / identifiers. This case supplies a populated modern identifier. Related PR #50635 updates Fetch, Aggregate and Join handling with a proto bump to v0.63.0; its current changes do not include URN resolution.

The reproduction is against 25.0.1; cpp/src/arrow/engine/substrait is unchanged on main since f14ae5b.

Five focused cases with controls include the modern URNs, legacy URI counterparts and exact decimal values.

[KHARSHAVARDHAN-eng]

Hi @alexandrefimov, I’d like to investigate and work on this issue. The modern extension URN resolution and name-only fallback behavior look particularly interesting, and I’d like to explore the root cause and add appropriate regression coverage. I’ll first verify the behavior on the current main branch before proposing a fix.