[C++][Acero] Preserve input nullability in direct field projections

[alexandrefimov]

Could Acero preserve input nullability when a project only references top-level input fields? Currently, such a projection changes a required input field to nullable in the output schema. A table_source and Table.select preserve the same field's nullability. The projected values are unchanged.

This also affects Substrait RelCommon.emit: even an identity mapping [0, 1] changes a required field to nullable, while a bare read preserves it.

To reproduce

Install pyarrow==25.0.1 and run:

import pyarrow as pa
import pyarrow.acero as ac
import pyarrow.compute as pc

schema = pa.schema([
    pa.field("r", pa.int64(), nullable=False),
    pa.field("n", pa.int64(), nullable=True),
])
table = pa.Table.from_arrays([
    pa.array([1, 2], type=pa.int64()),
    pa.array([None, 3], type=pa.int64()),
], schema=schema)
source = ac.Declaration("table_source", ac.TableSourceNodeOptions(table))
projected = ac.Declaration(
    "project",
    ac.ProjectNodeOptions([pc.field("r"), pc.field("n")], ["r", "n"]),
    inputs=[source],
).to_table(use_threads=False)

for name, value in [
    ("source", source.to_table(use_threads=False)),
    ("select", table.select([0, 1])),
    ("project", projected),
]:
    print(name, [field.nullable for field in value.schema])
print("values unchanged:", projected.to_pylist() == table.to_pylist())

Output:

source [False, True]
select [False, True]
project [True, True]
values unchanged: True

I expected the direct-reference projection to retain [False, True]. It only selects the two input fields; there are no functions or casts, and the required input field contains no nulls.

Schema compatibility impact

Writing the projected table to an IPC stream created with the original schema fails. Run this after the example above:

import pyarrow.ipc as ipc

with ipc.new_stream(pa.BufferOutputStream(), schema) as writer:
    writer.write_table(projected)

This raises ArrowInvalid: Tried to write record batch with different schema. Replacing projected with table or table.select([0, 1]) succeeds. The values and column order are identical; the schema differs only in nullability.

Where it happens

In ProjectNode::Make, output fields are created with the expression name and type, without carrying input nullability. The same construction is present at main commit 397b3d0023030f6c6bc69d214ea7b27a687256f8; the runtime reproduction above is against 25.0.1.

Substrait's ProcessEmit adds a field-reference project, which explains the identity emit result. Related issue #20108 concerns Substrait type serialization and deserialization; the native Acero example above reproduces this behavior without either step.

The same direct-projection behavior was already reported in #35730. That issue was closed by #35860, which added a custom output schema for dataset writes. It did not change ProjectNode. This request follows up on preserving input nullability in direct projections.

The Acero Substrait documentation notes incomplete support for non-nullable types. I have not found an explicit guarantee that ProjectNode preserves input nullability, so I am requesting a schema-preservation improvement here. This request is limited to direct references to top-level input fields; general expression nullability inference is outside its scope.

Four related Substrait cases, including identity and reordered emit with controls, are in the corpus.