GH-50967: [C++] Allow CSV reader to ignore extra columns in rows with more columns by HuaHuaY · Pull Request #51118 · apache/arrow

HuaHuaY · 2026-09-01T06:35:03Z

Rationale for this change

Currently, the C++ CSV reader rejects rows with more columns than expected. We can allow users to ignore the extra values instead of throwing exception.

What changes are included in this PR?

Add an option ignore_extra_columns in CSV struct ParseOptions that ignores extra columns.

Are these changes tested?

Yes.

Are there any user-facing changes?

Add an option ignore_extra_columns in CSV struct ParseOptions.

GitHub Issue: [C++] Allow CSV reader to ignore extra columns in rows with more columns #50967

Copilot

Pull request overview

Adds a new C++ CSV parsing option to tolerate rows with more fields than expected by ignoring any extra columns, instead of treating such rows as invalid. This extends the existing “pad short rows” behavior on the other side of the mismatch spectrum and makes the CSV reader more flexible for imperfect inputs.

Changes:

Introduces ParseOptions::ignore_extra_columns (default false) to ignore surplus fields when the expected column count is known.
Updates the CSV block parser to avoid materializing/parsing data for ignored extra fields while still scanning correctly for delimiters/newlines.
Adds parser- and reader-level tests covering extra columns (including trailing delimiters) and updates dataset CsvFileFormat::Equals to account for the new option.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
cpp/src/arrow/dataset/file_csv.cc	Includes `ignore_extra_columns` in `CsvFileFormat::Equals` so format equality reflects the new parse behavior.
cpp/src/arrow/csv/options.h	Adds the `ignore_extra_columns` parse option to the public `ParseOptions` API.
cpp/src/arrow/csv/parser.cc	Implements ignoring of extra fields in the core parsing state machine (including bulk filter path).
cpp/src/arrow/csv/parser_test.cc	Adds a unit test validating that extra columns are ignored at the block parser level.
cpp/src/arrow/csv/reader_test.cc	Adds an integration test validating `TableReader` behavior with extra columns.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

wgtmac

Looks reasonable to me. It would be good if @pitrou could take a look.

pitrou · 2026-09-03T16:04:37Z

@ursabot please benchmark lang=C++

rok · 2026-09-03T16:04:44Z

Benchmark runs are scheduled for commit 3e198d8. Watch https://buildkite.com/apache-arrow and https://conbench.arrow-dev.org for updates. A comment will be posted here when the runs are complete.

HuaHuaY · 2026-09-04T08:05:21Z

@ursabot please benchmark lang=C++

rok · 2026-09-04T08:05:27Z

Benchmark runs are scheduled for commit 0f0c8c3. Watch https://buildkite.com/apache-arrow and https://conbench.arrow-dev.org for updates. A comment will be posted here when the runs are complete.

Copilot

🟢 Approval recommended

The change is opt-in, localized to CSV parsing, and is covered by new parser- and reader-level tests for the intended behavior.

Review details

Files reviewed: 5/5 changed files
Comments generated: 0 new
Review effort level: Lite

HuaHuaY · 2026-09-08T05:44:45Z

+template <typename... CompiledBools, typename Fn, typename... Rest>
+  requires std::invocable<Fn, CompiledBools...,
+                          std::conditional_t<true, std::true_type, Rest>...>
+decltype(auto) DispatchBool(Fn&& fn, Rest... rest) {


I place this utility function here because it can also be used later when handling SpecializedOptions in chunker.cc.

HuaHuaY · 2026-09-08T05:45:08Z

@ursabot please benchmark lang=C++

rok · 2026-09-08T05:45:13Z

Benchmark runs are scheduled for commit a02ebcf. Watch https://buildkite.com/apache-arrow and https://conbench.arrow-dev.org for updates. A comment will be posted here when the runs are complete.

Copilot

🟡 Changes recommended

There are confirmed build/correctness issues (missing required header includes for DispatchBool, and incorrect handling of implicit trailing empty field metadata at EOF) that should be fixed before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 6/6 changed files
Comments generated: 3
Review effort level: Lite

HuaHuaY · 2026-09-08T05:58:15Z

  AbortLine:
    // Not a full line except perhaps if in final block
    if (is_final) {
+      if constexpr (IgnoreExtraColumns) {
+        // Handle an implicit trailing empty field after a delimiter.
+        ignoring_extra_field = IsExtraField();
+      }
      goto LineEnd;
    }


This appears to be a legacy issue and was not introduced by the current PR.

Copilot

🔵 Needs a closer look

It changes core CSV parsing state-machine logic and warrants careful human validation beyond the automated review.

Review details

Files reviewed: 6/6 changed files
Comments generated: 1
Review effort level: Lite

HuaHuaY · 2026-09-08T06:11:32Z

+template <typename... CompiledBools, typename Fn, typename... Rest>
+  requires std::invocable<Fn, CompiledBools...,
+                          std::conditional_t<true, std::true_type, Rest>...>
+decltype(auto) DispatchBool(Fn&& fn, Rest... rest) {


I feel this is a bit too strict; it actually makes the code complicated.

HuaHuaY · 2026-09-08T07:15:49Z

@ursabot please benchmark lang=C++

rok · 2026-09-08T07:15:55Z

Benchmark runs are scheduled for commit a80a3aa. Watch https://buildkite.com/apache-arrow and https://conbench.arrow-dev.org for updates. A comment will be posted here when the runs are complete.

Copilot

🟢 Approval recommended

The change is localized, maintains default behavior, and includes targeted unit tests at both parser and reader levels.

Review details

Files reviewed: 6/6 changed files
Comments generated: 0 new
Review effort level: Lite

HuaHuaY · 2026-09-08T11:49:06Z

@ursabot please benchmark lang=C++

rok · 2026-09-08T11:49:11Z

Benchmark runs are scheduled for commit 2ac3639. Watch https://buildkite.com/apache-arrow and https://conbench.arrow-dev.org for updates. A comment will be posted here when the runs are complete.

HuaHuaY · 2026-09-09T02:39:36Z

A comment will be posted here when the runs are complete.

Is this comment no longer being sent?

HuaHuaY · 2026-09-09T03:27:26Z

I think the benchmark performance regression caused by this PR has basically disappeared. @pitrou Please take a look again.

pitrou · 2026-09-09T14:17:36Z


+template <bool... CompiledBools, typename Fn, typename... Rest>
+decltype(auto) DispatchBool(Fn&& fn, Rest... rest)
+  requires requires {


Double "requires"?

Yes. The first requires is a syntactic construct. The second requires is a requires expression. We need the second requires when checking whether a statement compiles, rather than when passing a concept.

pitrou · 2026-09-09T14:21:36Z

Is this comment no longer being sent?

Indeed, it's not, because we disabled some of the benchmarking machines due to resource consumption issues (cc @rok ).

Or at least I think that's the reason.

rok · 2026-09-09T14:29:14Z

Is this comment no longer being sent?

Indeed, it's not, because we disabled some of the benchmarking machines due to resource consumption issues (cc @rok ).

Or at least I think that's the reason.

Machines are still running, but maybe during optimization (I was changing node sizes etc) some callbacks were not executed. Please benchmark again if you need to and ping me if conbench doesn't post back.

pitrou · 2026-09-09T15:04:39Z

When walking through the benchmark results directly from https://conbench.arrow-dev.org/, we see indeed that there's no regression anymore.

HuaHuaY · 2026-09-09T15:26:32Z

I rebase the code and add one more commit, which adds some comments about DispatchBool and a unit test using both ignore_extra_columns and pad_short_rows.

Copilot

🟢 Approval recommended

The new option is consistently wired through parsing and dataset inspection, and is covered by targeted parser and reader tests.

Review details

Files reviewed: 6/6 changed files
Comments generated: 0 new
Review effort level: Lite

pitrou · 2026-09-09T16:11:08Z

@github-actions crossbow submit -g cpp

github-actions · 2026-09-09T16:14:44Z

Revision: 3000cb4

Submitted crossbow builds: ursacomputing/crossbow @ actions-5d85f52df3

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

Copilot AI lite review requested due to automatic review settings September 1, 2026 06:35

HuaHuaY requested a review from pitrou as a code owner September 1, 2026 06:35

HuaHuaY requested a review from wgtmac September 1, 2026 06:35

github-actions Bot added Component: C++ awaiting review Awaiting review labels Sep 1, 2026

Copilot started reviewing on behalf of HuaHuaY September 1, 2026 06:37 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

HuaHuaY force-pushed the strengthen_csv branch from 52277b3 to 3e198d8 Compare September 1, 2026 07:22

Copilot AI review requested due to automatic review settings September 1, 2026 07:22

Copilot started reviewing on behalf of HuaHuaY September 1, 2026 07:24 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

wgtmac reviewed Sep 2, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Sep 2, 2026

HuaHuaY force-pushed the strengthen_csv branch from 3e198d8 to 0f0c8c3 Compare September 4, 2026 08:03

Copilot AI review requested due to automatic review settings September 4, 2026 08:03

Copilot started reviewing on behalf of HuaHuaY September 4, 2026 08:05 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

HuaHuaY force-pushed the strengthen_csv branch from 0f0c8c3 to a02ebcf Compare September 8, 2026 05:42

Copilot AI review requested due to automatic review settings September 8, 2026 05:42

Copilot started reviewing on behalf of HuaHuaY September 8, 2026 05:43 View session

HuaHuaY commented Sep 8, 2026

View reviewed changes

Copilot AI reviewed Sep 8, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 8, 2026 05:49

Copilot AI review requested due to automatic review settings September 8, 2026 05:59

HuaHuaY force-pushed the strengthen_csv branch from 855f742 to a80a3aa Compare September 8, 2026 05:59

Copilot started reviewing on behalf of HuaHuaY September 8, 2026 06:00 View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 8, 2026 07:57

Copilot started reviewing on behalf of HuaHuaY September 8, 2026 07:58 View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

pitrou reviewed Sep 9, 2026

View reviewed changes

HuaHuaY added 4 commits September 9, 2026 23:02

allow CSV reader to ignore extra columns

f747b52

minor performance optimizations

c220f52

minor file changes

3cfdb0e

minor refactor

83c0bb9

address review

3000cb4

Copilot AI review requested due to automatic review settings September 9, 2026 15:23

HuaHuaY force-pushed the strengthen_csv branch from 2ac3639 to 3000cb4 Compare September 9, 2026 15:24

Copilot started reviewing on behalf of HuaHuaY September 9, 2026 15:29 View session

Copilot AI reviewed Sep 9, 2026

View reviewed changes

pitrou approved these changes Sep 9, 2026

View reviewed changes

Uh oh!

Conversation

HuaHuaY commented Sep 1, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou commented Sep 3, 2026

Uh oh!

rok commented Sep 3, 2026

Uh oh!

HuaHuaY commented Sep 4, 2026

Uh oh!

rok commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

HuaHuaY Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuaHuaY commented Sep 8, 2026

Uh oh!

rok commented Sep 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

HuaHuaY Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

HuaHuaY Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

HuaHuaY commented Sep 8, 2026

Uh oh!

rok commented Sep 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

HuaHuaY commented Sep 8, 2026

Uh oh!

rok commented Sep 8, 2026

Uh oh!

HuaHuaY commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuaHuaY commented Sep 9, 2026

Uh oh!

Uh oh!

pitrou Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

HuaHuaY Sep 9, 2026

Choose a reason for hiding this comment

HuaHuaY commented Sep 1, 2026 •

edited by github-actions Bot

Loading

HuaHuaY Sep 8, 2026 •

edited

Loading

HuaHuaY commented Sep 9, 2026 •

edited

Loading

pitrou commented Sep 9, 2026 •

edited

Loading

rok commented Sep 9, 2026 •

edited

Loading