Skip to content

Keep multipart CSV rows intact when a part lacks a trailing newline - #1800

Open
DMZ22 wants to merge 1 commit into
frictionlessdata:mainfrom
DMZ22:multipart-part-trailing-newline
Open

Keep multipart CSV rows intact when a part lacks a trailing newline#1800
DMZ22 wants to merge 1 commit into
frictionlessdata:mainfrom
DMZ22:multipart-part-trailing-newline

Conversation

@DMZ22

@DMZ22 DMZ22 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Closes #1778.

The multipart byte stream concatenates the parts line by line, so a CSV part whose last line has no trailing newline glued that row onto the first data row of the next part — chunk1.csv = id,name\n1,english plus chunk2.csv = id,name\n2,german\n produced the single corrupted row {"id": 1, "name": "english2"} instead of two rows. The same happens for headless multipart CSVs.

The fix appends the missing newline to the last line of a CSV part when another part follows. Two deliberate boundaries:

  • Non-tabular parts are untouched. For any other format the parts are chunks of one file (the loader's existing format != "csv" distinction), and inserting bytes would corrupt the reassembled file. A regression test pins this: b"\x00\x01NOEOL" + b"\x02\x03\n" still reassembles verbatim.
  • The final part is left as-is — with no following part there is nothing to glue, so the stream stays byte-faithful there too.

Three tests added (header CSV, headless CSV, binary verbatim); the two CSV tests fail on main and pass with the fix. The full multipart scheme suite passes (8 passed, 4 skipped — the skips are the pre-existing remote/vcr ones). Also verified with CRLF endings and a three-part resource whose middle part lacks the newline.

While touching the constructor I switched its List[str] annotation to list[str] — ruff's UP006 flags the line once edited; the file's other pre-existing findings are left alone.

The multipart byte stream concatenates the parts' lines, so a CSV part
whose last line has no trailing newline glued that row onto the first
data row of the next part: chunk1 'id,name\n1,english' + chunk2
'id,name\n2,german\n' produced the single row {'id': 1, 'name':
'english2'} instead of two rows.

A newline is now appended to the last line of a CSV part when it is
missing and another part follows. Non-tabular parts are untouched: they
are chunks of one file and must be concatenated verbatim.

Closes frictionlessdata#1778
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multipart CSV parsing corrupts rows when a file part lacks a trailing newline

1 participant