Keep multipart CSV rows intact when a part lacks a trailing newline - #1800
Open
DMZ22 wants to merge 1 commit into
Open
Keep multipart CSV rows intact when a part lacks a trailing newline#1800DMZ22 wants to merge 1 commit into
DMZ22 wants to merge 1 commit into
Conversation
The multipart byte stream concatenates the parts' lines, so a CSV part
whose last line has no trailing newline glued that row onto the first
data row of the next part: chunk1 'id,name\n1,english' + chunk2
'id,name\n2,german\n' produced the single row {'id': 1, 'name':
'english2'} instead of two rows.
A newline is now appended to the last line of a CSV part when it is
missing and another part follows. Non-tabular parts are untouched: they
are chunks of one file and must be concatenated verbatim.
Closes frictionlessdata#1778
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1778.
The multipart byte stream concatenates the parts line by line, so a CSV part whose last line has no trailing newline glued that row onto the first data row of the next part —
chunk1.csv=id,name\n1,englishpluschunk2.csv=id,name\n2,german\nproduced the single corrupted row{"id": 1, "name": "english2"}instead of two rows. The same happens for headless multipart CSVs.The fix appends the missing newline to the last line of a CSV part when another part follows. Two deliberate boundaries:
format != "csv"distinction), and inserting bytes would corrupt the reassembled file. A regression test pins this:b"\x00\x01NOEOL"+b"\x02\x03\n"still reassembles verbatim.Three tests added (header CSV, headless CSV, binary verbatim); the two CSV tests fail on
mainand pass with the fix. The full multipart scheme suite passes (8 passed, 4 skipped — the skips are the pre-existing remote/vcr ones). Also verified with CRLF endings and a three-part resource whose middle part lacks the newline.While touching the constructor I switched its
List[str]annotation tolist[str]— ruff's UP006 flags the line once edited; the file's other pre-existing findings are left alone.