GH-3751: Preserve field repetition when truncating recursive proto fields - #3752
Draft
puskarpeter wants to merge 2 commits into
Draft
GH-3751: Preserve field repetition when truncating recursive proto fields#3752puskarpeter wants to merge 2 commits into
puskarpeter wants to merge 2 commits into
Conversation
Parquet forbids empty groups, so a message containing a field whose type is an empty proto message converted into a schema that writer construction rejects with "Cannot write a schema with an empty group" (InvalidSchemaException) - a single such field made the whole message type unwritable. Terminate such fields as BINARY holding the serialized message (zero bytes when the field is set), the same mechanism PARQUET-1711 uses for recursion beyond maxRecursion, preserving the field's repetition: LIST-wrapped binary in parquet-specs mode, repeated binary in the old style, optional binary for map values inside key_value. Field presence and cardinality round-trip; only a message that is empty at the root is still rejected. ProtoWriteSupport's truncated-field detection now looks through the LIST/MAP wrapper (getContentType) so BinaryWriter lines up with these schemas. Signed-off-by: Puškár, Peter <peter.puskar@firma.seznam.cz>
…oto fields The maxRecursion truncation (PARQUET-1711) replaced recursive fields with a hardcoded optional binary, while ProtoWriteSupport still wraps repeated/map fields' writers in ArrayWriter/RepeatedWriter/MapWriter. Writing data that nests deeper than maxRecursion through a repeated recursive field crashed with a ClassCastException in parquet-specs mode and corrupted the file in the old style (inconsistent repetition levels: reads fail with ParquetDecodingException or return a wrong tree). A map field exhausting the recursion budget collapsed entirely - keys included - into one binary, and writing data through it crashed the same way. Truncate to proto bytes preserving the field's shape instead, reusing the terminate-as-bytes path introduced for empty message types (apacheGH-2142): LIST-wrapped binary for repeated fields in specs mode, repeated binary in the old style, and the MAP structure kept with the recursive value truncated inside key_value (the map branch now runs before the recursion check). Truncated optional fields are unchanged; proto2 required fields now keep their required repetition. Each truncated cell round-trips as the serialized subtree. Signed-off-by: Puškár, Peter <peter.puskar@firma.seznam.cz>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Draft — stacked on #3750. This fix reuses the terminate-as-proto-bytes machinery introduced
by the empty-message fix (GH-2142), so only the top commit belongs to this PR; the base commit
is #3750's. I will rebase onto master and mark this ready for review once #3750 merges.
Rationale for this change
PARQUET-1711 truncates recursive proto fields at
parquet.proto.maxRecursiondepth by replacingthem with the serialized proto bytes, but hardcodes the replacement column as
optional binary,ignoring the field's actual repetition.
ProtoWriteSupportstill wraps repeated fields' writers inArrayWriter/RepeatedWriterand map fields inMapWriter, which emit record structure theoptional binarycolumn cannot hold. As a result (see #3751):maxRecursionthrough a repeatedrecursive field crashes with
ClassCastException: PrimitiveColumnIO cannot be cast to GroupColumnIO— a data-dependent failure that passes schema creation and shallow rows;repetition levels and corrupts the file — depending on the data, reading it back either
fails with
ParquetDecodingExceptionor silently returns a wrong tree (elements lost orattached to phantom duplicate nodes);
google.protobuf.Structmapsreached through
list_valuebranches), the whole MAP — including its keys — collapses into onebinary in the schema, and writing data through it crashes with the same
ClassCastException.The existing mock-based tests (
ProtoWriteSupportTest.testRepeatedRecursion/testMapRecursion)never validate against a real
MessageColumnIO, which is why the mismatch went unnoticed.What changes are included in this PR?
ProtoSchemaConverter.addMessageFieldkeeps the field's shape when truncating, reusing theterminate-as-proto-bytes path introduced for empty message types (#3750, which this builds on):
addRepeatedPrimitive;builder.primitive(BINARY, getRepetition(descriptor))—repeated binaryin the oldstyle; truncated optional fields stay
optional binary, byte-for-byte identical to before(proto2
requiredfields in a recursion cycle now keep theirrequiredrepetition instead ofbeing forced
optional);(typed key) is always preserved and a recursive value type is truncated to
optional binaryinside
key_valuewhenaddMapFieldrecurses into the value field — same recursion budget,applied at the level where the recursion actually is.
The writer side needs no further changes: the
getContentTypecheck from #3750 already selectsBinaryWriterbehind LIST/MAP wrappers, so the existingArrayWriter/RepeatedWriter/MapWriterwrapping then lines up with the schema.Are these changes tested?
Yes. New
ProtoRecursionTruncationTest(5 tests) writes recursive data deeper thanmaxRecursionthrough the real write path (
ProtoParquetWriter→MessageColumnIO) and reads it back:Trees.WideTree), specs-compliant and old style — previously theClassCastExceptionand the file-corrupting write, respectively; now every element at thetruncation depth round-trips as the serialized subtree;
google.protobuf.Structbehindlist_value) —previously the whole-MAP collapse plus the same
ClassCastExceptionwhen data reached it; nowkeys stay typed and queryable, values round-trip as serialized protos;
Structpath and optional recursion (Trees.BinaryTree) asregression guards for the shapes that already worked.
Expected-schema fixtures were regenerated for the new truncation shape:
WideTree.par,Value.par,Struct.par, the inline schemas inProtoSchemaConverterTest, and thetestDeepRecursionStruct fan-out series (now2n+5— a truncated map keeps its key column). Thefull parquet-protobuf suite passes.
Are there any user-facing changes?
Schemas containing repeated or map recursive fields change shape at the truncation depth
(LIST-of-binary / MAP-with-binary-value instead of a single
optional binary) — but writing morethan one element at that depth previously crashed (specs mode) or corrupted the file (old style),
so no valid existing files carry meaningful multi-element data in the old shape. Truncated
optional fields are unchanged; proto2
requiredfields in a recursion cycle now map torequired binary(previously forcedoptional). Data past the truncation depth now round-trips losslessly:each binary cell is the serialized subtree, recoverable with
X.parseFrom(bytes).Closes #3751