(parquet-avro) Automatically detect list encodings in AvroReadSupport

[clairemcginty]

Rationale for this change

parquet-avro supports writing both "old" and "new" list encodings via the parquet.avro.write-old-list-structure config. "old" encodings (aka "2-level"), which wrap the list in a repeated group array schema, are the default; "new" encodings (aka "3-level") are opt-in.

On the reader side, if you're using ParquetAvroReader to read data that was written using ParquetAvroWriter, and don't specify a projection, both type sof list encoding get parsed automatically from a combination of the file schema + the parquet.avro.schema metadata key. There's no need to set parquet.avro.write-old-list-structure key in your Configuration.

However, if you're either:

• specifying a projection (AvroReadSupport.setRequestedProjection(...)), or
• reading data not written using ParquetAvroWriter (and thus not containing the parquet.avro.schema metadata key),

3-levle list encodings will not be parsed correctly - the reader will inject an extra nested record, named element, into the list item type.

As a reader this introduces some pain, since you have to look up the underlying file metadata of the upstream Parquet file, and risk reading incorrect data. This PR attempts to automatically detect new list encodings based on the writer file schema.

lmk what you think of this change. Automatic inference is always a bit risky, but I tried to be conservative with the approach (only set the list structure property if all list fields in the schema use 3-level encoding; don't override parquet.avro.write-old-list-structure if the user is already setting it). any ideas for a better approach here are welcome - this is becoming more of a pain point as 3-level lists become a more popular option among other writer sdks.

What changes are included in this PR?

A new read configuration property parquet.avro.read.autoDetectListStructure (defaulting to true) that will instruct AvroReadSupport to automatically set List configuration properties based on parsing the writer file schema.

Are these changes tested?

Yes, unit tests + locally on real data.

Are there any user-facing changes?

Yes, since the new property defaults to true - it would impact anyone who's reading 3-level list data without setting the parquet.avro.write-old-list-structure key and who's relying on/working around the incorrectly formatted data (e.g. {"locations": [{"element": {"latitude": 0.0, "longitude": 180.0}}, ...]} instead of {"locations": [{"latitude": 0.0, "longitude": 180.0}, ...]} .

additionally, this change also modifies the underlying Configuration object to add the properties.