AVRO-4241: [Java] BinaryDecoder should verify available bytes before reading by iemejia · Pull Request #3725 · apache/avro

iemejia · 2026-04-06T19:58:58Z

Add ensureAvailableBytes() pre-check in readString, readBytes, readArrayStart, arrayNext, readMapStart, and mapNext to verify the source has sufficient data before proceeding.

Byte-array-backed sources return an exact remaining count. Stream-backed sources return buffered bytes plus InputStream.available(), which is reliable for the finite streams used by DataFileReader and DataFileStream.

Includes regression tests and updated array/map limit tests.

R: @RyanSkraba @martin-g

What is the purpose of the change

(For example: This pull request improves file read performance by buffering data, fixing AVRO-XXXX.)

Verifying this change

(Please pick one of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Extended interop tests to verify consistent valid schema names between SDKs
Added test that validates that Java throws an AvroRuntimeException on invalid binary data
Manually verified the change by building the website and checking the new redirect

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

…reading Add ensureAvailableBytes() pre-check in readString, readBytes, readArrayStart, arrayNext, readMapStart, and mapNext to verify the source has sufficient data before proceeding. Byte-array-backed sources return an exact remaining count. Stream-backed sources return buffered bytes plus InputStream.available(), which is reliable for the finite streams used by DataFileReader and DataFileStream. Includes regression tests and updated array/map limit tests.

RyanSkraba · 2026-04-09T11:18:50Z

To cherry pick to 1.12!

Copilot

Pull request overview

This PR strengthens Java binary decoding against truncated/malicious inputs by adding “bytes remaining” awareness and using it to fail fast (EOF) before allocating large buffers or collection backing structures.

Changes:

Add a Decoder#remainingBytes() API (default -1) and plumb it through BinaryDecoder and ValidatingDecoder.
Add early “ensure available bytes” checks for BinaryDecoder.readString/readBytes and schema-aware prechecks for array/map block counts in GenericDatumReader.
Add/adjust regression tests covering string/bytes length validation and array/map byte-limit behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
lang/java/avro/src/main/java/org/apache/avro/io/Decoder.java	Introduces `remainingBytes()` default API for decoders.
lang/java/avro/src/main/java/org/apache/avro/io/BinaryDecoder.java	Implements `remainingBytes()` and adds early byte-availability checks for string/bytes reads.
lang/java/avro/src/main/java/org/apache/avro/io/ValidatingDecoder.java	Delegates `remainingBytes()` to the underlying decoder.
lang/java/avro/src/main/java/org/apache/avro/generic/GenericDatumReader.java	Adds schema-aware byte validation for arrays/maps before allocating/reading blocks.
lang/java/avro/src/main/java/org/apache/avro/util/ByteBufferInputStream.java	Implements `available()` to support accurate remaining-byte reporting.
lang/java/avro/src/test/java/org/apache/avro/io/TestBinaryDecoder.java	Adds tests ensuring EOF is thrown before large allocations for string/bytes.
lang/java/avro/src/test/java/org/apache/avro/generic/TestGenericDatumReader.java	Adds unit tests for `minBytesPerElement` and end-to-end collection byte validation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+      int buffered = ba.getLim() - ba.getPos();
+      try {
+        if (in.getClass() == ByteArrayInputStream.class || in.getClass() == ByteBufferInputStream.class) {
+          return buffered + in.available();


+    int minBytesPerEntry = 1 + minBytesPerElement(valueSchema);
+    if (count > 0) {
+      int remaining = decoder.remainingBytes();
+      if (remaining >= 0 && count * (long) minBytesPerEntry > remaining) {


steveloughran · 2026-04-28T19:29:12Z

+   */
+  private static void ensureAvailableMapBytes(Decoder decoder, long count, Schema valueSchema) throws EOFException {
+    // Map keys are always strings: at least 1 byte for the length varint
+    int minBytesPerEntry = 1 + minBytesPerElement(valueSchema);


go with copilot and use longs here to avoid problems on very large files

steveloughran · 2026-04-28T19:29:59Z

+  private long arrayNext(ResolvingDecoder in, Schema elementType) throws IOException {
+    long l = in.arrayNext();
+    if (l > 0) {
+      ensureAvailableCollectionBytes(in, l, elementType);


if this a no-op on l <= 0 then you wouldn't need to guard all the uses

Got it fixing it

steveloughran · 2026-04-28T19:35:17Z

  }

+  @Override
+  public int available() throws IOException {


there's this real oddness with available(), in that some interpret it as "all that is left in the stream" but it can also be interpreted as "bytes you can read() before blocking for new data". that's probably the correct one.

it does hold here, just important not to use available() as measures of how much is left in a stream, which may be larger. looks like you are doing the right thing and testing it later on.

steveloughran · 2026-04-28T19:40:21Z

+        return 0; // break recursion for self-referencing schemas
+      }
+      long sum = 0;
+      for (Schema.Field f : schema.getFields()) {


I worry about the cost of this operation on a complex wide and recursive structure, as it'll be invoked once per record.

I added a JMH benchmark test to measure the impact of this for wide and deeply nested structures the results are promising apparently the extra cost is negligible.
https://gist.github.com/iemejia/bae3302ec0f3d2abf92e99911ccba606

that's cool. I'm hitting serious field enum problems on variants in parquet, as shredded variants explode the schema.

…t overflow per PR comments

…n on GenericDatumReader Benchmark to measure the overhead of {@code minBytesPerElement} computation during array/map decoding via {@link GenericDatumReader}. Tests complex, wide, and recursive schema structures to verify that the per-block schema traversal cost is acceptable.

iemejia · 2026-04-30T19:12:24Z

All PR review comments addressed
PTAL again @steveloughran and @RyanSkraba

steveloughran

LGTM

github-actions Bot added the Java Pull Requests for Java binding label Apr 6, 2026

iemejia force-pushed the AVRO-4241-binarydecoder-verify-available-bytes-before-read branch 3 times, most recently from ef9c52a to ce192d2 Compare April 7, 2026 13:44

github-advanced-security AI found potential problems Apr 7, 2026

View reviewed changes

Comment thread lang/java/avro/src/test/java/org/apache/avro/io/TestBinaryDecoder.java Fixed

iemejia force-pushed the AVRO-4241-binarydecoder-verify-available-bytes-before-read branch from ce192d2 to 8471fc2 Compare April 7, 2026 15:35

iemejia requested review from RyanSkraba and martin-g April 8, 2026 07:25

iemejia changed the title ~~AVRO-4241: [Java] BinaryDecoder should verify available bytes before …~~ AVRO-4241: [Java] BinaryDecoder should verify available bytes before reading Apr 8, 2026

iemejia requested a review from Copilot April 10, 2026 19:10

Copilot started reviewing on behalf of iemejia April 10, 2026 19:10 View session

Copilot AI reviewed Apr 10, 2026

View reviewed changes

steveloughran reviewed Apr 28, 2026

View reviewed changes

iemejia added 2 commits April 30, 2026 21:10

AVRO-4241: [Java] Simplify no-op validatioin code and fix possible in…

3b6e1ae

…t overflow per PR comments

steveloughran approved these changes Apr 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AVRO-4241: [Java] BinaryDecoder should verify available bytes before reading#3725

AVRO-4241: [Java] BinaryDecoder should verify available bytes before reading#3725
iemejia wants to merge 3 commits intoapache:mainfrom
iemejia:AVRO-4241-binarydecoder-verify-available-bytes-before-read

iemejia commented Apr 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

RyanSkraba commented Apr 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

steveloughran Apr 28, 2026

Uh oh!

steveloughran Apr 28, 2026

Uh oh!

iemejia Apr 30, 2026

Uh oh!

steveloughran Apr 28, 2026

Uh oh!

steveloughran Apr 28, 2026

Uh oh!

iemejia Apr 30, 2026

Uh oh!

steveloughran Apr 30, 2026

Uh oh!

iemejia commented Apr 30, 2026

Uh oh!

steveloughran left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

iemejia commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the change

Verifying this change

Documentation

Uh oh!

Uh oh!

RyanSkraba commented Apr 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

steveloughran Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

steveloughran Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

iemejia Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

steveloughran Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

steveloughran Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

iemejia Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

steveloughran Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

iemejia commented Apr 30, 2026

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

iemejia commented Apr 6, 2026 •

edited

Loading