Skip to content

Carry a value of any size through the Flink serializer - #66

Merged
estebanzimanyi merged 1 commit into
MobilityDB:mainfrom
estebanzimanyi:test/value-of-any-size
Sep 28, 2026
Merged

estebanzimanyi merged 1 commit into
MobilityDB:mainfrom
estebanzimanyi:test/value-of-any-size

Conversation

@estebanzimanyi

Copy link
Copy Markdown
Member

A MEOS value crosses Flink as the WKB bytes of its type where the catalog states that byte
codec, and the serializer writes the length of the form and its bytes. GeneratedSqlSurfaceTest
builds its hex literals from a value's toString, the hex of its WKB, and
aValueOfAnySizeCrossesTheSerializer serializes a 20,000-instant tgeompoint and reads it back
to the same value with its 20,000 instants. GENERATION.md states the codec order: WKB bytes,
then hex WKB, then text.

Witness. The new test fails on the generator of JMEOS main with UTFDataFormatException:
Encoded string is too long: 1160018, the hex form of that trajectory.

Measured. Against JMEOS #111, the catalog and libmeos of MobilityDB 438528db44 with MEOS-API
20efba81be and the jar JMEOS builds from them, the binding runs 17 tests and the benchmark 12,
all passing.

Why. A trajectory of an ordinary day passes the 65,535 bytes a string write carries, and a
value that cannot be serialized cannot be shuffled or checkpointed.

A MEOS value crosses Flink as the WKB bytes of its type where the catalog states that byte
codec, and the serializer writes the length of the form and its bytes. GeneratedSqlSurfaceTest
builds its hex literals from a value's toString, the hex of its WKB, and
aValueOfAnySizeCrossesTheSerializer serializes a 20,000-instant tgeompoint and reads it back
to the same value with its 20,000 instants. GENERATION.md states the codec order: WKB bytes,
then hex WKB, then text.

Witness. The new test fails on the generator of JMEOS main with UTFDataFormatException:
Encoded string is too long: 1160018, the hex form of that trajectory.

Measured. Against JMEOS #111, the catalog and libmeos of MobilityDB 438528db44 with MEOS-API
20efba81be and the jar JMEOS builds from them, the binding runs 17 tests and the benchmark 12,
all passing.

Why. A trajectory of an ordinary day passes the 65,535 bytes a string write carries, and a
value that cannot be serialized cannot be shuffled or checkpointed.
@estebanzimanyi
estebanzimanyi merged commit e15afc0 into MobilityDB:main Sep 28, 2026
2 checks passed
@estebanzimanyi
estebanzimanyi deleted the test/value-of-any-size branch September 28, 2026 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant