Skip to content

[Feature] Support Lance file format #313

Description

@wangyong9999

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

Apache Paimon Java supports Lance as a data file format through the paimon-lance module. Paimon C++ currently cannot read or write tables configured with file.format=lance, which prevents Java and C++ engines from interoperating on those tables.

Solution

Add optional Lance file format support to Paimon C++, aligned with the Java implementation and Lance 0.39.0:

  • register a Lance file format factory behind a disabled-by-default CMake option;
  • support reading and writing Lance files through the existing file format interfaces;
  • support column projection, precise row selection, row counts, and file row-id reporting;
  • validate the same supported and unsupported logical types as Java;
  • add C++ round-trip tests and Java/C++ compatibility fixtures in both directions.

The implementation should follow the existing Mosaic Rust FFI integration and lifecycle patterns.

Anything else?

Java reference module: https://github.com/apache/paimon/tree/master/paimon-lance

Are you willing to submit a PR?

  • I am willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions