Is your feature request related to a problem or challenge?
Currently everything in DataFusion is released together which makes the upgrade path painful and causes dependency conflicts if you have a project which depends on multiple 3rd party dependencies which themselves depend on DataFusion.
For example, if I am building a query engine which should support multiple different table formats and want to use 3rd party dependencies which expose DataFusion TableProviders (e.g https://github.com/apache/iceberg-rust and https://github.com/delta-io/delta-rs), but also implement the core execution engine on top of DataFusion then I need to ensure all dependencies are in sync in terms of their DF dependency.
Describe the solution you'd like
Separate the stable API for internal implementation as separate crates. For instance instead of having just a datafusion-catalog crate with both the stable API and implementations we can have two crate:
datafusion-catalog-apidatafusion-catalog (which depends on datafusion-catalog-api
The -api crate would contain only trait definitions like:
TableProvider
CatalogProvider
etc.
The -api should (theoretically) evolve much more slowly than the implementation so 3rd party crates that want to expose a TableProvider can depend only on datafusion-catalog-api and I can consume that dependency while using any version of datafusion-catalog I want (provided it uses the same stable API).
Obviously there are quite a few details to work out about which APIs to "stabilize" in this way, which I will hand-wave past :)
Describe alternatives you've considered
A lot of alternative suggestions to the exact same issue have been proposed in #16622
Additional context
No response
Is your feature request related to a problem or challenge?
Currently everything in DataFusion is released together which makes the upgrade path painful and causes dependency conflicts if you have a project which depends on multiple 3rd party dependencies which themselves depend on DataFusion.
For example, if I am building a query engine which should support multiple different table formats and want to use 3rd party dependencies which expose DataFusion
TableProviders (e.g https://github.com/apache/iceberg-rust and https://github.com/delta-io/delta-rs), but also implement the core execution engine on top of DataFusion then I need to ensure all dependencies are in sync in terms of their DF dependency.Describe the solution you'd like
Separate the stable API for internal implementation as separate crates. For instance instead of having just a
datafusion-catalogcrate with both the stable API and implementations we can have two crate:datafusion-catalog-apidatafusion-catalog(which depends ondatafusion-catalog-apiThe
-apicrate would contain only trait definitions like:TableProviderCatalogProvideretc.
The
-apishould (theoretically) evolve much more slowly than the implementation so 3rd party crates that want to expose aTableProvidercan depend only ondatafusion-catalog-apiand I can consume that dependency while using any version ofdatafusion-catalogI want (provided it uses the same stable API).Obviously there are quite a few details to work out about which APIs to "stabilize" in this way, which I will hand-wave past :)
Describe alternatives you've considered
A lot of alternative suggestions to the exact same issue have been proposed in #16622
Additional context
No response