Add feature to link datafusion as dylib#23767
Draft
sergiimk wants to merge 1 commit into
Draft
Conversation
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #23767 +/- ##
==========================================
- Coverage 80.71% 80.71% -0.01%
==========================================
Files 1089 1089
Lines 368750 368748 -2
Branches 368750 368748 -2
==========================================
- Hits 297631 297621 -10
- Misses 53375 53379 +4
- Partials 17744 17748 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes: #6470
Rationale for this change
As we build many tools with
datafusionwe have many dozens of crates in our workspace depending on it, and every test and example ends up linking DF library statically, resulting in:devprofile, linking DF dynamically reduces ourdevbuild footprint from 112GiB to 57GiBWe believe offering an option for dynamic linking can provide a good productivity boost for local development and shorted interations.
The proposed implementation mirrors dynamic linking in Bevy engine which has provided this option for years for similar reasons. The mechanism is opt-in, and imposes zero cost on consumers who don't enable it.
What changes are included in this PR?
This PR proposes a new feature
datafusion/dynamicwhich when enabled builds adylibfor dynamic linking.How it Works
datafusioncrate is renamed todatafusion-core. The package rename is invisible to consumersbecause [lib] name = "datafusion" preserves the import name — all use datafusion::... statements in
downstream code (and in datafusion's own tests, benches, and bins) continue to work unchanged.
datafusion-dylibsub-crate is added withcrate-type = ["dylib"]. It contains a singleuse datafusion_core as _;statement, which is what forces the dynamic library to contain thedatafusion-coresymbol table.datafusionfacade crate is introduced that re-exports everything fromdatafusion-coreand forwards every existing feature 1:1.datafusion-dylibvia#[cfg(feature = "dynamic")] use datafusion_dylib as _;Dependency graph:
When
dynamicfeature is off (default),datafusion-dylibis not compiled at all — no cost.Downsides & Limitations
[lib] name = "datafusion"on core crate which can cause confusion with the facade crate. This is only to keepuse datafusion::*inside thedatafusion-corecrate working unchanged. Full PR should probably change those statements touse crate::*.Alternatives
dylib- impacts how people distribute their client apps, can impact performancecrate-type = ["rlib", "dylib"]with-C prefer-dynamiccompile option - currently this will always build both types of libraries, adding overhead for users that don't need dynamic libAre these changes tested?
We have tested it in our builds against a fork, but more tweaks may be necessary for different target platforms (e.g. wasm).
Are there any user-facing changes?
No. The new facade library should be fully substitutable for the original one.