feat: Native execution via Ballista [experimental]#4800

Draft

andygrove wants to merge 42 commits into

apache:mainfrom

andygrove:poc/ffi-execution-plan

Commits on Jul 2, 2026

feat(native): execute native plans across a datafusion-ffi boundary
andygrove
committed
test(native): cover the datafusion-ffi plan export end to end
andygrove
committed
feat(ballista): add datafusion-comet-ballista crate
andygrove
committed
test(ballista): codec round-trip for the Comet FFI leaf
andygrove
committed
test(ballista): end-to-end distributed shuffle over the Comet scan
andygrove
committed
fix(ballista): address review — honor scan projection, derive Default, tidy deps/docs
andygrove
committed
feat(ballista): prove JVM -> native -> in-process Ballista -> JVM Arrow round-trip
andygrove
committed
docs(contributor-guide): add experimental Ballista execution mode page
andygrove
committed
feat(ballista): offload single-stage Comet collect() to in-process Ballista
andygrove
committed
test(ballista): strengthen offload guard + zero-task proofs, note DPP caveat
andygrove
committed
docs(contributor-guide): mark R1-T1/T2 done, note DPP/subquery caveat
andygrove
committed
test(ballista): offload TPC-H Q1 pre-aggregation slice via in-process Ballista
andygrove
committed
docs(contributor-guide): record R1-T3 result (Q1 single-stage subset; full Q1 needs R2)
andygrove
committed
feat(scan): let ScanExec pull from a native RecordBatch stream
andygrove
committed
feat(ballista): add CometFragmentExec fed by DataFusion child streams
andygrove
committed
feat(ballista): distribute a GROUP BY offload across a Ballista hash shuffle
andygrove
committed

Commits on Jul 3, 2026