feat: re-expose the embedder public API a distributed driver needs#29
Draft
mdashti wants to merge 1 commit into
Draft
feat: re-expose the embedder public API a distributed driver needs#29mdashti wants to merge 1 commit into
mdashti wants to merge 1 commit into
Conversation
This was referenced Jun 30, 2026
mdashti
force-pushed
the
moe/protocol-grpc-shm
branch
from
deb340e to
02ff23b
Compare
mdashti
force-pushed
the
moe/embedder-api
branch
from
1728fb5 to
328e142
Compare
An out-of-crate driver dispatches a query and drives the head stage itself, so it needs prepare_in_process_plan, the metrics store and a public insert, the partition-routing seam on NetworkBoundary, and a no-gRPC decode_task_metrics to file proto metric frames into the plain store.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR re-exposes the public API an out-of-crate driver/embedder needs to run distributed execution, which datafusion-contrib#512 removed. It stacks on the transport-completion PR (#27).
What
DistributedExec::prepare_in_process_plan: routes and dispatches every stage through the registered transport and returns the head stage, for an embedder that owns the runtime and drives it synchronously (no backgroundexecute()task).DistributedExec::metrics_store()+ publicMetricsStore/MetricsStore::insert: so a driver can collect worker metrics out of crate.NetworkBoundary::route_partition/partitions_per_consumer_task+PartitionRoute: the shuffle routing a transport or embedder that places produced partitions reads, instead of re-deriving it.NetworkCoalesceExecoverridesroute_partitionwith an error (its consumers read per-producer-task groups, not slices).decode_task_metrics: the worker-metrics proto to plainTaskMetricsdecode.Why
datafusion-contrib#512 dropped these as not-transport-concerns, with "what's worth making public in a follow up." This is that follow-up: a real embedder (
pg_search) and a real transport's own in-process test need them. If a transport or embedder cannot be written without re-adding core API, the abstraction was not complete; this is the missing surface.Note
prepare_in_process_plandispatches through the internalQueryCoordinator, which spawns background tasks, so it stashes the coordinator and the caller must keep theDistributedExecalive while driving the head stage. Surfacing the end-of-query / metrics-drain lifecycle for an embedder-owned runtime is a follow-up.