fix: let the shm transport absorb the coordinator's plan_proto encode#30
Draft
mdashti wants to merge 2 commits into
Draft
fix: let the shm transport absorb the coordinator's plan_proto encode#30mdashti wants to merge 2 commits into
plan_proto encode#30mdashti wants to merge 2 commits into
Conversation
The coordinator serializes every stage subplan before handing it to the channel, but the shm transport ships plans over DSM and discards that request. `ShmDiscardedPlanCodec` lets the throwaway encode succeed for nodes DataFusion's proto cannot represent, such as `SortMergeJoinExec`.
The shm fragment path has no task-level stamps, but the decoder requires the field present; emitting an empty set instead of `None` keeps every worker metrics frame decodable so the per-task EXPLAIN rewrite runs.
mdashti
force-pushed
the
moe/shm-on-protocol-grpc
branch
from
992673d to
ea63b5e
Compare
mdashti
force-pushed
the
moe/shm-plan-proto-discard
branch
from
a09c34a to
741f0c1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes the shm embedder work under the channel-protocol coordinator, which serializes every stage subplan into
plan_protobefore dispatch even though the shm channel ships plans over DSM and drains-and-discards that request.Two changes, both under
src/shm/, so#28stays shm-only:src/shm/plan_codec.rsaddsShmDiscardedPlanCodec, aPhysicalExtensionCodecthe embedder registers viawith_distributed_user_codec. It sits behind the built-inDistributedCodecinnew_combined_with_user, so it is consulted only for nodes DataFusion's proto can't represent (custom scans,SortMergeJoinExec), and encodes each as empty. The bytes are never read, so it refuses to decode.collect_task_metricsreports an empty task-levelMetricsSetinstead ofNone, so the unchangeddecode_task_metricsaccepts the shm path's frames and the per-task EXPLAIN rewrite runs.Surfaced by the
pg_searchMPP regress in paradedb #5448: without the first, aSortMergeJoinExecquery errored mid-dispatch (DataFusion's proto can't encode it and no codec absorbed it); without the second, joins showed no workeroutput_rowsinEXPLAIN ANALYZE(every metrics frame failed to decode, so the rewrite timed out).Stacked on
#28. Draft for review.