Skip to content

Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion #186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alamb merged 79 commits into from
Jul 20, 2026
Merged

Add blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion #186

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
94c67b9
Add blog: Sort Pushdown in DataFusion: Skip Sorts, Skip I/O
zhuqi-lucas May 14, 2026
f137266
Push draft date to 2026-05-25
zhuqi-lucas May 14, 2026
f897e38
Add Phase 3 section covering #21956 runtime reorder pipeline
zhuqi-lucas May 15, 2026
247fc55
Add empirical note: why we keep an out-of-tree RG-level Exact reverse
zhuqi-lucas May 16, 2026
8d3ebed
Correct internal RG-Exact description and trim arrow-rs #9937 duplica…
zhuqi-lucas May 16, 2026
0fc60d4
Mark #21956 as landed (merged via merge queue)
zhuqi-lucas May 17, 2026
e6ab133
Reframe blog from Phase N → capability-based sections
zhuqi-lucas May 17, 2026
75553f6
Clarify two-layer composability: same key (min), not 'nest by constru…
zhuqi-lucas May 17, 2026
b1d03c1
Strip inline PR refs from narrative; collect all links in References
zhuqi-lucas May 17, 2026
5fd2f61
Restructure: split into merged-features / bottlenecks / roadmap
zhuqi-lucas May 17, 2026
1862d8f
Tighten verbose paragraphs; drop unrelated limit-pruning blog reference
zhuqi-lucas May 17, 2026
1da0e07
Fold reverse as case of Inexact runtime reorder; make two-trigger asy…
zhuqi-lucas May 17, 2026
4391fb3
Drop EnsureRequirements + OFFSET pushdown — both unrelated to sort pu…
zhuqi-lucas May 17, 2026
5d5e509
Roadmap RG-level early termination: split into non-overlap vs overlap…
zhuqi-lucas May 17, 2026
e7baa8a
Rewrite blog around #22450: three-layer pruning + topk_tpch headline
zhuqi-lucas Jun 28, 2026
7ace058
Update content/blog/2026-07-05-sort-pushdown.md
zhuqi-lucas Jul 8, 2026
d6fac61
Update content/blog/2026-07-05-sort-pushdown.md
zhuqi-lucas Jul 8, 2026
2822f47
Update content/blog/2026-07-05-sort-pushdown.md
zhuqi-lucas Jul 8, 2026
1b75d5a
Update content/blog/2026-07-05-sort-pushdown.md
zhuqi-lucas Jul 8, 2026
89b422d
Update content/blog/2026-07-05-sort-pushdown.md
zhuqi-lucas Jul 8, 2026
06f5bba
Update content/blog/2026-07-05-sort-pushdown.md
zhuqi-lucas Jul 8, 2026
efec00e
Update content/blog/2026-07-05-sort-pushdown.md
zhuqi-lucas Jul 8, 2026
4daaa2d
Address alamb review round 2 (46 comments)
zhuqi-lucas Jul 8, 2026
9b48aa8
Remove transition_anatomy and pruner_loop figures
zhuqi-lucas Jul 8, 2026
126ad20
Revert "Remove transition_anatomy and pruner_loop figures"
zhuqi-lucas Jul 8, 2026
4e8a2d6
Regenerate transition_anatomy and pruner_loop figures
zhuqi-lucas Jul 8, 2026
9037e8a
Move Three-Layer Pruning section earlier
zhuqi-lucas Jul 8, 2026
ead1ddc
Add @alamb as second author
zhuqi-lucas Jul 8, 2026
25e259b
Wordsmith into, add blog post
alamb Jul 8, 2026
f37e1f0
wordsmith
alamb Jul 8, 2026
6200078
wordsmith
alamb Jul 8, 2026
3ba83a5
wordsmith
alamb Jul 8, 2026
2f9db08
wordsmith
alamb Jul 8, 2026
817dbe7
wordsmith
alamb Jul 8, 2026
73403bb
wordsmith
alamb Jul 8, 2026
338a610
wordsmith
alamb Jul 8, 2026
92a3605
nits
alamb Jul 8, 2026
4fe7b46
make headings more consistent
alamb Jul 8, 2026
06caf86
Drop PR-number-as-concept + stale (Future B) label
zhuqi-lucas Jul 9, 2026
22de1bc
Update content/blog/2026-07-05-sort-pushdown.md
zhuqi-lucas Jul 10, 2026
6e1d134
Update figure 2, also wordsmit intro
alamb Jul 10, 2026
75409dc
refine
alamb Jul 10, 2026
44fc3a4
shorten
alamb Jul 10, 2026
ca3373f
upate
alamb Jul 10, 2026
435cd97
tweak
alamb Jul 10, 2026
da5794b
move buffering without sorting to an appendix"
alamb Jul 10, 2026
a59c71e
rework
alamb Jul 10, 2026
e210040
more
alamb Jul 16, 2026
354f0d8
reorder
alamb Jul 16, 2026
ebef378
improve narrative
alamb Jul 16, 2026
8c43f4e
exits
alamb Jul 16, 2026
65db179
convert
alamb Jul 16, 2026
5142a5f
convert + tighten
alamb Jul 16, 2026
e806e69
update
alamb Jul 16, 2026
0df396b
rework
alamb Jul 16, 2026
754da45
hone
alamb Jul 16, 2026
f9a0b57
remove redundancy
alamb Jul 16, 2026
c92908c
remove redundancy
alamb Jul 16, 2026
d9169f1
typos
alamb Jul 16, 2026
3bd4135
updated with related work
alamb Jul 17, 2026
07575fe
cit
alamb Jul 17, 2026
f913317
move
alamb Jul 17, 2026
7a2cfbf
cleanup
alamb Jul 17, 2026
1279880
more
alamb Jul 17, 2026
ac27fd8
update
alamb Jul 17, 2026
1666952
small tweaks to alamb edits: #22450 attribution + DF version + alt-text
zhuqi-lucas Jul 17, 2026
a83fdea
add link to raw sort_tpch benchmark numbers (addresses alamb review)
zhuqi-lucas Jul 17, 2026
994da14
move merged arrow-rs#10158 to Major PRs; clarify #23067 uses it
zhuqi-lucas Jul 17, 2026
71133a6
chore: update publish date to 2026-07-20
zhuqi-lucas Jul 17, 2026
6ff7cd7
update date, fix bold
alamb Jul 17, 2026
a0f953d
wordsmith
alamb Jul 17, 2026
c7c860c
wordsmith
alamb Jul 17, 2026
a1a0985
wordsmith
alamb Jul 17, 2026
0bc3cff
cut
alamb Jul 17, 2026
298ac80
wordsmit
alamb Jul 17, 2026
258cf5b
wordsmit
alamb Jul 17, 2026
01341c1
remove layer1/2/3 terminology
alamb Jul 17, 2026
b8966c1
more
alamb Jul 17, 2026
3af340a
fix links
alamb Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
528 changes: 528 additions & 0 deletions content/blog/2026-07-20-sort-pushdown.md
View file
Open in desktop

Large diffs are not rendered by default.

Binary file added content/images/sort-pushdown/arch_one_glance.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
78 changes: 78 additions & 0 deletions content/images/sort-pushdown/benchmark.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
160 changes: 160 additions & 0 deletions content/images/sort-pushdown/buffer-exec-stall.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
89 changes: 89 additions & 0 deletions content/images/sort-pushdown/buffer-exec.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading