Fix DuckDB scans over Vortex being merged or stalling - #9803
Merged
Conversation
Merging this PR will not alter performance
|
joseph-isaacs
added
changelog/fix
Claude
Codecov Report✅ All modified and coverable lines are covered by tests. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
A Vortex runtime driven from a thread inside `tokio::runtime::Runtime::block_on` loses the wakeups that complete its I/O and parks forever. The DuckDB SLT runner drove each file from inside `rt.block_on`, so a long sequence of scans reliably wedged: tracing showed a read issuing `get_opts` and never being resumed, with every thread parked and the blocking worker already idle. The trigger is narrow and was worth isolating before fixing. It needs both an awaited Tokio task and a driving thread inside `Runtime::block_on`; neither alone reproduces. An `enter()` guard is fine, wakeups from ordinary foreign threads are fine, and a plain `tokio::spawn` stalls just as a `spawn_blocking` does, so this is not about the blocking pool or about worker counts -- a multi-threaded runtime stalls identically. See #9817. DuckDB needs no Tokio here: `AsyncDB::run` is synchronous, `vortex-duckdb` drives its own `CurrentThreadRuntime`, and only DataFusion `.slt` files use the `sleep` and `system` directives. So the DuckDB path now drives its runner with `futures::executor::block_on` and the DataFusion path keeps its Tokio runtime. Also records the real hazard in `resolve_filesystem`, whose comment previously gave only the blocking-pool cost as the reason to keep local files off `Compat`. Fixes #9817 Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9w8fY73UoZKRboETTuBit
joseph-isaacs
force-pushed
the
claude/duckdb-vortex-scan-fixes
branch
from
September 10, 2026 16:03
385e8ec to
3675253
Compare
myrrc
approved these changes
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two bugs surfaced by running all 99 TPC-DS queries against Vortex through DuckDB in a single connection (the TPC-DS SLT suite is stacked on this PR). Both make DuckDB over Vortex either return wrong results or hang, so they are split out here for separate review.