Skip to content

Async runtime for io_uring - #472

Draft
proteetpaul wants to merge 27 commits into
datafusion-contrib:mainfrom
proteetpaul:io-uring-runtime
Draft

proteetpaul wants to merge 27 commits into
datafusion-contrib:mainfrom
proteetpaul:io-uring-runtime

Conversation

@proteetpaul

Copy link
Copy Markdown
Contributor

This commit introduces a multi-threaded runtime that integrates io_uring. Each worker thread has the following components:

  • A local async executor
  • An io_uring instance

Tasks are submitted to the worker thread using a crossbeam channel, and results are propagated back to the caller using a oneshot channel. When a task requires IO, it will yield to the runtime. The runtime is responsible for submitting IO's to the ring and polling IO completions. Upon completion, the runtime will unpark the corresponding task.

TODO():

  • Batch IO submissions to amortize the syscall overhead
  • Better admission control

@codecov

codecov Bot commented Jan 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.74359% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.89%. Comparing base (f3a8da2) to head (533869e).

Files with missing lines Patch % Lines
src/parquet/src/io/io_uring/tasks.rs 67.85% 5 Missing and 4 partials ⚠️
src/parquet/src/io/io_uring/runtime.rs 95.70% 3 Missing and 4 partials ⚠️
src/common/src/io_mode.rs 0.00% 2 Missing ⚠️
src/parquet/src/io/io_backend.rs 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #472      +/-   ##
==========================================
+ Coverage   83.83%   83.89%   +0.05%     
==========================================
  Files          97       98       +1     
  Lines       21526    21721     +195     
  Branches    21526    21721     +195     
==========================================
+ Hits        18047    18222     +175     
- Misses       3054     3066      +12     
- Partials      425      433       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

proteetpaul and others added 18 commits February 24, 2026 02:31
This commit implements a buffer pool for fixed buffers in order to remove memory allocation
and pinning overheads during io submission. The buffer pool is integrated into the uring
threadpool mechanism. Also implements minor optimizations in the uring threadpool, such as syscall batching.
- Syscall batching
- Handle multiple sqes per IO task
- Spawn a batch of tasks on the runtime
Partition batches more evenly, add support for other io modes, minor code cleanup
- Add error handling in pool.rs
- Run cargo fmt
- Minor fixes
@codacy-production

codacy-production Bot commented Apr 1, 2026

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 50 high · 10 medium

Results:
60 new issues

Category Results
Security 50 high
Complexity 10 medium

View in Codacy

🟢 Metrics 1436 complexity · 209 duplication

Metric Results
Complexity 1436
Duplication 209

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

- Steal tasks only from the global queue
- Basic admission control: Only a fixed number of tasks are admitted per worker
- Table formatting for results
- Use a tokio runtime if io mode is std-blocking
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant