feat: Adds instrumentation to collect tokio task metrics

[devanbenz]

This PR adds metrics collection for tokio tasks and polling statistics. Gathered some ideas from here and put together the following metrics to be collected.

struct LiquidTaskMetrics {
	total_tasks_n: AtomicU64,
	total_slow_poll_n: AtomicU64,
	total_poll_n: AtomicU64,
	mean_poll_duration_ms: AtomicU64,
	mean_idle_duration_ms: AtomicU64,
};

• total_tasks_n is the amount of tasks spawned during runtime
• total_slow_poll_n is the number of slow tasks
• total_poll_n is the amount of tasks in a polling state
• mean_poll_duration_ms shows how long polls are taking on average in milliseconds
• mean_idle_duration_ms shows the average idle duration in milliseconds

I think the most interesting metric as we explore the different IO methods will be mean_idle_duration_ms as described by the link I referenced above:

Did mean_idle_duration increase?

• This metric reflects the mean duration that tasks spent in the idle state. The idle state is the duration spanning the instant a task completes a poll, and the instant that it is next awoken. Tasks inhabit this state when they are waiting for task-external events to complete (e.g., an asynchronous sleep, a network request, file I/O, etc.). If this metric increases, tasks, in aggregate, spent more time waiting for task-external events to complete.

Overall different reasons for accounting these metrics can be seen here and in various sections throughout this document.

This PR is meant to be a building block so we can include these metrics via an admin server endpoint.

[codecov]

Codecov Report

❌ Patch coverage is 95.79832% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.94%. Comparing base (c1f5855) to head (0183e06).
⚠️ Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
...datafusion-server/src/admin_server/task_monitor.rs	95.87%	2 Missing and 2 partials ⚠️
src/datafusion/src/optimizers/lineage_opt.rs	50.00%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (c1f5855) and HEAD (0183e06). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (c1f5855)	HEAD (0183e06)
	4	3

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #208       +/-   ##
===========================================
- Coverage   83.83%   44.94%   -38.90%     
===========================================
  Files          91       82        -9     
  Lines       20246    13552     -6694     
  Branches    20246    13552     -6694     
===========================================
- Hits        16974     6091    -10883     
- Misses       2926     7251     +4325     
+ Partials      346      210      -136     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[devanbenz]

@XiangpengHao I just need to collect some metrics both without and with collecting tokio task metrics and compare whether this impacts the performance of the application overall.

[devanbenz]

Currently with my naive implementation I'm seeing that the performance metrics timings are a bit better without the task metrics. The flamegraphs also show a large chunk dealing with collecting task metrics. I'll likely need to refactor this to improve performance.

Without task metrics

With task metrics

[XiangpengHao]

Currently with my naive implementation I'm seeing that the performance metrics timings are a bit better without the task metrics. The flamegraphs also show a large chunk dealing with collecting task metrics. I'll likely need to refactor this to improve performance.

Without task metrics

With task metrics

happy to see that you're using this dashboard! This looks very cool

[devanbenz]

Alrighty, after a few adjustments I'm seeing much much better performance for tokio task metrics collection.

[devanbenz]

I'll set this PR ready for review and we can make any refinements you see fit. Could be nice to incorporate these metrics in to the admin dashboard as well. @XiangpengHao

[XiangpengHao]

Thank you @devanbenz , I plan to take a look at this in the next few days!

[XiangpengHao]

Sorry for the delay, I took a high level look -- this is super cool @devanbenz

I plan to educate myself a bit more about tokio_unstable and understand how the metrics work under the hood.
will come back to merge this in a few days!

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 10 complexity · 0 duplication

Metric	Results
Complexity	10
Duplication	0

View in Codacy

_{TIP This summary will be updated as you push new changes. Give us feedback}