[Diagnostics] Updating docs and adding performance investigation guidance - #55913
[Diagnostics] Updating docs and adding performance investigation guidance#55913mdh1418 wants to merge 7 commits into
Conversation
Describe dotnet-trace without requiring command-verb context, present record-trace as the user-facing OneCollect tool, and identify PerfCollect as the earlier LTTng-based Linux workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retain the original profiler-selection context, keep the Linux subsection focused on collect-linux, distinguish record-trace and perf workflows, and modernize .NET terminology. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Distinguish platform facilities from collectors, remove PerfCollect from the EventPipe comparison, keep the specialized diagnostics overview focused on mechanisms, and document PerfCollect as the earlier LTTng-dependent workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document PerfView and TraceEvent 3.2.1 analysis-time symbol lookup for .NET, R2R, and Azure Linux binaries, plus local symbol paths and a valid offline dotnet-symbol workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Clarify the recommended memory-leak and deadlock artifacts, use dotnet-dump consistently for deadlock collection, fix the ThreadPool trace command, and prefer collect-linux for intermittent starvation analysis on .NET 10+ Linux. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Provide a cross-tool decision path from observed symptoms to metrics, traces, snapshots, and dumps, and simplify the diagnostics landing page around that workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Provide one runnable sample with distinct CPU, memory, GC, blocking, I/O, exception, startup, process, mixed-cause, and healthy-control scenarios. Organize the walkthrough around the collection strategy and analysis pivot each symptom requires. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
A few documentation details are inconsistent or potentially misleading (PerfView platform note, --duration format, and one sentence that contradicts analysis-time symbol resolution), and there’s a user-facing typo.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This pull request expands the .NET diagnostics documentation with a symptom-first path for performance investigations, introduces dedicated dotnet-trace collect-linux guidance plus runnable Linux scenarios, and updates existing diagnostics pages to reflect current tooling and workflows.
Changes:
- Adds a new performance decision guide and two Linux-focused
collect-linuxarticles (workflow guidance plus scenario-based practice). - Updates the diagnostics landing page and navigation TOC to make the new performance path discoverable.
- Refreshes multiple existing diagnostics articles to align terminology and tool recommendations (PerfCollect vs
collect-linux, OneCollectrecord-trace, symbols guidance, and updated tutorials).
File summaries
| File | Description |
|---|---|
| docs/navigate/tools-diagnostics/toc.yml | Adds navigation entries for the new performance decision guide and Linux collect-linux articles. |
| docs/core/diagnostics/trace-perfcollect-lttng.md | Adds guidance to prefer collect-linux for .NET 10+ Linux, and clarifies LTTng version constraints. |
| docs/core/diagnostics/tools-overview.md | Updates tool descriptions and adds OneCollect record-trace context alongside PerfCollect/collect-linux. |
| docs/core/diagnostics/specialized-diagnostics-overview.md | Refreshes tracing/tool references and links into collect-linux guidance. |
| docs/core/diagnostics/performance-diagnostics.md | New symptom-first performance troubleshooting guide with platform-specific collection recommendations. |
| docs/core/diagnostics/index.md | Reorganizes the landing page to point readers toward the new performance decision guide and updated tutorial grouping. |
| docs/core/diagnostics/eventpipe.md | Updates EventPipe comparisons and points to collect-linux and OneCollect for platform context. |
| docs/core/diagnostics/dotnet-trace.md | Adds collect-linux cross-links and expands symbol-resolution guidance for Linux traces. |
| docs/core/diagnostics/dotnet-trace-collect-linux-scenarios.md | New tutorial with runnable Linux performance investigation scenarios. |
| docs/core/diagnostics/dotnet-trace-collect-linux-performance.md | New how-to guide for collecting and analyzing collect-linux traces across common symptom areas. |
| docs/core/diagnostics/debug-threadpool-starvation.md | Updates commands and adds collect-linux guidance for Linux blocking/ThreadPool investigations. |
| docs/core/diagnostics/debug-memory-leak.md | Refreshes prerequisites/terminology and adds guidance for choosing dotnet-gcdump vs dumps. |
| docs/core/diagnostics/debug-highcpu.md | Refreshes the tutorial and adds Linux guidance for collect-linux, OneCollect, and perf workflows. |
| docs/core/diagnostics/debug-deadlock.md | Refreshes the tutorial to emphasize dumps for deadlocks and points to Linux trace guidance for deadlock formation. |
Review details
- Files reviewed: 14/14 changed files
- Comments generated: 6
- Review effort level: Lite
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
| Running Bombadier to send load to the `api/diagscenario/taskasyncwait` endpoint shows that the ThreadPool thread count stays much lower and average latency remains near 500ms when using the async/await approach: | ||
|
|
||
| ```dotnetcli | ||
| >bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/taskasyncwait | ||
| bombardier https://localhost:5001/api/diagscenario/taskasyncwait |
| sudo dotnet-trace collect-linux --profile dotnet-common,cpu-sampling | ||
| ``` | ||
|
|
||
| Start collection shortly before reproducing the problem, capture a representative 10-30 second interval, and stop collection after the symptom occurs. Keeping the first trace short bounds collection overhead and trace size while retaining enough context to choose the next investigation step. Press <kbd>Enter</kbd> or <kbd>Ctrl</kbd>+<kbd>C</kbd> to stop interactively, or add a duration such as `--duration 00:00:00:30`. The profile in the preceding command is the default configuration, shown explicitly. |
|
|
||
| Resolve native and ReadyToRun frames as described in [Get symbols for native runtime frames](dotnet-trace.md#get-symbols-for-native-runtime-frames). | ||
|
|
||
| Open the trace in a current version of PerfView and filter the result to the relevant process and time range. |
| - Linux that meets the [`collect-linux` prerequisites](dotnet-trace.md#prerequisites). | ||
| - The [.NET 10 SDK](https://dotnet.microsoft.com/download/dotnet/10.0). | ||
| - The latest [`dotnet-trace`](dotnet-trace.md) global tool. | ||
| - A current version of [PerfView](https://github.com/microsoft/perfview) for analysis. |
| @@ -727,6 +735,8 @@ To download native runtime symbols, use [dotnet-symbol](./dotnet-symbol.md): | |||
|
|
|||
| After you place the symbols, `collect-linux` resolves native method names when it collects the trace. | |||
| ### dotnet-trace | ||
|
|
||
| .NET Core includes `EventPipe`, which exposes diagnostics data. The [dotnet-trace](dotnet-trace.md) tool allows you to consume interesting profiling data from your app that can help in scenarios where you need to root-cause apps running that are running slowly. | ||
| The [dotnet-trace](dotnet-trace.md) tool is a cross-platform .NET diagnostic tool that collects traces from running applications without using a native profiler. On Linux, it can also combine .NET runtime and application events with machine-wide CPU samples, native call stacks, and Linux kernel events collected through the `perf_events` facility. For a symptom-driven workflow, see [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md). |
noahfalk
left a comment
noahfalk
left a comment
There was a problem hiding this comment.
I put some comments inline but this might be something you want to iterate on with @lateralusX. I think you may want to adjust the structure and scope somewhat before worrying about the smaller stuff.
| The API request to the site will stop responding. Let the request run for about 10-15 seconds. Then create the core dump using the following command: | ||
| The API request to the site will stop responding. Let the request run for about 10-15 seconds. | ||
|
|
||
| A dump is the recommended artifact for an existing deadlock because it preserves the current threads, lock owners, and wait cycle. If the deadlock is intermittent or you need to understand how it formed, start a contention and thread-time trace before reproducing it. For a Linux example, see [Capture deadlock formation](dotnet-trace-collect-linux-scenarios.md#capture-deadlock-formation). |
There was a problem hiding this comment.
A deadlock normally means two or more threads that block on each other indefinitely so anything that is intermittent isn't usually considered a deadlock. Without trying to be pedantic though I'd suggest not mentioning contention or the link here just to stay focused on a single symptom and its resolution. In the future if we suspected people were coming to this tutorial in error and really they had an intermittent latency issue rather than a deadlock then we might want to put some clarifying text/links in the intro to redirect them. Ideally we'd also redirect them to something that is more OS agnostic rather than just a linux specific example.
|
|
||
| ### [Linux](#tab/linux) | ||
|
|
||
| Prefer `dotnet-trace collect-linux` for the .NET-oriented Linux workflow. Use OneCollect `record-trace` when you need its lower-level scripting, filtering, or output controls, and use `perf` directly only when you need `perf.data`, perf-native analysis, or hardware performance counters. |
There was a problem hiding this comment.
Since this is a tutorial I think its better to offer fewer options when possible. I'd suggest we only describe dotnet-trace collect-linux for .NET 10+ and perf as an alternative covering scenarios where collect-linux isn't available. Currently the user experience for record-trace doesn't feel mature enough that its worth complicating the tutorial to add it (the project README describes it as pre-release with no clear distribution or usage documentation)
| Open the `.nettrace` with [`PerfView`](https://github.com/microsoft/perfview/blob/main/documentation/Downloading.md) and use the **CPU Stacks** view to identify the methods consuming the most CPU time. | ||
|
|
||
| For information about resolving native runtime symbols in the trace, see [Get symbols for native runtime frames](dotnet-trace.md#get-symbols-for-native-runtime-frames). | ||
| PerfView and TraceEvent 3.2.1 or later can resolve .NET native and R2R symbols at analysis time. In PerfView, select unresolved module frames and choose **Lookup Symbols**. For other native libraries, configure a local symbol path. For more information, see [Get symbols for native runtime frames](dotnet-trace.md#get-symbols-for-native-runtime-frames). |
There was a problem hiding this comment.
| PerfView and TraceEvent 3.2.1 or later can resolve .NET native and R2R symbols at analysis time. In PerfView, select unresolved module frames and choose **Lookup Symbols**. For other native libraries, configure a local symbol path. For more information, see [Get symbols for native runtime frames](dotnet-trace.md#get-symbols-for-native-runtime-frames). | |
| PerfView 3.2.1 or later can resolve .NET native and R2R symbols at analysis time. In PerfView, select unresolved module frames and choose **Lookup Symbols**. For other native libraries, configure a local symbol path. For more information, see [Get symbols for native runtime frames](dotnet-trace.md#get-symbols-for-native-runtime-frames). |
Tutorial users will probably be using GUI tools, not writing custom tooling using TraceEvent.
| @@ -191,11 +191,28 @@ Let it run for about 20-30 seconds, then press <kbd>Ctrl+C</kbd> or <kbd>Enter</ | |||
|
|
|||
| Open the `.nettrace` with [`PerfView`](https://github.com/microsoft/perfview/blob/main/documentation/Downloading.md) and use the **CPU Stacks** view to identify the methods consuming the most CPU time. | |||
There was a problem hiding this comment.
Is PerfView still a better experience than Visual Studio here? If it is thats fine but hopefully we'd be working with the VS team to resolve whatever issues make us recommend PerfView instead of VS.
|
|
||
| For a broader workflow that covers CPU, blocking, GC, exceptions, I/O, and startup, see [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md). | ||
|
|
||
| #### Use OneCollect `record-trace` |
There was a problem hiding this comment.
Suggest we remove this section for brevity and clarity.
| ### Performance tutorials | ||
|
|
||
| [Tutorial: Measure performance using EventCounters in .NET](event-counter-perf.md) shows you how to use the <xref:System.Diagnostics.Tracing.EventCounter> API to measure performance in your .NET app. | ||
| Use [Diagnose performance issues in .NET applications](performance-diagnostics.md) to choose the recommended workflow for a performance symptom. The guide links to the applicable detailed tutorials and hands-on exercises. |
There was a problem hiding this comment.
I think folks will be more likely to click through this if you offer some specific examples of performance problems that this guide is going to help them diagnose.
| [Tutorial: Debug a StackOverflow](debug-stackoverflow.md) demonstrates how to debug a <xref:System.StackOverflowException> on Linux. | ||
|
|
||
| ### Debug Linux dumps | ||
| .NET supports a number of [CLI tools](./tools-overview.md) that can be used to diagnose your applications. To automate a custom diagnostic workflow, use the [diagnostics client library](diagnostics-client-library.md) and <xref:Microsoft.Diagnostics.NETCore.Client>. |
There was a problem hiding this comment.
The linked content actually includes VS and VS code so we probably don't want to describe it as just "CLI tools".
|
|
||
| ## Start with the symptom | ||
|
|
||
| Before collecting a large trace or dump, record: |
There was a problem hiding this comment.
I'm not sure how helpful this will be.
I think we should assume the audience reading this already knows:
- a rough description of the problem
- some context about the scope of the problem (a machine, a service, a process)
I don't think we should assume they necessarily know:
- The time range in which the problem occurred - this is probably irrelevant if we are assuming the problem is reproducable and we'll monitor future instances of it
- cpu usage, memory usage, request rate, latency, error rate - figuring out some of this info might be exactly what the reader hopes this guide will help them do.
- Whether the machine, container, or only one process is resource constrained - ditto
| #Customer intent: As a .NET developer, I want to choose the right diagnostic tools and data to find the cause of a performance problem. | ||
| --- | ||
|
|
||
| # Diagnose performance issues in .NET applications |
There was a problem hiding this comment.
If we are going to funnel all the performance investigation tutorials through here then I'd expect this guide to start approximately where the existing tutorials start by showing people how to use dotnet-counters and/or dotnet-trace in detail to get from initial symptoms to a more specific workflow. Its a noble effort to try creating a universal triage workflow but it might be easier to just skip this part and let folks explore the existing performance investigation tutorials one by one.
| | macOS | Use Xcode Instruments for CPU and native profiling. Use [`dotnet-trace collect`](dotnet-trace.md#dotnet-trace-collect) separately when you need .NET runtime events or managed stack samples. | | ||
| | Automated production or container collection | Use [`dotnet-monitor`](dotnet-monitor.md) for automated .NET diagnostics, and pair it with the platform profiler when the investigation requires operating-system or native context. | | ||
|
|
||
| On Linux or Windows, use OneCollect [`record-trace`](https://github.com/microsoft/one-collect/tree/main/record-trace) when you need lower-level scripts, event selection, process or CPU filtering, or alternate output formats. On Linux, use `perf` directly when you specifically require `perf.data`, perf-native analysis, or hardware performance counters. [PerfCollect](trace-perfcollect-lttng.md) is the earlier Linux workflow and its runtime-event collection requires LTTng 2.12; it isn't the default fallback for `collect-linux`. |
There was a problem hiding this comment.
Same as elsewhere, I don't think OneCollect's project advertises a mature enough state for our docs to be pointing at it.
Summary
Improve the .NET diagnostics documentation so readers can move from an observed performance symptom to the appropriate platform collector, diagnostic artifact, detailed workflow, and hands-on exercise.
This change:
dotnet-trace collect-linuxinvestigation guide covering broad-first collection, focused GC, blocking, exception, I/O, and startup configurations, PerfView analysis, collection overhead, and the limits that require another diagnostic artifact.dotnet-trace collect-linux, OneCollectrecord-trace, Linuxperf, and the earlier LTTng-based PerfCollect workflow.The scenario walkthrough uses the companion
dotnet/samplessample at/samples/dotnet/samples/dotnet-trace-collect-linux-performance-scenarios/. The samples PR should merge before or with this PR so that route is available when the walkthrough is published.Validation included checking all changed relative links and heading anchors, parsing the diagnostics TOC as YAML, checking every commit independently for forward references, and verifying Markdown formatting.
Internal previews
Toggle expand/collapse
Build report