From 71b26d71cb6cba545fd5e0d1d4dfcd7c9a89d942 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:07:43 +0100 Subject: [PATCH 001/110] Add docs for Microsoft.Testing.Platform integration Added a new guide for running benchmarks with Microsoft.Testing.Platform (MTP), covering setup, usage, and caveats. Updated the table of contents to include the new page and added a note to the VSTest docs about the MTP adapter option. --- docs/articles/features/testingplatform.md | 177 ++++++++++++++++++++++ docs/articles/features/toc.yml | 4 +- docs/articles/features/vstest.md | 5 + 3 files changed, 185 insertions(+), 1 deletion(-) create mode 100644 docs/articles/features/testingplatform.md diff --git a/docs/articles/features/testingplatform.md b/docs/articles/features/testingplatform.md new file mode 100644 index 0000000000..85fffeeb0b --- /dev/null +++ b/docs/articles/features/testingplatform.md @@ -0,0 +1,177 @@ +--- +uid: docs.testingplatform +name: Running with Microsoft.Testing.Platform +--- + +# Running with Microsoft.Testing.Platform + +BenchmarkDotNet can discover and execute benchmarks through + [Microsoft.Testing.Platform](https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-intro) (MTP), + the test platform that succeeds VSTest. +This gives you the same "benchmarks as tests" experience as [the VSTest adapter](xref:docs.vstest), + but on the platform that `dotnet test` and modern IDE integrations are moving to. + +If you are looking for the VSTest adapter, see [Running with VSTest](xref:docs.vstest) instead. +You only need one of the two. + +## VSTest or Microsoft.Testing.Platform? + +The two adapters solve the same problem on different platforms, and the difference that matters most is *where your + benchmarks run*: + +* With **VSTest**, an external `testhost` process loads your benchmark assembly and the adapter reflects into it. +* With **Microsoft.Testing.Platform**, your benchmark project *is* the test host. + There is no separate host process, so BenchmarkDotNet behaves exactly as it does when you run the app from the CLI. + +The practical consequences of the MTP model are: + +* The adapter no longer needs the child `AppDomain` that the VSTest adapter uses to load your assemblies correctly. +* Your project's entry point is generated by the platform and starts the test application, + so it no longer calls `BenchmarkSwitcher`. See [Keeping a BenchmarkSwitcher entry point](#keeping-a-benchmarkswitcher-entry-point). + +## Caveats and things to know + +* **The benchmark measurements may be affected by the test host and your IDE!** + If you want accurate measurements, + it is still recommended to run benchmarks through the CLI without other processes impacting performance. + The measurements remain useful during development when comparing different approaches. +* **The adapter will not display or execute benchmarks if optimizations are disabled.** + Please ensure you are compiling in Release mode or with `Optimize` set to true. + Using an `InProcess` toolchain will let you run your benchmarks with optimizations disabled + and will let you attach the debugger as well. +* **The adapter will not call your application's entry point.** + If you use the entry point to customize how your benchmarks are run, + you will need to do this through other means such as an assembly-level `IConfigSource`, + as shown in [Setting a default configuration](xref:docs.vstest#setting-a-default-configuration). +* **The adapter will generate an entry point for you automatically.** + Unlike the VSTest adapter, the generated entry point starts the test application rather than `BenchmarkSwitcher`. + +## Getting started + +* **Step 1.** Install the NuGet package. + Only one package is needed; it brings in `Microsoft.Testing.Platform` and the MSBuild integration for you: + +```xml + + + +``` + +* **Step 2.** Make sure the project is an executable and does not define its own entry point. + Microsoft.Testing.Platform applications are executables, and the package generates the entry point for you. + Here is a complete `.csproj` based on the default Console Application template: + +```xml + + + + Exe + net10.0 + enable + enable + + + + + + + +``` + +> [!NOTE] +> The name of your project file must match the name of the produced assembly. +> This is a general BenchmarkDotNet requirement: it rebuilds your project to run benchmarks out of process. + +* **Step 3.** Opt into the Microsoft.Testing.Platform mode of `dotnet test`. + On the .NET 10 SDK and later this is required, because `dotnet test` runs in VSTest mode by default. + Add a `global.json` next to your solution: + +```json +{ + "test": { + "runner": "Microsoft.Testing.Platform" + } +} +``` + + On the .NET 9 SDK and earlier this step is not needed: + the package sets `TestingPlatformDotnetTestSupport` for you, which routes `dotnet test` to the platform. + +* **Step 4.** Switch to the `Release` configuration. + As mentioned above, the adapter does not discover or run benchmarks with optimizations disabled (by design). + +* **Step 5.** Build and run. + +```console +dotnet test -c Release +``` + + You can also run the produced executable directly, which is the same thing without going through MSBuild: + +```console +dotnet run -c Release +``` + +If this doesn't work for you, don't hesitate to file [a new GitHub issue](https://github.com/dotnet/BenchmarkDotNet/issues/new). + +## Listing and filtering benchmarks + +The benchmark project is a normal Microsoft.Testing.Platform application, so it accepts the platform's options. +Run it with `--help` to see all of them; the ones you are most likely to want are: + +```console +# List the benchmarks without running them. +dotnet run -c Release -- --list-tests + +# Run every benchmark of a class. +dotnet run -c Release -- --treenode-filter "/*/*/MyBenchmarks/*" + +# Run every benchmark of a category. +dotnet run -c Release -- --treenode-filter "/*/*/*/*[Category=Fast]" + +# Run one specific benchmark, by the exact id reported by the platform. +dotnet run -c Release -- --filter-uid "MyProject.MyBenchmarks.Add(x: 1) [DefaultJob]" +``` + +The tree node filter path is `////`, + and `[BenchmarkCategory]` attributes are exposed as a `Category` trait that the filter can match on. + +## Keeping a BenchmarkSwitcher entry point + +The generated entry point starts the test application, which means it replaces the `BenchmarkSwitcher` entry point that + a benchmark project normally has. +If you want to keep your own entry point, turn off the generated one and register BenchmarkDotNet yourself: + +```xml + + false + +``` + +```csharp +using BenchmarkDotNet.TestAdapter.TestingPlatform; +using Microsoft.Testing.Platform.Builder; + +public static class Program +{ + public static async Task Main(string[] args) + { + var builder = await TestApplication.CreateBuilderAsync(args); + builder.AddBenchmarkDotNet(); + using var app = await builder.BuildAsync(); + return await app.RunAsync(); + } +} +``` + +From there you are free to decide when to start the test application and when to hand over to `BenchmarkSwitcher`, + for example by looking at the arguments your CI passes. + +## Viewing the results + +The full BenchmarkDotNet output, including the summary table that compares benchmarks with each other, + is written to the test run output. + +In addition, each individual benchmark reports its own output, containing a histogram and various statistics for that + single benchmark case. +Depending on your IDE, this is shown when selecting the test after running it. diff --git a/docs/articles/features/toc.yml b/docs/articles/features/toc.yml index b456ed66be..f2fb0728d2 100644 --- a/docs/articles/features/toc.yml +++ b/docs/articles/features/toc.yml @@ -17,4 +17,6 @@ - name: VSProfiler href: vsprofiler.md - name: VSTest - href: vstest.md \ No newline at end of file + href: vstest.md +- name: Microsoft.Testing.Platform + href: testingplatform.md \ No newline at end of file diff --git a/docs/articles/features/vstest.md b/docs/articles/features/vstest.md index 3e94901bdd..e86527211c 100644 --- a/docs/articles/features/vstest.md +++ b/docs/articles/features/vstest.md @@ -5,6 +5,11 @@ name: Running with VSTest # Running with VSTest +> [!NOTE] +> BenchmarkDotNet also ships an adapter for [Microsoft.Testing.Platform](xref:docs.testingplatform), +> the test platform that succeeds VSTest. +> You only need one of the two. + BenchmarkDotNet supports discovering and executing benchmarks through VSTest. This provides an alternative user experience to running benchmarks with the CLI and may be preferable for those who like their IDE's VSTest integrations that they may have used when running unit tests. From 8918420561a72698e012f764bd80472c134874e3 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:08:20 +0100 Subject: [PATCH 002/110] Add InternalsVisibleTo for TestAdapter.TestingPlatform Added InternalsVisibleTo attribute in AssemblyInfo.cs to expose internal members to the BenchmarkDotNet.TestAdapter.TestingPlatform assembly, ensuring it uses the same public key as other related assemblies. --- src/BenchmarkDotNet/Properties/AssemblyInfo.cs | 1 + 1 file changed, 1 insertion(+) diff --git a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs index aa93984520..5a5ff709d3 100644 --- a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs +++ b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs @@ -15,3 +15,4 @@ [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning.MultipleFrameworks,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] +[assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter.TestingPlatform,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] From 3dd579644cfe191d745fff973a63a2082d410ec5 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:08:40 +0100 Subject: [PATCH 003/110] Remove GetUnrandomizedJobDisplayInfo method Deleted the internal static method GetUnrandomizedJobDisplayInfo from BenchmarkCaseExtensions.cs. This method handled normalization of job display info by removing randomness from job IDs for consistent benchmark referencing. No other code changes were made. --- .../BenchmarkCaseExtensions.cs | 21 ------------------- 1 file changed, 21 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseExtensions.cs b/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseExtensions.cs index 06e5782b6a..08ad640e26 100644 --- a/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseExtensions.cs +++ b/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseExtensions.cs @@ -1,5 +1,4 @@ using BenchmarkDotNet.Attributes; -using BenchmarkDotNet.Characteristics; using BenchmarkDotNet.Exporters; using BenchmarkDotNet.Extensions; using BenchmarkDotNet.Running; @@ -65,26 +64,6 @@ internal static TestCase ToVsTestCase(this BenchmarkCase benchmarkCase, string a return vsTestCase; } - /// - /// If an ID is not provided, a random string is used for the ID. This method will identify if randomness was - /// used for the ID and return the Job's DisplayInfo with that randomness removed so that the same benchmark - /// can be referenced across multiple processes. - /// - /// The benchmark case. - /// The benchmark case' job's DisplayInfo without randomness. - internal static string GetUnrandomizedJobDisplayInfo(this BenchmarkCase benchmarkCase) - { - var jobDisplayInfo = benchmarkCase.Job.DisplayInfo; - if (!benchmarkCase.Job.HasValue(CharacteristicObject.IdCharacteristic) && - benchmarkCase.Job.ResolvedId.StartsWith("Job-", StringComparison.OrdinalIgnoreCase)) - { - // Replace Job-ABCDEF with Job - jobDisplayInfo = "Job" + jobDisplayInfo.Substring(benchmarkCase.Job.ResolvedId.Length); - } - - return jobDisplayInfo; - } - /// /// Gets an ID for a given BenchmarkCase that is uniquely identifiable from discovery to execution phase. /// From 695ee488ec41dee8f7908a4619b1f3fa95c8b516 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:08:56 +0100 Subject: [PATCH 004/110] Add BenchmarkCaseIdentityExtensions for stable job IDs Introduced BenchmarkCaseIdentityExtensions with GetUnrandomizedJobDisplayInfo to normalize Job DisplayInfo by removing random ID components. This ensures consistent benchmark identification across processes for test adapters. --- .../BenchmarkCaseIdentityExtensions.cs | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs b/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs new file mode 100644 index 0000000000..78cb64f5ce --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs @@ -0,0 +1,32 @@ +using BenchmarkDotNet.Characteristics; +using BenchmarkDotNet.Running; + +namespace BenchmarkDotNet.TestAdapter +{ + /// + /// Helpers for deriving stable identities for a BenchmarkCase. Shared by the VSTest and the + /// Microsoft.Testing.Platform adapters, because both need identities that survive across processes. + /// + internal static class BenchmarkCaseIdentityExtensions + { + /// + /// If an ID is not provided, a random string is used for the ID. This method will identify if randomness was + /// used for the ID and return the Job's DisplayInfo with that randomness removed so that the same benchmark + /// can be referenced across multiple processes. + /// + /// The benchmark case. + /// The benchmark case' job's DisplayInfo without randomness. + internal static string GetUnrandomizedJobDisplayInfo(this BenchmarkCase benchmarkCase) + { + var jobDisplayInfo = benchmarkCase.Job.DisplayInfo; + if (!benchmarkCase.Job.HasValue(CharacteristicObject.IdCharacteristic) && + benchmarkCase.Job.ResolvedId.StartsWith("Job-", StringComparison.OrdinalIgnoreCase)) + { + // Replace Job-ABCDEF with Job + jobDisplayInfo = "Job" + jobDisplayInfo.Substring(benchmarkCase.Job.ResolvedId.Length); + } + + return jobDisplayInfo; + } + } +} From 770bd0d39ed02216c2525835f1a01d27078dd960 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:09:26 +0100 Subject: [PATCH 005/110] Refactor benchmark extraction into reusable method Introduce GetBenchmarksFromAssembly to extract benchmarks from an already loaded Assembly. Refactor existing logic to use this method, improving code reuse and enabling benchmark retrieval from both loaded assemblies and file paths. --- .../BenchmarkEnumerator.cs | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs b/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs index e8cbc4f130..f001ef45f8 100644 --- a/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs +++ b/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs @@ -48,8 +48,16 @@ public static BenchmarkRunInfo[] GetBenchmarksFromAssemblyPath(string assemblyPa }; #endif - var assembly = Assembly.LoadFrom(assemblyPath); + return GetBenchmarksFromAssembly(Assembly.LoadFrom(assemblyPath)); + } + /// + /// Returns all the BenchmarkRunInfo objects from an already loaded assembly. + /// + /// The assembly of the benchmark project. + /// The benchmarks inside the assembly. + public static BenchmarkRunInfo[] GetBenchmarksFromAssembly(Assembly assembly) + { var isDebugAssembly = assembly.IsJitOptimizationDisabled() ?? false; return GenericBenchmarksBuilder.GetRunnableBenchmarks(assembly.GetRunnableBenchmarks()) @@ -59,7 +67,7 @@ public static BenchmarkRunInfo[] GetBenchmarksFromAssemblyPath(string assemblyPa if (isDebugAssembly) { // If the assembly is a debug assembly, then only display them if they will run in-process - // This will allow people to debug their benchmarks using VSTest if they wish. + // This will allow people to debug their benchmarks from a test runner if they wish. benchmarkRunInfo = new BenchmarkRunInfo( benchmarkRunInfo.BenchmarksCases.Where(c => c.GetToolchain().IsInProcess).ToArray(), benchmarkRunInfo.Type, From 0fa20cd8f7a4525876060389dac9fac10c45ebe1 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:09:57 +0100 Subject: [PATCH 006/110] Integrate BenchmarkDotNet with Microsoft.Testing.Platform Add MSBuild props to enable TestingPlatform integration, set defaults for `dotnet test` compatibility, disable parallel TFM runs, and auto-register BenchmarkDotNet builder hook. --- ...rkDotNet.TestAdapter.TestingPlatform.props | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props new file mode 100644 index 0000000000..7b6f563554 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props @@ -0,0 +1,31 @@ + + + + true + + + true + + + false + + + + + + BenchmarkDotNet + BenchmarkDotNet.TestAdapter.TestingPlatform.TestingPlatformBuilderHook + + + From a97a8f185552576ad4a16c63f43880a463dc2395 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:10:37 +0100 Subject: [PATCH 007/110] Add AsyncWorkQueue for async work item processing Introduced AsyncWorkQueue, an internal sealed class in BenchmarkDotNet.TestAdapter.TestingPlatform. It enables ordered, thread-safe queuing of asynchronous work items, allowing synchronous producers and asynchronous consumers. Utilizes ConcurrentQueue and SemaphoreSlim, supports completion signaling, and implements IDisposable for resource cleanup. --- .../AsyncWorkQueue.cs | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs new file mode 100644 index 0000000000..128eb51b0c --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs @@ -0,0 +1,61 @@ +using System.Collections.Concurrent; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// An ordered queue of asynchronous work items that are produced synchronously and consumed asynchronously. + /// + /// + /// BenchmarkDotNet reports its progress through synchronous callbacks (EventProcessor and ILogger) while the + /// platform message bus and output device are asynchronous. Blocking on those from inside a callback risks + /// deadlocking against the synchronization context BenchmarkDotNet installs while it runs, so the callbacks + /// enqueue instead and the caller of DrainAsync does the awaiting. + /// + internal sealed class AsyncWorkQueue : IDisposable + { + private readonly ConcurrentQueue> queue = new(); + private readonly SemaphoreSlim available = new(0); + private volatile bool completed; + + /// + /// Queues a work item. Safe to call from any thread. + /// + /// The work to perform. + public void Enqueue(Func work) + { + queue.Enqueue(work); + available.Release(); + } + + /// + /// Signals that no more work items will be enqueued, which lets DrainAsync return once the already queued + /// items have been processed. + /// + public void Complete() + { + completed = true; + available.Release(); + } + + /// + /// Processes queued work items in order until has been called and the queue is empty. + /// + /// A task that completes when the queue has been drained. + public async Task DrainAsync() + { + while (true) + { + await available.WaitAsync().ConfigureAwait(false); + + // Draining everything on each wake-up means a lost or surplus permit can never strand a work item. + while (queue.TryDequeue(out var work)) + await work().ConfigureAwait(false); + + if (completed && queue.IsEmpty) + return; + } + } + + public void Dispose() => available.Dispose(); + } +} From ff50c125601df76e6bef44e804e38b1f5efc4c76 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:11:47 +0100 Subject: [PATCH 008/110] Add BenchmarkDotNet.TestAdapter.TestingPlatform project Created a new .csproj targeting netstandard2.0 for the TestingPlatform adapter. Configured project metadata, packaging, and references. Integrated Microsoft.Testing.Platform.MSBuild and BenchmarkDotNet, and linked shared source files for benchmark enumeration. Set IsTestingPlatformApplication to false to avoid test app behavior. --- ...kDotNet.TestAdapter.TestingPlatform.csproj | 46 +++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj new file mode 100644 index 0000000000..c3dd5c4cb3 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj @@ -0,0 +1,46 @@ + + + + netstandard2.0 + BenchmarkDotNet.TestAdapter.TestingPlatform + BenchmarkDotNet.TestAdapter.TestingPlatform + BenchmarkDotNet.TestAdapter.TestingPlatform + Runs BenchmarkDotNet benchmarks as tests through Microsoft.Testing.Platform + README.md + True + BenchmarkDotNet.TestAdapter.TestingPlatform + + + false + + + + + + + + + + + + + + + + + + + + + + + + + + From 46e4df0b5fa744a7d6893e294c38001f0b6d08a5 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:13:27 +0100 Subject: [PATCH 009/110] Add BenchmarkDotNetExtension for platform integration Introduced BenchmarkDotNetExtension class implementing IExtension to provide extension metadata and enablement status for Microsoft.Testing.Platform integration. --- .../BenchmarkDotNetExtension.cs | 34 +++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs new file mode 100644 index 0000000000..789fbc611a --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs @@ -0,0 +1,34 @@ +using Microsoft.Testing.Platform.Extensions; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// Identifies BenchmarkDotNet to Microsoft.Testing.Platform. + /// + /// + /// Several platform services, such as the tree node filter behind --filter, are registered on behalf of an + /// extension rather than of the test framework itself, so the identity lives in its own type. + /// + internal sealed class BenchmarkDotNetExtension : IExtension + { + /// + /// The uid shared by every extension this package registers. + /// + public const string ExtensionUid = "BenchmarkDotNet.TestAdapter.TestingPlatform"; + + /// + public string Uid => ExtensionUid; + + /// + public string Version => typeof(BenchmarkDotNetExtension).Assembly.GetName().Version?.ToString() ?? "0.0.0"; + + /// + public string DisplayName => "BenchmarkDotNet"; + + /// + public string Description => "Runs BenchmarkDotNet benchmarks as tests."; + + /// + public Task IsEnabledAsync() => Task.FromResult(true); + } +} From 2a004ccb4db49c58f442e29b7cf488f28bd62467 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:14:56 +0100 Subject: [PATCH 010/110] Add BenchmarkEventProcessor for test node event handling Introduced BenchmarkEventProcessor to process BenchmarkDotNet events and translate them into test node updates for the testing platform. Handles validation errors, build results, benchmark execution, and ensures all benchmarks have published results. Includes logic for error aggregation, output formatting, and timing information. --- .../BenchmarkEventProcessor.cs | 191 ++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs new file mode 100644 index 0000000000..b48e5cd6e6 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs @@ -0,0 +1,191 @@ +using BenchmarkDotNet.EventProcessors; +using BenchmarkDotNet.Extensions; +using BenchmarkDotNet.Reports; +using BenchmarkDotNet.Running; +using BenchmarkDotNet.Toolchains.Results; +using BenchmarkDotNet.Validators; +using Microsoft.Testing.Platform.Extensions.Messages; +using Perfolizer.Mathematics.Histograms; +using System.Diagnostics; +using System.Globalization; +using System.Text; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// Translates the BenchmarkDotNet run into the stream of test node updates the platform expects. + /// + internal sealed class BenchmarkEventProcessor : EventProcessor + { + private readonly IReadOnlyDictionary nodes; + private readonly Action publish; + private readonly Stopwatch runTimerStopwatch = new(); + private readonly Dictionary pendingResults = []; + private readonly HashSet publishedResults = []; + + public BenchmarkEventProcessor(IReadOnlyDictionary nodes, Action publish) + { + this.nodes = nodes; + this.publish = publish; + } + + public override void OnValidationError(ValidationError validationError) + { + // If the error is not linked to a benchmark case, then set the error on all benchmarks. + var affected = validationError.BenchmarkCase == null + ? nodes.Values + : [nodes[BenchmarkTestNode.GetUid(validationError.BenchmarkCase)]]; + + foreach (var node in affected) + { + var pending = GetOrCreatePendingResult(node); + + if (validationError.IsCritical) + { + // The result is not published yet, in case there are more validation errors to append. + pending.ErrorMessages.Add(validationError.Message); + } + else + { + pending.Output.AppendLine($"WARNING: {validationError.Message}"); + } + } + } + + public override void OnBuildComplete(BuildPartition buildPartition, BuildResult buildResult) + { + // Only build failures need to be reported, successful builds are followed by a run. + if (buildResult.IsBuildSuccess) + return; + + foreach (var benchmarkBuildInfo in buildPartition.Benchmarks) + { + var node = nodes[BenchmarkTestNode.GetUid(benchmarkBuildInfo.BenchmarkCase)]; + var pending = GetOrCreatePendingResult(node); + + if (buildResult.GenerateException != null) + pending.ErrorMessages.Add($"// Generate Exception: {buildResult.GenerateException.Message}"); + else if (buildResult.TryToExplainFailureReason(out string? reason)) + pending.ErrorMessages.Add($"// Build Error: {reason}"); + else if (buildResult.ErrorMessage != null) + pending.ErrorMessages.Add($"// Build Error: {buildResult.ErrorMessage}"); + + // A benchmark that failed to build will never run, so the result can be published immediately. + publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); + PublishResult(node, pending, new FailedTestNodeStateProperty(pending.GetErrorMessage() ?? "The benchmark failed to build.")); + } + } + + public override void OnStartRunBenchmark(BenchmarkCase benchmarkCase) + { + var node = nodes[BenchmarkTestNode.GetUid(benchmarkCase)]; + var pending = GetOrCreatePendingResult(node); + pending.StartTime = DateTimeOffset.UtcNow; + + publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); + runTimerStopwatch.Restart(); + } + + public override void OnEndRunBenchmark(BenchmarkCase benchmarkCase, BenchmarkReport report) + { + var node = nodes[BenchmarkTestNode.GetUid(benchmarkCase)]; + var pending = GetOrCreatePendingResult(node); + pending.Duration = runTimerStopwatch.Elapsed; + pending.EndTime = DateTimeOffset.UtcNow; + + AppendMeasurementSummary(pending.Output, report); + + TestNodeStateProperty state = report.Success && pending.ErrorMessages.Count == 0 + ? PassedTestNodeStateProperty.CachedInstance + : new FailedTestNodeStateProperty(pending.GetErrorMessage() ?? "The benchmark did not complete successfully."); + + PublishResult(node, pending, state); + } + + /// + /// Publishes a result for every benchmark that was scheduled to run but never reported one, which happens when + /// a critical validation error stopped it or when BenchmarkDotNet never reached it. + /// + public void PublishOutstandingResults() + { + foreach (var node in nodes.Values) + { + if (publishedResults.Contains(node.Uid)) + continue; + + var pending = GetOrCreatePendingResult(node); + var errorMessage = pending.GetErrorMessage(); + + TestNodeStateProperty state = errorMessage != null + ? new FailedTestNodeStateProperty(errorMessage) + : SkippedTestNodeStateProperty.CachedInstance; + + publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); + PublishResult(node, pending, state); + } + } + + private void PublishResult(BenchmarkTestNode node, PendingResult pending, TestNodeStateProperty state) + { + var properties = new List(); + + if (pending.StartTime is { } startTime) + { + var duration = pending.Duration ?? TimeSpan.Zero; + properties.Add(new TimingProperty(new TimingInfo(startTime, pending.EndTime ?? startTime + duration, duration))); + } + + if (pending.Output.Length > 0) + properties.Add(new StandardOutputProperty(pending.Output.ToString())); + + publish(node.ToTestNode(state, properties.ToArray())); + publishedResults.Add(node.Uid); + } + + private PendingResult GetOrCreatePendingResult(BenchmarkTestNode node) + { + if (!pendingResults.TryGetValue(node.Uid, out var pending)) + { + pending = new PendingResult(); + pendingResults[node.Uid] = pending; + } + + return pending; + } + + private static void AppendMeasurementSummary(StringBuilder output, BenchmarkReport report) + { + var resultRuns = report.GetResultRuns(); + if (resultRuns.Count == 0) + return; + + output.AppendLine(report.BenchmarkCase.DisplayInfo); + output.AppendLine($"Runtime = {report.GetRuntimeInfo()}; GC = {report.GetGcInfo()}"); + + var statistics = resultRuns.GetStatistics(); + var cultureInfo = CultureInfo.InvariantCulture; + var formatter = statistics.CreateNanosecondFormatter(cultureInfo); + + var histogram = HistogramBuilder.Adaptive.Build(statistics.Sample.Values); + output.AppendLine("-------------------- Histogram --------------------"); + output.AppendLine(histogram.ToString(formatter)); + output.AppendLine("---------------------------------------------------"); + output.AppendLine(statistics.ToString(cultureInfo, formatter, calcHistogram: false)); + } + + private sealed class PendingResult + { + public List ErrorMessages { get; } = []; + + public StringBuilder Output { get; } = new(); + + public DateTimeOffset? StartTime { get; set; } + + public DateTimeOffset? EndTime { get; set; } + + public TimeSpan? Duration { get; set; } + + public string? GetErrorMessage() => ErrorMessages.Count == 0 ? null : string.Join("\n", ErrorMessages); + } + } +} From 76fd7da7eb2df1991a7feab6f28dce7499fd26f8 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:15:44 +0100 Subject: [PATCH 011/110] Add BenchmarkTestFramework for test discovery/execution Introduce BenchmarkTestFramework to integrate BenchmarkDotNet with Microsoft.Testing.Platform, enabling benchmark discovery and execution. Implements session management, filtering, event processing, and output routing. Handles test node updates and cancellation, with support for experimental platform features. --- .../BenchmarkTestFramework.cs | 195 ++++++++++++++++++ 1 file changed, 195 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs new file mode 100644 index 0000000000..569bf063cb --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs @@ -0,0 +1,195 @@ +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Loggers; +using BenchmarkDotNet.Running; +using Microsoft.Testing.Platform.Capabilities.TestFramework; +using Microsoft.Testing.Platform.Extensions.Messages; +using Microsoft.Testing.Platform.Extensions.OutputDevice; +using Microsoft.Testing.Platform.Extensions.TestFramework; +using Microsoft.Testing.Platform.Requests; +using Microsoft.Testing.Platform.Services; +using System.Reflection; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// Discovers and executes the benchmarks of the running assembly through Microsoft.Testing.Platform. + /// + internal sealed class BenchmarkTestFramework : ITestFramework, IDataProducer, IOutputDeviceDataProducer + { + private readonly BenchmarkDotNetExtension extension = new(); + private readonly IServiceProvider serviceProvider; + private readonly Assembly assembly; + + public BenchmarkTestFramework(ITestFrameworkCapabilities capabilities, IServiceProvider serviceProvider, Assembly assembly) + { + Capabilities = capabilities; + this.serviceProvider = serviceProvider; + this.assembly = assembly; + } + + /// + public string Uid => extension.Uid; + + /// + public string Version => extension.Version; + + /// + public string DisplayName => extension.DisplayName; + + /// + public string Description => extension.Description; + + /// + public Type[] DataTypesProduced => [typeof(TestNodeUpdateMessage)]; + + /// + /// Gets the capabilities the framework was registered with. + /// + public ITestFrameworkCapabilities Capabilities { get; } + + /// + public Task IsEnabledAsync() => extension.IsEnabledAsync(); + + /// + public Task CreateTestSessionAsync(CreateTestSessionContext context) + => Task.FromResult(new CreateTestSessionResult { IsSuccess = true }); + + /// + public Task CloseTestSessionAsync(CloseTestSessionContext context) + => Task.FromResult(new CloseTestSessionResult { IsSuccess = true }); + + /// + public async Task ExecuteRequestAsync(ExecuteRequestContext context) + { + try + { + switch (context.Request) + { + case DiscoverTestExecutionRequest discoverRequest: + await DiscoverAsync(discoverRequest, context).ConfigureAwait(false); + break; + case RunTestExecutionRequest runRequest: + await RunAsync(runRequest, context).ConfigureAwait(false); + break; + } + } + finally + { + context.Complete(); + } + } + + private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRequestContext context) + { + foreach (var (_, node) in GetMatchingBenchmarks(request.Filter)) + { + context.CancellationToken.ThrowIfCancellationRequested(); + + var message = new TestNodeUpdateMessage( + request.Session.SessionUid, + node.ToTestNode(DiscoveredTestNodeStateProperty.CachedInstance)); + + await context.MessageBus.PublishAsync(this, message).ConfigureAwait(false); + } + } + + private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestContext context) + { + var matches = GetMatchingBenchmarks(request.Filter); + if (matches.Count == 0) + return; + + var nodes = matches.ToDictionary(match => match.Node.Uid, match => match.Node); + var sessionUid = request.Session.SessionUid; + var cancellationToken = context.CancellationToken; + + using var workQueue = new AsyncWorkQueue(); + + var eventProcessor = new BenchmarkEventProcessor(nodes, testNode => + { + var message = new TestNodeUpdateMessage(sessionUid, testNode); + workQueue.Enqueue(() => context.MessageBus.PublishAsync(this, message)); + }); + + // BenchmarkDotNet's own console output is replaced so that everything goes through the output device, + // which keeps it in the right place when the platform runs in server mode or inside an IDE. + var logger = new OutputDeviceLogger(serviceProvider.GetOutputDevice(), this, workQueue, cancellationToken); + + var runInfos = matches + .GroupBy(match => match.RunInfo) + .Select(group => new BenchmarkRunInfo( + group.Select(match => match.Node.BenchmarkCase).ToArray(), + group.Key.Type, + group.Key.Config + .AddEventProcessor(eventProcessor) + .AddLogger(logger) + .RemoveLoggersOfType() + .CreateImmutableConfig(), + group.Key.CompositeInProcessDiagnoser)) + .ToArray(); + + // BenchmarkDotNet blocks the calling thread for the whole run, so it gets a thread of its own and the + // queued messages are published from here as they are produced. + var runTask = Task.Run( + () => + { + try + { + BenchmarkRunner.Run(runInfos, cancellationToken); + } + finally + { + // Benchmarks that never reported a result still need one, unless the run was cancelled: the + // platform expects an OperationCanceledException in that case, and publishing results + // afterwards would contradict it. + if (!cancellationToken.IsCancellationRequested) + eventProcessor.PublishOutstandingResults(); + + logger.Flush(); + workQueue.Complete(); + } + }, + CancellationToken.None); + + await workQueue.DrainAsync().ConfigureAwait(false); + await runTask.ConfigureAwait(false); + } + + /// + /// Enumerates the benchmarks of the assembly and keeps the ones the request asked for. + /// + /// The filter of the request. + /// The matching benchmarks, paired with the run info they belong to. + private List<(BenchmarkRunInfo RunInfo, BenchmarkTestNode Node)> GetMatchingBenchmarks(ITestExecutionFilter filter) + { + var matches = new List<(BenchmarkRunInfo, BenchmarkTestNode)>(); + + foreach (var runInfo in BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly)) + { + // The job only earns a place in the display name when the benchmark actually runs under several jobs. + // This is computed before filtering so that a benchmark keeps the same name however it was selected. + var includeJobInName = runInfo.BenchmarksCases.Select(c => c.Job.DisplayInfo).Distinct().Count() > 1; + + foreach (var benchmarkCase in runInfo.BenchmarksCases) + { + var node = BenchmarkTestNode.Create(benchmarkCase, includeJobInName); + if (Matches(filter, node)) + matches.Add((runInfo, node)); + } + } + + return matches; + } + +#pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. + private static bool Matches(ITestExecutionFilter filter, BenchmarkTestNode node) => filter switch + { + TestNodeUidListFilter uidListFilter => uidListFilter.TestNodeUids.Any(uid => uid.Value == node.Uid), + TreeNodeFilter treeNodeFilter => treeNodeFilter.MatchesFilter(node.Path, node.GetFilterableProperties()), + + // NopFilter, and anything the platform adds later, means "everything". + _ => true + }; +#pragma warning restore TPEXP + } +} From aeba977c2212f6ed5e463b48156265113a9597b4 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:16:04 +0100 Subject: [PATCH 012/110] Add BenchmarkTestNode for test platform integration Introduced the internal sealed class BenchmarkTestNode to encapsulate immutable BenchmarkCase data for Microsoft.Testing.Platform integration. This includes stable UID generation, display name and path construction, property management, and support for test filtering and message bus conversion. --- .../BenchmarkTestNode.cs | 163 ++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs new file mode 100644 index 0000000000..200b22b775 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs @@ -0,0 +1,163 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Exporters; +using BenchmarkDotNet.Extensions; +using BenchmarkDotNet.Running; +using Microsoft.Testing.Platform.Extensions.Messages; +using System.Reflection; +using System.Text; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// The Microsoft.Testing.Platform view of a single . + /// + /// + /// A carries mutable state (the property bag holds the current outcome), so a fresh node is + /// created for every message published on the bus. This class holds the parts that never change. + /// + internal sealed class BenchmarkTestNode + { + private readonly IProperty[] staticProperties; + + private BenchmarkTestNode(BenchmarkCase benchmarkCase, string uid, string displayName, string path, IProperty[] staticProperties) + { + BenchmarkCase = benchmarkCase; + Uid = uid; + DisplayName = displayName; + Path = path; + this.staticProperties = staticProperties; + } + + /// + /// Gets the benchmark this node represents. + /// + public BenchmarkCase BenchmarkCase { get; } + + /// + /// Gets the stable identifier of the node. It has to be identical in the discovery and the execution phase, + /// which may happen in different processes. + /// + public string Uid { get; } + + /// + /// Gets the name shown by test runners. + /// + public string DisplayName { get; } + + /// + /// Gets the '/' separated path used by . + /// + public string Path { get; } + + /// + /// Gets the stable identifier of a benchmark case, without building the whole node. + /// + /// The benchmark case to identify. + /// The uid of the node representing the benchmark case. + /// + /// The job is always part of the uid, otherwise two cases of the same benchmark that only differ by job would + /// collide. The parameters are already part of the method name. + /// + public static string GetUid(BenchmarkCase benchmarkCase) + { + var fullClassName = benchmarkCase.Descriptor.Type.GetCorrectCSharpTypeName(prefixWithGlobal: false); + return $"{fullClassName}.{FullNameProvider.GetMethodName(benchmarkCase)} [{benchmarkCase.GetUnrandomizedJobDisplayInfo()}]"; + } + + /// + /// Creates the node for a benchmark case. + /// + /// The benchmark case to describe. + /// + /// Whether the display name should be suffixed with the job, which is only useful when the benchmark runs + /// under more than one job. + /// + /// The created node. + public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool includeJobInName) + { + var benchmarkMethod = benchmarkCase.Descriptor.WorkloadMethod; + var type = benchmarkCase.Descriptor.Type; + var fullClassName = type.GetCorrectCSharpTypeName(prefixWithGlobal: false); + var parametrizedMethodName = FullNameProvider.GetMethodName(benchmarkCase); + var jobDisplayInfo = benchmarkCase.GetUnrandomizedJobDisplayInfo(); + + // Unlike the uid, the job is only part of the display name when it actually adds information. + var uid = GetUid(benchmarkCase); + var displayName = $"{fullClassName}.{parametrizedMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); + + var properties = new List + { + new TestMethodIdentifierProperty( + type.Assembly.FullName, + type.Namespace ?? string.Empty, + type.GetCorrectCSharpTypeName(prefixWithGlobal: false, includeNamespace: false), + benchmarkMethod.Name, + benchmarkMethod.IsGenericMethodDefinition ? benchmarkMethod.GetGenericArguments().Length : 0, + benchmarkMethod.GetParameters().Select(p => p.ParameterType.FullName ?? p.ParameterType.Name).ToArray(), + benchmarkMethod.ReturnType.FullName ?? benchmarkMethod.ReturnType.Name), + }; + + var benchmarkAttribute = benchmarkMethod.ResolveAttribute(); + if (benchmarkAttribute?.SourceCodeFile != null) + { + // BenchmarkAttribute captures the line of the attribute itself, and the platform expects 0-based lines. + var line = Math.Max(0, benchmarkAttribute.SourceCodeLineNumber - 1); + var position = new LinePosition(line, 0); + properties.Add(new TestFileLocationProperty(benchmarkAttribute.SourceCodeFile, new LinePositionSpan(position, position))); + } + + foreach (var category in DefaultCategoryDiscoverer.Instance.GetCategories(benchmarkMethod)) + properties.Add(new TestMetadataProperty("Category", category)); + + var path = BuildPath(type.Assembly, type.Namespace, fullClassName, parametrizedMethodName, jobDisplayInfo); + + return new BenchmarkTestNode(benchmarkCase, uid, displayName, path, properties.ToArray()); + } + + /// + /// Creates a message-bus ready node in the given state. + /// + /// The state of the benchmark, e.g. discovered, passed or failed. + /// Any additional properties, such as timing or captured output. + /// The created test node. + public TestNode ToTestNode(TestNodeStateProperty state, params IProperty[] extraProperties) + { + var properties = new PropertyBag(staticProperties); + properties.Add(state); + foreach (var property in extraProperties) + properties.Add(property); + + return new TestNode + { + Uid = new TestNodeUid(Uid), + DisplayName = DisplayName, + Properties = properties + }; + } + + /// + /// Gets the properties a can match against, + /// which is what makes `--filter "/*/*/*/*[Category=Fast]"` work. + /// + /// The filterable properties. + public PropertyBag GetFilterableProperties() => new PropertyBag(staticProperties); + + private static string BuildPath(Assembly assembly, string? @namespace, string fullClassName, string methodName, string jobDisplayInfo) + { + // The convention followed by the other test frameworks is ////. + var className = @namespace == null || !fullClassName.StartsWith(@namespace + ".", StringComparison.Ordinal) + ? fullClassName + : fullClassName.Substring(@namespace.Length + 1); + + return new StringBuilder() + .Append('/').Append(Escape(assembly.GetName().Name)) + .Append('/').Append(Escape(@namespace ?? string.Empty)) + .Append('/').Append(Escape(className)) + .Append('/').Append(Escape($"{methodName} [{jobDisplayInfo}]")) + .ToString(); + } + + // Benchmark parameters are stringified user values, so they can contain the path separator. + private static string Escape(string segment) => segment.Replace("/", "\\/"); + } +} From d0808862a1274c6db5ea865602b144da47f2fe46 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:16:33 +0100 Subject: [PATCH 013/110] Add OutputDeviceLogger to forward logs to output device Introduced OutputDeviceLogger class implementing ILogger to forward BenchmarkDotNet logs to the platform output device. Handles log kinds, buffers lines, and asynchronously displays output to ensure build progress and results are visible in test run output. --- .../OutputDeviceLogger.cs | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs new file mode 100644 index 0000000000..fb17503219 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs @@ -0,0 +1,77 @@ +using BenchmarkDotNet.Loggers; +using Microsoft.Testing.Platform.Extensions.OutputDevice; +using Microsoft.Testing.Platform.OutputDevice; +using System.Text; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// Forwards the BenchmarkDotNet log to the platform output device, so that build progress and the result summary + /// show up in the test run output. + /// + internal sealed class OutputDeviceLogger : ILogger + { + private readonly IOutputDevice outputDevice; + private readonly IOutputDeviceDataProducer producer; + private readonly AsyncWorkQueue workQueue; + private readonly CancellationToken cancellationToken; + private readonly StringBuilder currentLine = new(); + private LogKind currentLineKind = LogKind.Default; + + public OutputDeviceLogger( + IOutputDevice outputDevice, + IOutputDeviceDataProducer producer, + AsyncWorkQueue workQueue, + CancellationToken cancellationToken) + { + this.outputDevice = outputDevice; + this.producer = producer; + this.workQueue = workQueue; + this.cancellationToken = cancellationToken; + } + + public string Id => nameof(OutputDeviceLogger); + + public int Priority => 0; + + public void Write(LogKind logKind, string text) + { + currentLine.Append(text); + + // Assume that if any part of the line is an error or a warning, the whole line is. + // The kind is reset when the line is flushed. + if (logKind == LogKind.Error || (logKind == LogKind.Warning && currentLineKind != LogKind.Error)) + currentLineKind = logKind; + } + + public void WriteLine() + { + var text = currentLine.ToString(); + var kind = currentLineKind; + + currentLine.Clear(); + currentLineKind = LogKind.Default; + + IOutputDeviceData data = kind switch + { + LogKind.Error => new ErrorMessageOutputDeviceData(text), + LogKind.Warning => new WarningMessageOutputDeviceData(text), + _ => new TextOutputDeviceData(text) + }; + + workQueue.Enqueue(() => outputDevice.DisplayAsync(producer, data, cancellationToken)); + } + + public void WriteLine(LogKind logKind, string text) + { + Write(logKind, text); + WriteLine(); + } + + public void Flush() + { + if (currentLine.Length > 0) + WriteLine(); + } + } +} From d155facb7801334ddd63f52a695c8cf9b12ef52a Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:16:52 +0100 Subject: [PATCH 014/110] Add extension methods to register BenchmarkDotNet as test framework Introduce TestApplicationBuilderExtensions with AddBenchmarkDotNet methods for integrating BenchmarkDotNet benchmarks into Microsoft.Testing.Platform. Includes overloads for entry assembly and specific assemblies, null checks, test framework registration, and tree node filter service support. --- .../TestApplicationBuilderExtensions.cs | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs new file mode 100644 index 0000000000..0f72cb90c6 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs @@ -0,0 +1,50 @@ +using Microsoft.Testing.Platform.Builder; +using Microsoft.Testing.Platform.Capabilities.TestFramework; +using Microsoft.Testing.Platform.Helpers; +using System.Reflection; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// Extensions for registering BenchmarkDotNet with a Microsoft.Testing.Platform application. + /// + public static class TestApplicationBuilderExtensions + { + /// + /// Registers BenchmarkDotNet as the test framework, so that the benchmarks of the entry assembly are exposed + /// as tests. + /// + /// The builder of the test application. + /// The same builder, so that calls can be chained. + public static ITestApplicationBuilder AddBenchmarkDotNet(this ITestApplicationBuilder builder) + => builder.AddBenchmarkDotNet( + Assembly.GetEntryAssembly() ?? throw new InvalidOperationException( + "There is no entry assembly to look for benchmarks in. Use the overload that takes an assembly.")); + + /// + /// Registers BenchmarkDotNet as the test framework, so that the benchmarks of the given assembly are exposed + /// as tests. + /// + /// The builder of the test application. + /// The assembly to look for benchmarks in. + /// The same builder, so that calls can be chained. + public static ITestApplicationBuilder AddBenchmarkDotNet(this ITestApplicationBuilder builder, Assembly assembly) + { + if (builder == null) + throw new ArgumentNullException(nameof(builder)); + if (assembly == null) + throw new ArgumentNullException(nameof(assembly)); + + builder.RegisterTestFramework( + _ => new TestFrameworkCapabilities(), + (capabilities, serviceProvider) => new BenchmarkTestFramework(capabilities, serviceProvider, assembly)); + + // Opts into the tree node filter, which is what backs `--filter "/*/*/MyBenchmarks/*"`. +#pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. + builder.AddTreeNodeFilterService(new BenchmarkDotNetExtension()); +#pragma warning restore TPEXP + + return builder; + } + } +} From 41e20991394917adfe79f392e3430c4432ef65c2 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:17:07 +0100 Subject: [PATCH 015/110] Add TestingPlatformBuilderHook for BenchmarkDotNet integration Introduced a static TestingPlatformBuilderHook class in the BenchmarkDotNet.TestAdapter.TestingPlatform namespace. This class provides an AddExtensions method to register BenchmarkDotNet with the test application builder, intended for use by generated code and hidden from IntelliSense. --- .../TestingPlatformBuilderHook.cs | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs new file mode 100644 index 0000000000..47aa1705a7 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs @@ -0,0 +1,26 @@ +using Microsoft.Testing.Platform.Builder; +using System.ComponentModel; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// The hook Microsoft.Testing.Platform.MSBuild calls from the entry point it generates for the benchmark project. + /// + /// + /// This is wired up by the TestingPlatformBuilderHook item in BenchmarkDotNet.TestAdapter.TestingPlatform.props. + /// It is public because the generated code lives in the benchmark assembly, but it is not meant to be called + /// directly; use + /// instead when writing an entry point by hand. + /// + [EditorBrowsable(EditorBrowsableState.Never)] + public static class TestingPlatformBuilderHook + { + /// + /// Registers BenchmarkDotNet with the test application being built. + /// + /// The builder of the test application. + /// The command line arguments of the process. Unused. + public static void AddExtensions(ITestApplicationBuilder builder, string[] arguments) + => builder.AddBenchmarkDotNet(); + } +} From e7fb11938622c092e320624492b5b36d2197a1d3 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:17:53 +0100 Subject: [PATCH 016/110] Add test runner config to global.json Added a "test" section to global.json to specify "Microsoft.Testing.Platform" as the test runner. This configures the project to use the designated testing platform. --- .../global.json | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/global.json diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/global.json b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/global.json new file mode 100644 index 0000000000..3140116df3 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/global.json @@ -0,0 +1,5 @@ +{ + "test": { + "runner": "Microsoft.Testing.Platform" + } +} From 9838f5788fc1cb7be43e0186053c8c8a743a1e70 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:18:18 +0100 Subject: [PATCH 017/110] Add IntegrationTests.TestingPlatform project for net10.0 Introduce BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj targeting net10.0 as an executable. The project includes assembly metadata, references BenchmarkDotNet.TestAdapter.TestingPlatform, manually imports its build props, and uses shared common.props and common.targets for build configuration. --- ...et.IntegrationTests.TestingPlatform.csproj | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj new file mode 100644 index 0000000000..b89575026a --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj @@ -0,0 +1,19 @@ + + + + net10.0 + Exe + BenchmarkDotNet.IntegrationTests.TestingPlatform + BenchmarkDotNet.IntegrationTests.TestingPlatform + BenchmarkDotNet.IntegrationTests.TestingPlatform + + + + + + + + + + + From b84f3b86781266e33b6c8b8dc20f387b59e5b9d1 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:19:08 +0100 Subject: [PATCH 018/110] Add SampleBenchmarks for fast in-process BDN testing Introduced SampleBenchmarks class in BenchmarkDotNet.IntegrationTests.TestingPlatform. Defines Add and Multiply benchmarks with parameterized Size, categorized as "Fast" and "Slow". Uses a custom FastConfig to run benchmarks in-process with a single dry iteration for quick end-to-end testing. --- .../SampleBenchmarks.cs | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SampleBenchmarks.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SampleBenchmarks.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SampleBenchmarks.cs new file mode 100644 index 0000000000..27bce02ef5 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SampleBenchmarks.cs @@ -0,0 +1,31 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// Benchmarks used to exercise the Microsoft.Testing.Platform adapter end to end. They run in-process with a + /// single iteration so that a full run stays fast. + /// + [Config(typeof(FastConfig))] + public class SampleBenchmarks + { + [Params(1, 2)] + public int Size { get; set; } + + [Benchmark] + [BenchmarkCategory("Fast")] + public int Add() => Size + Size; + + [Benchmark] + [BenchmarkCategory("Slow")] + public int Multiply() => Size * Size; + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} From d5d3c0b066247c4810f877a5468ceaf68e46278b Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Fri, 7 Aug 2026 16:19:20 +0100 Subject: [PATCH 019/110] Add TestingPlatform projects and update test build configs Added BenchmarkDotNet.TestAdapter.TestingPlatform and BenchmarkDotNet.IntegrationTests.TestingPlatform projects to the solution. Updated MonoBenchmarks and SharedDiagnosers integration test projects to only build for the solution in Debug configuration. --- BenchmarkDotNet.slnx | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index 2d5a3cf0dd..9b62a63f29 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -20,6 +20,7 @@ + @@ -39,9 +40,14 @@ - - + + + + + + + From f420e45161888411469e72303328aad06fafef35 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Wed, 12 Aug 2026 18:33:56 +0100 Subject: [PATCH 020/110] Update slnx to exclude two projects from Debug build Explicitly set BenchmarkDotNet.TestAdapter.TestingPlatform and BenchmarkDotNet.IntegrationTests.TestingPlatform to not build in the Debug configuration by adding in BenchmarkDotNet.slnx. No other changes made. --- BenchmarkDotNet.slnx | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index 9b62a63f29..119a22f6ca 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -20,7 +20,9 @@ - + + + @@ -47,7 +49,9 @@ - + + + From 61ce3f14609829aad418aad65801374ccafa2a5f Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:12:00 +0100 Subject: [PATCH 021/110] Update docs, namespaces, and add GetBenchmarkUid method - Correct NuGet package and namespace in documentation - Add GetBenchmarkUid for stable benchmark identification - Change namespace in BenchmarkCaseIdentityExtensions - Update InternalsVisibleTo for TestingPlatform assembly --- docs/articles/features/testingplatform.md | 6 +++--- .../Exporters/FullNameProvider.cs | 18 ++++++++++++++++++ .../BenchmarkCaseIdentityExtensions.cs | 6 +++--- src/BenchmarkDotNet/Properties/AssemblyInfo.cs | 2 +- 4 files changed, 25 insertions(+), 7 deletions(-) rename src/{BenchmarkDotNet.TestAdapter => BenchmarkDotNet/Extensions}/BenchmarkCaseIdentityExtensions.cs (88%) diff --git a/docs/articles/features/testingplatform.md b/docs/articles/features/testingplatform.md index 85fffeeb0b..a54ccab78e 100644 --- a/docs/articles/features/testingplatform.md +++ b/docs/articles/features/testingplatform.md @@ -53,7 +53,7 @@ The practical consequences of the MTP model are: ```xml - + ``` @@ -72,7 +72,7 @@ The practical consequences of the MTP model are: - + @@ -149,7 +149,7 @@ If you want to keep your own entry point, turn off the generated one and registe ``` ```csharp -using BenchmarkDotNet.TestAdapter.TestingPlatform; +using BenchmarkDotNet.TestingPlatform; using Microsoft.Testing.Platform.Builder; public static class Program diff --git a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs index 19555908d2..922d3590c4 100644 --- a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs +++ b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs @@ -63,6 +63,24 @@ public static string GetBenchmarkName(BenchmarkCase benchmarkCase) return name.ToString(); } + /// + /// Gets an identifier of a benchmark case that stays the same across processes, which is what lets a benchmark + /// discovered in one process be selected for execution in another (for example by a test adapter). + /// + /// The benchmark case to identify. + /// The unique identifier of the benchmark case. + /// + /// The job is always part of the uid, otherwise two cases of the same benchmark that only differ by job would + /// collide. The parameters are already part of the method name. + /// + [PublicAPI] + public static string GetBenchmarkUid(BenchmarkCase benchmarkCase) + { + var fullClassName = benchmarkCase.Descriptor.Type.GetCorrectCSharpTypeName(prefixWithGlobal: false); + return $"{fullClassName}.{GetMethodName(benchmarkCase)} [{benchmarkCase.GetUnrandomizedJobDisplayInfo()}]"; + } + + private static string GetNestedTypes(Type type) { string nestedTypes = ""; diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs b/src/BenchmarkDotNet/Extensions/BenchmarkCaseIdentityExtensions.cs similarity index 88% rename from src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs rename to src/BenchmarkDotNet/Extensions/BenchmarkCaseIdentityExtensions.cs index 78cb64f5ce..948ea1d7fe 100644 --- a/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseIdentityExtensions.cs +++ b/src/BenchmarkDotNet/Extensions/BenchmarkCaseIdentityExtensions.cs @@ -1,11 +1,10 @@ using BenchmarkDotNet.Characteristics; using BenchmarkDotNet.Running; -namespace BenchmarkDotNet.TestAdapter +namespace BenchmarkDotNet.Extensions { /// - /// Helpers for deriving stable identities for a BenchmarkCase. Shared by the VSTest and the - /// Microsoft.Testing.Platform adapters, because both need identities that survive across processes. + /// Helpers for deriving stable identities for a BenchmarkCase. /// internal static class BenchmarkCaseIdentityExtensions { @@ -30,3 +29,4 @@ internal static string GetUnrandomizedJobDisplayInfo(this BenchmarkCase benchmar } } } + diff --git a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs index 5a5ff709d3..7bd16f9307 100644 --- a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs +++ b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs @@ -15,4 +15,4 @@ [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning.MultipleFrameworks,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] -[assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter.TestingPlatform,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] +[assembly: InternalsVisibleTo("BenchmarkDotNet.TestingPlatform,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] From 4a27390fdb7aa93db4710dd3e112dbb7d0caf3b6 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:12:32 +0100 Subject: [PATCH 022/110] Remove BenchmarkDotNet.TestAdapter.TestingPlatform project Deleted all source, project, and props files from BenchmarkDotNet.TestAdapter.TestingPlatform. This removes all implementation and integration for running benchmarks as tests via Microsoft.Testing.Platform, including test discovery, execution, and result processing logic. --- .../AsyncWorkQueue.cs | 61 ------ ...kDotNet.TestAdapter.TestingPlatform.csproj | 46 ----- .../BenchmarkDotNetExtension.cs | 34 --- .../BenchmarkEventProcessor.cs | 191 ----------------- .../BenchmarkTestFramework.cs | 195 ------------------ .../BenchmarkTestNode.cs | 163 --------------- .../OutputDeviceLogger.cs | 77 ------- .../TestApplicationBuilderExtensions.cs | 50 ----- .../TestingPlatformBuilderHook.cs | 26 --- ...rkDotNet.TestAdapter.TestingPlatform.props | 31 --- 10 files changed, 874 deletions(-) delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs deleted file mode 100644 index 128eb51b0c..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/AsyncWorkQueue.cs +++ /dev/null @@ -1,61 +0,0 @@ -using System.Collections.Concurrent; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// An ordered queue of asynchronous work items that are produced synchronously and consumed asynchronously. - /// - /// - /// BenchmarkDotNet reports its progress through synchronous callbacks (EventProcessor and ILogger) while the - /// platform message bus and output device are asynchronous. Blocking on those from inside a callback risks - /// deadlocking against the synchronization context BenchmarkDotNet installs while it runs, so the callbacks - /// enqueue instead and the caller of DrainAsync does the awaiting. - /// - internal sealed class AsyncWorkQueue : IDisposable - { - private readonly ConcurrentQueue> queue = new(); - private readonly SemaphoreSlim available = new(0); - private volatile bool completed; - - /// - /// Queues a work item. Safe to call from any thread. - /// - /// The work to perform. - public void Enqueue(Func work) - { - queue.Enqueue(work); - available.Release(); - } - - /// - /// Signals that no more work items will be enqueued, which lets DrainAsync return once the already queued - /// items have been processed. - /// - public void Complete() - { - completed = true; - available.Release(); - } - - /// - /// Processes queued work items in order until has been called and the queue is empty. - /// - /// A task that completes when the queue has been drained. - public async Task DrainAsync() - { - while (true) - { - await available.WaitAsync().ConfigureAwait(false); - - // Draining everything on each wake-up means a lost or surplus permit can never strand a work item. - while (queue.TryDequeue(out var work)) - await work().ConfigureAwait(false); - - if (completed && queue.IsEmpty) - return; - } - } - - public void Dispose() => available.Dispose(); - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj deleted file mode 100644 index c3dd5c4cb3..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNet.TestAdapter.TestingPlatform.csproj +++ /dev/null @@ -1,46 +0,0 @@ - - - - netstandard2.0 - BenchmarkDotNet.TestAdapter.TestingPlatform - BenchmarkDotNet.TestAdapter.TestingPlatform - BenchmarkDotNet.TestAdapter.TestingPlatform - Runs BenchmarkDotNet benchmarks as tests through Microsoft.Testing.Platform - README.md - True - BenchmarkDotNet.TestAdapter.TestingPlatform - - - false - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs deleted file mode 100644 index 789fbc611a..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkDotNetExtension.cs +++ /dev/null @@ -1,34 +0,0 @@ -using Microsoft.Testing.Platform.Extensions; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// Identifies BenchmarkDotNet to Microsoft.Testing.Platform. - /// - /// - /// Several platform services, such as the tree node filter behind --filter, are registered on behalf of an - /// extension rather than of the test framework itself, so the identity lives in its own type. - /// - internal sealed class BenchmarkDotNetExtension : IExtension - { - /// - /// The uid shared by every extension this package registers. - /// - public const string ExtensionUid = "BenchmarkDotNet.TestAdapter.TestingPlatform"; - - /// - public string Uid => ExtensionUid; - - /// - public string Version => typeof(BenchmarkDotNetExtension).Assembly.GetName().Version?.ToString() ?? "0.0.0"; - - /// - public string DisplayName => "BenchmarkDotNet"; - - /// - public string Description => "Runs BenchmarkDotNet benchmarks as tests."; - - /// - public Task IsEnabledAsync() => Task.FromResult(true); - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs deleted file mode 100644 index b48e5cd6e6..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkEventProcessor.cs +++ /dev/null @@ -1,191 +0,0 @@ -using BenchmarkDotNet.EventProcessors; -using BenchmarkDotNet.Extensions; -using BenchmarkDotNet.Reports; -using BenchmarkDotNet.Running; -using BenchmarkDotNet.Toolchains.Results; -using BenchmarkDotNet.Validators; -using Microsoft.Testing.Platform.Extensions.Messages; -using Perfolizer.Mathematics.Histograms; -using System.Diagnostics; -using System.Globalization; -using System.Text; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// Translates the BenchmarkDotNet run into the stream of test node updates the platform expects. - /// - internal sealed class BenchmarkEventProcessor : EventProcessor - { - private readonly IReadOnlyDictionary nodes; - private readonly Action publish; - private readonly Stopwatch runTimerStopwatch = new(); - private readonly Dictionary pendingResults = []; - private readonly HashSet publishedResults = []; - - public BenchmarkEventProcessor(IReadOnlyDictionary nodes, Action publish) - { - this.nodes = nodes; - this.publish = publish; - } - - public override void OnValidationError(ValidationError validationError) - { - // If the error is not linked to a benchmark case, then set the error on all benchmarks. - var affected = validationError.BenchmarkCase == null - ? nodes.Values - : [nodes[BenchmarkTestNode.GetUid(validationError.BenchmarkCase)]]; - - foreach (var node in affected) - { - var pending = GetOrCreatePendingResult(node); - - if (validationError.IsCritical) - { - // The result is not published yet, in case there are more validation errors to append. - pending.ErrorMessages.Add(validationError.Message); - } - else - { - pending.Output.AppendLine($"WARNING: {validationError.Message}"); - } - } - } - - public override void OnBuildComplete(BuildPartition buildPartition, BuildResult buildResult) - { - // Only build failures need to be reported, successful builds are followed by a run. - if (buildResult.IsBuildSuccess) - return; - - foreach (var benchmarkBuildInfo in buildPartition.Benchmarks) - { - var node = nodes[BenchmarkTestNode.GetUid(benchmarkBuildInfo.BenchmarkCase)]; - var pending = GetOrCreatePendingResult(node); - - if (buildResult.GenerateException != null) - pending.ErrorMessages.Add($"// Generate Exception: {buildResult.GenerateException.Message}"); - else if (buildResult.TryToExplainFailureReason(out string? reason)) - pending.ErrorMessages.Add($"// Build Error: {reason}"); - else if (buildResult.ErrorMessage != null) - pending.ErrorMessages.Add($"// Build Error: {buildResult.ErrorMessage}"); - - // A benchmark that failed to build will never run, so the result can be published immediately. - publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); - PublishResult(node, pending, new FailedTestNodeStateProperty(pending.GetErrorMessage() ?? "The benchmark failed to build.")); - } - } - - public override void OnStartRunBenchmark(BenchmarkCase benchmarkCase) - { - var node = nodes[BenchmarkTestNode.GetUid(benchmarkCase)]; - var pending = GetOrCreatePendingResult(node); - pending.StartTime = DateTimeOffset.UtcNow; - - publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); - runTimerStopwatch.Restart(); - } - - public override void OnEndRunBenchmark(BenchmarkCase benchmarkCase, BenchmarkReport report) - { - var node = nodes[BenchmarkTestNode.GetUid(benchmarkCase)]; - var pending = GetOrCreatePendingResult(node); - pending.Duration = runTimerStopwatch.Elapsed; - pending.EndTime = DateTimeOffset.UtcNow; - - AppendMeasurementSummary(pending.Output, report); - - TestNodeStateProperty state = report.Success && pending.ErrorMessages.Count == 0 - ? PassedTestNodeStateProperty.CachedInstance - : new FailedTestNodeStateProperty(pending.GetErrorMessage() ?? "The benchmark did not complete successfully."); - - PublishResult(node, pending, state); - } - - /// - /// Publishes a result for every benchmark that was scheduled to run but never reported one, which happens when - /// a critical validation error stopped it or when BenchmarkDotNet never reached it. - /// - public void PublishOutstandingResults() - { - foreach (var node in nodes.Values) - { - if (publishedResults.Contains(node.Uid)) - continue; - - var pending = GetOrCreatePendingResult(node); - var errorMessage = pending.GetErrorMessage(); - - TestNodeStateProperty state = errorMessage != null - ? new FailedTestNodeStateProperty(errorMessage) - : SkippedTestNodeStateProperty.CachedInstance; - - publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); - PublishResult(node, pending, state); - } - } - - private void PublishResult(BenchmarkTestNode node, PendingResult pending, TestNodeStateProperty state) - { - var properties = new List(); - - if (pending.StartTime is { } startTime) - { - var duration = pending.Duration ?? TimeSpan.Zero; - properties.Add(new TimingProperty(new TimingInfo(startTime, pending.EndTime ?? startTime + duration, duration))); - } - - if (pending.Output.Length > 0) - properties.Add(new StandardOutputProperty(pending.Output.ToString())); - - publish(node.ToTestNode(state, properties.ToArray())); - publishedResults.Add(node.Uid); - } - - private PendingResult GetOrCreatePendingResult(BenchmarkTestNode node) - { - if (!pendingResults.TryGetValue(node.Uid, out var pending)) - { - pending = new PendingResult(); - pendingResults[node.Uid] = pending; - } - - return pending; - } - - private static void AppendMeasurementSummary(StringBuilder output, BenchmarkReport report) - { - var resultRuns = report.GetResultRuns(); - if (resultRuns.Count == 0) - return; - - output.AppendLine(report.BenchmarkCase.DisplayInfo); - output.AppendLine($"Runtime = {report.GetRuntimeInfo()}; GC = {report.GetGcInfo()}"); - - var statistics = resultRuns.GetStatistics(); - var cultureInfo = CultureInfo.InvariantCulture; - var formatter = statistics.CreateNanosecondFormatter(cultureInfo); - - var histogram = HistogramBuilder.Adaptive.Build(statistics.Sample.Values); - output.AppendLine("-------------------- Histogram --------------------"); - output.AppendLine(histogram.ToString(formatter)); - output.AppendLine("---------------------------------------------------"); - output.AppendLine(statistics.ToString(cultureInfo, formatter, calcHistogram: false)); - } - - private sealed class PendingResult - { - public List ErrorMessages { get; } = []; - - public StringBuilder Output { get; } = new(); - - public DateTimeOffset? StartTime { get; set; } - - public DateTimeOffset? EndTime { get; set; } - - public TimeSpan? Duration { get; set; } - - public string? GetErrorMessage() => ErrorMessages.Count == 0 ? null : string.Join("\n", ErrorMessages); - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs deleted file mode 100644 index 569bf063cb..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestFramework.cs +++ /dev/null @@ -1,195 +0,0 @@ -using BenchmarkDotNet.Configs; -using BenchmarkDotNet.Loggers; -using BenchmarkDotNet.Running; -using Microsoft.Testing.Platform.Capabilities.TestFramework; -using Microsoft.Testing.Platform.Extensions.Messages; -using Microsoft.Testing.Platform.Extensions.OutputDevice; -using Microsoft.Testing.Platform.Extensions.TestFramework; -using Microsoft.Testing.Platform.Requests; -using Microsoft.Testing.Platform.Services; -using System.Reflection; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// Discovers and executes the benchmarks of the running assembly through Microsoft.Testing.Platform. - /// - internal sealed class BenchmarkTestFramework : ITestFramework, IDataProducer, IOutputDeviceDataProducer - { - private readonly BenchmarkDotNetExtension extension = new(); - private readonly IServiceProvider serviceProvider; - private readonly Assembly assembly; - - public BenchmarkTestFramework(ITestFrameworkCapabilities capabilities, IServiceProvider serviceProvider, Assembly assembly) - { - Capabilities = capabilities; - this.serviceProvider = serviceProvider; - this.assembly = assembly; - } - - /// - public string Uid => extension.Uid; - - /// - public string Version => extension.Version; - - /// - public string DisplayName => extension.DisplayName; - - /// - public string Description => extension.Description; - - /// - public Type[] DataTypesProduced => [typeof(TestNodeUpdateMessage)]; - - /// - /// Gets the capabilities the framework was registered with. - /// - public ITestFrameworkCapabilities Capabilities { get; } - - /// - public Task IsEnabledAsync() => extension.IsEnabledAsync(); - - /// - public Task CreateTestSessionAsync(CreateTestSessionContext context) - => Task.FromResult(new CreateTestSessionResult { IsSuccess = true }); - - /// - public Task CloseTestSessionAsync(CloseTestSessionContext context) - => Task.FromResult(new CloseTestSessionResult { IsSuccess = true }); - - /// - public async Task ExecuteRequestAsync(ExecuteRequestContext context) - { - try - { - switch (context.Request) - { - case DiscoverTestExecutionRequest discoverRequest: - await DiscoverAsync(discoverRequest, context).ConfigureAwait(false); - break; - case RunTestExecutionRequest runRequest: - await RunAsync(runRequest, context).ConfigureAwait(false); - break; - } - } - finally - { - context.Complete(); - } - } - - private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRequestContext context) - { - foreach (var (_, node) in GetMatchingBenchmarks(request.Filter)) - { - context.CancellationToken.ThrowIfCancellationRequested(); - - var message = new TestNodeUpdateMessage( - request.Session.SessionUid, - node.ToTestNode(DiscoveredTestNodeStateProperty.CachedInstance)); - - await context.MessageBus.PublishAsync(this, message).ConfigureAwait(false); - } - } - - private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestContext context) - { - var matches = GetMatchingBenchmarks(request.Filter); - if (matches.Count == 0) - return; - - var nodes = matches.ToDictionary(match => match.Node.Uid, match => match.Node); - var sessionUid = request.Session.SessionUid; - var cancellationToken = context.CancellationToken; - - using var workQueue = new AsyncWorkQueue(); - - var eventProcessor = new BenchmarkEventProcessor(nodes, testNode => - { - var message = new TestNodeUpdateMessage(sessionUid, testNode); - workQueue.Enqueue(() => context.MessageBus.PublishAsync(this, message)); - }); - - // BenchmarkDotNet's own console output is replaced so that everything goes through the output device, - // which keeps it in the right place when the platform runs in server mode or inside an IDE. - var logger = new OutputDeviceLogger(serviceProvider.GetOutputDevice(), this, workQueue, cancellationToken); - - var runInfos = matches - .GroupBy(match => match.RunInfo) - .Select(group => new BenchmarkRunInfo( - group.Select(match => match.Node.BenchmarkCase).ToArray(), - group.Key.Type, - group.Key.Config - .AddEventProcessor(eventProcessor) - .AddLogger(logger) - .RemoveLoggersOfType() - .CreateImmutableConfig(), - group.Key.CompositeInProcessDiagnoser)) - .ToArray(); - - // BenchmarkDotNet blocks the calling thread for the whole run, so it gets a thread of its own and the - // queued messages are published from here as they are produced. - var runTask = Task.Run( - () => - { - try - { - BenchmarkRunner.Run(runInfos, cancellationToken); - } - finally - { - // Benchmarks that never reported a result still need one, unless the run was cancelled: the - // platform expects an OperationCanceledException in that case, and publishing results - // afterwards would contradict it. - if (!cancellationToken.IsCancellationRequested) - eventProcessor.PublishOutstandingResults(); - - logger.Flush(); - workQueue.Complete(); - } - }, - CancellationToken.None); - - await workQueue.DrainAsync().ConfigureAwait(false); - await runTask.ConfigureAwait(false); - } - - /// - /// Enumerates the benchmarks of the assembly and keeps the ones the request asked for. - /// - /// The filter of the request. - /// The matching benchmarks, paired with the run info they belong to. - private List<(BenchmarkRunInfo RunInfo, BenchmarkTestNode Node)> GetMatchingBenchmarks(ITestExecutionFilter filter) - { - var matches = new List<(BenchmarkRunInfo, BenchmarkTestNode)>(); - - foreach (var runInfo in BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly)) - { - // The job only earns a place in the display name when the benchmark actually runs under several jobs. - // This is computed before filtering so that a benchmark keeps the same name however it was selected. - var includeJobInName = runInfo.BenchmarksCases.Select(c => c.Job.DisplayInfo).Distinct().Count() > 1; - - foreach (var benchmarkCase in runInfo.BenchmarksCases) - { - var node = BenchmarkTestNode.Create(benchmarkCase, includeJobInName); - if (Matches(filter, node)) - matches.Add((runInfo, node)); - } - } - - return matches; - } - -#pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. - private static bool Matches(ITestExecutionFilter filter, BenchmarkTestNode node) => filter switch - { - TestNodeUidListFilter uidListFilter => uidListFilter.TestNodeUids.Any(uid => uid.Value == node.Uid), - TreeNodeFilter treeNodeFilter => treeNodeFilter.MatchesFilter(node.Path, node.GetFilterableProperties()), - - // NopFilter, and anything the platform adds later, means "everything". - _ => true - }; -#pragma warning restore TPEXP - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs deleted file mode 100644 index 200b22b775..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/BenchmarkTestNode.cs +++ /dev/null @@ -1,163 +0,0 @@ -using BenchmarkDotNet.Attributes; -using BenchmarkDotNet.Exporters; -using BenchmarkDotNet.Extensions; -using BenchmarkDotNet.Running; -using Microsoft.Testing.Platform.Extensions.Messages; -using System.Reflection; -using System.Text; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// The Microsoft.Testing.Platform view of a single . - /// - /// - /// A carries mutable state (the property bag holds the current outcome), so a fresh node is - /// created for every message published on the bus. This class holds the parts that never change. - /// - internal sealed class BenchmarkTestNode - { - private readonly IProperty[] staticProperties; - - private BenchmarkTestNode(BenchmarkCase benchmarkCase, string uid, string displayName, string path, IProperty[] staticProperties) - { - BenchmarkCase = benchmarkCase; - Uid = uid; - DisplayName = displayName; - Path = path; - this.staticProperties = staticProperties; - } - - /// - /// Gets the benchmark this node represents. - /// - public BenchmarkCase BenchmarkCase { get; } - - /// - /// Gets the stable identifier of the node. It has to be identical in the discovery and the execution phase, - /// which may happen in different processes. - /// - public string Uid { get; } - - /// - /// Gets the name shown by test runners. - /// - public string DisplayName { get; } - - /// - /// Gets the '/' separated path used by . - /// - public string Path { get; } - - /// - /// Gets the stable identifier of a benchmark case, without building the whole node. - /// - /// The benchmark case to identify. - /// The uid of the node representing the benchmark case. - /// - /// The job is always part of the uid, otherwise two cases of the same benchmark that only differ by job would - /// collide. The parameters are already part of the method name. - /// - public static string GetUid(BenchmarkCase benchmarkCase) - { - var fullClassName = benchmarkCase.Descriptor.Type.GetCorrectCSharpTypeName(prefixWithGlobal: false); - return $"{fullClassName}.{FullNameProvider.GetMethodName(benchmarkCase)} [{benchmarkCase.GetUnrandomizedJobDisplayInfo()}]"; - } - - /// - /// Creates the node for a benchmark case. - /// - /// The benchmark case to describe. - /// - /// Whether the display name should be suffixed with the job, which is only useful when the benchmark runs - /// under more than one job. - /// - /// The created node. - public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool includeJobInName) - { - var benchmarkMethod = benchmarkCase.Descriptor.WorkloadMethod; - var type = benchmarkCase.Descriptor.Type; - var fullClassName = type.GetCorrectCSharpTypeName(prefixWithGlobal: false); - var parametrizedMethodName = FullNameProvider.GetMethodName(benchmarkCase); - var jobDisplayInfo = benchmarkCase.GetUnrandomizedJobDisplayInfo(); - - // Unlike the uid, the job is only part of the display name when it actually adds information. - var uid = GetUid(benchmarkCase); - var displayName = $"{fullClassName}.{parametrizedMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); - - var properties = new List - { - new TestMethodIdentifierProperty( - type.Assembly.FullName, - type.Namespace ?? string.Empty, - type.GetCorrectCSharpTypeName(prefixWithGlobal: false, includeNamespace: false), - benchmarkMethod.Name, - benchmarkMethod.IsGenericMethodDefinition ? benchmarkMethod.GetGenericArguments().Length : 0, - benchmarkMethod.GetParameters().Select(p => p.ParameterType.FullName ?? p.ParameterType.Name).ToArray(), - benchmarkMethod.ReturnType.FullName ?? benchmarkMethod.ReturnType.Name), - }; - - var benchmarkAttribute = benchmarkMethod.ResolveAttribute(); - if (benchmarkAttribute?.SourceCodeFile != null) - { - // BenchmarkAttribute captures the line of the attribute itself, and the platform expects 0-based lines. - var line = Math.Max(0, benchmarkAttribute.SourceCodeLineNumber - 1); - var position = new LinePosition(line, 0); - properties.Add(new TestFileLocationProperty(benchmarkAttribute.SourceCodeFile, new LinePositionSpan(position, position))); - } - - foreach (var category in DefaultCategoryDiscoverer.Instance.GetCategories(benchmarkMethod)) - properties.Add(new TestMetadataProperty("Category", category)); - - var path = BuildPath(type.Assembly, type.Namespace, fullClassName, parametrizedMethodName, jobDisplayInfo); - - return new BenchmarkTestNode(benchmarkCase, uid, displayName, path, properties.ToArray()); - } - - /// - /// Creates a message-bus ready node in the given state. - /// - /// The state of the benchmark, e.g. discovered, passed or failed. - /// Any additional properties, such as timing or captured output. - /// The created test node. - public TestNode ToTestNode(TestNodeStateProperty state, params IProperty[] extraProperties) - { - var properties = new PropertyBag(staticProperties); - properties.Add(state); - foreach (var property in extraProperties) - properties.Add(property); - - return new TestNode - { - Uid = new TestNodeUid(Uid), - DisplayName = DisplayName, - Properties = properties - }; - } - - /// - /// Gets the properties a can match against, - /// which is what makes `--filter "/*/*/*/*[Category=Fast]"` work. - /// - /// The filterable properties. - public PropertyBag GetFilterableProperties() => new PropertyBag(staticProperties); - - private static string BuildPath(Assembly assembly, string? @namespace, string fullClassName, string methodName, string jobDisplayInfo) - { - // The convention followed by the other test frameworks is ////. - var className = @namespace == null || !fullClassName.StartsWith(@namespace + ".", StringComparison.Ordinal) - ? fullClassName - : fullClassName.Substring(@namespace.Length + 1); - - return new StringBuilder() - .Append('/').Append(Escape(assembly.GetName().Name)) - .Append('/').Append(Escape(@namespace ?? string.Empty)) - .Append('/').Append(Escape(className)) - .Append('/').Append(Escape($"{methodName} [{jobDisplayInfo}]")) - .ToString(); - } - - // Benchmark parameters are stringified user values, so they can contain the path separator. - private static string Escape(string segment) => segment.Replace("/", "\\/"); - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs deleted file mode 100644 index fb17503219..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/OutputDeviceLogger.cs +++ /dev/null @@ -1,77 +0,0 @@ -using BenchmarkDotNet.Loggers; -using Microsoft.Testing.Platform.Extensions.OutputDevice; -using Microsoft.Testing.Platform.OutputDevice; -using System.Text; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// Forwards the BenchmarkDotNet log to the platform output device, so that build progress and the result summary - /// show up in the test run output. - /// - internal sealed class OutputDeviceLogger : ILogger - { - private readonly IOutputDevice outputDevice; - private readonly IOutputDeviceDataProducer producer; - private readonly AsyncWorkQueue workQueue; - private readonly CancellationToken cancellationToken; - private readonly StringBuilder currentLine = new(); - private LogKind currentLineKind = LogKind.Default; - - public OutputDeviceLogger( - IOutputDevice outputDevice, - IOutputDeviceDataProducer producer, - AsyncWorkQueue workQueue, - CancellationToken cancellationToken) - { - this.outputDevice = outputDevice; - this.producer = producer; - this.workQueue = workQueue; - this.cancellationToken = cancellationToken; - } - - public string Id => nameof(OutputDeviceLogger); - - public int Priority => 0; - - public void Write(LogKind logKind, string text) - { - currentLine.Append(text); - - // Assume that if any part of the line is an error or a warning, the whole line is. - // The kind is reset when the line is flushed. - if (logKind == LogKind.Error || (logKind == LogKind.Warning && currentLineKind != LogKind.Error)) - currentLineKind = logKind; - } - - public void WriteLine() - { - var text = currentLine.ToString(); - var kind = currentLineKind; - - currentLine.Clear(); - currentLineKind = LogKind.Default; - - IOutputDeviceData data = kind switch - { - LogKind.Error => new ErrorMessageOutputDeviceData(text), - LogKind.Warning => new WarningMessageOutputDeviceData(text), - _ => new TextOutputDeviceData(text) - }; - - workQueue.Enqueue(() => outputDevice.DisplayAsync(producer, data, cancellationToken)); - } - - public void WriteLine(LogKind logKind, string text) - { - Write(logKind, text); - WriteLine(); - } - - public void Flush() - { - if (currentLine.Length > 0) - WriteLine(); - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs deleted file mode 100644 index 0f72cb90c6..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestApplicationBuilderExtensions.cs +++ /dev/null @@ -1,50 +0,0 @@ -using Microsoft.Testing.Platform.Builder; -using Microsoft.Testing.Platform.Capabilities.TestFramework; -using Microsoft.Testing.Platform.Helpers; -using System.Reflection; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// Extensions for registering BenchmarkDotNet with a Microsoft.Testing.Platform application. - /// - public static class TestApplicationBuilderExtensions - { - /// - /// Registers BenchmarkDotNet as the test framework, so that the benchmarks of the entry assembly are exposed - /// as tests. - /// - /// The builder of the test application. - /// The same builder, so that calls can be chained. - public static ITestApplicationBuilder AddBenchmarkDotNet(this ITestApplicationBuilder builder) - => builder.AddBenchmarkDotNet( - Assembly.GetEntryAssembly() ?? throw new InvalidOperationException( - "There is no entry assembly to look for benchmarks in. Use the overload that takes an assembly.")); - - /// - /// Registers BenchmarkDotNet as the test framework, so that the benchmarks of the given assembly are exposed - /// as tests. - /// - /// The builder of the test application. - /// The assembly to look for benchmarks in. - /// The same builder, so that calls can be chained. - public static ITestApplicationBuilder AddBenchmarkDotNet(this ITestApplicationBuilder builder, Assembly assembly) - { - if (builder == null) - throw new ArgumentNullException(nameof(builder)); - if (assembly == null) - throw new ArgumentNullException(nameof(assembly)); - - builder.RegisterTestFramework( - _ => new TestFrameworkCapabilities(), - (capabilities, serviceProvider) => new BenchmarkTestFramework(capabilities, serviceProvider, assembly)); - - // Opts into the tree node filter, which is what backs `--filter "/*/*/MyBenchmarks/*"`. -#pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. - builder.AddTreeNodeFilterService(new BenchmarkDotNetExtension()); -#pragma warning restore TPEXP - - return builder; - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs deleted file mode 100644 index 47aa1705a7..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/TestingPlatformBuilderHook.cs +++ /dev/null @@ -1,26 +0,0 @@ -using Microsoft.Testing.Platform.Builder; -using System.ComponentModel; - -namespace BenchmarkDotNet.TestAdapter.TestingPlatform -{ - /// - /// The hook Microsoft.Testing.Platform.MSBuild calls from the entry point it generates for the benchmark project. - /// - /// - /// This is wired up by the TestingPlatformBuilderHook item in BenchmarkDotNet.TestAdapter.TestingPlatform.props. - /// It is public because the generated code lives in the benchmark assembly, but it is not meant to be called - /// directly; use - /// instead when writing an entry point by hand. - /// - [EditorBrowsable(EditorBrowsableState.Never)] - public static class TestingPlatformBuilderHook - { - /// - /// Registers BenchmarkDotNet with the test application being built. - /// - /// The builder of the test application. - /// The command line arguments of the process. Unused. - public static void AddExtensions(ITestApplicationBuilder builder, string[] arguments) - => builder.AddBenchmarkDotNet(); - } -} diff --git a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props b/src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props deleted file mode 100644 index 7b6f563554..0000000000 --- a/src/BenchmarkDotNet.TestAdapter.TestingPlatform/build/BenchmarkDotNet.TestAdapter.TestingPlatform.props +++ /dev/null @@ -1,31 +0,0 @@ - - - - true - - - true - - - false - - - - - - BenchmarkDotNet - BenchmarkDotNet.TestAdapter.TestingPlatform.TestingPlatformBuilderHook - - - From e54444463aeb8e5acc9cd280dabb5d4f537f5cff Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:13:09 +0100 Subject: [PATCH 023/110] Integrate BenchmarkDotNet with Microsoft.Testing.Platform Add BenchmarkDotNet.TestingPlatform.props to enable seamless integration with Microsoft.Testing.Platform. This includes setting required properties for Testing Platform application behavior, ensuring `dotnet test` compatibility on older SDKs, disabling parallel test execution for multi-targeted projects by default, and registering BenchmarkDotNet as a builder hook. --- .../BenchmarkDotNet.TestingPlatform.props | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props diff --git a/src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props b/src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props new file mode 100644 index 0000000000..5f1a9412ef --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props @@ -0,0 +1,31 @@ + + + + true + + + true + + + false + + + + + + BenchmarkDotNet + BenchmarkDotNet.TestingPlatform.TestingPlatformBuilderHook + + + From 7da9979d012a49636079956884b442e961f53b5a Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:14:45 +0100 Subject: [PATCH 024/110] Add AsyncWorkQueue for ordered async work processing Introduced AsyncWorkQueue in BenchmarkDotNet.TestingPlatform to enable thread-safe, ordered queuing of asynchronous work items. Supports synchronous enqueuing, asynchronous draining, completion signaling, and resource disposal using ConcurrentQueue and SemaphoreSlim. --- .../AsyncWorkQueue.cs | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs b/src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs new file mode 100644 index 0000000000..89818c96c2 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs @@ -0,0 +1,61 @@ +using System.Collections.Concurrent; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// An ordered queue of asynchronous work items that are produced synchronously and consumed asynchronously. + /// + /// + /// BenchmarkDotNet reports its progress through synchronous callbacks (EventProcessor and ILogger) while the + /// platform message bus and output device are asynchronous. Blocking on those from inside a callback risks + /// deadlocking against the synchronization context BenchmarkDotNet installs while it runs, so the callbacks + /// enqueue instead and the caller of DrainAsync does the awaiting. + /// + internal sealed class AsyncWorkQueue : IDisposable + { + private readonly ConcurrentQueue> queue = new(); + private readonly SemaphoreSlim available = new(0); + private volatile bool completed; + + /// + /// Queues a work item. Safe to call from any thread. + /// + /// The work to perform. + public void Enqueue(Func work) + { + queue.Enqueue(work); + available.Release(); + } + + /// + /// Signals that no more work items will be enqueued, which lets DrainAsync return once the already queued + /// items have been processed. + /// + public void Complete() + { + completed = true; + available.Release(); + } + + /// + /// Processes queued work items in order until has been called and the queue is empty. + /// + /// A task that completes when the queue has been drained. + public async Task DrainAsync() + { + while (true) + { + await available.WaitAsync().ConfigureAwait(false); + + // Draining everything on each wake-up means a lost or surplus permit can never strand a work item. + while (queue.TryDequeue(out var work)) + await work().ConfigureAwait(false); + + if (completed && queue.IsEmpty) + return; + } + } + + public void Dispose() => available.Dispose(); + } +} From 2d945bf0b6c64ed56f38d8cfcb43c5e6f8b9991c Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:18:07 +0100 Subject: [PATCH 025/110] Add BenchmarkDotNet.TestingPlatform integration Introduce a new project to integrate BenchmarkDotNet with Microsoft.Testing.Platform, enabling benchmarks to be discovered and executed as tests. Implements extension identification, test framework, event processing, test node representation, and output logging. Provides builder extensions for easy registration and an MSBuild hook for automatic integration. Updates project configuration for packaging and dependencies. --- .../BenchmarkDotNet.TestingPlatform.csproj | 46 ++++ .../BenchmarkDotNetExtension.cs | 34 +++ .../BenchmarkEventProcessor.cs | 193 +++++++++++++++++ .../BenchmarkTestFramework.cs | 197 ++++++++++++++++++ .../BenchmarkTestNode.cs | 148 +++++++++++++ .../OutputDeviceLogger.cs | 77 +++++++ .../TestApplicationBuilderExtensions.cs | 50 +++++ .../TestingPlatformBuilderHook.cs | 26 +++ 8 files changed, 771 insertions(+) create mode 100644 src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj create mode 100644 src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNetExtension.cs create mode 100644 src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs create mode 100644 src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs create mode 100644 src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs create mode 100644 src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs create mode 100644 src/BenchmarkDotNet.TestingPlatform/TestApplicationBuilderExtensions.cs create mode 100644 src/BenchmarkDotNet.TestingPlatform/TestingPlatformBuilderHook.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj new file mode 100644 index 0000000000..a97893d407 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj @@ -0,0 +1,46 @@ + + + + netstandard2.0 + BenchmarkDotNet.TestingPlatform + BenchmarkDotNet.TestingPlatform + BenchmarkDotNet.TestingPlatform + Runs BenchmarkDotNet benchmarks as tests through Microsoft.Testing.Platform + README.md + True + BenchmarkDotNet.TestingPlatform + + + false + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNetExtension.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNetExtension.cs new file mode 100644 index 0000000000..d2bc8722d8 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNetExtension.cs @@ -0,0 +1,34 @@ +using Microsoft.Testing.Platform.Extensions; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// Identifies BenchmarkDotNet to Microsoft.Testing.Platform. + /// + /// + /// Several platform services, such as the tree node filter behind --filter, are registered on behalf of an + /// extension rather than of the test framework itself, so the identity lives in its own type. + /// + internal sealed class BenchmarkDotNetExtension : IExtension + { + /// + /// The uid shared by every extension this package registers. + /// + public const string ExtensionUid = "BenchmarkDotNet.TestingPlatform"; + + /// + public string Uid => ExtensionUid; + + /// + public string Version => typeof(BenchmarkDotNetExtension).Assembly.GetName().Version?.ToString() ?? "0.0.0"; + + /// + public string DisplayName => "BenchmarkDotNet"; + + /// + public string Description => "Runs BenchmarkDotNet benchmarks as tests."; + + /// + public Task IsEnabledAsync() => Task.FromResult(true); + } +} diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs new file mode 100644 index 0000000000..a7d50256da --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs @@ -0,0 +1,193 @@ +using BenchmarkDotNet.EventProcessors; +using BenchmarkDotNet.Exporters; +using BenchmarkDotNet.Extensions; +using BenchmarkDotNet.Reports; +using BenchmarkDotNet.Running; +using BenchmarkDotNet.Toolchains.Results; +using BenchmarkDotNet.Validators; +using Microsoft.Testing.Platform.Extensions.Messages; +using Perfolizer.Mathematics.Histograms; +using System.Diagnostics; +using System.Globalization; +using System.Text; + +namespace BenchmarkDotNet.TestingPlatform +{ + + /// + /// Translates the BenchmarkDotNet run into the stream of test node updates the platform expects. + /// + internal sealed class BenchmarkEventProcessor : EventProcessor + { + private readonly IReadOnlyDictionary nodes; + private readonly Action publish; + private readonly Stopwatch runTimerStopwatch = new(); + private readonly Dictionary pendingResults = []; + private readonly HashSet publishedResults = []; + + public BenchmarkEventProcessor(IReadOnlyDictionary nodes, Action publish) + { + this.nodes = nodes; + this.publish = publish; + } + + public override void OnValidationError(ValidationError validationError) + { + // If the error is not linked to a benchmark case, then set the error on all benchmarks. + var affected = validationError.BenchmarkCase == null + ? nodes.Values + : [nodes[FullNameProvider.GetBenchmarkUid(validationError.BenchmarkCase)]]; + + foreach (var node in affected) + { + var pending = GetOrCreatePendingResult(node); + + if (validationError.IsCritical) + { + // The result is not published yet, in case there are more validation errors to append. + pending.ErrorMessages.Add(validationError.Message); + } + else + { + pending.Output.AppendLine($"WARNING: {validationError.Message}"); + } + } + } + + public override void OnBuildComplete(BuildPartition buildPartition, BuildResult buildResult) + { + // Only build failures need to be reported, successful builds are followed by a run. + if (buildResult.IsBuildSuccess) + return; + + foreach (var benchmarkBuildInfo in buildPartition.Benchmarks) + { + var node = nodes[FullNameProvider.GetBenchmarkUid(benchmarkBuildInfo.BenchmarkCase)]; + var pending = GetOrCreatePendingResult(node); + + if (buildResult.GenerateException != null) + pending.ErrorMessages.Add($"// Generate Exception: {buildResult.GenerateException.Message}"); + else if (buildResult.TryToExplainFailureReason(buildPartition.GetInProcessDiagnoserHandlerTypes(), out string? reason)) + pending.ErrorMessages.Add($"// Build Error: {reason}"); + else if (buildResult.ErrorMessage != null) + pending.ErrorMessages.Add($"// Build Error: {buildResult.ErrorMessage}"); + + // A benchmark that failed to build will never run, so the result can be published immediately. + publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); + PublishResult(node, pending, new FailedTestNodeStateProperty(pending.GetErrorMessage() ?? "The benchmark failed to build.")); + } + } + + public override void OnStartRunBenchmark(BenchmarkCase benchmarkCase) + { + var node = nodes[FullNameProvider.GetBenchmarkUid(benchmarkCase)]; + var pending = GetOrCreatePendingResult(node); + pending.StartTime = DateTimeOffset.UtcNow; + + publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); + runTimerStopwatch.Restart(); + } + + public override void OnEndRunBenchmark(BenchmarkCase benchmarkCase, BenchmarkReport report) + { + var node = nodes[FullNameProvider.GetBenchmarkUid(benchmarkCase)]; + var pending = GetOrCreatePendingResult(node); + pending.Duration = runTimerStopwatch.Elapsed; + pending.EndTime = DateTimeOffset.UtcNow; + + AppendMeasurementSummary(pending.Output, report); + + TestNodeStateProperty state = report.Success && pending.ErrorMessages.Count == 0 + ? PassedTestNodeStateProperty.CachedInstance + : new FailedTestNodeStateProperty(pending.GetErrorMessage() ?? "The benchmark did not complete successfully."); + + PublishResult(node, pending, state); + } + + /// + /// Publishes a result for every benchmark that was scheduled to run but never reported one, which happens when + /// a critical validation error stopped it or when BenchmarkDotNet never reached it. + /// + public void PublishOutstandingResults() + { + foreach (var node in nodes.Values) + { + if (publishedResults.Contains(node.Uid)) + continue; + + var pending = GetOrCreatePendingResult(node); + var errorMessage = pending.GetErrorMessage(); + + TestNodeStateProperty state = errorMessage != null + ? new FailedTestNodeStateProperty(errorMessage) + : SkippedTestNodeStateProperty.CachedInstance; + + publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); + PublishResult(node, pending, state); + } + } + + private void PublishResult(BenchmarkTestNode node, PendingResult pending, TestNodeStateProperty state) + { + var properties = new List(); + + if (pending.StartTime is { } startTime) + { + var duration = pending.Duration ?? TimeSpan.Zero; + properties.Add(new TimingProperty(new TimingInfo(startTime, pending.EndTime ?? startTime + duration, duration))); + } + + if (pending.Output.Length > 0) + properties.Add(new StandardOutputProperty(pending.Output.ToString())); + + publish(node.ToTestNode(state, properties.ToArray())); + publishedResults.Add(node.Uid); + } + + private PendingResult GetOrCreatePendingResult(BenchmarkTestNode node) + { + if (!pendingResults.TryGetValue(node.Uid, out var pending)) + { + pending = new PendingResult(); + pendingResults[node.Uid] = pending; + } + + return pending; + } + + private static void AppendMeasurementSummary(StringBuilder output, BenchmarkReport report) + { + var resultRuns = report.GetResultRuns(); + if (resultRuns.Count == 0) + return; + + output.AppendLine(report.BenchmarkCase.DisplayInfo); + output.AppendLine($"Runtime = {report.GetRuntimeInfo()}; GC = {report.GetGcInfo()}"); + + var statistics = resultRuns.GetStatistics(); + var cultureInfo = CultureInfo.InvariantCulture; + var formatter = statistics.CreateNanosecondFormatter(cultureInfo); + + var histogram = HistogramBuilder.Adaptive.Build(statistics.Sample.Values); + output.AppendLine("-------------------- Histogram --------------------"); + output.AppendLine(histogram.ToString(formatter)); + output.AppendLine("---------------------------------------------------"); + output.AppendLine(statistics.ToString(cultureInfo, formatter, calcHistogram: false)); + } + + private sealed class PendingResult + { + public List ErrorMessages { get; } = []; + + public StringBuilder Output { get; } = new(); + + public DateTimeOffset? StartTime { get; set; } + + public DateTimeOffset? EndTime { get; set; } + + public TimeSpan? Duration { get; set; } + + public string? GetErrorMessage() => ErrorMessages.Count == 0 ? null : string.Join("\n", ErrorMessages); + } + } +} diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs new file mode 100644 index 0000000000..48fcfa8d72 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs @@ -0,0 +1,197 @@ +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Loggers; +using BenchmarkDotNet.Running; +// BenchmarkEnumerator is compiled into this assembly from the VSTest adapter, where it keeps its own namespace. +using BenchmarkDotNet.TestAdapter; +using Microsoft.Testing.Platform.Capabilities.TestFramework; +using Microsoft.Testing.Platform.Extensions.Messages; +using Microsoft.Testing.Platform.Extensions.OutputDevice; +using Microsoft.Testing.Platform.Extensions.TestFramework; +using Microsoft.Testing.Platform.Requests; +using Microsoft.Testing.Platform.Services; +using System.Reflection; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// Discovers and executes the benchmarks of the running assembly through Microsoft.Testing.Platform. + /// + internal sealed class BenchmarkTestFramework : ITestFramework, IDataProducer, IOutputDeviceDataProducer + { + private readonly BenchmarkDotNetExtension extension = new(); + private readonly IServiceProvider serviceProvider; + private readonly Assembly assembly; + + public BenchmarkTestFramework(ITestFrameworkCapabilities capabilities, IServiceProvider serviceProvider, Assembly assembly) + { + Capabilities = capabilities; + this.serviceProvider = serviceProvider; + this.assembly = assembly; + } + + /// + public string Uid => extension.Uid; + + /// + public string Version => extension.Version; + + /// + public string DisplayName => extension.DisplayName; + + /// + public string Description => extension.Description; + + /// + public Type[] DataTypesProduced => [typeof(TestNodeUpdateMessage)]; + + /// + /// Gets the capabilities the framework was registered with. + /// + public ITestFrameworkCapabilities Capabilities { get; } + + /// + public Task IsEnabledAsync() => extension.IsEnabledAsync(); + + /// + public Task CreateTestSessionAsync(CreateTestSessionContext context) + => Task.FromResult(new CreateTestSessionResult { IsSuccess = true }); + + /// + public Task CloseTestSessionAsync(CloseTestSessionContext context) + => Task.FromResult(new CloseTestSessionResult { IsSuccess = true }); + + /// + public async Task ExecuteRequestAsync(ExecuteRequestContext context) + { + try + { + switch (context.Request) + { + case DiscoverTestExecutionRequest discoverRequest: + await DiscoverAsync(discoverRequest, context).ConfigureAwait(false); + break; + case RunTestExecutionRequest runRequest: + await RunAsync(runRequest, context).ConfigureAwait(false); + break; + } + } + finally + { + context.Complete(); + } + } + + private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRequestContext context) + { + foreach (var (_, node) in GetMatchingBenchmarks(request.Filter)) + { + context.CancellationToken.ThrowIfCancellationRequested(); + + var message = new TestNodeUpdateMessage( + request.Session.SessionUid, + node.ToTestNode(DiscoveredTestNodeStateProperty.CachedInstance)); + + await context.MessageBus.PublishAsync(this, message).ConfigureAwait(false); + } + } + + private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestContext context) + { + var matches = GetMatchingBenchmarks(request.Filter); + if (matches.Count == 0) + return; + + var nodes = matches.ToDictionary(match => match.Node.Uid, match => match.Node); + var sessionUid = request.Session.SessionUid; + var cancellationToken = context.CancellationToken; + + using var workQueue = new AsyncWorkQueue(); + + var eventProcessor = new BenchmarkEventProcessor(nodes, testNode => + { + var message = new TestNodeUpdateMessage(sessionUid, testNode); + workQueue.Enqueue(() => context.MessageBus.PublishAsync(this, message)); + }); + + // BenchmarkDotNet's own console output is replaced so that everything goes through the output device, + // which keeps it in the right place when the platform runs in server mode or inside an IDE. + var logger = new OutputDeviceLogger(serviceProvider.GetOutputDevice(), this, workQueue, cancellationToken); + + var runInfos = matches + .GroupBy(match => match.RunInfo) + .Select(group => new BenchmarkRunInfo( + group.Select(match => match.Node.BenchmarkCase).ToArray(), + group.Key.Type, + group.Key.Config + .AddEventProcessor(eventProcessor) + .AddLogger(logger) + .RemoveLoggersOfType() + .CreateImmutableConfig(), + group.Key.CompositeInProcessDiagnoser)) + .ToArray(); + + // BenchmarkDotNet blocks the calling thread for the whole run, so it gets a thread of its own and the + // queued messages are published from here as they are produced. + var runTask = Task.Run( + () => + { + try + { + BenchmarkRunner.Run(runInfos, cancellationToken); + } + finally + { + // Benchmarks that never reported a result still need one, unless the run was cancelled: the + // platform expects an OperationCanceledException in that case, and publishing results + // afterwards would contradict it. + if (!cancellationToken.IsCancellationRequested) + eventProcessor.PublishOutstandingResults(); + + logger.Flush(); + workQueue.Complete(); + } + }, + CancellationToken.None); + + await workQueue.DrainAsync().ConfigureAwait(false); + await runTask.ConfigureAwait(false); + } + + /// + /// Enumerates the benchmarks of the assembly and keeps the ones the request asked for. + /// + /// The filter of the request. + /// The matching benchmarks, paired with the run info they belong to. + private List<(BenchmarkRunInfo RunInfo, BenchmarkTestNode Node)> GetMatchingBenchmarks(ITestExecutionFilter filter) + { + var matches = new List<(BenchmarkRunInfo, BenchmarkTestNode)>(); + + foreach (var runInfo in BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly)) + { + // The job only earns a place in the display name when the benchmark actually runs under several jobs. + // This is computed before filtering so that a benchmark keeps the same name however it was selected. + var includeJobInName = runInfo.BenchmarksCases.Select(c => c.Job.DisplayInfo).Distinct().Count() > 1; + + foreach (var benchmarkCase in runInfo.BenchmarksCases) + { + var node = BenchmarkTestNode.Create(benchmarkCase, includeJobInName); + if (Matches(filter, node)) + matches.Add((runInfo, node)); + } + } + + return matches; + } + +#pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. + private static bool Matches(ITestExecutionFilter filter, BenchmarkTestNode node) => filter switch + { + TestNodeUidListFilter uidListFilter => uidListFilter.TestNodeUids.Any(uid => uid.Value == node.Uid), + TreeNodeFilter treeNodeFilter => treeNodeFilter.MatchesFilter(node.Path, node.GetFilterableProperties()), + + // NopFilter, and anything the platform adds later, means "everything". + _ => true + }; +#pragma warning restore TPEXP + } +} diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs new file mode 100644 index 0000000000..61d6cfb7a5 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs @@ -0,0 +1,148 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Exporters; +using BenchmarkDotNet.Extensions; +using BenchmarkDotNet.Running; +using Microsoft.Testing.Platform.Extensions.Messages; +using System.Reflection; +using System.Text; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// The Microsoft.Testing.Platform view of a single . + /// + /// + /// A carries mutable state (the property bag holds the current outcome), so a fresh node is + /// created for every message published on the bus. This class holds the parts that never change. + /// + internal sealed class BenchmarkTestNode + { + private readonly IProperty[] staticProperties; + + private BenchmarkTestNode(BenchmarkCase benchmarkCase, string uid, string displayName, string path, IProperty[] staticProperties) + { + BenchmarkCase = benchmarkCase; + Uid = uid; + DisplayName = displayName; + Path = path; + this.staticProperties = staticProperties; + } + + /// + /// Gets the benchmark this node represents. + /// + public BenchmarkCase BenchmarkCase { get; } + + /// + /// Gets the stable identifier of the node. It has to be identical in the discovery and the execution phase, + /// which may happen in different processes. + /// + public string Uid { get; } + + /// + /// Gets the name shown by test runners. + /// + public string DisplayName { get; } + + /// + /// Gets the '/' separated path used by . + /// + public string Path { get; } + + /// + /// Creates the node for a benchmark case. + /// + /// The benchmark case to describe. + /// + /// Whether the display name should be suffixed with the job, which is only useful when the benchmark runs + /// under more than one job. + /// + /// The created node. + public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool includeJobInName) + { + var benchmarkMethod = benchmarkCase.Descriptor.WorkloadMethod; + var type = benchmarkCase.Descriptor.Type; + var fullClassName = type.GetCorrectCSharpTypeName(prefixWithGlobal: false); + var parametrizedMethodName = FullNameProvider.GetMethodName(benchmarkCase); + var jobDisplayInfo = benchmarkCase.GetUnrandomizedJobDisplayInfo(); + + // Unlike the uid, the job is only part of the display name when it actually adds information. + var uid = FullNameProvider.GetBenchmarkUid(benchmarkCase); + var displayName = $"{fullClassName}.{parametrizedMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); + + var properties = new List + { + new TestMethodIdentifierProperty( + type.Assembly.FullName, + type.Namespace ?? string.Empty, + type.GetCorrectCSharpTypeName(prefixWithGlobal: false, includeNamespace: false), + benchmarkMethod.Name, + benchmarkMethod.IsGenericMethodDefinition ? benchmarkMethod.GetGenericArguments().Length : 0, + benchmarkMethod.GetParameters().Select(p => p.ParameterType.FullName ?? p.ParameterType.Name).ToArray(), + benchmarkMethod.ReturnType.FullName ?? benchmarkMethod.ReturnType.Name), + }; + + var benchmarkAttribute = benchmarkMethod.ResolveAttribute(); + if (benchmarkAttribute?.SourceCodeFile != null) + { + // BenchmarkAttribute captures the line of the attribute itself, and the platform expects 0-based lines. + var line = Math.Max(0, benchmarkAttribute.SourceCodeLineNumber - 1); + var position = new LinePosition(line, 0); + properties.Add(new TestFileLocationProperty(benchmarkAttribute.SourceCodeFile, new LinePositionSpan(position, position))); + } + + foreach (var category in DefaultCategoryDiscoverer.Instance.GetCategories(benchmarkMethod)) + properties.Add(new TestMetadataProperty("Category", category)); + + var path = BuildPath(type.Assembly, type.Namespace, fullClassName, parametrizedMethodName, jobDisplayInfo); + + return new BenchmarkTestNode(benchmarkCase, uid, displayName, path, properties.ToArray()); + } + + /// + /// Creates a message-bus ready node in the given state. + /// + /// The state of the benchmark, e.g. discovered, passed or failed. + /// Any additional properties, such as timing or captured output. + /// The created test node. + public TestNode ToTestNode(TestNodeStateProperty state, params IProperty[] extraProperties) + { + var properties = new PropertyBag(staticProperties); + properties.Add(state); + foreach (var property in extraProperties) + properties.Add(property); + + return new TestNode + { + Uid = new TestNodeUid(Uid), + DisplayName = DisplayName, + Properties = properties + }; + } + + /// + /// Gets the properties a can match against, + /// which is what makes `--filter "/*/*/*/*[Category=Fast]"` work. + /// + /// The filterable properties. + public PropertyBag GetFilterableProperties() => new PropertyBag(staticProperties); + + private static string BuildPath(Assembly assembly, string? @namespace, string fullClassName, string methodName, string jobDisplayInfo) + { + // The convention followed by the other test frameworks is ////. + var className = @namespace == null || !fullClassName.StartsWith(@namespace + ".", StringComparison.Ordinal) + ? fullClassName + : fullClassName.Substring(@namespace.Length + 1); + + return new StringBuilder() + .Append('/').Append(Escape(assembly.GetName().Name)) + .Append('/').Append(Escape(@namespace ?? string.Empty)) + .Append('/').Append(Escape(className)) + .Append('/').Append(Escape($"{methodName} [{jobDisplayInfo}]")) + .ToString(); + } + + // Benchmark parameters are stringified user values, so they can contain the path separator. + private static string Escape(string segment) => segment.Replace("/", "\\/"); + } +} diff --git a/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs b/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs new file mode 100644 index 0000000000..917ca5b25e --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs @@ -0,0 +1,77 @@ +using BenchmarkDotNet.Loggers; +using Microsoft.Testing.Platform.Extensions.OutputDevice; +using Microsoft.Testing.Platform.OutputDevice; +using System.Text; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// Forwards the BenchmarkDotNet log to the platform output device, so that build progress and the result summary + /// show up in the test run output. + /// + internal sealed class OutputDeviceLogger : ILogger + { + private readonly IOutputDevice outputDevice; + private readonly IOutputDeviceDataProducer producer; + private readonly AsyncWorkQueue workQueue; + private readonly CancellationToken cancellationToken; + private readonly StringBuilder currentLine = new(); + private LogKind currentLineKind = LogKind.Default; + + public OutputDeviceLogger( + IOutputDevice outputDevice, + IOutputDeviceDataProducer producer, + AsyncWorkQueue workQueue, + CancellationToken cancellationToken) + { + this.outputDevice = outputDevice; + this.producer = producer; + this.workQueue = workQueue; + this.cancellationToken = cancellationToken; + } + + public string Id => nameof(OutputDeviceLogger); + + public int Priority => 0; + + public void Write(LogKind logKind, string text) + { + currentLine.Append(text); + + // Assume that if any part of the line is an error or a warning, the whole line is. + // The kind is reset when the line is flushed. + if (logKind == LogKind.Error || (logKind == LogKind.Warning && currentLineKind != LogKind.Error)) + currentLineKind = logKind; + } + + public void WriteLine() + { + var text = currentLine.ToString(); + var kind = currentLineKind; + + currentLine.Clear(); + currentLineKind = LogKind.Default; + + IOutputDeviceData data = kind switch + { + LogKind.Error => new ErrorMessageOutputDeviceData(text), + LogKind.Warning => new WarningMessageOutputDeviceData(text), + _ => new TextOutputDeviceData(text) + }; + + workQueue.Enqueue(() => outputDevice.DisplayAsync(producer, data, cancellationToken)); + } + + public void WriteLine(LogKind logKind, string text) + { + Write(logKind, text); + WriteLine(); + } + + public void Flush() + { + if (currentLine.Length > 0) + WriteLine(); + } + } +} diff --git a/src/BenchmarkDotNet.TestingPlatform/TestApplicationBuilderExtensions.cs b/src/BenchmarkDotNet.TestingPlatform/TestApplicationBuilderExtensions.cs new file mode 100644 index 0000000000..9df284d376 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/TestApplicationBuilderExtensions.cs @@ -0,0 +1,50 @@ +using Microsoft.Testing.Platform.Builder; +using Microsoft.Testing.Platform.Capabilities.TestFramework; +using Microsoft.Testing.Platform.Helpers; +using System.Reflection; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// Extensions for registering BenchmarkDotNet with a Microsoft.Testing.Platform application. + /// + public static class TestApplicationBuilderExtensions + { + /// + /// Registers BenchmarkDotNet as the test framework, so that the benchmarks of the entry assembly are exposed + /// as tests. + /// + /// The builder of the test application. + /// The same builder, so that calls can be chained. + public static ITestApplicationBuilder AddBenchmarkDotNet(this ITestApplicationBuilder builder) + => builder.AddBenchmarkDotNet( + Assembly.GetEntryAssembly() ?? throw new InvalidOperationException( + "There is no entry assembly to look for benchmarks in. Use the overload that takes an assembly.")); + + /// + /// Registers BenchmarkDotNet as the test framework, so that the benchmarks of the given assembly are exposed + /// as tests. + /// + /// The builder of the test application. + /// The assembly to look for benchmarks in. + /// The same builder, so that calls can be chained. + public static ITestApplicationBuilder AddBenchmarkDotNet(this ITestApplicationBuilder builder, Assembly assembly) + { + if (builder == null) + throw new ArgumentNullException(nameof(builder)); + if (assembly == null) + throw new ArgumentNullException(nameof(assembly)); + + builder.RegisterTestFramework( + _ => new TestFrameworkCapabilities(), + (capabilities, serviceProvider) => new BenchmarkTestFramework(capabilities, serviceProvider, assembly)); + + // Opts into the tree node filter, which is what backs `--filter "/*/*/MyBenchmarks/*"`. +#pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. + builder.AddTreeNodeFilterService(new BenchmarkDotNetExtension()); +#pragma warning restore TPEXP + + return builder; + } + } +} diff --git a/src/BenchmarkDotNet.TestingPlatform/TestingPlatformBuilderHook.cs b/src/BenchmarkDotNet.TestingPlatform/TestingPlatformBuilderHook.cs new file mode 100644 index 0000000000..5e77ef6b88 --- /dev/null +++ b/src/BenchmarkDotNet.TestingPlatform/TestingPlatformBuilderHook.cs @@ -0,0 +1,26 @@ +using Microsoft.Testing.Platform.Builder; +using System.ComponentModel; + +namespace BenchmarkDotNet.TestingPlatform +{ + /// + /// The hook Microsoft.Testing.Platform.MSBuild calls from the entry point it generates for the benchmark project. + /// + /// + /// This is wired up by the TestingPlatformBuilderHook item in BenchmarkDotNet.TestingPlatform.props. + /// It is public because the generated code lives in the benchmark assembly, but it is not meant to be called + /// directly; use + /// instead when writing an entry point by hand. + /// + [EditorBrowsable(EditorBrowsableState.Never)] + public static class TestingPlatformBuilderHook + { + /// + /// Registers BenchmarkDotNet with the test application being built. + /// + /// The builder of the test application. + /// The command line arguments of the process. Unused. + public static void AddExtensions(ITestApplicationBuilder builder, string[] arguments) + => builder.AddBenchmarkDotNet(); + } +} From 2158052015ce471adb0fbf1f1cfb705384698609 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:18:20 +0100 Subject: [PATCH 026/110] Update project references to TestingPlatform project Updated BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj to reference BenchmarkDotNet.TestingPlatform instead of BenchmarkDotNet.TestAdapter.TestingPlatform. Adjusted both the ProjectReference and Import paths accordingly. --- .../BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj index b89575026a..59913fd0b3 100644 --- a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj @@ -9,11 +9,11 @@ - + - + From 67771650f6dfabfc94640a8939ce0d7e38256bd5 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 11:18:33 +0100 Subject: [PATCH 027/110] Refactor solution file project entries Simplified project definitions in BenchmarkDotNet.slnx by removing custom build entries and updating project paths, including renaming the TestingPlatform project. --- BenchmarkDotNet.slnx | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index 119a22f6ca..ca8e06a6dd 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -20,10 +20,8 @@ - - - + @@ -42,16 +40,10 @@ - - - - - - + + - - - + From 83961038d37868cf4f5720b585ad6897527d51f4 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 15 Aug 2026 14:02:44 +0100 Subject: [PATCH 028/110] Update props packaging and cSpell dictionary - Add "testingplatform" to cSpell.json to suppress spelling warnings. - Refactor .props packaging in csproj to use a single entry with multiple paths, ensuring cross-platform compatibility and resolving NU5129. --- build/cSpell.json | 1 + .../BenchmarkDotNet.TestingPlatform.csproj | 5 +++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/build/cSpell.json b/build/cSpell.json index e90695fa7f..4cb6c27117 100644 --- a/build/cSpell.json +++ b/build/cSpell.json @@ -34,6 +34,7 @@ "vsprofiler", "vstest", "Tailcall", + "testingplatform", "toolchains", "unmanaged" ], diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj index a97893d407..7bcea6e705 100644 --- a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj @@ -38,8 +38,9 @@ - - + + From cd523d53ce876c78fa586c02144602852414c759 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 03:36:56 +0100 Subject: [PATCH 029/110] Add generic benchmark for closed generic type testing Introduced GenericProbe benchmark class in a new BenchmarkDotNet.IntegrationTests.TestingPlatform namespace to evaluate how test runners handle closed generic types. The benchmark tests instance creation for int, char, and List type arguments. Added GenericProbeConfig to configure the benchmark with InProcessEmit toolchain and a dry job. --- .../GenericProbe.cs | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/GenericProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/GenericProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/GenericProbe.cs new file mode 100644 index 0000000000..24ffa27b9c --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/GenericProbe.cs @@ -0,0 +1,25 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// A generic benchmark, used to check how closed generic types are named and grouped by test runners. + /// + [Config(typeof(GenericProbeConfig))] + [GenericTypeArguments(typeof(int))] + [GenericTypeArguments(typeof(char))] + [GenericTypeArguments(typeof(System.Collections.Generic.List))] + public class GenericProbe where T : new() + { + [Benchmark] + public T Create() => new T(); + } + + internal class GenericProbeConfig : ManualConfig + { + public GenericProbeConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } +} From 9f2cacb1f7924972fd73b20fb3758aa33933ad67 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 03:38:15 +0100 Subject: [PATCH 030/110] Handle benchmark UID collisions during discovery/run Benchmarks are now grouped by UID to detect collisions. When multiple benchmarks share a UID, `PublishCollisionAsync` reports the issue as a failed test node, allowing other benchmarks to proceed. Only benchmarks with unique UIDs are executed. The refactor introduces a `Match` class, improves cancellation and exception handling, and ensures proper resource cleanup during async operations. --- .../BenchmarkTestFramework.cs | 149 +++++++++++++++--- 1 file changed, 125 insertions(+), 24 deletions(-) diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs index 48fcfa8d72..368c12dfa0 100644 --- a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs @@ -9,7 +9,9 @@ using Microsoft.Testing.Platform.Extensions.TestFramework; using Microsoft.Testing.Platform.Requests; using Microsoft.Testing.Platform.Services; +using Microsoft.Testing.Platform.TestHost; using System.Reflection; +using System.Runtime.ExceptionServices; namespace BenchmarkDotNet.TestingPlatform { @@ -83,13 +85,15 @@ public async Task ExecuteRequestAsync(ExecuteRequestContext context) private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRequestContext context) { - foreach (var (_, node) in GetMatchingBenchmarks(request.Filter)) + foreach (var benchmarks in GetMatchingBenchmarks(request.Filter)) { context.CancellationToken.ThrowIfCancellationRequested(); + // Exactly one node per uid: publishing a colliding uid twice would leave the platform with two nodes + // it cannot tell apart. The collision itself is reported when the benchmarks are run. var message = new TestNodeUpdateMessage( request.Session.SessionUid, - node.ToTestNode(DiscoveredTestNodeStateProperty.CachedInstance)); + benchmarks[0].Node.ToTestNode(DiscoveredTestNodeStateProperty.CachedInstance)); await context.MessageBus.PublishAsync(this, message).ConfigureAwait(false); } @@ -97,16 +101,29 @@ private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRe private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestContext context) { - var matches = GetMatchingBenchmarks(request.Filter); - if (matches.Count == 0) - return; - - var nodes = matches.ToDictionary(match => match.Node.Uid, match => match.Node); var sessionUid = request.Session.SessionUid; var cancellationToken = context.CancellationToken; + var runnable = new List(); + foreach (var benchmarks in GetMatchingBenchmarks(request.Filter)) + { + if (benchmarks.Count == 1) + runnable.Add(benchmarks[0]); + else + await PublishCollisionAsync(context, sessionUid, benchmarks).ConfigureAwait(false); + } + + if (runnable.Count == 0) + return; + + var nodes = runnable.ToDictionary(match => match.Node.Uid, match => match.Node); + using var workQueue = new AsyncWorkQueue(); + // A failure while publishing has to stop the benchmarks as well, otherwise the run would carry on with + // nobody listening and would keep writing to a queue that is about to be disposed. + using var runCancellation = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + var eventProcessor = new BenchmarkEventProcessor(nodes, testNode => { var message = new TestNodeUpdateMessage(sessionUid, testNode); @@ -117,7 +134,7 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte // which keeps it in the right place when the platform runs in server mode or inside an IDE. var logger = new OutputDeviceLogger(serviceProvider.GetOutputDevice(), this, workQueue, cancellationToken); - var runInfos = matches + var runInfos = runnable .GroupBy(match => match.RunInfo) .Select(group => new BenchmarkRunInfo( group.Select(match => match.Node.BenchmarkCase).ToArray(), @@ -137,34 +154,93 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte { try { - BenchmarkRunner.Run(runInfos, cancellationToken); + BenchmarkRunner.Run(runInfos, runCancellation.Token); } finally { - // Benchmarks that never reported a result still need one, unless the run was cancelled: the - // platform expects an OperationCanceledException in that case, and publishing results - // afterwards would contradict it. - if (!cancellationToken.IsCancellationRequested) - eventProcessor.PublishOutstandingResults(); - - logger.Flush(); - workQueue.Complete(); + try + { + // Benchmarks that never reported a result still need one, unless the run was cancelled: + // the platform expects an OperationCanceledException in that case, and publishing results + // afterwards would contradict it. + if (!runCancellation.IsCancellationRequested) + eventProcessor.PublishOutstandingResults(); + + logger.Flush(); + } + finally + { + // The drain only ends once the queue is completed, so this has to happen no matter what + // else went wrong. + workQueue.Complete(); + } } }, CancellationToken.None); - await workQueue.DrainAsync().ConfigureAwait(false); - await runTask.ConfigureAwait(false); + ExceptionDispatchInfo? drainFailure = null; + try + { + await workQueue.DrainAsync().ConfigureAwait(false); + } + catch (Exception exception) + { + // Nothing consumes the queue anymore, so the run has to be stopped rather than left orphaned. This is + // also the path a cancelled run takes, since the queued writes are handed the platform's token. + drainFailure = ExceptionDispatchInfo.Capture(exception); + runCancellation.Cancel(); + } + + try + { + // BenchmarkDotNet keeps writing to the queue until its thread returns, so the run always has to be + // over before the queue is disposed. + await runTask.ConfigureAwait(false); + } + catch when (drainFailure != null) + { + // The run was stopped because publishing failed, so that failure is the one worth reporting. + } + + drainFailure?.Throw(); + } + + /// + /// Reports benchmarks that share a uid as a single failed test. + /// + /// + /// The platform identifies test nodes by uid, so benchmarks that produce the same one cannot be reported + /// separately. Failing them keeps the rest of the run going, which is more useful than aborting the request. + /// + private async Task PublishCollisionAsync(ExecuteRequestContext context, SessionUid sessionUid, List collision) + { + var node = collision[0].Node; + var error = + $"{collision.Count} benchmarks are identified as '{node.Uid}', so they cannot be told apart and none " + + "of them were run. Benchmarks are identified by the string representation of their parameters: give " + + "the colliding values distinct ToString() results."; + + await context.MessageBus.PublishAsync( + this, + new TestNodeUpdateMessage(sessionUid, node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance))).ConfigureAwait(false); + + await context.MessageBus.PublishAsync( + this, + new TestNodeUpdateMessage(sessionUid, node.ToTestNode(new FailedTestNodeStateProperty(error)))).ConfigureAwait(false); } /// /// Enumerates the benchmarks of the assembly and keeps the ones the request asked for. /// /// The filter of the request. - /// The matching benchmarks, paired with the run info they belong to. - private List<(BenchmarkRunInfo RunInfo, BenchmarkTestNode Node)> GetMatchingBenchmarks(ITestExecutionFilter filter) + /// + /// The matching benchmarks in enumeration order, grouped by uid. A group holding more than one benchmark is a + /// uid collision. + /// + private List> GetMatchingBenchmarks(ITestExecutionFilter filter) { - var matches = new List<(BenchmarkRunInfo, BenchmarkTestNode)>(); + var matches = new List>(); + var matchesByUid = new Dictionary>(StringComparer.Ordinal); foreach (var runInfo in BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly)) { @@ -175,8 +251,17 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte foreach (var benchmarkCase in runInfo.BenchmarksCases) { var node = BenchmarkTestNode.Create(benchmarkCase, includeJobInName); - if (Matches(filter, node)) - matches.Add((runInfo, node)); + if (!Matches(filter, node)) + continue; + + if (!matchesByUid.TryGetValue(node.Uid, out var sameUid)) + { + sameUid = new List(); + matchesByUid.Add(node.Uid, sameUid); + matches.Add(sameUid); + } + + sameUid.Add(new Match(runInfo, node)); } } @@ -193,5 +278,21 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte _ => true }; #pragma warning restore TPEXP + + /// + /// A benchmark that matched the request, together with the run info it belongs to. + /// + private sealed class Match + { + public Match(BenchmarkRunInfo runInfo, BenchmarkTestNode node) + { + RunInfo = runInfo; + Node = node; + } + + public BenchmarkRunInfo RunInfo { get; } + + public BenchmarkTestNode Node { get; } + } } } From 539cc5b7994979121b5e9ba89c8ec83871442c41 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 03:38:29 +0100 Subject: [PATCH 031/110] Adjust whitespace around [PublicAPI] attribute Only whitespace was changed above the GetBenchmarkUid method; no functional or logical modifications were made. --- src/BenchmarkDotNet/Exporters/FullNameProvider.cs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs index 922d3590c4..79de14dc1c 100644 --- a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs +++ b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs @@ -73,14 +73,13 @@ public static string GetBenchmarkName(BenchmarkCase benchmarkCase) /// The job is always part of the uid, otherwise two cases of the same benchmark that only differ by job would /// collide. The parameters are already part of the method name. /// - [PublicAPI] + [PublicAPI] public static string GetBenchmarkUid(BenchmarkCase benchmarkCase) { var fullClassName = benchmarkCase.Descriptor.Type.GetCorrectCSharpTypeName(prefixWithGlobal: false); return $"{fullClassName}.{GetMethodName(benchmarkCase)} [{benchmarkCase.GetUnrandomizedJobDisplayInfo()}]"; } - private static string GetNestedTypes(Type type) { string nestedTypes = ""; From f5af28f97efbe07b2db67a1226273d13f8cb3b0a Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 04:02:29 +0100 Subject: [PATCH 032/110] Remove AsyncWorkQueue and related async queue logic Deleted AsyncWorkQueue.cs, removing the AsyncWorkQueue class and all associated methods for managing and draining asynchronous work items. This eliminates the custom ordered async work queue implementation. --- .../AsyncWorkQueue.cs | 61 ------------------- 1 file changed, 61 deletions(-) delete mode 100644 src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs b/src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs deleted file mode 100644 index 89818c96c2..0000000000 --- a/src/BenchmarkDotNet.TestingPlatform/AsyncWorkQueue.cs +++ /dev/null @@ -1,61 +0,0 @@ -using System.Collections.Concurrent; - -namespace BenchmarkDotNet.TestingPlatform -{ - /// - /// An ordered queue of asynchronous work items that are produced synchronously and consumed asynchronously. - /// - /// - /// BenchmarkDotNet reports its progress through synchronous callbacks (EventProcessor and ILogger) while the - /// platform message bus and output device are asynchronous. Blocking on those from inside a callback risks - /// deadlocking against the synchronization context BenchmarkDotNet installs while it runs, so the callbacks - /// enqueue instead and the caller of DrainAsync does the awaiting. - /// - internal sealed class AsyncWorkQueue : IDisposable - { - private readonly ConcurrentQueue> queue = new(); - private readonly SemaphoreSlim available = new(0); - private volatile bool completed; - - /// - /// Queues a work item. Safe to call from any thread. - /// - /// The work to perform. - public void Enqueue(Func work) - { - queue.Enqueue(work); - available.Release(); - } - - /// - /// Signals that no more work items will be enqueued, which lets DrainAsync return once the already queued - /// items have been processed. - /// - public void Complete() - { - completed = true; - available.Release(); - } - - /// - /// Processes queued work items in order until has been called and the queue is empty. - /// - /// A task that completes when the queue has been drained. - public async Task DrainAsync() - { - while (true) - { - await available.WaitAsync().ConfigureAwait(false); - - // Draining everything on each wake-up means a lost or surplus permit can never strand a work item. - while (queue.TryDequeue(out var work)) - await work().ConfigureAwait(false); - - if (completed && queue.IsEmpty) - return; - } - } - - public void Dispose() => available.Dispose(); - } -} From fc1a8da2d4849b41ebf4e0282c4d8a1a3ffea266 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 04:02:42 +0100 Subject: [PATCH 033/110] Refactor to use ChannelWriter for log task queuing Replaces custom AsyncWorkQueue with ChannelWriter> for queuing log display tasks. Updates constructor and field types, and switches from Enqueue to TryWrite for task scheduling. This enhances integration with .NET's built-in concurrency primitives. --- src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs b/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs index 917ca5b25e..aad7dc8fae 100644 --- a/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs +++ b/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs @@ -2,6 +2,7 @@ using Microsoft.Testing.Platform.Extensions.OutputDevice; using Microsoft.Testing.Platform.OutputDevice; using System.Text; +using System.Threading.Channels; namespace BenchmarkDotNet.TestingPlatform { @@ -13,7 +14,7 @@ internal sealed class OutputDeviceLogger : ILogger { private readonly IOutputDevice outputDevice; private readonly IOutputDeviceDataProducer producer; - private readonly AsyncWorkQueue workQueue; + private readonly ChannelWriter> workQueue; private readonly CancellationToken cancellationToken; private readonly StringBuilder currentLine = new(); private LogKind currentLineKind = LogKind.Default; @@ -21,7 +22,7 @@ internal sealed class OutputDeviceLogger : ILogger public OutputDeviceLogger( IOutputDevice outputDevice, IOutputDeviceDataProducer producer, - AsyncWorkQueue workQueue, + ChannelWriter> workQueue, CancellationToken cancellationToken) { this.outputDevice = outputDevice; @@ -59,7 +60,7 @@ public void WriteLine() _ => new TextOutputDeviceData(text) }; - workQueue.Enqueue(() => outputDevice.DisplayAsync(producer, data, cancellationToken)); + workQueue.TryWrite(() => outputDevice.DisplayAsync(producer, data, cancellationToken)); } public void WriteLine(LogKind logKind, string text) From 52d46c7a0160bbafef53cb9bcc3bfebf357788b9 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 04:02:55 +0100 Subject: [PATCH 034/110] Replace AsyncWorkQueue with Channel for event processing Switch to System.Threading.Channels for the benchmark event work queue to improve thread safety and prevent deadlocks. Update event processor and logger to use the channel writer, and add a DrainAsync method to process queued work items sequentially until completion. --- .../BenchmarkTestFramework.cs | 39 +++++++++++++++---- 1 file changed, 31 insertions(+), 8 deletions(-) diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs index 368c12dfa0..a08a70a7c4 100644 --- a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs @@ -12,6 +12,7 @@ using Microsoft.Testing.Platform.TestHost; using System.Reflection; using System.Runtime.ExceptionServices; +using System.Threading.Channels; namespace BenchmarkDotNet.TestingPlatform { @@ -118,21 +119,30 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte var nodes = runnable.ToDictionary(match => match.Node.Uid, match => match.Node); - using var workQueue = new AsyncWorkQueue(); + // BenchmarkDotNet reports its progress through synchronous callbacks (EventProcessor and ILogger) while + // the message bus and the output device are asynchronous. Blocking on those from inside a callback risks + // deadlocking against the synchronization context BenchmarkDotNet installs while it runs, so the callbacks + // write to this channel and the drain below does the awaiting. Synchronous continuations are left off, so + // that a write can never end up publishing on BenchmarkDotNet's own thread. + var workQueue = Channel.CreateUnbounded>(new UnboundedChannelOptions + { + SingleReader = true, + AllowSynchronousContinuations = false + }); // A failure while publishing has to stop the benchmarks as well, otherwise the run would carry on with - // nobody listening and would keep writing to a queue that is about to be disposed. + // nobody listening to it. using var runCancellation = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); var eventProcessor = new BenchmarkEventProcessor(nodes, testNode => { var message = new TestNodeUpdateMessage(sessionUid, testNode); - workQueue.Enqueue(() => context.MessageBus.PublishAsync(this, message)); + workQueue.Writer.TryWrite(() => context.MessageBus.PublishAsync(this, message)); }); // BenchmarkDotNet's own console output is replaced so that everything goes through the output device, // which keeps it in the right place when the platform runs in server mode or inside an IDE. - var logger = new OutputDeviceLogger(serviceProvider.GetOutputDevice(), this, workQueue, cancellationToken); + var logger = new OutputDeviceLogger(serviceProvider.GetOutputDevice(), this, workQueue.Writer, cancellationToken); var runInfos = runnable .GroupBy(match => match.RunInfo) @@ -172,7 +182,7 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte { // The drain only ends once the queue is completed, so this has to happen no matter what // else went wrong. - workQueue.Complete(); + workQueue.Writer.TryComplete(); } } }, @@ -181,7 +191,7 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte ExceptionDispatchInfo? drainFailure = null; try { - await workQueue.DrainAsync().ConfigureAwait(false); + await DrainAsync(workQueue.Reader).ConfigureAwait(false); } catch (Exception exception) { @@ -193,8 +203,8 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte try { - // BenchmarkDotNet keeps writing to the queue until its thread returns, so the run always has to be - // over before the queue is disposed. + // The run has to be over before the request completes, otherwise it would carry on in the background + // and its failure would go unobserved. await runTask.ConfigureAwait(false); } catch when (drainFailure != null) @@ -205,6 +215,19 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte drainFailure?.Throw(); } + /// + /// Runs the queued work items in order, until the queue is completed and empty. + /// + /// The reader of the work queue. + private static async Task DrainAsync(ChannelReader> reader) + { + while (await reader.WaitToReadAsync().ConfigureAwait(false)) + { + while (reader.TryRead(out var work)) + await work().ConfigureAwait(false); + } + } + /// /// Reports benchmarks that share a uid as a single failed test. /// From 8b4492e37d5f0c92ada9ad68ab5cb9c6defa3bf2 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 05:11:19 +0100 Subject: [PATCH 035/110] Switch to GetUniqueId for benchmark case identification Standardized benchmark case UID generation by replacing all usages of FullNameProvider.GetBenchmarkUid with BenchmarkCase.GetUniqueId. Removed the obsolete GetBenchmarkUid method. Updated comments for clarity. Also adjusted .slnx to control build for TestingPlatform projects. --- BenchmarkDotNet.slnx | 8 ++++++-- .../BenchmarkEventProcessor.cs | 8 ++++---- .../BenchmarkTestNode.cs | 6 ++++-- .../Exporters/FullNameProvider.cs | 17 ----------------- 4 files changed, 14 insertions(+), 25 deletions(-) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index ca8e06a6dd..ec83125a62 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -21,7 +21,9 @@ - + + + @@ -43,7 +45,9 @@ - + + + diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs index a7d50256da..9aab6ff3e8 100644 --- a/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs @@ -36,7 +36,7 @@ public override void OnValidationError(ValidationError validationError) // If the error is not linked to a benchmark case, then set the error on all benchmarks. var affected = validationError.BenchmarkCase == null ? nodes.Values - : [nodes[FullNameProvider.GetBenchmarkUid(validationError.BenchmarkCase)]]; + : [nodes[validationError.BenchmarkCase.GetUniqueId()]]; foreach (var node in affected) { @@ -62,7 +62,7 @@ public override void OnBuildComplete(BuildPartition buildPartition, BuildResult foreach (var benchmarkBuildInfo in buildPartition.Benchmarks) { - var node = nodes[FullNameProvider.GetBenchmarkUid(benchmarkBuildInfo.BenchmarkCase)]; + var node = nodes[benchmarkBuildInfo.BenchmarkCase.GetUniqueId()]; var pending = GetOrCreatePendingResult(node); if (buildResult.GenerateException != null) @@ -80,7 +80,7 @@ public override void OnBuildComplete(BuildPartition buildPartition, BuildResult public override void OnStartRunBenchmark(BenchmarkCase benchmarkCase) { - var node = nodes[FullNameProvider.GetBenchmarkUid(benchmarkCase)]; + var node = nodes[benchmarkCase.GetUniqueId()]; var pending = GetOrCreatePendingResult(node); pending.StartTime = DateTimeOffset.UtcNow; @@ -90,7 +90,7 @@ public override void OnStartRunBenchmark(BenchmarkCase benchmarkCase) public override void OnEndRunBenchmark(BenchmarkCase benchmarkCase, BenchmarkReport report) { - var node = nodes[FullNameProvider.GetBenchmarkUid(benchmarkCase)]; + var node = nodes[benchmarkCase.GetUniqueId()]; var pending = GetOrCreatePendingResult(node); pending.Duration = runTimerStopwatch.Elapsed; pending.EndTime = DateTimeOffset.UtcNow; diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs index 61d6cfb7a5..f67ed910fd 100644 --- a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs +++ b/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs @@ -66,8 +66,10 @@ public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool include var parametrizedMethodName = FullNameProvider.GetMethodName(benchmarkCase); var jobDisplayInfo = benchmarkCase.GetUnrandomizedJobDisplayInfo(); - // Unlike the uid, the job is only part of the display name when it actually adds information. - var uid = FullNameProvider.GetBenchmarkUid(benchmarkCase); + // The uid is the hash BenchmarkDotNet itself uses (and reports through `--list json`), so that a benchmark + // keeps the same identity across processes and across tools. The job is only part of the display name + // when it actually adds information. + var uid = benchmarkCase.GetUniqueId(); var displayName = $"{fullClassName}.{parametrizedMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); var properties = new List diff --git a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs index 79de14dc1c..19555908d2 100644 --- a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs +++ b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs @@ -63,23 +63,6 @@ public static string GetBenchmarkName(BenchmarkCase benchmarkCase) return name.ToString(); } - /// - /// Gets an identifier of a benchmark case that stays the same across processes, which is what lets a benchmark - /// discovered in one process be selected for execution in another (for example by a test adapter). - /// - /// The benchmark case to identify. - /// The unique identifier of the benchmark case. - /// - /// The job is always part of the uid, otherwise two cases of the same benchmark that only differ by job would - /// collide. The parameters are already part of the method name. - /// - [PublicAPI] - public static string GetBenchmarkUid(BenchmarkCase benchmarkCase) - { - var fullClassName = benchmarkCase.Descriptor.Type.GetCorrectCSharpTypeName(prefixWithGlobal: false); - return $"{fullClassName}.{GetMethodName(benchmarkCase)} [{benchmarkCase.GetUnrandomizedJobDisplayInfo()}]"; - } - private static string GetNestedTypes(Type type) { string nestedTypes = ""; From 9f3298023851eb494a362d8e477d00c9965996dc Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:24:02 +0100 Subject: [PATCH 036/110] Move TestingPlatform config to .targets; cleanup props/csproj Refactor BenchmarkDotNet.TestingPlatform integration by moving configuration logic from .props to BenchmarkDotNet.TestAdapter.targets. Remove obsolete .props and .csproj files. Update cSpell dictionary to use "testadapter" instead of "testingplatform". --- build/cSpell.json | 2 +- .../{testingplatform.md => testadapter.md} | 0 .../BenchmarkDotNetExtension.cs | 0 .../BenchmarkEventProcessor.cs | 0 .../BenchmarkTestFramework.cs | 0 .../TestingPlatform}/BenchmarkTestNode.cs | 0 .../TestingPlatform}/OutputDeviceLogger.cs | 0 .../TestApplicationBuilderExtensions.cs | 0 .../TestingPlatformBuilderHook.cs | 0 .../build/BenchmarkDotNet.TestAdapter.targets | 34 ++++++++++++++ .../BenchmarkDotNet.TestingPlatform.csproj | 47 ------------------- .../BenchmarkDotNet.TestingPlatform.props | 31 ------------ 12 files changed, 35 insertions(+), 79 deletions(-) rename docs/articles/features/{testingplatform.md => testadapter.md} (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/BenchmarkDotNetExtension.cs (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/BenchmarkEventProcessor.cs (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/BenchmarkTestFramework.cs (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/BenchmarkTestNode.cs (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/OutputDeviceLogger.cs (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/TestApplicationBuilderExtensions.cs (100%) rename src/{BenchmarkDotNet.TestingPlatform => BenchmarkDotNet.TestAdapter/TestingPlatform}/TestingPlatformBuilderHook.cs (100%) create mode 100644 src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets delete mode 100644 src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj delete mode 100644 src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props diff --git a/build/cSpell.json b/build/cSpell.json index 4cb6c27117..db68aadb88 100644 --- a/build/cSpell.json +++ b/build/cSpell.json @@ -34,7 +34,7 @@ "vsprofiler", "vstest", "Tailcall", - "testingplatform", + "testadapter", "toolchains", "unmanaged" ], diff --git a/docs/articles/features/testingplatform.md b/docs/articles/features/testadapter.md similarity index 100% rename from docs/articles/features/testingplatform.md rename to docs/articles/features/testadapter.md diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNetExtension.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkDotNetExtension.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNetExtension.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkDotNetExtension.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/BenchmarkEventProcessor.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/BenchmarkTestFramework.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/BenchmarkTestNode.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/OutputDeviceLogger.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/OutputDeviceLogger.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/OutputDeviceLogger.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/TestApplicationBuilderExtensions.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/TestApplicationBuilderExtensions.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs diff --git a/src/BenchmarkDotNet.TestingPlatform/TestingPlatformBuilderHook.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestingPlatformBuilderHook.cs similarity index 100% rename from src/BenchmarkDotNet.TestingPlatform/TestingPlatformBuilderHook.cs rename to src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestingPlatformBuilderHook.cs diff --git a/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets new file mode 100644 index 0000000000..339a7fd56b --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets @@ -0,0 +1,34 @@ + + + + + true + false + + + true + + + + + + BenchmarkDotNet + BenchmarkDotNet.TestAdapter.TestingPlatform.TestingPlatformBuilderHook + + + diff --git a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj b/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj deleted file mode 100644 index 7bcea6e705..0000000000 --- a/src/BenchmarkDotNet.TestingPlatform/BenchmarkDotNet.TestingPlatform.csproj +++ /dev/null @@ -1,47 +0,0 @@ - - - - netstandard2.0 - BenchmarkDotNet.TestingPlatform - BenchmarkDotNet.TestingPlatform - BenchmarkDotNet.TestingPlatform - Runs BenchmarkDotNet benchmarks as tests through Microsoft.Testing.Platform - README.md - True - BenchmarkDotNet.TestingPlatform - - - false - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props b/src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props deleted file mode 100644 index 5f1a9412ef..0000000000 --- a/src/BenchmarkDotNet.TestingPlatform/build/BenchmarkDotNet.TestingPlatform.props +++ /dev/null @@ -1,31 +0,0 @@ - - - - true - - - true - - - false - - - - - - BenchmarkDotNet - BenchmarkDotNet.TestingPlatform.TestingPlatformBuilderHook - - - From 1f73e5b3979769d7ed253acc1bf321f84dd4b308 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:39:23 +0100 Subject: [PATCH 037/110] Update docs for new TestAdapter and MTP integration Rewrote and reorganized documentation to focus on the new BenchmarkDotNet.TestAdapter package and its integration with Microsoft.Testing.Platform (MTP) and VSTest. Clarified default behaviors, entry point handling, and configuration steps. Updated code samples and project file snippets. Revised table of contents to reflect the new structure and clarified the relationship between MTP and VSTest. Added notes on IDE support and caveats. --- docs/articles/features/testadapter.md | 86 ++++++++++++++++++--------- docs/articles/features/toc.yml | 6 +- docs/articles/features/vstest.md | 26 +++++--- 3 files changed, 79 insertions(+), 39 deletions(-) diff --git a/docs/articles/features/testadapter.md b/docs/articles/features/testadapter.md index a54ccab78e..90e159768b 100644 --- a/docs/articles/features/testadapter.md +++ b/docs/articles/features/testadapter.md @@ -1,33 +1,32 @@ --- -uid: docs.testingplatform -name: Running with Microsoft.Testing.Platform +uid: docs.testadapter +name: Running benchmarks as tests --- -# Running with Microsoft.Testing.Platform +# Running benchmarks as tests -BenchmarkDotNet can discover and execute benchmarks through - [Microsoft.Testing.Platform](https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-intro) (MTP), - the test platform that succeeds VSTest. -This gives you the same "benchmarks as tests" experience as [the VSTest adapter](xref:docs.vstest), - but on the platform that `dotnet test` and modern IDE integrations are moving to. +`BenchmarkDotNet.TestAdapter` lets your IDE and `dotnet test` discover and execute your benchmarks the way they do + unit tests. +This provides an alternative user experience to running benchmarks with the CLI + and may be preferable for those who like their IDE's test integrations that they may have used when running unit tests. -If you are looking for the VSTest adapter, see [Running with VSTest](xref:docs.vstest) instead. -You only need one of the two. +Below is an example of running some benchmarks from the BenchmarkDotNet samples project in Visual Studio's Test Explorer. -## VSTest or Microsoft.Testing.Platform? +![](../../images/vs-testexplorer-demo.png) -The two adapters solve the same problem on different platforms, and the difference that matters most is *where your - benchmarks run*: +The adapter supports two test platforms: -* With **VSTest**, an external `testhost` process loads your benchmark assembly and the adapter reflects into it. -* With **Microsoft.Testing.Platform**, your benchmark project *is* the test host. - There is no separate host process, so BenchmarkDotNet behaves exactly as it does when you run the app from the CLI. +* [Microsoft.Testing.Platform](https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-intro) (MTP), + the platform that `dotnet test` and modern IDE integrations are moving to. **This is the default.** +* [VSTest](xref:docs.vstest), for tooling without Microsoft.Testing.Platform support (such as Visual Studio 2019) + and for solutions that mix benchmark projects with VSTest based test projects. -The practical consequences of the MTP model are: +The difference that matters most is *where your benchmarks run*: -* The adapter no longer needs the child `AppDomain` that the VSTest adapter uses to load your assemblies correctly. -* Your project's entry point is generated by the platform and starts the test application, - so it no longer calls `BenchmarkSwitcher`. See [Keeping a BenchmarkSwitcher entry point](#keeping-a-benchmarkswitcher-entry-point). +* With **VSTest**, an external `testhost` process loads your benchmark assembly and the adapter reflects into it. +* With **Microsoft.Testing.Platform**, your benchmark project *is* the test host. + There is no separate host process, so BenchmarkDotNet behaves exactly as it does when you run the app from the CLI, + and the adapter does not need the child `AppDomain` that the VSTest adapter uses to load your assemblies correctly. ## Caveats and things to know @@ -44,7 +43,8 @@ The practical consequences of the MTP model are: you will need to do this through other means such as an assembly-level `IConfigSource`, as shown in [Setting a default configuration](xref:docs.vstest#setting-a-default-configuration). * **The adapter will generate an entry point for you automatically.** - Unlike the VSTest adapter, the generated entry point starts the test application rather than `BenchmarkSwitcher`. + The generated entry point starts the test application. + See [Keeping your own entry point](#keeping-your-own-entry-point) if your project already has one. ## Getting started @@ -53,7 +53,7 @@ The practical consequences of the MTP model are: ```xml - + ``` @@ -72,7 +72,7 @@ The practical consequences of the MTP model are: - + @@ -128,28 +128,39 @@ dotnet run -c Release -- --treenode-filter "/*/*/MyBenchmarks/*" # Run every benchmark of a category. dotnet run -c Release -- --treenode-filter "/*/*/*/*[Category=Fast]" - -# Run one specific benchmark, by the exact id reported by the platform. -dotnet run -c Release -- --filter-uid "MyProject.MyBenchmarks.Add(x: 1) [DefaultJob]" ``` The tree node filter path is `////`, and `[BenchmarkCategory]` attributes are exposed as a `Category` trait that the filter can match on. -## Keeping a BenchmarkSwitcher entry point +## Keeping your own entry point The generated entry point starts the test application, which means it replaces the `BenchmarkSwitcher` entry point that a benchmark project normally has. -If you want to keep your own entry point, turn off the generated one and register BenchmarkDotNet yourself: +There are two ways to keep your own. + +To keep a plain `BenchmarkSwitcher` entry point, tell the adapter that the project generates its own: + +```xml + + + false + +``` + +The project is then a normal console application again, and no test platform integration is set up for it. + +To keep an entry point *and* the test integration, start the test application yourself: ```xml + true false ``` ```csharp -using BenchmarkDotNet.TestingPlatform; +using BenchmarkDotNet.TestAdapter.TestingPlatform; using Microsoft.Testing.Platform.Builder; public static class Program @@ -167,6 +178,23 @@ public static class Program From there you are free to decide when to start the test application and when to hand over to `BenchmarkSwitcher`, for example by looking at the arguments your CI passes. +## Using VSTest instead + +Set `BenchmarkDotNetUseVSTest` and add the VSTest host package: + +```xml + + true + + + + + + +``` + +See [Running with VSTest](xref:docs.vstest) for the details, including the IDE settings that VSTest integration needs. + ## Viewing the results The full BenchmarkDotNet output, including the summary table that compares benchmarks with each other, diff --git a/docs/articles/features/toc.yml b/docs/articles/features/toc.yml index f2fb0728d2..db72e5c3e2 100644 --- a/docs/articles/features/toc.yml +++ b/docs/articles/features/toc.yml @@ -16,7 +16,7 @@ href: event-pipe-profiler.md - name: VSProfiler href: vsprofiler.md +- name: Benchmarks as tests + href: testadapter.md - name: VSTest - href: vstest.md -- name: Microsoft.Testing.Platform - href: testingplatform.md \ No newline at end of file + href: vstest.md \ No newline at end of file diff --git a/docs/articles/features/vstest.md b/docs/articles/features/vstest.md index e86527211c..2e0b3368d3 100644 --- a/docs/articles/features/vstest.md +++ b/docs/articles/features/vstest.md @@ -6,9 +6,9 @@ name: Running with VSTest # Running with VSTest > [!NOTE] -> BenchmarkDotNet also ships an adapter for [Microsoft.Testing.Platform](xref:docs.testingplatform), -> the test platform that succeeds VSTest. -> You only need one of the two. +> `BenchmarkDotNet.TestAdapter` runs your benchmarks through +> [Microsoft.Testing.Platform](xref:docs.testadapter) by default, which is the platform that succeeds VSTest. +> VSTest is opt-in, as described below. BenchmarkDotNet supports discovering and executing benchmarks through VSTest. This provides an alternative user experience to running benchmarks with the CLI @@ -59,7 +59,17 @@ In addition, we can still make use of this boolean output to indicate You need to install two packages into your benchmark project: * `BenchmarkDotNet.TestAdapter`: Implements the VSTest protocol for BenchmarkDotNet * `Microsoft.NET.Test.Sdk`: Includes all the pieces needed for the VSTest host to run and load the VSTest adapter. -* **Step 2.** Make sure that the entry point is configured correctly. +* **Step 2.** Ask the adapter for VSTest. + `BenchmarkDotNet.TestAdapter` uses [Microsoft.Testing.Platform](xref:docs.testadapter) unless you set + `BenchmarkDotNetUseVSTest` in your project file: + +```xml + + true + +``` + +* **Step 3.** Make sure that the entry point is configured correctly. As mentioned in the caveats section, `BenchmarkDotNet.TestAdapter` will generate an entry point for you automatically. So, if you have an entry point already, you will either need to delete it or set `GenerateProgramFile` to `false` in your project file to continue using your existing one. @@ -73,6 +83,8 @@ In addition, we can still make use of this boolean output to indicate net8.0 enable enable + + true false @@ -85,7 +97,7 @@ In addition, we can still make use of this boolean output to indicate ``` -* **Step 3.** Make sure that your IDE supports VSTest integration. +* **Step 4.** Make sure that your IDE supports VSTest integration. In Visual Studio, everything works out of the box. In Rider/R#, the VSTest integration might need to be activated: * Go to the "Unit Testing" settings page. @@ -93,9 +105,9 @@ In addition, we can still make use of this boolean output to indicate * R#: Extensions -> ReSharper -> Options -> Tools -> Unit Testing -> Test Frameworks -> VSTest * Make sure that the "Enable VSTest adapter support" checkbox is checked. In recent versions of Rider, this may be enabled by default. -* **Step 4.** Switch to the `Release` configuration. +* **Step 5.** Switch to the `Release` configuration. As mentioned above, the TestAdapter is not able to discover and run benchmarks with optimizations disabled (by design). -* **Step 5.** Build the project. +* **Step 6.** Build the project. In order to discover the benchmarks, the VSTest adapter needs to be able to find the assembly. Once you build the project, you should observe the discovered benchmarks in your IDE's Unit Test Explorer. From 68be1e28b006c4ed41404c37b2320fa8ffe88533 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:39:52 +0100 Subject: [PATCH 038/110] Update namespaces and extension UID for adapter alignment Refactor namespaces from BenchmarkDotNet.TestingPlatform to BenchmarkDotNet.TestAdapter.TestingPlatform throughout the codebase. Update the extension UID and related comments to match the new adapter naming convention and integration targets. --- .../TestingPlatform/BenchmarkDotNetExtension.cs | 8 ++++---- .../TestingPlatform/BenchmarkEventProcessor.cs | 2 +- .../TestingPlatform/BenchmarkTestFramework.cs | 4 +--- .../TestingPlatform/BenchmarkTestNode.cs | 2 +- .../TestingPlatform/OutputDeviceLogger.cs | 2 +- .../TestingPlatform/TestApplicationBuilderExtensions.cs | 2 +- .../TestingPlatform/TestingPlatformBuilderHook.cs | 4 ++-- 7 files changed, 11 insertions(+), 13 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkDotNetExtension.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkDotNetExtension.cs index d2bc8722d8..5f6a54403e 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkDotNetExtension.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkDotNetExtension.cs @@ -1,6 +1,6 @@ using Microsoft.Testing.Platform.Extensions; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// /// Identifies BenchmarkDotNet to Microsoft.Testing.Platform. @@ -14,11 +14,11 @@ internal sealed class BenchmarkDotNetExtension : IExtension /// /// The uid shared by every extension this package registers. /// - public const string ExtensionUid = "BenchmarkDotNet.TestingPlatform"; + public const string ExtensionUid = "BenchmarkDotNet.TestAdapter"; /// public string Uid => ExtensionUid; - + /// public string Version => typeof(BenchmarkDotNetExtension).Assembly.GetName().Version?.ToString() ?? "0.0.0"; @@ -26,7 +26,7 @@ internal sealed class BenchmarkDotNetExtension : IExtension public string DisplayName => "BenchmarkDotNet"; /// - public string Description => "Runs BenchmarkDotNet benchmarks as tests."; + public string Description => "Runs BenchmarkDotNet benchmarks as tests."; /// public Task IsEnabledAsync() => Task.FromResult(true); diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs index 9aab6ff3e8..523f7a89a9 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs @@ -11,7 +11,7 @@ using System.Globalization; using System.Text; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs index a08a70a7c4..bdce27cd9d 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs @@ -1,8 +1,6 @@ using BenchmarkDotNet.Configs; using BenchmarkDotNet.Loggers; using BenchmarkDotNet.Running; -// BenchmarkEnumerator is compiled into this assembly from the VSTest adapter, where it keeps its own namespace. -using BenchmarkDotNet.TestAdapter; using Microsoft.Testing.Platform.Capabilities.TestFramework; using Microsoft.Testing.Platform.Extensions.Messages; using Microsoft.Testing.Platform.Extensions.OutputDevice; @@ -14,7 +12,7 @@ using System.Runtime.ExceptionServices; using System.Threading.Channels; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// /// Discovers and executes the benchmarks of the running assembly through Microsoft.Testing.Platform. diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs index f67ed910fd..14f6bd171e 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs @@ -6,7 +6,7 @@ using System.Reflection; using System.Text; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// /// The Microsoft.Testing.Platform view of a single . diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/OutputDeviceLogger.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/OutputDeviceLogger.cs index aad7dc8fae..a2d1b0d2b4 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/OutputDeviceLogger.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/OutputDeviceLogger.cs @@ -4,7 +4,7 @@ using System.Text; using System.Threading.Channels; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// /// Forwards the BenchmarkDotNet log to the platform output device, so that build progress and the result summary diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs index 9df284d376..0f72cb90c6 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs @@ -3,7 +3,7 @@ using Microsoft.Testing.Platform.Helpers; using System.Reflection; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// /// Extensions for registering BenchmarkDotNet with a Microsoft.Testing.Platform application. diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestingPlatformBuilderHook.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestingPlatformBuilderHook.cs index 5e77ef6b88..bc7c7db03c 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestingPlatformBuilderHook.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestingPlatformBuilderHook.cs @@ -1,13 +1,13 @@ using Microsoft.Testing.Platform.Builder; using System.ComponentModel; -namespace BenchmarkDotNet.TestingPlatform +namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// /// The hook Microsoft.Testing.Platform.MSBuild calls from the entry point it generates for the benchmark project. /// /// - /// This is wired up by the TestingPlatformBuilderHook item in BenchmarkDotNet.TestingPlatform.props. + /// This is wired up by the TestingPlatformBuilderHook item in BenchmarkDotNet.TestAdapter.targets. /// It is public because the generated code lives in the benchmark assembly, but it is not meant to be called /// directly; use /// instead when writing an entry point by hand. From 9deb942cbb07465dc4518bcac6f891b92548fae6 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:40:23 +0100 Subject: [PATCH 039/110] Manually import TestAdapter build files in sample projects Added explicit imports for BenchmarkDotNet.TestAdapter .props and .targets files in both F# and C# sample projects to ensure adapter build logic is applied. Also imported common.targets. This preserves custom entry points and prevents conversion to Microsoft.Testing.Platform applications. --- .../BenchmarkDotNet.Samples.FSharp.fsproj | 8 ++++++++ .../BenchmarkDotNet.Samples.csproj | 8 ++++++++ 2 files changed, 16 insertions(+) diff --git a/samples/BenchmarkDotNet.Samples.FSharp/BenchmarkDotNet.Samples.FSharp.fsproj b/samples/BenchmarkDotNet.Samples.FSharp/BenchmarkDotNet.Samples.FSharp.fsproj index eeee8b90f2..fd51c0e8bf 100644 --- a/samples/BenchmarkDotNet.Samples.FSharp/BenchmarkDotNet.Samples.FSharp.fsproj +++ b/samples/BenchmarkDotNet.Samples.FSharp/BenchmarkDotNet.Samples.FSharp.fsproj @@ -25,5 +25,13 @@ + + + + diff --git a/samples/BenchmarkDotNet.Samples/BenchmarkDotNet.Samples.csproj b/samples/BenchmarkDotNet.Samples/BenchmarkDotNet.Samples.csproj index 4aa0c95e8c..ffa2cd63d3 100644 --- a/samples/BenchmarkDotNet.Samples/BenchmarkDotNet.Samples.csproj +++ b/samples/BenchmarkDotNet.Samples/BenchmarkDotNet.Samples.csproj @@ -38,5 +38,13 @@ + + + + From 9c81559e3e8f89e9f1f6088a85a98b18cde473ee Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:40:41 +0100 Subject: [PATCH 040/110] Update project to use BenchmarkDotNet.TestAdapter Switched project reference and build file imports from BenchmarkDotNet.TestingPlatform to BenchmarkDotNet.TestAdapter, including .props and .targets files. --- ...BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj index 59913fd0b3..6cea87bd71 100644 --- a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj @@ -9,11 +9,12 @@ - + - - + + + From 8b128f593a4b152806169426e39a5e63cbf45e79 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:40:57 +0100 Subject: [PATCH 041/110] Update packaging and entry point logic for test adapter Add NuGet description and set IsTestingPlatformApplication to false to avoid treating the adapter as a test app. Add Microsoft.Testing.Platform.MSBuild as a dependency for downstream projects. Update .props and .targets packaging for cross-platform compatibility. Only generate entry point for VSTest scenarios; clarify comments. --- .../BenchmarkDotNet.TestAdapter.csproj | 20 ++++++++++++++++++- .../build/BenchmarkDotNet.TestAdapter.props | 14 ++++++++----- 2 files changed, 28 insertions(+), 6 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkDotNet.TestAdapter.csproj b/src/BenchmarkDotNet.TestAdapter/BenchmarkDotNet.TestAdapter.csproj index 3deb70b1cd..7cdb59410a 100644 --- a/src/BenchmarkDotNet.TestAdapter/BenchmarkDotNet.TestAdapter.csproj +++ b/src/BenchmarkDotNet.TestAdapter/BenchmarkDotNet.TestAdapter.csproj @@ -5,8 +5,16 @@ BenchmarkDotNet.TestAdapter BenchmarkDotNet.TestAdapter BenchmarkDotNet.TestAdapter + Runs BenchmarkDotNet benchmarks as tests, through Microsoft.Testing.Platform or VSTest README.md True + + + false @@ -20,6 +28,13 @@ + + + + + @@ -28,7 +43,10 @@ - + + + diff --git a/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.props b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.props index 7184842aba..5d1eabd4f6 100644 --- a/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.props +++ b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.props @@ -1,15 +1,19 @@ - + $(MSBuildThisFileDirectory)..\entrypoints\ false - - + From bd91fe3638c6ff970adc151c3366caf306287a4c Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 16 Aug 2026 21:41:19 +0100 Subject: [PATCH 042/110] Remove BenchmarkDotNet.TestingPlatform project Removed BenchmarkDotNet.TestingPlatform from the solution and deleted its InternalsVisibleTo entry from AssemblyInfo.cs, as it no longer requires access to internal members. No other InternalsVisibleTo changes were made. --- BenchmarkDotNet.slnx | 3 --- src/BenchmarkDotNet/Properties/AssemblyInfo.cs | 3 +-- 2 files changed, 1 insertion(+), 5 deletions(-) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index ec83125a62..f681138fe7 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -21,9 +21,6 @@ - - - diff --git a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs index 7bd16f9307..4b0041b9e3 100644 --- a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs +++ b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs @@ -14,5 +14,4 @@ [assembly: InternalsVisibleTo("BenchmarkDotNet.Diagnostics.dotMemory,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning.MultipleFrameworks,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] -[assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] -[assembly: InternalsVisibleTo("BenchmarkDotNet.TestingPlatform,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] +[assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] From fe3143e2f8c670050dbb1028331b86b41c6dc91a Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 22 Aug 2026 16:55:55 +0100 Subject: [PATCH 043/110] Simplify project entry in BenchmarkDotNet.slnx Replaced the explicit configuration for BenchmarkDotNet.IntegrationTests.TestingPlatform with a standard entry to align with the format used for other projects in the solution. --- BenchmarkDotNet.slnx | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index f681138fe7..c3c733fb82 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -42,9 +42,7 @@ - - - + From 857dc665d3f2213c11466e11dab3eb9987a9d1ea Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 22 Aug 2026 18:32:28 +0100 Subject: [PATCH 044/110] Update assistant introduction message Replaced default introduction with a personalized message identifying as GitHub Copilot and offering software development assistance. --- .../Exporters/FullNameProvider.cs | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs index 19555908d2..5411f7243b 100644 --- a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs +++ b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs @@ -99,6 +99,22 @@ internal static string GetMethodName(BenchmarkCase benchmarkCase) return name.ToString(); } + /// + /// Gets the method name to show to a user, which is the [Benchmark(Description = ...)] when one is set and + /// the method name otherwise, followed by the parameters. + /// + /// The benchmark case. + /// The method name to display. + internal static string GetMethodDisplayName(BenchmarkCase benchmarkCase) + { + var name = new StringBuilder(benchmarkCase.Descriptor.WorkloadMethodDisplayInfo); + + if (benchmarkCase.HasParameters) + name.Append(GetBenchmarkParameters(benchmarkCase.Descriptor.WorkloadMethod, benchmarkCase.Parameters)); + + return name.ToString(); + } + private static string GetBenchmarkParameters(MethodInfo method, ParameterInstances benchmarkParameters) { var methodArguments = method.GetParameters(); From 580eb353eff527e25974ca149c0b03933ebe7732 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 22 Aug 2026 18:33:05 +0100 Subject: [PATCH 045/110] Refactor method display name handling in BenchmarkDotNet Refactored the logic for generating method display names in BenchmarkDotNet. Improved separation of concerns by extracting display name generation to a dedicated provider. Updated the display name formatting to include job information conditionally. Enhanced maintainability and clarity in the test adapter's method identification process. --- .../TestingPlatform/BenchmarkTestNode.cs | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs index 14f6bd171e..87b9b820a5 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs @@ -70,7 +70,13 @@ public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool include // keeps the same identity across processes and across tools. The job is only part of the display name // when it actually adds information. var uid = benchmarkCase.GetUniqueId(); - var displayName = $"{fullClassName}.{parametrizedMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); + + // Microsoft.Testing.Platform keeps the display name and the identity apart, so the name is free to be the + // [Benchmark(Description = ...)] the author chose. GetMethodDisplayName falls back to the method name when + // no description is set. The path keeps the method name, so that a filter still matches what + // BenchmarkDotNet's own --filter matches. + var displayMethodName = FullNameProvider.GetMethodDisplayName(benchmarkCase); + var displayName = $"{fullClassName}.{displayMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); var properties = new List { From 5311e65ec7c8d89ffc302b19926ddd440e1dab31 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 22 Aug 2026 18:33:26 +0100 Subject: [PATCH 046/110] Add DescribedProbe benchmark class with custom config Introduced DescribedProbe to test benchmarks with and without custom descriptions. Includes FastConfig for quick in-process execution and a configurable Size parameter. --- .../DescribedProbe.cs | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DescribedProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DescribedProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DescribedProbe.cs new file mode 100644 index 0000000000..72922bc2a7 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DescribedProbe.cs @@ -0,0 +1,28 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// Benchmarks whose display name comes from the description rather than the method name. + /// + [Config(typeof(FastConfig))] + public class DescribedProbe + { + [Params(1)] + public int Size { get; set; } + + [Benchmark(Description = "A described benchmark")] + public int Described() => Size; + + [Benchmark] + public int Undescribed() => Size; + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} From fdeac602c0ee7bb8a49ff3eedff177210c439f74 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 30 Aug 2026 18:50:02 +0100 Subject: [PATCH 047/110] Add BenchmarkDotNet.IntegrationTests.TestingPlatform to tests Added tests/BenchmarkDotNet.IntegrationTests.TestingPlatform to run-tests-selected.yaml. Included a comment clarifying that this project is a Microsoft.Testing.Platform application, relies on global.json for dotnet test routing, and requires the workflow to set the working directory for correct resolution. --- .github/workflows/run-tests-selected.yaml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/run-tests-selected.yaml b/.github/workflows/run-tests-selected.yaml index 0e2f436810..e4e8be56fe 100644 --- a/.github/workflows/run-tests-selected.yaml +++ b/.github/workflows/run-tests-selected.yaml @@ -26,6 +26,10 @@ on: - tests/BenchmarkDotNet.IntegrationTests - tests/BenchmarkDotNet.IntegrationTests.ManualRunning - tests/BenchmarkDotNet.IntegrationTests.ManualRunning.MultipleFrameworks + # This one is a Microsoft.Testing.Platform application. It carries the `global.json` which routes + # `dotnet test` to the platform, and that file is resolved from the working directory, which this + # workflow sets to the project. + - tests/BenchmarkDotNet.IntegrationTests.TestingPlatform - samples/BenchmarkDotNet.Samples framework: type: choice From b6e386d1d92a9c4bd549cc50dc5a0f8826e981a8 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 30 Aug 2026 18:51:56 +0100 Subject: [PATCH 048/110] Format: Re-add InternalsVisibleTo line without changes No functional changes; removed and immediately re-added the InternalsVisibleTo attribute line for formatting consistency. --- src/BenchmarkDotNet/Properties/AssemblyInfo.cs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs index 4b0041b9e3..aa93984520 100644 --- a/src/BenchmarkDotNet/Properties/AssemblyInfo.cs +++ b/src/BenchmarkDotNet/Properties/AssemblyInfo.cs @@ -14,4 +14,4 @@ [assembly: InternalsVisibleTo("BenchmarkDotNet.Diagnostics.dotMemory,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] [assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.ManualRunning.MultipleFrameworks,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] -[assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] +[assembly: InternalsVisibleTo("BenchmarkDotNet.TestAdapter,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] From 32a3793e143cdf01af7f889e7205345b50994694 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 30 Aug 2026 18:52:11 +0100 Subject: [PATCH 049/110] Improve MSBuild property logic in TestAdapter.targets Update logic for IsTestingPlatformApplication and GenerateProgramFile to ensure correct opt-out handling for Microsoft.Testing.Platform. Explicitly set IsTestingPlatformApplication to false when BenchmarkDotNetUseVSTest is true or GenerateProgramFile is false, and default to true otherwise. Set GenerateProgramFile to false when IsTestingPlatformApplication is true to prevent entry point conflicts. Add comments to clarify the changes. --- .../build/BenchmarkDotNet.TestAdapter.targets | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets index 339a7fd56b..09d71557cf 100644 --- a/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets +++ b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets @@ -13,11 +13,26 @@ * GenerateProgramFile, which is how a project says it already has an entry point of its own, typically one that calls BenchmarkSwitcher. Generating a second one would not compile. --> - true - false + + false + true true + + + false + + + net10.0 + Exe + enable + enable + + $(MSBuildThisFileDirectory)..\..\..\artifacts + + + + + + + + From f1b10dee9b7162887dd25197c0b6fe017e7210c9 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:57:28 +0100 Subject: [PATCH 056/110] Add .csproj for TestingPlatform.Failures integration tests A new BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures.csproj was added targeting .NET 10.0 as an executable. The project includes assembly metadata, enforces code optimization for consistent benchmarks, and references BenchmarkDotNet.TestAdapter with manual imports of its .props and .targets files. Common build property and target files are also imported to ensure correct MSBuild behavior. --- ...ationTests.TestingPlatform.Failures.csproj | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures.csproj diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures.csproj b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures.csproj new file mode 100644 index 0000000000..610c70c194 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures.csproj @@ -0,0 +1,32 @@ + + + + net10.0 + Exe + BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures + BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures + BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures + + + true + + + + + + + + + + + + From 5ff52b24a947f677eef896482923bb5e88d0a28c Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:57:46 +0100 Subject: [PATCH 057/110] Add SeparatorProbe benchmark with FastConfig setup Added SeparatorProbe class in BenchmarkDotNet.IntegrationTests.TestingPlatform. This benchmark uses a parameter with '/' as a tree separator, includes a Length() method, and applies a custom FastConfig with a dry job and InProcessEmitToolchain for faster execution. --- .../SeparatorProbe.cs | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SeparatorProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SeparatorProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SeparatorProbe.cs new file mode 100644 index 0000000000..ab01cae901 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/SeparatorProbe.cs @@ -0,0 +1,26 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// A benchmark whose parameter contains the character Microsoft.Testing.Platform uses to separate the levels of the + /// tree a --treenode-filter walks. It has to stay at the same level of that tree as every other benchmark. + /// + [Config(typeof(FastConfig))] + public class SeparatorProbe + { + [Params("a/b")] + public string Value { get; set; } = ""; + + [Benchmark] + public int Length() => Value.Length; + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} From 53c3d60756c3e617da4a41889cd89cccee50b4a9 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:58:00 +0100 Subject: [PATCH 058/110] Add BuildFailureProbe for build failure testing Introduce BuildFailureProbe in BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures. This class uses a custom toolchain (FailingBuildConfig) with a NoopGenerator, FailingBuilder, and UnreachableExecutor to reliably simulate build failures for adapter testing, without relying on uncompilable code. --- .../BuildFailureProbe.cs | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BuildFailureProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BuildFailureProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BuildFailureProbe.cs new file mode 100644 index 0000000000..3665f8592a --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BuildFailureProbe.cs @@ -0,0 +1,50 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Loggers; +using BenchmarkDotNet.Running; +using BenchmarkDotNet.Toolchains; +using BenchmarkDotNet.Toolchains.Parameters; +using BenchmarkDotNet.Toolchains.Results; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures +{ + /// + /// A benchmark whose build always fails, so that the adapter has to turn the build failure into a failed test. + /// The failure is faked by the toolchain rather than by uncompilable code, which keeps it quick and keeps the + /// error message the test asserts on under this file's control. + /// + [Config(typeof(FailingBuildConfig))] + public class BuildFailureProbe + { + internal const string ErrorMessage = "The build of this benchmark always fails, on purpose."; + + [Benchmark] + public int Add() => 1 + 1; + + private class FailingBuildConfig : ManualConfig + { + public FailingBuildConfig() + => AddJob(Job.Dry.WithToolchain(new Toolchain("FailingBuild", new NoopGenerator(), new FailingBuilder(), new UnreachableExecutor()))); + } + + private sealed class NoopGenerator : IGenerator + { + public ValueTask GenerateProjectAsync(BuildPartition buildPartition, ILogger logger, string rootArtifactsFolderPath, CancellationToken cancellationToken) + => new(GenerateResult.Success(ArtifactsPaths.Empty, [])); + } + + private sealed class FailingBuilder : IBuilder + { + public ValueTask BuildAsync(GenerateResult generateResult, BuildPartition buildPartition, ILogger logger, CancellationToken cancellationToken) + => new(BuildResult.Failure(generateResult, ErrorMessage)); + } + + private sealed class UnreachableExecutor : IExecutor + { + // A benchmark that failed to build is never executed. + public ValueTask ExecuteAsync(ExecuteParameters executeParameters, CancellationToken cancellationToken) + => throw new InvalidOperationException("The benchmark should never have been executed."); + } + } +} From c30b2ce0f19bf3dbb8ab053582c6b2f0872d3235 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:58:17 +0100 Subject: [PATCH 059/110] Add CollisionProbe test for parameter collision handling Added CollisionProbe class to test BenchmarkDotNet's behavior when benchmark parameters have identical string representations, using a custom Ambiguous type. Ensures the adapter reports collisions instead of running ambiguous benchmarks. --- .../CollisionProbe.cs | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/CollisionProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/CollisionProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/CollisionProbe.cs new file mode 100644 index 0000000000..f40a600703 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/CollisionProbe.cs @@ -0,0 +1,36 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures +{ + /// + /// Two benchmark cases that BenchmarkDotNet identifies as one, because a benchmark is identified by the string + /// representation of its parameters and both values stringify the same way. The platform cannot tell the two apart + /// either, so the adapter is expected to report the collision instead of running them. + /// + [Config(typeof(FastConfig))] + public class CollisionProbe + { + public IEnumerable Values => [new Ambiguous(1), new Ambiguous(2)]; + + [ParamsSource(nameof(Values))] + public Ambiguous? Value { get; set; } + + [Benchmark] + public int Identity() => Value!.Number; + + public class Ambiguous(int number) + { + public int Number { get; } = number; + + public override string ToString() => "ambiguous"; + } + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} From 8b4994ede2a0bcb8689a978ecb4d5721e019c271 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:58:32 +0100 Subject: [PATCH 060/110] Add OutOfProcessProbe for out-of-process benchmark test Added OutOfProcessProbe class in BenchmarkDotNet.IntegrationTests.TestingPlatform with an Add() benchmark method. Configured to run out-of-process using a custom OutOfProcessConfig and Job.Dry to ensure a real build/execute cycle for adapter testing, unlike in-process probes. --- .../OutOfProcessProbe.cs | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/OutOfProcessProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/OutOfProcessProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/OutOfProcessProbe.cs new file mode 100644 index 0000000000..cba4a1748b --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/OutOfProcessProbe.cs @@ -0,0 +1,25 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// A benchmark that runs out of process, on the default toolchain. The other probes stay in process to keep the run + /// fast, which skips the generate/build/execute cycle entirely, so this one is what makes the adapter see a real + /// . + /// + [Config(typeof(OutOfProcessConfig))] + public class OutOfProcessProbe + { + [Benchmark] + public int Add() => 1 + 1; + + private class OutOfProcessConfig : ManualConfig + { + // A dry job on the default toolchain: one iteration, but a separate executable is still generated, built + // and run. + public OutOfProcessConfig() => AddJob(Job.Dry); + } + } +} From 09485e9ee072cc306cd141889cb16baf11247ddc Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:58:53 +0100 Subject: [PATCH 061/110] Add conditional refs for TestingPlatform probe projects Added conditional project references to BenchmarkDotNet.IntegrationTests.TestingPlatform and .Failures in BenchmarkDotNet.IntegrationTests.csproj. These are included only for .NETCoreApp targets with ReferenceOutputAssembly set to false, ensuring correct build order for probe apps used in TestingPlatformAdapterTests. --- .../BenchmarkDotNet.IntegrationTests.csproj | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj b/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj index 768cccaaa7..fbb0355c9c 100644 --- a/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj +++ b/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj @@ -39,6 +39,15 @@ + + + + + From 22759aa1574f4faea526a820d2c2d22dc8614c45 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:59:22 +0100 Subject: [PATCH 062/110] Add TestingPlatformAdapterTests for integration testing Added TestingPlatformAdapterTests (under #if NETCOREAPP) using BenchmarkDotNet to perform integration tests on Microsoft.Testing.Platform probe apps. Tests cover benchmark discovery, UID consistency, filtering, build/run behavior, and error reporting by running probe apps as separate processes and asserting on their output. Introduced helper methods for process execution, output parsing, and result summarization. --- .../TestingPlatformAdapterTests.cs | 258 ++++++++++++++++++ 1 file changed, 258 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs new file mode 100644 index 0000000000..2e8520bf77 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs @@ -0,0 +1,258 @@ +#if NETCOREAPP +using BenchmarkDotNet.Detectors; +using System.Diagnostics; +using System.Text; +using System.Text.Json; +using System.Text.RegularExpressions; + +namespace BenchmarkDotNet.IntegrationTests +{ + /// + /// Drives the two Microsoft.Testing.Platform probe applications through their command line and asserts on what + /// BenchmarkDotNet.TestAdapter reports back. Everything here goes through a separate process on purpose: the + /// adapter's job is to keep a benchmark identifiable and addressable from the outside, and discovery and execution + /// are two different processes when a test runner drives it. + /// + public class TestingPlatformAdapterTests(ITestOutputHelper output) + { + private const string PassingProbes = "BenchmarkDotNet.IntegrationTests.TestingPlatform"; + private const string FailingProbes = "BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures"; + + // Both probe projects are single targeted, see their .csproj files. + private const string ProbeTargetFramework = "net10.0"; + + // A run that has to build a benchmark pays for a restore and a build of the generated project. + private static readonly TimeSpan Timeout = TimeSpan.FromMinutes(10); + + [Fact] + public void EveryBenchmarkIsDiscoveredUnderItsOwnName() + { + string[] expected = + [ + // The description of a [Benchmark(Description = ...)] is what a user recognises it by, so it is used + // instead of the method name. Without one the method name is used, and the parameters are appended to + // both. + "DescribedProbe.'A described benchmark'(Size: 1)", + "DescribedProbe.Undescribed(Size: 1)", + + // A generic benchmark is named after the type arguments it was closed over. + "GenericProbe.Create", + "GenericProbe>.Create", + "GenericProbe.Create", + + "OutOfProcessProbe.Add", + "SampleBenchmarks.Add(Size: 1)", + "SampleBenchmarks.Add(Size: 2)", + "SampleBenchmarks.Multiply(Size: 1)", + "SampleBenchmarks.Multiply(Size: 2)", + "SeparatorProbe.Length(Value: \"a/b\")", + ]; + + var discovered = Discover(PassingProbes); + + Assert.Equal( + expected, + discovered.Select(test => test.DisplayName.Substring(PassingProbes.Length + 1)).OrderBy(name => name, StringComparer.Ordinal)); + + // The platform identifies a node by its uid, so two benchmarks sharing one cannot be told apart. + Assert.Equal(discovered.Count, discovered.Select(test => test.Uid).Distinct().Count()); + } + + [Fact] + public void TheUidOfABenchmarkIsTheSameInEveryProcess() + { + var first = Discover(PassingProbes).ToDictionary(test => test.Uid, test => test.DisplayName); + var second = Discover(PassingProbes).ToDictionary(test => test.Uid, test => test.DisplayName); + + Assert.Equal(first, second); + } + + [Fact] + public void ABenchmarkCanBeRunByTheUidItWasDiscoveredWith() + { + // This is the contract a test runner relies on: it discovers in one process and asks for a uid in another. + var uid = Discover(PassingProbes) + .Single(test => test.DisplayName.EndsWith("SampleBenchmarks.Add(Size: 2)", StringComparison.Ordinal)) + .Uid; + + var summary = RunAndSummarize(PassingProbes, "--filter-uid", uid); + + Assert.Equal(1, summary.Total); + Assert.Equal(1, summary.Succeeded); + Assert.Equal(0, summary.Failed); + } + + [Fact] + public void ATreeNodeFilterMatchesTheCategoriesOfABenchmark() + { + // The categories are published as filterable properties, which is what makes this expression work. + var discovered = Discover(PassingProbes, "--treenode-filter", "/*/*/*/*[Category=Fast]"); + + Assert.Equal( + new[] { "SampleBenchmarks.Add(Size: 1)", "SampleBenchmarks.Add(Size: 2)" }, + discovered.Select(test => test.DisplayName.Substring(PassingProbes.Length + 1)).OrderBy(name => name, StringComparer.Ordinal)); + } + + [Fact] + public void ATreeNodeFilterMatchesTheClassAndTheMethodOfABenchmark() + { + var discovered = Discover(PassingProbes, "--treenode-filter", "/*/*/SampleBenchmarks/Multiply*"); + + Assert.Equal(2, discovered.Count); + Assert.All(discovered, test => Assert.Contains("SampleBenchmarks.Multiply", test.DisplayName, StringComparison.Ordinal)); + } + + [Fact] + public void ABenchmarkStaysAtTheSameLevelOfTheTreeWhenAParameterContainsTheSeparator() + { + // The platform splits the tree path on every '/' and never unescapes it, so a parameter containing one has + // to be encoded rather than escaped: otherwise the benchmark sits one level deeper and this filter, which + // matches every other benchmark, would miss it. + var discovered = Discover(PassingProbes, "--treenode-filter", "/*/*/SeparatorProbe/*"); + + Assert.Single(discovered); + Assert.Contains("SeparatorProbe.Length(Value: \"a/b\")", discovered[0].DisplayName, StringComparison.Ordinal); + } + + [Fact] + public void AnOutOfProcessBenchmarkIsBuiltAndRun() + { + // The only probe that is not pinned to an in-process toolchain, so the only one that makes the adapter see + // a real generate/build/execute cycle. + var summary = RunAndSummarize(PassingProbes, "--treenode-filter", "/*/*/OutOfProcessProbe/*"); + + Assert.Equal(1, summary.Total); + Assert.Equal(1, summary.Succeeded); + Assert.Equal(0, summary.Failed); + } + + [Fact] + public void ABuildFailureIsReportedAsAFailedTest() + { + var (summary, standardOutput) = Run(FailingProbes, "--treenode-filter", "/*/*/BuildFailureProbe/*"); + + Assert.Equal(1, summary.Total); + Assert.Equal(1, summary.Failed); + Assert.Contains("// Build Error: The build of this benchmark always fails, on purpose.", standardOutput, StringComparison.Ordinal); + } + + [Fact] + public void BenchmarksSharingAUidAreReportedAsOneFailedTest() + { + // Two benchmarks the platform cannot tell apart are published as a single node during discovery, and the + // collision is reported when they are asked to run. + Assert.Single(Discover(FailingProbes, "--treenode-filter", "/*/*/CollisionProbe/*")); + + var (summary, standardOutput) = Run(FailingProbes, "--treenode-filter", "/*/*/CollisionProbe/*"); + + Assert.Equal(1, summary.Total); + Assert.Equal(1, summary.Failed); + Assert.Contains("2 benchmarks are identified as", standardOutput, StringComparison.Ordinal); + } + + private IReadOnlyList Discover(string project, params string[] arguments) + { + var (exitCode, standardOutput) = Execute(project, ["--list-tests", "json", .. arguments]); + + Assert.Equal(0, exitCode); + + using var document = JsonDocument.Parse(standardOutput); + + return document.RootElement.GetProperty("tests") + .EnumerateArray() + .Select(test => new DiscoveredTest(test.GetProperty("uid").GetString()!, test.GetProperty("displayName").GetString()!)) + .ToArray(); + } + + private TestRunSummary RunAndSummarize(string project, params string[] arguments) => Run(project, arguments).Summary; + + private (TestRunSummary Summary, string StandardOutput) Run(string project, params string[] arguments) + { + var (_, standardOutput) = Execute(project, arguments); + + return (TestRunSummary.Parse(standardOutput), standardOutput); + } + + private (int ExitCode, string StandardOutput) Execute(string project, string[] arguments) + { + var application = GetProbeApplication(project); + var startInfo = new ProcessStartInfo(application) + { + WorkingDirectory = Path.GetDirectoryName(application), + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false + }; + + // Progress reporting redraws the screen in place, which is noise once the output is redirected. + foreach (var argument in arguments.Concat(["--no-ansi", "--progress", "off"])) + startInfo.ArgumentList.Add(argument); + + var standardOutput = new StringBuilder(); + var standardError = new StringBuilder(); + + using var process = new Process { StartInfo = startInfo }; + process.OutputDataReceived += (_, e) => { if (e.Data != null) lock (standardOutput) standardOutput.AppendLine(e.Data); }; + process.ErrorDataReceived += (_, e) => { if (e.Data != null) lock (standardError) standardError.AppendLine(e.Data); }; + + process.Start(); + process.BeginOutputReadLine(); + process.BeginErrorReadLine(); + + if (!process.WaitForExit((int)Timeout.TotalMilliseconds)) + { + process.Kill(entireProcessTree: true); + throw new TimeoutException($"'{Path.GetFileName(application)} {string.Join(" ", arguments)}' did not finish within {Timeout}."); + } + + // Lets the redirected output be flushed before it is read. + process.WaitForExit(); + + output.WriteLine($"$ {application} {string.Join(" ", startInfo.ArgumentList)}"); + output.WriteLine(standardOutput.ToString()); + + if (standardError.Length > 0) + output.WriteLine($"stderr:{Environment.NewLine}{standardError}"); + + return (process.ExitCode, standardOutput.ToString()); + } + + private static string GetProbeApplication(string project) + { + // The tests run from /tests/BenchmarkDotNet.IntegrationTests/bin///, and + // the probes are built next to them, by the ProjectReferences of this project. + var binaries = new DirectoryInfo(AppContext.BaseDirectory); + var configuration = binaries.Parent!.Name; + var testsFolder = binaries.Parent!.Parent!.Parent!.Parent!.FullName; + + var fileName = OsDetector.IsWindows() ? $"{project}.exe" : project; + var path = Path.Combine(testsFolder, project, "bin", configuration, ProbeTargetFramework, fileName); + + if (!File.Exists(path)) + throw new FileNotFoundException($"The probe application was not built. Expected it at '{path}'.", path); + + return path; + } + + private sealed record DiscoveredTest(string Uid, string DisplayName); + + private sealed record TestRunSummary(int Total, int Failed, int Succeeded, int Skipped) + { + public static TestRunSummary Parse(string standardOutput) + { + // The platform ends a run with a block of " : " lines under "Test run summary:". + int Read(string name) + { + var match = Regex.Match(standardOutput, $@"^\s*{name}:\s*(?\d+)\s*$", RegexOptions.Multiline); + + return match.Success + ? int.Parse(match.Groups["count"].Value) + : throw new InvalidOperationException($"The test run did not report a '{name}' count.{Environment.NewLine}{standardOutput}"); + } + + return new TestRunSummary(Read("total"), Read("failed"), Read("succeeded"), Read("skipped")); + } + } + } +} +#endif From af7c0190ff356e3a589bd6fe01ef214cea0d0384 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:59:38 +0100 Subject: [PATCH 063/110] Escape '/' as '%2F' in BenchmarkTestNode parameters Updated the Escape method in BenchmarkTestNode.cs to percent-encode '/' as '%2F' and '%' as '%25'. This prevents path segmentation issues in Microsoft.Testing.Platform, ensuring correct tree structure and benchmark addressability. Filters must now use '%2F' instead of '/'. --- .../TestingPlatform/BenchmarkTestNode.cs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs index 764067fb67..9d79bd7f35 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs @@ -150,7 +150,11 @@ private static string BuildPath(Assembly assembly, string? @namespace, string fu .ToString(); } - // Benchmark parameters are stringified user values, so they can contain the path separator. - private static string Escape(string segment) => segment.Replace("/", "\\/"); + // Benchmark parameters are stringified user values, so they can contain the path separator. A '/' cannot be + // escaped into a segment: Microsoft.Testing.Platform splits the path on every '/' without ever unescaping it, + // and TreeNodeFilter rejects a filter whose segment contains one, so a raw '/' would both deepen the tree and + // leave the benchmark unmatchable. Percent encoding keeps the path four levels deep and the segment + // addressable, at the price of a filter having to spell the separator as '%2F'. + private static string Escape(string segment) => segment.Replace("%", "%25").Replace("/", "%2F"); } } From ba88cc9761b22fc8150e381c7b5a1eaa88edd84e Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:59:48 +0100 Subject: [PATCH 064/110] Document percent-encoding in filter path Updated documentation to clarify that benchmark parameter values containing slashes (/) or percent signs (%) are percent-encoded in the tree node filter path (e.g., a/b as a%2Fb, % as %25). This encoding applies only to the filter, not to the displayed benchmark name. --- docs/articles/features/testadapter.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/articles/features/testadapter.md b/docs/articles/features/testadapter.md index 90e159768b..8a091886fb 100644 --- a/docs/articles/features/testadapter.md +++ b/docs/articles/features/testadapter.md @@ -132,6 +132,10 @@ dotnet run -c Release -- --treenode-filter "/*/*/*/*[Category=Fast]" The tree node filter path is `////`, and `[BenchmarkCategory]` attributes are exposed as a `Category` trait that the filter can match on. +Because the platform separates the levels of that path with `/`, a benchmark parameter whose value contains one is + percent encoded in the path: a parameter value of `a/b` is written `a%2Fb` in a filter, and a literal `%` is + written `%25`. +This affects the filter only; the name the benchmark is displayed under is unchanged. ## Keeping your own entry point From 61fa92eb615dce35bf5b380c3ffc25aedb2378bc Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 11:59:58 +0100 Subject: [PATCH 065/110] Add smoke test script for BenchmarkDotNet.TestAdapter Added test-adapter-consumer.ps1 to perform smoke tests on the packed BenchmarkDotNet.TestAdapter NuGet package. The script restores and builds a consumer project, checks MSBuild property resolutions, and verifies benchmark discovery to ensure correct adapter behavior when used as a package. Includes detailed comments, parameter handling, and error checking. --- build/smoke-tests/test-adapter-consumer.ps1 | 115 ++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100644 build/smoke-tests/test-adapter-consumer.ps1 diff --git a/build/smoke-tests/test-adapter-consumer.ps1 b/build/smoke-tests/test-adapter-consumer.ps1 new file mode 100644 index 0000000000..47c9e586a5 --- /dev/null +++ b/build/smoke-tests/test-adapter-consumer.ps1 @@ -0,0 +1,115 @@ +#!/usr/bin/env pwsh + +<# +.SYNOPSIS + Smoke tests the packed BenchmarkDotNet.TestAdapter against a project that consumes it as a NuGet package. + +.DESCRIPTION + Everything in this repository that uses the adapter's build files imports them by path from the project file, + which MSBuild evaluates before nuget.g.targets. A package consumer gets the opposite order: NuGet imports + Microsoft.Testing.Platform.MSBuild's targets, which default IsTestingPlatformApplication to true, and only then + the adapter's, which have to overwrite that default for the opt-outs to work. Nothing in the solution can + reproduce that order, so this restores the real package and asserts on how the properties resolve. + + Run `build.cmd pack` first, so that the packages exist. + +.PARAMETER ArtifactsDirectory + The directory `build.cmd pack` wrote the packages to. + +.PARAMETER Configuration + The configuration to build the consuming project in. +#> + +[CmdletBinding()] +param( + [string] $ArtifactsDirectory = [System.IO.Path]::Combine($PSScriptRoot, '..', '..', 'artifacts'), + [string] $Configuration = 'Release' +) + +$ErrorActionPreference = 'Stop' +$PSNativeCommandUseErrorActionPreference = $true + +$project = [System.IO.Path]::Combine($PSScriptRoot, 'TestAdapterConsumer', 'TestAdapterConsumer.csproj') +$targetFramework = 'net10.0' + +# $IsWindows only exists on PowerShell Core, where it is the only way to tell; Windows PowerShell is Windows by definition. +$onWindows = ($null -eq $IsWindows) -or $IsWindows + +# build.cmd installs the SDK the repository is pinned to into .dotnet, and only puts it on PATH for its own run. +$dotnet = [System.IO.Path]::Combine($PSScriptRoot, '..', '..', '.dotnet', $(if ($onWindows) { 'dotnet.exe' } else { 'dotnet' })) +if (-not (Test-Path $dotnet)) { + $dotnet = 'dotnet' +} + +$package = Get-ChildItem -Path $ArtifactsDirectory -Filter 'BenchmarkDotNet.TestAdapter.*.nupkg' | + Where-Object { $_.Name -notlike '*.symbols.nupkg' } | + Select-Object -First 1 + +if ($null -eq $package) { + throw "No BenchmarkDotNet.TestAdapter package was found in '$ArtifactsDirectory'. Run 'build.cmd pack' first." +} + +$version = $package.BaseName -replace '^BenchmarkDotNet\.TestAdapter\.', '' +Write-Output "Consuming BenchmarkDotNet.TestAdapter $version from $ArtifactsDirectory" + +function Invoke-Dotnet { + param([Parameter(ValueFromRemainingArguments = $true)] [string[]] $Arguments) + + $output = & $dotnet @Arguments 2>&1 | Out-String + + if ($LASTEXITCODE -ne 0) { + Write-Output $output + throw "'dotnet $($Arguments -join ' ')' failed with exit code $LASTEXITCODE." + } + + return $output +} + +function Assert-Property { + param( + [string] $Name, + [string] $Expected, + [string[]] $With = @() + ) + + $arguments = @($project, '-nologo', '-tl:off', "-p:BenchmarkDotNetVersion=$version", "-p:Configuration=$Configuration") + $With + @("-getProperty:$Name") + $actual = (Invoke-Dotnet msbuild @arguments).Trim() + + $description = if ($With.Count -eq 0) { 'by default' } else { "with $($With -join ' ')" } + + if ($actual -ne $Expected) { + throw "Expected $Name to be '$Expected' $description, but it was '$actual'." + } + + Write-Output " OK: $Name is '$Expected' $description" +} + +Write-Output '##[group]Restoring the consuming project' +Invoke-Dotnet restore $project "-p:BenchmarkDotNetVersion=$version" '-tl:off' | Write-Output +Write-Output '##[endgroup]' + +Write-Output 'Checking how the packaged build files resolve the test platform:' + +# Microsoft.Testing.Platform is the default, and the adapter leaves the entry point to it. +Assert-Property -Name 'IsTestingPlatformApplication' -Expected 'true' +Assert-Property -Name 'GenerateProgramFile' -Expected 'false' + +# The two opt-outs have to win over the default Microsoft.Testing.Platform.MSBuild sets in its own targets, which a +# package consumer imports before the adapter's. +Assert-Property -Name 'IsTestingPlatformApplication' -Expected 'false' -With '-p:BenchmarkDotNetUseVSTest=true' +Assert-Property -Name 'IsTestingPlatformApplication' -Expected 'false' -With '-p:GenerateProgramFile=false' + +Write-Output '##[group]Building the consuming project' +Invoke-Dotnet build $project '--no-restore' '-c' $Configuration "-p:BenchmarkDotNetVersion=$version" '-tl:off' | Write-Output +Write-Output '##[endgroup]' + +Write-Output 'Listing the benchmarks through the entry point Microsoft.Testing.Platform generated:' +$application = [System.IO.Path]::Combine($PSScriptRoot, 'TestAdapterConsumer', 'bin', $Configuration, $targetFramework, 'TestAdapterConsumer.dll') +$listed = Invoke-Dotnet $application '--list-tests' '--no-ansi' +Write-Output $listed + +if ($listed -notmatch 'TestAdapterConsumer\.ConsumedBenchmark\.Add') { + throw 'The packaged adapter did not list the benchmark of the consuming project.' +} + +Write-Output 'The packaged BenchmarkDotNet.TestAdapter behaves as expected.' From 8745d4aa27fc9ad2484e533e08a25a83470224b4 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 12:00:13 +0100 Subject: [PATCH 066/110] Enable optimizations in project file for benchmarks Add true to ensure the assembly is always built with optimizations, preventing BenchmarkEnumerator from hiding out-of-process benchmarks in non-Release builds. Expand comments to clarify manual build file imports and MSBuild processing order. --- ...kDotNet.IntegrationTests.TestingPlatform.csproj | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj index 6cea87bd71..cdbdbc7edc 100644 --- a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BenchmarkDotNet.IntegrationTests.TestingPlatform.csproj @@ -6,13 +6,25 @@ BenchmarkDotNet.IntegrationTests.TestingPlatform BenchmarkDotNet.IntegrationTests.TestingPlatform BenchmarkDotNet.IntegrationTests.TestingPlatform + + + true - + From 3ad898433b620ad3a036a672755355ca21ef8606 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 12:00:26 +0100 Subject: [PATCH 067/110] Document TestingPlatform.Failures project in README Added a section to README.md describing the BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures project. The documentation explains its purpose as a collection of intentionally failing benchmarks for testing BenchmarkDotNet.TestAdapter error handling, including UID collision and build failure mapping. It also clarifies the relationship with TestingPlatformAdapterTests and the location of passing benchmarks. --- .../README.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/README.md diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/README.md b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/README.md new file mode 100644 index 0000000000..1905d04e12 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/README.md @@ -0,0 +1,9 @@ +# BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures + +Every benchmark in this Microsoft.Testing.Platform application is expected to be reported as failed, so a plain +`dotnet test` over it fails by design. It exists for the paths of `BenchmarkDotNet.TestAdapter` that only a broken +benchmark reaches: the uid collision report and the mapping of a build failure onto failed tests. + +`TestingPlatformAdapterTests` in `BenchmarkDotNet.IntegrationTests` drives it, one probe at a time, and asserts on +what the platform reports. The benchmarks that are expected to pass live in +`BenchmarkDotNet.IntegrationTests.TestingPlatform` instead. From 67c9996016cf41558a9856c457fd7f0ea81f0e84 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 31 Aug 2026 12:00:37 +0100 Subject: [PATCH 068/110] Add Failures test project to BenchmarkDotNet.slnx Added BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures to the solution file, ensuring it is included with other integration test projects. --- BenchmarkDotNet.slnx | 1 + 1 file changed, 1 insertion(+) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index c3c733fb82..f6801feaba 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -43,6 +43,7 @@ + From 28d34f3e76ccf1bae3804d101cfcf8017071bc89 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 5 Sep 2026 16:41:19 +0100 Subject: [PATCH 069/110] Clarify comment on IsTestingPlatformApplication logic Updated the comment explaining the opt-out logic for IsTestingPlatformApplication to clarify that the property may already be set to true by the project or another package, making empty checks unreliable. The revised comment aligns with MSTest.TestAdapter.targets and improves accuracy. No functional code changes were made. --- .../build/BenchmarkDotNet.TestAdapter.targets | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets index 09d71557cf..331d980cfd 100644 --- a/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets +++ b/src/BenchmarkDotNet.TestAdapter/build/BenchmarkDotNet.TestAdapter.targets @@ -14,9 +14,9 @@ that calls BenchmarkSwitcher. Generating a second one would not compile. --> false From 8a0c01f9775271f72645e82dd6c4b5cf299e0f9c Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 5 Sep 2026 16:41:49 +0100 Subject: [PATCH 070/110] Add CategoryProbe and DisposableProbe benchmark tests CategoryProbe.cs introduces a benchmark using a custom ICategoryDiscoverer to assign categories based on method names, ensuring custom categories are recognized by BenchmarkDotNet. DisposableProbe.cs adds a benchmark with disposable parameter values, tracking their creation and disposal, and writing a report on process exit to address potential runtime hangs from undisposed parameters. Both use custom ManualConfig for job and toolchain settings. --- .../CategoryProbe.cs | 37 ++++++++++++ .../DisposableProbe.cs | 56 +++++++++++++++++++ 2 files changed, 93 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/CategoryProbe.cs create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DisposableProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/CategoryProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/CategoryProbe.cs new file mode 100644 index 0000000000..44c4e3bd80 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/CategoryProbe.cs @@ -0,0 +1,37 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Running; +using BenchmarkDotNet.Toolchains.InProcess.Emit; +using System.Reflection; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// A benchmark whose category comes from a custom rather than from a + /// [BenchmarkCategory]. The adapter has to publish the categories the config resolved: rediscovering them with the + /// default discoverer would leave a --treenode-filter on a custom category matching nothing, even though + /// BenchmarkDotNet's own --anyCategories and the summary do see it. + /// + [Config(typeof(DiscoveredCategoryConfig))] + public class CategoryProbe + { + [Benchmark] + public int Identity() => 1; + + private class CategoryFromMethodName : ICategoryDiscoverer + { + // The default discoverer only reads [BenchmarkCategory], so this category exists nowhere else. + public string[] GetCategories(MethodInfo method) => [$"Discovered{method.Name}"]; + } + + private class DiscoveredCategoryConfig : ManualConfig + { + public DiscoveredCategoryConfig() + { + AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + WithCategoryDiscoverer(new CategoryFromMethodName()); + } + } + } +} diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DisposableProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DisposableProbe.cs new file mode 100644 index 0000000000..e13db876d9 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DisposableProbe.cs @@ -0,0 +1,56 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// A benchmark whose parameter values are disposable. Enumerating the assembly creates every one of them, whether + /// the case it belongs to is going to run or not, and BenchmarkDotNet only disposes the ones it was handed - a + /// parameter with a locking finalizer hangs the runtime otherwise, see dotnet/BenchmarkDotNet#1383. What is left + /// undisposed only shows at the end, so the counts are written out when the process exits. + /// + [Config(typeof(FastConfig))] + public class DisposableProbe + { + /// + /// The name of the file the counts are written to, next to the probe application. + /// + public const string ReportFileName = "disposable-probe.txt"; + + // Created once, so that re-reading the source cannot change the count. + private static readonly Tracked[] Instances = [new Tracked(1), new Tracked(2), new Tracked(3)]; + + public IEnumerable Values => Instances; + + [ParamsSource(nameof(Values))] + public Tracked? Value { get; set; } + + [Benchmark] + public int Identity() => Value!.Number; + + public class Tracked : IDisposable + { + private static int disposed; + + static Tracked() => + AppDomain.CurrentDomain.ProcessExit += (_, _) => File.WriteAllText( + Path.Combine(AppContext.BaseDirectory, ReportFileName), + $"created={Instances.Length} disposed={Volatile.Read(ref disposed)}"); + + public Tracked(int number) => Number = number; + + public int Number { get; } + + public void Dispose() => Interlocked.Increment(ref disposed); + + public override string ToString() => $"tracked-{Number}"; + } + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} From 4c1de8a33adc6288dc4b30e7264d7fa2ec9b64d7 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 5 Sep 2026 16:42:02 +0100 Subject: [PATCH 071/110] Add tests for categories, disposal, and UID collisions Expanded TestingPlatformAdapterTests with cases for custom categories, descriptions, and disposal of parameter values. Added tests for handling UID collisions in benchmarks. Introduced ReadDisposalReport helper and updated expected benchmark discovery list. --- .../TestingPlatformAdapterTests.cs | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs index 2e8520bf77..93a9d63812 100644 --- a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs +++ b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs @@ -29,12 +29,18 @@ public void EveryBenchmarkIsDiscoveredUnderItsOwnName() { string[] expected = [ + "CategoryProbe.Identity", + // The description of a [Benchmark(Description = ...)] is what a user recognises it by, so it is used // instead of the method name. Without one the method name is used, and the parameters are appended to // both. "DescribedProbe.'A described benchmark'(Size: 1)", "DescribedProbe.Undescribed(Size: 1)", + "DisposableProbe.Identity(Value: tracked-1)", + "DisposableProbe.Identity(Value: tracked-2)", + "DisposableProbe.Identity(Value: tracked-3)", + // A generic benchmark is named after the type arguments it was closed over. "GenericProbe.Create", "GenericProbe>.Create", @@ -93,6 +99,41 @@ public void ATreeNodeFilterMatchesTheCategoriesOfABenchmark() discovered.Select(test => test.DisplayName.Substring(PassingProbes.Length + 1)).OrderBy(name => name, StringComparer.Ordinal)); } + [Fact] + public void ATreeNodeFilterMatchesTheCategoriesOfACustomCategoryDiscoverer() + { + // The node has to carry the categories the config resolved, not the ones the default discoverer finds: + // this category is produced by an ICategoryDiscoverer and exists on no [BenchmarkCategory] anywhere. + var discovered = Discover(PassingProbes, "--treenode-filter", "/*/*/*/*[Category=DiscoveredIdentity]"); + + Assert.Equal( + new[] { "CategoryProbe.Identity" }, + discovered.Select(test => test.DisplayName.Substring(PassingProbes.Length + 1))); + } + + [Fact] + public void ParameterValuesAreDisposedWhenBenchmarksAreOnlyListed() + { + // Listing runs nothing, so BenchmarkDotNet disposes nothing: every value the enumeration created is the + // adapter's to dispose. + Assert.Equal("created=3 disposed=3", ReadDisposalReport(() => Discover(PassingProbes))); + } + + [Fact] + public void ParameterValuesAreDisposedWhenOnlyOneBenchmarkOfASetIsRun() + { + // BenchmarkDotNet only disposes the case it was handed, so the two that were filtered out would leak. A + // value shared with the case that runs must not be disposed early either, which the count would catch as + // a disposal too many. + var uid = Discover(PassingProbes) + .Single(test => test.DisplayName.EndsWith("DisposableProbe.Identity(Value: tracked-1)", StringComparison.Ordinal)) + .Uid; + + var report = ReadDisposalReport(() => RunAndSummarize(PassingProbes, "--filter-uid", uid)); + + Assert.Equal("created=3 disposed=3", report); + } + [Fact] public void ATreeNodeFilterMatchesTheClassAndTheMethodOfABenchmark() { @@ -150,6 +191,41 @@ public void BenchmarksSharingAUidAreReportedAsOneFailedTest() Assert.Contains("2 benchmarks are identified as", standardOutput, StringComparison.Ordinal); } + [Fact] + public void BenchmarksSharingAUidThroughTheirDescriptionAreReportedWithTheirMethodNames() + { + // Nothing here is parameterized: what collides is the description of one benchmark against the method + // name of the other, so the message has to name the two methods and point at the description. + var (summary, standardOutput) = Run(FailingProbes, "--treenode-filter", "/*/*/DescriptionCollisionProbe/*"); + + Assert.Equal(1, summary.Total); + Assert.Equal(1, summary.Failed); + Assert.Contains("none of them were run: Described, Twin.", standardOutput, StringComparison.Ordinal); + Assert.Contains("[Benchmark(Description = \"...\")]", standardOutput, StringComparison.Ordinal); + } + + /// + /// Runs the probe application and reads back what it reported about the disposal of its parameter values. + /// + /// + /// The counts are written to a file rather than to the output, because the discovery output is parsed as json. + /// + /// The way the probe application is driven. + /// The counts the probe reported when it exited. + private static string ReadDisposalReport(Action execute) + { + // The probe projects are referenced with ReferenceOutputAssembly="false", so the name is repeated here + // rather than taken from DisposableProbe.ReportFileName. + var report = Path.Combine(Path.GetDirectoryName(GetProbeApplication(PassingProbes))!, "disposable-probe.txt"); + + File.Delete(report); + execute(); + + Assert.True(File.Exists(report), $"The probe application did not write '{report}'."); + + return File.ReadAllText(report); + } + private IReadOnlyList Discover(string project, params string[] arguments) { var (exitCode, standardOutput) = Execute(project, ["--list-tests", "json", .. arguments]); From 0f21efed949aabd963ad8750f0a8f5abed059e49 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 5 Sep 2026 16:42:12 +0100 Subject: [PATCH 072/110] Add test for benchmark description/name collision Introduced DescriptionCollisionProbe class with two benchmarks to test identity collisions when a benchmark's description matches another's method name. Added FastConfig for dry job using InProcessEmitToolchain. --- .../DescriptionCollisionProbe.cs | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/DescriptionCollisionProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/DescriptionCollisionProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/DescriptionCollisionProbe.cs new file mode 100644 index 0000000000..f79a582aa0 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/DescriptionCollisionProbe.cs @@ -0,0 +1,27 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures +{ + /// + /// Two benchmarks without any parameter that BenchmarkDotNet still identifies as one, because the description of + /// the first is the method name of the second and the identity of a benchmark is built from the name it displays. + /// The reported collision has to point at that rather than at the parameters. + /// + [Config(typeof(FastConfig))] + public class DescriptionCollisionProbe + { + [Benchmark(Description = "Twin")] + public int Described() => 1; + + [Benchmark] + public int Twin() => 2; + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} From 3d57bdc81c44d3c400fb792d352ca19e39a23a85 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 5 Sep 2026 16:42:32 +0100 Subject: [PATCH 073/110] Refactor benchmark logic; fix artifact packing - Update .csproj to pack .props/.targets only in build folder, preventing test platform settings from leaking to consumers - Refactor BenchmarkTestFramework.cs to track all enumerated benchmarks, dispose unused parameter values, and improve collision error messages - Add internal classes for grouping enumeration results and reference-based IDisposable comparison - Use resolved categories from descriptors in BenchmarkTestNode.cs for consistency with BenchmarkDotNet and custom discoverers --- .../BenchmarkDotNet.TestAdapter.csproj | 15 +- .../TestingPlatform/BenchmarkTestFramework.cs | 138 +++++++++++++++--- .../TestingPlatform/BenchmarkTestNode.cs | 6 +- 3 files changed, 134 insertions(+), 25 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkDotNet.TestAdapter.csproj b/src/BenchmarkDotNet.TestAdapter/BenchmarkDotNet.TestAdapter.csproj index 7cdb59410a..6d61995d42 100644 --- a/src/BenchmarkDotNet.TestAdapter/BenchmarkDotNet.TestAdapter.csproj +++ b/src/BenchmarkDotNet.TestAdapter/BenchmarkDotNet.TestAdapter.csproj @@ -43,10 +43,17 @@ - - - + + + diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs index bdce27cd9d..ed5def52f1 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs @@ -9,6 +9,7 @@ using Microsoft.Testing.Platform.Services; using Microsoft.Testing.Platform.TestHost; using System.Reflection; +using System.Runtime.CompilerServices; using System.Runtime.ExceptionServices; using System.Threading.Channels; @@ -84,17 +85,27 @@ public async Task ExecuteRequestAsync(ExecuteRequestContext context) private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRequestContext context) { - foreach (var benchmarks in GetMatchingBenchmarks(request.Filter)) + var enumeration = GetMatchingBenchmarks(request.Filter); + + try { - context.CancellationToken.ThrowIfCancellationRequested(); + foreach (var benchmarks in enumeration.Matches) + { + context.CancellationToken.ThrowIfCancellationRequested(); - // Exactly one node per uid: publishing a colliding uid twice would leave the platform with two nodes - // it cannot tell apart. The collision itself is reported when the benchmarks are run. - var message = new TestNodeUpdateMessage( - request.Session.SessionUid, - benchmarks[0].Node.ToTestNode(DiscoveredTestNodeStateProperty.CachedInstance)); + // Exactly one node per uid: publishing a colliding uid twice would leave the platform with two + // nodes it cannot tell apart. The collision itself is reported when the benchmarks are run. + var message = new TestNodeUpdateMessage( + request.Session.SessionUid, + benchmarks[0].Node.ToTestNode(DiscoveredTestNodeStateProperty.CachedInstance)); - await context.MessageBus.PublishAsync(this, message).ConfigureAwait(false); + await context.MessageBus.PublishAsync(this, message).ConfigureAwait(false); + } + } + finally + { + // Discovery runs nothing, so every value the enumeration created is this method's to dispose. + DisposeUnusedParameterValues(enumeration.All, []); } } @@ -104,7 +115,8 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte var cancellationToken = context.CancellationToken; var runnable = new List(); - foreach (var benchmarks in GetMatchingBenchmarks(request.Filter)) + var enumeration = GetMatchingBenchmarks(request.Filter); + foreach (var benchmarks in enumeration.Matches) { if (benchmarks.Count == 1) runnable.Add(benchmarks[0]); @@ -112,6 +124,10 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte await PublishCollisionAsync(context, sessionUid, benchmarks).ConfigureAwait(false); } + // A benchmark that was filtered out or that collided is never handed to BenchmarkDotNet, so nothing else + // would dispose the values the enumeration created for it. + DisposeUnusedParameterValues(enumeration.All, runnable.Select(match => match.Node.BenchmarkCase)); + if (runnable.Count == 0) return; @@ -168,9 +184,16 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte { try { - // Benchmarks that never reported a result still need one, unless the run was cancelled: - // the platform expects an OperationCanceledException in that case, and publishing results - // afterwards would contradict it. + // Benchmarks that never reported a result still need one, unless the run was cancelled. + // Two things make the cancelled run the exception: + // + // * The platform's contract for a cancelled request is an OperationCanceledException, + // not a terminal state per node. CancelledTestNodeStateProperty is obsolete for + // exactly this reason, so a node left in progress is the shape it asks for. + // BenchmarkDotNet rethrows the cancellation, and awaiting the run task below surfaces + // it out of the request. + // * The token is also cancelled when publishing itself failed. Nothing drains the queue + // at that point, so results published here would be dropped anyway. if (!runCancellation.IsCancellationRequested) eventProcessor.PublishOutstandingResults(); @@ -236,10 +259,18 @@ private static async Task DrainAsync(ChannelReader> reader) private async Task PublishCollisionAsync(ExecuteRequestContext context, SessionUid sessionUid, List collision) { var node = collision[0].Node; + + // The colliding benchmarks share every string the uid is built from, so the method names are the only + // thing left that can tell them apart. They are the same name when it is the parameters that collide. + var methodNames = string.Join(", ", collision + .Select(match => match.Node.BenchmarkCase.Descriptor.WorkloadMethod.Name) + .Distinct(StringComparer.Ordinal)); var error = - $"{collision.Count} benchmarks are identified as '{node.Uid}', so they cannot be told apart and none " + - "of them were run. Benchmarks are identified by the string representation of their parameters: give " + - "the colliding values distinct ToString() results."; + $"{collision.Count} benchmarks are identified as '{node.Uid}' and cannot be told apart, so none of " + + $"them were run: {methodNames}. The identity is built from the type, the benchmark name " + + "([Benchmark(Description = \"...\")] when set, the method name otherwise), the job, and the string " + + "representation of the parameters. Give the colliding benchmarks distinct descriptions, distinct " + + "jobs, or distinct parameter ToString() results."; await context.MessageBus.PublishAsync( this, @@ -255,15 +286,16 @@ await context.MessageBus.PublishAsync( /// /// The filter of the request. /// - /// The matching benchmarks in enumeration order, grouped by uid. A group holding more than one benchmark is a - /// uid collision. + /// The matching benchmarks in enumeration order and grouped by uid, together with everything the assembly + /// declares. A group holding more than one benchmark is a uid collision. /// - private List> GetMatchingBenchmarks(ITestExecutionFilter filter) + private Enumeration GetMatchingBenchmarks(ITestExecutionFilter filter) { var matches = new List>(); var matchesByUid = new Dictionary>(StringComparer.Ordinal); + var runInfos = BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly); - foreach (var runInfo in BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly)) + foreach (var runInfo in runInfos) { // The job only earns a place in the display name when the benchmark actually runs under several jobs. // This is computed before filtering so that a benchmark keeps the same name however it was selected. @@ -286,9 +318,41 @@ private List> GetMatchingBenchmarks(ITestExecutionFilter filter) } } - return matches; + return new Enumeration(matches, runInfos); + } + + /// + /// Disposes the parameter values of the benchmarks that were enumerated but will not be run. + /// + /// + /// Enumerating an assembly instantiates the values of every [Params] and [ArgumentsSource], and BenchmarkDotNet + /// only disposes the ones belonging to the benchmarks it was handed. The values are matched by reference + /// instead of being disposed case by case, because BenchmarkConverter gives the same ParameterInstance to + /// every job and every argument set of a benchmark: disposing a filtered out case wholesale would take down + /// values that a benchmark which is about to run still owns. + /// + /// Everything the assembly declares. + /// The benchmarks that are going to be run, if any. + private static void DisposeUnusedParameterValues(BenchmarkRunInfo[] enumerated, IEnumerable retained) + { + var unused = new HashSet(ReferenceComparer.Instance); + + foreach (var value in GetDisposableParameterValues(enumerated.SelectMany(runInfo => runInfo.BenchmarksCases))) + unused.Add(value); + + foreach (var value in GetDisposableParameterValues(retained)) + unused.Remove(value); + + foreach (var value in unused) + value.Dispose(); } + private static IEnumerable GetDisposableParameterValues(IEnumerable benchmarkCases) + => benchmarkCases + .SelectMany(benchmarkCase => benchmarkCase.Parameters.Items) + .Select(parameter => parameter.Value) + .OfType(); + #pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. private static bool Matches(ITestExecutionFilter filter, BenchmarkTestNode node) => filter switch { @@ -300,6 +364,40 @@ private List> GetMatchingBenchmarks(ITestExecutionFilter filter) }; #pragma warning restore TPEXP + /// + /// The result of enumerating the assembly for a request. + /// + private sealed class Enumeration + { + public Enumeration(List> matches, BenchmarkRunInfo[] all) + { + Matches = matches; + All = all; + } + + /// + /// Gets the benchmarks the request asked for, grouped by uid. + /// + public List> Matches { get; } + + /// + /// Gets every benchmark the assembly declares, matching or not. + /// + public BenchmarkRunInfo[] All { get; } + } + + /// + /// Compares by reference, so that a parameter value which overrides Equals is still disposed once per instance. + /// + private sealed class ReferenceComparer : IEqualityComparer + { + public static readonly ReferenceComparer Instance = new ReferenceComparer(); + + public bool Equals(IDisposable? x, IDisposable? y) => ReferenceEquals(x, y); + + public int GetHashCode(IDisposable obj) => RuntimeHelpers.GetHashCode(obj); + } + /// /// A benchmark that matched the request, together with the run info it belongs to. /// diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs index 9d79bd7f35..d6c30b1de6 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs @@ -99,7 +99,11 @@ public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool include properties.Add(new TestFileLocationProperty(benchmarkAttribute.SourceCodeFile, new LinePositionSpan(position, position))); } - foreach (var category in DefaultCategoryDiscoverer.Instance.GetCategories(benchmarkMethod)) + // The categories come from the descriptor rather than from DefaultCategoryDiscoverer, because + // BenchmarkConverter has already resolved them through the config's ICategoryDiscoverer. Rediscovering + // them here would hide the categories of a custom discoverer from --treenode-filter, even though + // BenchmarkDotNet's own --anyCategories and the summary do see them. + foreach (var category in benchmarkCase.Descriptor.Categories) properties.Add(new TestMetadataProperty("Category", category)); var path = BuildPath(type.Assembly, type.Namespace, fullClassName, parametrizedMethodName, jobDisplayInfo); From 619b92cfdd0256d1fe66ee9dfe74f197bb977a46 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sun, 6 Sep 2026 05:53:19 +0100 Subject: [PATCH 074/110] Adapt BuildFailureProbe to the abstract Toolchain base Toolchain became abstract and gained a Runtime parameter in the toolchain rework merged from master, so the probe's fake toolchain is now a named subclass. UnknownRuntime is enough: the build always fails, so nothing ever resolves a default toolchain from it. --- .../BuildFailureProbe.cs | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BuildFailureProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BuildFailureProbe.cs index 3665f8592a..ba1708bdb9 100644 --- a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BuildFailureProbe.cs +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures/BuildFailureProbe.cs @@ -1,5 +1,6 @@ using BenchmarkDotNet.Attributes; using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Environments; using BenchmarkDotNet.Jobs; using BenchmarkDotNet.Loggers; using BenchmarkDotNet.Running; @@ -25,7 +26,12 @@ public class BuildFailureProbe private class FailingBuildConfig : ManualConfig { public FailingBuildConfig() - => AddJob(Job.Dry.WithToolchain(new Toolchain("FailingBuild", new NoopGenerator(), new FailingBuilder(), new UnreachableExecutor()))); + => AddJob(Job.Dry.WithToolchain(new FailingBuildToolchain())); + } + + private sealed class FailingBuildToolchain() + : Toolchain("FailingBuild", UnknownRuntime.Instance, new NoopGenerator(), new FailingBuilder(), new UnreachableExecutor()) + { } private sealed class NoopGenerator : IGenerator From 701d02340340672d439d9f6db1a61b7dc697130f Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 7 Sep 2026 09:52:19 +0100 Subject: [PATCH 075/110] Improve restore process and error handling in test adapter - Add detailed comments to .csproj explaining RestorePackagesPath usage for local NuGet package restore, ensuring correct versions and avoiding stale extractions. - Update PowerShell script to set $PSNativeCommandUseErrorActionPreference for better error logging. - Add check for artifacts directory existence with clear error message. - Resolve artifacts directory to absolute path for consistency. - Sort and select latest BenchmarkDotNet.TestAdapter package if multiple are found, with informative messaging. - Clean previous benchmarkdotnet* directories before restore for a fresh extraction. - Pass artifacts directory explicitly as a global property during restore. - Add clarifying comments throughout the script regarding restore logic and behavior. --- .../TestAdapterConsumer.csproj | 19 ++++++++- build/smoke-tests/test-adapter-consumer.ps1 | 42 ++++++++++++++++--- 2 files changed, 55 insertions(+), 6 deletions(-) diff --git a/build/smoke-tests/TestAdapterConsumer/TestAdapterConsumer.csproj b/build/smoke-tests/TestAdapterConsumer/TestAdapterConsumer.csproj index f88d1a27f0..d1fb804f5e 100644 --- a/build/smoke-tests/TestAdapterConsumer/TestAdapterConsumer.csproj +++ b/build/smoke-tests/TestAdapterConsumer/TestAdapterConsumer.csproj @@ -14,8 +14,25 @@ Exe enable enable - + $(MSBuildThisFileDirectory)..\..\..\artifacts + + $(MSBuildThisFileDirectory)..\packages\ diff --git a/build/smoke-tests/test-adapter-consumer.ps1 b/build/smoke-tests/test-adapter-consumer.ps1 index 47c9e586a5..8865365f1f 100644 --- a/build/smoke-tests/test-adapter-consumer.ps1 +++ b/build/smoke-tests/test-adapter-consumer.ps1 @@ -27,7 +27,11 @@ param( ) $ErrorActionPreference = 'Stop' -$PSNativeCommandUseErrorActionPreference = $true + +# Invoke-Dotnet checks $LASTEXITCODE itself, and prints the log before it throws. Leaving this on would make a failing +# `dotnet` throw at the call itself on PowerShell Core - which is what the workflow runs - so the log would never be +# printed and a broken restore or build would report nothing at all. +$PSNativeCommandUseErrorActionPreference = $false $project = [System.IO.Path]::Combine($PSScriptRoot, 'TestAdapterConsumer', 'TestAdapterConsumer.csproj') $targetFramework = 'net10.0' @@ -41,14 +45,30 @@ if (-not (Test-Path $dotnet)) { $dotnet = 'dotnet' } -$package = Get-ChildItem -Path $ArtifactsDirectory -Filter 'BenchmarkDotNet.TestAdapter.*.nupkg' | +if (-not (Test-Path -LiteralPath $ArtifactsDirectory)) { + throw "The artifacts directory '$ArtifactsDirectory' does not exist. Run 'build.cmd pack' first." +} + +# Both the package this reads the version from and the source the restore below resolves it through, so it has to be +# the same absolute path in both: a relative one would otherwise be resolved against the consuming project. +$ArtifactsDirectory = (Resolve-Path -LiteralPath $ArtifactsDirectory).ProviderPath + +# `build.cmd pack` never cleans, so the folder can hold several versions. The newest is the one that was just packed, +# which is the one worth smoke testing. +$packages = @(Get-ChildItem -Path $ArtifactsDirectory -Filter 'BenchmarkDotNet.TestAdapter.*.nupkg' | Where-Object { $_.Name -notlike '*.symbols.nupkg' } | - Select-Object -First 1 + Sort-Object -Property LastWriteTime -Descending) -if ($null -eq $package) { +if ($packages.Count -eq 0) { throw "No BenchmarkDotNet.TestAdapter package was found in '$ArtifactsDirectory'. Run 'build.cmd pack' first." } +$package = $packages[0] + +if ($packages.Count -gt 1) { + Write-Output "'$ArtifactsDirectory' holds $($packages.Count) BenchmarkDotNet.TestAdapter packages, taking the most recently written one." +} + $version = $package.BaseName -replace '^BenchmarkDotNet\.TestAdapter\.', '' Write-Output "Consuming BenchmarkDotNet.TestAdapter $version from $ArtifactsDirectory" @@ -84,8 +104,20 @@ function Assert-Property { Write-Output " OK: $Name is '$Expected' $description" } +# The project restores into this folder rather than into the global one, see its .csproj. NuGet never re-extracts a +# version it already has, and the version does not change between runs, so the packages this repository produces are +# dropped before the restore; everything else in there is an ordinary cache and is left alone. +$packagesDirectory = [System.IO.Path]::Combine($PSScriptRoot, 'packages') +if (Test-Path -LiteralPath $packagesDirectory) { + Get-ChildItem -Path $packagesDirectory -Directory -Filter 'benchmarkdotnet*' | Remove-Item -Recurse -Force +} + Write-Output '##[group]Restoring the consuming project' -Invoke-Dotnet restore $project "-p:BenchmarkDotNetVersion=$version" '-tl:off' | Write-Output +# The project assigns RestoreAdditionalProjectSources too, but a global property wins over that assignment, which is +# what makes a custom -ArtifactsDirectory restore from the folder the version was read off. Only the restore needs it: +# nuget.g.props bakes the result in for every later invocation. +Invoke-Dotnet restore $project "-p:BenchmarkDotNetVersion=$version" ` + "-p:RestoreAdditionalProjectSources=$ArtifactsDirectory" '-tl:off' | Write-Output Write-Output '##[endgroup]' Write-Output 'Checking how the packaged build files resolve the test platform:' From a9e3474d7ba3c983aacc74a94a329d2968bebc10 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 7 Sep 2026 09:52:54 +0100 Subject: [PATCH 076/110] Refactor generic benchmark type handling and errors Introduce GenericBenchmarkType struct for encapsulating Type and error state in BenchmarkDotNet.Helpers. Update GenericBenchmarksBuilder to use this struct instead of tuples, improving clarity and error reporting. Add TryGetGenericTypeArguments for safe attribute reading. Enhance BuildGenericTypes to provide detailed failure reasons. Improves error handling and code readability for generic benchmarks. --- .../Helpers/GenericBenchmarkType.cs | 31 ++++++++++ .../Helpers/GenericBenchmarksBuilder.cs | 56 +++++++++++++++---- 2 files changed, 76 insertions(+), 11 deletions(-) create mode 100644 src/BenchmarkDotNet/Helpers/GenericBenchmarkType.cs diff --git a/src/BenchmarkDotNet/Helpers/GenericBenchmarkType.cs b/src/BenchmarkDotNet/Helpers/GenericBenchmarkType.cs new file mode 100644 index 0000000000..6ef0f3ff2c --- /dev/null +++ b/src/BenchmarkDotNet/Helpers/GenericBenchmarkType.cs @@ -0,0 +1,31 @@ +namespace BenchmarkDotNet.Helpers +{ + /// + /// A type that was considered for running benchmarks, and the reason it cannot be run when it cannot. + /// + internal readonly struct GenericBenchmarkType + { + private GenericBenchmarkType(Type type, string? error) + { + Type = type; + Error = error; + } + + /// + /// Gets the type. It is the type the benchmarks are read from when , and the type that + /// was rejected otherwise. + /// + internal Type Type { get; } + + /// + /// Gets the reason the type cannot be run, or null when it can. + /// + internal string? Error { get; } + + internal bool IsSuccess => Error == null; + + internal static GenericBenchmarkType Runnable(Type type) => new GenericBenchmarkType(type, null); + + internal static GenericBenchmarkType Failed(Type type, string error) => new GenericBenchmarkType(type, error); + } +} diff --git a/src/BenchmarkDotNet/Helpers/GenericBenchmarksBuilder.cs b/src/BenchmarkDotNet/Helpers/GenericBenchmarksBuilder.cs index 471087713c..3a86a54ea7 100644 --- a/src/BenchmarkDotNet/Helpers/GenericBenchmarksBuilder.cs +++ b/src/BenchmarkDotNet/Helpers/GenericBenchmarksBuilder.cs @@ -8,24 +8,58 @@ internal static class GenericBenchmarksBuilder internal static Type[] GetRunnableBenchmarks(IEnumerable types) => types.Where(type => type.ContainsRunnableBenchmarks()) .SelectMany(BuildGenericsIfNeeded) - .Where(x => x.isSuccess) - .Select(x => x.result) + .Where(x => x.IsSuccess) + .Select(x => x.Type) .ToArray(); - internal static IEnumerable<(bool isSuccess, Type result)> BuildGenericsIfNeeded(Type type) + internal static IEnumerable BuildGenericsIfNeeded(Type type) { - var typeArguments = type.GetCustomAttributes(true).OfType() - .Select(x => x.GenericTypeArguments) - .ToArray(); + if (!TryGetGenericTypeArguments(type, out var typeArguments, out var error)) + return [GenericBenchmarkType.Failed(type, error)]; - if (typeArguments.Any()) + if (typeArguments.Length > 0) return BuildGenericTypes(type, typeArguments); - return [(true, type)]; + return [GenericBenchmarkType.Runnable(type)]; + } + + /// + /// Reads the [GenericTypeArguments] of a type, if the attributes of that type can be read at all. + /// + /// + /// Reflection constructs every attribute of a type in order to hand any of them back, so a single attribute + /// whose constructor throws makes the whole read throw: [Config(typeof(SomeAbstractConfig))] fails inside + /// ConfigAttribute's own constructor, and there is no way to ask for the [GenericTypeArguments] alone. The + /// type is unusable once that happens - BenchmarkConverter would throw on the very same read - so it is + /// reported as a failure and dropped, rather than aborting the enumeration of every other benchmark in the + /// assembly. + /// + /// The type to read the attributes of. + /// The type argument sets the type is to be closed over. + /// The reason the attributes could not be read. + /// Whether the attributes could be read. + private static bool TryGetGenericTypeArguments(Type type, out Type[][] typeArguments, out string error) + { + try + { + typeArguments = type.GetCustomAttributes(true).OfType() + .Select(x => x.GenericTypeArguments) + .ToArray(); + error = string.Empty; + return true; + } + catch (Exception e) + { + typeArguments = []; + error = $"Type {type.Name} was ignored because its attributes could not be read: {e.Message}"; + return false; + } } - private static IEnumerable<(bool isSuccess, Type result)> BuildGenericTypes(Type type, IEnumerable typeArguments) - => typeArguments.Select(genericArg => (type.TryMakeGenericType(genericArg, out var builtType), builtType)); + private static IEnumerable BuildGenericTypes(Type type, IEnumerable typeArguments) + => typeArguments.Select(genericArg => type.TryMakeGenericType(genericArg, out var builtType) + ? GenericBenchmarkType.Runnable(builtType) + : GenericBenchmarkType.Failed(builtType, $"Generic type {builtType.Name} failed to build due to wrong type argument or arguments count, ignoring.")); private static bool TryMakeGenericType(this Type type, Type[] typeArguments, out Type result) { @@ -41,4 +75,4 @@ private static bool TryMakeGenericType(this Type type, Type[] typeArguments, out } } } -} \ No newline at end of file +} From 1f38c145fb821ccf3779b54ee599f492d36bcddd Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 7 Sep 2026 09:53:11 +0100 Subject: [PATCH 077/110] Add unoptimized test project for parameter disposal checks Added BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized with false and BenchmarkDotNet.TestAdapter reference. Introduced DroppedProbe and SharedValueProbe benchmarks to test parameter disposal in hidden/unoptimized scenarios. Implemented TrackedValue for instance tracking and disposal verification. Included README.md detailing project purpose and disposal issue coverage. --- ...onTests.TestingPlatform.Unoptimized.csproj | 33 ++++++++++++++++ .../DroppedProbe.cs | 33 ++++++++++++++++ .../README.md | 12 ++++++ .../SharedValueProbe.cs | 38 +++++++++++++++++++ .../TrackedValue.cs | 34 +++++++++++++++++ 5 files changed, 150 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized.csproj create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/DroppedProbe.cs create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/README.md create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/SharedValueProbe.cs create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/TrackedValue.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized.csproj b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized.csproj new file mode 100644 index 0000000000..fcd02ada6e --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized.csproj @@ -0,0 +1,33 @@ + + + + net10.0 + Exe + BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized + BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized + BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized + + + false + + + + + + + + + + + + diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/DroppedProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/DroppedProbe.cs new file mode 100644 index 0000000000..55a40e47b3 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/DroppedProbe.cs @@ -0,0 +1,33 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized +{ + /// + /// A benchmark that only runs out of process, so an unoptimized assembly hides every one of its cases and nothing + /// downstream can reach the parameter values they own. This is the leak: they are disposed by nobody unless the + /// enumeration disposes them itself. + /// + [Config(typeof(OutOfProcessConfig))] + public class DroppedProbe + { + // Created once, so that re-reading the source cannot change the count. + private static readonly TrackedValue[] Instances = [new TrackedValue("dropped-1"), new TrackedValue("dropped-2")]; + + public IEnumerable Values => Instances; + + [ParamsSource(nameof(Values))] + public TrackedValue? Value { get; set; } + + [Benchmark] + public int Length() => Value!.Name.Length; + + private class OutOfProcessConfig : ManualConfig + { + // A dry job on the default toolchain, which generates, builds and runs a separate executable - except + // that it is never reached here, because the enumeration hides it before anything is built. + public OutOfProcessConfig() => AddJob(Job.Dry.WithId("OutOfProcess")); + } + } +} diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/README.md b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/README.md new file mode 100644 index 0000000000..27127548b5 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/README.md @@ -0,0 +1,12 @@ +# BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized + +A Microsoft.Testing.Platform application built with `false`, which is the one thing that makes it +different from `BenchmarkDotNet.IntegrationTests.TestingPlatform`. + +`BenchmarkEnumerator` hides the benchmarks that would run out of process when the assembly is not optimized, so that +they can be debugged from a test runner. Those benchmarks are enumerated first and BenchmarkDotNet never sees them +afterwards, so the parameter values they own are the adapter's to dispose - a value with a locking finalizer hangs the +runtime otherwise, see dotnet/BenchmarkDotNet#1383. Every other project in this repository is optimized, deliberately, +which leaves that path unreachable in a Release CI run. + +`TestingPlatformAdapterTests` in `BenchmarkDotNet.IntegrationTests` drives it. diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/SharedValueProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/SharedValueProbe.cs new file mode 100644 index 0000000000..5901939dfc --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/SharedValueProbe.cs @@ -0,0 +1,38 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized +{ + /// + /// A benchmark that runs under an in-process and an out-of-process job over the same parameter values. An + /// unoptimized assembly hides the out-of-process cases only, and BenchmarkConverter hands the very same + /// ParameterInstance to both jobs, so disposing what is hidden case by case would take down values the surviving + /// benchmarks still own. + /// + [Config(typeof(BothToolchainsConfig))] + public class SharedValueProbe + { + // Created once, so that re-reading the source cannot change the count. + private static readonly TrackedValue[] Instances = [new TrackedValue("shared-1"), new TrackedValue("shared-2")]; + + public IEnumerable Values => Instances; + + [ParamsSource(nameof(Values))] + public TrackedValue? Value { get; set; } + + [Benchmark] + public int Length() => Value!.Name.Length; + + private class BothToolchainsConfig : ManualConfig + { + // The ids are set explicitly, so that the two jobs stay distinguishable by name. + public BothToolchainsConfig() + { + AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default).WithId("InProcess")); + AddJob(Job.Dry.WithId("OutOfProcess")); + } + } + } +} diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/TrackedValue.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/TrackedValue.cs new file mode 100644 index 0000000000..eefb07a368 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized/TrackedValue.cs @@ -0,0 +1,34 @@ +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized +{ + /// + /// A disposable parameter value that keeps count of how many of its kind were created and disposed. What is left + /// undisposed only shows at the end, so the counts are written out when the process exits. + /// + public sealed class TrackedValue : IDisposable + { + /// + /// The name of the file the counts are written to, next to the probe application. + /// + public const string ReportFileName = "unoptimized-probe.txt"; + + private static int created; + private static int disposed; + + static TrackedValue() => + AppDomain.CurrentDomain.ProcessExit += (_, _) => File.WriteAllText( + Path.Combine(AppContext.BaseDirectory, ReportFileName), + $"created={Volatile.Read(ref created)} disposed={Volatile.Read(ref disposed)}"); + + public TrackedValue(string name) + { + Name = name; + Interlocked.Increment(ref created); + } + + public string Name { get; } + + public void Dispose() => Interlocked.Increment(ref disposed); + + public override string ToString() => Name; + } +} From 63beac27a1877d286b64d86e4114ad96272cc096 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 7 Sep 2026 09:53:44 +0100 Subject: [PATCH 078/110] Remove obsolete AppDomain remoting wrappers Removed BenchmarkEnumeratorWrapper, BenchmarkExecutorWrapper, MessageLoggerWrapper, SerializationHelpers, and TestExecutionRecorderWrapper from BenchmarkDotNet.TestAdapter.Remoting. These classes enabled AppDomain boundary communication for VSTest integration; their removal reflects a refactor or architectural change eliminating the need for such wrappers. --- .../Remoting/BenchmarkEnumeratorWrapper.cs | 31 --------------- .../Remoting/BenchmarkExecutorWrapper.cs | 20 ---------- .../Remoting/MessageLoggerWrapper.cs | 22 ----------- .../Remoting/SerializationHelpers.cs | 26 ------------- .../Remoting/TestExecutionRecorderWrapper.cs | 38 ------------------- 5 files changed, 137 deletions(-) delete mode 100644 src/BenchmarkDotNet.TestAdapter/Remoting/BenchmarkEnumeratorWrapper.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter/Remoting/BenchmarkExecutorWrapper.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter/Remoting/MessageLoggerWrapper.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter/Remoting/SerializationHelpers.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter/Remoting/TestExecutionRecorderWrapper.cs diff --git a/src/BenchmarkDotNet.TestAdapter/Remoting/BenchmarkEnumeratorWrapper.cs b/src/BenchmarkDotNet.TestAdapter/Remoting/BenchmarkEnumeratorWrapper.cs deleted file mode 100644 index 9012328a1f..0000000000 --- a/src/BenchmarkDotNet.TestAdapter/Remoting/BenchmarkEnumeratorWrapper.cs +++ /dev/null @@ -1,31 +0,0 @@ -namespace BenchmarkDotNet.TestAdapter.Remoting -{ - /// - /// A wrapper around the BenchmarkEnumerator for passing data across AppDomain boundaries. - /// - internal class BenchmarkEnumeratorWrapper : MarshalByRefObject - { - /// - /// Gets a list of VSTest TestCases from the given assembly. - /// Each test case is serialized into a string so that it can be used across AppDomain boundaries. - /// - /// The dll or exe of the benchmark project. - /// The serialized test cases. - public List GetTestCasesFromAssemblyPathSerialized(string assemblyPath) - { - var serializedTestCases = new List(); - foreach (var runInfo in BenchmarkEnumerator.GetBenchmarksFromAssemblyPath(assemblyPath)) - { - // If all the benchmarks have the same job, then no need to include job info. - var needsJobInfo = runInfo.BenchmarksCases.Select(c => c.Job.DisplayInfo).Distinct().Count() > 1; - foreach (var benchmarkCase in runInfo.BenchmarksCases) - { - var testCase = benchmarkCase.ToVsTestCase(assemblyPath, needsJobInfo); - serializedTestCases.Add(SerializationHelpers.Serialize(testCase)); - } - } - - return serializedTestCases; - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter/Remoting/BenchmarkExecutorWrapper.cs b/src/BenchmarkDotNet.TestAdapter/Remoting/BenchmarkExecutorWrapper.cs deleted file mode 100644 index 98b5b9a35f..0000000000 --- a/src/BenchmarkDotNet.TestAdapter/Remoting/BenchmarkExecutorWrapper.cs +++ /dev/null @@ -1,20 +0,0 @@ -namespace BenchmarkDotNet.TestAdapter.Remoting -{ - /// - /// A wrapper around the BenchmarkExecutor that works across AppDomain boundaries. - /// - internal class BenchmarkExecutorWrapper : MarshalByRefObject - { - private readonly BenchmarkExecutor benchmarkExecutor = new(); - - public void RunBenchmarks(string assemblyPath, TestExecutionRecorderWrapper recorder, HashSet? benchmarkIds = null) - { - benchmarkExecutor.RunBenchmarks(assemblyPath, recorder, benchmarkIds); - } - - public void Cancel() - { - benchmarkExecutor.Cancel(); - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter/Remoting/MessageLoggerWrapper.cs b/src/BenchmarkDotNet.TestAdapter/Remoting/MessageLoggerWrapper.cs deleted file mode 100644 index 9d2bc6f3b3..0000000000 --- a/src/BenchmarkDotNet.TestAdapter/Remoting/MessageLoggerWrapper.cs +++ /dev/null @@ -1,22 +0,0 @@ -using Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging; - -namespace BenchmarkDotNet.TestAdapter.Remoting -{ - /// - /// A wrapper around an IMessageLogger that works across AppDomain boundaries. - /// - internal class MessageLoggerWrapper : MarshalByRefObject, IMessageLogger - { - private readonly IMessageLogger logger; - - public MessageLoggerWrapper(IMessageLogger logger) - { - this.logger = logger; - } - - public void SendMessage(TestMessageLevel testMessageLevel, string message) - { - logger.SendMessage(testMessageLevel, message); - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter/Remoting/SerializationHelpers.cs b/src/BenchmarkDotNet.TestAdapter/Remoting/SerializationHelpers.cs deleted file mode 100644 index 7a99a741bc..0000000000 --- a/src/BenchmarkDotNet.TestAdapter/Remoting/SerializationHelpers.cs +++ /dev/null @@ -1,26 +0,0 @@ -using Microsoft.VisualStudio.TestPlatform.CommunicationUtilities; - -namespace BenchmarkDotNet.TestAdapter.Remoting -{ - /// - /// A set of helper methods for serializing and deserializing the VSTest TestCases and TestReports. - /// - internal static class SerializationHelpers - { - // Version number of the VSTest protocol that the adapter supports. Only needs to be updated when - // the VSTest protocol has a change and this test adapter wishes to take a dependency on it. - // A list of protocol versions and a summary of the changes that were made in them can be found here: - // https://github.com/microsoft/vstest/blob/main/docs/Overview.md#protocolversion-request - private const int VsTestProtocolVersion = 7; - - public static string Serialize(T data) - { - return JsonDataSerializer.Instance.Serialize(data, version: VsTestProtocolVersion); - } - - public static T Deserialize(string data) - { - return JsonDataSerializer.Instance.Deserialize(data, version: VsTestProtocolVersion)!; - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter/Remoting/TestExecutionRecorderWrapper.cs b/src/BenchmarkDotNet.TestAdapter/Remoting/TestExecutionRecorderWrapper.cs deleted file mode 100644 index 728039ade8..0000000000 --- a/src/BenchmarkDotNet.TestAdapter/Remoting/TestExecutionRecorderWrapper.cs +++ /dev/null @@ -1,38 +0,0 @@ -using Microsoft.VisualStudio.TestPlatform.ObjectModel; -using Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter; - -namespace BenchmarkDotNet.TestAdapter.Remoting -{ - /// - /// A wrapper around the ITestExecutionRecorder which works across AppDomain boundaries. - /// - internal class TestExecutionRecorderWrapper : MarshalByRefObject - { - private readonly ITestExecutionRecorder testExecutionRecorder; - - public TestExecutionRecorderWrapper(ITestExecutionRecorder testExecutionRecorder) - { - this.testExecutionRecorder = testExecutionRecorder; - } - - public MessageLoggerWrapper GetLogger() - { - return new MessageLoggerWrapper(testExecutionRecorder); - } - - internal void RecordStart(string serializedTestCase) - { - testExecutionRecorder.RecordStart(SerializationHelpers.Deserialize(serializedTestCase)); - } - - internal void RecordEnd(string serializedTestCase, TestOutcome testOutcome) - { - testExecutionRecorder.RecordEnd(SerializationHelpers.Deserialize(serializedTestCase), testOutcome); - } - - internal void RecordResult(string serializedTestResult) - { - testExecutionRecorder.RecordResult(SerializationHelpers.Deserialize(serializedTestResult)); - } - } -} \ No newline at end of file From 35b58a92fc5e67b4d22888b6adeb02cb600cdbff Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 7 Sep 2026 09:53:58 +0100 Subject: [PATCH 079/110] Add wrappers for cross-AppDomain VSTest communication Introduced BenchmarkDotNet.TestAdapter.VSTest.Remoting namespace with wrappers for enumerating, executing, and logging benchmarks across AppDomain boundaries. Added serialization helpers for TestCases and TestReports using VSTest protocol. Implemented wrappers for IMessageLogger and ITestExecutionRecorder to support remote logging and test event recording. --- .../Remoting/BenchmarkEnumeratorWrapper.cs | 31 +++++++++++++++ .../Remoting/BenchmarkExecutorWrapper.cs | 20 ++++++++++ .../VSTest/Remoting/MessageLoggerWrapper.cs | 22 +++++++++++ .../VSTest/Remoting/SerializationHelpers.cs | 26 +++++++++++++ .../Remoting/TestExecutionRecorderWrapper.cs | 38 +++++++++++++++++++ 5 files changed, 137 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/BenchmarkEnumeratorWrapper.cs create mode 100644 src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/BenchmarkExecutorWrapper.cs create mode 100644 src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/MessageLoggerWrapper.cs create mode 100644 src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/SerializationHelpers.cs create mode 100644 src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/TestExecutionRecorderWrapper.cs diff --git a/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/BenchmarkEnumeratorWrapper.cs b/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/BenchmarkEnumeratorWrapper.cs new file mode 100644 index 0000000000..eb9a888896 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/BenchmarkEnumeratorWrapper.cs @@ -0,0 +1,31 @@ +namespace BenchmarkDotNet.TestAdapter.VSTest.Remoting +{ + /// + /// A wrapper around the BenchmarkEnumerator for passing data across AppDomain boundaries. + /// + internal class BenchmarkEnumeratorWrapper : MarshalByRefObject + { + /// + /// Gets a list of VSTest TestCases from the given assembly. + /// Each test case is serialized into a string so that it can be used across AppDomain boundaries. + /// + /// The dll or exe of the benchmark project. + /// The serialized test cases. + public List GetTestCasesFromAssemblyPathSerialized(string assemblyPath) + { + var serializedTestCases = new List(); + foreach (var runInfo in BenchmarkEnumerator.GetBenchmarksFromAssemblyPath(assemblyPath)) + { + // If all the benchmarks have the same job, then no need to include job info. + var needsJobInfo = runInfo.BenchmarksCases.Select(c => c.Job.DisplayInfo).Distinct().Count() > 1; + foreach (var benchmarkCase in runInfo.BenchmarksCases) + { + var testCase = benchmarkCase.ToVsTestCase(assemblyPath, needsJobInfo); + serializedTestCases.Add(SerializationHelpers.Serialize(testCase)); + } + } + + return serializedTestCases; + } + } +} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/BenchmarkExecutorWrapper.cs b/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/BenchmarkExecutorWrapper.cs new file mode 100644 index 0000000000..9318769a8c --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/BenchmarkExecutorWrapper.cs @@ -0,0 +1,20 @@ +namespace BenchmarkDotNet.TestAdapter.VSTest.Remoting +{ + /// + /// A wrapper around the BenchmarkExecutor that works across AppDomain boundaries. + /// + internal class BenchmarkExecutorWrapper : MarshalByRefObject + { + private readonly BenchmarkExecutor benchmarkExecutor = new(); + + public void RunBenchmarks(string assemblyPath, TestExecutionRecorderWrapper recorder, HashSet? benchmarkIds = null) + { + benchmarkExecutor.RunBenchmarks(assemblyPath, recorder, benchmarkIds); + } + + public void Cancel() + { + benchmarkExecutor.Cancel(); + } + } +} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/MessageLoggerWrapper.cs b/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/MessageLoggerWrapper.cs new file mode 100644 index 0000000000..5d04e17739 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/MessageLoggerWrapper.cs @@ -0,0 +1,22 @@ +using Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging; + +namespace BenchmarkDotNet.TestAdapter.VSTest.Remoting +{ + /// + /// A wrapper around an IMessageLogger that works across AppDomain boundaries. + /// + internal class MessageLoggerWrapper : MarshalByRefObject, IMessageLogger + { + private readonly IMessageLogger logger; + + public MessageLoggerWrapper(IMessageLogger logger) + { + this.logger = logger; + } + + public void SendMessage(TestMessageLevel testMessageLevel, string message) + { + logger.SendMessage(testMessageLevel, message); + } + } +} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/SerializationHelpers.cs b/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/SerializationHelpers.cs new file mode 100644 index 0000000000..872530190e --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/SerializationHelpers.cs @@ -0,0 +1,26 @@ +using Microsoft.VisualStudio.TestPlatform.CommunicationUtilities; + +namespace BenchmarkDotNet.TestAdapter.VSTest.Remoting +{ + /// + /// A set of helper methods for serializing and deserializing the VSTest TestCases and TestReports. + /// + internal static class SerializationHelpers + { + // Version number of the VSTest protocol that the adapter supports. Only needs to be updated when + // the VSTest protocol has a change and this test adapter wishes to take a dependency on it. + // A list of protocol versions and a summary of the changes that were made in them can be found here: + // https://github.com/microsoft/vstest/blob/main/docs/Overview.md#protocolversion-request + private const int VsTestProtocolVersion = 7; + + public static string Serialize(T data) + { + return JsonDataSerializer.Instance.Serialize(data, version: VsTestProtocolVersion); + } + + public static T Deserialize(string data) + { + return JsonDataSerializer.Instance.Deserialize(data, version: VsTestProtocolVersion)!; + } + } +} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/TestExecutionRecorderWrapper.cs b/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/TestExecutionRecorderWrapper.cs new file mode 100644 index 0000000000..7625bd444a --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/VSTest/Remoting/TestExecutionRecorderWrapper.cs @@ -0,0 +1,38 @@ +using Microsoft.VisualStudio.TestPlatform.ObjectModel; +using Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter; + +namespace BenchmarkDotNet.TestAdapter.VSTest.Remoting +{ + /// + /// A wrapper around the ITestExecutionRecorder which works across AppDomain boundaries. + /// + internal class TestExecutionRecorderWrapper : MarshalByRefObject + { + private readonly ITestExecutionRecorder testExecutionRecorder; + + public TestExecutionRecorderWrapper(ITestExecutionRecorder testExecutionRecorder) + { + this.testExecutionRecorder = testExecutionRecorder; + } + + public MessageLoggerWrapper GetLogger() + { + return new MessageLoggerWrapper(testExecutionRecorder); + } + + internal void RecordStart(string serializedTestCase) + { + testExecutionRecorder.RecordStart(SerializationHelpers.Deserialize(serializedTestCase)); + } + + internal void RecordEnd(string serializedTestCase, TestOutcome testOutcome) + { + testExecutionRecorder.RecordEnd(SerializationHelpers.Deserialize(serializedTestCase), testOutcome); + } + + internal void RecordResult(string serializedTestResult) + { + testExecutionRecorder.RecordResult(SerializationHelpers.Deserialize(serializedTestResult)); + } + } +} \ No newline at end of file From 6d7829732f9ed36acf8481e13a6eb35ecd6d099d Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 7 Sep 2026 09:55:09 +0100 Subject: [PATCH 080/110] Add unoptimized probe & enhance benchmark discovery tests Add BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized to test suite and update .csproj references. Add and update tests in TestingPlatformAdapterTests.cs for unoptimized assembly discovery, benchmark description formatting, ECMA-335 type name usage, and handling of special characters in benchmark names. Refactor ReadDisposalReport for multiple probe projects. Extend DiscoveredTest with TypeName and update Discover method to extract it. --- .../BenchmarkDotNet.IntegrationTests.csproj | 1 + .../TestingPlatformAdapterTests.cs | 108 +++++++++++++++--- 2 files changed, 96 insertions(+), 13 deletions(-) diff --git a/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj b/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj index cbf674e9d1..0eeff3cf50 100644 --- a/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj +++ b/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj @@ -48,6 +48,7 @@ + diff --git a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs index 93a9d63812..a49a6e59e1 100644 --- a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs +++ b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs @@ -17,8 +17,9 @@ public class TestingPlatformAdapterTests(ITestOutputHelper output) { private const string PassingProbes = "BenchmarkDotNet.IntegrationTests.TestingPlatform"; private const string FailingProbes = "BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures"; + private const string UnoptimizedProbes = "BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized"; - // Both probe projects are single targeted, see their .csproj files. + // Every probe project is single targeted, see their .csproj files. private const string ProbeTargetFramework = "net10.0"; // A run that has to build a benchmark pays for a restore and a build of the generated project. @@ -29,12 +30,15 @@ public void EveryBenchmarkIsDiscoveredUnderItsOwnName() { string[] expected = [ + "BracketProbe.Length(Value: \"[Dry]\")", "CategoryProbe.Identity", - // The description of a [Benchmark(Description = ...)] is what a user recognises it by, so it is used - // instead of the method name. Without one the method name is used, and the parameters are appended to - // both. - "DescribedProbe.'A described benchmark'(Size: 1)", + // The description of a [Benchmark(Description = ...)] is what a user recognises it by, so it is + // used instead of the method name, and spelled the way it was written. Descriptor quotes a + // description containing a space so that BenchmarkDotNet's own --filter can delimit it, which an IDE + // label has no use for. Without a description the method name is used, and the parameters are + // appended to both. + "DescribedProbe.A described benchmark(Size: 1)", "DescribedProbe.Undescribed(Size: 1)", "DisposableProbe.Identity(Value: tracked-1)", @@ -46,6 +50,7 @@ public void EveryBenchmarkIsDiscoveredUnderItsOwnName() "GenericProbe>.Create", "GenericProbe.Create", + "NestedProbe.Inner.Identity", "OutOfProcessProbe.Add", "SampleBenchmarks.Add(Size: 1)", "SampleBenchmarks.Add(Size: 2)", @@ -56,6 +61,8 @@ public void EveryBenchmarkIsDiscoveredUnderItsOwnName() var discovered = Discover(PassingProbes); + // InvalidConfigProbe's benchmarks are deliberately absent: their [Config] cannot be constructed, and a + // type whose attributes cannot be read is dropped rather than allowed to abort the whole discovery. Assert.Equal( expected, discovered.Select(test => test.DisplayName.Substring(PassingProbes.Length + 1)).OrderBy(name => name, StringComparer.Ordinal)); @@ -64,6 +71,27 @@ public void EveryBenchmarkIsDiscoveredUnderItsOwnName() Assert.Equal(discovered.Count, discovered.Select(test => test.Uid).Distinct().Count()); } + [Fact] + public void TheTypeOfABenchmarkIsIdentifiedByItsEcmaName() + { + // Microsoft.Testing.Platform documents TestMethodIdentifierProperty as ECMA-335, which is the form a + // test runner - Visual Studio's Test Explorer above all - matches a type by. A generic type is named + // after its arity there, and its arguments are no part of it. + var generic = Discover(PassingProbes, "--treenode-filter", "/*/*/GenericProbe*/*"); + + Assert.Equal(3, generic.Count); + Assert.All(generic, test => Assert.Equal("GenericProbe`1", test.TypeName)); + + // The arguments are still what tells one closed generic from another, in the name the user reads. + Assert.Equal(3, generic.Select(test => test.DisplayName).Distinct(StringComparer.Ordinal).Count()); + + // A nested type is qualified by its declaring types rather than by its namespace, which the property + // carries separately. + var nested = Discover(PassingProbes, "--treenode-filter", "/*/*/NestedProbe*/*"); + + Assert.Equal("NestedProbe+Inner", Assert.Single(nested).TypeName); + } + [Fact] public void TheUidOfABenchmarkIsTheSameInEveryProcess() { @@ -116,7 +144,9 @@ public void ParameterValuesAreDisposedWhenBenchmarksAreOnlyListed() { // Listing runs nothing, so BenchmarkDotNet disposes nothing: every value the enumeration created is the // adapter's to dispose. - Assert.Equal("created=3 disposed=3", ReadDisposalReport(() => Discover(PassingProbes))); + Assert.Equal( + "created=3 disposed=3", + ReadDisposalReport(PassingProbes, "disposable-probe.txt", () => Discover(PassingProbes))); } [Fact] @@ -129,11 +159,39 @@ public void ParameterValuesAreDisposedWhenOnlyOneBenchmarkOfASetIsRun() .Single(test => test.DisplayName.EndsWith("DisposableProbe.Identity(Value: tracked-1)", StringComparison.Ordinal)) .Uid; - var report = ReadDisposalReport(() => RunAndSummarize(PassingProbes, "--filter-uid", uid)); + var report = ReadDisposalReport( + PassingProbes, + "disposable-probe.txt", + () => RunAndSummarize(PassingProbes, "--filter-uid", uid)); Assert.Equal("created=3 disposed=3", report); } + [Fact] + public void OutOfProcessBenchmarksAreHiddenWhenTheAssemblyIsNotOptimized() + { + // The point of the unoptimized probe application: a benchmark that would leave the process is hidden, so + // that it can be debugged from a test runner. DroppedProbe has no other job and disappears entirely, + // SharedValueProbe keeps its in-process cases - which is also why the job is no part of their names. + var discovered = Discover(UnoptimizedProbes); + + Assert.Equal( + new[] { "SharedValueProbe.Length(Value: shared-1)", "SharedValueProbe.Length(Value: shared-2)" }, + discovered.Select(test => test.DisplayName.Substring(UnoptimizedProbes.Length + 1)).OrderBy(name => name, StringComparer.Ordinal)); + } + + [Fact] + public void ParameterValuesAreDisposedWhenBenchmarksAreHiddenByAnUnoptimizedAssembly() + { + // The benchmarks hidden above are never handed to BenchmarkDotNet by either adapter, so the values they + // own are the enumeration's to dispose: the two of DroppedProbe are unreachable from anything that + // survives. The two of SharedValueProbe are shared with cases that do survive, so disposing them here + // would be a disposal too many, which the count catches just as well as a leak. + var report = ReadDisposalReport(UnoptimizedProbes, "unoptimized-probe.txt", () => Discover(UnoptimizedProbes)); + + Assert.Equal("created=4 disposed=4", report); + } + [Fact] public void ATreeNodeFilterMatchesTheClassAndTheMethodOfABenchmark() { @@ -155,6 +213,25 @@ public void ABenchmarkStaysAtTheSameLevelOfTheTreeWhenAParameterContainsTheSepar Assert.Contains("SeparatorProbe.Length(Value: \"a/b\")", discovered[0].DisplayName, StringComparison.Ordinal); } + [Fact] + public void ABenchmarkIsAddressableWhenItsPathContainsAPropertyFilterDelimiter() + { + // A TreeNodeFilter reads '[' and ']' as the delimiters of a property filter, so a segment carrying them + // has to be encoded rather than left to be parsed - unlike the parentheses around the parameters, which a + // filter escapes with a backslash. That is true of a parameter that contains them... + var byParameter = Discover(PassingProbes, "--treenode-filter", @"/*/*/BracketProbe/Length\(Value: ""%5BDry%5D""\)*"); + + Assert.Single(byParameter); + Assert.Contains("BracketProbe.Length(Value: \"[Dry]\")", byParameter[0].DisplayName, StringComparison.Ordinal); + + // ...and of the job that every leaf ends in, which is what an exact path would otherwise trip over. The + // trailing wildcard stands in for the job name, so that this does not pin how a job is displayed. + var byJob = Discover(PassingProbes, "--treenode-filter", @"/*/*/BracketProbe/Length\(Value: ""%5BDry%5D""\) %5B*"); + + Assert.Single(byJob); + Assert.Equal(byParameter[0].Uid, byJob[0].Uid); + } + [Fact] public void AnOutOfProcessBenchmarkIsBuiltAndRun() { @@ -210,13 +287,15 @@ public void BenchmarksSharingAUidThroughTheirDescriptionAreReportedWithTheirMeth /// /// The counts are written to a file rather than to the output, because the discovery output is parsed as json. /// + /// The probe application that writes the counts. + /// The name of the file the probe writes them to. /// The way the probe application is driven. /// The counts the probe reported when it exited. - private static string ReadDisposalReport(Action execute) + private static string ReadDisposalReport(string project, string reportFileName, Action execute) { - // The probe projects are referenced with ReferenceOutputAssembly="false", so the name is repeated here - // rather than taken from DisposableProbe.ReportFileName. - var report = Path.Combine(Path.GetDirectoryName(GetProbeApplication(PassingProbes))!, "disposable-probe.txt"); + // The probe projects are referenced with ReferenceOutputAssembly="false", so the names are repeated here + // rather than taken from the ReportFileName constants of the probes themselves. + var report = Path.Combine(Path.GetDirectoryName(GetProbeApplication(project))!, reportFileName); File.Delete(report); execute(); @@ -236,7 +315,10 @@ private IReadOnlyList Discover(string project, params string[] a return document.RootElement.GetProperty("tests") .EnumerateArray() - .Select(test => new DiscoveredTest(test.GetProperty("uid").GetString()!, test.GetProperty("displayName").GetString()!)) + .Select(test => new DiscoveredTest( + test.GetProperty("uid").GetString()!, + test.GetProperty("displayName").GetString()!, + test.GetProperty("type").GetProperty("typeName").GetString()!)) .ToArray(); } @@ -310,7 +392,7 @@ private static string GetProbeApplication(string project) return path; } - private sealed record DiscoveredTest(string Uid, string DisplayName); + private sealed record DiscoveredTest(string Uid, string DisplayName, string TypeName); private sealed record TestRunSummary(int Total, int Failed, int Succeeded, int Skipped) { From 6eb58893259c060e18d7722d28b789101791b821 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 7 Sep 2026 09:55:39 +0100 Subject: [PATCH 081/110] Refactor disposal logic, improve display names & ECMA names Refactored disposal of unused parameter values by replacing DisposeUnusedParameterValues with ParameterValueDisposer.DisposeUnused on flattened benchmark cases. Improved display name logic to use [Benchmark(Description = ...)] attributes and avoid quoting for IDE labels. Added GetEcmaTypeName to generate ECMA-335 compliant type names for TestMethodIdentifierProperty, handling generics and nesting. Enhanced Escape method to percent-encode '[', ']', and '/' for TreeNodeFilter compatibility. Removed ReferenceComparer and related code as disposal is now handled elsewhere. --- .../TestingPlatform/BenchmarkTestFramework.cs | 51 ++--------------- .../TestingPlatform/BenchmarkTestNode.cs | 55 +++++++++++++++---- 2 files changed, 47 insertions(+), 59 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs index ed5def52f1..718ca2f4b2 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs @@ -9,7 +9,6 @@ using Microsoft.Testing.Platform.Services; using Microsoft.Testing.Platform.TestHost; using System.Reflection; -using System.Runtime.CompilerServices; using System.Runtime.ExceptionServices; using System.Threading.Channels; @@ -105,7 +104,7 @@ private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRe finally { // Discovery runs nothing, so every value the enumeration created is this method's to dispose. - DisposeUnusedParameterValues(enumeration.All, []); + ParameterValueDisposer.DisposeUnused(enumeration.All.SelectMany(runInfo => runInfo.BenchmarksCases), []); } } @@ -126,7 +125,9 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte // A benchmark that was filtered out or that collided is never handed to BenchmarkDotNet, so nothing else // would dispose the values the enumeration created for it. - DisposeUnusedParameterValues(enumeration.All, runnable.Select(match => match.Node.BenchmarkCase)); + ParameterValueDisposer.DisposeUnused( + enumeration.All.SelectMany(runInfo => runInfo.BenchmarksCases), + runnable.Select(match => match.Node.BenchmarkCase)); if (runnable.Count == 0) return; @@ -321,38 +322,6 @@ private Enumeration GetMatchingBenchmarks(ITestExecutionFilter filter) return new Enumeration(matches, runInfos); } - /// - /// Disposes the parameter values of the benchmarks that were enumerated but will not be run. - /// - /// - /// Enumerating an assembly instantiates the values of every [Params] and [ArgumentsSource], and BenchmarkDotNet - /// only disposes the ones belonging to the benchmarks it was handed. The values are matched by reference - /// instead of being disposed case by case, because BenchmarkConverter gives the same ParameterInstance to - /// every job and every argument set of a benchmark: disposing a filtered out case wholesale would take down - /// values that a benchmark which is about to run still owns. - /// - /// Everything the assembly declares. - /// The benchmarks that are going to be run, if any. - private static void DisposeUnusedParameterValues(BenchmarkRunInfo[] enumerated, IEnumerable retained) - { - var unused = new HashSet(ReferenceComparer.Instance); - - foreach (var value in GetDisposableParameterValues(enumerated.SelectMany(runInfo => runInfo.BenchmarksCases))) - unused.Add(value); - - foreach (var value in GetDisposableParameterValues(retained)) - unused.Remove(value); - - foreach (var value in unused) - value.Dispose(); - } - - private static IEnumerable GetDisposableParameterValues(IEnumerable benchmarkCases) - => benchmarkCases - .SelectMany(benchmarkCase => benchmarkCase.Parameters.Items) - .Select(parameter => parameter.Value) - .OfType(); - #pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. private static bool Matches(ITestExecutionFilter filter, BenchmarkTestNode node) => filter switch { @@ -386,18 +355,6 @@ public Enumeration(List> matches, BenchmarkRunInfo[] all) public BenchmarkRunInfo[] All { get; } } - /// - /// Compares by reference, so that a parameter value which overrides Equals is still disposed once per instance. - /// - private sealed class ReferenceComparer : IEqualityComparer - { - public static readonly ReferenceComparer Instance = new ReferenceComparer(); - - public bool Equals(IDisposable? x, IDisposable? y) => ReferenceEquals(x, y); - - public int GetHashCode(IDisposable obj) => RuntimeHelpers.GetHashCode(obj); - } - /// /// A benchmark that matched the request, together with the run info it belongs to. /// diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs index d6c30b1de6..da241cc1d1 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs @@ -72,10 +72,16 @@ public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool include var uid = benchmarkCase.GetUniqueId(); // Microsoft.Testing.Platform keeps the display name and the identity apart, so the name is free to be the - // [Benchmark(Description = ...)] the author chose. GetMethodDisplayName falls back to the method name when - // no description is set. The path keeps the method name, so that a filter still matches what - // BenchmarkDotNet's own --filter matches. - var displayMethodName = FullNameProvider.GetMethodDisplayName(benchmarkCase); + // [Benchmark(Description = ...)] the author chose, spelled the way it was written: Descriptor's + // WorkloadMethodDisplayInfo is the console table form, which quotes a description containing a space or a + // bracket so that BenchmarkDotNet's own --filter can delimit it, and an IDE label does not want that. It + // is still the fallback, so that a hand-built Descriptor with no attribute keeps its name. The path keeps + // the method name, so that a filter still matches what --filter matches. + var benchmarkAttribute = benchmarkMethod.ResolveAttribute(); + var benchmarkName = string.IsNullOrEmpty(benchmarkAttribute?.Description) + ? benchmarkCase.Descriptor.WorkloadMethodDisplayInfo + : benchmarkAttribute!.Description!; + var displayMethodName = FullNameProvider.GetMethodDisplayName(benchmarkCase, benchmarkName); var displayName = $"{fullClassName}.{displayMethodName}" + (includeJobInName ? $" [{jobDisplayInfo}]" : ""); var properties = new List @@ -83,14 +89,13 @@ public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool include new TestMethodIdentifierProperty( type.Assembly.FullName, type.Namespace ?? string.Empty, - type.GetCorrectCSharpTypeName(prefixWithGlobal: false, includeNamespace: false), + GetEcmaTypeName(type), benchmarkMethod.Name, benchmarkMethod.IsGenericMethodDefinition ? benchmarkMethod.GetGenericArguments().Length : 0, benchmarkMethod.GetParameters().Select(p => p.ParameterType.FullName ?? p.ParameterType.Name).ToArray(), benchmarkMethod.ReturnType.FullName ?? benchmarkMethod.ReturnType.Name), }; - var benchmarkAttribute = benchmarkMethod.ResolveAttribute(); if (benchmarkAttribute?.SourceCodeFile != null) { // BenchmarkAttribute captures the line of the attribute itself, and the platform expects 0-based lines. @@ -139,6 +144,28 @@ public TestNode ToTestNode(TestNodeStateProperty state, params IProperty[] extra /// The filterable properties. public PropertyBag GetFilterableProperties() => new PropertyBag(staticProperties); + /// + /// Gets the name of a type in the form Microsoft.Testing.Platform documents for + /// , which is the ECMA-335 one rather than the C# one. + /// + /// + /// A generic type is named after its arity, `GenericProbe`1`, and its type arguments are no part of the name - + /// they belong to the display name, which carries them. A nested type is qualified by its declaring types, + /// separated by '+'; the namespace is left out, because the property carries it separately. + /// + /// The type declaring the benchmark. + /// The ECMA-335 name of the type. + private static string GetEcmaTypeName(Type type) + { + // Type.Name is already the arity form, for an open and for a closed generic type alike. + var name = type.Name; + + for (var declaringType = type.DeclaringType; declaringType != null; declaringType = declaringType.DeclaringType) + name = declaringType.Name + "+" + name; + + return name; + } + private static string BuildPath(Assembly assembly, string? @namespace, string fullClassName, string methodName, string jobDisplayInfo) { // The convention followed by the other test frameworks is ////. @@ -154,11 +181,15 @@ private static string BuildPath(Assembly assembly, string? @namespace, string fu .ToString(); } - // Benchmark parameters are stringified user values, so they can contain the path separator. A '/' cannot be - // escaped into a segment: Microsoft.Testing.Platform splits the path on every '/' without ever unescaping it, - // and TreeNodeFilter rejects a filter whose segment contains one, so a raw '/' would both deepen the tree and - // leave the benchmark unmatchable. Percent encoding keeps the path four levels deep and the segment - // addressable, at the price of a filter having to spell the separator as '%2F'. - private static string Escape(string segment) => segment.Replace("%", "%25").Replace("/", "%2F"); + // Benchmark parameters are stringified user values, and the leaf ends in the job between brackets, so a + // segment can contain the characters TreeNodeFilter gives a meaning to. None of them can be escaped into a + // segment: Microsoft.Testing.Platform splits the path on every '/' without ever unescaping it, and + // TreeNodeFilter rejects a filter whose segment contains one, so a raw '/' would both deepen the tree and + // leave the benchmark unmatchable; '[' and ']' delimit a property filter, so a filter spelling a leaf out in + // full would have its ' [Dry]' parsed as one instead of matched. Percent encoding keeps the path four levels + // deep and every segment addressable, at the price of a filter having to spell those characters as '%2F', + // '%5B' and '%5D'. + private static string Escape(string segment) + => segment.Replace("%", "%25").Replace("/", "%2F").Replace("[", "%5B").Replace("]", "%5D"); } } From 0280c3f16194370950d0fb719ba2d3a4d67982b2 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 7 Sep 2026 09:55:53 +0100 Subject: [PATCH 082/110] Add probes for special params, invalid configs, and nesting Added BracketProbe, InvalidConfigProbe, and NestedProbe in BenchmarkDotNet.IntegrationTests.TestingPlatform. BracketProbe tests property filters with special characters. InvalidConfigProbe checks error handling for invalid [Config] attributes. NestedProbe verifies benchmark discovery for nested types. All use BenchmarkDotNet attributes and custom configs where needed. --- .../BracketProbe.cs | 27 +++++++++++++ .../InvalidConfigProbe.cs | 39 +++++++++++++++++++ .../NestedProbe.cs | 26 +++++++++++++ 3 files changed, 92 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BracketProbe.cs create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/InvalidConfigProbe.cs create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/NestedProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BracketProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BracketProbe.cs new file mode 100644 index 0000000000..dac2397d2d --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/BracketProbe.cs @@ -0,0 +1,27 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// A benchmark whose parameter contains the characters a --treenode-filter uses to delimit a property + /// filter. The leaf of every path ends in the job between those same characters, so a filter has to be able to + /// spell them out rather than have them parsed. + /// + [Config(typeof(FastConfig))] + public class BracketProbe + { + [Params("[Dry]")] + public string Value { get; set; } = ""; + + [Benchmark] + public int Length() => Value.Length; + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/InvalidConfigProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/InvalidConfigProbe.cs new file mode 100644 index 0000000000..3bd8daccff --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/InvalidConfigProbe.cs @@ -0,0 +1,39 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// Benchmarks whose [Config] cannot be constructed. Reflection builds every attribute of a type in order to hand + /// any of them back, so reading the attributes of these types throws, and it throws while the list of types is + /// being built - before any benchmark of the assembly has been converted. They have to be dropped rather than + /// take the discovery of every other benchmark down with them. + /// + public static class InvalidConfigProbe + { + /// + /// ConfigAttribute instantiates the type it is given, and an abstract one cannot be instantiated. + /// + [Config(typeof(DebugConfig))] + public class WithAbstractConfig + { + [Benchmark] + public int Identity() => 1; + } + + /// + /// Same read, a different reason: the config has no public parameterless constructor. + /// + [Config(typeof(NoPublicConstructorConfig))] + public class WithInaccessibleConfig + { + [Benchmark] + public int Identity() => 1; + + private class NoPublicConstructorConfig : ManualConfig + { + private NoPublicConstructorConfig() { } + } + } + } +} diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/NestedProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/NestedProbe.cs new file mode 100644 index 0000000000..d1e372bb90 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/NestedProbe.cs @@ -0,0 +1,26 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// A benchmark declared inside another type. ECMA-335 qualifies a nested type by its declaring types rather than + /// by its namespace, which the identity a test runner reads has to follow. + /// + public static class NestedProbe + { + [Config(typeof(FastConfig))] + public class Inner + { + [Benchmark] + public int Identity() => 1; + } + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} From 24ecc1f886a125e0027301013d5dd439d6cfc8ac Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 7 Sep 2026 09:56:15 +0100 Subject: [PATCH 083/110] Remove all contents from LoggerHelper.cs and TestCaseFilter.cs Both LoggerHelper.cs and TestCaseFilter.cs have been cleared of all code, including class definitions, methods, and using directives. The files are now empty. --- .../Utility/LoggerHelper.cs | 56 ------ .../Utility/TestCaseFilter.cs | 164 ------------------ 2 files changed, 220 deletions(-) delete mode 100644 src/BenchmarkDotNet.TestAdapter/Utility/LoggerHelper.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter/Utility/TestCaseFilter.cs diff --git a/src/BenchmarkDotNet.TestAdapter/Utility/LoggerHelper.cs b/src/BenchmarkDotNet.TestAdapter/Utility/LoggerHelper.cs deleted file mode 100644 index ff06c93189..0000000000 --- a/src/BenchmarkDotNet.TestAdapter/Utility/LoggerHelper.cs +++ /dev/null @@ -1,56 +0,0 @@ -using Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging; -using System.Diagnostics; - -namespace BenchmarkDotNet.TestAdapter; - -internal class LoggerHelper -{ - public LoggerHelper(IMessageLogger logger, Stopwatch stopwatch) - { - InnerLogger = logger; - Stopwatch = stopwatch; - } - - public IMessageLogger InnerLogger { get; private set; } - - public Stopwatch Stopwatch { get; private set; } - - public void Log(string format, params object[] args) - { - SendMessage(TestMessageLevel.Informational, null, string.Format(format, args)); - } - - public void LogWithSource(string source, string format, params object[] args) - { - SendMessage(TestMessageLevel.Informational, source, string.Format(format, args)); - } - - public void LogError(string format, params object[] args) - { - SendMessage(TestMessageLevel.Error, null, string.Format(format, args)); - } - - public void LogErrorWithSource(string source, string format, params object[] args) - { - SendMessage(TestMessageLevel.Error, source, string.Format(format, args)); - } - - public void LogWarning(string format, params object[] args) - { - SendMessage(TestMessageLevel.Warning, null, string.Format(format, args)); - } - - public void LogWarningWithSource(string source, string format, params object[] args) - { - SendMessage(TestMessageLevel.Warning, source, string.Format(format, args)); - } - - private void SendMessage(TestMessageLevel level, string? assemblyName, string message) - { - var assemblyText = assemblyName == null - ? "" : - $"{Path.GetFileNameWithoutExtension(assemblyName)}: "; - - InnerLogger.SendMessage(level, $"[BenchmarkDotNet {Stopwatch.Elapsed:hh\\:mm\\:ss\\.ff}] {assemblyText}{message}"); - } -} diff --git a/src/BenchmarkDotNet.TestAdapter/Utility/TestCaseFilter.cs b/src/BenchmarkDotNet.TestAdapter/Utility/TestCaseFilter.cs deleted file mode 100644 index 47f9941e2e..0000000000 --- a/src/BenchmarkDotNet.TestAdapter/Utility/TestCaseFilter.cs +++ /dev/null @@ -1,164 +0,0 @@ -using Microsoft.VisualStudio.TestPlatform.ObjectModel; -using Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter; -using System.Reflection; -using System.Runtime.ExceptionServices; - -namespace BenchmarkDotNet.TestAdapter; - -internal class TestCaseFilter -{ - private const string DisplayNameString = "DisplayName"; - private const string FullyQualifiedNameString = "FullyQualifiedName"; - - private readonly HashSet knownTraits; - private List supportedPropertyNames; - private readonly ITestCaseFilterExpression? filterExpression; - private readonly bool successfullyGotFilter; - private readonly bool isDiscovery; - - public TestCaseFilter(IDiscoveryContext discoveryContext, LoggerHelper logger) - { - // Traits are not known at discovery time because we load them from benchmarks - isDiscovery = true; - knownTraits = []; - supportedPropertyNames = GetSupportedPropertyNames(); - successfullyGotFilter = GetTestCaseFilterExpressionFromDiscoveryContext(discoveryContext, logger, out filterExpression); - } - - public TestCaseFilter(IRunContext runContext, LoggerHelper logger, string assemblyFileName, HashSet knownTraits) - { - this.knownTraits = knownTraits; - supportedPropertyNames = GetSupportedPropertyNames(); - successfullyGotFilter = GetTestCaseFilterExpression(runContext, logger, assemblyFileName, out filterExpression); - } - - public string GetTestCaseFilterValue() - { - return successfullyGotFilter - ? filterExpression?.TestCaseFilterValue ?? "" - : ""; - } - - public bool MatchTestCase(TestCase testCase) - { - if (!successfullyGotFilter) - { - // Had an error while getting filter, match no testcase to ensure discovered test list is empty - return false; - } - else if (filterExpression == null) - { - // No filter specified, keep every testcase - return true; - } - - return filterExpression.MatchTestCase(testCase, p => PropertyProvider(testCase, p)); - } - - public object? PropertyProvider(TestCase testCase, string name) - { - // Traits filtering - if (isDiscovery || knownTraits.Contains(name)) - { - var result = new List(); - - foreach (var trait in GetTraits(testCase)) - if (string.Equals(trait.Key, name, StringComparison.OrdinalIgnoreCase)) - result.Add(trait.Value); - - if (result.Count > 0) - return result.ToArray(); - } - - // Property filtering - switch (name.ToLowerInvariant()) - { - // FullyQualifiedName - case "fullyqualifiedname": - return testCase.FullyQualifiedName; - // DisplayName - case "displayname": - return testCase.DisplayName; - default: - return null; - } - } - - private bool GetTestCaseFilterExpression(IRunContext runContext, LoggerHelper logger, string assemblyFileName, out ITestCaseFilterExpression? filter) - { - filter = null; - - try - { - filter = runContext.GetTestCaseFilter(supportedPropertyNames, null!); - return true; - } - catch (TestPlatformFormatException e) - { - logger.LogWarning("{0}: Exception filtering tests: {1}", Path.GetFileNameWithoutExtension(assemblyFileName), e.Message); - return false; - } - } - - private bool GetTestCaseFilterExpressionFromDiscoveryContext(IDiscoveryContext discoveryContext, LoggerHelper logger, out ITestCaseFilterExpression? filter) - { - filter = null; - - if (discoveryContext is IRunContext runContext) - { - try - { - filter = runContext.GetTestCaseFilter(supportedPropertyNames, null!); - return true; - } - catch (TestPlatformException e) - { - logger.LogWarning("Exception filtering tests: {0}", e.Message); - return false; - } - } - else - { - try - { - // GetTestCaseFilter is present on DiscoveryContext but not in IDiscoveryContext interface - var method = discoveryContext.GetType().GetRuntimeMethod("GetTestCaseFilter", [typeof(IEnumerable), typeof(Func)]); - filter = (ITestCaseFilterExpression)method?.Invoke(discoveryContext, [supportedPropertyNames, null])!; - - return true; - } - catch (TargetInvocationException e) - { - if (e.InnerException is TestPlatformException ex) - { - logger.LogWarning("Exception filtering tests: {0}", ex.Message); - return false; - } - - ExceptionDispatchInfo.Capture(e.InnerException ?? e).Throw(); - return default!;// It's required to suppress error CS0161. - } - } - } - - private List GetSupportedPropertyNames() - { - // Returns the set of well-known property names usually used with the Test Plugins (Used Test Traits + DisplayName + FullyQualifiedName) - if (supportedPropertyNames == null) - { - supportedPropertyNames = knownTraits.ToList(); - supportedPropertyNames.Add(DisplayNameString); - supportedPropertyNames.Add(FullyQualifiedNameString); - } - - return supportedPropertyNames; - } - - private static IEnumerable> GetTraits(TestCase testCase) - { - var traitProperty = TestProperty.Find("TestObject.Traits"); - return traitProperty != null - ? testCase.GetPropertyValue(traitProperty, Array.Empty>()) - : []; - } -} From 424d848761eb7bf27858452c8320c95dc67eccf1 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 7 Sep 2026 09:56:30 +0100 Subject: [PATCH 084/110] Add VSTest adapter integration for BenchmarkDotNet Implemented VSTest adapter to enable discovery and execution of BenchmarkDotNet benchmarks as VSTest test cases. Added VsTestAdapter for test discovery and execution, BenchmarkCaseExtensions for mapping benchmarks to VSTest TestCase objects, and BenchmarkExecutor for running and filtering benchmarks. Introduced LoggerHelper for standardized logging, TestCaseFilter for VSTest-compatible filtering, VsTestEventProcessor for translating BenchmarkDotNet events to VSTest results, and VsTestLogger for bridging logging systems. Defined custom VsTestProperties for benchmark data. All code is under BenchmarkDotNet.TestAdapter.VSTest and integrates with VSTest extensibility points. --- .../VSTest/BenchmarkCaseExtensions.cs | 83 ++++++ .../VSTest/BenchmarkExecutor.cs | 87 +++++++ .../VSTest/Utility/LoggerHelper.cs | 56 ++++ .../VSTest/Utility/TestCaseFilter.cs | 164 ++++++++++++ .../VSTest/VSTestAdapter.cs | 244 ++++++++++++++++++ .../VSTest/VSTestEventProcessor.cs | 194 ++++++++++++++ .../VSTest/VSTestLogger.cs | 62 +++++ .../VSTest/VSTestProperties.cs | 22 ++ 8 files changed, 912 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter/VSTest/BenchmarkCaseExtensions.cs create mode 100644 src/BenchmarkDotNet.TestAdapter/VSTest/BenchmarkExecutor.cs create mode 100644 src/BenchmarkDotNet.TestAdapter/VSTest/Utility/LoggerHelper.cs create mode 100644 src/BenchmarkDotNet.TestAdapter/VSTest/Utility/TestCaseFilter.cs create mode 100644 src/BenchmarkDotNet.TestAdapter/VSTest/VSTestAdapter.cs create mode 100644 src/BenchmarkDotNet.TestAdapter/VSTest/VSTestEventProcessor.cs create mode 100644 src/BenchmarkDotNet.TestAdapter/VSTest/VSTestLogger.cs create mode 100644 src/BenchmarkDotNet.TestAdapter/VSTest/VSTestProperties.cs diff --git a/src/BenchmarkDotNet.TestAdapter/VSTest/BenchmarkCaseExtensions.cs b/src/BenchmarkDotNet.TestAdapter/VSTest/BenchmarkCaseExtensions.cs new file mode 100644 index 0000000000..ab0121ad0c --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/VSTest/BenchmarkCaseExtensions.cs @@ -0,0 +1,83 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Exporters; +using BenchmarkDotNet.Extensions; +using BenchmarkDotNet.Running; +using Microsoft.TestPlatform.AdapterUtilities; +using Microsoft.VisualStudio.TestPlatform.ObjectModel; + +namespace BenchmarkDotNet.TestAdapter.VSTest +{ + /// + /// A set of extensions for BenchmarkCase to support converting to VSTest TestCase objects. + /// + internal static class BenchmarkCaseExtensions + { + /// + /// Converts a BDN BenchmarkCase to a VSTest TestCase. + /// + /// The BenchmarkCase to convert. + /// The dll or exe of the benchmark project. + /// Whether or not the display name should include the job name. + /// The VSTest TestCase. + internal static TestCase ToVsTestCase(this BenchmarkCase benchmarkCase, string assemblyPath, bool includeJobInName = false) + { + var benchmarkMethod = benchmarkCase.Descriptor.WorkloadMethod; + var fullClassName = benchmarkCase.Descriptor.Type.GetCorrectCSharpTypeName(prefixWithGlobal: false); + var parametrizedMethodName = FullNameProvider.GetMethodName(benchmarkCase); + + var displayJobInfo = benchmarkCase.GetUnrandomizedJobDisplayInfo(); + var displayMethodName = parametrizedMethodName + (includeJobInName ? $" [{displayJobInfo}]" : ""); + var displayName = $"{fullClassName}.{displayMethodName}"; + + // We use displayName as FQN to workaround the Rider/R# problem with FQNs processing + // See: https://github.com/dotnet/BenchmarkDotNet/issues/2494 + var fullyQualifiedName = displayName; + + // Use benchmark method FQN on Visual Studio environment to avoid TestExplorer hierarchy split + // when job display name contains '.' which is interpreted as a namespace separator by VS. + // See: https://github.com/dotnet/BenchmarkDotNet/issues/2793 + if (Environment.GetEnvironmentVariable("VSAPPIDNAME") != null) + { + var benchmarkMethodName = benchmarkMethod.Name; + fullyQualifiedName = $"{fullClassName}.{benchmarkMethodName}"; + } + + var vsTestCase = new TestCase(fullyQualifiedName, VsTestAdapter.ExecutorUri, assemblyPath) + { + DisplayName = displayName, + Id = GetTestCaseId(benchmarkCase) + }; + + var benchmarkAttribute = benchmarkMethod.ResolveAttribute(); + if (benchmarkAttribute != null) + { + vsTestCase.CodeFilePath = benchmarkAttribute.SourceCodeFile; + vsTestCase.LineNumber = benchmarkAttribute.SourceCodeLineNumber; + } + + var categories = DefaultCategoryDiscoverer.Instance.GetCategories(benchmarkMethod); + foreach (var category in categories) + vsTestCase.Traits.Add("Category", category); + + vsTestCase.Traits.Add("", "BenchmarkDotNet"); + + return vsTestCase; + } + + /// + /// Gets an ID for a given BenchmarkCase that is uniquely identifiable from discovery to execution phase. + /// + /// The benchmark case. + /// The test case ID. + internal static Guid GetTestCaseId(this BenchmarkCase benchmarkCase) + { + var testIdProvider = new TestIdProvider(); + testIdProvider.AppendString(VsTestAdapter.ExecutorUriString); + testIdProvider.AppendString(benchmarkCase.Descriptor.Type.Namespace ?? string.Empty); + testIdProvider.AppendString(benchmarkCase.Descriptor.DisplayInfo); + testIdProvider.AppendString(benchmarkCase.GetUnrandomizedJobDisplayInfo()); + testIdProvider.AppendString(benchmarkCase.Parameters.DisplayInfo); + return testIdProvider.GetId(); + } + } +} \ No newline at end of file diff --git a/src/BenchmarkDotNet.TestAdapter/VSTest/BenchmarkExecutor.cs b/src/BenchmarkDotNet.TestAdapter/VSTest/BenchmarkExecutor.cs new file mode 100644 index 0000000000..214176ef3a --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/VSTest/BenchmarkExecutor.cs @@ -0,0 +1,87 @@ +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Loggers; +using BenchmarkDotNet.Running; +using BenchmarkDotNet.TestAdapter.VSTest.Remoting; +using Microsoft.VisualStudio.TestPlatform.ObjectModel; + +namespace BenchmarkDotNet.TestAdapter.VSTest +{ + /// + /// A class used for executing benchmarks + /// + internal class BenchmarkExecutor + { + private readonly CancellationTokenSource cts = new(); + + /// + /// Runs all the benchmarks in the given assembly, updating the TestExecutionRecorder as they get run. + /// + /// The dll or exe of the benchmark project. + /// The interface used to record the current test execution progress. + /// + /// An optional list of benchmark IDs specifying which benchmarks to run. + /// These IDs are the same as the ones generated for the VSTest TestCase. + /// + public void RunBenchmarks(string assemblyPath, TestExecutionRecorderWrapper recorder, HashSet? benchmarkIds = null) + { + var benchmarks = BenchmarkEnumerator.GetBenchmarksFromAssemblyPath(assemblyPath); + var testCases = new List(); + + var filteredBenchmarks = new List(); + foreach (var benchmark in benchmarks) + { + var needsJobInfo = benchmark.BenchmarksCases.Select(c => c.Job.DisplayInfo).Distinct().Count() > 1; + var filteredCases = new List(); + foreach (var benchmarkCase in benchmark.BenchmarksCases) + { + var testId = benchmarkCase.GetTestCaseId(); + if (benchmarkIds == null || benchmarkIds.Contains(testId)) + { + filteredCases.Add(benchmarkCase); + testCases.Add(benchmarkCase.ToVsTestCase(assemblyPath, needsJobInfo)); + } + } + + if (filteredCases.Count > 0) + { + filteredBenchmarks.Add(new BenchmarkRunInfo(filteredCases.ToArray(), benchmark.Type, benchmark.Config, benchmark.CompositeInProcessDiagnoser)); + } + } + + benchmarks = filteredBenchmarks.ToArray(); + + if (benchmarks.Length == 0) + return; + + // Create an event processor which will subscribe to events and push them to VSTest + var eventProcessor = new VsTestEventProcessor(testCases, recorder, cts.Token); + + // Create a logger which will forward all log messages in BDN to the VSTest logger. + var logger = new VsTestLogger(recorder.GetLogger()); + + // Modify all the benchmarks so that the event process and logger is added. + benchmarks = benchmarks + .Select(b => new BenchmarkRunInfo( + b.BenchmarksCases, + b.Type, + b.Config.AddEventProcessor(eventProcessor) + .AddLogger(logger) + .RemoveLoggersOfType() // Console logs are also outputted by VSTestLogger. + .CreateImmutableConfig(), + b.CompositeInProcessDiagnoser)) + .ToArray(); + + // Run all the benchmarks, and ensure that any tests that don't have a result yet are sent. + BenchmarkRunner.Run(benchmarks); + eventProcessor.SendUnsentTestResults(); + } + + /// + /// Stop the benchmarks when next able. + /// + public void Cancel() + { + cts.Cancel(); + } + } +} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTest/Utility/LoggerHelper.cs b/src/BenchmarkDotNet.TestAdapter/VSTest/Utility/LoggerHelper.cs new file mode 100644 index 0000000000..594beba206 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/VSTest/Utility/LoggerHelper.cs @@ -0,0 +1,56 @@ +using Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging; +using System.Diagnostics; + +namespace BenchmarkDotNet.TestAdapter.VSTest; + +internal class LoggerHelper +{ + public LoggerHelper(IMessageLogger logger, Stopwatch stopwatch) + { + InnerLogger = logger; + Stopwatch = stopwatch; + } + + public IMessageLogger InnerLogger { get; private set; } + + public Stopwatch Stopwatch { get; private set; } + + public void Log(string format, params object[] args) + { + SendMessage(TestMessageLevel.Informational, null, string.Format(format, args)); + } + + public void LogWithSource(string source, string format, params object[] args) + { + SendMessage(TestMessageLevel.Informational, source, string.Format(format, args)); + } + + public void LogError(string format, params object[] args) + { + SendMessage(TestMessageLevel.Error, null, string.Format(format, args)); + } + + public void LogErrorWithSource(string source, string format, params object[] args) + { + SendMessage(TestMessageLevel.Error, source, string.Format(format, args)); + } + + public void LogWarning(string format, params object[] args) + { + SendMessage(TestMessageLevel.Warning, null, string.Format(format, args)); + } + + public void LogWarningWithSource(string source, string format, params object[] args) + { + SendMessage(TestMessageLevel.Warning, source, string.Format(format, args)); + } + + private void SendMessage(TestMessageLevel level, string? assemblyName, string message) + { + var assemblyText = assemblyName == null + ? "" : + $"{Path.GetFileNameWithoutExtension(assemblyName)}: "; + + InnerLogger.SendMessage(level, $"[BenchmarkDotNet {Stopwatch.Elapsed:hh\\:mm\\:ss\\.ff}] {assemblyText}{message}"); + } +} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTest/Utility/TestCaseFilter.cs b/src/BenchmarkDotNet.TestAdapter/VSTest/Utility/TestCaseFilter.cs new file mode 100644 index 0000000000..457b07e98c --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/VSTest/Utility/TestCaseFilter.cs @@ -0,0 +1,164 @@ +using Microsoft.VisualStudio.TestPlatform.ObjectModel; +using Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter; +using System.Reflection; +using System.Runtime.ExceptionServices; + +namespace BenchmarkDotNet.TestAdapter.VSTest; + +internal class TestCaseFilter +{ + private const string DisplayNameString = "DisplayName"; + private const string FullyQualifiedNameString = "FullyQualifiedName"; + + private readonly HashSet knownTraits; + private List supportedPropertyNames; + private readonly ITestCaseFilterExpression? filterExpression; + private readonly bool successfullyGotFilter; + private readonly bool isDiscovery; + + public TestCaseFilter(IDiscoveryContext discoveryContext, LoggerHelper logger) + { + // Traits are not known at discovery time because we load them from benchmarks + isDiscovery = true; + knownTraits = []; + supportedPropertyNames = GetSupportedPropertyNames(); + successfullyGotFilter = GetTestCaseFilterExpressionFromDiscoveryContext(discoveryContext, logger, out filterExpression); + } + + public TestCaseFilter(IRunContext runContext, LoggerHelper logger, string assemblyFileName, HashSet knownTraits) + { + this.knownTraits = knownTraits; + supportedPropertyNames = GetSupportedPropertyNames(); + successfullyGotFilter = GetTestCaseFilterExpression(runContext, logger, assemblyFileName, out filterExpression); + } + + public string GetTestCaseFilterValue() + { + return successfullyGotFilter + ? filterExpression?.TestCaseFilterValue ?? "" + : ""; + } + + public bool MatchTestCase(TestCase testCase) + { + if (!successfullyGotFilter) + { + // Had an error while getting filter, match no testcase to ensure discovered test list is empty + return false; + } + else if (filterExpression == null) + { + // No filter specified, keep every testcase + return true; + } + + return filterExpression.MatchTestCase(testCase, p => PropertyProvider(testCase, p)); + } + + public object? PropertyProvider(TestCase testCase, string name) + { + // Traits filtering + if (isDiscovery || knownTraits.Contains(name)) + { + var result = new List(); + + foreach (var trait in GetTraits(testCase)) + if (string.Equals(trait.Key, name, StringComparison.OrdinalIgnoreCase)) + result.Add(trait.Value); + + if (result.Count > 0) + return result.ToArray(); + } + + // Property filtering + switch (name.ToLowerInvariant()) + { + // FullyQualifiedName + case "fullyqualifiedname": + return testCase.FullyQualifiedName; + // DisplayName + case "displayname": + return testCase.DisplayName; + default: + return null; + } + } + + private bool GetTestCaseFilterExpression(IRunContext runContext, LoggerHelper logger, string assemblyFileName, out ITestCaseFilterExpression? filter) + { + filter = null; + + try + { + filter = runContext.GetTestCaseFilter(supportedPropertyNames, null!); + return true; + } + catch (TestPlatformFormatException e) + { + logger.LogWarning("{0}: Exception filtering tests: {1}", Path.GetFileNameWithoutExtension(assemblyFileName), e.Message); + return false; + } + } + + private bool GetTestCaseFilterExpressionFromDiscoveryContext(IDiscoveryContext discoveryContext, LoggerHelper logger, out ITestCaseFilterExpression? filter) + { + filter = null; + + if (discoveryContext is IRunContext runContext) + { + try + { + filter = runContext.GetTestCaseFilter(supportedPropertyNames, null!); + return true; + } + catch (TestPlatformException e) + { + logger.LogWarning("Exception filtering tests: {0}", e.Message); + return false; + } + } + else + { + try + { + // GetTestCaseFilter is present on DiscoveryContext but not in IDiscoveryContext interface + var method = discoveryContext.GetType().GetRuntimeMethod("GetTestCaseFilter", [typeof(IEnumerable), typeof(Func)]); + filter = (ITestCaseFilterExpression)method?.Invoke(discoveryContext, [supportedPropertyNames, null])!; + + return true; + } + catch (TargetInvocationException e) + { + if (e.InnerException is TestPlatformException ex) + { + logger.LogWarning("Exception filtering tests: {0}", ex.Message); + return false; + } + + ExceptionDispatchInfo.Capture(e.InnerException ?? e).Throw(); + return default!;// It's required to suppress error CS0161. + } + } + } + + private List GetSupportedPropertyNames() + { + // Returns the set of well-known property names usually used with the Test Plugins (Used Test Traits + DisplayName + FullyQualifiedName) + if (supportedPropertyNames == null) + { + supportedPropertyNames = knownTraits.ToList(); + supportedPropertyNames.Add(DisplayNameString); + supportedPropertyNames.Add(FullyQualifiedNameString); + } + + return supportedPropertyNames; + } + + private static IEnumerable> GetTraits(TestCase testCase) + { + var traitProperty = TestProperty.Find("TestObject.Traits"); + return traitProperty != null + ? testCase.GetPropertyValue(traitProperty, Array.Empty>()) + : []; + } +} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTest/VSTestAdapter.cs b/src/BenchmarkDotNet.TestAdapter/VSTest/VSTestAdapter.cs new file mode 100644 index 0000000000..6d4f5bb13b --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/VSTest/VSTestAdapter.cs @@ -0,0 +1,244 @@ +using BenchmarkDotNet.TestAdapter.VSTest.Remoting; +using Microsoft.VisualStudio.TestPlatform.ObjectModel; +using Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter; +using Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging; +using System.Diagnostics; +using System.Reflection; + +namespace BenchmarkDotNet.TestAdapter.VSTest +{ + /// + /// Discovers and executes benchmarks using the VSTest protocol. + /// + [ExtensionUri(ExecutorUriString)] + [DefaultExecutorUri(ExecutorUriString)] + [FileExtension(".dll")] + [FileExtension(".exe")] + public class VsTestAdapter : ITestExecutor, ITestDiscoverer + { + // This URI is used to identify the adapter. + internal const string ExecutorUriString = "executor://BenchmarkDotNet.TestAdapter"; + internal static readonly Uri ExecutorUri = new Uri(ExecutorUriString); + + /// + /// Cancellation token used to stop any benchmarks that are currently running. + /// + private CancellationTokenSource? cts = null; + + /// + /// Discovers the benchmarks. + /// + /// List of assemblies to search for benchmarks in. + /// A context that the discovery is performed in. + /// Logger that sends messages back to VSTest host. + /// Interface that provides methods for sending discovered benchmarks back to the host. + public void DiscoverTests( + IEnumerable sources, + IDiscoveryContext discoveryContext, + IMessageLogger logger, + ITestCaseDiscoverySink discoverySink) + { + var stopwatch = Stopwatch.StartNew(); + var loggerHelper = new LoggerHelper(logger, stopwatch); + var testCaseFilter = new TestCaseFilter(discoveryContext, loggerHelper); + + foreach (var source in sources) + { + ValidateSourceIsAssemblyOrThrow(source); + foreach (var testCase in GetVsTestCasesFromAssembly(source, logger)) + { + if (!testCaseFilter.MatchTestCase(testCase)) + continue; + + discoverySink.SendTestCase(testCase); + } + } + } + + /// + /// Runs a given set of test cases that represent benchmarks. + /// + /// The tests to run. + /// A context that the run is performed in. + /// Interface used for communicating with the VSTest host. + public void RunTests(IEnumerable? tests, IRunContext? runContext, IFrameworkHandle? frameworkHandle) + { + if (tests == null) + throw new ArgumentNullException(nameof(tests)); + if (frameworkHandle == null) + throw new ArgumentNullException(nameof(frameworkHandle)); + + cts ??= new CancellationTokenSource(); + + var stopwatch = Stopwatch.StartNew(); + var logger = new LoggerHelper(frameworkHandle, stopwatch); + + foreach (var testsPerAssembly in tests.GroupBy(t => t.Source)) + { + RunBenchmarks(testsPerAssembly.Key, frameworkHandle, testsPerAssembly); + } + + cts = null; + } + + /// + /// Runs all/filtered benchmarks in the given set of sources (assemblies). + /// + /// The assemblies to run. + /// A context that the run is performed in. + /// Interface used for communicating with the VSTest host. + public void RunTests(IEnumerable? sources, IRunContext? runContext, IFrameworkHandle? frameworkHandle) + { + if (sources == null) + throw new ArgumentNullException(nameof(sources)); + if (frameworkHandle == null) + throw new ArgumentNullException(nameof(frameworkHandle)); + + cts ??= new CancellationTokenSource(); + + var stopwatch = Stopwatch.StartNew(); + var logger = new LoggerHelper(frameworkHandle, stopwatch); + + foreach (var source in sources) + { + var filter = new TestCaseFilter(runContext!, logger, source, ["Category"]); + if (filter.GetTestCaseFilterValue() != "") + { + var discoveredBenchmarks = GetVsTestCasesFromAssembly(source, frameworkHandle); + var filteredTestCases = discoveredBenchmarks.Where(x => filter.MatchTestCase(x)) + .ToArray(); + + if (filteredTestCases.Length == 0) + continue; + + // Run filtered tests. + RunBenchmarks(source, frameworkHandle, filteredTestCases); + } + else + { + // Run all benchmarks + RunBenchmarks(source, frameworkHandle); + } + } + + + cts = null; + } + + /// + /// Stops any currently running benchmarks. + /// + public void Cancel() + { + cts?.Cancel(); + } + + /// + /// Gets the VSTest test cases in the given assembly. + /// + /// The dll or exe of the benchmark project. + /// A logger that sends logs to VSTest. + /// The VSTest test cases inside the given assembly. + private static List GetVsTestCasesFromAssembly(string assemblyPath, IMessageLogger logger) + { + try + { + // Ensure that the test enumeration is done inside the context of the source directory. + var enumerator = (BenchmarkEnumeratorWrapper)CreateIsolatedType(typeof(BenchmarkEnumeratorWrapper), assemblyPath); + var testCases = enumerator + .GetTestCasesFromAssemblyPathSerialized(assemblyPath) + .Select(SerializationHelpers.Deserialize) + .ToList(); + + // Validate that all test ids are unique + var idLookup = new Dictionary(); + foreach (var testCase in testCases) + { + if (idLookup.TryGetValue(testCase.Id, out var matchingCase)) + throw new Exception($"Encountered Duplicate Test ID: '{testCase.DisplayName}' and '{matchingCase}'"); + + idLookup[testCase.Id] = testCase.DisplayName; + } + + return testCases; + } + catch (Exception ex) + { + logger.SendMessage(TestMessageLevel.Error, $"Failed to load benchmarks from assembly\n{ex}"); + throw; + } + } + + /// + /// Runs the benchmarks in the given source. + /// + /// The dll or exe of the benchmark project. + /// An interface used to communicate with the VSTest host. + /// + /// The specific test cases to be run if specified. + /// If unspecified, runs all the test cases in the source. + /// + private void RunBenchmarks(string source, IFrameworkHandle frameworkHandle, IEnumerable? testCases = null) + { + ValidateSourceIsAssemblyOrThrow(source); + + // Create a HashSet of all the TestCase IDs to be run if specified. + var caseIds = testCases == null ? null : new HashSet(testCases.Select(c => c.Id)); + + try + { + // Ensure that test execution is done inside the context of the source directory. + var executor = (BenchmarkExecutorWrapper)CreateIsolatedType(typeof(BenchmarkExecutorWrapper), source); + cts?.Token.Register(executor.Cancel); + + executor.RunBenchmarks(source, new TestExecutionRecorderWrapper(frameworkHandle), caseIds); + } + catch (Exception ex) + { + frameworkHandle.SendMessage(TestMessageLevel.Error, $"Failed to run benchmarks in assembly\n{ex}"); + throw; + } + } + + /// + /// This will create the given type in a child AppDomain when used in .NET Framework. + /// If not in the .NET Framework, it will use the current AppDomain. + /// + /// The type to create. + /// The dll or exe of the benchmark project. + /// The created object. + private static object CreateIsolatedType(Type type, string assemblyPath) + { + // .NET Framework runs require a custom AppDomain to be set up to run the benchmarks in because otherwise, + // all the assemblies will be loaded from the VSTest console rather than from the directory that the BDN + // program under test lives in. .NET Core assembly resolution is smarter and will correctly load the right + // assembly versions as needed and does not require a custom AppDomain. Unfortunately, the APIs needed to + // create the AppDomain for .NET Framework are not part of .NET Standard, and so a multi-targeting solution + // such as this is required to get this to work. This same approach is also used by other .NET unit testing + // libraries as well, further justifying this approach to solving how to get the correct assemblies loaded. +#if NETFRAMEWORK + var appBase = Path.GetDirectoryName(assemblyPath); + var setup = new AppDomainSetup { ApplicationBase = appBase }; + var domainName = $"Isolated Domain for {type.Name}"; + var appDomain = AppDomain.CreateDomain(domainName, null, setup); + return appDomain.CreateInstanceAndUnwrap( + type.Assembly.FullName, type.FullName, false, BindingFlags.Default, null, null, null, null); +#else + return Activator.CreateInstance(type); +#endif + } + + private static void ValidateSourceIsAssemblyOrThrow(string source) + { + if (string.IsNullOrEmpty(source)) + throw new ArgumentException($"'{nameof(source)}' cannot be null or whitespace.", nameof(source)); + + if (!Path.HasExtension(source)) + throw new NotSupportedException($"Missing extension on source '{source}', must have the extension '.dll' or '.exe'."); + + var extension = Path.GetExtension(source); + if (!string.Equals(extension, ".dll", StringComparison.OrdinalIgnoreCase) && !string.Equals(extension, ".exe", StringComparison.OrdinalIgnoreCase)) + throw new NotSupportedException($"Unsupported extension on source '{source}', must have the extension '.dll' or '.exe'."); + } + } +} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTest/VSTestEventProcessor.cs b/src/BenchmarkDotNet.TestAdapter/VSTest/VSTestEventProcessor.cs new file mode 100644 index 0000000000..14570c748e --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/VSTest/VSTestEventProcessor.cs @@ -0,0 +1,194 @@ +using BenchmarkDotNet.EventProcessors; +using BenchmarkDotNet.Extensions; +using BenchmarkDotNet.Reports; +using BenchmarkDotNet.Running; +using BenchmarkDotNet.TestAdapter.VSTest.Remoting; +using BenchmarkDotNet.Toolchains.Results; +using BenchmarkDotNet.Validators; +using Microsoft.VisualStudio.TestPlatform.ObjectModel; +using Perfolizer.Mathematics.Histograms; +using System.Diagnostics; +using System.Globalization; +using System.Text; + +namespace BenchmarkDotNet.TestAdapter.VSTest +{ + /// + /// An event processor which will pass on benchmark execution information to VSTest. + /// + internal class VsTestEventProcessor : EventProcessor + { + private readonly Dictionary cases; + private readonly TestExecutionRecorderWrapper recorder; + private readonly CancellationToken cancellationToken; + private readonly Stopwatch runTimerStopwatch = new(); + private readonly Dictionary testResults = []; + private readonly HashSet sentTestResults = []; + + public VsTestEventProcessor( + List cases, + TestExecutionRecorderWrapper recorder, + CancellationToken cancellationToken) + { + this.cases = cases.ToDictionary(c => c.Id); + this.recorder = recorder; + this.cancellationToken = cancellationToken; + } + + public override void OnValidationError(ValidationError validationError) + { + // If the error is not linked to a benchmark case, then set the error on all benchmarks + var errorCases = validationError.BenchmarkCase == null + ? cases.Values.ToList() + : [cases[validationError.BenchmarkCase.GetTestCaseId()]]; + foreach (var testCase in errorCases) + { + var testResult = GetOrCreateTestResult(testCase); + + if (validationError.IsCritical) + { + // Fail if there is a critical validation error + testResult.Outcome = TestOutcome.Failed; + + // Append validation error message to end of test case error message + testResult.ErrorMessage = testResult.ErrorMessage == null + ? validationError.Message + : $"{testResult.ErrorMessage}\n{validationError.Message}"; + + // The test result is not sent yet, in case there are multiple validation errors that need to be sent. + } + else + { + // If the validation error is not critical, append it as a message + testResult.Messages.Add(new TestResultMessage(TestResultMessage.StandardOutCategory, $"WARNING: {validationError.Message}\n")); + } + } + } + + public override void OnBuildComplete(BuildPartition buildPartition, BuildResult buildResult) + { + // Only need to handle build failures + if (!buildResult.IsBuildSuccess) + { + foreach (var benchmarkBuildInfo in buildPartition.Benchmarks) + { + var testCase = cases[benchmarkBuildInfo.BenchmarkCase.GetTestCaseId()]; + var testResult = GetOrCreateTestResult(testCase); + + if (buildResult.GenerateException != null) + testResult.ErrorMessage = $"// Generate Exception: {buildResult.GenerateException.Message}"; + else if (!buildResult.IsBuildSuccess && buildResult.TryToExplainFailureReason(buildPartition.GetInProcessDiagnoserHandlerTypes(), out string? reason)) + testResult.ErrorMessage = $"// Build Error: {reason}"; + else if (buildResult.ErrorMessage != null) + testResult.ErrorMessage = $"// Build Error: {buildResult.ErrorMessage}"; + testResult.Outcome = TestOutcome.Failed; + + // Send the result immediately + RecordStart(testCase); + RecordEnd(testCase, testResult.Outcome); + RecordResult(testResult); + sentTestResults.Add(testCase.Id); + } + } + } + + public override void OnStartRunBenchmark(BenchmarkCase benchmarkCase) + { + // TODO: add proper cancellation support to BDN so that we don't need to do cancellation through the event processor + cancellationToken.ThrowIfCancellationRequested(); + + var testCase = cases[benchmarkCase.GetTestCaseId()]; + var testResult = GetOrCreateTestResult(testCase); + testResult.StartTime = DateTimeOffset.UtcNow; + + RecordStart(testCase); + runTimerStopwatch.Restart(); + } + + public override void OnEndRunBenchmark(BenchmarkCase benchmarkCase, BenchmarkReport report) + { + var testCase = cases[benchmarkCase.GetTestCaseId()]; + var testResult = GetOrCreateTestResult(testCase); + testResult.EndTime = DateTimeOffset.UtcNow; + testResult.Duration = runTimerStopwatch.Elapsed; + testResult.Outcome = report.Success ? TestOutcome.Passed : TestOutcome.Failed; + + var resultRuns = report.GetResultRuns(); + + // Provide the raw result runs data. + testResult.SetPropertyValue(VsTestProperties.Measurement, resultRuns.Select(m => m.Nanoseconds.ToString()).ToArray()); + + // Add a message to the TestResult which contains the results summary. + testResult.Messages.Add(new TestResultMessage(TestResultMessage.StandardOutCategory, report.BenchmarkCase.DisplayInfo + "\n")); + testResult.Messages.Add(new TestResultMessage(TestResultMessage.StandardOutCategory, $"Runtime = {report.GetRuntimeInfo()}; GC = {report.GetGcInfo()}\n")); + + var statistics = resultRuns.GetStatistics(); + var cultureInfo = CultureInfo.InvariantCulture; + var formatter = statistics.CreateNanosecondFormatter(cultureInfo); + + var builder = new StringBuilder(); + var histogram = HistogramBuilder.Adaptive.Build(statistics.Sample.Values); + builder.AppendLine("-------------------- Histogram --------------------"); + builder.AppendLine(histogram.ToString(formatter)); + builder.AppendLine("---------------------------------------------------"); + + var statisticsOutput = statistics.ToString(cultureInfo, formatter, calcHistogram: false); + builder.AppendLine(statisticsOutput); + + testResult.Messages.Add(new TestResultMessage(TestResultMessage.StandardOutCategory, builder.ToString())); + + RecordEnd(testResult.TestCase, testResult.Outcome); + RecordResult(testResult); + sentTestResults.Add(testCase.Id); + } + + /// + /// Iterate through all the benchmarks that were scheduled to run, and if they haven't been sent yet, send the result through. + /// + public void SendUnsentTestResults() + { + foreach (var testCase in cases.Values) + { + if (!sentTestResults.Contains(testCase.Id)) + { + var testResult = GetOrCreateTestResult(testCase); + if (testResult.Outcome == TestOutcome.None) + testResult.Outcome = TestOutcome.Skipped; + RecordStart(testCase); + RecordEnd(testCase, testResult.Outcome); + RecordResult(testResult); + } + } + } + + private TestResult GetOrCreateTestResult(TestCase testCase) + { + if (testResults.TryGetValue(testCase.Id, out var testResult)) + return testResult; + + var newResult = new TestResult(testCase) + { + ComputerName = Environment.MachineName, + DisplayName = testCase.DisplayName + }; + + testResults[testCase.Id] = newResult; + return newResult; + } + + private void RecordStart(TestCase testCase) + { + recorder.RecordStart(SerializationHelpers.Serialize(testCase)); + } + + private void RecordEnd(TestCase testCase, TestOutcome testOutcome) + { + recorder.RecordEnd(SerializationHelpers.Serialize(testCase), testOutcome); + } + + private void RecordResult(TestResult testResult) + { + recorder.RecordResult(SerializationHelpers.Serialize(testResult)); + } + } +} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTest/VSTestLogger.cs b/src/BenchmarkDotNet.TestAdapter/VSTest/VSTestLogger.cs new file mode 100644 index 0000000000..78b17a9f40 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/VSTest/VSTestLogger.cs @@ -0,0 +1,62 @@ +using BenchmarkDotNet.Loggers; +using Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging; +using System.Text; + +namespace BenchmarkDotNet.TestAdapter.VSTest +{ + /// + /// A class to send logs from BDN to the VSTest output log. + /// + internal sealed class VsTestLogger : ILogger + { + private readonly IMessageLogger messageLogger; + private readonly StringBuilder currentLine = new StringBuilder(); + private TestMessageLevel currentLevel = TestMessageLevel.Informational; + + public VsTestLogger(IMessageLogger logger) + { + messageLogger = logger; + } + + public string Id => nameof(VsTestLogger); + + public int Priority => 0; + + public void Flush() + { + WriteLine(); + } + + public void Write(LogKind logKind, string text) + { + currentLine.Append(text); + + // Assume that if the log kind is an error, that the whole line is treated as an error + // The level will be reset to Informational when WriteLine() is called. + currentLevel = logKind switch + { + LogKind.Error => TestMessageLevel.Error, + LogKind.Warning => TestMessageLevel.Warning, + _ => currentLevel + }; + } + + public void WriteLine() + { + // The VSTest logger throws an error on logging empty or whitespace strings, so skip them. + if (currentLine.Length == 0) + return; + + messageLogger.SendMessage(currentLevel, currentLine.ToString()); + + currentLevel = TestMessageLevel.Informational; + currentLine.Clear(); + } + + public void WriteLine(LogKind logKind, string text) + { + Write(logKind, text); + WriteLine(); + } + } +} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTest/VSTestProperties.cs b/src/BenchmarkDotNet.TestAdapter/VSTest/VSTestProperties.cs new file mode 100644 index 0000000000..ae07b803f9 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/VSTest/VSTestProperties.cs @@ -0,0 +1,22 @@ +using Microsoft.VisualStudio.TestPlatform.ObjectModel; + +namespace BenchmarkDotNet.TestAdapter.VSTest +{ + /// + /// A class that contains all the custom properties that can be set on VSTest TestCase and TestResults. + /// Some of these properties are well known as they are also used by VSTest adapters for other test libraries. + /// + internal static class VsTestProperties + { + /// + /// A test property used for storing the test results so that they could be accessed + /// programmatically from a custom VSTest runner. + /// + internal static readonly TestProperty Measurement = TestProperty.Register( + "BenchmarkDotNet.TestAdapter.Measurements", + "Measurements", + typeof(string[]), + TestPropertyAttributes.Hidden, + typeof(TestResult)); + } +} From ee4516b51891f010e43bd9717fca1d46f4601fb9 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 7 Sep 2026 09:57:17 +0100 Subject: [PATCH 085/110] Refactor adapter: improve disposal, remove legacy files Refactored BenchmarkDotNet.TestAdapter for better resource management by introducing ParameterValueDisposer to handle IDisposable benchmark parameters. Updated BenchmarkEnumerator to use the new disposer. Removed obsolete files (BenchmarkCaseExtensions.cs, BenchmarkExecutor.cs, VSTestAdapter.cs, VSTestEventProcessor.cs, VSTestLogger.cs, VSTestProperties.cs) to simplify and clean up the codebase. --- .../BenchmarkCaseExtensions.cs | 83 ------ .../BenchmarkEnumerator.cs | 38 +-- .../BenchmarkExecutor.cs | 87 ------- .../ParameterValueDisposer.cs | 59 +++++ .../VSTestAdapter.cs | 244 ------------------ .../VSTestEventProcessor.cs | 194 -------------- .../VSTestLogger.cs | 62 ----- .../VSTestProperties.cs | 22 -- 8 files changed, 80 insertions(+), 709 deletions(-) delete mode 100644 src/BenchmarkDotNet.TestAdapter/BenchmarkCaseExtensions.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter/BenchmarkExecutor.cs create mode 100644 src/BenchmarkDotNet.TestAdapter/ParameterValueDisposer.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter/VSTestAdapter.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter/VSTestEventProcessor.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter/VSTestLogger.cs delete mode 100644 src/BenchmarkDotNet.TestAdapter/VSTestProperties.cs diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseExtensions.cs b/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseExtensions.cs deleted file mode 100644 index 08ad640e26..0000000000 --- a/src/BenchmarkDotNet.TestAdapter/BenchmarkCaseExtensions.cs +++ /dev/null @@ -1,83 +0,0 @@ -using BenchmarkDotNet.Attributes; -using BenchmarkDotNet.Exporters; -using BenchmarkDotNet.Extensions; -using BenchmarkDotNet.Running; -using Microsoft.TestPlatform.AdapterUtilities; -using Microsoft.VisualStudio.TestPlatform.ObjectModel; - -namespace BenchmarkDotNet.TestAdapter -{ - /// - /// A set of extensions for BenchmarkCase to support converting to VSTest TestCase objects. - /// - internal static class BenchmarkCaseExtensions - { - /// - /// Converts a BDN BenchmarkCase to a VSTest TestCase. - /// - /// The BenchmarkCase to convert. - /// The dll or exe of the benchmark project. - /// Whether or not the display name should include the job name. - /// The VSTest TestCase. - internal static TestCase ToVsTestCase(this BenchmarkCase benchmarkCase, string assemblyPath, bool includeJobInName = false) - { - var benchmarkMethod = benchmarkCase.Descriptor.WorkloadMethod; - var fullClassName = benchmarkCase.Descriptor.Type.GetCorrectCSharpTypeName(prefixWithGlobal: false); - var parametrizedMethodName = FullNameProvider.GetMethodName(benchmarkCase); - - var displayJobInfo = benchmarkCase.GetUnrandomizedJobDisplayInfo(); - var displayMethodName = parametrizedMethodName + (includeJobInName ? $" [{displayJobInfo}]" : ""); - var displayName = $"{fullClassName}.{displayMethodName}"; - - // We use displayName as FQN to workaround the Rider/R# problem with FQNs processing - // See: https://github.com/dotnet/BenchmarkDotNet/issues/2494 - var fullyQualifiedName = displayName; - - // Use benchmark method FQN on Visual Studio environment to avoid TestExplorer hierarchy split - // when job display name contains '.' which is interpreted as a namespace separator by VS. - // See: https://github.com/dotnet/BenchmarkDotNet/issues/2793 - if (Environment.GetEnvironmentVariable("VSAPPIDNAME") != null) - { - var benchmarkMethodName = benchmarkMethod.Name; - fullyQualifiedName = $"{fullClassName}.{benchmarkMethodName}"; - } - - var vsTestCase = new TestCase(fullyQualifiedName, VsTestAdapter.ExecutorUri, assemblyPath) - { - DisplayName = displayName, - Id = GetTestCaseId(benchmarkCase) - }; - - var benchmarkAttribute = benchmarkMethod.ResolveAttribute(); - if (benchmarkAttribute != null) - { - vsTestCase.CodeFilePath = benchmarkAttribute.SourceCodeFile; - vsTestCase.LineNumber = benchmarkAttribute.SourceCodeLineNumber; - } - - var categories = DefaultCategoryDiscoverer.Instance.GetCategories(benchmarkMethod); - foreach (var category in categories) - vsTestCase.Traits.Add("Category", category); - - vsTestCase.Traits.Add("", "BenchmarkDotNet"); - - return vsTestCase; - } - - /// - /// Gets an ID for a given BenchmarkCase that is uniquely identifiable from discovery to execution phase. - /// - /// The benchmark case. - /// The test case ID. - internal static Guid GetTestCaseId(this BenchmarkCase benchmarkCase) - { - var testIdProvider = new TestIdProvider(); - testIdProvider.AppendString(VsTestAdapter.ExecutorUriString); - testIdProvider.AppendString(benchmarkCase.Descriptor.Type.Namespace ?? string.Empty); - testIdProvider.AppendString(benchmarkCase.Descriptor.DisplayInfo); - testIdProvider.AppendString(benchmarkCase.GetUnrandomizedJobDisplayInfo()); - testIdProvider.AppendString(benchmarkCase.Parameters.DisplayInfo); - return testIdProvider.GetId(); - } - } -} \ No newline at end of file diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs b/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs index f001ef45f8..2d492f0769 100644 --- a/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs +++ b/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs @@ -58,27 +58,31 @@ public static BenchmarkRunInfo[] GetBenchmarksFromAssemblyPath(string assemblyPa /// The benchmarks inside the assembly. public static BenchmarkRunInfo[] GetBenchmarksFromAssembly(Assembly assembly) { - var isDebugAssembly = assembly.IsJitOptimizationDisabled() ?? false; + var all = GenericBenchmarksBuilder.GetRunnableBenchmarks(assembly.GetRunnableBenchmarks()) + .Select(type => BenchmarkConverter.TypeToBenchmarks(type)) + .ToArray(); - return GenericBenchmarksBuilder.GetRunnableBenchmarks(assembly.GetRunnableBenchmarks()) - .Select(type => - { - var benchmarkRunInfo = BenchmarkConverter.TypeToBenchmarks(type); - if (isDebugAssembly) - { - // If the assembly is a debug assembly, then only display them if they will run in-process - // This will allow people to debug their benchmarks from a test runner if they wish. - benchmarkRunInfo = new BenchmarkRunInfo( - benchmarkRunInfo.BenchmarksCases.Where(c => c.GetToolchain().IsInProcess).ToArray(), - benchmarkRunInfo.Type, - benchmarkRunInfo.Config, - benchmarkRunInfo.CompositeInProcessDiagnoser); - } + if (!(assembly.IsJitOptimizationDisabled() ?? false)) + return all.Where(runInfo => runInfo.BenchmarksCases.Length > 0).ToArray(); - return benchmarkRunInfo; - }) + // If the assembly is a debug assembly, then only display the benchmarks that will run in-process. This + // will allow people to debug their benchmarks from a test runner if they wish. + var runnable = all + .Select(runInfo => new BenchmarkRunInfo( + runInfo.BenchmarksCases.Where(c => c.GetToolchain().IsInProcess).ToArray(), + runInfo.Type, + runInfo.Config, + runInfo.CompositeInProcessDiagnoser)) .Where(runInfo => runInfo.BenchmarksCases.Length > 0) .ToArray(); + + // BenchmarkConverter has already constructed every parameter value by now, and a case hidden here is never + // handed to BenchmarkDotNet by either adapter, so nothing downstream can dispose what is dropped. + ParameterValueDisposer.DisposeUnused( + all.SelectMany(runInfo => runInfo.BenchmarksCases), + runnable.SelectMany(runInfo => runInfo.BenchmarksCases)); + + return runnable; } } } diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkExecutor.cs b/src/BenchmarkDotNet.TestAdapter/BenchmarkExecutor.cs deleted file mode 100644 index 8fea8f0c9d..0000000000 --- a/src/BenchmarkDotNet.TestAdapter/BenchmarkExecutor.cs +++ /dev/null @@ -1,87 +0,0 @@ -using BenchmarkDotNet.Configs; -using BenchmarkDotNet.Loggers; -using BenchmarkDotNet.Running; -using BenchmarkDotNet.TestAdapter.Remoting; -using Microsoft.VisualStudio.TestPlatform.ObjectModel; - -namespace BenchmarkDotNet.TestAdapter -{ - /// - /// A class used for executing benchmarks - /// - internal class BenchmarkExecutor - { - private readonly CancellationTokenSource cts = new(); - - /// - /// Runs all the benchmarks in the given assembly, updating the TestExecutionRecorder as they get run. - /// - /// The dll or exe of the benchmark project. - /// The interface used to record the current test execution progress. - /// - /// An optional list of benchmark IDs specifying which benchmarks to run. - /// These IDs are the same as the ones generated for the VSTest TestCase. - /// - public void RunBenchmarks(string assemblyPath, TestExecutionRecorderWrapper recorder, HashSet? benchmarkIds = null) - { - var benchmarks = BenchmarkEnumerator.GetBenchmarksFromAssemblyPath(assemblyPath); - var testCases = new List(); - - var filteredBenchmarks = new List(); - foreach (var benchmark in benchmarks) - { - var needsJobInfo = benchmark.BenchmarksCases.Select(c => c.Job.DisplayInfo).Distinct().Count() > 1; - var filteredCases = new List(); - foreach (var benchmarkCase in benchmark.BenchmarksCases) - { - var testId = benchmarkCase.GetTestCaseId(); - if (benchmarkIds == null || benchmarkIds.Contains(testId)) - { - filteredCases.Add(benchmarkCase); - testCases.Add(benchmarkCase.ToVsTestCase(assemblyPath, needsJobInfo)); - } - } - - if (filteredCases.Count > 0) - { - filteredBenchmarks.Add(new BenchmarkRunInfo(filteredCases.ToArray(), benchmark.Type, benchmark.Config, benchmark.CompositeInProcessDiagnoser)); - } - } - - benchmarks = filteredBenchmarks.ToArray(); - - if (benchmarks.Length == 0) - return; - - // Create an event processor which will subscribe to events and push them to VSTest - var eventProcessor = new VsTestEventProcessor(testCases, recorder, cts.Token); - - // Create a logger which will forward all log messages in BDN to the VSTest logger. - var logger = new VsTestLogger(recorder.GetLogger()); - - // Modify all the benchmarks so that the event process and logger is added. - benchmarks = benchmarks - .Select(b => new BenchmarkRunInfo( - b.BenchmarksCases, - b.Type, - b.Config.AddEventProcessor(eventProcessor) - .AddLogger(logger) - .RemoveLoggersOfType() // Console logs are also outputted by VSTestLogger. - .CreateImmutableConfig(), - b.CompositeInProcessDiagnoser)) - .ToArray(); - - // Run all the benchmarks, and ensure that any tests that don't have a result yet are sent. - BenchmarkRunner.Run(benchmarks); - eventProcessor.SendUnsentTestResults(); - } - - /// - /// Stop the benchmarks when next able. - /// - public void Cancel() - { - cts.Cancel(); - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter/ParameterValueDisposer.cs b/src/BenchmarkDotNet.TestAdapter/ParameterValueDisposer.cs new file mode 100644 index 0000000000..c07047770c --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/ParameterValueDisposer.cs @@ -0,0 +1,59 @@ +using BenchmarkDotNet.Running; +using System.Runtime.CompilerServices; + +namespace BenchmarkDotNet.TestAdapter +{ + /// + /// Disposes the parameter values of the benchmarks that were enumerated but will not be run. + /// + /// + /// Enumerating an assembly instantiates the values of every [Params], [ParamsSource] and [ArgumentsSource], and + /// BenchmarkDotNet only disposes the ones belonging to the benchmarks it was handed. A value with a locking + /// finalizer hangs the runtime when it is left to the finalizer thread instead, see dotnet/BenchmarkDotNet#1383, + /// which is what makes this worse than an ordinary leak. + /// + internal static class ParameterValueDisposer + { + /// + /// Disposes every value that the enumerated benchmarks own and the retained ones do not. + /// + /// + /// The values are matched by reference instead of being disposed case by case, because BenchmarkConverter + /// gives the same ParameterInstance to every job and every argument set of a benchmark: disposing a dropped + /// case wholesale would take down values that a benchmark which is about to run still owns. + /// + /// Everything the assembly declares. + /// The benchmarks that are kept, if any. + internal static void DisposeUnused(IEnumerable enumerated, IEnumerable retained) + { + var unused = new HashSet(ReferenceComparer.Instance); + + foreach (var value in GetDisposableParameterValues(enumerated)) + unused.Add(value); + + foreach (var value in GetDisposableParameterValues(retained)) + unused.Remove(value); + + foreach (var value in unused) + value.Dispose(); + } + + private static IEnumerable GetDisposableParameterValues(IEnumerable benchmarkCases) + => benchmarkCases + .SelectMany(benchmarkCase => benchmarkCase.Parameters.Items) + .Select(parameter => parameter.Value) + .OfType(); + + /// + /// Compares by reference, so that a parameter value which overrides Equals is still disposed once per instance. + /// + private sealed class ReferenceComparer : IEqualityComparer + { + public static readonly ReferenceComparer Instance = new ReferenceComparer(); + + public bool Equals(IDisposable? x, IDisposable? y) => ReferenceEquals(x, y); + + public int GetHashCode(IDisposable obj) => RuntimeHelpers.GetHashCode(obj); + } + } +} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTestAdapter.cs b/src/BenchmarkDotNet.TestAdapter/VSTestAdapter.cs deleted file mode 100644 index ced77fae7a..0000000000 --- a/src/BenchmarkDotNet.TestAdapter/VSTestAdapter.cs +++ /dev/null @@ -1,244 +0,0 @@ -using BenchmarkDotNet.TestAdapter.Remoting; -using Microsoft.VisualStudio.TestPlatform.ObjectModel; -using Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter; -using Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging; -using System.Diagnostics; -using System.Reflection; - -namespace BenchmarkDotNet.TestAdapter -{ - /// - /// Discovers and executes benchmarks using the VSTest protocol. - /// - [ExtensionUri(ExecutorUriString)] - [DefaultExecutorUri(ExecutorUriString)] - [FileExtension(".dll")] - [FileExtension(".exe")] - public class VsTestAdapter : ITestExecutor, ITestDiscoverer - { - // This URI is used to identify the adapter. - internal const string ExecutorUriString = "executor://BenchmarkDotNet.TestAdapter"; - internal static readonly Uri ExecutorUri = new Uri(ExecutorUriString); - - /// - /// Cancellation token used to stop any benchmarks that are currently running. - /// - private CancellationTokenSource? cts = null; - - /// - /// Discovers the benchmarks. - /// - /// List of assemblies to search for benchmarks in. - /// A context that the discovery is performed in. - /// Logger that sends messages back to VSTest host. - /// Interface that provides methods for sending discovered benchmarks back to the host. - public void DiscoverTests( - IEnumerable sources, - IDiscoveryContext discoveryContext, - IMessageLogger logger, - ITestCaseDiscoverySink discoverySink) - { - var stopwatch = Stopwatch.StartNew(); - var loggerHelper = new LoggerHelper(logger, stopwatch); - var testCaseFilter = new TestCaseFilter(discoveryContext, loggerHelper); - - foreach (var source in sources) - { - ValidateSourceIsAssemblyOrThrow(source); - foreach (var testCase in GetVsTestCasesFromAssembly(source, logger)) - { - if (!testCaseFilter.MatchTestCase(testCase)) - continue; - - discoverySink.SendTestCase(testCase); - } - } - } - - /// - /// Runs a given set of test cases that represent benchmarks. - /// - /// The tests to run. - /// A context that the run is performed in. - /// Interface used for communicating with the VSTest host. - public void RunTests(IEnumerable? tests, IRunContext? runContext, IFrameworkHandle? frameworkHandle) - { - if (tests == null) - throw new ArgumentNullException(nameof(tests)); - if (frameworkHandle == null) - throw new ArgumentNullException(nameof(frameworkHandle)); - - cts ??= new CancellationTokenSource(); - - var stopwatch = Stopwatch.StartNew(); - var logger = new LoggerHelper(frameworkHandle, stopwatch); - - foreach (var testsPerAssembly in tests.GroupBy(t => t.Source)) - { - RunBenchmarks(testsPerAssembly.Key, frameworkHandle, testsPerAssembly); - } - - cts = null; - } - - /// - /// Runs all/filtered benchmarks in the given set of sources (assemblies). - /// - /// The assemblies to run. - /// A context that the run is performed in. - /// Interface used for communicating with the VSTest host. - public void RunTests(IEnumerable? sources, IRunContext? runContext, IFrameworkHandle? frameworkHandle) - { - if (sources == null) - throw new ArgumentNullException(nameof(sources)); - if (frameworkHandle == null) - throw new ArgumentNullException(nameof(frameworkHandle)); - - cts ??= new CancellationTokenSource(); - - var stopwatch = Stopwatch.StartNew(); - var logger = new LoggerHelper(frameworkHandle, stopwatch); - - foreach (var source in sources) - { - var filter = new TestCaseFilter(runContext!, logger, source, ["Category"]); - if (filter.GetTestCaseFilterValue() != "") - { - var discoveredBenchmarks = GetVsTestCasesFromAssembly(source, frameworkHandle); - var filteredTestCases = discoveredBenchmarks.Where(x => filter.MatchTestCase(x)) - .ToArray(); - - if (filteredTestCases.Length == 0) - continue; - - // Run filtered tests. - RunBenchmarks(source, frameworkHandle, filteredTestCases); - } - else - { - // Run all benchmarks - RunBenchmarks(source, frameworkHandle); - } - } - - - cts = null; - } - - /// - /// Stops any currently running benchmarks. - /// - public void Cancel() - { - cts?.Cancel(); - } - - /// - /// Gets the VSTest test cases in the given assembly. - /// - /// The dll or exe of the benchmark project. - /// A logger that sends logs to VSTest. - /// The VSTest test cases inside the given assembly. - private static List GetVsTestCasesFromAssembly(string assemblyPath, IMessageLogger logger) - { - try - { - // Ensure that the test enumeration is done inside the context of the source directory. - var enumerator = (BenchmarkEnumeratorWrapper)CreateIsolatedType(typeof(BenchmarkEnumeratorWrapper), assemblyPath); - var testCases = enumerator - .GetTestCasesFromAssemblyPathSerialized(assemblyPath) - .Select(SerializationHelpers.Deserialize) - .ToList(); - - // Validate that all test ids are unique - var idLookup = new Dictionary(); - foreach (var testCase in testCases) - { - if (idLookup.TryGetValue(testCase.Id, out var matchingCase)) - throw new Exception($"Encountered Duplicate Test ID: '{testCase.DisplayName}' and '{matchingCase}'"); - - idLookup[testCase.Id] = testCase.DisplayName; - } - - return testCases; - } - catch (Exception ex) - { - logger.SendMessage(TestMessageLevel.Error, $"Failed to load benchmarks from assembly\n{ex}"); - throw; - } - } - - /// - /// Runs the benchmarks in the given source. - /// - /// The dll or exe of the benchmark project. - /// An interface used to communicate with the VSTest host. - /// - /// The specific test cases to be run if specified. - /// If unspecified, runs all the test cases in the source. - /// - private void RunBenchmarks(string source, IFrameworkHandle frameworkHandle, IEnumerable? testCases = null) - { - ValidateSourceIsAssemblyOrThrow(source); - - // Create a HashSet of all the TestCase IDs to be run if specified. - var caseIds = testCases == null ? null : new HashSet(testCases.Select(c => c.Id)); - - try - { - // Ensure that test execution is done inside the context of the source directory. - var executor = (BenchmarkExecutorWrapper)CreateIsolatedType(typeof(BenchmarkExecutorWrapper), source); - cts?.Token.Register(executor.Cancel); - - executor.RunBenchmarks(source, new TestExecutionRecorderWrapper(frameworkHandle), caseIds); - } - catch (Exception ex) - { - frameworkHandle.SendMessage(TestMessageLevel.Error, $"Failed to run benchmarks in assembly\n{ex}"); - throw; - } - } - - /// - /// This will create the given type in a child AppDomain when used in .NET Framework. - /// If not in the .NET Framework, it will use the current AppDomain. - /// - /// The type to create. - /// The dll or exe of the benchmark project. - /// The created object. - private static object CreateIsolatedType(Type type, string assemblyPath) - { - // .NET Framework runs require a custom AppDomain to be set up to run the benchmarks in because otherwise, - // all the assemblies will be loaded from the VSTest console rather than from the directory that the BDN - // program under test lives in. .NET Core assembly resolution is smarter and will correctly load the right - // assembly versions as needed and does not require a custom AppDomain. Unfortunately, the APIs needed to - // create the AppDomain for .NET Framework are not part of .NET Standard, and so a multi-targeting solution - // such as this is required to get this to work. This same approach is also used by other .NET unit testing - // libraries as well, further justifying this approach to solving how to get the correct assemblies loaded. -#if NETFRAMEWORK - var appBase = Path.GetDirectoryName(assemblyPath); - var setup = new AppDomainSetup { ApplicationBase = appBase }; - var domainName = $"Isolated Domain for {type.Name}"; - var appDomain = AppDomain.CreateDomain(domainName, null, setup); - return appDomain.CreateInstanceAndUnwrap( - type.Assembly.FullName, type.FullName, false, BindingFlags.Default, null, null, null, null); -#else - return Activator.CreateInstance(type); -#endif - } - - private static void ValidateSourceIsAssemblyOrThrow(string source) - { - if (string.IsNullOrEmpty(source)) - throw new ArgumentException($"'{nameof(source)}' cannot be null or whitespace.", nameof(source)); - - if (!Path.HasExtension(source)) - throw new NotSupportedException($"Missing extension on source '{source}', must have the extension '.dll' or '.exe'."); - - var extension = Path.GetExtension(source); - if (!string.Equals(extension, ".dll", StringComparison.OrdinalIgnoreCase) && !string.Equals(extension, ".exe", StringComparison.OrdinalIgnoreCase)) - throw new NotSupportedException($"Unsupported extension on source '{source}', must have the extension '.dll' or '.exe'."); - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTestEventProcessor.cs b/src/BenchmarkDotNet.TestAdapter/VSTestEventProcessor.cs deleted file mode 100644 index 8b4546b034..0000000000 --- a/src/BenchmarkDotNet.TestAdapter/VSTestEventProcessor.cs +++ /dev/null @@ -1,194 +0,0 @@ -using BenchmarkDotNet.EventProcessors; -using BenchmarkDotNet.Extensions; -using BenchmarkDotNet.Reports; -using BenchmarkDotNet.Running; -using BenchmarkDotNet.TestAdapter.Remoting; -using BenchmarkDotNet.Toolchains.Results; -using BenchmarkDotNet.Validators; -using Microsoft.VisualStudio.TestPlatform.ObjectModel; -using Perfolizer.Mathematics.Histograms; -using System.Diagnostics; -using System.Globalization; -using System.Text; - -namespace BenchmarkDotNet.TestAdapter -{ - /// - /// An event processor which will pass on benchmark execution information to VSTest. - /// - internal class VsTestEventProcessor : EventProcessor - { - private readonly Dictionary cases; - private readonly TestExecutionRecorderWrapper recorder; - private readonly CancellationToken cancellationToken; - private readonly Stopwatch runTimerStopwatch = new(); - private readonly Dictionary testResults = []; - private readonly HashSet sentTestResults = []; - - public VsTestEventProcessor( - List cases, - TestExecutionRecorderWrapper recorder, - CancellationToken cancellationToken) - { - this.cases = cases.ToDictionary(c => c.Id); - this.recorder = recorder; - this.cancellationToken = cancellationToken; - } - - public override void OnValidationError(ValidationError validationError) - { - // If the error is not linked to a benchmark case, then set the error on all benchmarks - var errorCases = validationError.BenchmarkCase == null - ? cases.Values.ToList() - : [cases[validationError.BenchmarkCase.GetTestCaseId()]]; - foreach (var testCase in errorCases) - { - var testResult = GetOrCreateTestResult(testCase); - - if (validationError.IsCritical) - { - // Fail if there is a critical validation error - testResult.Outcome = TestOutcome.Failed; - - // Append validation error message to end of test case error message - testResult.ErrorMessage = testResult.ErrorMessage == null - ? validationError.Message - : $"{testResult.ErrorMessage}\n{validationError.Message}"; - - // The test result is not sent yet, in case there are multiple validation errors that need to be sent. - } - else - { - // If the validation error is not critical, append it as a message - testResult.Messages.Add(new TestResultMessage(TestResultMessage.StandardOutCategory, $"WARNING: {validationError.Message}\n")); - } - } - } - - public override void OnBuildComplete(BuildPartition buildPartition, BuildResult buildResult) - { - // Only need to handle build failures - if (!buildResult.IsBuildSuccess) - { - foreach (var benchmarkBuildInfo in buildPartition.Benchmarks) - { - var testCase = cases[benchmarkBuildInfo.BenchmarkCase.GetTestCaseId()]; - var testResult = GetOrCreateTestResult(testCase); - - if (buildResult.GenerateException != null) - testResult.ErrorMessage = $"// Generate Exception: {buildResult.GenerateException.Message}"; - else if (!buildResult.IsBuildSuccess && buildResult.TryToExplainFailureReason(buildPartition.GetInProcessDiagnoserHandlerTypes(), out string? reason)) - testResult.ErrorMessage = $"// Build Error: {reason}"; - else if (buildResult.ErrorMessage != null) - testResult.ErrorMessage = $"// Build Error: {buildResult.ErrorMessage}"; - testResult.Outcome = TestOutcome.Failed; - - // Send the result immediately - RecordStart(testCase); - RecordEnd(testCase, testResult.Outcome); - RecordResult(testResult); - sentTestResults.Add(testCase.Id); - } - } - } - - public override void OnStartRunBenchmark(BenchmarkCase benchmarkCase) - { - // TODO: add proper cancellation support to BDN so that we don't need to do cancellation through the event processor - cancellationToken.ThrowIfCancellationRequested(); - - var testCase = cases[benchmarkCase.GetTestCaseId()]; - var testResult = GetOrCreateTestResult(testCase); - testResult.StartTime = DateTimeOffset.UtcNow; - - RecordStart(testCase); - runTimerStopwatch.Restart(); - } - - public override void OnEndRunBenchmark(BenchmarkCase benchmarkCase, BenchmarkReport report) - { - var testCase = cases[benchmarkCase.GetTestCaseId()]; - var testResult = GetOrCreateTestResult(testCase); - testResult.EndTime = DateTimeOffset.UtcNow; - testResult.Duration = runTimerStopwatch.Elapsed; - testResult.Outcome = report.Success ? TestOutcome.Passed : TestOutcome.Failed; - - var resultRuns = report.GetResultRuns(); - - // Provide the raw result runs data. - testResult.SetPropertyValue(VsTestProperties.Measurement, resultRuns.Select(m => m.Nanoseconds.ToString()).ToArray()); - - // Add a message to the TestResult which contains the results summary. - testResult.Messages.Add(new TestResultMessage(TestResultMessage.StandardOutCategory, report.BenchmarkCase.DisplayInfo + "\n")); - testResult.Messages.Add(new TestResultMessage(TestResultMessage.StandardOutCategory, $"Runtime = {report.GetRuntimeInfo()}; GC = {report.GetGcInfo()}\n")); - - var statistics = resultRuns.GetStatistics(); - var cultureInfo = CultureInfo.InvariantCulture; - var formatter = statistics.CreateNanosecondFormatter(cultureInfo); - - var builder = new StringBuilder(); - var histogram = HistogramBuilder.Adaptive.Build(statistics.Sample.Values); - builder.AppendLine("-------------------- Histogram --------------------"); - builder.AppendLine(histogram.ToString(formatter)); - builder.AppendLine("---------------------------------------------------"); - - var statisticsOutput = statistics.ToString(cultureInfo, formatter, calcHistogram: false); - builder.AppendLine(statisticsOutput); - - testResult.Messages.Add(new TestResultMessage(TestResultMessage.StandardOutCategory, builder.ToString())); - - RecordEnd(testResult.TestCase, testResult.Outcome); - RecordResult(testResult); - sentTestResults.Add(testCase.Id); - } - - /// - /// Iterate through all the benchmarks that were scheduled to run, and if they haven't been sent yet, send the result through. - /// - public void SendUnsentTestResults() - { - foreach (var testCase in cases.Values) - { - if (!sentTestResults.Contains(testCase.Id)) - { - var testResult = GetOrCreateTestResult(testCase); - if (testResult.Outcome == TestOutcome.None) - testResult.Outcome = TestOutcome.Skipped; - RecordStart(testCase); - RecordEnd(testCase, testResult.Outcome); - RecordResult(testResult); - } - } - } - - private TestResult GetOrCreateTestResult(TestCase testCase) - { - if (testResults.TryGetValue(testCase.Id, out var testResult)) - return testResult; - - var newResult = new TestResult(testCase) - { - ComputerName = Environment.MachineName, - DisplayName = testCase.DisplayName - }; - - testResults[testCase.Id] = newResult; - return newResult; - } - - private void RecordStart(TestCase testCase) - { - recorder.RecordStart(SerializationHelpers.Serialize(testCase)); - } - - private void RecordEnd(TestCase testCase, TestOutcome testOutcome) - { - recorder.RecordEnd(SerializationHelpers.Serialize(testCase), testOutcome); - } - - private void RecordResult(TestResult testResult) - { - recorder.RecordResult(SerializationHelpers.Serialize(testResult)); - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTestLogger.cs b/src/BenchmarkDotNet.TestAdapter/VSTestLogger.cs deleted file mode 100644 index 03bf860cba..0000000000 --- a/src/BenchmarkDotNet.TestAdapter/VSTestLogger.cs +++ /dev/null @@ -1,62 +0,0 @@ -using BenchmarkDotNet.Loggers; -using Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging; -using System.Text; - -namespace BenchmarkDotNet.TestAdapter -{ - /// - /// A class to send logs from BDN to the VSTest output log. - /// - internal sealed class VsTestLogger : ILogger - { - private readonly IMessageLogger messageLogger; - private readonly StringBuilder currentLine = new StringBuilder(); - private TestMessageLevel currentLevel = TestMessageLevel.Informational; - - public VsTestLogger(IMessageLogger logger) - { - messageLogger = logger; - } - - public string Id => nameof(VsTestLogger); - - public int Priority => 0; - - public void Flush() - { - WriteLine(); - } - - public void Write(LogKind logKind, string text) - { - currentLine.Append(text); - - // Assume that if the log kind is an error, that the whole line is treated as an error - // The level will be reset to Informational when WriteLine() is called. - currentLevel = logKind switch - { - LogKind.Error => TestMessageLevel.Error, - LogKind.Warning => TestMessageLevel.Warning, - _ => currentLevel - }; - } - - public void WriteLine() - { - // The VSTest logger throws an error on logging empty or whitespace strings, so skip them. - if (currentLine.Length == 0) - return; - - messageLogger.SendMessage(currentLevel, currentLine.ToString()); - - currentLevel = TestMessageLevel.Informational; - currentLine.Clear(); - } - - public void WriteLine(LogKind logKind, string text) - { - Write(logKind, text); - WriteLine(); - } - } -} diff --git a/src/BenchmarkDotNet.TestAdapter/VSTestProperties.cs b/src/BenchmarkDotNet.TestAdapter/VSTestProperties.cs deleted file mode 100644 index ac039045f5..0000000000 --- a/src/BenchmarkDotNet.TestAdapter/VSTestProperties.cs +++ /dev/null @@ -1,22 +0,0 @@ -using Microsoft.VisualStudio.TestPlatform.ObjectModel; - -namespace BenchmarkDotNet.TestAdapter -{ - /// - /// A class that contains all the custom properties that can be set on VSTest TestCase and TestResults. - /// Some of these properties are well known as they are also used by VSTest adapters for other test libraries. - /// - internal static class VsTestProperties - { - /// - /// A test property used for storing the test results so that they could be accessed - /// programmatically from a custom VSTest runner. - /// - internal static readonly TestProperty Measurement = TestProperty.Register( - "BenchmarkDotNet.TestAdapter.Measurements", - "Measurements", - typeof(string[]), - TestPropertyAttributes.Hidden, - typeof(TestResult)); - } -} From 55f5b6d2590190665ea1391c7e2fa429e1cb00fa Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Mon, 7 Sep 2026 09:58:47 +0100 Subject: [PATCH 086/110] Improve test infra: new project, docs, and validation .gitignore updated for BenchmarkDotNet.TestAdapter packages. Added BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized to solution. Enhanced testadapter.md with details on benchmark name/category encoding and display. Refactored FullNameProvider.GetMethodDisplayName for better parameter handling. Updated TypeFilter and GenericBenchmarksValidator to use new GenericBenchmarksBuilder properties (IsSuccess, Type, Error). Improved error reporting and robustness in GenericBuilderTests for unreadable attributes. --- .gitignore | 3 ++ BenchmarkDotNet.slnx | 1 + docs/articles/features/testadapter.md | 16 +++++-- .../Exporters/FullNameProvider.cs | 18 +++++--- src/BenchmarkDotNet/Running/TypeFilter.cs | 2 +- .../Validators/GenericBenchmarksValidator.cs | 4 +- .../GenericBuilderTests.cs | 43 +++++++++++++++++++ 7 files changed, 74 insertions(+), 13 deletions(-) diff --git a/.gitignore b/.gitignore index 4226473653..ba9270b557 100644 --- a/.gitignore +++ b/.gitignore @@ -67,6 +67,9 @@ Resource.designer.cs # Tests TestResults +# The smoke test restores BenchmarkDotNet.TestAdapter into a packages folder of its own, see its .csproj +build/smoke-tests/packages/ + ## Mac OS # General diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index 2c55c5d3e7..6b04cdbbd9 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -45,6 +45,7 @@ + diff --git a/docs/articles/features/testadapter.md b/docs/articles/features/testadapter.md index 8a091886fb..6f039328bb 100644 --- a/docs/articles/features/testadapter.md +++ b/docs/articles/features/testadapter.md @@ -131,12 +131,20 @@ dotnet run -c Release -- --treenode-filter "/*/*/*/*[Category=Fast]" ``` The tree node filter path is `////`, - and `[BenchmarkCategory]` attributes are exposed as a `Category` trait that the filter can match on. -Because the platform separates the levels of that path with `/`, a benchmark parameter whose value contains one is - percent encoded in the path: a parameter value of `a/b` is written `a%2Fb` in a filter, and a literal `%` is - written `%25`. + and the categories of a benchmark are exposed as a `Category` trait that the filter can match on. +The last level ends with the job between brackets, as in `MyBenchmark(Size: 1) [Dry]`. + +The characters the filter itself gives a meaning to are percent encoded in that path: + `/`, which separates the levels, is written `%2F`; + `[` and `]`, which delimit a property filter, are written `%5B` and `%5D`; + and a literal `%` is written `%25`. +So a parameter value of `a/b` is spelled `a%2Fb` in a filter, and the `[Dry]` above is spelled `%5BDry%5D`. +The parentheses around the parameters are not encoded - escape them with a backslash in the filter instead. This affects the filter only; the name the benchmark is displayed under is unchanged. +A benchmark is displayed under its `[Benchmark(Description = "...")]` when it has one, and under the name of its + method otherwise, followed by its parameters. + ## Keeping your own entry point The generated entry point starts the test application, which means it replaces the `BenchmarkSwitcher` entry point that diff --git a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs index 5411f7243b..6d58947988 100644 --- a/src/BenchmarkDotNet/Exporters/FullNameProvider.cs +++ b/src/BenchmarkDotNet/Exporters/FullNameProvider.cs @@ -100,19 +100,25 @@ internal static string GetMethodName(BenchmarkCase benchmarkCase) } /// - /// Gets the method name to show to a user, which is the [Benchmark(Description = ...)] when one is set and - /// the method name otherwise, followed by the parameters. + /// Gets the method name to show to a user, which is the given name followed by the parameters. /// + /// + /// The name is taken as an argument rather than read off the descriptor, because + /// is the console table form: it wraps a + /// description containing a space, a quote or a bracket in single quotes, so that BenchmarkDotNet's own + /// --filter can delimit it. A label shown by an IDE wants the description as it was written. + /// /// The benchmark case. + /// The name of the benchmark, without its parameters. /// The method name to display. - internal static string GetMethodDisplayName(BenchmarkCase benchmarkCase) + internal static string GetMethodDisplayName(BenchmarkCase benchmarkCase, string name) { - var name = new StringBuilder(benchmarkCase.Descriptor.WorkloadMethodDisplayInfo); + var builder = new StringBuilder(name); if (benchmarkCase.HasParameters) - name.Append(GetBenchmarkParameters(benchmarkCase.Descriptor.WorkloadMethod, benchmarkCase.Parameters)); + builder.Append(GetBenchmarkParameters(benchmarkCase.Descriptor.WorkloadMethod, benchmarkCase.Parameters)); - return name.ToString(); + return builder.ToString(); } private static string GetBenchmarkParameters(MethodInfo method, ParameterInstances benchmarkParameters) diff --git a/src/BenchmarkDotNet/Running/TypeFilter.cs b/src/BenchmarkDotNet/Running/TypeFilter.cs index b9420915b5..ff3fb548b2 100644 --- a/src/BenchmarkDotNet/Running/TypeFilter.cs +++ b/src/BenchmarkDotNet/Running/TypeFilter.cs @@ -42,7 +42,7 @@ public static (bool allTypesValid, IReadOnlyList runnable) GetTypesWithRun { if (type.ContainsRunnableBenchmarks()) { - validRunnableTypes.AddRange(GenericBenchmarksBuilder.BuildGenericsIfNeeded(type).Where(tuple => tuple.isSuccess).Select(tuple => tuple.result)); + validRunnableTypes.AddRange(GenericBenchmarksBuilder.BuildGenericsIfNeeded(type).Where(built => built.IsSuccess).Select(built => built.Type)); } else { diff --git a/src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs b/src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs index be0522c138..70a85e0ec0 100644 --- a/src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs +++ b/src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs @@ -16,8 +16,8 @@ public IAsyncEnumerable ValidateAsync(ValidationParameters vali .Distinct() .SelectMany(assembly => assembly.GetRunnableBenchmarks()) .SelectMany(GenericBenchmarksBuilder.BuildGenericsIfNeeded) - .Where(result => !result.isSuccess) - .Select(result => new ValidationError(false, $"Generic type {result.result.Name} failed to build due to wrong type argument or arguments count, ignoring.")) + .Where(built => !built.IsSuccess) + .Select(built => new ValidationError(false, built.Error!)) .ToAsyncEnumerable(); } } \ No newline at end of file diff --git a/tests/BenchmarkDotNet.Tests/GenericBuilderTests.cs b/tests/BenchmarkDotNet.Tests/GenericBuilderTests.cs index dc41b9e426..5f5bca2166 100644 --- a/tests/BenchmarkDotNet.Tests/GenericBuilderTests.cs +++ b/tests/BenchmarkDotNet.Tests/GenericBuilderTests.cs @@ -1,4 +1,5 @@ using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; using BenchmarkDotNet.Helpers; namespace BenchmarkDotNet.Tests @@ -122,5 +123,47 @@ public class GenericBenchmarkWithConstraintsWrongArgs where T1 : struct [Benchmark] public T2 CreateT2() => Activator.CreateInstance(); } + + [Fact] + public void TestTypeWithUnreadableAttributesIsDropped() + { + // Reflection constructs every attribute of a type in order to hand any of them back, so a [Config] that + // cannot be instantiated makes the read of the [GenericTypeArguments] throw. The type is unusable at that + // point - BenchmarkConverter would throw on the very same read - so it is dropped, and the benchmarks + // that were listed next to it are still returned. + var types = GenericBenchmarksBuilder.GetRunnableBenchmarks( + [typeof(BenchmarkWithAbstractConfig), typeof(BenchmarkWithInaccessibleConfig), typeof(OneArgGenericBenchmark<>)]); + + Assert.Equal(2, types.Length); + Assert.Single(types, typeof(OneArgGenericBenchmark)); + Assert.Single(types, typeof(OneArgGenericBenchmark)); + } + + [Fact] + public void TestTypeWithUnreadableAttributesIsReportedAsAFailure() + { + var built = GenericBenchmarksBuilder.BuildGenericsIfNeeded(typeof(BenchmarkWithAbstractConfig)).ToArray(); + + var failure = Assert.Single(built); + Assert.False(failure.IsSuccess); + Assert.Contains(nameof(BenchmarkWithAbstractConfig), failure.Error); + } + + [Config(typeof(DebugConfig))] // abstract, so ConfigAttribute's constructor throws + public class BenchmarkWithAbstractConfig + { + [Benchmark] public int Identity() => 1; + } + + [Config(typeof(NoPublicConstructorConfig))] + public class BenchmarkWithInaccessibleConfig + { + [Benchmark] public int Identity() => 1; + + private class NoPublicConstructorConfig : ManualConfig + { + private NoPublicConstructorConfig() { } + } + } } } \ No newline at end of file From 3b61b934badcf68f8766d7007a8e62ab80cb593b Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Tue, 8 Sep 2026 09:54:14 +0100 Subject: [PATCH 087/110] Add IsUnreadable to GenericBenchmarkType and update builder GenericBenchmarkType now has an IsUnreadable property to indicate types rejected before reading benchmarks. Added Unreadable static method and updated constructor to accept isUnreadable. Failed static method now sets isUnreadable to false. GenericBenchmarksBuilder uses BuildRunnableBenchmarks to return all built and rejected types. BuildGenericsIfNeeded returns Unreadable for types with missing arguments. Added XML docs for new property and method. --- .../Helpers/GenericBenchmarkType.cs | 16 +++++++++++++--- .../Helpers/GenericBenchmarksBuilder.cs | 15 ++++++++++++--- 2 files changed, 25 insertions(+), 6 deletions(-) diff --git a/src/BenchmarkDotNet/Helpers/GenericBenchmarkType.cs b/src/BenchmarkDotNet/Helpers/GenericBenchmarkType.cs index 6ef0f3ff2c..46677d02f8 100644 --- a/src/BenchmarkDotNet/Helpers/GenericBenchmarkType.cs +++ b/src/BenchmarkDotNet/Helpers/GenericBenchmarkType.cs @@ -5,10 +5,11 @@ namespace BenchmarkDotNet.Helpers /// internal readonly struct GenericBenchmarkType { - private GenericBenchmarkType(Type type, string? error) + private GenericBenchmarkType(Type type, string? error, bool isUnreadable) { Type = type; Error = error; + IsUnreadable = isUnreadable; } /// @@ -22,10 +23,19 @@ private GenericBenchmarkType(Type type, string? error) /// internal string? Error { get; } + /// + /// Gets whether the type was rejected before any of its benchmarks could be read, rather than because one set + /// of [GenericTypeArguments] did not fit it. Nothing downstream reports such a type - GenericBenchmarksValidator + /// only runs once at least one benchmark of the assembly survived - so whoever drops it has to say so. + /// + internal bool IsUnreadable { get; } + internal bool IsSuccess => Error == null; - internal static GenericBenchmarkType Runnable(Type type) => new GenericBenchmarkType(type, null); + internal static GenericBenchmarkType Runnable(Type type) => new GenericBenchmarkType(type, null, false); + + internal static GenericBenchmarkType Failed(Type type, string error) => new GenericBenchmarkType(type, error, false); - internal static GenericBenchmarkType Failed(Type type, string error) => new GenericBenchmarkType(type, error); + internal static GenericBenchmarkType Unreadable(Type type, string error) => new GenericBenchmarkType(type, error, true); } } diff --git a/src/BenchmarkDotNet/Helpers/GenericBenchmarksBuilder.cs b/src/BenchmarkDotNet/Helpers/GenericBenchmarksBuilder.cs index 3a86a54ea7..ddffd86be9 100644 --- a/src/BenchmarkDotNet/Helpers/GenericBenchmarksBuilder.cs +++ b/src/BenchmarkDotNet/Helpers/GenericBenchmarksBuilder.cs @@ -6,16 +6,25 @@ namespace BenchmarkDotNet.Helpers internal static class GenericBenchmarksBuilder { internal static Type[] GetRunnableBenchmarks(IEnumerable types) - => types.Where(type => type.ContainsRunnableBenchmarks()) - .SelectMany(BuildGenericsIfNeeded) + => BuildRunnableBenchmarks(types) .Where(x => x.IsSuccess) .Select(x => x.Type) .ToArray(); + /// + /// Builds the benchmark types of the given types, keeping the ones that could not be built so that a caller + /// with somewhere to report them can. + /// + /// The types to consider. + /// Every type that was built, and every one that was rejected. + internal static IEnumerable BuildRunnableBenchmarks(IEnumerable types) + => types.Where(type => type.ContainsRunnableBenchmarks()) + .SelectMany(BuildGenericsIfNeeded); + internal static IEnumerable BuildGenericsIfNeeded(Type type) { if (!TryGetGenericTypeArguments(type, out var typeArguments, out var error)) - return [GenericBenchmarkType.Failed(type, error)]; + return [GenericBenchmarkType.Unreadable(type, error)]; if (typeArguments.Length > 0) return BuildGenericTypes(type, typeArguments); From 18685fe7e7051690c6e8e75da818ceb561ab7fe9 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Tue, 8 Sep 2026 09:54:38 +0100 Subject: [PATCH 088/110] Add server mode session class and expand adapter tests Added TestingPlatformServerModeSession for JSON-RPC server mode integration, simulating VS/VS Code test host interactions. Expanded TestingPlatformAdapterTests to cover async-disposable parameter handling and server mode parameter lifecycle. Improved test node state tracking and request/response synchronization. --- .../TestingPlatformAdapterTests.cs | 60 ++++ .../TestingPlatformServerModeSession.cs | 308 ++++++++++++++++++ 2 files changed, 368 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests/TestingPlatformServerModeSession.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs index a49a6e59e1..153f585be3 100644 --- a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs +++ b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs @@ -30,6 +30,11 @@ public void EveryBenchmarkIsDiscoveredUnderItsOwnName() { string[] expected = [ + // The values of this one are IAsyncDisposable and not IDisposable, which is what an async + // [ParamsSource] can produce since #3248. + "AsyncDisposableProbe.Identity(Value: async-1)", + "AsyncDisposableProbe.Identity(Value: async-2)", + "BracketProbe.Length(Value: \"[Dry]\")", "CategoryProbe.Identity", @@ -149,6 +154,61 @@ public void ParameterValuesAreDisposedWhenBenchmarksAreOnlyListed() ReadDisposalReport(PassingProbes, "disposable-probe.txt", () => Discover(PassingProbes))); } + [Fact] + public void ParameterValuesThatAreOnlyAsyncDisposableAreDisposedToo() + { + // An async [ParamsSource] can hand back a value that implements IAsyncDisposable and not IDisposable. + // Disposal goes through ParameterInstance, which knows both, rather than casting the value to + // IDisposable - which would drop these on the floor and leave them to the finalizer. + Assert.Equal( + "created=2 disposed=2", + ReadDisposalReport(PassingProbes, "async-disposable-probe.txt", () => Discover(PassingProbes))); + } + + [Fact] + public void ParameterValuesSurviveADiscoveryWhenTheSameProcessRunsThemAfterwards() + { + // Server mode is how Visual Studio and the Visual Studio Code Test Explorer drive the platform: one + // process serves the discovery and then the runs. Every request enumerates the assembly again, and a + // source backed by a cached collection hands back the very same values, so disposing them when the + // discovery request ends would leave the run executing against disposed objects - which DisposableProbe + // turns into an ObjectDisposedException rather than letting it pass unnoticed. + IReadOnlyList discovered = []; + IReadOnlyList ran = []; + + var report = ReadDisposalReport( + PassingProbes, + "disposable-probe.txt", + () => (discovered, ran) = TestingPlatformServerModeSession.DiscoverThenRun( + GetProbeApplication(PassingProbes), + "DisposableProbe.Identity(Value: tracked", + Timeout)); + + Assert.NotEmpty(discovered); + Assert.Equal(3, ran.Count); + Assert.All(ran, node => Assert.Equal("passed", node.ExecutionState)); + + // Once each: BenchmarkDotNet disposes what it ran, and the adapter must not have done so beforehand. + Assert.Equal("created=3 disposed=3", report); + } + + [Fact] + public void ParameterValuesOfBenchmarksNoRequestRanAreDisposedWhenTheApplicationEnds() + { + // The mirror image of the test above: the values of every benchmark that neither request ran are the + // adapter's to dispose, and holding them for the application rather than for the request must not turn + // into either a leak or a second disposal. + var report = ReadDisposalReport( + PassingProbes, + "async-disposable-probe.txt", + () => TestingPlatformServerModeSession.DiscoverThenRun( + GetProbeApplication(PassingProbes), + "DisposableProbe.Identity(Value: tracked", + Timeout)); + + Assert.Equal("created=2 disposed=2", report); + } + [Fact] public void ParameterValuesAreDisposedWhenOnlyOneBenchmarkOfASetIsRun() { diff --git a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformServerModeSession.cs b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformServerModeSession.cs new file mode 100644 index 0000000000..60b7fb61c1 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformServerModeSession.cs @@ -0,0 +1,308 @@ +#if NETCOREAPP +using System.Diagnostics; +using System.Net; +using System.Net.Sockets; +using System.Text; +using System.Text.Json; + +namespace BenchmarkDotNet.IntegrationTests +{ + /// + /// Drives a Microsoft.Testing.Platform application through its server mode, which is how Visual Studio and the + /// Visual Studio Code Test Explorer run tests: one process serves a discovery request and then the run requests + /// that follow it, rather than being started again for each of them. + /// + /// + /// The platform speaks JSON-RPC framed the way the language server protocol frames it, over a socket that the + /// client listens on and the test host connects back to. Only the handful of messages this needs are implemented: + /// `initialize`, `testing/discoverTests`, `testing/runTests` and `exit`. + /// + internal sealed class TestingPlatformServerModeSession : IDisposable + { + private readonly Process process; + private readonly TcpClient client; + private readonly NetworkStream stream; + private readonly Dictionary> pendingRequests = []; + private readonly Dictionary> pendingRuns = []; + private readonly List nodes = []; + private readonly TimeSpan timeout; + private int lastRequestId; + + private TestingPlatformServerModeSession(Process process, TcpClient client, TimeSpan timeout) + { + this.process = process; + this.client = client; + this.timeout = timeout; + stream = client.GetStream(); + + Task.Run(ReadLoop); + } + + /// + /// Discovers every benchmark of the application and then runs the ones whose display name contains the given + /// text, from the one process. + /// + /// The probe application to drive. + /// The text the display name of a benchmark has to contain to be run. + /// How long any one step may take. + /// The nodes the discovery reported, and the last state each ran node was reported in. + public static (IReadOnlyList Discovered, IReadOnlyList Ran) DiscoverThenRun( + string application, + string runFilter, + TimeSpan timeout) + { + var listener = new TcpListener(IPAddress.Loopback, 0); + listener.Start(); + + try + { + var port = ((IPEndPoint)listener.LocalEndpoint).Port; + var startInfo = new ProcessStartInfo(application) + { + WorkingDirectory = Path.GetDirectoryName(application), + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false + }; + + foreach (var argument in new[] { "--server", "--client-host", "127.0.0.1", "--client-port", port.ToString(), "--no-ansi" }) + startInfo.ArgumentList.Add(argument); + + using var process = Process.Start(startInfo)!; + process.OutputDataReceived += (_, _) => { }; + process.ErrorDataReceived += (_, _) => { }; + process.BeginOutputReadLine(); + process.BeginErrorReadLine(); + + using var client = listener.AcceptTcpClient(); + using var session = new TestingPlatformServerModeSession(process, client, timeout); + + return session.Run(runFilter); + } + finally + { + listener.Stop(); + } + } + + private (IReadOnlyList, IReadOnlyList) Run(string runFilter) + { + SendRequest("initialize", new + { + processId = Environment.ProcessId, + clientInfo = new { name = "BenchmarkDotNet.IntegrationTests", version = "1.0.0" }, + capabilities = new { testing = new { debuggerProvider = false } } + }); + Send(new { jsonrpc = "2.0", method = "initialized", @params = new { } }); + + var discovered = Exchange("testing/discoverTests", runId => new { runId }); + + var selected = discovered + .Where(node => node.DisplayName.Contains(runFilter, StringComparison.Ordinal)) + .ToArray(); + + if (selected.Length == 0) + throw new InvalidOperationException($"No discovered benchmark matched '{runFilter}'."); + + var ran = Exchange("testing/runTests", runId => new + { + runId, + tests = selected + .Select(node => new Dictionary { ["uid"] = node.Uid, ["display-name"] = node.DisplayName }) + .ToArray() + }); + + Send(new { jsonrpc = "2.0", method = "exit", @params = new { } }); + + if (!process.WaitForExit((int)timeout.TotalMilliseconds)) + { + process.Kill(entireProcessTree: true); + throw new TimeoutException("The test host did not exit after the session was closed."); + } + + // Lets the probe write the report file its process exit handler produces. + process.WaitForExit(); + + return (discovered, ran); + } + + /// + /// Sends one request and collects the node updates the platform reports for it. + /// + private ServerNode[] Exchange(string method, Func parameters) + { + var runId = Guid.NewGuid().ToString(); + var completion = new TaskCompletionSource(); + + lock (pendingRuns) + pendingRuns[runId] = completion; + + lock (nodes) + nodes.Clear(); + + SendRequest(method, parameters(runId)); + + if (!completion.Task.Wait(timeout)) + throw new TimeoutException($"'{method}' did not complete within {timeout}."); + + lock (nodes) + { + // The platform reports a node again whenever its state changes, and the last one is the outcome. + return nodes + .GroupBy(node => node.Uid, StringComparer.Ordinal) + .Select(group => group.Last()) + .ToArray(); + } + } + + private void SendRequest(string method, object parameters) + { + var id = Interlocked.Increment(ref lastRequestId); + var completion = new TaskCompletionSource(); + + lock (pendingRequests) + pendingRequests[id] = completion; + + Send(new { jsonrpc = "2.0", id, method, @params = parameters }); + + if (!completion.Task.Wait(timeout)) + throw new TimeoutException($"'{method}' was not answered within {timeout}."); + } + + private void Send(object message) + { + var body = Encoding.UTF8.GetBytes(JsonSerializer.Serialize(message)); + var header = Encoding.UTF8.GetBytes($"Content-Length: {body.Length}\r\n\r\n"); + + lock (stream) + { + stream.Write(header, 0, header.Length); + stream.Write(body, 0, body.Length); + stream.Flush(); + } + } + + private void ReadLoop() + { + var buffer = new List(); + var chunk = new byte[8192]; + + while (true) + { + int read; + + try + { + read = stream.Read(chunk, 0, chunk.Length); + } + catch + { + return; + } + + if (read == 0) + return; + + buffer.AddRange(chunk.Take(read)); + + while (TryReadMessage(buffer, out var json)) + Handle(json); + } + } + + private static bool TryReadMessage(List buffer, out string json) + { + json = string.Empty; + + // The header is ASCII, so the byte offsets of the separator and of the character offsets agree. + var text = Encoding.ASCII.GetString(buffer.ToArray()); + var headerEnd = text.IndexOf("\r\n\r\n", StringComparison.Ordinal); + if (headerEnd < 0) + return false; + + var lengthHeader = text.Substring(0, headerEnd) + .Split(["\r\n"], StringSplitOptions.None) + .First(header => header.StartsWith("Content-Length", StringComparison.OrdinalIgnoreCase)); + var length = int.Parse(lengthHeader.Split(':')[1].Trim()); + var bodyStart = headerEnd + 4; + + if (buffer.Count < bodyStart + length) + return false; + + json = Encoding.UTF8.GetString(buffer.GetRange(bodyStart, length).ToArray()); + buffer.RemoveRange(0, bodyStart + length); + + return true; + } + + private void Handle(string json) + { + using var document = JsonDocument.Parse(json); + var root = document.RootElement; + + if (root.TryGetProperty("id", out var id) && id.ValueKind == JsonValueKind.Number) + { + lock (pendingRequests) + { + if (pendingRequests.TryGetValue(id.GetInt32(), out var request)) + request.TrySetResult(true); + } + } + + if (!root.TryGetProperty("method", out var method) + || method.GetString() != "testing/testUpdates/tests" + || !root.TryGetProperty("params", out var parameters)) + { + return; + } + + if (parameters.TryGetProperty("changes", out var changes) && changes.ValueKind == JsonValueKind.Array) + { + lock (nodes) + { + foreach (var change in changes.EnumerateArray()) + { + var node = change.GetProperty("node"); + if (node.TryGetProperty("node-type", out var nodeType) && nodeType.GetString() == "action") + { + nodes.Add(new ServerNode( + node.GetProperty("uid").GetString()!, + node.GetProperty("display-name").GetString()!, + node.TryGetProperty("execution-state", out var state) ? state.GetString()! : "")); + } + } + } + + return; + } + + // A null "changes" is how the platform says the request is over. + if (parameters.TryGetProperty("runId", out var runId)) + { + lock (pendingRuns) + { + if (pendingRuns.TryGetValue(runId.GetString()!, out var run)) + run.TrySetResult(true); + } + } + } + + public void Dispose() + { + try + { + if (!process.HasExited) + process.Kill(entireProcessTree: true); + } + catch + { + // The process is gone, which is what was wanted. + } + + client.Dispose(); + } + + internal sealed record ServerNode(string Uid, string DisplayName, string ExecutionState); + } +} +#endif From 12a9e36384fb82a3f02345110d6d569b7239cec0 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Tue, 8 Sep 2026 09:54:50 +0100 Subject: [PATCH 089/110] Add AsyncDisposableProbe and improve Tracked disposal logic Added AsyncDisposableProbe for benchmarking async-disposables and tracking their disposal, writing counts on process exit. Updated Tracked in DisposableProbe.cs to use a private number field, added isDisposed flag, throw on access after disposal, updated ToString(), and set isDisposed in Dispose(). --- .../AsyncDisposableProbe.cs | 69 +++++++++++++++++++ .../DisposableProbe.cs | 20 ++++-- 2 files changed, 85 insertions(+), 4 deletions(-) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/AsyncDisposableProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/AsyncDisposableProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/AsyncDisposableProbe.cs new file mode 100644 index 0000000000..ddb8bf03e4 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/AsyncDisposableProbe.cs @@ -0,0 +1,69 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// A benchmark whose parameter values come from an async source and are only . They + /// are disposed by exactly the same rules as a synchronously disposable one, and casting a value to + /// would drop them on the floor instead - which the finalizer of a value holding a lock + /// then turns into the hang of dotnet/BenchmarkDotNet#1383. + /// + [Config(typeof(FastConfig))] + public class AsyncDisposableProbe + { + /// + /// The name of the file the counts are written to, next to the probe application. + /// + public const string ReportFileName = "async-disposable-probe.txt"; + + // Created once, so that re-reading the source cannot change the count. + private static readonly AsyncTracked[] Instances = [new AsyncTracked(1), new AsyncTracked(2)]; + + public static async IAsyncEnumerable GetValues() + { + await Task.Yield(); + + foreach (var instance in Instances) + yield return instance; + } + + [ParamsSource(nameof(GetValues))] + public AsyncTracked? Value { get; set; } + + [Benchmark] + public int Identity() => Value!.Number; + + /// + /// Deliberately not : the whole point of the probe. + /// + public class AsyncTracked : IAsyncDisposable + { + private static int disposed; + + static AsyncTracked() => + AppDomain.CurrentDomain.ProcessExit += (_, _) => File.WriteAllText( + Path.Combine(AppContext.BaseDirectory, ReportFileName), + $"created={Instances.Length} disposed={Volatile.Read(ref disposed)}"); + + public AsyncTracked(int number) => Number = number; + + public int Number { get; } + + public ValueTask DisposeAsync() + { + Interlocked.Increment(ref disposed); + return default; + } + + public override string ToString() => $"async-{Number}"; + } + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DisposableProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DisposableProbe.cs index e13db876d9..7749a1c0c3 100644 --- a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DisposableProbe.cs +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/DisposableProbe.cs @@ -34,18 +34,30 @@ public class Tracked : IDisposable { private static int disposed; + private readonly int number; + + private bool isDisposed; + static Tracked() => AppDomain.CurrentDomain.ProcessExit += (_, _) => File.WriteAllText( Path.Combine(AppContext.BaseDirectory, ReportFileName), $"created={Instances.Length} disposed={Volatile.Read(ref disposed)}"); - public Tracked(int number) => Number = number; + public Tracked(int number) => this.number = number; - public int Number { get; } + /// + /// Reading this after the value was disposed is the failure a run that executes against the values a + /// discovery already disposed would otherwise get away with, so it is made loud rather than counted. + /// + public int Number => isDisposed ? throw new ObjectDisposedException(ToString()) : number; - public void Dispose() => Interlocked.Increment(ref disposed); + public void Dispose() + { + isDisposed = true; + Interlocked.Increment(ref disposed); + } - public override string ToString() => $"tracked-{Number}"; + public override string ToString() => $"tracked-{number}"; } private class FastConfig : ManualConfig From c9e63592404718d183179bf564d16571ee381751 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Tue, 8 Sep 2026 09:55:03 +0100 Subject: [PATCH 090/110] Improve error handling for unreadable configs in tests Added tests to ensure types with unreadable [Config] attributes are properly reported and do not silently fail benchmark discovery. Updated GenericBuilderTests and TypeFilterTests with assertions and new test classes to simulate config instantiation failures. --- .../GenericBuilderTests.cs | 14 +++++++++ .../BenchmarkDotNet.Tests/TypeFilterTests.cs | 30 +++++++++++++++++++ 2 files changed, 44 insertions(+) diff --git a/tests/BenchmarkDotNet.Tests/GenericBuilderTests.cs b/tests/BenchmarkDotNet.Tests/GenericBuilderTests.cs index 5f5bca2166..7d1ac5c363 100644 --- a/tests/BenchmarkDotNet.Tests/GenericBuilderTests.cs +++ b/tests/BenchmarkDotNet.Tests/GenericBuilderTests.cs @@ -147,6 +147,20 @@ public void TestTypeWithUnreadableAttributesIsReportedAsAFailure() var failure = Assert.Single(built); Assert.False(failure.IsSuccess); Assert.Contains(nameof(BenchmarkWithAbstractConfig), failure.Error); + + // Told apart from a [GenericTypeArguments] that did not fit, because only this kind has to be reported + // by whoever drops it: GenericBenchmarksValidator needs a surviving benchmark before it ever runs. + Assert.True(failure.IsUnreadable); + } + + [Fact] + public void TestGenericTypeThatFailedToBuildIsNotReportedAsUnreadable() + { + var built = GenericBenchmarksBuilder.BuildGenericsIfNeeded(typeof(GenericBenchmarkWithConstraintsWrongArgs<,>)).ToArray(); + + var failure = Assert.Single(built, candidate => !candidate.IsSuccess); + Assert.False(failure.IsUnreadable); + Assert.Contains("wrong type argument", failure.Error); } [Config(typeof(DebugConfig))] // abstract, so ConfigAttribute's constructor throws diff --git a/tests/BenchmarkDotNet.Tests/TypeFilterTests.cs b/tests/BenchmarkDotNet.Tests/TypeFilterTests.cs index 1f3e6e7918..ce924c2136 100644 --- a/tests/BenchmarkDotNet.Tests/TypeFilterTests.cs +++ b/tests/BenchmarkDotNet.Tests/TypeFilterTests.cs @@ -1,4 +1,5 @@ using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; using BenchmarkDotNet.ConsoleArguments; using BenchmarkDotNet.Extensions; using BenchmarkDotNet.Loggers; @@ -207,6 +208,23 @@ public void GenericTypesCanBeFilteredByDisplayName() Assert.Contains("SomeGeneric.Create", benchmarks); } + [Fact] + public void ReportsATypeWhoseAttributesCannotBeRead() + { + // The [Config] of this type throws while reflection constructs it, so the type is dropped rather than + // allowed to abort the whole run - but it has to be said out loud, because GenericBenchmarksValidator + // never gets to report it when nothing of the assembly survives, and "No benchmarks were found" on its + // own sends the user looking in the wrong place. + var logger = new AccumulationLogger(); + + var benchmarks = Filter([typeof(ClassWithUnreadableConfig), typeof(ClassA)], ["--filter", "*"], logger); + + Assert.Equal(2, benchmarks.Count); + Assert.Contains("ClassA.Method1", benchmarks); + Assert.DoesNotContain("ClassWithUnreadableConfig.Method1", benchmarks); + Assert.Contains(nameof(ClassWithUnreadableConfig), logger.GetLog(), StringComparison.Ordinal); + } + private HashSet Filter(Type[] types, string[] args, ILogger? logger = null) { var nonNullLogger = logger ?? new OutputLogger(Output); @@ -251,6 +269,18 @@ public void Method2() { } public void Method3() { } } + [Config(typeof(AbstractConfig))] + public class ClassWithUnreadableConfig + { + [Benchmark] + public void Method1() { } + } + + // ConfigAttribute instantiates the type it is given, and an abstract one cannot be instantiated. + public abstract class AbstractConfig : ManualConfig + { + } + public class ClassC { // None of these methods are actually Benchmarks!! From 55bfa5a9767f8032754419c80378f23cfc30c8e5 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Tue, 8 Sep 2026 09:55:18 +0100 Subject: [PATCH 091/110] Improve error handling in TypeFilter AddRunnable logic Refactored TypeFilter.cs to introduce AddRunnable, which adds runnable benchmark types and logs errors for types with unreadable attributes. Replaced LINQ-based addition with explicit error reporting, enhancing clarity on excluded types and improving diagnostics during benchmark discovery. --- src/BenchmarkDotNet/Running/TypeFilter.cs | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/src/BenchmarkDotNet/Running/TypeFilter.cs b/src/BenchmarkDotNet/Running/TypeFilter.cs index e509a2336d..76c5c8a090 100644 --- a/src/BenchmarkDotNet/Running/TypeFilter.cs +++ b/src/BenchmarkDotNet/Running/TypeFilter.cs @@ -39,11 +39,28 @@ public static (bool allTypesValid, IReadOnlyList runnable) GetTypesWithRun return (false, Array.Empty()); } + // A type whose attributes cannot be read at all - [Config(typeof(SomeAbstractConfig))], say, where the + // attribute throws while reflection constructs it - is dropped rather than allowed to abort the run, but + // that has to be said out loud: GenericBenchmarksValidator never gets to report it, because it only runs + // once at least one benchmark of the assembly survived, and "No benchmarks were found" on its own sends + // the user looking in the wrong place. A type that failed on its [GenericTypeArguments] is left to the + // validator, which is where that has always been reported. + void AddRunnable(IEnumerable built) + { + foreach (var candidate in built) + { + if (candidate.IsSuccess) + validRunnableTypes.Add(candidate.Type); + else if (candidate.IsUnreadable) + logger.WriteLineError(candidate.Error!); + } + } + foreach (var type in types) { if (type.ContainsRunnableBenchmarks()) { - validRunnableTypes.AddRange(GenericBenchmarksBuilder.BuildGenericsIfNeeded(type).Where(built => built.IsSuccess).Select(built => built.Type)); + AddRunnable(GenericBenchmarksBuilder.BuildGenericsIfNeeded(type)); } else { @@ -55,7 +72,7 @@ public static (bool allTypesValid, IReadOnlyList runnable) GetTypesWithRun foreach (var assembly in assemblies) { - validRunnableTypes.AddRange(GenericBenchmarksBuilder.GetRunnableBenchmarks(assembly.GetRunnableBenchmarks())); + AddRunnable(GenericBenchmarksBuilder.BuildRunnableBenchmarks(assembly.GetRunnableBenchmarks())); } return (true, validRunnableTypes); From 954717fb55b6e9e2a89bd078616dfff65661ab72 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Tue, 8 Sep 2026 09:55:42 +0100 Subject: [PATCH 092/110] Exclude unreadable builds from validation errors Previously, all unsuccessful builds were reported as validation errors, including those that were unreadable and already handled by the TypeFilter. The updated LINQ query now excludes builds that are both unsuccessful and unreadable, preventing duplicate error reporting for unreadable types. --- src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs b/src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs index 70a85e0ec0..4513dfa0f5 100644 --- a/src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs +++ b/src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs @@ -16,7 +16,8 @@ public IAsyncEnumerable ValidateAsync(ValidationParameters vali .Distinct() .SelectMany(assembly => assembly.GetRunnableBenchmarks()) .SelectMany(GenericBenchmarksBuilder.BuildGenericsIfNeeded) - .Where(built => !built.IsSuccess) + // An unreadable type is reported by TypeFilter, which sees it even when nothing else survives. + .Where(built => !built.IsSuccess && !built.IsUnreadable) .Select(built => new ValidationError(false, built.Error!)) .ToAsyncEnumerable(); } From 9a69f40974b33c73a2fa72c47cc16855cd3d2c20 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Tue, 8 Sep 2026 09:56:01 +0100 Subject: [PATCH 093/110] Add ParameterValueLifetime for safe parameter disposal Introduce ParameterValueLifetime to manage benchmark parameter lifetimes, preventing premature disposal in server mode. Update BenchmarkTestFramework and related classes to use this for tracking and disposing parameter values. Improve benchmark state handling in BenchmarkEventProcessor and avoid publishing empty source file locations. Update service registration and filtering logic for safety and correctness. --- .../BenchmarkEventProcessor.cs | 14 ++- .../TestingPlatform/BenchmarkTestFramework.cs | 33 ++++-- .../TestingPlatform/BenchmarkTestNode.cs | 5 +- .../TestingPlatform/ParameterValueLifetime.cs | 112 ++++++++++++++++++ .../TestApplicationBuilderExtensions.cs | 8 +- 5 files changed, 157 insertions(+), 15 deletions(-) create mode 100644 src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs index 70b0839469..dd9a924491 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs @@ -124,11 +124,17 @@ public void PublishOutstandingResults() continue; var pending = GetOrCreatePendingResult(node); - var errorMessage = pending.GetErrorMessage(); - TestNodeStateProperty state = errorMessage != null - ? new FailedTestNodeStateProperty(errorMessage) - : SkippedTestNodeStateProperty.CachedInstance; + // A benchmark that reported a start and never an end did not finish: the run was torn down under it + // by something this processor was never told about, such as BenchmarkRunnerClean throwing. Only a + // benchmark that never started at all was really never run. + TestNodeStateProperty state = (pending.GetErrorMessage(), pending.StartTime) switch + { + (string errorMessage, _) => new FailedTestNodeStateProperty(errorMessage), + (null, not null) => new FailedTestNodeStateProperty( + "The benchmark started but never reported a result, so the run did not complete."), + (null, null) => SkippedTestNodeStateProperty.CachedInstance + }; publish(node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance)); PublishResult(node, pending, state); diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs index 718ca2f4b2..58302d83f7 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs @@ -22,12 +22,18 @@ internal sealed class BenchmarkTestFramework : ITestFramework, IDataProducer, IO private readonly BenchmarkDotNetExtension extension = new(); private readonly IServiceProvider serviceProvider; private readonly Assembly assembly; + private readonly ParameterValueLifetime parameterValues; - public BenchmarkTestFramework(ITestFrameworkCapabilities capabilities, IServiceProvider serviceProvider, Assembly assembly) + public BenchmarkTestFramework( + ITestFrameworkCapabilities capabilities, + IServiceProvider serviceProvider, + Assembly assembly, + ParameterValueLifetime parameterValues) { Capabilities = capabilities; this.serviceProvider = serviceProvider; this.assembly = assembly; + this.parameterValues = parameterValues; } /// @@ -103,8 +109,10 @@ private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRe } finally { - // Discovery runs nothing, so every value the enumeration created is this method's to dispose. - ParameterValueDisposer.DisposeUnused(enumeration.All.SelectMany(runInfo => runInfo.BenchmarksCases), []); + // Discovery runs nothing, so every value the enumeration created is unused - but not necessarily for + // good: under server mode a run request follows in this same process, so the disposal waits until the + // application is done rather than happening here. + parameterValues.Track(enumeration.All.SelectMany(runInfo => runInfo.BenchmarksCases), []); } } @@ -124,8 +132,9 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte } // A benchmark that was filtered out or that collided is never handed to BenchmarkDotNet, so nothing else - // would dispose the values the enumeration created for it. - ParameterValueDisposer.DisposeUnused( + // would dispose the values the enumeration created for it. As in DiscoverAsync, a later request of the + // same application may still enumerate and run them, so the disposal waits for the end of the run. + parameterValues.Track( enumeration.All.SelectMany(runInfo => runInfo.BenchmarksCases), runnable.Select(match => match.Node.BenchmarkCase)); @@ -294,7 +303,7 @@ private Enumeration GetMatchingBenchmarks(ITestExecutionFilter filter) { var matches = new List>(); var matchesByUid = new Dictionary>(StringComparer.Ordinal); - var runInfos = BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly); + var runInfos = BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly, parameterValues.TrackHidden); foreach (var runInfo in runInfos) { @@ -327,9 +336,15 @@ private Enumeration GetMatchingBenchmarks(ITestExecutionFilter filter) { TestNodeUidListFilter uidListFilter => uidListFilter.TestNodeUids.Any(uid => uid.Value == node.Uid), TreeNodeFilter treeNodeFilter => treeNodeFilter.MatchesFilter(node.Path, node.GetFilterableProperties()), - - // NopFilter, and anything the platform adds later, means "everything". - _ => true + NopFilter => true, + + // A filter the platform adds later has to be implemented here before it can be honoured. Falling back to + // "everything" would run the whole assembly instead of the subset that was asked for, which is a wrong + // answer rather than a visible failure. + _ => throw new NotSupportedException( + $"BenchmarkDotNet.TestAdapter does not support the '{filter.GetType().FullName}' test execution " + + "filter, and will not run every benchmark in its place. Please report this at " + + "https://github.com/dotnet/BenchmarkDotNet/issues.") }; #pragma warning restore TPEXP diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs index da241cc1d1..b65f820cf2 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestNode.cs @@ -96,7 +96,10 @@ public static BenchmarkTestNode Create(BenchmarkCase benchmarkCase, bool include benchmarkMethod.ReturnType.FullName ?? benchmarkMethod.ReturnType.Name), }; - if (benchmarkAttribute?.SourceCodeFile != null) + // SourceCodeFile is a non-nullable string that a [CallerFilePath] fills in, so it is empty rather than + // null on an attribute built without caller information - by an analyzer, or by hand. Publishing a + // location of "" at line 0 would send an IDE to a file that does not exist. + if (benchmarkAttribute != null && !string.IsNullOrEmpty(benchmarkAttribute.SourceCodeFile)) { // BenchmarkAttribute captures the line of the attribute itself, and the platform expects 0-based lines. var line = Math.Max(0, benchmarkAttribute.SourceCodeLineNumber - 1); diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs new file mode 100644 index 0000000000..5106650092 --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs @@ -0,0 +1,112 @@ +using BenchmarkDotNet.Helpers; +using BenchmarkDotNet.Parameters; +using BenchmarkDotNet.Running; +using Microsoft.Testing.Platform.Extensions.TestHost; + +namespace BenchmarkDotNet.TestAdapter.TestingPlatform +{ + /// + /// Holds the parameter values that the requests of a test application enumerated but never ran, and disposes them + /// once the application is done. + /// + /// + /// Disposing them at the end of the request that enumerated them is wrong under server mode, which is how Visual + /// Studio and the VS Code Test Explorer drive the platform: a discovery request and the run requests that follow + /// it are served by the same process, every request enumerates the assembly again, and a [ParamsSource] backed by + /// a cached collection - a static field, or a property over a readonly array - hands back the very same objects. + /// Discovery would then dispose the values the run is about to execute against. The platform builds a + /// per request but this extension only once, so it is what can hold the + /// values until nothing can ask for them again. + /// + internal sealed class ParameterValueLifetime : ITestHostApplicationLifetime + { + private readonly BenchmarkDotNetExtension extension = new(); + + // Requests are served one at a time, but the platform is free to change that, and getting this wrong would + // leak or double dispose rather than fail visibly. + private readonly object gate = new(); + + // Everything the requests enumerated, keyed by the value so that a ParameterInstance which BenchmarkConverter + // handed to several jobs is only disposed once, and everything that was handed to BenchmarkDotNet, which + // disposes what it was given itself. Retention is remembered for the whole application: a value that one + // request ran is disposed by BenchmarkDotNet, however many later requests enumerate it again. + private readonly Dictionary enumerated = new(ParameterValueDisposer.ByReference); + private readonly HashSet retained = new(ParameterValueDisposer.ByReference); + + /// + public string Uid => extension.Uid + ".ParameterValueLifetime"; + + /// + public string Version => extension.Version; + + /// + public string DisplayName => extension.DisplayName; + + /// + public string Description => extension.Description; + + /// + public Task IsEnabledAsync() => extension.IsEnabledAsync(); + + /// + /// Records what a request enumerated and what of it was handed to BenchmarkDotNet, without disposing + /// anything yet. + /// + /// Everything the request enumerated. + /// The benchmarks that were handed to BenchmarkDotNet, if any. + public void Track(IEnumerable enumeratedCases, IEnumerable retainedCases) + { + lock (gate) + { + foreach (var parameter in ParameterValueDisposer.GetDisposableParameters(enumeratedCases)) + enumerated[parameter.Value!] = parameter; + + foreach (var parameter in ParameterValueDisposer.GetDisposableParameters(retainedCases)) + retained.Add(parameter.Value!); + } + } + + /// + /// Records the values of the benchmarks that the enumeration hid, which no request will ever be handed. + /// + /// + /// Being kept by the enumeration is not the same as being handed to BenchmarkDotNet, so the kept values are + /// only excluded from what is collected here: whether they are ever run is for to say. + /// + /// Everything the assembly declares. + /// The benchmarks the enumeration returned. + public void TrackHidden(IEnumerable enumeratedCases, IEnumerable keptCases) + { + lock (gate) + { + var kept = new HashSet( + ParameterValueDisposer.GetDisposableParameters(keptCases).Select(parameter => parameter.Value!), + ParameterValueDisposer.ByReference); + + foreach (var parameter in ParameterValueDisposer.GetDisposableParameters(enumeratedCases)) + { + if (!kept.Contains(parameter.Value!)) + enumerated[parameter.Value!] = parameter; + } + } + } + + /// + public Task BeforeRunAsync(CancellationToken cancellationToken) => Task.CompletedTask; + + /// + public async Task AfterRunAsync(int exitCode, CancellationToken cancellationToken) + { + List unused; + + lock (gate) + { + unused = enumerated.Where(pair => !retained.Contains(pair.Key)).Select(pair => pair.Value).ToList(); + enumerated.Clear(); + retained.Clear(); + } + + await unused.DisposeAllAsync().ConfigureAwait(false); + } + } +} diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs index e1eecf053a..ea2f997fa3 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/TestApplicationBuilderExtensions.cs @@ -35,9 +35,15 @@ public static ITestApplicationBuilder AddBenchmarkDotNet(this ITestApplicationBu if (assembly == null) throw new ArgumentNullException(nameof(assembly)); + // The platform builds a test framework per request but this once, which is what makes it able to hold + // the parameter values of one request until the whole application is done. See ParameterValueLifetime. + var parameterValues = new ParameterValueLifetime(); + builder.RegisterTestFramework( _ => new TestFrameworkCapabilities(), - (capabilities, serviceProvider) => new BenchmarkTestFramework(capabilities, serviceProvider, assembly)); + (capabilities, serviceProvider) => new BenchmarkTestFramework(capabilities, serviceProvider, assembly, parameterValues)); + + builder.TestHost.AddTestHostApplicationLifetime(_ => parameterValues); // Opts into the tree node filter, which is what backs `--treenode-filter "/*/*/MyBenchmarks/*"`. #pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. From c3addfb61abc1edb62538b370e119bf93b1d0c90 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Tue, 8 Sep 2026 09:56:12 +0100 Subject: [PATCH 094/110] Add async disposal for benchmark parameters Introduce DisposeUnusedAsync in ParameterValueDisposer to support asynchronous disposal of unused benchmark parameter values (IDisposable and IAsyncDisposable). Refactor disposal logic to operate on ParameterInstance objects for correct semantics. Overload GetBenchmarksFromAssembly to accept a custom disposal action, enhancing flexibility. Maintain reference-based equality for parameter value tracking. Synchronous DisposeUnused now wraps the async method using BenchmarkSynchronizationContext. --- .../BenchmarkEnumerator.cs | 20 ++++++- .../ParameterValueDisposer.cs | 53 ++++++++++++------- 2 files changed, 53 insertions(+), 20 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs b/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs index 2d492f0769..bb5604e8bf 100644 --- a/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs +++ b/src/BenchmarkDotNet.TestAdapter/BenchmarkEnumerator.cs @@ -57,6 +57,22 @@ public static BenchmarkRunInfo[] GetBenchmarksFromAssemblyPath(string assemblyPa /// The assembly of the benchmark project. /// The benchmarks inside the assembly. public static BenchmarkRunInfo[] GetBenchmarksFromAssembly(Assembly assembly) + => GetBenchmarksFromAssembly(assembly, ParameterValueDisposer.DisposeUnused); + + /// + /// Returns all the BenchmarkRunInfo objects from an already loaded assembly. + /// + /// The assembly of the benchmark project. + /// + /// What to do with the parameter values of the benchmarks that are hidden here, given everything the assembly + /// declares and the benchmarks that are kept. Disposing them right away is only right when nothing will + /// enumerate the assembly again: a host that serves several requests from one process hides the same + /// benchmarks every time, and a cached source hands back the same values, so it takes the disposal over. + /// + /// The benchmarks inside the assembly. + internal static BenchmarkRunInfo[] GetBenchmarksFromAssembly( + Assembly assembly, + Action, IEnumerable> disposeHidden) { var all = GenericBenchmarksBuilder.GetRunnableBenchmarks(assembly.GetRunnableBenchmarks()) .Select(type => BenchmarkConverter.TypeToBenchmarks(type)) @@ -77,8 +93,8 @@ public static BenchmarkRunInfo[] GetBenchmarksFromAssembly(Assembly assembly) .ToArray(); // BenchmarkConverter has already constructed every parameter value by now, and a case hidden here is never - // handed to BenchmarkDotNet by either adapter, so nothing downstream can dispose what is dropped. - ParameterValueDisposer.DisposeUnused( + // handed to BenchmarkDotNet by either adapter, so nothing downstream will ever dispose what is dropped. + disposeHidden( all.SelectMany(runInfo => runInfo.BenchmarksCases), runnable.SelectMany(runInfo => runInfo.BenchmarksCases)); diff --git a/src/BenchmarkDotNet.TestAdapter/ParameterValueDisposer.cs b/src/BenchmarkDotNet.TestAdapter/ParameterValueDisposer.cs index c07047770c..899f891e13 100644 --- a/src/BenchmarkDotNet.TestAdapter/ParameterValueDisposer.cs +++ b/src/BenchmarkDotNet.TestAdapter/ParameterValueDisposer.cs @@ -1,3 +1,6 @@ +using BenchmarkDotNet.Engines; +using BenchmarkDotNet.Helpers; +using BenchmarkDotNet.Parameters; using BenchmarkDotNet.Running; using System.Runtime.CompilerServices; @@ -14,46 +17,60 @@ namespace BenchmarkDotNet.TestAdapter /// internal static class ParameterValueDisposer { + /// + internal static void DisposeUnused(IEnumerable enumerated, IEnumerable retained) + { + using var context = BenchmarkSynchronizationContext.CreateAndSetCurrent(); + context.ExecuteUntilComplete(DisposeUnusedAsync(enumerated, retained)); + } + /// /// Disposes every value that the enumerated benchmarks own and the retained ones do not. /// /// /// The values are matched by reference instead of being disposed case by case, because BenchmarkConverter /// gives the same ParameterInstance to every job and every argument set of a benchmark: disposing a dropped - /// case wholesale would take down values that a benchmark which is about to run still owns. + /// case wholesale would take down values that a benchmark which is about to run still owns. Disposal goes + /// through ParameterInstance so that a value which is only IAsyncDisposable is disposed as well. /// /// Everything the assembly declares. /// The benchmarks that are kept, if any. - internal static void DisposeUnused(IEnumerable enumerated, IEnumerable retained) + internal static ValueTask DisposeUnusedAsync(IEnumerable enumerated, IEnumerable retained) { - var unused = new HashSet(ReferenceComparer.Instance); + var unused = new Dictionary(ReferenceComparer.Instance); - foreach (var value in GetDisposableParameterValues(enumerated)) - unused.Add(value); + foreach (var parameter in GetDisposableParameters(enumerated)) + unused[parameter.Value!] = parameter; - foreach (var value in GetDisposableParameterValues(retained)) - unused.Remove(value); + foreach (var parameter in GetDisposableParameters(retained)) + unused.Remove(parameter.Value!); - foreach (var value in unused) - value.Dispose(); + return unused.Values.DisposeAllAsync(); } - private static IEnumerable GetDisposableParameterValues(IEnumerable benchmarkCases) - => benchmarkCases - .SelectMany(benchmarkCase => benchmarkCase.Parameters.Items) - .Select(parameter => parameter.Value) - .OfType(); + /// + /// Compares values by reference, so that a parameter value which overrides Equals is still disposed once per + /// instance. + /// + internal static IEqualityComparer ByReference => ReferenceComparer.Instance; /// - /// Compares by reference, so that a parameter value which overrides Equals is still disposed once per instance. + /// Gets the parameters of the given benchmarks whose value needs disposing. /// - private sealed class ReferenceComparer : IEqualityComparer + /// The benchmarks to read the parameters of. + /// The parameters holding a disposable value. + internal static IEnumerable GetDisposableParameters(IEnumerable benchmarkCases) + => benchmarkCases + .SelectMany(benchmarkCase => benchmarkCase.Parameters.Items) + .Where(parameter => parameter.Value is IDisposable or IAsyncDisposable); + + private sealed class ReferenceComparer : IEqualityComparer { public static readonly ReferenceComparer Instance = new ReferenceComparer(); - public bool Equals(IDisposable? x, IDisposable? y) => ReferenceEquals(x, y); + public new bool Equals(object? x, object? y) => ReferenceEquals(x, y); - public int GetHashCode(IDisposable obj) => RuntimeHelpers.GetHashCode(obj); + public int GetHashCode(object obj) => RuntimeHelpers.GetHashCode(obj); } } } From bcd4e204f7a97dfcd2b62714e7101e818ce6b4b2 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Wed, 9 Sep 2026 08:27:04 +0100 Subject: [PATCH 095/110] Clarify IsUnreadable XML doc comment Updated the XML documentation for the IsUnreadable property to clarify its meaning. The new comment explains that types rejected due to incompatible [GenericTypeArguments] still declare benchmarks, so "no benchmarks were found" should not be used for them. It also specifies responsibility for explaining dropped types. No code logic was changed. --- src/BenchmarkDotNet/Helpers/GenericBenchmarkType.cs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/BenchmarkDotNet/Helpers/GenericBenchmarkType.cs b/src/BenchmarkDotNet/Helpers/GenericBenchmarkType.cs index 46677d02f8..a597bf70d7 100644 --- a/src/BenchmarkDotNet/Helpers/GenericBenchmarkType.cs +++ b/src/BenchmarkDotNet/Helpers/GenericBenchmarkType.cs @@ -25,8 +25,9 @@ private GenericBenchmarkType(Type type, string? error, bool isUnreadable) /// /// Gets whether the type was rejected before any of its benchmarks could be read, rather than because one set - /// of [GenericTypeArguments] did not fit it. Nothing downstream reports such a type - GenericBenchmarksValidator - /// only runs once at least one benchmark of the assembly survived - so whoever drops it has to say so. + /// of [GenericTypeArguments] did not fit it. Such a type still declares benchmarks, which is what "no + /// benchmarks were found" must not be said about, and whoever drops it when nothing else survives is the + /// only one left to say why. /// internal bool IsUnreadable { get; } From 0b2f49d01eeccfb9db3e8cc859cb3c4a3609da5a Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Wed, 9 Sep 2026 08:29:54 +0100 Subject: [PATCH 096/110] Refactor benchmark type collection logic in GetTypesWithRunnableBenchmarks Refactored to build runnable benchmark types once per assembly and reuse the result, improving efficiency and clarity. Updated logic to check for runnable benchmarks using assemblyTypes.Any with IsSuccess or IsUnreadable. Enhanced comments to clarify reporting of unreadable types and the validator's role. --- src/BenchmarkDotNet/Running/TypeFilter.cs | 24 ++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/src/BenchmarkDotNet/Running/TypeFilter.cs b/src/BenchmarkDotNet/Running/TypeFilter.cs index 76c5c8a090..eed7e591df 100644 --- a/src/BenchmarkDotNet/Running/TypeFilter.cs +++ b/src/BenchmarkDotNet/Running/TypeFilter.cs @@ -13,8 +13,16 @@ public static (bool allTypesValid, IReadOnlyList runnable) GetTypesWithRun { var validRunnableTypes = new List(); + // Built once: the guard below and the list at the end both need it. + var assemblyTypes = assemblies + .SelectMany(assembly => GenericBenchmarksBuilder.BuildRunnableBenchmarks(assembly.GetRunnableBenchmarks())) + .ToArray(); + bool hasRunnableTypeBenchmarks = types.Any(type => type.ContainsRunnableBenchmarks()); - bool hasRunnableAssemblyBenchmarks = assemblies.Any(assembly => GenericBenchmarksBuilder.GetRunnableBenchmarks(assembly.GetRunnableBenchmarks()).Length > 0); + + // A type whose attributes cannot be read counts as declaring benchmarks here - it does, and telling the + // user that no [Benchmark] was found would be wrong. It is reported below, in place of being run. + bool hasRunnableAssemblyBenchmarks = assemblyTypes.Any(built => built.IsSuccess || built.IsUnreadable); if (!hasRunnableTypeBenchmarks && !hasRunnableAssemblyBenchmarks) { @@ -41,10 +49,11 @@ public static (bool allTypesValid, IReadOnlyList runnable) GetTypesWithRun // A type whose attributes cannot be read at all - [Config(typeof(SomeAbstractConfig))], say, where the // attribute throws while reflection constructs it - is dropped rather than allowed to abort the run, but - // that has to be said out loud: GenericBenchmarksValidator never gets to report it, because it only runs - // once at least one benchmark of the assembly survived, and "No benchmarks were found" on its own sends - // the user looking in the wrong place. A type that failed on its [GenericTypeArguments] is left to the - // validator, which is where that has always been reported. + // that has to be said out loud. GenericBenchmarksValidator says so too, and on the paths that never come + // through here it is the only one that can, but it only runs once at least one benchmark survived: when + // the unreadable type was the only one, this is the one place left to say why nothing was found. Saying + // it twice on the way to a run is the lesser problem. A type that failed on its [GenericTypeArguments] + // is left to the validator alone, which is where that has always been reported. void AddRunnable(IEnumerable built) { foreach (var candidate in built) @@ -70,10 +79,7 @@ void AddRunnable(IEnumerable built) } } - foreach (var assembly in assemblies) - { - AddRunnable(GenericBenchmarksBuilder.BuildRunnableBenchmarks(assembly.GetRunnableBenchmarks())); - } + AddRunnable(assemblyTypes); return (true, validRunnableTypes); } From 1e681a2f46e119aeed4ccaf9b5f110d25a2f949f Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Wed, 9 Sep 2026 08:32:52 +0100 Subject: [PATCH 097/110] Improve parameter value lifetime management in BDN runs Refactor parameter value disposal logic to track usage across requests, preventing premature disposal and memory leaks, especially in server mode scenarios. BenchmarkEventProcessor now exposes ParameterValuesDisposed. Update filter matching to handle unknown types gracefully and warn users. Add WarnAboutUnrecognisedFilterAsync to BenchmarkTestFramework. Update comments and docs to clarify new strategy. --- .../BenchmarkEventProcessor.cs | 16 ++ .../TestingPlatform/BenchmarkTestFramework.cs | 160 ++++++++++++------ .../TestingPlatform/ParameterValueLifetime.cs | 114 +++++++++---- 3 files changed, 209 insertions(+), 81 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs index dd9a924491..cfbba9096b 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs @@ -30,12 +30,28 @@ internal sealed class BenchmarkEventProcessor : EventProcessor // first benchmark runs, and the benchmarks themselves run one after another. private readonly object buildCompleteGate = new(); + // Written on BenchmarkDotNet's thread, read on the request's once the run is over. + private volatile bool parameterValuesDisposed; + public BenchmarkEventProcessor(IReadOnlyDictionary nodes, Action publish) { this.nodes = nodes; this.publish = publish; } + /// + /// Gets whether BenchmarkDotNet disposed the parameter values of the benchmarks it was handed. + /// + /// + /// BenchmarkRunnerClean.Run disposes them in the finally of its run stage, which is where OnEndRunStage is + /// raised from - after the disposal, and whether the run completed, threw or was cancelled. It never gets + /// there when a critical validation error makes it return before the run stage, and nothing disposes the + /// values then, so whoever handed them over has to. + /// + public bool ParameterValuesDisposed => parameterValuesDisposed; + + public override void OnEndRunStage() => parameterValuesDisposed = true; + public override void OnValidationError(ValidationError validationError) { // If the error is not linked to a benchmark case, then set the error on all benchmarks. diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs index 58302d83f7..158c454ecd 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs @@ -5,6 +5,7 @@ using Microsoft.Testing.Platform.Extensions.Messages; using Microsoft.Testing.Platform.Extensions.OutputDevice; using Microsoft.Testing.Platform.Extensions.TestFramework; +using Microsoft.Testing.Platform.OutputDevice; using Microsoft.Testing.Platform.Requests; using Microsoft.Testing.Platform.Services; using Microsoft.Testing.Platform.TestHost; @@ -90,10 +91,13 @@ public async Task ExecuteRequestAsync(ExecuteRequestContext context) private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRequestContext context) { - var enumeration = GetMatchingBenchmarks(request.Filter); - + // The enumeration sits inside the try: it records the values it creates as it goes, and completing the + // request is what hands them over, so nothing that throws between the two can lose them. try { + var enumeration = GetMatchingBenchmarks(request.Filter); + await WarnAboutUnrecognisedFilterAsync(enumeration, context.CancellationToken).ConfigureAwait(false); + foreach (var benchmarks in enumeration.Matches) { context.CancellationToken.ThrowIfCancellationRequested(); @@ -109,10 +113,9 @@ private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRe } finally { - // Discovery runs nothing, so every value the enumeration created is unused - but not necessarily for - // good: under server mode a run request follows in this same process, so the disposal waits until the - // application is done rather than happening here. - parameterValues.Track(enumeration.All.SelectMany(runInfo => runInfo.BenchmarksCases), []); + // Discovery runs nothing, so BenchmarkDotNet disposed nothing. The values are not disposed here + // either: under server mode a run request follows in this very process, see ParameterValueLifetime. + await parameterValues.CompleteRequestAsync([]).ConfigureAwait(false); } } @@ -121,49 +124,77 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte var sessionUid = request.Session.SessionUid; var cancellationToken = context.CancellationToken; + // Declared ahead of the try, because the finally needs both to say what BenchmarkDotNet disposed - and the + // event processor is created out here too, so that a run which threw after BenchmarkDotNet had already + // disposed the values, as one does when publishing fails, still has that on record. var runnable = new List(); - var enumeration = GetMatchingBenchmarks(request.Filter); - foreach (var benchmarks in enumeration.Matches) - { - if (benchmarks.Count == 1) - runnable.Add(benchmarks[0]); - else - await PublishCollisionAsync(context, sessionUid, benchmarks).ConfigureAwait(false); - } - - // A benchmark that was filtered out or that collided is never handed to BenchmarkDotNet, so nothing else - // would dispose the values the enumeration created for it. As in DiscoverAsync, a later request of the - // same application may still enumerate and run them, so the disposal waits for the end of the run. - parameterValues.Track( - enumeration.All.SelectMany(runInfo => runInfo.BenchmarksCases), - runnable.Select(match => match.Node.BenchmarkCase)); - - if (runnable.Count == 0) - return; - - var nodes = runnable.ToDictionary(match => match.Node.Uid, match => match.Node); + BenchmarkEventProcessor? eventProcessor = null; // BenchmarkDotNet reports its progress through synchronous callbacks (EventProcessor and ILogger) while // the message bus and the output device are asynchronous. Blocking on those from inside a callback risks // deadlocking against the synchronization context BenchmarkDotNet installs while it runs, so the callbacks - // write to this channel and the drain below does the awaiting. Synchronous continuations are left off, so - // that a write can never end up publishing on BenchmarkDotNet's own thread. + // write to this channel and the drain in RunAsync does the awaiting. Synchronous continuations are left + // off, so that a write can never end up publishing on BenchmarkDotNet's own thread. var workQueue = Channel.CreateUnbounded>(new UnboundedChannelOptions { SingleReader = true, AllowSynchronousContinuations = false }); + try + { + var enumeration = GetMatchingBenchmarks(request.Filter); + await WarnAboutUnrecognisedFilterAsync(enumeration, cancellationToken).ConfigureAwait(false); + + foreach (var benchmarks in enumeration.Matches) + { + if (benchmarks.Count == 1) + runnable.Add(benchmarks[0]); + else + await PublishCollisionAsync(context, sessionUid, benchmarks).ConfigureAwait(false); + } + + if (runnable.Count == 0) + return; + + eventProcessor = new BenchmarkEventProcessor( + runnable.ToDictionary(match => match.Node.Uid, match => match.Node), + testNode => + { + var message = new TestNodeUpdateMessage(sessionUid, testNode); + workQueue.Writer.TryWrite(() => context.MessageBus.PublishAsync(this, message)); + }); + + await RunAsync(context, runnable, eventProcessor, workQueue, cancellationToken).ConfigureAwait(false); + } + finally + { + // A benchmark that was filtered out or that collided was never handed to BenchmarkDotNet, so nothing + // else disposes its values; and the ones that were handed over are only disposed by BenchmarkDotNet + // once its run stage began, not when it bailed out on a critical validation error. As in + // DiscoverAsync, nothing is disposed here that a later request could still run. + var ranCases = eventProcessor is { ParameterValuesDisposed: true } + ? runnable.Select(match => match.Node.BenchmarkCase) + : []; + + await parameterValues.CompleteRequestAsync(ranCases).ConfigureAwait(false); + } + } + + /// + /// Runs the given benchmarks through BenchmarkDotNet, publishing their results as they are produced. + /// + private async Task RunAsync( + ExecuteRequestContext context, + List runnable, + BenchmarkEventProcessor eventProcessor, + Channel> workQueue, + CancellationToken cancellationToken) + { // A failure while publishing has to stop the benchmarks as well, otherwise the run would carry on with // nobody listening to it. using var runCancellation = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); - var eventProcessor = new BenchmarkEventProcessor(nodes, testNode => - { - var message = new TestNodeUpdateMessage(sessionUid, testNode); - workQueue.Writer.TryWrite(() => context.MessageBus.PublishAsync(this, message)); - }); - // BenchmarkDotNet's own console output is replaced so that everything goes through the output device, // which keeps it in the right place when the platform runs in server mode or inside an IDE. var logger = new OutputDeviceLogger(serviceProvider.GetOutputDevice(), this, workQueue.Writer, cancellationToken); @@ -246,6 +277,23 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte drainFailure?.Throw(); } + /// + /// Tells the user that the request's filter is one this adapter does not know, and is being treated as + /// matching everything. + /// + private Task WarnAboutUnrecognisedFilterAsync(Enumeration enumeration, CancellationToken cancellationToken) + { + if (enumeration.UnrecognisedFilter is not { } filterType) + return Task.CompletedTask; + + var warning = new WarningMessageOutputDeviceData( + $"BenchmarkDotNet.TestAdapter does not recognise the '{filterType.FullName}' test execution filter " + + "and is treating it as matching every benchmark. Please report this at " + + "https://github.com/dotnet/BenchmarkDotNet/issues."); + + return serviceProvider.GetOutputDevice().DisplayAsync(this, warning, cancellationToken); + } + /// /// Runs the queued work items in order, until the queue is completed and empty. /// @@ -301,10 +349,15 @@ await context.MessageBus.PublishAsync( /// private Enumeration GetMatchingBenchmarks(ITestExecutionFilter filter) { - var matches = new List>(); + var (matches, unrecognisedFilter) = CreateMatcher(filter); + var matchingGroups = new List>(); var matchesByUid = new Dictionary>(StringComparer.Ordinal); var runInfos = BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly, parameterValues.TrackHidden); + // Recorded before anything else is done with them, so that whatever throws from here on - a node that + // cannot be built, a message that cannot be published - cannot lose them. + parameterValues.Track(runInfos.SelectMany(runInfo => runInfo.BenchmarksCases)); + foreach (var runInfo in runInfos) { // The job only earns a place in the display name when the benchmark actually runs under several jobs. @@ -314,37 +367,35 @@ private Enumeration GetMatchingBenchmarks(ITestExecutionFilter filter) foreach (var benchmarkCase in runInfo.BenchmarksCases) { var node = BenchmarkTestNode.Create(benchmarkCase, includeJobInName); - if (!Matches(filter, node)) + if (!matches(node)) continue; if (!matchesByUid.TryGetValue(node.Uid, out var sameUid)) { sameUid = new List(); matchesByUid.Add(node.Uid, sameUid); - matches.Add(sameUid); + matchingGroups.Add(sameUid); } sameUid.Add(new Match(runInfo, node)); } } - return new Enumeration(matches, runInfos); + return new Enumeration(matchingGroups, runInfos, unrecognisedFilter); } #pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. - private static bool Matches(ITestExecutionFilter filter, BenchmarkTestNode node) => filter switch + private static (Func Matches, Type? UnrecognisedFilter) CreateMatcher(ITestExecutionFilter filter) => filter switch { - TestNodeUidListFilter uidListFilter => uidListFilter.TestNodeUids.Any(uid => uid.Value == node.Uid), - TreeNodeFilter treeNodeFilter => treeNodeFilter.MatchesFilter(node.Path, node.GetFilterableProperties()), - NopFilter => true, - - // A filter the platform adds later has to be implemented here before it can be honoured. Falling back to - // "everything" would run the whole assembly instead of the subset that was asked for, which is a wrong - // answer rather than a visible failure. - _ => throw new NotSupportedException( - $"BenchmarkDotNet.TestAdapter does not support the '{filter.GetType().FullName}' test execution " + - "filter, and will not run every benchmark in its place. Please report this at " + - "https://github.com/dotnet/BenchmarkDotNet/issues.") + TestNodeUidListFilter uidListFilter => (node => uidListFilter.TestNodeUids.Any(uid => uid.Value == node.Uid), null), + TreeNodeFilter treeNodeFilter => (node => treeNodeFilter.MatchesFilter(node.Path, node.GetFilterableProperties()), null), + NopFilter => (_ => true, null), + + // ITestExecutionFilter is a public extension point, and a consumer can resolve a newer platform than this + // was built against, so a filter this does not know is bound to turn up one day. Failing the request + // over it would report zero tests - and discovery runs nothing, so nothing is protected by that. It is + // treated as matching everything instead, and said so, which keeps the wrong subset visible. + _ => (_ => true, filter.GetType()) }; #pragma warning restore TPEXP @@ -353,12 +404,19 @@ private Enumeration GetMatchingBenchmarks(ITestExecutionFilter filter) /// private sealed class Enumeration { - public Enumeration(List> matches, BenchmarkRunInfo[] all) + public Enumeration(List> matches, BenchmarkRunInfo[] all, Type? unrecognisedFilter) { Matches = matches; All = all; + UnrecognisedFilter = unrecognisedFilter; } + /// + /// Gets the type of the request's filter when it is one this adapter does not know, and was therefore + /// taken to match every benchmark. + /// + public Type? UnrecognisedFilter { get; } + /// /// Gets the benchmarks the request asked for, grouped by uid. /// diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs index 5106650092..10e97771a7 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs @@ -6,17 +6,29 @@ namespace BenchmarkDotNet.TestAdapter.TestingPlatform { /// - /// Holds the parameter values that the requests of a test application enumerated but never ran, and disposes them - /// once the application is done. + /// Owns the parameter values that the requests of a test application enumerate and do not run, and disposes them + /// once no request can hand them back again. /// /// - /// Disposing them at the end of the request that enumerated them is wrong under server mode, which is how Visual - /// Studio and the VS Code Test Explorer drive the platform: a discovery request and the run requests that follow - /// it are served by the same process, every request enumerates the assembly again, and a [ParamsSource] backed by - /// a cached collection - a static field, or a property over a readonly array - hands back the very same objects. - /// Discovery would then dispose the values the run is about to execute against. The platform builds a - /// per request but this extension only once, so it is what can hold the - /// values until nothing can ask for them again. + /// + /// Disposing a value at the end of the request that enumerated it is wrong under server mode, which is how Visual + /// Studio and the VS Code Test Explorer drive the platform: one process serves a discovery request and the run + /// requests that follow it, every request enumerates the assembly again, and a [ParamsSource] backed by a cached + /// collection - a static field, a property over a readonly array - hands back the very same objects. Discovery + /// would dispose the values the run is about to execute against. + /// + /// + /// Holding every value until the application ends is wrong the other way round: a source that constructs per + /// read - yield return new FileStream(...), the common shape - produces fresh objects on every request, + /// none of which a later request can reuse, and a long session would pile them up. The two are told apart by + /// what the next request enumerates: a value that comes back is cached and stays, a value that does not is gone + /// for good and is disposed then. The same rule bounds what is remembered about the values BenchmarkDotNet + /// disposed itself. + /// + /// + /// The platform builds a per request but this extension only once, which is + /// why the values live here. + /// /// internal sealed class ParameterValueLifetime : ITestHostApplicationLifetime { @@ -26,12 +38,14 @@ internal sealed class ParameterValueLifetime : ITestHostApplicationLifetime // leak or double dispose rather than fail visibly. private readonly object gate = new(); - // Everything the requests enumerated, keyed by the value so that a ParameterInstance which BenchmarkConverter - // handed to several jobs is only disposed once, and everything that was handed to BenchmarkDotNet, which - // disposes what it was given itself. Retention is remembered for the whole application: a value that one - // request ran is disposed by BenchmarkDotNet, however many later requests enumerate it again. - private readonly Dictionary enumerated = new(ParameterValueDisposer.ByReference); - private readonly HashSet retained = new(ParameterValueDisposer.ByReference); + // Keyed by the value rather than by the ParameterInstance, because BenchmarkConverter hands the same value + // to every job and every argument set of a benchmark, and it is to be disposed once. + // + // What the request in flight has enumerated so far; what the last completed request enumerated, and which + // of those BenchmarkDotNet has disposed itself because it ran them. + private Dictionary inFlight = new(ParameterValueDisposer.ByReference); + private Dictionary held = new(ParameterValueDisposer.ByReference); + private readonly HashSet disposedByBenchmarkDotNet = new(ParameterValueDisposer.ByReference); /// public string Uid => extension.Uid + ".ParameterValueLifetime"; @@ -49,20 +63,15 @@ internal sealed class ParameterValueLifetime : ITestHostApplicationLifetime public Task IsEnabledAsync() => extension.IsEnabledAsync(); /// - /// Records what a request enumerated and what of it was handed to BenchmarkDotNet, without disposing - /// anything yet. + /// Records the values the request in flight enumerated. Nothing is disposed until the request completes. /// - /// Everything the request enumerated. - /// The benchmarks that were handed to BenchmarkDotNet, if any. - public void Track(IEnumerable enumeratedCases, IEnumerable retainedCases) + /// The benchmarks the request enumerated. + public void Track(IEnumerable enumeratedCases) { lock (gate) { foreach (var parameter in ParameterValueDisposer.GetDisposableParameters(enumeratedCases)) - enumerated[parameter.Value!] = parameter; - - foreach (var parameter in ParameterValueDisposer.GetDisposableParameters(retainedCases)) - retained.Add(parameter.Value!); + inFlight[parameter.Value!] = parameter; } } @@ -70,8 +79,8 @@ public void Track(IEnumerable enumeratedCases, IEnumerable /// - /// Being kept by the enumeration is not the same as being handed to BenchmarkDotNet, so the kept values are - /// only excluded from what is collected here: whether they are ever run is for to say. + /// Being kept by the enumeration is not the same as being run, so the kept values are only excluded from what + /// is collected here: they are recorded through once the enumeration has returned. /// /// Everything the assembly declares. /// The benchmarks the enumeration returned. @@ -86,11 +95,48 @@ public void TrackHidden(IEnumerable enumeratedCases, IEnumerable< foreach (var parameter in ParameterValueDisposer.GetDisposableParameters(enumeratedCases)) { if (!kept.Contains(parameter.Value!)) - enumerated[parameter.Value!] = parameter; + inFlight[parameter.Value!] = parameter; } } } + /// + /// Completes the request in flight: disposes the values of the previous request that this one did not + /// enumerate again, and keeps the rest for the next one. + /// + /// + /// The benchmarks whose values BenchmarkDotNet disposed itself, which it does once its run stage began - and + /// not at all when it bailed out before that, on a critical validation error. + /// + public async ValueTask CompleteRequestAsync(IEnumerable ranCases) + { + List gone; + + lock (gate) + { + var current = inFlight; + inFlight = new Dictionary(ParameterValueDisposer.ByReference); + + // A value the previous request enumerated and this one did not comes from a source that constructs + // per read: no request can hand it back again, so it goes now rather than at exit. A value that came + // back is cached, and stays until nothing can ask for it. + gone = held + .Where(pair => !current.ContainsKey(pair.Key) && !disposedByBenchmarkDotNet.Contains(pair.Key)) + .Select(pair => pair.Value) + .ToList(); + + // Only the values that keep coming back need remembering as already disposed; a fresh one that was + // run is gone with its request. + disposedByBenchmarkDotNet.RemoveWhere(value => !current.ContainsKey(value)); + foreach (var parameter in ParameterValueDisposer.GetDisposableParameters(ranCases)) + disposedByBenchmarkDotNet.Add(parameter.Value!); + + held = current; + } + + await gone.DisposeAllAsync().ConfigureAwait(false); + } + /// public Task BeforeRunAsync(CancellationToken cancellationToken) => Task.CompletedTask; @@ -101,9 +147,17 @@ public async Task AfterRunAsync(int exitCode, CancellationToken cancellationToke lock (gate) { - unused = enumerated.Where(pair => !retained.Contains(pair.Key)).Select(pair => pair.Value).ToList(); - enumerated.Clear(); - retained.Clear(); + // Every request completes through CompleteRequestAsync, so nothing should be in flight here, but a + // value that somehow is would otherwise be leaked for good. + unused = held.Concat(inFlight) + .Where(pair => !disposedByBenchmarkDotNet.Contains(pair.Key)) + .GroupBy(pair => pair.Key, ParameterValueDisposer.ByReference) + .Select(group => group.First().Value) + .ToList(); + + held.Clear(); + inFlight.Clear(); + disposedByBenchmarkDotNet.Clear(); } await unused.DisposeAllAsync().ConfigureAwait(false); From 48faefa2339bf0c246a243fb10d2848e308d2444 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Wed, 9 Sep 2026 08:33:25 +0100 Subject: [PATCH 098/110] Improve test disposal checks, add discoverAgain option Added tests for parameter value disposal in TestingPlatformAdapterTests.cs, including cases for per-read sources and BenchmarkDotNet validation failures. Updated TestingPlatformServerModeSession.cs to support an optional discoverAgain parameter, enabling a second discovery after test runs to simulate IDE refresh. Modified Run and DiscoverThenRun methods to handle discoverAgain, and enhanced test infrastructure for better real-world simulation. --- .../TestingPlatformAdapterTests.cs | 54 +++++++++++++++++++ .../TestingPlatformServerModeSession.cs | 11 ++-- 2 files changed, 62 insertions(+), 3 deletions(-) diff --git a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs index 153f585be3..80aa5f4706 100644 --- a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs +++ b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs @@ -50,6 +50,9 @@ public void EveryBenchmarkIsDiscoveredUnderItsOwnName() "DisposableProbe.Identity(Value: tracked-2)", "DisposableProbe.Identity(Value: tracked-3)", + "FreshValueProbe.Identity(Value: fresh-1)", + "FreshValueProbe.Identity(Value: fresh-2)", + // A generic benchmark is named after the type arguments it was closed over. "GenericProbe.Create", "GenericProbe>.Create", @@ -209,6 +212,57 @@ public void ParameterValuesOfBenchmarksNoRequestRanAreDisposedWhenTheApplication Assert.Equal("created=2 disposed=2", report); } + [Fact] + public void ParameterValuesOfASourceThatConstructsPerReadAreDisposedAsRequestsGoBy() + { + // FreshValueProbe's source hands back new values on every read, so nothing one request enumerated can + // ever be reused by the next. Holding them for the whole session would turn a long Test Explorer session + // into the handle leak the disposal exists to prevent: they have to go as soon as the following request + // shows they did not come back. The probe reports what had been disposed by the time each read happened. + var report = ReadDisposalReport( + PassingProbes, + "fresh-value-probe.txt", + () => TestingPlatformServerModeSession.DiscoverThenRun( + GetProbeApplication(PassingProbes), + "DisposableProbe.Identity(Value: tracked", + Timeout, + discoverAgain: true)); + + Assert.Equal( + [ + "read=1 created=2 disposed=0", + + // The values of the first request are only known to be unreusable once this read did not hand + // them back, which is after it. + "read=2 created=4 disposed=0", + + // By the third request they are gone, and so on: the session holds one request's worth. + "read=3 created=6 disposed=2", + "exit created=6 disposed=6", + ], + report.Split([Environment.NewLine], StringSplitOptions.RemoveEmptyEntries)); + } + + [Fact] + public void ParameterValuesAreDisposedWhenBenchmarkDotNetBailsOutOnValidation() + { + // The unoptimized probe application fails JitOptimizationsValidator, which is critical: BenchmarkRunnerClean + // returns before the try whose finally disposes the values it was handed, so nothing disposes them. The + // adapter must not take "handed to BenchmarkDotNet" for "disposed by BenchmarkDotNet" - that assumption + // would leave exactly these values, of a run that never started, to the finalizer for good. + IReadOnlyList discovered = []; + TestRunSummary? summary = null; + + var report = ReadDisposalReport( + UnoptimizedProbes, + "unoptimized-probe.txt", + () => summary = RunAndSummarize(UnoptimizedProbes, "--treenode-filter", "/*/*/SharedValueProbe/*")); + + Assert.Equal(2, summary!.Total); + Assert.Equal(2, summary.Failed); + Assert.Equal("created=4 disposed=4", report); + } + [Fact] public void ParameterValuesAreDisposedWhenOnlyOneBenchmarkOfASetIsRun() { diff --git a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformServerModeSession.cs b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformServerModeSession.cs index 60b7fb61c1..b0280badc8 100644 --- a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformServerModeSession.cs +++ b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformServerModeSession.cs @@ -45,11 +45,13 @@ private TestingPlatformServerModeSession(Process process, TcpClient client, Time /// The probe application to drive. /// The text the display name of a benchmark has to contain to be run. /// How long any one step may take. + /// Whether to discover a second time once the run is over, as an IDE refreshing does. /// The nodes the discovery reported, and the last state each ran node was reported in. public static (IReadOnlyList Discovered, IReadOnlyList Ran) DiscoverThenRun( string application, string runFilter, - TimeSpan timeout) + TimeSpan timeout, + bool discoverAgain = false) { var listener = new TcpListener(IPAddress.Loopback, 0); listener.Start(); @@ -77,7 +79,7 @@ public static (IReadOnlyList Discovered, IReadOnlyList R using var client = listener.AcceptTcpClient(); using var session = new TestingPlatformServerModeSession(process, client, timeout); - return session.Run(runFilter); + return session.Run(runFilter, discoverAgain); } finally { @@ -85,7 +87,7 @@ public static (IReadOnlyList Discovered, IReadOnlyList R } } - private (IReadOnlyList, IReadOnlyList) Run(string runFilter) + private (IReadOnlyList, IReadOnlyList) Run(string runFilter, bool discoverAgain) { SendRequest("initialize", new { @@ -112,6 +114,9 @@ public static (IReadOnlyList Discovered, IReadOnlyList R .ToArray() }); + if (discoverAgain) + Exchange("testing/discoverTests", runId => new { runId }); + Send(new { jsonrpc = "2.0", method = "exit", @params = new { } }); if (!process.WaitForExit((int)timeout.TotalMilliseconds)) From acb489a461ad58ea3e51182f8406f9fbb72b0620 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Wed, 9 Sep 2026 08:33:55 +0100 Subject: [PATCH 099/110] Improve TypeFilterTests for unreadable attribute scenarios Added tests to ensure proper logging when benchmark types or assemblies have unreadable attributes. Introduced a helper to emit assemblies with unreadable benchmark types for testing. --- .../BenchmarkDotNet.Tests/TypeFilterTests.cs | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/tests/BenchmarkDotNet.Tests/TypeFilterTests.cs b/tests/BenchmarkDotNet.Tests/TypeFilterTests.cs index ce924c2136..0f1e4c4186 100644 --- a/tests/BenchmarkDotNet.Tests/TypeFilterTests.cs +++ b/tests/BenchmarkDotNet.Tests/TypeFilterTests.cs @@ -225,6 +225,68 @@ public void ReportsATypeWhoseAttributesCannotBeRead() Assert.Contains(nameof(ClassWithUnreadableConfig), logger.GetLog(), StringComparison.Ordinal); } + [Fact] + public void ReportsATypeWhoseAttributesCannotBeReadWhenItIsTheOnlyOne() + { + // With nothing else to run, the validator never gets a chance to say anything, so this is the one place + // left to explain why nothing was found - and the message must not be that no [Benchmark] was found. + var logger = new AccumulationLogger(); + + var benchmarks = Filter([typeof(ClassWithUnreadableConfig)], ["--filter", "*"], logger); + + Assert.Empty(benchmarks); + Assert.Contains(nameof(ClassWithUnreadableConfig), logger.GetLog(), StringComparison.Ordinal); + Assert.DoesNotContain("No [Benchmark] attribute found", logger.GetLog(), StringComparison.Ordinal); + } + +#if NETCOREAPP + [Fact] + public void ReportsAnAssemblyWhoseOnlyBenchmarkTypeCannotBeRead() + { + // BenchmarkSwitcher.FromAssembly(assembly).Run(args) comes in through the assembly rather than through + // types, where "does this assembly declare benchmarks" used to be answered from the readable types only: + // an assembly whose only benchmark class is unreadable was then told it has no [Benchmark] at all, which + // is both silent about the cause and wrong. Emitted rather than compiled, because every real assembly + // of this repository declares readable benchmarks too. + var assembly = EmitAssemblyWithAnUnreadableBenchmarkType(); + var logger = new AccumulationLogger(); + + var (allTypesValid, runnable) = TypeFilter.GetTypesWithRunnableBenchmarks([], [assembly], logger); + + Assert.True(allTypesValid); + Assert.Empty(runnable); + Assert.Contains("Unreadable was ignored because its attributes could not be read", logger.GetLog(), StringComparison.Ordinal); + Assert.DoesNotContain("No [Benchmark] attribute found", logger.GetLog(), StringComparison.Ordinal); + } + + private static System.Reflection.Assembly EmitAssemblyWithAnUnreadableBenchmarkType() + { + var assembly = System.Reflection.Emit.AssemblyBuilder.DefineDynamicAssembly( + new System.Reflection.AssemblyName("UnreadableBenchmarks"), + System.Reflection.Emit.AssemblyBuilderAccess.Run); + var type = assembly.DefineDynamicModule("UnreadableBenchmarks").DefineType( + "Unreadable", + System.Reflection.TypeAttributes.Public | System.Reflection.TypeAttributes.Class); + + // [Config(typeof(AbstractConfig))]: reflection constructs the attribute in order to hand it back, and + // ConfigAttribute instantiates the config it is given, which an abstract one cannot be. + type.SetCustomAttribute(new System.Reflection.Emit.CustomAttributeBuilder( + typeof(ConfigAttribute).GetConstructor([typeof(Type)])!, + [typeof(AbstractConfig)])); + + // [Benchmark] public void Method1() { } + var method = type.DefineMethod("Method1", System.Reflection.MethodAttributes.Public, typeof(void), Type.EmptyTypes); + method.SetCustomAttribute(new System.Reflection.Emit.CustomAttributeBuilder( + typeof(BenchmarkAttribute).GetConstructor([typeof(int), typeof(string)])!, + [0, ""])); + method.GetILGenerator().Emit(System.Reflection.Emit.OpCodes.Ret); + + type.CreateType(); + + return assembly; + } +#endif + private HashSet Filter(Type[] types, string[] args, ILogger? logger = null) { var nonNullLogger = logger ?? new OutputLogger(Output); From d1fef31033cf5f6b1c2a92fbc30b5a69882a4b1a Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Wed, 9 Sep 2026 08:34:11 +0100 Subject: [PATCH 100/110] Report all unsuccessful benchmarks as validation errors Previously, only unsuccessful and readable benchmarks were reported as validation errors. Now, the logic includes all unsuccessful benchmarks, ensuring unreadable types are also reported. This change improves error reporting for scenarios without TypeFilter, such as BenchmarkRunner.Run() and some test adapters. --- src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs b/src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs index 4513dfa0f5..1c5b7eb0d9 100644 --- a/src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs +++ b/src/BenchmarkDotNet/Validators/GenericBenchmarksValidator.cs @@ -16,8 +16,9 @@ public IAsyncEnumerable ValidateAsync(ValidationParameters vali .Distinct() .SelectMany(assembly => assembly.GetRunnableBenchmarks()) .SelectMany(GenericBenchmarksBuilder.BuildGenericsIfNeeded) - // An unreadable type is reported by TypeFilter, which sees it even when nothing else survives. - .Where(built => !built.IsSuccess && !built.IsUnreadable) + // An unreadable type is reported here as well as by TypeFilter: BenchmarkRunner.Run() and both + // test adapters never go through TypeFilter, so for them this is the only report there is. + .Where(built => !built.IsSuccess) .Select(built => new ValidationError(false, built.Error!)) .ToAsyncEnumerable(); } From 51e49603ee744cb41ff1793d264e28f9bcc23668 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Wed, 9 Sep 2026 08:34:27 +0100 Subject: [PATCH 101/110] Add FreshValueProbe benchmark for per-read resource tests Introduced FreshValueProbe in BenchmarkDotNet.IntegrationTests.TestingPlatform to benchmark parameter sources that yield new resources on each read. Tracks reads, creations, and disposals of Fresh disposable instances, and writes reports to a file per read and at process exit. Includes FastConfig for in-process dry job execution. --- .../FreshValueProbe.cs | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/FreshValueProbe.cs diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/FreshValueProbe.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/FreshValueProbe.cs new file mode 100644 index 0000000000..be82e6d48a --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform/FreshValueProbe.cs @@ -0,0 +1,76 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform +{ + /// + /// A benchmark whose parameter source constructs its values on every read - the common shape of a source over + /// resources, yield return new FileStream(...) - as opposed to , which hands + /// back the same instances every time. Under server mode every request reads the source again, and nothing a + /// later request could reuse comes of it, so the values of one request must not be held for the whole session. + /// + /// + /// The report is a line per read, written as the read happens, so that a test driving several requests from one + /// process can see what had been disposed by the time each request enumerated - and a last line at exit. + /// + [Config(typeof(FastConfig))] + public class FreshValueProbe + { + /// + /// The name of the file the counts are written to, next to the probe application. + /// + public const string ReportFileName = "fresh-value-probe.txt"; + + private static int reads; + private static int created; + private static int disposed; + + static FreshValueProbe() => + AppDomain.CurrentDomain.ProcessExit += (_, _) => Report($"exit created={Volatile.Read(ref created)} disposed={Volatile.Read(ref disposed)}"); + + public static IEnumerable Values + { + get + { + var read = Interlocked.Increment(ref reads); + Fresh[] values = [new Fresh(1), new Fresh(2)]; + + Report($"read={read} created={Volatile.Read(ref created)} disposed={Volatile.Read(ref disposed)}"); + + return values; + } + } + + [ParamsSource(nameof(Values))] + public Fresh? Value { get; set; } + + [Benchmark] + public int Identity() => Value!.Number; + + private static void Report(string line) + => File.AppendAllText(Path.Combine(AppContext.BaseDirectory, ReportFileName), line + Environment.NewLine); + + public class Fresh : IDisposable + { + public Fresh(int number) + { + Number = number; + Interlocked.Increment(ref created); + } + + public int Number { get; } + + public void Dispose() => Interlocked.Increment(ref disposed); + + // The same name on every read, so that a benchmark keeps its identity across the requests of a session. + public override string ToString() => $"fresh-{Number}"; + } + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} From acdbf335b62048267414136b220db327879f3a09 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Thu, 10 Sep 2026 10:10:48 +0100 Subject: [PATCH 102/110] Improve error handling and deduplication in BenchmarkRunner - Use ExceptionDispatchInfo to capture/rethrow exceptions during async disposal, ensuring cleanup and event signaling always occur - Update artifacts cleanup log message for consistency - Prevent duplicate ValidationError entries by tracking errors in a HashSet during validation --- .../Running/BenchmarkRunnerClean.cs | 30 +++++++++++++++++-- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/src/BenchmarkDotNet/Running/BenchmarkRunnerClean.cs b/src/BenchmarkDotNet/Running/BenchmarkRunnerClean.cs index 775b02bfe5..042a3381d2 100644 --- a/src/BenchmarkDotNet/Running/BenchmarkRunnerClean.cs +++ b/src/BenchmarkDotNet/Running/BenchmarkRunnerClean.cs @@ -22,6 +22,7 @@ using Perfolizer.Horology; using System.Collections.Immutable; using System.Runtime.CompilerServices; +using System.Runtime.ExceptionServices; using System.Text.RegularExpressions; using RunMode = BenchmarkDotNet.Jobs.RunMode; @@ -197,14 +198,29 @@ private static async ValueTask RunCore(BenchmarkRunInfo[] benchmarkRu // some benchmarks might be using parameters that have locking finalizers // so we need to dispose them after we are done running the benchmarks // see https://github.com/dotnet/BenchmarkDotNet/issues/1383 and https://github.com/dotnet/runtime/issues/314 for more - await benchmarkRunInfos.DisposeAllAsync().ConfigureAwait(); + // + // DisposeAllAsync goes through every value before it reports a failure, so one that threw leaves + // nothing else undisposed - and must not skip the teardown below either. Whoever is listening is told + // that the run stage ended whatever the disposal did, and the failure is raised after them. + ExceptionDispatchInfo? disposeFailure = null; + + try + { + await benchmarkRunInfos.DisposeAllAsync().ConfigureAwait(); + } + catch (Exception exception) + { + disposeFailure = ExceptionDispatchInfo.Capture(exception); + } compositeLogger.WriteLineHeader("// * Artifacts cleanup *"); Cleanup(compositeLogger, new HashSet(artifactsToCleanup.Distinct())); - compositeLogger.WriteLineInfo("Artifacts cleanup is finished"); + compositeLogger.WriteLineInfo("Artifacts cleanup is finished."); compositeLogger.Flush(); eventProcessor.OnEndRunStage(); + + disposeFailure?.Throw(); } } @@ -404,6 +420,13 @@ private static async ValueTask> Validate(BenchmarkR { var errors = new List(); + // The validators run once per BenchmarkRunInfo, so one that looks at the whole assembly - as + // GenericBenchmarksValidator does - reports the same thing again for every type in it. PrintValidationErrors + // has always shown those once, but the event processors are handed every copy, and an error that names no + // benchmark case is fanned out by the test adapters to every node: N types then put N copies of the same + // warning on each of N nodes. The same error, is one error however many times it is raised. + var reported = new HashSet(); + foreach (var benchmark in benchmarks) { var validationParameters = new ValidationParameters(benchmark.BenchmarksCases, benchmark.Config); @@ -411,7 +434,8 @@ private static async ValueTask> Validate(BenchmarkR await foreach (var error in benchmark.Config.GetCompositeValidator().ValidateAsync(validationParameters).ConfigureAwait(cancellationToken)) #pragma warning restore CA2007 { - errors.Add(error); + if (reported.Add(error)) + errors.Add(error); } } From 44c05976202e8114d4c1129d424b4dad19e1216a Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Thu, 10 Sep 2026 10:12:22 +0100 Subject: [PATCH 103/110] Add test for validation warnings; extend ServerNode output Added AnAssemblyWideValidationWarningIsReportedOncePerNode test to ensure assembly-wide validation warnings are reported once per node. Updated ServerNode in TestingPlatformServerModeSession.cs to include StandardOutput property and populated it from node JSON data. --- .../TestingPlatformAdapterTests.cs | 20 ++++++++++++++++++- .../TestingPlatformServerModeSession.cs | 5 +++-- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs index 80aa5f4706..fc6e314efa 100644 --- a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs +++ b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs @@ -243,6 +243,25 @@ public void ParameterValuesOfASourceThatConstructsPerReadAreDisposedAsRequestsGo report.Split([Environment.NewLine], StringSplitOptions.RemoveEmptyEntries)); } + [Fact] + public void AnAssemblyWideValidationWarningIsReportedOncePerNode() + { + // GenericBenchmarksValidator looks at the whole assembly, however BenchmarkDotNet runs the validators once + // per benchmark type, so an unreadable type is reported again for every type that runs. An error that + // names no benchmark case is put on every node, so without deduplication N types leave N copies of the + // same warning on each of the N types' nodes. + var (_, ran) = TestingPlatformServerModeSession.DiscoverThenRun( + GetProbeApplication(PassingProbes), + "Probe.Identity", + Timeout); + + Assert.True(ran.Count >= 4, $"Expected several benchmark types to run, but {ran.Count} node(s) did."); + Assert.All(ran, node => Assert.Equal("passed", node.ExecutionState)); + Assert.All( + ran, + node => Assert.Single(Regex.Matches(node.StandardOutput, "WithAbstractConfig was ignored"))); + } + [Fact] public void ParameterValuesAreDisposedWhenBenchmarkDotNetBailsOutOnValidation() { @@ -250,7 +269,6 @@ public void ParameterValuesAreDisposedWhenBenchmarkDotNetBailsOutOnValidation() // returns before the try whose finally disposes the values it was handed, so nothing disposes them. The // adapter must not take "handed to BenchmarkDotNet" for "disposed by BenchmarkDotNet" - that assumption // would leave exactly these values, of a run that never started, to the finalizer for good. - IReadOnlyList discovered = []; TestRunSummary? summary = null; var report = ReadDisposalReport( diff --git a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformServerModeSession.cs b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformServerModeSession.cs index b0280badc8..8a1dda83fd 100644 --- a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformServerModeSession.cs +++ b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformServerModeSession.cs @@ -273,7 +273,8 @@ private void Handle(string json) nodes.Add(new ServerNode( node.GetProperty("uid").GetString()!, node.GetProperty("display-name").GetString()!, - node.TryGetProperty("execution-state", out var state) ? state.GetString()! : "")); + node.TryGetProperty("execution-state", out var state) ? state.GetString()! : "", + node.TryGetProperty("standardOutput", out var output) ? output.GetString() ?? "" : "")); } } } @@ -307,7 +308,7 @@ public void Dispose() client.Dispose(); } - internal sealed record ServerNode(string Uid, string DisplayName, string ExecutionState); + internal sealed record ServerNode(string Uid, string DisplayName, string ExecutionState, string StandardOutput); } } #endif From aed3f0b0599e5e7f66142b2472780fef8c9ca1fa Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Thu, 10 Sep 2026 10:12:59 +0100 Subject: [PATCH 104/110] Refactor parameter value lifetime with RequestScope Refactored parameter value lifetime management by introducing a RequestScope class to encapsulate parameter values per request, improving isolation and handling of overlapping requests. Moved tracking, hiding, and completion logic into RequestScope and updated BenchmarkTestFramework to use the new scope. Updated disposal logic to ensure correct cleanup and clarified comments. Improved handling of unrecognized filters: discovery lists all benchmarks, run requests reject unsupported filters. --- .../BenchmarkEventProcessor.cs | 8 +- .../TestingPlatform/BenchmarkTestFramework.cs | 56 ++++-- .../TestingPlatform/ParameterValueLifetime.cs | 189 +++++++++++------- 3 files changed, 154 insertions(+), 99 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs index cfbba9096b..aaa64f506a 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkEventProcessor.cs @@ -43,10 +43,10 @@ public BenchmarkEventProcessor(IReadOnlyDictionary no /// Gets whether BenchmarkDotNet disposed the parameter values of the benchmarks it was handed. /// /// - /// BenchmarkRunnerClean.Run disposes them in the finally of its run stage, which is where OnEndRunStage is - /// raised from - after the disposal, and whether the run completed, threw or was cancelled. It never gets - /// there when a critical validation error makes it return before the run stage, and nothing disposes the - /// values then, so whoever handed them over has to. + /// BenchmarkRunnerClean disposes them in the finally of its run stage, which is where OnEndRunStage is + /// raised from - after the disposal has been through every value, whether the run completed, threw, was + /// cancelled, or the disposal itself threw. It never gets there when a critical validation error makes it + /// return before the run stage, and nothing disposes the values then, so whoever handed them over has to. /// public bool ParameterValuesDisposed => parameterValuesDisposed; diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs index 158c454ecd..0c72702c66 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs @@ -93,9 +93,14 @@ private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRe { // The enumeration sits inside the try: it records the values it creates as it goes, and completing the // request is what hands them over, so nothing that throws between the two can lose them. + var parameterValueScope = parameterValues.BeginRequest(); + try { - var enumeration = GetMatchingBenchmarks(request.Filter); + var enumeration = GetMatchingBenchmarks(request.Filter, parameterValueScope); + + // Discovery runs nothing, so a filter this adapter does not know costs nothing but a wrong list, and + // reporting every benchmark beats reporting none. The run path refuses it instead. await WarnAboutUnrecognisedFilterAsync(enumeration, context.CancellationToken).ConfigureAwait(false); foreach (var benchmarks in enumeration.Matches) @@ -115,7 +120,7 @@ private async Task DiscoverAsync(DiscoverTestExecutionRequest request, ExecuteRe { // Discovery runs nothing, so BenchmarkDotNet disposed nothing. The values are not disposed here // either: under server mode a run request follows in this very process, see ParameterValueLifetime. - await parameterValues.CompleteRequestAsync([]).ConfigureAwait(false); + await parameterValueScope.CompleteAsync([]).ConfigureAwait(false); } } @@ -129,6 +134,7 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte // disposed the values, as one does when publishing fails, still has that on record. var runnable = new List(); BenchmarkEventProcessor? eventProcessor = null; + var parameterValueScope = parameterValues.BeginRequest(); // BenchmarkDotNet reports its progress through synchronous callbacks (EventProcessor and ILogger) while // the message bus and the output device are asynchronous. Blocking on those from inside a callback risks @@ -143,8 +149,19 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte try { - var enumeration = GetMatchingBenchmarks(request.Filter); - await WarnAboutUnrecognisedFilterAsync(enumeration, cancellationToken).ConfigureAwait(false); + var enumeration = GetMatchingBenchmarks(request.Filter, parameterValueScope); + + // Unlike discovery, a run cannot treat a filter it does not know as matching everything: that would + // spend the machine's next hour benchmarking the whole assembly instead of the subset that was asked + // for, and a warning on the output device is not something an IDE is bound to surface. Refusing the + // request keeps the filter visible and costs nothing but a re-run once it is supported. + if (enumeration.UnrecognisedFilter is { } unrecognisedFilter) + { + throw new NotSupportedException( + $"BenchmarkDotNet.TestAdapter does not support the '{unrecognisedFilter.FullName}' test " + + "execution filter, and will not run every benchmark of the assembly in its place. Please " + + "report this at https://github.com/dotnet/BenchmarkDotNet/issues."); + } foreach (var benchmarks in enumeration.Matches) { @@ -177,7 +194,7 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte ? runnable.Select(match => match.Node.BenchmarkCase) : []; - await parameterValues.CompleteRequestAsync(ranCases).ConfigureAwait(false); + await parameterValueScope.CompleteAsync(ranCases).ConfigureAwait(false); } } @@ -343,20 +360,21 @@ await context.MessageBus.PublishAsync( /// Enumerates the benchmarks of the assembly and keeps the ones the request asked for. /// /// The filter of the request. + /// The scope the values this creates are recorded in. /// - /// The matching benchmarks in enumeration order and grouped by uid, together with everything the assembly - /// declares. A group holding more than one benchmark is a uid collision. + /// The matching benchmarks in enumeration order and grouped by uid. A group holding more than one benchmark + /// is a uid collision. /// - private Enumeration GetMatchingBenchmarks(ITestExecutionFilter filter) + private Enumeration GetMatchingBenchmarks(ITestExecutionFilter filter, ParameterValueLifetime.RequestScope parameterValueScope) { var (matches, unrecognisedFilter) = CreateMatcher(filter); var matchingGroups = new List>(); var matchesByUid = new Dictionary>(StringComparer.Ordinal); - var runInfos = BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly, parameterValues.TrackHidden); + var runInfos = BenchmarkEnumerator.GetBenchmarksFromAssembly(assembly, parameterValueScope.TrackHidden); // Recorded before anything else is done with them, so that whatever throws from here on - a node that // cannot be built, a message that cannot be published - cannot lose them. - parameterValues.Track(runInfos.SelectMany(runInfo => runInfo.BenchmarksCases)); + parameterValueScope.Track(runInfos.SelectMany(runInfo => runInfo.BenchmarksCases)); foreach (var runInfo in runInfos) { @@ -381,7 +399,7 @@ private Enumeration GetMatchingBenchmarks(ITestExecutionFilter filter) } } - return new Enumeration(matchingGroups, runInfos, unrecognisedFilter); + return new Enumeration(matchingGroups, unrecognisedFilter); } #pragma warning disable TPEXP // The tree node filter is still marked as experimental by the platform. @@ -391,10 +409,10 @@ private Enumeration GetMatchingBenchmarks(ITestExecutionFilter filter) TreeNodeFilter treeNodeFilter => (node => treeNodeFilter.MatchesFilter(node.Path, node.GetFilterableProperties()), null), NopFilter => (_ => true, null), - // ITestExecutionFilter is a public extension point, and a consumer can resolve a newer platform than this - // was built against, so a filter this does not know is bound to turn up one day. Failing the request - // over it would report zero tests - and discovery runs nothing, so nothing is protected by that. It is - // treated as matching everything instead, and said so, which keeps the wrong subset visible. + // ITestExecutionFilter is a public extension point, and a consumer can resolve a newer platform than + // this was built against, so a filter this does not know is bound to turn up one day. It matches + // everything and is reported as unrecognised; what that costs differs between the two requests, so what + // to do about it is left to each of them - discovery lists the lot and says so, a run refuses. _ => (_ => true, filter.GetType()) }; #pragma warning restore TPEXP @@ -404,10 +422,9 @@ private Enumeration GetMatchingBenchmarks(ITestExecutionFilter filter) /// private sealed class Enumeration { - public Enumeration(List> matches, BenchmarkRunInfo[] all, Type? unrecognisedFilter) + public Enumeration(List> matches, Type? unrecognisedFilter) { Matches = matches; - All = all; UnrecognisedFilter = unrecognisedFilter; } @@ -421,11 +438,6 @@ public Enumeration(List> matches, BenchmarkRunInfo[] all, Type? unre /// Gets the benchmarks the request asked for, grouped by uid. /// public List> Matches { get; } - - /// - /// Gets every benchmark the assembly declares, matching or not. - /// - public BenchmarkRunInfo[] All { get; } } /// diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs index 10e97771a7..9166caf157 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs @@ -26,6 +26,10 @@ namespace BenchmarkDotNet.TestAdapter.TestingPlatform /// disposed itself. /// /// + /// Each request collects into a of its own, so that requests the platform chooses to + /// overlap cannot take each other's values down; only completing a request touches what is held. + /// + /// /// The platform builds a per request but this extension only once, which is /// why the values live here. /// @@ -34,16 +38,13 @@ internal sealed class ParameterValueLifetime : ITestHostApplicationLifetime { private readonly BenchmarkDotNetExtension extension = new(); - // Requests are served one at a time, but the platform is free to change that, and getting this wrong would - // leak or double dispose rather than fail visibly. private readonly object gate = new(); - // Keyed by the value rather than by the ParameterInstance, because BenchmarkConverter hands the same value - // to every job and every argument set of a benchmark, and it is to be disposed once. + // Keyed by the value rather than by the ParameterInstance, because BenchmarkConverter hands the same value to + // every job and every argument set of a benchmark, and it is to be disposed once. // - // What the request in flight has enumerated so far; what the last completed request enumerated, and which - // of those BenchmarkDotNet has disposed itself because it ran them. - private Dictionary inFlight = new(ParameterValueDisposer.ByReference); + // What the last completed request enumerated, and which of those BenchmarkDotNet has disposed itself because + // it ran them. private Dictionary held = new(ParameterValueDisposer.ByReference); private readonly HashSet disposedByBenchmarkDotNet = new(ParameterValueDisposer.ByReference); @@ -63,104 +64,146 @@ internal sealed class ParameterValueLifetime : ITestHostApplicationLifetime public Task IsEnabledAsync() => extension.IsEnabledAsync(); /// - /// Records the values the request in flight enumerated. Nothing is disposed until the request completes. + /// Starts collecting the parameter values of one request. /// - /// The benchmarks the request enumerated. - public void Track(IEnumerable enumeratedCases) + /// The scope to record that request's values in, and to complete when it is over. + public RequestScope BeginRequest() => new RequestScope(this); + + /// + public Task BeforeRunAsync(CancellationToken cancellationToken) => Task.CompletedTask; + + /// + public async Task AfterRunAsync(int exitCode, CancellationToken cancellationToken) { + List unused; + lock (gate) { - foreach (var parameter in ParameterValueDisposer.GetDisposableParameters(enumeratedCases)) - inFlight[parameter.Value!] = parameter; + unused = held + .Where(pair => !disposedByBenchmarkDotNet.Contains(pair.Key)) + .Select(pair => pair.Value) + .ToList(); + + held.Clear(); + disposedByBenchmarkDotNet.Clear(); } + + await unused.DisposeAllAsync().ConfigureAwait(false); } - /// - /// Records the values of the benchmarks that the enumeration hid, which no request will ever be handed. - /// - /// - /// Being kept by the enumeration is not the same as being run, so the kept values are only excluded from what - /// is collected here: they are recorded through once the enumeration has returned. - /// - /// Everything the assembly declares. - /// The benchmarks the enumeration returned. - public void TrackHidden(IEnumerable enumeratedCases, IEnumerable keptCases) + private async ValueTask CompleteAsync(RequestScope request, IEnumerable ranCases) { + List gone = []; + lock (gate) { - var kept = new HashSet( - ParameterValueDisposer.GetDisposableParameters(keptCases).Select(parameter => parameter.Value!), - ParameterValueDisposer.ByReference); + var enumerated = request.Enumerated; - foreach (var parameter in ParameterValueDisposer.GetDisposableParameters(enumeratedCases)) + if (!request.HasEnumerated) { - if (!kept.Contains(parameter.Value!)) - inFlight[parameter.Value!] = parameter; - } - } - } + // The request never reached the end of its enumeration - the assembly failed to load, a source + // threw partway - so its absences say nothing about what a source would hand back, and what is + // held has to stay. Whatever it did manage to create joins it, rather than being lost. + foreach (var pair in enumerated) + held[pair.Key] = pair.Value; - /// - /// Completes the request in flight: disposes the values of the previous request that this one did not - /// enumerate again, and keeps the rest for the next one. - /// - /// - /// The benchmarks whose values BenchmarkDotNet disposed itself, which it does once its run stage began - and - /// not at all when it bailed out before that, on a critical validation error. - /// - public async ValueTask CompleteRequestAsync(IEnumerable ranCases) - { - List gone; + return; + } - lock (gate) - { - var current = inFlight; - inFlight = new Dictionary(ParameterValueDisposer.ByReference); + // Recorded before anything is chosen for disposal, so that a value BenchmarkDotNet has just disposed + // can never also be a candidate here. + foreach (var parameter in ParameterValueDisposer.GetDisposableParameters(ranCases)) + disposedByBenchmarkDotNet.Add(parameter.Value!); - // A value the previous request enumerated and this one did not comes from a source that constructs - // per read: no request can hand it back again, so it goes now rather than at exit. A value that came - // back is cached, and stays until nothing can ask for it. + // A value the last completed request enumerated and this one did not comes from a source that + // constructs per read: no request can hand it back again, so it goes now rather than at exit. A value + // that came back is cached, and stays until nothing can ask for it. gone = held - .Where(pair => !current.ContainsKey(pair.Key) && !disposedByBenchmarkDotNet.Contains(pair.Key)) + .Where(pair => !enumerated.ContainsKey(pair.Key) && !disposedByBenchmarkDotNet.Contains(pair.Key)) .Select(pair => pair.Value) .ToList(); // Only the values that keep coming back need remembering as already disposed; a fresh one that was // run is gone with its request. - disposedByBenchmarkDotNet.RemoveWhere(value => !current.ContainsKey(value)); - foreach (var parameter in ParameterValueDisposer.GetDisposableParameters(ranCases)) - disposedByBenchmarkDotNet.Add(parameter.Value!); + disposedByBenchmarkDotNet.RemoveWhere(value => !enumerated.ContainsKey(value)); - held = current; + held = enumerated; } await gone.DisposeAllAsync().ConfigureAwait(false); } - /// - public Task BeforeRunAsync(CancellationToken cancellationToken) => Task.CompletedTask; - - /// - public async Task AfterRunAsync(int exitCode, CancellationToken cancellationToken) + /// + /// The parameter values one request enumerated. + /// + internal sealed class RequestScope { - List unused; - - lock (gate) + private readonly ParameterValueLifetime owner; + + internal RequestScope(ParameterValueLifetime owner) => this.owner = owner; + + /// + /// Gets the values this request enumerated, keyed by the value itself. + /// + internal Dictionary Enumerated { get; } = new(ParameterValueDisposer.ByReference); + + /// + /// Gets whether the request got as far as enumerating the assembly. It tells "this request enumerated + /// nothing" apart from "this request never got to enumerate", which is the difference between concluding + /// that a source no longer hands a value back and having asked it nothing at all. + /// + internal bool HasEnumerated { get; private set; } + + /// + /// Records the values of the benchmarks the enumeration returned, and marks the enumeration as reached. + /// + /// The benchmarks the enumeration returned. + public void Track(IEnumerable enumeratedCases) { - // Every request completes through CompleteRequestAsync, so nothing should be in flight here, but a - // value that somehow is would otherwise be leaked for good. - unused = held.Concat(inFlight) - .Where(pair => !disposedByBenchmarkDotNet.Contains(pair.Key)) - .GroupBy(pair => pair.Key, ParameterValueDisposer.ByReference) - .Select(group => group.First().Value) - .ToList(); + lock (owner.gate) + { + foreach (var parameter in ParameterValueDisposer.GetDisposableParameters(enumeratedCases)) + Enumerated[parameter.Value!] = parameter; - held.Clear(); - inFlight.Clear(); - disposedByBenchmarkDotNet.Clear(); + HasEnumerated = true; + } } - await unused.DisposeAllAsync().ConfigureAwait(false); + /// + /// Records the values of the benchmarks that the enumeration hid, which no request will ever be handed. + /// + /// + /// Called from inside the enumeration, which may still throw afterwards, so it deliberately does not mark + /// the enumeration as reached: being kept by the enumeration is not the same as being run, and the kept + /// values are recorded by once the enumeration has returned. + /// + /// Everything the assembly declares. + /// The benchmarks the enumeration returned. + public void TrackHidden(IEnumerable enumeratedCases, IEnumerable keptCases) + { + lock (owner.gate) + { + var kept = new HashSet( + ParameterValueDisposer.GetDisposableParameters(keptCases).Select(parameter => parameter.Value!), + ParameterValueDisposer.ByReference); + + foreach (var parameter in ParameterValueDisposer.GetDisposableParameters(enumeratedCases)) + { + if (!kept.Contains(parameter.Value!)) + Enumerated[parameter.Value!] = parameter; + } + } + } + + /// + /// Completes the request: disposes the values of the last completed request that this one did not + /// enumerate again, and keeps the rest for the next one. + /// + /// + /// The benchmarks whose values BenchmarkDotNet disposed itself, which it does once its run stage began - + /// and not at all when it bailed out before that, on a critical validation error. + /// + public ValueTask CompleteAsync(IEnumerable ranCases) => owner.CompleteAsync(this, ranCases); } } } From 204c68d5b1a13a3cd1fdaeb6bb2324be1e5a404e Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Thu, 10 Sep 2026 10:56:10 +0100 Subject: [PATCH 105/110] Simplify cleanup logic and ensure resource disposal Refactored post-benchmark cleanup to remove ExceptionDispatchInfo usage. Cleanup and event notification now occur in a finally block, guaranteeing artifact removal and stage completion even if disposal throws, simplifying error handling and ensuring consistent resource management. --- .../Running/BenchmarkRunnerClean.cs | 25 +++++++------------ 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/src/BenchmarkDotNet/Running/BenchmarkRunnerClean.cs b/src/BenchmarkDotNet/Running/BenchmarkRunnerClean.cs index 042a3381d2..30bc2a92df 100644 --- a/src/BenchmarkDotNet/Running/BenchmarkRunnerClean.cs +++ b/src/BenchmarkDotNet/Running/BenchmarkRunnerClean.cs @@ -22,7 +22,6 @@ using Perfolizer.Horology; using System.Collections.Immutable; using System.Runtime.CompilerServices; -using System.Runtime.ExceptionServices; using System.Text.RegularExpressions; using RunMode = BenchmarkDotNet.Jobs.RunMode; @@ -200,27 +199,21 @@ private static async ValueTask RunCore(BenchmarkRunInfo[] benchmarkRu // see https://github.com/dotnet/BenchmarkDotNet/issues/1383 and https://github.com/dotnet/runtime/issues/314 for more // // DisposeAllAsync goes through every value before it reports a failure, so one that threw leaves - // nothing else undisposed - and must not skip the teardown below either. Whoever is listening is told - // that the run stage ended whatever the disposal did, and the failure is raised after them. - ExceptionDispatchInfo? disposeFailure = null; - + // nothing else undisposed - and must not take the teardown down with it either: whoever is listening + // is told that the run stage ended whatever the disposal did. try { await benchmarkRunInfos.DisposeAllAsync().ConfigureAwait(); } - catch (Exception exception) + finally { - disposeFailure = ExceptionDispatchInfo.Capture(exception); - } - - compositeLogger.WriteLineHeader("// * Artifacts cleanup *"); - Cleanup(compositeLogger, new HashSet(artifactsToCleanup.Distinct())); - compositeLogger.WriteLineInfo("Artifacts cleanup is finished."); - compositeLogger.Flush(); + compositeLogger.WriteLineHeader("// * Artifacts cleanup *"); + Cleanup(compositeLogger, new HashSet(artifactsToCleanup.Distinct())); + compositeLogger.WriteLineInfo("Artifacts cleanup is finished."); + compositeLogger.Flush(); - eventProcessor.OnEndRunStage(); - - disposeFailure?.Throw(); + eventProcessor.OnEndRunStage(); + } } } From 69d84f87fa2bc4585a3526b9cf51ed51a22794ac Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 12 Sep 2026 14:03:00 +0100 Subject: [PATCH 106/110] Enable internals access for integration test assembly Add InternalsVisibleTo for BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals using BenchmarkDotNetInfo.PublicKey. This allows direct testing of Microsoft.Testing.Platform internals, supporting scenarios like test execution filters and incomplete application requests not accessible via a real test host. Added comments to clarify intent. --- src/BenchmarkDotNet.TestAdapter/Properties/AssemblyInfo.cs | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 src/BenchmarkDotNet.TestAdapter/Properties/AssemblyInfo.cs diff --git a/src/BenchmarkDotNet.TestAdapter/Properties/AssemblyInfo.cs b/src/BenchmarkDotNet.TestAdapter/Properties/AssemblyInfo.cs new file mode 100644 index 0000000000..5b0bbc532c --- /dev/null +++ b/src/BenchmarkDotNet.TestAdapter/Properties/AssemblyInfo.cs @@ -0,0 +1,7 @@ +using BenchmarkDotNet.Properties; +using System.Runtime.CompilerServices; + +// Drives the adapter's Microsoft.Testing.Platform types directly, to cover what a real test host cannot reach: a +// test execution filter no platform this was built against can produce, and the end of an application whose request +// never completed. See that project's README. +[assembly: InternalsVisibleTo("BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals,PublicKey=" + BenchmarkDotNetInfo.PublicKey)] From 9bca8aed146fb3b0eab108d7efbfe308666e27e1 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 12 Sep 2026 14:03:34 +0100 Subject: [PATCH 107/110] Improve error reporting and resource disposal logic Refactored BenchmarkTestFramework to mark benchmarks as failed with clear messages for unrecognized filters, enhancing IDE feedback. Updated ParameterValueLifetime to track live request scopes, ensuring all parameter values are disposed properly and preventing resource leaks. Added comments to clarify resource management and error handling changes. --- .../TestingPlatform/BenchmarkTestFramework.cs | 43 +++++++++++++++++-- .../TestingPlatform/ParameterValueLifetime.cs | 27 +++++++++++- 2 files changed, 64 insertions(+), 6 deletions(-) diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs index 0c72702c66..452d95b6f6 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/BenchmarkTestFramework.cs @@ -157,10 +157,8 @@ private async Task RunAsync(RunTestExecutionRequest request, ExecuteRequestConte // request keeps the filter visible and costs nothing but a re-run once it is supported. if (enumeration.UnrecognisedFilter is { } unrecognisedFilter) { - throw new NotSupportedException( - $"BenchmarkDotNet.TestAdapter does not support the '{unrecognisedFilter.FullName}' test " + - "execution filter, and will not run every benchmark of the assembly in its place. Please " + - "report this at https://github.com/dotnet/BenchmarkDotNet/issues."); + await RefuseUnrecognisedFilterAsync(context, sessionUid, enumeration, unrecognisedFilter).ConfigureAwait(false); + return; } foreach (var benchmarks in enumeration.Matches) @@ -294,6 +292,43 @@ private async Task RunAsync( drainFailure?.Throw(); } + /// + /// Refuses a run whose filter this adapter does not know, reporting every benchmark it could have selected as + /// failed. + /// + /// + /// Throwing instead would leave the platform with nothing per benchmark to show: the request is completed by + /// the finally in before the exception is observed, so an IDE gets a run + /// that finished with no feedback on any test. A failed node per benchmark says it where the user is looking. + /// + private async Task RefuseUnrecognisedFilterAsync( + ExecuteRequestContext context, + SessionUid sessionUid, + Enumeration enumeration, + Type unrecognisedFilter) + { + var error = + $"BenchmarkDotNet.TestAdapter does not support the '{unrecognisedFilter.FullName}' test execution " + + "filter, so it cannot tell which benchmarks this run asked for, and it will not run every benchmark " + + "of the assembly in its place. Please report this at " + + "https://github.com/dotnet/BenchmarkDotNet/issues."; + + foreach (var benchmarks in enumeration.Matches) + { + context.CancellationToken.ThrowIfCancellationRequested(); + + var node = benchmarks[0].Node; + + await context.MessageBus.PublishAsync( + this, + new TestNodeUpdateMessage(sessionUid, node.ToTestNode(InProgressTestNodeStateProperty.CachedInstance))).ConfigureAwait(false); + + await context.MessageBus.PublishAsync( + this, + new TestNodeUpdateMessage(sessionUid, node.ToTestNode(new FailedTestNodeStateProperty(error)))).ConfigureAwait(false); + } + } + /// /// Tells the user that the request's filter is one this adapter does not know, and is being treated as /// matching everything. diff --git a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs index 9166caf157..29fa031b86 100644 --- a/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs +++ b/src/BenchmarkDotNet.TestAdapter/TestingPlatform/ParameterValueLifetime.cs @@ -48,6 +48,11 @@ internal sealed class ParameterValueLifetime : ITestHostApplicationLifetime private Dictionary held = new(ParameterValueDisposer.ByReference); private readonly HashSet disposedByBenchmarkDotNet = new(ParameterValueDisposer.ByReference); + // The scopes of the requests that have not completed yet. A request hands its values over by completing, so + // without this the values of one that never got there - the client sent `exit`, or the IDE cancelled, while + // the request was still in flight - would be reachable from nothing by the time the application ends. + private readonly HashSet live = []; + /// public string Uid => extension.Uid + ".ParameterValueLifetime"; @@ -67,7 +72,15 @@ internal sealed class ParameterValueLifetime : ITestHostApplicationLifetime /// Starts collecting the parameter values of one request. /// /// The scope to record that request's values in, and to complete when it is over. - public RequestScope BeginRequest() => new RequestScope(this); + public RequestScope BeginRequest() + { + var request = new RequestScope(this); + + lock (gate) + live.Add(request); + + return request; + } /// public Task BeforeRunAsync(CancellationToken cancellationToken) => Task.CompletedTask; @@ -79,12 +92,18 @@ public async Task AfterRunAsync(int exitCode, CancellationToken cancellationToke lock (gate) { + // Nothing can ask for any of these again. That includes the values of a request still in flight: the + // application is going away, so its scope will never be completed, and these are the only reference + // left to them - a value left to the finalizer instead is the dotnet/BenchmarkDotNet#1383 hang. unused = held + .Concat(live.SelectMany(request => request.Enumerated)) .Where(pair => !disposedByBenchmarkDotNet.Contains(pair.Key)) - .Select(pair => pair.Value) + .GroupBy(pair => pair.Key, ParameterValueDisposer.ByReference) + .Select(group => group.First().Value) .ToList(); held.Clear(); + live.Clear(); disposedByBenchmarkDotNet.Clear(); } @@ -97,6 +116,10 @@ private async ValueTask CompleteAsync(RequestScope request, IEnumerable Date: Sat, 12 Sep 2026 14:03:59 +0100 Subject: [PATCH 108/110] Enhance test coverage and internals probe support - Added project reference to BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals. - Introduced `InternalsProbe` constant in `TestingPlatformAdapterTests.cs`. - Improved test to count distinct benchmark types. - Added tests for unrecognized filters and parameter disposal. - Implemented `RunInternalsProbe` helper and `InternalsReport` record for probe execution and parsing. --- .../BenchmarkDotNet.IntegrationTests.csproj | 1 + .../TestingPlatformAdapterTests.cs | 99 ++++++++++++++++++- 2 files changed, 99 insertions(+), 1 deletion(-) diff --git a/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj b/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj index 4e07fe4d69..bf5a148547 100644 --- a/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj +++ b/tests/BenchmarkDotNet.IntegrationTests/BenchmarkDotNet.IntegrationTests.csproj @@ -48,6 +48,7 @@ + diff --git a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs index fc6e314efa..3caecaaf2a 100644 --- a/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs +++ b/tests/BenchmarkDotNet.IntegrationTests/TestingPlatformAdapterTests.cs @@ -18,6 +18,7 @@ public class TestingPlatformAdapterTests(ITestOutputHelper output) private const string PassingProbes = "BenchmarkDotNet.IntegrationTests.TestingPlatform"; private const string FailingProbes = "BenchmarkDotNet.IntegrationTests.TestingPlatform.Failures"; private const string UnoptimizedProbes = "BenchmarkDotNet.IntegrationTests.TestingPlatform.Unoptimized"; + private const string InternalsProbe = "BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals"; // Every probe project is single targeted, see their .csproj files. private const string ProbeTargetFramework = "net10.0"; @@ -255,7 +256,14 @@ public void AnAssemblyWideValidationWarningIsReportedOncePerNode() "Probe.Identity", Timeout); - Assert.True(ran.Count >= 4, $"Expected several benchmark types to run, but {ran.Count} node(s) did."); + // The dedup only does anything when more than one BenchmarkRunInfo is validated, so the benchmarks that + // ran have to span several types for this to be exercising it at all - which counting nodes would not say. + var types = ran + .Select(node => node.DisplayName.Substring(PassingProbes.Length + 1).Split('.')[0]) + .Distinct(StringComparer.Ordinal) + .ToArray(); + + Assert.True(types.Length >= 2, $"Expected several benchmark types to run, but only {string.Join(", ", types)} did."); Assert.All(ran, node => Assert.Equal("passed", node.ExecutionState)); Assert.All( ran, @@ -364,6 +372,95 @@ public void ABenchmarkIsAddressableWhenItsPathContainsAPropertyFilterDelimiter() Assert.Equal(byParameter[0].Uid, byJob[0].Uid); } + [Fact] + public void ADiscoveryWithAnUnrecognisedFilterListsEveryBenchmarkAndSaysSo() + { + // Microsoft.Testing.Platform 2.3.3 has no filter the adapter does not handle, and the extension point for + // adding one is internal to it, so this branch is unreachable from a real test host - FilterProbe drives + // the framework itself to reach it. Discovery runs nothing, so listing too much is the cheap mistake and + // reporting nothing is the expensive one; the warning is what makes the wrong list visible. + var report = RunInternalsProbe(); + + Assert.Contains( + report.Discover, + line => line.StartsWith("output ", StringComparison.Ordinal) + && line.Contains("does not recognise", StringComparison.Ordinal) + && line.Contains("UnrecognisedFilter", StringComparison.Ordinal)); + + // Everything the probe assembly declares, rather than a count that a benchmark added to it would break. + Assert.NotEmpty(report.Discovered); + Assert.Contains("complete True", report.Discover); + } + + [Fact] + public void ARunWithAnUnrecognisedFilterIsRefusedWithAFailedNodePerBenchmark() + { + // The other half of the same branch: a run cannot list too much, because it would spend the machine's next + // hour on it. Refusing by throwing would be invisible - the request is completed before the exception is + // observed - so every benchmark it could have selected is reported failed instead, which is where an IDE + // shows it. + var report = RunInternalsProbe(); + + var failed = report.Run + .Where(line => line.StartsWith("failed(", StringComparison.Ordinal)) + .ToArray(); + + // Every benchmark the same filter listed during discovery is reported, so that none of them is left + // looking like it was quietly skipped. + Assert.Equal(report.Discovered.Length, failed.Length); + Assert.All(failed, line => Assert.Contains("does not support", line, StringComparison.Ordinal)); + Assert.All(failed, line => Assert.Contains("UnrecognisedFilter", line, StringComparison.Ordinal)); + + // Every one of them was reported as started too, and the request finished rather than throwing. + Assert.Equal(failed.Length, report.Run.Count(line => line.StartsWith("in-progress ", StringComparison.Ordinal))); + Assert.DoesNotContain(report.Run, line => line.StartsWith("threw ", StringComparison.Ordinal)); + Assert.Contains("complete True", report.Run); + } + + [Fact] + public void ParameterValuesOfARequestStillInFlightAreDisposedWhenTheApplicationEnds() + { + // A request hands its values over by completing. One that never gets there - the client sent `exit`, or + // the IDE cancelled, while it was still in flight - leaves them reachable from nothing else, and a value + // left to the finalizer instead is the dotnet/BenchmarkDotNet#1383 hang this disposal exists to prevent. + var report = RunInternalsProbe(); + + Assert.Equal("created=2 disposed=2", Assert.Single(report.Abandoned)); + } + + /// + /// Runs the application that drives the adapter's platform types directly, and splits what it reported into + /// its sections. + /// + /// The lines of each section, and the benchmarks the discovery request listed. + private InternalsReport RunInternalsProbe() + { + var (exitCode, standardOutput) = Execute(InternalsProbe, []); + + Assert.Equal(0, exitCode); + + var lines = standardOutput.Split([Environment.NewLine], StringSplitOptions.RemoveEmptyEntries); + + var abandonedStart = Array.IndexOf(lines, "== abandoned"); + var discoverStart = Array.IndexOf(lines, "== discover"); + var runStart = Array.IndexOf(lines, "== run"); + var end = Array.IndexOf(lines, "== done"); + + Assert.True( + abandonedStart >= 0 && discoverStart > abandonedStart && runStart > discoverStart && end > runStart, + $"The internals probe did not report every section:{Environment.NewLine}{standardOutput}"); + + var discover = lines[(discoverStart + 1)..runStart]; + + return new InternalsReport( + lines[(abandonedStart + 1)..discoverStart], + discover, + lines[(runStart + 1)..end], + discover.Where(line => line.StartsWith("discovered ", StringComparison.Ordinal)).ToArray()); + } + + private sealed record InternalsReport(string[] Abandoned, string[] Discover, string[] Run, string[] Discovered); + [Fact] public void AnOutOfProcessBenchmarkIsBuiltAndRun() { From 0f83c8ede45819daf672a90ae96cd1278427dc15 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 12 Sep 2026 14:04:34 +0100 Subject: [PATCH 109/110] Add BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals Introduce a new project to test BenchmarkDotNet's Microsoft.Testing.Platform integration. Includes benchmarks for resource cleanup and filtering, a platform stub for simulation, and a test orchestrator. Adds project file and documentation detailing test scenarios not possible from a real test host. --- .../AbandonedBenchmarks.cs | 46 +++++++ ...tionTests.TestingPlatform.Internals.csproj | 25 ++++ .../FilteredBenchmarks.cs | 30 +++++ .../PlatformStub.cs | 115 ++++++++++++++++++ .../Program.cs | 100 +++++++++++++++ .../README.md | 26 ++++ 6 files changed, 342 insertions(+) create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/AbandonedBenchmarks.cs create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals.csproj create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/FilteredBenchmarks.cs create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/PlatformStub.cs create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/Program.cs create mode 100644 tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/README.md diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/AbandonedBenchmarks.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/AbandonedBenchmarks.cs new file mode 100644 index 0000000000..ed14864ba3 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/AbandonedBenchmarks.cs @@ -0,0 +1,46 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals +{ + /// + /// Benchmarks whose parameter values are disposable, used to check what becomes of the values of a request that + /// is still in flight when the application ends. + /// + [Config(typeof(FastConfig))] + public class AbandonedBenchmarks + { + // Created once, so that re-reading the source cannot change the count. + private static readonly Abandoned[] Instances = [new Abandoned(1), new Abandoned(2)]; + + public IEnumerable Values => Instances; + + [ParamsSource(nameof(Values))] + public Abandoned? Value { get; set; } + + [Benchmark] + public int Identity() => Value!.Number; + + public class Abandoned : IDisposable + { + public Abandoned(int number) => Number = number; + + public static int Created => Instances.Length; + + public static int Disposed { get; private set; } + + public int Number { get; } + + public void Dispose() => Disposed++; + + public override string ToString() => $"abandoned-{Number}"; + } + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals.csproj b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals.csproj new file mode 100644 index 0000000000..e848a7ef77 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals.csproj @@ -0,0 +1,25 @@ + + + + net10.0 + Exe + BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals + BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals + BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals + + + false + + + true + + + + + + + + diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/FilteredBenchmarks.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/FilteredBenchmarks.cs new file mode 100644 index 0000000000..9850bcf8d5 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/FilteredBenchmarks.cs @@ -0,0 +1,30 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Jobs; +using BenchmarkDotNet.Toolchains.InProcess.Emit; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals +{ + /// + /// The benchmarks this application's requests enumerate. They are never executed - the run request under test is + /// refused before anything is handed to BenchmarkDotNet - so what they do does not matter, only that there are + /// several of them to be reported about. + /// + [Config(typeof(FastConfig))] + public class FilteredBenchmarks + { + [Params(1, 2)] + public int Size { get; set; } + + [Benchmark] + public int Add() => Size + Size; + + [Benchmark] + public int Multiply() => Size * Size; + + private class FastConfig : ManualConfig + { + public FastConfig() => AddJob(Job.Dry.WithToolchain(InProcessEmitToolchain.Default)); + } + } +} diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/PlatformStub.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/PlatformStub.cs new file mode 100644 index 0000000000..4232021bac --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/PlatformStub.cs @@ -0,0 +1,115 @@ +using Microsoft.Testing.Platform.Extensions.Messages; +using Microsoft.Testing.Platform.Extensions.OutputDevice; +using Microsoft.Testing.Platform.Messages; +using Microsoft.Testing.Platform.OutputDevice; +using Microsoft.Testing.Platform.Requests; +using Microsoft.Testing.Platform.TestHost; +using System.Reflection; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals +{ + /// + /// The parts of Microsoft.Testing.Platform that surround a test framework while it serves one request, recorded + /// rather than acted on. + /// +#pragma warning disable TPEXP // Completing a request is still marked as experimental by the platform. + internal sealed class PlatformStub : IMessageBus, IOutputDevice, IExecuteRequestCompletionNotifier +#pragma warning restore TPEXP + { + private readonly List messages = []; + + /// + /// Gets the node states the framework published, as "<state> <display name>" lines, and the + /// output device data it displayed, as "output <text>" lines. + /// + public IReadOnlyList Messages => messages; + + /// + /// Gets whether the framework completed the request. + /// + public bool IsComplete { get; private set; } + + public Task PublishAsync(IDataProducer dataProducer, IData data) + { + if (data is TestNodeUpdateMessage update) + { + var state = update.TestNode.Properties.SingleOrDefault(); + + lock (messages) + messages.Add($"{StateName(state)} {update.TestNode.DisplayName}"); + } + + return Task.CompletedTask; + } + + public Task DisplayAsync(IOutputDeviceDataProducer producer, IOutputDeviceData data, CancellationToken cancellationToken) + { + var text = data switch + { + WarningMessageOutputDeviceData warning => warning.Message, + ErrorMessageOutputDeviceData error => error.Message, + TextOutputDeviceData plain => plain.Text, + _ => data.ToString() ?? string.Empty + }; + + lock (messages) + messages.Add($"output {text}"); + + return Task.CompletedTask; + } + + public void Complete() => IsComplete = true; + + /// + /// Builds the service provider the framework resolves the output device from. + /// + /// + /// GetOutputDevice casts to the platform's own provider rather than asking any + /// for the service, so this has to be that type - which the platform keeps to + /// itself - rather than something implemented here. + /// + /// The output device to resolve. + /// The created service provider. + public static IServiceProvider CreateServiceProvider(IOutputDevice outputDevice) + { + var type = typeof(IOutputDevice).Assembly.GetType("Microsoft.Testing.Platform.Services.ServiceProvider", throwOnError: true)!; + var provider = (IServiceProvider)Activator.CreateInstance(type)!; + var addService = type.GetMethod("AddService", BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic) + ?? throw new InvalidOperationException("The platform's service provider no longer has an AddService method."); + + addService.Invoke(provider, [outputDevice, false]); + + return provider; + } + + /// + /// Creates the session a request belongs to. The platform keeps the constructor to itself, and there is no + /// other way to build one of these from outside it. + /// + /// The created session context. + public static TestSessionContext CreateSessionContext() + { + var constructor = typeof(TestSessionContext).GetConstructor( + BindingFlags.Instance | BindingFlags.NonPublic, + binder: null, + [typeof(SessionUid)], + modifiers: null) + ?? throw new InvalidOperationException( + $"{nameof(TestSessionContext)} no longer has a constructor taking a {nameof(SessionUid)}."); + + return (TestSessionContext)constructor.Invoke([new SessionUid(Guid.NewGuid().ToString())]); + } + + private static string StateName(TestNodeStateProperty? state) => state switch + { + null => "none", + DiscoveredTestNodeStateProperty => "discovered", + InProgressTestNodeStateProperty => "in-progress", + PassedTestNodeStateProperty => "passed", + FailedTestNodeStateProperty failed => $"failed({failed.Explanation})", + ErrorTestNodeStateProperty error => $"error({error.Explanation})", + SkippedTestNodeStateProperty => "skipped", + _ => state.GetType().Name + }; + } +} diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/Program.cs b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/Program.cs new file mode 100644 index 0000000000..3328433481 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/Program.cs @@ -0,0 +1,100 @@ +using BenchmarkDotNet.Running; +using BenchmarkDotNet.TestAdapter.TestingPlatform; +using Microsoft.Testing.Platform.Capabilities.TestFramework; +using Microsoft.Testing.Platform.Extensions.TestFramework; +using Microsoft.Testing.Platform.Requests; +using System.Reflection; + +namespace BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals +{ + /// + /// Drives the adapter's Microsoft.Testing.Platform types the way a real test host cannot, and prints what they + /// did. See the README next to this file. + /// + internal static class Program + { + private static async Task Main() + { + // First, so that the values it counts have not already been disposed by the requests below - they are + // benchmarks of this assembly too, so those enumerate and dispose them as well. + Console.WriteLine("== abandoned"); + Console.WriteLine(await ReportAbandonedRequestAsync().ConfigureAwait(false)); + + Console.WriteLine("== discover"); + foreach (var line in await ExecuteAsync(session => new DiscoverTestExecutionRequest(session, new UnrecognisedFilter())).ConfigureAwait(false)) + Console.WriteLine(line); + + Console.WriteLine("== run"); + foreach (var line in await ExecuteAsync(session => new RunTestExecutionRequest(session, new UnrecognisedFilter())).ConfigureAwait(false)) + Console.WriteLine(line); + + Console.WriteLine("== done"); + + return 0; + } + + /// + /// Ends an application while a request is still in flight, which is what the client sending `exit` or an IDE + /// cancelling looks like from in here, and reports what became of the values that request had enumerated. + /// + /// + /// Nothing else holds them: the request hands its values over by completing, and this one never does. Left + /// undisposed they reach the finalizer, which is the dotnet/BenchmarkDotNet#1383 hang. + /// + /// The counts, as a line. + private static async Task ReportAbandonedRequestAsync() + { + var lifetime = new ParameterValueLifetime(); + var request = lifetime.BeginRequest(); + + request.Track(BenchmarkConverter.TypeToBenchmarks(typeof(AbandonedBenchmarks)).BenchmarksCases); + + // Deliberately no CompleteAsync: that is the point of this one. + await lifetime.AfterRunAsync(0, CancellationToken.None).ConfigureAwait(false); + + return $"created={AbandonedBenchmarks.Abandoned.Created} disposed={AbandonedBenchmarks.Abandoned.Disposed}"; + } + + private static async Task> ExecuteAsync(Func createRequest) + { + var platform = new PlatformStub(); + var lifetime = new ParameterValueLifetime(); + var framework = new BenchmarkTestFramework( + new TestFrameworkCapabilities(), + PlatformStub.CreateServiceProvider(platform), + Assembly.GetExecutingAssembly(), + lifetime); + + var request = createRequest(PlatformStub.CreateSessionContext()); +#pragma warning disable TPEXP // Building a request context is still marked as experimental by the platform. + var context = new ExecuteRequestContext(request, platform, platform, CancellationToken.None); +#pragma warning restore TPEXP + + // The framework is expected to deal with the filter rather than throw at the caller, which is the whole + // point: an exception here escapes after the request has already been completed, so the platform reports + // a finished request that said nothing about any benchmark. + try + { + await framework.ExecuteRequestAsync(context).ConfigureAwait(false); + } + catch (Exception exception) + { + return [.. platform.Messages, $"threw {exception.GetType().Name}: {exception.Message}"]; + } + finally + { + await lifetime.AfterRunAsync(0, CancellationToken.None).ConfigureAwait(false); + } + + return [.. platform.Messages, $"complete {platform.IsComplete}"]; + } + + /// + /// A filter of a kind the adapter has never been told about, which is what a newer Microsoft.Testing.Platform + /// or an extension registering its own filter factory would hand it. + /// + private sealed class UnrecognisedFilter : ITestExecutionFilter + { + } + } +} diff --git a/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/README.md b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/README.md new file mode 100644 index 0000000000..90bc6cdec1 --- /dev/null +++ b/tests/BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals/README.md @@ -0,0 +1,26 @@ +# BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals + +Drives the adapter's Microsoft.Testing.Platform types directly, to reach what a real test host cannot. + +The first of those is `BenchmarkTestFramework` with a test execution filter that `BenchmarkDotNet.TestAdapter` does not recognise, +which is the one thing the other probe applications cannot do: Microsoft.Testing.Platform 2.3.3 ships `NopFilter`, +`TestNodeUidListFilter` and `TreeNodeFilter`, the adapter handles all three, and `ITestExecutionFilterFactory` — the +extension point a consumer would register another one through — is internal to the platform. So the branch that +handles an unknown filter is unreachable from a real test host, and a regression in it would be silent. + +`ITestExecutionFilter` itself is public, so this application implements one, hands it to the framework through a +discovery request and a run request, and prints what the framework did with each. It is not a Microsoft.Testing.Platform +application: it has its own entry point, and stands in for the platform around the framework. + +The two requests are deliberately treated differently, which is what this pins: + +* **discovery** lists every benchmark and warns on the output device — a wrong list costs nothing to correct; +* **a run** refuses, reporting every benchmark it could have selected as failed — running the whole assembly instead + of the subset that was asked for would cost the machine the next hour. + +It also ends an application while a request is still in flight - the client sending `exit`, or an IDE cancelling - +which no driven test host can be made to do on cue either. The values that request had enumerated are reachable from +nothing else, so `ParameterValueLifetime` has to dispose them on its way out; left to the finalizer they are the +dotnet/BenchmarkDotNet#1383 hang. + +`TestingPlatformAdapterTests` in `BenchmarkDotNet.IntegrationTests` runs it and asserts on the report. From fd51095fec962abe3985bcd99edd4274a07d63a9 Mon Sep 17 00:00:00 2001 From: Ifeanyi Shadrach Odom Date: Sat, 12 Sep 2026 14:04:56 +0100 Subject: [PATCH 110/110] Add Internals project to BenchmarkDotNet.slnx Added BenchmarkDotNet.IntegrationTests.TestingPlatform.Internals project to the solution file for improved test coverage and integration. --- BenchmarkDotNet.slnx | 1 + 1 file changed, 1 insertion(+) diff --git a/BenchmarkDotNet.slnx b/BenchmarkDotNet.slnx index 852a4a36c6..26a5ea5a5d 100644 --- a/BenchmarkDotNet.slnx +++ b/BenchmarkDotNet.slnx @@ -46,6 +46,7 @@ +