Skip to content

Block --help and --list-tests for MTP test apps when passed via non-CLI channels#54487

Open
Evangelink wants to merge 4 commits into
dotnet:mainfrom
Evangelink:dev/amauryleve/block-help-listtests-mtp
Open

Block --help and --list-tests for MTP test apps when passed via non-CLI channels#54487
Evangelink wants to merge 4 commits into
dotnet:mainfrom
Evangelink:dev/amauryleve/block-help-listtests-mtp

Conversation

@Evangelink
Copy link
Copy Markdown
Member

@Evangelink Evangelink commented May 28, 2026

Fixes #50654.

Problem

When users pass --help, -?, or --list-tests to a Microsoft Testing Platform (MTP) test app through channels the .NET CLI doesn't observe, the SDK either crashes via FailFast or silently reports zero tests run. Affected channels:

  1. TestingPlatformCommandLineArguments MSBuild property
  2. RunArguments MSBuild property
  3. launchSettings.json's commandLineArgs

Fix

Added a pre-launch validation step in TestApplication.RunAsync that tokenizes args from all three sources and throws a GracefulException with a clear, actionable message identifying the offending option and its source. The user is told to pass the option directly to dotnet test instead.

The check is skipped when the user legitimately passed --help to dotnet test (TestOptions.IsHelp), and similarly for --list-tests during discovery (TestOptions.IsDiscovery).

Also added a catch (GracefulException) block in TestApplicationActionQueue.Read so the new error message is printed cleanly (no stack trace) and returns ExitCode.GenericFailure.

The existing OnCommandLineOptionMessages FailFast safety net is retained as an invariant guard for IPC-layer issues, but should no longer be reached for user-error scenarios.

Tests

Added 10 new test cases in GivenDotnetTestBuildsAndRunsHelp.cs:

  • PassingHelpOrListTestsViaTestingPlatformCommandLineArguments_ShouldFailWithClearError (6 cases: --help / -? / --list-tests × Debug / Release)
  • PassingHelpOrListTestsViaUnmatchedTokens_ShouldFailWithClearError (4 cases: --help / --list-tests × Debug / Release; -? excluded because the CLI parser intercepts it)

All 10 new tests pass; existing baseline tests (RunHelpOnTestProject_ShouldReturnExitCodeSuccess, DiscoverTestProjectWithCustomRunArguments*) still pass.

Localization

Added 4 entries to CliCommandStrings.resx and auto-populated the 13 .xlf files via dotnet build /t:UpdateXlf.

@Evangelink
Block --help and --list-tests for MTP when passed via non-CLI channels
936438c 
Fixes dotnet#50654.

When users pass --help, -?, or --list-tests to a Microsoft Testing
Platform (MTP) test app through channels the .NET CLI doesn't observe
(TestingPlatformCommandLineArguments MSBuild property, RunArguments, or
launchSettings.json's commandLineArgs), the SDK previously either
crashed via FailFast or silently reported zero tests run.

This change adds a pre-launch validation step in TestApplication.RunAsync
that tokenizes args from all three sources and throws a GracefulException
with a clear, actionable message identifying the offending option and
the channel it came through.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 28, 2026 11:52
@Evangelink Evangelink requested a review from a team as a code owner May 28, 2026 11:52
Copilot AI reviewed May 28, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR prevents MTP test apps from receiving --help, -?, or --list-tests through argument channels that bypass normal dotnet test parsing, replacing crash/zero-test behavior with a clear user-facing failure.

Changes:

  • Adds pre-launch validation for forbidden test-app arguments from CLI passthrough, MSBuild RunArguments, and launch profile args.
  • Handles GracefulException cleanly in the MTP test application queue.
  • Adds localized resource strings and regression tests for key argument sources.
Show a summary per file
File Description
src/Cli/dotnet/Commands/Test/MTP/TestApplication.cs Adds forbidden-option validation and tokenization before launching test apps.
src/Cli/dotnet/Commands/Test/MTP/TestApplicationActionQueue.cs Prints GracefulException messages without wrapping/stack traces.
src/Cli/dotnet/Commands/CliCommandStrings.resx Adds user-facing error/source strings.
src/Cli/dotnet/Commands/xlf/CliCommandStrings.cs.xlf Adds localization units for the new strings.
src/Cli/dotnet/Commands/xlf/CliCommandStrings.de.xlf Adds localization units for the new strings.
src/Cli/dotnet/Commands/xlf/CliCommandStrings.es.xlf Adds localization units for the new strings.
src/Cli/dotnet/Commands/xlf/CliCommandStrings.fr.xlf Adds localization units for the new strings.
src/Cli/dotnet/Commands/xlf/CliCommandStrings.it.xlf Adds localization units for the new strings.
src/Cli/dotnet/Commands/xlf/CliCommandStrings.ja.xlf Adds localization units for the new strings.
src/Cli/dotnet/Commands/xlf/CliCommandStrings.ko.xlf Adds localization units for the new strings.
src/Cli/dotnet/Commands/xlf/CliCommandStrings.pl.xlf Adds localization units for the new strings.
src/Cli/dotnet/Commands/xlf/CliCommandStrings.pt-BR.xlf Adds localization units for the new strings.
src/Cli/dotnet/Commands/xlf/CliCommandStrings.ru.xlf Adds localization units for the new strings.
src/Cli/dotnet/Commands/xlf/CliCommandStrings.tr.xlf Adds localization units for the new strings.
src/Cli/dotnet/Commands/xlf/CliCommandStrings.zh-Hans.xlf Adds localization units for the new strings.
src/Cli/dotnet/Commands/xlf/CliCommandStrings.zh-Hant.xlf Adds localization units for the new strings.
test/dotnet.Tests/CommandTests/Test/GivenDotnetTestBuildsAndRunsHelp.cs Adds regression tests for forbidden options via MSBuild property and CLI passthrough.

Copilot's findings

  • Files reviewed: 17/17 changed files
  • Comments generated: 2

Comment thread test/dotnet.Tests/CommandTests/Test/GivenDotnetTestBuildsAndRunsHelp.cs Outdated
Comment thread src/Cli/dotnet/Commands/Test/MTP/TestApplication.cs
Evangelink and others added 3 commits May 28, 2026 14:24
@Evangelink
Address PR review: sanitize test identifiers and add launchSettings t…
6796b73 
…ests

- Sanitize forbiddenOption before using it in CopyTestAsset identifier:
  '-?' was producing a test-asset directory containing '?', invalid on
  Windows file systems.
- Add PassingHelpOrListTestsViaLaunchSettings_ShouldFailWithClearError
  to cover the launchSettings.json commandLineArgs validation path.
- Add PassingHelpInLaunchSettings_IsBypassedByNoLaunchProfileArguments
  to verify '--no-launch-profile-arguments' opts out of the check.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Evangelink
Defensive: split TokenizeArguments on ';' as well as whitespace
1711673 
Addresses a reviewer concern about MSBuild list-separator semantics.
While the OS argv parser does not in practice split on ';' (verified
empirically with a dotnet exec test), splitting defensively here is
zero-cost (the three forbidden tokens never appear inside legitimate
arg values) and future-proofs the validation against any upstream
pre-processing that might.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Evangelink
Tighten bypass test assertion to verify our validation didn't fire
0247f87 
The previous '.NotBe(GenericFailure)' assertion passed coincidentally
because the test asset's Program.cs throws on missing
'--from-launch-settings', producing exit code -532462766 (.NET unhandled
exception) rather than 1. That meant if the bypass logic silently broke,
the test would still pass.

Now assert directly that the launch-settings validation error fragment
does not appear in any output and that the test app actually proceeded
to its own pre-launch check (proving '--help' was dropped from argv).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Evangelink Evangelink force-pushed the dev/amauryleve/block-help-listtests-mtp branch from 0cb5efc to 0247f87 Compare May 28, 2026 12:44
This was referenced May 28, 2026
Open
Open
Open
Open
@Evangelink
Copy link
Copy Markdown
Member Author

Evangelink commented May 31, 2026

@copilot resolve the merge conflicts in this pull request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

Area-CLI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Block --help (probably --list-tests as well) for MTP when not known by .NET CLI

2 participants

@Evangelink