Skip to content

Add slnf and slnx solution formats to run-tests skill#857

Merged
Evangelink merged 1 commit into
mainfrom
dev/amauryleve/curly-doodle
Jul 6, 2026
Merged

Add slnf and slnx solution formats to run-tests skill#857
Evangelink merged 1 commit into
mainfrom
dev/amauryleve/curly-doodle

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Summary

Extends the run-tests skill (plugins/dotnet-test/skills/run-tests) to document .slnf and .slnx solution formats alongside .sln.

Changes

  • Inputs table: the solution path description now reads .sln, .slnf, .slnx.
  • Examples: added explicit dotnet test --solution commands for .slnf and .slnx under the existing "(sln, slnf, slnx)" example.

Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 5, 2026 09:09
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Skill Coverage Report

Plugin Skill Covered Coverage
dotnet-test run-tests 16/16 100%

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the run-tests skill documentation to explicitly include additional solution file formats (.slnf, .slnx) alongside .sln, ensuring users know they can target these with the skill’s --solution workflow.

Changes:

  • Expanded the Inputs table description to list .sln, .slnf, and .slnx as supported solution formats.
  • Added example dotnet test --solution commands demonstrating .slnf and .slnx usage.
Show a summary per file
File Description
plugins/dotnet-test/skills/run-tests/SKILL.md Documents .slnf/.slnx solution formats in inputs and provides explicit example commands.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 1/1 changed files
  • Comments generated: 0

@Evangelink Evangelink enabled auto-merge (squash) July 5, 2026 09:13
@Evangelink

Copy link
Copy Markdown
Member Author

/evaluate

@github-actions github-actions Bot added the waiting-on-review PR state label label Jul 5, 2026
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Skill Validation Results

Skill Scenario Quality Skills Loaded Overfit Verdict
run-tests Run tests in a VSTest MSTest project 4.0/5 → 5.0/5 🟢 ✅ run-tests; tools: skill, bash / ⚠️ NOT ACTIVATED ✅ 0.13 [1]
run-tests Run tests with trx reporting on MTP project (SDK 9) 2.0/5 → 5.0/5 🟢 ✅ run-tests; tools: skill, report_intent, view ✅ 0.13
run-tests Run tests with blame-hang on MTP project (SDK 10) 2.3/5 → 4.0/5 🟢 ⚠️ NOT ACTIVATED / ✅ run-tests; tools: skill ✅ 0.13 [2]
run-tests Run tests on a specific TFM with TRX in a multi-TFM MTP project (SDK 9) 5.0/5 → 5.0/5 ⚠️ NOT ACTIVATED / ✅ run-tests; tools: skill ✅ 0.13 [3]
run-tests Filter MSTest tests by category on VSTest 5.0/5 → 4.7/5 🔴 ⚠️ NOT ACTIVATED / ✅ run-tests; tools: skill, glob ✅ 0.13 [4]
run-tests Filter NUnit tests by class name on VSTest 3.0/5 → 5.0/5 🟢 ✅ run-tests; tools: skill, view, bash / ⚠️ NOT ACTIVATED ✅ 0.13
run-tests Filter xUnit v3 tests by class on MTP 1.0/5 → 5.0/5 🟢 ✅ run-tests; tools: skill, view ✅ 0.13 [5]
run-tests Filter xUnit v3 tests by trait on MTP 1.0/5 → 5.0/5 🟢 ✅ run-tests; tools: skill, view / ⚠️ NOT ACTIVATED ✅ 0.13 [6]
run-tests Filter xUnit v3 tests by class pattern and trait using query filter language 1.0/5 → 1.0/5 ⚠️ NOT ACTIVATED ✅ 0.13 [7]
run-tests Filter TUnit tests by class using treenode-filter 2.3/5 → 4.0/5 🟢 ⚠️ NOT ACTIVATED ✅ 0.13 [8]
run-tests Combine multiple filter criteria on VSTest MSTest 4.0/5 → 5.0/5 🟢 ✅ run-tests; tools: skill, bash, report_intent, view / ✅ run-tests; tools: skill, report_intent, view ✅ 0.13 [9]
run-tests MTP project on SDK 9 must use -- separator for args 1.0/5 → 3.0/5 🟢 ⚠️ NOT ACTIVATED ✅ 0.13 [10]
run-tests MTP project on SDK 10 passes args directly 3.0/5 → 4.3/5 🟢 ⚠️ NOT ACTIVATED ✅ 0.13 [11]
run-tests Detect test platform from Directory.Build.props 1.3/5 → 5.0/5 🟢 ✅ run-tests; tools: skill / ⚠️ NOT ACTIVATED ✅ 0.13
run-tests Negative test: do not use MTP syntax for a VSTest project 4.0/5 → 5.0/5 🟢 ✅ run-tests; tools: skill, view / ⚠️ NOT ACTIVATED ✅ 0.13 [12]

[1] (Plugin) Quality unchanged but weighted score is -2.4% due to: tokens (25249 → 34603), time (10.5s → 12.6s)
[2] ⚠️ High run-to-run variance (CV=2105%) — consider re-running with --runs 5. (Isolated) Quality improved but weighted score is -28.2% due to: quality, judgment, tool calls (2 → 3), time (11.9s → 15.6s)
[3] ⚠️ High run-to-run variance (CV=61%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -7.1% due to: tokens (25489 → 55142), tool calls (2 → 3), time (14.3s → 19.2s)
[4] ⚠️ High run-to-run variance (CV=106%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -7.4% due to: tokens (29892 → 59681), tool calls (2 → 5), time (14.3s → 20.6s)
[5] ⚠️ High run-to-run variance (CV=191%) — consider re-running with --runs 5. (Plugin) Quality improved but weighted score is -7.1% due to: tokens (24952 → 43780), tool calls (1 → 2), time (6.8s → 9.1s)
[6] ⚠️ High run-to-run variance (CV=321%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -0.2% due to: tokens (37923 → 45128)
[7] (Isolated) Quality unchanged but weighted score is -9.4% due to: tokens (12635 → 26577), tool calls (0 → 2), time (8.6s → 15.2s)
[8] ⚠️ High run-to-run variance (CV=127%) — consider re-running with --runs 5
[9] ⚠️ High run-to-run variance (CV=191%) — consider re-running with --runs 5
[10] ⚠️ High run-to-run variance (CV=70%) — consider re-running with --runs 5
[11] ⚠️ High run-to-run variance (CV=1159%) — consider re-running with --runs 5
[12] ⚠️ High run-to-run variance (CV=118%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -13.8% due to: judgment, tokens (25590 → 34931)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 857 in dotnet/skills, download eval artifacts with gh run download 28735886995 --repo dotnet/skills --pattern "skill-validator-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/1dd3255dac6fd5318302306afa84a7bc7119dbfe/eng/skill-validator/src/docs/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

✅ Evaluation passed for 1dd3255. cc @dotnet/dotnet-testing — please review.

@Evangelink Evangelink merged commit 72af853 into main Jul 6, 2026
38 checks passed
@Evangelink Evangelink deleted the dev/amauryleve/curly-doodle branch July 6, 2026 04:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-on-review PR state label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants