Adding agentic-perf and eval skills#839
Open
leslierichardson95 wants to merge 83 commits into
Open
Conversation
Installs an always-on rules block into a .NET MAF/Aspire/Foundry agentic app's instructions file (.github/copilot-instructions.md) so coding agents volunteer perf concerns by default — agent count, handoff edges, per-agent model selection, message-history strategy, per-turn token cost, and post-change measurement. The managed block is delimited by sentinel HTML comments and embeds the skill version so updates can preserve user-edited threshold values cleanly. Includes SKILL.md (201 lines), three reference docs (managed-block template, threshold defaults, per-rule rationales), eval.yaml with 5 scenarios covering install / append-preserving / idempotent / version-update / AGENTS.md-stub flows, and a CODEOWNERS entry matching the dotnet-ai convention (@leslierichardson95 @mikekistler — co-owner placeholder). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Read-only audit skill for .NET agentic apps. Runs seven check classes (topology, tool inventory, message-history strategy, prompt weight, parallelism, OTel coverage, model assignment) and writes a severity-tagged Markdown report to .copilot/perf-reports/audit-<timestamp>.md plus latest-audit.md. Findings carry file:line citations, evidence, a one-paragraph why, and a concrete next action that can route into select-agent-models, setup-maf-evals, or configure-agentic-perf-rules. Never edits source. Includes SKILL.md, eight reference docs, eval.yaml with five scenarios (clean / sprawl / full-history / prompt-bloat / missing-otel), and a CODEOWNERS entry matching the dotnet-ai convention. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Recommends per-agent model assignments based on classified roles (router, planner, decomposer, worker, validator, formatter, summarizer) and a curated role->model matrix balancing latency, quality, and cost. Two modes: read-only recommend (default, writes .copilot/perf-reports/model-plan-<timestamp>.md) and apply (diff-preview-and-confirm). Never applies without explicit confirmation. Includes SKILL.md, role-model matrix, AppHost multi-client + per-agent resolution templates, plan template, and eval.yaml with four scenarios (recommend mode / single-model defaulting / single-agent abort / apply mode requires confirmation). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scaffolds an <App>.Evals project alongside an existing .NET agentic app (MAF + Aspire + Foundry) with three modes: telemetry (per-call latency / tokens / cost), quality (LLM-as-judge against a rubric and golden conversations), and compare (two model assignments side by side). Outputs Markdown + JSON + JUnit-XML to .copilot/perf-reports/evals/. Optionally adds an Aspire-served static-file panel showing per-agent token/latency live during dev (apply mode required for AppHost edits). Includes SKILL.md, project template, telemetry capture, quality modes, compare mode, optional dashboard panel, and eval.yaml with four scenarios (scaffold fresh / skip non-agentic / stub run / update preserves user edits). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Companion architect agent for .NET agentic apps (MAF + Aspire + Foundry). Runs a three-pass review: Pass 1 — direct read (no skills): inventory, topology, obvious smells. Pass 2 — deep audit: orchestrates audit-agentic-app-perf, then routes into select-agent-models / setup-maf-evals / configure-agentic-perf-rules based on the report. Pass 3 — synthesis: prioritized action list with skill routes, expected effects, and validation steps. Registers the agent path in plugins/dotnet-ai/plugin.json (and the .codex-plugin manifest) using an explicit file path per the validator. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses critique findings: - project-template.md: remove invalid semicolons in <None Update> attribute lists (was breaking MSBuild parse), add explicit Version attributes to PackageReferences with CPM fallback note, add ProjectReference to agent-under-test, define IEvalRunner + EvalReport contract that Program.cs depends on. - SKILL.md: split step 1 into discover + update-mode matrix that explicitly classifies infra vs user-data vs generated files. - SKILL.md: rewrite step 7 (Aspire panel) as an explicit apply-mode step with diff-preview-and-confirm; declined path scaffolds files outside the AppHost. - telemetry-capture.md: split EVAL_USE_REAL_MODELS into two toggles EVAL_USE_REAL_AGENT and EVAL_USE_REAL_JUDGE so quality/compare modes have a coherent stub story. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses critique findings: - SKILL.md step 4: drop the .gitignore edit instruction. The skill now only writes under .copilot/perf-reports/. Recommendation about ignoring the report folder is surfaced in chat only. - SKILL.md step 3: add explicit evidence gate that requires re-reading the cited file before emitting a finding. Adds check_id (T1/T2/...) to the schema for deterministic referencing across reports and reviews. - SKILL.md pitfalls: spell out which files are off-limits explicitly (.gitignore, source, config) and require evidence verification. - references/report-template.md: render check_id in finding heading. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses critique findings: - Rename heading from Two-Pass to Three-Pass to match the actual number of passes defined. - Remove edit from the tool list. Agent claims no source edits in its boundaries; the tool is the actual mechanism by which it could violate that boundary, so removing it eliminates the footgun. - Add explicit exclusion in the description so routing prefers optimizing-dotnet-performance for non-agentic .NET perf reviews. - Pass 1 step 1: define detection signals concretely (AppHost, Microsoft.Agents.AI / Microsoft.Extensions.AI references, ChatClientAgent, Foundry config) so Pass 1 can stop cleanly when the project is not agentic. - Pass 2 step 2: replace MA-/OTel-category language with the actual report schema (check_id MA1-MA4, O1-O4) and require the agent to read the report file rather than infer from memory. - Pass 3 step 3: require qualitative effects unless setup-maf-evals has produced a citable report; no fabricated numeric estimates. - Boundaries: clarify apply-mode chaining — user intent in initial turn is not user confirmation; the invoked skill still owns its own confirmation flow. - Add explicit quick-triage escape hatch so Pass 2 isn't unconditional when the user has asked for a fast read. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses critique findings: - SKILL.md frontmatter: add explicit 'version: 0.1.0' field as the authoritative source. Update SKILL.md step 2 references to read the version from frontmatter rather than assume. - SKILL.md step 2: spell out sentinel parse rules with exact full-line regexes for BEGIN and END, refuse-on-malformed contract (zero/multiple/mismatched/out-of-order sentinels), and a same-version-but-structurally-invalid path that no longer silently no-ops when the block is missing rule headings or has unparseable threshold YAML. - SKILL.md step 2: replace the brief 'merge thresholds' phrase with a concrete five-step preservation algorithm (parse, validate types, override known keys, drop unknown with warning, write). - SKILL.md step 2: add path-safety rules (resolve project root, reject paths that normalize outside it, reject absolute paths and .. segments). - SKILL.md: add target-file selection table for the AGENTS.md vs .github/copilot-instructions.md case so 'both have a managed block' refuses cleanly instead of doubling up. - SKILL.md frontmatter: relax 'with Aspire and Foundry' to 'optionally with Aspire/Foundry' so MAF-only projects route correctly (matches the eval fixture). - references/managed-block-template.md: switch outer fence from three to four backticks so the inner three-backtick yaml fence in the rendered template is no longer ambiguous to Markdown parsers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses critique findings: - SKILL.md step 6: split apply mode into seven explicit sub-steps. - Add provider-resolution step that distinguishes public OpenAI from Azure OpenAI / Foundry deployments by inspecting AppHost connection string and appsettings keys, and refuses to write a public OpenAI id into an Azure project's appsettings.json. The agent must ask the user which deployment alias maps to each recommended role or stop apply mode if no suitable alias exists. - Add explicit confirmation contract that distinguishes the initial apply request (intent) from the confirmation that follows the diff preview, so phrasings like 'apply and confirm' or 'just do it' do not collapse the two-step contract. - Add pre-write parse / build-dryrun validation step before any write. - Add atomic-write contract: if any write fails midway, restore all touched files from their pre-write content. - Add build-failure rollback: on dotnet build failure after apply, revert all written changes and report apply failed (build) instead of declaring success. - references/role-model-matrix.md: add router sub-types section that promotes the router from gpt-4o-mini to gpt-4o / o4-mini when the router generates tool args, validates input schema, has more than five destinations, or routes to expensive workflows. - references/role-model-matrix.md: add planner decision criteria for o4-mini vs gpt-4o based on dependency depth, latency budget, and output structure. - references/role-model-matrix.md: add multi-role classification table for agents that fit more than one role. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Dogfood pass against interview-coach-v2 / behavioral-interview-coach surfaced that the project-template.md illustrative versions (9.4.0-preview.1.*, Microsoft.Extensions.* 9.0.0) were significantly stale. Verified via 'dotnet package search Microsoft.Extensions.AI.Evaluation' that the current GA family is 10.7.0 across: - Microsoft.Extensions.AI - Microsoft.Extensions.AI.Evaluation - Microsoft.Extensions.AI.Evaluation.Quality - Microsoft.Extensions.AI.Evaluation.Reporting Switch the template to 10.* floating versions and clarify in the lead-in that the listed numbers reflect the latest stable family (not just an arbitrary placeholder). The skill body still requires the agent to query nuget.org at scaffold time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
User feedback: the separate category field was confusing because the check_id prefix already encodes the category (T=topology, MH=history, etc.). Also: the skill produced a report and stopped without offering to route into the apply-capable downstream skills. Changes: - SKILL.md step 3: remove category field from finding schema. The check_id prefix is now the sole category indicator. Add a glossary table mapping each prefix to its category and reference doc. - SKILL.md step 4: sort findings by check_id within severity bands instead of by category. - SKILL.md step 5: render findings with check_id + title (no parenthetical category). - SKILL.md add step 6 (Offer to route): after surfacing top findings, ask the user once whether they want to follow up on any routed ref: targets. Render only the lettered options that correspond to refs actually present in the findings. Explicitly does not edit source — just routes into the appropriate sibling skill which owns its own diff-and-confirm flow. Apply-mode chaining language mirrors the agent: intent is not confirmation. - SKILL.md step 7: previous step 6 (Stop) renumbered. - references/report-template.md: render finding heading as '[severity] [check_id] title' and add a glossary line under the ## Findings heading. Update the per-finding block to drop the separate Check field. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… follow-ups in Pass 3 Pairs with the audit-agentic-app-perf change that drops the separate category field. The agent's Pass 2 routing rules now read the check_id prefix (MA*, O*, etc.) instead of looking for a 'category' column that no longer exists in the report. Pass 3 also gains a final sub-step that asks the user once whether to run any of the routed skills now, mirroring the new step 6 in the audit skill. Apply-mode chaining language is preserved: the invoked skill still owns its own diff-and-confirm. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
User feedback: 'audit' overlaps with security audits and the companion agentic-perf-reviewer agent's review framing. 'Scan' is shorter, conveys the read-only sweep more clearly, and avoids the overlap. Internal narrative still uses 'audit' as a synonym where natural; trigger phrases keep both. - Rename plugins/dotnet-ai/skills/audit-agentic-app-perf -> scan-agentic-app-perf - Rename tests/dotnet-ai/audit-agentic-app-perf -> scan-agentic-app-perf - SKILL.md: name field, # heading, lead description verb (Scan...), trigger phrases (added 'scan my agentic app', kept 'audit my... agentic app' for back-compat search) - Report filenames: audit-<timestamp>.md -> scan-<timestamp>.md, latest-audit.md -> latest-scan.md (changes the on-disk artifact the agent expects to read; agent.md updated on its branch) - references/report-template.md: filename refs - eval.yaml: skill key + prompts - CODEOWNERS: path entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pairs with the rename on skills/audit-agentic-app-perf branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pairs with the rename on skills/audit-agentic-app-perf branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pairs with the rename on skills/audit-agentic-app-perf branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pairs with the rename on skills/audit-agentic-app-perf branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…detection) Rewrites SKILL.md and references to specify the v2 overhaul: - Replaces hand-rolled markdown runner with a Microsoft.Extensions.AI.Evaluation.Reporting + `aieval` HTML report pipeline (GA 10.7.0). - Switches the scaffolded project from a console runner to an MSTest `<App>.Evals.Tests` project matching the canonical Learn-docs pattern. - Categorizes evaluators into three independent tiers (NLP / Quality / Safety) with separate env knobs (EVAL_USE_REAL_AGENT, EVAL_USE_REAL_JUDGE, EVAL_USE_FOUNDRY_SAFETY). - Adds IChatClient auto-detection so generated AgentChatClientFactory wires to the app's existing chat client registration. - Adds opt-in Safety tier (ContentHarmEvaluator via Azure AI Foundry) and opt-in GitHub Actions workflow. - New eval.yaml has 8 scenarios covering scaffold, update-mode preservation, IChatClient detection, reporting wiring, schema v2, CI workflow, safety opt-in. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three gaps surfaced when scaffolding ELI5Agent.Evals.Tests against an Aspire 13.2 + Foundry app: 1. **Detection table missing the Aspire Inference pattern.** `builder.AddAzureChatCompletionsClient(\"chat\").AddChatClient(\"chat\")` is the standard Aspire 13.2 way of wiring an IChatClient against a Foundry chat deployment, and was not in the v2 detection table. Added to ichatclient-detection.md with a note that the argument is the connection-string name. 2. **Connection-string setup not surfaced for standalone runs.** When the app uses Aspire orchestration, `ConnectionStrings:<alias>` is populated by the AppHost — but `dotnet test` runs outside the host and gets a silent missing-config NRE on first real-agent run. Added a `Connection-string setup for standalone test runs` section to ichatclient-detection.md that surfaces both user-secrets and env-var setup paths and points to `azd env get-values`. 3. **WordCountEvaluator implementation not pinned.** The catalog called the evaluator out as `always scaffolded (custom)` but never gave a verbatim template. Added the Learn-doc canonical implementation to evaluators-catalog.md so every scaffold gets the same (correct) IEvaluator skeleton. Validated end-to-end against ELI5Agent: scaffolded ELI5Agent.Evals.Tests, `dotnet test` exits 0 in stub tier with 4 scenarios x 4 metrics (Words/BLEU/GLEU/F1) producing a 670 KB report.html. Real-judge tier deferred (no Azure creds set up locally for ELI5Agent — itself the trigger for finding 2). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…gnostic 1. New references/metrics-glossary.md authored as the source of truth for metric definitions/scales/thresholds across NLP, Quality, and Safety tiers. Includes the canonical Reporting/MetricsGlossary.cs template that the skill emits into <App>.Evals.Tests/Reporting/. 2. Factory template in references/ichatclient-detection.md now wraps DI resolution in try/catch and throws a friendly InvalidOperationException naming the connection-string key + the exact 'dotnet user-secrets set' command + the env-var alternative + 'azd env get-values' pointer. Replaces the silent NRE that ELI5Agent dogfooding hit when EVAL_USE_REAL_AGENT=1 with no creds set up. 3. Two new eval.yaml scenarios bind the new behavior: - scaffold-emits-metrics-glossary: asserts MetricsGlossary.cs exists, references metrics-glossary.md, has [AssemblyCleanup] - factory-emits-friendly-secrets-diagnostic: asserts AgentChatClientFactory mentions 'dotnet user-secrets' + 'ConnectionStrings' 4. Captured two MSTest constraints that turned up while implementing the glossary writer: - quality-modes.md ReportingConfig: ExecutionName must be cached at class load, NOT re-evaluated per call (otherwise AievalReport and MetricsGlossary land in different timestamped folders 5 s apart) - metrics-glossary.md template now declares MetricsGlossary as a plain static class; MSTest forbids more than one [AssemblyCleanup] per assembly (UTA014). Glossary write is chained from AievalReport's single AssemblyCleanup, wrapped in try/catch so a glossary-write failure doesn't mask the report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Spec was creeping toward the validator's 'comprehensive' threshold (3,574 BPE tokens / 14,238 chars / 265 lines). Pulled prose-heavy sections into references and kept SKILL.md focused on decision-relevant content. - Step 3 (Scaffold): replaced 27-line file tree with a one-sentence summary + link to project-template.md. Kept the post-write powershell as a fenced code block so the agent has concrete commands. - Steps 4-9 (telemetry/quality/compare/safety/panel/CI): collapsed bulleted prose into 3-4 line stubs that retain the decision facts (default ON/OFF, opt-in semantics, env knob) and link the corresponding reference. - Step 11 (Surface in chat): added the metrics-glossary.md path to the 'Paths' bullet and to the trailing 'see also' line. - ## Common pitfalls: extracted to references/common-pitfalls.md (also adds two new entries from this work: the multi-AssemblyCleanup MSTest constraint and the AgentChatClientFactory friendly-NRE pattern). SKILL.md keeps a one-line link. - ## References: trimmed each bullet to one short line; consolidated the three external links onto a single line. Net effect: SKILL.md 14,238 -> 10,449 chars, 265 -> 189 lines, 3,574 -> 2,672 BPE tokens (-25%). Validator still passes; eval.yaml scenarios are unaffected (assertions are file-based, not prose-based). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Dogfood findings against ELI5Agent surfaced 3 real failure modes when promoting from stub to judge tier: 1. user-secrets silently not loading: dotnet test runs under testhost.exe as the entry assembly, so Host.CreateApplicationBuilder() does NOT pick up the secrets store keyed off the test project's UserSecretsId. Fix: factory template now calls builder.Configuration.AddUserSecrets(typeof(...).Assembly, optional: true). 2. services.ai.azure.com hostname strips dashes from the resource name (foundry-abc -> foundryabc.services.ai.azure.com). The legacy properties.endpoint value points at cognitiveservices.azure.com which 404s for the /models route. 3. Foundry resources provisioned by Aspire/azd usually have disableLocalAuth=true. Key-based auth returns 403; drop the Key= segment and rely on DefaultAzureCredential. Updates: - references/ichatclient-detection.md: factory template adds AddUserSecrets; diagnostic message now lists Entra and Key options + endpoint gotchas. - references/project-template.md: GlobalUsings.cs adds Microsoft.Extensions .Configuration / .DependencyInjection / .Hosting (needed by the factory). - references/common-pitfalls.md: 3 new entries covering the above. - SKILL.md: step 11 now surfaces the exact 2-command judge-promotion path. - tests/dotnet-ai/setup-maf-evals/eval.yaml: new factory-loads-user-secrets-explicitly scenario. Verified end-to-end: ELI5Agent judge tier 4/4 passing in 2m 24s, report.html shows 9 metric families (Words/BLEU/GLEU/F1 + Relevance/ Coherence/Fluency/Completeness/Equivalence). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Dogfooding the Judge tier against ELI5Agent's gpt-5-mini judge surfaced a silent failure: reasoning models (gpt-5*, o-series) reject the max_tokens parameter that Azure.AI.Inference still sends, returning HTTP 400 unsupported_parameter. The MEAI Quality evaluators swallow the 400 and record it as a per-metric error row -- tests exit 0 but every Quality column is an error. Pitfalls doc now covers: - which model families are affected (gpt-5*, o1/o3/o-series) - how the failure manifests (test pass + errors in report.html, not a hard test failure) - the workaround (pick a non-reasoning judge: gpt-4o / gpt-4o-mini / gpt-4-turbo; if the agent uses reasoning, set EVAL_JUDGE_DEPLOYMENT_NAME=<non-reasoning-alias> to split agent and judge deployments) - az CLI snippet to list deployment model families SKILL.md step 11's judge-promotion block now flags this alongside the existing dash/key-auth gotchas. eval.yaml gets a smoke assertion that common-pitfalls.md mentions both max_tokens and the env-var workaround. Verified end-to-end: re-pointed ELI5Agent at a gpt-4o-mini Foundry deployment, cleared _store/, re-ran judge tier: 4/4 passing in 44s, clean report (no max_tokens / unsupported_parameter strings anywhere), all 9 metric families populate (4 NLP + Relevance / Coherence / Fluency / Completeness / Equivalence). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
/evaluate --run 5 |
Contributor
Skill Validation Results
[1]
Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
…st doc; make dotnet steps deferrable Round-2 eval showed the isolated scaffold arm still read 11-14 reference docs and created zero files (then hit the 600s timeout), because the default-mode file bodies were scattered across 6+ refs. Reading them all exhausts the turn budget before anything is written. - Add references/default-scaffold.md: the complete, copy-pasteable body of every default-mode file (Telemetry + Quality + NLP + Reporting + Wire) in create order, so the agent reads ONE doc and creates the whole project. Defines the previously-undefined helpers (Wire, ProjectRoot/RepoRoot, loaders, Thresholds, records) so the emitted project is self-consistent. - SKILL.md step 3: read default-scaffold.md once and create files first; mark the network-bound dotnet steps (add package / tool install / build / test) as explicitly deferrable. Success = files on disk, not a green build. dotnet new stays the default shell-creation step. - project-template.md: point at default-scaffold.md for the default modes. Verified: fresh extraction of the doc into a throwaway MSTest project builds clean and 6/6 tests pass offline in stub tier; reports write and the aieval tool absence degrades gracefully. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…isk deliverable Reorder the execution-discipline block so the agent's FIRST write is the scan.md report (right after inventory, before any category check), add an explicit 'Now create the report file' instruction at the end of step 1, and reframe step 2 to discourage pre-reading all seven reference docs (the top cause of runs that finish with no report written). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tivation
Lead the description with the WHEN trigger phrases ('set up evals', 'add
evaluation harness/coverage', ...) so the skill activates more reliably in
isolated selection, while preserving all assertion-relevant tokens
(max_completion_tokens, stylistic/summarizer, EVAL_USE_REAL_AGENT). Stays
under the 1024-char limit (1019).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment on lines
+1
to
+5
| --- | ||
| name: setup-maf-evals | ||
| description: | | ||
| Scaffold a .NET agentic-app (MAF; Aspire/Foundry optional) evaluation harness — `<App>.Evals.Tests` MSTest project wired to the GA Microsoft.Extensions.AI.Evaluation (MEAI) reporting pipeline. WHEN: "set up evals", "add evaluation harness/coverage", "validate quality after a model change", "compare gpt-4o vs gpt-4o-mini", "add safety evaluators"; or troubleshooting — "why are my Quality columns erroring/empty", "reasoning model breaks evals / max_tokens vs max_completion_tokens", "which evaluators fail my stylistic/summarizer agent". Categories: **NLP** (BLEU/GLEU/F1, no key), **Quality** (LLM-as-judge), **Safety** (content-harm via Foundry). Auto-installs `aieval`, detects `IChatClient`, generates a factory (`EVAL_USE_REAL_AGENT=1`), emits an HTML report; optional PR workflow. Topologies: Aspire/console/ASP.NET/worker. NOT-WHEN: one-shot audit (scan-agentic-app-perf), install rules (configure-agentic-perf-rules), reasoning-model questions unrelated to evals, or running an existing suite (dotnet test).--- | ||
|
|
Comment on lines
+50
to
+56
| public static readonly string StorageRoot = | ||
| // _store lives next to the test project (the directory that holds | ||
| // .config/dotnet-tools.json) so the runtime AssemblyCleanup report | ||
| // call, the CI safety-net step, and `dotnet tool run aieval` — all | ||
| // executed from the project directory — resolve the same store. | ||
| public static readonly string StorageRoot = | ||
| Path.Combine(ProjectRoot.Find(), "_store"); |
Comment on lines
+84
to
+87
| foreach ($m in [regex]::Matches($line, $ep)) { | ||
| $src = [System.IO.Path]::GetFileNameWithoutExtension($f.Directory.Name) # best-effort: source project/dir | ||
| if ([string]::IsNullOrWhiteSpace($src)) { $src = [System.IO.Path]::GetFileNameWithoutExtension($f.Name) } | ||
| $edges.Add([pscustomobject]@{ source = $src; target = $m.Groups['target'].Value; file = $rel; line = ($i + 1) }) |
Contributor
Author
|
/evaluate --run 5 |
Contributor
Skill Validation Results
[1]
Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Round-3 eval showed scan regressing below baseline (0/5): the skill wrote scan.md but ended with an empty/one-line chat message, and the judge scores the chat output. Rebalance the workflow so the chat answer is primary: - Reframe execution-discipline: chat response is the deliverable; scan.md is a persisted copy. Remove the file-first framing (the prior 'write the report FIRST, before any analysis' directive starved the chat answer). - Analyze first, then present findings in chat AND persist to scan.md; the chat summary is the terminal step. - On a small app, read sources directly and skip the detection scripts (a large-codebase accelerator) to save the turn budget for the answer. - Make step 5 (surface in chat) a hard requirement: never end with an empty or one-line message. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…fold Round-3 eval showed ~7 scaffold arms hitting the 600s timeout (600k-1.6M tokens, 45-63 tool calls) even though the assertions only check file_exists / csproj-contains. The agent was grinding dotnet build/test/run/restore to 'verify' a scaffold that was already complete on disk. - Add a hard execution-discipline rule: do NOT run dotnet build/test/run/ restore as part of scaffolding; hand those commands to the user instead. - Reframe step 10 (Validation) as commands to hand to the user rather than something the agent runs inline; only run on explicit user request. The version-less <PackageReference> entries created in step 3.2 already carry the correct package set, so the scaffold is complete once files exist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
/evaluate --run 5 |
Contributor
Skill Validation Results
[1] (Isolated) Quality improved but weighted score is -45.0% due to: judgment, quality, tokens (52448 → 215416), tool calls (7 → 21), time (36.5s → 98.0s) Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
…head Round-4 eval showed scan now answers in chat and scores 4.0, but still loses close calls on tiny fixtures by over-tooling (19-22 tool calls, ~220k tokens, ~120s vs baseline's 9 calls / 65k / 39s) — the token/tool/time sub-scores drag an otherwise-competitive audit negative. Make the small-app discipline concrete: read each source file once, skip the detection scripts, open no reference docs, and aim for well under ~10 tool calls on apps of ~5 files or fewer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The clean-project-zero-criticals scenario asserts 'critical: 0' but its fixture contained real reliability issues an unconstrained baseline flags (hard-coded API key, ignored Aspire connection string, agent never exposed via an endpoint, and an invalid AddServiceDefaults() on the AppHost builder) — so baseline out-audited the scoped perf skill and won every run (-0.45 consistently, not variance). These are out of scan's perf/cost/topology scope. Align the fixture with the scenario's stated intent: consume the connection string via AddAzureOpenAIClient (no hard-coded key), expose the agent via /chat, WaitFor the resource, and drop the invalid AppHost AddServiceDefaults call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment on lines
+1
to
+5
| --- | ||
| name: setup-maf-evals | ||
| description: | | ||
| Scaffold a .NET agentic-app (MAF; Aspire/Foundry optional) evaluation harness — `<App>.Evals.Tests` MSTest project wired to the GA Microsoft.Extensions.AI.Evaluation (MEAI) reporting pipeline. WHEN: "set up evals", "add evaluation harness/coverage", "validate quality after a model change", "compare gpt-4o vs gpt-4o-mini", "add safety evaluators"; or troubleshooting — "why are my Quality columns erroring/empty", "reasoning model breaks evals / max_tokens vs max_completion_tokens", "which evaluators fail my stylistic/summarizer agent". Categories: **NLP** (BLEU/GLEU/F1, no key), **Quality** (LLM-as-judge), **Safety** (content-harm via Foundry). Auto-installs `aieval`, detects `IChatClient`, generates a factory (`EVAL_USE_REAL_AGENT=1`), emits an HTML report; optional PR workflow. Topologies: Aspire/console/ASP.NET/worker. NOT-WHEN: one-shot audit (scan-agentic-app-perf), install rules (configure-agentic-perf-rules), reasoning-model questions unrelated to evals, or running an existing suite (dotnet test).--- | ||
|
|
Comment on lines
+50
to
+56
| public static readonly string StorageRoot = | ||
| // _store lives next to the test project (the directory that holds | ||
| // .config/dotnet-tools.json) so the runtime AssemblyCleanup report | ||
| // call, the CI safety-net step, and `dotnet tool run aieval` — all | ||
| // executed from the project directory — resolve the same store. | ||
| public static readonly string StorageRoot = | ||
| Path.Combine(ProjectRoot.Find(), "_store"); |
Comment on lines
+46
to
+64
| # Node-declaration patterns. Each captures the agent's name/alias where present. | ||
| $nodePatterns = @( | ||
| # Aspire AppHost project registration: AddProject<Projects.X>("alias") | ||
| @{ kind = 'project'; rx = 'AddProject<[^>]+>\s*\(\s*"(?<name>[^"]+)"' }, | ||
| # MAF host agent registration: AddAIAgent("name", ...) | ||
| @{ kind = 'agent'; rx = 'AddAIAgent\s*\(\s*"(?<name>[^"]+)"' }, | ||
| # AsAIAgent(name: "name", ...) | ||
| @{ kind = 'agent'; rx = 'As(?:AI)?Agent\s*\(\s*name\s*:\s*"(?<name>[^"]+)"' }, | ||
| # CreateAIAgent(name: "name") | ||
| @{ kind = 'agent'; rx = 'CreateAIAgent\s*\(\s*(?:name\s*:\s*)?"(?<name>[^"]+)"' }, | ||
| # new ChatClientAgent(... ) — name not always present; recorded per-file | ||
| @{ kind = 'agent'; rx = 'new\s+ChatClientAgent\s*\(' } | ||
| ) | ||
|
|
||
| # Handoff-edge patterns: AddHandoff("target") / WithHandoff("target") | ||
| $edgePatterns = @( | ||
| 'Add(?:Handoff|Edge)\s*\(\s*"(?<target>[^"]+)"', | ||
| 'WithHandoff\s*\(\s*"(?<target>[^"]+)"' | ||
| ) |
Contributor
Author
|
/evaluate --run 5 |
Contributor
Skill Validation Results
[1]
Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Contributor
Author
|
/evaluate --run 5 |
Contributor
Skill Validation Results
[1]
Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
…; stop assuming detection scripts ran on small apps Softens the step-1 abort so an Aspire AppHost or any IChatClient/agent surface is in scope (fixes the missing-otel refusal where the skill declined because the app didn't reference Microsoft.Agents.AI). Also reworks step-2 category seeding so categories 1/6/7 are read directly on small apps where the detection scripts were skipped, instead of implying the scripts always run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… agentic service The fixture was AppHost-only, which pushed the skill onto its 'no agentic app -> abort' path. Add a Coach service that wires an IChatClient + ChatClientAgent with zero OpenTelemetry so the OTel gap is an unambiguous, in-scope finding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracks #837.
Summary
Consolidated polish + retirement across the
dotnet-aiperf-skills cluster(
configure-agentic-perf-rules,scan-agentic-app-perf,setup-maf-evals),dogfooded against a reference MAF + Aspire + Foundry app over the last
several weeks. See issue #837 for the full motivation and per-skill
breakdown — this PR is the implementation.
Retirements
agentic-perf-reviewer.agent.md— collapses intoscan-agentic-app-perfconfigure-agentic-perf-rules.select-agent-modelsskill — folded intoconfigure-agentic-perf-rules(rule feat: Add WinForms Expert #3) and
scan-agentic-app-perf(checkmodel.same-default).Rename
audit-agentic-app-perf→scan-agentic-app-perf(better intent matchper CONTRIBUTING.md naming guidance).
Per-skill highlights
configure-agentic-perf-rulesv0.3.0scan-agentic-app-perfmodel.same-default); single overwrittenscan.md; pruned guesswork thresholdssetup-maf-evalsv2aievalauto-install; MEAI HTML report; rubric-driven evaluatorBug fix + deprecation note (in setup-maf-evals)
JsonNamingPolicy.SnakeCaseLower+PropertyNameCaseInsensitive. Theoriginal templates relied on STJ defaults, causing PascalCase
properties to bind to
null(loud in Quality mode, silentlycorrupting in Telemetry mode).
references/ichatclient-detection.mdadds the
AddOpenAIClient(...).AddChatClient(...)detection row,marks the legacy
AddAzureChatCompletionsClientrow as legacy, andlinks the
Foundry migration guide.
references/common-pitfalls.mdupdates the reasoning-modelmax_tokenspitfall to recommend the OpenAI SDK migration as thedurable fix. Related: MEAI.Evaluation.Quality: judge HTTP errors recorded as silent per-metric error rows; tests exit 0 with operationally-useless report extensions#7580.
Cross-cutting doc updates
MAF + Aspire + Foundry→
MAF; Aspire/Foundry optional); add "Supported topologies" bodysections covering plain console, ASP.NET Core, worker service.
Detection logic unchanged in this PR; deeper non-Aspire support is a
candidate follow-up.
WHEN/NOT-WHENphrasing across all three skills.CODEOWNERS: rotate the perf-skills second owner; delete thedangling entries for the retired skill/agent.
skills.
Validation
(
code-review-buddy):scan-agentic-app-perfagainst current code: 0 critical / 0 warn /2 info (both intentional-by-design —
model.same-defaultandotel.no-per-agent-source).setup-maf-evalsscaffolds, builds, and all three modes(telemetry, quality, compare) produce clean reports with the stub
IChatClientand with a real Foundry deployment.tests/dotnet-ai/configure-agentic-perf-rules/eval.yamlupdated forthe rule reframing and the v0.2.0 → v0.3.0 idempotent upgrade path.
configure-agentic-perf-rules 966, scan-agentic-app-perf 994,
setup-maf-evals 1001.
Commit history
Intentionally fine-grained (~46 commits) — each reflects a real
dogfooding iteration. Happy to squash on request before merge.
Out of scope (deferred to follow-ups)
topology question).
mcp-csharp-*CODEOWNERS rotation.