Add Excel skill-value case study and reproducible formatting example

CHANGELOG.md

@@ -6,6 +6,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [Unreleased]
 
+### Added
+
+- Added an Excel skill-value case study and four-case formatting example using
+  the public CopilotEval API, frozen historical guidance, offline checks, and
+  opt-in desktop Excel verification. The historical comparison found equal
+  correctness and higher recorded token usage with broad skills; the small
+  example and narrow formatting skills have not had paid comparisons.
+
 ### Changed
 
 - Refreshed runtime, development, documentation, and build dependency minimums

docs/use-cases/excel-skill-value.md

@@ -0,0 +1,107 @@
+# Do broad Excel skills help an agent?
+
+ExcelMcp tested whether a skill improved an agent's use of two equal product
+entry points: native MCP tools and `excelcli`. The comparison found no
+correctness advantage on the selected tasks, while recorded tokens increased.
+Useful documentation and useful automatically loaded instructions are not
+the same thing.
+
+## What was compared
+
+The completed comparison used public **pytest-skill-engineering 1.0.3**,
+`github-copilot-sdk` **1.0.16**, `mcp` **2.2.0**, and **gpt-6.1-sol**.
+Six tasks, two entry points, skill/no-skill conditions, and two repetitions
+produced **48 unique cases**. Requests, tools, permissions, fixtures, model,
+and budgets were held constant within each entry point. Only the matching
+skill's availability changed.
+
+Four tasks used pinned public SpreadsheetBench workbooks: grouped transaction
+totals, monthly inventory comparison, last-record deduplication, and conditional
+date/text totals. Two authored tasks repaired Power Query and refreshed an
+existing Data Model/DAX/PivotTable/PivotChart report. This was an adapted pilot,
+not an official SpreadsheetBench score.
+
+The input revision was `49b73a94775fb489063f60ca1865e3a650079a79`; the archive
+SHA256 was `10ef893dd29cb13ab97143ea787e68cdc9574a13873ab9a54e50b31dc03fc949`.
+SpreadsheetBench declares CC BY-SA 4.0. No upstream workbook, answer file, or
+adapted benchmark instruction is included in this example.
+
+## Evidence, not completion claims
+
+Fixed checks ran outside the model-writable task directory. They reopened
+saved workbooks through desktop Excel COM, checked every requested result,
+and protected unrelated values, formulas, formats, calculation mode, and
+named analysis objects. Formula cases also changed inputs to reject constants
+that happened to match the original answer.
+
+Before paying for comparisons, checker proofs rejected unchanged or deliberately
+incorrect workbooks. Model prose and successful tool calls alone could not pass.
+Missing correlated completion evidence was a capture failure, not a task success.
+
+An early lesson was that supplying a skill directory did **not** establish
+availability. In the SDK's isolated `empty` mode, skills needed explicit
+`enable_skills=True`. Each real directory was validated with public `load_skill`,
+then SDK discovery checked the exact enabled name and resolved source path.
+Discovery probes forbade `send` and `send_and_wait`; they made no model requests.
+Actual treatment calls separately established that the skills were read.
+
+## Measured result
+
+| Entry point | Without skill | With broad skill | Correct cases per condition |
+|---|---:|---:|---:|
+| MCP | 4,312,584 recorded tokens | 5,320,118 recorded tokens | 12/12 |
+| CLI | 1,794,556 recorded tokens | 3,092,967 recorded tokens | 12/12 |
+
+Recorded token usage increased **23.4% for MCP** and **72.4% for CLI**.
+All **24 treatment cases read the matching skill**. This was not a
+missing-treatment result.
+
+The resumed round reserved **50 of 100 authorized attempts**: 48 verified
+unique cases and two interrupted reservations. Interrupted usage is unknown,
+excluded from those percentages, and never represented as zero. Earlier
+package-1.0.2 comparisons had a separate allowance and are not pooled here.
+
+The [compact receipt](https://github.com/sbroenne/pytest-skill-engineering/blob/main/examples/excel-skill-value/evidence/real-world-1.0.3.json)
+preserves case metrics, frozen hashes, and invocation boundaries without private
+workbook paths or raw conversations.
+
+## Limits and the recording problem
+
+One capable model, six tasks, and two repetitions do not establish universal
+skill value. Repeated cases reused input contents; public cases may already be
+known to the model. Equal success creates a ceiling: there is no observed
+correctness difference to estimate.
+
+The runner resumed across three invocations and a recorder-code change.
+Two interrupted runners returned no final result; their cause was not proven.
+The recorder later saved returned execution before workbook verification and
+used atomic attempt snapshots. Those protections worked. The configured public
+`on_event` callback received **zero live events** in the actual comparison.
+Mocked callback tests are not evidence that live journaling works.
+
+## Product decision
+
+The broad default skills were replaced by two optional report-formatting
+skills. General workflows, limitations, recovery, and command references moved
+to ordinary documentation, with the existing website addresses preserved.
+The `skills` directory now contains actual skills, not a documentation corpus.
+Native schemas/help and recovery messages remain responsible for product
+behavior and essential safety.
+
+This decision is supported for the tested model and tasks. It does **not**
+establish that the new narrow skills help. Their separate, unrun comparison has
+positive formatting tasks and ordinary non-trigger tasks; it checks selection,
+workbook correctness, and usage separately.
+
+## Small runnable example
+
+The [Excel example](https://github.com/sbroenne/pytest-skill-engineering/tree/main/examples/excel-skill-value) demonstrates the
+method with one self-authored workbook task, both entry points, and paired
+skill/no-skill cases using the public 1.x API. Its frozen broad fixtures match
+the historical manifests. A formatting-profile run instead uses explicitly
+supplied prepared narrow skills and must not be pooled with the broad result.
+
+The example does **not** reproduce the full 48-case workload or promise the
+same token percentages. Default tests are offline. Desktop-Excel checker
+proofs and paid live comparisons are separate commands. No new live example
+run was used to support the historical conclusion.

examples/README.md

@@ -14,6 +14,13 @@ Its harder checkout workflow adds durable state, duplicate/conflicting requests,
 and 42 fixed checks. The [use case](../docs/use-cases/companion-skill.md) records
 the comparison design, actual results, and decision not to ship the skill.
 
+The [Excel skill-value example](excel-skill-value/) pairs native MCP and CLI
+execution with and without a skill, then checks saved workbooks independently
+through desktop Excel. Default checks are offline; Excel checker proofs and
+paid comparisons are explicit. Its [case study](../docs/use-cases/excel-skill-value.md)
+records equal correctness and higher recorded token usage with broad skills,
+and separates that outcome from future narrow-formatting comparisons.
+
 The repository's own integration tests demonstrate advanced scenarios:
 
 - `test_02_models.py` — model comparison

examples/excel-skill-value/.gitattributes

@@ -0,0 +1 @@
+fixtures/** -text whitespace=cr-at-eol,-blank-at-eof

examples/excel-skill-value/.gitignore

@@ -0,0 +1,5 @@
+.venv/
+__pycache__/
+.pytest_cache/
+receipts/
+*.xlsx

examples/excel-skill-value/Close-OwnedWorkbook.ps1

@@ -0,0 +1,76 @@
+param(
+    [Parameter(Mandatory)][string]$Path,
+    [switch]$InspectOnly
+)
+
+$ErrorActionPreference = 'Stop'
+Add-Type -TypeDefinition @'
+using System;
+using System.Runtime.InteropServices;
+using System.Runtime.InteropServices.ComTypes;
+public static class OwnedWorkbookRot {
+    [DllImport("ole32.dll")]
+    private static extern int GetRunningObjectTable(int reserved, out IRunningObjectTable table);
+    [DllImport("ole32.dll")]
+    private static extern int CreateBindCtx(int reserved, out IBindCtx context);
+    public static object Find(string path) {
+        IRunningObjectTable table = null;
+        IBindCtx context = null;
+        IEnumMoniker enumerator = null;
+        try {
+            Marshal.ThrowExceptionForHR(GetRunningObjectTable(0, out table));
+            Marshal.ThrowExceptionForHR(CreateBindCtx(0, out context));
+            table.EnumRunning(out enumerator);
+            var monikers = new IMoniker[1];
+            while (enumerator.Next(1, monikers, IntPtr.Zero) == 0) {
+                var moniker = monikers[0];
+                try {
+                    string name;
+                    moniker.GetDisplayName(context, null, out name);
+                    if (String.Equals(name.TrimStart('!'), path, StringComparison.OrdinalIgnoreCase)) {
+                        object book;
+                        Marshal.ThrowExceptionForHR(table.GetObject(moniker, out book));
+                        return book;
+                    }
+                } finally { Marshal.FinalReleaseComObject(moniker); }
+            }
+            return null;
+        } finally {
+            if (enumerator != null) Marshal.FinalReleaseComObject(enumerator);
+            if (context != null) Marshal.FinalReleaseComObject(context);
+            if (table != null) Marshal.FinalReleaseComObject(table);
+        }
+    }
+}
+'@
+
+$resolved = [IO.Path]::GetFullPath($Path)
+$book = $null
+$excel = $null
+$books = $null
+$matched = 0
+$closed = 0
+try {
+    $book = [OwnedWorkbookRot]::Find($resolved)
+    if ($null -ne $book) {
+        if (-not [string]::Equals($book.FullName, $resolved, [StringComparison]::OrdinalIgnoreCase)) {
+            throw 'Running workbook identity does not match the owned fixture.'
+        }
+        $matched++
+        if (-not $InspectOnly) {
+            $excel = $book.Application
+            $books = $excel.Workbooks
+            $soleWorkbook = $books.Count -eq 1
+            $book.Close($false)
+            $closed++
+            if ($soleWorkbook) { $excel.Quit() }
+        }
+    }
+    @{ matched = $matched; closed = $closed } | ConvertTo-Json -Compress
+} finally {
+    foreach ($owned in @($books, $excel, $book)) {
+        if ($null -ne $owned -and [Runtime.InteropServices.Marshal]::IsComObject($owned)) {
+            [void][Runtime.InteropServices.Marshal]::FinalReleaseComObject($owned)
+        }
+    }
+}

examples/excel-skill-value/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Stefan Broenner
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

examples/excel-skill-value/README.md

@@ -0,0 +1,83 @@
+# Excel skill-value example
+
+This small project compares one self-authored formatting request through native
+MCP tools and `excelcli`, with and without the matching skill. Fixed checks
+inspect the saved workbook using desktop Excel COM. The model cannot edit the
+checks or expected answers.
+
+This is a method demonstration, not the 48-case benchmark. The historical
+[case study](../../docs/use-cases/excel-skill-value.md) and
+[receipt](evidence/real-world-1.0.3.json) describe broad skills. No paid run of
+this small example or the new formatting skills is claimed.
+
+## Requirements and setup
+
+Live and checker-proof commands require Windows, PowerShell 7, desktop Excel,
+and built ExcelMcp MCP/CLI executables. Live commands also require authenticated
+GitHub Copilot and `gpt-6.1-sol`. GitHub-hosted runners do not supply Excel.
+Place this project under `examples/excel-skill-value` in the
+pytest-skill-engineering repository first. It uses that repository's local
+framework source and committed root `uv.lock`, not a separate dependency setup.
+Run the following commands from this example's directory:
+
+```powershell
+uv sync --project ..\.. --frozen --all-extras
+$env:EXCEL_MCP_EXECUTABLE = 'C:\Tools\ExcelMcp\Sbroenne.ExcelMcp.McpServer.exe'
+$env:EXCEL_CLI_EXECUTABLE = 'C:\Tools\ExcelMcp\excelcli.exe'
+uv run --project ..\.. --frozen pytest -q
+uv run --project ..\.. --frozen pytest tests\offline\test_checks.py --prove-excel -q
+```
+
+Default checks need neither Excel nor model calls. Keeping the existing root
+lock avoids duplicating dependencies or rewriting the upstream registry.
+The preparation was validated in an isolated public-source snapshot of
+upstream v1.0.3; no existing upstream checkout was modified.
+The opt-in proof rejects the unformatted input, accepts proper formatting, and
+rejects a changed fractional rate in a real workbook.
+
+## Explicit paid comparison
+
+```powershell
+uv run --project ..\.. --frozen pytest tests\live --run-live --paid-limit 4 --receipt-dir receipts\new-broad-run --aitest-json receipts\new-broad-native.json -v
+```
+
+Four selected cases reserve four attempts before model execution. Existing
+receipt directories are rejected. Supply `--prior-attempts` for a resumed
+allowance and never exceed the user's authorized ceiling, at most 100.
+This example makes one attempt per case, with no retries or judging model.
+Do not add iterations; the example rejects multiplicative iteration counts.
+Every paid smoke or rerun counts, including interrupted executions.
+
+The default `--skill-profile broad` uses frozen `fixtures\excel-mcp` and
+`fixtures\excel-cli`. These reproduce the historical *guidance*, not all
+historical tasks. Git preserves these fixture bytes without line-ending
+conversion so their hashes match the historical receipt on every checkout.
+To test the new, unproven narrow skills instead:
+
+```powershell
+$env:EXCEL_MCP_SKILL_DIRECTORY = 'C:\PreparedSkills\excel-mcp-report-formatting'
+$env:EXCEL_CLI_SKILL_DIRECTORY = 'C:\PreparedSkills\excel-cli-report-formatting'
+uv run --project ..\.. --frozen pytest tests\live --run-live --skill-profile formatting --paid-limit 4 --receipt-dir receipts\new-formatting-run --aitest-json receipts\new-formatting-native.json -v
+```
+
+Within each entry point the request, tools, permissions, model, and budgets
+are unchanged between conditions. The CLI is one custom SDK tool that accepts
+an argument array and runs without a shell. Native MCP schemas are unchanged.
+The workspace tool can only read the task and skill directories and write
+non-executable task inputs. No expected result is put in the task directory.
+
+The public skill loader and no-send SDK discovery run before paid execution,
+including exact treatment names/source paths and empty baselines. Returned
+execution is saved before verification; discovery, skill reads, nullable token
+usage, and independent verification are separate fields. `passed: true` means
+the workbook check passed, not merely that the agent said it finished.
+There is no claimed live event journal. `reserved` or `verifying` records left
+by interruption are incomplete, not free attempts or verified successes.
+
+Raw native evidence contains local paths and tool outputs; keep it private.
+Only the compact historical receipt is intended for publication. Cleanup closes
+only the exact test-owned workbook without saving, then stops the private CLI
+pipe. It does not stop a user's default service or kill Excel by name.
+
+The authored Python/PowerShell example and frozen ExcelMcp guidance are MIT
+licensed; preserve the included licence. No SpreadsheetBench content ships here.