Skip to content

Add Excel skill-value case study and reproducible formatting example - #114

Merged
sbroenne merged 1 commit into
mainfrom
sbroenne-excel-skill-value-case-study
Oct 2, 2026
Merged

sbroenne merged 1 commit into
mainfrom
sbroenne-excel-skill-value-case-study

Conversation

@sbroenne

@sbroenne sbroenne commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

Description

Add the prepared Excel skill-value case study and a small four-case example comparing one self-authored formatting request through native MCP and CLI, with and without matching skills. The example uses the public CopilotEval API, default offline checks, no-send skill discovery, explicit paid-attempt limits, and independent saved-workbook verification through desktop Excel COM. It reuses the existing root uv.lock; no framework runtime or dependency changes are included.

The separate historical receipt records 48/48 independently verified unique cases and 24/24 treatments that read their skills. Recorded tokens increased 23.4% for MCP and 72.4% for CLI, with equal correctness. Its 50 reserved attempts include two interrupted attempts whose usage remains unknown, not zero. These results are not pooled with older runs or presented as results from the four-case example. No working live event journal is claimed; the callback received zero live events. New narrow formatting skills remain unproven.

Include 27 MCP and 26 CLI frozen historical guidance files, their MIT license, and the sanitized receipt. These are test fixtures, not framework companion/product skills. Git attributes preserve their exact bytes and historical whitespace; all 53 committed fixture hashes match the receipt. Add website navigation, the examples index, root type-check coverage, and an Unreleased changelog entry. No SpreadsheetBench input/golden workbooks, raw native traces, customer data, preparation notes, or duplicate lockfiles are included.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

Checklist

  • I ran the deterministic checks required for this change
  • I ran the relevant real-Copilot integration file when behavior changed
  • I added contract or integration coverage that proves the outcome
  • I updated user and contributor documentation where needed
  • I did not include credentials or unsanitized private prompts

Testing

Commands run from this isolated worktree's repository root:

uv run --frozen ruff check examples\excel-skill-value
Passed.

uv run --frozen ruff format --check examples\excel-skill-value
Passed; 58 files already formatted.

uv run --frozen pyright
Passed; 0 errors, 0 warnings, 0 informations across the repository's configured coverage.

uv run --frozen python -X utf8 -m mkdocs build --strict
Passed; full documentation site including the new case study and navigation.

uv run --frozen python -m pytest tests\contracts -q -o addopts=
Passed; 173 tests, no model requests.

uv build
Passed; source distribution and wheel built.

uv run --frozen pre-commit run
Passed; lint, formatting, type checks, and documentation build.

Normal git commit hooks
Passed again; lint, formatting, type checks, and documentation build. No hooks were skipped.

git diff --cached --check
Passed after declaring the preserved historical fixtures' CRLF and intentional final blank line in their scoped Git attributes.

Commands run from examples\excel-skill-value:

uv run --project ..\.. --frozen python -m pytest -q
Passed; 2 passed, 1 opt-in desktop Excel proof skipped.

uv run --project ..\.. --frozen python -m pytest tests\live\test_comparison.py --collect-only -q
Passed; exactly 4 cases collected, no model requests.

uv run --project ..\.. --frozen python -m pytest tests\offline\test_checks.py --prove-excel -q
Passed; 3 passed in 68.27 seconds against actual desktop Excel in this worktree.

The desktop Excel proof ran after the other session released Excel; no Excel-dependent commands overlapped. It rejected the unformatted input, accepted properly formatted output, and rejected a changed fractional-rate value. This verifies the fixed workbook checker, not model performance.

Additional checks verified all 53 fixture hashes in both copied files and staged Git blobs, 48 distinct historical case identifiers, historical group token totals, and the unchanged root lock.

No paid model runs, real-Copilot integration runs, --run-live executions, or new narrow-skill comparisons were authorized or performed. Framework execution behavior is unchanged. Offline checks establish framework/checker contracts, not model performance. The live example remains explicitly opt-in and unproven.

Related Issues

N/A — user-requested upstream contribution; no linked issue.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@sbroenne
sbroenne merged commit 7c08c6e into main Oct 2, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant