Fix all review findings and release 1.0.3

.github/workflows/docs.yml

@@ -20,13 +20,13 @@ on:
   workflow_call:
     inputs:
       version:
-        description: "Version tag (e.g. v1.0.2) to validate and build from"
+        description: "Version tag (e.g. v1.0.3) to validate and build from"
         required: false
         type: string
   workflow_dispatch:
     inputs:
       version:
-        description: "Version tag (e.g. v1.0.2) to validate and build from"
+        description: "Version tag (e.g. v1.0.3) to validate and build from"
         required: false
         type: string
 

CHANGELOG.md

@@ -19,6 +19,33 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
   frozen guidance exists only as a historical experiment fixture. Evaluating
   consumer-provided skills remains supported.
 
+## [1.0.3] - 2026-10-01
+
+### Fixed
+
+- Repeated pytest cases preserve grouped parameters and custom case IDs.
+  Saved outcomes include setup and cleanup failures, along with verification
+  recorded during cleanup.
+- Plugin and project custom agents preserve declared names, models, tools,
+  reasoning effort, skills, inference settings, and MCP configuration.
+  Duplicate declared names are rejected consistently.
+- Skill reference tools expose enabled skills only and retain nested relative
+  Markdown paths without silently replacing duplicate references.
+- Framework-owned child executions retain separate results and descendants.
+  Overall token, premium-request, cost, and evidence completeness totals include
+  child work without counting it twice.
+- Tool completion evidence preserves images, including multiple images and
+  explicit capture errors for malformed or conflicting payloads.
+- Direct CLI execution preserves Windows paths, grouping quotes, spaces, and
+  empty arguments.
+- Saved configuration records the actual prepared runtime settings instead of
+  reconstructing them from the original eval. A/B roles remain separate.
+- Missing-price warnings include every captured usage model. Unknown model
+  identities remain null instead of receiving an invented default.
+- Native schema-4.0 loading requires every current producer field and preserves
+  explicit nulls, empty values, and image bytes. Regenerate older incomplete
+  reports through their producer.
+
 ## [1.0.2] - 2026-10-01
 
 ### Fixed

docs/contributing/evidence-structure.md

@@ -18,6 +18,10 @@ Schema 4.0 saves test IDs, outcomes, assertion errors, verification properties,
 execution configuration, turns, tool arguments and outputs, completion status,
 capture errors, request audits, stop reasons, usage, and missing-pricing models.
 Private SDK/session fields are not serialized.
+Prepared session settings reflect overrides and persona additions. A missing
+snapshot or unknown model is explicit `null`. Framework-owned child results
+remain nested under invocation IDs, with inclusive top-level accounting; tool
+image bytes and any additional images round-trip without loss.
 
 `eval_result.skill_discovery` saves actual public SDK availability separately
 from requested configuration and observed reads. It preserves metadata, loading
@@ -28,11 +32,14 @@ availability from supplied paths or patch saved JSON.
 
 Preserve missing versus empty output and nullable usage. Do not replace unsupported
 property values with strings or synthesize uncaptured successful assertions.
-The loader rejects old schemas; there is no compatibility reader.
+The loader rejects old schemas and missing current-schema fields at every record
+level; there is no compatibility reader or reconstruction from default values.
 
 Each captured run retains its pytest outcome. Multiple executions in one test
 share that outcome, so their entries are not independently graded task verdicts.
 Preserve side-specific properties and captured comparison roles.
+Finalize outcomes and verification properties after fixture cleanup; setup and
+cleanup errors with captured executions must not be saved as passes.
 
 ## Validation
 

docs/contributing/releasing.md

@@ -85,6 +85,10 @@ Check the [breaking-change guide](../migration.md), complete current documentati
 schema-4.0 evidence contracts, package version metadata, and the installed wheel
 before publishing. There is no companion skill to install or publish.
 Reading saved evidence must work without Copilot.
+Confirm strict current-field loading, null model identity, image and child-result
+round trips, and outcomes finalized after fixture cleanup. A/B roles are saved
+separately from prepared runtime settings. Regenerate evidence missing required
+schema-4.0 fields through its producer; do not patch old reports.
 
 Verify the quickstart and the
 [historical customer workflow sample](https://github.com/sbroenne/pytest-skill-engineering/tree/main/examples/skill-dogfood)

docs/getting-started/custom-agents.md

@@ -9,6 +9,12 @@ A **custom agent** is a `.agent.md` definition file that you load with `load_cus
 It is **not** inherently a subagent.
 It only becomes a **subagent** when Copilot dispatches work to it at runtime.
 
+Standalone, plugin, and project loaders preserve declared agent names, model,
+tools, reasoning effort, inference settings, skills, and `mcp-servers` settings.
+When no name is declared, the filename supplies it. Directory include/exclude
+filters and per-agent overrides use that resolved name. Duplicate declared names
+are errors; an explicit empty tool list remains empty.
+
 ## Load one custom agent
 
 ```python
@@ -37,6 +43,9 @@ async def test_dispatches_to_reviewer(copilot_eval):
 ```
 
 `result.subagent_invocations` reports runtime subagent events such as `selected`, `started`, and `completed`.
+Framework-owned invocations also retain a separate child `result`. Inspect its
+tools, usage, and capture errors without mixing its conversation into the parent.
+See [child evidence and accounting](../reference/result.md#child-execution-evidence).
 
 Each nested invocation receives the remaining custom agent registry without its
 own definition. That registry shrinks along the dispatch path, so a custom agent

docs/getting-started/iterations.md

@@ -15,6 +15,13 @@ execution cost. Authorize the scope before running a large case matrix.
 Each repetition has its own pytest outcome. Native JSON preserves that execution,
 its iteration number, configuration, usage, cost estimate, duration, and recorded
 checks. There is no grouped dashboard verdict or automatic explanation.
+Grouped and stacked pytest parameters retain their user-supplied IDs. Only the
+framework's repetition axis is removed from the saved case name; the iteration
+number remains a separate field.
+
+Saved outcomes are finalized after fixture cleanup. A cleanup failure changes
+the captured execution's outcome to failed and retains cleanup-recorded checks,
+even when the test body passed.
 
 There is no AI flakiness explanation. Inspect the individual traces and source
 to distinguish intermittent interface behavior from unstable fixtures or

docs/getting-started/skills.md

@@ -47,8 +47,9 @@ For detailed budgeting advice, use the reference document.
 
 References are documents the agent can look up **on demand** rather than having
 them always in context. Copilot loads skills natively and can read their
-`references/` files using its file tools. The plugin does not inject synthetic
-reference tools.
+`references/` files using its file tools. The supplied personas also expose
+`list_skill_references` and `read_skill_reference` for validated enabled
+references. `client_mode="empty"` does not inject persona tools.
 
 Reference documents may be organized in subdirectories, for example
 `references/commands/read.md`. Link to them directly from `SKILL.md` using paths
@@ -59,6 +60,10 @@ The package validates all reference files recursively as nonempty UTF-8 Markdown
 `Skill.references` uses paths relative to `references/`, with `/` separators on
 every platform: `commands/read.md`. Flat names such as `budgeting-guide.md` are
 unchanged, and equal basenames in different folders remain distinct.
+Reference tools use these same relative paths. Disabled skills are not
+advertised or readable through the reference tools. This does not block ordinary
+filesystem access by other tools. Duplicate relative reference paths across
+enabled skills are rejected rather than silently replacing a document.
 Links are accepted only when their resolved targets stay within `references/`;
 the `references/` directory itself must resolve within the skill directory.
 Broken links, directory cycles, unreadable entries, and non-file entries are

docs/how-to/cost-estimation.md

@@ -20,9 +20,16 @@ The generated starter writes explicit rates without replacing existing ones.
 
 A missing model rate produces a placeholder zero estimate and a missing-pricing
 warning. The suite saves `models_without_pricing` in native JSON evidence.
+Warnings cover every captured usage model, including earlier models and
+framework-owned child sessions, not just the test's final display model.
+They are calculated for the current suite and current pricing file.
 **This is not measured free usage.** Do not choose
 a supposedly cheapest configuration from unavailable prices.
 
 Missing SDK usage values remain `None`, not known zero. Estimates from available
 input/output counts are not complete billing evidence; unknown cache reads
 receive no cache discount. See [result evidence](../reference/result.md).
+
+Parent cost and premium-request totals include framework-owned child work once;
+the saved child records remain separately inspectable. Do not sum inclusive
+parent and child totals together.

docs/how-to/image-assertions.md

@@ -10,6 +10,12 @@ Tool-returned images are captured in results and preserved in native JSON.
 Inspect `image_content` (bytes) and `image_media_type` on calls returned by
 `result.tool_calls_for(...)`.
 
+The first image remains in those fields. All later images from the same
+completion are in `call.additional_images`, as `ImageContent` objects with
+`data` and `media_type`. Saved `EvalResult.tool_images_for(name)` returns every
+image for that tool, in order. SDK content blocks and binary image results are
+captured; malformed image data produces an explicit capture error.
+
 Semantic image judging is not supported. Use ordinary pytest assertions for
 image presence and metadata. Your coding agent can inspect the captured images;
 the framework does not automatically interpret them.

docs/how-to/inspect-evidence.md

@@ -48,7 +48,9 @@ saved under `tests[].properties`; recording a property alone does not determine
 the outcome. Record the observed values before asserting, so failures retain
 the check's evidence. Use JSON-compatible values and never record secrets.
 
-Setup failures, ordinary tests without eval execution, and opt-out skips remain
+Setup failures with a captured eval and fixture-cleanup failures are reflected
+in the saved execution outcome. Cleanup-recorded properties are included too.
+Ordinary tests, setup failures without eval execution, and opt-out skips remain
 in pytest and JUnit output. Their absence from execution evidence is not a pass.
 
 ## Investigate with your coding agent
@@ -74,6 +76,8 @@ No companion skill is required.
 
 `ab_run` saves both captured configurations and comparison roles without changing
 runtime names. Both entries share the enclosing pytest outcome and properties.
+The role is stored in `eval_result.comparison_role`, separately from prepared
+session settings.
 Record `baseline_verification` and `treatment_verification` separately. A failed
 combined assertion does not prove both sides failed.
 
@@ -96,5 +100,8 @@ for case in evidence.tests:
 ```
 
 Reading schema-4.0 evidence is local and needs no Copilot process or credentials.
+Every current producer field is required, including nested child and tool records.
+Present `null`, empty collections, empty output, and known zero remain distinct
+from absent fields. Unknown models remain `null`, not a guessed model name.
 Regenerate older evidence by running its producer with framework 1.x; never
 hand-edit JSON or invent missing fields.

docs/how-to/test-cli-tools.md

@@ -43,6 +43,11 @@ git_server = CLIServer(
 
 ## Configuration options
 
+With `shell="none"`, both `command` and tool `args` are split into direct process
+arguments, without shell expansion. On Windows, backslashes in paths are
+preserved; quote paths containing spaces. Grouping quotes are removed, and
+explicit empty quoted arguments remain empty arguments.
+
 ```python
 CLIServer(
     command="git",

docs/how-to/test-plugins.md

@@ -105,6 +105,10 @@ async def test_plugin_tool_usage(copilot_eval):
 ```
 
 `CopilotEval.from_plugin()` loads instructions, skills, and custom agents from the directory. The Copilot SDK handles model selection and tool access.
+Custom-agent definitions retain their declared names and supported SDK settings,
+including model, tools, skills, reasoning effort, and MCP servers, just as they do
+when loaded through the project factories. Missing names use the filename;
+duplicate resolved names are errors.
 
 ### Overriding Plugin Settings
 

docs/reference/result.md

@@ -13,8 +13,9 @@ description: "CopilotResult reference: tool calls, responses, token usage, files
 | `success` | Whether the eval completed successfully |
 | `error` | Error message when `success` is false |
 | `turns` | Normalized conversation turns |
-| `usage` | Per-turn token usage entries |
-| `model_used` | Actual Copilot model used |
+| `usage` | Per-turn token usage entries for this session |
+| `all_usage` | Captured usage across this session and framework-owned descendants |
+| `model_used` | Actual Copilot model used, or `None` when unknown |
 | `total_premium_requests` | Premium-request accounting |
 | `subagent_invocations` | Runtime subagent events from custom agent dispatch |
 | `permission_requested` | Whether Copilot requested permissions |
@@ -25,6 +26,7 @@ description: "CopilotResult reference: tool calls, responses, token usage, files
 | `tool_calls_admitted` | Calls admitted by the framework after caller guards; not proof of execution or application success |
 | `request_audit` | Actual outgoing request records when `audit_requests=True` |
 | `skill_discovery` | Actual pre-execution SDK skill metadata and diagnostics for explicit directories; `None` means no discovery check was attempted |
+| `configuration` | Safe snapshot of the prepared session settings; `None` means the snapshot was not captured |
 
 `success` describes session execution, not whether the application did the right
 thing. A tool can fail and the session can recover. Check application state with
@@ -99,7 +101,23 @@ result.tool_was_called_with("transfer", amount=500.0)
 ```
 
 Tool-returned images are available on individual tool calls as `image_content`
-(bytes) and `image_media_type`. See [Tool-returned images](../how-to/image-assertions.md).
+(bytes) and `image_media_type`. Further images from the same call are retained in
+`additional_images`, in order. See [Tool-returned images](../how-to/image-assertions.md).
+
+## Child execution evidence
+
+Each `subagent_invocations` entry retains its `invocation_id`, name, status, and
+duration. For a framework-owned child session, `invocation.result` contains that
+child's separate `CopilotResult`, including its own turns, tools, usage, request
+audit, capture errors, prepared settings, and further children. Native SDK events
+without an independently captured child result leave `result=None`; the framework
+does not invent a child trace.
+
+`execution_results` walks this session and its framework-owned descendants.
+Parent `turns`, `all_tool_calls`, and `usage` remain session-local. Token totals,
+premium-request totals, and saved cost estimates include descendant work once.
+Do not add already-inclusive child totals to the parent's totals again.
+Parent `evidence_complete` also checks captured child evidence.
 
 ## Token helpers
 
@@ -113,11 +131,25 @@ result.token_usage
 Each `usage` entry retains `model`, `input_tokens`, `output_tokens`,
 `cache_read_tokens`, `cache_write_tokens`, `reasoning_tokens`,
 `reasoning_effort`, and `duration_ms`. Missing SDK values are `None`, not zero.
-Token totals are `None` when any corresponding count is unavailable (or no
-usage was captured); `token_usage` omits those totals. Known zero remains zero.
+Token totals are `None` when any corresponding count is unavailable, or a
+captured session in the execution tree has no usage; `token_usage` omits those
+totals. Known zero remains zero.
 Cost is an estimate from reported input/output counts, not a claim of complete
 billing evidence; unknown cache reads receive no cache discount in that estimate.
 
+## Prepared settings
+
+`configuration` records the supplied settings after `extra_config` overrides,
+persona instructions, and tool setup, rather than rebuilding them during report
+conversion. Native evidence's `effective_system_prompt` is this supplied system
+message; it is not a claim to capture hidden SDK instructions. If preparation
+fails before the snapshot, both remain `None`.
+
+Server credentials, commands, arguments, environment variables, and executable
+handlers are excluded. Request audit remains the separate source of evidence
+about actual outgoing requests. A/B comparison roles are report metadata in
+`eval_result.comparison_role`, not part of the runtime configuration.
+
 ## Request audit
 
 Each `request_audit` record contains:

examples/skill-dogfood/src/skill_dogfood/checkout_workflow.py

@@ -76,6 +76,7 @@ def check_checkout_consumer(
     assert case.eval_result is not None
     result = case.eval_result
     assert result.success and result.evidence_complete, result.error
+    assert result.configuration is not None, "Prepared session settings were not captured"
     assert result.configuration["model"] == MODEL
     calls = [call for turn in result.turns for call in turn.tool_calls]
     assert len(calls) == 1 and calls[0].name == "checkout"

examples/skill-dogfood/src/skill_dogfood/workflow.py

@@ -261,6 +261,7 @@ def _check_consumer(
     assert case.eval_result is not None, run.stdout + run.stderr
     result = case.eval_result
     assert result.success and result.evidence_complete, result.error
+    assert result.configuration is not None, "Prepared session settings were not captured"
     assert result.configuration["model"] == MODEL
     calls = [call for turn in result.turns for call in turn.tool_calls]
     assert len(calls) == 1 and calls[0].name == "invoice"

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pytest-skill-engineering"
-version = "1.0.2"
+version = "1.0.3"
 description = "Reliable Copilot execution and evidence for coding-agent-led testing of MCP servers, CLI tools, skills, and custom agents."
 readme = "README.md"
 license = { text = "MIT" }