Align execution patterns and strengthen behavioral evaluations

README.md

@@ -118,6 +118,13 @@ Only tracked `assistant-*` directories are first-class release skills.
 ### assistant-workflow
 Core development pipeline: idea-to-action decomposition, discover, proportional planning, build and verification, independent review, bounded repair, and evidence-backed documentation.
 
+| Pattern | Use | Combine with |
+|---|---|---|
+| Linear | One focused change with dependent steps; scale planning and review to risk. | Goal-driven checks and fresh review; a low-risk local change may use the direct lightweight path. |
+| Skill-driven | A recurring task has a matching installed skill and contract. | Any other pattern; the skill supplies the method. |
+| Parallel | Independent work can progress together. Source-changing packets remain sequential in a shared or unknown workspace; overlap requires runtime-proven isolated workspaces. | Goal-driven integration: dependents wait for VERIFIED prerequisites, then full-scope validation precedes fresh review; cross-slice validation applies to multi-slice manifests. |
+| Goal-driven | Acceptance checks must remain the completion condition through repair. | Linear, skill-driven, or parallel execution; bounded repair escalates instead of claiming false completion. |
+
 For dependency-shaped uncertainty, the workflow defaults to
 `uncertainty_shape=bounded`: size alone does not activate progressive Discover.
 It enters that substate only when a predecessor decision must unlock an
@@ -196,8 +203,7 @@ requirement. Dependent slices wait for verified prerequisites. Read-only analysi
 may run in parallel for independent packets with non-overlapping ownership.
 Source-changing packets in a shared or unknown workspace remain sequential;
 parallel source-changing packets require runtime proof of isolated workspaces.
-After integration, rerun cross-slice and full-scope validation and perform a
-fresh review before completion.
+After all slices are integrated, full-scope validation is required before fresh Review. Cross-slice validation applies only when slice_manifest contains more than one item; when it contains one item, record cross-slice validation as not_applicable using the one-item manifest and single_slice_rationale. Single-slice full-scope validation still covers integration with existing code. After these integration checks, perform a fresh review before completion.
 
 For compression-safe work, the orchestrator keeps concise, root-scoped task and
 session state under `.codex/`. Resume reconciles that journal with the newest

docs/evals/README.md

@@ -28,6 +28,63 @@ under common operating conditions:
 - pivot/restart decisions for stagnation and Code Writer blockers
 - terminal max 10 review/QA round behavior
 
+## Observed execution-pattern evidence
+
+The pattern cases distinguish a deterministic policy fixture from evidence the
+Codex adapter actually observed in its JSONL event stream.
+
+- `small-fix-stays-lightweight` requires its exact target-file discovery probe
+  to be the first workspace command or file action; the matching command-start
+  event may precede its successful completion. This rule and its no-web/MCP
+  check apply only to the disposable local typo fixture, not delegated work.
+  Its admitted command vocabulary is limited to that exact read-only probe,
+  including bounded raw, argv, and shell-wrapper forms. Any other started or
+  completed command is an unsupported scope observation, regardless of whether
+  a later target-file event is present. The adapter reports unavailable only
+  when response, final-content, plan, scope, observed-path, discovery-order, and
+  external-tool checks reveal no independent failure. A grading artifact alone
+  cannot pass the case. Every observed file-change event must name only the
+  target or grading artifact; combined or separate out-of-scope paths fail even
+  when the final workspace diff is clean. Malformed raw event structure follows
+  the pre-grading unavailable policy, while well-formed unsafe paths and failed
+  target completions remain observed failures. With no `file_change`, command
+  text and final state still cannot prove which command edited the target or
+  when, so the existing unavailable route remains. Observable pre-discovery
+  actions, external calls, and incorrect final content remain completed
+  failures.
+- `pivot-restart-on-stagnation-or-code-writer-blocker` seeds a trusted failing
+  check, fixture-owned failure/recovery receipts, recovery action, and fresh
+  check. Its recovery artifact retains `terminal_completed=false`: a fresh-check
+  pass validates the recovery protocol, not repair of the legacy bug or workflow
+  completion. It admits only the three trusted fixture scripts and optional
+  read-only `cat RECOVERY.md`; any other started or completed command fails the
+  bounded case. Completed commands require numeric exit codes; a started command
+  needs no exit or output. For the three trusted marker commands, the selected
+  `aggregated_output`/`output` value must be a string. A numeric nonzero exit or
+  wrong marker in a string remains behavioral failure evidence; small-fix
+  output, passive message content, and optional recovery-read output are not
+  consumed. A present command start must have a nonempty id and one later
+  completion with the same admitted command kind; duplicate, unmatched, or
+  mismatched starts, and duplicate nonempty completion ids fail. Each recovery
+  artifact file-change event must also occur after the recovery start (or the
+  completion when no start exists) and before the fresh-check start (or its
+  completion when no start exists). Events before the trusted failure or
+  recovery, after the fresh-check start, and after fresh-check completion fail;
+  events during a matched recovery and after recovery completion but before the
+  fresh check remain valid. Once that unique fresh check completes, any later
+  started or completed command or file-change event also fails. This boundary
+  applies only to the disposable recovery fixture.
+- `isolated-parallel-a-b-then-c-with-integration` records the required A/B/C
+  dependency and integration policy, but current Codex CLI JSONL does not expose
+  authoritative worker, workspace, isolation, or overlap telemetry. The adapter
+  therefore emits `adapter_unavailable` with `unknown_event_shape`, no metrics,
+  and an excluded incomplete pair. Agent-message narrative never upgrades that
+  result to observed native parallel execution.
+
+The workflow policy fixtures still test shared-workspace sequencing,
+runtime-proven isolation, VERIFIED prerequisites, integration validation, and
+fresh review deterministically. They do not supply native execution telemetry.
+
 ## Generated workflow references
 
 `assistant-workflow` phase and plan views are generated from their authoritative
@@ -343,7 +400,10 @@ model-selection evidence and counts incomplete pairs. A second incomplete pair
 stops the batch before any later call,
 leaves remaining attempt records `not_started`, and withholds comparison and
 semantic-review artifacts. The exact limit is bound into the run plan as
-`max_incomplete_pairs=1`. The runner never retries an uncertain call.
+`max_incomplete_pairs=1`. Only the known pre-dispatch unavailable traces for the
+`isolated-parallel-a-b-then-c-with-integration` fixture are excluded; every other
+incomplete pair counts toward the limit. The runner never retries an uncertain
+call.
 
 An `in_flight` record without a valid trace is quota-uncertain: resume exits
 before every model call, reports only the bounded run ID, and requires separate
@@ -713,10 +773,14 @@ tools/evals/run-skill-evals.sh --emit-prompts /tmp/skill-eval-prompts
 tools/evals/run-skill-evals.sh --emit-prompts /tmp/clarify-eval-prompts --skill assistant-clarify
 ```
 
-Prompt packets are written under `<output>/<skill>/<case-id>.md` and include the
-setup context, prompt, expected behavior, pass criteria, fail signals, optional
-seeded defects / measurable assertions, machine expectations, and an optional
-Structured JSON Assertions section when the case declares one.
+Prompt packets are written under `<output>/<skill>/<case-id>.md`. By default,
+and when a case declares `prompt_packet_mode: annotated`, they include the setup
+context, prompt, expected behavior, pass criteria, fail signals, optional seeded
+defects / measurable assertions, machine expectations, and an optional Structured
+JSON Assertions section when the case declares one. A case may instead declare
+`prompt_packet_mode: task_only`; its target packet contains only a neutral header,
+skill identity and path, setup context, and prompt. The local grader always keeps
+the complete fixture, including its grading-only expectations.
 
 Run each prompt packet with the target assistant and save captured responses as
 `<response-dir>/<skill>/<case-id>.txt` or `<response-dir>/<skill>/<case-id>.md`.
@@ -742,15 +806,31 @@ Cases may additionally define `machine_expectations.structured_json_assertions`.
 For these per-skill cases, the response must contain exactly one valid JSON
 value. The local grader applies only the fixed provider-neutral operators:
 `equals`, `one_of`, `nonempty_string`, `nonempty_array`, `empty_array`, `array_type`, `array_nonblank_strings`, `path_absent`, `absent_or_empty_array`, `equals_path`,
-`required_when_equals`, `array_field_values_exact`, `array_object_values_exact`, and
-`array_items_nonempty_fields`, and `array_items_nonempty_array_fields`. Assertion paths are JSON arrays for safe
+`required_when_equals`, `array_field_values_exact`, `array_object_values_exact`,
+`object_keys_exact`, `array_items_nonempty_fields`, and
+`array_items_nonempty_array_fields`. Assertion paths are JSON arrays for safe
 `getpath` access. They are grader-only declarations, never executable fixture
 content: arbitrary jq, code, or expressions are not accepted. `array_items_nonempty_fields`
 requires the target array to contain at least one object, and every listed field
 in every object must be a non-empty string.
 `array_items_nonempty_array_fields` requires the target array to contain at
 least one object, and every listed field in every object must be a non-empty
 array whose every member is a nonblank string.
+`object_keys_exact` requires the target to be an object whose keys exactly match
+the declared `fields`; it rejects missing keys, extra keys, and non-object
+values. Its path may be `[]` to check the JSON response root, or a declared
+object path. It accepts at most 16 unique field names. Assertion paths follow
+the declared shape one segment at a time: each numeric segment consumes one
+array level, and string segments traverse fields only on an object. This admits
+object-array element fields and primitive-array elements while rejecting
+repeated or skipped indexes; schema descriptors are cloned when resolving an
+array element so the declared root remains unchanged.
+The assistant-workflow eval-only root registry explicitly projects the
+`decision_item` and `decision_resolution` object arrays as single objects for
+the `progressive-collaborative-contributor-evidence` fixture, whose prompt asks
+for those projected roots. Their canonical array descriptors remain available
+for indexed paths; this bounded compatibility does not permit skipped indexes
+on other arrays.
 `empty_array` requires the target path to resolve to an empty array.
 `array_type` requires the target path to resolve to an array and permits an empty array. `array_nonblank_strings` requires every member to be a nonblank string and a required boolean `allow_empty` declares whether an empty array is valid.
 In this exhaustive fixed operator list, `path_absent` passes only when its target

docs/evals/fixtures/pivot-restart-on-stagnation/RECOVERY.md

@@ -0,0 +1,30 @@
+# Stagnation recovery fixture
+
+The first trusted check writes a fixture-owned failure receipt, emits
+`STAGNATION_TRUSTED_FAILURE`, and fails. The recovery action validates that
+receipt and this bounded decision, then writes a pending recovery artifact and
+emits `RECOVERY_APPLIED`. Only then may the fresh check validate the receipts,
+mark the recovery artifact fresh-check passed, and emit
+`STAGNATION_FRESH_CHECK_PASS`.
+
+For this disposable fixture only, admitted workspace commands are:
+
+```text
+bash tests/stagnation-contracts.sh
+bash tests/recovery-contracts.sh
+bash tests/stagnation-contracts.sh --after-recovery
+cat RECOVERY.md (optional, read-only)
+```
+
+Any other started or completed command is rejected. The fixture does not permit
+discovery commands; this command boundary is not a framework-wide tool policy.
+
+```text
+terminal_completed=false
+next_action=assistant-debugging
+recovery_pointer=fixture-owned-recovery-artifacts
+```
+
+A passing fresh check proves the bounded recovery protocol was observed. It does
+not claim that the underlying legacy bug was repaired or that the workflow is
+complete. A patch or retry after the bound is rejected.

docs/evals/fixtures/pivot-restart-on-stagnation/tests/recovery-contracts.sh

@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+fixture_root="$(cd "$(dirname "$0")/.." && pwd -P)"
+cd "$fixture_root"
+
+grep -Fqx 'terminal_completed=false' RECOVERY.md
+grep -Fqx 'next_action=assistant-debugging' RECOVERY.md
+grep -Fqx 'recovery_pointer=fixture-owned-recovery-artifacts' RECOVERY.md
+[[ -d .assistant-eval && ! -L .assistant-eval && ! -L .assistant-eval/stagnation-failure-receipt.json && ! -L .assistant-eval/stagnation-recovery.json ]]
+[[ -f .assistant-eval/stagnation-failure-receipt.json ]]
+jq -e '
+  . == {
+    schema_version:"1.0",
+    terminal_completed:false,
+    next_action:"assistant-debugging",
+    recovery_pointer:"fixture-owned-recovery-artifacts"
+  }
+' .assistant-eval/stagnation-failure-receipt.json >/dev/null
+mkdir -p .assistant-eval
+[[ ! -L .assistant-eval && ! -L .assistant-eval/stagnation-recovery.json ]] || exit 1
+jq -cnS '
+  {
+    schema_version:"1.0",
+    trusted_failure:"observed",
+    recovery:"applied",
+    fresh_check:"pending",
+    terminal_completed:false
+  }
+' >.assistant-eval/stagnation-recovery.json
+
+printf '%s\n' 'RECOVERY_APPLIED'

docs/evals/fixtures/pivot-restart-on-stagnation/tests/stagnation-contracts.sh

@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+fixture_root="$(cd "$(dirname "$0")/.." && pwd -P)"
+cd "$fixture_root"
+
+if [[ "${1:-}" == "--after-recovery" ]]; then
+    failure_receipt=".assistant-eval/stagnation-failure-receipt.json"
+    recovery_artifact=".assistant-eval/stagnation-recovery.json"
+    [[ -d .assistant-eval && ! -L .assistant-eval && ! -L "$failure_receipt" && ! -L "$recovery_artifact" ]] || exit 1
+    [[ -f "$failure_receipt" && ! -L "$failure_receipt" && -f "$recovery_artifact" && ! -L "$recovery_artifact" ]] \
+        && jq -e '
+          . == {
+            schema_version:"1.0",
+            terminal_completed:false,
+            next_action:"assistant-debugging",
+            recovery_pointer:"fixture-owned-recovery-artifacts"
+          }
+        ' "$failure_receipt" >/dev/null \
+        && jq -e '
+          . == {
+            schema_version:"1.0",
+            trusted_failure:"observed",
+            recovery:"applied",
+            fresh_check:"pending",
+            terminal_completed:false
+          }
+        ' "$recovery_artifact" >/dev/null \
+        && grep -Fqx 'terminal_completed=false' RECOVERY.md \
+        && grep -Fqx 'next_action=assistant-debugging' RECOVERY.md \
+        && grep -Fqx 'recovery_pointer=fixture-owned-recovery-artifacts' RECOVERY.md \
+        || exit 1
+    jq -cnS '
+      {schema_version:"1.0",trusted_failure:"observed",recovery:"applied",fresh_check:"passed",terminal_completed:false}
+    ' >"$recovery_artifact"
+    printf '%s\n' 'STAGNATION_FRESH_CHECK_PASS'
+    exit 0
+fi
+
+if [[ -L .assistant-eval || ( -e .assistant-eval && ! -d .assistant-eval ) ]]; then
+    exit 1
+fi
+mkdir -p .assistant-eval
+[[ -d .assistant-eval && ! -L .assistant-eval && ! -L .assistant-eval/stagnation-failure-receipt.json ]] || exit 1
+jq -cnS '
+  {schema_version:"1.0",terminal_completed:false,next_action:"assistant-debugging",recovery_pointer:"fixture-owned-recovery-artifacts"}
+' >.assistant-eval/stagnation-failure-receipt.json
+printf '%s\n' 'STAGNATION_TRUSTED_FAILURE'
+exit 1