From d3c9653d022e929d3fb5c1ad8d70d4a9f1647d0d Mon Sep 17 00:00:00 2001 From: Sree Raman <1271305+sreejithraman@users.noreply.github.com> Date: Sun, 20 Sep 2026 22:34:37 -0400 Subject: [PATCH 1/2] Expand manual verification for web and iOS --- skills/manual-verify/SKILL.md | 83 ++++++++++++++++++-------- skills/manual-verify/references/ios.md | 46 ++++++++++++++ skills/manual-verify/references/web.md | 26 ++++++++ 3 files changed, 131 insertions(+), 24 deletions(-) create mode 100644 skills/manual-verify/references/ios.md create mode 100644 skills/manual-verify/references/web.md diff --git a/skills/manual-verify/SKILL.md b/skills/manual-verify/SKILL.md index 3986b41..15cc904 100644 --- a/skills/manual-verify/SKILL.md +++ b/skills/manual-verify/SKILL.md @@ -5,27 +5,62 @@ description: Verify changes through real user workflows when hands-on testing wo # Manual Verify -Think like someone who uses this product: who are they, what are they trying to -get done, and how could these changes affect them? - -1. **Choose workflows.** Read the changes and identify the affected audience - and tasks. Use judgment to pick realistic workflows, including nearby behavior - that could break. Cover relevant error or edge cases without turning every - change into a full product audit. - -2. **Use the product.** Use a browser for web apps and iOS Simulator for iOS apps. - For other tools, use the interface their users would use. Carry out the chosen - workflows and compare what happens with what the user needs. Check that the - result makes sense and is usable, beyond whether the action succeeds. - - If login or another step needs the user, tell them exactly what to do and - where, then resume once they finish. Continue any independent checks meanwhile. - If the environment cannot support a needed check, report the gap. - -3. **Report what happened.** Briefly state the workflows tested, expected and - observed results, and any failures or gaps. Include screenshots or other - evidence when useful. Base conclusions on what you exercised; distinguish - untested behavior from passing checks. - - After UI work, use `showroom` to package useful visual checkpoints. Keep - workflow selection and the pass or fail judgment here. +Exercise the product through the interface its users operate. Judge whether the +workflow works and remains usable; a successful command or build is supporting +evidence, not the verdict. + +## Workflow + +1. **Choose workflows.** Read the request and changes. Identify the affected + users, their goal, and nearby behavior the change could break. Select the + smallest useful set of realistic workflows covering the main path and any + material error or edge case. State the expected result of each workflow. + +2. **Choose the interface.** Use the product surface its users use. For a web + app, read [web verification](references/web.md). For an iOS app, read + [iOS verification](references/ios.md). Use the same observe-act-observe loop + for other interactive products with the best available interface tooling. + +3. **Observe, act, observe.** Start from a known state and wait for the interface + to settle. Identify the environment and external effects before exercising a + consequential workflow. Use disposable accounts, fixtures, and provider test + modes when available. Complete an irreversible or externally visible action + only when the task authorizes it; otherwise stop at the last safe step and + report the remaining gap. + + Inspect both the rendered appearance and semantic representation when + available. Perform one meaningful interaction, then inspect the resulting + state before continuing. Prefer semantic targets such as roles, labels, and + identifiers; use coordinates only when the interface exposes no stable target. + + If an interaction appears to fail because the interface moved or was still + loading, recapture its state and retry once. Treat a repeated failure as + evidence instead of retrying until it disappears. + + When the reported behavior is intermittent, choose a bounded attempt count + before testing, repeat the same controlled workflow, and record every outcome. + Report `reproduced` or `not reproduced in N attempts`; do not turn one + successful attempt into a pass for the intermittent report. + +4. **Judge the result.** Compare the observed result with the workflow's expected + result. Check function and usability, including relevant layout, readability, + focus, input, navigation, loading, empty, disabled, and error states. Distinguish: + + - functional failures, such as the wrong destination or an unresponsive action; + - visual failures, such as clipped, overlapping, unreadable, or misplaced content; + - crashes and unexpected exits; + - transient states that settle correctly; and + - expected states, such as a disabled submit action for incomplete input. + +5. **Report evidence.** For each workflow, state what was exercised, the expected + result, the observed result, and its status: `passed`, `failed` or `reproduced`, + `not reproduced in N attempts`, or `blocked` or `untested`. A result that was + not reproduced is not a pass. Include screenshots, logs, or semantic snapshots + when they explain the conclusion. Name any untested behavior or environmental + gap rather than treating it as passing. + +If login or another step requires the user, tell them exactly what to do and +where, then resume after they finish. Continue independent checks meanwhile. + +After UI work, use `showroom` to package useful visual checkpoints. Keep workflow +selection and the pass or fail judgment in this skill. diff --git a/skills/manual-verify/references/ios.md b/skills/manual-verify/references/ios.md new file mode 100644 index 0000000..b3ee122 --- /dev/null +++ b/skills/manual-verify/references/ios.md @@ -0,0 +1,46 @@ +# iOS Verification + +Use a Simulator that supports the app's deployment target. Identify the project +or workspace, scheme, build configuration, Simulator model, and runtime used for +the check. Prefer XcodeBuildMCP when it is available; otherwise use the host's +Simulator and Xcode tooling. + +## Build and launch + +- Build and run the intended scheme. If the build fails, preserve the failure and + determine whether it comes from the product or verification environment. Report + it as the workflow result. Fix code only when the active task includes fixing + failures, then restart verification against the changed revision. +- After launch, confirm that the expected app and screen are visible. Capture a + screenshot and inspect the accessibility hierarchy when the available tooling + exposes it. +- Preserve the workflow's data assumptions. Reset, reinstall, seed, or relaunch + only when the required starting state calls for it, and report material setup. + +## Observe and interact + +- Target controls by accessibility identifier or label when supported. When the + available tooling exposes only rendered state, interact from the current view, + recapture after every action, and report semantic or accessibility coverage as + untested. Derive coordinate targets from current hierarchy or bounds when they + are available rather than from a stale screenshot. +- Recapture the hierarchy after navigation, presentation, rotation, animation, or + any interaction that may move the target. +- Exercise text input, gestures, scrolling, orientation, permissions, and + background or foreground transitions when they belong to the selected workflow. +- Check both function and presentation: hit targets, clipping, Dynamic Type where + relevant, safe areas, keyboard avoidance, and expected system dialogs. +- Capture application logs when the app crashes, exits, hangs, or behaves + differently from the visible state. Use the process and bundle identifier to + separate app output from unrelated Simulator noise. + +Keep an explicitly requested Simulator check in Simulator. Exercise the portions +the Simulator supports and name the exact device-only step that remains unverified. +Use a physical device when the request includes device verification and suitable +hardware is available. + +## Evidence + +Record the scheme, Simulator model and runtime, workflow states, and relevant +screenshots or logs. A successful build proves that the app compiled; the +interaction and resulting state prove whether the workflow worked. diff --git a/skills/manual-verify/references/web.md b/skills/manual-verify/references/web.md new file mode 100644 index 0000000..711a3da --- /dev/null +++ b/skills/manual-verify/references/web.md @@ -0,0 +1,26 @@ +# Web Verification + +Use the browser environment already attached to the task when possible. Start or +locate the development server, open the exact changed route, and record the +viewport or device mode when it affects the result. + +## Observe and interact + +- Capture a browser snapshot or inspect the DOM and accessibility tree before + interacting. Pair semantic inspection with a screenshot when appearance matters. +- Target elements by role, accessible name, label, or another stable locator. + Reinspect the page after navigation, re-rendering, or layout changes before + reusing a target. +- Exercise pointer, keyboard, focus, scrolling, and responsive behavior when the + change affects them. Test only the viewports and input methods relevant to the + requested workflow and likely regression risk. +- Wait for visible loading to settle. If an action produces no clear result, + inspect the console and failed network requests before deciding whether the + product or the environment failed. + +## Evidence + +Capture the states that establish the result: the initial condition, the outcome, +and any failure. Record the route, viewport, and relevant console or network +evidence. Do not infer success from a screenshot when behavior required an +interaction. From 6f699117f4573197440625a2f1dbc0e9ab866711 Mon Sep 17 00:00:00 2001 From: Sree Raman <1271305+sreejithraman@users.noreply.github.com> Date: Sun, 20 Sep 2026 22:55:18 -0400 Subject: [PATCH 2/2] Add macOS manual verification --- skills/manual-verify/SKILL.md | 51 +++++++++++++++++------- skills/manual-verify/references/ios.md | 4 ++ skills/manual-verify/references/macos.md | 49 +++++++++++++++++++++++ 3 files changed, 89 insertions(+), 15 deletions(-) create mode 100644 skills/manual-verify/references/macos.md diff --git a/skills/manual-verify/SKILL.md b/skills/manual-verify/SKILL.md index 15cc904..0dee32d 100644 --- a/skills/manual-verify/SKILL.md +++ b/skills/manual-verify/SKILL.md @@ -1,6 +1,6 @@ --- name: manual-verify -description: Verify changes through real user workflows when hands-on testing would add confidence, including web apps in a browser and iOS apps in Simulator. +description: Verify changes through real user workflows when hands-on testing would add confidence, including web apps in a browser and native iOS or macOS apps. --- # Manual Verify @@ -18,8 +18,9 @@ evidence, not the verdict. 2. **Choose the interface.** Use the product surface its users use. For a web app, read [web verification](references/web.md). For an iOS app, read - [iOS verification](references/ios.md). Use the same observe-act-observe loop - for other interactive products with the best available interface tooling. + [iOS verification](references/ios.md). For a native Mac app, read + [macOS verification](references/macos.md). Use the same observe-act-observe + loop for other interactive products with the best available interface tooling. 3. **Observe, act, observe.** Start from a known state and wait for the interface to settle. Identify the environment and external effects before exercising a @@ -33,14 +34,18 @@ evidence, not the verdict. state before continuing. Prefer semantic targets such as roles, labels, and identifiers; use coordinates only when the interface exposes no stable target. - If an interaction appears to fail because the interface moved or was still - loading, recapture its state and retry once. Treat a repeated failure as - evidence instead of retrying until it disappears. + If an interaction produces no expected change, record that outcome before any + retry. Retry once only when observable evidence shows that the target moved or + the interface had not settled, and count the retry as a separate attempt. + Treat a repeated failure as evidence instead of retrying until it disappears. When the reported behavior is intermittent, choose a bounded attempt count - before testing, repeat the same controlled workflow, and record every outcome. - Report `reproduced` or `not reproduced in N attempts`; do not turn one - successful attempt into a pass for the intermittent report. + before testing. Restore an equivalent known state with disposable or uniquely + identified data before every planned attempt so earlier outcomes cannot affect + later ones. Record every outcome. Recovery retries are separate from the + planned attempt count and must be recorded separately. Report `reproduced` or + `not reproduced in N attempts`; do not turn one successful attempt into a pass + for the intermittent report. 4. **Judge the result.** Compare the observed result with the workflow's expected result. Check function and usability, including relevant layout, readability, @@ -52,12 +57,28 @@ evidence, not the verdict. - transient states that settle correctly; and - expected states, such as a disabled submit action for incomplete input. -5. **Report evidence.** For each workflow, state what was exercised, the expected - result, the observed result, and its status: `passed`, `failed` or `reproduced`, - `not reproduced in N attempts`, or `blocked` or `untested`. A result that was - not reproduced is not a pass. Include screenshots, logs, or semantic snapshots - when they explain the conclusion. Name any untested behavior or environmental - gap rather than treating it as passing. + When the expected result includes persisted data or an output artifact, verify + it through a fresh read path rather than relying on the current screen alone. + +5. **Report evidence.** For each workflow, state what was exercised and the + expected and observed result. Give each required acceptance assertion a status: + + - `passed` when the observed result matched the expected assertion; + - `failed` when the observed result differed from the expected assertion; + - `blocked` when a missing prerequisite prevented the check; or + - `untested` when the selected check was not exercised. + + Report defect reproduction separately as `reproduced` or + `not reproduced in N attempts`. A reproduced defect makes its affected + assertion and the overall workflow `failed`. When the defect is not reproduced, + report that outcome beside the acceptance status and state that it does not + establish that the defect is fixed. + + A workflow passes when every required assertion passed. Otherwise report every + non-passing assertion and use `failed` when any assertion failed, then + `blocked`, then `untested` as the overall status. Include screenshots, logs, or + semantic snapshots when they explain the conclusion. Name any untested + behavior or environmental gap rather than treating it as passing. If login or another step requires the user, tell them exactly what to do and where, then resume after they finish. Continue independent checks meanwhile. diff --git a/skills/manual-verify/references/ios.md b/skills/manual-verify/references/ios.md index b3ee122..b4d7b09 100644 --- a/skills/manual-verify/references/ios.md +++ b/skills/manual-verify/references/ios.md @@ -28,6 +28,10 @@ Simulator and Xcode tooling. any interaction that may move the target. - Exercise text input, gestures, scrolling, orientation, permissions, and background or foreground transitions when they belong to the selected workflow. +- When a save or update should persist, verify it through a fresh read path by + relaunching and refetching, inspecting the stored value through an independent + interface, or reopening through a path known to create a new model and reload + durable storage. Record the persisted value observed. - Check both function and presentation: hit targets, clipping, Dynamic Type where relevant, safe areas, keyboard avoidance, and expected system dialogs. - Capture application logs when the app crashes, exits, hangs, or behaves diff --git a/skills/manual-verify/references/macos.md b/skills/manual-verify/references/macos.md new file mode 100644 index 0000000..f061afb --- /dev/null +++ b/skills/manual-verify/references/macos.md @@ -0,0 +1,49 @@ +# macOS Verification + +Build and launch the intended app configuration. Record the macOS version, app +build or revision, and any fixture or account state that affects the check. Use +the host's native app or computer-use tooling, with accessibility inspection when +it is available. + +## Observe and interact + +- Confirm that the expected app and window are active before interacting. Inspect + the rendered interface and accessibility hierarchy when the tooling exposes it. +- Target controls by accessibility role, label, or identifier when supported. + With rendered-state tooling, recapture after each action and report semantic or + accessibility coverage as untested. +- Exercise menus, keyboard shortcuts, focus, window resizing, sheets, popovers, + multiple windows, and full-screen behavior when they belong to the selected + workflow. Give multiple windows distinct observable states, confirm which one + is active, then inspect both the active window's expected change and the + inactive window's unchanged state. +- For open, save, import, export, or drag-and-drop workflows, use disposable + copies while preserving the source fixtures. Record the paths and expected + changes. To verify an explicit save action, capture an independent on-disk + baseline and the document's dirty state immediately before invoking it, then + observe the command-specific state transition and inspect the on-disk result. + If autosave removes the dirty precondition before the command or makes + attribution impossible, report persistence separately and mark the save-action + assertion `blocked`. Mark it `untested` when the command was not exercised. If + the document was dirty, the command was exercised, and neither the expected + transition nor persisted output appears after settling, mark it `failed`. + Closing and reopening the recorded path can provide additional persistence + evidence. Record the persisted value observed. +- Treat permission prompts and other system UI as part of the workflow. Record + any permission state that a later run would need to reproduce. A controlled + denial requested by the workflow is a test precondition: keep it denied unless + the task authorizes changing it, then check the app's error or fallback behavior + and confirm that it did not produce restricted output. +- Capture application logs or crash reports when the app exits, hangs, or behaves + differently from the visible state. + +When the verifier cannot establish or observe a required state because hardware, +services, entitlements, permissions, or system configuration are unavailable, +exercise the supported portion and mark only the affected assertions blocked. + +## Evidence + +Record the macOS version, app build or revision, workflow states, affected and +unaffected windows, and relevant screenshots, inspected output files, or logs. A +successful build proves that the app compiled; the interaction and resulting +state prove whether the workflow worked.