diff --git a/skills/manual-verify/SKILL.md b/skills/manual-verify/SKILL.md index 3986b41..0dee32d 100644 --- a/skills/manual-verify/SKILL.md +++ b/skills/manual-verify/SKILL.md @@ -1,31 +1,87 @@ --- name: manual-verify -description: Verify changes through real user workflows when hands-on testing would add confidence, including web apps in a browser and iOS apps in Simulator. +description: Verify changes through real user workflows when hands-on testing would add confidence, including web apps in a browser and native iOS or macOS apps. --- # Manual Verify -Think like someone who uses this product: who are they, what are they trying to -get done, and how could these changes affect them? +Exercise the product through the interface its users operate. Judge whether the +workflow works and remains usable; a successful command or build is supporting +evidence, not the verdict. -1. **Choose workflows.** Read the changes and identify the affected audience - and tasks. Use judgment to pick realistic workflows, including nearby behavior - that could break. Cover relevant error or edge cases without turning every - change into a full product audit. +## Workflow -2. **Use the product.** Use a browser for web apps and iOS Simulator for iOS apps. - For other tools, use the interface their users would use. Carry out the chosen - workflows and compare what happens with what the user needs. Check that the - result makes sense and is usable, beyond whether the action succeeds. +1. **Choose workflows.** Read the request and changes. Identify the affected + users, their goal, and nearby behavior the change could break. Select the + smallest useful set of realistic workflows covering the main path and any + material error or edge case. State the expected result of each workflow. - If login or another step needs the user, tell them exactly what to do and - where, then resume once they finish. Continue any independent checks meanwhile. - If the environment cannot support a needed check, report the gap. +2. **Choose the interface.** Use the product surface its users use. For a web + app, read [web verification](references/web.md). For an iOS app, read + [iOS verification](references/ios.md). For a native Mac app, read + [macOS verification](references/macos.md). Use the same observe-act-observe + loop for other interactive products with the best available interface tooling. -3. **Report what happened.** Briefly state the workflows tested, expected and - observed results, and any failures or gaps. Include screenshots or other - evidence when useful. Base conclusions on what you exercised; distinguish - untested behavior from passing checks. +3. **Observe, act, observe.** Start from a known state and wait for the interface + to settle. Identify the environment and external effects before exercising a + consequential workflow. Use disposable accounts, fixtures, and provider test + modes when available. Complete an irreversible or externally visible action + only when the task authorizes it; otherwise stop at the last safe step and + report the remaining gap. - After UI work, use `showroom` to package useful visual checkpoints. Keep - workflow selection and the pass or fail judgment here. + Inspect both the rendered appearance and semantic representation when + available. Perform one meaningful interaction, then inspect the resulting + state before continuing. Prefer semantic targets such as roles, labels, and + identifiers; use coordinates only when the interface exposes no stable target. + + If an interaction produces no expected change, record that outcome before any + retry. Retry once only when observable evidence shows that the target moved or + the interface had not settled, and count the retry as a separate attempt. + Treat a repeated failure as evidence instead of retrying until it disappears. + + When the reported behavior is intermittent, choose a bounded attempt count + before testing. Restore an equivalent known state with disposable or uniquely + identified data before every planned attempt so earlier outcomes cannot affect + later ones. Record every outcome. Recovery retries are separate from the + planned attempt count and must be recorded separately. Report `reproduced` or + `not reproduced in N attempts`; do not turn one successful attempt into a pass + for the intermittent report. + +4. **Judge the result.** Compare the observed result with the workflow's expected + result. Check function and usability, including relevant layout, readability, + focus, input, navigation, loading, empty, disabled, and error states. Distinguish: + + - functional failures, such as the wrong destination or an unresponsive action; + - visual failures, such as clipped, overlapping, unreadable, or misplaced content; + - crashes and unexpected exits; + - transient states that settle correctly; and + - expected states, such as a disabled submit action for incomplete input. + + When the expected result includes persisted data or an output artifact, verify + it through a fresh read path rather than relying on the current screen alone. + +5. **Report evidence.** For each workflow, state what was exercised and the + expected and observed result. Give each required acceptance assertion a status: + + - `passed` when the observed result matched the expected assertion; + - `failed` when the observed result differed from the expected assertion; + - `blocked` when a missing prerequisite prevented the check; or + - `untested` when the selected check was not exercised. + + Report defect reproduction separately as `reproduced` or + `not reproduced in N attempts`. A reproduced defect makes its affected + assertion and the overall workflow `failed`. When the defect is not reproduced, + report that outcome beside the acceptance status and state that it does not + establish that the defect is fixed. + + A workflow passes when every required assertion passed. Otherwise report every + non-passing assertion and use `failed` when any assertion failed, then + `blocked`, then `untested` as the overall status. Include screenshots, logs, or + semantic snapshots when they explain the conclusion. Name any untested + behavior or environmental gap rather than treating it as passing. + +If login or another step requires the user, tell them exactly what to do and +where, then resume after they finish. Continue independent checks meanwhile. + +After UI work, use `showroom` to package useful visual checkpoints. Keep workflow +selection and the pass or fail judgment in this skill. diff --git a/skills/manual-verify/references/ios.md b/skills/manual-verify/references/ios.md new file mode 100644 index 0000000..b4d7b09 --- /dev/null +++ b/skills/manual-verify/references/ios.md @@ -0,0 +1,50 @@ +# iOS Verification + +Use a Simulator that supports the app's deployment target. Identify the project +or workspace, scheme, build configuration, Simulator model, and runtime used for +the check. Prefer XcodeBuildMCP when it is available; otherwise use the host's +Simulator and Xcode tooling. + +## Build and launch + +- Build and run the intended scheme. If the build fails, preserve the failure and + determine whether it comes from the product or verification environment. Report + it as the workflow result. Fix code only when the active task includes fixing + failures, then restart verification against the changed revision. +- After launch, confirm that the expected app and screen are visible. Capture a + screenshot and inspect the accessibility hierarchy when the available tooling + exposes it. +- Preserve the workflow's data assumptions. Reset, reinstall, seed, or relaunch + only when the required starting state calls for it, and report material setup. + +## Observe and interact + +- Target controls by accessibility identifier or label when supported. When the + available tooling exposes only rendered state, interact from the current view, + recapture after every action, and report semantic or accessibility coverage as + untested. Derive coordinate targets from current hierarchy or bounds when they + are available rather than from a stale screenshot. +- Recapture the hierarchy after navigation, presentation, rotation, animation, or + any interaction that may move the target. +- Exercise text input, gestures, scrolling, orientation, permissions, and + background or foreground transitions when they belong to the selected workflow. +- When a save or update should persist, verify it through a fresh read path by + relaunching and refetching, inspecting the stored value through an independent + interface, or reopening through a path known to create a new model and reload + durable storage. Record the persisted value observed. +- Check both function and presentation: hit targets, clipping, Dynamic Type where + relevant, safe areas, keyboard avoidance, and expected system dialogs. +- Capture application logs when the app crashes, exits, hangs, or behaves + differently from the visible state. Use the process and bundle identifier to + separate app output from unrelated Simulator noise. + +Keep an explicitly requested Simulator check in Simulator. Exercise the portions +the Simulator supports and name the exact device-only step that remains unverified. +Use a physical device when the request includes device verification and suitable +hardware is available. + +## Evidence + +Record the scheme, Simulator model and runtime, workflow states, and relevant +screenshots or logs. A successful build proves that the app compiled; the +interaction and resulting state prove whether the workflow worked. diff --git a/skills/manual-verify/references/macos.md b/skills/manual-verify/references/macos.md new file mode 100644 index 0000000..f061afb --- /dev/null +++ b/skills/manual-verify/references/macos.md @@ -0,0 +1,49 @@ +# macOS Verification + +Build and launch the intended app configuration. Record the macOS version, app +build or revision, and any fixture or account state that affects the check. Use +the host's native app or computer-use tooling, with accessibility inspection when +it is available. + +## Observe and interact + +- Confirm that the expected app and window are active before interacting. Inspect + the rendered interface and accessibility hierarchy when the tooling exposes it. +- Target controls by accessibility role, label, or identifier when supported. + With rendered-state tooling, recapture after each action and report semantic or + accessibility coverage as untested. +- Exercise menus, keyboard shortcuts, focus, window resizing, sheets, popovers, + multiple windows, and full-screen behavior when they belong to the selected + workflow. Give multiple windows distinct observable states, confirm which one + is active, then inspect both the active window's expected change and the + inactive window's unchanged state. +- For open, save, import, export, or drag-and-drop workflows, use disposable + copies while preserving the source fixtures. Record the paths and expected + changes. To verify an explicit save action, capture an independent on-disk + baseline and the document's dirty state immediately before invoking it, then + observe the command-specific state transition and inspect the on-disk result. + If autosave removes the dirty precondition before the command or makes + attribution impossible, report persistence separately and mark the save-action + assertion `blocked`. Mark it `untested` when the command was not exercised. If + the document was dirty, the command was exercised, and neither the expected + transition nor persisted output appears after settling, mark it `failed`. + Closing and reopening the recorded path can provide additional persistence + evidence. Record the persisted value observed. +- Treat permission prompts and other system UI as part of the workflow. Record + any permission state that a later run would need to reproduce. A controlled + denial requested by the workflow is a test precondition: keep it denied unless + the task authorizes changing it, then check the app's error or fallback behavior + and confirm that it did not produce restricted output. +- Capture application logs or crash reports when the app exits, hangs, or behaves + differently from the visible state. + +When the verifier cannot establish or observe a required state because hardware, +services, entitlements, permissions, or system configuration are unavailable, +exercise the supported portion and mark only the affected assertions blocked. + +## Evidence + +Record the macOS version, app build or revision, workflow states, affected and +unaffected windows, and relevant screenshots, inspected output files, or logs. A +successful build proves that the app compiled; the interaction and resulting +state prove whether the workflow worked. diff --git a/skills/manual-verify/references/web.md b/skills/manual-verify/references/web.md new file mode 100644 index 0000000..711a3da --- /dev/null +++ b/skills/manual-verify/references/web.md @@ -0,0 +1,26 @@ +# Web Verification + +Use the browser environment already attached to the task when possible. Start or +locate the development server, open the exact changed route, and record the +viewport or device mode when it affects the result. + +## Observe and interact + +- Capture a browser snapshot or inspect the DOM and accessibility tree before + interacting. Pair semantic inspection with a screenshot when appearance matters. +- Target elements by role, accessible name, label, or another stable locator. + Reinspect the page after navigation, re-rendering, or layout changes before + reusing a target. +- Exercise pointer, keyboard, focus, scrolling, and responsive behavior when the + change affects them. Test only the viewports and input methods relevant to the + requested workflow and likely regression risk. +- Wait for visible loading to settle. If an action produces no clear result, + inspect the console and failed network requests before deciding whether the + product or the environment failed. + +## Evidence + +Capture the states that establish the result: the initial condition, the outcome, +and any failure. Record the route, viewport, and relevant console or network +evidence. Do not infer success from a screenshot when behavior required an +interaction.