diff --git a/.agents/skills/argent-create-flow/SKILL.md b/.agents/skills/argent-create-flow/SKILL.md index 664861b26e..ce9c90ee4d 100644 --- a/.agents/skills/argent-create-flow/SKILL.md +++ b/.agents/skills/argent-create-flow/SKILL.md @@ -1,299 +1,50 @@ --- name: argent-create-flow -description: Record a reusable flow (scripted sequence of MCP tool calls) that can be replayed later with a single command. Use when the user asks to create, record, or build a flow, or to script a sequence of device actions. Also used proactively, without an explicit request, when a multi-step interaction sequence is about to be repeated (re-profiling, re-testing, or a complex path worth saving). +description: Create, record, edit, replay, or repair reusable Argent flow YAML files. Use when the user asks to record or replay a repeatable device path, set up profiling or an A/B comparison, or invoke the authoring engine behind argent-qa-flows. Also use before repeating three or more interactions. For one-off UI checks, acceptance-driven regression tests, or screen video, use argent-test-ui-flow, argent-qa-flows, or argent-screen-recording respectively. --- -## Overview +# Create an Argent flow -A flow is a sequence of steps saved to a `.yaml` file in the `.argent/flows/` directory. Each recorded step is **executed live** as you add it, so you verify it works before it becomes part of the flow. Replay a finished flow with `flow-execute`, or — for an e2e flow — headlessly with `argent flow run `. +An Argent flow is a replayable sequence in `.argent/flows/.yaml`. -Flows store **no device id**: the runner binds a device (the single booted one, or pass `device`/`platform`). A recorded coordinate `gesture-tap` is captured as a portable `tap: { selector }` step whenever the tapped element has stable text/identifier. +For a saved QA test case, ticket, or acceptance criterion, load `argent-qa-flows` first. It adds deterministic setup, acceptance evidence, and two-pass proof. -**Two flow types** +## Read the relevant reference -- **e2e** — begins with a `launch:` step, which starts that app from scratch (terminate + relaunch), so the flow controls its own start state. No `executionPrerequisite`. May `run:` other flows, and (on iOS/Android) may itself be a `run:` target — when nested, its `launch` runs inline, restarting the app for that sub-scenario. **Chromium is the exception:** the runner boots one Electron app per run (the top-level flow's), so a nested chromium e2e flow's `launch` can't boot its own instance and fails the run — keep chromium e2e flows top-level. Record one by adding a `restart-app` of the app under test as the **first** step — it is captured as the `launch` step. -- **fragment** — doesn't begin with a launch; runs against the device's current state. May declare an `executionPrerequisite` (a documented entry-state contract). Invoked from other flows via a `run:` step, or directly by you at any time. +- Before creating or changing a flow, read [Live authoring](references/live-authoring.md) completely. +- When polishing, composing, or manually reviewing YAML, read [Flow YAML](references/flow-yaml.md). For Vega, read its platform limits before recording remote or keyboard tools. +- Flows run on physical iPhones (an iOS `list-devices` entry with kind `"device"`), but replay never auto-binds one, even when no simulator is booted: pass the phone's udid as `device` (CLI `--device`), and only a `connected` phone can run. `pinch`/`rotate` steps fail there like the live tools. On hardware the flow tree is the `describe` tree: same ids and roles, no UIView hierarchy. See `argent-ios-device-interact` for the hardware contract. +- On capture warnings, raw coordinates, unavailable trees, mistimed transitions, overlays, or replay failures, read [Reliability and recovery](references/reliability-and-recovery.md). -Both run via `argent flow run ` — a fragment simply runs against whatever is on screen (its prerequisite is printed as a reminder). Only e2e flows are meaningful CI/suite entries, since only they give a deterministic verdict from a clean start. +## Non-negotiable rules -### Step directives +1. **Record the first walkthrough.** Start the recorder before the first launch or in-app action. Do not reconstruct a rehearsed path. +2. **Record checks when their states appear.** Record `await-ui-element` live, then convert it during polish. An echo records intent or diagnostic context, not app behavior or a verdict. A screenshot is human evidence, not an executable verdict. For absence, record the same selector as `visible`, perform the removing action, then record it as `hidden`. +3. **Use semantic targets.** Prefer a strict id, then stable text or an accessibility label. Use `scroll-to` for off-screen elements. Resolve every raw-point warning immediately through the [coordinate fallback gate](references/reliability-and-recovery.md#coordinate-fallback-gate). +4. **Prove every screen change.** Record a destination-only identity check. During polish, follow it with `await: { idle: true }`. Stillness does not prove identity, and `idle` can pass with a warning. +5. **Polish only executed behavior.** Convert recorded steps without changing their meaning. Record any missing action or structural check live. The only unrecorded insertions are a planned `snapshot:`, a navigation `await: { idle: true }`, and the documented Chromium packaging `launch:`. +6. **Replay the final YAML end to end.** A normal flow needs one uninterrupted full pass. `argent-qa-flows` requires two consecutive passes. -Beyond raw `tool:` steps and `echo:`, flows support declarative directives interpreted by the runner (they are **not** agent-callable tools). **Every directive hard-stops the flow on failure**; later steps are reported `skip`. +### Stable selectors -| Directive | YAML | Meaning | -| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `launch` | `- launch: com.acme.app` or `- launch: { ios: …, android: … }` | start the app from scratch (terminate + relaunch) and wait until ready | -| `tap` | `- tap: Login`, `- tap: { x: 0.5, y: 0.57 }`, `- tap: { on: Login, times: 2 }`, `- tap: { on: { x: 0.5, y: 0.57 }, times: 2 }` | tap by selector (auto-waits) or raw point; `times` (2 = double-tap) needs the target nested under `on:` — a selector or a point (`{ x, y, times }` is rejected) | -| `long-press` | `- long-press: Row 3`, `- long-press: { x: 0.5, y: 0.6 }`, `- long-press: { on: , duration: 1200 }`, `- long-press: { on: { x: 0.5, y: 0.6 }, duration: 1200 }` | press and hold an element or raw point (default 800ms; Chromium: mouse press-hold); `duration` needs the target nested under `on:` — a selector or a point | -| `type` | `- type: { into: email, text: "a@b.com" }` | focus a field, type, then press Enter to submit + dismiss the keyboard | -| `scroll-to` | `- scroll-to: "Order #1234"` (scrolls down) or `- scroll-to: { target: …, direction: right, within: … }` | momentum-free scroll until the target is visible | -| `pinch` | `- pinch: { on: "Map", scale: 3 }` or `- pinch: { scale: 0.5 }` | two-finger zoom in (`scale` > 1) or out (`< 1`); big scales chain gestures; `on` optional — defaults to screen center; open-loop — assert the visible result | -| `rotate` | `- rotate: { on: "Map", by: 90 }` or `- rotate: { by: -45 }` | two-finger rotation by degrees (+ CW, − CCW, within ±3000°; options map only); `on` optional — screen center default; not `tool: rotate` (orientation) | -| `await` | `- await: { visible: Home }` | wait for a UI condition | -| `wait` | `- wait: 500` | pause for a fixed number of milliseconds (last resort — prefer `await`) | -| `assert` | `- assert: { visible: Welcome }` | check a condition, hard-fail if it never holds | -| `snapshot` | `- snapshot: home` or `- snapshot: { name: home, maxMismatch: 0.5, cropOn: { id: order-summary } }` | diff a screenshot — or one element's region — against a stored baseline | -| `run` | `- run: login` | execute another flow's steps inline (fragment or e2e) | -| `when` | `- when: { visible: "What's new" }` + `steps: [...]` | run a guarded step block only when the condition holds (no else) | +A stable selector is fixed by app code and survives account, data, time, count, order, and every locale and environment the flow supports. Prefer ids such as `settings-screen`. Do not gate on values such as `Today`, `Item 4`, usernames, counters, or timestamps. -### Selectors +### Flow-only selector scopes -A **selector** is `{ text?, id?, role? }` plus the optional scopes below (all-must-match; `text`/`role` are case-insensitive substrings, `id` matches the element's testID / accessibilityIdentifier / resource-id exactly, case-insensitive, also accepting the unqualified Android resource-id name — `submit` matches `com.example.app:id/submit`) — the same semantics `await-ui-element` uses, though that tool spells the `id` field `identifier` (flow YAML also accepts `identifier` as an alias for `id`, but `id` is the canonical spelling and what the recorder writes). A bare string is a _loose_ selector: it resolves **identifier-first, then falls back to text** (label/value), so `tap: Login` matches a `testID="Login"` or, failing that, visible text "Login" — no need to know which. Loose fallback applies uniformly to every selector slot (`tap`, `type.into`, `await`, `assert`, `scroll-to`). Use the map form to be strict: `{ id: submit-btn }` (identifier only) or `{ text: Login }` (text only, no fallback). +During polish, use `within`, `after`, and `next` to disambiguate repeated elements. Read [Flow YAML: Relational scopes](references/flow-yaml.md#relational-scopes) for their frame-based semantics and failure cases. -`text` also takes a **regex matcher map** — `{ text: { matches: '^Order #\d+$' } }`, in any selector slot — for dynamic text no literal can pin. It tests each node's native **own** label/value (not the adapter-hoisted `subtreeText`), though on iOS a container's own label may itself aggregate descendant text, so a wrapper and its leaf can both match. Same regex rules as `text.in`'s `matches` (see _`await` and `assert`_): unanchored, **case-sensitive**, single-quoted, invalid pattern fails at parse. So `assert: { visible: { text: { matches: '^Taps: \d+$' } } }` asserts a counter is on screen with no locator at all, and `tap: { text: { matches: '^Order #\d+$' } }` taps a dynamic row — though a stable `id` stays the more robust action target. +## Workflow -#### Scopes — CSS combinators, read off frames +1. Choose the flow type: + - **e2e:** the first non-echo step is `launch:`. The flow controls process start. + - **fragment:** there is no leading launch. Declare a precise `executionPrerequisite`. +2. Follow [Live authoring](references/live-authoring.md): start, record one verified step at a time, finish, polish, audit, and replay. +3. Report the file, replay command, result, prerequisite or side effects, and every coordinate or raw-gesture exception. -A map selector may also carry a **scope**: a nested selector naming another element the match must sit in a given spatial relation to. These are the geometric readings of the CSS combinators that survive a flattened tree — the child combinator `>` has no analog, since parent/child structure does not reach replay. They combine, and nest up to six scopes per selector: +## Proactive recording -| Scope | CSS | Reads as | Example | -| --------------- | ------- | --------------------------------------------------------------- | --------------------------------------------------------------------- | -| `within: ` | `A B` | the match's frame sits **inside** the scope element's frame | `tap: { text: Delete, within: { id: profile-card } }` | -| `after: ` | `A ~ B` | the match **follows** the scope element in reading order | `assert: { visible: { role: Button, after: { text: Danger zone } } }` | -| `next: ` | `A + B` | as `after`, narrowed to the **nearest** follower of each anchor | `tap: { role: Switch, next: { text: Wi-Fi } }` | +Before repeating three or more interactions, tell the user and start a recording. Record that run and replay it afterward. A completed path cannot be recorded retroactively. -And `any: true` is the CSS `*` universal selector. It carries no locator of its own, so the parser enforces two rules: it needs **at least one scope** (a bare `any: true` would match the whole screen), and it may **not** sit beside `text`/`id`/`role` (which it would only make redundant — write `{ role: Switch, next: … }`, not `{ any: true, role: Switch, next: … }`). Only the literal `true` is accepted. `assert: { hidden: { any: true, within: { id: empty-state } } }` asserts an empty container. It works for actions too — `tap: { any: true, next: { text: Airplane Mode } }` taps whatever sits right after that label — but on a real tree "whatever" includes spacers and wrappers, so **name the target** (`role`, `id`) whenever you can and keep `any` for conditions and for `next`, which reduces to a single element per anchor. +## Repair -All of it is **visual (frame-based), not tree ancestry** — flow trees are flattened, and "inside the card" / "the switch after this label" mean what the screen shows — the same frame-based reading of "within" that `scroll-to`'s container anchor uses. Every scope needs a **distinct element** (nothing scopes itself, so `{ id: card, within: { id: card } }` needs two nested elements), the synthetic screen root never counts, and a scope only narrows _where_ to look — the selector still needs its own `text`/`id`/`role` naming _what_ to find there, or `any: true`. The nested slot takes every selector form: a bare string keeps the loose identifier-first fallback (`within: profile-card`), the map form stays strict, the regex matcher works (`within: { text: { matches: '^Card \d+$' } }`). Scopes are flow-YAML only; the raw `await-ui-element` tool's selector accepts none of them. Prefer a unique `id` on the target itself when one exists — reach for a scope when the target has no unique locator of its own (repeated row actions, per-card buttons, list cells). - -**`within`** — `tap: { text: Delete, within: { id: profile-card } }` taps the Delete button in the profile card even when other cards show identical ones; `tap: { text: "Pin feed", within: { text: "For You" } }` picks one card's button out of a whole list. Scope to a container with a **tight frame** (a row, card, dialog, toast): a full-screen wrapper contains everything and scopes nothing. It chains outward: `{ text: Save, within: { id: cards, within: Settings } }` reads "Save inside cards inside Settings", each container's frame inside the next. - -**`after` / `next`** — reading order is row-band aware: an element **follows** the anchor when it starts below the anchor's bottom edge, _or_ shares its row band and sits entirely to its right. That is what makes `{ role: Switch, next: { text: Wi-Fi } }` resolve the Wi-Fi row's own switch even though the taller switch's frame starts a couple of pixels _higher_ than the label's. `next` keeps only the nearest follower — a match in the anchor's own row beats anything on the rows below, leftmost first — while `after` keeps them all, so `assert: { hidden: { role: Button, after: { text: Danger zone } } }` holds when nothing button-like appears past that heading. Both union over anchors exactly as CSS does: with three rows on screen, `{ role: Switch, next: { role: AXStaticText } }` yields all three switches, one per label. Note that "follows" is **not transitive** — against a tall anchor, an element can follow something that itself follows the anchor without following the anchor directly — so nesting `after` scopes is not the same as chaining CSS `~`: `{ after: { …, after: … } }` can match elements a single `after` excludes. Nest them only when each link is a container-sized step. An element sitting _inside_ the anchor does not follow **that** anchor — containment is not reading order — so scope by `within` for that. - -**Prefer the map form for a scope's anchor.** A bare-string scope (`next: wifi-row`) keeps the loose identifier-first fallback, and the runner takes the first pass that finds a _visible_ match — so if some unrelated element carries `testID="wifi-row"`, the identifier pass wins and the text pass never runs. Reproduced: with a decoy `testID="Wi-Fi"` elsewhere on screen, `tap: { role: Switch, next: Wi-Fi }` taps the decoy's neighbour and reports a pass. This is the ordinary identifier-first doctrine, but a scope makes a decoy likelier to "succeed": a decoy _container_ only wins if it actually holds a match, while a decoy _anchor_ wins if anything at all sits after it — which on a real screen it usually does. Spell an anchor you care about as a map: `next: { text: Wi-Fi }` or `next: { id: wifi-row }`. - -One way `next` is deliberately **looser than CSS `+`**: where `A + B` matches nothing unless the very next sibling is a `B`, `next` keeps looking and returns the nearest match further on. That is what makes it survive the wrapper and spacer nodes a flattened tree is full of, but it also means a row that is _missing_ the control you asked for silently resolves to the next row's — `{ role: Switch, next: { text: Wi-Fi } }` on a Wi-Fi row rendered without a switch returns the _Bluetooth_ row's switch rather than failing. When a row may legitimately lack the control, assert it first (`assert: { visible: { role: Switch, within: { id: wifi-row } } }`) or scope by `within` instead. - -Scopes compose, and it matters which one carries them. `{ role: Button, next: { text: Name, within: { id: card-b } } }` scopes the **anchor** — one label, so one pick, but that pick may land outside card-b. `{ role: Button, next: { text: Name }, within: { id: card-b } }` scopes the **target** — every label is still an anchor, but only card-b's buttons can be picked. They agree on a well-formed screen and diverge when card-b has no button: the first reaches on to the next card's, the second returns nothing. Scope the target when the container is the thing you trust. Conditions honor scopes like any other selector: `assert: { hidden: { text: Saved, within: { id: toast-area } } }` holds when nothing matching "Saved" is inside the toast area — matches elsewhere on screen don't count, and a missing scope element satisfies `hidden` (and fails `visible`/`exists`). - -Selectors resolve against the **full native hierarchy** (iOS: the UIView tree; Android: the complete accessibility hierarchy including not-important views) — strictly more than `describe` or the raw `await-ui-element` tool see (both use the trimmed tree), with complete `testID`/`resource-id` coverage. So an `id` selector works even when `describe` collapses or omits the element — don't fall back to coordinate taps just because a testID isn't visible in `describe` output. And when several elements match — including wrappers whose native text aggregates descendant content — the action directives (`tap`, `type`, `scroll-to`) pick the **most specific** match: an exact text/identifier match beats a substring hit (for a regex matcher, a pattern consuming the element's whole text counts as exact), then the smallest frame wins. (A universal `any: true` selector has no field to be exact about, so its matches rank by reading order instead — the first element in the scope, which is the element a condition reads too. Where two matches share a top-left corner, an action breaks the tie toward the smaller, more specific one and a condition does not, so those two can name different elements.) - -**Quote strings YAML would mangle.** An unquoted `#` starts a YAML comment — `tap: Order #1234` silently parses as `tap: Order` — and bare `yes`/`no`/`on`/`off`/numbers coerce to non-strings. When a selector or typed text contains `#`, `:`, quotes, or could read as a boolean/number, wrap it: `tap: "Order #1234"`. - -### `await` and `assert` - -The **condition is the key**, and its value is the selector: - -- `{ visible: Home }`, `{ exists: { id: row } }`, `{ hidden: spinner }` -- `{ text: { in: , contains: "Taps:" } }` or `{ text: { in: , equals: "Taps: 0" } }` — `text` locates an element (`in`) and checks its rendered content against exactly one of `contains` (case-insensitive substring) or `equals` (case-insensitive exact match — use it when boundaries matter: `contains: "Taps: 3"` is also satisfied by "Taps: 30"). Reach for `text` only when the locator is an identifier/role; to assert a string is simply on screen, prefer `{ visible: "Taps: 0" }`. -- `{ text: { in: total, matches: 'Total: \$\d+\.\d{2}' } }` — the third comparator: a JS regex for dynamic content (counters, prices, dates) that neither literal mode can pin. Unanchored like `contains` (anchor with `^…$` for the `equals` analog) and — unlike the literal modes — **case-sensitive**: the pattern carries its own semantics. An invalid pattern fails at parse time. **Quote the pattern in single quotes**: single-quoted and plain YAML scalars keep backslashes; double quotes would need `\\d`. To assert a dynamic string is simply on screen with no locator, prefer a regex **selector** — `{ visible: { text: { matches: '^Taps: \d+$' } } }` (see Selectors); `text.in` + `matches` is for checking a specific element's aggregated text. -- A container's text aggregates its descendants' text (space-joined), so `text` can assert what a testID wrapper visibly shows even when the string lives in a child node. That also means `equals` against a wrapper must match _everything_ it shows or exactly the wrapper's own label/value — targeting the leaf holding exactly the value (or using `contains`) stays the clearer spelling. - -This condition-as-key form is the only spelling. `await` also accepts an optional `timeout` sibling key in milliseconds — `- await: { visible: Home, timeout: 15000 }` — for a transition that legitimately needs longer than the default budget. **Omit `timeout` by default**: the default budget covers normal transitions, and a habitual generous override just delays failure reporting on every broken step. Add one only after a step demonstrably needs it — it timed out at the default and the wait is legitimately slow (a cold start, a network round-trip, a long animation). `assert` has no timeout override: a check that needs seconds to become true is a wait — spell it `await`. - -For a custom poll interval or bundleId, drop to an explicit `- tool: await-ui-element` step — but the raw tool polls the trimmed `describe` tree, so a testID it reports as not found can still resolve fine as an `await:` directive (see Selectors). Prefer the directive. - -### `type` and `scroll-to` - -`type` presses Enter after typing to commit the value and dismiss the keyboard, so it can't cover later targets. For a chained form whose fields feed one explicit submit — e.g. email then password then a `tap: "Log in"` — set `submit: false` on the intermediate fields so a premature Enter doesn't fire the form early: `type: { into: password, text: "hunter2", submit: false }`. - -Never record a real credential into a flow — the YAML is committed to the repo. Use a secret placeholder instead: `type: { into: password, text: "{{secret:APP_PASSWORD}}" }`. The placeholder is stored verbatim (the YAML stays secret-free) and is resolved at run time by the tool-server from the `ARGENT_SECRET_APP_PASSWORD` environment variable — including agent-less `argent flow run` in CI, where the variable comes from the job's secrets. - -`scroll-to` takes an optional `direction` (`up` | `down` | `left` | `right`, default `down` — so the common case is just `- scroll-to: `) and optionally a `within: ` that anchors the scroll inside a specific container — required to drive a **nested** scroller (e.g. a horizontal carousel inside a vertical list), since the device can't be asked which container to scroll. This step-level `within` (a sibling of `target`) anchors the _gesture_, and it is the **only** scope key the step body takes — `after:`/`next:`/`any:` beside `target` are rejected. It is distinct from a selector's scopes (see Selectors), which `target` may itself carry — `scroll-to: { target: { text: Delete, within: { id: cards } }, within: { id: settings-list } }` scrolls the settings list until the Delete button _inside the cards container_ is visible. It scrolls in bounded momentum-free increments, re-checks after each, and stops if a scroll reveals nothing new (end of the container). `tap`/`type` do **not** scroll — add a `scroll-to` before any target that may be off-screen. It's a no-op when the target is already visible, so a defensive `scroll-to` costs nothing on replay and keeps the flow working on smaller screens. - -### `snapshot` cropping - -`cropOn: ` narrows a snapshot's comparison to one element's region — `- snapshot: { name: cart-total, cropOn: { id: order-summary } }`. The selector resolves like a directive target (settled tree, auto-wait, the standard not-found failure; selector-only, no point form), and the **cropped** image is what gets compared, stored as the baseline, and reported as the `current` artifact; the baseline filename still keys on the full capture's resolution — plus a `-crop-` selector suffix — so device-class drift is still caught. A cropped comparison never masks any region — every pixel of the crop is compared — so prefer elements clear of the top status-bar band (a crop overlapping it leans on best-effort status-bar pinning, and the clock/battery may diff). Crop **fixed-size containers**, addressed by `id`: a text selector resolves to the smallest matching node — typically the label itself, whose frame tracks text metrics — and the crop tracks the element's frame, so an element that grew or shrank by a pixel fails on dimensions ("nothing was compared") rather than on content. Baseline storage and seeding are covered under _Standalone runner_. - -### TV targets (Vega) - -A Vega (Fire TV) device is remote-driven — there is no touch input, so the touch directives (`tap`, `long-press`, `type`, `scroll-to`, `pinch`, `rotate`) fail on it with guidance. Drive focus with `tool: tv-remote` steps and type with `tool: keyboard` instead; everything else (`launch`, `await`, `assert`, `wait`, `snapshot`, `echo`, `run`, selectors) works unchanged — the tree comes from the on-device automation toolkit, which attaches at app launch (the `launch` step waits for it, so a leading `launch` also guarantees selectors resolve). - -```yaml -steps: - - launch: com.example.app.main # the interactive component id from manifest.toml - - await: { visible: Home } - - tool: tv-remote - args: { button: [down, select] } # move focus, then confirm — one step per navigation - - await: { visible: Explore Screen } - - snapshot: explore -``` - -Since a `tv-remote` path is positional (like a coordinate tap), gate each navigation with an `await` on the destination screen and echo where focus should be — that is what makes the flow diagnosable when the focus order changes. - -### Standalone runner - -`argent flow run [--device ] [--platform ios|android|chromium|vega] [--update-baselines] [--output ] [--json]` runs a flow with no LLM in the loop and exits non-zero on any failure — suitable for CI (e2e flows; a fragment runs against the current device state, useful while authoring). `snapshot` baselines live in `.argent/flows/__baselines__//`, keyed by platform + resolution; a `snapshot` step **fails** when no baseline exists for the run's device class, so seed baselines with `--update-baselines` and have the user review and commit `__baselines__/` — and pin the device class in CI (`--device`/`--platform`, same simulator model) so runs compare against the committed key. The status bar is pinned (iOS `simctl status_bar`, Android demo mode) for the run so it doesn't drive visual diffs. `--output ` writes each failed snapshot's baseline/current/diff images to `//` — a stable path for CI artifact upload. - -## Tools - -| Tool | Purpose | -| ------------------------ | --------------------------------------------------------------------------------------------------------- | -| `flow-start-recording` | Start recording — takes a name and (fragments only) an optional `executionPrerequisite`; creates the file | -| `flow-add-step` | Execute a tool call live and record it if it succeeds | -| `flow-add-echo` | Add a label/comment that prints during replay | -| `flow-finish-recording` | Stop recording and get a summary | -| `flow-read-prerequisite` | Read a flow's execution prerequisite without running it | -| `flow-execute` | Replay a saved flow by name | - -Every tool during recording returns the current flow file contents, so you can track what has been recorded. Rules: - -- **Every step runs live.** You see the real tool result (including screenshots) — verify the step worked before continuing. **Only successful steps are recorded**: a failed call writes nothing to the flow file; fix the issue and try again. -- **Pass `project_root` once.** Give the absolute `project_root` (an error is returned if the path is not absolute) to `flow-start-recording` — it is stored for the session and used by all subsequent flow tools. You do **not** pass a flow name to `flow-add-step`, `flow-add-echo`, or `flow-finish-recording` — the active flow is tracked automatically. -- **Start before adding.** Calling those tools without an active recording returns _"No active flow. Call flow-start-recording first."_ -- **One flow at a time.** `flow-start-recording` while already recording switches to the new flow — the response tells you which flow was abandoned and which is now active; the old flow's file remains on disk. -- **Mistakes can be edited out.** Edit the `.yaml` file directly to remove or reorder steps. - -### flow-add-step arguments - -The `command` parameter is the MCP tool name; `args` is a **JSON string** (not an object), omitted entirely for tools with no arguments: - -``` -command: "gesture-tap" -args: "{\"udid\": \"\", \"x\": 0.5, \"y\": 0.35}" - -command: "await-ui-element" -args: "{\"udid\": \"\", \"condition\": \"visible\", \"selector\": {\"text\": \"Continue\"}}" -``` - -Record an `await-ui-element` step to **gate** the next step on a screen transition — it blocks until the element is `visible`/`hidden` (or contains `text`), so the following step runs only once the screen has actually settled; prefer this over a fixed `delayMs`. If its condition is not met before the timeout, replay **stops at that step** (the steps after it assume the transition happened). See the `await-ui-element` section of `argent-device-interact` for the full condition/selector reference. The live call sees only the trimmed `describe` tree — if it can't find an identifier you know exists, gate on visible text to get the step recorded, then retarget the identifier in the `await:` form during polish (the directive resolves the full hierarchy — see Selectors); don't conclude the testID is unusable in the flow. - -## Recording - -1. **Start, then launch as the first step (e2e) or set the stage yourself (fragment).** Call `flow-start-recording` with a descriptive name and the absolute `project_root`. For an **e2e** flow, record a `restart-app` of the app under test as the **first** step — it runs live (resetting the device for the rest of the recording) and is captured as the flow's `launch` step. For a **fragment**, bring the device to the entry state _before_ recording and pass an `executionPrerequisite` describing it (e.g. "App on the login screen") to `flow-start-recording` instead. -2. **Build step-by-step**: for each action, call `flow-add-step` with the tool name and args. The tool runs immediately — check the result before moving on, and gate each navigation with an `await-ui-element` step. -3. **Add labels**: use `flow-add-echo` between steps — echo the expected state, not just the action (see _Making flows resilient_). -4. **Finish**: call `flow-finish-recording`. It returns the file path where the flow was saved and a summary of all steps. -5. **Polish**: **read the saved `.yaml` file** and convert the raw `tool:` steps that have a cleaner directive form (the recorder leaves these as tools): - - `tool: keyboard` typing into a field → `type: { into: "", text: "…" }`, folding in the `tap` that focused the field. - - `tool: await-ui-element` gating a transition → `await: { visible: "…" }` / `{ hidden: … }` / `{ text: { in: …, equals: … } }`, carrying a custom `timeoutMs` over as a `timeout` sibling key. Converting also upgrades the wait from the trimmed `describe` tree to the flow's full-hierarchy tree (see Selectors). Keep the raw `tool: await-ui-element` step only when it sets a custom `pollIntervalMs`/`bundleId` the directive can't express. - - A scroll-to-reach-an-element — a `tool: gesture-swipe` (or its chromium analog, `gesture-scroll`) used to bring a specific element on screen before interacting with it (a `tap`, `type`, `assert`, …) → `scroll-to: { target: "", direction: … }`, dropping the swipe. This is far more robust than a fixed-distance swipe: it scrolls momentum-free and stops exactly when the target appears, so it survives layout and content changes. (`tap`/`type` do not scroll, so a raw swipe whose fling lands differently on another device leaves the following tap unresolved — always prefer the `scroll-to` rewrite.) Keep a `gesture-swipe` as a raw `tool:` step when it isn't scrolling toward a specific element — especially a velocity-dependent gesture like swipe-to-dismiss, edge-swipe-back, or swipe-to-reveal a row action, which a momentum-free `scroll-to` would not reproduce. - - `tool: gesture-pinch` → `pinch: { on: "", scale: … }`, deriving `scale` as `endDistance / startDistance`. Set `on:` to the element under the pinch center when the pinch was aimed at one (the map or image being zoomed); omit it for a screen-center pinch. Don't carry the recorded distances/angle over — the directive re-derives the geometry (finger placement, system-edge avoidance, chaining of large scales) at run time, so the conversion swaps device-specific coordinates for a portable selector with auto-wait. Keep the raw `tool: gesture-pinch` step when the pinch is anchored at a specific point _inside_ a large element (zooming toward a particular map location, not the map's center) or deliberately pans via `endCenterX`/`endCenterY` — `on:` takes only a selector and re-centers the pinch on the element's frame center, so converting would silently move the zoom anchor. - - `tool: gesture-rotate` → `rotate: { on: "", by: … }`, deriving `by` as `endAngle − startAngle` (the tool's `endAngle` > `startAngle` turns clockwise, matching the directive's positive `by`). Set `on:` to the element under the rotation center when the rotation was aimed at one (the map or image being rotated); omit it for a screen-center rotation. Don't carry the recorded `centerX`/`centerY`, radii (`radius` or `radiusX`/`radiusY`), `startAngle`, or `durationMs` over — the directive re-derives the geometry (finger placement, physical-circle radius, system-edge avoidance) and runs at a fixed pace (~90° per 300 ms), so the conversion swaps device-specific coordinates for a portable selector with auto-wait. Keep the raw `tool: gesture-rotate` step when the rotation is anchored at a specific point _inside_ a large element rather than its center (the directive re-centers on the element's frame center, so converting would silently move the pivot), when the gesture's speed itself matters (the directive's pace is fixed), or when the sweep exceeds the directive's ±3000° bound. - -Every other recorded tool (a velocity-dependent `gesture-swipe`, a fixed-distance `gesture-scroll` not aimed at an element, `button`, `screenshot`, …) has no directive form — leave it as a `tool:` step. The recorder already handles the rest: coordinate `gesture-tap`s are captured as portable `tap:` selector steps, a `restart-app` is captured as a `launch:` step, a `flow-execute` of a sibling fragment is captured as a `run: ` composition directive, and device ids are stripped. Captured selectors are emitted in the strict map form (`tap: { text: General }`), never as a loose bare string — the recorder verified the exact element the tap hit, and a bare string would re-parse as loose and route through the identifier-first fallback it was never checked against. After editing, re-run with `flow-execute` to confirm the cleaned flow still passes. - -### Example session - -``` -flow-start-recording { name: "open-about", project_root: "/Users/dev/MyApp" } -flow-add-echo { message: "Start Settings from scratch" } -flow-add-step { command: "restart-app", args: "{\"udid\": \"ABC\", \"bundleId\": \"com.apple.Preferences\"}" } # ⇒ captured as `- launch: com.apple.Preferences` — this is now an e2e flow -flow-add-echo { message: "On the Settings root list, tapping the 'General' row" } -flow-add-step { command: "gesture-tap", args: "{\"udid\": \"ABC\", \"x\": 0.5, \"y\": 0.35}" } # ⇒ captured as `- tap: { text: General }` (portable selector, no udid) -flow-add-step { command: "await-ui-element", args: "{\"udid\": \"ABC\", \"condition\": \"visible\", \"selector\": {\"text\": \"About\"}}" } # gate the transition -flow-add-echo { message: "On Settings > General, tapping 'About'" } -flow-add-step { command: "gesture-tap", args: "{\"udid\": \"ABC\", \"x\": 0.5, \"y\": 0.17}" } -flow-add-step { command: "await-ui-element", args: "{\"udid\": \"ABC\", \"condition\": \"visible\", \"selector\": {\"text\": \"Model Name\"}}" } -flow-finish-recording {} -``` - -Then polish the saved file: the two `await-ui-element` steps become `await:` directives (see the file below). - -## Replaying - -Call `flow-execute` with the flow name (and `project_root`, unless a recording this session already stored it). If the flow has an execution prerequisite, the tool returns a **notice** with the prerequisite text instead of running — verify the prerequisite is met (you can also inspect it beforehand with `flow-read-prerequisite`) and call `flow-execute` again with `prerequisiteAcknowledged: true`. A flow without a prerequisite runs immediately. The run executes all steps in order and returns a structured report: `{ ok, passed, failed, skipped, errored, steps }`. - -**What each step reports.** Raw `tool:` steps include the underlying tool's full `result` (screenshots and other outputs render as usual). The directive steps are summarized: `tap`/`type`/`await`/`assert` report only `status` + `reason`, and `snapshot` adds `artifacts` only when there is something to look at — a failed comparison (baseline/current/diff paths), a missing-baseline failure (`current` only), or a baseline write; a clean pass reports just `status` + `reason`. So converting a `tool: gesture-tap` into a `tap:` directive during cleanup drops only that tap's (uninteresting) raw result — output-bearing tools like `screenshot` have no directive form and stay `tool:` steps, so their results keep flowing through. - -## Flow file format - -The top-level is an object with `steps` (array) and — fragments only — `executionPrerequisite` (an e2e flow, one beginning with `launch:`, has none). Besides the directives above: - -- `- echo: ` — a label printed during replay -- `- tool: ` with optional `args:` — a raw tool call. A tool step may also carry `delayMs: ` to sleep that long before it runs. (`await-ui-element` is an ordinary tool step; see _flow-add-step arguments_ and _Making flows resilient_ for when to gate a transition with one.) -- **`when:` blocks** handle one-sided divergences (interstitials, coach marks): `- when: { visible: "What's new" }` with a sibling `steps: [...]` list runs the block only if the condition holds — checked once with the short assert grace (~1s), so a skipped block barely costs a clean run. Guards are one condition key (`exists`/`visible`/`hidden`/`text`, the await/assert shapes) or `platform: ios|android|chromium|vega`. **No else** (parse-rejected): a block exists to dismiss the divergence and reconverge, never to test two paths — two paths are two flows. Failures inside an entered block are real failures; a skipped block reports `skip` lines. Tap-if-present is a one-step block (`when: { visible: "Got it" }` + `steps: [tap: "Got it"]`); there is NO per-step `optional:` key — it is rejected at parse with a pointer to `when:`. - -The polished result of the example session above: - -```yaml -steps: - - echo: Start Settings from scratch - - launch: com.apple.Preferences - - echo: On the Settings root list, tapping the 'General' row - - tap: { text: General } - - await: { visible: About } - - echo: On Settings > General, tapping 'About' - - tap: { text: About } - - await: { visible: Model Name } -``` - -Note there is **no device id** anywhere in the file — the recorder strips them and the runner injects the bound device. - -## When to proactively record a flow - -Proactive recording is part of this skill's scope (see the description). Record a flow without waiting to be asked — telling the user you are doing so — when you recognize any of these patterns: - -- **About to re-profile**: You completed a profiling session and are about to apply a fix and re-profile. Record the interaction steps now so the re-profile replays them identically (see `argent-react-native-profiler` and `argent-native-profiler` skills). -- **Repeating steps**: You have already performed a multi-step interaction sequence once and the task requires doing it again (comparison, retry, re-test). -- **Complex path discovered**: You worked through a non-trivial sequence of taps/swipes/navigation to reach a desired app state. Capture it before it is lost. -- **User says "again" / "one more time"**: Any request to redo what you just did is a signal to record first, then replay. - -## Flow self-improvement - -Flows break. UI layouts change, coordinates drift, screens get added or removed. When `flow-execute` returns a failure, follow this procedure to diagnose and fix the flow instead of silently re-recording or giving up. - -### Classify the result - -After every `flow-execute`, classify the outcome before proceeding: - -| Outcome | Signal | Action | -| ---------------------- | --------------------------------------------------------------------- | ------------------ | -| **Success** | All steps completed, final screenshot shows expected state | Continue with task | -| **Hard error** | A step has `ERROR` in the result — engine stopped there | Enter **Diagnose** | -| **Silent misfire** | All steps completed but final screenshot shows wrong screen | Enter **Diagnose** | -| **Partial divergence** | Intermediate screenshot shows wrong state even though later steps ran | Enter **Diagnose** | - -For silent misfires and partial divergence, echo annotations (see _Making flows resilient_) are your reference for what each screen _should_ look like. - -### Diagnose - -1. Note the failure step index and error message (if hard error). -2. Call `screenshot` to see where the app actually is now. -3. Call `describe` or `debugger-component-tree` to get the current element tree. Remember `describe` shows less than the flow tree — a testID missing from its output can still resolve as a selector (see Selectors). -4. Compare current state to what the failed step expected. Classify the root cause: - -| Root cause | Symptoms | -| ---------------- | --------------------------------------------------------------- | -| Coordinate drift | Tap succeeded but hit wrong element; elements shifted positions | -| Missing element | Target element not present in element tree | -| Wrong screen | Screenshot shows entirely different page than expected | -| Timing | Element exists in tree but tap missed; loading spinner visible | -| State mismatch | First step fails — executionPrerequisite was not actually met | - -5. State the diagnosis in one sentence before attempting any correction. - -### Correct - -Choose the lightest strategy that fits: - -**Strategy 1 — Edit the YAML** (coordinate drift, parameter changes). -Read `.argent/flows/.yaml`, update the broken step's `x`/`y`, `bundleId`, `text`, or other args. Re-run `flow-execute` to verify. - -**Strategy 2 — Manual recovery + continue** (timing/transient issues, one-off replay). -Manually execute the failed step with corrected coordinates from the Diagnose step, then manually execute remaining steps. Does not fix the YAML — use only when re-recording is not worth it. - -**Strategy 3 — Re-record from failure point** (structural changes, new intermediate screens). -Navigate the app to the state just before the failure point. Call `flow-start-recording` with the same flow name (overwrites). Re-add the working prefix steps via `flow-add-step`, then continue recording new steps from the divergence point. Call `flow-finish-recording`. - -**Strategy 4 — Full re-record** (major changes, unclear diagnosis, or 3+ broken steps). -Reset the app to prerequisite state (`restart-app` + `launch-app`). Record from scratch with the same flow name. - -**Decision heuristic:** - -- 1 step broken, parameter-only change → Strategy 1 -- 1 step broken, transient issue, not worth persisting → Strategy 2 -- 2–3 steps broken or flow structure partially changed → Strategy 3 -- 3+ steps broken, or unclear root cause → Strategy 4 -- Flow used for profiling comparison (must be identical) → Strategy 4 - -### Verify and bound retries - -After applying a correction, re-run `flow-execute` to verify. - -- If it succeeds → done. Report what changed (e.g. "Fixed step 4: updated tap coordinates from 0.5,0.35 to 0.5,0.42"). -- If it fails at a **different** step → return to Diagnose for a second attempt. -- If this is already the second correction attempt → **stop**. Report the diagnosis to the user and recommend a full re-record or manual investigation. - -**Hard cap: 2 correction cycles.** Do not enter an unbounded fix loop. - -### Making flows resilient - -Apply these when recording new flows to reduce future breakage: - -- **Echo expected state, not just actions.** Write `"On Settings > General screen, about to tap About"` not `"Tap About"`. During diagnosis these tell you what the screen _should_ look like. -- **Gate transitions with `await-ui-element`, not fixed delays.** After a tap that triggers a navigation, record an `await-ui-element` step that waits for the next screen's element to be `visible` (or a spinner to be `hidden`) before the following step — converted to an `await:` directive during polish. This removes the **Timing** failure mode in Diagnose (the element is in the tree but the tap fired before the screen settled) and is more reliable than `delayMs` or an extra `screenshot`. An unmet wait stops replay at that step, so a mistimed step can never run blind. -- **Add screenshot steps after critical navigation.** Insert `screenshot` steps after screen transitions. These produce images in the flow result you can inspect during diagnosis. -- **Write specific executionPrerequisites.** `"App on home tab, user logged in, simulator UDID is "` — not `"App running"`. Verify with `screenshot` + `describe` before acknowledging. -- **Prefer launch-app / open-url over navigation chains.** Deep links are more resilient to layout changes than tap sequences. -- **Echo accessibility labels for coordinate taps.** When recording a tap, add an echo with the target's label or testID: `"Tapping 'Submit' button (testID: submit-btn) at 0.5, 0.82"`. During repair, use `describe` to find the element by label and update coordinates. Only use `screenshot` for permission or system overlays when `describe` cannot expose the target reliably. +When replay fails, follow [Reliability and recovery](references/reliability-and-recovery.md). Inspect the first divergence, correct the smallest justified unit, audit, and replay the full flow. Stop after two unsuccessful correction cycles. Never weaken a requested check to obtain a pass. diff --git a/.agents/skills/argent-create-flow/references/flow-yaml.md b/.agents/skills/argent-create-flow/references/flow-yaml.md new file mode 100644 index 0000000000..3f82860200 --- /dev/null +++ b/.agents/skills/argent-create-flow/references/flow-yaml.md @@ -0,0 +1,240 @@ +# Flow YAML + +Read this reference when polishing, composing, or manually reviewing a flow. + +- [File shape and flow type](#file-shape-and-flow-type) +- [Selectors](#selectors) +- [Directives](#directives) +- [Verification conditions](#verification-conditions) +- [Prove a navigation](#prove-a-navigation-identity-then-readiness) +- [Optional divergences](#optional-divergences) +- [Composition and platform limits](#composition-and-platform-limits) +- [Snapshots and standalone runs](#snapshots-and-standalone-runs) +- [YAML safety](#yaml-safety) + +## File shape and flow type + +```yaml +steps: + - launch: com.example.app + - await: { visible: { id: home-screen } } + - await: { idle: true } +``` + +An e2e flow has a literal `launch:` as its first non-echo step. It cannot declare `executionPrerequisite`. Put the named start state in a leading echo. + +A leading `run:` does not classify the outer flow as e2e, but the runner still follows the chain to the launch it reaches, and on Chromium that launch boots the app before step 1. A flow whose `run:` chain reaches a launch is refused an `executionPrerequisite` too: parse accepts the file, then the run rejects it. The one exception is a run pinned to a Chromium instance you brought to the required state yourself (`--device chromium-cdp-`), where that leading launch only attaches. + +A fragment reaches no leading launch, by its own step or through a `run:` chain, and can declare: + +```yaml +executionPrerequisite: User is signed in and viewing Settings +steps: [] +``` + +Flows never store a device id. The runner binds the device. `launch:` restarts the process but does not clear app, account, or backend data. + +The one exception is a device _scope_ rather than a target: `stop-all-simulator-servers`' `devices` list **is** kept in the YAML, because without it the step means the machine-wide sweep and would tear down devices other agents are mid-session on. Replay rebinds a recorded scope only when you pass `device` explicitly — an auto-detected device would retarget the teardown at a device the flow never named. So the recorded ids are what run when you replay without `device`; on another host they reap nothing and come back in `unmatched`, so re-record the cleanup flow there or pass `device`. A step that recorded no scope is narrowed onto the run's device **only when the run resolved one**. A cleanup flow whose only step is that teardown needs no device, so with none or several booted it resolves none, replays as the machine-wide sweep, and still reports a pass. Record the scope, or pass `device` at replay, whenever the sweep must stay confined. + +## Selectors + +Use values that meet the [stable-selector definition](../SKILL.md#stable-selectors). Always write explicit maps: + +```yaml +{ id: save-button } +{ text: Save } +{ role: button } +{ id: settings-row, text: Notifications } +``` + +All provided fields must match. `id` is exact and case-insensitive. `text` and `role` are case-insensitive substrings. An unqualified Android id also matches its qualified resource id. `identifier` is accepted as an alias, but `id` is canonical. Never author a bare string. It is loose shorthand that tries id before text. + +Use single quotes for anchored, case-sensitive regexes: + +```yaml +{ text: { matches: '^Order #\d+$' } } +``` + +### The runner tree is not the discovery tree + +Flow selectors and live discovery use different screen projections: + +| Platform | Runner tree | `describe` / `await-ui-element` | Important difference | +| -------- | --------------------------------------------------------- | ------------------------------- | ------------------------------------------------------ | +| iOS | projected UIView hierarchy | accessibility tree | `native-full-hierarchy` is raw; nodes and roles differ | +| Android | full accessibility hierarchy | trimmed interactables | Discovery can omit testID containers or merge nodes | +| Chromium | filtered DOM nodes with id, label, value, click, or focus | shorter DOM walk | Projections and node limits differ (12,000 vs. 5,000) | +| Vega | toolkit page source | same source | Same elements, different shape | + +On iOS, Android, and Chromium, an id absent from `describe` can still resolve in a flow. Verify it in a scratch fragment. Chromium exposes password fields to the runner as `[password]`; select them by id or role. + +The recorder rechecks each successful `await-ui-element` against the runner tree. Follow any `message` warning and replay each conversion. On Vega, a mismatch usually means the screen changed. A `text` check can also select different elements from the same source. See [Live waits and checks](live-authoring.md#live-waits-and-checks). + +**On iOS, a `launch:` step also decides which app the runner reads.** A successful `launch:` pins later runner-tree reads to that app, so a read probes only that app instead of fanning out over every connected one to find the frontmost. A pinned read still refuses, naming the reason, when the app has no foreground presence left, when it stops answering after an earlier read got through, when its devtools connection dropped, or when the pinned id is a `com.apple.*` system app. + +Any raw `tool:` step ends the pin, because its effect on the screen is opaque to the runner, and reads auto-detect the frontmost app again until the next `launch:` re-pins. A tool that cannot change the foreground app leaves the launched id as an unpinned fallback, which takes the read only when auto-detection times out and the launched app vouches for itself with a probe of its own. `launch-app`, `restart-app`, `reinstall-app`, `open-url`, and `button` drop even that; `launch-app` and `restart-app` replace it with the app they just started, still unpinned. Nested `run:` fragments inherit both the pin and its clearing. + +So on iOS recording and replay can read different apps, not only different projections: recording has no run state and always auto-detects the frontmost connected app, while a replay read between a `launch:` and the next raw `tool:` step reads the launched app. + +**On iOS, never copy a `role` from `describe` into a flow selector.** The runner derives iOS roles from the UIView class name and `describe` from accessibility traits, so a React Native `Pressable` (class `RCTView`) is `AXGroup` to the runner and `AXButton` to `describe`. Select on `id`/`text`, or confirm the role against the runner's own tree. + +When several nodes match, the directive decides: + +- **Actions** (`tap`, `long-press`, `type`, `scroll-to`, `pinch`, `rotate`) take the most specific visible match: exact text/id beats substring, then the smallest frame, then reading order. +- **Conditions** (`await`, `assert`) do not rank. `exists`/`visible` hold if any match qualifies and `hidden` only if none does; `text` reads the first visible match in reading order. + +A container that aggregates a child's text therefore splits them: `tap` hits the leaf while `text.in` reads the container, so `equals` fails against correct UI. Use an `id` or a [relational scope](#relational-scopes) when an action and a check must agree, and a stricter selector when ranking can still choose the wrong element. + +### Relational scopes + +Flow selectors support frame-based `within`, `after`, and `next` in every selector slot. Live `await-ui-element` does not support them. + +```yaml +- tap: { text: Delete, within: { id: profile-card } } # inside a container +- assert: { visible: { role: Button, after: { text: Danger zone } } } # any follower +- tap: { role: Switch, next: { text: Wi-Fi } } # nearest matching follower +``` + +`within` means visual frame containment, not source-tree ancestry. Overflowing children and anchored popovers can fall outside it. `after` and `next` use top-to-bottom, left-to-right reading order. A target cannot satisfy its own `within`, `after`, or `next` anchor. The synthetic root never counts. + +`next` finds the nearest matching follower and skips non-matches. It can therefore reach the next row when the intended row lacks a control. Prefer a stable row container with `within`, or assert the row-local control first. + +Scopes can combine and nest, with at most six scope keys. Use strict selectors for anchors. Scope through a trusted container when a missing control must fail instead of reaching another row. + +## Directives + +Directives stop the flow on failure and skip later steps. `flow-execute` documents their shapes. The available directives are `launch`, `tap`, `long-press`, `swipe`, `type`, `scroll-to`, `pinch`, `rotate`, `await`, `assert`, `wait`, `snapshot`, `run`, `when`, `echo`, and `tool`. + +Use the launch map for cross-platform flows. A bare launch applies everywhere and becomes an app path on Chromium. The map takes `native:`, `ios:`, `android:`, `vega:`, and `chromium:`. `native:` is one id shared by iOS, Android, and Vega, and a per-platform key overrides it for that platform. `chromium:` accepts a relative or absolute app path. A launch that declares no id for the run's platform is an error, not a cue to switch platforms. On iOS, a successful launch also pins later tree reads to that app until the next raw `tool:` step, so read [The runner tree is not the discovery tree](#the-runner-tree-is-not-the-discovery-tree) when a read describes the wrong screen. + +```yaml +- launch: { native: com.acme.app, chromium: ../../app } +- launch: { ios: com.acme.app, android: com.acme.app.android, chromium: ../../app } +``` + +An Android app that needs a non-launcher activity has no `launch:` form. Record `restart-app` with `activity` and keep the flow as a fragment. + +In a `scroll-to` map, put the selector under `target:`. The map supports `up`, `down`, `left`, and `right` directions. The default is `down`; set it explicitly to reach a target above the viewport or along a horizontal carousel. If the target is already visible, the step is a safe no-op. `tap`, `type`, and `long-press` do not auto-scroll. Add `scroll-to` when the target can be off-screen. Use `within` for a nested scroller. + +### `swipe` + +`swipe` is one semantic finger flick where the gesture itself is the action — dismiss a card, page a carousel, open a drawer, pull-to-refresh: `- swipe: left`, `- swipe: { from: Card, direction: left }`, `- swipe: { by: { y: -0.4 } }`. + +**Never use `swipe` to scroll.** Whenever the goal is "bring X on screen so the next step can act on it", write `scroll-to: ` instead: it is goal-seeking (stops exactly when the target appears), momentum-free, and a no-op if the target is already visible. + +**`direction` is the finger's travel** (the Maestro convention): `swipe: left` flings content leftward and reveals what is to the right, the opposite sense of `scroll-to`'s content direction. Declare exactly one travel: + +- `direction` - Maestro-compatible screen geometry, with edge-gesture-safe start and end points. +- `by: { x?, y? }` - signed 0-1 screen fractions, one axis or both for a diagonal. +- `to: ` - an explicit endpoint, selector or point. + +`from` anchors the start on a selector or `{ x, y }` point. Without it the start is the direction preset, or screen centre for `to` and `by`. + +All three must clear a 0.03 minimum travel on the combined start-to-end vector; shorter reads as a tap and will be rejected/fail. + +Anchoring decides how a travel that does not fit is resolved: + +- `direction` keeps the preset magnitude from where the finger lands (0.8 of the width for `left`/`right`, 0.7 of the height for `down`, 0.4 for `up`), clamped at the screen edge. It fails only when the clamp leaves it under 0.03, so a drawer handle or bottom sheet near the edge still swipes. +- `by` delivers its exact delta or nothing. With `from`, a delta that runs off-screen fails. + +Only the direction presets carry OS-edge margins. A resolved `from` point is used verbatim, so use `from` for app-level gestures only and write system-edge gestures such as system back as raw `tool: gesture-swipe` steps. + +`momentum: false` removes the fling, so the swipe lands where the finger stops. The default `momentum: true` is a natural flick that flings on. `duration` (ms) is the travel time: default 300, minimum 150. A shorter gesture gives the content too few frames to track the travel, so it overshoots and may fling backwards. Every `swipe` then waits for the tree to settle, best effort, so its momentum does not swallow the next step's touch. + +On Chromium a swipe is a mouse drag (`gesture-drag`), so a `from` on an ``, an `` or a `draggable="true"` node starts the browser's native drag-and-drop: the page gets `pointerdown` then `pointercancel` and sees no travel. On Vega, `swipe` fails upfront like the other touch directives. + +`type` presses Enter in a second `keyboard` call unless `submit: false`. A polished focus tap plus one text-only `keyboard` call usually needs `submit: false`. Store external values as `{{secret:NAME}}`. The runner uses the first source that defines the name: environment `ARGENT_SECRET_NAME`; project `.argent/secrets.env`; project `.env.local`, then `.env`; then `~/.argent/secrets.env`. The two `secrets.env` files accept the bare `NAME`, but the shared dotenv files expose only `ARGENT_SECRET_`-prefixed keys, so a bare `NAME=…` in `.env` or `.env.local` stays unresolved. The runner redacts every resolved value, so do not use a placeholder for content a report must show. + +A **selector-less gesture** — a coordinate `tap`/`long-press`/`swipe`, or a `pinch`/`rotate` with no `on:` — resolves no frame, so a tree source it cannot read does not fail it. It settles best effort, dispatches anyway, and the step **passes carrying a warning** that quotes the source's own error. That green says the gesture was sent, not that it landed: one aimed at a moving element can miss it entirely. Restore the tree source, usually by relaunching the app so the instrumentation loads. Accept the warning only where the app serves no tree at all, and put an explicit `wait:` before a gesture that follows a transition. The first such gesture proves the outage and later ones spend that verdict without paying the settle window again. A tree read that comes back, or a relaunch, retires that verdict — which only makes the next gesture pay a fresh window, and it warns again if the source is still down. + +## Verification conditions + +```yaml +- await: { visible: { id: settings-screen } } +- await: { hidden: { id: loading-spinner }, timeout: 15000 } +- assert: { exists: { id: notifications-toggle } } +- assert: { text: { in: { id: preference-status }, equals: Enabled } } +- assert: { text: { in: { id: result-count }, matches: '^\d+ results$' } } +``` + +`text.in` locates one element. It compares that element's rendered and descendant text with exactly one comparator: + +- `contains`: case-insensitive substring. +- `equals`: case-insensitive full match. +- `matches`: case-sensitive JavaScript regex. + +Use `equals` or an anchored regex when boundaries matter. For example, `contains: "Taps: 3"` also matches `Taps: 30`. + +Use `await` for an outcome that can take time. Its default is 7500 ms. Add a larger timeout only after the default expires. Use `assert` for settled state. It has a fixed 1000 ms grace and rejects `timeout`. + +A negative condition proves only that the current tree has no visible match. It also passes before the element appears, for a typo, or on the wrong screen. First prove the containing screen and the same stable selector as `visible`. Then perform the removing action and check `hidden`. Prefer an additional positive replacement or empty state. + +## Prove a navigation: identity, then readiness + +Every screen change needs both checks: + +```yaml +- await: { visible: { id: profile-screen } } # identity +- await: { idle: true } # readiness +``` + +The identity selector must exist only on the destination. A dropped tap can leave the source screen idle. A destination element can enter the tree before its animation finishes. Therefore neither check replaces the other. + +### `idle` readiness + +`idle` waits until the screen has content and stops changing in both the UI tree and pixels. + +```yaml +- await: { idle: true, stableFor: 400, timeout: 9000 } +``` + +`stableFor` (default 250) is how long stillness must hold. `timeout` (default 7500) is the budget for the whole wait, and parse rejects one that cannot contain a settle. A settle spans three reads across two 200ms polls. The floor is the longer of `stableFor` and the 400ms a settle spans, plus the 200ms of budget the closing round needs to start. The hold runs during those polls rather than after them, so only the longer of the two counts: the default 250ms hold needs 600ms and an 800ms hold needs 1000ms. Stillness is measured across intervals, so `stableFor: 0` means the first two agreeing intervals, not the first read. `idle` has no assert form and no `when:` form. + +It **never fails a run.** Every outcome short of a clean settle passes with a warning, because readiness is not an acceptance criterion. Read that warning rather than stepping over it: + +- **the screen never held still** — it spent the timeout and was still moving on the last interval. A video, shimmer, carousel, or live text stays healthy while moving, and a load that never finished looks identical. +- **a small part of it was still changing** — a spinner, caret, or progress dot moved through the hold. Too small to count as the screen moving, so the settle completed anyway. +- **the screen was still for the last Nms** — the wait ran out mid-hold, so no settle was confirmed. It names the term that was short: a second agreeing interval, or the rest of `stableFor`. Raise this step's `timeout:`, because the wait was too short rather than the screen too busy. +- **the tree stayed empty** — the screen rendered no accessible content: a canvas or video surface, or a screen that never arrived. +- **settled on the UI tree alone** — no screenshot pair could be read, so presentation-layer animation was never waited out. +- **too few reads** — a settle needs three reads across two intervals and this step got fewer, so it ended with no evidence either way. + +Only a tree source this step could not read stops the run, as an errored step — one still failing when the wait ends, one that wedges after answering, one that answers with an empty tree it flags as degraded (an unattached Vega toolkit, an AX service asking to be relaunched), or one that never answers (raise `timeout` before suspecting the app). The run is then not ok and every later step is skipped. A single failed read is not that: the hold restarts from the next good read. The same outage stops no [selector-less gesture](#directives), which needs no frame and passes with its own warning instead. + +`idle` proves readiness only and never identifies the screen, so it cannot serve as acceptance evidence or replace the identity gate. Gate the next action on a stable element. Add `idle` during polish after each screen change, not after every step. + +## Optional divergences + +Use `when:` only for optional setup or an interstitial that reconverges: + +```yaml +- when: { visible: { text: Got it } } + steps: + - tap: { text: Got it } +``` + +The guard accepts one `exists`, `visible`, `hidden`, or `text` condition, or `{ platform: ios|android|chromium|vega }`. UI guards use the short assert grace and reject `timeout`. There is no `else` or per-step `optional`. Put separate behavioral paths in separate flows. Never place a required acceptance check inside `when:`. + +## Composition and platform limits + +A `run:` target is a YAML path resolved against the directory of the flow file containing the step, so `../shared/login.yaml` reaches a sibling directory rather than the project root. The `.yaml` suffix is optional: `run: login` and `run: login.yaml` both name `login.yaml` beside the flow. + +- iOS and Android can run fragments or e2e flows inline. A nested e2e launch restarts its app. +- Chromium boots one instance per launch **step**, not one per run. The leading launch — the flow's own, or the one its leading `run:` chain reaches — boots before step 1, unless you pinned the run with an explicit `device`, where it only attaches. Every later launch boots a fresh instance, moves the run onto it, and tears down the instance the run already owned for that app path. Nesting a Chromium e2e flow with its own launch is therefore the supported way to give a sub-scenario its own restart. Chromium rejects `pinch` and `rotate`. Use the app's own zoom or rotate controls. +- Vega uses `tool: tv-remote` and raw `tool: keyboard`. The touch directives (`tap`, `long-press`, `swipe`, `type`, `scroll-to`, `pinch`, `rotate`) are unsupported. Gate focus and navigation results with `await`. + +## Snapshots and standalone runs + +`argent flow run [--device ] [--platform ios|android|chromium|vega] [--update-baselines] [--output ] [--json]` runs without an LLM and exits non-zero on failure. + +A screenshot is human evidence. A `snapshot:` is executable visual verification. A missing baseline or excessive mismatch fails. A `cropOn` size change also fails. Use snapshots for color, layout, size, spacing, typography, clipping, overflow, images, icons, or stable component appearance. Use full screen for global changes and `cropOn` for one component. + +Do not use a snapshot as the only proof of navigation, persistence, data, accessibility state, logs, or network behavior. Avoid unstable timestamps, live data, ads, animation, and device drift. First establish deterministic state, identity, and readiness. + +Baselines live under `.argent/flows/__baselines__//` and are keyed by platform and full-capture geometry; `cropOn` also contributes its selector. Seed from a known-good state with `--update-baselines`. Inspect every baseline and require user review. Do not commit it yourself. Baseline creation or update is not a test pass. Never update a baseline only to make a diff pass. The default `maxMismatch` is 0.5 percent. + +Pin `--platform` and `--device` for iOS, Android, or Vega. For Chromium the device class is the window's own pixel size, which the app sets and no launch argument changes: pass `--platform chromium` and omit `--device` so the runner boots the declared app path instead of attaching to a running window of another size. A window sized from host or session state produces a key CI cannot reproduce, and the step fails for a missing baseline. The runner pins mobile status bars during visual runs. `--output ` writes failed baseline, current, and diff images under `//` for CI artifact upload. + +## YAML safety + +Quote strings containing `#`, `:`, or quotes. Quote numbers and `true` or `false` in text slots. Use single quotes for regexes with backslashes. Parsing rejects invalid directives, selectors, regexes, `else`, unsupported options, and e2e flows that also declare `executionPrerequisite`. diff --git a/.agents/skills/argent-create-flow/references/live-authoring.md b/.agents/skills/argent-create-flow/references/live-authoring.md new file mode 100644 index 0000000000..47539e828a --- /dev/null +++ b/.agents/skills/argent-create-flow/references/live-authoring.md @@ -0,0 +1,264 @@ +# Live authoring + +Read this file before creating or changing a flow. Exercise the saved path through the recorder. Perform syntax cleanup only after finishing. + +- [Recorder contract](#recorder-contract) +- [Start in the correct order](#start-in-the-correct-order) +- [Record the first walkthrough](#record-the-first-walkthrough) +- [Finish and polish](#finish-and-polish) +- [Worked example](#worked-example) +- [Blocking audit](#blocking-audit) +- [Replay](#replay) + +## Recorder contract + +`flow-add-step.command` is an MCP tool name. `args` is a JSON string, not an object. Omit `args` for a no-argument tool. + +```text +command: "gesture-tap" +args: "{\"udid\":\"DEVICE\",\"x\":0.5,\"y\":0.35}" +``` + +A recorded `flow-execute` has two names. The top-level `name` identifies the recording. `args.name` identifies the sibling flow captured as `run:`. + +Obey these lifecycle rules: + +1. Pass the same `name` and absolute `project_root` to every recording tool. +2. Choose a name unique to the task. Another caller can take over the same pair without an ownership check. The pair is keyed by the file the filesystem resolves to, not the spelling you passed, so a differently-cased name or a symlinked `.argent/flows` collides too. That collision is reported: the second start says `restarted`, and the first recording's next call fails naming both spellings. +3. Give concurrent recordings separate devices. Their files are isolated, but their live device actions are not. +4. Treat `flow-start-recording` as destructive. It always truncates the named YAML, including a finished or committed flow. `restarted` reports only a displaced live take. +5. If a call says the recording is inactive, do not restart under that name. The completed take can still be on disk. Copy it aside or record under a fresh name. +6. Inspect `toolResult`, `message`, and `recorded` after each call. A call that errors records nothing, but a call that returns normally while reporting an unmet condition **does** append the step, and `message` says the step was added either way. `await-ui-element` is the case that turns up in practice (see [Live waits and checks](#live-waits-and-checks)). Only `flow-start-recording` and `flow-finish-recording` return the whole YAML as `flowFile`. A step call returns `recorded` — one summary line for the step it appended — plus a running `stepCount`. Read `recorded`: the recorder does not always store the tool call you made, and that line is where a rewrite shows up. To see the whole file mid-recording, read it at `savedTo`. A `savedTo` that comes back `null` means the write failed on your side. The step is still in the recording, so continue: the next step rewrites the whole file, and `flow-finish-recording` returns `flowFile` regardless. +7. Edit or reorder the YAML only after `flow-finish-recording`. An active remote recording can overwrite mid-recording edits. + +## Start in the correct order + +### iOS, Android, and Vega e2e flows + +1. Call `flow-start-recording` before launching or touching the app. +2. Record a plain `restart-app` as the first non-echo action. Pass only the device id and app id. The recorder converts it to `launch:`. +3. Record `await-ui-element` for the real first screen immediately after restart. + +Extra restart arguments prevent `launch:` conversion. An Android `activity`, for example, leaves a raw tool step and therefore a fragment. + +Do not use splash content as a selector or landmark. Wait for the first real screen. + +On iOS, only `restart-app` guarantees an instrumented launch. `launch-app` can foreground an uninstrumented process. Use [iOS selector recovery](reliability-and-recovery.md#ios-selector-recovery) when the tree is missing. + +### Chromium e2e flows + +**The app sets the window size; no boot argument overrides it.** Electron ignores `--window-size` — the size comes from the app's own `BrowserWindow` options, and Argent never resizes the window. Boot a fresh target with `boot-device` and `electronAppPath`, then take one `screenshot` and record its pixel dimensions: that capture size is the snapshot device class ([Flow YAML: Snapshots](flow-yaml.md#snapshots-and-standalone-runs)) and must match when CI replays. If the app sizes its window from host or session state, say so in the report. + +After boot, start the recorder before the first in-app action. Record the first-screen wait. `restart-app` has no Chromium support, so the call errors and records nothing, and a recorded Chromium flow is always a fragment: its launch is written in during polish rather than captured, and any `executionPrerequisite` the recording declared must go with it. A launch-first flow must not carry one. During polish, add the matching launch: + +```yaml +steps: + - launch: + chromium: + path: ../../app + args: ["--enable-feature-flag"] +``` + +The path is relative to `.argent/flows/`. Copy the live boot arguments verbatim, and omit `args` when the boot passed none. This packaging exception represents the boot already exercised live. It does not permit a rehearsed UI path. + +### Fragments + +Stage the entry state before recording. Then start with a precise `executionPrerequisite` that names UI, account, and platform state. Do not store a device id. Start recording before the fragment's first interaction. + +## Record the first walkthrough + +For Vega, first read [Flow YAML: Composition and platform limits](flow-yaml.md#composition-and-platform-limits). Vega is remote-driven and does not use touch gestures. + +Reach each screen through the app's UI. Do not replace tested navigation with `open-url`. Starting the app is not navigation. + +For every action: + +1. **Discover without mutation.** Use `describe`, iOS native discovery, `debugger-component-tree`, or `screenshot`. Do not record discovery or `debugger-*` calls: `port` is not a device-bind key, so a recorded one replays against whatever Metro owns that port. +2. **Choose a durable target.** Prefer a stable id, then stable text or an accessibility label. On iOS, use native discovery for ids that trimmed accessibility output omits. +3. **Add an echo.** Name the current state, action, and expected outcome before the action can fail. +4. **Execute through `flow-add-step`.** Inspect the result and the `recorded` line immediately. +5. **Verify immediately.** Record outcome checks when their states first appear. After navigation, prove identity then readiness: record the identity check live, and add the readiness gate during polish. + +### Record identity, then readiness, after every navigation + +1. Record `await-ui-element visible` on an element unique to the destination. +2. If a specific control marks application readiness, record another visible wait on that control before the next action. +3. During polish, add `await: { idle: true }` after the identity check. + +A shared tab bar, source element, or positional id does not prove navigation. `idle` cannot identify the screen or prove application data is ready. Keep the control wait because asynchronous loading can continue after stillness. If the screen intentionally moves, disclose it and gate the next action on a stable element. Read [Flow YAML: Prove a navigation](flow-yaml.md#prove-a-navigation-identity-then-readiness) for the directive semantics. + +### Record absence in three steps + +Use the same stable selector for both checks: + +1. Record it as `visible`. +2. Record the action that removes it. +3. Record it as `hidden`. + +Without step 1, `hidden` also passes for a typo or an element that never existed. A role-only or regex-only first locator does not establish a specific element. + +### Taps + +`flow-add-step` cannot receive a flow selector directly. Discover the element first, then record `gesture-tap` at its frame center; the live coordinates are transport for the gesture, not a final locator. The recorder reads the pre-tap tree and derives the selector in a fixed order — `id`, then `text`, then `role` — giving three outcomes. Read the `recorded` line after every tap, because only two of them warn. It names the derived form — a selector map, or the kept point: + +1. **`tap: { id: ... }` or `tap: { text: ... }`** — the good case. +2. **`tap: { role: ... }`, appended with no warning.** An icon-only button with neither id nor visible label lands here. `role` matches as a case-insensitive substring, so a replay screen holding a second control of that role can win the [ranking](flow-yaml.md#the-runner-tree-is-not-the-discovery-tree) and the tap reports a pass on the wrong control. +3. **A kept raw point**, with a warning naming the reason and the retarget. + +Treat outcomes 2 and 3 alike. Restore the source screen with direct MCP calls, record a corrected tap, then remove the weak step after finishing. Keep a point or a bare role only through the [coordinate fallback gate](reliability-and-recovery.md#coordinate-fallback-gate). + +Never tap the on-screen keyboard through the recorder. Some platforms expose it as one large node, so replay can tap the wrong key while reporting success. Record text with `keyboard`. + +### Typing + +Record the focus tap, then record `keyboard` with `text`. A `keyboard` call carries `text` or `key`, never both. To submit, record a second `keyboard` step with `key: "enter"`. Verify the complete value with `describe` or an app validation marker. + +**Never `describe` or `screenshot` a non-secure field you just filled from `{{secret:…}}`.** Only a password field is redacted; a plain text input hands the resolved value back into your context, and an API key or token typed into one is the ordinary case. Submit or navigate away first, then verify the resulting screen. + +**`describe` reports focus on Chromium only.** iOS and Android leave it unset — it is a Vega/D-pad signal there — so those platforms have no live pre-typing focus check, and the value check afterwards is what proves the keys landed. On Chromium, read `focused` before recording `keyboard`. + +If characters are lost, restore the field with direct calls. Do not record a duplicate typing step. Polish the valid pair into `type:`. Its replay focus wait reads the runner's own tree, which does report focus on iOS, Android, and Chromium, but an unconfirmed poll falls through to typing rather than failing — so retain the committed-value check. Store credentials as `{{secret:NAME}}`. Never record a literal credential. + +### Scrolling and swiping + +Record the required live gesture. During polish: + +- Convert element-seeking movement to selector-based `scroll-to`. +- Convert a swipe that is a gesture in its own right - swipe-to-dismiss, paging a carousel, revealing a row action - to `swipe:`, anchoring `from` on the gesture's **subject** (the card being dismissed, the row being revealed), not on whatever content happened to be under the finger. +- Retain a raw gesture tool only for what `swipe` deliberately doesn't express - an edge swipe (system back), a multi-touch `gesture-custom`, or exotic velocity control. + +For every retained raw gesture, add an echo and a recorded result check. + +### Live waits and checks + +Record `await-ui-element` through `flow-add-step`. The recorder writes the step even when `toolResult.success` is false. Read `success` and `cause` after each check: + +- `unmet`: The tree was readable, but the condition was false. Restore the expected state or correct the selector or timeout. Record the check again, then delete the failed step after `flow-finish-recording`. +- `unreadable`: The wait ended without a trustworthy read. Restore the tree source and record the check again. Keep the failed step: the condition is unknown, not false. +- `cancelled`: The caller stopped the wait. Record the check again. Keep the failed step: the condition is unknown, not false. + +Only `unmet` disproves the condition. Never delete a step during the recording. + +A stale `hidden` whose selector matches nothing replays as a silent pass — the unfalsifiable gate that [Record absence in three steps](#record-absence-in-three-steps) exists to prevent. Never proceed as though the gate passed. See the `await-ui-element` section of `argent-device-interact` for the full live condition and selector reference. + +A wait inside `run-sequence` gets no recorder warning. Inspect the nested result. Any `success: false` fails the sequence during replay. + +The live tool and flow runner use [different trees](flow-yaml.md#the-runner-tree-is-not-the-discovery-tree). After a successful wait, the recorder checks the same condition on the runner tree: + +- No warning: The condition holds on both trees. +- Mismatch: For `text`, first rule out a selector that matches more than one element. Then rule out a changed screen. If the trees really differ, use a runner-tree selector and replay. +- Unreadable, slow, or cancelled check: The conversion is unknown. Restore the source or re-record before conversion. + +A warning does not reject the step. `flow-finish-recording` repeats each warning below its step and reports dropped warnings. + +Do not edit YAML before finishing because edits can drop recorded verdicts. If the finish reports drops, record the waits again. Replay every conversion. Keep a raw tool only for `pollIntervalMs` or `bundleId`. + +### Wrong turns + +Stop immediately. Restore the last valid screen with direct MCP calls, not `flow-add-step`. Continue only from verified state. Remove the bad step after finishing. If recovery changed or skipped meaningful behavior, re-record that portion live. + +## Finish and polish + +Call `flow-finish-recording`, then read the saved YAML. Apply only meaning-preserving conversions: + +| Recorded form | Finished form | +| ------------------------------------------ | ------------------------------------------------------------------ | +| focus tap + `tool: keyboard` | `type:` | +| text `keyboard` + `key: enter` `keyboard` | submitted `type:` without Enter in its text | +| `tool: await-ui-element` | `await:` or `assert:` | +| element-seeking movement | `scroll-to:` | +| `tool: gesture-swipe` as the action itself | `swipe:` with `from` on the gesture's subject | +| coordinate tap or long-press | strict selector after the fallback gate | +| `tool: gesture-pinch` | selector-based `pinch:` with `scale = endDistance / startDistance` | +| `tool: gesture-rotate` | selector-based `rotate:` with `by = endAngle - startAngle` | +| sibling `tool: flow-execute` | recorder-captured `run:` | + +Copy the recorded `selector:` map when you convert a wait. Do not use the loose bare-string form. Flow YAML accepts `identifier`; rename it to `id` only for style. Convert `textMatch: equals` to `equals:` and other text checks to `contains:`. + +Only these unrecorded insertions are allowed, at states observed live: + +- A planned `snapshot:` for pixel-level evidence. +- `await: { idle: true }` after a navigation identity check. +- The Chromium launch that packages the live boot. + +Keep raw forms only when conversion changes behavior. Examples include point-anchored or panning pinch, an edge swipe or one with exotic velocity control, or rotation with a tested start angle, radius, pivot, duration, or speed. Keep screenshots for human evidence. Use `snapshot:` for automated visual comparison. Read [Flow YAML](flow-yaml.md) for syntax. + +If polish reveals a missing action or structural check, restore its preceding state and record it. Do not add remembered behavior directly to YAML. + +## Worked example + +`FLOW` below abbreviates `name: "open-settings", project_root: "/Users/dev/AcmeNotes"`. Repeat both fields in every call. + +```text +flow-start-recording { FLOW } +flow-add-echo { FLOW, message: "Restart Acme Notes; expect Home" } +flow-add-step { FLOW, command: "restart-app", args: "{\"udid\":\"ABC\",\"bundleId\":\"com.acme.notes\"}" } +# captured as: - launch: com.acme.notes +flow-add-step { FLOW, command: "await-ui-element", args: "{\"udid\":\"ABC\",\"condition\":\"visible\",\"selector\":{\"identifier\":\"home-screen\"}}" } +flow-add-echo { FLOW, message: "On Home; open Settings" } +flow-add-step { FLOW, command: "gesture-tap", args: "{\"udid\":\"ABC\",\"x\":0.91,\"y\":0.94}" } +# pre-tap capture resolves to: - tap: { id: settings-tab } +flow-add-step { FLOW, command: "await-ui-element", args: "{\"udid\":\"ABC\",\"condition\":\"visible\",\"selector\":{\"identifier\":\"settings-screen\"}}" } +flow-finish-recording { FLOW } +``` + +After meaning-preserving conversion: + +```yaml +steps: + - echo: Restart Acme Notes. Expect Home + - launch: com.acme.notes + - await: { visible: { id: home-screen } } + - await: { idle: true } + - echo: On Home. Open Settings + - tap: { id: settings-tab } + - await: { visible: { id: settings-screen } } + - await: { idle: true } +``` + +## Blocking audit + +Run these checks before replay: + +```text +# Weak targets: coordinates, raw gestures, role-only selectors +rg -n '(\{ *x:|^ +(x|centerX|fromX|toX):|gesture-(tap|swipe|scroll|drag|pinch|rotate|custom))' .argent/flows/.yaml +rg -n -B2 '^ +role:' .argent/flows/.yaml +# Stored device ids +rg -n '(udid|device_id)' .argent/flows/.yaml +# Positional ids and loose condition selectors +rg -n '(-selector-\d+|selector-\d+\b)' .argent/flows/.yaml +rg -n '(visible|hidden|exists) *: *["'"'"'A-Za-z0-9]' .argent/flows/.yaml +# Fixed waits and skipped navigation +rg -n '^\s*- wait:|open-url' .argent/flows/.yaml +``` + +The condition grep matches a condition key with a scalar after it — `visible: Save` — and not `visible:` opening a map. Recorder output is always block style, so both forms are one line below their `await:`/`assert:`. It covers `when:` guards too, which take the same loose fallback. + +Resolve every hit and confirm: + +- Every element action uses a stable selector unless the fallback gate cleared and documented it. +- **No `role:` stands as the only key under a `tap:`/`long-press:`.** That is the recorder's silent fallback, which warned about nothing. Replace it, or clear it through the fallback gate. A `role:` beside another field or a scope (`within`, `after`, `next`) is deliberate and needs no defence. +- Every element-seeking gesture became `scroll-to`. +- No device id or literal credential remains. +- Every selector-bearing condition uses an explicit selector map without positional or data-derived values. +- Every fixed wait has an echo and a following hard check. Prefer a condition or `idle`. +- No `open-url` replaces tested navigation. +- Every snapshot is intentional, deterministic, non-mutating, and ready for reviewed baseline creation. +- The e2e launch and real first-screen gate are present. Only Chromium permits an inserted launch. +- Every screen change has a destination-only identity check and an `idle` readiness check. The two are repaired differently: record a missing identity check live on the restored screen, but add a missing readiness gate in YAML, because `await: { idle: true }` has no recorder form. +- Every `hidden` check follows a `visible` check on the same stable selector and the removing action. A proven containing screen is not a substitute, because it is no evidence the selector itself ever resolved. + +## Replay + +Run `flow-execute` on the complete YAML with the absolute project root. For a fragment, verify its prerequisite before setting `prerequisiteAcknowledged: true`. + +`flow-execute` takes exactly one flow source: `name`, for a flow saved under `.argent/flows/`, or `flow_path`, an absolute path to any flow `.yaml`. `run:` targets and baselines resolve on the tool server's filesystem, beside the YAML it actually reads. `flow_path` therefore requires the agent and the tool server to share a filesystem and is refused when they do not. `name` still runs remotely, but the server receives only that one YAML in a fresh temp directory, so a `run:` target fails as a missing fragment and a `snapshot` fails for a missing baseline. Replay self-contained flows remotely; a composing or snapshotting flow needs one shared filesystem. + +Manual rescue invalidates the pass. An `errored` step was never evaluated: an `idle` wait whose tree source could not be read, a step that threw, an unresolvable `run:` target, or a `launch:` that did not start the app. Read the reason — most name the environment, but a failed `launch:` is a verdict about the app. Unconfirmed focus is not in this class at all: the replay focus poll has no failure return, so a `type:` step whose focus was never confirmed is scored a **pass**, and only the value check after typing catches it. + +**A passing step that carries a `warning` is a finding, not noise.** `await: { idle: true }` raises [six different warnings](flow-yaml.md#idle-readiness) and they do not share one meaning. Two say the screen was moving; one says the wait ran out mid-hold and is repaired by raising the step's `timeout:`; one says the tree stayed empty; one says the tree did hold still and only the screenshot pairs were missing, so the capture path is what to check; one says the step ended with no evidence either way. No report separates intended motion from a load that never finished. Read which one it is, look at that screen, disclose what you found, and confirm the following step targets a stable element rather than stillness. + +A [selector-less gesture](flow-yaml.md#directives) raises a warning of a different shape, not one of those six: a tree-source outage left it unsettled, so it dispatched blind and the green says only that the gesture was sent. Restore the source, usually by relaunching the app so the instrumentation loads. Accept it only where the app serves no tree at all, such as the [injection-free iOS form](reliability-and-recovery.md#terminally-non-injectable-ios-apps). + +One uninterrupted full pass completes a normal flow. `argent-qa-flows` requires two consecutive passes of unchanged YAML. For CI, use `argent flow run [--platform ...]`; it exits non-zero on failure. diff --git a/.agents/skills/argent-create-flow/references/reliability-and-recovery.md b/.agents/skills/argent-create-flow/references/reliability-and-recovery.md new file mode 100644 index 0000000000..d8e0a7664a --- /dev/null +++ b/.agents/skills/argent-create-flow/references/reliability-and-recovery.md @@ -0,0 +1,154 @@ +# Reliability and recovery + +Read this file for selector warnings, raw coordinates, unavailable trees, swallowed actions, overlays, or replay failures. + +- [Coordinate fallback gate](#coordinate-fallback-gate) +- [iOS selector recovery](#ios-selector-recovery) +- [Other tree sources](#tree-source-recovery-on-android-chromium-and-vega) +- [Strong transition gates](#strong-transition-gates) +- [Obscured targets](#obscured-targets-and-persistent-overlays) +- [Replay diagnosis](#diagnose-a-replay-failure) +- [Corrections](#correct-the-smallest-justified-unit) + +## Coordinate fallback gate + +Use this target order: + +1. A strict stable id. +2. Narrow, stable text or accessibility label. +3. A stable role only when it is unique. +4. `scroll-to` plus one of those selectors for an off-screen target. +5. Raw coordinates only after the checks below. + +Convert element-seeking swipes to `scroll-to`. Keep a coordinate swipe only when the gesture itself is the tested action and no directive expresses it. + +### Run the gate when capture warns + +Work this gate as soon as capture warns that it kept a raw point — and equally when it silently recorded a role-only selector, which warns about nothing. Keep the source screen available and do these checks: + +1. **iOS:** query plausible ids or labels with `native-find-views`. If no term is useful, call `native-full-hierarchy` with narrow fields and `maxDepth: 100`. `describe` and `native-describe-screen` are accessibility projections. They cannot prove that no flow selector exists. +2. **Other platforms:** use `debugger-component-tree` for React Native; otherwise, use `describe`. Verify Android and Chromium candidates in step 3. Their discovery trees can omit runner elements. +3. Test each candidate in a scratch fragment with `assert: { visible: }` on the valid screen. Inspect every failure before trying a better id, label, app, or container. +4. If source is available, inspect its `testID`, `accessibilityIdentifier`, or `resource-id`. If none exists, report the missing stable id as the real fix. + +An unavailable tree makes the candidate test void. It proves the tree was absent, not that the selector failed, so it never authorizes coordinates. Relevant failures include `native devtools is unavailable` or `No native-devtools-connected apps are available` on iOS, an unreachable Android helper, an unreachable Chromium CDP session, or missing Vega page source. The recorder quotes the same reason back in its `selector capture failed` warning, so read that warning before treating it as a verdict about the element. Restore the tree and repeat the test. + +Keep coordinates only for a genuinely unlabeled target or after all plausible labeled candidates fail against a working flow tree. Add an echo naming the target and a hard check on the action's outcome. Report the point, discovery results, and candidate failures. Re-record every uncleared point. + +QA flows are stricter. They can keep a coordinate only for a genuinely unlabeled target. Failed selector candidates alone are insufficient. + +## iOS selector recovery + +The full iOS flow tree exists only for an app launched by Argent with instrumentation. + +1. If Metro, Expo, Xcode, an icon, or a prior process launched the app, call `restart-app`. Restore the source screen and retry capture. `launch-app` can only foreground the existing process. +2. Tap capture does **not** wait for that connection. It makes one tree read and turns any failure straight into the kept-coordinates warning. A recording-time `restart-app` returns before the devtools connection opens, so the first tap after a restart can warn transiently. Re-record that tap once before escalating; only a warning that survives the retry is evidence of a real fault. +3. If the warning survives, call `native-devtools-status` with the same UDID and bundle id and follow its `message`, which names the one action that helps and says when to stop. `requiresRestart` covers only the states a fresh process fixes: an `unregistered` or `connecting` app reports it false, because it already launched under the terms a restart would recreate. +4. If an injectable app remains disconnected, call `stop-all-simulator-servers` once, **scoped to `devices: []`**. One tool-server serves every agent on this Argent install, so an unscoped call tears down their devices too. This does not change app or account data. Then restart and check status again. +5. If it still fails, report an instrumentation blocker. Do not replace selectors with coordinates in a QA flow. + +Use the same explicit UDID throughout. Multiple booted simulators are not an injection fault. Pass `--device ` when standalone selection is ambiguous. + +### Terminally non-injectable iOS apps + +This fallback applies only to `com.apple.*` system apps. A connection failure in another app never authorizes it. + +Argent refuses `com.apple.*` bundle ids at every native-devtools read that names one, because a system app is never the app under test. The instrumentation has been seen both loading and not loading into one, depending on the simulator runtime — either way it is no basis for a selector. `restart-app`, `launch-app`, and `describe` still work on one; it just never gets a flow tree. + +Give the flow a `launch:` step as usual. On iOS the launch waits the full devtools budget out, then passes for one of these bundle ids: starting the app is all that step is for, and a coordinate-driven flow needs nothing more. The flow stays e2e; it just pays roughly sixteen seconds at the launch. Where the refusal bites is selector resolution, and the first selector step reports it there — terminally, naming the coordinate remedy — rather than as a launch failure. The rest of the tree-free form: + +- Raw `tool: await-ui-element` accessibility checks. +- Point taps or long-presses derived from `describe`, each named by an echo. +- A point focus tap plus a raw text-only `keyboard` with `delayMs: 500`, and a second raw `keyboard` with `key: "enter"` to submit. +- Raw swipes with `momentum: false` because `scroll-to` needs the missing flow tree. Momentum-free scrolling keeps later coordinate taps valid. `momentum: false` needs `durationMs` of at least 150 and is rejected below it, so keep the 300 default or raise it. + +Every point tap, long-press or coordinate swipe in such a flow passes **carrying a warning** for as long as the app serves no tree: each [selector-less gesture](flow-yaml.md#directives) dispatches unsettled. Nothing here repairs it. Accept the warnings, read each green as "the gesture was sent, not that it landed", and put an explicit `wait:` or a raw `tool: await-ui-element` before a gesture that follows a transition. Raw `tool:` steps take no settle, so they never carry that warning. + +A recorded wait carries a different warning: it adds about one second and reports that the runner tree is unavailable. That warning is expected too. Keep the wait as a raw `tool:` step. + +Report that the flow has no flow tree and its coordinates are not portable. It cannot satisfy the QA contract. Report the artifact and platform blocker instead. + +A normally injectable app that is broken in the environment gets the same coordinate-only treatment, but not the same launch: there the `launch:` step fails, since the gate withholds its verdict only for a bundle id argent refuses outright. Start such a flow with a raw `tool: restart-app`, which terminates and relaunches without the readiness gate, and accept that the result is a **fragment** — its first non-echo step is not `launch:`, so the runner never classifies it as e2e, and it cannot complete `argent-qa-flows`, which requires a leading `launch:`. Report the blocker rather than labeling that fallback a completed QA test. + +## Tree source recovery on Android, Chromium, and Vega + +While the required source is down, selector failures and raw-point capture are void. Restore the source and re-record affected taps. + +| Platform | Symptom | Recovery | +| -------- | -------------------------------- | ------------------------------------------------------ | +| Android | Cannot reach the devtools helper | Unlock the device, allow `adb install -t`, and rerun | +| Chromium | No reachable CDP session | Boot again with `electronAppPath` and remote debugging | +| Vega | Toolkit returns no page source | Relaunch an app built with automation support | + +On Android, healthy `describe` output does not prove the flow tree is available. It can fall back to legacy `uiautomator`, while the runner refuses that trimmed fallback. + +## Strong transition gates + +Every navigation needs destination identity followed by readiness. Do not identify a screen with a shared header, persistent tab bar, source element, positional id, counter, username, timestamp, or other data-derived value. + +Prefer navigation with a fixed destination. A back button or swipe pops one stack entry, so repeated visits can change its destination. Use back only when back navigation is under test, and gate its result like any other screen change. + +## Obscured targets and persistent overlays + +A selector tap can resolve the intended element while an overlay receives the touch. + +When an overlay intersects the next target: + +1. Record the overlay as `visible` while it exists. +2. Record its real dismissal action. +3. Record the same selector as `hidden`. +4. Only then touch the covered region. + +Do not rely on auto-dismiss timers. Prefer an app e2e affordance that disables transient overlays. On iOS, use `native-user-interactable-view-at-point` for hit-test diagnosis. Other platforms rely on the recorded visibility trio. + +Keep a dismissal swipe only when the UI supports it. Pass it through the coordinate gate and hard-check that the overlay disappeared. + +## Diagnose a replay failure + +Classify before editing: + +| Outcome | Meaning | Response | +| ------------------ | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Hard failure | A step fails and later steps skip | Inspect that step and actual state | +| Environment error | The reason says the check could not run | Repair the environment and rerun; it is no verdict about the app. A failed `launch:` is `errored` too but **is** a verdict — treat it as a hard failure | +| Silent misfire | The run passes but final state is wrong | Restore the first wrong screen and record a stronger gate | +| Partial divergence | An intermediate result disagrees with its echo | Find the first divergent transition | +| Acceptance failure | Actions pass but a requested check fails | Preserve the check and investigate behavior | +| Idle warning | A readiness step passes without settling | Read [which of the six warnings](flow-yaml.md#idle-readiness) it is, then gate the next action on a stable element | +| Unsettled gesture | A selector-less gesture passes unsettled | Restore the tree source, usually by relaunching the app; the green says [only that the gesture was sent](flow-yaml.md#directives) | + +Then: + +1. Record the first failure or divergence index and message. +2. Capture `screenshot` and `describe`. Use native or React Native discovery when needed. +3. Compare actual state with the preceding echo and expected destination. +4. Classify the cause: selector, screen, missing element, readiness, stale data, optional interstitial, or product behavior. +5. State the diagnosis in one sentence before correcting it. + +## Correct the smallest justified unit + +- For one parameter or selector error, edit the YAML and prefer a stable selector. +- For readiness or identity failure, repair that gate and audit every transition with the same shape. +- For one missing transition or two to three structural steps, copy the working prefix and re-record the affected span live. +- For four or more broken steps, unclear state, or a comparison or profiling flow, fully re-record. +- Treat manual recovery as diagnosis only. It never counts as a replay pass. + +Starting again under the same name truncates the YAML. Copy any working prefix before re-recording. + +### Make every replacement gate stronger + +| Weak gate | Do not use | Add the missing proof | +| ----------------------------- | -------------- | ----------------------------------------- | +| Shared or positional identity | Longer timeout | Destination-only root or control | +| Tap lost during motion | Fixed wait | `idle` after destination identity | +| Toast absorbs tap | Retry | Verified overlay dismissal | +| `hidden` never established | Longer timeout | Same-selector `visible`, action, `hidden` | +| Typed value is wrong | Retype | Assert the committed value | + +State the added proof before rerunning. + +### Correction limit + +After each correction, audit and replay from the declared start. Stop after two unsuccessful correction cycles and report the remaining blocker. If failures move while the flow grows, re-record the affected span instead of adding more patches. + +Never weaken, remove, or hide a requested check to obtain a pass. Keep a failing product check and report the flow as an unproven regression artifact. QA remains incomplete until its two-pass gate succeeds. diff --git a/.agents/skills/argent-device-interact/SKILL.md b/.agents/skills/argent-device-interact/SKILL.md index 871ebe1b60..38a1002515 100644 --- a/.agents/skills/argent-device-interact/SKILL.md +++ b/.agents/skills/argent-device-interact/SKILL.md @@ -15,6 +15,8 @@ All interaction tools below accept a `udid` parameter and auto-dispatch iOS vs A > **TV targets (Apple TV / Android TV) are not covered by this skill.** A TV target is **focus-driven, not touch-driven** — the `gesture-*` tools are the wrong tools for it. This applies to both Apple TV simulators (UUID-shaped, identical to iOS) and Android TV / leanback devices (serial-shaped, identical to a phone emulator). If `list-devices` tags your target `runtimeKind: "tv"`, stop and use the `argent-tv-interact` skill: `describe` to read focus, `tv-remote` for remote / D-pad presses, and `keyboard` to type. +> **Physical iPhones (an iOS entry with kind `"device"`) follow a different contract:** automation is app-scoped and only a subset of these tools exists there. Use the `argent-ios-device-interact` skill instead. + For platform-specific caveats (Metro `adb reverse`, locked-screen describe errors, etc.), see § 9 Platform-specific notes at the bottom. ## 1. Before You Start @@ -29,10 +31,10 @@ Use `list-devices` to get a target id. Results are tagged with `platform` (`ios` 1. **Always refer to tapping_rule** from your argent.md rule before tapping. 2. Before performing interactions, consider whether they can be **dispatched sequentially** - more on that in `run-sequence`. -3. **Use `gesture-swipe` for lists/scrolling**, not `gesture-custom`, unless you need non-linear movement. On Chromium use `gesture-scroll` instead — `gesture-swipe` is touch-only. Consider whether you need multiple swipes, if yes - use `run-sequence`. +3. **Use `gesture-swipe` for lists/scrolling**, not `gesture-custom`, unless you need non-linear movement. On Chromium use `gesture-scroll` instead — `gesture-swipe` is touch-only. Consider whether you need multiple swipes, if yes - use `run-sequence`. Pass `momentum: false` when the swipe should decelerate before ending for a precise movement. 4. **Tap a text field before typing**, then use `keyboard` to enter text. 5. **Coordinates are normalized** — always 0.0–1.0, not pixels. -6. **For app navigation, prefer `describe` first.** It works on any screen without app restart. Do not navigate from screenshots on regular in-app screens unless `describe` failed to expose a reliable target. Use `native-describe-screen` only when you need app-scoped UIKit properties. +6. **For app navigation, use the element tree returned after each action** (`--- Elements after action (describe) ---`); call `describe` only when no fresh tree is available for the current screen. It works on any screen without app restart. Do not navigate from screenshot pixels on regular in-app screens unless the tree failed to expose a reliable target. Use `native-describe-screen` only when you need app-scoped UIKit properties. ## 3. Opening Apps @@ -56,25 +58,28 @@ Common schemes: `messages://`, `settings://`, `maps://?q=`, `tel://