diff --git a/.agents/skills/kane-cli/SKILL.md b/.agents/skills/kane-cli/SKILL.md index 32231a7..5399e4d 100644 --- a/.agents/skills/kane-cli/SKILL.md +++ b/.agents/skills/kane-cli/SKILL.md @@ -5,7 +5,7 @@ description: Browser automation + AI test authoring via kane-cli - run browser o # Kane CLI β€” Browser Automation Skill -Use `kane-cli` for **any task that requires a real browser**: navigating websites, clicking elements, filling forms, searching, testing web UI, taking screenshots, or verifying deployments. Do NOT use Playwright, Puppeteer, or Selenium directly. Always run with `--agent` so output is structured NDJSON you can parse. +Use `kane-cli` for **any task that requires a real browser**: navigating websites, clicking elements, filling forms, searching, testing web UI, taking screenshots, or verifying deployments. Do NOT use Playwright, Puppeteer, or Selenium directly. Use `--agent` for `run`, `testmd run`, and `generate`. `testrun run` has no `--agent`: it emits NDJSON when **stdin** is not a TTY (use `< /dev/null` for terminal automation). Assurance conversational commands use `--mode agent`. **Authoring test cases or scenarios?** Never write them by hand β€” kane-cli has two authoring pipelines, and the routing matters: @@ -36,7 +36,7 @@ After that, run kane-cli normally β€” the variable is inherited: kane-cli run "" --agent ``` -Bash blocks until kane-cli exits, then hands you the complete stdout. Parse it, summarize what happened, and present the results table. Same pattern for `kane-cli testmd run` and `kane-cli generate`. +Bash blocks until kane-cli exits, then hands you the complete stdout. Parse it, summarize what happened, and present the results table. Wait for process completion on `testmd run` and `generate` too, but parse their own completion events: `test_md_done` and `generate_done`, respectively. An intermediate `run_end` does not finish a saved test. Set a generous timeout (up to 600000ms) since browser runs can take a while. @@ -100,7 +100,7 @@ The terminal event has `type: "run_end"` and stable fields: `status`, `summary`, | 🟒 **Result** | Passed | | 🎯 **Task** | | | ⏱️ **Duration** | s | -| πŸ‘£ **Steps taken** | | +| πŸ‘£ **Steps taken** | | | πŸ“ **What happened** | | | πŸ”— **View details** | [Open in KaneAI Dashboard]() | ``` @@ -184,7 +184,7 @@ kane-cli run "" --agent [options] | `--variables ` | Inline variables JSON (for `{{key}}` in objective) | None | | `--variables-file ` | Load variables from a JSON file | None | | `--ws-endpoint ` | Remote browser (LambdaTest grid) | Local Chrome | -| `--code-export` | Generate code export after upload | Off | +| `--code-export` | Generate code export after upload | config (`true` by default) | | `--bug-detection ` | Flag suspected product bugs while authoring: `off`/`stop`/`continue` (`stop` halts on a confirmed bug; `continue` records and keeps going) | config value (`off`) | Other flags (`--global-context`, `--local-context`, `--cdp-endpoint`, `--allow-missing-url`) and the full variables precedence chain live in `references/setup-and-config.md`. @@ -222,11 +222,11 @@ How you phrase the objective string determines what the agent does. Four pattern | 🎯 **Action** | "go to", "click", "type", "search", "fill" | Performs browser actions | | βœ… **Assertion** | "assert", "verify", "confirm", "check that" | Pass/fail check on a condition | | πŸ“¦ **Extraction** | "store X as 'name'" | Persists a value into `run_end.final_state` | -| πŸ”Œ **API call** | "call", "POST/GET a URL", a pasted `curl` | The agent makes the HTTP request itself; "save the response as X", then assert/reference `{{X.status}}` / `{{X.response_body…}}` | +| πŸ”Œ **API call** | "call", "POST/GET a URL", a pasted `curl` | The agent makes the HTTP request itself; "save the response as X", then assert on it in plain English: "assert the response status is 200", "store the id from the response body as 'order_id'" | ### Two rules that make an objective replayable -1. **End every flow in a terminal assertion.** Close with a check of the resulting page state (`verify the cart shows 1 item`), not a bare `submit` or `confirm the dialog`. A run only earns a replayable pass/fail from a verify/assert β€” a pure-action objective (`add a laptop to the cart`) gets none. If the objective has several phases, each ends in its own check. Two traps: `confirm the dialog` is an action, not a check; and `verify the Submit button is visible` fails exactly when the action worked (the control disappears on success) β€” assert the outcome, not the trigger. +1. **End every flow in a terminal assertion.** Close with a check of the resulting page state (`verify the cart shows 1 item`), not a bare `submit` or `confirm the dialog`. A run only earns a replayable pass/fail from a verify/assert β€” a pure-action objective (`add a laptop to the cart`) gets none. If the objective has several phases, each ends in its own check. Phrase the closing check as what is on screen once the last action completes, so it verifies with no further click or navigation; if the evidence is elsewhere, make getting there an explicit action (`place the order, open Order History, then verify the newest order shows "Processing"`), never `verify the order succeeded by checking Order History`. Two traps: `confirm the dialog` is an action, not a check; and `verify the Submit button is visible` fails exactly when the action worked (the control disappears on success) β€” assert the outcome, not the trigger. 2. **Intent for the actions, literal for the data.** Phrase actions as goals (`Log in with {{user}}`) so the run absorbs layout drift; keep exact values literal or in `{{variables}}`. An expected-optional branch (a sometimes-there cookie banner) goes in an `if/else`, not assumed away. **Shape:** an intent action carrying literal data, then a verify of an observable end state β€” `Search for "{{query}}" and open the first result, then verify the title contains "{{query}}"`. Full grammar in `references/objectives-cookbook.md Β§1`. @@ -238,7 +238,7 @@ Vague phrasing like "read", "tell me", "report" does NOT reliably extract data ❌ `"go to example.com and read the page title"` βœ… `"go to example.com, store the page title as 'page_title'"` -Stored values appear in `run_end.final_state` and become the second results table per Β§1.4. +Stored values appear in `run_end.final_state` and become the second results table per Β§1.4. Refer back to a stored value in plain English (`the stored price value`), never as `{{price}}`; `{{name}}` is for global variables and secrets only. ### Calling APIs directly @@ -246,10 +246,10 @@ The agent can make API calls itself β€” not just observe the page's traffic. Phr ```text "Call POST https://api.example.com/login with body {...}, save the response as login, - assert {{login.status}} is 200" + assert the response status is 200" ``` -Reference the saved response as `{{login.status}}`, `{{login.response_body}}`, or `{{login.response_body.}}`; a pasted `curl` works too. Full grammar in `references/objectives-cookbook.md` Β§3.5. +Use the response in plain English: `the response status`, `store the id from the response body as 'order_id'`, and later `the stored order_id value`. Never `{{login.status}}`: `{{name}}` is reserved for global variables and secrets. A pasted `curl` works too. Full grammar in `references/objectives-cookbook.md` Β§3.5. ### Chaining @@ -269,18 +269,19 @@ Action β†’ extraction β†’ assertion in one objective: | Specific: "click the 'Add to Cart' button" | Vague: "add the item" | | Name extractions: "store X as 'price'" | Hope for values: "tell me the price" | | `{{variables}}` for credentials/URLs | Hardcode secrets in the objective | +| Plain English for values the run produces: "the stored price value", "the response status" | `{{price}}` / `{{login.status}}` for a stored value or an API response | | Always include starting URL | Assume the agent knows where to start | | Split mega-objectives (>15 steps) into multiple runs | Cram everything into one | --- -## 5. Parsing `--agent` output β€” essentials +## 5. Parsing one-shot `run --agent` output β€” essentials > Internal reference only. Never expose these field names to the user β€” translate them per Β§1. Stdout is NDJSON, one event per line. There are two shapes: -- **Progress events** (most events) have `step` (1-based), `status` (`passed`/`failed`), `remark` β€” and **no `type` field**. +- **Progress events** (most events) have `step` (1-based), `status` (`running` at start, `done`/`failed` at completion), `remark` β€” and **no `type` field**. - **Typed events** have a `type` field: `project_folder_auto_defaulted` (run-startup gate, fires before any progress when no project/folder is configured), `bifurcation`, `child_agent_start`, `child_agent_end`, `ask_user`, `error` (an `error` with `code: "unresolved_variables"` is a pre-run refusal and the **only** line β€” no `run_end` follows; handle per Β§3), and finally `run_end`. Parsing strategy: @@ -292,7 +293,7 @@ for each line: else if obj.step exists β†’ progress event β†’ summarize per Β§1.3 ``` -`run_end` is the only event with a stable cross-version schema β€” build all post-run logic on it. +For one-shot `run`, build post-run logic on `run_end` and process exit. Saved tests and suites use their own completion events (see Command-specific completion below). For full event schemas (`bifurcation` flow fields, `child_agent_*`, `ask_user` semantics, `cancel`/`user_response` outbound events, complete `run_end` field list), Read `references/parsing.md`. @@ -363,3 +364,12 @@ Internal event/field names (`generate_snapshot`, `request_id`, …) are for pars | Browse / create projects or folders, or parse the auto-default event | `references/test-manager.md` | | First-time install, auth, or full config | `references/setup-and-config.md` | | Compare / evaluate / benchmark kane-cli vs another tool or approach (cost, tokens, effort, ROI) | `references/fair-evaluation.md` | + +## Command-specific completion + +The `run_end` parsing strategy applies to one-shot `run` only. For `testmd run`, collect `test_md_done.overall_status`, `duration_s`, `session_id`, and optional `share_url`; embedded `run_end` events can finish individual steps. Local suites emit `testrun_done`; dispatched remote suites then emit `remote_done` (retain `status`, `exit`, `sessions_path`). `generate` emits `generate_done`. Assurance conversational agent streams end in `done`; review/read verbs have their own contracts. Always check process exit too: early refusal, invalid plan or dry-run can exit without the normal completion event. + +Progress is for live display: count only `done`/`failed` completions, retaining child and execution context when step indices repeat. + + +For assertion mode, optional final validation, current-page `--analyzer-only` checks, streaming-network capture, and code-export defaults, read [Execution controls](references/execution-controls.md). diff --git a/.agents/skills/kane-cli/references/assurance.md b/.agents/skills/kane-cli/references/assurance.md index 24f994e..cab6a28 100644 --- a/.agents/skills/kane-cli/references/assurance.md +++ b/.agents/skills/kane-cli/references/assurance.md @@ -229,3 +229,13 @@ Never `context ingest` the new version first β€” reconcile does its own re-inges - **One stream.** The re-extract child's events ride the reconcile stream itself, stamped `verb: "reconcile"` β€” parse per `references/assurance-parsing.md`. For staleness that arrived outside a reconcile, `kane-cli maintain evolve` re-designs a use-case β€” it is interactive-only (the blast-radius confirmation is the point); suggest the user run it in a terminal rather than scripting around it. + +## Sharing a context store + +For location setup, sync, pull, push, clone and rebase recovery, see [Sharing the context store](context-sync.md). + +## Lifecycle automation contracts + +`context name`, `context retire`, and `context revert` accept `--mode agent` and emit an envelope ending in `done`. Destructive agent-mode operations require `--yes` even when stdin is a TTY. Other modes retain human output. Read commands use their documented `--json` flags; do not assume they all accept `--mode`. + +`context review --mode agent` / `--json` returns review outcome rows on success rather than the conversational `done` contract. Early agent-mode errors may emit `error` plus `done`; check command-specific output and process exit. diff --git a/.agents/skills/kane-cli/references/evidence.md b/.agents/skills/kane-cli/references/evidence.md index f11b229..e299539 100644 --- a/.agents/skills/kane-cli/references/evidence.md +++ b/.agents/skills/kane-cli/references/evidence.md @@ -73,3 +73,9 @@ Combines packs (execution ids or paths, order-significant) into one, sealed by d ## Debugging with a pack The pack is the first place to look on a failure: the failed step's **failure record** (error + page state), its **console/network slice** (4xx/5xx or JS errors usually explain it), and the **annotated screenshot** (what the agent actually acted on). Full failure workflow: `references/debug.md`. + +## Evidence merge identity + +For ordinary runs, default merge identity distinguishes the test and commit (`external_id.test_id`, `external_id.commit_id`) and the environment (`environment.os`, `environment.os_version`, `environment.browser`, `environment.browser_version`). Re-runs with the same identity nest as attempts; a different environment produces a separate sibling. + +Explicit collision policies can change grouping; a custom `--rules` file replaces the default rules rather than extending them. Check the selected identity rules before interpreting two runs as retries of the same test. diff --git a/.agents/skills/kane-cli/references/execution-controls.md b/.agents/skills/kane-cli/references/execution-controls.md new file mode 100644 index 0000000..5cae455 --- /dev/null +++ b/.agents/skills/kane-cli/references/execution-controls.md @@ -0,0 +1,11 @@ +# Execution controls + +## Assertion controls and current-page analysis + +`run` and `testmd run` support `--assertion-mode dom|visual` (default `dom` with vision fallback) and `--final-validation on|off` (default off). Persist with `config set-assertion-mode` and `config set-final-validation`. Final validation controls the synthesized `cp_final` checkpoint independently of action/testing mode; keep explicit terminal assertions in objectives. + +`run --analyzer-only --condition ""` checks the current desktop browser page without an objective, action steps or saved test. Repeat `--condition` for multiple checks. Use `--agent` or non-TTY input. Results contain `condition_results: boolean[]`; exit `0` means all conditions were judged, **not** that all are true. Exit `1` means a result was missing, and `3` means cancelled. This mode rejects mobile target/app/device options and code-export/name options. + +Experimental `run --network-ws` and `run --network-sse` enable WebSocket and SSE capture; both default off. Persist with `config set-network-ws on|off` and `config set-network-sse on|off`. SSE capture is Chromium-only. Do not copy these run-only flags onto `testmd` or `testrun` commands. + +Code export defaults to enabled, subject to saved configuration, and supports `python` (default) or `javascript`. `run`/`testmd run` use `--code-language`; `testmd export` uses `--language`. Upload eligibility is independent of action/testing mode. \ No newline at end of file diff --git a/.agents/skills/kane-cli/references/fair-evaluation.md b/.agents/skills/kane-cli/references/fair-evaluation.md index 5eced3f..2b8f1e6 100644 --- a/.agents/skills/kane-cli/references/fair-evaluation.md +++ b/.agents/skills/kane-cli/references/fair-evaluation.md @@ -17,7 +17,7 @@ The most common evaluation error is pitting kane-cli **authoring** against the o |---|---|---| | **Create the test** | AI authoring β€” tokens, one-time | Agent/human **generates** the script β€” tokens and/or engineer hours, one-time | | **Run the test** | Replay from cache β€” ~0 LLM | Execute the script β€” ~0 LLM | -| **UI changes / locator breaks** | Re-author only the failing step + downstream (`--retry` shrinking window); cost ∝ change | Human/agent finds & fixes broken selectors; debug the script | +| **UI changes / locator breaks** | Re-author only the failing step + downstream (default adaptive healing); cost ∝ change | Human/agent finds & fixes broken selectors; debug the script | | **Ongoing maintenance** | Edit plain-English Markdown; cascade re-authors only forward; shared `@import` helpers fix once | Edit code; flaky-wait/fixture upkeep; selector churn | | **Verify pass/fail** | Deterministic asserts (URL/title/DOM/network/console/cookies); AI vision only for ~10% visual checks | Code assertions; or a separate LLM judge if the check is semantic/visual | diff --git a/.agents/skills/kane-cli/references/mobile.md b/.agents/skills/kane-cli/references/mobile.md index d434ec3..222a8c4 100644 --- a/.agents/skills/kane-cli/references/mobile.md +++ b/.agents/skills/kane-cli/references/mobile.md @@ -144,8 +144,8 @@ What to present after `testrun_done`/`remote_done`: the suite rollup (per `refer The **same natural-language objective grammar** applies (`references/objectives-cookbook.md`): action verbs, assertions, extractions ("store as"), if/else, chaining, and variables all carry over. A mobile run just drives an app instead of a page. -The exception is **browser/DevTools-only checkpoints**, which are **web-only** and do not apply to a mobile run: - -- Network (HTTP traffic), Console, DOM/selectors, Cookies, localStorage, Core Web Vitals (LCP/CLS/INP/FCP/TTFB). +Mobile capability depends on the platform and app. The implementation includes native network operations and Android cookie/storage access for visible Chrome or debuggable WebViews. This does not establish full desktop DevTools parity, general native DOM support, or equivalent iOS support. Verify device/app prerequisites and the specific checkpoint before relying on it; live platform parity remains unverified. Write mobile objectives around what the app shows and does (open a screen, tap, type, assert visible text/state, store a value). And never point a mobile run at a URL: a mobile run drives an app, not a website. + +In the interactive TUI, `/mobile` and `/desktop` can switch targets before the first dispatch. The target locks after dispatch; use `/new` to start a new session before switching. diff --git a/.agents/skills/kane-cli/references/objectives-cookbook.md b/.agents/skills/kane-cli/references/objectives-cookbook.md index 920745d..be6b2a9 100644 --- a/.agents/skills/kane-cli/references/objectives-cookbook.md +++ b/.agents/skills/kane-cli/references/objectives-cookbook.md @@ -4,7 +4,7 @@ Read this whenever you're constructing the prose objective for `kane-cli run ""` or the body of a `## Step` in a `_test.md` file. Both surfaces feed the same agent and accept the same grammar. -> **Mobile runs** (`--target emulator|simulator`, macOS Apple Silicon) share this same objective grammar, but the browser/DevTools-only checkpoints below (Network, Console, DOM/selectors, Cookies, localStorage, Core Web Vitals) are **web-only** and do not apply. Read `references/mobile.md`. +> **Mobile runs** (`--target emulator|simulator`) share this objective grammar. Checkpoint support depends on platform and app: native network and Android Chrome/debuggable-WebView storage paths exist, but desktop parity and iOS equivalents must be verified. Read `references/mobile.md` for the capability boundary and local/cloud prerequisites. --- @@ -38,6 +38,15 @@ Close each objective with a claim about the resulting **page state** β€” not a b If an objective has several phases (log in, then search, then check out), **each phase ends in its own assertion** β€” an intermediate phase with no check runs unverified, and a replay can't tell you which phase drifted. +**The closing assertion is a look, not a trip.** Phrase it as a claim about what is on screen once the last action completes, so the agent verifies it without clicking, navigating, scrolling, or typing anywhere else. If the evidence lives somewhere else (Order History, the cart, a details drawer), add the action that gets there as its own step, then assert. + +```text +❌ Place the order, then verify it succeeded by checking that it appears in Order History +βœ… Place the order, open Order History, then verify the newest order shows "Processing" +``` + +The ❌ version hides an action inside a check. The agent has to find its own way to Order History on every run, so the route can differ between runs and is never a step you can read or fix. The βœ… version keeps the route in the actions and leaves the assertion a pure observation of the page it lands on. + Two things look like assertions but are not: - **A UI-commit "confirm" is an action, not a check.** `confirm the dialog` / `click OK to confirm` presses a control; it verifies nothing about the outcome. Follow it with a real check: `… then verify the confirmation reads "Order placed"`. @@ -277,19 +286,22 @@ A pasted `curl` works too and is kept verbatim (method, headers, body, auth): curl -X POST https://api.example.com/login -H 'Content-Type: application/json' -d '{"u":"a","p":"b"}', save the response as login ``` -Once saved, reference the response by name: +Once the call is made, use the response in plain English. `{{name}}` is reserved for global variables and secrets (Β§6); a value the run produces is never written that way. -| Reference | Resolves to | +| To use | Say | |---|---| -| `{{order.status}}` | the HTTP status code (e.g. `201`) | -| `{{order.response_body}}` | the whole response body | -| `{{order.response_body.}}` | a field from the JSON response body | +| the HTTP status code | `assert the response status is 201` | +| a field from the JSON body | `store the id from the response body as 'order_id'` | +| the whole body | `store the response body as 'order_body'` | +| a value later in the run | `the stored order_id value` | + +When one objective makes several calls, name each response (`save the response as order`) and say which one you mean: `assert the order response status is 201`. Then assert or chain on it β€” API calls and browser actions mix freely in one objective: ```text Call POST https://api.example.com/login with body {"u": "{{user}}", "p": "{{password}}"}, save the response as login, -assert {{login.status}} is 200, +assert the response status is 200, then open https://app.example.com and verify the dashboard loads ``` @@ -322,6 +334,8 @@ For DevTools extractions, the same rule applies β€” use "store" or "extract": Stored values land in `run_end.final_state` and feed the second results table per `SKILL.md Β§1.4`. +Refer back to a stored value in plain English (`the stored price value`, `the same version as the stored api_tag value`), never as `{{price}}`. The `{{name}}` form is for global variables and secrets only (Β§6). + --- ## 5. Chaining β€” action β†’ extraction β†’ assertion @@ -374,6 +388,7 @@ Use `{{name}}` syntax for values that should be parameterized: **Always parameterize:** credentials, API keys, tokens, environment-specific URLs. **OK to hardcode:** one-off URLs, static UI text, navigation paths. +**Never `{{name}}`:** anything the run itself produces. A stored value or an API response is referenced in plain English (Β§3.5, Β§4): `the stored order_id value`, `the response status`. Mark credentials with `secret: true` in the variables JSON so they're masked in logs and routed to the secrets store: @@ -430,6 +445,7 @@ Positional assertions check where something is on the page: | Cram 25 operations into one objective | Split at logical boundaries (login, navigate, action, verify) | Long runs drift and stall. | | "Check the page is fast" | "Assert LCP is under 2500ms and CLS is below 0.1" | Use the explicit web-vital metric, not a vague "fast." | | "Make sure no errors" | "Assert no console errors and no API calls returned 5xx" | Be explicit about which kind of error you're checking. | +| "Place the order, then verify it succeeded by checking Order History" | "Place the order, open Order History, then verify the newest order shows 'Processing'" | A check that needs its own navigation is an action hiding inside an assertion. Put the route in the actions and keep the assertion a pure observation of the screen it lands on. | | "Use DevTools to intercept the response and issue a GET to the invoice endpoint" | "Open the invoice from the order page, then verify the request to `/api/orders/42/invoice` returned 200 with `content-disposition: attachment`" | Describe the user action and the observable outcome, not the mechanism β€” the agent picks the method, and a prescribed mechanism it cannot perform leaves it stuck. | --- @@ -483,7 +499,7 @@ The step body is exactly the same grammar as `kane-cli run`. Everything in this ```text "Call POST https://api.example.com/orders with body {"item": "sku_42", "qty": 1}, save the response as order, - assert {{order.status}} is 201, + assert the response status is 201, then open https://app.example.com/orders, assert an order for 'sku_42' is visible" ``` diff --git a/.agents/skills/kane-cli/references/parsing.md b/.agents/skills/kane-cli/references/parsing.md index 9af3442..7622576 100644 --- a/.agents/skills/kane-cli/references/parsing.md +++ b/.agents/skills/kane-cli/references/parsing.md @@ -8,18 +8,19 @@ With `--agent`, kane-cli outputs one JSON object per line to **stdout**. Progres ## Event Types -**Progress events** (bulk of the output β€” one per step): +**Progress events** (a start and completion event per step): ```json -{"step": 1, "status": "passed", "remark": "Navigated to amazon.in"} -{"step": 2, "status": "passed", "remark": "Typed 'laptop' in search box"} +{"step": 1, "status": "running", "remark": "Navigate to amazon.in"} +{"step": 1, "status": "done", "remark": "Navigated to amazon.in"} +{"step": 2, "status": "done", "remark": "Typed 'laptop' in search box"} {"step": 3, "status": "failed", "remark": "Could not find Add to Cart button"} ``` | Field | Type | Description | |-------|------|-------------| | `step` | number | Step index (1-based) | -| `status` | string | `"passed"` or `"failed"` | +| `status` | string | `"running"` at start; `"done"` or `"failed"` at completion | | `remark` | string | What the agent did or why it failed | These are **untyped** β€” they have no `type` field. Do **not** key on `event.type === 'step_start'` or `'step_end'`; those event types are not emitted. @@ -38,7 +39,7 @@ These are **untyped** β€” they have no `type` field. Do **not** key on `event.ty | `test_md_bundle_sync` | `status: "ok"\|"failed"`, `commit_id`, `bytes?` (success) / `stage?` (failure) | `testmd run` / `testmd sync`: test bundle pushed to the cloud after an authored commit. Informational. | | `testrun_*` family | see `references/testrun.md` | Emitted only by `kane-cli testrun run`; terminal event is `testrun_done`, not `run_end`. | -**Note:** There is no `run_start` event β€” the first line is either a `bifurcation` or a progress object. +**Note:** The `run` stream has no `run_start` event; startup metadata or errors can precede progress. ### `error` with `code: "unresolved_variables"` (0.8.12+) @@ -64,7 +65,7 @@ Terminal: do not re-run the same command. Supply values (`--variables`, or fill **Note:** `ask_user` is auto-disabled when stdin is not a TTY. Since agents typically run kane-cli as a subprocess, ask_user events will not be emitted. Write objectives that don't require interactive input. -## Parsing Strategy +## Parsing Strategy for one-shot `run` Since progress events lack a `type` field, distinguish them from typed events like this: @@ -76,9 +77,9 @@ for each line of NDJSON: if obj.step exists β†’ progress event (step/status/remark) ``` -**Build automation on `run_end`** β€” it is the only event guaranteed to have a stable schema across versions. Use progress events for live status display only. +For one-shot `run`, build automation on `run_end` and process exit; other commands use the completion events listed below. Use progress events for live status display only. -**Terminal event** (always the last line): +**One-shot `run` completion event** (early refusals may exit without it): ```json { @@ -129,3 +130,9 @@ To cancel a run: ```json {"type": "cancel"} ``` + +## Command-specific completion + +The `run_end` parsing strategy applies to one-shot `run` only. For `testmd run`, collect `test_md_done.overall_status`, `duration_s`, `session_id`, and optional `share_url`; embedded `run_end` events can finish individual steps. Local suites emit `testrun_done`; dispatched remote suites then emit `remote_done` (retain `status`, `exit`, `sessions_path`). `generate` emits `generate_done`. Assurance conversational agent streams end in `done`; review/read verbs have their own contracts. Always check process exit too: early refusal, invalid plan or dry-run can exit without the normal completion event. + +Progress is for live display: count only `done`/`failed` completions, retaining child and execution context when step indices repeat. diff --git a/.agents/skills/kane-cli/references/testmd.md b/.agents/skills/kane-cli/references/testmd.md index 4efdbe5..470cbb4 100644 --- a/.agents/skills/kane-cli/references/testmd.md +++ b/.agents/skills/kane-cli/references/testmd.md @@ -54,7 +54,7 @@ Four parts in order: 1. **YAML frontmatter** β€” between `--- ... ---` at the very top. 2. **`# Title`** β€” decorative; everything before the first `## ` is ignored. -3. **`## H2` step headings** β€” one per step. The agent reads the step body, not the heading. +3. **`## H2` step headings** β€” one per step. The agent reads the step body; the parser also interprets heading markers for replay-only steps, imports and structured control flow. 4. **Step body** β€” either prose **or** a single `@import ` line. Never both. Prose bodies are objectives with the same grammar as `kane-cli run` β€” for the full pattern catalog (action verbs, assertion analyze methods, checkpoint types, chaining, worked examples), Read `references/objectives-cookbook.md`. Per-step `yaml` overrides go immediately under the heading, in a fenced block: @@ -75,7 +75,7 @@ Click submit and verify the confirmation banner. | `mode` | root | `action` (halts on auth walls) or `testing` (default β€” pushes through so negative-test assertions can fire) | | `url` | root | Start URL for the first step (bare domains get `https://`). Overridden by `--url`; falls back to config `default_url`. | | `tags` | root | Labels for batch selection: YAML list, `[a, b]`, or bare `a, b`. Trimmed, lowercased, deduped. Selected with `testrun --tags` (`references/testrun.md`); shown by `testmd list`. Empty β†’ parse error; per-step β†’ parse error. | -| `max_steps` | root + step | Max agent reasoning steps. Default `30`. | +| `max_steps` | root + step | Max agent reasoning steps. Engine fallback `30`; CLI default `50` (step config overrides). | | `timeout` | root + step | Hard kill per step in seconds. | | `headless` | root | No browser window. | | `variables` | root + step | `{{name}}` params, same shape as Β§3, with `secret: true` for credentials. Counts as a value for the pre-run check (0.8.12+) β€” a `{{name}}` with no value anywhere refuses the run before it starts | @@ -88,19 +88,19 @@ Files ending in `_test.md` are tests (valid entry points). Any other `.md` is a On the **first** run of a test, the agent authors each step and saves a recording. On **every later run**, each step replays from its recording β€” no agent, no LLM cost, much faster. -A step replays only if **all** of these hold: +An authorable step replays only if **all** of these hold (replay-only marked steps always require their recording): - A recording for that step exists, - Its prose is unchanged since the recording, - Its `yaml` block is unchanged, - No earlier step in the file invalidated it. -**Editing step N re-authors step N AND every step after it in the same file.** Each step starts where the previous step left off (URL, login, tabs). When step 3 changes, step 4 cannot safely replay against state that no longer exists. +**Editing step N re-authors authorable steps from N onwards in that file. Replay-only marked steps still replay.** Each step starts where the previous step left off (URL, login, tabs). When step 3 changes, step 4 cannot safely replay against state that no longer exists. Consequences when editing tests: -- A one-line tweak at the top of a 20-step test re-authors all 20 steps on the next run. +- A one-line tweak at the top of a 20-step test re-authors all authorable steps on the next run; replay-only steps keep their recordings. - To re-record only one step, edit only that step (or steps after it). -- `--author` forces full authoring for one run (debugging only). -- `rm -rf output-/` wipes the cache entirely. +- `--author` forces authorable steps to re-author; replay-only marked steps still require their recordings. +- Do not delete output recordings for replay-only marked steps; they cannot be recovered by re-authoring. ## `@import` for reusing flows @@ -129,7 +129,7 @@ Editing a helper re-authors that step in **every test that imports it**, plus ev | `kane-cli testmd run --agent [flags]` | Run a test | | `kane-cli testmd list` | List `*_test.md` files under cwd (NDJSON when non-TTY; records include `tags`) | | `kane-cli testmd status ` | Test Manager identity + local-sync state | -| `kane-cli testmd export [--code-language python\|javascript]` | Regenerate code export from existing recordings (no browser launch) | +| `kane-cli testmd export [--language python\|javascript]` | Regenerate code export from existing recordings (no browser launch) | | `kane-cli testmd delete ` | Local-only delete: removes source + `output-/`. Does NOT delete from Test Manager. | | `kane-cli testmd sync ` | Re-push the test bundle (test + imports + outputs) to the cloud. Rarely needed β€” it happens automatically after every authored commit; use only to recover a failed auto-sync. | @@ -143,12 +143,11 @@ Editing a helper re-authors that step in **every test that imports it**, plus ev | `--allow-missing-url` | off | Non-TTY only: start from the browser's current page instead of failing when the first step has no start URL. | | `--name ` | none | Persist the run under this name. Regex `[a-zA-Z0-9_-]+`. | | `--on-lock-conflict ` | none | Behavior when another user holds the test's edit lock. `readonly` = replay-only / no upload, `fail` = exit 2, `wait` = block until released | -| `--retry` | off | On replay failure, restart with a shrinking replay window | -| `--retry-count ` | `3` | Max retry restarts before falling back to full re-author | -| `--author` | off | Force authoring every step (skip replay decision) | +| `--no-adaptive-heal` | healing enabled | Disable adaptive healing after replay failure | +| `--author` | off | Force authorable steps to re-author; replay-only steps still replay | | `--bug-detection ` | config (`off`) | Flag suspected product bugs while **authoring** (`stop` halts on a confirmed bug; `continue` records it). Replay failures always investigate regardless. | -All Β§3 `run` flags also apply (`--agent`, `--headless`, `--max-steps`, `--timeout`, `--variables`, etc.). +Common flags include `--agent`, `--headless`, `--max-steps`, `--timeout`, and `--variables`. Command-specific flags differ; check `kane-cli testmd run --help`. Flag wins over frontmatter for everything **except** `variables` β€” the file owns variables; you can add new keys via flags but cannot override file-defined ones. @@ -195,14 +194,13 @@ On exit, kane-cli writes `/.testmuai/tests/amazon-search_test.md`. Move tha kane-cli testmd run ./tests/checkout_test.md \ --agent \ --headless \ - --on-lock-conflict wait \ - --retry + --on-lock-conflict wait ``` - `--agent` β€” NDJSON to stdout (auto-enabled when stdin is not a TTY; pass explicitly anyway). - `--headless` β€” no window. - `--on-lock-conflict wait` β€” block instead of failing if a teammate is editing the same test. -- `--retry` β€” automatically recover transient replay failures. +- Adaptive healing recovers replay failures by default; `--no-adaptive-heal` opts out. In non-TTY runs, interactive `ask_user` prompts are disabled β€” a step that would wait for input fails cleanly instead of hanging forever. Write CI test steps that don't depend on mid-run prompts. (Likewise, supply a start URL via the objective, `url:` frontmatter, or `--url`; a non-TTY run with no resolvable start URL fails unless you pass `--allow-missing-url`.) @@ -234,3 +232,13 @@ Parse errors abort **before** any browser launch with exit `2`. Common ones and | `auth/identity keys are CLI-only` | Pass `username` / `access_key` as CLI flags, not in frontmatter | When the user reports a parse error, fix the file before retrying β€” don't loop on the same error. + +## Replay policy and completion + +Adaptive healing is enabled by default: after a replay failure the runner tries up to three shrinking replay windows, then full re-authoring of authorable steps. `--no-adaptive-heal` disables this recovery. Retired `--retry` and `--retry-count` are accepted only to print a notice; they do not enable healing or change the fixed budget. + +Headings marked `@db`, `@api`, `@js`, `@smartui`, `@network_query`, or `@network_assertion` are replay-only. They still replay under `--author`, downstream divergence and adaptive healing. Missing recordings cannot be re-authored through these markers: restore the recording or recreate it through the originating workflow. Do not delete their tapes to force authoring; adding a marker does not create an operation. + +Structured control flow uses balanced heading markers: `@if`, `@elif`, `@else`, `@end-if`, `@while`, `@end-while`. An `@else` must be last in its conditional; end markers must match the opened block type. These are distinct from natural-language conditionals. Markers are excluded from the step body hash. Only one replay-only kind is allowed per step, and an import cannot also be marked replay-only. + +Under `--agent`, wait for `test_md_done` (file-level `overall_status`, `duration_s`, `session_id`, optional `share_url`) and process exit. Individual `run_end` events do not complete the file. diff --git a/.agents/skills/kane-cli/references/testrun.md b/.agents/skills/kane-cli/references/testrun.md index 3604258..6691f4c 100644 --- a/.agents/skills/kane-cli/references/testrun.md +++ b/.agents/skills/kane-cli/references/testrun.md @@ -15,7 +15,7 @@ ## Command ```bash -kane-cli testrun run [paths...] [flags] # NDJSON is automatic when stdout is piped β€” there is NO --agent flag on testrun +kane-cli testrun run [paths...] [flags] # NDJSON is automatic when stdin is not a TTY (use < /dev/null in terminal automation) β€” there is NO --agent flag on testrun ``` `[paths...]` is optional β€” omit it to auto-discover every `*_test.md` under the cwd. Explicit paths must end in `_test.md`. @@ -24,11 +24,11 @@ kane-cli testrun run [paths...] [flags] # NDJSON is automatic when stdout is |---|---|---| | `--match ` | Filter candidates by project-relative path regex | β€” | | `--tags ` | ANY-match on frontmatter `tags:` (repeatable or comma-separated, case-insensitive) | β€” | -| `--parallel ` | Worker count; each worker gets an isolated Chrome with a fresh temp profile | `1` | +| `--parallel ` | Worker count; each desktop worker gets an isolated Chrome with a fresh temp profile | `1` | | `--on-failure ` | `continue` (run everything) \| `fail-fast` (stop dispatching new members after a failure) | `continue` | | `--name