Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 22 additions & 12 deletions .agents/skills/kane-cli/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Browser automation + AI test authoring via kane-cli - run browser o

# Kane CLI — Browser Automation Skill

Use `kane-cli` for **any task that requires a real browser**: navigating websites, clicking elements, filling forms, searching, testing web UI, taking screenshots, or verifying deployments. Do NOT use Playwright, Puppeteer, or Selenium directly. Always run with `--agent` so output is structured NDJSON you can parse.
Use `kane-cli` for **any task that requires a real browser**: navigating websites, clicking elements, filling forms, searching, testing web UI, taking screenshots, or verifying deployments. Do NOT use Playwright, Puppeteer, or Selenium directly. Use `--agent` for `run`, `testmd run`, and `generate`. `testrun run` has no `--agent`: it emits NDJSON when **stdin** is not a TTY (use `< /dev/null` for terminal automation). Assurance conversational commands use `--mode agent`.

**Authoring test cases or scenarios?** Never write them by hand — kane-cli has two authoring pipelines, and the routing matters:

Expand Down Expand Up @@ -36,7 +36,7 @@ After that, run kane-cli normally — the variable is inherited:
kane-cli run "<objective>" --agent <other-flags>
```

Bash blocks until kane-cli exits, then hands you the complete stdout. Parse it, summarize what happened, and present the results table. Same pattern for `kane-cli testmd run` and `kane-cli generate`.
Bash blocks until kane-cli exits, then hands you the complete stdout. Parse it, summarize what happened, and present the results table. Wait for process completion on `testmd run` and `generate` too, but parse their own completion events: `test_md_done` and `generate_done`, respectively. An intermediate `run_end` does not finish a saved test.

Set a generous timeout (up to 600000ms) since browser runs can take a while.

Expand Down Expand Up @@ -100,7 +100,7 @@ The terminal event has `type: "run_end"` and stable fields: `status`, `summary`,
| 🟢 **Result** | Passed |
| 🎯 **Task** | <one_liner> |
| ⏱️ **Duration** | <duration>s |
| 👣 **Steps taken** | <count of progress events> |
| 👣 **Steps taken** | <count of completed progress events (done/failed), retaining child/execution context> |
| 📝 **What happened** | <summary> |
| 🔗 **View details** | [Open in KaneAI Dashboard](<test_url>) |
```
Expand Down Expand Up @@ -184,7 +184,7 @@ kane-cli run "<objective>" --agent [options]
| `--variables <json>` | Inline variables JSON (for `{{key}}` in objective) | None |
| `--variables-file <path>` | Load variables from a JSON file | None |
| `--ws-endpoint <url>` | Remote browser (LambdaTest grid) | Local Chrome |
| `--code-export` | Generate code export after upload | Off |
| `--code-export` | Generate code export after upload | config (`true` by default) |
| `--bug-detection <mode>` | Flag suspected product bugs while authoring: `off`/`stop`/`continue` (`stop` halts on a confirmed bug; `continue` records and keeps going) | config value (`off`) |

Other flags (`--global-context`, `--local-context`, `--cdp-endpoint`, `--allow-missing-url`) and the full variables precedence chain live in `references/setup-and-config.md`.
Expand Down Expand Up @@ -222,11 +222,11 @@ How you phrase the objective string determines what the agent does. Four pattern
| 🎯 **Action** | "go to", "click", "type", "search", "fill" | Performs browser actions |
| ✅ **Assertion** | "assert", "verify", "confirm", "check that" | Pass/fail check on a condition |
| 📦 **Extraction** | "store X as 'name'" | Persists a value into `run_end.final_state` |
| 🔌 **API call** | "call", "POST/GET a URL", a pasted `curl` | The agent makes the HTTP request itself; "save the response as X", then assert/reference `{{X.status}}` / `{{X.response_body…}}` |
| 🔌 **API call** | "call", "POST/GET a URL", a pasted `curl` | The agent makes the HTTP request itself; "save the response as X", then assert on it in plain English: "assert the response status is 200", "store the id from the response body as 'order_id'" |

### Two rules that make an objective replayable

1. **End every flow in a terminal assertion.** Close with a check of the resulting page state (`verify the cart shows 1 item`), not a bare `submit` or `confirm the dialog`. A run only earns a replayable pass/fail from a verify/assert — a pure-action objective (`add a laptop to the cart`) gets none. If the objective has several phases, each ends in its own check. Two traps: `confirm the dialog` is an action, not a check; and `verify the Submit button is visible` fails exactly when the action worked (the control disappears on success) — assert the outcome, not the trigger.
1. **End every flow in a terminal assertion.** Close with a check of the resulting page state (`verify the cart shows 1 item`), not a bare `submit` or `confirm the dialog`. A run only earns a replayable pass/fail from a verify/assert — a pure-action objective (`add a laptop to the cart`) gets none. If the objective has several phases, each ends in its own check. Phrase the closing check as what is on screen once the last action completes, so it verifies with no further click or navigation; if the evidence is elsewhere, make getting there an explicit action (`place the order, open Order History, then verify the newest order shows "Processing"`), never `verify the order succeeded by checking Order History`. Two traps: `confirm the dialog` is an action, not a check; and `verify the Submit button is visible` fails exactly when the action worked (the control disappears on success) — assert the outcome, not the trigger.
2. **Intent for the actions, literal for the data.** Phrase actions as goals (`Log in with {{user}}`) so the run absorbs layout drift; keep exact values literal or in `{{variables}}`. An expected-optional branch (a sometimes-there cookie banner) goes in an `if/else`, not assumed away.

**Shape:** an intent action carrying literal data, then a verify of an observable end state — `Search for "{{query}}" and open the first result, then verify the title contains "{{query}}"`. Full grammar in `references/objectives-cookbook.md §1`.
Expand All @@ -238,18 +238,18 @@ Vague phrasing like "read", "tell me", "report" does NOT reliably extract data
❌ `"go to example.com and read the page title"`
✅ `"go to example.com, store the page title as 'page_title'"`

Stored values appear in `run_end.final_state` and become the second results table per §1.4.
Stored values appear in `run_end.final_state` and become the second results table per §1.4. Refer back to a stored value in plain English (`the stored price value`), never as `{{price}}`; `{{name}}` is for global variables and secrets only.

### Calling APIs directly

The agent can make API calls itself — not just observe the page's traffic. Phrase an explicit call and name the response:

```text
"Call POST https://api.example.com/login with body {...}, save the response as login,
assert {{login.status}} is 200"
assert the response status is 200"
```

Reference the saved response as `{{login.status}}`, `{{login.response_body}}`, or `{{login.response_body.<field>}}`; a pasted `curl` works too. Full grammar in `references/objectives-cookbook.md` §3.5.
Use the response in plain English: `the response status`, `store the id from the response body as 'order_id'`, and later `the stored order_id value`. Never `{{login.status}}`: `{{name}}` is reserved for global variables and secrets. A pasted `curl` works too. Full grammar in `references/objectives-cookbook.md` §3.5.

### Chaining

Expand All @@ -269,18 +269,19 @@ Action → extraction → assertion in one objective:
| Specific: "click the 'Add to Cart' button" | Vague: "add the item" |
| Name extractions: "store X as 'price'" | Hope for values: "tell me the price" |
| `{{variables}}` for credentials/URLs | Hardcode secrets in the objective |
| Plain English for values the run produces: "the stored price value", "the response status" | `{{price}}` / `{{login.status}}` for a stored value or an API response |
| Always include starting URL | Assume the agent knows where to start |
| Split mega-objectives (>15 steps) into multiple runs | Cram everything into one |

---

## 5. Parsing `--agent` output — essentials
## 5. Parsing one-shot `run --agent` output — essentials

> Internal reference only. Never expose these field names to the user — translate them per §1.

Stdout is NDJSON, one event per line. There are two shapes:

- **Progress events** (most events) have `step` (1-based), `status` (`passed`/`failed`), `remark` — and **no `type` field**.
- **Progress events** (most events) have `step` (1-based), `status` (`running` at start, `done`/`failed` at completion), `remark` — and **no `type` field**.
- **Typed events** have a `type` field: `project_folder_auto_defaulted` (run-startup gate, fires before any progress when no project/folder is configured), `bifurcation`, `child_agent_start`, `child_agent_end`, `ask_user`, `error` (an `error` with `code: "unresolved_variables"` is a pre-run refusal and the **only** line — no `run_end` follows; handle per §3), and finally `run_end`.

Parsing strategy:
Expand All @@ -292,7 +293,7 @@ for each line:
else if obj.step exists → progress event → summarize per §1.3
```

`run_end` is the only event with a stable cross-version schema — build all post-run logic on it.
For one-shot `run`, build post-run logic on `run_end` and process exit. Saved tests and suites use their own completion events (see Command-specific completion below).

For full event schemas (`bifurcation` flow fields, `child_agent_*`, `ask_user` semantics, `cancel`/`user_response` outbound events, complete `run_end` field list), Read `references/parsing.md`.

Expand Down Expand Up @@ -363,3 +364,12 @@ Internal event/field names (`generate_snapshot`, `request_id`, …) are for pars
| Browse / create projects or folders, or parse the auto-default event | `references/test-manager.md` |
| First-time install, auth, or full config | `references/setup-and-config.md` |
| Compare / evaluate / benchmark kane-cli vs another tool or approach (cost, tokens, effort, ROI) | `references/fair-evaluation.md` |

## Command-specific completion

The `run_end` parsing strategy applies to one-shot `run` only. For `testmd run`, collect `test_md_done.overall_status`, `duration_s`, `session_id`, and optional `share_url`; embedded `run_end` events can finish individual steps. Local suites emit `testrun_done`; dispatched remote suites then emit `remote_done` (retain `status`, `exit`, `sessions_path`). `generate` emits `generate_done`. Assurance conversational agent streams end in `done`; review/read verbs have their own contracts. Always check process exit too: early refusal, invalid plan or dry-run can exit without the normal completion event.

Progress is for live display: count only `done`/`failed` completions, retaining child and execution context when step indices repeat.


For assertion mode, optional final validation, current-page `--analyzer-only` checks, streaming-network capture, and code-export defaults, read [Execution controls](references/execution-controls.md).
10 changes: 10 additions & 0 deletions .agents/skills/kane-cli/references/assurance.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,3 +229,13 @@ Never `context ingest` the new version first — reconcile does its own re-inges
- **One stream.** The re-extract child's events ride the reconcile stream itself, stamped `verb: "reconcile"` — parse per `references/assurance-parsing.md`.

For staleness that arrived outside a reconcile, `kane-cli maintain evolve` re-designs a use-case — it is interactive-only (the blast-radius confirmation is the point); suggest the user run it in a terminal rather than scripting around it.

## Sharing a context store

For location setup, sync, pull, push, clone and rebase recovery, see [Sharing the context store](context-sync.md).

## Lifecycle automation contracts

`context name`, `context retire`, and `context revert` accept `--mode agent` and emit an envelope ending in `done`. Destructive agent-mode operations require `--yes` even when stdin is a TTY. Other modes retain human output. Read commands use their documented `--json` flags; do not assume they all accept `--mode`.

`context review --mode agent` / `--json` returns review outcome rows on success rather than the conversational `done` contract. Early agent-mode errors may emit `error` plus `done`; check command-specific output and process exit.
6 changes: 6 additions & 0 deletions .agents/skills/kane-cli/references/evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,9 @@ Combines packs (execution ids or paths, order-significant) into one, sealed by d
## Debugging with a pack

The pack is the first place to look on a failure: the failed step's **failure record** (error + page state), its **console/network slice** (4xx/5xx or JS errors usually explain it), and the **annotated screenshot** (what the agent actually acted on). Full failure workflow: `references/debug.md`.

## Evidence merge identity

For ordinary runs, default merge identity distinguishes the test and commit (`external_id.test_id`, `external_id.commit_id`) and the environment (`environment.os`, `environment.os_version`, `environment.browser`, `environment.browser_version`). Re-runs with the same identity nest as attempts; a different environment produces a separate sibling.

Explicit collision policies can change grouping; a custom `--rules` file replaces the default rules rather than extending them. Check the selected identity rules before interpreting two runs as retries of the same test.
11 changes: 11 additions & 0 deletions .agents/skills/kane-cli/references/execution-controls.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Execution controls

## Assertion controls and current-page analysis

`run` and `testmd run` support `--assertion-mode dom|visual` (default `dom` with vision fallback) and `--final-validation on|off` (default off). Persist with `config set-assertion-mode` and `config set-final-validation`. Final validation controls the synthesized `cp_final` checkpoint independently of action/testing mode; keep explicit terminal assertions in objectives.

`run --analyzer-only --condition "<condition>"` checks the current desktop browser page without an objective, action steps or saved test. Repeat `--condition` for multiple checks. Use `--agent` or non-TTY input. Results contain `condition_results: boolean[]`; exit `0` means all conditions were judged, **not** that all are true. Exit `1` means a result was missing, and `3` means cancelled. This mode rejects mobile target/app/device options and code-export/name options.

Experimental `run --network-ws` and `run --network-sse` enable WebSocket and SSE capture; both default off. Persist with `config set-network-ws on|off` and `config set-network-sse on|off`. SSE capture is Chromium-only. Do not copy these run-only flags onto `testmd` or `testrun` commands.

Code export defaults to enabled, subject to saved configuration, and supports `python` (default) or `javascript`. `run`/`testmd run` use `--code-language`; `testmd export` uses `--language`. Upload eligibility is independent of action/testing mode.
2 changes: 1 addition & 1 deletion .agents/skills/kane-cli/references/fair-evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The most common evaluation error is pitting kane-cli **authoring** against the o
|---|---|---|
| **Create the test** | AI authoring — tokens, one-time | Agent/human **generates** the script — tokens and/or engineer hours, one-time |
| **Run the test** | Replay from cache — ~0 LLM | Execute the script — ~0 LLM |
| **UI changes / locator breaks** | Re-author only the failing step + downstream (`--retry` shrinking window); cost ∝ change | Human/agent finds & fixes broken selectors; debug the script |
| **UI changes / locator breaks** | Re-author only the failing step + downstream (default adaptive healing); cost ∝ change | Human/agent finds & fixes broken selectors; debug the script |
| **Ongoing maintenance** | Edit plain-English Markdown; cascade re-authors only forward; shared `@import` helpers fix once | Edit code; flaky-wait/fixture upkeep; selector churn |
| **Verify pass/fail** | Deterministic asserts (URL/title/DOM/network/console/cookies); AI vision only for ~10% visual checks | Code assertions; or a separate LLM judge if the check is semantic/visual |

Expand Down
6 changes: 3 additions & 3 deletions .agents/skills/kane-cli/references/mobile.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,8 @@ What to present after `testrun_done`/`remote_done`: the suite rollup (per `refer

The **same natural-language objective grammar** applies (`references/objectives-cookbook.md`): action verbs, assertions, extractions ("store as"), if/else, chaining, and variables all carry over. A mobile run just drives an app instead of a page.

The exception is **browser/DevTools-only checkpoints**, which are **web-only** and do not apply to a mobile run:

- Network (HTTP traffic), Console, DOM/selectors, Cookies, localStorage, Core Web Vitals (LCP/CLS/INP/FCP/TTFB).
Mobile capability depends on the platform and app. The implementation includes native network operations and Android cookie/storage access for visible Chrome or debuggable WebViews. This does not establish full desktop DevTools parity, general native DOM support, or equivalent iOS support. Verify device/app prerequisites and the specific checkpoint before relying on it; live platform parity remains unverified.

Write mobile objectives around what the app shows and does (open a screen, tap, type, assert visible text/state, store a value). And never point a mobile run at a URL: a mobile run drives an app, not a website.

In the interactive TUI, `/mobile` and `/desktop` can switch targets before the first dispatch. The target locks after dispatch; use `/new` to start a new session before switching.
Loading