diff --git a/docs/agent-assurance-ci-cd.md b/docs/agent-assurance-ci-cd.md index 868fe654b..f24eaeabe 100644 --- a/docs/agent-assurance-ci-cd.md +++ b/docs/agent-assurance-ci-cd.md @@ -94,6 +94,22 @@ Use CI only after the same agent, profile, and scenarios work locally. Commit re This recipe targets **Rook 0.1.3**. It runs an explicit suite, preserves evidence, and checks completion and verdict counts instead of interpreting a successful CLI process as a successful agent test. +## Choose Your CI/CD Platform + +| Platform | What the dedicated guide provides | +| --- | --- | +| [GitHub Actions](/support/docs/rook-github-actions/) | A protected-environment workflow, secret configuration, and evidence upload after success or failure. | +| [Jenkins](/support/docs/rook-jenkins/) | A declarative Jenkinsfile with scoped credentials, isolated Rook state, and archived results. | +| [Argo CD](/support/docs/rook-argocd/) | A PostSync Kubernetes Job, a reviewed-suite image, secret references, and persistent evidence. | + +These guides share a downloadable [reviewed-suite gate script](pathname:///support/resources/rook/rook-ci.sh), which you review and commit as `ci/rook-ci.sh` in your agent repository. It calls the CLI directly; a coding-agent skill is not required on the runner. For interactive authoring, use the [coding-agent setup guides](/support/docs/rook-coding-agents/). + +### Choose a Verdict Policy Deliberately + +The dedicated platform examples use a **strict release gate**: every explicitly selected scenario must pass, and missing evidence, Unable to Verify, incomplete work, or compromised results block the job. Blocking the job does not relabel an Unable to Verify verdict as Fail. + +The [public skill's general CI recipe](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/references/ci.md) is more permissive: it reports Unable to Verify and unrunnable gaps without failing on those outcomes alone. Choose and review the policy for your application; do not silently switch policies to get a green build. All versions must check completion, use the current run ID, preserve evidence, and reject malformed or missing result fields. + ## Prepare a Reviewed Suite 1. Select the correct environment, project, and agent. diff --git a/docs/agent-assurance-quickstart.md b/docs/agent-assurance-quickstart.md index 9b13e1e7c..556356db9 100644 --- a/docs/agent-assurance-quickstart.md +++ b/docs/agent-assurance-quickstart.md @@ -302,6 +302,8 @@ Enter /exit to leave Rook. Stop the sample server with Ctrl+C ## Continue After Your First Test {#continue-after-your-first-test} +To repeat this workflow through your coding assistant, [choose a client-specific Rook skill guide](/support/docs/rook-coding-agents/). To automate the reviewed suite, use [GitHub Actions](/support/docs/rook-github-actions/), [Jenkins](/support/docs/rook-jenkins/), or [Argo CD](/support/docs/rook-argocd/). + Use `rook status` at any point to check the selected project, active agent, and local/upstream state. In the TUI, bare `/project`, `/agent`, and `/profile` open pickers; select with the arrow keys and Enter. In a shell, their bare forms list the available records. ### Ask in Plain Language diff --git a/docs/agent-skills.md b/docs/agent-skills.md index ae9d5ac81..269b39447 100644 --- a/docs/agent-skills.md +++ b/docs/agent-skills.md @@ -131,6 +131,8 @@ Skills work with Claude Code, GitHub Copilot, Cursor, Gemini CLI, and any assist ## How to Install a Skill +Testing an AI agent with **Rook**? Its skill is distributed from the [public Rook repository](https://github.com/LambdaTest/rook/tree/main/skill-installer/skills), separately from the framework skills below. Follow the [Rook coding-agent setup guides](/support/docs/rook-coding-agents/) for Claude Code, Codex, Gemini CLI, GitHub Copilot CLI, OpenCode, Cursor CLI, Antigravity CLI, VS Code, Windsurf, or Antigravity IDE. The Rook skill installer is `npx @testmuai/rook-skill@latest` for Claude Code, Codex, and Gemini CLI; the other guides use manual project installation. Install Rook CLI separately. + Use the `agentskillsforall` installer to add a skill straight from the repository. No manual cloning is required. diff --git a/docs/rook-antigravity-cli.md b/docs/rook-antigravity-cli.md new file mode 100644 index 000000000..8f60c17ba --- /dev/null +++ b/docs/rook-antigravity-cli.md @@ -0,0 +1,130 @@ +--- +id: rook-antigravity-cli +title: Use Rook with Antigravity CLI +sidebar_label: Antigravity CLI +description: Set up the public Rook skill in Antigravity CLI, verify it with agy, safely test your AI agent, and review local and hosted evidence. +toc_max_heading_level: 2 +hide_title: false +slug: rook-antigravity-cli/ +--- + +# Use Rook with Antigravity CLI + +Use Antigravity CLI (`agy`) to operate Rook from your agent repository: inspect the implementation, prepare a profile, propose scenarios, and explain recorded results. Antigravity is the coding assistant; Rook CLI performs the agent testing. Installing this skill does not configure Antigravity itself as the agent under test or create a Rook MCP server. + +Prefer an editor? Use the [Antigravity IDE guide](/support/docs/rook-antigravity-ide/). Both guides use the same project skill folder, so you only need to copy the bundle once per repository. + +## Before You Start + +- Follow Google's [Antigravity CLI installation and authentication guide](https://antigravity.google/docs/cli/install/). Check `agy --version` and `agy --help` in your terminal. Your Antigravity account and model usage are separate from Rook authentication and credits. +- [Install the public Rook CLI](/support/docs/rook-installation/), run `rook --version` and `rook doctor`, and use `rook login` if needed. +- Open a repository containing an AI agent you own and a safe test environment. Complete the [sample quickstart](/support/docs/agent-assurance-quickstart/) first if you have not used Rook before. +- The copy example below requires Git and Bash. On Windows, use a compatible shell such as WSL and keep the client, repository, and Rook installation in the same execution environment. Do not put target credentials in the skill or chat. + +## Install the Rook Skill + +Antigravity is **not a named target of `@testmuai/rook-skill`**. Use a manual project installation of the complete [public Rook skill bundle](https://github.com/LambdaTest/rook/tree/main/skill-installer/skills). This is a standards-based setup, not an Antigravity-specific Rook plugin. The installer's `--agent gemini-cli` option installs for Gemini CLI, not Antigravity. + +Run this Bash block from your agent repository. It uses the same pinned public revision as the other manual-client guides and refuses to replace an existing file, directory, or symlink: + +```bash +( + set -eu + if test -e .agents/skills/rook || test -L .agents/skills/rook; then + echo "An existing rook skill needs review; nothing was overwritten." >&2 + exit 1 + fi + rook_skill_checkout=$(mktemp -d) + git clone --filter=blob:none https://github.com/LambdaTest/rook.git "$rook_skill_checkout/rook" + git -C "$rook_skill_checkout/rook" checkout --detach f199cb99c589b565e48a3a29781475daee907533 + mkdir -p .agents/skills + cp -R "$rook_skill_checkout/rook/skill-installer/skills" .agents/skills/rook +) +``` + +Keep `.agents/skills/rook/SKILL.md` and the entire `references/` directory together. Review the bundle before committing it for teammates. The temporary checkout contains public source only and can be removed after review. If you already installed it through the IDE guide, inspect that copy and skip this step. + +For updates, review a newer public revision, preserve any local changes, and replace the whole bundle in a reviewed change. The npm installer does not update or uninstall this manual copy. To remove it, preserve custom changes and remove only this project's `.agents/skills/rook`; other compatible assistants may also use it. + +## Confirm the CLI Loaded It + +Launch Antigravity from the repository root: + +```bash +agy +``` + +Inside the interactive CLI, enter `/skills` and check for `rook`. Skills are exposed as slash commands, so use `/rook` for the first request below. These are Antigravity prompt commands, not shell commands. If the skill was added while a session was running, start a fresh session. See Google's [CLI reference](https://antigravity.google/docs/cli/reference/) and [skill locations](https://antigravity.google/docs/skills#cli-skill-locations). + +Start with inspection only: + +```text +/rook Inspect this repository and explain the next safe Rook setup step. +Confirm the skill path and its bundled references. Do not invoke the target, +run paid Rook commands, install software, or change profiles yet. +``` + +Check that the response identifies `.agents/skills/rook/SKILL.md`, missing setup, and the next proposed command. Skill discovery alone is not a successful Rook test. + +## Run a Small, Reviewed Test + +After reviewing the target and cost, adapt this prompt to your agent: + +```text +/rook Help test the refund agent in this repository against its refund policy. +Use the staging profile and test fixtures only. Propose up to three scenarios. +Before each paid operation or target invocation, show the command, profile, +selected scenario, hooks, possible writes, and expected spending; ask me to approve. +After approval, run one selected scenario. Report its exact run ID and the +Pass, Fail, and Unable to Verify results with criterion-level evidence. +Do not run paid RCA or retry automatically. +``` + +Review the [profile and hooks](/support/docs/rook-profiles-and-hooks/) and `agent.yaml` before execution. Profile creation, repair, and testing can reach the target too; Rook cannot undo target-side changes. Use [Rook's environment store](/support/docs/rook-environment-and-secrets/) or approved environment variables for secrets. + +Keep Antigravity command approvals and Rook permissions in place. Do not enable `--dangerously-skip-permissions` just to complete this walkthrough. A prompt is not a spending cap: review both the client's model usage and Rook credits. See [permissions and safety](/support/docs/rook-permissions-and-safety/). + +## Review the Actual Result + +Require the current invocation's run ID, completion state, and evidence. Exit code zero does not guarantee that scenarios passed. From the same Rook workspace, run: + +```bash +rook report --json +rook ui --local +``` + +The local UI reads workspace evidence. For a normal shared run, sync the selected agent before execution with `rook sync`. Then use `rook ui` or open [Rook projects](https://rook.lambdatest.com/projects) and select project → agent → run → scenario. Deliberate `--test` runs stay local and do not appear on the shared timeline. + +### Local UI + +Inspect the scenario criteria and recorded files. This existing HTTP smoke-test screenshot illustrates Rook result review; it is not an Antigravity CLI capture or proof of a live Antigravity-driven run. + +Local Rook scenario result showing criterion-level HTTP smoke-test evidence + +### Hosted Web UI + +Match the uploaded run ID to the CLI report, then review **Verdict** and **Artefacts**. The screenshot is an example result view, not client-specific test evidence. + +Hosted Rook Verdict tab for reviewing an uploaded scenario result + +See [results and evidence](/support/docs/agent-assurance-results-and-evidence/) for incomplete or unverifiable outcomes and [both UI walkthroughs](/support/docs/rook-web-ui/) for navigation. + +## Troubleshooting + +| Symptom | What to check | +| --- | --- | +| `agy` is not found | Follow Google's CLI installation guide and reopen the terminal. Installing the IDE alone is not a CLI verification. | +| `/skills` does not list `rook` | Start `agy` in the repository containing `.agents/skills/rook/SKILL.md`, then start a fresh session. Check the selected skill path for duplicate copies. | +| Installed with `--agent gemini-cli`, but no skill appears | That installer target is for Gemini CLI. Use the project bundle above for Antigravity. | +| References cannot be read | Copy the complete `references/` folder beside `SKILL.md`, not only the Markdown entry file. | +| Rook works in another terminal but not through Antigravity | Check PATH, working directory, and access in the client's command environment. Review denied operations individually instead of disabling protections. | +| Login, sandbox, or network failure | Verify Rook authentication in the execution environment and allow only the access needed for the approved operation. Never paste credentials into the prompt. | +| Assistant says “passed,” but no run is visible | Ask for the run ID and saved report. Check whether execution was declined, halted, incomplete, or intentionally local-only. | + +## Next Steps and Sources + +- [Use the same project skill in Antigravity IDE](/support/docs/rook-antigravity-ide/). +- [Choose another coding client](/support/docs/rook-coding-agents/) or [automate a reviewed suite in CI/CD](/support/docs/agent-assurance-ci-cd/). +- [Public Rook skill](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/SKILL.md), [Antigravity CLI overview](https://antigravity.google/docs/cli/overview/), and [Google's Agent Skills documentation](https://antigravity.google/docs/skills). + +Setup instructions were checked against public documentation. The locally available `agy` 1.1.2 version/help commands were checked; this is not a minimum-version recommendation or a live end-to-end integration test. Client versions, account policies, and execution environments can differ. diff --git a/docs/rook-antigravity-ide.md b/docs/rook-antigravity-ide.md new file mode 100644 index 000000000..16c636ddc --- /dev/null +++ b/docs/rook-antigravity-ide.md @@ -0,0 +1,127 @@ +--- +id: rook-antigravity-ide +title: Use Rook with Antigravity IDE +sidebar_label: Antigravity IDE +description: Set up the public Rook skill in Antigravity IDE, verify workspace discovery, safely test your AI agent, and review local and hosted evidence. +toc_max_heading_level: 2 +hide_title: false +slug: rook-antigravity-ide/ +--- + +# Use Rook with Antigravity IDE + +Use Antigravity IDE's agent side panel to work through a Rook test in your agent repository. The assistant can inspect code, propose a profile and scenarios, and use Rook CLI after you approve the operations. The Rook skill is an instruction bundle, not an editor extension or MCP server; it does not make the IDE itself the test target. + +This page covers the standalone **Antigravity IDE**. For the terminal client (`agy`), use [Antigravity CLI](/support/docs/rook-antigravity-cli/). Google documents Antigravity 2.0 and IDE extensions separately; do not assume every interface has the same controls. Google also currently directs enterprise customers to Antigravity 2.0 or CLI rather than the standalone IDE. See the [IDE overview](https://antigravity.google/docs/ide/overview/). + +## Before You Start + +- Install Antigravity IDE from [Google's official download page](https://antigravity.google/download), complete sign-in, and open the agent repository as your workspace. +- [Install the public Rook CLI](/support/docs/rook-installation/). In the IDE's integrated terminal, run `rook --version` and `rook doctor`, then `rook login` if needed. Signing in to Antigravity does not sign you in to Rook. +- Use an AI agent you own and a safe test environment. Start with the [sample quickstart](/support/docs/agent-assurance-quickstart/) if you do not yet have a Rook workspace. +- Use Git and Bash for the copy example below. If using WSL or another remote environment, keep the workspace, skill, and Rook executable in the environment the IDE agent actually uses. Keep credentials out of committed files and chat. + +## Install the Workspace Skill + +Antigravity is **not a named target of `@testmuai/rook-skill`**. Copy the complete [public Rook skill bundle](https://github.com/LambdaTest/rook/tree/main/skill-installer/skills) into the workspace. This is manual Agent Skills compatibility, not an automatic installer integration. Do not substitute the installer's Gemini CLI target for Antigravity. + +Run this Bash block from your repository's root in the integrated terminal: + +```bash +( + set -eu + if test -e .agents/skills/rook || test -L .agents/skills/rook; then + echo "An existing rook skill needs review; nothing was overwritten." >&2 + exit 1 + fi + rook_skill_checkout=$(mktemp -d) + git clone --filter=blob:none https://github.com/LambdaTest/rook.git "$rook_skill_checkout/rook" + git -C "$rook_skill_checkout/rook" checkout --detach f199cb99c589b565e48a3a29781475daee907533 + mkdir -p .agents/skills + cp -R "$rook_skill_checkout/rook/skill-installer/skills" .agents/skills/rook +) +``` + +Keep `.agents/skills/rook/SKILL.md` and every file under `references/` together. Review the files before committing them for the team. The temporary checkout holds public source only and can be removed after review. If you already followed the Antigravity CLI guide in this repository, inspect and reuse its bundle instead of installing a second copy. + +Google's [skill documentation](https://antigravity.google/docs/skills#antigravity-ide-skill-locations) specifies `.agents/skills` for project skills. The older `.agent/skills` spelling remains compatible, but use the current path for this setup and avoid duplicate `rook` copies. + +For updates, review a newer source revision and replace the whole bundle after preserving local changes. The npm installer does not manage this copy. To stop using it, preserve custom changes and remove only this project's `.agents/skills/rook`; the CLI and other compatible assistants may share it. + +## Confirm the IDE Loaded It + +Open a new conversation in the [agent side panel](https://antigravity.google/docs/ide/agent-side-panel/), then open its **Customizations** menu and inspect the active skills for `rook`. Confirm that the selected workspace contains the copied bundle. Reopen the workspace and start a fresh conversation if it was installed during an existing session. + +Use an inspection-only request first: + +```text +Use the rook skill from .agents/skills/rook to inspect this repository. +Confirm the skill path and its bundled references, then explain the next safe +Rook setup step. Do not invoke the target, run paid Rook commands, install +software, or change profiles yet. +``` + +Ask the assistant to identify missing setup and its next proposed command. A generated plan or an IDE artifact is not proof that a Rook test ran. This walkthrough uses a named skill request; the CLI's `/skills` commands are not instructions to type in the IDE terminal. + +## Approve and Run a Bounded Test + +Adapt the following to your target agent and policy: + +```text +Use the rook skill to help test the refund agent against its refund policy. +Use the staging profile and test fixtures only. Propose up to three scenarios. +Before each paid operation or target invocation, show the command, profile, +selected scenario, hooks, possible writes, and expected spending; ask me to approve. +After approval, run one selected scenario. Report its exact run ID and the +Pass, Fail, and Unable to Verify results with criterion-level evidence. +Do not run paid RCA or retry automatically. +``` + +Review the assistant's plan and proposed terminal commands. Keep command review enabled; approving a plan is not blanket approval for later target writes or paid operations. Antigravity permissions and Rook grants are separate controls, and a prompt is not a hard budget cap. Client model usage and Rook credits are separate too. + +Check [profiles and hooks](/support/docs/rook-profiles-and-hooks/) and write-capable calls in `agent.yaml`. Profile creation, repair, and testing can invoke the target before a scenario run. Use [Rook's environment store](/support/docs/rook-environment-and-secrets/) or approved environment variables for credentials, and follow [permissions and safety](/support/docs/rook-permissions-and-safety/). + +## Review Rook Evidence Alongside the IDE + +Keep the IDE plan, code changes, and Rook verdict distinct. Require the current invocation's run ID and completion state; a successful command exit does not mean every scenario passed. In the same project terminal: + +```bash +rook report --json +rook ui --local +``` + +The local UI shows workspace evidence. For a shared run, sync the selected agent before execution with `rook sync`, then use `rook ui` or [Rook projects](https://rook.lambdatest.com/projects). Select project → agent → run → scenario and match the run ID. An intentional `--test` run remains local. + +### Local UI + +Review each criterion and its recorded files. This page-only HTTP smoke-test capture illustrates Rook's result view; it is not an Antigravity IDE screenshot or evidence that this integration was executed. + +Local Rook scenario criteria and evidence for the HTTP smoke-test sample + +### Hosted Web UI + +For uploaded results, inspect the **Verdict** and **Artefacts** tabs. This screenshot is illustrative Rook evidence, not an IDE-specific test capture. + +Hosted Rook scenario Verdict tab with uploaded result evidence + +For missing observations or incomplete runs, see [results and evidence](/support/docs/agent-assurance-results-and-evidence/). Use [both UI walkthroughs](/support/docs/rook-web-ui/) for the full navigation flow. + +## Troubleshooting + +| Symptom | What to check | +| --- | --- | +| `rook` is absent from active skills | Check the selected workspace, `.agents/skills/rook/SKILL.md`, and the **Customizations** menu. Start a new conversation after copying the bundle. | +| Skill instructions load but references fail | Keep the complete `references/` folder next to `SKILL.md`. | +| Duplicate or outdated instructions | Inspect the loaded path. Check legacy `.agent/skills` and user-level copies before moving or replacing anything. | +| CLI setup worked, but the IDE cannot find Rook | Check PATH and authentication in the IDE's integrated terminal and agent execution environment; restart the IDE after installation if needed. | +| Works locally but fails in WSL or a remote workspace | The execution environment needs its own reachable target, Rook installation, and approved credentials. Do not assume the host's PATH or login is shared. | +| A command needs extra permissions | Review the exact command and requested access. Do not disable all terminal or filesystem protections to make it run. | +| The IDE produced a report, but Rook shows no run | Require a saved Rook run ID and report, not only an IDE summary. Check for declined, halted, or local-only execution. | + +## Next Steps and Sources + +- [Use Rook from Antigravity CLI](/support/docs/rook-antigravity-cli/) or [choose another coding client](/support/docs/rook-coding-agents/). +- [Automate a reviewed suite in CI/CD](/support/docs/agent-assurance-ci-cd/). +- [Public Rook skill](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/SKILL.md), [Google's Agent Skills documentation](https://antigravity.google/docs/skills), and [Antigravity IDE overview](https://antigravity.google/docs/ide/overview/). + +Setup instructions are based on these public sources. This guide does not claim a live end-to-end Rook test inside Antigravity IDE; discovery and execution depend on your installed client, account policy, and workspace environment. diff --git a/docs/rook-argocd.md b/docs/rook-argocd.md new file mode 100644 index 000000000..559a05cbb --- /dev/null +++ b/docs/rook-argocd.md @@ -0,0 +1,224 @@ +--- +id: rook-argocd +title: Verify Agent Deployments with Rook and Argo CD +sidebar_label: Argo CD +description: Run Rook as a Kubernetes Job after Argo CD has applied an application and its resources are healthy. +hide_title: false +toc_max_heading_level: 2 +slug: rook-argocd/ +--- + +# Verify Agent Deployments with Rook and Argo CD + +Run Rook as a Kubernetes Job after Argo CD has applied an application and its resources are healthy. This is a post-deployment assurance check, not a CI build job or an Argo Workflows template. A failed Rook gate marks the sync operation failed; it does not automatically undo the deployment. + +## Prepare the Reviewed Suite + +Use a safe target that is reachable from the runner. In a local rehearsal, select the project and agent, create and test the profile, review its hooks and possible writes, and prove the selected scenarios work. Commit the reviewed `.testmuai/rook/` definitions and required hook scripts without credentials or old run histories. Do not generate new scenarios inside the release gate. + +Download [rook-ci.sh](pathname:///support/resources/rook/rook-ci.sh), inspect it, and commit it as `ci/rook-ci.sh` in **your agent repository**. The examples below call that checked-in file, not a script downloaded at execution time. The script requires Bash, jq, tar, and Rook CLI. + +Configure these values: + +| Setting | Meaning | +| --- | --- | +| `LT_USERNAME`, `LT_ACCESS_KEY` | Rook account credentials, supplied only by the platform's secret store. | +| `ROOK_PROJECT_ID`, `ROOK_AGENT_ID` | IDs matching the committed workspace definitions. | +| `ROOK_PROFILE` | A reviewed, reachable test profile, for example `staging`. | +| `ROOK_SCENARIO_IDS` | An explicit comma-separated suite, for example `SC-001,SC-004,SC-014`. Use your own IDs. | +| `ROOK_ALLOW_RULES` | Optional newline-separated, exact tool grants observed during rehearsal. No blanket approval is added. | +| Target credentials | Variables required by your profile, such as `AGENT_TOKEN`; these are separate from Rook credentials. | + +The `staging` profile names **your test target**, not the Rook service. `ROOK_ENV=prod` selects the public Rook service. The script uses the LT environment credential pair and isolated `ROOK_HOME`; it does not copy a developer's browser session. + +Execution calls the real target and can spend Rook credits. Review the suite, tool grants, fixture isolation, and spending before enabling a job. Use a fresh checkout/output directory and avoid concurrent jobs against shared mutable fixtures. + +## What Makes the Job Pass + +The shared script selects the project, agent, and profile; syncs the reviewed agent; and runs only the selected scenarios. It checks the process status, `halted`, `discarded`, and the run ID, then fetches **that run's** JSON report. + +This example deliberately uses a **strict release policy**: every selected scenario must pass, with no Unable to Verify, unjudged, skipped/not-run, unrunnable, or compromised results. Missing fields, malformed JSON, a mismatched run ID, or an unexpected scenario count fail closed. A command exit of zero by itself is insufficient. + +Unable to Verify remains its own verdict even when it blocks a release. The public skill's [general CI recipe](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/references/ci.md) allows that outcome alone; this stricter policy is a documented choice, not a change to Rook's verdict semantics. Add criterion-level requirements if your release needs evidence beyond scenario totals. + +## Choose the Right Hook + +Use **PostSync** to test the newly deployed agent. A PreSync hook runs before the new application resources are applied, so it cannot validate a service that exists only after this deployment. Use a separate pre-promotion environment or pipeline if the check must block production rollout. + +Argo CD does not run hooks during selective sync. Require a normal full-application sync for this verification, and do not treat a selective sync as having passed Rook. See [Argo CD phases and waves](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/). + +## Prepare the Namespace + +Before enabling the hook, provision the following in the same application namespace: + +- A Secret named `rook-ci-credentials`, managed through your secret-management process, with `LT_USERNAME`, `LT_ACCESS_KEY`, and the target's required keys. The example includes `AGENT_TOKEN`; remove or replace it if your profile differs. Never commit literal credentials. +- A PVC named `rook-evidence`, with enough space, an appropriate retention policy, and write access for UID/GID 10001. For example: + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: rook-evidence +spec: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 2Gi +``` + +This example uses the cluster's default StorageClass. Change it for your storage provisioner. A PVC is persistent storage, not an automatic off-cluster backup; configure backup/export and access controls before using it as release evidence. + +The target profile must reach the deployed service through its cluster DNS or a routable test URL. Add appropriate network policy and image-pull credentials for your environment. The job does not need a Kubernetes API token unless your reviewed hooks specifically require one. + +## Build a Reviewed-Suite Image + +Download the [Dockerfile](pathname:///support/resources/rook/Dockerfile) and save it as `ci/rook.Dockerfile`. Build from the reviewed **agent repository root** after adding `ci/rook-ci.sh` and the committed workspace definitions. + +```dockerfile +FROM debian:bookworm-slim +ARG ROOK_VERSION=0.1.3 +RUN apt-get update && apt-get install -y --no-install-recommends \ + bash ca-certificates curl jq tar gzip git \ + && curl -fsSL https://raw.githubusercontent.com/LambdaTest/rook/main/install.sh \ + -o /tmp/install-rook.sh \ + && bash /tmp/install-rook.sh --version "$ROOK_VERSION" --dir /usr/local/bin \ + && useradd --create-home --uid 10001 rook +WORKDIR /workspace +# Build with the reviewed repository root as the context; never COPY personal home state. +COPY --chown=10001:10001 .testmuai/rook/ .testmuai/rook/ +COPY --chown=10001:10001 ci/rook-ci.sh ci/rook-ci.sh +# Add only reviewed hook scripts and their runtime dependencies if your profile needs them. +USER 10001 +ENV ROOK_ENV=prod ROOK_HOME=/tmp/rook-home +ENTRYPOINT ["bash", "ci/rook-ci.sh"] +``` + +Add only the hook files, agent code, and language runtimes the selected profile actually requires. A remote HTTP profile may need none; a command-based profile does. Keep credentials and old run histories out of the build context and image. Pin your base image by digest according to your organization's policy. + +Build, scan, and publish this image through your existing trusted CI. Record its source revision and immutable image digest alongside the application revision. There is no prebuilt public image implied by this example. + +## Add the PostSync Job + +Put this manifest in the path rendered by your Argo CD Application. [Download the Job manifest](pathname:///support/resources/rook/argocd-job.yaml). Replace the registry/image digest, project ID, agent ID, profile, and scenario list before syncing. + +```yaml +apiVersion: batch/v1 +kind: Job +metadata: + name: rook-assurance + annotations: + argocd.argoproj.io/hook: PostSync + argocd.argoproj.io/hook-delete-policy: BeforeHookCreation +spec: + backoffLimit: 0 + activeDeadlineSeconds: 1800 + template: + spec: + restartPolicy: Never + automountServiceAccountToken: false + securityContext: + runAsNonRoot: true + runAsUser: 10001 + runAsGroup: 10001 + fsGroup: 10001 + containers: + - name: rook + # Build this image from the reviewed suite; replace with your registry and digest. + image: registry.example.com/team/rook-assurance@sha256:REPLACE_IMAGE_DIGEST + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: [ALL] + resources: + requests: + cpu: 250m + memory: 256Mi + limits: + cpu: '1' + memory: 1Gi + env: + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid + - name: ROOK_RESULTS_DIR + value: /evidence/$(POD_UID) + - name: ROOK_RUN_NAME + value: argocd-$(POD_UID) + - name: ROOK_PROJECT_ID + value: REPLACE_PROJECT_ID + - name: ROOK_AGENT_ID + value: REPLACE_AGENT_ID + - name: ROOK_PROFILE + value: staging + - name: ROOK_SCENARIO_IDS + value: SC-001,SC-004,SC-014 + - name: ROOK_ALLOW_RULES + value: '' + - name: LT_USERNAME + valueFrom: + secretKeyRef: + name: rook-ci-credentials + key: LT_USERNAME + - name: LT_ACCESS_KEY + valueFrom: + secretKeyRef: + name: rook-ci-credentials + key: LT_ACCESS_KEY + - name: AGENT_TOKEN + valueFrom: + secretKeyRef: + name: rook-ci-credentials + key: AGENT_TOKEN + volumeMounts: + - name: evidence + mountPath: /evidence + volumes: + - name: evidence + persistentVolumeClaim: + claimName: rook-evidence +``` + +The pod UID gives each attempt a distinct evidence directory on the PVC. `backoffLimit: 0` prevents Kubernetes from automatically retrying a paid test; the deadline bounds the job's lifetime. Argo CD sync retries and manual re-syncs can still start new paid runs. Make those retries deliberate. + +`BeforeHookCreation` replaces the previous named Job on the next sync. Evidence on the PVC survives that Job replacement. The example does not delete successful/failed hook Jobs immediately, so their status and logs remain inspectable until the next attempt. A hard kill or node failure can prevent the shell's exit trap from finishing the evidence archive; retain partial JSON and central logs, and do not treat missing evidence as success. + +## Verify the Deployment Outcome + +After a full sync, inspect the `rook-assurance` Job in the Argo CD resource tree and the `rook` container logs. A successful Kubernetes Job means the gate exited zero; a failed Job requires inspecting the Rook outcome and evidence, not just retrying the deployment. + +Retrieve `/evidence//` through your approved PVC backup/export or read-only maintenance-pod process. A completed pod cannot be relied on for `kubectl exec` or `kubectl cp`. Keep the PVC and its evidence after deleting a Job, and ensure later pruning does not remove your only copy. + +## Troubleshooting Argo CD + +- **Hook never starts:** check application health, the hook annotation, and whether selective sync bypassed hooks. +- **Image pull fails:** replace the example registry/digest and configure namespace image-pull access. +- **Pod remains pending:** inspect PVC binding, scheduling, and storage access mode. +- **Evidence permission denied:** verify volume ownership and the storage driver's `fsGroup` support. +- **Target unavailable:** confirm the profile points at the deployed service, not a developer's localhost. +- **Sync failed after rollout:** the application may still be running; use your existing rollback/promotion policy. A PostSync failure is not automatic rollback. + +See [Argo CD hook lifecycle](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/) for hook ordering, failure behavior, and deletion policies. + +## Review Results Locally and on the Web + +The gate prints the run ID and saves `run.json`, `report.json`, and, when a run directory exists, `evidence.tar.gz`. Keep failure output too. The archive contains this agent's run folders, not `ROOK_HOME`; a clean checkout avoids including earlier runs. Set retention and access controls because target responses and evidence can contain sensitive data. + +To investigate locally, restore the reviewed checkout and extract the **trusted** evidence archive at its repository root, preserving the `.testmuai/rook/projects/…` hierarchy. Select the matching project and agent, then run `rook report --json` or `rook ui --local`. Inspect a run's scenario → criteria → files. + +Local Rook files panel illustrating retained request, response, hook, snapshot, and verdict evidence + +For uploaded results, open [Rook projects](https://rook.lambdatest.com/projects) → project → agent → run. Match the run ID before reviewing **Response**, **Verdict**, and **Artefacts**. A hosted record is useful for sharing, but the pipeline's validated report determines the gate. + +Hosted Rook run review illustrating completion, profile, and scenario outcomes + +These are existing smoke-test result captures, not screenshots of this CI integration executing. A local URL on a CI runner is not accessible to teammates; do not expose the local UI publicly. If a hosted upload is missing, retain the evidence and investigate [result synchronization](/support/docs/rook-web-ui/#troubleshooting) rather than rerunning paid tests blindly. + +## Related Guides + +- [CI/CD policy and common setup](/support/docs/agent-assurance-ci-cd/) +- [Profiles, hooks, and phases](/support/docs/rook-profiles-and-hooks/) +- [Public Rook headless contract](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/references/headless-contract.md) + +These are reference integrations for your reviewed environment. Syntax and gate behavior can be validated locally, but a successful deployment/build must be demonstrated in your own configured platform. diff --git a/docs/rook-claude-code.md b/docs/rook-claude-code.md new file mode 100644 index 000000000..43b703633 --- /dev/null +++ b/docs/rook-claude-code.md @@ -0,0 +1,128 @@ +--- +id: rook-claude-code +title: Use Rook with Claude Code +sidebar_label: Claude Code +description: Set up the public Rook skill in Claude Code, verify discovery, test an AI agent safely, and review local and hosted evidence. +toc_max_heading_level: 2 +hide_title: false +slug: rook-claude-code/ +--- + +# Use Rook with Claude Code + +Use Claude Code to operate Rook from your agent repository: inspect the implementation, prepare a test profile, generate scenarios, and explain recorded results. The coding assistant runs Rook CLI; installing a skill does not turn Rook into an MCP server or automatically test the assistant itself. + +## Before You Start + +- Install and sign in to Claude Code; its account or model-provider access is separate from your Rook account. +- [Install the public Rook CLI](/support/docs/rook-installation/) and run `rook --version`, `rook doctor`, then `rook login` if needed. +- Open a repository containing an AI agent you own and a safe test environment. If this is your first run, complete the [sample quickstart](/support/docs/agent-assurance-quickstart/) first. +- Use Node.js 22+ and npm for the skill installer. Keep target credentials in Rook's environment store or approved environment variables, never in the skill or chat. + +## Install the Rook Skill + +The [public Rook installer](https://github.com/LambdaTest/rook/tree/main/skill-installer) has a dedicated target for this client. It installs the skill, not the Rook executable. + +```bash +npx @testmuai/rook-skill@latest install --agent claude-code +``` + +The bundle lives in `~/.claude/skills/rook/`. Check that both `SKILL.md` and `references/` exist. The skill has its own version; updating Rook CLI does not automatically update it. + +To update only this client's installer-owned copy: + +```bash +npx @testmuai/rook-skill@latest update --agent claude-code +``` + +Only when you want to remove the skill: + +```bash +npx @testmuai/rook-skill@latest uninstall --agent claude-code +``` + +Do not run uninstall as a setup step. If the installer reports modified files, preserve your changes before updating; it intentionally refuses to overwrite an edited or unowned skill. + +## Confirm the Client Loaded It + +From the agent repository, launch the client: + +```bash +claude +``` + +Start a new Claude Code session in the agent repository, type `/`, and select `rook`. If it is missing, ask “What skills are available?” and check the personal skill directory. + +Claude Code's own approval settings still apply. Loading `/rook` does not authorize shell commands, network access, target writes, or credit spending. This page covers local Claude Code sessions; a personal skill is not automatically installed in a remote session. + +## Test Your Agent Through a Prompt + +Start with an inspection-only request: + +```text +/rook Inspect this repository and explain the next safe setup step. Do not run paid commands or invoke the target yet. +``` + +A useful response identifies the selected project/agent, available profiles, missing setup, and the next command. It should not claim that tests ran merely because it read the skill. + +Once you have checked the target and approved the cost, give a bounded testing request: + +```text +Use Rook to test the refund agent in this repository against its refund policy. +Use the staging profile and test fixtures only. Propose up to three scenarios. +Before invoking the target, show me the selected scenario, hooks, possible writes, +and expected credit spending, then ask for confirmation. +After approval, run one selected scenario and report its run ID, Pass, Fail, +Unable to Verify, and criterion-level evidence. Do not run paid RCA or retry +automatically. +``` + +Substitute your agent and policy. The assistant should discover/select the agent, configure or test its [profile and hooks](/support/docs/rook-profiles-and-hooks/), and review generated scenarios before execution. A normal shared run requires `rook sync`; use `--test` only when you deliberately want a local-only result. + +Profile creation, repair, testing, and scenario execution can invoke the real target. Inspect `agent.yaml` for write-capable calls before approving them. Discovery, generation, execution, and other model-backed operations can spend credits; the client's own model costs are separate. A prompt is not a hard spending cap. See [permissions and safety](/support/docs/rook-permissions-and-safety/). + +## Review the Actual Result + +Ask for the current invocation's run ID and report. A zero process exit code does not mean every scenario passed. Keep **Pass**, **Fail**, and **Unable to Verify** separate; include incomplete work and the recorded evidence for each criterion. + +From the same project workspace: + +```bash +rook report --json +rook ui --local +``` + +The local UI reads workspace evidence. For uploaded runs, use `rook ui` or open [Rook projects](https://rook.lambdatest.com/projects), then select the project → agent → run → scenario. An intentional `--test` run stays local; do not expect it on the shared timeline. + +### Local UI + +Open the scenario's criteria and files to inspect what actually happened. This existing HTTP smoke-test capture illustrates result review; it is not a screenshot of this coding client or proof that this integration was executed. + +Local Rook scenario result with criterion-level evidence for the HTTP smoke-test sample + +### Hosted Web UI + +Open the uploaded scenario's **Verdict** and **Artefacts** tabs and match its run ID to the CLI report. This is an example result view, not a client-specific test capture. + +Hosted Rook Verdict tab illustrating review of an uploaded scenario result + +See [results and evidence](/support/docs/agent-assurance-results-and-evidence/) for interpreting missing observations and [both UI walkthroughs](/support/docs/rook-web-ui/) for navigation. + +## Troubleshooting + +| Symptom | What to check | +| --- | --- | +| Skill is missing | Confirm the exact `SKILL.md` filename, its directory, and the active workspace; restart or reload the client's skills. | +| Skill loads but references fail | Copy the full `references/` folder beside `SKILL.md`; do not install just one raw Markdown file. | +| `rook` is not found | Check PATH in the client's command-execution environment, not only your regular terminal. | +| Wrong instructions are loaded | Look for another `rook` skill at project or user scope; inspect the selected path before changing files. | +| Agent cannot run a command | Review client approvals and Rook grants separately; approve the specific operation, not all future commands. | +| “Passed” but no evidence | Require the exact run ID, completion state, and saved verdict; a natural-language claim is insufficient. | + +## Next Steps and Sources + +- [Choose another coding client](/support/docs/rook-coding-agents/). +- [Automate a reviewed suite in CI/CD](/support/docs/agent-assurance-ci-cd/). +- [Public Rook skill](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/SKILL.md) and [Claude Code skill documentation](https://code.claude.com/docs/en/skills). + +Installation/discovery instructions are based on the public sources above. Availability still depends on your client version, account policy, and execution environment; this guide does not claim a live end-to-end test in every client. diff --git a/docs/rook-codex.md b/docs/rook-codex.md new file mode 100644 index 000000000..c03346b8f --- /dev/null +++ b/docs/rook-codex.md @@ -0,0 +1,128 @@ +--- +id: rook-codex +title: Use Rook with Codex CLI +sidebar_label: Codex CLI +description: Set up the public Rook skill in Codex CLI, verify discovery, test an AI agent safely, and review local and hosted evidence. +toc_max_heading_level: 2 +hide_title: false +slug: rook-codex/ +--- + +# Use Rook with Codex CLI + +Use Codex CLI to operate Rook from your agent repository: inspect the implementation, prepare a test profile, generate scenarios, and explain recorded results. The coding assistant runs Rook CLI; installing a skill does not turn Rook into an MCP server or automatically test the assistant itself. + +## Before You Start + +- Install and sign in to Codex CLI; its account or model-provider access is separate from your Rook account. +- [Install the public Rook CLI](/support/docs/rook-installation/) and run `rook --version`, `rook doctor`, then `rook login` if needed. +- Open a repository containing an AI agent you own and a safe test environment. If this is your first run, complete the [sample quickstart](/support/docs/agent-assurance-quickstart/) first. +- Use Node.js 22+ and npm for the skill installer. Keep target credentials in Rook's environment store or approved environment variables, never in the skill or chat. + +## Install the Rook Skill + +The [public Rook installer](https://github.com/LambdaTest/rook/tree/main/skill-installer) has a dedicated target for this client. It installs the skill, not the Rook executable. + +```bash +npx @testmuai/rook-skill@latest install --agent codex +``` + +The bundle lives in `~/.agents/skills/rook/`. Check that both `SKILL.md` and `references/` exist. The skill has its own version; updating Rook CLI does not automatically update it. + +To update only this client's installer-owned copy: + +```bash +npx @testmuai/rook-skill@latest update --agent codex +``` + +Only when you want to remove the skill: + +```bash +npx @testmuai/rook-skill@latest uninstall --agent codex +``` + +Do not run uninstall as a setup step. If the installer reports modified files, preserve your changes before updating; it intentionally refuses to overwrite an edited or unowned skill. + +## Confirm the Client Loaded It + +From the agent repository, launch the client: + +```bash +codex +``` + +Run `/skills` in Codex and select `rook`, or type `$rook` in the prompt. Restart Codex if a newly installed skill is not visible. + +Codex's sandbox and approval policy remain in control of command execution. If the environment blocks a network request or target command, resolve that specific restriction; do not disable the sandbox just to make a test pass. Install the skill and CLI inside a remote environment if that is where Codex executes commands. + +## Test Your Agent Through a Prompt + +Start with an inspection-only request: + +```text +$rook Inspect this repository and explain the next safe setup step. Do not run paid commands or invoke the target yet. +``` + +A useful response identifies the selected project/agent, available profiles, missing setup, and the next command. It should not claim that tests ran merely because it read the skill. + +Once you have checked the target and approved the cost, give a bounded testing request: + +```text +Use Rook to test the refund agent in this repository against its refund policy. +Use the staging profile and test fixtures only. Propose up to three scenarios. +Before invoking the target, show me the selected scenario, hooks, possible writes, +and expected credit spending, then ask for confirmation. +After approval, run one selected scenario and report its run ID, Pass, Fail, +Unable to Verify, and criterion-level evidence. Do not run paid RCA or retry +automatically. +``` + +Substitute your agent and policy. The assistant should discover/select the agent, configure or test its [profile and hooks](/support/docs/rook-profiles-and-hooks/), and review generated scenarios before execution. A normal shared run requires `rook sync`; use `--test` only when you deliberately want a local-only result. + +Profile creation, repair, testing, and scenario execution can invoke the real target. Inspect `agent.yaml` for write-capable calls before approving them. Discovery, generation, execution, and other model-backed operations can spend credits; the client's own model costs are separate. A prompt is not a hard spending cap. See [permissions and safety](/support/docs/rook-permissions-and-safety/). + +## Review the Actual Result + +Ask for the current invocation's run ID and report. A zero process exit code does not mean every scenario passed. Keep **Pass**, **Fail**, and **Unable to Verify** separate; include incomplete work and the recorded evidence for each criterion. + +From the same project workspace: + +```bash +rook report --json +rook ui --local +``` + +The local UI reads workspace evidence. For uploaded runs, use `rook ui` or open [Rook projects](https://rook.lambdatest.com/projects), then select the project → agent → run → scenario. An intentional `--test` run stays local; do not expect it on the shared timeline. + +### Local UI + +Open the scenario's criteria and files to inspect what actually happened. This existing HTTP smoke-test capture illustrates result review; it is not a screenshot of this coding client or proof that this integration was executed. + +Local Rook scenario result with criterion-level evidence for the HTTP smoke-test sample + +### Hosted Web UI + +Open the uploaded scenario's **Verdict** and **Artefacts** tabs and match its run ID to the CLI report. This is an example result view, not a client-specific test capture. + +Hosted Rook Verdict tab illustrating review of an uploaded scenario result + +See [results and evidence](/support/docs/agent-assurance-results-and-evidence/) for interpreting missing observations and [both UI walkthroughs](/support/docs/rook-web-ui/) for navigation. + +## Troubleshooting + +| Symptom | What to check | +| --- | --- | +| Skill is missing | Confirm the exact `SKILL.md` filename, its directory, and the active workspace; restart or reload the client's skills. | +| Skill loads but references fail | Copy the full `references/` folder beside `SKILL.md`; do not install just one raw Markdown file. | +| `rook` is not found | Check PATH in the client's command-execution environment, not only your regular terminal. | +| Wrong instructions are loaded | Look for another `rook` skill at project or user scope; inspect the selected path before changing files. | +| Agent cannot run a command | Review client approvals and Rook grants separately; approve the specific operation, not all future commands. | +| “Passed” but no evidence | Require the exact run ID, completion state, and saved verdict; a natural-language claim is insufficient. | + +## Next Steps and Sources + +- [Choose another coding client](/support/docs/rook-coding-agents/). +- [Automate a reviewed suite in CI/CD](/support/docs/agent-assurance-ci-cd/). +- [Public Rook skill](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/SKILL.md) and [Codex CLI skill documentation](https://learn.chatgpt.com/docs/build-skills). + +Installation/discovery instructions are based on the public sources above. Availability still depends on your client version, account policy, and execution environment; this guide does not claim a live end-to-end test in every client. diff --git a/docs/rook-coding-agents.md b/docs/rook-coding-agents.md new file mode 100644 index 000000000..bceeb18e3 --- /dev/null +++ b/docs/rook-coding-agents.md @@ -0,0 +1,56 @@ +--- +id: rook-coding-agents +title: Use Rook with Coding Agents +sidebar_label: Choose a Coding Agent +description: Choose a Rook skill setup guide for coding CLIs, VS Code, Windsurf, or Antigravity IDE, using the public Rook skill bundle. +toc_max_heading_level: 2 +hide_title: false +slug: rook-coding-agents/ +--- + +# Use Rook with Coding Agents + +The Rook skill teaches a coding assistant to use Rook CLI to test an AI agent in your repository. You describe the testing goal; the assistant follows Rook's setup, execution, permission, and evidence workflow. You still review the target, spending, and real-world effects before execution. + +The skill is **not** the Rook executable, an editor extension, or a Rook MCP server. Install the [public CLI](/support/docs/rook-installation/) separately. Testing the coding assistant itself also requires a suitable agent profile; installing this skill alone does not configure that target. + +## Choose Your Client + +Each page includes its own installation, discovery check, first prompt, troubleshooting, and local/hosted result review. + +| Client | Setup provided here | Skill location | +| --- | --- | --- | +| [Claude Code](/support/docs/rook-claude-code/) | Rook npm skill installer | `~/.claude/skills/rook/` | +| [Codex CLI](/support/docs/rook-codex/) | Rook npm skill installer | `~/.agents/skills/rook/` | +| [Gemini CLI](/support/docs/rook-gemini-cli/) | Rook npm skill installer | `~/.gemini/skills/rook/` | +| [GitHub Copilot CLI](/support/docs/rook-copilot-cli/) | Public skill in a project directory | `.github/skills/rook/` | +| [OpenCode](/support/docs/rook-opencode/) | Public skill in a project directory | `.opencode/skills/rook/` | +| [Cursor CLI](/support/docs/rook-cursor-cli/) | Public skill in a project directory | `.cursor/skills/rook/` | +| [Antigravity CLI](/support/docs/rook-antigravity-cli/) | Public skill in a project directory | `.agents/skills/rook/` | +| [VS Code with GitHub Copilot](/support/docs/rook-vscode/) | Public skill in a project directory | `.github/skills/rook/` | +| [Windsurf Cascade](/support/docs/rook-windsurf/) | Public skill in a project directory | `.windsurf/skills/rook/` | +| [Antigravity IDE](/support/docs/rook-antigravity-ide/) | Public skill in a project directory | `.agents/skills/rook/` | + +The first three are explicit targets of the [public Rook skill installer](https://github.com/LambdaTest/rook/tree/main/skill-installer). The other guides use the clients' documented Agent Skills compatibility; they do not add new installer flags or imply a live certification of every client/version. Their pages link the corresponding vendor documentation. + +## Install for the Three Packaged Clients + +With Node.js 22+ and npm: + +```bash +npx @testmuai/rook-skill@latest +``` + +This installs for Claude Code, Codex, and Gemini CLI under your home directory. To avoid installing all three, follow the selected client's page and use its `--agent` flag. The currently published skill package is 0.1.0; its version is independent of Rook CLI 0.1.3. + +For a team-scoped installation or another compatible client, copy the complete `skill-installer/skills` bundle from [LambdaTest/rook](https://github.com/LambdaTest/rook/tree/main/skill-installer/skills), including `references/`. The individual guides provide non-overwriting, revision-pinned examples. Do not substitute a similarly named skill from an unrelated repository. + +## What a Useful First Session Looks Like + +1. The assistant loads the public skill and checks the installed CLI and workspace state. +2. It identifies the target agent, authentication needs, invocation profile, hooks, and possible writes. +3. You approve a small, explicitly scoped test and its expected cost. +4. It records the run ID and presents Pass, Fail, and Unable to Verify with evidence—not just a successful shell exit. +5. You inspect the same run in the [local or hosted UI](/support/docs/rook-web-ui/). + +For automation after that rehearsal, choose [GitHub Actions](/support/docs/rook-github-actions/), [Jenkins](/support/docs/rook-jenkins/), or [Argo CD](/support/docs/rook-argocd/). These pipelines invoke the CLI directly; they do not require a coding assistant or skill installer on the runner. diff --git a/docs/rook-copilot-cli.md b/docs/rook-copilot-cli.md new file mode 100644 index 000000000..50b2d76a9 --- /dev/null +++ b/docs/rook-copilot-cli.md @@ -0,0 +1,129 @@ +--- +id: rook-copilot-cli +title: Use Rook with GitHub Copilot CLI +sidebar_label: GitHub Copilot CLI +description: Set up the public Rook skill in GitHub Copilot CLI, verify discovery, test an AI agent safely, and review local and hosted evidence. +toc_max_heading_level: 2 +hide_title: false +slug: rook-copilot-cli/ +--- + +# Use Rook with GitHub Copilot CLI + +Use GitHub Copilot CLI to operate Rook from your agent repository: inspect the implementation, prepare a test profile, generate scenarios, and explain recorded results. The coding assistant runs Rook CLI; installing a skill does not turn Rook into an MCP server or automatically test the assistant itself. + +## Before You Start + +- Install and sign in to GitHub Copilot CLI; its account or model-provider access is separate from your Rook account. +- [Install the public Rook CLI](/support/docs/rook-installation/) and run `rook --version`, `rook doctor`, then `rook login` if needed. +- Open a repository containing an AI agent you own and a safe test environment. If this is your first run, complete the [sample quickstart](/support/docs/agent-assurance-quickstart/) first. +- Use Git and Bash for the project-skill example below. On Windows, use a compatible shell such as WSL. Keep target credentials in Rook's environment store or approved environment variables, never in the skill or chat. + +## Install the Rook Skill + +This client supports the Agent Skills format, but it is **not a named target of `@testmuai/rook-skill`**. Use its project skill directory with the unmodified bundle from the [public Rook repository](https://github.com/LambdaTest/rook/tree/main/skill-installer/skills). This is a standards-based integration, not a separate Rook plugin. + +Run this Bash example from your agent repository. It copies only the skill bundle from a pinned public source revision, and stops if a destination already exists: + +```bash +( + set -eu + rook_skill_checkout=$(mktemp -d) + git clone --filter=blob:none https://github.com/LambdaTest/rook.git "$rook_skill_checkout/rook" + git -C "$rook_skill_checkout/rook" checkout --detach f199cb99c589b565e48a3a29781475daee907533 + if test -e .github/skills/rook; then + echo "An existing rook skill needs review; nothing was overwritten." >&2 + exit 1 + fi + mkdir -p .github/skills + cp -R "$rook_skill_checkout/rook/skill-installer/skills" .github/skills/rook +) +``` + +Keep `.github/skills/rook/SKILL.md` and the entire `references/` directory together. Review the files before committing the project skill for teammates. The temporary checkout contains public source only and can be removed after review. + +For an update, review a newer public revision, back up the existing skill, and replace the **whole bundle** in a reviewed change. The Rook npm installer does not manage this manual copy. To stop using it, remove only this project's `.github/skills/rook` after preserving custom changes. + +## Confirm the Client Loaded It + +From the agent repository, launch the client: + +```bash +copilot +``` + +Start Copilot in this repository and run `/skills list`, then `/skills info rook`. Use `/skills reload` if you added the folder while the session was open. + +The `.github/skills/rook` project folder can also be used by GitHub Copilot in VS Code. This is the standalone `copilot` CLI, not the older `gh copilot` command. Keep its tool approvals enabled and approve only the reviewed Rook commands. + +## Test Your Agent Through a Prompt + +Start with an inspection-only request: + +```text +/rook Inspect this repository and explain the next safe setup step. Do not run paid commands or invoke the target yet. +``` + +A useful response identifies the selected project/agent, available profiles, missing setup, and the next command. It should not claim that tests ran merely because it read the skill. + +Once you have checked the target and approved the cost, give a bounded testing request: + +```text +Use Rook to test the refund agent in this repository against its refund policy. +Use the staging profile and test fixtures only. Propose up to three scenarios. +Before invoking the target, show me the selected scenario, hooks, possible writes, +and expected credit spending, then ask for confirmation. +After approval, run one selected scenario and report its run ID, Pass, Fail, +Unable to Verify, and criterion-level evidence. Do not run paid RCA or retry +automatically. +``` + +Substitute your agent and policy. The assistant should discover/select the agent, configure or test its [profile and hooks](/support/docs/rook-profiles-and-hooks/), and review generated scenarios before execution. A normal shared run requires `rook sync`; use `--test` only when you deliberately want a local-only result. + +Profile creation, repair, testing, and scenario execution can invoke the real target. Inspect `agent.yaml` for write-capable calls before approving them. Discovery, generation, execution, and other model-backed operations can spend credits; the client's own model costs are separate. A prompt is not a hard spending cap. See [permissions and safety](/support/docs/rook-permissions-and-safety/). + +## Review the Actual Result + +Ask for the current invocation's run ID and report. A zero process exit code does not mean every scenario passed. Keep **Pass**, **Fail**, and **Unable to Verify** separate; include incomplete work and the recorded evidence for each criterion. + +From the same project workspace: + +```bash +rook report --json +rook ui --local +``` + +The local UI reads workspace evidence. For uploaded runs, use `rook ui` or open [Rook projects](https://rook.lambdatest.com/projects), then select the project → agent → run → scenario. An intentional `--test` run stays local; do not expect it on the shared timeline. + +### Local UI + +Open the scenario's criteria and files to inspect what actually happened. This existing HTTP smoke-test capture illustrates result review; it is not a screenshot of this coding client or proof that this integration was executed. + +Local Rook scenario result with criterion-level evidence for the HTTP smoke-test sample + +### Hosted Web UI + +Open the uploaded scenario's **Verdict** and **Artefacts** tabs and match its run ID to the CLI report. This is an example result view, not a client-specific test capture. + +Hosted Rook Verdict tab illustrating review of an uploaded scenario result + +See [results and evidence](/support/docs/agent-assurance-results-and-evidence/) for interpreting missing observations and [both UI walkthroughs](/support/docs/rook-web-ui/) for navigation. + +## Troubleshooting + +| Symptom | What to check | +| --- | --- | +| Skill is missing | Confirm the exact `SKILL.md` filename, its directory, and the active workspace; restart or reload the client's skills. | +| Skill loads but references fail | Copy the full `references/` folder beside `SKILL.md`; do not install just one raw Markdown file. | +| `rook` is not found | Check PATH in the client's command-execution environment, not only your regular terminal. | +| Wrong instructions are loaded | Look for another `rook` skill at project or user scope; inspect the selected path before changing files. | +| Agent cannot run a command | Review client approvals and Rook grants separately; approve the specific operation, not all future commands. | +| “Passed” but no evidence | Require the exact run ID, completion state, and saved verdict; a natural-language claim is insufficient. | + +## Next Steps and Sources + +- [Choose another coding client](/support/docs/rook-coding-agents/). +- [Automate a reviewed suite in CI/CD](/support/docs/agent-assurance-ci-cd/). +- [Public Rook skill](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/SKILL.md) and [GitHub Copilot CLI skill documentation](https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/add-skills). + +Installation/discovery instructions are based on the public sources above. Availability still depends on your client version, account policy, and execution environment; this guide does not claim a live end-to-end test in every client. diff --git a/docs/rook-cursor-cli.md b/docs/rook-cursor-cli.md new file mode 100644 index 000000000..a28add75b --- /dev/null +++ b/docs/rook-cursor-cli.md @@ -0,0 +1,129 @@ +--- +id: rook-cursor-cli +title: Use Rook with Cursor CLI +sidebar_label: Cursor CLI +description: Set up the public Rook skill in Cursor CLI, verify discovery, test an AI agent safely, and review local and hosted evidence. +toc_max_heading_level: 2 +hide_title: false +slug: rook-cursor-cli/ +--- + +# Use Rook with Cursor CLI + +Use Cursor CLI to operate Rook from your agent repository: inspect the implementation, prepare a test profile, generate scenarios, and explain recorded results. The coding assistant runs Rook CLI; installing a skill does not turn Rook into an MCP server or automatically test the assistant itself. + +## Before You Start + +- Install and sign in to Cursor CLI; its account or model-provider access is separate from your Rook account. +- [Install the public Rook CLI](/support/docs/rook-installation/) and run `rook --version`, `rook doctor`, then `rook login` if needed. +- Open a repository containing an AI agent you own and a safe test environment. If this is your first run, complete the [sample quickstart](/support/docs/agent-assurance-quickstart/) first. +- Use Git and Bash for the project-skill example below. On Windows, use a compatible shell such as WSL. Keep target credentials in Rook's environment store or approved environment variables, never in the skill or chat. + +## Install the Rook Skill + +This client supports the Agent Skills format, but it is **not a named target of `@testmuai/rook-skill`**. Use its project skill directory with the unmodified bundle from the [public Rook repository](https://github.com/LambdaTest/rook/tree/main/skill-installer/skills). This is a standards-based integration, not a separate Rook plugin. + +Run this Bash example from your agent repository. It copies only the skill bundle from a pinned public source revision, and stops if a destination already exists: + +```bash +( + set -eu + rook_skill_checkout=$(mktemp -d) + git clone --filter=blob:none https://github.com/LambdaTest/rook.git "$rook_skill_checkout/rook" + git -C "$rook_skill_checkout/rook" checkout --detach f199cb99c589b565e48a3a29781475daee907533 + if test -e .cursor/skills/rook; then + echo "An existing rook skill needs review; nothing was overwritten." >&2 + exit 1 + fi + mkdir -p .cursor/skills + cp -R "$rook_skill_checkout/rook/skill-installer/skills" .cursor/skills/rook +) +``` + +Keep `.cursor/skills/rook/SKILL.md` and the entire `references/` directory together. Review the files before committing the project skill for teammates. The temporary checkout contains public source only and can be removed after review. + +For an update, review a newer public revision, back up the existing skill, and replace the **whole bundle** in a reviewed change. The Rook npm installer does not manage this manual copy. To stop using it, remove only this project's `.cursor/skills/rook` after preserving custom changes. + +## Confirm the Client Loaded It + +From the agent repository, launch the client: + +```bash +agent +``` + +Start a fresh Cursor CLI session in the repository and ask it to load `rook`. Confirm it reads `.cursor/skills/rook/SKILL.md` and can resolve the bundled `references/` files. + +The same project skill folder works with Cursor's editor Agent. This guide uses the local Cursor CLI (`agent`); it does not provision a Cloud Agent or copy your local credentials to one. See the [Cursor CLI overview](https://cursor.com/docs/cli/overview) if the executable is missing. + +## Test Your Agent Through a Prompt + +Start with an inspection-only request: + +```text +Use the rook skill from .cursor/skills/rook to inspect this repository. Explain the next safe setup step. Do not run paid commands or invoke the target yet. +``` + +A useful response identifies the selected project/agent, available profiles, missing setup, and the next command. It should not claim that tests ran merely because it read the skill. + +Once you have checked the target and approved the cost, give a bounded testing request: + +```text +Use Rook to test the refund agent in this repository against its refund policy. +Use the staging profile and test fixtures only. Propose up to three scenarios. +Before invoking the target, show me the selected scenario, hooks, possible writes, +and expected credit spending, then ask for confirmation. +After approval, run one selected scenario and report its run ID, Pass, Fail, +Unable to Verify, and criterion-level evidence. Do not run paid RCA or retry +automatically. +``` + +Substitute your agent and policy. The assistant should discover/select the agent, configure or test its [profile and hooks](/support/docs/rook-profiles-and-hooks/), and review generated scenarios before execution. A normal shared run requires `rook sync`; use `--test` only when you deliberately want a local-only result. + +Profile creation, repair, testing, and scenario execution can invoke the real target. Inspect `agent.yaml` for write-capable calls before approving them. Discovery, generation, execution, and other model-backed operations can spend credits; the client's own model costs are separate. A prompt is not a hard spending cap. See [permissions and safety](/support/docs/rook-permissions-and-safety/). + +## Review the Actual Result + +Ask for the current invocation's run ID and report. A zero process exit code does not mean every scenario passed. Keep **Pass**, **Fail**, and **Unable to Verify** separate; include incomplete work and the recorded evidence for each criterion. + +From the same project workspace: + +```bash +rook report --json +rook ui --local +``` + +The local UI reads workspace evidence. For uploaded runs, use `rook ui` or open [Rook projects](https://rook.lambdatest.com/projects), then select the project → agent → run → scenario. An intentional `--test` run stays local; do not expect it on the shared timeline. + +### Local UI + +Open the scenario's criteria and files to inspect what actually happened. This existing HTTP smoke-test capture illustrates result review; it is not a screenshot of this coding client or proof that this integration was executed. + +Local Rook scenario result with criterion-level evidence for the HTTP smoke-test sample + +### Hosted Web UI + +Open the uploaded scenario's **Verdict** and **Artefacts** tabs and match its run ID to the CLI report. This is an example result view, not a client-specific test capture. + +Hosted Rook Verdict tab illustrating review of an uploaded scenario result + +See [results and evidence](/support/docs/agent-assurance-results-and-evidence/) for interpreting missing observations and [both UI walkthroughs](/support/docs/rook-web-ui/) for navigation. + +## Troubleshooting + +| Symptom | What to check | +| --- | --- | +| Skill is missing | Confirm the exact `SKILL.md` filename, its directory, and the active workspace; restart or reload the client's skills. | +| Skill loads but references fail | Copy the full `references/` folder beside `SKILL.md`; do not install just one raw Markdown file. | +| `rook` is not found | Check PATH in the client's command-execution environment, not only your regular terminal. | +| Wrong instructions are loaded | Look for another `rook` skill at project or user scope; inspect the selected path before changing files. | +| Agent cannot run a command | Review client approvals and Rook grants separately; approve the specific operation, not all future commands. | +| “Passed” but no evidence | Require the exact run ID, completion state, and saved verdict; a natural-language claim is insufficient. | + +## Next Steps and Sources + +- [Choose another coding client](/support/docs/rook-coding-agents/). +- [Automate a reviewed suite in CI/CD](/support/docs/agent-assurance-ci-cd/). +- [Public Rook skill](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/SKILL.md) and [Cursor CLI skill documentation](https://cursor.com/docs/skills). + +Installation/discovery instructions are based on the public sources above. Availability still depends on your client version, account policy, and execution environment; this guide does not claim a live end-to-end test in every client. diff --git a/docs/rook-gemini-cli.md b/docs/rook-gemini-cli.md new file mode 100644 index 000000000..0a95b68e2 --- /dev/null +++ b/docs/rook-gemini-cli.md @@ -0,0 +1,128 @@ +--- +id: rook-gemini-cli +title: Use Rook with Gemini CLI +sidebar_label: Gemini CLI +description: Set up the public Rook skill in Gemini CLI, verify discovery, test an AI agent safely, and review local and hosted evidence. +toc_max_heading_level: 2 +hide_title: false +slug: rook-gemini-cli/ +--- + +# Use Rook with Gemini CLI + +Use Gemini CLI to operate Rook from your agent repository: inspect the implementation, prepare a test profile, generate scenarios, and explain recorded results. The coding assistant runs Rook CLI; installing a skill does not turn Rook into an MCP server or automatically test the assistant itself. + +## Before You Start + +- Install and sign in to Gemini CLI; its account or model-provider access is separate from your Rook account. +- [Install the public Rook CLI](/support/docs/rook-installation/) and run `rook --version`, `rook doctor`, then `rook login` if needed. +- Open a repository containing an AI agent you own and a safe test environment. If this is your first run, complete the [sample quickstart](/support/docs/agent-assurance-quickstart/) first. +- Use Node.js 22+ and npm for the skill installer. Keep target credentials in Rook's environment store or approved environment variables, never in the skill or chat. + +## Install the Rook Skill + +The [public Rook installer](https://github.com/LambdaTest/rook/tree/main/skill-installer) has a dedicated target for this client. It installs the skill, not the Rook executable. + +```bash +npx @testmuai/rook-skill@latest install --agent gemini-cli +``` + +The bundle lives in `~/.gemini/skills/rook/`. Check that both `SKILL.md` and `references/` exist. The skill has its own version; updating Rook CLI does not automatically update it. + +To update only this client's installer-owned copy: + +```bash +npx @testmuai/rook-skill@latest update --agent gemini-cli +``` + +Only when you want to remove the skill: + +```bash +npx @testmuai/rook-skill@latest uninstall --agent gemini-cli +``` + +Do not run uninstall as a setup step. If the installer reports modified files, preserve your changes before updating; it intentionally refuses to overwrite an edited or unowned skill. + +## Confirm the Client Loaded It + +From the agent repository, launch the client: + +```bash +gemini +``` + +Run `/skills list` in Gemini CLI. If `rook` is missing, run `/skills reload`. Review Gemini's skill-activation request before allowing it to load the instructions and supporting files. + +If a second `rook` skill exists in `.agents/skills` or `~/.agents/skills`, check which copy Gemini selected. The shared `.agents` location takes precedence over the `.gemini` alias at the same scope. Avoid keeping conflicting copies. + +## Test Your Agent Through a Prompt + +Start with an inspection-only request: + +```text +Use the rook skill to inspect this repository and explain the next safe setup step. Do not run paid commands or invoke the target yet. +``` + +A useful response identifies the selected project/agent, available profiles, missing setup, and the next command. It should not claim that tests ran merely because it read the skill. + +Once you have checked the target and approved the cost, give a bounded testing request: + +```text +Use Rook to test the refund agent in this repository against its refund policy. +Use the staging profile and test fixtures only. Propose up to three scenarios. +Before invoking the target, show me the selected scenario, hooks, possible writes, +and expected credit spending, then ask for confirmation. +After approval, run one selected scenario and report its run ID, Pass, Fail, +Unable to Verify, and criterion-level evidence. Do not run paid RCA or retry +automatically. +``` + +Substitute your agent and policy. The assistant should discover/select the agent, configure or test its [profile and hooks](/support/docs/rook-profiles-and-hooks/), and review generated scenarios before execution. A normal shared run requires `rook sync`; use `--test` only when you deliberately want a local-only result. + +Profile creation, repair, testing, and scenario execution can invoke the real target. Inspect `agent.yaml` for write-capable calls before approving them. Discovery, generation, execution, and other model-backed operations can spend credits; the client's own model costs are separate. A prompt is not a hard spending cap. See [permissions and safety](/support/docs/rook-permissions-and-safety/). + +## Review the Actual Result + +Ask for the current invocation's run ID and report. A zero process exit code does not mean every scenario passed. Keep **Pass**, **Fail**, and **Unable to Verify** separate; include incomplete work and the recorded evidence for each criterion. + +From the same project workspace: + +```bash +rook report --json +rook ui --local +``` + +The local UI reads workspace evidence. For uploaded runs, use `rook ui` or open [Rook projects](https://rook.lambdatest.com/projects), then select the project → agent → run → scenario. An intentional `--test` run stays local; do not expect it on the shared timeline. + +### Local UI + +Open the scenario's criteria and files to inspect what actually happened. This existing HTTP smoke-test capture illustrates result review; it is not a screenshot of this coding client or proof that this integration was executed. + +Local Rook scenario result with criterion-level evidence for the HTTP smoke-test sample + +### Hosted Web UI + +Open the uploaded scenario's **Verdict** and **Artefacts** tabs and match its run ID to the CLI report. This is an example result view, not a client-specific test capture. + +Hosted Rook Verdict tab illustrating review of an uploaded scenario result + +See [results and evidence](/support/docs/agent-assurance-results-and-evidence/) for interpreting missing observations and [both UI walkthroughs](/support/docs/rook-web-ui/) for navigation. + +## Troubleshooting + +| Symptom | What to check | +| --- | --- | +| Skill is missing | Confirm the exact `SKILL.md` filename, its directory, and the active workspace; restart or reload the client's skills. | +| Skill loads but references fail | Copy the full `references/` folder beside `SKILL.md`; do not install just one raw Markdown file. | +| `rook` is not found | Check PATH in the client's command-execution environment, not only your regular terminal. | +| Wrong instructions are loaded | Look for another `rook` skill at project or user scope; inspect the selected path before changing files. | +| Agent cannot run a command | Review client approvals and Rook grants separately; approve the specific operation, not all future commands. | +| “Passed” but no evidence | Require the exact run ID, completion state, and saved verdict; a natural-language claim is insufficient. | + +## Next Steps and Sources + +- [Choose another coding client](/support/docs/rook-coding-agents/). +- [Automate a reviewed suite in CI/CD](/support/docs/agent-assurance-ci-cd/). +- [Public Rook skill](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/SKILL.md) and [Gemini CLI skill documentation](https://geminicli.com/docs/cli/skills/). + +Installation/discovery instructions are based on the public sources above. Availability still depends on your client version, account policy, and execution environment; this guide does not claim a live end-to-end test in every client. diff --git a/docs/rook-github-actions.md b/docs/rook-github-actions.md new file mode 100644 index 000000000..dcb8f95a2 --- /dev/null +++ b/docs/rook-github-actions.md @@ -0,0 +1,150 @@ +--- +id: rook-github-actions +title: Run Rook in GitHub Actions +sidebar_label: GitHub Actions +description: Run a reviewed Rook suite from a manually triggered GitHub Actions workflow. +hide_title: false +toc_max_heading_level: 2 +slug: rook-github-actions/ +--- + +# Run Rook in GitHub Actions + +Run a reviewed Rook suite from a manually triggered GitHub Actions workflow. This example uses the public CLI, a protected environment, explicit scenario selection, and evidence upload even when the gate fails. It does not install a coding-agent skill or require a dedicated Rook action. + +## Prepare the Reviewed Suite + +Use a safe target that is reachable from the runner. In a local rehearsal, select the project and agent, create and test the profile, review its hooks and possible writes, and prove the selected scenarios work. Commit the reviewed `.testmuai/rook/` definitions and required hook scripts without credentials or old run histories. Do not generate new scenarios inside the release gate. + +Download [rook-ci.sh](pathname:///support/resources/rook/rook-ci.sh), inspect it, and commit it as `ci/rook-ci.sh` in **your agent repository**. The examples below call that checked-in file, not a script downloaded at execution time. The script requires Bash, jq, tar, and Rook CLI. + +Configure these values: + +| Setting | Meaning | +| --- | --- | +| `LT_USERNAME`, `LT_ACCESS_KEY` | Rook account credentials, supplied only by the platform's secret store. | +| `ROOK_PROJECT_ID`, `ROOK_AGENT_ID` | IDs matching the committed workspace definitions. | +| `ROOK_PROFILE` | A reviewed, reachable test profile, for example `staging`. | +| `ROOK_SCENARIO_IDS` | An explicit comma-separated suite, for example `SC-001,SC-004,SC-014`. Use your own IDs. | +| `ROOK_ALLOW_RULES` | Optional newline-separated, exact tool grants observed during rehearsal. No blanket approval is added. | +| Target credentials | Variables required by your profile, such as `AGENT_TOKEN`; these are separate from Rook credentials. | + +The `staging` profile names **your test target**, not the Rook service. `ROOK_ENV=prod` selects the public Rook service. The script uses the LT environment credential pair and isolated `ROOK_HOME`; it does not copy a developer's browser session. + +Execution calls the real target and can spend Rook credits. Review the suite, tool grants, fixture isolation, and spending before enabling a job. Use a fresh checkout/output directory and avoid concurrent jobs against shared mutable fixtures. + +## What Makes the Job Pass + +The shared script selects the project, agent, and profile; syncs the reviewed agent; and runs only the selected scenarios. It checks the process status, `halted`, `discarded`, and the run ID, then fetches **that run's** JSON report. + +This example deliberately uses a **strict release policy**: every selected scenario must pass, with no Unable to Verify, unjudged, skipped/not-run, unrunnable, or compromised results. Missing fields, malformed JSON, a mismatched run ID, or an unexpected scenario count fail closed. A command exit of zero by itself is insufficient. + +Unable to Verify remains its own verdict even when it blocks a release. The public skill's [general CI recipe](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/references/ci.md) allows that outcome alone; this stricter policy is a documented choice, not a change to Rook's verdict semantics. Add criterion-level requirements if your release needs evidence beyond scenario totals. + +## Configure GitHub + +1. Add the reviewed definitions, hook scripts, and `ci/rook-ci.sh` to your trusted default branch. +2. Create an environment named `rook-assurance`. Restrict deployment branches to your reviewed branch and configure required reviewers where your GitHub plan supports them. +3. Add environment secrets `LT_USERNAME`, `LT_ACCESS_KEY`, and any target secrets. The example uses `AGENT_TOKEN`; rename or remove it to match your profile. +4. Add environment variables `ROOK_PROJECT_ID`, `ROOK_AGENT_ID`, `ROOK_PROFILE`, `ROOK_SCENARIO_IDS`, and optional `ROOK_ALLOW_RULES`. +5. Ensure the target is reachable from the runner. For a private endpoint, use an approved network-connected runner; localhost on your laptop is not reachable from GitHub's runner. + +## Add the Workflow + +Save this as `.github/workflows/rook-assurance.yml`. [Download the workflow](pathname:///support/resources/rook/github-actions.yml). + +```yaml +name: Rook assurance +on: + workflow_dispatch: +permissions: + contents: read +concurrency: + group: rook-assurance + cancel-in-progress: false +jobs: + assurance: + if: github.ref == 'refs/heads/main' + runs-on: ubuntu-24.04 + timeout-minutes: 30 + environment: rook-assurance + env: + ROOK_ENV: prod + ROOK_HOME: ${{ runner.temp }}/rook-home + ROOK_PROJECT_ID: ${{ vars.ROOK_PROJECT_ID }} + ROOK_AGENT_ID: ${{ vars.ROOK_AGENT_ID }} + ROOK_PROFILE: ${{ vars.ROOK_PROFILE }} + ROOK_SCENARIO_IDS: ${{ vars.ROOK_SCENARIO_IDS }} + ROOK_ALLOW_RULES: ${{ vars.ROOK_ALLOW_RULES }} + ROOK_RUN_NAME: github-${{ github.run_id }}-${{ github.run_attempt }} + steps: + - uses: actions/checkout@v7 + with: + persist-credentials: false + - name: Install pinned public Rook CLI + shell: bash + run: | + set -euo pipefail + curl -fsSL https://raw.githubusercontent.com/LambdaTest/rook/main/install.sh \ + -o "$RUNNER_TEMP/install-rook.sh" + bash "$RUNNER_TEMP/install-rook.sh" --version 0.1.3 --dir "$RUNNER_TEMP/rook-bin" + echo "$RUNNER_TEMP/rook-bin" >> "$GITHUB_PATH" + command -v jq + - name: Run the reviewed suite + shell: bash + env: + LT_USERNAME: ${{ secrets.LT_USERNAME }} + LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }} + AGENT_TOKEN: ${{ secrets.AGENT_TOKEN }} + run: bash ci/rook-ci.sh + - name: Preserve results even on failure + if: always() + uses: actions/upload-artifact@v7 + with: + name: rook-${{ github.run_id }}-${{ github.run_attempt }} + path: rook-results/ + if-no-files-found: warn + retention-days: 7 +``` + +Change the `main` branch condition if your reviewed default branch has another name. The manual trigger avoids exposing credentials to untrusted pull-request code. Do not replace it with `pull_request_target` plus a checkout of an untrusted PR. Serialize jobs that share test fixtures. + +The workflow pins Rook CLI to the public 0.1.3 release and installs it only in the job's temporary directory. Action major versions are shown for readability; review and pin action commit SHAs for your organization's production policy. The hosted Ubuntu runner provides jq. A self-hosted runner needs the same tools and the minimum runtime requirements of the selected actions. + +## Run and Check the Workflow + +Open **Actions → Rook assurance → Run workflow** on the reviewed branch, then approve the environment if required. Inspect **Run the reviewed suite** for the run ID and verdict counts. The **Preserve results even on failure** step publishes a seven-day artifact from `rook-results/`; an early setup failure can legitimately produce no artifact. + +A red assurance step must remain red after uploading evidence. Do not add `continue-on-error` to the gate. If you later make this a required check, ensure its trigger runs for every event where branch protection expects it; a manual-only workflow is not an automatic PR check. + +## Troubleshooting GitHub Actions + +- **Missing credentials or variables:** check the environment name and approval status, not just repository-level settings. +- **Branch skipped:** match the job's branch condition to your reviewed branch. +- **Runner cannot reach the agent:** use a routable test endpoint or an approved private runner. +- **No evidence artifact:** inspect setup/sync failure first. Only `rook-results/` is uploaded, never home credentials. +- **Permission refusal:** reproduce the requested tool rule locally and approve the exact reviewed rule in `ROOK_ALLOW_RULES`. + +The workflow follows [GitHub workflow syntax](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax), [checkout](https://github.com/actions/checkout), and [artifact upload](https://github.com/actions/upload-artifact) documentation. + +## Review Results Locally and on the Web + +The gate prints the run ID and saves `run.json`, `report.json`, and, when a run directory exists, `evidence.tar.gz`. Keep failure output too. The archive contains this agent's run folders, not `ROOK_HOME`; a clean checkout avoids including earlier runs. Set retention and access controls because target responses and evidence can contain sensitive data. + +To investigate locally, restore the reviewed checkout and extract the **trusted** evidence archive at its repository root, preserving the `.testmuai/rook/projects/…` hierarchy. Select the matching project and agent, then run `rook report --json` or `rook ui --local`. Inspect a run's scenario → criteria → files. + +Local Rook files panel illustrating retained request, response, hook, snapshot, and verdict evidence + +For uploaded results, open [Rook projects](https://rook.lambdatest.com/projects) → project → agent → run. Match the run ID before reviewing **Response**, **Verdict**, and **Artefacts**. A hosted record is useful for sharing, but the pipeline's validated report determines the gate. + +Hosted Rook run review illustrating completion, profile, and scenario outcomes + +These are existing smoke-test result captures, not screenshots of this CI integration executing. A local URL on a CI runner is not accessible to teammates; do not expose the local UI publicly. If a hosted upload is missing, retain the evidence and investigate [result synchronization](/support/docs/rook-web-ui/#troubleshooting) rather than rerunning paid tests blindly. + +## Related Guides + +- [CI/CD policy and common setup](/support/docs/agent-assurance-ci-cd/) +- [Profiles, hooks, and phases](/support/docs/rook-profiles-and-hooks/) +- [Public Rook headless contract](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/references/headless-contract.md) + +These are reference integrations for your reviewed environment. Syntax and gate behavior can be validated locally, but a successful deployment/build must be demonstrated in your own configured platform. diff --git a/docs/rook-installation.md b/docs/rook-installation.md index 16f3236b1..587e77c3b 100644 --- a/docs/rook-installation.md +++ b/docs/rook-installation.md @@ -271,6 +271,8 @@ If a public install path still fails, [open a Rook issue](https://github.com/Lam ## Next Step +If you use a coding assistant, [choose its Rook skill setup guide](/support/docs/rook-coding-agents/) after installing the CLI. Claude Code, Codex, Gemini CLI, Copilot CLI, OpenCode, Cursor CLI, Antigravity CLI, VS Code, Windsurf, and Antigravity IDE each have their own instructions. Installing the skill does not install or authenticate the CLI. + Continue with [How to Get Started With Agent Assurance](/support/docs/agent-assurance-quickstart/). diff --git a/docs/rook-jenkins.md b/docs/rook-jenkins.md new file mode 100644 index 000000000..a980a29b0 --- /dev/null +++ b/docs/rook-jenkins.md @@ -0,0 +1,153 @@ +--- +id: rook-jenkins +title: Run Rook in Jenkins +sidebar_label: Jenkins +description: Use a Jenkins declarative Pipeline to run a reviewed Rook suite and archive its results. +hide_title: false +toc_max_heading_level: 2 +slug: rook-jenkins/ +--- + +# Run Rook in Jenkins + +Use a Jenkins declarative Pipeline to run a reviewed Rook suite and archive its results. Jenkins executes the CLI directly; no coding assistant, skill package, or Rook-specific Jenkins plugin is required. + +## Prepare the Reviewed Suite + +Use a safe target that is reachable from the runner. In a local rehearsal, select the project and agent, create and test the profile, review its hooks and possible writes, and prove the selected scenarios work. Commit the reviewed `.testmuai/rook/` definitions and required hook scripts without credentials or old run histories. Do not generate new scenarios inside the release gate. + +Download [rook-ci.sh](pathname:///support/resources/rook/rook-ci.sh), inspect it, and commit it as `ci/rook-ci.sh` in **your agent repository**. The examples below call that checked-in file, not a script downloaded at execution time. The script requires Bash, jq, tar, and Rook CLI. + +Configure these values: + +| Setting | Meaning | +| --- | --- | +| `LT_USERNAME`, `LT_ACCESS_KEY` | Rook account credentials, supplied only by the platform's secret store. | +| `ROOK_PROJECT_ID`, `ROOK_AGENT_ID` | IDs matching the committed workspace definitions. | +| `ROOK_PROFILE` | A reviewed, reachable test profile, for example `staging`. | +| `ROOK_SCENARIO_IDS` | An explicit comma-separated suite, for example `SC-001,SC-004,SC-014`. Use your own IDs. | +| `ROOK_ALLOW_RULES` | Optional newline-separated, exact tool grants observed during rehearsal. No blanket approval is added. | +| Target credentials | Variables required by your profile, such as `AGENT_TOKEN`; these are separate from Rook credentials. | + +The `staging` profile names **your test target**, not the Rook service. `ROOK_ENV=prod` selects the public Rook service. The script uses the LT environment credential pair and isolated `ROOK_HOME`; it does not copy a developer's browser session. + +Execution calls the real target and can spend Rook credits. Review the suite, tool grants, fixture isolation, and spending before enabling a job. Use a fresh checkout/output directory and avoid concurrent jobs against shared mutable fixtures. + +## What Makes the Job Pass + +The shared script selects the project, agent, and profile; syncs the reviewed agent; and runs only the selected scenarios. It checks the process status, `halted`, `discarded`, and the run ID, then fetches **that run's** JSON report. + +This example deliberately uses a **strict release policy**: every selected scenario must pass, with no Unable to Verify, unjudged, skipped/not-run, unrunnable, or compromised results. Missing fields, malformed JSON, a mismatched run ID, or an unexpected scenario count fail closed. A command exit of zero by itself is insufficient. + +Unable to Verify remains its own verdict even when it blocks a release. The public skill's [general CI recipe](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/references/ci.md) allows that outcome alone; this stricter policy is a documented choice, not a change to Rook's verdict semantics. Add criterion-level requirements if your release needs evidence beyond scenario totals. + +## Configure Jenkins + +1. Use a Pipeline job pointing at a reviewed branch. Do not provide these credentials to jobs running untrusted PR code. +2. Provide a disposable Linux agent with label `rook-linux`, Bash, jq, tar, curl, Git, and network access to the public Rook service and your test target. +3. Install the standard Pipeline, Git, and Credentials Binding capabilities needed by this Jenkinsfile. +4. Add **Secret text** credentials with IDs `rook-lt-username` and `rook-lt-access-key`. Add `rook-agent-token` only if your profile needs that target credential. +5. Replace the project/agent placeholders, profile, selected scenario IDs, and tool grants in the Jenkinsfile. An empty grants value does not automatically authorize prompted operations. + +## Add the Jenkinsfile + +Commit this at your repository root beside the reviewed definitions and `ci/rook-ci.sh`. [Download the Jenkinsfile](pathname:///support/resources/rook/Jenkinsfile). + +```groovy +pipeline { + agent { label 'rook-linux' } + options { + skipDefaultCheckout(true) + disableConcurrentBuilds() + timeout(time: 30, unit: 'MINUTES') + } + environment { + ROOK_ENV = 'prod' + ROOK_PROJECT_ID = 'REPLACE_PROJECT_ID' + ROOK_AGENT_ID = 'REPLACE_AGENT_ID' + ROOK_PROFILE = 'staging' + ROOK_SCENARIO_IDS = 'SC-001,SC-004,SC-014' + ROOK_ALLOW_RULES = '' + } + stages { + stage('Reviewed checkout and assurance') { + steps { + // A new directory per build avoids stale result files in reused workspaces. + dir("rook-job-${env.BUILD_NUMBER}") { + checkout scm + withCredentials([ + string(credentialsId: 'rook-lt-username', variable: 'LT_USERNAME'), + string(credentialsId: 'rook-lt-access-key', variable: 'LT_ACCESS_KEY'), + string(credentialsId: 'rook-agent-token', variable: 'AGENT_TOKEN') + ]) { + sh '''#!/usr/bin/env bash +set -euo pipefail +set +x +job_temp=$(mktemp -d "${TMPDIR:-/tmp}/rook-jenkins.XXXXXX") +export ROOK_HOME="$job_temp/home" +export ROOK_RUN_NAME="jenkins-$BUILD_NUMBER" +curl -fsSL https://raw.githubusercontent.com/LambdaTest/rook/main/install.sh \ + -o "$job_temp/install-rook.sh" +bash "$job_temp/install-rook.sh" --version 0.1.3 --dir "$job_temp/bin" +export PATH="$job_temp/bin:$PATH" +bash ci/rook-ci.sh +''' + } + } + } + } + } + post { + always { + dir("rook-job-${env.BUILD_NUMBER}") { + archiveArtifacts artifacts: 'rook-results/**', allowEmptyArchive: true + } + } + } +} +``` + +The pipeline uses a build-specific checkout directory and temporary Rook home. Use disposable agents or your organization's secure temporary-directory cleanup policy so credentials and transcripts do not remain on a shared machine after a job. `disableConcurrentBuilds()` serializes this job, not every other job targeting the same fixtures. + +The shell script is single-quoted in Groovy so secrets are expanded by Bash, not interpolated into Groovy strings. Shell tracing is disabled. Jenkins masking is a safeguard, not permission to print secrets. + +## Run and Inspect the Build + +Run **Build Now** for the trusted job. A successful build requires the complete selected suite to satisfy the strict gate. A failed shell step leaves the build failed while `post { always { … } }` archives results. + +Open the build's **Artifacts** and download `rook-results/run.json`, `report.json`, and `evidence.tar.gz` when present under the build-specific directory. Configure Jenkins build/artifact retention and access control for your team's evidence policy. + +Rook's JSON report is not JUnit XML. This example uses `archiveArtifacts`, not a `junit` publisher. If you build a JUnit adapter later, preserve Unable to Verify and incomplete outcomes instead of reporting them as passed tests. + +## Troubleshooting Jenkins + +- **Job waits for an executor:** provision a node matching `rook-linux`. +- **Credential binding fails:** verify the credential IDs, Secret text type, and folder/job access. +- **Interpreter or jq missing:** install the prerequisites in the agent image, not on the Jenkins controller. +- **Job cannot reach the target:** check networking from the build agent. +- **Stale output directory:** use a new build/checkout; do not reuse a prior successful report. +- **Artifacts absent:** inspect checkout, installation, authentication, and sync before assuming a test ran. + +See [Jenkins Pipeline and credential handling](https://www.jenkins.io/doc/book/pipeline/jenkinsfile/) for the platform behavior used here. + +## Review Results Locally and on the Web + +The gate prints the run ID and saves `run.json`, `report.json`, and, when a run directory exists, `evidence.tar.gz`. Keep failure output too. The archive contains this agent's run folders, not `ROOK_HOME`; a clean checkout avoids including earlier runs. Set retention and access controls because target responses and evidence can contain sensitive data. + +To investigate locally, restore the reviewed checkout and extract the **trusted** evidence archive at its repository root, preserving the `.testmuai/rook/projects/…` hierarchy. Select the matching project and agent, then run `rook report --json` or `rook ui --local`. Inspect a run's scenario → criteria → files. + +Local Rook files panel illustrating retained request, response, hook, snapshot, and verdict evidence + +For uploaded results, open [Rook projects](https://rook.lambdatest.com/projects) → project → agent → run. Match the run ID before reviewing **Response**, **Verdict**, and **Artefacts**. A hosted record is useful for sharing, but the pipeline's validated report determines the gate. + +Hosted Rook run review illustrating completion, profile, and scenario outcomes + +These are existing smoke-test result captures, not screenshots of this CI integration executing. A local URL on a CI runner is not accessible to teammates; do not expose the local UI publicly. If a hosted upload is missing, retain the evidence and investigate [result synchronization](/support/docs/rook-web-ui/#troubleshooting) rather than rerunning paid tests blindly. + +## Related Guides + +- [CI/CD policy and common setup](/support/docs/agent-assurance-ci-cd/) +- [Profiles, hooks, and phases](/support/docs/rook-profiles-and-hooks/) +- [Public Rook headless contract](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/references/headless-contract.md) + +These are reference integrations for your reviewed environment. Syntax and gate behavior can be validated locally, but a successful deployment/build must be demonstrated in your own configured platform. diff --git a/docs/rook-opencode.md b/docs/rook-opencode.md new file mode 100644 index 000000000..afecdc0d3 --- /dev/null +++ b/docs/rook-opencode.md @@ -0,0 +1,129 @@ +--- +id: rook-opencode +title: Use Rook with OpenCode +sidebar_label: OpenCode +description: Set up the public Rook skill in OpenCode, verify discovery, test an AI agent safely, and review local and hosted evidence. +toc_max_heading_level: 2 +hide_title: false +slug: rook-opencode/ +--- + +# Use Rook with OpenCode + +Use OpenCode to operate Rook from your agent repository: inspect the implementation, prepare a test profile, generate scenarios, and explain recorded results. The coding assistant runs Rook CLI; installing a skill does not turn Rook into an MCP server or automatically test the assistant itself. + +## Before You Start + +- Install and sign in to OpenCode; its account or model-provider access is separate from your Rook account. +- [Install the public Rook CLI](/support/docs/rook-installation/) and run `rook --version`, `rook doctor`, then `rook login` if needed. +- Open a repository containing an AI agent you own and a safe test environment. If this is your first run, complete the [sample quickstart](/support/docs/agent-assurance-quickstart/) first. +- Use Git and Bash for the project-skill example below. On Windows, use a compatible shell such as WSL. Keep target credentials in Rook's environment store or approved environment variables, never in the skill or chat. + +## Install the Rook Skill + +This client supports the Agent Skills format, but it is **not a named target of `@testmuai/rook-skill`**. Use its project skill directory with the unmodified bundle from the [public Rook repository](https://github.com/LambdaTest/rook/tree/main/skill-installer/skills). This is a standards-based integration, not a separate Rook plugin. + +Run this Bash example from your agent repository. It copies only the skill bundle from a pinned public source revision, and stops if a destination already exists: + +```bash +( + set -eu + rook_skill_checkout=$(mktemp -d) + git clone --filter=blob:none https://github.com/LambdaTest/rook.git "$rook_skill_checkout/rook" + git -C "$rook_skill_checkout/rook" checkout --detach f199cb99c589b565e48a3a29781475daee907533 + if test -e .opencode/skills/rook; then + echo "An existing rook skill needs review; nothing was overwritten." >&2 + exit 1 + fi + mkdir -p .opencode/skills + cp -R "$rook_skill_checkout/rook/skill-installer/skills" .opencode/skills/rook +) +``` + +Keep `.opencode/skills/rook/SKILL.md` and the entire `references/` directory together. Review the files before committing the project skill for teammates. The temporary checkout contains public source only and can be removed after review. + +For an update, review a newer public revision, back up the existing skill, and replace the **whole bundle** in a reviewed change. The Rook npm installer does not manage this manual copy. To stop using it, remove only this project's `.opencode/skills/rook` after preserving custom changes. + +## Confirm the Client Loaded It + +From the agent repository, launch the client: + +```bash +opencode +``` + +Start OpenCode in this repository and ask it to load `rook` with its skill tool. Confirm the loaded source is `.opencode/skills/rook/SKILL.md`, not just an assistant response claiming familiarity with Rook. + +OpenCode discovers this bundle through its native `skill` tool. If `rook` is not offered, check that skill permissions do not deny it and that the skill tool is enabled for the current agent. Discovery alone does not grant permission to execute the test. + +## Test Your Agent Through a Prompt + +Start with an inspection-only request: + +```text +Load the rook skill and inspect this repository. Explain the next safe setup step. Do not run paid commands or invoke the target yet. +``` + +A useful response identifies the selected project/agent, available profiles, missing setup, and the next command. It should not claim that tests ran merely because it read the skill. + +Once you have checked the target and approved the cost, give a bounded testing request: + +```text +Use Rook to test the refund agent in this repository against its refund policy. +Use the staging profile and test fixtures only. Propose up to three scenarios. +Before invoking the target, show me the selected scenario, hooks, possible writes, +and expected credit spending, then ask for confirmation. +After approval, run one selected scenario and report its run ID, Pass, Fail, +Unable to Verify, and criterion-level evidence. Do not run paid RCA or retry +automatically. +``` + +Substitute your agent and policy. The assistant should discover/select the agent, configure or test its [profile and hooks](/support/docs/rook-profiles-and-hooks/), and review generated scenarios before execution. A normal shared run requires `rook sync`; use `--test` only when you deliberately want a local-only result. + +Profile creation, repair, testing, and scenario execution can invoke the real target. Inspect `agent.yaml` for write-capable calls before approving them. Discovery, generation, execution, and other model-backed operations can spend credits; the client's own model costs are separate. A prompt is not a hard spending cap. See [permissions and safety](/support/docs/rook-permissions-and-safety/). + +## Review the Actual Result + +Ask for the current invocation's run ID and report. A zero process exit code does not mean every scenario passed. Keep **Pass**, **Fail**, and **Unable to Verify** separate; include incomplete work and the recorded evidence for each criterion. + +From the same project workspace: + +```bash +rook report --json +rook ui --local +``` + +The local UI reads workspace evidence. For uploaded runs, use `rook ui` or open [Rook projects](https://rook.lambdatest.com/projects), then select the project → agent → run → scenario. An intentional `--test` run stays local; do not expect it on the shared timeline. + +### Local UI + +Open the scenario's criteria and files to inspect what actually happened. This existing HTTP smoke-test capture illustrates result review; it is not a screenshot of this coding client or proof that this integration was executed. + +Local Rook scenario result with criterion-level evidence for the HTTP smoke-test sample + +### Hosted Web UI + +Open the uploaded scenario's **Verdict** and **Artefacts** tabs and match its run ID to the CLI report. This is an example result view, not a client-specific test capture. + +Hosted Rook Verdict tab illustrating review of an uploaded scenario result + +See [results and evidence](/support/docs/agent-assurance-results-and-evidence/) for interpreting missing observations and [both UI walkthroughs](/support/docs/rook-web-ui/) for navigation. + +## Troubleshooting + +| Symptom | What to check | +| --- | --- | +| Skill is missing | Confirm the exact `SKILL.md` filename, its directory, and the active workspace; restart or reload the client's skills. | +| Skill loads but references fail | Copy the full `references/` folder beside `SKILL.md`; do not install just one raw Markdown file. | +| `rook` is not found | Check PATH in the client's command-execution environment, not only your regular terminal. | +| Wrong instructions are loaded | Look for another `rook` skill at project or user scope; inspect the selected path before changing files. | +| Agent cannot run a command | Review client approvals and Rook grants separately; approve the specific operation, not all future commands. | +| “Passed” but no evidence | Require the exact run ID, completion state, and saved verdict; a natural-language claim is insufficient. | + +## Next Steps and Sources + +- [Choose another coding client](/support/docs/rook-coding-agents/). +- [Automate a reviewed suite in CI/CD](/support/docs/agent-assurance-ci-cd/). +- [Public Rook skill](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/SKILL.md) and [OpenCode skill documentation](https://opencode.ai/docs/skills/). + +Installation/discovery instructions are based on the public sources above. Availability still depends on your client version, account policy, and execution environment; this guide does not claim a live end-to-end test in every client. diff --git a/docs/rook-vscode.md b/docs/rook-vscode.md new file mode 100644 index 000000000..b1404799b --- /dev/null +++ b/docs/rook-vscode.md @@ -0,0 +1,123 @@ +--- +id: rook-vscode +title: Use Rook in VS Code with GitHub Copilot +sidebar_label: VS Code +description: Set up the public Rook skill in VS Code with GitHub Copilot, verify discovery, test an AI agent safely, and review local and hosted evidence. +toc_max_heading_level: 2 +hide_title: false +slug: rook-vscode/ +--- + +# Use Rook in VS Code with GitHub Copilot + +Use VS Code with GitHub Copilot to operate Rook from your agent repository: inspect the implementation, prepare a test profile, generate scenarios, and explain recorded results. The coding assistant runs Rook CLI; installing a skill does not turn Rook into an MCP server or automatically test the assistant itself. + +## Before You Start + +- Install and sign in to VS Code with GitHub Copilot; its account or model-provider access is separate from your Rook account. +- [Install the public Rook CLI](/support/docs/rook-installation/) and run `rook --version`, `rook doctor`, then `rook login` if needed. +- Open a repository containing an AI agent you own and a safe test environment. If this is your first run, complete the [sample quickstart](/support/docs/agent-assurance-quickstart/) first. +- Use Git and Bash for the project-skill example below. On Windows, use a compatible shell such as WSL. Keep target credentials in Rook's environment store or approved environment variables, never in the skill or chat. + +## Install the Rook Skill + +This client supports the Agent Skills format, but it is **not a named target of `@testmuai/rook-skill`**. Use its project skill directory with the unmodified bundle from the [public Rook repository](https://github.com/LambdaTest/rook/tree/main/skill-installer/skills). This is a standards-based integration, not a separate Rook plugin. + +Run this Bash example from your agent repository. It copies only the skill bundle from a pinned public source revision, and stops if a destination already exists: + +```bash +( + set -eu + rook_skill_checkout=$(mktemp -d) + git clone --filter=blob:none https://github.com/LambdaTest/rook.git "$rook_skill_checkout/rook" + git -C "$rook_skill_checkout/rook" checkout --detach f199cb99c589b565e48a3a29781475daee907533 + if test -e .github/skills/rook; then + echo "An existing rook skill needs review; nothing was overwritten." >&2 + exit 1 + fi + mkdir -p .github/skills + cp -R "$rook_skill_checkout/rook/skill-installer/skills" .github/skills/rook +) +``` + +Keep `.github/skills/rook/SKILL.md` and the entire `references/` directory together. Review the files before committing the project skill for teammates. The temporary checkout contains public source only and can be removed after review. + +For an update, review a newer public revision, back up the existing skill, and replace the **whole bundle** in a reviewed change. The Rook npm installer does not manage this manual copy. To stop using it, remove only this project's `.github/skills/rook` after preserving custom changes. + +## Confirm the Client Loaded It + +Open the repository in VS Code, open Copilot Chat in Agent mode, and type `/skills` to open Configure Skills. Confirm `rook` appears, then select `/rook` in chat. + +VS Code is the editor; GitHub Copilot is the agent in this walkthrough. If you use the Codex or Claude Code extension instead, follow that client's skill setup. In WSL, Dev Containers, or Remote SSH, install Rook and the skill where the extension actually executes commands, then verify `rook --version` in that environment's terminal. + +## Test Your Agent Through a Prompt + +Start with an inspection-only request: + +```text +/rook Inspect this repository and explain the next safe setup step. Do not run paid commands or invoke the target yet. +``` + +A useful response identifies the selected project/agent, available profiles, missing setup, and the next command. It should not claim that tests ran merely because it read the skill. + +Once you have checked the target and approved the cost, give a bounded testing request: + +```text +Use Rook to test the refund agent in this repository against its refund policy. +Use the staging profile and test fixtures only. Propose up to three scenarios. +Before invoking the target, show me the selected scenario, hooks, possible writes, +and expected credit spending, then ask for confirmation. +After approval, run one selected scenario and report its run ID, Pass, Fail, +Unable to Verify, and criterion-level evidence. Do not run paid RCA or retry +automatically. +``` + +Substitute your agent and policy. The assistant should discover/select the agent, configure or test its [profile and hooks](/support/docs/rook-profiles-and-hooks/), and review generated scenarios before execution. A normal shared run requires `rook sync`; use `--test` only when you deliberately want a local-only result. + +Profile creation, repair, testing, and scenario execution can invoke the real target. Inspect `agent.yaml` for write-capable calls before approving them. Discovery, generation, execution, and other model-backed operations can spend credits; the client's own model costs are separate. A prompt is not a hard spending cap. See [permissions and safety](/support/docs/rook-permissions-and-safety/). + +## Review the Actual Result + +Ask for the current invocation's run ID and report. A zero process exit code does not mean every scenario passed. Keep **Pass**, **Fail**, and **Unable to Verify** separate; include incomplete work and the recorded evidence for each criterion. + +From the same project workspace: + +```bash +rook report --json +rook ui --local +``` + +The local UI reads workspace evidence. For uploaded runs, use `rook ui` or open [Rook projects](https://rook.lambdatest.com/projects), then select the project → agent → run → scenario. An intentional `--test` run stays local; do not expect it on the shared timeline. + +### Local UI + +Open the scenario's criteria and files to inspect what actually happened. This existing HTTP smoke-test capture illustrates result review; it is not a screenshot of this coding client or proof that this integration was executed. + +Local Rook scenario result with criterion-level evidence for the HTTP smoke-test sample + +### Hosted Web UI + +Open the uploaded scenario's **Verdict** and **Artefacts** tabs and match its run ID to the CLI report. This is an example result view, not a client-specific test capture. + +Hosted Rook Verdict tab illustrating review of an uploaded scenario result + +See [results and evidence](/support/docs/agent-assurance-results-and-evidence/) for interpreting missing observations and [both UI walkthroughs](/support/docs/rook-web-ui/) for navigation. + +## Troubleshooting + +| Symptom | What to check | +| --- | --- | +| Skill is missing | Confirm the exact `SKILL.md` filename, its directory, and the active workspace; restart or reload the client's skills. | +| Skill loads but references fail | Copy the full `references/` folder beside `SKILL.md`; do not install just one raw Markdown file. | +| `rook` is not found | Check PATH in the client's command-execution environment, not only your regular terminal. | +| Wrong instructions are loaded | Look for another `rook` skill at project or user scope; inspect the selected path before changing files. | +| Agent cannot run a command | Review client approvals and Rook grants separately; approve the specific operation, not all future commands. | +| “Passed” but no evidence | Require the exact run ID, completion state, and saved verdict; a natural-language claim is insufficient. | + +## Next Steps and Sources + +- [Choose another coding client](/support/docs/rook-coding-agents/). +- [Automate a reviewed suite in CI/CD](/support/docs/agent-assurance-ci-cd/). +- [Public Rook skill](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/SKILL.md) and [VS Code with GitHub Copilot skill documentation](https://code.visualstudio.com/docs/agent-customization/agent-skills). + +Installation/discovery instructions are based on the public sources above. Availability still depends on your client version, account policy, and execution environment; this guide does not claim a live end-to-end test in every client. diff --git a/docs/rook-windsurf.md b/docs/rook-windsurf.md new file mode 100644 index 000000000..9f991d692 --- /dev/null +++ b/docs/rook-windsurf.md @@ -0,0 +1,123 @@ +--- +id: rook-windsurf +title: Use Rook with Windsurf Cascade +sidebar_label: Windsurf +description: Set up the public Rook skill in Windsurf Cascade, verify discovery, test an AI agent safely, and review local and hosted evidence. +toc_max_heading_level: 2 +hide_title: false +slug: rook-windsurf/ +--- + +# Use Rook with Windsurf Cascade + +Use Windsurf Cascade to operate Rook from your agent repository: inspect the implementation, prepare a test profile, generate scenarios, and explain recorded results. The coding assistant runs Rook CLI; installing a skill does not turn Rook into an MCP server or automatically test the assistant itself. + +## Before You Start + +- Install and sign in to Windsurf Cascade; its account or model-provider access is separate from your Rook account. +- [Install the public Rook CLI](/support/docs/rook-installation/) and run `rook --version`, `rook doctor`, then `rook login` if needed. +- Open a repository containing an AI agent you own and a safe test environment. If this is your first run, complete the [sample quickstart](/support/docs/agent-assurance-quickstart/) first. +- Use Git and Bash for the project-skill example below. On Windows, use a compatible shell such as WSL. Keep target credentials in Rook's environment store or approved environment variables, never in the skill or chat. + +## Install the Rook Skill + +This client supports the Agent Skills format, but it is **not a named target of `@testmuai/rook-skill`**. Use its project skill directory with the unmodified bundle from the [public Rook repository](https://github.com/LambdaTest/rook/tree/main/skill-installer/skills). This is a standards-based integration, not a separate Rook plugin. + +Run this Bash example from your agent repository. It copies only the skill bundle from a pinned public source revision, and stops if a destination already exists: + +```bash +( + set -eu + rook_skill_checkout=$(mktemp -d) + git clone --filter=blob:none https://github.com/LambdaTest/rook.git "$rook_skill_checkout/rook" + git -C "$rook_skill_checkout/rook" checkout --detach f199cb99c589b565e48a3a29781475daee907533 + if test -e .windsurf/skills/rook; then + echo "An existing rook skill needs review; nothing was overwritten." >&2 + exit 1 + fi + mkdir -p .windsurf/skills + cp -R "$rook_skill_checkout/rook/skill-installer/skills" .windsurf/skills/rook +) +``` + +Keep `.windsurf/skills/rook/SKILL.md` and the entire `references/` directory together. Review the files before committing the project skill for teammates. The temporary checkout contains public source only and can be removed after review. + +For an update, review a newer public revision, back up the existing skill, and replace the **whole bundle** in a reviewed change. The Rook npm installer does not manage this manual copy. To stop using it, remove only this project's `.windsurf/skills/rook` after preserving custom changes. + +## Confirm the Client Loaded It + +Open Cascade, use its customization menu → Skills, and confirm the workspace skill `rook` appears. Type `@rook` to select it explicitly; Windsurf skills use an @-mention, not the workflow slash-command syntax. + +This page covers Cascade's workspace skills. A global installation uses `~/.codeium/windsurf/skills/rook`, not `~/.windsurf/skills`. Windsurf's documentation may redirect to the current Devin Desktop documentation; follow the Cascade instructions rather than assuming every agent in that editor uses the same configuration. + +## Test Your Agent Through a Prompt + +Start with an inspection-only request: + +```text +@rook Inspect this repository and explain the next safe setup step. Do not run paid commands or invoke the target yet. +``` + +A useful response identifies the selected project/agent, available profiles, missing setup, and the next command. It should not claim that tests ran merely because it read the skill. + +Once you have checked the target and approved the cost, give a bounded testing request: + +```text +Use Rook to test the refund agent in this repository against its refund policy. +Use the staging profile and test fixtures only. Propose up to three scenarios. +Before invoking the target, show me the selected scenario, hooks, possible writes, +and expected credit spending, then ask for confirmation. +After approval, run one selected scenario and report its run ID, Pass, Fail, +Unable to Verify, and criterion-level evidence. Do not run paid RCA or retry +automatically. +``` + +Substitute your agent and policy. The assistant should discover/select the agent, configure or test its [profile and hooks](/support/docs/rook-profiles-and-hooks/), and review generated scenarios before execution. A normal shared run requires `rook sync`; use `--test` only when you deliberately want a local-only result. + +Profile creation, repair, testing, and scenario execution can invoke the real target. Inspect `agent.yaml` for write-capable calls before approving them. Discovery, generation, execution, and other model-backed operations can spend credits; the client's own model costs are separate. A prompt is not a hard spending cap. See [permissions and safety](/support/docs/rook-permissions-and-safety/). + +## Review the Actual Result + +Ask for the current invocation's run ID and report. A zero process exit code does not mean every scenario passed. Keep **Pass**, **Fail**, and **Unable to Verify** separate; include incomplete work and the recorded evidence for each criterion. + +From the same project workspace: + +```bash +rook report --json +rook ui --local +``` + +The local UI reads workspace evidence. For uploaded runs, use `rook ui` or open [Rook projects](https://rook.lambdatest.com/projects), then select the project → agent → run → scenario. An intentional `--test` run stays local; do not expect it on the shared timeline. + +### Local UI + +Open the scenario's criteria and files to inspect what actually happened. This existing HTTP smoke-test capture illustrates result review; it is not a screenshot of this coding client or proof that this integration was executed. + +Local Rook scenario result with criterion-level evidence for the HTTP smoke-test sample + +### Hosted Web UI + +Open the uploaded scenario's **Verdict** and **Artefacts** tabs and match its run ID to the CLI report. This is an example result view, not a client-specific test capture. + +Hosted Rook Verdict tab illustrating review of an uploaded scenario result + +See [results and evidence](/support/docs/agent-assurance-results-and-evidence/) for interpreting missing observations and [both UI walkthroughs](/support/docs/rook-web-ui/) for navigation. + +## Troubleshooting + +| Symptom | What to check | +| --- | --- | +| Skill is missing | Confirm the exact `SKILL.md` filename, its directory, and the active workspace; restart or reload the client's skills. | +| Skill loads but references fail | Copy the full `references/` folder beside `SKILL.md`; do not install just one raw Markdown file. | +| `rook` is not found | Check PATH in the client's command-execution environment, not only your regular terminal. | +| Wrong instructions are loaded | Look for another `rook` skill at project or user scope; inspect the selected path before changing files. | +| Agent cannot run a command | Review client approvals and Rook grants separately; approve the specific operation, not all future commands. | +| “Passed” but no evidence | Require the exact run ID, completion state, and saved verdict; a natural-language claim is insufficient. | + +## Next Steps and Sources + +- [Choose another coding client](/support/docs/rook-coding-agents/). +- [Automate a reviewed suite in CI/CD](/support/docs/agent-assurance-ci-cd/). +- [Public Rook skill](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/SKILL.md) and [Windsurf Cascade skill documentation](https://docs.windsurf.com/windsurf/cascade/skills). + +Installation/discovery instructions are based on the public sources above. Availability still depends on your client version, account policy, and execution environment; this guide does not claim a live end-to-end test in every client. diff --git a/scripts/generate-static-md.js b/scripts/generate-static-md.js index 99386530e..f81ba9e88 100644 --- a/scripts/generate-static-md.js +++ b/scripts/generate-static-md.js @@ -140,6 +140,10 @@ function toPlainMarkdown(body) { body = body.replace(/^[ \t]*[-*+][ \t]*$/gm, ''); body = body.replace(/^[ \t]*\d+[.)][ \t]*$/gm, ''); + // Static downloads use Docusaurus' pathname:// escape to avoid route rewriting. + // Plain Markdown readers need an ordinary path, not that framework protocol. + body = body.replace(/\]\(pathname:\/\/(\/[^)]+)\)/g, ']($1)'); + // 9. Remove standalone thematic-break lines (redundant under ATX headings). body = body.replace(/^[ \t]*([-*_])(?:[ \t]*\1){2,}[ \t]*$/gm, ''); diff --git a/scripts/rook-docs.test.cjs b/scripts/rook-docs.test.cjs new file mode 100644 index 000000000..0775e6a18 --- /dev/null +++ b/scripts/rook-docs.test.cjs @@ -0,0 +1,295 @@ +// Offline checks only: no real agent calls, account credentials, or credit spending. +// Run: node --test scripts/rook-docs.test.cjs +const test = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const {spawnSync} = require('node:child_process'); +const yaml = require('js-yaml'); + +const root = path.resolve(__dirname, '..'); +const examples = path.join(root, 'static/resources/rook'); +const read = (name) => fs.readFileSync(path.join(examples, name), 'utf8'); +const clients = ['claude-code', 'codex', 'gemini-cli', 'copilot-cli', 'opencode', 'cursor-cli', 'antigravity-cli', 'vscode', 'windsurf', 'antigravity-ide']; +const pages = [...clients, 'coding-agents', 'github-actions', 'jenkins', 'argocd'].map(x => `rook-${x}`); + +test.before(() => { + // Fresh checkouts do not contain the ignored, generated Markdown exports. + const generated = spawnSync(process.execPath, [ + path.join(root, 'scripts/generate-static-md.js'), + ...pages.map(slug => path.join(root, 'docs', `${slug}.md`)), + ], {cwd: root, encoding: 'utf8'}); + assert.equal(generated.status, 0, generated.stderr); +}); + +test('all integration pages have valid frontmatter, navigation, images, and local links', () => { + const sidebar = JSON.stringify(require('../sidebars.js').AgentToAgentSidebar); + for (const slug of pages) { + const doc = fs.readFileSync(path.join(root, 'docs', `${slug}.md`), 'utf8'); + const fm = yaml.load(doc.match(/^---\n([\s\S]*?)\n---/)[1]); + assert.equal(fm.id, slug); + assert.equal(fm.slug, `${slug}/`); + assert.ok(sidebar.includes(`"${slug}"`), `${slug} is navigable`); + for (const [, target] of doc.matchAll(/\]\(\/support\/docs\/([^/#)]+)\/?(?:#[^)]*)?\)/g)) { + assert.ok(fs.existsSync(path.join(root, 'docs', `${target}.md`)), `${slug}: ${target}`); + } + for (const [, resource] of doc.matchAll(/\]\((?:pathname:\/\/)?\/support\/resources\/rook\/([^)]*)\)/g)) { + assert.ok(fs.existsSync(path.join(examples, resource)), `${slug}: ${resource}`); + } + for (const [, image] of doc.matchAll(/require\('([^']+)'\)/g)) { + assert.ok(fs.existsSync(path.resolve(root, 'docs', image)), `${slug}: ${image}`); + } + if (slug !== 'rook-coding-agents') { + assert.match(doc, /rook-local-/); + assert.match(doc, /rook-web-/); + assert.match(doc, /https:\/\/rook\.lambdatest\.com\/projects/); + } + assert.doesNotMatch(doc, /stage-rook\.lambdatestinternal/); + } +}); + +test('downloadable platform examples match the displayed code exactly', () => { + for (const [slug, file, language] of [ + ['rook-github-actions', 'github-actions.yml', 'yaml'], + ['rook-jenkins', 'Jenkinsfile', 'groovy'], + ['rook-argocd', 'Dockerfile', 'dockerfile'], + ['rook-argocd', 'argocd-job.yaml', 'yaml'], + ]) { + const doc = fs.readFileSync(path.join(root, 'docs', `${slug}.md`), 'utf8'); + assert.ok(doc.includes('```' + language + '\n' + read(file).trim() + '\n```'), file); + } +}); + +test('Bash snippets parse and Markdown exports preserve their code fences', () => { + for (const slug of pages) { + const doc = fs.readFileSync(path.join(root, 'docs', `${slug}.md`), 'utf8'); + const exported = fs.readFileSync(path.join(root, 'static/docs', `${slug}.md`), 'utf8'); + assert.doesNotMatch(exported, /\]\(pathname:\/\//, 'Markdown download links are portable'); + for (const [, resource] of doc.matchAll(/\]\(pathname:\/\/\/support\/resources\/rook\/([^)]*)\)/g)) { + assert.ok(exported.includes(`](/support/resources/rook/${resource})`)); + } + for (const [block, language, content] of doc.matchAll(/```([\w]+)\n([\s\S]*?)\n```/g)) { + assert.ok(exported.includes(block), `${slug}: ${language} block preserved in Markdown export`); + if (language === 'bash') { + const result = spawnSync('/bin/bash', ['-n'], {input: content, encoding: 'utf8'}); + assert.equal(result.status, 0, `${slug}: ${result.stderr}`); + } + } + } +}); + +test('Antigravity guides distinguish CLI and IDE discovery from Gemini installer support', () => { + const cli = fs.readFileSync(path.join(root, 'docs/rook-antigravity-cli.md'), 'utf8'); + const ide = fs.readFileSync(path.join(root, 'docs/rook-antigravity-ide.md'), 'utf8'); + const hub = fs.readFileSync(path.join(root, 'docs/rook-coding-agents.md'), 'utf8'); + for (const [slug, doc] of [['cli', cli], ['ide', ide]]) { + assert.ok(hub.includes(`/support/docs/rook-antigravity-${slug}/`)); + assert.match(doc, /not a named target of `@testmuai\/rook-skill`/); + assert.match(doc, /\.agents\/skills\/rook\/SKILL\.md/); + assert.match(doc, /https:\/\/antigravity\.google\/docs\/skills/); + assert.match(doc, /live end-to-end/); + assert.doesNotMatch(doc, /npx[^\n]*--agent antigravity/); + } + assert.match(cli, /```bash\nagy\n```/); + assert.match(cli, /enter `\/skills`/); + assert.match(cli, /\/rook Inspect/); + assert.match(cli, /rook-antigravity-ide/); + assert.match(ide, /\*\*Customizations\*\*/); + assert.match(ide, /rook-antigravity-cli/); +}); + +for (const surface of ['cli', 'ide']) { + test(`Antigravity ${surface} copy preserves the full bundle and refuses existing destinations`, t => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'rook-antigravity-doc-test-')); + t.after(() => fs.rmSync(dir, {recursive: true, force: true})); + const source = path.join(dir, 'public-source'); + const bundle = path.join(source, 'skill-installer/skills'); + fs.mkdirSync(path.join(bundle, 'references'), {recursive: true}); + fs.writeFileSync(path.join(bundle, 'SKILL.md'), 'fixture skill\n'); + fs.writeFileSync(path.join(bundle, 'references/profiles.md'), 'fixture profiles\n'); + const bin = path.join(dir, 'bin'); + fs.mkdirSync(bin); + // Only simulate fetching the public source; never call GitHub or a live assistant. + fs.writeFileSync(path.join(bin, 'git'), `#!/bin/bash +set -eu +printf '%s\\n' "$*" >> "$MOCK_GIT_CALLS" +case "$1" in + clone) + test "$2" = --filter=blob:none + test "$3" = https://github.com/LambdaTest/rook.git + cp -R "$MOCK_SKILL_SOURCE" "$4" + ;; + -C) + test "$3" = checkout + test "$4" = --detach + test "$5" = f199cb99c589b565e48a3a29781475daee907533 + ;; + *) exit 99 ;; +esac +`, {mode: 0o700}); + const workspace = path.join(dir, 'agent repository'); + const temp = path.join(dir, 'temp'); + fs.mkdirSync(workspace); + fs.mkdirSync(temp); + const calls = path.join(dir, 'git-calls.txt'); + const doc = fs.readFileSync(path.join(root, `docs/rook-antigravity-${surface}.md`), 'utf8'); + const snippet = [...doc.matchAll(/```bash\n([\s\S]*?)\n```/g)] + .map(match => match[1]).find(block => block.includes('git clone')); + assert.ok(snippet, 'manual installation example exists'); + const runCopy = () => spawnSync('/bin/bash', ['-c', snippet], { + cwd: workspace, encoding: 'utf8', timeout: 10000, + env: {...process.env, PATH: `${bin}:${process.env.PATH}`, TMPDIR: temp, + MOCK_SKILL_SOURCE: source, MOCK_GIT_CALLS: calls}, + }); + const result = runCopy(); + assert.equal(result.status, 0, result.stderr); + const destination = path.join(workspace, '.agents/skills/rook'); + assert.equal(fs.readFileSync(path.join(destination, 'SKILL.md'), 'utf8'), 'fixture skill\n'); + assert.equal(fs.readFileSync(path.join(destination, 'references/profiles.md'), 'utf8'), 'fixture profiles\n'); + const firstCalls = fs.readFileSync(calls, 'utf8'); + for (const type of ['directory', 'file', 'dangling symlink']) { + fs.rmSync(destination, {recursive: true, force: true}); + if (type === 'directory') { + fs.mkdirSync(destination); + fs.writeFileSync(path.join(destination, 'custom.md'), 'user change\n'); + } else if (type === 'file') { + fs.writeFileSync(destination, 'user change\n'); + } else { + fs.symlinkSync(path.join(dir, 'missing-skill'), destination); + } + const refused = runCopy(); + assert.notEqual(refused.status, 0, type); + assert.match(refused.stderr, /nothing was overwritten/); + assert.equal(fs.readFileSync(calls, 'utf8'), firstCalls, 'refuse before cloning again'); + if (type === 'directory') { + assert.equal(fs.readFileSync(path.join(destination, 'custom.md'), 'utf8'), 'user change\n'); + } else if (type === 'file') { + assert.equal(fs.readFileSync(destination, 'utf8'), 'user change\n'); + } else { + assert.equal(fs.readlinkSync(destination), path.join(dir, 'missing-skill')); + } + } + }); +} + +test('GitHub and Argo YAML preserve credential, artifact, and hook safeguards', () => { + const workflow = yaml.load(read('github-actions.yml')); + assert.deepEqual(Object.keys(workflow.on), ['workflow_dispatch']); + assert.equal(workflow.permissions.contents, 'read'); + const job = workflow.jobs.assurance; + assert.equal(job.environment, 'rook-assurance'); + assert.match(job.if, /refs\/heads\/main/); + assert.equal(job.steps.find(s => s.uses?.startsWith('actions/checkout')).with['persist-credentials'], false); + assert.equal(job.steps.find(s => s.uses?.startsWith('actions/upload-artifact')).if, 'always()'); + assert.equal(job.steps.find(s => s.uses?.startsWith('actions/upload-artifact')).with.path, 'rook-results/'); + const argo = yaml.load(read('argocd-job.yaml')); + assert.equal(argo.metadata.annotations['argocd.argoproj.io/hook'], 'PostSync'); + assert.equal(argo.spec.backoffLimit, 0); + assert.equal(argo.spec.template.spec.automountServiceAccountToken, false); + assert.ok(argo.spec.template.spec.volumes.some(v => v.persistentVolumeClaim)); + const environment = argo.spec.template.spec.containers[0].env; + for (const name of ['LT_USERNAME', 'LT_ACCESS_KEY', 'AGENT_TOKEN']) { + assert.ok(environment.find(v => v.name === name).valueFrom.secretKeyRef); + } +}); + +const goodReport = () => ({run_id: 'run-1', dir: '.testmuai/rook/projects/project-1/agents/agent-1/runs/run-1', report: { + run_id: 'run-1', totals: {planned: 3, executed: 3, passed: 3, failed: 0, unverifiable: 0, unjudged: 0, not_run: 0, unrunnable: 0}, clusters: [], +}}); +const goodRun = () => ({ok: true, halted: false, run_id: 'run-1', report: {totals: goodReport().report.totals}}); + +function runGate(t, {run = goodRun(), report = goodReport(), runExit = 0, env = {}, rawRun, syncExit = 0} = {}) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'rook-gate-test-')); + t.after(() => fs.rmSync(dir, {recursive: true, force: true})); + const bin = path.join(dir, 'bin'); + fs.mkdirSync(bin); + const runDir = path.join(dir, '.testmuai/rook/projects/project-1/agents/agent-1/runs/run-1'); + fs.mkdirSync(runDir, {recursive: true}); + fs.writeFileSync(path.join(runDir, 'verdict.yaml'), 'fixture: true\n'); + fs.writeFileSync(path.join(dir, 'mock-run.json'), rawRun ?? JSON.stringify(run)); + fs.writeFileSync(path.join(dir, 'mock-report.json'), JSON.stringify(report)); + fs.writeFileSync(path.join(bin, 'rook'), `#!/bin/bash +printf '%s\\n' "$*" >> "$MOCK_DIR/calls.txt" +case "$1" in + run) cat "$MOCK_DIR/mock-run.json"; exit "$MOCK_RUN_EXIT" ;; + report) cat "$MOCK_DIR/mock-report.json" ;; + sync) exit "$MOCK_SYNC_EXIT" ;; + *) exit 0 ;; +esac +`, {mode: 0o700}); + const result = spawnSync('/bin/bash', [path.join(examples, 'rook-ci.sh')], { + cwd: dir, encoding: 'utf8', timeout: 10000, + env: {PATH: `${bin}:${process.env.PATH}`, LT_USERNAME: 'test-only', LT_ACCESS_KEY: 'test-only', + ROOK_HOME: path.join(dir, 'isolated-home'), ROOK_PROJECT_ID: 'project-1', ROOK_AGENT_ID: 'agent-1', + ROOK_PROFILE: 'staging', ROOK_SCENARIO_IDS: 'SC-001,SC-004,SC-014', + MOCK_DIR: dir, MOCK_RUN_EXIT: String(runExit), MOCK_SYNC_EXIT: String(syncExit), ...env}, + }); + assert.equal(result.error, undefined); + return {...result, dir}; +} + +test('gate passes a complete suite, fetches its exact report, and archives evidence', t => { + const result = runGate(t); + assert.equal(result.status, 0, result.stderr); + assert.match(result.stdout, /Rook release gate passed/); + assert.ok(fs.existsSync(path.join(result.dir, 'rook-results/evidence.tar.gz'))); + const calls = fs.readFileSync(path.join(result.dir, 'calls.txt'), 'utf8'); + assert.match(calls, /report run-1 --json/); + assert.doesNotMatch(calls, /explore|generate|--yes|--test/); +}); + +for (const [name, change] of [ + ['failed verdict', r => {r.report.totals.failed = 1; r.report.totals.passed = 2;}], + ['unable to verify', r => {r.report.totals.unverifiable = 1; r.report.totals.passed = 2;}], + ['unjudged scenario', r => {r.report.totals.unjudged = 1; r.report.totals.passed = 2;}], + ['not run', r => {r.report.totals.not_run = 1; r.report.totals.executed = 2; r.report.totals.passed = 2;}], + ['unrunnable gap', r => {r.report.totals.unrunnable = 1;}], + ['missing counts', r => {delete r.report.totals.failed;}], + ['string counts', r => {r.report.totals.passed = '3';}], + ['wrong run ID', r => {r.run_id = 'older-run';}], + ['wrong nested run ID', r => {r.report.run_id = 'older-run';}], + ['unexpected suite size', r => {r.report.totals.planned = 2; r.report.totals.executed = 2; r.report.totals.passed = 2;}], + ['compromised evidence', r => {r.report.clusters = [{kind: 'compromised'}];}], +]) { + test(`gate blocks ${name} despite process exit zero`, t => { + const report = goodReport(); change(report); + const result = runGate(t, {report}); + assert.notEqual(result.status, 0, result.stdout); + assert.ok(fs.existsSync(path.join(result.dir, 'rook-results/evidence.tar.gz'))); + }); +} + +for (const [name, options] of [ + ['halted run', {run: {...goodRun(), halted: true}}], + ['declined run', {run: {...goodRun(), discarded: 'declined'}}], + ['refused run', {run: {...goodRun(), discarded: 'refused'}, runExit: 1}], + ['missing run ID', {run: {ok: true, halted: false, report: {}}}], + ['empty JSON', {rawRun: ''}], + ['malformed JSON', {rawRun: 'not json'}], + ['nonzero run exit', {runExit: 2}], + ['sync failure', {syncExit: 1}], + ['duplicate scenario IDs', {env: {ROOK_SCENARIO_IDS: 'SC-001,SC-001'}}], + ['missing credentials', {env: {LT_ACCESS_KEY: ''}}], +]) { + test(`gate rejects ${name}`, t => { + const result = runGate(t, options); + assert.notEqual(result.status, 0); + const calls = fs.existsSync(path.join(result.dir, 'calls.txt')) ? fs.readFileSync(path.join(result.dir, 'calls.txt'), 'utf8') : ''; + assert.doesNotMatch(calls, /report /, 'must not fall back to an old/default report'); + }); +} + +test('newline-separated grants remain literal tool rules', t => { + const result = runGate(t, {env: {ROOK_ALLOW_RULES: 'bash(npm test)\nhttp(https://test.example/api)'}}); + assert.equal(result.status, 0, result.stderr); + assert.match(fs.readFileSync(path.join(result.dir, 'calls.txt'), 'utf8'), /--allow bash\(npm test\) --allow http\(https:\/\/test.example\/api\)/); +}); + +test('reusing an output directory fails before executing a second run', t => { + const result = runGate(t); + assert.equal(result.status, 0); + const repeated = runGate(t, {env: {ROOK_RESULTS_DIR: path.join(result.dir, 'rook-results')}}); + assert.notEqual(repeated.status, 0); + assert.equal(fs.existsSync(path.join(repeated.dir, 'calls.txt')), false); +}); diff --git a/sidebars.js b/sidebars.js index 5f896a5cf..0a5421d41 100644 --- a/sidebars.js +++ b/sidebars.js @@ -450,7 +450,35 @@ module.exports = { { type: "doc", label: "Run Tests", id: "agent-assurance-run-tests" }, { type: "doc", label: "Local & Hosted UIs", id: "rook-web-ui" }, { type: "doc", label: "Verdicts & Reports", id: "agent-assurance-results-and-evidence" }, - { type: "doc", label: "CI/CD and Automation", id: "agent-assurance-ci-cd" }, + ], + }, + { + type: "category", + label: "Coding Agents & Skills", + collapsed: true, + link: { type: "doc", id: "rook-coding-agents" }, + items: [ + { type: "doc", label: "Claude Code", id: "rook-claude-code" }, + { type: "doc", label: "Codex CLI", id: "rook-codex" }, + { type: "doc", label: "Gemini CLI", id: "rook-gemini-cli" }, + { type: "doc", label: "GitHub Copilot CLI", id: "rook-copilot-cli" }, + { type: "doc", label: "OpenCode", id: "rook-opencode" }, + { type: "doc", label: "Cursor CLI", id: "rook-cursor-cli" }, + { type: "doc", label: "Antigravity CLI", id: "rook-antigravity-cli" }, + { type: "doc", label: "VS Code", id: "rook-vscode" }, + { type: "doc", label: "Windsurf", id: "rook-windsurf" }, + { type: "doc", label: "Antigravity IDE", id: "rook-antigravity-ide" }, + ], + }, + { + type: "category", + label: "CI/CD Integrations", + collapsed: true, + link: { type: "doc", id: "agent-assurance-ci-cd" }, + items: [ + { type: "doc", label: "GitHub Actions", id: "rook-github-actions" }, + { type: "doc", label: "Jenkins", id: "rook-jenkins" }, + { type: "doc", label: "Argo CD", id: "rook-argocd" }, ], }, { diff --git a/static/docs/agent-assurance-ci-cd.md b/static/docs/agent-assurance-ci-cd.md index 52e9c11a9..997a51c3e 100644 --- a/static/docs/agent-assurance-ci-cd.md +++ b/static/docs/agent-assurance-ci-cd.md @@ -6,6 +6,22 @@ Use CI only after the same agent, profile, and scenarios work locally. Commit re This recipe targets **Rook 0.1.3**. It runs an explicit suite, preserves evidence, and checks completion and verdict counts instead of interpreting a successful CLI process as a successful agent test. +## Choose Your CI/CD Platform + +| Platform | What the dedicated guide provides | +| --- | --- | +| [GitHub Actions](/support/docs/rook-github-actions/) | A protected-environment workflow, secret configuration, and evidence upload after success or failure. | +| [Jenkins](/support/docs/rook-jenkins/) | A declarative Jenkinsfile with scoped credentials, isolated Rook state, and archived results. | +| [Argo CD](/support/docs/rook-argocd/) | A PostSync Kubernetes Job, a reviewed-suite image, secret references, and persistent evidence. | + +These guides share a downloadable [reviewed-suite gate script](/support/resources/rook/rook-ci.sh), which you review and commit as `ci/rook-ci.sh` in your agent repository. It calls the CLI directly; a coding-agent skill is not required on the runner. For interactive authoring, use the [coding-agent setup guides](/support/docs/rook-coding-agents/). + +### Choose a Verdict Policy Deliberately + +The dedicated platform examples use a **strict release gate**: every explicitly selected scenario must pass, and missing evidence, Unable to Verify, incomplete work, or compromised results block the job. Blocking the job does not relabel an Unable to Verify verdict as Fail. + +The [public skill's general CI recipe](https://github.com/LambdaTest/rook/blob/main/skill-installer/skills/references/ci.md) is more permissive: it reports Unable to Verify and unrunnable gaps without failing on those outcomes alone. Choose and review the policy for your application; do not silently switch policies to get a green build. All versions must check completion, use the current run ID, preserve evidence, and reject malformed or missing result fields. + ## Prepare a Reviewed Suite 1. Select the correct environment, project, and agent. diff --git a/static/docs/agent-assurance-quickstart.md b/static/docs/agent-assurance-quickstart.md index 01bb85c79..24298aaf3 100644 --- a/static/docs/agent-assurance-quickstart.md +++ b/static/docs/agent-assurance-quickstart.md @@ -193,6 +193,8 @@ Enter /exit to leave Rook. Stop the sample server with Ctrl+C in its terminal. R ## Continue After Your First Test {#continue-after-your-first-test} +To repeat this workflow through your coding assistant, [choose a client-specific Rook skill guide](/support/docs/rook-coding-agents/). To automate the reviewed suite, use [GitHub Actions](/support/docs/rook-github-actions/), [Jenkins](/support/docs/rook-jenkins/), or [Argo CD](/support/docs/rook-argocd/). + Use `rook status` at any point to check the selected project, active agent, and local/upstream state. In the TUI, bare `/project`, `/agent`, and `/profile` open pickers; select with the arrow keys and Enter. In a shell, their bare forms list the available records. ### Ask in Plain Language diff --git a/static/docs/agent-skills.md b/static/docs/agent-skills.md index d9fcde0e9..79a4d43a0 100644 --- a/static/docs/agent-skills.md +++ b/static/docs/agent-skills.md @@ -10,6 +10,8 @@ Skills work with Claude Code, GitHub Copilot, Cursor, Gemini CLI, and any assist ## How to Install a Skill +Testing an AI agent with **Rook**? Its skill is distributed from the [public Rook repository](https://github.com/LambdaTest/rook/tree/main/skill-installer/skills), separately from the framework skills below. Follow the [Rook coding-agent setup guides](/support/docs/rook-coding-agents/) for Claude Code, Codex, Gemini CLI, GitHub Copilot CLI, OpenCode, Cursor CLI, Antigravity CLI, VS Code, Windsurf, or Antigravity IDE. The Rook skill installer is `npx @testmuai/rook-skill@latest` for Claude Code, Codex, and Gemini CLI; the other guides use manual project installation. Install Rook CLI separately. + Use the `agentskillsforall` installer to add a skill straight from the repository. No manual cloning is required. Add skills by name. Run the installer once per skill you want: diff --git a/static/docs/rook-installation.md b/static/docs/rook-installation.md index 9c0715153..af2682b29 100644 --- a/static/docs/rook-installation.md +++ b/static/docs/rook-installation.md @@ -149,6 +149,8 @@ If a public install path still fails, [open a Rook issue](https://github.com/Lam ## Next Step +If you use a coding assistant, [choose its Rook skill setup guide](/support/docs/rook-coding-agents/) after installing the CLI. Claude Code, Codex, Gemini CLI, Copilot CLI, OpenCode, Cursor CLI, Antigravity CLI, VS Code, Windsurf, and Antigravity IDE each have their own instructions. Installing the skill does not install or authenticate the CLI. + Continue with [How to Get Started With Agent Assurance](/support/docs/agent-assurance-quickstart/). ## Open the Local or Hosted UI {#select-the-web-ui-environment} diff --git a/static/resources/rook/Dockerfile b/static/resources/rook/Dockerfile new file mode 100644 index 000000000..4a8718f5e --- /dev/null +++ b/static/resources/rook/Dockerfile @@ -0,0 +1,16 @@ +FROM debian:bookworm-slim +ARG ROOK_VERSION=0.1.3 +RUN apt-get update && apt-get install -y --no-install-recommends \ + bash ca-certificates curl jq tar gzip git \ + && curl -fsSL https://raw.githubusercontent.com/LambdaTest/rook/main/install.sh \ + -o /tmp/install-rook.sh \ + && bash /tmp/install-rook.sh --version "$ROOK_VERSION" --dir /usr/local/bin \ + && useradd --create-home --uid 10001 rook +WORKDIR /workspace +# Build with the reviewed repository root as the context; never COPY personal home state. +COPY --chown=10001:10001 .testmuai/rook/ .testmuai/rook/ +COPY --chown=10001:10001 ci/rook-ci.sh ci/rook-ci.sh +# Add only reviewed hook scripts and their runtime dependencies if your profile needs them. +USER 10001 +ENV ROOK_ENV=prod ROOK_HOME=/tmp/rook-home +ENTRYPOINT ["bash", "ci/rook-ci.sh"] diff --git a/static/resources/rook/Jenkinsfile b/static/resources/rook/Jenkinsfile new file mode 100644 index 000000000..ba4131454 --- /dev/null +++ b/static/resources/rook/Jenkinsfile @@ -0,0 +1,51 @@ +pipeline { + agent { label 'rook-linux' } + options { + skipDefaultCheckout(true) + disableConcurrentBuilds() + timeout(time: 30, unit: 'MINUTES') + } + environment { + ROOK_ENV = 'prod' + ROOK_PROJECT_ID = 'REPLACE_PROJECT_ID' + ROOK_AGENT_ID = 'REPLACE_AGENT_ID' + ROOK_PROFILE = 'staging' + ROOK_SCENARIO_IDS = 'SC-001,SC-004,SC-014' + ROOK_ALLOW_RULES = '' + } + stages { + stage('Reviewed checkout and assurance') { + steps { + // A new directory per build avoids stale result files in reused workspaces. + dir("rook-job-${env.BUILD_NUMBER}") { + checkout scm + withCredentials([ + string(credentialsId: 'rook-lt-username', variable: 'LT_USERNAME'), + string(credentialsId: 'rook-lt-access-key', variable: 'LT_ACCESS_KEY'), + string(credentialsId: 'rook-agent-token', variable: 'AGENT_TOKEN') + ]) { + sh '''#!/usr/bin/env bash +set -euo pipefail +set +x +job_temp=$(mktemp -d "${TMPDIR:-/tmp}/rook-jenkins.XXXXXX") +export ROOK_HOME="$job_temp/home" +export ROOK_RUN_NAME="jenkins-$BUILD_NUMBER" +curl -fsSL https://raw.githubusercontent.com/LambdaTest/rook/main/install.sh \ + -o "$job_temp/install-rook.sh" +bash "$job_temp/install-rook.sh" --version 0.1.3 --dir "$job_temp/bin" +export PATH="$job_temp/bin:$PATH" +bash ci/rook-ci.sh +''' + } + } + } + } + } + post { + always { + dir("rook-job-${env.BUILD_NUMBER}") { + archiveArtifacts artifacts: 'rook-results/**', allowEmptyArchive: true + } + } + } +} diff --git a/static/resources/rook/argocd-job.yaml b/static/resources/rook/argocd-job.yaml new file mode 100644 index 000000000..f0a9c263b --- /dev/null +++ b/static/resources/rook/argocd-job.yaml @@ -0,0 +1,76 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: rook-assurance + annotations: + argocd.argoproj.io/hook: PostSync + argocd.argoproj.io/hook-delete-policy: BeforeHookCreation +spec: + backoffLimit: 0 + activeDeadlineSeconds: 1800 + template: + spec: + restartPolicy: Never + automountServiceAccountToken: false + securityContext: + runAsNonRoot: true + runAsUser: 10001 + runAsGroup: 10001 + fsGroup: 10001 + containers: + - name: rook + # Build this image from the reviewed suite; replace with your registry and digest. + image: registry.example.com/team/rook-assurance@sha256:REPLACE_IMAGE_DIGEST + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: [ALL] + resources: + requests: + cpu: 250m + memory: 256Mi + limits: + cpu: '1' + memory: 1Gi + env: + - name: POD_UID + valueFrom: + fieldRef: + fieldPath: metadata.uid + - name: ROOK_RESULTS_DIR + value: /evidence/$(POD_UID) + - name: ROOK_RUN_NAME + value: argocd-$(POD_UID) + - name: ROOK_PROJECT_ID + value: REPLACE_PROJECT_ID + - name: ROOK_AGENT_ID + value: REPLACE_AGENT_ID + - name: ROOK_PROFILE + value: staging + - name: ROOK_SCENARIO_IDS + value: SC-001,SC-004,SC-014 + - name: ROOK_ALLOW_RULES + value: '' + - name: LT_USERNAME + valueFrom: + secretKeyRef: + name: rook-ci-credentials + key: LT_USERNAME + - name: LT_ACCESS_KEY + valueFrom: + secretKeyRef: + name: rook-ci-credentials + key: LT_ACCESS_KEY + - name: AGENT_TOKEN + valueFrom: + secretKeyRef: + name: rook-ci-credentials + key: AGENT_TOKEN + volumeMounts: + - name: evidence + mountPath: /evidence + volumes: + - name: evidence + persistentVolumeClaim: + claimName: rook-evidence diff --git a/static/resources/rook/github-actions.yml b/static/resources/rook/github-actions.yml new file mode 100644 index 000000000..ad696ffe2 --- /dev/null +++ b/static/resources/rook/github-actions.yml @@ -0,0 +1,51 @@ +name: Rook assurance +on: + workflow_dispatch: +permissions: + contents: read +concurrency: + group: rook-assurance + cancel-in-progress: false +jobs: + assurance: + if: github.ref == 'refs/heads/main' + runs-on: ubuntu-24.04 + timeout-minutes: 30 + environment: rook-assurance + env: + ROOK_ENV: prod + ROOK_HOME: ${{ runner.temp }}/rook-home + ROOK_PROJECT_ID: ${{ vars.ROOK_PROJECT_ID }} + ROOK_AGENT_ID: ${{ vars.ROOK_AGENT_ID }} + ROOK_PROFILE: ${{ vars.ROOK_PROFILE }} + ROOK_SCENARIO_IDS: ${{ vars.ROOK_SCENARIO_IDS }} + ROOK_ALLOW_RULES: ${{ vars.ROOK_ALLOW_RULES }} + ROOK_RUN_NAME: github-${{ github.run_id }}-${{ github.run_attempt }} + steps: + - uses: actions/checkout@v7 + with: + persist-credentials: false + - name: Install pinned public Rook CLI + shell: bash + run: | + set -euo pipefail + curl -fsSL https://raw.githubusercontent.com/LambdaTest/rook/main/install.sh \ + -o "$RUNNER_TEMP/install-rook.sh" + bash "$RUNNER_TEMP/install-rook.sh" --version 0.1.3 --dir "$RUNNER_TEMP/rook-bin" + echo "$RUNNER_TEMP/rook-bin" >> "$GITHUB_PATH" + command -v jq + - name: Run the reviewed suite + shell: bash + env: + LT_USERNAME: ${{ secrets.LT_USERNAME }} + LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }} + AGENT_TOKEN: ${{ secrets.AGENT_TOKEN }} + run: bash ci/rook-ci.sh + - name: Preserve results even on failure + if: always() + uses: actions/upload-artifact@v7 + with: + name: rook-${{ github.run_id }}-${{ github.run_attempt }} + path: rook-results/ + if-no-files-found: warn + retention-days: 7 diff --git a/static/resources/rook/rook-ci.sh b/static/resources/rook/rook-ci.sh new file mode 100644 index 000000000..3aa3af52f --- /dev/null +++ b/static/resources/rook/rook-ci.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +# Reviewed-suite release gate. Requires Rook, bash, jq, and tar. +set -euo pipefail +set +x +umask 077 + +: "${LT_USERNAME:?Set LT_USERNAME through your CI secret store}" +: "${LT_ACCESS_KEY:?Set LT_ACCESS_KEY through your CI secret store}" +: "${ROOK_HOME:?Set ROOK_HOME to an isolated directory outside the checkout}" +: "${ROOK_PROJECT_ID:?Set the reviewed project ID}" +: "${ROOK_AGENT_ID:?Set the reviewed agent ID}" +: "${ROOK_PROFILE:?Set the reviewed profile name}" +: "${ROOK_SCENARIO_IDS:?Set a comma-separated list of reviewed scenario IDs}" + +for id in "$ROOK_PROJECT_ID" "$ROOK_AGENT_ID"; do + [[ "$id" =~ ^[a-zA-Z0-9_-]+$ ]] || { echo 'Invalid project/agent ID' >&2; exit 1; } +done +[[ "$ROOK_SCENARIO_IDS" =~ ^[a-zA-Z0-9_-]+(,[a-zA-Z0-9_-]+)*$ ]] || { + echo 'Scenario IDs must be comma-separated, without spaces or empty entries' >&2; exit 1; +} +expected=$(jq -en --arg ids "$ROOK_SCENARIO_IDS" ' + ($ids | split(",")) as $list | + if ($list | unique | length) == ($list | length) + then ($list | length) else error("Duplicate scenario IDs") end') +for dependency in rook jq tar; do command -v "$dependency" >/dev/null; done +agent_dir=".testmuai/rook/projects/$ROOK_PROJECT_ID/agents/$ROOK_AGENT_ID" +test -d "$agent_dir" || { echo "Missing committed agent definitions: $agent_dir" >&2; exit 1; } + +results=${ROOK_RESULTS_DIR:-rook-results} +mkdir -p "$(dirname "$results")" +# Refuse a reused output directory: old results must never pass a new job. +mkdir "$results" +preserve_evidence() { + code=$? + trap - EXIT + if test -d "$agent_dir/runs"; then + tar -czf "$results/evidence.tar.gz" "$agent_dir/runs" || code=1 + fi + exit "$code" +} +trap preserve_evidence EXIT + +rook --version +rook project use "$ROOK_PROJECT_ID" +rook agent use "$ROOK_AGENT_ID" +rook profile use "$ROOK_PROFILE" +# No discovery or generation here: sync only the reviewed checkout. +rook sync --agent "$ROOK_AGENT_ID" + +run_args=(run --only "$ROOK_SCENARIO_IDS" --profile "$ROOK_PROFILE" + --concurrency 1 --name "${ROOK_RUN_NAME:-ci-release-gate}") +while IFS= read -r rule; do + test -z "$rule" || run_args+=(--allow "$rule") +done <<< "${ROOK_ALLOW_RULES:-}" + +run_code=0 +rook "${run_args[@]}" --json > "$results/run.json" || run_code=$? +if test "$run_code" -ne 0; then + echo "Rook command failed (exit $run_code); inspect stderr and retained evidence." >&2 + exit "$run_code" +fi +jq -e '.ok == true and .halted == false and .discarded == null + and (.run_id | type == "string" and length > 0) + and (.report | type == "object")' "$results/run.json" >/dev/null +run_id=$(jq -er '.run_id' "$results/run.json") +rook report "$run_id" --json > "$results/report.json" +jq -e --arg id "$run_id" --argjson expected "$expected" ' + .run_id == $id and .report.run_id == $id + and (.dir | type == "string" and length > 0) + and (.report.totals | [.planned, .executed, .passed, .failed, + .unverifiable, .unjudged, .not_run, .unrunnable] | + all(.[]; type == "number" and . >= 0 and floor == .)) + and (.report.totals | .planned == $expected + and .planned == (.executed + .not_run) + and .executed == (.passed + .failed + .unverifiable + .unjudged)) + and (.report.clusters | type == "array") +' "$results/report.json" >/dev/null +jq -r '.report.totals | + "Pass: \(.passed) | Fail: \(.failed) | Unable to Verify: \(.unverifiable)", + "Unjudged: \(.unjudged) | Not run: \(.not_run) | Unrunnable: \(.unrunnable)"' \ + "$results/report.json" +printf 'Run ID: %s\n' "$run_id" +# Strict release policy: uncertainty is a blocked gate, not a fabricated Fail verdict. +jq -e --argjson expected "$expected" ' + (.report.totals | .executed == $expected and .passed == $expected + and .failed == 0 and .unverifiable == 0 and .unjudged == 0 + and .not_run == 0 and .unrunnable == 0) + and ([.report.clusters[] | select(.kind == "compromised")] | length == 0) +' "$results/report.json" >/dev/null +echo 'Rook release gate passed.'