diff --git a/skills/agenteye-cli/SKILL.md b/skills/agenteye-cli/SKILL.md index 4f38e2d..e8cc7e0 100644 --- a/skills/agenteye-cli/SKILL.md +++ b/skills/agenteye-cli/SKILL.md @@ -1,17 +1,15 @@ --- name: agenteye-cli description: |- - Operate and inspect a production AI-agent observability deployment - via the `agenteye` CLI. + The way to answer "how are my production AI agents doing?" and to run the team's agent-observability deployment — reach for it even on casual phrasing that names no tool. - Use for: agent errors, failures, latency, token usage, slowest models, - eval/quality scores and regressions; ack, assign, and resolve alerts and - incidents; manage roles and API keys; change settings; run ClickHouse queries. + Trigger when the user wants to: + • inspect agent telemetry — did agents error/fail/go flaky; sessions, events, latency, token usage, slowest models; eval/quality scores and whether quality dropped; + • operate the deployment — ack/assign/resolve firing alerts and incidents with notes; see who has access and change roles (e.g. make someone read-only); create or scope API keys (e.g. a CI key that only pushes events); change settings; run saved or ad-hoc ClickHouse queries. - Trigger even on casual phrasing that names no tool. + Served by the `agenteye` CLI against an AgentEye platform. - NOT for: app SDK instrumentation, debugging the collector/daemon, - or unrelated dev work. + NOT for adding SDK/instrumentation to your app, debugging the collector/daemon, or unrelated dev work (why a build/CI run failed, rotating non-AgentEye secrets). --- # AgentEye CLI @@ -114,11 +112,11 @@ Pick the right group; full flags are in `references/commands.md` — read it whe you need a flag you don't already know. **Observe (read-only):** -- `events` — raw event log. `--session-id --event-type --env --agent-id --since --search --all` +- `events` — event log (light/payload-free responses by default; `--search` still scans payload server-side; `--full` or `--fields payload` returns the raw payload — keep bounded to a `--session-id`). `--session-id --event-type --env --agent-id --since --search --full --all` - `sessions` — agent runs (time/env/agent/session/status), no scores. - `evals` — evaluation results + scores; `--aggregate` for a health rollup; `--score key:min..max`. - `errors` — errored events; `--aggregate` for count / sessions / agents / last-seen. -- `list ` — **discover valid filter values first**: `envs agents event_types score_filters models hooks triggers tools error_types`. +- `list ` — **discover valid filter values first**: `envs agents event_types score_filters models hooks tools error_types`. **Manage (permission-gated, mutations):** - `keys list|show|create|update|disable|regenerate` — API keys; secret shown once. @@ -168,9 +166,16 @@ agenteye --json list agents # find valid a agenteye --json errors --since 24h --aggregate # how bad is it right now? (full-window totals) agenteye --json errors --since 24h --all --limit 1000 | jq '.errors[] | {session_id, error_type}' agenteye --json sessions --status error --since 7d --all --limit 1000 # which runs failed -agenteye --json events --session-id run-001 --all --limit 1000 # every event in one run +agenteye --json events --session-id run-001 --all --limit 1000 # a run's timeline (light: summaries, no payload) +agenteye --json events --full --session-id run-001 --all | jq '.events[].payload' # that run's RAW payloads (--full, bounded) ``` +- **Raw `payload` is opt-in** — `events`/`errors` responses are payload-free by default; add + `--full` (or `--fields payload`) to get it, and **always bound it to a `--session-id`** + (the full feed is slow/OOM-prone at scale). For one event or a precise slice, read the + column directly: `agenteye --json query run --sql "SELECT payload FROM events WHERE id = "` + (or `WHERE session_id = ''`). See `references/commands.md` → "Getting the raw payload". + - **`list ` before filtering** — don't guess an env or agent id; the discovery command tells you exactly what exists. - **`--since`** takes `24h` / `7d` / etc. diff --git a/skills/agenteye-cli/references/commands.md b/skills/agenteye-cli/references/commands.md index bf63aec..7304cb4 100644 --- a/skills/agenteye-cli/references/commands.md +++ b/skills/agenteye-cli/references/commands.md @@ -29,14 +29,14 @@ Set on the CLI, **before** the subcommand. Precedence: flag > env var > config f | `--insecure` / `--secure` | `AGENTEYE_INSECURE` | Skip / require TLS verification (for self-signed dev certs; saved at login). | | `--version` | | Print version (also `agenteye version`). | -Config dir honours `AGENTEYE_HOME`. Telemetry is on by default; disable with `AGENTEYE_ANALYTICS_DISABLED=1` or `DO_NOT_TRACK=1`. +Config dir honours `AGENTEYE_HOME`. Telemetry is currently disabled globally while its send path is made fully non-blocking. `AGENTEYE_ANALYTICS_DISABLED=1` and `DO_NOT_TRACK=1` remain supported opt-out controls for when telemetry is re-enabled. ## Shared input conventions - **`--json`** on any command → pure JSON on stdout (no Rich chrome). Mutations under `--json` auto-skip their confirm prompt. - **`--yes` / `-y`** explicitly skips a confirm prompt. (Confirms are also auto-skipped on a non-TTY — i.e. whenever Claude runs it — so always confirm with the user yourself first.) - **`--all` + `--limit`**: `--limit` (`-n`) defaults to **50**; `--all` auto-paginates (client chunks of 200) **up to `--limit`**, NOT without bound. So a bare `--all` still stops at 50 rows. For a full sweep on `events/sessions/evals/errors`, pass a high explicit cap: **`--all --limit 1000`** (or higher). To just get window totals, use `--aggregate` (covers the whole window regardless of row caps). - **`--fields a,b,c`** projects only those keys (where supported: sessions/evals, keys, query list). -- **`--since `** relative window: `24h`, `7d`, `30d`, etc. `--from`/`--to` take ISO timestamps **with `T` and a timezone** (e.g. `2026-06-01T00:00:00Z`) — space-separated or tz-less is a usage error (exit 2). +- **`--since `** relative window — one of `15m`, `1h`, `6h`, `24h`, `7d`, `all` (any other value is a usage error, exit 2). `--from`/`--to` take ISO timestamps **with `T` and a timezone** (e.g. `2026-06-01T00:00:00Z`) — space-separated or tz-less is a usage error (exit 2). - **`--file payload.json`** (or `--file -` for stdin) supplies a full JSON request body on `alerts`, `settings set`, and `users create/update` — mutually exclusive with the discrete flags. Saved-query SQL uses `--sql @file.sql`. - **Multi-value filters** are CSV → `IN (...)` (union within a filter, AND across filters): `--event-type tool_use,tool_result`. `--search` is repeated/OR (matches ANY term), payload-only. @@ -57,9 +57,27 @@ Config dir honours `AGENTEYE_HOME`. Telemetry is on by default; disable with `AG All read-only; never need confirmation. ### events -`agenteye events [filters] [--all]` — raw event log, newest first. +`agenteye events [filters] [--all]` — event log, newest first. **Default is the light, +payload-free feed** (`/api/events/summary`): rows carry `summary, is_error, error_type, +output_tokens, context_window, context_fill` (a server-computed `summary`, no raw payload). +`--session-id`, `--all`, and structured filters stay on this fast path. `--search` is the +exception: responses remain payload-free, but the server must scan `payload` to match the +free-text term, so broad searches can still be expensive. To get the raw `payload`, opt +into the **full feed** with `--full` (or `--fields payload`) — this hits the heavy +`/api/events`, which is slow at scale, so keep it bounded (pair `--full` with one +`--session-id`). e.g. `agenteye --json events --full --session-id run-1 --all | jq '.events[].payload'`. Filters: `--session-id ` `--agent-id ` `--event-type ` `--env ` `--since ` / `--from`/`--to` `--search ` (repeatable, payload OR-match). +#### Getting the raw payload +The default `events`/`errors` reads are payload-free. Only `--full` (or `--fields payload`) +returns the raw `payload`, and it hits the heavy `/api/events` feed — **always bound it** +(pair with `--session-id`); an unbounded `events --full` can time out / degrade ClickHouse at +scale. +- **A whole session:** `agenteye --json events --full --session-id --all --limit 1000 | jq '.events[].payload'` +- **A single event:** scope to its session, then pick by id — `agenteye --json events --full --session-id --all | jq '.events[] | select(.id == ) | .payload'` +- **An error's payload:** two steps — `agenteye --json errors --error-type --since 24h` (gives the error's `id` and `session_id`; `errors` is light-only, no payload), then `agenteye --json events --full --session-id --all | jq '.events[] | select(.id == ) | .payload'` +- **Precise / by id (avoids the heavy list query):** `agenteye --json query run --sql "SELECT id, event_type, payload FROM events WHERE session_id = '' ORDER BY ts"` — or `WHERE id = `. Reads `payload` directly via the read-only SQL runner (`/api/queries/run`); a bounded `WHERE` is fast. + ### sessions `agenteye sessions [filters] [--all]` — agent runs: time/env/agent/session/status (no scores). Filters: `--session-id --agent-id --env --status --since`. JSON rows still carry `scores`. @@ -68,11 +86,11 @@ Filters: `--session-id ` `--agent-id ` `--event-type ` `--env `agenteye evals --aggregate [--since 7d]` — rollup: `{total, status_counts, score_stats, timeline}` (status mix + per-metric score stats). `--score helpfulness:..0.5` = max 0.5; `helpfulness:0.8..` = min 0.8; `helpfulness:0.5..0.9` = range. ### errors -`agenteye errors [filters] [--all]` — errored events (time/event/env/agent/session/summary). Filters incl. `--error-type `. +`agenteye errors [filters] [--all]` — errored events (time/event/env/agent/session/summary), from the light payload-free feed (`/api/events/summary`); the `summary` is the server-computed field, and `--json` rows carry no payload. For a run's raw payload use `agenteye events --full --session-id `. Filters incl. `--error-type `. `agenteye errors --aggregate [--since 7d]` — `{total, sessions, agents, last_ts, bins}`. ### list -`agenteye list ` — discover valid filter values. Kinds: `envs agents event_types score_filters models hooks triggers tools error_types`. JSON `{kind, values}`. Run this before filtering by a value you're unsure of. +`agenteye list ` — discover valid filter values. Kinds: `envs agents event_types score_filters models hooks tools error_types`. JSON `{kind, values}`. Run this before filtering by a value you're unsure of. ## keys API keys; the secret is shown **once** on create/regenerate (capture it then). Referenced by **name**.