Skip to content

Sync agenteye-cli skill (agenteye@1e6ff16) #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
NiveditJain wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Sync agenteye-cli skill (agenteye@1e6ff16) #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 16 additions & 11 deletions skills/agenteye-cli/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,15 @@
---
name: agenteye-cli
description: |-
Operate and inspect a production AI-agent observability deployment
via the `agenteye` CLI.
The way to answer "how are my production AI agents doing?" and to run the team's agent-observability deployment — reach for it even on casual phrasing that names no tool.

Use for: agent errors, failures, latency, token usage, slowest models,
eval/quality scores and regressions; ack, assign, and resolve alerts and
incidents; manage roles and API keys; change settings; run ClickHouse queries.
Trigger when the user wants to:
• inspect agent telemetry — did agents error/fail/go flaky; sessions, events, latency, token usage, slowest models; eval/quality scores and whether quality dropped;
• operate the deployment — ack/assign/resolve firing alerts and incidents with notes; see who has access and change roles (e.g. make someone read-only); create or scope API keys (e.g. a CI key that only pushes events); change settings; run saved or ad-hoc ClickHouse queries.

Trigger even on casual phrasing that names no tool.
Served by the `agenteye` CLI against an AgentEye platform.

NOT for: app SDK instrumentation, debugging the collector/daemon,
or unrelated dev work.
NOT for adding SDK/instrumentation to your app, debugging the collector/daemon, or unrelated dev work (why a build/CI run failed, rotating non-AgentEye secrets).
---

# AgentEye CLI
Expand Down Expand Up @@ -114,11 +112,11 @@ Pick the right group; full flags are in `references/commands.md` — read it whe
you need a flag you don't already know.

**Observe (read-only):**
- `events` — raw event log. `--session-id --event-type --env --agent-id --since --search --all`
- `events` — event log (light/payload-free responses by default; `--search` still scans payload server-side; `--full` or `--fields payload` returns the raw payload — keep bounded to a `--session-id`). `--session-id --event-type --env --agent-id --since --search --full --all`
- `sessions` — agent runs (time/env/agent/session/status), no scores.
- `evals` — evaluation results + scores; `--aggregate` for a health rollup; `--score key:min..max`.
- `errors` — errored events; `--aggregate` for count / sessions / agents / last-seen.
- `list <kind>` — **discover valid filter values first**: `envs agents event_types score_filters models hooks triggers tools error_types`.
- `list <kind>` — **discover valid filter values first**: `envs agents event_types score_filters models hooks tools error_types`.

**Manage (permission-gated, mutations):**
- `keys list|show|create|update|disable|regenerate` — API keys; secret shown once.
Expand Down Expand Up @@ -168,9 +166,16 @@ agenteye --json list agents # find valid a
agenteye --json errors --since 24h --aggregate # how bad is it right now? (full-window totals)
agenteye --json errors --since 24h --all --limit 1000 | jq '.errors[] | {session_id, error_type}'
agenteye --json sessions --status error --since 7d --all --limit 1000 # which runs failed
agenteye --json events --session-id run-001 --all --limit 1000 # every event in one run
agenteye --json events --session-id run-001 --all --limit 1000 # a run's timeline (light: summaries, no payload)
agenteye --json events --full --session-id run-001 --all | jq '.events[].payload' # that run's RAW payloads (--full, bounded)
```

- **Raw `payload` is opt-in** — `events`/`errors` responses are payload-free by default; add
`--full` (or `--fields payload`) to get it, and **always bound it to a `--session-id`**
(the full feed is slow/OOM-prone at scale). For one event or a precise slice, read the
column directly: `agenteye --json query run --sql "SELECT payload FROM events WHERE id = <id>"`
(or `WHERE session_id = '<id>'`). See `references/commands.md` → "Getting the raw payload".

- **`list <kind>` before filtering** — don't guess an env or agent id; the
discovery command tells you exactly what exists.
- **`--since`** takes `24h` / `7d` / etc.
Expand Down
28 changes: 23 additions & 5 deletions skills/agenteye-cli/references/commands.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,14 @@ Set on the CLI, **before** the subcommand. Precedence: flag > env var > config f
| `--insecure` / `--secure` | `AGENTEYE_INSECURE` | Skip / require TLS verification (for self-signed dev certs; saved at login). |
| `--version` | | Print version (also `agenteye version`). |

Config dir honours `AGENTEYE_HOME`. Telemetry is on by default; disable with `AGENTEYE_ANALYTICS_DISABLED=1` or `DO_NOT_TRACK=1`.
Config dir honours `AGENTEYE_HOME`. Telemetry is currently disabled globally while its send path is made fully non-blocking. `AGENTEYE_ANALYTICS_DISABLED=1` and `DO_NOT_TRACK=1` remain supported opt-out controls for when telemetry is re-enabled.

## Shared input conventions
- **`--json`** on any command → pure JSON on stdout (no Rich chrome). Mutations under `--json` auto-skip their confirm prompt.
- **`--yes` / `-y`** explicitly skips a confirm prompt. (Confirms are also auto-skipped on a non-TTY — i.e. whenever Claude runs it — so always confirm with the user yourself first.)
- **`--all` + `--limit`**: `--limit` (`-n`) defaults to **50**; `--all` auto-paginates (client chunks of 200) **up to `--limit`**, NOT without bound. So a bare `--all` still stops at 50 rows. For a full sweep on `events/sessions/evals/errors`, pass a high explicit cap: **`--all --limit 1000`** (or higher). To just get window totals, use `--aggregate` (covers the whole window regardless of row caps).
- **`--fields a,b,c`** projects only those keys (where supported: sessions/evals, keys, query list).
- **`--since <window>`** relative window: `24h`, `7d`, `30d`, etc. `--from`/`--to` take ISO timestamps **with `T` and a timezone** (e.g. `2026-06-01T00:00:00Z`) — space-separated or tz-less is a usage error (exit 2).
- **`--since <window>`** relative window — one of `15m`, `1h`, `6h`, `24h`, `7d`, `all` (any other value is a usage error, exit 2). `--from`/`--to` take ISO timestamps **with `T` and a timezone** (e.g. `2026-06-01T00:00:00Z`) — space-separated or tz-less is a usage error (exit 2).
- **`--file payload.json`** (or `--file -` for stdin) supplies a full JSON request body on `alerts`, `settings set`, and `users create/update` — mutually exclusive with the discrete flags. Saved-query SQL uses `--sql @file.sql`.
- **Multi-value filters** are CSV → `IN (...)` (union within a filter, AND across filters): `--event-type tool_use,tool_result`. `--search` is repeated/OR (matches ANY term), payload-only.

Expand All @@ -57,9 +57,27 @@ Config dir honours `AGENTEYE_HOME`. Telemetry is on by default; disable with `AG
All read-only; never need confirmation.

### events
`agenteye events [filters] [--all]` — raw event log, newest first.
`agenteye events [filters] [--all]` — event log, newest first. **Default is the light,
payload-free feed** (`/api/events/summary`): rows carry `summary, is_error, error_type,
output_tokens, context_window, context_fill` (a server-computed `summary`, no raw payload).
`--session-id`, `--all`, and structured filters stay on this fast path. `--search` is the
exception: responses remain payload-free, but the server must scan `payload` to match the
free-text term, so broad searches can still be expensive. To get the raw `payload`, opt
into the **full feed** with `--full` (or `--fields payload`) — this hits the heavy
`/api/events`, which is slow at scale, so keep it bounded (pair `--full` with one
Comment on lines +65 to +67

@coderabbitai coderabbitai Bot Jul 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== files ==\n'
git ls-files 'skills/agenteye-cli/*' | sed 's#^`#-` #'

printf '\n== search flags ==\n'
rg -n --hidden --glob '!**/.git/**' --glob '!*node_modules*' --glob '!*dist*' --glob '!*build*' \
  -- '--fields|--full|--limit|events|errors|session-id|payload' skills/agenteye-cli

printf '\n== outline key docs ==\n'
ast-grep outline skills/agenteye-cli/SKILL.md || true
ast-grep outline skills/agenteye-cli/references/commands.md || true

Repository: FailproofAI/skills

Length of output: 11271

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== commands doc excerpt ==\n'
sed -n '34,45p;59,90p' skills/agenteye-cli/references/commands.md | cat -n

printf '\n== skill doc excerpt ==\n'
sed -n '108,185p' skills/agenteye-cli/SKILL.md | cat -n

printf '\n== search for command contract / parser ==\n'
rg -n --hidden --glob '!**/.git/**' --glob '!*node_modules*' --glob '!*dist*' --glob '!*build*' \
  -e 'fields payload|--fields payload|where supported: sessions/evals, keys, query list|events .*--fields|errors .*--fields|--fields .*payload' .

Repository: FailproofAI/skills

Length of output: 12441

Align --fields payload across the docs. references/commands.md scopes --fields to sessions/evals, keys, and query list, but the events/errors guidance also advertises --fields payload. Either add events to the shared --fields scope or remove that event-specific wording.

📍 Affects 2 files
  • skills/agenteye-cli/references/commands.md#L65-L67 (this comment)
  • skills/agenteye-cli/SKILL.md#L115-L115
  • skills/agenteye-cli/SKILL.md#L173-L174
  • skills/agenteye-cli/references/commands.md#L72-L73
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/agenteye-cli/references/commands.md` around lines 65 - 67, Align the
documented scope of --fields with its event/error usage: in
skills/agenteye-cli/references/commands.md lines 65-67 and 72-73 and
skills/agenteye-cli/SKILL.md lines 115 and 173-174, either include events in the
shared --fields scope or remove the event-specific --fields payload guidance,
applying the same choice consistently at every listed site.

`--session-id`). e.g. `agenteye --json events --full --session-id run-1 --all | jq '.events[].payload'`.

@coderabbitai coderabbitai Bot Jul 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Make raw-payload examples explicit about result caps. Bare --all defaults to 50 rows, and --limit 1000 is still not guaranteed to cover an unusually large session.

  • skills/agenteye-cli/references/commands.md#L68-L68: add an explicit limit or label the example as capped.
  • skills/agenteye-cli/references/commands.md#L76-L78: add limits to the generic, single-event, and error examples, and avoid calling a capped query “whole session.”
  • skills/agenteye-cli/SKILL.md#L170-L170: add --limit <N> or direct readers to the exact by-id query.
📍 Affects 2 files
  • skills/agenteye-cli/references/commands.md#L68-L68 (this comment)
  • skills/agenteye-cli/references/commands.md#L76-L78
  • skills/agenteye-cli/SKILL.md#L170-L170
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/agenteye-cli/references/commands.md` at line 68, Make all raw-payload
examples explicit about result caps: in
skills/agenteye-cli/references/commands.md lines 68-68, add an explicit --limit
or label the --all example as capped; in lines 76-78, add limits to the generic,
single-event, and error examples and avoid describing capped queries as the
“whole session”; in skills/agenteye-cli/SKILL.md lines 170-170, add --limit <N>
or direct readers to the exact by-id query.

Filters: `--session-id <id>` `--agent-id <id>` `--event-type <csv>` `--env <csv>` `--since <window>` / `--from`/`--to` `--search <term>` (repeatable, payload OR-match).

#### Getting the raw payload
The default `events`/`errors` reads are payload-free. Only `--full` (or `--fields payload`)
returns the raw `payload`, and it hits the heavy `/api/events` feed — **always bound it**
(pair with `--session-id`); an unbounded `events --full` can time out / degrade ClickHouse at
scale.
- **A whole session:** `agenteye --json events --full --session-id <SESSION_ID> --all --limit 1000 | jq '.events[].payload'`
- **A single event:** scope to its session, then pick by id — `agenteye --json events --full --session-id <SESSION_ID> --all | jq '.events[] | select(.id == <EVENT_ID>) | .payload'`
- **An error's payload:** two steps — `agenteye --json errors --error-type <T> --since 24h` (gives the error's `id` and `session_id`; `errors` is light-only, no payload), then `agenteye --json events --full --session-id <SESSION_ID> --all | jq '.events[] | select(.id == <ERROR_EVENT_ID>) | .payload'`
- **Precise / by id (avoids the heavy list query):** `agenteye --json query run --sql "SELECT id, event_type, payload FROM events WHERE session_id = '<SESSION_ID>' ORDER BY ts"` — or `WHERE id = <EVENT_ID>`. Reads `payload` directly via the read-only SQL runner (`/api/queries/run`); a bounded `WHERE` is fast.

### sessions
`agenteye sessions [filters] [--all]` — agent runs: time/env/agent/session/status (no scores). Filters: `--session-id --agent-id --env --status <error|...> --since`. JSON rows still carry `scores`.

Expand All @@ -68,11 +86,11 @@ Filters: `--session-id <id>` `--agent-id <id>` `--event-type <csv>` `--env <csv>
`agenteye evals --aggregate [--since 7d]` — rollup: `{total, status_counts, score_stats, timeline}` (status mix + per-metric score stats). `--score helpfulness:..0.5` = max 0.5; `helpfulness:0.8..` = min 0.8; `helpfulness:0.5..0.9` = range.

### errors
`agenteye errors [filters] [--all]` — errored events (time/event/env/agent/session/summary). Filters incl. `--error-type <csv>`.
`agenteye errors [filters] [--all]` — errored events (time/event/env/agent/session/summary), from the light payload-free feed (`/api/events/summary`); the `summary` is the server-computed field, and `--json` rows carry no payload. For a run's raw payload use `agenteye events --full --session-id <id>`. Filters incl. `--error-type <csv>`.
`agenteye errors --aggregate [--since 7d]` — `{total, sessions, agents, last_ts, bins}`.

### list
`agenteye list <kind>` — discover valid filter values. Kinds: `envs agents event_types score_filters models hooks triggers tools error_types`. JSON `{kind, values}`. Run this before filtering by a value you're unsure of.
`agenteye list <kind>` — discover valid filter values. Kinds: `envs agents event_types score_filters models hooks tools error_types`. JSON `{kind, values}`. Run this before filtering by a value you're unsure of.

## keys
API keys; the secret is shown **once** on create/regenerate (capture it then). Referenced by **name**.
Expand Down