Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions docs/SKILLS.md
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,65 @@ They work well together:
hyperagent --skill web-scraper --profile web-research
```

## User Skills (Persist What You Learn)

Hyperagent ships with a curated set of *system skills* (the table above) — and
also lets you grow your own library of *user skills* during real work. When you
and the agent have just spent ten minutes figuring out how to do something
non-obvious, save the lesson so the next session starts where you left off.

### Saving a Skill

In any REPL session, run:

```
/save-skill # let the LLM pick a name
/save-skill teams-transcript-finder
```

What happens:

1. Hyperagent collects a structured summary of the session — tool calls,
MCP servers used, modules registered, recent errors.
2. That summary plus instructions is sent to the LLM as a synthetic user
turn.
3. The LLM calls the `generate_skill` tool with a proposed SKILL.md (and
optionally a companion module). You approve before anything is written.
4. The skill is persisted to `~/.hyperagent/skills/<name>/SKILL.md`.

User skills are loaded automatically on every startup. If a user skill has
the same name as a system skill, the user version wins (overrides are
flagged with `👤 (overrides built-in)` in `/skills`).

### Managing User Skills

```
/skills # list system + user skills (👤 = user)
/skills info <name> # show the SKILL.md contents
/skills edit <name> # print the path for your $EDITOR
/skills delete <name> # remove a user skill (system ones are immutable)
```

User skills live in `~/.hyperagent/skills/<name>/SKILL.md` and follow the
same format as system skills documented above. Edit them in your editor of
choice — changes apply on the next `/suggest_approach` invocation.

### When to Save vs Not Save

Save a skill when:

- The workflow took non-trivial effort to figure out and is likely to recur.
- The lesson would be lost between sessions (modules + skill capture it
together).
- A few well-chosen triggers will reliably match future prompts on the same
topic.

Don't save a skill when:

- The task was one-off (no expected recurrence).
- The lesson is generic ("use the right tool"). Skills are for *specific*
domain knowledge.

## See Also

- [PATTERNS.md](PATTERNS.md) - Code generation patterns
Expand Down
201 changes: 201 additions & 0 deletions docs/TESTING-USER-SKILLS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,201 @@
# Testing the User-Generated Skills Feature

A walkthrough for verifying the **user skills** feature end-to-end. The
feature lets a user persist what HyperAgent learned in a session as a
reusable skill at `~/.hyperagent/skills/<name>/SKILL.md`, surviving
upgrades and overriding system skills with the same name.

---

## Prerequisites

- A working HyperAgent checkout
- `just setup` already run (Rust addons built, deps installed) — see the
project [README](../README.md) and [DEVELOPMENT.md](DEVELOPMENT.md)
- A terminal where `just start` launches the agent successfully
- A working GitHub Copilot login for the agent's LLM calls

---

## 1. Smoke Test (~2 minutes)

This is the minimum bar — if this works, the feature is wired up
end-to-end.

```bash
# Use a throwaway skills dir so you don't pollute ~/.hyperagent/skills/
export HYPERAGENT_USER_SKILLS_DIR=/tmp/ha-skills-test
mkdir -p "$HYPERAGENT_USER_SKILLS_DIR"

just start
```

In the agent REPL:

```text
> /skills
```

Confirms baseline — only **system** skills should appear, none with the
👤 (user) badge.

Now do some work the agent will remember:

```text
> use the fetch plugin to grab https://example.com and tell me the title
```

Let it run to completion. Then ask the agent to save what it learned:

```text
> /save-skill fetch-page-title
```

**Expected behaviour:**

1. The agent receives a synthetic prompt summarising the session
context (tools used, MCP servers, modules registered, recent errors)
2. The LLM calls the `generate_skill(...)` tool
3. You see an interactive approval prompt showing a **summary** — the
skill name, the one-line description, a preview of the first few
triggers, the allowed-tools list, and a byte count for the guidance
body. (The full content is *not* echoed to stdout.)
4. Hit `y` to approve

Verify the file landed on disk:

```bash
cat /tmp/ha-skills-test/fetch-page-title/SKILL.md
```

You should see a valid SKILL.md with YAML frontmatter (`name`,
`description`, `triggers`, etc.) and a markdown guidance body.

If that file exists, **the feature works.** 🎉

---

## 2. Full Workout

Exercise every command path. From a fresh `just start`:

```text
> /skills # list both system + user skills
> /skills info kql-expert # show full detail for a bundled system skill
> /save-skill # no name → LLM picks one
> /skills # user skill now shows with 👤
> /skills info fetch-page-title # user skill detail
> /skills edit fetch-page-title # prints the user-skill path; open it in your editor
> exit
```

> `/skills edit <name>` does **not** spawn `$EDITOR`. It just prints
> the absolute path to the user-skill `SKILL.md` so you can open it
> in your own editor of choice. Save the file, then restart (or run
> `/suggest_approach`) and the change takes effect.

Then restart the agent and repeat the original task — the matching
`/suggest_approach` should surface the saved skill via its triggers.

---

## 3. Override Test

User skills must override system skills with the same name. Drop a user
skill that shadows an existing system one (pick any skill that `ls
skills/` shows — here we use `kql-expert`):

```bash
mkdir -p "$HYPERAGENT_USER_SKILLS_DIR/kql-expert"
cat > "$HYPERAGENT_USER_SKILLS_DIR/kql-expert/SKILL.md" << 'EOF'
---
name: kql-expert
description: My customised KQL skill
triggers: [kql, kusto, query]
allowed-tools: [execute_javascript]
---
This overrides the system version.
EOF

just start
```

In the REPL:

```text
> /skills
```

**Expected:** the `kql-expert` row appears with the **`👤 (overrides
built-in)`** badge in the list view. Running `/skills info kql-expert`
then shows the **user** description ("My customised KQL skill").

---

## 4. Negative / Boundary Tests

Validation should reject bad input cleanly without crashing the agent:

| Input | Expected outcome |
|-------|------------------|
| `/save-skill BadName` | Rejected — not kebab-case |
| `/save-skill ../escape` | Rejected — path traversal |
| `/save-skill thisnameisreallylongandshouldfailitsbeyondsixtyfourcharactersnowforsure` | Rejected — exceeds 64 chars |
| `/save-skill info` | Rejected — reserved subcommand name |
| `/save-skill fetch-page-title` (second time, fresh session) | `generate_skill` first errors with "already exists — set overwrite=true"; the LLM retries with `overwrite=true`, and you get an **"Overwrite existing user skill?"** confirmation before the file is replaced |

---

## 5. Cleanup

```bash
rm -rf /tmp/ha-skills-test
unset HYPERAGENT_USER_SKILLS_DIR
```

---

## Verification Checklist

| Symptom | Confirms |
|---------|----------|
| `generate_skill` appears in the tool log after `/save-skill` | LLM picked up the system-message guidance ✅ |
| Approval prompt shows a skill preview | Tool handler validation working ✅ |
| `.md` file lands on disk under `$HYPERAGENT_USER_SKILLS_DIR` | `writeUserSkill()` working ✅ |
| `/skills` shows the 👤 badge for the new skill | Multi-dir loader + `source` field working ✅ |
| `/skills` shows `👤 (overrides built-in)` for shadowed system skills | Name-collision detection working ✅ |
| Restarting the agent matches the skill on similar prompts | `loadSkillsFromDirs` + boot wiring working ✅ |

---

## Likely Failure Modes & Where to Look

- **`/save-skill` runs but the LLM never calls `generate_skill`** — the
synthetic prompt from `submitToLLM` may be too weak. See
[src/agent/slash-commands.ts](../src/agent/slash-commands.ts) (the
`/save-skill` handler) and
[src/agent/system-message.ts](../src/agent/system-message.ts)
("SAVING WHAT YOU LEARN" section).
- **Tool not allowed** — every new tool needs registration at THREE
points: `tools[]` array, `ALLOWED_TOOLS` in
[src/agent/tool-gating.ts](../src/agent/tool-gating.ts), and
`availableTools[]` in the session config. Triple-check.
- **File written but `/skills` doesn't list it** —
`loadSkillsFromDirs()` in
[src/agent/skill-loader.ts](../src/agent/skill-loader.ts) may not be
reading the user dir. Verify `skillDirectories` in
[src/agent/index.ts](../src/agent/index.ts) includes
`getUserSkillsDir()`.

---

## Reporting Results

If something doesn't work, please capture:

1. The full agent REPL transcript
2. Contents of `$HYPERAGENT_USER_SKILLS_DIR` after the test (`ls -laR`)
3. The agent's debug log (`~/.hyperagent/logs/debug-*.log`)
4. The output of `just check` from the same checkout

…and share with the implementer. Good hunting. 🎯
13 changes: 9 additions & 4 deletions src/agent/approach-resolver.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import type { Skill } from "./skill-loader.js";
import type { Pattern } from "./pattern-loader.js";
import { matchIntent } from "./intent-matcher.js";
import { loadSkills } from "./skill-loader.js";
import { loadSkills, loadSkillsFromDirs } from "./skill-loader.js";
import { loadPatterns } from "./pattern-loader.js";
import { loadModule, type ModuleHints } from "./module-store.js";

Expand Down Expand Up @@ -323,20 +323,25 @@ export function formatGuidance(guidance: MaterialisedGuidance): string {
*
* @param prompt - The user's prompt text
* @param preLoadedSkills - Pre-loaded skill names (from --skill flag)
* @param skillsDir - Path to skills/ directory
* @param skillsDir - Path to skills directory(ies). A single string loads
* only system skills; pass an array of `{ dir, source }` records to load
* user skills alongside system skills (user wins on name collision).
* @param patternsDir - Path to patterns/ directory
* @param debugLog - Optional debug logger
*/
export function runSuggestApproach(
prompt: string,
preLoadedSkills: string[],
skillsDir: string,
skillsDir: string | Array<{ dir: string; source: "system" | "user" }>,
patternsDir: string,
debugLog?: (msg: string) => void,
): SuggestApproachResult {
const log = debugLog ?? (() => {});

const skills = loadSkills(skillsDir);
const skills =
typeof skillsDir === "string"
? loadSkills(skillsDir)
: loadSkillsFromDirs(skillsDir);
const patterns = loadPatterns(patternsDir);

let matchedSkillNames: string[];
Expand Down
20 changes: 18 additions & 2 deletions src/agent/commands.ts
Original file line number Diff line number Diff line change
Expand Up @@ -488,12 +488,28 @@ const COMMANDS: readonly CommandEntry[] = Object.freeze([
help: "List and invoke available skills",
group: "General",
detail:
"/skills — list all available skills\n" +
"/skills — list system + user skills (user-authored marked 👤)\n" +
"/skills <name> — invoke a skill (injects domain expertise)\n" +
"/skills info <name> — show the full SKILL.md\n" +
"/skills edit <name> — print path of user skill for $EDITOR\n" +
"/skills delete <name> — remove a user skill (system ones are immutable)\n" +
"Skills are SKILL.md files in the skills/ directory.\n" +
"Invoke a skill to get specialised instructions for a task.\n" +
"User skills live in ~/.hyperagent/skills/ and override system ones.\n" +
"Example: /skills pptx-expert — expert at building PPTX presentations.",
},
{
completion: "/save-skill",
help: "Save session learnings as a reusable skill",
group: "General",
detail:
"/save-skill — capture what we learned this session as a SKILL.md\n" +
"/save-skill <name> — same, but suggest a name to the LLM\n" +
"Sends a structured summary of the session's tool activity, MCP\n" +
"servers used, modules registered, and errors hit to the LLM,\n" +
"which then calls the generate_skill tool to write SKILL.md to\n" +
"~/.hyperagent/skills/<name>/. The skill is loaded on next start\n" +
"and triggered automatically by /suggest_approach.",
},
{
completion: "/help",
help: "Show this help (or /help <topic> for details)",
Expand Down
Loading
Loading