diff --git a/examples/combined-app/README.md b/examples/combined-app/README.md index 3470abd..67f48a2 100644 --- a/examples/combined-app/README.md +++ b/examples/combined-app/README.md @@ -118,6 +118,11 @@ src/ └── ... Setup, controls, permissions, elicitation ``` +## Docs + +- [`docs/architecture.md`](docs/architecture.md) — how the server, hooks and block manager fit together. +- [`docs/pi-e2e-runbook.md`](docs/pi-e2e-runbook.md) — operator procedure for verifying the Pi path end to end against a live devbox (streaming, interrupt, steer/follow-up, suspend/resume, replay). + ## License MIT — part of the [`remote-agents-sdk`](https://github.com/runloopai/remote-agents-sdk) workspace. diff --git a/examples/combined-app/docs/pi-e2e-runbook.md b/examples/combined-app/docs/pi-e2e-runbook.md new file mode 100644 index 0000000..7133786 --- /dev/null +++ b/examples/combined-app/docs/pi-e2e-runbook.md @@ -0,0 +1,203 @@ +# Pi end-to-end verification runbook + +Operator procedure for verifying the Pi protocol path end to end: blueprint → +devbox → broker (`pi_json`) → `PiAxonConnection` → combined-app UI. Run it +against a live devbox on a broker that has Pi protocol selection deployed; +`compatibility.md` and the unit suites cover nothing that happens on the wire. + +Every step lists what to do and what to expect, so a failure localises to one +hop. The instrument for the UI steps is a scripted browser (vbrowser); nothing +here is wired into CI. Recurring live coverage belongs in +[`runloopai/remote-agents-sdk-smoke-tests`](https://github.com/runloopai/remote-agents-sdk-smoke-tests). + +## 0. Preconditions + +| Requirement | Check | +|-------------|-------| +| Broker with Pi protocol selection deployed to the target environment | Ask for the deployed broker version; a broker without it rejects the mount below with an unknown-protocol error | +| `RUNLOOP_API_KEY` (and `RUNLOOP_BASE_URL` if not production) | `env \| grep RUNLOOP_` | +| `NEBIUS_API_KEY`, `NEBIUS_BASE_URL` — Runloop's dedicated GLM-5.2 endpoint | `env \| grep NEBIUS_` | +| `axon-agents` blueprint built from a `Dockerfile` that pins `@earendil-works/pi-coding-agent@0.82.1` | Step 1 | + +Credentials come from the environment or a local `.env`. Never commit them. + +## 1. Build the blueprint + +```bash +# From the monorepo root +bun install && bun run build +bun run build-blueprint +``` + +Expected: the build succeeds and the image contains Pi at +`/usr/local/bin/pi`. Confirm the pin on a scratch devbox: + +```bash +pi --version # 0.82.1 +``` + +An `axon-agents` image built before Pi was added to +[`../../blueprint/Dockerfile`](../../blueprint/Dockerfile) fails step 4 with a +missing-binary agent error. Rebuild rather than debugging the mount. + +## 2. The mount under test + +The combined-app's Pi manager creates this broker mount +([`../src/server/pi-manager.ts`](../src/server/pi-manager.ts)): + +```json +{ + "type": "broker_mount", + "axon_id": "", + "protocol": "pi_json", + "agent_binary": "/usr/local/bin/pi", + "launch_args": ["--model", "nebius/glm-5.2"] +} +``` + +`--mode rpc` and `--session-dir` are **broker-owned**: the broker appends both, +and `--session-dir` must stay on the durable state root or resume will not +survive a devbox snapshot. If you see either in `launch_args` anywhere in this +repo, that is the bug. + +The devbox also runs a launch command that writes `~/.pi/agent/models.json` +with a `nebius` provider whose `apiKey` is the literal `"$NEBIUS_API_KEY"` — +Pi interpolates the environment variable itself, so the key never lands on +disk. On a devbox, verify: + +```bash +cat ~/.pi/agent/models.json # apiKey must read "$NEBIUS_API_KEY", not a key +``` + +## 3. Start the app + +```bash +cd examples/combined-app +cp .env.example .env # then fill in RUNLOOP_API_KEY, NEBIUS_API_KEY, NEBIUS_BASE_URL +bun run dev +``` + +Expected: the server logs `NEBIUS_API_KEY: set, NEBIUS_BASE_URL: set (Pi agents +need both)`. If either says `NOT SET`, `POST /api/start` returns 401 before any +devbox is created. + +Open http://localhost:5176. + +## 4. Provision a Pi agent + +In the UI: **Agent Type → Pi**, blueprint `axon-agents`, model +`nebius/glm-5.2`, then **Start Agent**. + +Expected: + +- Progress steps: `Creating Axon channel…` → `Provisioning sandbox…` → + `Connecting to Pi…`. +- The sidebar shows a `P` badge and the agent name. +- The Timeline tab shows an `agent_started` config event (`CFG` badge). +- No `initialize` event of any kind. Pi has no handshake; `connect()` is the + whole setup. + +Failure modes: an unknown-protocol error means the broker predates Pi support +(step 0); a missing-binary agent error means a stale blueprint (step 1). + +## 5. Stream a turn + +Prompt: `List the files in the working directory, then explain what this +project is in two sentences.` + +Expected, in the Timeline tab (`PI` badge): + +1. `turn/start` published outbound, then `response prompt` with `success: true` + — Pi *accepted* the prompt. This is not turn completion. +2. `agent_start`, `turn_start`. +3. `message_start`, then a run of `message_update` events: + `thinking_delta` (thinking blocks in the chat view) and `text_delta` + (streamed assistant text). +4. `tool_execution_start` / `_update` / `_end` around the directory listing, + with the tool name in the summary. +5. `message_end`, `turn_end`, `agent_end`. +6. `agent_settled` — **the only event that ends a turn.** The composer + re-enables here. + +The distinction in 5–6 is the point of this step. An `agent_end` with +`willRetry` shown in its summary is followed by more streaming, not by the end +of the turn. A UI that finalises on `agent_end` truncates retried turns; watch +for the composer re-enabling early. + +Capture: a screenshot of the streamed turn with both a thinking block and a +text block rendered, and one of the Timeline showing `agent_end` followed by +`agent_settled`. + +## 6. Interrupt a turn + +Send a prompt that runs long (`Count slowly from 1 to 200, one number per +line.`), then hit **Cancel** mid-stream. + +Expected: a `cancel` event goes out; streaming stops; the turn finalises and +the composer re-enables. Capture a screenshot of the interrupted turn. + +## 7. Steer and follow up + +With a turn in flight, `POST /api/pi/queue`: + +```bash +curl -sX POST localhost:3003/api/pi/queue \ + -H 'content-type: application/json' \ + -d '{"agentId":"","message":"Actually, keep it to one sentence.","mode":"steer"}' +``` + +Expected: the in-flight turn changes course without a new `turn/start`, and no +second turn appears. Repeat with `"mode":"follow_up"` and expect the message to +run after the current turn settles instead. Neither mode starts a turn, so +neither goes through the broker's turn tracking. + +## 8. Session state, suspend, resume + +```bash +curl -sX POST localhost:3003/api/pi/state \ + -H 'content-type: application/json' -d '{"agentId":""}' +``` + +Expected: `sessionId` and `sessionFile` are populated (the UI shows +`Session: ` above the chat), and `model` reflects `nebius/glm-5.2`. + +The devbox is provisioned with a 60s idle-suspend lifecycle and +`resume_triggers.axon_event`. Leave the agent idle past that window, then send +a prompt referring to something established earlier in the conversation. + +Expected: the devbox resumes on the published event, the turn runs, and the +answer shows the earlier context survived. Capture a screenshot of the +post-resume turn together with the `devbox.suspended` / `devbox.resumed` system +events in the Timeline. + +## 9. Reconnect with replay + +Reload the browser tab. The client re-issues `POST /api/subscribe`, which +re-wires the connection and replays history. + +Expected: the full prior conversation renders once, with no duplicated blocks, +and `sessionId` is unchanged. + +## 10. feature-examples suite + +```bash +bun run feature-compat --agent pi +``` + +Expected: `pass` for `single-prompt`, `agent-via-blueprint` and +`session-resume-pi`. This regenerates `compatibility.md` and `llms.txt` from +live results — commit the regenerated files rather than hand-editing them, and +regenerate from a full run (`bun run feature-compat`) so other agents' rows are +not downgraded to `skip` by a filtered run. + +Any row that cannot pass must carry a written reason in the use case, following +the `elicitation-acp` convention. + +## What this runbook does not cover + +- Nothing here runs in CI, by design. Add the Pi case to + `runloopai/remote-agents-sdk-smoke-tests` for recurring coverage. +- Image attachments: the combined-app flattens Pi prompts to a single string + and drops images (see [`../src/server/routes/prompt.ts`](../src/server/routes/prompt.ts)). +- Approvals: Pi has no approval protocol, so there is no permission step to + exercise. The auto-approve toggle is inert for Pi agents. diff --git a/examples/feature-examples/src/use-cases/session-resume-pi.ts b/examples/feature-examples/src/use-cases/session-resume-pi.ts index 2db23ae..c69d967 100644 --- a/examples/feature-examples/src/use-cases/session-resume-pi.ts +++ b/examples/feature-examples/src/use-cases/session-resume-pi.ts @@ -49,6 +49,9 @@ export default { if (secondState.sessionId === firstState.sessionId) { throw new Error("new_session did not create a new session"); } + if (secondState.sessionFile === firstSessionFile) { + throw new Error("new_session reused session A's transcript file"); + } // Resume the first session by its file path. await pi.switchSession(firstSessionFile); @@ -58,6 +61,11 @@ export default { `switch_session did not restore session A (got ${resumedState.sessionId}, want ${firstState.sessionId})`, ); } + if (resumedState.sessionFile !== firstSessionFile) { + throw new Error( + `switch_session reported ${resumedState.sessionFile}, want ${firstSessionFile}`, + ); + } ctx.log(`Resumed session A: ${resumedState.sessionId}`); // Turn 2 on the resumed session: the codeword must still be in context. diff --git a/examples/feature-examples/src/use-cases/single-prompt.ts b/examples/feature-examples/src/use-cases/single-prompt.ts index f54a6b9..26cf6cc 100644 --- a/examples/feature-examples/src/use-cases/single-prompt.ts +++ b/examples/feature-examples/src/use-cases/single-prompt.ts @@ -1,7 +1,11 @@ import { isAgentTextChunk } from "@runloop/remote-agents-sdk/acp"; import { isClaudeAssistantTextEvent, isClaudeResultEvent } from "@runloop/remote-agents-sdk/claude"; import { isCodexItemCompletedEvent } from "@runloop/remote-agents-sdk/codex"; -import { isPiAssistantTextDeltaEvent } from "@runloop/remote-agents-sdk/pi"; +import { + isPiAgentEndEvent, + isPiAgentSettledEvent, + isPiAssistantTextDeltaEvent, +} from "@runloop/remote-agents-sdk/pi"; import type { UseCase } from "../types.js"; import { waitFor } from "../validator.js"; @@ -100,9 +104,15 @@ export default { ctx.log("Running Pi path..."); let responseText = ""; + let sawAgentEnd = false; + let sawSettled = false; const unsub = ctx.pi.onTimelineEvent((event) => { if (isPiAssistantTextDeltaEvent(event)) { responseText += event.data.assistantMessageEvent.delta; + } else if (isPiAgentEndEvent(event)) { + sawAgentEnd = true; + } else if (isPiAgentSettledEvent(event)) { + sawSettled = true; } }); @@ -119,6 +129,20 @@ export default { if (!responseText.trim()) throw new Error("Agent did not respond with any text"); + // The turn boundary is the thing worth asserting: receiveTurn() must have + // returned on agent_settled, never on the agent_end that precedes it. + if (!sawSettled) { + throw new Error("Turn ended without agent_settled"); + } + ctx.log(`Turn settled (agent_end seen first: ${sawAgentEnd})`); + + // Pi is idle once the turn settles, which is what makes the next prompt + // safe to send. + const state = await ctx.pi.getState(); + if (state.isStreaming) { + throw new Error("Pi still reports isStreaming after agent_settled"); + } + ctx.log("Pass: Agent responded with text"); } else { ctx.skip("No connection available");