@W-22167805@ add sf agent trace read command by WillieRuemmele · Pull Request #411 · salesforcecli/plugin-agent

WillieRuemmele · 2026-04-30T20:53:07Z

Work Item

@W-22167805@

Summary

Adds the sf agent trace read command for reading and analyzing trace files from an agent preview session. No org connection required.

Changes

New command: sf agent trace read
Flags: --session-id (required), --format (summary|detail|raw, default: summary), --dimension (actions|grounding|routing|errors, required with --format detail), --turn (scope to a single turn)
--format summary: per-turn table showing topic, user input, agent response, actions executed, latency, errors
--format detail: dimension-specific drill-down (actions, grounding, routing, errors)
--format raw: prints unprocessed trace JSON
--turn N: scopes any format to a single conversation turn via turn-index.json
Falls back to listSessionTraces file order when no turn index exists
Messages: messages/agent.trace.read.md
Schema and snapshot updated

Files Changed

src/commands/agent/trace/read.ts
messages/agent.trace.read.md
schemas/agent-trace-read.json
command-snapshot.json
test/commands/agent/trace/read.test.ts

Test Plan

24 unit tests passing
npm run build and npm test pass cleanly
Schema and deprecation snapshot regenerated

Dependencies

Requires agents library PR: @W-22167805@ Add trace session utilities (listSessionTraces, readSessionTrace, readTurnIndex) forcedotcom/agents#273
Stacks on top of delete PR: @W-22167805@ add sf agent trace delete command #410

…ndle

W-22203672 Adds a hidden --agent-json flag that accepts a file path to a pre-compiled AgentJSON, bypassing the compile step in ScriptAgent. Intended for internal developer testing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-authored-by: Juliet Shackell <63259011+jshackell-sfdc@users.noreply.github.com>

…horing-bundle

W-22203672 - Apply jshackell's wording suggestion for flag summary - Add dependsOn authoring-bundle to --agent-json flag - Add telemetry for AgentJsonReadError - Add --agent-json flag to interactive agent preview command - Add NUT tests for --agent-json flag Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

W-22203672 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

test: fix NUTs @W-22293348@

W-22203672 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…-start

…PI call Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…and instanceof narrowing Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The static fixture had an empty defaultAgentUser, causing bypassUser to resolve to false and the preview sessions API to reject the request. Patch it at runtime with the real agent user from the shared test session, matching what the compile-then-start path does via string replacement. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The preview sessions API rejected "AgentforceAgent" with HTTP 400 "Define a valid value for ''agent_type''". Updated to "customer" which matches the AgentType union type in @salesforce/agents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

"customer" was also rejected by the preview sessions API. The unit test for --agent-json uses "AgentforceServiceAgent", which matches the value the compile API returns for service-type agents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…agents to 1.4.0 The preview sessions API rejected all prior agentType values. The bot-meta.xml for the test project's service agent uses "EinsteinServiceAgent", which matches what the compile API returns for this agent type. Also bumps @salesforce/agents to 1.4.0. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Setting defaultAgentUser to a valid org user made bypassUser=true, which caused the preview sessions API to validate permission sets that aren't assigned, resulting in a 500. An empty defaultAgentUser keeps bypassUser=false and skips that check entirely. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

A static fixture's agentVersion is rejected by the preview sessions API with a 500. Compile the agent at test time via Agent.init+compile(), write the real compiledArtifact to a temp file, and pass that as --agent-json. This tests the flag plumbing while ensuring the body is API-compatible. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Bumps [yaml](https://github.com/eemeli/yaml) from 2.8.3 to 2.8.4. - [Release notes](https://github.com/eemeli/yaml/releases) - [Commits](eemeli/yaml@v2.8.3...v2.8.4) --- updated-dependencies: - dependency-name: yaml dependency-version: 2.8.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

…ml-2.8.4 fix(deps): bump yaml from 2.8.3 to 2.8.4

…lag-preview-start W-22203672: add hidden --agent-json flag to agent preview start

@W-22203670 feat: add --verbose and --concise flags to agent publish authoring-bundle

…o @salesforce/agents W-22203426 Removes local copies of evalNormalizer, evalFormatter, and yamlSpecTranslator now that they live in the shared library. Rewrites run-eval.ts as a thin CLI shell that imports normalizePayload, translateTestSpec, resolveAgent, executeBatches, buildResultSummary, and formatResults from @salesforce/agents. Promotes the command from state='beta'/hidden to state='ga'. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Remove state='ga' (GA commands omit state to avoid oclif help regression) - Remove unused --wait flag - Wrap parseTestSpec/translateTestSpec in try/catch for named SfError - Validate test.steps is an array before injection loop - Hoist isYamlTestSpec to avoid double-call - Extract resolveAndInjectAgent helper to stay within complexity limit - Fix test stubs to use $$.SANDBOX, remove sinon.restore(), fix describe label Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…emoval Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Wr/ngt @W-22143479@ @W-22143478@

Six NUT cases asserted ensureExitCode: 1 but the commands fail at oclif flag-parse time (Flags.file({ exists: true }), parse callbacks that throw, validate callbacks, and missing required flags), which exits with code 2. Update the assertions to match. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

The exit code for these oclif parse failures differs by sf-plugins-core version: 12.2.6 (bundled in CLI 2.135.0) emits 2, while 12.2.13 (used in plugin-agent's own CI) catches the parse error and re-emits exit 1. Use 'nonZero' so the assertion is correct in both environments. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

fix: expect oclif parse exit code 2 for invalid-input NUTs @W-22403352

…or-to-library W-22203426: feat: promote agent test run-eval to GA and delegate logic to @salesforce/agents

sreckomileta · 2026-05-08T13:02:38Z

QA Release Notes: Agent Trace Read Command

The functionality of the new sf agent trace read command was verified across all supported parameters and flags. All test cases passed successfully.

🧪 Test Scenarios

Test Case	Command	Result
Session Filter	`sf agent trace read --session-id <id>`	✅ Pass
Summary Format	`sf agent trace read --format summary`	✅ Pass
Raw Format	`sf agent trace read --format raw`	✅ Pass
Detail Format w/ routing dim	`sf agent trace read --format detail --dimension routing`	✅ Pass
Detail Format w/ errors dim	`sf agent trace read --format detail --dimension errors`	✅ Pass
Turn Filter	`sf agent trace read --turn <number>`	✅ Pass

⚠️ Observation: Misleading Error Message

During negative testing (providing an invalid --session-id), a discrepancy was found in the error suggestion:

Action: Ran sf agent trace read --session-id <invalid_id>
Actual Error Message: ...Run "sf agent trace list" to see available sessions.
Issue: The suggested command sf agent trace list is incorrect. The actual command is sf agent preview trace list.
Recommendation: Update the error message output to reference the correct command.

Final Verdict: PASS WITH OBSERVATIONS (Logic is correct, but UI/UX needs a fix).

sreckomileta

LGTM

sreckomileta · 2026-05-08T13:20:36Z

@WillieRuemmele I've noticed a small typo in the error message. When the session ID is incorrect, the command-line interface suggests running sf agent trace list, but the current command is sf agent preview trace list. Could you please verify this? Thank you!

WillieRuemmele · 2026-05-08T14:11:20Z

@sreckomileta - the command/message is correct, it's

 ../../oss/plugin-agent/bin/run.js agent trace list 
┌───────────────────┬──────────────────────────────────────┬──────────────────────────────────────┬──────────────────────────┬────────┬───────┐
│ Agent             │ Session ID                           │ Plan ID                              │ Recorded At              │ Size   │ Path  │
├───────────────────┼──────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────┼────────┼───────┤
│ Local_Info_Agent  │ 85b4d4c5-956e-4bc0-b952-8ea162ac3b22 │ 9e21ae60-98d4-41a4-b1dd-4b5a9df994a5 │ 2026-04-07T18:32:45.183Z │ 17581B │ /Use… │

…written to the project dir Without cwd, preview start/send/end wrote the session cache relative to the runner's working directory, causing trace read/delete NUTs to throw SessionNotFound when they ran with cwd: session.project.dir.

listCachedPreviewSessions only returns sessions still in the index, but agent preview end removes the entry when it cleans up the active-session marker. Both trace read and trace delete now fall back to a filesystem scan of .sfdx/agents/<agentId>/sessions/ so they work on ended sessions.

When sessions are run with --simulate-actions, the turn index exists but planId entries are null. --turn N now falls back to positional trace file order rather than throwing TurnIndexNotFound. Also relaxes the grounding NUT assertion since simulated sessions may not produce LLMExecutionStep rows with React prompt names.

sreckomileta and others added 13 commits April 27, 2026 11:48

feat: add --verbose and --concise flags to agent publish authoring-bu…

065ed7d

…ndle

chore: update command snapshot for --verbose and --concise flags

0bb20b9

Update messages/agent.preview.start.md

3348722

Co-authored-by: Juliet Shackell <63259011+jshackell-sfdc@users.noreply.github.com>

test: add NUTs for --verbose and --concise flags on agent publish aut…

28fe2f1

…horing-bundle

chore: update command-snapshot.json with agent-json flag

1b77efd

W-22203672 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

test: fix NUTs

0da5975

docs: add error codes to command help

966347d

Merge pull request #408 from salesforcecli/er/fix-nuts

9a00b01

test: fix NUTs @W-22293348@

chore: bump @salesforce/agents to 1.3.0

ae959cf

W-22203672 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chore: Merge branch 'main' into ar/W-22203672/agent-json-flag-preview…

40197ff

…-start

feat: add agent trace read command

d14d199

WillieRuemmele requested a review from a team as a code owner April 30, 2026 20:53

WillieRuemmele and others added 16 commits April 30, 2026 15:14

feat: add NUTs for agent trace read

4eca978

test: use static fixture for --agent-json NUT to avoid live compile A…

ede5495

…PI call Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

refactor: extract loadAgentJson to common, simplify NUT fixture path …

8010fee

…and instanceof narrowing Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

test: add ensureExitCode=0 to surface actual agent-json NUT failure

5dc2713

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge pull request #412 from salesforcecli/dependabot-npm_and_yarn-ya…

884b830

…ml-2.8.4 fix(deps): bump yaml from 2.8.3 to 2.8.4

chore(release): 1.34.1 [skip ci]

144ad8e

Merge pull request #405 from salesforcecli/ar/W-22203672/agent-json-f…

a809026

…lag-preview-start W-22203672: add hidden --agent-json flag to agent preview start

chore(release): 1.35.0 [skip ci]

2dc1762

Merge branch 'main' into sem/addVerboseFlag

85f14a9

npiccolo and others added 14 commits May 4, 2026 12:56

Merge pull request #404 from salesforcecli/sem/addVerboseFlag

9d78e75

@W-22203670 feat: add --verbose and --concise flags to agent publish authoring-bundle

fix(agent/test/run-eval): update command snapshot to reflect --wait r…

7f9c353

…emoval Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chore(release): 1.36.0 [skip ci]

8a5b531

Merge pull request #407 from salesforcecli/wr/ngt

9f32ecb

Wr/ngt @W-22143479@ @W-22143478@

Merge pull request #414 from salesforcecli/sh/fix-NUTs

d1c8709

fix: expect oclif parse exit code 2 for invalid-input NUTs @W-22403352

chore(release): 1.36.1 [skip ci]

76324bd

Merge branch 'main' into afr/W-22203426/eval-refactor-to-library

af9e270

chore: bump agents

4633bbd

Merge pull request #413 from salesforcecli/afr/W-22203426/eval-refact…

0b1ef4d

…or-to-library W-22203426: feat: promote agent test run-eval to GA and delegate logic to @salesforce/agents

chore(release): 1.37.0 [skip ci]

00f6055

sreckomileta self-requested a review May 8, 2026 13:03

sreckomileta approved these changes May 8, 2026

View reviewed changes

Merge branch 'wr/deleteTraces' into wr/readTraces

ca2a3bc

WillieRuemmele added 2 commits May 8, 2026 07:51

Merge branch 'main' into wr/readTraces

32a5448

chore: merge main, resolve conflict, bump agents

79ec729

WillieRuemmele added 3 commits May 8, 2026 08:35

WillieRuemmele merged commit 1011ea2 into wr/deleteTraces May 8, 2026
13 checks passed

WillieRuemmele deleted the wr/readTraces branch May 8, 2026 16:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

@W-22167805@ add sf agent trace read command#411

@W-22167805@ add sf agent trace read command#411
WillieRuemmele merged 49 commits intowr/deleteTracesfrom
wr/readTraces

WillieRuemmele commented Apr 30, 2026

Uh oh!

sreckomileta commented May 8, 2026

Uh oh!

sreckomileta left a comment

Uh oh!

sreckomileta commented May 8, 2026

Uh oh!

WillieRuemmele commented May 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

WillieRuemmele commented Apr 30, 2026

Work Item

Summary

Changes

Files Changed

Test Plan

Dependencies

Uh oh!

sreckomileta commented May 8, 2026

QA Release Notes: Agent Trace Read Command

🧪 Test Scenarios

⚠️ Observation: Misleading Error Message

Uh oh!

sreckomileta left a comment

Choose a reason for hiding this comment

Uh oh!

sreckomileta commented May 8, 2026

Uh oh!

WillieRuemmele commented May 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants