From 422a87f3f15232d1f75023db639a9c88e61e90e9 Mon Sep 17 00:00:00 2001 From: Paulo Lacerda Date: Tue, 9 Jun 2026 10:11:32 -0300 Subject: [PATCH] docs: explain Foundry full conversation data source Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/tutorial-prompt-agent-quickstart.md | 33 +++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/docs/tutorial-prompt-agent-quickstart.md b/docs/tutorial-prompt-agent-quickstart.md index 43f17fe..14e9e51 100644 --- a/docs/tutorial-prompt-agent-quickstart.md +++ b/docs/tutorial-prompt-agent-quickstart.md @@ -862,7 +862,24 @@ Foundry portal. This preview evaluation scope evaluates a complete multi-turn conversation from start to finish, including overall conversation quality, task completion, and user satisfaction. -Use this Foundry portal path when you want to review the end-to-end +This part runs in the **Foundry portal**, not inside the AgentOps CLI. AgentOps +Accelerator still matters here: it owns the repo-controlled gate above and can +record the Foundry evaluation URL in Doctor, Cockpit, and release evidence. + +Use one of these data sources: + +| If you have... | Use this dataset source | +|---|---| +| No production conversations yet | Start with the synthetic conversation rows from `.agentops/data/travel-conversations.jsonl`. They are controlled examples for the tutorial. | +| A deployed agent with traffic | Use Foundry traces or exported conversation logs, then convert/select those conversations as the Foundry evaluation dataset. | +| A curated review set from your team | Upload that approved conversation dataset in the format the Foundry portal asks for. | + +For this tutorial, start with the local synthetic file you just created. If the +portal asks you to upload data, use `.agentops/data/travel-conversations.jsonl` +as the source content and adapt the column mapping in the wizard if prompted. +Later, replace that with real Foundry traces or approved conversation logs. + +Run the Foundry portal evaluation when you want to review the end-to-end conversation experience itself: 1. Open your Foundry project in . @@ -872,6 +889,20 @@ conversation experience itself: 5. Run the evaluation and keep the Foundry evaluation URL with the release review. +Then paste that URL into `agentops.yaml` so AgentOps can cite it as release +evidence: + +```yaml +observability: + evaluations_url: https://ai.azure.com/ +``` + +Re-run Doctor when you want the evidence pack to include that link: + +```powershell +agentops doctor --workspace . --evidence-pack +``` + Reference: [Run evaluations from the Microsoft Foundry portal](https://learn.microsoft.com/azure/foundry/how-to/evaluate-generative-ai-app#create-an-evaluation). If your Foundry project already has a real rubric evaluator, add it later as an