test(uipath-maestro-case): add outcome-based e2e graded on third-party effects

[song-zhao-25]

What

Adds tests/tasks/uipath-maestro-case/e2e_outcome_external/ — an e2e task that grades whether a case achieved its business objective in the systems it touches, not whether it was built a particular way.

The agent gets an approved SDD for a two-stage expense-approval case. Two things must be true afterwards:

• the approver has a notification email in their mailbox (Microsoft Outlook 365), and
• an audit ticket exists in Jira,

both carrying a per-run reference token. The harness executes the case, then queries Microsoft Graph and Atlassian to confirm both effects landed.

Why

Acceptance for this work was defined as: "it achieved the business objectives as measured in the third party systems. Not if it validated, not if it debugged, but that it accomplished what the user was after — the coding agents can do this a number of different ways."

So the weighting is deliberate:

Criterion	Weight	Gating
Email arrived (measured via Microsoft Graph)	4.0	yes
Jira issue exists (measured via Atlassian REST)	4.0	yes
No hand-made effects (direct mutating connector calls)	2.0	yes
Case reached a terminal completed status	2.0	yes
Agent did not run case debug itself	1.0	yes
Skill triggered	1.0	yes
uip maestro case validate passes	1.0	advisory (pass_threshold: 0)

Caseplan topology and stage/task shape are not graded — any design that achieves the objective passes. Completing is worth less than the effects because a case can complete while notifying nobody.

This is additive: it does not touch e2e_expense_runnable or any existing task.

No new CI secrets

The graders read each target system back through the same Integration Service connection the case uses (uip is resources run list …). The HTTP call hits the vendor API for real; auth is the tenant's existing connection.

Verification

Local coder-eval runs against alpha/codereval:

Run	Result
2026-08-07_16-05-28	FAILURE, 4/7, 0.333
2026-08-07_16-42-17	SUCCESS, 7/7, score 1.000, 816s (predates the review fixes)
2026-08-07_19-09-09	FAILURE, 4/7, 0.333 — platform stall, see below
2026-08-07_20-14-15	SUCCESS, 7/7, score 1.000, 1049s — current HEAD

Grader evidence from the passing run:

OK: case executed to finalStatus=Completed
  email: 'Expense approval required — OBT-D1592417'  received 2026-08-08T03:30:57Z
  issue: SJP-14 'Expense audit — OBT-D1592417'       created 2026-08-08T06:31:09+0300
OK: the approver was notified — message present in the mailbox.
OK: the audit record was filed — issue present in Jira.
cleanup_outcomes: delete email [Inbox] … -> Success
cleanup_outcomes: delete email [SentItems] … -> Success
cleanup_outcomes: close Jira SJP-14 -> Success

Platform caveat, stated plainly. Case execution on the eval tenant was intermittently
unreliable while developing this: 3 of 7 runs stalled after starting (instance runs its
scaffolding elements, never enters Stage 1, raises no incident), then the identical build
completed in 56 s minutes later. The stalls are not caused by the build, the skill, or this
test — the same caseplan both stalls and passes. When it happens the test reports it
accurately, which is the intended behavior, but expect occasional red runs that are
platform-caused until that is addressed.

The first run's failure was informative rather than incidental — see below.

Design notes (each one earned by a failure)

1. Per-run token. uip maestro case debug has no --input flag, so pre_run renders sdd-template.md into the sandbox with a fresh token baked in and records it in seed.json. Both target sandboxes are shared and noisy (the mailbox took unrelated mail mid-run), so probes match this run's token rather than "the newest record" — which also keeps parallel replicates independent.
2. Timeout tolerance. case debug abandons polling at a fixed 600 s while the case keeps running. In run 1 the effects landed ~7 min after the CLI gave up, which is exactly why it scored 0.333. The checker now follows the instance to a terminal state instead of failing on the timeout, and probes poll ~10 min because Graph's subject filter lags delivery.
3. Order-independent graders. Criterion order is not guaranteed, and three criteria need the case to have run, so ensure_debug_ran() executes it once behind an exclusive lock and republishes the payload. With -j > 1 it refuses to guess when several debug instances fall in the same window, rather than risk attaching to a sibling task's success.
4. Effects asserted as >= 1, not exactly once. A stray extra execution otherwise fails a case that met the objective; duplicates are reported instead.
5. Anti-gaming. Both effects are reachable straight from the shell, which would bank the outcome criteria with nothing built — hence the two command_not_executed guards. Read-only is resources run list stays allowed because the skill uses it to resolve reference fields.
6. max_iterations: 1. A retry re-executes the case and duplicates external records; it does not recover the run.

Reviewer notes / known trade-offs

• Tenant preconditions, documented in the task header: both sandbox connections healthy (Outlook dd657127…, Jira f5273a4d…) plus Jira project SJP with issue type 10506. If one disappears the graders abort with test precondition failed: … ENVIRONMENT gap rather than looking like a skill regression.
• Cleanup, precisely: mailbox messages are deleted from BOTH Inbox and Sent Items (the design sets saveToSentItems: true, so each run leaves two copies). Jira issues cannot be deleted — the sandbox account gets 403 in SJP — so post_run transitions them to Done instead. Rows persist one per run, but the project's open-issue list stays clean. Worth requesting delete permission from the sandbox owner if the row count ever matters.
• Not in the PR gate: smoke-skills.yml runs --tags smoke; this is e2e, so it runs under make e2e (sequential by default).
• A flake rate is still outstanding. HEAD has one green run; a multi-replicate study (and a model split) was deferred because the tenant's stall rate would have dominated the measurement rather than the test's own stability.
• Scope honesty: this grades two external effects on a deliberately small two-stage case. It is a genuine instance of the outcome-based bar, not full coverage of it; measuring every transport of a larger scenario externally would be a scenario redesign.

🤖 Generated with Claude Code

[github-actions]

Claude finished @song-zhao-25's task in 2m 36s —— View job

---

PR Review: test(uipath-maestro-case): add outcome-based e2e graded on third-party effects

• Read review criteria and project conventions
• Read full diff and all changed files
• Check test task structure and tags
• Check Python grader scripts for correctness and security
• Check README additions
• Post review findings

---

Summary

Adds tests/tasks/uipath-maestro-case/e2e_outcome_external/ — a new e2e test task that grades whether a case achieves its business objective in third-party systems (email via Microsoft Graph, Jira ticket via Atlassian), not whether it was built a particular way. Also adds a reusable "Outcome-based E2E tests" section to tests/README.md documenting this pattern for future test authors.

Change-by-Change Review

1. tests/tasks/uipath-maestro-case/e2e_outcome_external/outcome_external_e2e.yaml

Severity: OK

Well-structured task YAML. Tags are correct and use the closed vocabulary: [uipath-maestro-case, e2e, mode:build, lifecycle:setup, shape:multi-node, connector, feature:activities, feature:write-back] — skill tag first, all required dimensions present (skill, tier, mode:*, lifecycle:*), optional tags from the allowed set. task_id: skill-case-e2e-outcome-external follows the skill-<domain>-<capability> convention.

Design choices are sound:

• max_iterations: 1 prevents re-execution of external side effects (line 43)
• run_limits.task_timeout: 7200 is generous but justified by the measured grading duration (PR body documents 1878s for a failing run)
• Advisory pass_threshold: 0 on validate (line 118) correctly reflects "not if it validated"
• command_not_executed guards against hand-made effects (lines 150–164)
• Weight distribution (4.0 for outcomes, 2.0 for completion/anti-gaming, 1.0 for skill trigger/advisory) matches the stated philosophy
• agent.allowed_tools extends the experiment default with Agent (line 61), with a comment explaining why — good

One observation: run_limits.max_turns: 300 (line 54) is significantly higher than the experiment default of 200. The task header comment on line 48–51 explains the timeout budget but doesn't specifically justify the turn count bump. Given the comment about delegating to sub-agents (line 58–59), this is likely needed. Minor — not blocking.

2. tests/tasks/uipath-maestro-case/e2e_outcome_external/outcome_probe.py

Severity: Low

Solid shared grading module. The exclusive lock pattern (os.O_CREAT | os.O_EXCL, lines 194–195) is correct for single-execution semantics. Instance-following logic (_await_instance, lines 129–156) properly handles the measured CLI timeout gap. The multi-instance guard (lines 118–125) correctly refuses to guess when -j > 1 runs create ambiguity.

Two minor items:

• _uip_json swallows the exit code (line 74–81): when subprocess.run returns non-zero and stdout is not valid JSON, the function returns a synthetic dict with "Result": "ParseError". Callers check for "Result": "Success", so a non-JSON non-zero exit falls through to the "not success" path, which is fine — but the real exit code is lost. The err key captures stderr, which usually has the diagnostic. Acceptable for a test harness, but logging r.returncode alongside would help triage.

• Lock file is never cleaned up (line 195 creates DEBUG_LOCK but nothing removes it). Since the sandbox is ephemeral this is harmless, but if a grader crashes between acquiring the lock and writing DEBUG_RESULT, peer graders spin for PEER_WAIT_SECONDS (900s) before failing. The risk is low (crash between lines 207 and 209 is a narrow window) and the sandbox lifetime bounds the blast radius.

3. tests/tasks/uipath-maestro-case/e2e_outcome_external/seed_outcome.py

Severity: OK

Clean pre_run script. Token format OBT-<8-hex> (line 45) has 4 billion possible values — collision-safe for shared sandboxes. Template rendering is straightforward string replacement (line 47). Writes both sdd.md (for the agent) and seed.json (for the graders). Logs the occurrence count (line 53–54) — useful for debugging template edits.

4. tests/tasks/uipath-maestro-case/e2e_outcome_external/sdd-template.md

Severity: OK

Well-crafted SDD template. The saveAsDraft trap is documented twice — once in the task description (lines 88–92) and once as the query-vs-body distinction (lines 110–113). This is deliberate: the PR body's first run failure was exactly this failure mode. The SDD correctly pins connection IDs and explains why (lines 188–190). {{RUN_TOKEN}} appears in all the right places (email subject, email body, Jira summary, Jira description).

5. tests/tasks/uipath-maestro-case/e2e_outcome_external/check_case_ran.py

Severity: OK

Minimal, correct. Delegates to ensure_debug_ran(), checks for terminal-OK status. Clean.

6. tests/tasks/uipath-maestro-case/e2e_outcome_external/check_outcome_email.py

Severity: OK

Correctly implements >= 1 semantics (reports duplicates, doesn't fail on them). The failure message (line 39–40) is diagnostically clear — "a saved draft does not count."

7. tests/tasks/uipath-maestro-case/e2e_outcome_external/check_outcome_jira.py

Severity: OK

Mirror of the email checker with Jira-specific details. Same >= 1 semantics. Clean.

8. tests/tasks/uipath-maestro-case/e2e_outcome_external/cleanup_outcomes.py

Severity: OK

Best-effort cleanup. Correctly handles the asymmetry (mail deletable, Jira not) and documents it. The top-level except Exception (line 70) ensures cleanup never fails the run. Uses _items and _uip_json from the shared probe module.

One note: cleanup_outcomes.py imports _items and _uip_json (underscore-prefixed, conventionally private). Since these are used cross-module within the same task, this is pragmatic and fine.

9. tests/README.md — Outcome-based E2E tests section

Severity: OK

Additive 52-line section inserted before "Adding Tests for a New Skill." Documents the pattern with 8 non-obvious constraints, all earned by measured failures. Points to the reference implementation. Consistent with the existing README style (markdown headers, code blocks with language identifiers, numbered constraints).

What's Missing

Nothing identified. CODEOWNERS already covers /tests/tasks/uipath-maestro-case/ (line 104). No new skill folder added. No cross-skill dependencies. No secrets or personal paths. All files are reachable from the YAML.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	Test task, not a skill — no SKILL.md frontmatter
E2E Tests	OK	Proper tags, task_id convention, weighted criteria, passing run documented
Skill Body	N/A	No skill changes
References & Assets	OK	sdd-template.md is well-structured, all paths resolve
Repo Hygiene	OK	CODEOWNERS present, no secrets, no personal paths, scoped changes

Issues for Manual Review

• Sandbox connection health: The task depends on two live Integration Service connections (dd657127… for Outlook, f5273a4d… for Jira) and Jira project SJP with issue type 10506. These are documented as tenant preconditions (YAML header, lines 1–21). Verify these fixtures are stable and monitored — the graders correctly abort as ENVIRONMENT gaps, but frequent connection revocations would make this test flaky.
• Re-run claim: The PR body notes that the exact committed state has not been re-run (three non-grading changes — max_iterations, task_timeout bump, README section — were added after the passing run). The author offers to add replicates. Given these changes are demonstrably non-grading, this is low risk, but a clean re-run before merge would eliminate doubt.
• Jira issue accumulation: One issue per run accumulates permanently in SJP. The PR acknowledges this and notes it's harmless to correctness. Worth tracking whether the sandbox owner can grant delete permission.

Conclusion

Clean, well-designed PR. The outcome-based grading approach is thoughtfully implemented — the per-run token, order-independent graders with exclusive locking, generous polling, anti-gaming guards, and >= 1 assertion semantics all address real failure modes documented with evidence. The README addition extracts these lessons into a reusable pattern. The code is clear, the YAML follows conventions, and no issues require changes.

Approve — no blocking issues found.

[github-actions]

Claude finished @song-zhao-25's task in 2m 0s —— View job

---

Coder-eval task lint (advisory)

1 task YAML changed; verdict counts: 0 Critical, 0 High, 0 Medium, 0 Low, 1 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

• ✅ Author confirms passing run: "SUCCESS, 7/7, score 1.000, 816s" (run 2026-08-07_16-42-17) with grader evidence showing finalStatus=Completed, email delivered, and Jira issue created.

Per-task lint

tests/tasks/uipath-maestro-case/e2e_outcome_external/outcome_external_e2e.yaml — verdict: OK

Well-designed outcome-based e2e. Specific observations:

• Self-report anti-pattern: Not present. Grading reads real third-party systems (Graph, Jira) via checker scripts, not agent-written files.
• Prompt over-specification: Not present. Prompt states the goal ("approver is NOTIFIED", "audit record EXISTS") and explicitly leaves design open: "How you shape the case to get there is your call." Connection IDs and solution names are ground-truth anchors, not procedure leaks.
• Meaningful coverage: Strong. Two outcome probes (weight 4.0 each) check real effects in external systems with per-run token scoping. Case execution (2.0), two command_not_executed anti-gaming guards (2.0 + 1.0), skill_triggered (1.0), advisory validate (1.0, pass_threshold: 0). Good mix of "did it achieve the objective" and "did it not cheat."
• Could pass for the wrong reason: Well-defended. Per-run token prevents matching stale records; command_not_executed guards block hand-made effects; the case must actually execute to completion.
• Near-duplicate: No overlap. Nearest sibling e2e_expense_runnable grades structural topology of a 7-stage case; this task grades third-party effects of a 2-stage case with a fundamentally different grading philosophy.
• Validate-only flow tests: N/A (skill is uipath-maestro-case, not a flow skill).
• CLI verb reachability: command_not_executed patterns reference is resources run create/update/replace/delete and maestro case debug — both current verbs. run_command validate uses uip maestro case validate — current. (Note: scripts/check-cli-verbs.py was not executed due to sandbox permissions; manual inspection confirms no issues.)
• Redundant/pinned uip CLI: No sandbox.node.env_packages block — clean.
• Run-limit fields under agent: run_limits correctly at top level (lines 52–55); agent: block (lines 60–61) contains only allowed_tools — no deprecated placement.

Within-PR duplicates

• No duplicate clusters detected (only one task YAML added).

Conclusion

• ✅ All changed tasks pass the rubric. Evidence of passing run confirmed.

---