fix(guardrails): make Terra trajectory gates explicit - #2219
Conversation
|
Claude finished @apetraru-uipath's task in 3m 16s —— View job PR Review: fix(guardrails): harden Codex Terra routing and validation
SummaryThis PR hardens guardrail skill guidance for Codex/Terra by: (1) reordering the SDK WebFetch to be the first external operation, (2) adding explicit callable-rule recipes for deterministic coded guardrails, (3) adding a "fail closed" policy for net-new recommendations when catalog/list data is unavailable, (4) adding a guardrail routing gate to the review skill, (5) allowing the review skill to write one report file to an explicitly requested external path, and (6) replacing regex-based checker logic with AST-based analysis for deterministic guardrail validation. Includes three new regression test files and substantial checker rewrites. Change-by-Change Review1.
|
| Area | Status | Notes |
|---|---|---|
| Frontmatter | OK | uipath-review description updated consistently with functional change |
| E2E Tests | OK | Existing task checkers updated + 3 new regression test files added |
| Skill Body | OK | Guardrail routing gate, report-writing exception, and fail-closed semantics well-structured |
| References & Assets | OK | All relative links resolve; guardrails.md and guardrails-recommend.md changes are internally consistent |
| Repo Hygiene | OK | No secrets, no cross-skill deps, no unrelated drive-by changes, docs/superpowers correctly scoped |
Issues for Manual Review
- Semantic correctness of fail-closed vs. advisory exception: The distinction between "Recommend mode must fail closed" and "Validate mode catalog/list failures are advisory" is a domain-specific design decision. Verify this matches the intended product behavior — particularly that
GuardrailCatalogUnavailableshould indeed not block fixing an existing misplaced validator. - Review skill Write permission: The
allowed-toolsnow includesWrite. While the guardrails (only write one report to an explicitly requested external path) are clearly documented, this does expand the skill's blast radius. Verify that the evaluation framework accounts for this. - AST checker Python version: The checker uses
ast.Module | ast.FunctionDefunion syntax (Python 3.10+). Verify the CI test runners use Python 3.10+.
Conclusion
This is a focused, well-structured remediation PR. The skill documentation changes are internally consistent, the test coverage is thorough, and the AST-based checker is a significant quality improvement over the previous regex approach. No critical or high-severity issues found. The medium-severity note on the Write permission is intentional and well-documented. Approve.
bda1fc8 to
7d14791
Compare
71a422f to
0725aad
Compare
7ee0240 to
59f5dc1
Compare
4a1a015 to
0a29b10
Compare
ff380e3 to
51689e2
Compare
a6b6f53 to
340aa63
Compare
| description: "Agent fetched the UiPath Python SDK guardrail docs" | ||
| tool_name: "WebFetch" | ||
| command_pattern: 'uipath\.github\.io/uipath-python/.*guardrails' | ||
| min_count: 1 |
There was a problem hiding this comment.
I removed tool_name: "WebFetch" because it made the documentation-fetch criterion Claude-specific.
Claude Sonnet fetches the SDK page with WebFetch, while Codex/Terra typically uses Bash with curl. The exact UiPath SDK URL remains mandatory through command_pattern, so the test still verifies the required documentation step—it just accepts either transport.
Without this change, Codex could fetch the correct page and perform the task correctly but still receive a false failure solely because it didn’t use Claude’s tool name. The tradeoff is that any tool matching the exact URL is accepted, which is intentional for cross-agent tests.
What changed?
deterministic guardrail before built-in catalog ranking. Quoted text and a
distinct all-caps token such as
CONFIDENTIALare explicitly treated asliterals, not broadened into an LLM-based semantic category.
resource.jsonTool name,Tool scope, the matching custom rule (
word/containsfor a literal), and ablocking action. Before validation, re-read
agent.json; if a built-in waswritten, replace it with the required custom rule.
completion gate.
resources list, both theexact-match and no-match branches must attempt
uip solution resources getbefore editing, rejecting, or responding.uip agent validateafter refresh before reporting conversational workcomplete.
still require the exact UiPath SDK guardrail URL, but accept either Claude's
WebFetchor Codex's Bash-based fetch.uipath-agentsguardrail references and three codedguardrail task definitions. No non-guardrail skill or workflow is changed.
How has this been tested?
94 passed—pytest tests/scripts -qcoder-eval planundertests/experiments/smoke.yaml.gpt-5.6-terraon the exact PR head (2f9ec36e5):6/6 passed with every weighted and individual criterion score at
1.0.
These are the six non-Maestro guardrail failures from the official
2026-07-23_04-33-01daily Codex baseline.passed in isolation
on the same head, producing
$guardrailType: "custom", Tool scope, theexact
Send message to channelname,word/contains, andvalue: "CONFIDENTIAL".tasks passed at
1.0(including escalation), and the reviewer thresholdpassed. The broad
26-task aggregate
is 23/26 because three unrelated tasks failed by timeout/max-turn exhaustion:
coded push/pull, bindings sync, and coded Integration Service.
main.git diff origin/main --checkpasses.Are there any breaking changes?