feat(evals): add auth0 CLI B2B organization setup eval - #178
feat(evals): add auth0 CLI B2B organization setup eval#178developerkunal wants to merge 2 commits into
Conversation
Goal-only CLI eval that drives a full B2B organizations setup through the auth0 CLI: a protected API with scopes, RBAC roles, a regular-web app and an M2M app, an M2M client grant, two organizations each with an enabled connection, and an org-admin invitation. Grading combines event graders over the successful command trace with a trace-aware judge over the manifest the agent writes. Includes an optional harness.json and verify.js for runners that provision a live tenant; both are inert to the framework loader and build.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
sanchitmehtagit
left a comment
sanchitmehtagit
left a comment
There was a problem hiding this comment.
Nice eval! The live-tenant verify.js is a great addition — asserting the full B2B graph exists on the server (not just in the agent's manifest) closes a gap that the trace and file/judge graders can't cover on their own. The PROMPT/graders structure looks clean and follows the level conventions.
Just one small nit inline, and one non-blocking follow-up that's out of scope for this PR:
- The
ranCommandsInOrderprimitive this eval relies on (already onmain) passes vacuously if it's ever called with an emptystepsarray — worth a construction-time guard + test at some point, but nothing to change here.
|
|
||
| // Management API list endpoints answer either a bare array or a wrapped object | ||
| // (e.g. { clients: [...] }) depending on pagination params. Normalize both. | ||
| function asList(res, key) { |
There was a problem hiding this comment.
Nit / totally optional: asList is already nicely defensive, and since api() throws on both a non-zero exit and non-JSON output, it only ever sees valid parsed JSON here — so this is safe as-is. If you ever want to make a malformed response easier to debug, distinguishing "key present but not an array" from "resource genuinely empty" could help, but definitely not worth holding the PR for. Just flagging in case it's useful later. 🙂
Move the live-tenant declaration for the B2B org setup eval from a colocated harness.json into PROMPT.md frontmatter (provision: auth0-tenant), alongside the existing skills field. The runner reads the frontmatter directly, so harness.json is removed and verify.js is auto-detected from the eval directory.
3d6a1b2 to
56d9d7d
Compare
2 participants
Adds an
auth0_cli_b2b_org_setupeval under a newclicategory that measures how well agents configure a complete B2B (organizations) setup using only theauth0CLI, against a tenant the CLI is already logged into. It complements the SDK-integration quickstarts with a pure tenant-configuration task.The prompt is goal-only and names no command or flag, so the agent has to work out the CLI surface itself. It asks for a protected API with
read:reports/write:reports/manage:members, two RBAC roles wired to those scopes, a Regular Web App and a Machine-to-Machine app, an M2M client grant between them, two organizations each with an enabled login connection, and an org-admin invitation, then asmoke-b2b-manifest.jsoncapturing every created resource.Grading combines event graders over the agent's successful command trace (
ranCommand/ranCommandOneOf/wroteFile) with a trace-aware judge (includeCommandTrace: true) that reads the manifest and cross-checks it against the commands actually run. The client grant and org enabled-connections have no public CLI subcommand, so those steps are matched by their Management API endpoint (client-grants,enabled_connections) rather than a dedicated command. A dedicated L3 judge fails the run if any client secret leaks into the manifest or the trace. Structure mirrors the existing evals: ascending L3/L4/L5 graders and a final levelless holistic judge.The eval also ships an optional
harness.jsonandverify.jsfor a runner that provisions a live Auth0 tenant. Both are inert to the framework here (the loader reads onlyPROMPT.mdandgraders.ts, and tsc compiles onlysrc/**/*.ts), so they do not affect build, lint, or discovery.Test plan
tsc --noEmitpasses for the auth0-evals appeslintpasses on the newgraders.tsprettier --checkclean on all new filesauth0_cli_b2b_org_setupunder categorycli