feat(mfa): live-tenant MFA CLI eval + framework-injected platform context - #182
Conversation
Adds mfa_tenant_cli, a goal-only CLI eval that configures MFA on a real throwaway Auth0 tenant (requested via 'provision: auth0-tenant' in PROMPT.md frontmatter, which the eval runner reads; the framework loader ignores it). The agent enables an MFA factor and enforces MFA via guardian/policies using the pre-authenticated auth0 CLI — the live-tenant analogue of the mocked Guardian scenario, exercising the real Management API instead of a mock. The eval writes no files, so grading is trace-based: ranCommandOneOf (enable factor), ranCommand guardian/policies all-applications (enforce), ranCommandsInOrder (enable before enforce), plus a holistic trace-aware judge.
📝 WalkthroughWalkthroughThe PR adds provision-aware CLI guidance to eval execution and introduces a tenant MFA CLI evaluation. The evaluation requires MFA factor enablement followed by all-application enforcement. Graders validate command order and reject dashboard or Terraform configuration. ChangesTenant CLI MFA evaluation
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Agent
participant Auth0CLI
participant Tenant
participant MFAGraders
Agent->>Auth0CLI: enable required MFA factor
Auth0CLI->>Tenant: update MFA factor
Agent->>Auth0CLI: enforce MFA for all applications
Auth0CLI->>Tenant: update guardian/policies
MFAGraders->>Tenant: inspect events and command trace
MFAGraders-->>Agent: validate order and CLI-only compliance
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/auth0-evals/src/evals/mfa/tenant-cli/graders.ts`:
- Around line 17-43: The existing event graders only match command substrings
and do not verify successful MFA writes or final state. Strengthen the graders
around ranCommandOneOf, ranCommand, and ranCommandsInOrder to reject
read/disabled or partial/contradictory writes, validate the resulting enabled
factor and all-applications policy (using a verify.js/final-state check if
supported), and add deterministic notContains checks for dashboard and Terraform
configuration.
In `@apps/auth0-evals/src/evals/mfa/tenant-cli/PROMPT.md`:
- Around line 23-27: Update the MFA tenant CLI prompt to explicitly require the
all-applications enforcement policy, matching the guardian/policies expectation
in the grader. Keep the instructions focused on enabling the required factor and
enforcing MFA for all applications via non-interactive Auth0 CLI commands.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 49278d0d-b331-43a7-85a1-6667711cebe2
📒 Files selected for processing (2)
apps/auth0-evals/src/evals/mfa/tenant-cli/PROMPT.mdapps/auth0-evals/src/evals/mfa/tenant-cli/graders.ts
…ontext file
The mfa_tenant_cli eval carried a '## System' platform preamble (you are a
platform engineer, use 'auth0 api ...'), but '## System' only feeds baseline
mode — agent runners receive only '## Task', so that context never reached the
agent in agent mode, which is the mode the eval targets.
Add a framework mechanism to inject it, keeping evals-core provider-agnostic:
- Loader parses the 'provision' frontmatter field onto EvalDefinition.
- writeAgentGuidance takes an options object { compileCommand?, extraContext? }
and appends extraContext after the base guidance (order: base guidance →
extraContext → compile guidance).
- FrameworkConfig gains provisionContext: Record<string,string> (empty default);
run.ts resolves evalDef.provision against it and passes the text as
extraContext, so the platform context lands in the agent's context file
(CLAUDE.md / AGENTS.md / ...) on both local and sandbox paths.
- The Auth0-specific wording moves to eval.config.js provisionContext
['auth0-tenant'], mirroring scoring.docUrlSources: mechanism in the framework,
provider text in the app.
- Drop the '## System' section from the eval; it's now framework-injected.
Tests: loader parses provision (present/absent); writeAgentGuidance injects and
orders extraContext correctly. Docs: ADDING_EVALS frontmatter table + AGENTS.md
context-injection note.
sanchitmehtagit
changed the title
…injected text - Rename the config field FrameworkConfig.provisionContext -> cliContext: the name should describe what it holds (CLI/platform context) rather than the provision trigger. Renamed everywhere (defaults, run.ts, eval.config.js, docs). - Rename the writeAgentGuidance option extraContext -> cliContext and keep AgentGuidanceOptions internal (unexported) — compileCommand is a per-run framework detail, not public API. - Trim the injected auth0-tenant context: drop the 'platform engineer' persona (no behavioral value) and the 'not IaC/not dashboard' line (a task constraint that already lives in the eval's ## Task). Keep the facts the agent genuinely can't discover — the shell is already authenticated to a live tenant, don't re-login, don't hardcode/look up domain/client-id/secret, stay non-interactive, never print secrets — plus the generic 'auth0 api' Management API passthrough affordance (many resources have no dedicated subcommand). No MFA/B2B endpoints, so the graded answer isn't leaked.
| 'CLI (`auth0 login` has already run and that tenant is active). Do not run `auth0 login`, ' + | ||
| 'and do not hardcode or look up the tenant domain, client ID, or client secret — the CLI ' + | ||
| 'is already authenticated. Run commands non-interactively and never print or store secrets.' + | ||
| '\n\nTenant configuration that has no dedicated `auth0` subcommand can be done through the ' + |
There was a problem hiding this comment.
Tenant configuration that has no dedicated auth0 subcommand can be done through the ' +
"Management API passthrough, which takes an HTTP method and a Management API path:\n\n" +
this should not be added
There was a problem hiding this comment.
Done in 3d0245f — removed the auth0 api <METHOD> <path> passthrough hint from cliContext; the agent discovers the CLI command surface itself.
| // resources have no dedicated subcommand; the specific paths are the agent's | ||
| // to find. | ||
| cliContext: { | ||
| 'auth0-tenant': |
There was a problem hiding this comment.
do we need to map this to auth0 tenant
There was a problem hiding this comment.
Keeping cliContext keyed by provision kind. The key is the same value the eval already declares in frontmatter (provision: auth0-tenant), and run.ts resolves cliContext[evalDef.provision] — so an eval with no provision injects nothing, and a future second environment kind (e.g. a different tenant type or a mock) maps to its own context instead of reusing this one. Costs one line of nesting; no new concept. No code change.
|
|
||
| Using the Auth0 CLI, enable the required MFA factor on the tenant and then enforce MFA so it is required for users. Work out the commands yourself. Run everything non-interactively (no interactive prompts should block you). If a command errors, read the message and adjust. Never print or store any client secret. | ||
|
|
||
| Do not configure the tenant through the dashboard or Terraform — the change must be made via the Auth0 CLI. |
There was a problem hiding this comment.
Work out the commands yourself. Run everything non-interactively (no interactive prompts should block you). If a command errors, read the message and adjust. Never print or store any client secret.Do not configure the tenant through the dashboard or Terraform — the change must be made via the Auth0 CLI.
this is not needed
There was a problem hiding this comment.
Done in 3d0245f — trimmed the prompt to the goal-only requirement. The non-interactive / never-print-secrets / no-dashboard-or-Terraform guidance is now injected via cliContext, so it no longer needs to live in the prompt.
- Drop the auth0 api Management API passthrough hint from cliContext (leave the CLI command surface for the agent to discover). - Trim the eval prompt to the goal-only requirement; the non-interactive / no-secrets / no-dashboard-or-Terraform guidance is already injected via cliContext.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/evals/src/cli/run.ts`:
- Around line 137-145: Add Vitest coverage in the packages/evals tests directory
for the runAgentJob cliContext resolution: verify a matching evalDef.provision
passes the configured context to writeAgentGuidance, while missing and unmapped
provision keys pass undefined. Mock or spy on writeAgentGuidance and
getFrameworkConfig as needed, covering each branch introduced in the
runner-level logic.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 71140709-8ad5-4f0b-93d8-2448dc499e91
📒 Files selected for processing (12)
AGENTS.mdapps/auth0-evals/eval.config.jsapps/auth0-evals/src/evals/mfa/tenant-cli/PROMPT.mddocs/ADDING_EVALS.mdpackages/evals-core/src/config/defaults.tspackages/evals-core/src/config/framework.tspackages/evals-core/src/loader.tspackages/evals-core/src/types/eval.tspackages/evals-core/src/workspace/workspace.tspackages/evals-core/tests/loader.test.tspackages/evals-core/tests/workspace.test.tspackages/evals/src/cli/run.ts
| // local execution paths so every runner picks it up. When the eval declares a | ||
| // `provision` kind, also inject the app-configured platform context for it — | ||
| // the `## System` section only feeds baseline mode, so this is how that | ||
| // guidance reaches the agent. | ||
| const cliContext = evalDef.provision ? getFrameworkConfig().cliContext?.[evalDef.provision] : undefined; | ||
| writeAgentGuidance(workspace, agentType, { | ||
| compileCommand: evalDef.compileCommand, | ||
| cliContext, | ||
| }); |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Add runner-level coverage for cliContext resolution.
Add Vitest coverage in the packages/evals package. Test a matching provision key, no provision key, and an unmapped provision key. The current core-package tests do not verify that runAgentJob passes the configured context to writeAgentGuidance.
As per coding guidelines, “Every new function and every logic change must be accompanied by Vitest tests in the package's own tests/ directory” and non-trivial logic changes must cover each new branch.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/evals/src/cli/run.ts` around lines 137 - 145, Add Vitest coverage in
the packages/evals tests directory for the runAgentJob cliContext resolution:
verify a matching evalDef.provision passes the configured context to
writeAgentGuidance, while missing and unmapped provision keys pass undefined.
Mock or spy on writeAgentGuidance and getFrameworkConfig as needed, covering
each branch introduced in the runner-level logic.
Source: Coding guidelines
2 participants
What
Two related changes:
New eval
mfa_tenant_cli— a goal-only CLI eval that configures MFA on a real, throwaway Auth0 tenant (requested viaprovision: auth0-tenantfrontmatter), rather than a mocked Guardian surface. Enable an MFA factor, then enforce MFA viaguardian/policiesusing the pre-authenticatedauth0CLI. Live-tenant style from feat(evals): add auth0 CLI B2B organization setup eval #178; MFA scenario from feat(mfa): MFA Leg 2 (tenant config: CLI) #83.Framework: inject provisioned-env platform context into the agent context file. The eval originally carried a
## Systemplatform preamble ("you are a platform engineer, useauth0 api ..."). But## Systemonly feeds baseline mode — agent runners receive only## Task, so that context never reached the agent in agent mode (the mode this eval targets). This adds a proper mechanism to inject it.Framework design (mechanism in framework, text in consumer)
evals-corestays provider-agnostic:provisionfrontmatter field ontoEvalDefinition.writeAgentGuidancenow takes{ compileCommand?, extraContext? }and appendsextraContextafter the base guidance (order: base guidance → extraContext → compile guidance).FrameworkConfig.provisionContext: Record<string,string>(empty default).run.tsresolvesevalDef.provisionagainst it and passes the text asextraContext, so the platform context lands in the agent's context file (CLAUDE.md/AGENTS.md/…) on both local and Docker-sandbox paths (written before dispatch on the bind-mounted workspace).eval.config.jsprovisionContext['auth0-tenant'], mirroringscoring.docUrlSources.Grading (unchanged, trace-based)
ranCommandOneOf(['guardian/factors/otp','push','sms'])— enabled a factorranCommand('guardian/policies', ['all-applications'])— enforced MFAranCommandsInOrder([...factors, 'guardian/policies'])— factor before enforcementincludeCommandTrace: trueTests
provision(present/absent).writeAgentGuidanceinjectsextraContextand orders sections correctly (base → extra → compile); existing call sites updated to the options object.Verification
npm run build && npm run lint && npm run format && npm test— all pass (loader + workspace suites include the new cases).loadConfigreads appprovisionContext, loader surfacesprovision: auth0-tenant, and the platform-engineer +auth0 apitext lands in the workspaceCLAUDE.mdalongside the base guidance — proving the context now reaches the agent.Notes
verify.js(deferred).docs/ADDING_EVALS.mdfrontmatter table (provision),AGENTS.mdcontext-injection note + docs-map row.Summary by CodeRabbit
New Features
Documentation
Tests