feat(mock-http): run the real Auth0 CLI against a mock Management API - #176
Closed
sanchitmehtagit wants to merge 2 commits into
Closed
feat(mock-http): run the real Auth0 CLI against a mock Management API#176sanchitmehtagit wants to merge 2 commits into
sanchitmehtagit wants to merge 2 commits into
Conversation
Adds hermetic HTTP-level mocking for tenant-config evals: instead of replacing the auth0 binary, run the REAL CLI and intercept its HTTPS calls to the Management API. Real command parsing and request construction are exercised; only the network hop is faked (no auth, no network, no live side effects). Interception is DNS+TLS (the CLI has no base-URL override / --insecure): a mock HTTPS server binds 127.0.0.1:8443 driven by declarative *.routes.json manifests; the CLI config is seeded with a fake tenant so it targets the mock and skips login; a test-only CA (docker/mock-ca) is baked into the container trust store so the Go CLI trusts the loopback leaf cert; telemetry is disabled. - packages/evals-core/src/mock-http: engine, HTTPS server, manifest/verbs/ matcher/state, CLI-config seeding, startMockCliForEval lifecycle helper - loader: http-routes/ convention sets EvalDefinition.httpRoutesDir and auto-attaches shared CLI platform context - wire lifecycle into sandbox-runner (in-container) and run.ts (local, isolated HOME, --workers 1) - Dockerfile: install pinned auth0 CLI v1.32.0, bake mock CA, ship leaf cert - example eval cli/guardian_otp_cli graded with L4 event graders - gen-mock-ca.mjs generator; docs in ADDING_EVALS, AGENTS, mock-http/README Tests: build, lint, format, and full suite pass; real-TLS server e2e covers create -> reflect read-after-write -> fallthrough.
📝 WalkthroughWalkthroughAdds a declarative HTTPS mock runtime for real Auth0 CLI evaluations, local lifecycle integration, route discovery, TLS assets, authoring guidance, and a Guardian OTP evaluation with stateful fixtures and event-based graders. ChangesHTTP-mocked CLI evaluation flow
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant EvalLoader
participant EvalRunner
participant MockCliLifecycle
participant Auth0CLI
participant MockManagementAPI
EvalLoader->>EvalRunner: load eval with httpRoutesDir
EvalRunner->>MockCliLifecycle: start local mock and seed CLI config
MockCliLifecycle->>MockManagementAPI: start loopback HTTPS server
EvalRunner->>Auth0CLI: run Guardian OTP evaluation
Auth0CLI->>MockManagementAPI: PUT guardian/factors/otp
MockManagementAPI-->>Auth0CLI: enabled OTP response
Auth0CLI->>MockManagementAPI: GET guardian/factors
MockManagementAPI-->>Auth0CLI: reflected factor configuration
EvalRunner->>MockCliLifecycle: stop local mock
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 13
🧹 Nitpick comments (5)
packages/evals-core/src/utils/env.ts (1)
4-8: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueCorrect the scope claim in the comment.
The comment states the three keys "only take effect when the mock-CLI lifecycle sets them".
filteredEnvforwards any value already present inprocess.env. If a developer exportsSSL_CERT_FILEin their shell, every agent child process now inherits it, including evals that do not mock the CLI. The values are trust-store paths and a telemetry flag, so this is not a credential exposure, but the stated scope is wider than the comment says.Reword the comment to state that the keys are forwarded whenever they are present in the parent environment.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/evals-core/src/utils/env.ts` around lines 4 - 8, Update the comment describing SSL_CERT_FILE, SSL_CERT_DIR, and AUTH0_CLI_ANALYTICS to state that filteredEnv forwards these keys whenever they are present in the parent environment, rather than claiming they only apply during the mock-CLI lifecycle.packages/evals-core/src/mock-http/cli-config.ts (1)
25-29: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueThe secret-scanner hit on
DUMMY_ACCESS_TOKENis a false positive.The value is a static, inert JWT whose signature segment is the literal
mock-signature-not-a-real-token. It grants nothing. Add a scanner allowlist entry (for example agitleaks:allowstyle annotation matching your tool) so this line does not block future scans.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/evals-core/src/mock-http/cli-config.ts` around lines 25 - 29, Add the repository’s supported secret-scanner allowlist annotation to DUMMY_ACCESS_TOKEN, targeting the static inert JWT declaration so future scans ignore this known false positive. Keep the token value and surrounding documentation unchanged.Source: Linters/SAST tools
packages/evals-core/src/mock-http/server.ts (1)
50-57: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winGuard the error branch against already-sent headers.
res.end(payload)on line 52 runs afterres.writeHead. If it throws, the catch block callsres.writeHeadagain. Node then throwsERR_HTTP_HEADERS_SENTinside thevoid-ed async function, which becomes an unhandled rejection and can terminate the eval process.Check
res.headersSentbefore you write the error response.♻️ Proposed fix
} catch (e) { // Never crash the server on a bad request — answer with a 500 JSON body. + if (res.headersSent) { + res.destroy(); + return; + } res.writeHead(500, { 'content-type': 'application/json' }); res.end(JSON.stringify({ error: 'mock_server_error', message: e instanceof Error ? e.message : String(e) })); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/evals-core/src/mock-http/server.ts` around lines 50 - 57, Update the catch branch around the request handler’s response serialization to check res.headersSent before calling res.writeHead(500) and res.end(...). Only send the JSON error response when headers have not already been sent, preventing a second header write after res.end(payload) fails.packages/evals-core/src/loader.ts (2)
169-195: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMove the app-specific context path out of the framework package.
CLI_PLATFORM_CONTEXThard-codessrc/evals/contexts/cli-platform/AGENTS.md.packages/evals-coreis the generic framework package, and that layout belongs toapps/auth0-evals. Any other consumer ofevals-corethat ships anhttp-routes/directory silently gets no platform context.Move the path into
FrameworkConfigwith the current value as the default.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/evals-core/src/loader.ts` around lines 169 - 195, Move the CLI platform context path from the module-level CLI_PLATFORM_CONTEXT constant into FrameworkConfig, defaulting it to the current src/evals/contexts/cli-platform/AGENTS.md value. Update loadSharedCliContext to receive and use the configured path, and ensure the framework configuration flows through the HTTP-routes context-loading logic so consumers can override it.
57-65: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winLog both context-resolution outcomes.
Two silent outcomes affect what the agent reads:
- Line 65 merges
contextover the scaffold, so a sharedAGENTS.mdreplaces an eval-authoredAGENTS.mdwith no signal to the eval author.loadSharedCliContextreturns{}when the shared file is missing, so anhttp-routes/eval runs with no CLI guidance and likely scores near zero for a reason that is not visible in the run output.Emit a
logger.warnfor each case.♻️ Proposed change in `loadSharedCliContext` and `loadEval`
function loadSharedCliContext(frameworkRoot: string): Record<string, string> { const contextFile = join(frameworkRoot, CLI_PLATFORM_CONTEXT); if (existsSync(contextFile) && statSync(contextFile).isFile()) { return { 'AGENTS.md': content }; } + logger.warn(`[loader] http-routes eval found but shared CLI context is missing at ${contextFile}`); return {}; }const httpRoutesDir = resolveHttpRoutesDir(evalPath); const context = httpRoutesDir ? loadSharedCliContext(frameworkRoot) : {}; - const scaffold = { ...loadScaffold(scaffoldDir), ...context }; + const baseScaffold = loadScaffold(scaffoldDir); + if (context['AGENTS.md'] && baseScaffold['AGENTS.md']) { + logger.warn(`[loader] ${evalConfig.id}: shared CLI context overrides the scaffold AGENTS.md`); + } + const scaffold = { ...baseScaffold, ...context };🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/evals-core/src/loader.ts` around lines 57 - 65, Update loadSharedCliContext and loadEval to emit logger.warn messages for both outcomes: when shared CLI context is missing for an eval with httpRoutesDir, and when shared context overrides an eval-authored AGENTS.md during scaffold merging. Keep the existing merge precedence and empty-context behavior unchanged, while making each warning identify the affected eval/context.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/auth0-evals/src/evals/cli/guardian-otp/graders.ts`:
- Line 37: Update the read grader’s guardian factors check near ranCommand in
the relevant grader flow to use ranCommandOneOf with the exact GET variants “get
guardian/factors” and “GET guardian/factors”, ensuring write commands such as
PUT guardian/factors/otp do not satisfy the confirmation.
- Around line 16-37: Update defineGraders to add an ordered trace grader after
the existing OTP command checks, requiring a successful PUT or PATCH to
guardian/factors/otp followed by a later successful GET of guardian/factors.
Ensure the GET response is validated to contain otp.enabled set to true, rather
than relying on the broad ranCommand auth0 api matching.
In
`@apps/auth0-evals/src/evals/cli/guardian-otp/http-routes/guardian.routes.json`:
- Around line 6-9: The guardian OTP mock route manifest currently supports only
PUT while the graders accept PATCH; update the route definitions around the
“guardian/factors/otp” mutation so PATCH applies the same “guardian.otp” state
transition and uses “otp-enabled.json”, or remove PATCH from the accepted
commands in the relevant grader. Keep the mock behavior and grader expectations
aligned.
In `@docs/ADDING_EVALS.md`:
- Around line 331-341: Update the directory-tree fenced block near the
my-cli-eval structure in the documentation to include a suitable language
identifier, such as text, on its opening fence so it satisfies markdownlint
MD040.
In `@packages/evals-core/src/mock-http/lifecycle.ts`:
- Line 55: Update the default port selection in the lifecycle flow around
startMockCliForEval to use port 0 when options.port is absent, allowing the OS
to assign an available port. Preserve the existing explicit-port behavior and
the writeAuth0CliConfig(..., { port: server.port }) wiring.
- Around line 83-95: Update startMockCliForEval so its returned stop function
removes the CLI config written by writeAuth0CliConfig and restores or unsets the
prior AUTH0_CLI_ANALYTICS value, in addition to closing the server. Capture the
original environment state before mutation and ensure cleanup runs reliably for
the default real-home path.
In `@packages/evals-core/src/mock-http/manifest.ts`:
- Around line 28-30: The manifest validation only checks that a handler name is
present, allowing unknown matched handlers to fall through to a successful
fallback response. In packages/evals-core/src/mock-http/manifest.ts#L28-L30,
validate handler names against the loaded handlers, or update dispatch to throw
when a matched handler is absent; in
packages/evals-core/tests/mock-http/engine.test.ts#L165-L177, add a regression
test confirming an unknown matched handler fails rather than returning a read or
write fallback.
In `@packages/evals-core/src/mock-http/server.ts`:
- Around line 75-79: Update the mock server’s close method to call
server.closeAllConnections() immediately after server.close(), before resolving
the Promise, so keep-alive sockets are terminated and teardown completes.
In `@packages/evals-core/src/mock-http/state.ts`:
- Around line 15-17: Replace the lossy markerName encoding with a collision-free
encoding of the complete key, such as Base64URL, while preserving valid
marker-name usage. Add tests covering normal key encoding and the edge case that
distinct keys such as guardian/otp and guardian_otp produce different names and
remain isolated in reflect state behavior.
In `@packages/evals-core/tests/mock-http/cli-config.test.ts`:
- Line 16: Update the path assertion in the relevant test to construct the
expected Auth0 config path with node:path join, matching writeAuth0CliConfig’s
platform-specific separators instead of hard-coding “/”.
In `@packages/evals-core/tests/mock-http/server.test.ts`:
- Around line 50-55: Update the committed mock TLS certificates referenced by
the HTTPS test to renew the CA and leaf certificates before expiration, using
the documented generator command and preserving the existing certificate paths.
Add the current expiration date for mockCA.pem and mockServer.pem to the
docker/mock-ca documentation.
In `@packages/evals/src/cli/run.ts`:
- Around line 242-247: Update the mock server startup call in the local
evaluation flow around mockHttp.startMockCliForEval to avoid the lifecycle
default port 8443: allocate and pass a unique per-job port, or explicitly reject
workers > 1 for local HTTP-mocked evaluations. Add regression coverage that
verifies concurrent startup does not produce EADDRINUSE.
In `@packages/evals/src/cli/sandbox-runner.ts`:
- Around line 183-184: Update the failure handling in main so it sets
process.exitCode to 1 and returns after writing the error result instead of
calling process.exit, allowing the finally block to await mockCli?.stop(). Add a
failure-path test that verifies the mock server’s stop method is invoked.
---
Nitpick comments:
In `@packages/evals-core/src/loader.ts`:
- Around line 169-195: Move the CLI platform context path from the module-level
CLI_PLATFORM_CONTEXT constant into FrameworkConfig, defaulting it to the current
src/evals/contexts/cli-platform/AGENTS.md value. Update loadSharedCliContext to
receive and use the configured path, and ensure the framework configuration
flows through the HTTP-routes context-loading logic so consumers can override
it.
- Around line 57-65: Update loadSharedCliContext and loadEval to emit
logger.warn messages for both outcomes: when shared CLI context is missing for
an eval with httpRoutesDir, and when shared context overrides an eval-authored
AGENTS.md during scaffold merging. Keep the existing merge precedence and
empty-context behavior unchanged, while making each warning identify the
affected eval/context.
In `@packages/evals-core/src/mock-http/cli-config.ts`:
- Around line 25-29: Add the repository’s supported secret-scanner allowlist
annotation to DUMMY_ACCESS_TOKEN, targeting the static inert JWT declaration so
future scans ignore this known false positive. Keep the token value and
surrounding documentation unchanged.
In `@packages/evals-core/src/mock-http/server.ts`:
- Around line 50-57: Update the catch branch around the request handler’s
response serialization to check res.headersSent before calling
res.writeHead(500) and res.end(...). Only send the JSON error response when
headers have not already been sent, preventing a second header write after
res.end(payload) fails.
In `@packages/evals-core/src/utils/env.ts`:
- Around line 4-8: Update the comment describing SSL_CERT_FILE, SSL_CERT_DIR,
and AUTH0_CLI_ANALYTICS to state that filteredEnv forwards these keys whenever
they are present in the parent environment, rather than claiming they only apply
during the mock-CLI lifecycle.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 28d4deb7-479e-4c4c-a171-8896c39e286c
⛔ Files ignored due to path filters (2)
docker/mock-ca/mockCA.pemis excluded by!**/*.pemdocker/mock-ca/mockServer.pemis excluded by!**/*.pem
📒 Files selected for processing (36)
AGENTS.mdapps/auth0-evals/scripts/gen-mock-ca.mjsapps/auth0-evals/src/evals/cli/guardian-otp/PROMPT.mdapps/auth0-evals/src/evals/cli/guardian-otp/graders.tsapps/auth0-evals/src/evals/cli/guardian-otp/http-routes/fixtures/guardian/factors-otp-off.jsonapps/auth0-evals/src/evals/cli/guardian-otp/http-routes/fixtures/guardian/factors-otp-on.jsonapps/auth0-evals/src/evals/cli/guardian-otp/http-routes/fixtures/guardian/otp-enabled.jsonapps/auth0-evals/src/evals/cli/guardian-otp/http-routes/guardian.routes.jsonapps/auth0-evals/src/evals/contexts/cli-platform/AGENTS.mddocker/Dockerfiledocker/mock-ca/mockServer.keydocs/ADDING_EVALS.mdpackages/evals-core/src/index.tspackages/evals-core/src/loader.tspackages/evals-core/src/mock-http/README.mdpackages/evals-core/src/mock-http/cli-config.tspackages/evals-core/src/mock-http/engine.tspackages/evals-core/src/mock-http/handlers.tspackages/evals-core/src/mock-http/index.tspackages/evals-core/src/mock-http/lifecycle.tspackages/evals-core/src/mock-http/manifest.tspackages/evals-core/src/mock-http/matcher.tspackages/evals-core/src/mock-http/server.tspackages/evals-core/src/mock-http/state.tspackages/evals-core/src/mock-http/types.tspackages/evals-core/src/mock-http/verbs.tspackages/evals-core/src/types/eval.tspackages/evals-core/src/utils/env.tspackages/evals-core/tests/loader.test.tspackages/evals-core/tests/mock-http/cli-config.test.tspackages/evals-core/tests/mock-http/engine.test.tspackages/evals-core/tests/mock-http/lifecycle.test.tspackages/evals-core/tests/mock-http/server.test.tspackages/evals/src/cli/constants.tspackages/evals/src/cli/run.tspackages/evals/src/cli/sandbox-runner.ts
Comment on lines
+16
to
+37
| export function defineGraders() { | ||
| return [ | ||
| // ── L4: Structural — the right CLI calls were made ──────────────────────── | ||
| // Enable OTP via the Management API passthrough: PUT/PATCH guardian/factors/otp. | ||
| ranCommand( | ||
| 'auth0 api', | ||
| ['guardian/factors/otp'], | ||
| 'Enabled the OTP factor via the Management API passthrough', | ||
| GraderLevel.L4, | ||
| ), | ||
| ranCommandOneOf( | ||
| [ | ||
| 'put guardian/factors/otp', | ||
| 'patch guardian/factors/otp', | ||
| 'PUT guardian/factors/otp', | ||
| 'PATCH guardian/factors/otp', | ||
| ], | ||
| 'Used PUT/PATCH to toggle the OTP factor', | ||
| GraderLevel.L4, | ||
| ), | ||
| // Confirm: read the factor configuration back. | ||
| ranCommand('auth0 api', ['guardian/factors'], 'Read the factor configuration back to confirm', GraderLevel.L4), |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== candidate file =="
fd -a 'graders\.ts$' . | sed 's#^\./##'
echo
if [ -f apps/auth0-evals/src/evals/cli/guardian-otp/graders.ts ]; then
echo "== file outline =="
ast-grep outline apps/auth0-evals/src/evals/cli/guardian-otp/graders.ts || true
echo
echo "== file contents =="
cat -n apps/auth0-evals/src/evals/cli/guardian-otp/graders.ts
else
fd 'graders\.ts$' . -x sh -c 'echo "--- $1"; sed -n "1,220p" "$1"' sh {}
fi
echo
echo "== search helpers definitions/usages =="
rg -n "function ranCommand|r an Command|ranCommandOneOf|defineGraders|Guardian|guardian" -S .Repository: auth0/auth0-evals
Length of output: 16646
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== guardian routes and fixtures =="
cat -n apps/auth0-evals/src/evals/cli/guardian-otp/http-routes/guardian.routes.json
find apps/auth0-evals/src/evals/cli/guardian-otp -maxdepth 3 -type f \( -name '*.json' -o -name '*.js' -o -name '*.ts' \) -print | sort | sed 's#^\./##'
echo
for f in apps/auth0-evals/src/evals/cli/guardian-otp/http-routes/guardian.routes.json; do
echo "--- $f"
cat -n "$f"
done
echo
echo "== grader primitives implementation =="
sed -n '1,180p' packages/evals-graders/src/primitives.ts | cat -n
echo
echo "== grader engine behavior tests around ranCommand =="
sed -n '980,1090p' packages/evals-core/tests/graders/engine.test.ts | cat -n
sed -n '240,280p' packages/evals-graders/tests/primitives.test.ts | cat -n
echo
echo "== deterministic substring/order probe for current predicate semantics =="
python3 - <<'PY'
cases = [
[("command auth0 api guardian/filters/otp", "success")],
[("command put guardian/factors/otp other", "success")],
[("command auth0 api guardian/factors/otp error", "error")],
[("command auth0 api guardian/filters/otp (success)", "success"),
("command auth0 api guardian/factors", "success")],
[("command auth0 api guardian/factors", "success"),
("command auth0 api guardian/factors/otp", "success")],
]
for i, calls in enumerate(cases):
strings = [a[0] for a in calls]
has_otp_api = any("command auth0 api guardian/filters/otp" in s or "command auth0 api guardian/factors/otp" in s for s in strings)
has_write = any("guardian/factors/otp" in s and s.startswith("command put ") or s.startswith("command patch ") or s.startswith("command PUT ") or s.startswith("command PATCH ") or "guardian/factors/otp" in s for s in strings)
has_read = any("guardian/factors" in s for s in strings)
has_read_first = len(calls) >= 2 and "guardian/factors" in calls[0][0]
print(f"case {i+1}: has_enable_endpoint={has_otp_api}, has_write_like={has_write}, has_read_like={has_read}, read_before_enable={has_read_first}")
PYRepository: auth0/auth0-evals
Length of output: 17367
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== auth0-evals guardian OTP fixtures =="
mkdir -p tmp_guardian_fixtures
find apps/auth0-evals/src/evals/cli/guardian-otp -type f \( -path '*fixtures*' -o -path '*https*' \) -print | sort
find apps/auth0-evals/src/evals/cli/guardian-otp -type f | sort | rg 'otp|routes|handlers|json' | xargs -r -n1 sh -c 'echo "--- $0"; sed -n "1,160p" "$0"' sh {}
echo
echo "== mock HTTP engine route/state handling =="
fd -a 'engine\.test\.ts$|engine\.(ts|js)$|types\.ts$|readme|README' packages/evals-core/src packages/evals-core/tests | sed 's#^\./##'
echo "route files:"
fd -a '(routes\.json|handlers\.js|README.md)$' packages/evals-core/src/mock-http apps/auth0-evals/src/evals/cli/guardian-otp | sed 's#^\./##'
for f in $(fd -a 'engine\.ts$|handlers\.js$|README.md$' packages/evals-core/src/mock-http packages/evals-core/tests/mock-http | sort); do
echo "--- $f"
sed -n "1,240p" "$f" | cat -n
doneRepository: auth0/auth0-evals
Length of output: 797
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== auth0-evals guardian OTP fixture contents =="
while IFS= read -r f; do
echo "--- $f"
cat -n "$f"
done < <(find apps/auth0-evals/src/evals/cli/guardian-otp/http-routes -type f -print | sort)
echo
echo "== mock HTTP implementation candidates =="
find packages/evals-core/src/mock-http -maxdepth 3 -type f ! -type d | sort
for f in $(find packages/evals-core/src/mock-http -maxdepth 3 -type f ! -type d -print | sort); do
echo "--- $f"
sed -n "1,220p" "$f" | cat -n
doneRepository: auth0/auth0-evals
Length of output: 255
🏁 Script executed:
#!/bin/bash
set -euo pipefail
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
echo "== auth0-evals guardian OTP fixture contents =="
find apps/auth0-evals/src/evals/cli/guardian-otp/http-routes -type f | sort > "$tmp/files"
while IFS= read -r f; do
echo "--- $f"
cat -n "$f"
done < "$tmp/files"
echo
echo "== mock HTTP implementation candidates =="
find packages/evals-core/src/mock-http -type f ! -type d | sort > "$tmp/files"
while IFS= read -r f; do
echo "--- $f"
sed -n "1,220p" "$f" | cat -n
done < "$tmp/files"Repository: auth0/auth0-evals
Length of output: 33203
Grade the enabled OTP response in order.
ranCommand* only matches non-errored shell command text. As written, get guardian/factors matches auth0 api guardian/factors/otp, so the read grader can pass without a real GET /api/v2/guardian/factors. Add an ordered trace grader that requires a successful PUT/PATCH guardian/factors/otp and a later GET guardian/factors whose response contains otp.enabled: true.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/auth0-evals/src/evals/cli/guardian-otp/graders.ts` around lines 16 - 37,
Update defineGraders to add an ordered trace grader after the existing OTP
command checks, requiring a successful PUT or PATCH to guardian/factors/otp
followed by a later successful GET of guardian/factors. Ensure the GET response
is validated to contain otp.enabled set to true, rather than relying on the
broad ranCommand auth0 api matching.
| GraderLevel.L4, | ||
| ), | ||
| // Confirm: read the factor configuration back. | ||
| ranCommand('auth0 api', ['guardian/factors'], 'Read the factor configuration back to confirm', GraderLevel.L4), |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate file:"
fd -a 'graders\.ts$' . | sed 's#^\./##'
echo
echo "Outline candidate:"
ast-grep outline apps/auth0-evals/src/evals/cli/guardian-otp/graders.ts --view compact || true
echo
echo "Relevant source around L20-L50:"
sed -n '1,90p' apps/auth0-evals/src/evals/cli/guardian-otp/graders.ts | cat -n
echo
echo "Search ranCommand definitions/usages:"
rg -n "function ranCommand|const ranCommand|ranCommand\\(" .Repository: auth0/auth0-evals
Length of output: 9988
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Graders primitive implementation:"
sed -n '1,160p' packages/evals-graders/src/primitives.ts | cat -n
echo
echo "Search for command recording format/tests around command methods:"
rg -n "put guardian/factors/otp|guardian/factors/otp|guardian/factors|successful command|ran successful" packages/evals-graders packages/evals-core apps/auth0-evals -S
echo
echo "Inspect primitive tests section:"
sed -n '200,260p' packages/evals-graders/tests/primitives.test.ts | cat -nRepository: auth0/auth0-evals
Length of output: 10755
🌐 Web query:
Auth0 CLI guardiann factors read command GET auth0 api guardian/factors
💡 Result:
The Auth0 CLI does not have a dedicated, hardcoded command (e.g., auth0 guardian factors) for retrieving multi-factor authentication factors [1][2]. However, you can interact with the underlying Management API endpoint using the generic auth0 api command [3][4]. To retrieve the multi-factor authentication factors for your tenant, you should use the auth0 api command to make a GET request to the /guardian/factors endpoint [5][3]. Execute the following command in your terminal: auth0 api get "guardian/factors" This command interfaces directly with the Management API [3], which provides the details of all MFA factors associated with your tenant [5][6]. Ensure that your CLI is authenticated [4][2] and that your token has the necessary read:guardian_factors scope [5][6].
Citations:
- 1: https://github.com/auth0/auth0-cli/
- 2: https://auth0.github.io/auth0-cli/
- 3: https://auth0.github.io/auth0-cli/auth0_api.html
- 4: https://auth0.com/docs/deploy-monitor/auth0-cli
- 5: https://auth0.com/docs/api/management/v2/guardian/get-factors
- 6: https://apis.io/apis/auth0/auth0-guardian-api/
Require GET guardian/factors in the read grader.
ranCommand('auth0 api', ['guardian/factors']) matches any auth0 api command containing guardian/factors, including auth0 api put guardian/factors/otp. Use ranCommandOneOf(['get guardian/factors', 'GET guardian/factors']) so the confirm step requires a read-path command.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/auth0-evals/src/evals/cli/guardian-otp/graders.ts` at line 37, Update
the read grader’s guardian factors check near ranCommand in the relevant grader
flow to use ranCommandOneOf with the exact GET variants “get guardian/factors”
and “GET guardian/factors”, ensuring write commands such as PUT
guardian/factors/otp do not satisfy the confirmation.
Comment on lines
+6
to
+9
| "match": "PUT guardian/factors/otp", | ||
| "verb": "create", | ||
| "state": "guardian.otp", | ||
| "body": "otp-enabled.json" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Align the mocked mutation methods with the grader.
apps/auth0-evals/src/evals/cli/guardian-otp/graders.ts accepts both PUT and PATCH, but this manifest handles only PUT. The documented unmatched-write fallback returns {"ok":true} without writing guardian.otp. A PATCH attempt can therefore satisfy the mutation graders while the mock remains disabled. Add a PATCH route with the same state transition, or remove PATCH from the accepted grader commands.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/auth0-evals/src/evals/cli/guardian-otp/http-routes/guardian.routes.json`
around lines 6 - 9, The guardian OTP mock route manifest currently supports only
PUT while the graders accept PATCH; update the route definitions around the
“guardian/factors/otp” mutation so PATCH applies the same “guardian.otp” state
transition and uses “otp-enabled.json”, or remove PATCH from the accepted
commands in the relevant grader. Keep the mock behavior and grader expectations
aligned.
Comment on lines
+331
to
+341
| ``` | ||
| my-cli-eval/ | ||
| ├── PROMPT.md | ||
| ├── graders.ts | ||
| └── http-routes/ | ||
| ├── <surface>.routes.json # one manifest per Management API surface | ||
| ├── fixtures/ | ||
| │ └── <surface>/ | ||
| │ └── *.json # response bodies referenced by routes | ||
| └── handlers.js # optional: request-shaped logic | ||
| ``` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add a language to the directory-tree fence.
markdownlint-cli2 reports MD040 for this fenced block. Add text or another suitable language identifier to the opening fence.
🧰 Tools
🪛 markdownlint-cli2 (0.23.2)
[warning] 331-331: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/ADDING_EVALS.md` around lines 331 - 341, Update the directory-tree
fenced block near the my-cli-eval structure in the documentation to include a
suitable language identifier, such as text, on its opening fence so it satisfies
markdownlint MD040.
Source: Linters/SAST tools
| */ | ||
| export async function startMockCliForEval(options: StartMockCliOptions): Promise<MockCliHandle> { | ||
| const { httpRoutesDir, stateDir, certDir } = options; | ||
| const port = options.port ?? MOCK_CLI_PORT; |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Determine whether eval jobs run concurrently and whether any caller passes a port.
rg -nP -C 6 'Promise\.all|p-limit|concurrenc|maxWorkers|parallel' --type=ts packages/evals/src
rg -nP -C 4 'startMockCliForEval\s*\(|MOCK_CLI_PORT' --type=tsRepository: auth0/auth0-evals
Length of output: 6201
🏁 Script executed:
#!/bin/bash
set -u
echo "== lifecycle.ts =="
fd -a 'lifecycle\.ts$' . | sed 's#^\./##' | while read -r f; do
wc -l "$f"
cat -n "$f"
done
echo "== run.ts relevant section =="
sed -n '420,580p' packages/evals/src/cli/run.ts | cat -n
echo "== sandbox-runner.ts =="
fd -a 'sandbox-runner\.ts$' . | sed 's#^\./##' | while read -r f; do
wc -l "$f"
cat -n "$f"
done
echo "== port usages =="
rg -n -C 3 'startMockCliForEval|MOCK_CLI_PORT|MOCK_CLI_(PORT|HOST)|listen\(|server\.port|port:' packages --glob '*.ts'
echo "== package files =="
fd -a 'package\.json$' . | sed 's#^\./##' | while_read() { :; }; for f in $(fd 'package\.json$' packages -t f); do echo "--- $f"; sed -n '1,160p' "$f"; doneRepository: auth0/auth0-evals
Length of output: 39404
🏁 Script executed:
#!/bin/bash
set -u
echo "== workers option and local mock helper =="
rg -n -C 5 'workers|workers:|startLocalMockCli|port\s*:' packages/evals/src/cli/run.ts
echo "== spawnEval implementation =="
rg -n -C 8 'function spawnEval|const spawnEval|spawnEval\s*=' packages/evals/src/cli/run.ts
echo "== lifecycle helpers around local mock =="
sed -n '124,260p' packages/evals/src/cli/run.ts | cat -n
echo "== worker-related tests/config =="
rg -n -C 4 'workers|parallel|concurrency|maxWorkers|Promise\.all|Promise\.allSettled' packages/evals/tests packages/evals/src packages/evals-core/tests --type=ts || trueRepository: auth0/auth0-evals
Length of output: 29509
Use a dynamic default port for the local mock CLI.
The local helper starts startMockCliForEval without a port, so it uses constant MOCK_CLI_PORT. With the default --workers value, two different eval jobs running on the same host can bind the same 8443 port concurrently and fail. Bind to port 0 and keep the existing writeAuth0CliConfig(..., { port: server.port }) wiring.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/evals-core/src/mock-http/lifecycle.ts` at line 55, Update the
default port selection in the lifecycle flow around startMockCliForEval to use
port 0 when options.port is absent, allowing the OS to assign an available port.
Preserve the existing explicit-port behavior and the writeAuth0CliConfig(..., {
port: server.port }) wiring.
Comment on lines
+15
to
+17
| function markerName(key: string): string { | ||
| return key.replace(/[^A-Za-z0-9._-]/g, '_'); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Use a collision-free state-key encoding.
markerName maps distinct keys to the same marker name. For example, guardian/otp and guardian_otp both map to guardian_otp. A reflect route can then read state written by another route.
Encode the complete key, such as with Base64URL. Add an edge-case test for two colliding keys.
As per coding guidelines, every new function must have “at least one happy-path test and one failure or edge-case test.”
Proposed fix
function markerName(key: string): string {
- return key.replace(/[^A-Za-z0-9._-]/g, '_');
+ return Buffer.from(key, 'utf-8').toString('base64url');
}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/evals-core/src/mock-http/state.ts` around lines 15 - 17, Replace the
lossy markerName encoding with a collision-free encoding of the complete key,
such as Base64URL, while preserving valid marker-name usage. Add tests covering
normal key encoding and the edge case that distinct keys such as guardian/otp
and guardian_otp produce different names and remain isolated in reflect state
behavior.
Source: Coding guidelines
| it('seeds a fake tenant pointing at 127.0.0.1:<port> with a far-future expiry', () => { | ||
| const home = tmp(); | ||
| const path = writeAuth0CliConfig(home, { port: 8443 }); | ||
| expect(path).toBe(`${home}/.config/auth0/config.json`); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Build the expected path with join instead of a hard-coded /.
writeAuth0CliConfig returns a path from node:path join. On Windows that path uses backslashes, so this assertion fails. The repository supports Windows; packages/evals-core/src/utils/env.ts keeps a dedicated WIN32_KEYS list.
💚 Proposed fix
-import { readFileSync } from 'node:fs';
+import { readFileSync } from 'node:fs';
+import { join } from 'node:path';- expect(path).toBe(`${home}/.config/auth0/config.json`);
+ expect(path).toBe(join(home, '.config', 'auth0', 'config.json'));📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| expect(path).toBe(`${home}/.config/auth0/config.json`); | |
| expect(path).toBe(join(home, '.config', 'auth0', 'config.json')); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/evals-core/tests/mock-http/cli-config.test.ts` at line 16, Update
the path assertion in the relevant test to construct the expected Auth0 config
path with node:path join, matching writeAuth0CliConfig’s platform-specific
separators instead of hard-coding “/”.
Comment on lines
+50
to
+55
| it('serves create→reflect and fallthrough over HTTPS with the trusted CA', async () => { | ||
| // The committed certs must exist; regenerated via scripts/gen-mock-ca.mjs. | ||
| expect(existsSync(join(CERT_DIR, 'mockServer.pem'))).toBe(true); | ||
| const cert = readFileSync(join(CERT_DIR, 'mockServer.pem'), 'utf-8'); | ||
| const key = readFileSync(join(CERT_DIR, 'mockServer.key'), 'utf-8'); | ||
| const ca = readFileSync(join(CERT_DIR, 'mockCA.pem'), 'utf-8'); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Print the validity window of the committed mock CA and leaf certificate.
fd -H -t f -g '*.pem' docker --exec sh -c 'echo "== $1"; openssl x509 -in "$1" -noout -subject -dates -ext subjectAltName' _ {}
# Show the lifetime configured by the generator.
fd -H -t f -g 'gen-mock-ca.mjs' --exec rg -n -i 'days|notAfter|validity|expir' {}Repository: auth0/auth0-evals
Length of output: 477
🏁 Script executed:
#!/bin/bash
set -eu
echo "== files =="
fd -H -t f -g '*.pem' docker/mock-ca || true
fd -H -t f -g 'gen-mock-ca.mjs' || true
echo "== certificate dates via OpenSSL if available =="
if command -v openssl >/dev/null 2>&1; then
fd -H -t f -g '*.pem' docker --exec sh -c 'echo "== $1"; openssl x509 -in "$1" -noout -subject -dates -ext subjectAltName 2>&1 || true' _ {}
else
for f in $(fd -H -t f -g '*.pem' docker || true); do
echo "== $f =="
openssl x509 -in "$f" -noout -subject -dates -ext subjectAltName 2>&1 || true
done
done
echo "== generator context =="
for f in $(fd -H -t f -g 'gen-mock-ca.mjs' || true); do
echo "== $f =="
sed -n '1,110p' "$f"
done
echo "== references =="
rg -n "docker/mock-ca|gen-mock-ca|mock-ca|mockCA\.pem|mockServer\.pem|MockServer|createMock" README.md packages eval docker scripts docs 2>/dev/null || trueRepository: auth0/auth0-evals
Length of output: 370
🏁 Script executed:
#!/bin/bash
set -eu
echo "== files =="
fd -H -t f -g '*.pem' docker/mock-ca || true
fd -H -t f -g 'gen-mock-ca.mjs' || true
echo "== certificate dates via OpenSSL if available =="
for f in docker/mock-ca/mockCA.pem docker/mock-ca/mockServer.pem; do
if [ -f "$f" ]; then
echo "== $f =="
if command -v openssl >/dev/null 2>&1; then
openssl x509 -in "$f" -noout -subject -dates -ext subjectAltName || true
else
echo "openssl not available"
fi
else
echo "missing: $f"
fi
done
echo "== generator context =="
f="apps/auth0-evals/scripts/gen-mock-ca.mjs"
if [ -f "$f" ]; then
sed -n '1,120p' "$f"
fi
echo "== references =="
rg -n "docker/mock-ca|gen-mock-ca|mock-ca|mockCA\.pem|mockServer\.pem|MockServer|createMock" README.md packages eval docker scripts docs 2>/dev/null || trueRepository: auth0/auth0-evals
Length of output: 9032
Rotate the mock TLS certificates before they expire.
The generator uses 3650-day lifetimes for both CA and leaf, and the generated docker/mock-ca/mockCA.pem / docker/mock-ca/mockServer.pem pair already exists. The regeneration command is documented for docker/mock-ca/, so add the current certificate expiration date there too.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/evals-core/tests/mock-http/server.test.ts` around lines 50 - 55,
Update the committed mock TLS certificates referenced by the HTTPS test to renew
the CA and leaf certificates before expiration, using the documented generator
command and preserving the existing certificate paths. Add the current
expiration date for mockCA.pem and mockServer.pem to the docker/mock-ca
documentation.
Comment on lines
+242
to
+247
| return mockHttp.startMockCliForEval({ | ||
| httpRoutesDir, | ||
| stateDir: mkdtempSync(join(tmpdir(), 'a0-mock-state-')), | ||
| certDir, | ||
| homeDir, | ||
| }); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Avoid the fixed mock-server port for concurrent local jobs.
This call uses the lifecycle default port, 8443. Concurrent --dangerously-skip-sandbox jobs can start separate processes that bind the same port. The later job fails with EADDRINUSE.
Allocate a per-job port, or reject workers > 1 for local HTTP-mocked evaluations. Add regression coverage for concurrent startup.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/evals/src/cli/run.ts` around lines 242 - 247, Update the mock server
startup call in the local evaluation flow around mockHttp.startMockCliForEval to
avoid the lifecycle default port 8443: allocate and pass a unique per-job port,
or explicitly reject workers > 1 for local HTTP-mocked evaluations. Add
regression coverage that verifies concurrent startup does not produce
EADDRINUSE.
Comment on lines
+183
to
+184
| } finally { | ||
| await mockCli?.stop(); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Allow finally to stop the mock server on failures.
The preceding process.exit(1) terminates the process before this finally block runs. A failed mocked evaluation therefore skips mockCli.stop().
Set process.exitCode = 1 and return from main after writing the error result. Add a failure-path test that verifies stop() runs.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/evals/src/cli/sandbox-runner.ts` around lines 183 - 184, Update the
failure handling in main so it sets process.exitCode to 1 and returns after
writing the error result instead of calling process.exit, allowing the finally
block to await mockCli?.stop(). Add a failure-path test that verifies the mock
server’s stop method is invoked.
Defer Docker sandbox support for HTTP-mocked CLI evals to a follow-up. The sandbox path was unverified (needs a Docker build to confirm the real auth0 CLI accepts an IP:port tenant + trusts the baked CA), so keep it out of this PR and ship only the local --dangerously-skip-sandbox path. - docker/Dockerfile: drop the auth0 CLI install, mock-CA trust-store bake, and leaf-cert copy - sandbox-runner.ts: remove the mock lifecycle wiring; reject an httpRoutesDir eval with a clear 'run it locally' error instead - constants.ts: drop the now-unused SANDBOX_MOCK_CERT_DIR - docs + gen-mock-ca comments: reflect local-only (SSL_CERT_FILE) trust The core mock-http engine and the local run.ts path (isolated HOME, SSL_CERT_FILE) are unchanged; the committed mock-ca certs are still used by the local path.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/evals/src/cli/sandbox-runner.ts`:
- Around line 104-112: Add a Vitest regression test under the packages/evals
tests directory covering the sandbox-runner branch when evalDef.httpRoutesDir is
set. Assert that the runner writes an error result and that the error message
includes --dangerously-skip-sandbox --workers 1, while preserving existing
behavior for supported evaluations.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 61f10064-f0cf-4544-8407-ee5bc83b0892
📒 Files selected for processing (5)
AGENTS.mdapps/auth0-evals/scripts/gen-mock-ca.mjsdocs/ADDING_EVALS.mdpackages/evals-core/src/mock-http/README.mdpackages/evals/src/cli/sandbox-runner.ts
🚧 Files skipped from review as they are similar to previous changes (4)
- docs/ADDING_EVALS.md
- AGENTS.md
- packages/evals-core/src/mock-http/README.md
- apps/auth0-evals/scripts/gen-mock-ca.mjs
Comment on lines
+104
to
+112
| // HTTP-mocked CLI evals are not yet supported in the Docker sandbox (the | ||
| // image ships neither the auth0 CLI nor the mock CA). Run them locally with | ||
| // `--dangerously-skip-sandbox --workers 1`. | ||
| if (evalDef.httpRoutesDir) { | ||
| throw new Error( | ||
| `[sandbox] HTTP-mocked CLI eval '${evalId}' is not supported in the Docker sandbox yet; ` + | ||
| `run it locally with --dangerously-skip-sandbox --workers 1`, | ||
| ); | ||
| } |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Add a Vitest regression test for this branch.
evalDef.httpRoutesDir now changes the sandbox outcome. Add a Vitest test in the packages/evals package’s tests/ directory. Verify that the runner writes an error result and includes --dangerously-skip-sandbox --workers 1 in the message. The supplied changes contain no test for this logic.
As per coding guidelines, every new function and logic change must include Vitest tests in the tests/ directory of the package containing the changed code.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/evals/src/cli/sandbox-runner.ts` around lines 104 - 112, Add a
Vitest regression test under the packages/evals tests directory covering the
sandbox-runner branch when evalDef.httpRoutesDir is set. Assert that the runner
writes an error result and that the error message includes
--dangerously-skip-sandbox --workers 1, while preserving existing behavior for
supported evaluations.
Source: Coding guidelines
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Hermetic HTTP-level mocking for tenant-config evals. Instead of replacing the
auth0binary with a stub (the approach in #82), this runs the realauth0CLI and intercepts only its HTTPS calls to the Management API. Real command parsing and request construction are exercised end-to-end; only the network hop is faked — no auth, no network, no live side effects.Built fresh on
main(independent of the unmerged #82).Why interception is DNS+TLS, not config-pointing
The Auth0 CLI (Go) has no env var / flag to override the Management API base URL and no
--insecure. It always buildshttps://<domain>/api/v2/<path>from~/.config/auth0/config.json. So the mock works at the DNS+TLS layer:default_tenant: 127.0.0.1:8443, dummy token, far-future expiry) so the CLI targets the mock and skips login.127.0.0.1:8443(non-privileged loopback) with a leaf cert whose SAN is IP127.0.0.1.docker/mock-ca/) is trusted for the run viaSSL_CERT_FILE, so Go'snet/httpaccepts the mock's TLS without touching the system trust store. The CA signs one loopback leaf and guards nothing real.AUTH0_CLI_ANALYTICS=falsedisables telemetry.Components
packages/evals-core/src/mock-http/— declarative engine (verbscreate/set/reflect/static/handler, path normalization, filesystem-backed state for read-after-write), anhttpsserver, CLI-config seeding, and astartMockCliForEvallifecycle helper. Exported asmockHttp.http-routes/dir opts in by convention:EvalDefinition.httpRoutesDiris set and the shared CLI platform context (src/evals/contexts/cli-platform/AGENTS.md) is auto-attached. No frontmatter needed.run.ts(local--dangerously-skip-sandbox, isolated tempHOME, requires--workers 1) starts the mock before the agent and tears it down infinally. The sandbox runner rejects anhttpRoutesDireval with a clear "run it locally" error until Docker support lands.cli/guardian_otp_cli(enable Guardian OTP → confirm), graded with L4 event graders (ranCommand). File-less CLI eval, so no holistic judge (the trace-aware judge story is deferred).gen-mock-ca.mjscert generator;docs/ADDING_EVALS.md,AGENTS.md, andpackages/evals-core/src/mock-http/README.md.Deferred to a follow-up (Docker sandbox support)
The Docker path is not in this PR. To land it later:
auth0CLI indocker/Dockerfileand bake the mock CA into the container trust store; copy the leaf cert/key into the image.sandbox-runner.ts(replacing the current reject-with-error guard).auth0 api GET tenants/settingsagainst the mock to confirm the real binary accepts anIP:porttenant domain. If it rejectsIP:port, the fallback is a hostname +/etc/hostsentry (the leaf cert SAN already includesDNS:localhost).Testing
npm run build,npm run lint,npm run format, andnpm testall pass (evals-core 522, evals 717, reporter 75). New coverage: mock-http engine/verbs/state/manifest, a real-TLS server e2e (create → reflect read-after-write → fallthrough with the CA trusted), cli-config, lifecycle, and loaderhttp-routes/detection.🤖 Generated with Claude Code