diff --git a/CHANGELOG.md b/CHANGELOG.md index 90f9544..df9e7f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,16 @@ `--expect`; the MCP tools' run summaries list each journey's verdict; and `prescale compare` diffs each journey by name (terminal, `--json`, `--markdown`). A drop past the threshold in any journey counts as a regression for `--fail-on-regression`, even when it isn't the weakest journey. +- **Journeys in CI.** `prescale ci` and the GitHub Action replay the checkout's journeys when + `prescale/journeys/` exists. The PR comment shows each journey's verdict, interference, and + journeys that no longer replay (listed always; `--stale fail` / `stale: fail` fails the check). + A journey that gets no answer, or a 5xx with a single user, always fails the check (that's the + app, not a stale file). Response bodies never reach the PR comment or uploads; locally, + `journeys verify` shows them with every secret the journeys read masked. + New Action inputs: `stale` and `allow-writes` (the action only passes them when set, so a + pinned older `prescale-version` keeps working). Until the dashboard compares journey runs only + with journey baselines, a journey run's PR comment shows no change instead of a possibly + misleading one, and never fails on it. - **`--expect N`** on `run` says whether you'd hold N users at peak. In a journey run, N is split across journeys by weight: each must hold its share alone, and all together must hold N. diff --git a/README.md b/README.md index 19d7d27..1515994 100644 --- a/README.md +++ b/README.md @@ -219,6 +219,8 @@ Interference checkout broke with 15 of its own users, but held 20 alone. `prescale show`, `--html` reports and the MCP tools show journeys too, and `prescale compare` diffs each journey: a drop past the threshold in any journey is a regression (so `--fail-on-regression` catches it), even when it isn't the weakest one; it's marked confident when the confidence bands don't overlap. A journey that held everything it was tested to, below the old level, is inconclusive, not a drop. A URL run and a journey run measure different things, so comparing them is reported as not comparable. +In CI, `prescale ci` (and the GitHub Action) replays the checkout's journeys instead of URLs whenever `prescale/journeys/` exists, and the PR comment shows each journey's verdict, any interference, and journeys that no longer replay. Those are always listed; `--stale fail` (Action input `stale: fail`) also fails the check. With `app:` the replica is local, so journeys that change data can run; against a preview host they need `allow-writes`. Journey secrets (`{{env.NAME}}`) come from the step's `env:`. + `--no-combined` skips this pass. `--expect 500` says whether you'd hold 500 users at peak: in a journey run, each journey must hold its share of them (by weight) on its own, and all journeys together must hold 500; a journey that wasn't load-tested makes the answer unknown. It reports, it doesn't change the exit code (use `--fail-under` to gate CI). - `{{name}}` is a value an earlier step pulled out with `extract` (`body.` or `header.`); `{{unique.email}}` and `{{unique.id}}` are fresh each run; `{{env.NAME}}` reads an environment variable. diff --git a/action/README.md b/action/README.md index cf589e5..9ba1f23 100644 --- a/action/README.md +++ b/action/README.md @@ -78,6 +78,25 @@ jobs: fail-on-regression: true ``` +## Journeys + +If the checkout has journey files (`prescale/journeys/*.yaml`, see the main README), the action replays them at load instead of testing URLs: each journey alone, then all together. The PR comment shows each journey's verdict, any interference, and journeys that no longer replay against the build. Give journeys their secrets through the step's `env:` (for `{{env.NAME}}`): + +```yaml + - uses: pyjeebz/prescale/action@v1 + with: + token: ${{ secrets.PRESCALE_TOKEN }} + target: myapp.com + app: npm run build && npm start + stale: fail + env: + SHOP_PASSWORD: ${{ secrets.STAGING_TEST_PASSWORD }} +``` + +A journey that **no longer replays** (the app answers differently, e.g. a 404) is listed on the PR; `stale: fail` also fails the check. A journey that gets **no answer, or a 5xx with a single user**, always fails the check: that's the app broken or down, not a stale file. Response bodies never appear in the PR comment. + +Deploy workflows (`on: deployment_status`) that also check out the code replay journeys too. Journeys that change data then need the preview's host in `allow-writes`; preview hosts change per deploy, so prefer replica mode (`app:`) for journeys that write. + ## How it works 1. A pull request is pushed (replica mode) or a deploy finishes (`deployment_status`). @@ -122,5 +141,7 @@ uploaded is the run's Result file, which you can inspect with | `app` | — | Replica mode: build-and-start command, tested at `url` (default `http://localhost:3000`) | | `app-timeout` | `600` | Seconds to wait for the app to answer | | `working-directory` | `.` | Where to run `app` (monorepos) | +| `stale` | `warn` | Journey runs: a journey that no longer replays is always listed on the PR; `fail` also fails the check | +| `allow-writes` | — | Journey runs: hosts (one per line) where journeys that change data may run; not needed with `app` | | `app-url` | `https://prescale.dev/dashboard` | Dashboard URL | | `prescale-version` | latest | CLI version | diff --git a/action/action.yml b/action/action.yml index 9fcf082..ff1b6e0 100644 --- a/action/action.yml +++ b/action/action.yml @@ -48,6 +48,14 @@ inputs: description: Confirm the deploy's host allows load testing. Vercel only permits it on Enterprise plans with notice, and Netlify and Cloudflare need it arranged; without this, PreScale skips their deploys and says why on the PR. required: false default: "false" + stale: + description: "Journey runs (when the checkout has prescale/journeys/): 'warn' or 'fail'. A journey that no longer replays against this build is always listed on the PR; 'fail' also fails the check." + required: false + default: "warn" + allow-writes: + description: "Journey runs: hosts (one per line, or comma-separated) where journeys that change data may run, e.g. a staging host. Not needed with app: the replica is local. Journeys read their secrets from env variables you set on this step." + required: false + default: "" app-url: description: PreScale dashboard URL. required: false @@ -105,6 +113,8 @@ runs: INPUT_HOST_PERMISSION: ${{ inputs.host-permission }} INPUT_APP: ${{ inputs.app }} INPUT_APP_TIMEOUT: ${{ inputs.app-timeout }} + INPUT_STALE: ${{ inputs.stale }} + INPUT_ALLOW_WRITES: ${{ inputs.allow-writes }} run: | args=(--max-users "$INPUT_MAX_USERS" --stage-seconds "$INPUT_STAGE_SECONDS" --store "$RUNNER_TEMP/prescale") [ "$INPUT_FAIL" = "true" ] && args+=(--fail-on-regression) @@ -112,5 +122,12 @@ runs: [ "$INPUT_ALLOW_PRODUCTION" = "true" ] && args+=(--allow-production) [ "$INPUT_HOST_PERMISSION" = "true" ] && args+=(--host-permission) [ -n "$INPUT_APP" ] && args+=(--app "$INPUT_APP" --app-timeout "$INPUT_APP_TIMEOUT") + # New flags only when set, so a pinned older prescale-version keeps working. + stale="${INPUT_STALE,,}" + [ -n "$stale" ] && [ "$stale" != "warn" ] && args+=(--stale "$stale") + while IFS= read -r host; do + host="${host//[[:space:]]/}" + [ -n "$host" ] && args+=(--allow-writes "$host") + done <<< "${INPUT_ALLOW_WRITES//,/$'\n'}" [ -n "$INPUT_URL" ] && args+=("$INPUT_URL") prescale ci "${args[@]}" diff --git a/cli/src/prescale_cli/ci.py b/cli/src/prescale_cli/ci.py index dfc8385..08929e3 100644 --- a/cli/src/prescale_cli/ci.py +++ b/cli/src/prescale_cli/ci.py @@ -16,6 +16,7 @@ import httpx from prescale_cli.bars import bar_rows, route_rows +from prescale_cli.journeys.load import LABELS from prescale_cli.throughput import VISITOR_NOTE, capacity_rps, visitors_for _SHA = re.compile(r"^[0-9a-f]{40}$") @@ -181,11 +182,91 @@ def render_host_skip(rule, *, target: str) -> str: ]) +def comparison_for(result: dict, upload: dict) -> dict | None: + """The dashboard's comparison, if it compared like with like. A journey run is + only comparable with a journey baseline; a dashboard that doesn't say which + kind of run it compared against (`mode`) may have used a URL run, so its + comparison isn't trusted for journey runs (never a false regression).""" + comparison = upload.get("comparison") + if not comparison: + return None + if _mode(result) == "journeys" and comparison.get("mode") != "journeys": + return None + return comparison + + +def _mode(result: dict) -> str: + return (result.get("config") or {}).get("mode") or "urls" + + +def _cell(text) -> str: + """Text from journey files or the app, safe in a PR comment: it can't break the + table, render HTML, or @mention (and so notify) anyone.""" + return (str(text).replace("|", "\\|").replace("\n", " ").replace("<", "<") + .replace(">", ">").replace("@", "@\u200b")) + + +def _code(text) -> str: + """TEXT as inline code that a backtick inside can't break out of.""" + return "`" + _cell(text).replace("`", "'") + "`" + + +def render_journeys(result: dict) -> list[str]: + """Journey runs: each journey's verdict, the stale ones, and interference.""" + journeys = result.get("journeys") or [] + if not journeys: + return [] + lines = ["", "**Journeys** (each ramped on its own)", "", + "| journey | survives | breaks at step | why |", "|---|---:|---|---|"] + stale = [] + for j in journeys: + if j["status"] != "ok": + stale.append(j) + continue + survives = (f"{j['max_tested']}+ (held)" if j["onset_users"] is None + else f"~{j['survives_users']}") + step = j.get("culprit_step") + where = f"{step['index']}. {_cell(step['name'])} {_code(step['route'])}" if step else "—" + lines.append(f"| {_cell(j['name'])} | {survives} | {where} | " + f"{_cell(j.get('bottleneck') or '')} |") + if stale: + lines += ["", "**Not load-tested** (they don't replay against this build, so the app " + "changed or it's broken; update the journey files if the change was on " + "purpose):"] + lines += [f"- ⚠️ **{_cell(j['name'])}** ({j['status']}): {_cell(j.get('reason') or '')}" + for j in stale] + inter = (result.get("combined") or {}).get("interference") + if inter: + lines += ["", f"**Interference:** {_cell(inter['journey'])} broke with " + f"{inter['users_at_break']} of its own users, but held " + f"{inter['held_alone']} alone. It competes with " + f"{_cell(', '.join(inter['with']))} for something they share " + "(a database pool, CPU, a cache)."] + return lines + + +def render_stale_only(runs, *, target: str) -> str: + """The PR comment when no journey replays, so nothing was load-tested.""" + lines = [comment_marker(target), "### ⚠️ PreScale: no journey replays against this build", "", + "Every journey failed with a single user, so there's no capacity to measure. " + "If the app changed on purpose, update the journey files (`prescale journeys " + "verify` shows what changed); if not, it's a bug.", ""] + for run in runs: + failed = run.check.failed + index = run.check.steps.index(failed) + 1 if failed else "?" + what = LABELS[run.check.outcome] + lines.append(f"- **{_cell(run.journey.name)}** ({what}): step {index} " + f"({_cell(failed.step.name if failed else '?')}): " + f"{_cell(failed.problem if failed else '')}") + return "\n".join(lines) + "\n" + + def render_comment(result: dict, upload: dict, *, target: str, sha: str | None) -> str: """The PR comment: the change vs baseline, what breaks first, why, the fix.""" verdict = result["verdict"] conf = verdict.get("confidence") or {} - comparison = upload.get("comparison") + comparison = comparison_for(result, upload) + unmatched = comparison is None and bool(upload.get("comparison")) target_info = upload.get("target") or {} baseline_branch = target_info.get("baselineBranch") or "main" tested_host = result["target"].get("host") or result["target"]["url"] @@ -238,13 +319,20 @@ def render_comment(result: dict, upload: dict, *, target: str, sha: str | None) lines += ["", "The confidence bands overlap, so this may be noise. Another run " "will confirm or clear it."] else: - lines.append(f"### ⚪ First capacity run: survives {survives}") + title = "Capacity" if unmatched else "First capacity run" + lines.append(f"### ⚪ {title}: survives {survives}") lines.append(f"`{tested_host}` · **{target}**") if rps: lines += ["", f"That's ≈ {rps:,.0f} req/s, roughly {visitors_for(rps):,} real " f"visitors. Visitors assume {VISITOR_NOTE}."] - lines += ["", f"No `{baseline_branch}` baseline yet. Later pushes to this PR compare " - f"with this run; once `{baseline_branch}` has a run, PRs compare with it."] + if unmatched: + lines += ["", "This run replayed journeys. The baseline it would be compared with " + "may have tested URLs, which measures something else, so no change is " + "shown."] + else: + lines += ["", f"No `{baseline_branch}` baseline yet. Later pushes to this PR " + f"compare with this run; once `{baseline_branch}` has a run, PRs " + "compare with it."] investigation = result.get("investigation") lines.append("") @@ -252,7 +340,9 @@ def render_comment(result: dict, upload: dict, *, target: str, sha: str | None) lines.append(f"**Held:** {_onset(verdict)} without crossing a failure threshold.") else: culprit = verdict.get("culprit_route") - lines.append(f"**Breaks first:** `{culprit}` · {_onset(verdict)}" if culprit + journey = verdict.get("culprit_journey") + where = f"{_cell(journey)} › {_code(culprit)}" if journey and culprit else _code(culprit) + lines.append(f"**Breaks first:** {where} · {_onset(verdict)}" if culprit else f"**Breaks first:** {_onset(verdict)}") if investigation: lines.append(f"**Why:** `{investigation['bottleneck_class']}` " @@ -267,6 +357,8 @@ def render_comment(result: dict, upload: dict, *, target: str, sha: str | None) caption = f"p95 per route at {stage['users']} users · ┆ = the {wall:g}s latency wall" lines += ["", f"{caption}", "", "```text", *route_rows(stage, wall), "```"] + lines += render_journeys(result) + links = [f"[Open the run]({upload['url']})"] if upload.get("url") else [] if target_info.get("url"): links.append(f"[Capacity timeline]({target_info['url']})") diff --git a/cli/src/prescale_cli/commands/ci.py b/cli/src/prescale_cli/commands/ci.py index 197a9e0..1626655 100644 --- a/cli/src/prescale_cli/commands/ci.py +++ b/cli/src/prescale_cli/commands/ci.py @@ -1,7 +1,8 @@ """`prescale ci` - capacity-check a deploy from CI and report it on the PR. Built for GitHub Actions (the PreScale action wraps it), usable from any CI: -investigate the deploy URL, upload the Result, compare it with the baseline +investigate the deploy URL (or, when the checkout has prescale/journeys/, +replay the journeys at load), upload the Result, compare it with the baseline (server-side, the same rule as the dashboard), write a job summary, and keep one capacity comment on the PR up to date. """ @@ -18,18 +19,25 @@ from prescale_cli.ci import ( comment_marker, + comparison_for, find_pull_request, github_context, render_comment, render_host_skip, + render_stale_only, upsert_pr_comment, ) from prescale_cli.cloud import CloudError, parse_headers, require_credentials, upload_result +from prescale_cli.commands.journeys import dir_option, refuse_missing_env, refuse_unallowed_writes from prescale_cli.commands.replicaopts import app_option, app_timeout_option, start_replica +from prescale_cli.commands.run import run_journeys from prescale_cli.hosts import PERMISSION_ENV from prescale_cli.investigate import investigate as run_investigate +from prescale_cli.journeys.load import NoJourneyReplaysError +from prescale_cli.journeys.model import load_dir from prescale_cli.live import ramp_progress -from prescale_cli.loadtest import HostPolicyError, LoadError, default_levels +from prescale_cli.loadtest import HostPolicyError, LoadError, default_levels, probe_host +from prescale_cli.result import write_result console = Console() @@ -67,13 +75,22 @@ def _annotate(level: str, message: str) -> None: @click.option("--host-permission", "host_permission", is_flag=True, envvar=PERMISSION_ENV, help="Confirm the deploy's host allows load testing (Vercel, Netlify and " f"Cloudflare restrict it; otherwise PreScale skips). Env: {PERMISSION_ENV}=1.") +@click.option("--stale", "stale", type=click.Choice(["warn", "fail"]), default="warn", + show_default=True, + help="Journey runs: what a journey that no longer replays does to the check. It's " + "always reported on the PR; 'fail' also fails the job.") +@click.option("--allow-writes", "allow_writes", multiple=True, metavar="HOST", + help="Let journeys that change data run against this non-local host " + "(repeatable). Not needed with --app: the replica is local.") +@dir_option @app_option @app_timeout_option @click.option("--store", "store", type=click.Path(file_okay=False), default=None, help="Directory for saved runs (default: ./.prescale).") def ci(url: str | None, target: str, paths: tuple[str, ...], header_values: tuple[str, ...], max_users: int, stage_seconds: float, max_rps: float | None, fail_on_regression: bool, - comment: bool, allow_production: bool, host_permission: bool, app: str | None, + comment: bool, allow_production: bool, host_permission: bool, stale: str, + allow_writes: tuple[str, ...], directory: Path, app: str | None, app_timeout: float, store: str | None) -> None: """Capacity-check a deploy from CI, upload it, and comment on the PR. @@ -92,6 +109,9 @@ def ci(url: str | None, target: str, paths: tuple[str, ...], header_values: tupl \b Example (replica mode, on: pull_request): prescale ci --target myapp.com --app "npm run build && npm start" + + When prescale/journeys/ (or --dir) has journeys, they're replayed at load + instead of the URL routes, and the PR comment shows each journey's verdict. """ ctx = github_context() @@ -110,6 +130,28 @@ def ci(url: str | None, target: str, paths: tuple[str, ...], header_values: tupl if not url: raise click.UsageError("No URL to test. Pass one, or run on a deployment_status event.") + # A checkout with journey files replays them; without one (a deploy event with no + # checkout) it's a URL run as before. Broken files and missing secrets stop the + # job here, before an app is built. + journeys, errors = load_dir(directory) + for error in errors: + console.print(f"[red]Error:[/red] {error}", markup=False) + _annotate("error", f"PreScale: {error}") + if errors: + raise SystemExit(2) + if journeys: + host = (httpx.URL(url).host or "").lower() + try: + refuse_unallowed_writes(journeys, host, {h.lower() for h in allow_writes}) + refuse_missing_env(journeys) + except SystemExit: + _annotate("error", "PreScale refused to replay the journeys (see the log: " + "journeys that change data need allow-writes for this host, " + "and every {{env.NAME}} they read must be set).") + raise + if paths or os.environ.get("PRESCALE_PATHS"): + console.print("[dim]Journeys are replayed, so --path / paths are ignored.[/dim]") + try: headers = parse_headers(header_values + tuple(_lines(os.environ.get("PRESCALE_HEADERS")))) creds = require_credentials() @@ -136,6 +178,14 @@ def ci(url: str | None, target: str, paths: tuple[str, ...], header_values: tupl if app: start_replica(app, url, timeout=app_timeout, console=console, quiet=False) + if journeys: + _ci_journeys(journeys, url, target=target, ctx=ctx, creds=creds, headers=headers, + max_users=max_users, stage_seconds=stage_seconds, max_rps=max_rps, + store=store, comment=comment, pr_number=pr_number, + github_token=github_token, host_permission=host_permission, + replica=bool(app), stale=stale, fail_on_regression=fail_on_regression) + return + extra_paths = tuple(paths) + tuple(_lines(os.environ.get("PRESCALE_PATHS"))) console.print(f"[bold]PreScale[/bold] — capacity check of [cyan]{url}[/cyan] for " f"[bold]{target}[/bold]" @@ -163,28 +213,82 @@ def ci(url: str | None, target: str, paths: tuple[str, ...], header_values: tupl _annotate("error", f"PreScale couldn't load-test {url}: {exc}") raise SystemExit(1) from None + if _report(result, url=url, target=target, ctx=ctx, creds=creds, comment=comment, + pr_number=pr_number, github_token=github_token, + fail_on_regression=fail_on_regression): + raise SystemExit(1) + + +def _ci_journeys(journeys, url: str, *, target: str, ctx, creds, headers: dict[str, str], + max_users: int, stage_seconds: float, max_rps: float | None, + store: str | None, comment: bool, pr_number: int | None, + github_token: str | None, host_permission: bool, replica: bool, stale: str, + fail_on_regression: bool) -> None: + """Replay the checkout's journeys at load, upload, comment, and gate.""" + # Journey ramps skip the preflight that checks the host, so check it here. + if not replica and not host_permission: + rule = asyncio.run(probe_host(url, headers=headers)) + if rule is not None: + _skip_restricted_host(rule, target=target, ctx=ctx, comment=comment, + pr_number=pr_number, github_token=github_token) + return + console.print(f"[bold]PreScale[/bold] — journey check for [bold]{target}[/bold]" + + (f" (PR #{pr_number})" if pr_number else "")) + config = {"method": "GET", "max_users": max_users, "stage_seconds": stage_seconds, + "latency_wall_s": 2.0, "error_threshold": 0.02, "max_rps": max_rps, + "warmup": True, "repeat": 1, "think_time_s": 0.0, "fail_under": None, + "profile": None} try: - upload = upload_result(result, creds, pr=pr_number, target=target) - except CloudError as exc: - console.print(f"[red]✗ upload failed:[/red] {exc}") - _annotate("error", f"PreScale upload failed: {exc}") + _, result, _ = run_journeys(journeys, url, config, latency_wall=2.0, + error_threshold=0.02, timeout=10.0, headers=headers, + replica=replica, as_json=False) + except NoJourneyReplaysError as exc: + body = render_stale_only(exc.runs, target=target) + _publish(body, target=target, ctx=ctx, comment=comment, pr_number=pr_number, + github_token=github_token) + # Only "stale" (the app answered differently) follows --stale. An app that + # doesn't answer, or fails with a single user, fails the check like URL mode. + broken = [r.journey.name for r in exc.runs if r.check.outcome != "stale"] + if broken: + _annotate("error", f"PreScale: {', '.join(broken)} failed with a single user " + "(no answer, or a 5xx), so nothing was load-tested.") + raise SystemExit(1) + level = "error" if stale == "fail" else "warning" + _annotate(level, "PreScale: no journey replays against this build, so nothing was " + "load-tested. Update the journey files if the app changed on purpose.") + if stale == "fail": + raise SystemExit(1) + return + except LoadError as exc: + console.print(f"[red]Error:[/red] {exc}") + _annotate("error", f"PreScale couldn't load-test {url}: {exc}") raise SystemExit(1) from None + write_result(result, store=store) + + stale_names = [j["name"] for j in result["journeys"] if j["status"] == "stale"] + broken = [j["name"] for j in result["journeys"] if j["status"] in ("error", "unreachable")] + regressed = _report(result, url=url, target=target, ctx=ctx, creds=creds, comment=comment, + pr_number=pr_number, github_token=github_token, + fail_on_regression=fail_on_regression) + if stale_names: + level = "error" if stale == "fail" else "warning" + what = ("replays against this build, so it wasn't" if len(stale_names) == 1 + else "replay against this build, so they weren't") + _annotate(level, f"PreScale: {', '.join(stale_names)} no longer {what} load-tested.") + if broken: + _annotate("error", f"PreScale: {', '.join(broken)} failed with a single user (no " + "answer, or a 5xx), so they weren't load-tested.") + if regressed or broken or (stale_names and stale == "fail"): + raise SystemExit(1) - body = render_comment(result, upload, target=target, sha=ctx.sha) - comparison = upload.get("comparison") - verdict = result["verdict"] - sent = sum(stage.get("total", 0) for stage in result.get("stages", [])) - console.print(f"[dim]Sent {sent:,} requests while measuring.[/dim]") - console.print(f"survives ~{verdict['survives_users']} users" - + (f" · {comparison['kind']} {round(comparison['pct'] * 100):+d}% vs " - f"{comparison['against']}" if comparison else " · first run") - + f"\n[green]✓[/green] uploaded → [cyan]{upload.get('url')}[/cyan]") +def _publish(body: str, *, target: str, ctx, comment: bool, pr_number: int | None, + github_token: str | None) -> None: + """The job summary, and the one PreScale comment on the PR.""" summary_path = os.environ.get("GITHUB_STEP_SUMMARY") if summary_path: with Path(summary_path).open("a", encoding="utf-8") as fh: fh.write(body + "\n") - if comment and pr_number and github_token and ctx.repository: try: action = upsert_pr_comment(ctx, github_token, pr_number, body, comment_marker(target)) @@ -195,15 +299,42 @@ def ci(url: str | None, target: str, paths: tuple[str, ...], header_values: tupl elif comment and not pr_number: console.print("[dim]No open PR for this commit, so no PR comment.[/dim]") + +def _report(result: dict, *, url: str, target: str, ctx, creds, comment: bool, + pr_number: int | None, github_token: str | None, + fail_on_regression: bool) -> bool: + """Upload the run, comment on the PR, and annotate a regression. Returns + whether the job should fail (a regression with --fail-on-regression); the + caller exits, after any notes of its own.""" + try: + upload = upload_result(result, creds, pr=pr_number, target=target) + except CloudError as exc: + console.print(f"[red]✗ upload failed:[/red] {exc}") + _annotate("error", f"PreScale upload failed: {exc}") + raise SystemExit(1) from None + + body = render_comment(result, upload, target=target, sha=ctx.sha) + comparison = comparison_for(result, upload) + verdict = result["verdict"] + sent = sum(stage.get("total", 0) for stage in result.get("stages", [])) + console.print(f"[dim]Sent {sent:,} requests while measuring.[/dim]") + console.print(f"survives ~{verdict['survives_users']} users" + + (f" · {comparison['kind']} {round(comparison['pct'] * 100):+d}% vs " + f"{comparison['against']}" if comparison else " · first run") + + f"\n[green]✓[/green] uploaded → [cyan]{upload.get('url')}[/cyan]") + + _publish(body, target=target, ctx=ctx, comment=comment, pr_number=pr_number, + github_token=github_token) + kind = comparison["kind"] if comparison else None if kind == "regressed": _annotate("error", f"Capacity regressed {round(comparison['pct'] * 100)}% vs " f"{comparison['against']}.") - if fail_on_regression: - raise SystemExit(1) - elif kind == "possible": + return fail_on_regression + if kind == "possible": _annotate("warning", f"Possible capacity regression {round(comparison['pct'] * 100)}% " f"vs {comparison['against']} (bands overlap).") + return False def _skip_restricted_host(rule, *, target: str, ctx, comment: bool, pr_number: int | None, diff --git a/cli/src/prescale_cli/commands/journeys.py b/cli/src/prescale_cli/commands/journeys.py index 5918f59..f5cb898 100644 --- a/cli/src/prescale_cli/commands/journeys.py +++ b/cli/src/prescale_cli/commands/journeys.py @@ -96,12 +96,16 @@ def verify(url: str, directory: Path, only: tuple[str, ...], allow_writes: tuple broken = [r for r in results if not r.ok] if broken: - stale = [r.journey.name for r in broken if r.stale] + stale = [r.journey.name for r in broken if r.outcome == "stale"] if stale: console.print(f"\n[yellow]{len(stale)} stale:[/yellow] {', '.join(stale)}. The app " "no longer answers the way these journeys expect. If the change was " "on purpose, update the journey files; if not, it's a bug.") - unreached = [r.journey.name for r in broken if not r.stale] + errors = [r.journey.name for r in broken if r.outcome == "error"] + if errors: + console.print(f"\n[red]{len(errors)} failing with a 5xx:[/red] {', '.join(errors)}. " + "With no load, that's a bug in the app, not a stale journey.") + unreached = [r.journey.name for r in broken if r.outcome == "unreachable"] if unreached: console.print(f"\n[red]{len(unreached)} couldn't run:[/red] {', '.join(unreached)}.") raise SystemExit(1) @@ -151,8 +155,8 @@ async def _check_all(found: list[Journey], base: str, *, timeout: float, def _print_check(result: JourneyCheck) -> None: journey = result.journey - mark = "[green]✓[/green]" if result.ok else ("[yellow]stale[/yellow]" if result.stale - else "[red]✗[/red]") + mark = {"ok": "[green]✓[/green]", "stale": "[yellow]stale[/yellow]"}.get( + result.outcome, "[red]✗[/red]") console.print(f"\n{mark} [bold]{escape(journey.name)}[/bold] [dim]({journey.type}, " f"{len(journey.steps)} steps)[/dim]") for i, step in enumerate(journey.steps, 1): @@ -166,6 +170,6 @@ def _print_check(result: JourneyCheck) -> None: else: console.print(f" {i}. {name} [dim]{label}[/dim] " f"[red]{escape(done.problem or '')}[/red]", highlight=False) - if done.status is not None and done.status >= 500: - console.print(" [dim]A 5xx with no load is usually a bug in the app, " - "not a stale journey.[/dim]") + if done.snippet: + # Local terminal only, secrets masked; never in CI output or uploads. + console.print(f" [dim]{escape(done.snippet)}[/dim]", highlight=False) diff --git a/cli/src/prescale_cli/commands/run.py b/cli/src/prescale_cli/commands/run.py index 0dbe69c..d3da268 100644 --- a/cli/src/prescale_cli/commands/run.py +++ b/cli/src/prescale_cli/commands/run.py @@ -22,7 +22,9 @@ from prescale_cli.commands.replicaopts import app_option, app_timeout_option, start_replica from prescale_cli.commands.upload import header_option, upload_after_run, upload_option from prescale_cli.journeys.load import ( + LABELS, JourneyRun, + NoJourneyReplaysError, combined_block, first_to_break, journeys_block, @@ -205,10 +207,19 @@ def run(url: str, paths: tuple[str, ...], from_sitemap: bool, max_users: int, as_json=as_json, upload=upload, fail_under=fail_under, expect=expect) if journeys: - report, result, live_mode = _run_journeys( - journeys, url, config, latency_wall=latency_wall, - error_threshold=error_threshold, timeout=timeout, headers=headers, - replica=bool(app), as_json=as_json, combined=not no_combined) + try: + report, result, live_mode = run_journeys( + journeys, url, config, latency_wall=latency_wall, + error_threshold=error_threshold, timeout=timeout, headers=headers, + replica=bool(app), as_json=as_json, combined=not no_combined) + except LoadError as exc: + (Console(stderr=True) if as_json else console).print(f"[red]Error:[/red] {exc}") + raise SystemExit(1) + except NoJourneyReplaysError: + (Console(stderr=True) if as_json else console).print( + "[red]Error:[/red] no journey replays against this app, so there's nothing " + "to load-test. Run `prescale journeys verify` for details.") + raise SystemExit(1) _finish(result, report, live_mode=live_mode, **finish) return @@ -373,12 +384,14 @@ def _journeys_to_run(directory: Path, only: tuple[str, ...], *, urls_only: bool, return found -def _run_journeys(journeys: list[Journey], url: str, config: dict, *, latency_wall: float, +def run_journeys(journeys: list[Journey], url: str, config: dict, *, latency_wall: float, error_threshold: float, timeout: float, headers: dict[str, str], replica: bool, as_json: bool, combined: bool = True) -> tuple: """The isolation pass: check each journey once, then ramp each one that - replays on its own. Returns (report, result, live_mode) for the first - journey to break, with every journey's verdict in result["journeys"].""" + replays on its own; then, with two or more, the combined pass. Returns + (report, result, live_mode) for the first journey to break, with every + journey's verdict in result["journeys"]. Raises NoJourneyReplaysError if no + journey replays, and LoadError if a ramp can't run. Shared by `run` and `ci`.""" if not as_json: console.print(f"\n[bold]PreScale[/bold] — replaying {len(journeys)} " f"journey{'s' if len(journeys) != 1 else ''} against [cyan]{url}[/cyan]") @@ -392,7 +405,7 @@ def _run_journeys(journeys: list[Journey], url: str, config: dict, *, latency_wa name = escape(run.journey.name) if not run.check.ok: failed = run.check.failed - what = "stale" if run.check.stale else "couldn't run" + what = LABELS[run.check.outcome] out.print(f"[yellow]⚠ {name}: {what}, not load-tested[/yellow] " f"[dim]({escape(failed.step.name)}: {escape(failed.problem or '')})" "[/dim]", highlight=False) @@ -402,25 +415,19 @@ def _run_journeys(journeys: list[Journey], url: str, config: dict, *, latency_wa f"{len(run.journey.steps)} steps)[/dim]", highlight=False) progress = ramp_progress(console, latency_wall=latency_wall, as_json=as_json) live_mode = isinstance(progress, LiveRamp) - try: - with progress: - stages, run.warning = asyncio.run(ramp( - run.journey, url, levels=levels, stage_seconds=config["stage_seconds"], - timeout=timeout, max_rps=config["max_rps"], warmup=config["warmup"], - repeat=config["repeat"], headers=headers, think=config["think_time_s"], - progress_cb=progress.starting, on_stage=progress.finished)) - except LoadError as exc: - out.print(f"[red]Error:[/red] {exc}") - raise SystemExit(1) + with progress: + stages, run.warning = asyncio.run(ramp( + run.journey, url, levels=levels, stage_seconds=config["stage_seconds"], + timeout=timeout, max_rps=config["max_rps"], warmup=config["warmup"], + repeat=config["repeat"], headers=headers, think=config["think_time_s"], + progress_cb=progress.starting, on_stage=progress.finished)) run.report = analyze(stages, latency_wall=latency_wall, error_threshold=error_threshold, rate_capped=config["max_rps"] is not None) worst = first_to_break(runs) if worst is None: - out.print("[red]Error:[/red] no journey replays against this app, so there's " - "nothing to load-test. Run `prescale journeys verify` for details.") - raise SystemExit(1) + raise NoJourneyReplaysError(runs) warnings = list(dict.fromkeys(r.warning for r in runs if r.warning)) result = build_result(worst.report, url=url, targets=[s.label for s in worst.journey.steps], @@ -439,17 +446,13 @@ def _run_journeys(journeys: list[Journey], url: str, config: dict, *, latency_wa highlight=False) progress = ramp_progress(console, latency_wall=latency_wall, as_json=as_json) live_mode = isinstance(progress, LiveRamp) - try: - with progress: - stages, together_warning = asyncio.run(ramp( - [r.journey for r in ramped], url, levels=levels, - stage_seconds=config["stage_seconds"], timeout=timeout, - max_rps=config["max_rps"], warmup=config["warmup"], - repeat=config["repeat"], headers=headers, think=config["think_time_s"], - progress_cb=progress.starting, on_stage=progress.finished)) - except LoadError as exc: - out.print(f"[red]Error:[/red] {exc}") - raise SystemExit(1) + with progress: + stages, together_warning = asyncio.run(ramp( + [r.journey for r in ramped], url, levels=levels, + stage_seconds=config["stage_seconds"], timeout=timeout, + max_rps=config["max_rps"], warmup=config["warmup"], + repeat=config["repeat"], headers=headers, think=config["think_time_s"], + progress_cb=progress.starting, on_stage=progress.finished)) together = analyze(stages, latency_wall=latency_wall, error_threshold=error_threshold, rate_capped=config["max_rps"] is not None) result["combined"] = combined_block(together, runs) diff --git a/cli/src/prescale_cli/journeys/load.py b/cli/src/prescale_cli/journeys/load.py index 9cd24a8..ae8920e 100644 --- a/cli/src/prescale_cli/journeys/load.py +++ b/cli/src/prescale_cli/journeys/load.py @@ -27,6 +27,17 @@ ) from prescale_cli.loadtest import RunReport, StageResult, run_loadtest +# How each non-ok outcome reads in reports. +LABELS = {"stale": "stale", "error": "5xx with one user", "unreachable": "couldn't run"} + + +class NoJourneyReplaysError(Exception): + """Every journey was stale or unreachable, so nothing could be load-tested.""" + + def __init__(self, runs: list[JourneyRun]) -> None: + super().__init__("no journey replays against this app") + self.runs = runs + @dataclass class JourneyRun: @@ -39,7 +50,7 @@ class JourneyRun: def status(self) -> str: if self.report is not None: return "ok" - return "stale" if self.check.stale else "unreachable" + return self.check.outcome async def ramp(journey: Journey | list[Journey], base_url: str, *, levels: list[int], diff --git a/cli/src/prescale_cli/journeys/model.py b/cli/src/prescale_cli/journeys/model.py index 699133f..e98fc51 100644 --- a/cli/src/prescale_cli/journeys/model.py +++ b/cli/src/prescale_cli/journeys/model.py @@ -53,7 +53,7 @@ _PLACEHOLDER = re.compile(r"\{\{\s*([A-Za-z_][\w.]*)\s*\}\}") # A key is secret-looking when it is, or ends with, one of these words: # password, client_secret, x-api-key, access_token, Authorization, Cookie. -_SENSITIVE_KEY = re.compile( +SENSITIVE_KEY = re.compile( r"(^|[-_.])(password|passwd|pwd|secret|token|api[-_]?key|authorization|cookie)$", re.IGNORECASE) # The secret itself must be a placeholder, optionally after an auth scheme. @@ -279,7 +279,7 @@ def _check_secrets(value: Any, where: str, key: str = "") -> None: elif isinstance(value, list): for v in value: _check_secrets(v, where, key) - elif value is not None and key and _SENSITIVE_KEY.search(key) \ + elif value is not None and key and SENSITIVE_KEY.search(key) \ and not (isinstance(value, str) and _SECRET_OK.fullmatch(value.strip())): raise JourneyError(f"{where}: {key} has a literal value. Put secrets in an " f"environment variable and write \"{{{{env.NAME}}}}\" instead") diff --git a/cli/src/prescale_cli/journeys/replay.py b/cli/src/prescale_cli/journeys/replay.py index 633f226..d6e4f7b 100644 --- a/cli/src/prescale_cli/journeys/replay.py +++ b/cli/src/prescale_cli/journeys/replay.py @@ -25,7 +25,14 @@ import httpx -from prescale_cli.journeys.model import Journey, Step, TemplateError, fresh_uniques, render +from prescale_cli.journeys.model import ( + SENSITIVE_KEY, + Journey, + Step, + TemplateError, + fresh_uniques, + render, +) from prescale_cli.loadtest import client_headers _SNIPPET = 160 @@ -76,8 +83,11 @@ class StepCheck: status: int | None = None latency: float = 0.0 ok: bool = False - problem: str | None = None # why it failed, in plain words + problem: str | None = None # why it failed, in plain words (no response body) kind: str | None = None # a short failure kind for load reports + # The start of the response body, secrets masked. Shown in a local terminal + # only: never in CI output, a PR comment or an uploaded Result. + snippet: str | None = None headers: httpx.Headers | None = None @@ -95,10 +105,20 @@ def failed(self) -> StepCheck | None: return next((s for s in self.steps if not s.ok), None) @property - def stale(self) -> bool: - """The app answered, but not the way the journey expects.""" + def outcome(self) -> str: + """ok; stale (the app answered, but not as the journey expects, e.g. a + 404: the app changed); error (a 5xx with a single user: the app is + broken); unreachable (no answer at all).""" failed = self.failed - return failed is not None and failed.status is not None + if failed is None and self.ok: + return "ok" + if failed is None or failed.status is None: + return "unreachable" + return "error" if failed.status >= 500 else "stale" + + @property + def stale(self) -> bool: + return self.outcome == "stale" async def check(journey: Journey, base_url: str, *, timeout: float = 10.0, @@ -108,12 +128,13 @@ async def check(journey: Journey, base_url: str, *, timeout: float = 10.0, env = dict(os.environ) if env is None else env result = JourneyCheck(journey) variables: dict[str, Any] = fresh_uniques() + secrets = secret_values(journey, env) jar = httpx.Cookies() async with httpx.AsyncClient(timeout=timeout, follow_redirects=True, headers=client_headers(headers), transport=transport, **client_options(base_url)) as client: for step in journey.steps: - outcome = await run_step(client, step, base_url, variables, env, jar) + outcome = await run_step(client, step, base_url, variables, env, jar, secrets) result.steps.append(outcome) if not outcome.ok: break @@ -195,9 +216,25 @@ def _record(sink, outcome: StepCheck, prefix: str = "") -> None: sink.record_error(label, outcome.kind or f"unexpected {outcome.status}") +def secret_values(journey: Journey, env: dict[str, str]) -> set[str]: + """The secrets a journey sends: the environment variables it reads.""" + return {env[name] for name in journey.env_names() if env.get(name)} + + +def _mask(text: str, secrets: set[str], variables: dict[str, Any]) -> str: + """TEXT with every secret replaced by ***: the journey's env values, and any + extracted value whose name looks secret (a token, a session id).""" + hidden = set(secrets) | {str(v) for k, v in variables.items() + if SENSITIVE_KEY.search(k) and isinstance(v, str)} + for value in sorted(hidden, key=len, reverse=True): + if len(value) >= 4: + text = text.replace(value, "***") + return text + + async def run_step(client: httpx.AsyncClient, step: Step, base_url: str, variables: dict[str, Any], env: dict[str, str], - jar: httpx.Cookies) -> StepCheck: + jar: httpx.Cookies, secrets: set[str] | None = None) -> StepCheck: """Send one step with this journey run's cookies (JAR), judge it against `expect`, and pull its `extract` values into VARIABLES for the steps after it.""" outcome = StepCheck(step) @@ -236,9 +273,9 @@ async def run_step(client: httpx.AsyncClient, step: Step, base_url: str, if not step.accepts(resp.status_code): wanted = (" or ".join(map(str, step.expect)) if step.expect else "a status below 400") outcome.problem = f"returned {resp.status_code}, expected {wanted}" - snippet = resp.text.strip().replace("\n", " ")[:_SNIPPET] - if snippet: - outcome.problem += f": {snippet}" + # Bodies can echo what was sent (validation errors often do), secrets included. + snippet = _mask(resp.text.strip().replace("\n", " "), secrets or set(), variables) + outcome.snippet = snippet[:_SNIPPET] or None return outcome for name, source in step.extract.items(): diff --git a/cli/src/prescale_cli/render.py b/cli/src/prescale_cli/render.py index fe5e821..ef37f61 100644 --- a/cli/src/prescale_cli/render.py +++ b/cli/src/prescale_cli/render.py @@ -9,6 +9,7 @@ from rich.table import Table from prescale_cli.bars import text_bar +from prescale_cli.journeys.load import LABELS from prescale_cli.throughput import VISITOR_NOTE, describe console = Console() @@ -160,7 +161,8 @@ def _expect_lines(exp: dict) -> list[str]: mark = {True: "✅", False: "🛑", None: "❔"}[c["meets"]] lines.append(f" {mark} all together: holds ~{c['survives_users']}") for name in exp.get("untested") or []: - lines.append(f" ❔ {escape(name)}: not load-tested (stale or unreachable)") + lines.append(f" ❔ {escape(name)}: not load-tested " + "(stale, failing or unreachable)") return lines @@ -176,7 +178,7 @@ def render_journeys(result: dict) -> None: for j in result["journeys"]: name = escape(j["name"]) if j["status"] != "ok": - label = "stale" if j["status"] == "stale" else "couldn't run" + label = LABELS.get(j["status"], j["status"]) table.add_row(name, f"[yellow]{label}[/yellow]", "—", f"[dim]{escape(j.get('reason') or '')}[/dim]") continue diff --git a/cli/src/prescale_cli/report.py b/cli/src/prescale_cli/report.py index 07720b6..2ea2d0a 100644 --- a/cli/src/prescale_cli/report.py +++ b/cli/src/prescale_cli/report.py @@ -10,6 +10,8 @@ import html from datetime import datetime +from prescale_cli.journeys.load import LABELS + _CSS = """ :root{ --canvas:#010102;--s1:#0f1011;--s2:#141516;--hair:#23252a; @@ -275,7 +277,7 @@ def _journeys_html(result: dict) -> str: rows = [] for j in journeys: if j["status"] != "ok": - label = "stale" if j["status"] == "stale" else "couldn't run" + label = LABELS.get(j["status"], j["status"]) rows.append(f"{_esc(j['name'])}{label}" f"—" f"{_esc(j.get('reason') or '')}") @@ -334,7 +336,7 @@ def _expect_html(exp: dict | None) -> str: if combined: items.append(f"
  • all together: holds ~{combined['survives_users']}
    • ") for name in exp.get("untested") or []: - items.append(f"
  • {_esc(name)}: not load-tested (stale or unreachable)
    • ") + items.append(f"
  • {_esc(name)}: not load-tested (stale, failing or unreachable)
    • ") if not exp.get("journeys"): items.append(f"
  • survives ~{exp['survives_users']}
    • ") return (f'
    Expected ~{exp["users"]} users: {answer}' diff --git a/cli/src/prescale_cli/schema/result.schema.json b/cli/src/prescale_cli/schema/result.schema.json index 7751c93..622f890 100644 --- a/cli/src/prescale_cli/schema/result.schema.json +++ b/cli/src/prescale_cli/schema/result.schema.json @@ -182,7 +182,7 @@ "name": { "type": "string" }, "type": { "type": "string" }, "weight": { "type": "number" }, - "status": { "enum": ["ok", "stale", "unreachable"] }, + "status": { "enum": ["ok", "stale", "error", "unreachable"], "description": "stale: answered, but not as the journey expects (the app changed); error: a 5xx with one user; unreachable: no answer." }, "reason": { "type": ["string", "null"], "description": "Why a stale or unreachable journey wasn't load-tested." }, "survives_users": { "type": "integer" }, "max_tested": { "type": "integer" }, diff --git a/cli/tests/test_action.py b/cli/tests/test_action.py index 06b4c4f..6fcf9a5 100644 --- a/cli/tests/test_action.py +++ b/cli/tests/test_action.py @@ -15,3 +15,31 @@ def test_metadata_has_no_expressions_outside_defaults_and_steps(): for name, spec in action["inputs"].items(): assert "${{" not in spec["description"], name assert "secrets." not in str(spec.get("default", "")), name + + +def test_capacity_check_passes_stale_and_each_allow_writes_host(tmp_path): + import os + import subprocess + + stub = tmp_path / "prescale" + stub.write_text('#!/bin/sh\nfor a in "$@"; do printf "[%s]" "$a"; done\n') + stub.chmod(0o755) + action = yaml.safe_load(ACTION.read_text()) + script = next(s for s in action["runs"]["steps"] if s.get("name") == "Capacity check")["run"] + env = {"PATH": f"{tmp_path}:{os.environ['PATH']}", "RUNNER_TEMP": "/tmp", + "INPUT_MAX_USERS": "100", "INPUT_STAGE_SECONDS": "5", "INPUT_FAIL": "false", + "INPUT_COMMENT": "true", "INPUT_ALLOW_PRODUCTION": "false", + "INPUT_HOST_PERMISSION": "false", "INPUT_APP": "", "INPUT_APP_TIMEOUT": "600", + "INPUT_URL": "", "INPUT_STALE": "fail", + "INPUT_ALLOW_WRITES": "staging.myapp.com\n qa.myapp.com \n\n"} + def run(**inputs): + return subprocess.run(["bash", "-eo", "pipefail", "-c", script], env={**env, **inputs}, + capture_output=True, text=True).stdout + + assert run().endswith("[--stale][fail][--allow-writes][staging.myapp.com]" + "[--allow-writes][qa.myapp.com]") + # the default passes no new flag, so a pinned older prescale-version keeps working + assert "--stale" not in run(INPUT_STALE="warn", INPUT_ALLOW_WRITES="") + assert "--stale" not in run(INPUT_STALE="", INPUT_ALLOW_WRITES="") + assert run(INPUT_STALE="Fail", INPUT_ALLOW_WRITES="a.com, b.com").endswith( + "[--stale][fail][--allow-writes][a.com][--allow-writes][b.com]") diff --git a/cli/tests/test_ci_journeys.py b/cli/tests/test_ci_journeys.py new file mode 100644 index 0000000..f48e46d --- /dev/null +++ b/cli/tests/test_ci_journeys.py @@ -0,0 +1,248 @@ +"""`prescale ci` with journeys: the comment, stale handling, and the safe comparison.""" + +import json + +import pytest +from click.testing import CliRunner + +from prescale_cli import ci as ci_mod +from prescale_cli.journeys.load import JourneyRun, NoJourneyReplaysError +from prescale_cli.journeys.model import parse +from prescale_cli.journeys.replay import JourneyCheck, StepCheck +from prescale_cli.main import cli + +CHECKOUT = {"version": 1, "name": "checkout", "type": "checkout", + "steps": [{"name": "place order", + "request": {"method": "POST", "path": "/api/orders"}}]} +BROWSE = {"version": 1, "name": "browse", "steps": [{"name": "home", "request": {"path": "/"}}]} + + +@pytest.fixture(autouse=True) +def clean_env(monkeypatch, tmp_path): + for name in ("GITHUB_ACTIONS", "GITHUB_EVENT_NAME", "GITHUB_EVENT_PATH", "GITHUB_REPOSITORY", + "GITHUB_SHA", "GITHUB_REF", "GITHUB_HEAD_REF", "GITHUB_TOKEN", + "GITHUB_STEP_SUMMARY", "PRESCALE_HEADERS", "PRESCALE_PATHS", + "PRESCALE_GIT_COMMIT", "PRESCALE_GIT_BRANCH", "PRESCALE_TARGET"): + monkeypatch.delenv(name, raising=False) + monkeypatch.setenv("PRESCALE_CONFIG_DIR", str(tmp_path / "config")) + monkeypatch.setenv("PRESCALE_TOKEN", "psk_test") + monkeypatch.setenv("GITHUB_ACTIONS", "true") + + +def _journey_result(stale=False): + journeys = [ + {"name": "checkout", "type": "checkout", "weight": 5, "status": "ok", "reason": None, + "survives_users": 20, "max_tested": 100, "onset_users": 30, "onset_reason": "errors", + "confidence": {"survives_low": 20, "survives_high": 30, "stable": False}, + "culprit_step": {"index": 1, "name": "place order", "route": "POST /api/orders"}, + "bottleneck": "5xx under load"}, + {"name": "browse", "type": "browse", "weight": 60, + "status": "stale" if stale else "ok", + "reason": "step 1 (home): returned 404, expected a status below 400" if stale else None, + **({} if stale else {"survives_users": 100, "max_tested": 100, "onset_users": None, + "onset_reason": None, "confidence": {}, "culprit_step": None, + "bottleneck": None})}, + ] + return { + "id": "r1", "target": {"url": "http://localhost:3000", "host": "localhost"}, + "config": {"mode": "journeys", "replica": True}, + "verdict": {"survives_users": 20, "max_tested": 100, "onset_users": 30, + "onset_reason": "errors", "culprit_route": "POST /api/orders", + "culprit_journey": "checkout", "peak_rps": 120.0, + "confidence": {"survives_low": 20, "survives_high": 30, "stable": False}}, + "environment": {"git_commit": "a41f9c2"}, + "journeys": journeys, + "combined": {"interference": {"journey": "checkout", "users_at_break": 15, + "held_alone": 20, "with": ["browse"]}}, + } + + +def _upload(mode=None, kind="regressed"): + comparison = {"kind": kind, "pct": -0.5, "confident": True, "against": "main @ 71aa0c9", + "baseline": {"survives": 40, "low": 35, "high": 45, "peakRps": 200.0, + "commit": "71aa0c9"}} + if mode: + comparison["mode"] = mode + return {"url": "https://prescale.dev/dashboard/runs/x", "comparison": comparison, + "target": {"baselineBranch": "main", "verified": True}} + + +# --- the comment ----------------------------------------------------------------- + +def test_a_journey_run_ignores_a_comparison_that_may_be_against_a_url_run(): + assert ci_mod.comparison_for(_journey_result(), _upload()) is None + assert ci_mod.comparison_for(_journey_result(), _upload(mode="journeys"))["kind"] == "regressed" + url_run = {**_journey_result(), "config": {}} + assert ci_mod.comparison_for(url_run, _upload())["kind"] == "regressed" + + body = ci_mod.render_comment(_journey_result(), _upload(), target="myapp.com", sha=None) + assert "Capacity regressed" not in body + assert "### ⚪ Capacity: survives ~20 users" in body + assert "may have tested URLs" in body + + +def test_journey_comment_lists_journeys_stale_ones_and_interference(): + body = ci_mod.render_comment(_journey_result(stale=True), _upload(mode="journeys"), + target="myapp.com", sha=None) + assert "### 🔴" in body or "Capacity regressed" in body + assert "**Breaks first:** checkout › `POST /api/orders`" in body + assert "| checkout | ~20 | 1. place order `POST /api/orders` | 5xx under load |" in body + assert "- ⚠️ **browse** (stale): step 1 (home): returned 404" in body + assert "checkout broke with 15 of its own users, but held 20 alone" in body + + +def test_stale_only_comment(): + runs = [] + for doc in (CHECKOUT, BROWSE): + j = parse(doc) + c = JourneyCheck(j) + c.steps = [StepCheck(j.steps[0], status=404, problem="returned 404, expected 201")] + runs.append(JourneyRun(j, c)) + body = ci_mod.render_stale_only(runs, target="myapp.com") + assert body.startswith(ci_mod.comment_marker("myapp.com")) + assert "no journey replays against this build" in body + assert "- **checkout** (stale): step 1 (place order): returned 404, expected 201" in body + + +# --- the command ------------------------------------------------------------------ + +def _dir(tmp_path): + d = tmp_path / "journeys" + d.mkdir(exist_ok=True) + for doc in (CHECKOUT, BROWSE): + (d / f"{doc['name']}.yaml").write_text(json.dumps(doc)) + return str(d) + + +def _wire(monkeypatch, *, result=None, raises=None, upload=None): + seen = {} + + def fake_run_journeys(journeys, url, config, **kwargs): + seen["journeys"] = [j.name for j in journeys] + seen["url"], seen["replica"] = url, kwargs["replica"] + if raises: + raise raises + return None, result, False + + def fake_upsert(ctx, token, pr_number, body, marker): + seen["comment"] = body + return "created" + + monkeypatch.setattr("prescale_cli.commands.ci.run_journeys", fake_run_journeys) + monkeypatch.setattr("prescale_cli.commands.ci.upload_result", + lambda *a, **k: upload or _upload()) + monkeypatch.setattr("prescale_cli.commands.ci.upsert_pr_comment", fake_upsert) + monkeypatch.setattr("prescale_cli.commands.ci.find_pull_request", + lambda ctx, token: {"number": 7, "head": {"ref": "feat/x"}}) + monkeypatch.setattr("prescale_cli.commands.ci.run_investigate", + lambda *a, **k: pytest.fail("journey runs don't investigate URLs")) + + async def no_rule(url, **kwargs): + return None + monkeypatch.setattr("prescale_cli.commands.ci.probe_host", no_rule) + monkeypatch.setenv("GITHUB_TOKEN", "ghs_x") + monkeypatch.setenv("GITHUB_REPOSITORY", "acme/shop") + return seen + + +def _ci(tmp_path, *extra): + return CliRunner().invoke(cli, ["ci", "http://localhost:3000", "--target", "myapp.com", + "--dir", _dir(tmp_path), "--store", str(tmp_path / "s"), + *extra]) + + +def test_ci_replays_journeys_and_warns_on_stale(monkeypatch, tmp_path): + seen = _wire(monkeypatch, result=_journey_result(stale=True)) + res = _ci(tmp_path) + assert res.exit_code == 0, res.output + assert seen["journeys"] == ["browse", "checkout"] + assert "::warning::PreScale: browse no longer replays" in res.output + assert "Not load-tested" in seen["comment"] + + +def test_ci_stale_fail_fails_the_job(monkeypatch, tmp_path): + _wire(monkeypatch, result=_journey_result(stale=True)) + res = _ci(tmp_path, "--stale", "fail") + assert res.exit_code == 1 + assert "::error::PreScale: browse no longer replays" in res.output + + +def test_ci_never_fails_a_journey_run_on_an_unmarked_comparison(monkeypatch, tmp_path): + _wire(monkeypatch, result=_journey_result()) + assert _ci(tmp_path, "--fail-on-regression").exit_code == 0 + _wire(monkeypatch, result=_journey_result(), upload=_upload(mode="journeys")) + assert _ci(tmp_path, "--fail-on-regression").exit_code == 1 + + +def test_ci_comments_when_no_journey_replays(monkeypatch, tmp_path): + seen = _wire(monkeypatch, raises=NoJourneyReplaysError([])) + res = _ci(tmp_path) + assert res.exit_code == 0 + assert "no journey replays against this build" in seen["comment"] + assert "::warning::PreScale: no journey replays" in res.output + _wire(monkeypatch, raises=NoJourneyReplaysError([])) + assert _ci(tmp_path, "--stale", "fail").exit_code == 1 + + +def test_ci_refuses_journey_writes_on_a_preview_host(monkeypatch, tmp_path): + _wire(monkeypatch, result=_journey_result()) + res = CliRunner().invoke(cli, ["ci", "https://preview.example.com", "--target", "myapp.com", + "--dir", _dir(tmp_path)]) + assert res.exit_code == 2 + assert "--allow-writes preview.example.com" in " ".join(res.output.split()) + + +# --- review fixes -------------------------------------------------------------------- + +def _runs(outcomes): + runs = [] + for name, status in outcomes: + j = parse({**BROWSE, "name": name}) + c = JourneyCheck(j) + c.steps = [StepCheck(j.steps[0], status=status, problem=f"returned {status}" + if status else "couldn't reach the app (ConnectError)")] + runs.append(JourneyRun(j, c)) + return runs + + +def test_an_app_that_doesnt_answer_fails_the_check_even_with_stale_warn(monkeypatch, tmp_path): + _wire(monkeypatch, raises=NoJourneyReplaysError(_runs([("a", None), ("b", 404)]))) + res = _ci(tmp_path) + assert res.exit_code == 1 + assert "::error::PreScale: a failed with a single user" in res.output + + +def test_a_5xx_with_one_user_fails_the_check(monkeypatch, tmp_path): + _wire(monkeypatch, raises=NoJourneyReplaysError(_runs([("a", 500)]))) + assert _ci(tmp_path).exit_code == 1 + result = _journey_result() + result["journeys"][1] = {"name": "browse", "type": "browse", "weight": 60, + "status": "error", "reason": "step 1 (home): returned 500"} + _wire(monkeypatch, result=result) + res = _ci(tmp_path) + assert res.exit_code == 1 + assert "::error::PreScale: browse failed with a single user" in res.output + + +def test_only_stale_follows_the_stale_policy(monkeypatch, tmp_path): + _wire(monkeypatch, raises=NoJourneyReplaysError(_runs([("a", 404)]))) + assert _ci(tmp_path).exit_code == 0 + + +def test_comment_text_from_files_and_apps_cant_mention_render_html_or_break_code(): + result = _journey_result(stale=True) + result["journeys"][1]["reason"] = "returned 422 for @octocat | ok" + result["journeys"][0]["culprit_step"]["route"] = "POST /a`b" + body = ci_mod.render_comment(result, _upload(mode="journeys"), target="t", sha=None) + assert "@octocat" not in body and "@​octocat" in body + assert "