feat(journeys): prescale ci and the Action replay journeys, with --stale

CHANGELOG.md

@@ -29,6 +29,16 @@
   `--expect`; the MCP tools' run summaries list each journey's verdict; and `prescale compare`
   diffs each journey by name (terminal, `--json`, `--markdown`). A drop past the threshold in any
   journey counts as a regression for `--fail-on-regression`, even when it isn't the weakest journey.
+- **Journeys in CI.** `prescale ci` and the GitHub Action replay the checkout's journeys when
+  `prescale/journeys/` exists. The PR comment shows each journey's verdict, interference, and
+  journeys that no longer replay (listed always; `--stale fail` / `stale: fail` fails the check).
+  A journey that gets no answer, or a 5xx with a single user, always fails the check (that's the
+  app, not a stale file). Response bodies never reach the PR comment or uploads; locally,
+  `journeys verify` shows them with every secret the journeys read masked.
+  New Action inputs: `stale` and `allow-writes` (the action only passes them when set, so a
+  pinned older `prescale-version` keeps working). Until the dashboard compares journey runs only
+  with journey baselines, a journey run's PR comment shows no change instead of a possibly
+  misleading one, and never fails on it.
 - **`--expect N`** on `run` says whether you'd hold N users at peak. In a journey run, N is split
   across journeys by weight: each must hold its share alone, and all together must hold N.
 

README.md

@@ -219,6 +219,8 @@ Interference  checkout broke with 15 of its own users, but held 20 alone.
 
 `prescale show`, `--html` reports and the MCP tools show journeys too, and `prescale compare` diffs each journey: a drop past the threshold in any journey is a regression (so `--fail-on-regression` catches it), even when it isn't the weakest one; it's marked confident when the confidence bands don't overlap. A journey that held everything it was tested to, below the old level, is inconclusive, not a drop. A URL run and a journey run measure different things, so comparing them is reported as not comparable.
 
+In CI, `prescale ci` (and the GitHub Action) replays the checkout's journeys instead of URLs whenever `prescale/journeys/` exists, and the PR comment shows each journey's verdict, any interference, and journeys that no longer replay. Those are always listed; `--stale fail` (Action input `stale: fail`) also fails the check. With `app:` the replica is local, so journeys that change data can run; against a preview host they need `allow-writes`. Journey secrets (`{{env.NAME}}`) come from the step's `env:`.
+
 `--no-combined` skips this pass. `--expect 500` says whether you'd hold 500 users at peak: in a journey run, each journey must hold its share of them (by weight) on its own, and all journeys together must hold 500; a journey that wasn't load-tested makes the answer unknown. It reports, it doesn't change the exit code (use `--fail-under` to gate CI).
 
 - `{{name}}` is a value an earlier step pulled out with `extract` (`body.<field>` or `header.<name>`); `{{unique.email}}` and `{{unique.id}}` are fresh each run; `{{env.NAME}}` reads an environment variable.

action/README.md

@@ -78,6 +78,25 @@ jobs:
           fail-on-regression: true
 ```
 
+## Journeys
+
+If the checkout has journey files (`prescale/journeys/*.yaml`, see the main README), the action replays them at load instead of testing URLs: each journey alone, then all together. The PR comment shows each journey's verdict, any interference, and journeys that no longer replay against the build. Give journeys their secrets through the step's `env:` (for `{{env.NAME}}`):
+
+```yaml
+      - uses: pyjeebz/prescale/action@v1
+        with:
+          token: ${{ secrets.PRESCALE_TOKEN }}
+          target: myapp.com
+          app: npm run build && npm start
+          stale: fail
+        env:
+          SHOP_PASSWORD: ${{ secrets.STAGING_TEST_PASSWORD }}
+```
+
+A journey that **no longer replays** (the app answers differently, e.g. a 404) is listed on the PR; `stale: fail` also fails the check. A journey that gets **no answer, or a 5xx with a single user**, always fails the check: that's the app broken or down, not a stale file. Response bodies never appear in the PR comment.
+
+Deploy workflows (`on: deployment_status`) that also check out the code replay journeys too. Journeys that change data then need the preview's host in `allow-writes`; preview hosts change per deploy, so prefer replica mode (`app:`) for journeys that write.
+
 ## How it works
 
 1. A pull request is pushed (replica mode) or a deploy finishes (`deployment_status`).
@@ -122,5 +141,7 @@ uploaded is the run's Result file, which you can inspect with
 | `app` | — | Replica mode: build-and-start command, tested at `url` (default `http://localhost:3000`) |
 | `app-timeout` | `600` | Seconds to wait for the app to answer |
 | `working-directory` | `.` | Where to run `app` (monorepos) |
+| `stale` | `warn` | Journey runs: a journey that no longer replays is always listed on the PR; `fail` also fails the check |
+| `allow-writes` | — | Journey runs: hosts (one per line) where journeys that change data may run; not needed with `app` |
 | `app-url` | `https://prescale.dev/dashboard` | Dashboard URL |
 | `prescale-version` | latest | CLI version |

action/action.yml

@@ -48,6 +48,14 @@ inputs:
     description: Confirm the deploy's host allows load testing. Vercel only permits it on Enterprise plans with notice, and Netlify and Cloudflare need it arranged; without this, PreScale skips their deploys and says why on the PR.
     required: false
     default: "false"
+  stale:
+    description: "Journey runs (when the checkout has prescale/journeys/): 'warn' or 'fail'. A journey that no longer replays against this build is always listed on the PR; 'fail' also fails the check."
+    required: false
+    default: "warn"
+  allow-writes:
+    description: "Journey runs: hosts (one per line, or comma-separated) where journeys that change data may run, e.g. a staging host. Not needed with app: the replica is local. Journeys read their secrets from env variables you set on this step."
+    required: false
+    default: ""
   app-url:
     description: PreScale dashboard URL.
     required: false
@@ -105,12 +113,21 @@ runs:
         INPUT_HOST_PERMISSION: ${{ inputs.host-permission }}
         INPUT_APP: ${{ inputs.app }}
         INPUT_APP_TIMEOUT: ${{ inputs.app-timeout }}
+        INPUT_STALE: ${{ inputs.stale }}
+        INPUT_ALLOW_WRITES: ${{ inputs.allow-writes }}
       run: |
         args=(--max-users "$INPUT_MAX_USERS" --stage-seconds "$INPUT_STAGE_SECONDS" --store "$RUNNER_TEMP/prescale")
         [ "$INPUT_FAIL" = "true" ] && args+=(--fail-on-regression)
         [ "$INPUT_COMMENT" = "true" ] || args+=(--no-comment)
         [ "$INPUT_ALLOW_PRODUCTION" = "true" ] && args+=(--allow-production)
         [ "$INPUT_HOST_PERMISSION" = "true" ] && args+=(--host-permission)
         [ -n "$INPUT_APP" ] && args+=(--app "$INPUT_APP" --app-timeout "$INPUT_APP_TIMEOUT")
+        # New flags only when set, so a pinned older prescale-version keeps working.
+        stale="${INPUT_STALE,,}"
+        [ -n "$stale" ] && [ "$stale" != "warn" ] && args+=(--stale "$stale")
+        while IFS= read -r host; do
+          host="${host//[[:space:]]/}"
+          [ -n "$host" ] && args+=(--allow-writes "$host")
+        done <<< "${INPUT_ALLOW_WRITES//,/$'\n'}"
         [ -n "$INPUT_URL" ] && args+=("$INPUT_URL")
         prescale ci "${args[@]}"

cli/src/prescale_cli/ci.py

@@ -16,6 +16,7 @@
 import httpx
 
 from prescale_cli.bars import bar_rows, route_rows
+from prescale_cli.journeys.load import LABELS
 from prescale_cli.throughput import VISITOR_NOTE, capacity_rps, visitors_for
 
 _SHA = re.compile(r"^[0-9a-f]{40}$")
@@ -181,11 +182,91 @@ def render_host_skip(rule, *, target: str) -> str:
     ])
 
 
+def comparison_for(result: dict, upload: dict) -> dict | None:
+    """The dashboard's comparison, if it compared like with like. A journey run is
+    only comparable with a journey baseline; a dashboard that doesn't say which
+    kind of run it compared against (`mode`) may have used a URL run, so its
+    comparison isn't trusted for journey runs (never a false regression)."""
+    comparison = upload.get("comparison")
+    if not comparison:
+        return None
+    if _mode(result) == "journeys" and comparison.get("mode") != "journeys":
+        return None
+    return comparison
+
+
+def _mode(result: dict) -> str:
+    return (result.get("config") or {}).get("mode") or "urls"
+
+
+def _cell(text) -> str:
+    """Text from journey files or the app, safe in a PR comment: it can't break the
+    table, render HTML, or @mention (and so notify) anyone."""
+    return (str(text).replace("|", "\\|").replace("\n", " ").replace("<", "&lt;")
+            .replace(">", "&gt;").replace("@", "@\u200b"))
+
+
+def _code(text) -> str:
+    """TEXT as inline code that a backtick inside can't break out of."""
+    return "`" + _cell(text).replace("`", "'") + "`"
+
+
+def render_journeys(result: dict) -> list[str]:
+    """Journey runs: each journey's verdict, the stale ones, and interference."""
+    journeys = result.get("journeys") or []
+    if not journeys:
+        return []
+    lines = ["", "**Journeys** <sub>(each ramped on its own)</sub>", "",
+             "| journey | survives | breaks at step | why |", "|---|---:|---|---|"]
+    stale = []
+    for j in journeys:
+        if j["status"] != "ok":
+            stale.append(j)
+            continue
+        survives = (f"{j['max_tested']}+ (held)" if j["onset_users"] is None
+                    else f"~{j['survives_users']}")
+        step = j.get("culprit_step")
+        where = f"{step['index']}. {_cell(step['name'])} {_code(step['route'])}" if step else "—"
+        lines.append(f"| {_cell(j['name'])} | {survives} | {where} | "
+                     f"{_cell(j.get('bottleneck') or '')} |")
+    if stale:
+        lines += ["", "**Not load-tested** (they don't replay against this build, so the app "
+                      "changed or it's broken; update the journey files if the change was on "
+                      "purpose):"]
+        lines += [f"- ⚠️ **{_cell(j['name'])}** ({j['status']}): {_cell(j.get('reason') or '')}"
+                  for j in stale]
+    inter = (result.get("combined") or {}).get("interference")
+    if inter:
+        lines += ["", f"**Interference:** {_cell(inter['journey'])} broke with "
+                      f"{inter['users_at_break']} of its own users, but held "
+                      f"{inter['held_alone']} alone. It competes with "
+                      f"{_cell(', '.join(inter['with']))} for something they share "
+                      "(a database pool, CPU, a cache)."]
+    return lines
+
+
+def render_stale_only(runs, *, target: str) -> str:
+    """The PR comment when no journey replays, so nothing was load-tested."""
+    lines = [comment_marker(target), "### ⚠️ PreScale: no journey replays against this build", "",
+             "Every journey failed with a single user, so there's no capacity to measure. "
+             "If the app changed on purpose, update the journey files (`prescale journeys "
+             "verify` shows what changed); if not, it's a bug.", ""]
+    for run in runs:
+        failed = run.check.failed
+        index = run.check.steps.index(failed) + 1 if failed else "?"
+        what = LABELS[run.check.outcome]
+        lines.append(f"- **{_cell(run.journey.name)}** ({what}): step {index} "
+                     f"({_cell(failed.step.name if failed else '?')}): "
+                     f"{_cell(failed.problem if failed else '')}")
+    return "\n".join(lines) + "\n"
+
+
 def render_comment(result: dict, upload: dict, *, target: str, sha: str | None) -> str:
     """The PR comment: the change vs baseline, what breaks first, why, the fix."""
     verdict = result["verdict"]
     conf = verdict.get("confidence") or {}
-    comparison = upload.get("comparison")
+    comparison = comparison_for(result, upload)
+    unmatched = comparison is None and bool(upload.get("comparison"))
     target_info = upload.get("target") or {}
     baseline_branch = target_info.get("baselineBranch") or "main"
     tested_host = result["target"].get("host") or result["target"]["url"]
@@ -238,21 +319,30 @@ def render_comment(result: dict, upload: dict, *, target: str, sha: str | None)
             lines += ["", "The confidence bands overlap, so this may be noise. Another run "
                           "will confirm or clear it."]
     else:
-        lines.append(f"### ⚪ First capacity run: survives {survives}")
+        title = "Capacity" if unmatched else "First capacity run"
+        lines.append(f"### ⚪ {title}: survives {survives}")
         lines.append(f"`{tested_host}` · **{target}**")
         if rps:
             lines += ["", f"That's ≈ {rps:,.0f} req/s, roughly {visitors_for(rps):,} real "
                           f"visitors. <sub>Visitors assume {VISITOR_NOTE}.</sub>"]
-        lines += ["", f"No `{baseline_branch}` baseline yet. Later pushes to this PR compare "
-                      f"with this run; once `{baseline_branch}` has a run, PRs compare with it."]
+        if unmatched:
+            lines += ["", "This run replayed journeys. The baseline it would be compared with "
+                          "may have tested URLs, which measures something else, so no change is "
+                          "shown."]
+        else:
+            lines += ["", f"No `{baseline_branch}` baseline yet. Later pushes to this PR "
+                          f"compare with this run; once `{baseline_branch}` has a run, PRs "
+                          "compare with it."]
 
     investigation = result.get("investigation")
     lines.append("")
     if verdict.get("onset_users") is None:
         lines.append(f"**Held:** {_onset(verdict)} without crossing a failure threshold.")
     else:
         culprit = verdict.get("culprit_route")
-        lines.append(f"**Breaks first:** `{culprit}` · {_onset(verdict)}" if culprit
+        journey = verdict.get("culprit_journey")
+        where = f"{_cell(journey)} › {_code(culprit)}" if journey and culprit else _code(culprit)
+        lines.append(f"**Breaks first:** {where} · {_onset(verdict)}" if culprit
                      else f"**Breaks first:** {_onset(verdict)}")
         if investigation:
             lines.append(f"**Why:** `{investigation['bottleneck_class']}` "
@@ -267,6 +357,8 @@ def render_comment(result: dict, upload: dict, *, target: str, sha: str | None)
             caption = f"p95 per route at {stage['users']} users · ┆ = the {wall:g}s latency wall"
             lines += ["", f"<sub>{caption}</sub>", "", "```text", *route_rows(stage, wall), "```"]
 
+    lines += render_journeys(result)
+
     links = [f"[Open the run]({upload['url']})"] if upload.get("url") else []
     if target_info.get("url"):
         links.append(f"[Capacity timeline]({target_info['url']})")