Skip to content

feat(journeys): investigate diagnoses the journey step that breaks first - #12

Merged
pyjeebz merged 17 commits into
mainfrom
feat/investigate-journeys
Nov 3, 2026
Merged

pyjeebz merged 17 commits into
mainfrom
feat/investigate-journeys

Conversation

@pyjeebz

@pyjeebz pyjeebz commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Stacked on #11. The base is #11's branch, so the diff shows only this slice. It gets retargeted to main once #11 merges.

prescale run on journeys says which step breaks first. Now prescale investigate says why, and what to do about it. With journeys in prescale/journeys/, investigate:

  1. ramps them like run does
  2. probes the step that broke first, reaching it the way a user does (logged in, cart filled)
  3. names the bottleneck and the fix
🔬 Diagnosis
Likely cause: 5xx under load while the journey's other steps held — the app/backend is the wall (often a DB or upstream pool).
Bottleneck  connection_pool (high confidence)  ·  culprit checkout, step 3 place order (POST /api/carts/{{cart_id}}/order)
Evidence
  • culprit median 108ms at 1 user vs p95 153ms at 30 users
  • the journey's other steps (log in, add to cart) held at 30 users
  • errors under load: 5xx (5)
Try this
  → Increase the DB/upstream connection-pool size.
  → Add a pooler (e.g. pgbouncer) and check the pool checkout timeout.

How it works (journeys/diagnose.py):

  • Under load: the step's numbers come from the stage where it broke.
  • At 1 user: the journey is replayed from the start, so the step carries its token and cart id. That runs for about 2s after a 1s pause, within --max-rps. It uses the median below 20 samples. Below 5 samples there's no baseline, and the evidence says so.
  • The reference: the journey's other steps play the part static assets play for URLs. If they held while this step failed, the limit is something this step uses.
  • Classification and fixes are investigate's existing classify and remediate, so a journey step and a URL route get the same diagnosis for the same evidence. No model is involved.
  • Breaks only together: if every journey held alone but they broke when ramped together, the step that broke in the combined run is diagnosed instead. The evidence says so, and --fail-under gates on the combined number.
  • New class unexpected_answer: a step that answers an unexpected 4xx or loses an extract under load (users conflicting over shared data, or sessions failing). It's no longer misread as a pool or a queue.
  • Shared routes: steps that call the same route are flagged and not used as their own reference.

The investigation field gains culprit_journey, culprit_step and interference. These are additive; schema_version stays 1. New investigate flags, the same as run's: --journey, --no-combined, --urls, --allow-writes, --dir. run and ci never diagnose, since diagnose defaults to false.

How it was verified

  • The real CLI against examples/fragile-shop.py, three times:
    • connection_pool, high confidence, checkout step 3 "place order", every time
    • --json stdout is the Result alone and validates against result.schema.json
    • --fail-under 200 exits 1
    • the password never appears in the output
  • Tests (test_journeys_diagnose.py, 17 new):
    • the end-to-end investigate --json above, against the real example shop (started by the test)
    • run adds no investigation
    • breaks that only happen together, from real ramps on a mock app
    • shared routes; 409s read as unexpected_answer
    • no baseline when the app is down after the ramp
    • --max-rps pacing of the 1-user replays
    • --fail-under gating on the combined number
    • refusals before any request: writes on a non-local host, unset env vars
    • --urls and --path still taking URL mode
  • ruff check cli/src: clean · pytest cli/tests -q: 346 passed.

Risk

Tier: risky.

Public surface:

  • investigation gains fields
  • a new bottleneck_class value, unexpected_answer. The dashboard shows class names as-is, so it needs no change.
  • new investigate flags
  • with journey files present, investigate now tests journeys. --urls restores the old behaviour.
  • jsonschema added to the dev extras

Rollback: revert. Nothing stored by older versions changes meaning.

Fresh-context review: the first pass said fix first. Its blocker was that journeys that held alone but broke together printed "Held up — nothing to diagnose". It also had five should-fixes:

  • shared routes
  • 4xx misclassified as a queue
  • a fragile baseline
  • missing CLI tests
  • --max-rps ignored

All are fixed. The re-review said ok to ship. Its three small should-fixes are also done:

  • --fail-under counts a break that only happens together
  • clearer "users in total" and "median" wording
  • jsonschema as a dev dependency, so the schema check never silently skips

Notes for review

  • Next slice (J1.5b): the diagnosis in the PR comment (prescale ci), MCP and the HTML report, plus a paste-ready fix prompt for coding agents.
  • Then (J1.5c): named classes for "everyone hits the same row" and "N+1 at scale", each with a demo app.
  • Pre-existing, unchanged: a gunicorn server header adds "increase --workers" even for pool problems.

Generated by Claude Code

claude added 16 commits November 2, 2026 14:02
# Conflicts:
#	CHANGELOG.md
#	README.md
#	cli/src/prescale_cli/commands/run.py
#	cli/src/prescale_cli/journeys/load.py
#	cli/src/prescale_cli/journeys/replay.py
#	cli/src/prescale_cli/loadtest.py
#	cli/src/prescale_cli/render.py
#	cli/src/prescale_cli/schema/result.schema.json
#	cli/tests/test_journeys_load.py
@pyjeebz pyjeebz closed this Nov 3, 2026
@pyjeebz pyjeebz reopened this Nov 3, 2026
@pyjeebz
pyjeebz changed the base branch from feat/journeys-in-ci to main November 3, 2026 15:14
@mintlify

mintlify Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
prescale 🟢 Ready View Preview Nov 3, 2026, 3:14 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants