Skip to content

feat(journeys): count how many requests a step serves at once, and name one-at-a-time bottlenecks - #14

Merged
pyjeebz merged 3 commits into
mainfrom
feat/how-many-at-once
Nov 8, 2026
Merged

pyjeebz merged 3 commits into
mainfrom
feat/how-many-at-once

Conversation

@pyjeebz

@pyjeebz pyjeebz commented Nov 3, 2026

Copy link
Copy Markdown
Owner

The last slice of Journeys 1. investigate named the bottleneck class (#12, #13). Now it also says how many requests the breaking step serves at once, and it names a new bug PreScale should own: one request at a time. That covers a lock on one shared row (a flash sale on one product, a counter, a balance), a global lock, or a single worker.

How it's measured (Little's law, journeys/diagnose.py): the step's successful completions per second at the onset level, times its median 1-user time, gives the requests in service at once. It only counts when all of these hold:

  • The step's own requests queued. Under load their mean time is at least 3× the 1-user time. A cheap step that just completes at the pace of a busier step before it never queues.
  • Completions held steady. From the level before the break to the break they stayed within ±15%. A rise means demand-limited; a drop means a collapse.
  • PreScale's machine kept up. Event-loop lag was under the existing threshold.
  • No --max-rps cap, since the cap, not the app, would be what stopped it.
  • At least 5 baseline samples.

What it shows:

  • The count, as evidence: "it completed at most ~23 requests/s while users kept rising; at 43ms each that's ~1.0 request served at once". For a pool, "~25 at once" is often the pool size you're hitting.
  • A new class, serialized, below 1.5 at once:
    • High confidence when the journey's other steps held and the errors are waiting, not 5xx or 4xx; otherwise medium.
    • Fixes: find the SELECT … FOR UPDATE, mutex or single worker; hold it for less; lock less, e.g. an atomic UPDATE … SET stock = stock - 1 WHERE stock > 0, or split the hot row.

Demo: examples/fragile-shop.py gains POST /api/products/{id}/buy, which holds one product's stock lock for ~40ms. The new examples/journeys/flash-sale.yaml has everyone buy the mug.

Not in this PR, deliberately: N+1 at scale. From outside it's indistinguishable from slow_endpoint. Naming it honestly needs inside-the-server visibility (query counts), which is planned for replica mode. Guessing would cost the trust this product depends on.

How it was verified

  • Live on the example shop:
    • --journey flash-sale → serialized, high, "~23 requests/s at 43ms each ≈ 1.0 at once"
    • --journey checkout → still connection_pool, high, with no count: its think time keeps completions demand-limited
    • journeys verify on all 3 example journeys passes
    • the combined run with 3 journeys still works
  • Tests:
    • one at a time → serialized with the number
    • a limit of 5 → counted, not serialized
    • a pool with 5xx shows "~25 at once"
    • --max-rps → no count
    • the reviewer's false positive (a cheap step paced by a capped step) → not serialized
    • a collapse and a busy load generator → no count
    • end-to-end investigate --json names the flash sale on the real example shop
  • Flakiness: the timing tests passed 5× in a row, and under CPU load. One early full-suite run had a single failure I couldn't reproduce in 9 more runs, and the reviewer couldn't either.
  • ruff check cli/src cli/tests: clean · pytest cli/tests -q: 368 passed (7 new).

Risk

Tier: standard. Public surface:

  • the new bottleneck_class value serialized. Additive: the schema types it as a string, and the dashboard shows class names as-is.
  • a new evidence line
  • a new example journey

Rollback: revert.

Fresh-context review:

  • Blocker: a cheap step paced by an earlier capped step was called serialized/high, which also hid a real 5xx diagnosis. Reproduced and fixed with the queue check.
  • Should-fixes, all done:
    • a collapse counted as a plateau
    • PreScale's own CPU limit could look like a lock
    • "~0 requests" wording
    • confidence when errors point elsewhere
  • Re-review: ok to ship.

Generated by Claude Code

@mintlify

mintlify Bot commented Nov 3, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
prescale 🟢 Ready View Preview Nov 3, 2026, 10:13 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@mintlify

mintlify Bot commented Nov 3, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
prescale 🟡 Building – Nov 3, 2026, 10:13 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@pyjeebz
pyjeebz merged commit b279062 into main Nov 8, 2026
4 checks passed
@pyjeebz pyjeebz mentioned this pull request Nov 8, 2026

This branch was successfully deployed

1 active deployment
staging - docs — 55549083 Deployed Nov 3, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants