diff --git a/CLAUDE.md b/CLAUDE.md index 242b334..4ebd105 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,7 @@ history, monolith dropped). The full agreed plan is in [`docs/PLAN.md`](docs/PLA - Name prefix: `ms12p` (parent uses `flaskms`). Teardown deletes the whole resource group, so a shared name would delete the other project. -## Current state (complete: 12 of 12 live, plus demo 13; one Azure run done and torn down, #11) +## Current state (complete: 12 of 12 live, plus demos 13 and 14; one Azure run done and torn down, #11) - Folders: `services/{user,catalog,order,payment,inventory}/` (Flask, one DB each), `gateway/` (FastAPI), `registry/` (FastAPI), `shared/` (`shop_common`), `contracts/`, `frontend/`, `infra/`, `scripts/`, `tests/`. @@ -49,6 +49,14 @@ history, monolith dropped). The full agreed plan is in [`docs/PLAN.md`](docs/PLA are returned as they are. `saga.settle_charge`: APPROVED confirms, DECLINED 402, VOIDED 503 (both release), PROCESSING/no answer writes `pending_payment` (checkout and add-item answer 409) and returns 202; `reconcile.py` settles pending rows every 5 s (deleting and committing the row is the claim). Page `/patterns/slow-payment`. +- Demo 14 (compensation, from a LinkedIn comment): inventory switch `POST /api/inventory/chaos` + (`healthy|slow_reserve|slow_release`, 6 s, past order's 3 s inventory timeout). `stock.release` of an + unknown hold records a RELEASED row (order_id `NEVER_HELD` = 0, no items, logged), so a late reserve + returns it and holds nothing; release never 404s now (ids must be 8-36 chars). `releases_received` + counts release copies (old DBs gain it via `add_missing_columns`). `consumer.counts` has + `duplicates_skipped` and `not_handled` (in `/api/inventory/events`). Demo switches use + `shop_common.flask_support.DemoSwitch` and `switches_off()`. `POST /api/order/sagas//replay` (chaos, own + checkout) clears `published_at` so the relay resends the same ids. Page `/patterns/compensation`. - Events (pattern 7): `shop_common/events.py` (Redis Stream `shop.events`, consumer groups, at least once). Order writes `outbox_event` in the same transaction; `outbox.py` relays every 1 s; inventory's `consumer.py` commits the hold on OrderConfirmed and dedupes via `consumed_event`. `POST /debug/bus` diff --git a/README.md b/README.md index eef5009..a03671e 100644 --- a/README.md +++ b/README.md @@ -6,8 +6,8 @@ can click. > **Status: complete.** All 12 patterns are live, each with a working demo, locally and in one > real Azure run: provision, deploy through GitHub Actions with OIDC, smoke tests, screenshots, -> teardown. Demo 13, a slow payment in the middle of checkout, came from a reader's comment on -> the project's LinkedIn post. The foundation is copied from +> teardown. Demos 13 (a slow payment in the middle of checkout) and 14 (the saga's undo under +> timeouts and duplicate events) came from readers' comments on the project's LinkedIn post. The foundation is copied from > [Azure-Ecommerce-Monolith-Microservices-Migration](https://github.com/MAOFILHO/Azure-Ecommerce-Monolith-Microservices-Migration). ## Project summary @@ -27,15 +27,16 @@ React frontend with one page per pattern. Every demo calls the real services: no | M3a: async and packaging | 7, 8, 9 | [#4](https://github.com/MAOFILHO/Microservices-12-Patterns/pull/4) | | M3b: delivery and the Azure run | 3 | [#12](https://github.com/MAOFILHO/Microservices-12-Patterns/pull/12), [#13](https://github.com/MAOFILHO/Microservices-12-Patterns/pull/13), [#14](https://github.com/MAOFILHO/Microservices-12-Patterns/pull/14) | | Demo 13: a slow payment mid-checkout | 4 and 8, under a timeout | [#16](https://github.com/MAOFILHO/Microservices-12-Patterns/pull/16) | +| Demo 14: compensation under fire | 7 and 8, under timeouts and duplicates | [#17](https://github.com/MAOFILHO/Microservices-12-Patterns/pull/17) | M3b was planned as 7 GitHub issues ([#5 to #11](https://github.com/MAOFILHO/Microservices-12-Patterns/issues?q=is%3Aissue)), all closed. **How it was checked.** -- **181 unit tests** across 8 suites, each in its own virtualenv, built test-first from M3a on. +- **194 unit tests** across 8 suites, each in its own virtualenv, built test-first from M3a on. - **Pact contract tests**: the order service's contracts with payment, catalog and inventory, verified against the real providers in CI. -- **25 local smoke tests** against the running stack, and **4 Azure smoke tests** against the real +- **28 local smoke tests** against the running stack, and **4 Azure smoke tests** against the real deployment. - **CI on every pull request** runs only what the change touches; every image is built for linux/amd64, started, and rejected if it runs as root. `main` requires the "CI ok" check. @@ -62,6 +63,7 @@ or the cheapest SKUs, with a $25/month budget alert, then torn down and verified | 11 | [Contract Testing](#11-contract-testing) | Checks that APIs stay compatible across services | M2 | | 12 | [Service Discovery](#12-service-discovery) | Finds service instances automatically at runtime | M2 | | 13 | [Saga under a slow payment](#13-saga-under-a-slow-payment) | Follows the money and the stock when payment times out mid-checkout | Demo 13 | +| 14 | [Compensation under fire](#14-compensation-under-fire) | Times out and duplicates the saga's undo, then checks the stock ends where it began | Demo 14 | ## Architecture @@ -213,6 +215,24 @@ payment still said PROCESSING, so the order waited; 5 s later it settled as 1 ch Demo 13: payment slowed mid-checkout; the order waits as pending, then settles as 1 charge, 1 order, 2 mugs sold +### 14. Compensation under fire +An undo is a new action, not a rewind, and it crosses the same unreliable network. A second +reader asked for duplicate events and timeouts during compensation. Three cases: + +- **A release that beats its hold.** Inventory holds slowly (6 s); the saga gives up after 3 s and + sends a release as an event, which arrives first (about 4 s). Inventory records the release, so + the late hold holds nothing. Before this demo, that late hold kept the stock held for good. +- **A release that times out but goes through.** The saga also sends it as an event. The event copy + lands first, the slow direct call second. Release only acts on a hold still RESERVED, so the + stock comes back once; the page shows both copies arrived. +- **Duplicate events.** A replay button sends a checkout's events again with the same ids. + Inventory acts on 2 event types: one it has acted on before is skipped and counted. The other + types are not its business, so it ignores them (also counted). + +Here the hold arrived late: the release came first, the stock went 473 to 473, nothing held, 1 release received. + +Demo 14: inventory holds slowly; the saga's release arrives first, the late hold holds nothing, and the stock ends where it began + ## Run locally (no Docker) ```bash diff --git a/docs/screenshots/p14-compensation.jpg b/docs/screenshots/p14-compensation.jpg new file mode 100644 index 0000000..4a3d71f Binary files /dev/null and b/docs/screenshots/p14-compensation.jpg differ diff --git a/frontend/src/api/gatewayClient.ts b/frontend/src/api/gatewayClient.ts index 24e159d..51d2a74 100644 --- a/frontend/src/api/gatewayClient.ts +++ b/frontend/src/api/gatewayClient.ts @@ -349,3 +349,38 @@ export interface DeliveryView { export const deliveryApi = { view: () => request("/api/patterns/delivery"), }; + +// --- Demo 14: compensation under timeouts and duplicate events ----------------------------------- + +export interface HoldRow { + reservation_id: string; + order_id: number; + status: "RESERVED" | "SOLD" | "RELEASED" | string; + items: { product_id: number; quantity: number }[]; + releases_received: number; +} + +export interface HandledEvent { + event_id: string; + type: string; + reservation_id: string | null; + outcome: string; +} + +export const compensationApi = { + inventoryMode: () => request<{ mode: string; enabled: boolean }>("/api/shop/inventory/chaos"), + setInventoryMode: (mode: string) => + getAny<{ mode?: string; message?: string }>("/api/shop/inventory/chaos", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ mode }), + }), + holds: () => request<{ results: HoldRow[] }>("/api/shop/inventory/reservations"), + handled: () => + request<{ results: HandledEvent[]; duplicates_skipped: number; not_handled: number }>("/api/shop/inventory/events"), + replay: (token: string, sagaId: string) => + getAny<{ replayed?: number; message?: string }>(`/api/shop/order/sagas/${sagaId}/replay`, { + method: "POST", + headers: authHeaders(token), + }), +}; diff --git a/frontend/src/pages/Home.tsx b/frontend/src/pages/Home.tsx index 96c0256..96032c0 100644 --- a/frontend/src/pages/Home.tsx +++ b/frontend/src/pages/Home.tsx @@ -18,7 +18,7 @@ export default function Home() {

Patterns

-

The 12 patterns. All 12 are live, each with a working demo, plus demo 13: a slow payment in the middle of checkout.

+

The 12 patterns. All 12 are live, each with a working demo, plus demo 13 (a slow payment in the middle of checkout) and demo 14 (the saga's undo under timeouts and duplicate events).

diff --git a/frontend/src/pages/PatternPage.tsx b/frontend/src/pages/PatternPage.tsx index 8a951be..a30abee 100644 --- a/frontend/src/pages/PatternPage.tsx +++ b/frontend/src/pages/PatternPage.tsx @@ -12,6 +12,7 @@ import ObservabilityDemo from "../patterns/demos/ObservabilityDemo"; import SagaDemo from "../patterns/demos/SagaDemo"; import ServiceDiscoveryDemo from "../patterns/demos/ServiceDiscoveryDemo"; import SlowPaymentDemo from "../patterns/demos/SlowPaymentDemo"; +import CompensationDemo from "../patterns/demos/CompensationDemo"; import StatelessDemo from "../patterns/demos/StatelessDemo"; import { PATTERNS } from "../patterns/content"; @@ -29,6 +30,7 @@ const DEMOS: Record ReactNode> = { containerization: () => , "ci-cd-pipeline": () => , "slow-payment": () => , + compensation: () => , }; export default function PatternPage() { diff --git a/frontend/src/pages/Patterns.tsx b/frontend/src/pages/Patterns.tsx index fb59885..0764e8a 100644 --- a/frontend/src/pages/Patterns.tsx +++ b/frontend/src/pages/Patterns.tsx @@ -2,15 +2,15 @@ import { Link } from "react-router-dom"; import { PATTERNS } from "../patterns/content"; export default function Patterns() { - const patterns = PATTERNS.filter((p) => p.n <= 12); // demo 13 is a bonus, not a 13th pattern + const patterns = PATTERNS.filter((p) => p.n <= 12); // demos 13 and 14 are bonuses, not patterns const liveCount = patterns.filter((p) => p.live).length; return (
-

The 12 patterns, plus demo 13

+

The 12 patterns, plus demos 13 and 14

Each pattern has a page that explains it, says when not to use it, and runs a live demo against the real - services. {liveCount} of {patterns.length} are live. Demo 13 puts patterns 4 and 8 under a slow payment. + services. {liveCount} of {patterns.length} are live. Demo 13 puts patterns 4 and 8 under a slow payment; demo 14 puts the saga's undo under timeouts and duplicate events.

diff --git a/frontend/src/patterns/content.ts b/frontend/src/patterns/content.ts index 9b31e12..ffce541 100644 --- a/frontend/src/patterns/content.ts +++ b/frontend/src/patterns/content.ts @@ -303,6 +303,29 @@ export const PATTERNS: PatternInfo[] = [ ], source: { label: "Stripe — Idempotent requests", url: "https://docs.stripe.com/api/idempotent_requests" }, }, + { + n: 14, + slug: "compensation", + name: "Compensation under fire", + what: "Times out and duplicates the saga's undo, then checks the stock ends where it began.", + milestone: "M3", + live: true, + problem: + "An undo is a new action, not a rewind, and it travels over the same unreliable network. A release can time out yet go through, so it is sent twice. A release can overtake the slow hold it undoes, so the hold lands afterwards with nothing left to cancel it. And at-least-once delivery sends any event twice.", + inThisProject: [ + "Inventory has a demo switch: hold slowly (6 s, past the saga's 3 s wait) or release slowly.", + "When the hold times out, the saga sends its release as an event; it reaches inventory in about 4 s, before the 6 s hold. Inventory records a release for a hold it never saw as RELEASED (and logs it). When the late hold arrives, it finds that record and holds nothing.", + "A release that times out is also sent as an event. The event copy lands first, the slow direct call second. Release only acts on a hold that is still RESERVED, so the second copy changes nothing; the page counts both copies.", + "Inventory acts on 2 event types, OrderConfirmed and StockReleaseRequested. It remembers every id it acted on and skips one it has seen; other types are not its business and it ignores them. The replay button sends a checkout's events again with the same ids, and the page counts what was skipped and what was ignored.", + "Raised by a reader of the project's LinkedIn post, after demo 13.", + ], + whenNotToUse: [ + "Recording releases for unknown holds keeps rows that never held anything. At scale they need a retention window, longer than any request can be late.", + "An undo that touches the outside world (an email sent, money refunded) cannot be made invisible: the customer sees both actions. Here the undo only moves stock inside one service.", + "Event-id dedupe needs the ids kept as long as a duplicate can arrive. A bus with a short redelivery window can keep fewer.", + ], + source: { label: "Microsoft — Compensating Transaction pattern", url: "https://learn.microsoft.com/azure/architecture/patterns/compensating-transaction" }, + }, ]; export const SERVICE_NAMES = ["gateway", "user", "catalog", "order", "payment", "inventory"] as const; diff --git a/frontend/src/patterns/demos/CompensationDemo.tsx b/frontend/src/patterns/demos/CompensationDemo.tsx new file mode 100644 index 0000000..da0ba16 --- /dev/null +++ b/frontend/src/patterns/demos/CompensationDemo.tsx @@ -0,0 +1,213 @@ +import { useEffect, useState } from "react"; +import { Link } from "react-router-dom"; +import { + HandledEvent, + HoldRow, + SagaRun, + StockRow, + asyncApi, + compensationApi, + savedToken, +} from "../../api/gatewayClient"; +import { PILL, STEP_LABEL } from "./SagaDemo"; + +const DEMO_PRODUCT = 1; +const POLL_MS = 2000; + +const MODES: { mode: string; label: string; what: string }[] = [ + { mode: "slow_reserve", label: "Hold arrives late", what: "Inventory takes the hold 6 s late. The saga waits 3 s, gives up, and sends a release, which arrives first." }, + { mode: "slow_release", label: "Release times out", what: "Inventory releases 6 s late. The saga's direct release times out after 3 s, so it also sends the release as an event. The event copy lands first (about 4 s), the direct call second (6 s): 2 copies, 1 release." }, + { mode: "healthy", label: "Healthy", what: "Inventory answers at once." }, +]; + +/** Demo 14: compensation under timeouts and duplicate events. Does the stock end where it began? */ +export default function CompensationDemo() { + const token = savedToken(); + const [mode, setMode] = useState("unknown"); + const [error, setError] = useState(null); + const [busy, setBusy] = useState(false); + const [answer, setAnswer] = useState(null); + const [before, setBefore] = useState(null); + const [stock, setStock] = useState(null); + const [saga, setSaga] = useState(null); + const [hold, setHold] = useState(null); + const [handled, setHandled] = useState([]); + const [counts, setCounts] = useState<{ skipped: number; notHandled: number } | null>(null); + const [replayed, setReplayed] = useState<{ count: number; skippedAt: number; notHandledAt: number } | null>(null); + + async function refresh(): Promise { + if (!token) return null; + const [history, view, holds, events, chaos] = await Promise.all([ + asyncApi.sagas(token).catch(() => null), + asyncApi.events().catch(() => null), + compensationApi.holds().catch(() => null), + compensationApi.handled().catch(() => null), + compensationApi.inventoryMode().catch(() => null), + ]); + const latest = history?.results[0] ?? null; + const row = view?.stock?.find((s) => s.product_id === DEMO_PRODUCT) ?? null; + setSaga(latest); + setStock(row); + setHold(latest ? holds?.results.find((h) => h.reservation_id === latest.saga_id) ?? null : null); + setHandled(latest && events ? events.results.filter((e) => e.reservation_id === latest.saga_id) : []); + if (events) setCounts({ skipped: events.duplicates_skipped, notHandled: events.not_handled }); + if (chaos) setMode(chaos.mode); + return row; + } + + useEffect(() => { + refresh(); + const timer = setInterval(refresh, POLL_MS); + return () => clearInterval(timer); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, []); + + if (!token) { + return ( +

+ This demo checks out a real cart, so it needs a signed-in user. Register or log in on the Shop + page, then come back. +

+ ); + } + + async function switchInventory(next: string) { + setError(null); + const { status, body } = await compensationApi.setInventoryMode(next); + if (status === 200 && body.mode) setMode(body.mode); + else setError(body.message ?? `HTTP ${status}`); + } + + async function buy(decline: boolean) { + if (!token) return; + setBusy(true); + setAnswer(null); + setReplayed(null); + try { + setBefore(await refresh()); + await asyncApi.addToCart(token, DEMO_PRODUCT, 2); + const { status, body } = await asyncApi.checkout(token, decline); + setAnswer(`HTTP ${status}: ${body.message ?? (status === 200 ? "Order placed." : "")}`); + } catch (err) { + setAnswer(String(err)); + } finally { + await refresh(); + setBusy(false); + } + } + + async function replay() { + if (!token || !saga) return; + setError(null); + const { status, body } = await compensationApi.replay(token, saga.saga_id); + if (status === 200) + setReplayed({ count: body.replayed ?? 0, skippedAt: counts?.skipped ?? 0, notHandledAt: counts?.notHandled ?? 0 }); + else setError(body.message ?? `HTTP ${status}`); + } + + const restored = before && stock && stock.on_hand === before.on_hand && stock.reserved === before.reserved; + const sold = before && stock && stock.on_hand === before.on_hand - 2 && stock.reserved === before.reserved; + const verdict = !before || !stock + ? null + : restored + ? "Back where it began: same mugs on hand, nothing held." + : sold + ? "Sold once: 2 mugs fewer on hand, nothing held." + : `Not settled yet: on hand ${before.on_hand} -> ${stock.on_hand}, held ${before.reserved} -> ${stock.reserved}.`; + + return ( +
+
+
+

Step 1. Pick what goes wrong in inventory

+
+ {MODES.map((m) => ( + + ))} +
+

{MODES.find((m) => m.mode === mode)?.what ?? "Inventory mode unknown."}

+ {error &&

{error}

} + +

Step 2. Buy 2 Contoso Mugs

+
+ + +
+

+ "Hold arrives late" goes with Buy. "Release times out" goes with card declined, so the saga has to undo. +

+ {busy &&

Checking out…

} + {answer &&

{answer}

} + +

Step 3. Send this checkout's events again

+ + {replayed && counts && ( +

+ Sent {replayed.count} event(s) again with the same ids. Since then, inventory skipped{" "} + {counts.skipped - replayed.skippedAt} it had already acted on, and ignored{" "} + {counts.notHandled - replayed.notHandledAt} that are not its business (counts include any other checkout + running now). +

+ )} +
+ +
+

Where the stock is

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
Hold for this checkout{hold ? `${hold.status}${hold.items.length === 0 ? " (release came first, nothing held)" : ""}` : "none yet"}
Release requests received{hold ? hold.releases_received : "–"}
Mugs on hand{before && stock ? `${before.on_hand} -> ${stock.on_hand}` : stock?.on_hand ?? "?"}
Mugs held for checkouts{before && stock ? `${before.reserved} -> ${stock.reserved}` : stock?.reserved ?? "?"}
Events inventory acted on{handled.length === 0 ? "none" : handled.map((e) => `${e.type}: ${e.outcome}`).join("; ")}
Duplicate events skipped (all checkouts){counts?.skipped ?? "?"}
+ {verdict &&

{verdict}

} +

Refreshes every 2 s. A slow step lands about 6 s after the click.

+
+
+ + {saga && ( +
+

What the saga did

+
    + {saga.steps.map((s, i) => ( +
  1. + {s.status} + {STEP_LABEL[s.step] ?? s.step} + {s.detail && · {s.detail}} +
  2. + ))} +
+
+ )} +
+ ); +} diff --git a/services/inventory/application/__init__.py b/services/inventory/application/__init__.py index c200054..5ad15e5 100644 --- a/services/inventory/application/__init__.py +++ b/services/inventory/application/__init__.py @@ -34,6 +34,9 @@ def create_app(config_name: str | None = None) -> Flask: app.register_blueprint(inventory_api) db.create_all() + from shop_common.db_setup import add_missing_columns + + add_missing_columns(db) # demo 14: reservation.releases_received from .seed import seed_missing_stock seed_missing_stock() diff --git a/services/inventory/application/consumer.py b/services/inventory/application/consumer.py index 9102b84..503a29b 100644 --- a/services/inventory/application/consumer.py +++ b/services/inventory/application/consumer.py @@ -8,13 +8,18 @@ from .models import ConsumedEvent HANDLERS = {"OrderConfirmed": stock.commit, "StockReleaseRequested": stock.release} +# Demo 14: what this process did with events it did not act on. In memory: demo counters, not data. +# duplicates_skipped: an event id it already acted on. not_handled: a type that is not inventory's. +counts = {"duplicates_skipped": 0, "not_handled": 0} def handle(event: dict) -> None: action = HANDLERS.get(event.get("type")) if action is None: + counts["not_handled"] += 1 return if db.session.get(ConsumedEvent, event["event_id"]) is not None: + counts["duplicates_skipped"] += 1 return # seen before: at-least-once delivery sent it twice reservation_id = event["payload"].get("reservation_id") # action() commits on its own. If we crash before the ConsumedEvent row below, the event is diff --git a/services/inventory/application/models.py b/services/inventory/application/models.py index e77bf13..0d6f984 100644 --- a/services/inventory/application/models.py +++ b/services/inventory/application/models.py @@ -29,6 +29,10 @@ def _utcnow(): return datetime.now(timezone.utc) +# Demo 14: the order id of a hold recorded only by its release (the hold itself never arrived). +NEVER_HELD = 0 + + class Reservation(db.Model): """Stock held for one checkout attempt (pattern 8). Keyed by the order service's saga id, so a retried request or a redelivered event finds the same row instead of holding stock twice.""" @@ -39,6 +43,8 @@ class Reservation(db.Model): order_id = db.Column(db.Integer, nullable=False, index=True) items = db.Column(db.JSON, nullable=False) # [{"product_id": 1, "quantity": 2}] status = db.Column(db.String(10), nullable=False) # RESERVED -> SOLD | RELEASED + # Demo 14: how many release requests arrived (a direct call and its event copy can both come). + releases_received = db.Column(db.Integer, nullable=True, default=0) created_at = db.Column(db.DateTime, default=_utcnow) updated_at = db.Column(db.DateTime, default=_utcnow, onupdate=_utcnow) @@ -48,6 +54,7 @@ def to_json(self): "order_id": self.order_id, "items": self.items, "status": self.status, + "releases_received": self.releases_received or 0, "updated_at": self.updated_at.isoformat() if self.updated_at else None, } diff --git a/services/inventory/application/routes.py b/services/inventory/application/routes.py index 6587d5b..6f4b049 100644 --- a/services/inventory/application/routes.py +++ b/services/inventory/application/routes.py @@ -1,12 +1,32 @@ +import time + from flask import Blueprint, jsonify, make_response, request from pydantic import BaseModel, Field, ValidationError -from shop_common.flask_support import chaos_enabled +from shop_common.flask_support import DemoSwitch, chaos_enabled, switches_off -from . import db, stock +from . import consumer, db, stock from .models import ConsumedEvent, Reservation, StockItem inventory_api = Blueprint("inventory_api", __name__) +# Demo 14 switch: "slow_reserve" makes a hold land after the saga gave up on it (its timeout is +# 3 s), so the saga's release arrives first. "slow_release" makes the saga's direct release time +# out although it goes through, so the event copy arrives too. In memory: a lab switch, not data. +SWITCH = DemoSwitch("healthy", "slow_reserve", "slow_release") +SLOW_SECONDS = 6 +_SLOWED = {"slow_reserve": "inventory_api.reserve", "slow_release": "inventory_api.release"} + + +@inventory_api.before_request +def _apply_chaos(): + if _SLOWED.get(SWITCH.value) == request.endpoint: + time.sleep(SLOW_SECONDS) + + +@inventory_api.route("/api/inventory/chaos", methods=["GET", "POST"]) +def chaos(): + return SWITCH.respond() + class Line(BaseModel): product_id: int = Field(gt=0) @@ -47,10 +67,9 @@ def reserve(): @inventory_api.post("/api/inventory/reservations//release") def release(reservation_id: str): - reservation = stock.release(reservation_id) - if reservation is None: - return make_response(jsonify({"message": "No such reservation"}), 404) - return jsonify({"result": reservation.to_json()}) + if not 8 <= len(reservation_id) <= 36: # the same rule as a hold's id + return make_response(jsonify({"message": "A hold id is 8 to 36 characters"}), 400) + return jsonify({"result": stock.release(reservation_id).to_json()}) @inventory_api.get("/api/inventory/reservations") @@ -62,7 +81,7 @@ def list_reservations(): @inventory_api.get("/api/inventory/events") def consumed_events(): rows = ConsumedEvent.query.order_by(ConsumedEvent.handled_at.desc()).limit(20) - return jsonify({"results": [e.to_json() for e in rows]}) + return jsonify({"results": [e.to_json() for e in rows], **consumer.counts}) class RestockRequest(BaseModel): @@ -73,7 +92,7 @@ class RestockRequest(BaseModel): def restock(product_id: int): """Demo and smoke-test helper: adds units on hand. Only with CHAOS_ENABLED=1 (make run).""" if not chaos_enabled(): - return make_response(jsonify({"message": "Demo switches are off (CHAOS_ENABLED is not 1)"}), 403) + return switches_off() try: data = RestockRequest.model_validate(request.get_json(silent=True) or {}) except ValidationError as exc: diff --git a/services/inventory/application/stock.py b/services/inventory/application/stock.py index 7bc538a..f383f74 100644 --- a/services/inventory/application/stock.py +++ b/services/inventory/application/stock.py @@ -1,15 +1,20 @@ """Stock rules for the checkout saga (pattern 8) and the event consumer (pattern 7). reserve: available -> reserved (saga step 1; all lines or none) -release: reserved -> available (saga compensation) +release: reserved -> available (saga compensation; an unknown hold is recorded as RELEASED, + so a hold that arrives late is refused: demo 14) commit: reserved -> sold (on OrderConfirmed: on hand goes down) Each is idempotent: calling it again for the same reservation changes nothing. """ +import logging + from sqlalchemy.exc import IntegrityError from . import db -from .models import Reservation, StockItem +from .models import NEVER_HELD, Reservation, StockItem + +log = logging.getLogger(__name__) class OutOfStock(Exception): @@ -29,7 +34,7 @@ def _in_lock_order(items: list[dict]) -> list[dict]: def reserve(reservation_id: str, order_id: int, items: list[dict]) -> Reservation: existing = db.session.get(Reservation, reservation_id) - if existing is not None: + if existing is not None: # a retry, or a hold its release beat here: either way, hold nothing new return existing items = _in_lock_order(items) short = [] @@ -53,9 +58,24 @@ def reserve(reservation_id: str, order_id: int, items: list[dict]) -> Reservatio return reservation -def release(reservation_id: str) -> Reservation | None: +def release(reservation_id: str) -> Reservation: reservation = db.session.get(Reservation, reservation_id) - if reservation is None or reservation.status != "RESERVED": + if reservation is None: + # Demo 14: the release beat its hold here (the hold request was slow and the saga gave + # up). Record the release, so the late hold finds it and holds nothing. + log.warning("release for hold %s, which never arrived: recorded so a late hold is refused", reservation_id) + reservation = Reservation(id=reservation_id, order_id=NEVER_HELD, items=[], status="RELEASED", + releases_received=1) + db.session.add(reservation) + try: + db.session.commit() + except IntegrityError: # the hold or another release landed at the same moment + db.session.rollback() + return release(reservation_id) + return reservation + reservation.releases_received = (reservation.releases_received or 0) + 1 + if reservation.status != "RESERVED": + db.session.commit() return reservation for line in _in_lock_order(reservation.items): _lock(line["product_id"]).reserved -= line["quantity"] diff --git a/services/inventory/tests/test_compensation.py b/services/inventory/tests/test_compensation.py new file mode 100644 index 0000000..78ffa36 --- /dev/null +++ b/services/inventory/tests/test_compensation.py @@ -0,0 +1,135 @@ +"""Demo 14: compensation under timeouts and duplicate events. + +A saga's release can arrive before its hold (the hold request was slow, the saga gave up and sent +a release), and the same release can arrive twice (the direct call timed out but went through, +and the event copy follows). Either way the stock must end up exactly as before the checkout. +""" +import pytest + +from application import consumer, db, routes +from application.models import StockItem +from application.seed import seed_missing_stock + +RID = "saga-0014-aaaa" + + +@pytest.fixture +def seeded(app): + seed_missing_stock() + return app + + +def _stock(product_id=1): + return db.session.get(StockItem, product_id).to_json() + + +def _reserve(client, rid=RID, qty=2): + return client.post("/api/inventory/reservations", + json={"reservation_id": rid, "order_id": 7, "items": [{"product_id": 1, "quantity": qty}]}) + + +def _release(client, rid=RID): + return client.post(f"/api/inventory/reservations/{rid}/release") + + +def _mode(client, monkeypatch, mode): + monkeypatch.setenv("CHAOS_ENABLED", "1") + return client.post("/api/inventory/chaos", json={"mode": mode}) + + +@pytest.fixture(autouse=True) +def _fresh(): + """The switch and the counters live in memory for the whole test run: reset them per test.""" + consumer.counts.update(duplicates_skipped=0, not_handled=0) + yield + routes.SWITCH.value = "healthy" + + +def _counts(client): + body = client.get("/api/inventory/events").get_json() + return body["duplicates_skipped"], body["not_handled"] + + +def test_a_release_that_arrives_before_its_hold_refuses_the_late_hold(seeded, client): + before = _stock() + assert _release(client).get_json()["result"]["status"] == "RELEASED" # nothing held yet + late = _reserve(client) + assert late.status_code == 200 + assert late.get_json()["result"]["status"] == "RELEASED" # the late hold holds nothing + assert _stock() == before + + +def test_a_release_sent_twice_returns_the_stock_once_and_counts_both_copies(seeded, client): + before = _stock() + _reserve(client) + consumer.handle({"event_id": "evt-r1", "type": "StockReleaseRequested", "payload": {"reservation_id": RID}}) + late = _release(client) # the direct call that timed out, landing after the event copy + assert _stock() == before + assert late.get_json()["result"]["releases_received"] == 2 # the page shows both copies arrived + + +def test_a_release_for_a_hold_never_seen_is_logged(seeded, client, caplog): + with caplog.at_level("WARNING"): + _release(client) + assert RID in caplog.text + + +def test_a_release_id_must_look_like_a_hold_id(seeded, client): + assert _release(client, "x").status_code == 400 + assert _release(client, "y" * 37).status_code == 400 + + +def test_a_replayed_checkout_is_skipped_or_ignored_event_by_event(seeded, client): + _reserve(client, qty=3) + checkout = [{"event_id": f"evt-{t}", "type": t, "payload": {"reservation_id": RID}} + for t in ("StockReserved", "PaymentCharged", "OrderConfirmed")] + for event in checkout: + consumer.handle(event) + sold = _stock() + for event in checkout: # redelivered + consumer.handle(event) + assert _stock() == sold + assert _counts(client) == (1, 4) # OrderConfirmed skipped once; the other 2 are not inventory's, twice + + +def test_a_database_from_before_demo_14_gains_the_release_count(app): + from sqlalchemy import inspect, text + + from application import create_app + + db.drop_all() + db.session.execute(text("CREATE TABLE reservation (id VARCHAR(36) PRIMARY KEY, order_id INTEGER NOT NULL, " + "items JSON NOT NULL, status VARCHAR(10) NOT NULL, created_at DATETIME, updated_at DATETIME)")) + db.session.commit() + create_app("development") + assert "releases_received" in {c["name"] for c in inspect(db.engine).get_columns("reservation")} + + +def test_slow_reserve_delays_only_the_hold(seeded, client, monkeypatch): + slept = [] + monkeypatch.setattr(routes.time, "sleep", slept.append) + assert _mode(client, monkeypatch, "slow_reserve").status_code == 200 + _release(client, "saga-other-1") + assert slept == [] + _reserve(client) + assert slept == [routes.SLOW_SECONDS] + + +def test_slow_release_delays_only_the_release(seeded, client, monkeypatch): + slept = [] + monkeypatch.setattr(routes.time, "sleep", slept.append) + _mode(client, monkeypatch, "slow_release") + _reserve(client) + assert slept == [] + _release(client) + assert slept == [routes.SLOW_SECONDS] + + +def test_the_inventory_switch_is_off_unless_chaos_is_enabled(seeded, client, monkeypatch): + monkeypatch.delenv("CHAOS_ENABLED", raising=False) + assert client.post("/api/inventory/chaos", json={"mode": "slow_reserve"}).status_code == 403 + assert client.get("/api/inventory/chaos").get_json()["mode"] == "healthy" + + +def test_the_inventory_switch_rejects_an_unknown_mode(seeded, client, monkeypatch): + assert _mode(client, monkeypatch, "broken").status_code == 400 diff --git a/services/order/application/order_api/routes.py b/services/order/application/order_api/routes.py index 0cf12ad..87844ab 100644 --- a/services/order/application/order_api/routes.py +++ b/services/order/application/order_api/routes.py @@ -3,10 +3,10 @@ from flask import jsonify, make_response, request from . import order_api_blueprint -from shop_common.flask_support import chaos_enabled +from shop_common.flask_support import chaos_enabled, switches_off from .. import db, outbox, saga -from ..models import Order, OrderItem, OutboxEvent +from ..models import Order, OrderItem, OutboxEvent, SagaStep from ..schemas import OrderAddItemRequest from ..validation import validate_form from .api import clients @@ -80,7 +80,7 @@ def checkout(): options = request.get_json(silent=True) or request.form simulate = "decline" if str(options.get("simulate_decline", "")).lower() in ("1", "true", "yes") else "approve" if simulate == "decline" and not chaos_enabled(): - return make_response(jsonify({"message": "Demo switches are off (CHAOS_ENABLED is not 1)"}), 403) + return switches_off() status, body = saga.run_checkout(order_model, u_id, simulate) return make_response(jsonify(body), status) @@ -93,6 +93,27 @@ def sagas(): return jsonify({"results": saga.recent_sagas(u_id)}) +# Demo 14: send a checkout's events to the bus again, with the same ids, as at-least-once delivery +# can. Inventory should skip every one. Only your own checkout, only with the demo switches on. +@order_api_blueprint.route("/api/order/sagas//replay", methods=["POST"]) +def replay_saga_events(saga_id: str): + if not chaos_enabled(): + return switches_off() + u_id = _user_id() + if u_id is None: + return _not_logged_in() + first = SagaStep.query.filter_by(saga_id=saga_id, user_id=u_id).order_by(SagaStep.id).first() + if first is None: + return make_response(jsonify({"message": "No such checkout"}), 404) + # Its events are all written after its first step: scan from there, not the whole outbox. + since = OutboxEvent.query.filter(OutboxEvent.created_at >= first.at).all() + events = [e for e in since if e.payload.get("saga_id") == saga_id] + for event in events: + event.published_at = None # the relay sends it again within 1 s + db.session.commit() + return jsonify({"replayed": len(events), "event_ids": [e.id for e in events]}) + + # Pattern 7 demo: what is waiting in the outbox and what has been published. No personal data: # events carry ids, quantities and amounts only. @order_api_blueprint.route("/api/order/outbox", methods=["GET"]) @@ -106,7 +127,7 @@ def outbox_events(): @order_api_blueprint.route("/debug/bus", methods=["POST"]) def bus_switch(): if not chaos_enabled(): - return make_response(jsonify({"message": "Demo switches are off (CHAOS_ENABLED is not 1)"}), 403) + return switches_off() connected = bool((request.get_json(silent=True) or {}).get("connected", True)) outbox.set_connected(connected) return jsonify({"bus_connected": connected}) diff --git a/services/order/tests/test_saga_and_outbox.py b/services/order/tests/test_saga_and_outbox.py index 8e5b740..0880291 100644 --- a/services/order/tests/test_saga_and_outbox.py +++ b/services/order/tests/test_saga_and_outbox.py @@ -161,3 +161,27 @@ def timeout(rid, oid, items): assert r.status_code == 503 assert _events() == ["StockReleaseRequested", "CheckoutFailed"] assert OutboxEvent.query.filter_by(type="StockReleaseRequested").one().payload["reservation_id"] == r.get_json()["saga_id"] + + +# --- Demo 14: replay a checkout's events, to show inventory skips duplicates --------------------- + +def test_replay_sends_a_checkouts_events_again_with_the_same_ids(client, calls): + saga_id = _checkout(client).get_json()["saga_id"] + OutboxEvent.query.update({"published_at": outbox.datetime.now(outbox.timezone.utc)}) + ids = sorted(e.id for e in OutboxEvent.query) + r = client.post(f"/api/order/sagas/{saga_id}/replay", headers=AUTH) + assert r.status_code == 200 + assert r.get_json()["replayed"] == 3 + assert sorted(e.id for e in OutboxEvent.query.filter(OutboxEvent.published_at.is_(None))) == ids + + +def test_replay_is_only_for_your_own_checkout(client, calls, monkeypatch): + saga_id = _checkout(client).get_json()["saga_id"] + monkeypatch.setattr(clients, "current_user_id", lambda header: 2) + assert client.post(f"/api/order/sagas/{saga_id}/replay", headers=AUTH).status_code == 404 + + +def test_replay_is_off_unless_chaos_is_enabled(client, calls, monkeypatch): + saga_id = _checkout(client).get_json()["saga_id"] + monkeypatch.delenv("CHAOS_ENABLED") + assert client.post(f"/api/order/sagas/{saga_id}/replay", headers=AUTH).status_code == 403 diff --git a/services/payment/application/routes.py b/services/payment/application/routes.py index abfd895..0cfb396 100644 --- a/services/payment/application/routes.py +++ b/services/payment/application/routes.py @@ -4,7 +4,7 @@ from flask import Blueprint, jsonify, make_response, request from pydantic import BaseModel, Field, ValidationError -from shop_common.flask_support import chaos_enabled +from shop_common.flask_support import DemoSwitch from sqlalchemy.exc import IntegrityError from . import db @@ -16,9 +16,8 @@ # SLOW_SECONDS, longer than the order service waits). Held in memory: it is a lab switch, not data. # The switch makes charging sick, not settling: void and the list always answer, so the demo 13 # page can show where the money is while payment is slow. -MODES = ("healthy", "failing", "slow") +SWITCH = DemoSwitch("healthy", "failing", "slow") SLOW_SECONDS = 3 -_mode = {"value": "healthy"} _ALWAYS_ANSWERS = {"payment_api.chaos", "payment_api.void", "payment_api.list_payments"} # Demo 13: a charge still PROCESSING after this long lost its process (a crash or restart), so a # void may cancel it. Far longer than any charge takes, slow mode included. @@ -42,9 +41,9 @@ class ChargeRequest(VoidRequest): def _apply_chaos(): if request.endpoint in _ALWAYS_ANSWERS: return None - if _mode["value"] == "failing": + if SWITCH.value == "failing": return make_response(jsonify({"message": "Payment is failing (demo switch)"}), 500) - if _mode["value"] == "slow" and request.endpoint != "payment_api.charge": # charge() is slow itself + if SWITCH.value == "slow" and request.endpoint != "payment_api.charge": # charge() is slow itself time.sleep(SLOW_SECONDS) return None @@ -99,7 +98,7 @@ def charge(): if (payment.order_id, payment.amount_cents) != (data.order_id, data.amount_cents): return make_response(jsonify({"message": "This key was already used for a different charge"}), 409) return jsonify({"result": payment.to_json()}) - if _mode["value"] == "slow": + if SWITCH.value == "slow": time.sleep(SLOW_SECONDS) return jsonify({"result": _finish(payment, "DECLINED" if data.simulate == "decline" else "APPROVED").to_json()}) @@ -122,11 +121,4 @@ def void(key: str): @payment_api.route("/api/payment/chaos", methods=["GET", "POST"]) def chaos(): - if request.method == "POST": - if not chaos_enabled(): - return make_response(jsonify({"message": "Chaos switches are off (CHAOS_ENABLED is not 1)"}), 403) - mode = (request.get_json(silent=True) or request.form.to_dict()).get("mode") - if mode not in MODES: - return make_response(jsonify({"message": f"mode must be one of {list(MODES)}"}), 400) - _mode["value"] = mode - return jsonify({"mode": _mode["value"], "modes": list(MODES), "enabled": chaos_enabled()}) + return SWITCH.respond() diff --git a/services/payment/tests/test_contract_provider.py b/services/payment/tests/test_contract_provider.py index 7a95fd4..3e8a692 100644 --- a/services/payment/tests/test_contract_provider.py +++ b/services/payment/tests/test_contract_provider.py @@ -19,7 +19,7 @@ def test_payment_honours_the_order_services_contract(app): db.session.remove() def healthy() -> None: - routes._mode["value"] = "healthy" + routes.SWITCH.value = "healthy" def charge_exists() -> None: healthy() diff --git a/services/payment/tests/test_payment.py b/services/payment/tests/test_payment.py index 9ae2cf7..757f7d1 100644 --- a/services/payment/tests/test_payment.py +++ b/services/payment/tests/test_payment.py @@ -115,7 +115,7 @@ def test_a_charge_voided_while_it_ran_does_not_approve_when_it_finishes(client, try: assert _charge(client, "saga-1").get_json()["result"]["status"] == "VOIDED" finally: - routes._mode["value"] = "healthy" + routes.SWITCH.value = "healthy" def _force_void(key): @@ -135,7 +135,7 @@ def test_a_slow_retry_does_not_wait_again(client, monkeypatch): _charge(client, "saga-1") _charge(client, "saga-1") finally: - routes._mode["value"] = "healthy" + routes.SWITCH.value = "healthy" assert len(sleeps) == 1 @@ -150,7 +150,7 @@ def test_a_slow_charge_is_recorded_as_processing_before_the_slow_part(client, mo try: assert _charge(client, "saga-1").get_json()["result"]["status"] == "APPROVED" finally: - routes._mode["value"] = "healthy" + routes.SWITCH.value = "healthy" assert seen_while_slow == ["PROCESSING"] # a void during the slow part sees it in flight diff --git a/shared/shop_common/flask_support.py b/shared/shop_common/flask_support.py index d37c33e..230c4aa 100644 --- a/shared/shop_common/flask_support.py +++ b/shared/shop_common/flask_support.py @@ -46,6 +46,34 @@ def chaos_enabled() -> bool: return os.environ.get("CHAOS_ENABLED") == "1" +def switches_off() -> Response: + """The one answer every demo switch gives when CHAOS_ENABLED is not 1.""" + response = jsonify({"message": "Demo switches are off (CHAOS_ENABLED is not 1)"}) + response.status_code = 403 + return response + + +class DemoSwitch: + """A lab switch held in memory, one value per process: a demo setting, not data. + GET reads it; POST {"mode": ...} sets it, only when CHAOS_ENABLED=1.""" + + def __init__(self, *modes: str): + self.modes = modes + self.value = modes[0] + + def respond(self) -> Response: + if request.method == "POST": + if not chaos_enabled(): + return switches_off() + mode = (request.get_json(silent=True) or request.form.to_dict()).get("mode") + if mode not in self.modes: + response = jsonify({"message": f"mode must be one of {list(self.modes)}"}) + response.status_code = 400 + return response + self.value = mode + return jsonify({"mode": self.value, "modes": list(self.modes), "enabled": chaos_enabled()}) + + def install(app: Flask, service_name: str, db) -> None: init_tracing(service_name) this_instance = instance_id(service_name) diff --git a/tests/smoke/test_post_run.py b/tests/smoke/test_post_run.py index 2ea3fb1..97997e7 100644 --- a/tests/smoke/test_post_run.py +++ b/tests/smoke/test_post_run.py @@ -254,6 +254,68 @@ def test_demo_13_a_charge_payment_never_took_is_voided_and_the_stock_goes_back() assert _wait_for(lambda: {httpx.get(f"{api}/call").json()["outcome"] for _ in range(2)} == {"ok"}, seconds=20) +def _inventory_mode(mode: str) -> None: + assert httpx.post(f"{GATEWAY_URL}/api/shop/inventory/chaos", json={"mode": mode}).status_code == 200 + + +def _hold(saga_id: str) -> dict | None: + holds = httpx.get(f"{GATEWAY_URL}/api/shop/inventory/reservations").json()["results"] + return next((h for h in holds if h["reservation_id"] == saga_id), None) + + +def test_demo_14_a_release_that_beats_its_slow_hold_leaves_the_stock_untouched(): + _require_stack() + import time + + headers = _login_with_cart() + before = _stock(1) + _inventory_mode("slow_reserve") + try: + sent = time.monotonic() + r = httpx.post(f"{GATEWAY_URL}/api/shop/order/checkout", headers=headers, timeout=15) + assert r.status_code == 503, r.text # the saga gave up on the hold after 3 s + time.sleep(max(0.0, sent + 8 - time.monotonic())) # past the late hold, which lands at about 6 s + finally: + _inventory_mode("healthy") + hold = _hold(r.json()["saga_id"]) + assert hold is not None and hold["status"] == "RELEASED" and hold["items"] == [], hold + assert _stock(1) == before + + +def test_demo_14_a_release_that_times_out_and_arrives_twice_counts_once(): + _require_stack() + headers = _login_with_cart() + before = _stock(1) + _inventory_mode("slow_release") + try: + r = httpx.post(f"{GATEWAY_URL}/api/shop/order/checkout", json={"simulate_decline": True}, headers=headers, timeout=15) + assert r.status_code == 402, r.text + saga_id = r.json()["saga_id"] + assert _wait_for(lambda: (_hold(saga_id) or {}).get("status") == "RELEASED") # the event copy + finally: + _inventory_mode("healthy") + assert _wait_for(lambda: _hold(saga_id)["releases_received"] == 2), _hold(saga_id) # the direct call, at 6 s + assert _stock(1) == before + + +def test_demo_14_replayed_events_are_skipped_and_the_sale_counts_once(): + _require_stack() + headers = _login_with_cart() + before = _stock(1) + r = httpx.post(f"{GATEWAY_URL}/api/shop/order/checkout", headers=headers, timeout=15) + assert r.status_code == 200, r.text + assert _wait_for(lambda: _stock(1)["on_hand"] == before["on_hand"] - 3) + sold = _stock(1) + counts = lambda: httpx.get(f"{GATEWAY_URL}/api/shop/inventory/events").json() # noqa: E731 + start = counts() + replay = httpx.post(f"{GATEWAY_URL}/api/shop/order/sagas/{r.json()['saga_id']}/replay", headers=headers) + assert replay.json()["replayed"] == 3, replay.text + # OrderConfirmed again: skipped, not sold twice. StockReserved and PaymentCharged: ignored. + assert _wait_for(lambda: counts()["duplicates_skipped"] >= start["duplicates_skipped"] + 1) + assert _wait_for(lambda: counts()["not_handled"] >= start["not_handled"] + 2) + assert _stock(1) == sold + + def test_events_wait_in_the_outbox_while_the_bus_is_down_then_drain(): _require_stack() api = f"{GATEWAY_URL}/api/patterns/events"