diff --git a/CLAUDE.md b/CLAUDE.md index 3d52db5..242b334 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,7 @@ history, monolith dropped). The full agreed plan is in [`docs/PLAN.md`](docs/PLA - Name prefix: `ms12p` (parent uses `flaskms`). Teardown deletes the whole resource group, so a shared name would delete the other project. -## Current state (complete: 12 of 12 live; one Azure run done and torn down, #11) +## Current state (complete: 12 of 12 live, plus demo 13; one Azure run done and torn down, #11) - Folders: `services/{user,catalog,order,payment,inventory}/` (Flask, one DB each), `gateway/` (FastAPI), `registry/` (FastAPI), `shared/` (`shop_common`), `contracts/`, `frontend/`, `infra/`, `scripts/`, `tests/`. @@ -41,6 +41,14 @@ history, monolith dropped). The full agreed plan is in [`docs/PLAN.md`](docs/PLA charge_payment, confirm; compensation releases the hold (or emits StockReleaseRequested). Steps in `saga_step`; holds keyed by saga id in inventory's `reservation`. `{"simulate_decline": true}` on checkout makes payment answer DECLINED (200, not a breaker failure). +- Demo 13 (slow payment, from a LinkedIn comment): every charge carries `idempotency_key` = saga id + (payment's `payment.idempotency_key`, unique; old DBs gain it via `shop_common.db_setup.add_missing_columns`). + Payment records PROCESSING before the slow part; a reused key with another amount answers 409. + After a timeout the saga calls `POST /api/payment/charges//void` (exempt from chaos): an unseen key + becomes VOIDED (a late charge then gets VOIDED back), PROCESSING older than 30 s is voided, others + are returned as they are. `saga.settle_charge`: APPROVED confirms, DECLINED 402, VOIDED 503 (both + release), PROCESSING/no answer writes `pending_payment` (checkout and add-item answer 409) and returns 202; + `reconcile.py` settles pending rows every 5 s (deleting and committing the row is the claim). Page `/patterns/slow-payment`. - Events (pattern 7): `shop_common/events.py` (Redis Stream `shop.events`, consumer groups, at least once). Order writes `outbox_event` in the same transaction; `outbox.py` relays every 1 s; inventory's `consumer.py` commits the hold on OrderConfirmed and dedupes via `consumed_event`. `POST /debug/bus` diff --git a/README.md b/README.md index 31c9a1d..eef5009 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,8 @@ can click. > **Status: complete.** All 12 patterns are live, each with a working demo, locally and in one > real Azure run: provision, deploy through GitHub Actions with OIDC, smoke tests, screenshots, -> teardown. The foundation is copied from +> teardown. Demo 13, a slow payment in the middle of checkout, came from a reader's comment on +> the project's LinkedIn post. The foundation is copied from > [Azure-Ecommerce-Monolith-Microservices-Migration](https://github.com/MAOFILHO/Azure-Ecommerce-Monolith-Microservices-Migration). ## Project summary @@ -25,15 +26,16 @@ React frontend with one page per pattern. Every demo calls the real services: no | M2: resilience and independence | 4, 10, 11, 12 | [#3](https://github.com/MAOFILHO/Microservices-12-Patterns/pull/3) | | M3a: async and packaging | 7, 8, 9 | [#4](https://github.com/MAOFILHO/Microservices-12-Patterns/pull/4) | | M3b: delivery and the Azure run | 3 | [#12](https://github.com/MAOFILHO/Microservices-12-Patterns/pull/12), [#13](https://github.com/MAOFILHO/Microservices-12-Patterns/pull/13), [#14](https://github.com/MAOFILHO/Microservices-12-Patterns/pull/14) | +| Demo 13: a slow payment mid-checkout | 4 and 8, under a timeout | [#16](https://github.com/MAOFILHO/Microservices-12-Patterns/pull/16) | M3b was planned as 7 GitHub issues ([#5 to #11](https://github.com/MAOFILHO/Microservices-12-Patterns/issues?q=is%3Aissue)), all closed. **How it was checked.** -- **155 unit tests** across 8 suites, each in its own virtualenv, built test-first from M3a on. +- **181 unit tests** across 8 suites, each in its own virtualenv, built test-first from M3a on. - **Pact contract tests**: the order service's contracts with payment, catalog and inventory, verified against the real providers in CI. -- **23 local smoke tests** against the running stack, and **4 Azure smoke tests** against the real +- **25 local smoke tests** against the running stack, and **4 Azure smoke tests** against the real deployment. - **CI on every pull request** runs only what the change touches; every image is built for linux/amd64, started, and rejected if it runs as root. `main` requires the "CI ok" check. @@ -59,6 +61,7 @@ or the cheapest SKUs, with a $25/month budget alert, then torn down and verified | 10 | [Stateless Services](#10-stateless-services) | Stores session data outside the service | M2 | | 11 | [Contract Testing](#11-contract-testing) | Checks that APIs stay compatible across services | M2 | | 12 | [Service Discovery](#12-service-discovery) | Finds service instances automatically at runtime | M2 | +| 13 | [Saga under a slow payment](#13-saga-under-a-slow-payment) | Follows the money and the stock when payment times out mid-checkout | Demo 13 | ## Architecture @@ -191,6 +194,25 @@ The live registry. Stop an instance: its heartbeat stops, the registry expires i Service Discovery demo: registry table and an expired then re-registered order replica +### 13. Saga under a slow payment +A declined card is the easy case. The hard one is a timeout: the charge call runs long, the +breaker counts a failure, and a saga that releases the stock now may leave a customer charged +with no order, or charged twice by the retry. Two fixes, both suggested by a reader: + +- **An idempotency key on every charge** (the saga id). The retry sends the same key, so payment + returns the first charge instead of making a second one. +- **A status check before any undo.** The saga voids the key with payment, which answers how the + charge stands. APPROVED confirms the order; DECLINED releases the stock; a charge payment never + saw becomes VOIDED, so a late copy of it is refused. PROCESSING or no answer leaves the order + **pending**, stock held, until a background check (every 5 s) settles it. A charge stuck in + PROCESSING for 30 s is voided, so no order waits forever. + +Make payment slow, buy 2 mugs, and follow the money and the stock. Here the charge call timed out, +payment still said PROCESSING, so the order waited; 5 s later it settled as 1 charge, 1 order, +2 mugs sold. + +Demo 13: payment slowed mid-checkout; the order waits as pending, then settles as 1 charge, 1 order, 2 mugs sold + ## Run locally (no Docker) ```bash diff --git a/contracts/pacts/order-payment.json b/contracts/pacts/order-payment.json index 64303f3..22f3111 100644 --- a/contracts/pacts/order-payment.json +++ b/contracts/pacts/order-payment.json @@ -15,6 +15,7 @@ "body": { "content": { "amount_cents": 2598, + "idempotency_key": "saga-42", "order_id": 42 }, "contentType": "application/json", @@ -35,6 +36,14 @@ } ] }, + "$.idempotency_key": { + "combine": "AND", + "matchers": [ + { + "match": "type" + } + ] + }, "$.order_id": { "combine": "AND", "matchers": [ @@ -110,6 +119,7 @@ "body": { "content": { "amount_cents": 899, + "idempotency_key": "saga-43", "order_id": 43, "simulate": "decline" }, @@ -131,6 +141,14 @@ } ] }, + "$.idempotency_key": { + "combine": "AND", + "matchers": [ + { + "match": "type" + } + ] + }, "$.order_id": { "combine": "AND", "matchers": [ @@ -204,6 +222,171 @@ "status": 200 }, "type": "Synchronous/HTTP" + }, + { + "description": "a void for a charge payment never saw", + "pending": false, + "providerStates": [ + { + "name": "payment is healthy" + } + ], + "request": { + "body": { + "content": { + "amount_cents": 500, + "order_id": 44 + }, + "contentType": "application/json", + "encoded": false + }, + "headers": { + "Content-Type": [ + "application/json" + ] + }, + "matchingRules": { + "body": { + "$.amount_cents": { + "combine": "AND", + "matchers": [ + { + "match": "integer" + } + ] + }, + "$.order_id": { + "combine": "AND", + "matchers": [ + { + "match": "integer" + } + ] + } + } + }, + "method": "POST", + "path": "/api/payment/charges/saga-never-sent/void" + }, + "response": { + "body": { + "content": { + "result": { + "id": 3, + "status": "VOIDED" + } + }, + "contentType": "application/json", + "encoded": false + }, + "headers": { + "Content-Type": [ + "application/json" + ] + }, + "matchingRules": { + "body": { + "$.result.id": { + "combine": "AND", + "matchers": [ + { + "match": "integer" + } + ] + } + } + }, + "status": 200 + }, + "type": "Synchronous/HTTP" + }, + { + "description": "a void for a charge that went through", + "pending": false, + "providerStates": [ + { + "name": "a charge exists", + "params": { + "idempotency_key": "saga-42" + } + } + ], + "request": { + "body": { + "content": { + "amount_cents": 2598, + "order_id": 42 + }, + "contentType": "application/json", + "encoded": false + }, + "headers": { + "Content-Type": [ + "application/json" + ] + }, + "matchingRules": { + "body": { + "$.amount_cents": { + "combine": "AND", + "matchers": [ + { + "match": "integer" + } + ] + }, + "$.order_id": { + "combine": "AND", + "matchers": [ + { + "match": "integer" + } + ] + } + } + }, + "method": "POST", + "path": "/api/payment/charges/saga-42/void" + }, + "response": { + "body": { + "content": { + "result": { + "id": 1, + "status": "APPROVED" + } + }, + "contentType": "application/json", + "encoded": false + }, + "headers": { + "Content-Type": [ + "application/json" + ] + }, + "matchingRules": { + "body": { + "$.result.id": { + "combine": "AND", + "matchers": [ + { + "match": "integer" + } + ] + }, + "$.result.status": { + "combine": "AND", + "matchers": [ + { + "match": "type" + } + ] + } + } + }, + "status": 200 + }, + "type": "Synchronous/HTTP" } ], "metadata": { diff --git a/docs/screenshots/p13-slow-payment.jpg b/docs/screenshots/p13-slow-payment.jpg new file mode 100644 index 0000000..d7c478b Binary files /dev/null and b/docs/screenshots/p13-slow-payment.jpg differ diff --git a/frontend/src/api/gatewayClient.ts b/frontend/src/api/gatewayClient.ts index 29c3e70..24e159d 100644 --- a/frontend/src/api/gatewayClient.ts +++ b/frontend/src/api/gatewayClient.ts @@ -90,8 +90,9 @@ export const shopApi = { headers: authHeaders(token), }), + // 200 has the closed order. 202 means payment was too slow to tell (demo 13): the order waits as pending. checkout: (token: string) => - request<{ result: Order }>(`/api/shop/order/checkout`, { + request<{ result?: Order; pending?: boolean; message?: string }>(`/api/shop/order/checkout`, { method: "POST", headers: authHeaders(token), }), @@ -249,7 +250,7 @@ export interface SagaStep { export interface SagaRun { saga_id: string; order_id: number; - outcome: "completed" | "compensated" | "failed"; + outcome: "completed" | "compensated" | "failed" | "pending"; steps: SagaStep[]; } @@ -295,7 +296,7 @@ export interface RuntimeInfo { export const asyncApi = { checkout: (token: string, simulateDecline: boolean) => - getAny<{ message?: string; saga_id?: string; failed_step?: string }>("/api/shop/order/checkout", { + getAny<{ message?: string; saga_id?: string; failed_step?: string; pending?: boolean }>("/api/shop/order/checkout", { method: "POST", headers: { ...authHeaders(token), "Content-Type": "application/json" }, body: JSON.stringify({ simulate_decline: simulateDecline }), @@ -310,8 +311,20 @@ export const asyncApi = { body: JSON.stringify({ connected }), }), runtime: () => request<{ processes: RuntimeInfo[] }>("/api/patterns/runtime"), + payments: () => request<{ results: PaymentRow[] }>("/api/shop/payments"), }; +// --- Demo 13: a slow payment mid-checkout -------------------------------------------------------- + +export interface PaymentRow { + id: number; + order_id: number; + amount_cents: number; + status: "PROCESSING" | "APPROVED" | "DECLINED" | "VOIDED" | string; + idempotency_key: string | null; + created_at: string | null; +} + // --- M3b demo endpoint (pattern 3) --------------------------------------------------------------- export interface WorkflowRun { diff --git a/frontend/src/pages/Home.tsx b/frontend/src/pages/Home.tsx index 03f6e1f..96c0256 100644 --- a/frontend/src/pages/Home.tsx +++ b/frontend/src/pages/Home.tsx @@ -18,7 +18,7 @@ export default function Home() {

Patterns

-

The 12 patterns. All 12 are live, each with a working demo.

+

The 12 patterns. All 12 are live, each with a working demo, plus demo 13: a slow payment in the middle of checkout.

diff --git a/frontend/src/pages/PatternPage.tsx b/frontend/src/pages/PatternPage.tsx index a15224c..8a951be 100644 --- a/frontend/src/pages/PatternPage.tsx +++ b/frontend/src/pages/PatternPage.tsx @@ -11,6 +11,7 @@ import DatabasePerServiceDemo from "../patterns/demos/DatabasePerServiceDemo"; import ObservabilityDemo from "../patterns/demos/ObservabilityDemo"; import SagaDemo from "../patterns/demos/SagaDemo"; import ServiceDiscoveryDemo from "../patterns/demos/ServiceDiscoveryDemo"; +import SlowPaymentDemo from "../patterns/demos/SlowPaymentDemo"; import StatelessDemo from "../patterns/demos/StatelessDemo"; import { PATTERNS } from "../patterns/content"; @@ -27,6 +28,7 @@ const DEMOS: Record ReactNode> = { saga: () => , containerization: () => , "ci-cd-pipeline": () => , + "slow-payment": () => , }; export default function PatternPage() { diff --git a/frontend/src/pages/Patterns.tsx b/frontend/src/pages/Patterns.tsx index 4a0afbe..fb59885 100644 --- a/frontend/src/pages/Patterns.tsx +++ b/frontend/src/pages/Patterns.tsx @@ -2,14 +2,15 @@ import { Link } from "react-router-dom"; import { PATTERNS } from "../patterns/content"; export default function Patterns() { - const liveCount = PATTERNS.filter((p) => p.live).length; + const patterns = PATTERNS.filter((p) => p.n <= 12); // demo 13 is a bonus, not a 13th pattern + const liveCount = patterns.filter((p) => p.live).length; return (
-

The 12 patterns

+

The 12 patterns, plus demo 13

Each pattern has a page that explains it, says when not to use it, and runs a live demo against the real - services. {liveCount} of 12 are live; every one has a working demo. + services. {liveCount} of {patterns.length} are live. Demo 13 puts patterns 4 and 8 under a slow payment.

diff --git a/frontend/src/pages/Shop.tsx b/frontend/src/pages/Shop.tsx index 1936082..fef323c 100644 --- a/frontend/src/pages/Shop.tsx +++ b/frontend/src/pages/Shop.tsx @@ -121,8 +121,12 @@ export default function Shop() { setCartError(null); setCheckoutMessage(null); try { - const { result } = await shopApi.checkout(auth.token); - setCart(result); + const answer = await shopApi.checkout(auth.token); + if (answer.pending || !answer.result) { + setCheckoutMessage(answer.message ?? "Your order is pending."); + return; + } + setCart(answer.result); setCheckoutMessage("Order placed! Thank you."); } catch (err) { setCartError(String(err)); diff --git a/frontend/src/patterns/content.ts b/frontend/src/patterns/content.ts index 8169dba..9b31e12 100644 --- a/frontend/src/patterns/content.ts +++ b/frontend/src/patterns/content.ts @@ -279,6 +279,30 @@ export const PATTERNS: PatternInfo[] = [ ], source: { label: "microservices.io — Client-side service discovery", url: "https://microservices.io/patterns/client-side-discovery.html" }, }, + { + n: 13, + slug: "slow-payment", + name: "Saga under a slow payment", + what: "Follows the money and the stock when payment times out mid-checkout.", + milestone: "M3", + live: true, + problem: + "A declined card is the easy case: the answer is clear and the stock goes back. A timeout is the hard one. The charge call runs long, the breaker counts a failure, the saga releases the stock, and the customer was charged anyway. A retry can even charge them twice.", + inThisProject: [ + "Every charge carries an idempotency key: the saga id, one per checkout attempt. The retry sends the same key, so payment returns the first charge instead of making a second one.", + "Payment records a charge as PROCESSING when it arrives, before the slow part, so a check made meanwhile sees it in flight. A reused key with a different amount is refused.", + "Before any undo, the saga voids the key with payment, which answers how the charge stands. APPROVED confirms the order. DECLINED releases the stock. A charge payment never saw is recorded as VOIDED, so a late copy of it can never be charged.", + "PROCESSING or no answer: the order waits as pending, with the stock held. It cannot be checked out again or changed, so a second attempt cannot charge.", + "Every 5 s, each order replica asks payment about pending orders and settles them by the same rules. Deleting the pending row is the claim, so 2 replicas cannot settle one twice. A charge stuck in PROCESSING for 30 s lost its process, so payment voids it and no order waits forever.", + "Raised by a reader of the project's LinkedIn post, who named both fixes.", + ], + whenNotToUse: [ + "Idempotency keys are not optional for anything that moves money. What varies is the key: one per attempt (as here) lets a declined card be retried.", + "A pending state is honest but costs the shopper a wait. Use it only where guessing is worse, as with money.", + "\"Never seen\" is final only because payment records a charge on arrival. A provider that does not must also offer a way to refuse a key before the saga releases the stock.", + ], + source: { label: "Stripe — Idempotent requests", url: "https://docs.stripe.com/api/idempotent_requests" }, + }, ]; export const SERVICE_NAMES = ["gateway", "user", "catalog", "order", "payment", "inventory"] as const; diff --git a/frontend/src/patterns/demos/SagaDemo.tsx b/frontend/src/patterns/demos/SagaDemo.tsx index f0b8f71..4089308 100644 --- a/frontend/src/patterns/demos/SagaDemo.tsx +++ b/frontend/src/patterns/demos/SagaDemo.tsx @@ -2,15 +2,16 @@ import { useEffect, useState } from "react"; import { Link } from "react-router-dom"; import { SagaRun, StockRow, asyncApi, savedToken } from "../../api/gatewayClient"; -const STEP_LABEL: Record = { +export const STEP_LABEL: Record = { price: "1. Price the cart (catalog)", reserve_stock: "2. Hold the stock (inventory)", charge_payment: "3. Charge the card (payment)", + check_payment: "3b. Ask payment what happened (demo 13)", confirm: "4. Confirm the order", release_stock: "Undo step 2: release the stock", }; -const PILL: Record = { done: "done", failed: "failed", compensated: "running" }; +export const PILL: Record = { done: "done", failed: "failed", compensated: "running", timed_out: "failed" }; const DEMO_PRODUCT = 1; export default function SagaDemo() { @@ -115,7 +116,7 @@ export default function SagaDemo() {

Step 2. What the saga did

Saga {latest.saga_id.slice(0, 8)} for order #{latest.order_id}:{" "} - + {latest.outcome}

diff --git a/frontend/src/patterns/demos/SlowPaymentDemo.tsx b/frontend/src/patterns/demos/SlowPaymentDemo.tsx new file mode 100644 index 0000000..9a8fefa --- /dev/null +++ b/frontend/src/patterns/demos/SlowPaymentDemo.tsx @@ -0,0 +1,171 @@ +import { useEffect, useState } from "react"; +import { Link } from "react-router-dom"; +import { PaymentRow, SagaRun, StockRow, asyncApi, resilienceApi, savedToken } from "../../api/gatewayClient"; +import { PILL, STEP_LABEL } from "./SagaDemo"; + +const DEMO_PRODUCT = 1; +const POLL_MS = 2000; + +/** Demo 13: slow payment down mid-checkout, then follow where the money and the stock end up. */ +export default function SlowPaymentDemo() { + const token = savedToken(); + const [mode, setMode] = useState("unknown"); + const [modeError, setModeError] = useState(null); + const [busy, setBusy] = useState(false); + const [result, setResult] = useState<{ kind: "done" | "running" | "failed"; label: string; message: string } | null>(null); + const [saga, setSaga] = useState(null); + const [charges, setCharges] = useState([]); + const [stock, setStock] = useState(null); + + async function refresh() { + if (!token) return; + const [history, payments, view, breaker] = await Promise.all([ + asyncApi.sagas(token).catch(() => null), + asyncApi.payments().catch(() => null), + asyncApi.events().catch(() => null), + resilienceApi.breakerState().catch(() => null), + ]); + const latest = history?.results[0] ?? null; + setSaga(latest); + setCharges(latest && payments ? payments.results.filter((p) => p.idempotency_key === latest.saga_id) : []); + setStock(view?.stock?.find((s) => s.product_id === DEMO_PRODUCT) ?? null); + if (breaker) setMode(breaker.payment.mode); + } + + useEffect(() => { + refresh(); + const timer = setInterval(refresh, POLL_MS); // the reconciler settles pending orders every 5 s + return () => clearInterval(timer); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, []); + + if (!token) { + return ( +

+ This demo checks out a real cart, so it needs a signed-in user. Register or log in on the Shop + page, then come back. +

+ ); + } + + async function switchPayment(next: string) { + setModeError(null); + const { status, body } = await resilienceApi.setPaymentMode(next); + if (status === 200 && body.mode) setMode(body.mode); + else setModeError(body.message ?? `HTTP ${status}`); + } + + async function buy() { + if (!token) return; + setBusy(true); + setResult(null); + try { + await asyncApi.addToCart(token, DEMO_PRODUCT, 2); + const { status, body } = await asyncApi.checkout(token, false); + if (status === 200) setResult({ kind: "done", label: "Completed", message: "Order placed. Payment approved." }); + else if (status === 202) setResult({ kind: "running", label: "Pending", message: body.message ?? "Waiting for payment." }); + else setResult({ kind: "failed", label: "Not placed", message: body.message ?? `HTTP ${status}` }); + } catch (err) { + setResult({ kind: "failed", label: "Error", message: String(err) }); + } finally { + await refresh(); + setBusy(false); + } + } + + // A pending answer goes stale once the reconciler settles the order: show what it settled to. + const shown = + result?.kind === "running" && saga && saga.outcome !== "pending" + ? saga.outcome === "completed" + ? { kind: "done" as const, label: "Confirmed", message: "Payment answered APPROVED, so the order was confirmed." } + : { kind: "failed" as const, label: "Released", message: "Payment declined or voided the charge, so the stock went back." } + : result; + const approved = charges.filter((c) => c.status === "APPROVED").length; + const verdict = !saga + ? null + : saga.outcome === "pending" + ? "Waiting: payment has not said yet. The mugs stay held, and the order cannot be paid for again." + : saga.outcome === "completed" + ? `Settled: ${approved} charge, 1 order, 2 mugs sold.` + : `Settled: ${approved} charges, no order, the mugs went back to stock.`; + + return ( +
+
+
+

Step 1. Slow payment down

+

+ Payment now answers {mode === "slow" ? "after 3 s" : "at once"}. The order service waits 1 s, retries once, + then gives up on the call. +

+
+ + +
+ {modeError &&

{modeError}

} + +

Step 2. Buy 2 Contoso Mugs

+ + {shown && ( +

+ {shown.label} {shown.message} +

+ )} +
+ +
+

Step 3. Where the money and the stock are

+ + + + + + + + + + + + + + + + + + + +
Charges for this checkout + {charges.length === 0 + ? "none" + : charges.map((c) => `#${c.id} ${c.status} ${(c.amount_cents / 100).toFixed(2)}`).join(", ")} +
Mugs on hand{stock?.on_hand ?? "?"}
Mugs held for checkouts{stock?.reserved ?? "?"}
Order{saga ? `#${saga.order_id} ${saga.outcome}` : "–"}
+ {verdict &&

{verdict}

} +

+ Refreshes every 2 s. A pending order settles within about 5 s of payment finishing. +

+
+
+ + {saga && ( +
+

What the saga did

+
    + {saga.steps.map((s, i) => ( +
  1. + {s.status} + {STEP_LABEL[s.step] ?? s.step} + {s.detail && · {s.detail}} +
  2. + ))} +
+
+ )} +
+ ); +} diff --git a/services/order/application/__init__.py b/services/order/application/__init__.py index 047f088..59a296b 100644 --- a/services/order/application/__init__.py +++ b/services/order/application/__init__.py @@ -48,4 +48,9 @@ def create_app(config_name: str | None = None) -> Flask: from .outbox import start as start_outbox_relay start_outbox_relay(app, instance_id("order")) + + # Demo 13: settle checkouts left pending by a slow payment. + from .reconcile import start as start_reconciler + + start_reconciler(app) return app diff --git a/services/order/application/models.py b/services/order/application/models.py index 7eafd49..1355ac9 100644 --- a/services/order/application/models.py +++ b/services/order/application/models.py @@ -93,3 +93,17 @@ class CheckoutLock(db.Model): order_id = db.Column(db.Integer, primary_key=True, autoincrement=False) saga_id = db.Column(db.String(36), nullable=False) started_at = db.Column(db.DateTime, nullable=False, default=_utcnow) + + +class PendingPayment(db.Model): + """Demo 13: a checkout whose charge timed out and whose outcome payment could not yet tell. + The stock stays held and the order cannot be checked out or changed until reconcile.py settles + it. A row is deleted by whoever settles it, so two replicas cannot settle it twice.""" + + __tablename__ = "pending_payment" + + saga_id = db.Column(db.String(36), primary_key=True) # also the charge's idempotency key + order_id = db.Column(db.Integer, nullable=False, unique=True) + user_id = db.Column(db.Integer, nullable=False) + amount_cents = db.Column(db.Integer, nullable=False) + since = db.Column(db.DateTime, nullable=False, default=_utcnow) diff --git a/services/order/application/order_api/api/clients.py b/services/order/application/order_api/api/clients.py index 9be3984..3145eb2 100644 --- a/services/order/application/order_api/api/clients.py +++ b/services/order/application/order_api/api/clients.py @@ -108,8 +108,9 @@ def attempt() -> requests.Response: raise PaymentUnavailable("failed") -def charge(order_id: int, amount_cents: int, simulate: str = "approve") -> dict: - body = {"order_id": order_id, "amount_cents": amount_cents} +def charge(order_id: int, amount_cents: int, simulate: str = "approve", *, idempotency_key: str) -> dict: + """The retry above resends this same body, key included, so payment charges at most once (demo 13).""" + body = {"order_id": order_id, "amount_cents": amount_cents, "idempotency_key": idempotency_key} if simulate != "approve": body["simulate"] = simulate response = call_payment("POST", "/api/payment/charge", body) @@ -117,6 +118,22 @@ def charge(order_id: int, amount_cents: int, simulate: str = "approve") -> dict: return response.json()["result"] +VOID_TIMEOUT_SECONDS = 2.0 + + +def void_charge(idempotency_key: str, order_id: int, amount_cents: int) -> dict: + """Demo 13: cancels this key unless the charge went through, and returns the charge as it now + stands (status APPROVED, DECLINED, VOIDED or PROCESSING). Once VOIDED, a late copy of the + charge is refused. Raises requests.RequestException when payment does not answer. + + Not behind the breaker: it is one light call, and its answer is what lets the saga undo safely. + """ + response = requests.post(f"{base_url('payment')}/api/payment/charges/{idempotency_key}/void", + json={"order_id": order_id, "amount_cents": amount_cents}, timeout=VOID_TIMEOUT_SECONDS) + response.raise_for_status() + return response.json()["result"] + + def breaker_state() -> dict: retry_in = None opened_at = BREAKER._state_storage.opened_at # pybreaker keeps no public accessor for this diff --git a/services/order/application/order_api/routes.py b/services/order/application/order_api/routes.py index f4e0002..0cf12ad 100644 --- a/services/order/application/order_api/routes.py +++ b/services/order/application/order_api/routes.py @@ -37,6 +37,8 @@ def order_add_item(): # The cart lives in the order database, not in this process (pattern 10): any replica sees it. known_order = Order.query.filter_by(user_id=u_id, is_open=True).first() + if known_order is not None and saga.is_pending(known_order.id): # demo 13: the charge may have happened + return make_response(jsonify({"message": saga.PENDING_MESSAGE, "pending": True}), 409) if known_order is None: known_order = Order(user_id=u_id, is_open=True) known_order.items.append(OrderItem(data.product_id, data.qty)) diff --git a/services/order/application/reconcile.py b/services/order/application/reconcile.py new file mode 100644 index 0000000..0bc6950 --- /dev/null +++ b/services/order/application/reconcile.py @@ -0,0 +1,65 @@ +"""Settles checkouts left pending by a slow payment (demo 13). + +Every RECONCILE_SECONDS, each order replica voids every pending charge's key with payment, which +answers how the charge stands: APPROVED confirms the order, DECLINED or VOIDED releases the stock, +and PROCESSING or no answer leaves it pending for the next pass. Payment voids a charge stuck in +PROCESSING once its process is surely gone, so no order waits forever while payment answers. +The rules are the saga's own (saga.settle_charge). +""" +import logging +import threading +import time + +import requests + +from . import db, saga +from .models import Order, PendingPayment +from .order_api.api import clients + +RECONCILE_SECONDS = 5 +log = logging.getLogger(__name__) + + +def resolve_pending() -> int: + """One pass. Returns how many pending checkouts it settled.""" + settled = 0 + for row in PendingPayment.query.order_by(PendingPayment.since).all(): + saga_id, order_id, user_id, amount = row.saga_id, row.order_id, row.user_id, row.amount_cents + try: + charge = clients.void_charge(saga_id, order_id, amount) + except requests.RequestException: + continue # still no answer: ask again next pass + if charge["status"] == "PROCESSING": + continue + # Claim it by deleting the row, and commit at once: settling calls inventory, and the other + # replica must not wait on this database lock meanwhile. If the other replica already took + # it, nothing is deleted and this replica leaves it alone. + claimed = PendingPayment.query.filter_by(saga_id=saga_id).delete() == 1 + db.session.commit() + if not claimed: + continue + checkout = saga.Saga(db.session.get(Order, order_id), user_id, saga_id=saga_id) + saga.settle_charge(checkout, amount, charge) + settled += 1 + return settled + + +def start(app) -> None: + from shop_common.flask_support import is_serving_process + + if not is_serving_process(app): + return # unit tests and the reloader's watcher land here + + def loop() -> None: + while True: + time.sleep(RECONCILE_SECONDS) + with app.app_context(): + try: + resolve_pending() + except Exception: + log.exception("reconcile pass failed; the next pass retries") + db.session.rollback() + finally: + db.session.remove() + + threading.Thread(target=loop, name="payment-reconciler", daemon=True).start() diff --git a/services/order/application/saga.py b/services/order/application/saga.py index f098815..9de1084 100644 --- a/services/order/application/saga.py +++ b/services/order/application/saga.py @@ -7,6 +7,12 @@ outbox. Inventory turns the hold into a sale when it reads that event (7). Any failure: event CheckoutFailed. One checkout per order at a time (checkout_lock). +Demo 13, a charge that times out: the customer may have been charged. The charge carries the saga +id as its idempotency key, so the retry cannot charge twice. Before it undoes anything, the saga +voids that key with payment, which answers how the charge stands (settle_charge). APPROVED +confirms the order; DECLINED or VOIDED (never arrived, and now never will) releases the stock; +PROCESSING or no answer leaves the order pending, with the stock held, until reconcile.py settles it. + If a later step fails, the earlier steps are undone in reverse order. Only step 2 changes another service's data, so the only compensation is releasing the stock. If that direct call fails too, a StockReleaseRequested event goes to the outbox, so inventory releases it once it can. @@ -21,13 +27,13 @@ from sqlalchemy.exc import IntegrityError from . import db -from .models import CheckoutLock, Order, OutboxEvent, SagaStep +from .models import CheckoutLock, Order, OutboxEvent, PendingPayment, SagaStep from .order_api.api import clients class Saga: - def __init__(self, order: Order, user_id: int): - self.id = str(uuid.uuid4()) + def __init__(self, order: Order, user_id: int, saga_id: str | None = None): + self.id = saga_id or str(uuid.uuid4()) self.order = order self.user_id = user_id @@ -85,11 +91,17 @@ def _release_lock(order_id: int) -> None: db.session.commit() +PENDING_MESSAGE = ("Your last checkout is still waiting for payment. It is confirmed, or the stock released, " + "as soon as payment answers.") + + def run_checkout(order: Order, user_id: int, simulate: str = "approve") -> tuple[int, dict]: saga = Saga(order, user_id) if not _acquire_lock(order, saga.id): return 409, {"message": "A checkout for this order is already in progress."} try: + if is_pending(order.id): + return 409, {"message": PENDING_MESSAGE, "pending": True} return _run_steps(saga, simulate) finally: _release_lock(order.id) @@ -128,36 +140,77 @@ def _run_steps(saga: Saga, simulate: str) -> tuple[int, dict]: saga.log("reserve_stock", "done", f"held {units} unit(s)", event="StockReserved", payload={"reservation_id": saga.id, "items": items}) - # 3. charge payment (behind the M2 circuit breaker) + # 3. charge payment (behind the M2 circuit breaker), keyed by the saga id (demo 13) try: - payment = clients.charge(order.id, amount, simulate) + payment = clients.charge(order.id, amount, simulate, idempotency_key=saga.id) except clients.PaymentUnavailable as exc: - breaker = clients.breaker_state() if exc.reason == "open": # the breaker refused the call: payment was never contacted + breaker = clients.breaker_state() wait = f" Try again in {breaker['retry_in_seconds']:.0f} s." if breaker["retry_in_seconds"] else "" - message = f"Payment is unavailable right now, so nothing was charged; the stock was released. {kept}{wait}" - else: # timeout or 5xx: the charge may or may not have happened - message = f"Payment could not be confirmed, so the order was not placed and the stock was released. {kept}" - return saga.fail("charge_payment", f"payment unavailable ({exc.reason})", 503, message, - compensate=True, breaker=breaker) + return saga.fail("charge_payment", "payment unavailable (open)", 503, + f"Payment is unavailable right now, so nothing was charged; the stock was released. {kept}{wait}", + compensate=True, breaker=breaker) + return _charge_unknown(saga, amount, f"payment unavailable ({exc.reason})") except requests.RequestException as exc: - return saga.fail("charge_payment", f"payment error: {exc.__class__.__name__}", 502, - f"Payment could not be confirmed, so the order was not placed and the stock was released. {kept}", - compensate=True) + return _charge_unknown(saga, amount, f"payment error: {exc.__class__.__name__}") + if payment["status"] == "PROCESSING": # a retry answered while the first try is still running + return _charge_unknown(saga, amount, "payment still processing") if payment["status"] != "APPROVED": return saga.fail("charge_payment", "declined", 402, f"Your payment was declined. Nothing was charged; the stock was released. {kept}", compensate=True) - saga.log("charge_payment", "done", f"payment {payment['id']} APPROVED, {amount} cents", event="PaymentCharged", - payload={"payment_id": payment["id"], "amount_cents": amount}) + return _confirm(saga, amount, payment, "charge_payment") + - # 4. confirm: the order change, its log line and its event commit together, or not at all +def _confirm(saga: Saga, amount: int, payment: dict, step: str) -> tuple[int, dict]: + """The charge went through: record it, then 4. confirm. The order change, its log line and its + event commit together, or not at all.""" + order = saga.order + items = [{"product_id": i.product_id, "quantity": i.quantity} for i in order.items] + saga.log(step, "done", f"payment {payment['id']} APPROVED, {amount} cents", event="PaymentCharged", + payload={"payment_id": payment["id"], "amount_cents": amount}) order.is_open = False saga.log("confirm", "done", "order closed; OrderConfirmed in the outbox", event="OrderConfirmed", payload={"reservation_id": saga.id, "amount_cents": amount, "items": items}) return 200, {"result": order.to_json(), "payment": payment, "saga_id": saga.id} +def _charge_unknown(saga: Saga, amount: int, detail: str) -> tuple[int, dict]: + """Demo 13: the charge call failed after it may have reached payment, so the customer may have + been charged. Void the key first: payment then says how the charge stands.""" + saga.log("charge_payment", "timed_out", f"{detail}; asking payment what happened") + try: + charge = clients.void_charge(saga.id, saga.order.id, amount) + except requests.RequestException as exc: + charge = {"status": f"no answer ({exc.__class__.__name__})"} + return settle_charge(saga, amount, charge) + + +def settle_charge(saga: Saga, amount: int, charge: dict) -> tuple[int, dict]: + """How payment says the saga's charge stands decides the outcome (also used by reconcile.py).""" + status = charge["status"] + kept = "Your cart is kept." + if status == "APPROVED": + return _confirm(saga, amount, charge, "check_payment") + if status == "DECLINED": + return saga.fail("check_payment", "payment says DECLINED", 402, + f"Your payment was declined. Nothing was charged; the stock was released. {kept}", + compensate=True) + if status == "VOIDED": + return saga.fail("check_payment", "payment never took the charge; its key is now void", 503, + f"Payment did not go through. Nothing was charged; the stock was released. {kept}", + compensate=True) + db.session.add(PendingPayment(saga_id=saga.id, order_id=saga.order.id, user_id=saga.user_id, amount_cents=amount)) + saga.log("check_payment", "pending", f"payment says {status}; the order waits, the stock stays held") + return 202, {"message": "Payment is taking longer than usual. Your order is pending: it is confirmed, or the " + "stock released, as soon as payment answers. You cannot be charged twice.", + "saga_id": saga.id, "pending": True} + + +def is_pending(order_id: int) -> bool: + return PendingPayment.query.filter_by(order_id=order_id).first() is not None + + def _compensate(saga: Saga) -> None: try: clients.release_stock(saga.id) @@ -176,7 +229,8 @@ def recent_sagas(user_id: int, limit: int = 5) -> list[dict]: for saga_id in saga_ids: steps = SagaStep.query.filter_by(saga_id=saga_id).order_by(SagaStep.id).all() statuses = {s.status for s in steps} - outcome = "compensated" if "compensated" in statuses else "failed" if "failed" in statuses else "completed" + outcome = ("compensated" if "compensated" in statuses else "failed" if "failed" in statuses + else "pending" if steps[-1].status == "pending" else "completed") result.append({"saga_id": saga_id, "order_id": steps[0].order_id, "outcome": outcome, "steps": [s.to_json() for s in steps]}) return result diff --git a/services/order/tests/test_checkout_and_breaker.py b/services/order/tests/test_checkout_and_breaker.py index fd485ae..a52a1a8 100644 --- a/services/order/tests/test_checkout_and_breaker.py +++ b/services/order/tests/test_checkout_and_breaker.py @@ -46,18 +46,19 @@ def test_breaker_opens_after_three_failures_then_fails_fast(client): assert 0 < body["breaker"]["retry_in_seconds"] <= clients.RESET_SECONDS -def test_checkout_keeps_the_cart_when_payment_is_down(client): +def test_checkout_waits_as_pending_when_payment_is_down(client): + """Payment at :9 refuses the charge and the void: nobody can say whether it was charged.""" _cart(client) r = client.post("/api/order/checkout", headers=AUTH) - assert r.status_code == 503 - assert "Your cart is kept" in r.get_json()["message"] + assert r.status_code == 202 + assert r.get_json()["pending"] is True assert client.get("/api/order", headers=AUTH).get_json()["result"]["is_open"] is True def test_checkout_charges_the_catalog_price_and_closes_the_order(client, monkeypatch): charged = {} - def fake_charge(order_id, amount_cents, simulate="approve"): + def fake_charge(order_id, amount_cents, simulate="approve", *, idempotency_key): charged.update(order_id=order_id, amount_cents=amount_cents) return {"id": 1, "order_id": order_id, "amount_cents": amount_cents, "status": "APPROVED"} @@ -92,8 +93,8 @@ def test_a_payment_timeout_never_claims_nothing_was_charged(client): """Payment at :9 refuses the call, which looks like a timeout: the charge state is unknown.""" _cart(client) message = client.post("/api/order/checkout", headers=AUTH).get_json()["message"] - assert "could not be confirmed" in message - assert "nothing was charged" not in message + assert "pending" in message + assert "nothing was charged" not in message.lower() def test_an_open_breaker_does_say_nothing_was_charged(client): diff --git a/services/order/tests/test_contracts_consumer.py b/services/order/tests/test_contracts_consumer.py index 1159a16..bfea820 100644 --- a/services/order/tests/test_contracts_consumer.py +++ b/services/order/tests/test_contracts_consumer.py @@ -28,7 +28,8 @@ def test_order_to_payment_contract(monkeypatch): pact.upon_receiving("a charge for an order") .given("payment is healthy") .with_request("POST", "/api/payment/charge") - .with_body({"order_id": match.int(42), "amount_cents": match.int(2598)}, "application/json") + .with_body({"order_id": match.int(42), "amount_cents": match.int(2598), "idempotency_key": match.str("saga-42")}, + "application/json") .will_respond_with(200) .with_body({"result": { "id": match.int(1), @@ -41,10 +42,27 @@ def test_order_to_payment_contract(monkeypatch): pact.upon_receiving("a charge the bank declines") .given("payment is healthy") .with_request("POST", "/api/payment/charge") - .with_body({"order_id": match.int(43), "amount_cents": match.int(899), "simulate": "decline"}, "application/json") + .with_body({"order_id": match.int(43), "amount_cents": match.int(899), "simulate": "decline", + "idempotency_key": match.str("saga-43")}, "application/json") .will_respond_with(200) .with_body({"result": {"id": match.int(2), "status": "DECLINED"}}, "application/json") ) + ( + pact.upon_receiving("a void for a charge that went through") + .given("a charge exists", idempotency_key="saga-42") + .with_request("POST", "/api/payment/charges/saga-42/void") + .with_body({"order_id": match.int(42), "amount_cents": match.int(2598)}, "application/json") + .will_respond_with(200) + .with_body({"result": {"id": match.int(1), "status": match.str("APPROVED")}}, "application/json") + ) + ( + pact.upon_receiving("a void for a charge payment never saw") + .given("payment is healthy") + .with_request("POST", "/api/payment/charges/saga-never-sent/void") + .with_body({"order_id": match.int(44), "amount_cents": match.int(500)}, "application/json") + .will_respond_with(200) + .with_body({"result": {"id": match.int(3), "status": "VOIDED"}}, "application/json") + ) ( pact.upon_receiving("a health check through the circuit breaker") .given("payment is healthy") @@ -54,10 +72,12 @@ def test_order_to_payment_contract(monkeypatch): ) with pact.serve() as server: monkeypatch.setitem(clients.STATIC_URLS, "payment", str(server.url).rstrip("/")) - payment = clients.charge(42, 2598) + payment = clients.charge(42, 2598, idempotency_key="saga-42") assert payment["status"] == "APPROVED" assert payment["amount_cents"] == 2598 - assert clients.charge(43, 899, "decline")["status"] == "DECLINED" + assert clients.charge(43, 899, "decline", idempotency_key="saga-43")["status"] == "DECLINED" + assert clients.void_charge("saga-42", 42, 2598)["status"] == "APPROVED" + assert clients.void_charge("saga-never-sent", 44, 500)["status"] == "VOIDED" assert clients.call_payment("GET", "/api/payment/ping").json() == {"status": "ok"} pact.write_file(PACT_DIR, overwrite=True) diff --git a/services/order/tests/test_saga_and_outbox.py b/services/order/tests/test_saga_and_outbox.py index 2a546d7..8e5b740 100644 --- a/services/order/tests/test_saga_and_outbox.py +++ b/services/order/tests/test_saga_and_outbox.py @@ -17,7 +17,7 @@ def calls(monkeypatch): monkeypatch.setattr(clients, "product_prices", lambda: {1: 1299}) monkeypatch.setattr(clients, "reserve_stock", lambda rid, oid, items: log.append(("reserve", rid)) or {}) monkeypatch.setattr(clients, "release_stock", lambda rid: log.append(("release", rid)) or {}) - monkeypatch.setattr(clients, "charge", lambda oid, amount, simulate="approve": log.append(("charge", simulate)) or { + monkeypatch.setattr(clients, "charge", lambda oid, amount, simulate="approve", **key: log.append(("charge", simulate)) or { "id": 9, "status": "DECLINED" if simulate == "decline" else "APPROVED"}) clients.BREAKER.close() return log diff --git a/services/order/tests/test_slow_payment.py b/services/order/tests/test_slow_payment.py new file mode 100644 index 0000000..8caeb99 --- /dev/null +++ b/services/order/tests/test_slow_payment.py @@ -0,0 +1,158 @@ +"""Demo 13: payment is slow in the middle of checkout. The charge call times out, so the saga +cannot know whether the customer was charged. It voids the charge's key before it undoes anything, +and payment answers how the charge now stands: + + APPROVED confirm the order (the customer paid, so they get it) + DECLINED release the stock, 402 like any decline + VOIDED release the stock (payment never took it, and now never will) + PROCESSING / no answer the order waits as pending; reconcile.resolve_pending() settles it later +""" +import pytest +import requests + +from application import reconcile +from application.models import OutboxEvent, SagaStep +from application.order_api.api import clients + +AUTH = {"Authorization": "Bearer test"} + + +@pytest.fixture +def payment(monkeypatch): + """Payment that times out on charge. `status` is what a void answers.""" + state = {"status": "PROCESSING", "keys": [], "voids": [], "released": []} + + def timed_out(order_id, amount, simulate="approve", *, idempotency_key): + state["keys"].append(idempotency_key) + raise clients.PaymentUnavailable("failed") + + def void(key, order_id, amount_cents): + state["voids"].append((key, amount_cents)) + if state["status"] == "UNREACHABLE": + raise requests.ConnectTimeout("payment did not answer") + return {"id": 9, "status": state["status"]} + + monkeypatch.setattr(clients, "current_user_id", lambda header: 1 if header else None) + monkeypatch.setattr(clients, "product_prices", lambda: {1: 1299}) + monkeypatch.setattr(clients, "reserve_stock", lambda rid, oid, items: {}) + monkeypatch.setattr(clients, "release_stock", lambda rid: state["released"].append(rid) or {}) + monkeypatch.setattr(clients, "charge", timed_out) + monkeypatch.setattr(clients, "void_charge", void) + clients.BREAKER.close() + return state + + +def _checkout(client): + client.post("/api/order/add-item", data={"product_id": 1, "qty": 2}, headers=AUTH) + return client.post("/api/order/checkout", json={}, headers=AUTH) + + +def _events(): + return [e.type for e in OutboxEvent.query.order_by(OutboxEvent.created_at)] + + +def _cart_is_open(client): + return client.get("/api/order", headers=AUTH).get_json().get("result", {}).get("is_open") is True + + +def test_the_charge_and_the_void_carry_the_saga_id_as_the_key(client, payment): + saga_id = _checkout(client).get_json()["saga_id"] + assert payment["keys"] == [saga_id] + assert payment["voids"] == [(saga_id, 2 * 1299)] + + +def test_a_timed_out_charge_that_went_through_confirms_the_order(client, payment): + payment["status"] = "APPROVED" + r = _checkout(client) + assert r.status_code == 200 + assert payment["released"] == [] # the customer paid: never give their stock away + assert _events() == ["StockReserved", "PaymentCharged", "OrderConfirmed"] + assert not _cart_is_open(client) + + +@pytest.mark.parametrize("answer, http_status", [("VOIDED", 503), ("DECLINED", 402)]) +def test_a_timed_out_charge_that_did_not_happen_releases_the_stock(client, payment, answer, http_status): + payment["status"] = answer + r = _checkout(client) + assert r.status_code == http_status + assert payment["released"] == [r.get_json()["saga_id"]] + assert _events() == ["StockReserved", "CheckoutFailed"] + assert "nothing was charged" in r.get_json()["message"].lower() + assert _cart_is_open(client) + + +@pytest.mark.parametrize("answer", ["PROCESSING", "UNREACHABLE"]) +def test_an_unknown_charge_leaves_the_order_pending_and_the_stock_held(client, payment, answer): + payment["status"] = answer + r = _checkout(client) + assert r.status_code == 202 + assert r.get_json()["pending"] is True + assert payment["released"] == [] + assert _events() == ["StockReserved"] # no CheckoutFailed: nothing is decided yet + assert SagaStep.query.order_by(SagaStep.id.desc()).first().status == "pending" + + +def test_a_pending_order_cannot_be_checked_out_or_changed(client, payment): + _checkout(client) + second = client.post("/api/order/checkout", json={}, headers=AUTH) + assert second.status_code == 409 + changed = client.post("/api/order/add-item", data={"product_id": 1, "qty": 1}, headers=AUTH) + assert changed.status_code == 409 + assert second.get_json()["message"] == changed.get_json()["message"] + assert "waiting for payment" in changed.get_json()["message"] + assert len(payment["keys"]) == 1 # no second charge attempt + + +def test_reconcile_confirms_a_pending_order_once_payment_approves(client, payment): + _checkout(client) + payment["status"] = "APPROVED" + assert reconcile.resolve_pending() == 1 + assert _events() == ["StockReserved", "PaymentCharged", "OrderConfirmed"] + assert payment["released"] == [] + assert not _cart_is_open(client) + assert reconcile.resolve_pending() == 0 # settled once, never twice + + +def test_reconcile_releases_the_stock_when_payment_voided_the_charge(client, payment): + saga_id = _checkout(client).get_json()["saga_id"] + payment["status"] = "VOIDED" + assert reconcile.resolve_pending() == 1 + assert payment["released"] == [saga_id] + assert _events() == ["StockReserved", "CheckoutFailed"] + assert _cart_is_open(client) + payment["status"] = "APPROVED" # the cart can be checked out again + assert client.post("/api/order/checkout", json={}, headers=AUTH).status_code == 200 + + +def test_reconcile_leaves_an_order_pending_while_payment_still_has_no_answer(client, payment): + _checkout(client) + payment["status"] = "UNREACHABLE" + assert reconcile.resolve_pending() == 0 + assert _events() == ["StockReserved"] + + +def test_the_saga_history_shows_a_pending_checkout(client, payment): + _checkout(client) + assert client.get("/api/order/sagas", headers=AUTH).get_json()["results"][0]["outcome"] == "pending" + + +def test_reconcile_commits_its_claim_before_it_calls_inventory(client, payment, monkeypatch): + """The release is an HTTP call: the other replica must not wait on this one's database lock.""" + from application.models import PendingPayment + + _checkout(client) + payment["status"] = "VOIDED" + seen = [] + monkeypatch.setattr(clients, "release_stock", lambda rid: seen.append(_claim_is_committed()) or {}) + reconcile.resolve_pending() + assert seen == [True] + assert PendingPayment.query.count() == 0 + + +def _claim_is_committed() -> bool: + from sqlalchemy import text + + from application import db + + with db.engine.connect() as other: # a second connection sees only committed rows + return other.execute(text("SELECT COUNT(*) FROM pending_payment")).scalar() == 0 diff --git a/services/payment/application/__init__.py b/services/payment/application/__init__.py index f1c2782..a3de7a2 100644 --- a/services/payment/application/__init__.py +++ b/services/payment/application/__init__.py @@ -30,5 +30,9 @@ def create_app(config_name: str | None = None) -> Flask: app.register_blueprint(payment_api) db.create_all() + # Demo 13 added payment.idempotency_key: a database made before it keeps its rows. + from shop_common.db_setup import add_missing_columns + + add_missing_columns(db) install_observability(app, "payment", db) return app diff --git a/services/payment/application/models.py b/services/payment/application/models.py index 4768d7e..4b5ad10 100644 --- a/services/payment/application/models.py +++ b/services/payment/application/models.py @@ -13,7 +13,10 @@ class Payment(db.Model): id = db.Column(db.Integer, primary_key=True) order_id = db.Column(db.Integer, nullable=False, index=True) # the order service's id; no cross-DB FK amount_cents = db.Column(db.Integer, nullable=False) - status = db.Column(db.String(20), nullable=False) # APPROVED | DECLINED + status = db.Column(db.String(20), nullable=False) # PROCESSING | APPROVED | DECLINED | VOIDED + # Demo 13: the caller's key for this charge. A retry with the same key gets this row back + # instead of a second charge. Unique, so two racing requests cannot both insert it. + idempotency_key = db.Column(db.String(64), nullable=True, unique=True) created_at = db.Column(db.DateTime, default=_utcnow) def to_json(self): @@ -22,5 +25,6 @@ def to_json(self): "order_id": self.order_id, "amount_cents": self.amount_cents, "status": self.status, + "idempotency_key": self.idempotency_key, "created_at": self.created_at.isoformat() if self.created_at else None, } diff --git a/services/payment/application/routes.py b/services/payment/application/routes.py index 8245e57..abfd895 100644 --- a/services/payment/application/routes.py +++ b/services/payment/application/routes.py @@ -1,9 +1,11 @@ import time +from datetime import datetime, timedelta, timezone from typing import Literal from flask import Blueprint, jsonify, make_response, request from pydantic import BaseModel, Field, ValidationError from shop_common.flask_support import chaos_enabled +from sqlalchemy.exc import IntegrityError from . import db from .models import Payment @@ -12,26 +14,37 @@ # Pattern 4 demo switch: "healthy", "failing" (every call returns 500) or "slow" (every call takes # SLOW_SECONDS, longer than the order service waits). Held in memory: it is a lab switch, not data. +# The switch makes charging sick, not settling: void and the list always answer, so the demo 13 +# page can show where the money is while payment is slow. MODES = ("healthy", "failing", "slow") SLOW_SECONDS = 3 _mode = {"value": "healthy"} +_ALWAYS_ANSWERS = {"payment_api.chaos", "payment_api.void", "payment_api.list_payments"} +# Demo 13: a charge still PROCESSING after this long lost its process (a crash or restart), so a +# void may cancel it. Far longer than any charge takes, slow mode included. +PROCESSING_LIMIT_SECONDS = 30 -class ChargeRequest(BaseModel): +class VoidRequest(BaseModel): order_id: int = Field(gt=0) amount_cents: int = Field(gt=0) + + +class ChargeRequest(VoidRequest): # Pattern 8 demo: "decline" acts like a card the bank refuses. A decline is a normal business # answer (200, status DECLINED), not a failure, so it never trips the order service's breaker. simulate: Literal["approve", "decline"] = "approve" + # Demo 13: one key per checkout attempt, sent again on every retry of that attempt. + idempotency_key: str = Field(min_length=1, max_length=64) @payment_api.before_request def _apply_chaos(): - if request.endpoint == "payment_api.chaos": + if request.endpoint in _ALWAYS_ANSWERS: return None if _mode["value"] == "failing": return make_response(jsonify({"message": "Payment is failing (demo switch)"}), 500) - if _mode["value"] == "slow": + if _mode["value"] == "slow" and request.endpoint != "payment_api.charge": # charge() is slow itself time.sleep(SLOW_SECONDS) return None @@ -46,17 +59,64 @@ def ping(): return jsonify({"status": "ok"}) +def _invalid(exc: ValidationError): + return make_response(jsonify({"message": "Invalid request", "errors": exc.errors(include_url=False)}), 400) + + +def _record(key: str, data: VoidRequest, status: str) -> tuple[Payment, bool]: + """The row for this key: the existing one, or a new one with this status. True if new.""" + payment = Payment.query.filter_by(idempotency_key=key).first() + if payment is not None: + return payment, False + db.session.add(Payment(order_id=data.order_id, amount_cents=data.amount_cents, status=status, idempotency_key=key)) + try: + db.session.commit() + except IntegrityError: # another request with this key got there first: use its row + db.session.rollback() + return Payment.query.filter_by(idempotency_key=key).one(), False + return Payment.query.filter_by(idempotency_key=key).one(), True + + +def _finish(payment: Payment, status: str) -> Payment: + """PROCESSING -> status, only if nothing (a void) changed the row meanwhile.""" + Payment.query.filter_by(id=payment.id, status="PROCESSING").update({"status": status}) + db.session.commit() + db.session.refresh(payment) + return payment + + @payment_api.post("/api/payment/charge") def charge(): raw = request.get_json(silent=True) or request.form.to_dict() try: data = ChargeRequest.model_validate(raw) except ValidationError as exc: - return make_response(jsonify({"message": "Invalid request", "errors": exc.errors(include_url=False)}), 400) - status = "DECLINED" if data.simulate == "decline" else "APPROVED" - payment = Payment(order_id=data.order_id, amount_cents=data.amount_cents, status=status) - db.session.add(payment) - db.session.commit() + return _invalid(exc) + # Demo 13: the charge is recorded as PROCESSING before the slow part, so a void sent while it + # runs sees it in flight. A retry with the same key gets this row back, never a new charge. + payment, created = _record(data.idempotency_key, data, "PROCESSING") + if not created: + if (payment.order_id, payment.amount_cents) != (data.order_id, data.amount_cents): + return make_response(jsonify({"message": "This key was already used for a different charge"}), 409) + return jsonify({"result": payment.to_json()}) + if _mode["value"] == "slow": + time.sleep(SLOW_SECONDS) + return jsonify({"result": _finish(payment, "DECLINED" if data.simulate == "decline" else "APPROVED").to_json()}) + + +@payment_api.post("/api/payment/charges//void") +def void(key: str): + """Demo 13: the order service's check before it undoes anything. A charge that never arrived + is recorded as VOIDED, so it can never be charged later; one whose process died is voided. + Answers the charge as it now stands: APPROVED, DECLINED, VOIDED or still PROCESSING.""" + try: + data = VoidRequest.model_validate(request.get_json(silent=True) or {}) + except ValidationError as exc: + return _invalid(exc) + payment, _ = _record(key, data, "VOIDED") + created = payment.created_at.replace(tzinfo=payment.created_at.tzinfo or timezone.utc) + if payment.status == "PROCESSING" and datetime.now(timezone.utc) - created > timedelta(seconds=PROCESSING_LIMIT_SECONDS): + payment = _finish(payment, "VOIDED") return jsonify({"result": payment.to_json()}) diff --git a/services/payment/tests/test_contract_provider.py b/services/payment/tests/test_contract_provider.py index bbf9692..7a95fd4 100644 --- a/services/payment/tests/test_contract_provider.py +++ b/services/payment/tests/test_contract_provider.py @@ -11,9 +11,23 @@ def test_payment_honours_the_order_services_contract(app): - from application import routes + from application import db, routes + from application.models import Payment + + # Pact calls the state handlers on another thread: end this thread's transaction first (SQLite). + db.session.commit() + db.session.remove() def healthy() -> None: routes._mode["value"] = "healthy" - verify_provider(app, provider="payment", consumer="order", states={"payment is healthy": healthy}, pact_dir=PACT_DIR) + def charge_exists() -> None: + healthy() + with app.app_context(): + if Payment.query.filter_by(idempotency_key="saga-42").first() is None: + db.session.add(Payment(order_id=42, amount_cents=2598, status="APPROVED", idempotency_key="saga-42")) + db.session.commit() + db.session.remove() + + verify_provider(app, provider="payment", consumer="order", + states={"payment is healthy": healthy, "a charge exists": charge_exists}, pact_dir=PACT_DIR) diff --git a/services/payment/tests/test_payment.py b/services/payment/tests/test_payment.py index e38febd..9ae2cf7 100644 --- a/services/payment/tests/test_payment.py +++ b/services/payment/tests/test_payment.py @@ -1,5 +1,5 @@ def test_charge_records_an_approved_payment(client): - r = client.post("/api/payment/charge", data={"order_id": 7, "amount_cents": 3798}) + r = client.post("/api/payment/charge", data={"order_id": 7, "amount_cents": 3798, "idempotency_key": "k7"}) assert r.status_code == 200 assert r.get_json()["result"]["status"] == "APPROVED" assert client.get("/api/payments").get_json()["results"][0]["order_id"] == 7 @@ -17,7 +17,7 @@ def test_health_db_reports_only_own_tables(client): def test_charge_accepts_json(client): - r = client.post("/api/payment/charge", json={"order_id": 3, "amount_cents": 500}) + r = client.post("/api/payment/charge", json={"order_id": 3, "amount_cents": 500, "idempotency_key": "k3"}) assert r.get_json()["result"]["amount_cents"] == 500 @@ -35,6 +35,167 @@ def test_chaos_switch_is_off_unless_enabled(client, monkeypatch): def test_a_simulated_decline_is_a_normal_answer(client): - r = client.post("/api/payment/charge", json={"order_id": 4, "amount_cents": 900, "simulate": "decline"}) + r = client.post("/api/payment/charge", json={"order_id": 4, "amount_cents": 900, "simulate": "decline", "idempotency_key": "k4"}) assert r.status_code == 200 assert r.get_json()["result"]["status"] == "DECLINED" + + +# --- Demo 13: a slow payment mid-checkout. Idempotency key and void. ------------------------------ + +def _charge(client, key, **body): + return client.post("/api/payment/charge", json={"order_id": 5, "amount_cents": 700, "idempotency_key": key, **body}) + + +def test_a_retry_with_the_same_key_charges_once(client): + first = _charge(client, "saga-1").get_json()["result"] + again = _charge(client, "saga-1").get_json()["result"] + assert again["id"] == first["id"] + assert len(client.get("/api/payments").get_json()["results"]) == 1 + + +def test_a_new_key_is_a_new_charge(client): + _charge(client, "saga-1") + _charge(client, "saga-2") + assert len(client.get("/api/payments").get_json()["results"]) == 2 + + +def test_a_charge_needs_an_idempotency_key(client): + assert client.post("/api/payment/charge", json={"order_id": 5, "amount_cents": 700}).status_code == 400 + + +def _void(client, key, order_id=5, amount_cents=700): + return client.post(f"/api/payment/charges/{key}/void", json={"order_id": order_id, "amount_cents": amount_cents}) + + +def _slow(client, monkeypatch, on_sleep): + from application import routes + + monkeypatch.setattr(routes.time, "sleep", on_sleep) + monkeypatch.setenv("CHAOS_ENABLED", "1") + client.post("/api/payment/chaos", json={"mode": "slow"}) + + +def test_a_reused_key_with_a_different_charge_is_refused(client): + _charge(client, "saga-1") + r = client.post("/api/payment/charge", json={"order_id": 5, "amount_cents": 999, "idempotency_key": "saga-1"}) + assert r.status_code == 409 + assert len(client.get("/api/payments").get_json()["results"]) == 1 + + +def test_void_reports_a_charge_that_went_through_and_leaves_it(client): + _charge(client, "saga-1") + assert _void(client, "saga-1").get_json()["result"]["status"] == "APPROVED" + + +def test_a_voided_key_can_never_be_charged_later(client): + assert _void(client, "saga-1").get_json()["result"]["status"] == "VOIDED" + assert _charge(client, "saga-1").get_json()["result"]["status"] == "VOIDED" # the late charge + assert [p["status"] for p in client.get("/api/payments").get_json()["results"]] == ["VOIDED"] + + +def test_void_leaves_a_charge_in_flight_alone_until_it_is_too_old(client): + from datetime import timedelta + + from application import db, routes + from application.models import Payment + + db.session.add(Payment(order_id=5, amount_cents=700, status="PROCESSING", idempotency_key="saga-1")) + db.session.commit() + assert _void(client, "saga-1").get_json()["result"]["status"] == "PROCESSING" + row = Payment.query.one() + row.created_at = row.created_at - timedelta(seconds=routes.PROCESSING_LIMIT_SECONDS + 1) + db.session.commit() + assert _void(client, "saga-1").get_json()["result"]["status"] == "VOIDED" # its process died + + +def test_a_charge_voided_while_it_ran_does_not_approve_when_it_finishes(client, monkeypatch): + from application import routes + + _slow(client, monkeypatch, lambda s: _force_void("saga-1")) + try: + assert _charge(client, "saga-1").get_json()["result"]["status"] == "VOIDED" + finally: + routes._mode["value"] = "healthy" + + +def _force_void(key): + from application import db + from application.models import Payment + + Payment.query.filter_by(idempotency_key=key).update({"status": "VOIDED"}) + db.session.commit() + + +def test_a_slow_retry_does_not_wait_again(client, monkeypatch): + from application import routes + + sleeps = [] + _slow(client, monkeypatch, sleeps.append) + try: + _charge(client, "saga-1") + _charge(client, "saga-1") + finally: + routes._mode["value"] = "healthy" + assert len(sleeps) == 1 + + +def test_a_slow_charge_is_recorded_as_processing_before_the_slow_part(client, monkeypatch): + from application import routes + from application.models import Payment + + seen_while_slow = [] + monkeypatch.setattr(routes.time, "sleep", lambda s: seen_while_slow.append(Payment.query.one().status)) + monkeypatch.setenv("CHAOS_ENABLED", "1") + client.post("/api/payment/chaos", json={"mode": "slow"}) + try: + assert _charge(client, "saga-1").get_json()["result"]["status"] == "APPROVED" + finally: + routes._mode["value"] = "healthy" + assert seen_while_slow == ["PROCESSING"] # a void during the slow part sees it in flight + + +def test_void_answers_while_charges_fail(client, monkeypatch): + _charge(client, "saga-1") + monkeypatch.setenv("CHAOS_ENABLED", "1") + client.post("/api/payment/chaos", json={"mode": "failing"}) + try: + assert _charge(client, "saga-2").status_code == 500 + assert _void(client, "saga-1").status_code == 200 + finally: + client.post("/api/payment/chaos", json={"mode": "healthy"}) + + +def test_a_database_from_before_demo_13_gains_the_key_column_at_startup(app): + from sqlalchemy import inspect, text + + from application import create_app, db + + db.drop_all() + db.session.execute(text("CREATE TABLE payment (id INTEGER PRIMARY KEY, order_id INTEGER NOT NULL, " + "amount_cents INTEGER NOT NULL, status VARCHAR(20) NOT NULL, created_at DATETIME)")) + db.session.commit() + create_app("development") + assert "idempotency_key" in {c["name"] for c in inspect(db.engine).get_columns("payment")} + + +def test_two_processes_upgrading_the_same_database_at_once_both_start(app, monkeypatch): + """The other process added the column after this one looked: the upgrade must not crash.""" + import sqlalchemy + + from application import db + from shop_common.db_setup import add_missing_columns + + real = sqlalchemy.inspect + + class Stale: + def __init__(self, engine): + self._real = real(engine) + + def has_table(self, name): + return self._real.has_table(name) + + def get_columns(self, name): + return [c for c in self._real.get_columns(name) if c["name"] != "idempotency_key"] + + monkeypatch.setattr(sqlalchemy, "inspect", Stale) + add_missing_columns(db) # sees the column as missing, but it is already there diff --git a/shared/shop_common/db_setup.py b/shared/shop_common/db_setup.py index 9c8cdcb..296c7b4 100644 --- a/shared/shop_common/db_setup.py +++ b/shared/shop_common/db_setup.py @@ -16,3 +16,40 @@ def create_tables(db) -> None: raise db.session.rollback() db.create_all() + + +def add_missing_columns(db) -> None: + """Adds model columns that an existing table lacks, so a database made by an older version + keeps its rows (there are no Alembic revisions). Only for new nullable columns: a unique one + gets a unique index, which allows many NULLs on SQLite and MySQL. Safe when two processes + start at once: a column or index the other one just added is skipped. + """ + from sqlalchemy import inspect + + inspector = inspect(db.engine) + quote = db.engine.dialect.identifier_preparer.quote + for table in db.metadata.sorted_tables: + if not inspector.has_table(table.name): + continue + present = {c["name"] for c in inspector.get_columns(table.name)} + for column in table.columns: + if column.name in present or not column.nullable: + continue + kind = column.type.compile(dialect=db.engine.dialect) + _run_once(db, f"ALTER TABLE {quote(table.name)} ADD COLUMN {quote(column.name)} {kind}") + if column.unique: + index = quote(f"uq_{table.name}_{column.name}") + _run_once(db, f"CREATE UNIQUE INDEX {index} ON {quote(table.name)} ({quote(column.name)})") + + +def _run_once(db, statement: str) -> None: + """Runs one schema change, skipping it if another process already made it.""" + from sqlalchemy import text + + try: + db.session.execute(text(statement)) + db.session.commit() + except OperationalError as exc: + db.session.rollback() + if "duplicate" not in str(exc).lower() and "already exists" not in str(exc).lower(): + raise diff --git a/tests/smoke/test_post_run.py b/tests/smoke/test_post_run.py index 65800a7..2ea3fb1 100644 --- a/tests/smoke/test_post_run.py +++ b/tests/smoke/test_post_run.py @@ -208,6 +208,52 @@ def test_declined_payment_is_compensated_and_stock_is_unchanged(): assert httpx.get(f"{GATEWAY_URL}/api/shop/order", headers=headers).json()["result"]["is_open"] is True +def test_demo_13_a_slow_payment_ends_as_one_charge_and_one_sale(): + """Payment slows down mid-checkout: the order waits as pending, then settles as exactly 1 charge.""" + _require_stack() + api = f"{GATEWAY_URL}/api/patterns/breaker" + headers = _login_with_cart() + before = _stock(1) + assert httpx.post(f"{api}/payment-mode", json={"mode": "slow"}).status_code == 200 + try: + r = httpx.post(f"{GATEWAY_URL}/api/shop/order/checkout", headers=headers, timeout=15) + finally: + httpx.post(f"{api}/payment-mode", json={"mode": "healthy"}) + assert r.status_code == 202, r.text # the charge timed out and payment still says PROCESSING + saga_id = r.json()["saga_id"] + assert _stock(1)["reserved"] == before["reserved"] + 3 # held, not given away + + def charges() -> list[dict]: + return [p for p in httpx.get(f"{GATEWAY_URL}/api/shop/payments").json()["results"] if p["idempotency_key"] == saga_id] + + saga = lambda: httpx.get(f"{GATEWAY_URL}/api/shop/order/sagas", headers=headers).json()["results"][0] # noqa: E731 + assert _wait_for(lambda: saga()["outcome"] == "completed", seconds=15), saga() + assert [c["status"] for c in charges()] == ["APPROVED"] # the retry did not charge twice + assert _wait_for(lambda: _stock(1)["on_hand"] == before["on_hand"] - 3), _stock(1) + assert _stock(1)["reserved"] == before["reserved"] + + +def test_demo_13_a_charge_payment_never_took_is_voided_and_the_stock_goes_back(): + """Payment fails every charge: the saga voids the key, so a late copy can never charge.""" + _require_stack() + api = f"{GATEWAY_URL}/api/patterns/breaker" + headers = _login_with_cart() + before = _stock(1) + assert httpx.post(f"{api}/payment-mode", json={"mode": "failing"}).status_code == 200 + try: + r = httpx.post(f"{GATEWAY_URL}/api/shop/order/checkout", headers=headers, timeout=15) + finally: + httpx.post(f"{api}/payment-mode", json={"mode": "healthy"}) + assert r.status_code == 503, r.text + saga_id = r.json()["saga_id"] + payments = httpx.get(f"{GATEWAY_URL}/api/shop/payments").json()["results"] + assert [p["status"] for p in payments if p["idempotency_key"] == saga_id] == ["VOIDED"] + assert _stock(1) == before + assert httpx.get(f"{GATEWAY_URL}/api/shop/order", headers=headers).json()["result"]["is_open"] is True + # The failed charges count toward a breaker: one good call per replica resets the count. + assert _wait_for(lambda: {httpx.get(f"{api}/call").json()["outcome"] for _ in range(2)} == {"ok"}, seconds=20) + + def test_events_wait_in_the_outbox_while_the_bus_is_down_then_drain(): _require_stack() api = f"{GATEWAY_URL}/api/patterns/events"