Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,11 @@ INVENTORY_SERVICE_BASE_URL=http://127.0.0.1:5105
# make run sets this; with it unset, callers use the fixed URLs above (as in unit tests and Azure).
# REGISTRY_URL=http://127.0.0.1:8500

# --- Demo switches (patterns 4, 10, 12) ---
# --- Event bus (patterns 7 and 8) ---
# make run starts redis-server on 6390 and sets this. Unset: events wait in the order outbox.
# REDIS_URL=redis://127.0.0.1:6390/0

# --- Demo switches (patterns 4, 7, 10, 12) ---
# make run sets CHAOS_ENABLED=1 so the pages can stop a replica and break payment. Keep it off anywhere public.
# CHAOS_ENABLED=1

Expand Down
45 changes: 45 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ jobs:
requirements: services/inventory/requirements.txt
tests: services/inventory/tests
shared: flask
contracts: true
steps:
- uses: actions/checkout@v4

Expand Down Expand Up @@ -101,6 +102,50 @@ jobs:
- name: Type-check and build
run: npm run build

containers:
# Pattern 9. Builds every image exactly as Azure will run it (linux/amd64), starts it, and checks
# it answers /health and does not run as root. Nothing is pushed anywhere.
name: Container image (${{ matrix.image }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- { image: user, dir: services/user, port: 5101, shared: true }
- { image: catalog, dir: services/catalog, port: 5102, shared: true }
- { image: order, dir: services/order, port: 5103, shared: true }
- { image: payment, dir: services/payment, port: 5104, shared: true }
- { image: inventory, dir: services/inventory, port: 5105, shared: true }
- { image: gateway, dir: gateway, port: 8100, shared: true }
- { image: registry, dir: registry, port: 8500, shared: false }
steps:
- uses: actions/checkout@v4

- name: Stage shared package (shop_common)
if: matrix.shared
run: cp -r shared "${{ matrix.dir }}/_shared"

- name: Build for linux/amd64
run: >
docker build --platform linux/amd64 --build-arg GIT_SHA=${{ github.sha }}
-t ms12p-${{ matrix.image }}:${{ github.sha }} ${{ matrix.dir }}

- name: Run it, check health and that it is not root
run: |
docker run -d --name svc -p ${{ matrix.port }}:${{ matrix.port }} -e RUN_MODE=local \
ms12p-${{ matrix.image }}:${{ github.sha }}
for i in $(seq 1 30); do curl -sf http://127.0.0.1:${{ matrix.port }}/health && break; sleep 1; done
curl -sf http://127.0.0.1:${{ matrix.port }}/health
uid=$(docker exec svc id -u)
echo "runs as uid $uid"
test "$uid" != "0"
size=$(docker image inspect ms12p-${{ matrix.image }}:${{ github.sha }} --format '{{.Size}}')
echo "| ${{ matrix.image }} | $((size / 1024 / 1024)) MB | uid $uid | linux/amd64 |" >> "$GITHUB_STEP_SUMMARY"

- name: Container logs (on failure)
if: failure()
run: docker logs svc || true

bicep:
name: Bicep (build)
runs-on: ubuntu-latest
Expand Down
36 changes: 27 additions & 9 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ history, monolith dropped). The full agreed plan is in [`docs/PLAN.md`](docs/PLA
- **Local + Azure.** `make provision` / `make teardown` / `make verify`, $25/mo budget alert.
Nothing is deployed to Azure before milestone M3, and only with the owner's go-ahead.
- **Phased delivery**: M0 bootstrap, then M1 (patterns 1, 2, 5, 6), M2 (10, 12, 4, 11),
M3 (7, 8, 9, 3 + Azure). One PR per milestone.
M3a (8, 7, 9, local) and M3b (3 + Azure, owner go-ahead). One PR per milestone.
- Screenshots live in `docs/screenshots/` and are referenced by relative path. Never use GitHub
`user-attachments` links (they went dead in the parent project).

Expand All @@ -27,15 +27,27 @@ history, monolith dropped). The full agreed plan is in [`docs/PLAN.md`](docs/PLA
- Name prefix: `ms12p` (parent uses `flaskms`). Teardown deletes the whole resource group, so a
shared name would delete the other project.

## Current state (M2: patterns 1, 2, 4, 5, 6, 10, 11, 12 live)
## Current state (M3a: every pattern except 3 is live)
- Folders: `services/{user,catalog,order,payment,inventory}/` (Flask, one DB each), `gateway/`
(FastAPI), `registry/` (FastAPI), `shared/` (`shop_common`), `contracts/`, `frontend/`, `infra/`,
`scripts/`, `tests/`.
- **One service table**: `scripts/service_table.py` lists every local process, folder and port, plus
`REPLICA_PORTS` (order also on 5113) and `CONTRACT_SUITES`. `setup.py`, `run_local.py`,
`provision.py` and the smoke tests all read it. Add a service there.
- Ports: user 5101, catalog 5102, order 5103 + 5113, payment 5104, inventory 5105, gateway 8100,
registry 8500, frontend 5273.
registry 8500, Redis 6390, frontend 5273.
- Saga (pattern 8): `services/order/application/saga.py`, orchestrated: price, reserve_stock,
charge_payment, confirm; compensation releases the hold (or emits StockReleaseRequested). Steps in
`saga_step`; holds keyed by saga id in inventory's `reservation`. `{"simulate_decline": true}` on
checkout makes payment answer DECLINED (200, not a breaker failure).
- Events (pattern 7): `shop_common/events.py` (Redis Stream `shop.events`, consumer groups, at least
once). Order writes `outbox_event` in the same transaction; `outbox.py` relays every 1 s; inventory's
`consumer.py` commits the hold on OrderConfirmed and dedupes via `consumed_event`. `POST /debug/bus`
(chaos) disconnects a replica's relay. Without `REDIS_URL` events just wait in the outbox.
- Background threads (heartbeat, relay, consumer) start only where `is_serving_process(app)` is true:
Flask's debug reloader runs a watcher plus a worker, and both build the app.
- Containers (pattern 9): multi-stage Dockerfiles, uid 10001, `.dockerignore` per service, `GIT_SHA`
build arg, `/debug/runtime`. CI job "Container image" builds, runs and checks all 7 images.
- **`shop_common`** (`shared/`): every service calls `shop_common.flask_support.install(app, name, db)`
inside its app context. It adds `/health`, `/health/db`, `/debug/spans`, `/metrics`,
`POST /debug/crash` (only with `CHAOS_ENABLED=1`), the `X-Instance-Id` header, tracing, JSON logs,
Expand All @@ -52,17 +64,23 @@ history, monolith dropped). The full agreed plan is in [`docs/PLAN.md`](docs/PLA
- Circuit breaker (pattern 4): `services/order/application/order_api/api/clients.py`, pybreaker
(3 failures, 15 s) + tenacity (1 retry, 1 s timeout), only 5xx/timeouts count. Checkout prices from
catalog, charges payment, keeps the cart on 503. Payment has a `healthy|failing|slow` switch.
- Contracts (pattern 11): order's tests write `contracts/pacts/order-{payment,catalog}.json`; payment
and catalog verify them. CI fails if the regenerated files differ. Pact (pact-python 4.2) is
- Contracts (pattern 11): order's tests write `contracts/pacts/order-{payment,catalog,inventory}.json`;
those 3 services verify them. CI fails if the regenerated files differ. Pact (pact-python 4.2) is
test-only, installed from `contracts/requirements.txt`, never in images.
- `make run` sets `REGISTRY_URL`, `CHAOS_ENABLED=1`, `INSTANCE_ID`, and restarts a stopped service
after 20 s. Registry TTL 10 s, heartbeat 2 s. Azure uses Container Apps DNS (no registry yet).
- `make run` sets `REGISTRY_URL`, `REDIS_URL`, `CHAOS_ENABLED=1`, `INSTANCE_ID`, and restarts a stopped
service after 20 s. Inventory seeds stock for all 24 catalog products; `POST /api/inventory/<id>/restock`
(chaos only) keeps smoke runs from running out. Registry TTL 10 s, heartbeat 2 s. Azure uses Container Apps DNS (no registry yet).
- Gotchas solved: FastAPI tracing middleware must be added AFTER the request-ID middleware; Flask-
SQLAlchemy's engine is instrumented per engine; Pact's `Verifier` needs `host="127.0.0.1"`; DB writes
inside a Pact state handler deadlock on SQLite (seed before verifying instead).
SQLAlchemy's engine is instrumented per engine; Pact's `Verifier` needs `host="127.0.0.1"`; Pact state
handlers run on another thread, so end the test's DB transaction first or SQLite deadlocks; two
replicas racing `create_all()` (use `shop_common.db_setup.create_tables`).
- Smoke tests create users and products in the local SQLite DBs; delete `services/*/instance/`
for a clean catalog (this also deletes the local signing key, which logs everyone out).

## How to build (owner's preference)
Follow the Matt Pocock `implement` skill (read it from the mattpocock-skills plugin folder): TDD at
agreed seams, red test first; then `mattpocock-skills:code-review`; then commit.

## Stack
Python 3.12 (Flask 3.1 services, FastAPI gateway), Node 18+ (React 18, TypeScript 5, Vite 5),
SQLite locally, Azure MySQL Flexible Server (one server, one DB per service) in Azure.
Expand Down
52 changes: 37 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,25 +4,25 @@ A small Contoso e-commerce system that shows **12 microservices patterns working
Each pattern gets its own page with an explanation, when *not* to use it, and a live demo you
can click.

> **Status: M2, resilience and independence.** 8 of 12 patterns are live: M1's Database per
> Service, Bounded Context, API Gateway and Observability, plus M2's Circuit Breaker, Stateless
> Services, Contract Testing and Service Discovery. The foundation is copied from
> **Status: M3a, async and packaging.** 11 of 12 patterns are live. M3a adds the Saga,
> Event-Driven Architecture and Containerization to M1 (patterns 1, 2, 5, 6) and M2 (4, 10, 11, 12).
> The foundation is copied from
> [Azure-Ecommerce-Monolith-Microservices-Migration](https://github.com/MAOFILHO/Azure-Ecommerce-Monolith-Microservices-Migration).
> The last 4 arrive in milestone M3. See [docs/PLAN.md](docs/PLAN.md).
> CI-CD and the Azure run arrive in M3b. See [docs/PLAN.md](docs/PLAN.md).

## The 12 patterns

| # | Pattern | What it does | Status |
|---|---|---|---|
| 1 | Database per Service | Gives each service its own database | **Live (M1)** |
| 2 | Bounded Context | Splits one large domain into clear business domains | **Live (M1)** |
| 3 | CI-CD Pipeline | Builds, tests and deploys every service automatically | M3 |
| 3 | CI-CD Pipeline | Builds, tests and deploys every service automatically | M3b |
| 4 | Circuit Breaker | Stops calls to a failing service to prevent more failures | **Live (M2)** |
| 5 | API Gateway | Routes client requests to the right service | **Live (M1)** |
| 6 | Observability | Collects logs, metrics and traces from every service | **Live (M1)** |
| 7 | Event-Driven Architecture | Services communicate by publishing and consuming events | M3 |
| 8 | Saga Pattern | Coordinates distributed transactions and rolls back failures safely | M3 |
| 9 | Containerization | Packages each service with all its dependencies | M3 |
| 7 | Event-Driven Architecture | Services communicate by publishing and consuming events | **Live (M3a)** |
| 8 | Saga Pattern | Coordinates distributed transactions and rolls back failures safely | **Live (M3a)** |
| 9 | Containerization | Packages each service with all its dependencies | **Live (M3a)** |
| 10 | Stateless Services | Stores session data outside the service | **Live (M2)** |
| 11 | Contract Testing | Checks that APIs stay compatible across services | **Live (M2)** |
| 12 | Service Discovery | Finds service instances automatically at runtime | **Live (M2)** |
Expand All @@ -32,11 +32,12 @@ can click.
```
Browser → API gateway ──┬──► user → user_db (signs login tokens)
(checks tokens, ├──► catalog → catalog_db
request IDs, ├──► order ×2 → order_db ──► catalog (prices)
tracing, metrics, │ └─► payment, behind a circuit breaker
request IDs, ├──► order ×2 → order_db checkout saga: price → hold stock →
tracing, metrics, │ charge (circuit breaker) → confirm
aggregation, ├──► payment → payment_db
failover) └──► inventory → inventory_db

order_db outbox ──relay──► Redis Stream "shop.events" ──► inventory (sells the held stock)
Every service heartbeats into the registry (:8500); callers look each other up there.
```

Expand All @@ -47,11 +48,13 @@ Every service heartbeats into the registry (:8500); callers look each other up t
- **`shop_common`** (`shared/`): OpenTelemetry tracing, JSON logs with trace IDs, Prometheus
metrics, the `/health/db`, `/debug/spans`, `/metrics` endpoints, token checks and the discovery client.
- **Service registry** (`registry/`): about 90 lines of FastAPI. Local only; Azure uses Container Apps DNS.
- **Contracts** (`contracts/pacts/`): the Pact files the order service needs payment and catalog to honour.
- **Contracts** (`contracts/pacts/`): the Pact files the order service needs payment, catalog and inventory to honour.
- **Redis** (Homebrew `redis-server`, in memory, port 6390): the event bus. `make run` starts and stops it.
- **Dockerfiles**: one per service plus gateway and registry, multi-stage, non-root, built only in
the cloud or CI for `linux/amd64`.
- **React + TypeScript** frontend (`frontend/`) with a page per pattern.

Still to come (M3): Redis Streams events with an outbox, checkout saga, hardened containers,
deploy pipeline, and a real Azure run.
Still to come (M3b): path-filtered CI with a manual Deploy workflow (OIDC), and a real Azure run.

## The live demos

Expand Down Expand Up @@ -105,11 +108,30 @@ The live registry. Stop an instance: its heartbeat stops, the registry expires i

<img width="100%" alt="Service Discovery demo: registry table and an expired then re-registered order replica" src="docs/screenshots/p12-service-discovery.jpg" />

### 7. Event-Driven Architecture
Disconnect the bus and buy a mug: OrderConfirmed waits in the order service's outbox, and the mug
stays held in inventory. Reconnect: the event goes out and inventory turns the hold into a sale.

<img width="100%" alt="Event-Driven demo: bus disconnected, one OrderConfirmed waiting in the outbox, one mug held" src="docs/screenshots/p07-event-driven.jpg" />

### 8. Saga
Buy 2 mugs with "make the bank decline the card" ticked. The saga prices the cart, holds the stock,
the charge fails, and it releases the stock. Stock ends exactly where it started; the cart is kept.

<img width="100%" alt="Saga demo: a declined card, with the stock hold released by the compensation step" src="docs/screenshots/p08-saga.jpg" />

### 9. Containerization
How each process runs (uid, container or not, platform, build). Locally they are plain processes;
CI builds and runs every image for linux/amd64 and fails any that runs as root.

<img width="100%" alt="Containerization demo: every process, its uid and platform, and the Dockerfile shape" src="docs/screenshots/p09-containerization.jpg" />

## Run locally (no Docker)

```bash
brew install redis # once: the event bus (patterns 7 and 8)
make setup # one venv per service + gateway, shop_common, frontend packages
make run # frontend http://127.0.0.1:5273, gateway http://127.0.0.1:8100, registry :8500
make run # frontend :5273, gateway :8100, registry :8500, Redis :6390
make test # unit tests, each suite in its own venv
make contracts # Pact: order writes its contracts, payment and catalog verify them
make smoke # end-to-end checks against the running stack
Expand All @@ -122,5 +144,5 @@ collector. Without it, traces stay in each process's memory and the Observabilit

Azure Container Apps (one app per service plus the gateway, all scale-to-zero) + MySQL Flexible
Server, provisioned with `make provision` and removed with `make teardown`. About $0.21 for an
8-hour session. See [cost_estimate.md](cost_estimate.md). Nothing from M1 or M2 has been deployed
8-hour session. See [cost_estimate.md](cost_estimate.md). Nothing from M1, M2 or M3a has been deployed
yet. In Azure, discovery uses Container Apps DNS (no registry), and the demo switches stay off.
Loading
Loading