Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -14,20 +14,31 @@ SECRET_KEY=change-me-to-a-random-string
USER_SERVICE_PORT=5101
CATALOG_SERVICE_PORT=5102
ORDER_SERVICE_PORT=5103
PAYMENT_SERVICE_PORT=5104
INVENTORY_SERVICE_PORT=5105
GATEWAY_PORT=8100
FRONTEND_PORT=5273

# --- Service base URLs (used by the gateway and by services calling each other) ---
USER_SERVICE_BASE_URL=http://127.0.0.1:5101
CATALOG_SERVICE_BASE_URL=http://127.0.0.1:5102
ORDER_SERVICE_BASE_URL=http://127.0.0.1:5103
PAYMENT_SERVICE_BASE_URL=http://127.0.0.1:5104
INVENTORY_SERVICE_BASE_URL=http://127.0.0.1:5105

# --- SQLite (local mode, default) ---
# Optional overrides β€” each path is relative to that service's own directory.
# Leave unset to use instance/<service>.sqlite3 inside each service folder.
# USER_DB_PATH=instance/user.sqlite3
# CATALOG_DB_PATH=instance/catalog.sqlite3
# ORDER_DB_PATH=instance/order.sqlite3
# PAYMENT_DB_PATH=instance/payment.sqlite3
# INVENTORY_DB_PATH=instance/inventory.sqlite3

# --- Observability (pattern 6) ---
# Optional: also export spans over OTLP/HTTP, e.g. to otel-desktop-viewer (http://localhost:4318)
# or an OpenTelemetry collector. Unset = spans stay in each process's in-memory store only.
# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

# --- Azure Database for MySQL Flexible Server (azure mode only) ---
# Populated automatically into .env by infra/provision.py after provisioning.
Expand Down
16 changes: 16 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,27 @@ jobs:
- suite: gateway
requirements: gateway/requirements.txt
tests: gateway/tests
shared: fastapi
- suite: user
requirements: services/user/requirements.txt
tests: services/user/tests
shared: flask
- suite: catalog
requirements: services/catalog/requirements.txt
tests: services/catalog/tests
shared: flask
- suite: order
requirements: services/order/requirements.txt
tests: services/order/tests
shared: flask
- suite: payment
requirements: services/payment/requirements.txt
tests: services/payment/tests
shared: flask
- suite: inventory
requirements: services/inventory/requirements.txt
tests: services/inventory/tests
shared: flask
steps:
- uses: actions/checkout@v4

Expand All @@ -45,6 +57,10 @@ jobs:
- name: Install dependencies
run: pip install -r ${{ matrix.requirements }} pytest==8.3.4 pytest-asyncio==0.25.0

- name: Install shared observability package (shop_common)
if: matrix.shared
run: pip install "./shared[${{ matrix.shared }}]"

- name: Run tests
run: python -m pytest ${{ matrix.tests }} -v

Expand Down
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,10 @@ migrations/versions/__pycache__/

# --- Staged for az acr build, cleaned up after (see infra/provision.py build_images) ---
bff/scripts/

# --- Staged by infra/provision.py for az acr build ---
_shared/
shared/*.egg-info/

# --- TypeScript build cache ---
*.tsbuildinfo
38 changes: 26 additions & 12 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,20 +27,34 @@ history, monolith dropped). The full agreed plan is in [`docs/PLAN.md`](docs/PLA
- Name prefix: `ms12p` (parent uses `flaskms`). Teardown deletes the whole resource group, so a
shared name would delete the other project.

## Current state (M1a, layout)
- Folders: `services/{user,catalog,order}/` (Flask, one DB each), `gateway/` (FastAPI),
`frontend/`, `infra/`, `scripts/`, `tests/`. The monolith and the old migration engine are gone.
## Current state (M1, foundation: patterns 1, 2, 5, 6 live)
- Folders: `services/{user,catalog,order,payment,inventory}/` (Flask, one DB each), `gateway/`
(FastAPI), `shared/` (`shop_common`), `frontend/`, `infra/`, `scripts/`, `tests/`.
- **One service table**: `scripts/service_table.py` lists every local process, folder and port.
`setup.py`, `run_local.py`, `provision.py` and the smoke tests all read it. Add a service there.
- Ports: user 5101, catalog 5102, order 5103, gateway 8100, frontend 5273 (registry 8500 and
payment/inventory 5104-5105 arrive in later milestones).
- Gateway routes `/api/shop/<path>` to `/api/<path>` on the owning service by first path segment
(`gateway/app/config.py` `ROUTE_OWNERS`). No auth or aggregation yet (M1).
- The catalog service keeps the `product` resource name in its API (`/api/product(s)`) and its
`product_api` package. Only the service, its env vars (`CATALOG_*`) and its DB (`catalog_db`) are
named catalog.
- Azure: one Container App per service plus the gateway, all external ingress, images built with
`az acr build --platform linux/amd64`.
- Ports: user 5101, catalog 5102, order 5103, payment 5104, inventory 5105, gateway 8100,
frontend 5273 (registry 8500 arrives in M2).
- **`shop_common`** (`shared/`): every service calls `shop_common.flask_support.install(app, name, db)`
inside its app context. It adds `/health`, `/health/db`, `/debug/spans`, `/metrics`, OpenTelemetry
tracing (Flask, requests, SQLAlchemy per engine), JSON logs with trace/request IDs, and metrics.
The gateway uses `gateway/app/observability.py` (FastAPI + httpx). `setup.py` installs `shared/`
editable into every venv; `provision.py` stages it as `<service>/_shared` for `az acr build`.
- Traces: each process keeps its last 2,000 spans in memory (`SPAN_STORE`); the gateway stitches a
trace from every service's `/debug/spans`. Database span names are trimmed to the file name so
no local path leaks. Containers run gunicorn with 1 worker + 4 threads so one process holds all
spans. `OTEL_EXPORTER_OTLP_ENDPOINT` optionally exports too.
- Gotchas solved: FastAPI tracing middleware must be added AFTER the request-ID middleware (the
last-added Starlette middleware runs outermost); Flask-SQLAlchemy's engine is missed by the
global SQLAlchemy hook, so it is instrumented per engine; the httpx hook only loads where httpx
is installed.
- Gateway: `/api/shop/<path>` (GET/POST only) routed by `ROUTE_OWNERS`; `/api/order-summary`
aggregation; demo endpoints in `gateway/app/routers/patterns.py`. No auth check yet (JWT is M2).
- The catalog service keeps the `product` resource name in its API (`/api/product(s)`). The
unauthenticated `DELETE /api/product/<slug>` was removed.
- Frontend: `/patterns` grid, `/patterns/:slug` page from `frontend/src/patterns/content.ts`, demos
in `frontend/src/patterns/demos/`. Signed-in user is read from localStorage `shop_auth`.
- Smoke tests create users and products in the local SQLite DBs; delete `services/*/instance/`
for a clean catalog.

## Stack
Python 3.12 (Flask 3.1 services, FastAPI gateway), Node 18+ (React 18, TypeScript 5, Vite 5),
Expand Down
2 changes: 2 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ test:
services/user/.venv/bin/python -m pytest services/user/tests -v
services/catalog/.venv/bin/python -m pytest services/catalog/tests -v
services/order/.venv/bin/python -m pytest services/order/tests -v
services/payment/.venv/bin/python -m pytest services/payment/tests -v
services/inventory/.venv/bin/python -m pytest services/inventory/tests -v
gateway/.venv/bin/python -m pytest gateway/tests -v

smoke:
Expand Down
80 changes: 62 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,45 +4,89 @@ A small Contoso e-commerce system that shows **12 microservices patterns working
Each pattern gets its own page with an explanation, when *not* to use it, and a live demo you
can click.

> **Status: M1a, layout.** Three services, a gateway and the frontend run end to end. The foundation is copied from
> **Status: M1, foundation.** 4 of 12 patterns are live: Database per Service, Bounded Context,
> API Gateway and Observability. The foundation is copied from
> [Azure-Ecommerce-Monolith-Microservices-Migration](https://github.com/MAOFILHO/Azure-Ecommerce-Monolith-Microservices-Migration).
> The patterns arrive in 3 milestones. See [docs/PLAN.md](docs/PLAN.md).
> The rest arrive in milestones M2 and M3. See [docs/PLAN.md](docs/PLAN.md).

## The 12 patterns

| # | Pattern | What it does | Milestone |
| # | Pattern | What it does | Status |
|---|---|---|---|
| 1 | Database per Service | Gives each service its own database | M1 |
| 2 | Bounded Context | Splits one large domain into clear business domains | M1 |
| 1 | Database per Service | Gives each service its own database | **Live (M1)** |
| 2 | Bounded Context | Splits one large domain into clear business domains | **Live (M1)** |
| 3 | CI-CD Pipeline | Builds, tests and deploys every service automatically | M3 |
| 4 | Circuit Breaker | Stops calls to a failing service to prevent more failures | M2 |
| 5 | API Gateway | Routes client requests to the right service | M1 |
| 6 | Observability | Collects logs, metrics and traces from every service | M1 |
| 5 | API Gateway | Routes client requests to the right service | **Live (M1)** |
| 6 | Observability | Collects logs, metrics and traces from every service | **Live (M1)** |
| 7 | Event-Driven Architecture | Services communicate by publishing and consuming events | M3 |
| 8 | Saga Pattern | Coordinates distributed transactions and rolls back failures safely | M3 |
| 9 | Containerization | Packages each service with all its dependencies | M3 |
| 10 | Stateless Services | Stores session data outside the service | M2 |
| 11 | Contract Testing | Checks that APIs stay compatible across services | M2 |
| 12 | Service Discovery | Finds service instances automatically at runtime | M2 |

## Planned architecture
## Architecture today

- **5 Flask services**, each with its own database: `user`, `catalog`, `order`, `payment`, `inventory`.
- **FastAPI gateway**: single entry point, JWT check, aggregation, live demo feeds.
- **Service registry**: register, heartbeat, lookup (local); Azure Container Apps DNS in the cloud.
- **Redis Streams** event bus with a transactional outbox; a saga orchestrator for checkout.
- **OpenTelemetry** traces, JSON logs and metrics.
- **React + TypeScript** frontend with a page per pattern.
```
Browser β†’ API gateway ──┬──► user β†’ user_db
(request IDs, β”œβ”€β”€β–Ί catalog β†’ catalog_db
tracing, metrics, β”œβ”€β”€β–Ί order β†’ order_db ──HTTP──► user (validates the caller)
aggregation) β”œβ”€β”€β–Ί payment β†’ payment_db
└──► inventory β†’ inventory_db
```

- **5 Flask services** (`services/`), each with its own database: SQLite locally, one logical
MySQL database each in Azure.
- **FastAPI gateway** (`gateway/`): the only thing the browser talks to. Routes `/api/shop/*` to
the owning service, adds request IDs, aggregates `/api/order-summary`, serves the demo endpoints.
- **`shop_common`** (`shared/`): OpenTelemetry tracing, JSON logs with trace IDs, Prometheus
metrics, and the `/health/db`, `/debug/spans`, `/metrics` endpoints every service exposes.
- **React + TypeScript** frontend (`frontend/`) with a page per pattern.

Still to come: service registry, JWT and replicas, circuit breaker, Pact contract tests (M2);
Redis Streams events with an outbox, checkout saga, hardened containers, deploy pipeline (M3).

## The live demos

### 1. Database per Service
Each service reports the database it owns. Then the order service tries to read the catalog's
`product` table through its own connection, and the database refuses: `no such table: product`.

<img width="100%" alt="Database per Service demo: 5 services each own their tables; the order service is blocked from reading the catalog's product table" src="docs/screenshots/p01-database-per-service.jpg" />

### 2. Bounded Context
A context map of the 5 business contexts, and one product as two contexts see it: Catalog knows
its name and price, Inventory knows how many are on hand.

<img width="100%" alt="Bounded Context demo: context map and one product seen by Catalog and by Inventory" src="docs/screenshots/p02-bounded-context.jpg" />

### 5. API Gateway
One browser call to `/api/order-summary` becomes 4 service calls. The page shows each call, its
status and timing, and the one request ID forwarded to all of them.

<img width="100%" alt="API Gateway demo: one browser call fans out to user, order, catalog and inventory" src="docs/screenshots/p05-api-gateway.jpg" />

### 6. Observability
One click, traced through every process: 25 spans across 5 processes, including each service's
database queries, drawn as a waterfall. Metrics per service come from each `/metrics` endpoint.

<img width="100%" alt="Observability demo: a trace waterfall across gateway, user, order, catalog and inventory" src="docs/screenshots/p06-observability.jpg" />

## Run locally (no Docker)

```bash
make setup
make setup # one venv per service + gateway, shop_common, frontend packages
make run # frontend http://127.0.0.1:5273, gateway http://127.0.0.1:8100
make test
make test # unit tests, each suite in its own venv
make smoke # end-to-end checks against the running stack
```

Optional: set `OTEL_EXPORTER_OTLP_ENDPOINT` to also send spans to an OpenTelemetry viewer or
collector. Without it, traces stay in each process's memory and the Observability page draws them.

## Cloud

Azure Container Apps + MySQL Flexible Server, provisioned with `make provision` and removed with
`make teardown`. About $0.21 for an 8-hour session. See [cost_estimate.md](cost_estimate.md).
Azure Container Apps (one app per service plus the gateway, all scale-to-zero) + MySQL Flexible
Server, provisioned with `make provision` and removed with `make teardown`. About $0.21 for an
8-hour session. See [cost_estimate.md](cost_estimate.md). Nothing from M1 has been deployed yet.
Binary file added docs/screenshots/p01-database-per-service.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/screenshots/p02-bounded-context.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/screenshots/p05-api-gateway.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/screenshots/p06-observability.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions frontend/src/App.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import NavBar from "./components/NavBar";
import Home from "./pages/Home";
import Shop from "./pages/Shop";
import Patterns from "./pages/Patterns";
import PatternPage from "./pages/PatternPage";

export default function App() {
return (
Expand All @@ -13,6 +14,7 @@ export default function App() {
<Route path="/" element={<Home />} />
<Route path="/shop" element={<Shop />} />
<Route path="/patterns" element={<Patterns />} />
<Route path="/patterns/:slug" element={<PatternPage />} />
</Routes>
</main>
</div>
Expand Down
90 changes: 90 additions & 0 deletions frontend/src/api/gatewayClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,16 @@ async function request<T>(path: string, options: RequestInit = {}): Promise<T> {
return response.json() as Promise<T>;
}

/** Same as request(), but also returns the gateway's request and trace IDs for this call. */
async function requestWithIds<T>(path: string, options: RequestInit = {}): Promise<{ body: T; requestId: string | null; traceId: string | null }> {
const response = await fetch(`${GATEWAY_BASE_URL}${path}`, options);
const body = (await response.json()) as T;
if (!response.ok) {
throw Object.assign(new Error(`${response.status} ${response.statusText}`), { body });
}
return { body, requestId: response.headers.get("X-Request-Id"), traceId: response.headers.get("X-Trace-Id") };
}

function toForm(fields: Record<string, string | number>): URLSearchParams {
const form = new URLSearchParams();
for (const [key, value] of Object.entries(fields)) form.append(key, String(value));
Expand Down Expand Up @@ -79,3 +89,83 @@ export const shopApi = {
headers: authHeaders(apiKey),
}),
};

// --- Pattern demo endpoints --------------------------------------------------------------------

export interface ServiceCall {
service: string;
path: string;
status: number | string;
ms: number;
}

export interface ServiceDatabase {
service: string;
reachable: boolean;
engine?: string;
database?: string;
tables?: string[];
}

export interface IsolationResult {
attempted: string;
owner: string;
order_tables: string[];
blocked: boolean;
error: string | null;
}

export interface BoundedContextView {
product_id: number;
catalog: Product | null;
inventory: { product_id: number; on_hand: number; reserved: number; available: number } | null;
calls: ServiceCall[];
}

export interface OrderSummary {
user: { username: string };
lines: { product_id: number; name: string | null; unit_price_cents: number | null; quantity: number; available: number | null }[];
total_cents: number;
calls: ServiceCall[];
request_id: string;
trace_id: string | null;
}

export interface Span {
trace_id: string;
span_id: string;
parent_id: string | null;
name: string;
service: string;
kind: string;
start_ns: number;
end_ns: number;
status: string;
}

export interface ServiceMetrics {
service: string;
reachable: boolean;
requests?: number;
errors_5xx?: number;
avg_ms?: number | null;
}

export const patternsApi = {
databases: () => request<{ services: ServiceDatabase[] }>("/api/patterns/databases"),
isolationTest: (table: string) => request<IsolationResult>(`/api/patterns/isolation-test?table=${encodeURIComponent(table)}`),
boundedContext: (productId: number) => requestWithIds<BoundedContextView>(`/api/patterns/bounded-context/${productId}`),
orderSummary: (apiKey: string) => requestWithIds<OrderSummary>("/api/order-summary", { headers: authHeaders(apiKey) }),
trace: (traceId: string) => request<{ trace_id: string; spans: Span[] }>(`/api/observability/trace/${traceId}`),
metrics: () => request<{ services: ServiceMetrics[] }>("/api/observability/metrics"),
};

/** The Shop page stores the signed-in user here; demos that need a login reuse it. */
export function savedApiKey(): string | null {
try {
const raw = localStorage.getItem("shop_auth");
return raw ? (JSON.parse(raw) as { apiKey: string }).apiKey : null;
} catch {
return null;
}
}
12 changes: 6 additions & 6 deletions frontend/src/pages/Home.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,18 @@ export default function Home() {
</Link>
<Link to="/patterns" className="card" style={{ textDecoration: "none", color: "inherit" }}>
<h3>Patterns</h3>
<p className="muted">The 12 patterns, each with a live demo as its milestone lands.</p>
<p className="muted">The 12 patterns. Database per Service, Bounded Context, API Gateway and Observability are live now.</p>
</Link>
</div>

<section className="card">
<h2>Architecture today</h2>
<pre style={{ overflowX: "auto" }}>{`
Browser β†’ API gateway ──┬──► user β†’ user_db
β”œβ”€β”€β–Ί catalog β†’ catalog_db
└──► order β†’ order_db
β”‚
└──HTTP──► user (validates the caller)
Browser β†’ API gateway ──┬──► user β†’ user_db
(request IDs, β”œβ”€β”€β–Ί catalog β†’ catalog_db
tracing, metrics, β”œβ”€β”€β–Ί order β†’ order_db ──HTTP──► user (validates the caller)
aggregation) β”œβ”€β”€β–Ί payment β†’ payment_db
└──► inventory β†’ inventory_db
`}</pre>
</section>
</div>
Expand Down
Loading
Loading