diff --git a/locustfile.py b/locustfile.py new file mode 100644 index 0000000..0b6ea4c --- /dev/null +++ b/locustfile.py @@ -0,0 +1,33 @@ +"""Locust load-test for QuickTicket gateway. + +Runs in-cluster as a Job so traffic goes through kube-proxy and is +load-balanced across all gateway replicas. The task mix mirrors real +traffic: listing events dominates, some checkout, some health-polling. + +Reserves are spread across events 3 and 5 (500 + 80 tickets) so a 60s +run at 50-100 users doesn't exhaust a single event's inventory and +pollute the error count with 409 Conflicts. +""" +import random +from locust import HttpUser, task, between + + +class QuickTicketUser(HttpUser): + wait_time = between(0.5, 2.0) + + @task(7) + def list_events(self): + self.client.get("/events") + + @task(2) + def reserve(self): + event = random.choice([3, 3, 3, 5]) + self.client.post( + f"/events/{event}/reserve", + json={"quantity": 1}, + headers={"Content-Type": "application/json"}, + ) + + @task(1) + def health(self): + self.client.get("/health") diff --git a/submissions/lab10.md b/submissions/lab10.md new file mode 100644 index 0000000..5cee103 --- /dev/null +++ b/submissions/lab10.md @@ -0,0 +1,67 @@ +# QuickTicket Reliability Review + +## 1. SLO Compliance +| SLO | Target | Observed | Status | +| --- | ------ | -------- | ------ | +| Gateway 5xx Error Rate | < 0.5% | 20.57% (at 100 users) | Missed | +| Gateway p99 Latency | < 500ms | 2100 ms (at 100 users) | Missed | + +## 2. Load Test Results +| Users | Ramp | RPS | p50 | p95 | p99 | 5xx error rate | 409 (inventory) | +| ----: | ---: | ----: | ----: | -----: | -----: | -------------: | --------------: | +| 10 | 2/s | 7.60 | 19ms | 100ms | 480ms | 0.00% | 0 | +| 50 | 5/s | 35.96 | 14ms | 63ms | 100ms | 0.00% | 23 | +| 100 | 10/s | 58.02 | 210ms | 1300ms | 2100ms | 19.27%* | 45 | + +*\*Note: 5xx error rate calculated by subtracting 45 (409 Conflict) from 713 total failures = 668 actual 5xx errors (19.27% of total 3467 requests).* + +**Breaking Point Analysis:** +The system's capacity ceiling is approximately **36-40 RPS** at **50 concurrent users**. At 100 users, the system experienced a large degradation: p99 latency reached **2100ms** and the 5xx error rate soared to **19.27%** due to a flood of `502 Bad Gateway`, `503 Service Unavailable`, and `Connection refused` errors, signaling cascading service failure. + +## 3. DORA Metrics +| Metric | Value | Source / Calculation | +| --------------------------- | ---------- | ------------------------------------------------------------------------------------------ | +| **Deployment Frequency** | 9 rollouts | Total count of ReplicaSets (`kubectl get rs`) across all services. | +| **Lead Time for Changes** | ~5 minutes | CI build time (~2 min) + ArgoCD poll interval (~3 min). | +| **Change Failure Rate** | 0% | No failed Argo Rollouts AnalysisRuns detected in the current cluster state. | +| **Time to Restore Service** | ~3 minutes | Time to execute `git revert` and wait for ArgoCD sync, or instant abort via Argo Rollouts. | + +## 4. Top 3 Reliability Risks +1. **Ephemeral Database Storage** — Without a PVC, a single Postgres pod restart wipes all order data. *Fix: PersistentVolumeClaims.* +2. **Unbounded Database Connection Pool** — High user load directly spawns new Postgres connections, causing services to hit `Connection refused` and crash. *Fix: Implement PgBouncer.* +3. **Single Point of Failure in Redis** — A single Redis instance handling all ticket reservations limits capacity and acts as a SPOF. *Fix: Deploy a Redis cluster/sentinel setup.* + +## 5. Toil Identification +| Toil Task | Frequency | Proposed Automation | Time Saved | +| -------------------------------------------------- | ------------------------ | ---------------------------------------------------------------- | ------------------------ | +| Re-seeding DB manually after Postgres pod restarts | Every restart (Labs 1-8) | Attach a PVC so data persists across pod restarts (Lab 9 Bonus). | ~2 mins per incident | +| Re-establishing `kubectl port-forward` | Constantly during dev | Expose services via Ingress controller / NodePort. | Interruptions eliminated | +| Staring at terminal during canary rollouts | Every deploy (Lab 7) | Argo Rollouts + Prometheus AnalysisRuns to auto-promote/abort. | ~5 mins per deploy | + +## 6. Monitoring Gaps +- **Missing Dependency Latency Alerts:** During Lab 8 chaos experiments, if the database became slow (but didn't throw 5xx), our error-rate SLO wouldn't catch it until the gateway completely timed out. We needed latency-specific alerts for downstream services (events/payments). +- **Missing Redis Capacity Alerts:** If Redis runs out of memory or connections during a ticket spike, we currently don't have an alert for it until users start seeing 500s from the gateway. + +## 7. Capacity Plan + +**Per-pod metrics at breaking point (100 users / ~58 RPS, measured during 3-min sustained load):** +- **Gateway (5 replicas):** ~100m CPU, 42Mi Mem (per pod) +- **Events (1 replica):** ~413m CPU, 51Mi Mem +- **Payments (1 replica):** ~23m CPU, 36Mi Mem + +**Analysis of Bottleneck:** +Sustained load testing revealed that the single `events` pod is the main bottleneck, spiking to over 400m CPU. The system failure at 100 users (502 Bad Gateway` and `Connection refused`) `is caused by the events service struggling to process the concurrency queue. It either hits CPU throttling limits or exhausts its internal application thread/connection pool, causing upstream requests from the gateway to time out.` + +**Scaling Plan for 2x Traffic (~80-100 RPS target):** +- **Events Service:** Scale from 1 to 3 replicas to distribute the processing load. Set explicit resources requests/limits (e.g., Request: 200m, Limit: 500m). +- **Payments Service:** Scale from 1 to 2 replicas (primarily for High Availability, as CPU usage is very low). +- **Gateway Service:** Keep at 5 replicas (100m CPU per pod is perfectly healthy and well-distributed). +- **Database:** Introduce **PgBouncer** before scaling `events` to 3 replicas, to prevent the newly scaled pods from overwhelming the Postgres `max_connections` limit. +- **Cache:** Move Redis to a Sentinel/Cluster setup to remove the single point of failure. + +**Rough Cost Estimate:** +Assuming small cloud containers (e.g., ~$5/pod per month): +- 5 Gateway + 3 Events + 2 Payments + 1 Postgres + 3 Redis = 14 total pods. +- **Estimated Total Cost:** ~$70 / month for the compute footprint. + +