🚀 Real-Time Distributed Ad Bidding Platform

A production-style cloud-native distributed ad bidding system built using microservices, Apache Kafka, Kubernetes, and AWS EKS.

This project simulates how modern real-time bidding (RTB) systems process ad auctions in milliseconds using asynchronous event-driven workflows, distributed messaging, and scalable infrastructure.

---

🧠 Project Vision

Modern ad-tech systems operate under extreme scale and latency constraints.

When a user visits a webpage:

• advertisers compete for ad inventory
• bid requests are generated in real time
• multiple bidders respond asynchronously
• an auction engine selects the winner
• budgets are updated dynamically
• analytics and traces are generated across services

All of this must happen within milliseconds.

This project focuses not only on building backend APIs, but on operating a distributed cloud-native system using modern DevOps and platform engineering practices.

---

🏗️ Distributed System Architecture

                    ┌────────────────────┐
                    │    Client/User     │
                    └─────────┬──────────┘
                              │
                              ▼
                 AWS Application Load Balancer
                              │
                              ▼
                    Kubernetes Ingress
                              │
                              ▼
                    Campaign Service
                              │
                              ▼
                        Kafka Topics
                              │
                              ▼
                     Bidder Service
                              │
                              ▼
                        Kafka Topics
                              │
                              ▼
                     Auction Service
                              │
                              ▼
                           MySQL

------------------------------------------------------------

Observability Stack:

Prometheus → Metrics Collection
Grafana    → Visualization & Dashboards
Zipkin     → Distributed Tracing

---

⚙️ Tech Stack

Backend

• Java 21
• Spring Boot
• Spring Kafka
• Spring Data JPA
• Maven

Distributed Systems

• Apache Kafka
• Event-Driven Architecture
• Asynchronous Messaging
• Consumer Groups
• Partition-based Processing
• Idempotent Event Handling

Cloud & DevOps

• Docker
• Docker Compose
• Kubernetes
• AWS EKS
• Helm
• AWS Load Balancer Controller
• EBS CSI Driver

Observability

• Prometheus
• Grafana
• Zipkin
• Spring Boot Actuator
• Micrometer

Databases

• MySQL
• Persistent Volumes (EBS)

---

🚀 Core Microservices

Service	Responsibility
Campaign Service	Campaign creation, budget management
Bidder Service	Consumes bid requests and generates bids
Auction Service	Processes bids and selects auction winner
Kafka	Distributed event backbone
MySQL	Persistent campaign and auction storage

---

🔥 Event Flow

Client Request
      ↓
Campaign Service
      ↓
Kafka → BidRequestEvent
      ↓
Bidder Service
      ↓
Kafka → BidResponseEvent
      ↓
Auction Service
      ↓
Kafka → AuctionResultEvent
      ↓
Campaign Service
      ↓
Budget Update

---

☁️ AWS EKS Deployment

The entire platform is deployed on AWS EKS using Kubernetes.

Infrastructure components include:

• AWS EKS Cluster
• Managed Node Groups
• Kubernetes Deployments & Services
• AWS ALB Ingress
• Persistent Volume Claims
• EBS CSI Driver
• Namespace Isolation
• Kubernetes DNS-based Service Discovery

---

📊 Observability & Monitoring

The platform includes full observability support.

Prometheus

• JVM metrics
• Kubernetes metrics
• Pod resource monitoring
• Service metrics

Grafana

• Kubernetes dashboards
• JVM dashboards
• Infrastructure visualization
• Live monitoring

Zipkin

• Distributed request tracing
• Cross-service request visibility
• End-to-end event tracing

---

🧠 Distributed Systems Concepts Implemented

• Event-Driven Architecture
• Asynchronous Processing
• Distributed Messaging
• Consumer Groups
• Event Ordering
• Idempotency
• Retry Handling
• Fault Isolation
• Decoupled Services
• Stateful vs Stateless Workloads
• Distributed Tracing
• Scalable Event Pipelines

---

⚡ DevOps & Platform Engineering Practices

This project heavily focuses on infrastructure and operational engineering:

• Containerized Microservices using Docker
• Kubernetes Workload Orchestration
• AWS Cloud Deployment
• Helm-based Infrastructure Management
• Ingress Routing & Load Balancing
• Persistent Storage Provisioning
• Metrics Collection & Monitoring
• Distributed Observability
• Infrastructure Debugging
• Cloud-Native Networking
• Namespace Isolation
• Public API Exposure through ALB

---

🐳 Running Locally with Docker Compose

Start Infrastructure

docker compose up -d

---

☸️ Kubernetes Deployment

Apply Infrastructure

kubectl apply -f k8s/mysql
kubectl apply -f k8s/kafka

Deploy Services

kubectl apply -f k8s/campaign-service
kubectl apply -f k8s/bidder-service
kubectl apply -f k8s/auction-service

Deploy Observability Stack

helm install prometheus prometheus-community/kube-prometheus-stack -n observability

---

📡 Kafka Topics

Topic	Description
bid-requests	Incoming bid requests
bid-responses	Bidder responses
auction-results	Final auction results

---

🚀 Current Capabilities

✅ Distributed Event-Driven Communication
✅ Kubernetes Orchestration
✅ AWS Cloud Deployment
✅ Public Ingress Routing
✅ Persistent Stateful Workloads
✅ Distributed Tracing
✅ Metrics Monitoring
✅ Grafana Dashboards
✅ Kafka-based Async Processing
✅ Production-style Infrastructure Setup

---

🚧 Future Improvements

• K6 Load Testing
• Horizontal Pod Autoscaling (HPA)
• Redis-based Bid Aggregation
• Dead Letter Queue (DLQ)
• CI/CD Pipelines
• Multi-region Kafka Setup
• API Gateway Integration
• Chaos Testing
• Service Mesh (Istio)

---

🧠 Key Learnings

This project fundamentally changed how I think about backend engineering.

Building APIs is only one small part of distributed systems engineering.

The real complexity starts when:

• services become asynchronous
• infrastructure becomes distributed
• failures become non-deterministic
• observability becomes mandatory
• scalability becomes operational

This project provided hands-on experience with:

• distributed systems
• cloud-native infrastructure
• Kubernetes operations
• DevOps workflows
• production-style observability

---

👨‍💻 Author

Built as a deep dive into:

• Distributed Systems
• Event-Driven Architecture
• Cloud-Native Infrastructure
• Kubernetes
• DevOps Engineering
• Scalable Backend Systems

---

⭐ GitHub Repository

https://github.com/98001yash/ad-bidding-system