E-Commerce Clickstream & Inventory Watch

Real-time big data pipeline for e-commerce clickstream analysis. Generates synthetic events (50 users, 20 products), streams via Kafka, processes with PySpark (10-min windows, flash-sale detection), and runs daily batch segmentation via Airflow.

Components

• Producer: Generates 25 events/sec (50 users, 20 products)

• Kafka: Streams events via clickstream_topic (3 partitions)

• Stream Processor: PySpark 10-min windows, computes metrics, detects flash-sales (view≥100 & purchase≤5)

• Airflow: Daily DAG - user segmentation, reports, optional email

• PostgreSQL: 4 tables (clickstream_events, product_metrics, user_segments, daily_product_summary)

Features

• Real-time event generation with synthetic data
• 10-minute sliding window aggregations via PySpark
• Flash-sale detection (high views + low purchases)
• Daily batch user segmentation (Buyers vs. Window Shoppers)
• Report generation (CSV/TXT with optional email)
• Full Docker stack orchestration

Tech Stack

• Languages: Python 3.9+, SQL
• Streaming: Apache Spark 3.5, Kafka 7.5, Zookeeper
• Orchestration: Apache Airflow 2.7
• Database: PostgreSQL 15
• Containers: Docker & Docker Compose

Data Flow

1. Event Generation: Producer generates 25 events/sec → Kafka
2. Event Streaming: Kafka distributes events via clickstream_topic
3. Real-time Processing: PySpark reads Kafka, computes 10-min windows, detects flash-sales → PostgreSQL
4. Batch Processing: Airflow DAG segments users, generates reports
5. Analytics: Query PostgreSQL for real-time metrics and historical trends

How to Run the Project

Prerequisites

• Docker & Docker Compose
• 8GB+ RAM
• Ports: 2181, 9092, 5432, 8080, 8000

Quickstart

1. Start all services:

   docker compose up -d --build

2. Verify services:

   docker compose ps

3. Check logs:

   docker logs -f python-producer
   docker logs -f stream-processor

Access Services

Service	URL/Connection	Credentials
Airflow	http://localhost:8080	admin / admin
Kafka UI	http://localhost:8000	(none)
PostgreSQL	localhost:5432	airflow / airflow
Kafka	localhost:9092	(none)

Stop

docker compose down          # keep data
docker compose down -v       # remove all

Project Structure

.
├── producer.py                     # Event generator (50 users, 20 products)
├── stream_processor.py             # PySpark streaming (10-min windows)
├── config.py                       # Configuration management
├── init_db.sql                     # PostgreSQL schema
├── dags/
│   └── dag_segmentation.py         # Airflow DAG (daily batch)
├── docker-compose.yaml             # Service orchestration
├── Dockerfile.producer
├── Dockerfile.processor
├── Dockerfile.airflow
├── requirements.txt
└── .env.example

Events API

Clickstream event schema (Kafka clickstream_topic):

{
  "user_id": 23,
  "product_id": 7,
  "event_type": "view",
  "timestamp": "2026-05-24T10:30:45Z",
  "session_id": "session_23_1716540645",
  "device": "mobile"
}

Event Types: view (85%), add_to_cart (10%), purchase (5%)

Development & Monitoring

Logs

# All services
docker compose logs -f

# Producer
docker logs -f python-producer

# Stream processor
docker logs -f stream-processor

Query Metrics

# Real-time product metrics
docker compose exec postgres-db psql -U airflow -d clickstream_db -c \
  "SELECT product_id, view_count, purchase_count, flash_sale_suggested FROM product_metrics LIMIT 5;"

# Daily segments
docker compose exec postgres-db psql -U airflow -d clickstream_db -c \
  "SELECT * FROM user_segments WHERE segment_date = CURRENT_DATE LIMIT 10;"

Browse Kafka

• UI: http://localhost:8000
• CLI: docker exec kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic clickstream_topic --max-messages 5

License

This project is licensed under the MIT License - see the LICENSE file for details.