🛡️ J-DataPipe: Enterprise Real-Time Fraud Detection Platform

Project Reference: Real-time Architectural Analysis and Fraud Prediction System
Academic Institution: Ho Chi Minh City Open University (HCMOU)
Domain: FinTech, Core Banking, Big Data Analytics

📑 Table of Contents

1. Overview
2. Architectural Topologies
3. Component Reference
4. Performance Benchmarks
5. Prerequisites & Environment
6. Quick Start Deployment
7. API Reference (Gateway)
8. Observability & Monitoring

---

📖 Overview

J-DataPipe is a highly scalable, fault-tolerant Data Platform engineered for commercial transaction identification. Designed specifically for the FinTech and Banking sector, it addresses the critical tradeoff between Processing Latency and System Throughput by providing dual architectural strategies:

1. In-line Blocking (Pre-auth): Synchronous transaction interception.
2. Out-of-band (Post-auth): Asynchronous streaming via CDC.

At the core of the prediction layer is a high-speed XGBoost model served via gRPC Multiplexing to minimize serialization overhead, integrated directly with PostgreSQL and Redis Cache for state management.

---

🏛️ Architectural Topologies

Topology A: In-line Blocking (Synchronous API)

This architecture is mandatory for transactions that require instant authorization (e.g., ATM withdrawals, Point-of-Sale). The transaction is blocked until the AI model computes the fraud probability.

Workflow:

1. [Client] -> HTTP POST /transaction -> [Golang API Gateway]
2. [Golang API Gateway] -> MGET -> [Redis] (Fetch Source & Dest Balances)
3. [Golang API Gateway] -> gRPC PredictBatch -> [Python ML Model] (Compute Fraud Score)
4. [Python ML Model] -> Returns Fraud Probability Score -> [Golang API Gateway]
5. Decision Logic:
   • If Score < Threshold (0.5): Gateway inserts to transactions table, updates Redis balances, returns 200 OK (SUCCESS).
   • If Score >= Threshold (0.5): Gateway inserts to fraud_alerts table, returns 403 Forbidden (BLOCKED).

System Architecture Diagram:

graph TD
    Client([👤 Client]) -->|1. HTTP POST /transaction| Gateway[⚡ Golang API Gateway]
    
    Gateway -->|2. MGET Balances| Redis[(🛢️ Redis Cache)]
    Gateway <-->|3. gRPC PredictBatch| ML[🧠 Python ML Model XGBoost]
    Gateway -->|4. Update Balances| Redis
    
    Gateway -->|5a. Score < 0.5| TxDB[(🐘 PostgreSQL: transactions)]
    Gateway -->|5b. Score >= 0.5| FraudDB[(🐘 PostgreSQL: fraud_alerts)]
    Gateway -.->|Write-Behind Async| Kafka[[🚀 Apache Kafka]]
    
    classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px;
    classDef gateway fill:#e0f7fa,stroke:#00acc1,stroke-width:2px,color:#000;
    classDef ml fill:#fff3e0,stroke:#ff9800,stroke-width:2px,color:#000;
    classDef db fill:#e8eaf6,stroke:#3f51b5,stroke-width:2px,color:#000;
    classDef stream fill:#fce4ec,stroke:#e91e63,stroke-width:2px,color:#000;
    
    class Gateway gateway;
    class ML ml;
    class Redis,TxDB,FraudDB db;
    class Kafka stream;

Loading

---

🧩 Component Reference

1. Payment Gateway (Golang)

• Role: Entry point for In-line transactions.
• Why Go?: Replaced the legacy Python FastAPI implementation to solve the Global Interpreter Lock (GIL) and Context Switching bottlenecks. Uses goroutines for multiplexed network I/O.
• Connections: database/sql + lib/pq for PG connection pooling; go-redis/v8 for cache.

3. Machine Learning (Python gRPC)

• Model: Pre-trained XGBoost Classifier.
• Protocol: Protobuf binary serialization ensures payload size is minimized compared to JSON REST.
• Scaling: Managed by Kubernetes HPA (Horizontal Pod Autoscaler) to scale from 3 to 10+ Pods dynamically under load.

---

📈 Performance Benchmarks

All tests were conducted on a Kubernetes cluster with 8 vCPUs and 16GB RAM. The Banking Compliance standard mandates an end-to-end response time of < 2000 ms.

Benchmark A: Golang In-line Gateway

Test environment configured with 8 ML Pod Replicas to match CPU cores.

Concurrent TPS	Success Rate	Average Latency	Max Latency	Verdict
10 TPS	100%	105 ms	114 ms	Excellent for P2P Transfers
100 TPS	100%	228 ms	356 ms	Ideal for standard Payment Gateways
300 TPS	100%	836 ms	1.37 s	Meets < 2s compliance standard
500 TPS	98.2%	1.39 s	2.03 s	ML CPU saturation reached. Requires Cluster Scale-out.

Crucial Insight: The Golang migration eliminated gateway-level connection drops entirely. The system now perfectly transfers the bottleneck to the compute-heavy ML Inference layer, which scales linearly by adding more Kubernetes Nodes.

🛠️ Prerequisites & Environment

Hardware Requirements

• CPU: 8+ Physical Cores
• RAM: Minimum 16 GB (24 GB Recommended)
• Disk: SSD with 50 GB free space

Software Stack

• Docker Engine: v24.0+
• Minikube: v1.31+
• Kubernetes CLI (kubectl): v1.27+
• Helm: v3.0+

---

🚀 Quick Start Deployment

1. Initialize Kubernetes Cluster

minikube start --cpus=8 --memory=16384 --disk-size=50g

2. Execute Master Setup

Deploys Postgres, Redis, Kafka, Flink, MLflow, Airflow, and Grafana.

chmod +x set-up.sh
./set-up.sh

3. Deploy the Golang Payment Gateway

./deploy_inline_go.sh

4. Stress Test the Architecture

To evaluate the In-line Blocking architecture on your own hardware:

# Simulates 500 concurrent transaction threads
python run_inline_benchmark.py --tps 500

---

📡 API Reference (Gateway)

POST /api/v1/transaction

Description: Validates and processes a transaction synchronously.

Headers:

• Content-Type: application/json

Request Body:

{
  "step": 1,
  "type": "TRANSFER",
  "amount": 1500.50,
  "nameOrig": "C12345678",
  "nameDest": "M87654321",
  "isFlaggedFraud": 0
}

Response (200 OK - Approved):

{
  "transaction_id": "tx_abc123",
  "status": "SUCCESS",
  "fraud_probability": 0.012
}

Response (403 Forbidden - Blocked):

{
  "transaction_id": "tx_abc123",
  "status": "BLOCKED",
  "fraud_probability": 0.987,
  "message": "Transaction intercepted due to high fraud risk."
}

---

🔭 Observability & Monitoring

The platform provides out-of-the-box telemetry using the Prometheus & Grafana stack.

Accessing Dashboards

1. Port-forward Grafana:

   kubectl port-forward svc/monitor-stack-grafana 3000:80 -n monitoring

2. Login: Navigate to http://localhost:3000
   • Username: admin
   • Password: kubectl get secret -n monitoring monitor-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode
3. Import Layout: Upload assets/images/DB_Dashboard.json for the Fraud Analytics layout.

Key Metrics Tracked

• grpc_server_handled_total: Total inference requests processed.
• flink_taskmanager_job_task_operator_numRecordsIn: Kafka ingestion throughput.
• go_goroutines: Active concurrent gateway connections.
• pg_stat_activity: Database connection pool saturation.

---

Maintained by the J-DataPipe Architecture Team.