EduScale

Distributed real-time EdTech platform — Socket.io at scale, Redlock, circuit breaker, Prometheus

---

What it is

EduScale is an engineering learning platform with real-time coding battles. The interesting engineering problem: when two players join a battle room, the server that handles "player 1 joined" and the server that handles "player 2 joined" may be different Node.js instances. Without distributed coordination, the battle never starts.

This README focuses on the infrastructure decisions — the distributed architecture, not the features.

---

Architecture

Browser ──────────────────────────────────────────────────────────────
          WebSocket (wss://)        HTTP (REST/Next.js SSR)
               │                              │
         ┌─────▼──────┐               ┌───────▼───────┐
         │  Socket.io  │               │  Next.js 16   │
         │  Server A   │               │  App Router   │
         └─────┬───────┘               └───────┬───────┘
               │                              │
    @socket.io/redis-adapter          Prisma + PostgreSQL
               │
         ┌─────▼───────────────────────────────┐
         │              Redis                  │
         │  • Pub/Sub (socket.io cross-node)   │
         │  • Redlock (battle start mutex)     │
         │  • Bull queue (background jobs)     │
         └─────────────────────────────────────┘
               │
         ┌─────▼───────┐
         │  Socket.io  │
         │  Server B   │
         └─────────────┘

---

Key design decisions

1. @socket.io/redis-adapter — multi-instance Socket.io

The default Socket.io in-memory adapter only broadcasts events within a single Node.js process. When deployed on Vercel/Railway with multiple instances, socket.to(room).emit(...) would reach only sockets connected to the same instance.

The Redis adapter publishes every room event to a Redis Pub/Sub channel. All instances subscribe and re-emit to their local sockets:

import { createAdapter } from '@socket.io/redis-adapter';
import { createClient } from 'redis';

const pubClient = createClient({ url: process.env.REDIS_URL });
const subClient = pubClient.duplicate();
await Promise.all([pubClient.connect(), subClient.connect()]);

io.adapter(createAdapter(pubClient, subClient));

Without this: battles only work when both players hit the same server. With this: horizontal scaling works transparently.

2. Redlock — distributed mutex on battle start

When two players join simultaneously, both Socket.io servers may detect "room is full" at the same time and try to start the battle. Without a lock, both execute the start logic — double-starting a battle corrupts scoring.

Redlock implements the Redlock algorithm: acquire a lock across Redis with a TTL before executing the critical section.

import Redlock from 'redlock';

// Shared Redlock instance (cacheService.ts). Retries are sensible for general
// cache/section locks, so the shared config keeps a retry budget:
const redlock = new Redlock([redis], {
  retryCount: 10,
  retryDelay: 200,
  retryJitter: 200,
  driftFactor: 0.01,
});

// Battle start overrides to FAIL-FAST per acquisition (battleRepository.withBattleLock):
async function withBattleLock<T>(battleId: string, ttlMs: number, fn: () => Promise<T>) {
  // retryCount: 0 → if the lock is taken, the other instance already won the race
  const lock = await redlock.acquire([`battle:lock:${battleId}`], ttlMs, { retryCount: 0 });
  try {
    return await fn(); // only one instance reaches here
  } finally {
    await lock.release().catch(() => {}); // ignore "already expired/released"
  }
}

Why fail-fast (retryCount: 0) on battle start specifically: if the lock is taken, the other instance already won the race and is starting the battle. Retrying would queue a second start attempt that fires after the first completes — restarting an already-running battle. So the shared lock retries (good for caches), but the battle-start path overrides to fail-fast per acquisition. This override lives in battleRepository.ts.

3. opossum — circuit breaker on code execution

The code execution service (external sandbox) is the most likely failure point. If it goes down or becomes slow, every battle hangs waiting for execution results.

opossum wraps the execution call with a circuit breaker:

import CircuitBreaker from 'opossum';

const executionBreaker = new CircuitBreaker(executeCode, {
  timeout: 3000,          // 3s — execution should be fast
  errorThresholdPercentage: 50,  // open if >50% fail
  resetTimeout: 10000,    // try again after 10s
  volumeThreshold: 5,     // need 5 calls before tripping
});

executionBreaker.fallback(() => ({
  output: '',
  error: 'Code execution unavailable. Score based on test cases submitted.',
  timedOut: true,
}));

When the circuit is open, battles continue with the fallback — players can still submit, scoring just uses the already-submitted results.

4. prom-client — Prometheus /metrics

import { collectDefaultMetrics, Counter, Histogram, register } from 'prom-client';

collectDefaultMetrics();

export const battleStarted = new Counter({
  name: 'eduscale_battles_started_total',
  help: 'Total battles started',
  labelNames: ['mode'],   // 1v1 | ffa
});

export const executionDuration = new Histogram({
  name: 'eduscale_code_execution_duration_seconds',
  help: 'Code execution latency',
  buckets: [0.1, 0.5, 1, 2, 5],
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

Metrics exposed: http_request_duration_seconds histogram (method, route, status_code), active_connections gauge, memory_usage_bytes heap gauge, and default Node.js process metrics via prom-client.

5. Bull queues — async background processing

Background operations such as transactional email deliveries and periodic audit log retention cleanups are handled through Redis-backed Bull queues, decoupling asynchronous processing from request handlers:

import Queue from 'bull';

// Transactional email dispatch queue
export const emailQueue = new Queue('email-queue', REDIS_URL);

// Scheduled audit log retention cleanup
export const auditRetentionQueue = new Queue('audit-retention', REDIS_URL);

---

Running locally

Prerequisites

• Node.js 18+
• PostgreSQL (local or Supabase free tier)
• Redis (local or Upstash free tier)

1. Clone

git clone https://github.com/Shailesh93602/devscale.git
cd devscale

2. Backend

cd Backend
npm install
cp .env.example .env   # fill in values below
npx prisma generate
npx prisma db push
npm run dev            # starts on :5000

3. Frontend

cd Frontend
npm install
cp .env.example .env   # fill in values below
npm run dev            # starts on :3000

---

Environment variables

Backend (Backend/.env)

Variable	Required	Description
DATABASE_URL	Yes	PostgreSQL connection string
REDIS_URL	Yes	Redis URL — used by socket adapter, Redlock, Bull
JWT_SECRET	Yes	Sign JWT access tokens
PORT	No	API port (default: 5000)
CLOUDINARY_CLOUD_NAME	No	Media uploads
CLOUDINARY_API_KEY	No	Media uploads
CLOUDINARY_API_SECRET	No	Media uploads

Frontend (Frontend/.env)

Variable	Required	Description
NEXT_PUBLIC_API_BASE_URL	Yes	Backend REST URL, e.g. http://localhost:5000/api/v1
NEXT_PUBLIC_WS_URL	Yes	Socket.io server URL, e.g. http://localhost:5000
NEXT_PUBLIC_SUPABASE_URL	Yes	Supabase project URL
NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY	Yes	Supabase anon key

---

Tech stack

Layer	Package	Why
Real-time transport	socket.io	WebSocket + fallback, room/namespace model
Multi-instance scaling	@socket.io/redis-adapter	Cross-node Pub/Sub for Socket.io rooms
Distributed locking	redlock	Redlock algorithm — prevents double-start race condition
Circuit breaker	opossum	Protects code execution service; fallback keeps battles running
Metrics	prom-client	Prometheus-compatible /metrics endpoint
Background jobs	bull	Redis-backed queue for async email delivery and audit retention cleanup
ORM	prisma	Type-safe PostgreSQL queries
Frontend	next.js 16	App Router, SSR, Server Components
State	redux-toolkit	Battle state, user session

---

Live demo

eduscale.vercel.app

---

Built by Shailesh Chaudhari