Coder Legion - Recent posts tagged websocket

Building the Realtime Foundation of the MyZubster Metaverse

Wed, 09 Sep 2026 00:12:26 +0000

Building the Realtime Foundation of the MyZubster Metaverse

We are upgrading the MyZubster Metaverse from a REST-synchronized experience into an authenticated realtime multiplayer environment.

Before continuing with the next development phase, here is a transparent update on what is already working and what we are implementing next.

From polling to realtime communication

The first version of the Metaverse uses REST endpoints to manage:

player sessions;
character movement;
shared presence;
chat messages;
emotes;
verified identities;
world synchronization.

This gave us a reliable starting point, but periodic polling is not the final architecture we want for a persistent shared world.

We have now introduced a Socket.IO gateway mounted at:

/realtime

It supports WebSocket communication with HTTP polling available as a compatibility fallback.

The preview runtime is online

The realtime infrastructure has been deployed to an isolated Vercel preview environment.

We verified it using the public health endpoint:

GET /api/realtime/health

The current response is:

{
  "success": true,
  "status": "ok",
  "transport": "socket.io",
  "socketPath": "/realtime",
  "presence": "local-fallback",
  "privacy": "aggregate-only"
}

This confirms that the application is running, Socket.IO is attached correctly, the preview database is reachable, and observability is configured to avoid exposing private user data.

Secure realtime authentication

Clients cannot connect anonymously to protected realtime features.

An authenticated user first requests a short-lived token from:

POST /api/realtime/token

The server creates a dedicated JWT containing only the information required by the realtime gateway.

The gateway verifies:

signature and expiration;
issuer and audience;
realtime-specific purpose;
user identity;
user role.

Authentication alone does not grant access to every channel. Each subscription is authorized independently by the server.

Every user also receives a private channel:

user:<userId>

This provides a controlled foundation for private notifications and direct message delivery.

Presence, messaging, and reconnects

The new gateway already supports the core operations required by a multiplayer environment:

presence.join
presence.heartbeat
presence.leave
channel.subscribe
channel.unsubscribe
chat.send
realtime.resume

When a client reconnects, it can request restoration of its previous subscriptions. The server verifies every channel again before restoring access.

Messages are validated, persisted, deduplicated, and delivered only to authorized recipients.

We also added backpressure controls so the service can reject excess work safely during temporary overload.

Preview database isolation

The preview deployment does not reuse unrestricted production credentials.

We created:

a dedicated MongoDB preview user;
a custom least-privilege role;
access limited to the preview database;
cluster-level restrictions;
encrypted Vercel environment variables;
branch-specific configuration.

The MongoDB URI and realtime signing secret are never committed to GitHub or exposed to frontend code.

Privacy-aware observability

Realtime systems need monitoring, but operational visibility should not become user surveillance.

Our observability layer collects aggregate technical information such as:

connection attempts;
successful and rejected connections;
active connections;
disconnects;
message failures;
duplicate messages;
resume operations;
processing duration;
Redis failures.

Private conversations and raw personal identifiers are not included in public metrics.

The detailed metrics endpoint is restricted to authenticated administrators.

Why presence is still using a fallback

The health response currently reports:

presence: local-fallback

This means presence works inside the active application instance, but it is not yet coordinated across multiple Vercel instances.

It is an important distinction: the realtime runtime is operational, but distributed presence is still the next step.

Redis will provide:

shared presence state;
cross-instance Socket.IO delivery;
instance coordination;
presence expiration;
better horizontal scaling.

What we are working on next

Our next development phase is focused on:

connecting the Metaverse frontend to Socket.IO;
keeping REST synchronization as a temporary fallback;
introducing the Socket.IO Redis adapter;
configuring an isolated preview Redis instance;
testing multiple simultaneous clients;
verifying disconnect and resume behavior;
validating privacy-safe metrics;
completing the production review and merge.

Current project status

Realtime server: implemented
Vercel preview: online
MongoDB connection: verified
Health endpoint: verified
Socket authentication: implemented
Channel authorization: implemented
Presence lifecycle: implemented
Reconnect support: implemented
Aggregate observability: implemented
Frontend Socket.IO connection: in progress
Distributed Redis coordination: next
Production release: pending validation

Building a world, not just a page

The MyZubster Metaverse is intended to become a shared digital space connected to identities, creators, projects, collaboration, and community experiences.

Making that possible requires more than adding a WebSocket. It requires strong security boundaries, failure handling, database isolation, reconnect logic, privacy-aware monitoring, and distributed state management.

The server-side foundation is now online.

The next step is connecting the world to it.

MyZubster: https://www.myzubster.com

GitHub: https://github.com/MyZubster-Ecosystem

How I Moved Real-Time Out of Django (and Made Everything Simpler)

Thu, 16 Jul 2026 14:21:03 +0000

Your users want live updates: an order flips to paid, a notification pops, a dashboard number ticks up without a refresh. The obvious answer in Django is Channels. I reached for it first too. But as our scale grew, the costs of keeping connections in-process pushed me to rethink the architecture. I let Django stop handling WebSockets entirely, and almost everything got simpler. Here's the pattern that replaced it.

The Problem

Django is a request/response framework. It processes a request, returns a response, and moves on. Real-time notifications — a dashboard ticking up, an order status flipping live — need a server that holds a connection open for every connected client. That is a fundamentally different job.

The standard answer in the Django ecosystem is Channels. It moves you to ASGI, gives you consumers, and works well, especially when your real-time needs are deep and bidirectional.

Ours weren't. We needed server→client push: notify a user when an order completed, or when a payment landed. The connections were mostly idle. Channels gave us that, but it came with costs that started hurting at scale.

The concrete problem was this: every open WebSocket connection consumed memory and a small slice of CPU time on our app servers. A thousand connected dashboards was fine. Ten thousand, across dozens of tenants, each holding a persistent socket. Suddenly, our workers were doing two very different jobs at once. Serving fast API responses and babysitting long-lived, mostly-quiet connections. Both suffered.

What made it brittle was the reconnect storm. Deploy a new build, restart a server, or hit a brief network partition, and every connected client disconnects and immediately reconnects. All at once. That thundering herd of TLS handshakes, authentication, and subscription requests would spike CPU to 100% on every app server as it tried to spin up thousands of ASGI consumer instances simultaneously, occasionally knocking them over entirely before the retry backoff kicked in.

There's also the smaller but telling friction of auth. Authenticating the WebSocket handshake means reaching your existing auth (DRF/JWT) from inside an ASGI scope. I ended up writing middleware that reconstructed a DRF Request out of the raw scope just to reuse the auth I already had. It worked, but it was a sign: I was bending the framework to do something it wasn't built for.

The turning point was asking: why is my web framework in the business of holding connections at all?

For a fire-and-forget server→client push, it shouldn't be. There are servers built specifically to hold millions of open connections and do nothing else. The only two things your backend needs to do are say who's allowed to listen, and publish events.

The Solution

Put a dedicated pub/sub gateway in front of your app. I used Centrifugo, a standalone real-time server. Clients connect to it, not to Django. Django goes back to being a plain, boring WSGI/sync app and plays two small roles:

Issue a short-lived connection token that embeds which channels this user may subscribe to.
Publish events to the gateway over its HTTP API.

The gateway holds the connections. Django holds nothing.

Step 1: Issue a token that carries its own permissions

When a client wants to connect, it asks your backend (over a normal authenticated request) for a connection token. The token is a JWT whose claims list the exact channels the user may subscribe to. The gateway verifies the signature and authorizes subscriptions without a single database lookup:

# realtime/auth.py
import time
import jwt
from django.conf import settings


class RealtimeTokenService:
    """Issues short-lived JWTs for connecting to the realtime gateway."""

    def generate_connection_token(self, user) -> str:
        tenant_id = user.tenant_id

        # Channel grants are baked into the token, so the gateway authorizes
        # subscriptions with zero database lookups on connect.
        channels = [
            f"user_{user.id}",
            f"critical:user_{user.id}",
            f"tenant_{tenant_id}",
            f"critical:tenant_{tenant_id}",
        ]

        claims = {
            "sub": str(user.id),
            "exp": int(time.time()) + settings.REALTIME_TOKEN_TTL_SECONDS,
            "info": {"name": user.get_full_name(), "email": user.email},
            "channels": channels,
        }
        return jwt.encode(claims, settings.REALTIME_HMAC_SECRET, algorithm="HS256")

This is the key idea: token-as-permission. The channel names encode your authorization model. A user gets their own user_{id} channel and their tenant's tenant_{id} channel, and nothing else. Multi-tenant isolation falls out of the channel naming: a user simply has no grant for another tenant's channel, so the gateway will refuse the subscription.

Step 2: Hand the token out behind your normal auth

The endpoint that mints the token is an ordinary authenticated view. Your existing auth stack protects it; no ASGI gymnastics:

# realtime/views.py
class RealtimeTokenView(APIView):
    permission_classes = [IsAuthenticated]

    def post(self, request):
        token = RealtimeTokenService().generate_connection_token(request.user)
        return Response(
            {"token": token, "socket_url": settings.REALTIME_SOCKET_URL}
        )

The client takes {token, socket_url}, opens a WebSocket to the gateway, and subscribes to its channels. Django is now out of the connection entirely.

Step 3: Publish events over HTTP

To push an update, your backend POSTs to the gateway's publish API. No connection state, no sockets: just an HTTP call:

# realtime/publisher.py
import requests
from django.conf import settings


def push_event(channel, event_type, payload, *, recoverable=False, timeout=10.0):
    # The "critical:" namespace is configured on the gateway with message
    # history + recovery, so clients can catch up after a reconnect.
    final_channel = f"critical:{channel}" if recoverable else channel

    resp = requests.post(
        f"{settings.REALTIME_API_URL}/publish",
        json={
            "channel": final_channel,
            "data": {"type": event_type, "payload": payload},
        },
        headers={"X-API-Key": settings.REALTIME_API_KEY},
        timeout=timeout,
    )
    resp.raise_for_status()

Notice the critical: channel prefix. On the gateway, that namespace is configured to keep a short history so a client that briefly drops can recover missed messages on reconnect:

{
  "namespaces": [
    { "name": "critical", "history_size": 50, "history_ttl": "120s", "force_recovery": true }
  ]
}

Ephemeral events (a "user is typing" blip) publish to the bare channel; events you can't afford to lose (a completed order) publish to critical: and survive a reconnect.

Step 4: Make publishing reliable, and don't block the request

Publishing is a network call to a separate service. You do not want it on the request's critical path, and you want it to survive a transient blip. Wrap it in a Celery task with retries and backoff:

# realtime/tasks.py
import logging
from requests.exceptions import RequestException, HTTPError
from celery import shared_task
from .publisher import push_event

logger = logging.getLogger(__name__)


@shared_task(bind=True, max_retries=3, default_retry_delay=5, retry_backoff=True)
def publish_realtime_event(self, channel, event_type, payload, recoverable=False):
    try:
        push_event(channel, event_type, payload, recoverable=recoverable)
    except HTTPError as exc:
        # A 4xx error (bad API key, malformed payload) is a deterministic
        # failure that retries can't fix. Log it and drop it.
        # A 5xx error might be transient, so retry it.
        if 400 <= exc.response.status_code < 500:
            logger.error("Deterministic 4xx error on %s, dropping.", channel)
            return
        raise self.retry(exc=exc)
    except RequestException as exc:
        # Pure network timeouts or connection errors — retry with backoff.
        raise self.retry(exc=exc)
    except Exception:
        # A serialization bug won't fix itself on a retry. Log and drop it,
        # rather than burning three attempts on a guaranteed failure.
        # By not re-raising, Celery marks the task as successful and removes
        # it from the queue — no poison pill looping forever.
        logger.exception("Dropping unrecoverable realtime event on %s", channel)

That except split matters: retry the transient, drop the deterministic. A flaky network deserves another try; a payload that can't be JSON-encoded will fail identically three times in a row, so retrying it is just noise.

Firing an event from anywhere in your app is now a one-liner:

publish_realtime_event.delay(
    channel=f"tenant_{order.tenant_id}",
    event_type="order.completed",
    payload={"order_id": order.id, "total": str(order.total)},
    recoverable=True,
)

Why This Works

Your web tier goes back to being stateless. Django stays on WSGI. No ASGI migration, no async server, no sockets held in your workers. You can scale, restart, and deploy the web tier like any other stateless service. The open connections live on the gateway and aren't disturbed.

Connections scale independently. Holding a lot of idle connections is a specialized job. A purpose-built gateway does it on its own box, tuned for that, while your app servers stay sized for request throughput.

Auth has zero connect-time cost. Because the token carries its channel grants, the gateway never calls back into your database to authorize a subscription. Ten thousand clients reconnecting after a blip is ten thousand signature checks, not ten thousand DB queries.

Multi-tenancy is just naming. Tenant isolation isn't a separate access-control layer. It's the channel names in the token. There's no code path by which a user can subscribe to a channel they weren't granted.

Delivery is fire-and-forget but durable. Publishing happens off the request path via Celery, retries on transient failure, and the critical: namespace lets clients recover what they missed across a reconnect.

Design Decision: The Token Is a Cache, So It Can Go Stale

Baking permissions into the token is what buys you DB-free connects. But a token is a snapshot. If a user's access changes (you remove them from a tenant, say) and their token still has 24 hours to live, the gateway will keep honoring the old grants until it expires. You've traded revocation freshness for connect-time performance.

That's usually the right trade for a notification stream, but be deliberate about it:

Keep the TTL short and have the client transparently re-fetch a token. Shorter tokens = smaller stale window.
For true revocation, the gateway can be told to force-disconnect a user or refuse channels server-side. Reach for that only when a stream is sensitive enough to need it.

One more operational gotcha: a single gateway is tempting to share across environments, but tenant_42 in staging and tenant_42 in production are the same channel name: cross-talk waiting to happen. If you must share one broker, prefix channels with the environment (prod_tenant_42, staging_tenant_42) so the namespaces can't collide.

And to be honest about the path here: I shipped the Channels version first. It worked. But the friction of running an async server and bridging my existing auth into the ASGI scope is exactly what made the gateway approach worth it. If your real-time needs are deep and bidirectional (collaborative editing, presence), in-process Channels may genuinely fit better. For "push updates to subscribed clients," letting something else hold the sockets is the simpler system.

The Result

Adding a live update anywhere in the app is now one .delay() call. The web tier never learned what a WebSocket is. Connections, fan-out, history, and recovery are the gateway's problem; authorization is a signed list of channel names; delivery is a retrying background task. Django went back to doing the thing it's good at. And the real-time problem moved to a server built for it.

How are you doing real-time in your Django apps — in-process with Channels, or have you pushed connections out to a gateway? I'd love to hear what's worked for you in the comments.

¡Hasta luego!

How I Architected a Real-Time Gaming Backend Supporting 3,000+ Concurrent Users with WebSockets

Tue, 07 Apr 2026 11:30:41 +0000

Real-time systems are unforgiving. In a standard web application, a 500-millisecond delay is an inconvenience. In a real-time multiplayer gaming platform, 500 milliseconds is the difference between a fair outcome and a furious user demanding a refund.

Between October 2019 and December 2021, I served as Senior Software Engineer at Lordwin Group, where I led a team of five backend developers building dice.ng — a real-time gaming platform where thousands of users placed wagers, watched outcomes resolve live, and expected the entire experience to feel instantaneous. The platform also encompassed an investment management system and hotel booking service, but the gaming backend was the most technically demanding piece of infrastructure I have ever designed.

This article is a technical deep dive into how I architected that real-time gaming backend from the ground up — the WebSocket infrastructure, the event-driven architecture, the scaling strategy, and the hard lessons learned from operating a system where latency directly impacts revenue.

The Problem: Sub-100ms Latency for Thousands of Simultaneous Players

The requirements were aggressive from day one. Lordwin Group needed a gaming platform that could:

Support 3,000+ concurrent WebSocket connections during peak hours
Deliver game events (dice rolls, bet confirmations, payout calculations) with sub-100ms latency
Maintain absolute consistency in game state — in a wagering platform, a race condition is not a bug, it is a financial liability
Handle burst traffic patterns — user activity would spike dramatically around evening hours and weekends, sometimes tripling within minutes
Achieve 99.9% uptime — every minute of downtime was measurable lost revenue

The initial prototype used HTTP polling. Clients hit an API endpoint every two seconds to check for game state updates. At 500 concurrent users, this generated over 15,000 HTTP requests per minute, the database was drowning, and the "real-time" experience felt sluggish and broken. It was immediately clear that polling could not scale. We needed a fundamentally different architecture.

My Role: Technical Lead and System Architect

As the senior engineer leading a team of five, my responsibilities went beyond writing code. I made the core architecture decisions, designed the WebSocket infrastructure, established the event-driven messaging patterns, defined the horizontal scaling strategy, and mentored junior engineers on real-time system design.

Olamilekan Lamidi was not just a contributor on this project — I owned the technical direction. Every critical design decision described in this article was one I either made directly or guided the team toward after evaluating alternatives.

Technical Deep Dive: Building the Real-Time Infrastructure

1. WebSocket Server Architecture with Node.js

I chose Node.js as the runtime for the WebSocket server. Its event-driven, non-blocking I/O model is purpose-built for maintaining thousands of persistent connections with minimal resource overhead. PHP (our backend for the investment and hotel systems) was poorly suited for long-lived connections, so I designed the gaming layer as a separate Node.js service communicating with the main platform through Redis message channels.

The core WebSocket server was built on the ws library rather than Socket.IO. While Socket.IO offers convenience features like automatic reconnection and room management, it adds protocol overhead and abstractions that reduce control. For a latency-sensitive gaming platform, I needed raw WebSocket performance with custom protocol handling.

const WebSocket = require('ws');
const Redis = require('ioredis');
const { v4: uuidv4 } = require('uuid');

const wss = new WebSocket.Server({
  port: 8080,
  maxPayload: 1024 * 16,
  perMessageDeflate: false,
  clientTracking: true,
});

const connections = new Map();

wss.on('connection', (ws, req) => {
  const connectionId = uuidv4();
  const clientIp = req.headers['x-forwarded-for'] || req.socket.remoteAddress;

  connections.set(connectionId, {
    ws,
    userId: null,
    rooms: new Set(),
    connectedAt: Date.now(),
    lastHeartbeat: Date.now(),
  });

  ws.on('message', (raw) => {
    try {
      const message = JSON.parse(raw);
      routeMessage(connectionId, message);
    } catch (err) {
      ws.send(JSON.stringify({ type: 'error', code: 'INVALID_PAYLOAD' }));
    }
  });

  ws.on('close', () => {
    cleanupConnection(connectionId);
  });

  ws.on('pong', () => {
    const conn = connections.get(connectionId);
    if (conn) conn.lastHeartbeat = Date.now();
  });

  ws.send(JSON.stringify({
    type: 'connected',
    connectionId,
    serverTime: Date.now(),
  }));
});

I disabled perMessageDeflate deliberately. Compression adds CPU overhead per message, and since our payloads were small (typically under 500 bytes for game events), the bandwidth savings were negligible compared to the latency cost. This single configuration change reduced median message delivery time by 8ms across our benchmark tests.

2. Connection Management and Authentication

In a gaming platform handling real money, every WebSocket connection must be authenticated. I implemented a token-based authentication flow where clients first obtain a short-lived JWT via the REST API, then present it during WebSocket handshake.

const jwt = require('jsonwebtoken');

function routeMessage(connectionId, message) {
  const conn = connections.get(connectionId);
  if (!conn) return;

  switch (message.type) {
    case 'authenticate':
      handleAuthentication(connectionId, message.token);
      break;
    case 'join_game':
      if (!conn.userId) return sendError(conn.ws, 'NOT_AUTHENTICATED');
      handleJoinGame(connectionId, message.gameId);
      break;
    case 'place_bet':
      if (!conn.userId) return sendError(conn.ws, 'NOT_AUTHENTICATED');
      handlePlaceBet(connectionId, message);
      break;
    case 'ping':
      conn.ws.send(JSON.stringify({ type: 'pong', serverTime: Date.now() }));
      break;
    default:
      sendError(conn.ws, 'UNKNOWN_MESSAGE_TYPE');
  }
}

function handleAuthentication(connectionId, token) {
  const conn = connections.get(connectionId);
  try {
    const payload = jwt.verify(token, process.env.JWT_SECRET, {
      algorithms: ['HS256'],
      maxAge: '5m',
    });
    conn.userId = payload.userId;
    conn.ws.send(JSON.stringify({ type: 'authenticated', userId: payload.userId }));
    userConnectionIndex.set(payload.userId, connectionId);
  } catch (err) {
    conn.ws.send(JSON.stringify({ type: 'auth_failed', reason: 'INVALID_TOKEN' }));
    conn.ws.close(4001, 'Authentication failed');
  }
}

The JWT had a deliberately short expiry of five minutes. Since it was only used for the WebSocket handshake, there was no reason for it to live longer. Once authenticated, the connection was maintained through heartbeat pings rather than token refresh cycles.

3. Room-Based Game State Management

Each active game session functioned as a "room" — a logical grouping of WebSocket connections that should receive the same events. I implemented a lightweight room system without relying on external libraries:

const rooms = new Map();

function handleJoinGame(connectionId, gameId) {
  const conn = connections.get(connectionId);
  if (!conn) return;

  if (!rooms.has(gameId)) {
    rooms.set(gameId, new Set());
  }

  rooms.get(gameId).add(connectionId);
  conn.rooms.add(gameId);

  conn.ws.send(JSON.stringify({
    type: 'game_joined',
    gameId,
    players: rooms.get(gameId).size,
  }));

  broadcastToRoom(gameId, {
    type: 'player_joined',
    userId: conn.userId,
    playerCount: rooms.get(gameId).size,
  }, connectionId);
}

function broadcastToRoom(gameId, payload, excludeConnectionId = null) {
  const room = rooms.get(gameId);
  if (!room) return;

  const message = JSON.stringify(payload);
  for (const connId of room) {
    if (connId === excludeConnectionId) continue;
    const conn = connections.get(connId);
    if (conn && conn.ws.readyState === WebSocket.OPEN) {
      conn.ws.send(message);
    }
  }
}

One critical optimisation: I serialised the message payload once with JSON.stringify outside the loop, then sent the same string buffer to every client. For a room of 500 players, this avoided 499 redundant serialisation operations per broadcast — a saving that compounds rapidly under load.

4. Event-Driven Game Engine with Redis Pub/Sub

The game logic itself — dice rolling, bet resolution, payout calculation — ran in a separate process from the WebSocket server. This was a deliberate architectural decision. The WebSocket server's only responsibility was managing connections and delivering messages. Game logic, financial calculations, and database writes happened in dedicated worker processes.

Redis Pub/Sub was the communication backbone:

const publisher = new Redis(process.env.REDIS_URL);
const subscriber = new Redis(process.env.REDIS_URL);

subscriber.subscribe('game:events', 'game:results', 'system:announcements');

subscriber.on('message', (channel, message) => {
  const event = JSON.parse(message);

  switch (channel) {
    case 'game:events':
      broadcastToRoom(event.gameId, {
        type: 'game_event',
        event: event.eventType,
        data: event.payload,
        timestamp: event.timestamp,
      });
      break;

    case 'game:results':
      handleGameResult(event);
      break;

    case 'system:announcements':
      broadcastToAll({
        type: 'system_announcement',
        message: event.message,
      });
      break;
  }
});

async function handlePlaceBet(connectionId, message) {
  const conn = connections.get(connectionId);

  const betEvent = {
    gameId: message.gameId,
    userId: conn.userId,
    amount: message.amount,
    selection: message.selection,
    timestamp: Date.now(),
    idempotencyKey: message.idempotencyKey,
  };

  await publisher.publish('bets:incoming', JSON.stringify(betEvent));

  conn.ws.send(JSON.stringify({
    type: 'bet_acknowledged',
    idempotencyKey: message.idempotencyKey,
    status: 'processing',
  }));
}

This decoupling had three critical benefits:

Fault isolation: If the game engine crashed, WebSocket connections remained alive. Users saw a brief pause rather than a disconnection.
Independent scaling: I could run multiple game engine workers to process bets in parallel without changing the WebSocket layer.
Auditability: Every game event flowed through Redis, creating a natural event stream that I logged for regulatory compliance and dispute resolution.

5. Horizontal Scaling Across Multiple WebSocket Servers

A single Node.js process can handle approximately 10,000 concurrent WebSocket connections before memory and CPU become constraints. But scaling WebSockets horizontally introduces a problem that REST APIs do not have: connection affinity. If Player A is connected to Server 1 and Player B to Server 2, a room broadcast must reach both servers.

I solved this with Redis as the cross-server message bus:

const serverId = process.env.SERVER_ID || uuidv4();

subscriber.subscribe('ws:broadcast');

subscriber.on('message', (channel, raw) => {
  if (channel !== 'ws:broadcast') return;
  const message = JSON.parse(raw);

  if (message.originServer === serverId) return;

  if (message.targetRoom) {
    broadcastToRoom(message.targetRoom, message.payload);
  } else if (message.targetUser) {
    sendToUser(message.targetUser, message.payload);
  }
});

function clusterBroadcastToRoom(gameId, payload) {
  broadcastToRoom(gameId, payload);

  publisher.publish('ws:broadcast', JSON.stringify({
    originServer: serverId,
    targetRoom: gameId,
    payload,
  }));
}

Each WebSocket server instance subscribed to a shared Redis channel. When a game event needed to reach all players in a room, the originating server broadcast to its local connections and simultaneously published to Redis, where other servers picked up the message and relayed it to their local connections.

The originServer check prevented message loops — without it, a server would re-broadcast messages it had already delivered locally.

6. Heartbeat Monitoring and Connection Cleanup

Stale connections are a silent performance killer in WebSocket systems. Users close browser tabs, lose internet connectivity, or let their phones go to sleep. Without proactive cleanup, the server accumulates zombie connections that consume memory and distort room player counts.

I implemented a heartbeat interval that pinged every client every 30 seconds:

const HEARTBEAT_INTERVAL = 30000;
const HEARTBEAT_TIMEOUT = 45000;

setInterval(() => {
  const now = Date.now();
  for (const [connectionId, conn] of connections) {
    if (now - conn.lastHeartbeat > HEARTBEAT_TIMEOUT) {
      conn.ws.terminate();
      cleanupConnection(connectionId);
      continue;
    }
    if (conn.ws.readyState === WebSocket.OPEN) {
      conn.ws.ping();
    }
  }
}, HEARTBEAT_INTERVAL);

function cleanupConnection(connectionId) {
  const conn = connections.get(connectionId);
  if (!conn) return;

  for (const gameId of conn.rooms) {
    const room = rooms.get(gameId);
    if (room) {
      room.delete(connectionId);
      if (room.size === 0) {
        rooms.delete(gameId);
      } else {
        broadcastToRoom(gameId, {
          type: 'player_left',
          userId: conn.userId,
          playerCount: room.size,
        });
      }
    }
  }

  if (conn.userId) {
    userConnectionIndex.delete(conn.userId);
  }

  connections.delete(connectionId);
}

This heartbeat mechanism maintained accurate player counts and ensured we never wasted resources on dead connections — critical when operating at 3,000+ concurrent users where even small inefficiencies compound.

Ensuring Game Integrity: The Financial Safety Layer

In a platform where real money is at stake, game outcome integrity is non-negotiable. I designed several safeguards:

Idempotent Bet Processing: Every bet carried a client-generated idempotency key. If a network hiccup caused a duplicate submission, the game engine would recognise the duplicate key and return the original response rather than processing the bet twice.

Atomic Balance Operations: All balance changes used Redis WATCH/MULTI/EXEC transactions for in-memory operations and database-level row locking for persistence, ensuring no user could ever bet more than their balance — even under concurrent requests.

async function processBalanceDeduction(userId, amount, idempotencyKey) {
  const lockKey = `lock:balance:${userId}`;
  const lock = await acquireLock(lockKey, 5000);

  try {
    const processed = await redis.get(`idempotency:${idempotencyKey}`);
    if (processed) return JSON.parse(processed);

    const balance = parseFloat(await redis.get(`balance:${userId}`));
    if (balance < amount) {
      return { success: false, reason: 'INSUFFICIENT_BALANCE' };
    }

    const newBalance = balance - amount;
    await redis.set(`balance:${userId}`, newBalance.toString());

    const result = { success: true, newBalance, deducted: amount };
    await redis.setex(`idempotency:${idempotencyKey}`, 3600, JSON.stringify(result));

    return result;
  } finally {
    await releaseLock(lock);
  }
}

Provably Fair Outcomes: Game results were generated using a combination of server seed and client seed, hashed together. Users could verify after each round that the outcome was not manipulated. This was both a regulatory requirement and a trust-building feature.

Monitoring and Observability in Production

Operating a real-time gaming system requires comprehensive observability. I built custom monitoring dashboards tracking:

Active connections per server instance — to trigger auto-scaling
Message throughput — messages sent per second across all rooms
Event latency — time from game engine event emission to client delivery
Redis pub/sub lag — early warning for message bus saturation
Game round completion times — anomaly detection for stuck game sessions

I instrumented the WebSocket server with Prometheus metrics:

const client = require('prom-client');

const activeConnections = new client.Gauge({
  name: 'ws_active_connections',
  help: 'Number of active WebSocket connections',
});

const messageLatency = new client.Histogram({
  name: 'ws_message_latency_ms',
  help: 'Message delivery latency in milliseconds',
  buckets: [5, 10, 25, 50, 100, 250, 500],
});

const messagesPerSecond = new client.Counter({
  name: 'ws_messages_total',
  help: 'Total WebSocket messages sent',
  labelNames: ['type'],
});

These metrics fed into Grafana dashboards that gave me real-time visibility into system health. On several occasions, a spike in message latency gave us a 10-minute early warning before a Redis memory issue would have caused visible user impact.

The Results: Production Metrics

After three months of iterative development and load testing, Olamilekan Lamidi and the engineering team shipped the gaming backend into production. The measurable outcomes exceeded our original targets:

Metric	Target	Achieved
Concurrent WebSocket connections	3,000	3,200+ sustained peak
Event delivery latency (p50)	<100ms	32ms
Event delivery latency (p99)	<200ms	87ms
System uptime (monthly)	99.5%	99.9%
Bet processing throughput	500/min	850+/min
Connection recovery after deploy	<5s	~2.5s average

The sub-50ms median latency was particularly significant. Users experienced game events as truly instantaneous — dice rolls resolved, payouts appeared, and leaderboards updated in what felt like real time. This directly correlated with user engagement metrics: average session duration increased by 35% compared to the earlier HTTP polling prototype.

Lessons Learned

Building this system taught me principles that I have carried into every project since:

Separate connection management from business logic. The WebSocket server should be a dumb pipe. All intelligence belongs in backend workers communicating through message queues.
Design for graceful degradation. When the game engine was temporarily slow, the WebSocket layer continued serving heartbeats and acknowledged messages. Users experienced a delay, not a crash.
Serialise once, send many. In broadcast-heavy systems, the cost of JSON serialisation dwarfs the cost of network transmission. Serialise the payload once and reuse the buffer.
Monitor latency percentiles, not averages. A 30ms average means nothing if 5% of your users experience 500ms delays. The p99 is what your angriest users feel.
Test with realistic load patterns. Our load tests simulated bursty evening traffic with sudden spikes, not steady-state throughput. Production traffic is never smooth.

Final Thoughts

Architecting a real-time gaming backend was the most technically challenging and rewarding project of my career up to that point. It demanded a deep understanding of network protocols, distributed systems, concurrent programming, and financial transaction safety — all operating under latency constraints that left no room for architectural shortcuts.

The patterns I developed at Lordwin Group — event-driven architecture, Redis-backed horizontal scaling, idempotent financial operations — became foundational to how I approach every real-time system I have built since. At VacancySoft, I applied similar event-driven patterns to handle 50,000+ daily API requests. At 2am Tech, the idempotent transaction processing pattern directly informed the financial workflows I built for the Addio platform.

Real-time systems are hard. But the engineering discipline they dem