← Back to the shelf
Appearance

Dispatches / a short note

concept

Websockets 101

From bidirectional echoserver to building a robust multi-player game.

omg claude (I’ll share the link once it’s ready) https://claude.ai/share/ede94c96-26f1-4c1e-872d-74ce5c09364f. if you can’t see it for somereason, here’s a breakdown:

Phase 1 — WebSocket Foundations (~5 hrs)

1.1 How WebSockets work under the hood (1.5 hrs)

WebSocket is a full-duplex, persistent TCP connection initiated via an HTTP/1.1 Upgrade handshake. Understand the 101 Switching Protocols response, the Sec-WebSocket-Key/Accept handshake, and how frames are structured (FIN, RSV, opcode, MASK, payload length, masking key).

Tags: core protocol

Footgun: WebSocket has nothing to do with HTTP after the handshake. CORS does NOT apply — the server must validate the Origin header manually.

Resources:

1.2 Browser WebSocket API (1 hr)

new WebSocket(url), readyState, onopen/onmessage/onerror/onclose events, send(), close(code, reason). Sending both text and binary (ArrayBuffer / Blob). Inspecting frames in DevTools Network tab.

Tags: browser

Tip: Always set binaryType = 'arraybuffer' if you’re sending binary — the default 'blob' is awkward to work with in game loops.

Resources:

1.3 Opcodes, frames & control frames (1 hr)

Text (0x1) vs Binary (0x2) vs Continuation (0x0) frames. Control frames: Close (0x8), Ping (0x9), Pong (0xA). Fragmentation — splitting large messages across multiple frames. Close codes: 1000 normal, 1001 going away, 1006 abnormal, 1011 server error.

Tags: core protocol

Footgun: Client frames MUST be masked. Server frames must NOT be masked. Get this wrong and the peer drops the connection immediately.

Resources:

1.4 ws vs wss — TLS in production (0.5 hrs)

wss:// is WebSocket over TLS. Always use wss in production. Your load balancer terminates TLS and forwards plain ws:// internally. Browsers block mixed content (ws:// on an https:// page).

Tags: security

Footgun: Proxies that don’t understand WebSocket buffer the connection and break it. You MUST configure them to handle the Upgrade header and disable request buffering.

Resources:

1.5 Ping / Pong heartbeats (1 hr)

TCP connections silently die without data flow. The server should send Ping frames periodically (~30s) and disconnect clients that don’t Pong back within a timeout. Client-side application-level heartbeats are also common in game servers.

Tags: reliability

Footgun: NATs, firewalls, and mobile networks silently kill idle connections. Without heartbeats you’ll have thousands of zombie connections that appear open server-side but are dead.

Resources:

Phase 2 — Server-Side Implementation (~6 hrs)

2.1 Choosing a WebSocket library (1 hr)

Node.js: ws (bare, fast), socket.io (batteries included), uWebSockets.js (C++ backed, highest perf). Go: Gorilla WebSocket, nhooyr/websocket. Rust: tokio-tungstenite. For a game server, prefer raw ws or uWebSockets.js.

Tags: Node.js, Go, Rust

Resources:

2.2 Connection lifecycle management (1 hr)

Tracking connected clients, associating metadata (user ID, room, session) with each connection object. Clean teardown on close — removing from maps, notifying other players. Handling the error event (which precedes close).

Tags: server

Footgun: If you store client state in a Map and forget to delete on close/error, you have a memory leak that grows forever. Always clean up in BOTH onclose and onerror.

Resources:

2.3 Room / channel architecture (2 hrs)

Model rooms as Sets of WebSocket connections keyed by room ID. Broadcast to a room by iterating the Set and sending to each client (checking readyState === OPEN). Design a message protocol with a type field, e.g. { type: 'PLAYER_MOVE', payload: {...} }.

Tags: architecture

Tip: Use a discriminated union message protocol: { type: 'PLAYER_MOVE', payload: {...} }. This scales cleanly and lets you add message types without breaking existing handlers.

Resources:

2.4 Backpressure handling (1 hr)

If your server sends data faster than a slow client can consume, the send buffer fills up. Check ws.bufferedAmount before sending, or handle the drain event. uWebSockets.js has explicit backpressure callbacks.

Tags: reliability

Footgun: Ignoring backpressure in a game server means slow clients get their receive buffer filled, causing lag and eventual disconnection.

Resources:

2.5 Message serialization strategies (1 hr)

JSON is easy but slow for high-frequency game messages. MessagePack is JSON-equivalent but binary and 2–3x smaller. FlatBuffers/Protobuf require schema compilation but are 5–10x faster. Benchmark early — serialization is often the bottleneck.

Tags: performance

Resources:

Phase 3 — Footguns & Security (~5 hrs)

3.1 Origin validation (0.5 hrs)

WebSocket connections from any origin succeed by default — there is no CORS. Any website can connect to your ws:// server. Always validate the Origin header during the handshake.

Tags: security ⚠️

Footgun: This is the #1 most missed WebSocket security issue. A malicious site can CSRF your WebSocket endpoint unless you validate Origin.

Resources:

3.2 Authentication over WebSocket (1 hr)

Cookies are sent on the initial HTTP Upgrade request, so cookie-based auth works. For token auth: send the JWT in the first message after connect (not as a query param — those appear in server logs). Validate auth before sending any game state.

Tags: security ⚠️

Footgun: Never trust anything from the client. Re-validate permissions on every sensitive action (joining rooms, game moves), not just on connect.

Resources:

3.3 Rate limiting & DoS protection (1 hr)

Limit connection rate per IP, max connections per user, and max message rate per connection. Track message counts in a sliding window. Disconnect violators. Set max message size (ws: maxPayload option).

Tags: security ⚠️, reliability

Footgun: A single client can open hundreds of connections or flood your server with millions of tiny messages. Without rate limiting, your server is trivially DoS-able.

Resources:

3.4 Input validation & message schema (1 hr)

Every message from a client is untrusted user input. Validate message type, payload shape, and field ranges before acting. Use zod or ajv on the server. Reject and log malformed messages — don’t crash.

Tags: security ⚠️

Tip: Treat WebSocket messages exactly like HTTP request bodies: validate every field, sanitize strings, check numeric ranges, and never trust client-sent identifiers without server-side verification.

Resources:

3.5 SSL termination & proxy pitfalls (1.5 hrs)

nginx must set proxy_http_version 1.1, proxy_set_header Upgrade, and proxy_set_header Connection 'upgrade'. Disable proxy_buffering and raise proxy_read_timeout above 60s. HAProxy needs correct tunnel mode on the WS backend.

Tags: ops

Footgun: proxy_read_timeout 60s is nginx’s default — it silently closes WebSocket connections after 60 seconds of no data. Set to 3600s or higher in production.

Resources:

Phase 4 — Reconnection Strategies (~4 hrs)

4.1 Exponential backoff with jitter (1 hr)

On disconnect, wait before reconnecting. Start at 1s, double each failure, cap at ~30s. Add ±20% random jitter to prevent thundering herd — thousands of clients don’t all reconnect at the same instant after a server restart.

Formula: delay = min(cap, base × 2^attempt) × (0.8 + random() × 0.4)

Tags: reliability

Tip: Implement in a useWebSocket hook or ReconnectingWebSocket wrapper class so every feature doesn’t re-implement it.

Resources:

4.2 Client-side message queue during disconnect (1 hr)

Buffer messages that arrive while disconnected. On reconnect, flush the queue or send a full state sync. Cap queue size to prevent OOM. For game state, a full state sync snapshot is usually simpler and more correct than replaying all buffered actions.

Tags: reliability

Footgun: Unbounded queues during long disconnects can OOM the browser tab. Cap queue size and drop or summarize old messages.

Resources:

4.3 Session resumption & state recovery (2 hrs)

Issue a short-lived reconnect token on connect. On reconnect: client sends token → server restores session (room, game state, player position). Design a ‘catch-up’ snapshot mechanism to send current world state to a reconnecting player.

Tags: architecture

Tip: Design reconnection and state recovery from day 1. A player who drops for 5 seconds should rejoin seamlessly — not restart from a lobby.

Resources:

Phase 5 — Scaling WebSockets (Stateless Architecture) (~10 hrs)

5.1 The statefulness problem (1 hr)

A WebSocket connection lives on exactly one server process. When you scale horizontally to N servers, a message for “room 42” must reach all connections in that room — which may span all N servers. This is the core scaling challenge.

Tags: core concept

Footgun: Naive horizontal scaling silently breaks broadcasting — messages only reach players connected to the same server instance. This is almost always the first bug in a multiplayer implementation.

Resources:

5.2 Pub/Sub fan-out with Redis (3 hrs)

Each server subscribes to a Redis channel per room. When server A receives a message for room 42, it publishes to Redis channel room:42. All servers receive it and forward to their local connections. Use Redis Pub/Sub or Redis Streams.

Tags: scaling, Redis

Tip: This is the industry-standard approach. socket.io has @socket.io/redis-adapter. With raw ws, implement it yourself — it’s ~50 lines with ioredis.

Resources:

5.3 NATS & Kafka as alternatives (1.5 hrs)

NATS is purpose-built for high-throughput messaging with lower latency than Redis Pub/Sub — ideal for real-time game state. Kafka adds durability and replay but adds latency; better for matchmaking queues and event sourcing than live game state.

Tags: scaling

Resources:

5.4 Sticky sessions (and why to avoid them) (0.5 hrs)

Sticky sessions pin a client to one server via IP hash or cookie. This works but distributes load unevenly, makes deployments painful, and doesn’t solve the fan-out problem. Use pub/sub instead. Sticky sessions are a crutch, not a solution.

Tags: ⚠️ anti-pattern

Resources:

5.5 Dedicated WebSocket gateway pattern (2 hrs)

Separate your connection-holding layer (WS gateway) from your business logic layer. The gateway only handles connections + pub/sub routing. Business logic runs in stateless HTTP microservices. This is the pattern used at scale by Slack, Discord, etc.

Tags: architecture

Tip: This lets you scale compute-heavy game logic independently from the connection-holding layer.

Resources:

5.6 WebSocket hibernation — Cloudflare Durable Objects (1 hr)

Cloudflare Durable Objects can hibernate WebSocket connections to zero CPU when idle — the connection stays open but the process is evicted. On the next message, the process restores in microseconds. Compelling for game lobbies and presence.

Tags: serverless

Resources:

5.7 Load balancer configuration (1 hr)

AWS ALB: natively supports WebSockets. AWS NLB: lower latency, better for long-lived connections. nginx: proxy_http_version 1.1, disable buffering, set long timeouts. HAProxy: balance by source IP only if needed; prefer pub/sub instead.

Tags: ops

Resources:

Phase 6 — Production & Observability (~5 hrs)

6.1 Metrics to track (1 hr)

Active connections, connection rate (new/sec), message throughput (in/out per sec), message size distribution, error rate, close code distribution (watch for 1006 = abnormal), round-trip ping latency, backpressure events, reconnect rate.

Tags: observability

Resources:

6.2 Graceful shutdown (1 hr)

On SIGTERM: stop accepting new connections, send Close 1001 (going away) to all clients, wait for acknowledgment, drain, then exit. Coordinate with your load balancer to drain before killing. Gives reconnect logic time to find a new server.

Tags: ops

Footgun: Hard-killing a WS server sends TCP RST packets — clients see abnormal close (1006). A graceful close lets your backoff reconnect logic work properly.

Resources:

6.3 Connection limits & resource management (1 hr)

Each WS connection uses a file descriptor. Raise ulimit to 100k+ in /etc/security/limits.conf and tune net.core.somaxconn. Each Node.js connection uses ~50–100KB RAM. uWebSockets.js can hold 1M+ connections per GB; ws is closer to 10–50k.

Tags: ops

Tip: uWebSockets.js can hold 1M+ connections in ~1GB RAM. Node.js ws is closer to 10k–50k per GB. Design your instance sizing accordingly.

Resources:

6.4 Structured logging (0.5 hrs)

Log: connect (userId, IP, userAgent), disconnect (userId, code, reason, duration), auth failures, rate limit hits. Use structured JSON logs with correlation IDs. Don’t log every message in production — you’ll drown in noise.

Tags: observability

Resources:

6.5 Testing WebSocket servers (1.5 hrs)

Unit test message handlers with mock WS objects. Integration test with a real ws.WebSocket client in Jest/Vitest. Load test with k6 (native WS support). Chaos test: kill servers mid-game, simulate slow clients, test reconnect recovery.

Tags: testing

Resources:

Phase 7 — Multiplayer Game Architecture (~8 hrs)

7.1 Authoritative server model (2 hrs)

The server is the single source of truth for game state. Clients send intents (MOVE_LEFT, FIRE), never positions. The server validates, computes new state, and broadcasts. Prevents cheating — a client cannot lie about its position.

Tags: game architecture

Tip: This is non-negotiable for a competitive game. Client-authoritative games are trivially hackable.

Resources:

7.2 Client-side prediction & server reconciliation (2 hrs)

Apply the player’s own input immediately (prediction) so movement feels responsive. Server processes input and sends authoritative state. If client and server diverge, the client snaps to server state (reconciliation). Hides network latency from the player.

Tags: game architecture

Resources:

7.3 Entity interpolation & lag compensation (1.5 hrs)

For OTHER players, interpolate between the two most recent known positions (adds ~100ms delay but removes jitter). For hit detection, lag compensation rolls back world state to what the shooter saw when they fired.

Tags: game architecture

Resources:

7.4 Delta compression for state updates (1 hr)

Instead of broadcasting full world state every tick, send only what changed since the last acknowledged state. Track per-client what they’ve seen (sequence numbers). Can reduce bandwidth by 80–95% in a large game world.

Tags: performance

Resources:

7.5 Tick rate & game loop design (1.5 hrs)

Server-side game loop runs at a fixed tick rate (20Hz–64Hz). Each tick: collect inputs, advance simulation, broadcast delta. Decoupled from client framerate. Higher tick rate = better responsiveness but more CPU and bandwidth. Start at 20Hz, tune up.

Tags: game architecture

Resources:

Phase 8 — Matchmaking System Design (~5 hrs)

8.1 Matchmaking queue architecture (2 hrs)

Players enter a queue (Redis sorted set by ELO + timestamp). A matchmaker process polls the queue and groups compatible players. When a match is found: create a game session, notify players via WebSocket, move them to a game server. Dequeue on disconnect.

Tags: matchmaking

Resources:

8.2 ELO & skill-based matching (1 hr)

ELO for skill tracking. Expand match criteria over time: first try ±50 ELO, after 30s try ±200, after 2min try ±500. This ‘expanding window’ balances match quality vs wait time.

Tags: matchmaking

Resources:

8.3 Game session lifecycle (1 hr)

States: WAITING → IN_PROGRESS → FINISHED. Store in Redis with TTL. Each state transition sends WebSocket events. Handle player abandonment (forfeit vs reconnect window). Post-game: update ELO, persist match history, send results.

Tags: matchmaking

Resources:

8.4 Presence & lobby system (1 hr)

Track online players in Redis with a TTL key refreshed by heartbeats. Expose a lobby list via WebSocket subscription. Players see who’s online, in queue, or in a game. Use Redis keyspace notifications or a dedicated heartbeat channel.

Tags: matchmaking

Resources:

Phase 9 — Production Checklist & Best Practices (~2 hrs)

9.1 Security checklist (0.5 hrs)

  • [ ] Always wss:// in production
  • [ ] Validate Origin header on every connection
  • [ ] Authenticate before sending any game state
  • [ ] Set maxPayload to prevent OOM from large messages
  • [ ] Rate limit connections per IP and messages per connection
  • [ ] Validate and sanitize every message payload (zod / ajv)

Resources:

9.2 Reliability checklist (0.5 hrs)

  • [ ] Ping/pong heartbeats with disconnect timeout
  • [ ] Exponential backoff with jitter on reconnect
  • [ ] Clean up all listeners and Maps on disconnect
  • [ ] Graceful shutdown with Close 1001
  • [ ] Session resumption tokens issued on connect
  • [ ] Message queue with cap during disconnect

Resources:

9.3 Scaling checklist (0.5 hrs)

  • [ ] Pub/Sub via Redis or NATS for horizontal scaling — no sticky sessions
  • [ ] WS gateway separated from business logic services
  • [ ] Load test with k6 before launch
  • [ ] Monitor connection count, close code distribution, message throughput
  • [ ] Raise ulimit (file descriptors) on all servers
  • [ ] Disable proxy_buffering and raise proxy_read_timeout in nginx

Resources:

9.4 The essential reading list (0.5 hrs)

Three resources worth reading cover-to-cover — everything else is reference material:

  1. High Performance Browser Networking ch.17 — free, authoritative, covers the protocol deeply
  2. Gaffer on Games — the multiplayer game networking bible; read the whole blog
  3. Gabriel Gambetta: Fast-Paced Multiplayer (4 parts) — the clearest explanation of prediction, reconciliation, and interpolation on the internet

websockets 101, I’m learning and trying them out here in https://github.com/nijuyonkadesu/79-plus/tree/main/chat (no where near completion, but really interesting)