Two players, one clock, zero trust: building a distributed chess server on the BEAM

This is part 1 of 2: the decisions and the architecture. Part 2, the war stories, is coming soon.

On 22 August at 20:00 CET, I dropped the gate on playchessm8.com and sat there refreshing the dashboard like it owed me money. Soon strangers were playing rated chess on my server. No signup, no email verification, no cookie wall. One tap and you are in a real-time game against another human somewhere on Earth.

This is the story of everything that one tap sets in motion, which is really a distributed systems story wearing a chess costume: single-writer ownership, monotonic clocks, idempotent writes, snapshots, supervision. It is also a confession. I built the server in a language I could barely read, it spent weeks beating my Go instincts out of me, and I would do it again.

Chess looks like a toy problem. It is not.

Two players, one board, alternating turns. When I started, I honestly thought the data model would fit on a napkin.

Then I added clocks, and suddenly 300 milliseconds of lag decided who wins. Then disconnects, because phones go through tunnels. Then rage quits, refreshes mid-game, rematch requests racing each other, ratings that must never be applied twice, and a server that needs to restart while a thousand games are in flight.

Somewhere down that list it dawned on me what I was really building: a tiny stateful world with a hard deadline and two participants who each believe they are right. In other words, a distributed system.

Every clock you see here is a projection of the server’s clock. The browser is not trusted with time, for reasons this post will make clear.

Before writing any code I wrote down the rules I refused to break:

1. The server has the final word. Every move, clock tick, and result is decided in one place. Anyone can open devtools; nobody gets to checkmate you from the console.
2. Time is measured where it is authoritative. Never trust a clock you do not own.
3. A crash must not eat a game. Deploys and hardware failures happen. Games in progress survive them.
4. A result is recorded exactly once. Your rating does not get to be approximately correct.
5. The whole thing runs on a hobby budget. More on that at the end, including the exact number.

Why Elixir, and what the BEAM did to me

Here is the honest part. Go is my daily language and would have been the comfortable choice. I picked Elixir, a language I could barely read, and spent the first weeks losing arguments with it.

What pushed me was the shape of the problem. A chess site is thousands of tiny, independent, stateful worlds. Each game needs its own memory, its own timeline, and its own failure domain: if one game hits a bug, the other 999 must not notice. That calls for a lightweight process per game, with no shared memory, each able to crash and restart alone. That is the actor model, and the BEAM, the virtual machine under Erlang and Elixir, has been running it in telecom switches since before I could castle.

In Go I would have built all of that myself, out of goroutines, channels, mutexes, and discipline. Robert Virding, one of Erlang’s creators, has a rule about this: any sufficiently complicated concurrent program in another language contains an ad hoc, informally specified, bug-ridden, slow implementation of half of Erlang. I took the warning and skipped the middle step.

Skipping it was not free. Elixir spent weeks dismantling my habits. Nothing mutates: you do not update the board, you build a new one and hand it forward, and my fingers kept reaching for a variable to change. There are no loops; I rewrote the same recursion three times before it stopped feeling like a trick. Pattern matching replaced my if/else trees, read like hieroglyphics on day one, and is now the feature I miss most in every other language. And I earned the classic beginner scar: I made a GenServer call itself and deadlocked it, then stared at the timeout as if the runtime had betrayed me. The runtime was fine. The mental model was mine to fix: not threads sharing memory, but little mailboxes sending letters.

To see what that did to my code, here is the same operation, applying a move, written with each instinct. In Go I would reach for a struct, a mutex, and an if:

func (g *Game) Move(color Color, uci string) error {
	g.mu.Lock()
	defer g.mu.Unlock()
	if g.turn != color {
		return ErrNotYourTurn
	}
	g.board.Apply(uci) // mutate in place
	g.turn = g.turn.Other()
	return nil
}

In Elixir, the same logic is function heads on a GenServer, and there is nothing to lock:

def handle_call({:move, color, _uci}, _from, %{turn: turn} = state) when color != turn do
  {:reply, {:error, :not_your_turn}, state}
end

def handle_call({:move, color, uci}, _from, state) do
  {:reply, :ok, state |> apply_move(color, uci) |> checkpoint()}
end

“Not your turn” is not an if inside the function; it is a clause that matches and answers before the happy path is even considered. The production handler is a stack of these, one per way a move can be wrong: game already over, opponent not seated yet, not your turn. The board is never mutated; apply_move returns a new state and checkpoint snapshots it. And there is no mutex, because there is nothing to protect: the mailbox already serializes every caller.

It was worth it. Every live game on chessm8 is a GenServer: one Elixir process holding the board, the clocks, the seats, and any pending draw or rematch offers. A Registry maps game ids to processes; a DynamicSupervisor restarts anything that dies; the preemptive scheduler means one pathological game cannot starve the rest.

That mailbox is the property that quietly deletes a whole class of bugs, and distributed systems people call it the single-writer principle: exactly one owner per piece of state, every write through its mailbox, in order. Concurrency across games is free. Consistency within a game is free. I did not write a single lock in this codebase.

The architecture

The whole system. Players reach the edge network, the static web app comes from a CDN, and everything that matters happens in one realtime core with Postgres as the ledger and Redis for matchmaking and snapshots.

I kept the front end deliberately dumb: Next.js on Vercel, static, cached at the edge. It runs a chess library locally so legal-move hints feel instant, but everything it shows you is a projection of what the server said. An untrusted client gets to predict; it never gets to decide.

The machine that decides is one Fly.io box in Frankfurt running my Elixir umbrella app, with Phoenix Channels handling the WebSockets. Behind it, I gave every game’s state three homes, each with one job:

• Hot: the game process. Board, clocks, seats, offers, all in process memory. Every rule is enforced here, at memory speed.
• Warm: a Redis snapshot. After each move, the process serializes its state into a compact versioned binary and writes it to Redis, so a restart can resurrect live games. Redis also runs the matchmaking queues.
• Cold: the Postgres ledger. Finished games, every move with its timing (anti-cheat fuel for later), and ratings. The durable record of everything that must survive forever.

Fast state where the rules run, a safety net for restarts, and a permanent ledger. Nearly every design decision I describe in this post is this split, applied again and again. Boot for a game process starts with the question the split makes possible:

def init(game_id) do
  state =
    case Snapshot.load(game_id) do
      {:ok, snapshot} -> restore(snapshot)  # a previous life, continued
      :error -> Game.new(game_id)           # a fresh board
    end

  {:ok, state}
end

I tested the safety net the only convincing way: started a real game on production, killed the virtual machine, and watched. It came back, both clients reconnected, the board and both clocks restored to the exact position, and the game continued after a two-second hiccup. Erlang people say “let it crash”, and this is what they mean: not recklessness, but a system where a crash is a boring, recoverable event. It also happens to describe my opening repertoire.

The full picture lives on the public architecture page.

Time is a liar

Clocks were the part I underestimated most, which is embarrassing, because the chess clock is the oldest distributed systems problem in the building: two parties, one shared resource, and a hard deadline both of them resent.

The first lesson: never trust the client’s clock. Wall clocks lie. NTP nudges them, virtual machines pause them, laptops close their lids mid-game. I charge your clock on the server with the BEAM’s monotonic time, which only moves forward, from the moment your turn starts to the moment your move arrives. Charging a clock is a subtraction that cannot be fooled:

elapsed = System.monotonic_time(:millisecond) - state.turn_started_at
remaining = state.clocks[color] - elapsed

The second lesson: the client never counts down on its own authority. Every move broadcast carries the remaining time plus a timestamp, and the display derives the ticking from timestamp deltas. Browsers throttle timers in background tabs, so a self-counting clock would gift you time that never existed; with timestamp math, a throttled tab snaps back to the truth the instant you return.

The third lesson arrived the hard way, in review: park the clock until the game actually starts. Early on, the first player could move while their opponent was still connecting, banking thinking time against an empty chair. Now my server rejects moves until both seats are taken. The real guard is one pattern match:

def handle_call({:move, _color, _uci}, _from, %{turn_started_at: nil} = state) do
  {:reply, {:error, :waiting_for_opponent}, state}
end

Exactly once is also a liar

There are only two hard problems in distributed systems:

2. exactly once delivery 1. guaranteed message order 2. exactly once delivery

When a game ends, the game process notifies a sink that writes the result to Postgres and applies the rating change. The notification can fail, so it retries. Retries mean the same result can arrive twice. And two games can finish in the same instant, which means two writers on one rating row.

I made the fix boring on purpose: retries give at-least-once delivery, an existence check makes the write idempotent, and a row lock serializes rating updates. My whole strategy fits in one transaction:

Repo.transaction(fn ->
  users = Repo.all(from(u in User, where: u.id in ^player_ids, order_by: u.id, lock: "FOR UPDATE"))

  unless Repo.exists?(from(g in Game, where: g.id == ^game_id)) do
    persist_result_and_ratings!(game, users)
  end
end)

The order_by is quietly load-bearing: both players are always locked in the same order, so two games finishing at once cannot deadlock each other. At-least-once plus idempotent writes gives the effect of exactly once, which is the closest anyone honest ever gets.

The rating math is Glicko-2, which models not just your strength but how sure the system is about it. That uncertainty is why your first wins swing by 150 points while the system figures out whether you are secretly Magnus.

Scaling, honestly

Here is where I am supposed to show you the Kubernetes diagram. There isn’t one. chessm8 runs on exactly one machine, and that is a decision, not an accident.

A game lives inside one process on one node. Run two nodes without routing players to the right one, and a game can grow two brains, each convinced it owns the truth. Distributed systems people call that split brain; in chess it means two kings, and the rules are extremely clear about that. Distributed Erlang with sticky routing by game id is the well-trodden path, and the codebase is shaped for it. But clustering before you need it is paying the complexity tax before you have the income, and one BEAM node comfortably handles the community that would make this project a wild success. So my plan is deliberately boring: measure, scale up, and cluster only when sustained load demands it. Until then, the CAP theorem and I have an understanding: no partitions, no partition tolerance required, considerably more sleep.

The scaling path, published for anyone to hold me to: vertical first, clustering with sticky routing by game id only when the players demand it.

I made the economics public on purpose. The whole platform costs me about $10.30 per month, itemized line by line on the support page, which also shows the donation total and a runway ticker with the exact date chessm8 is funded until; when someone chips in, the date moves. Infrastructure with an audit trail felt more honest than a donate button pointing into a void.

The distributed systems cheat sheet

If you skimmed to the end, here is every idea I leaned on and the chess feature it hides inside:

• Single-writer principle: one process owns each game, so moves serialize in its mailbox and I never wrote a lock
• Monotonic time: clocks are charged where time cannot jump backwards, on the only machine allowed to measure it
• At-least-once delivery plus idempotency: results retry until acknowledged and still land in the ledger exactly once
• Pessimistic locking: rating updates take a row lock, because Glicko-2 does not forgive lost updates
• Snapshotting: every move persists a versioned binary, so a restart is a resurrection, not a funeral
• Supervision: crashes are contained, restarted, and above all boring
• Split-brain avoidance: one node until sticky routing earns its complexity, because a game with two brains is a game with two kings

Your move

Chess was born in India some fifteen centuries ago and has never once asked anyone for an email address. I tried to build the version of online chess that honors that.

Play it at playchessm8.com. Break it if you can; the telemetry will tell on you either way. And if the engineering made you smile, the support page is where a coffee’s worth of support keeps the server thinking.

Special thanks to Sir Lorenz Cornelis, a colleague from Microsoft and honorary knight of this board, who test-played chessm8 and sent back the kind of sharp, specific feedback that genuinely helped me debug. Every project needs a tester like him.

This was part 1. In part 2, I attack my own server, a rook refuses to move, and the credential bots arrive at midnight.

No signup. Just chess. Your move, m8. ♟