Everything that tried to break my chess server

In part 1 I told you how I built chessm8: one Elixir process per game, server-owned clocks, exactly-once ratings, snapshots that make crashes boring. That was the theory. This is the part where reality, my own testing, and the internet’s bots all took turns trying to break it.

The races I found by attacking my own system

A week before launch I stopped building and spent days trying to break chessm8 like a hostile reviewer. Best engineering decision of the project. Here is what fell out:

The rematch hijack. A rematch created a fresh game with two open seats, and the game id was in the URL. A fast stranger with the link could steal your opponent’s chair. Now a rematch reserves both seats for the original players, and everyone else bounces off.

The double queue. Join matchmaking for blitz, then click rapid, and depending on timing you could sit in both queues and get matched twice at once, like a simul you never agreed to. The matchmaker now dequeues before it enqueues, atomically.

The polling bill. The matchmaker polled every queue on a timer, so an idle site was paying Redis to confirm nobody was there. Now it tracks which queues have waiters and polls only those. An empty site issues zero Redis commands. My proudest optimization is the one where the server does nothing.

Magnus and magnus. Usernames were unique but case-sensitive, so both could register. One unique index on the lowercased name later, there is only one Magnus, as nature intended.

The nastiest bug was not on the server

Two days before launch, a friend sent screenshots that made my stomach drop: a rook that refused to move, two pieces stacked on one square. My beautiful authoritative server, and the board was lying to people.

The server was innocent. The bug was a stale closure in the React client: a fast reply applied to a copy of the piece sprites captured before the previous render finished, and the display drifted from the engine. The logic was fine; the pixels were wrong, which for a user is the same thing.

I fixed it by treating the UI as what it is: another unreliable replica. I moved the sprite state into a synchronous ref, outside the render cycle, and wrote a reconciler that, after every move, compares the sprites against the engine, rebuilds them on any mismatch, and fires a telemetry event. Distributed systems people call this read repair; I call it pixels that are not allowed to freelance. Then I made the computer prove it: 300 simulated games, 35,405 random moves, zero drift; 100 deliberately corrupted boards, 100 repaired. Since launch, that telemetry counter reads zero, which is my favorite dashboard.

Launch night

The gate dropped at 20:00 sharp. My battle station was not some Grafana wall; it was one terminal with the server’s pulse in it, and I have kept it open ever since:

chessm8’s heartbeat, as seen from my couch: a health check every 15 seconds, answered in a couple hundred microseconds.

The first real lesson arrived within hours. Postgres started freezing for five to ten seconds at a time. The database machine had 256MB of memory, real traffic arrived, and the kernel began applying memory pressure like a positional grind. One resize to 512MB and it has not blinked since. Capacity planning by prophecy is hard; capacity planning by dashboard is merely urgent.

Then a second kind of visitor arrived: vulnerability scanners, the bots that sweep every fresh hostname on the internet for forgotten phpinfo.php files and leaked cloud credentials. I watched one work through its entire wishlist live: Google service accounts, GCP keys, Firebase admin secrets, right down to ~/.config/gcloud, in case I had somehow published my home directory to the internet. My server speaks Elixir and the bots speak PHP, so the conversations were short: fifteen probes, fifteen 404s, the slowest answered in 751 microseconds.

Launch night, a few hours after the gate dropped: a scanner runs its credential wishlist and collects nothing but 404s. There is nothing here that speaks PHP.

What the war stories taught me

Part 1 was about designing for failure on purpose. This part was the failure showing up in person, and the lesson is the same one, three times over: the client will drift, so reconcile it against the truth; your own features will race each other, so attack them before strangers do; and the internet will knock on every door you have, so make sure every door answers quickly and says nothing.

Play the result at playchessm8.com. Break it in a way I have not thought of, and the telemetry will tell on you. And if these stories saved you a launch-night surprise of your own, the support page shows exactly what a coffee keeps running.

No signup. Just chess. Your move, m8. ♟