Incident Library

IL-0565 Devsisters · 2022-12-22 (published)databaseaws

Cookie Run: Kingdom's CockroachDB cluster failed when 16 of 24 nodes lost their filesystems, causing a 36-hour outage

Cookie Run: Kingdom stored its user data in CockroachDB on AWS EC2 m5d.8xlarge instances, a type with two local storage devices, with 24 nodes and data configured to keep 7 replicas. About four days after launch, storage use was rising fast, and a full disk on Linux would have stopped all writes. To prepare, the team ran a script that creates a ballast file, a dummy file that reserves space and can be deleted in an emergency. Because of a configuration issue, the script's path was wrong on nodes where it matched the primary mount, which EC2 picks at random among several local devices; the command overwrote the partition, and 16 of 24 nodes lost a usable filesystem. With more than a majority of nodes gone, the cluster stopped. AWS support could not help, and CockroachDB support advised restoring from the last backup, which would have lost all data written since that backup. Devsisters and the game team decided not to roll back user data and worked on three plans at once: repair the filesystems (Plan A), extract the data from the storage layer of the surviving nodes (Plan B), and replay analytics logs on top of the latest backup (Plan C, started for the most important server APIs).

Plan A failed: the filesystems could not be repaired, and the 7 remaining nodes could not form a cluster because of the Raft quorum check at start-up. Plan B worked. The team estimated the chance of fully losing data at C(16, 7) / C(24, 7), about 3.3%, found a debug command in the CockroachDB source that shows the inside of SST files, decoded their format, and by about 4:00 the next day had Spark code that converted SST files to CSV. A modified CockroachDB CLI made the conversion more than 10 times faster than the estimated 24 hours on a 150-machine Spark cluster. From about 10:30 the team converted about 7.5TB of SST files from the 7 surviving nodes. IMPORT INTO takes the table offline and reverts everything on any key conflict, so the CSV files had to be deduplicated and merged into one file. To save time, only the latest snapshot and the latest 300 journal entries per user were restored, into a new cluster with more storage; the Spark workers used up every available R-family instance in all AZs of the AWS Tokyo region. The journal, snapshot and user tables were restored by 17:22. Checks against a database restored from the last backup showed the recovered data contained every user of the backup, but the user table was badly corrupted, so it was rebuilt from snapshots and journals by server logic instead, and the other tables came from the backup. After sampling users against analytics logs, checking snapshot replay for every user and an internal test, the game reopened at 23:37 on the second day, 30 hours 50 minutes after the start.

The reopening brought record API traffic to DevPlay, whose AWS RDS database recovered after a failover. Load also built up on the new Kingdom cluster, which used i3 instances; disk write operations appeared to drive CPU load, and at 2:42 all queries failed. The team built a third cluster on more m5d instances, moved the data with CockroachDB's official backup and restore, using settings similar to those of the TPC-C benchmark on CockroachDB support's advice to cut the transfer time, and ended the maintenance at 8:30, 36 hours 29 minutes after it began.

Severity
–
Scope
Cookie Run: Kingdom was in emergency maintenance for 36 hours 29 minutes in total; the CockroachDB cluster holding user data was lost and rebuilt from data extracted from the surviving nodes
Users affected
Cookie Run: Kingdom players; the restored data belonged to users numbering in the millions
Region
AWS Tokyo region
Time to detect
–
Time to resolve
36h 29m

Symptoms

Right after a script ran on the CockroachDB nodes behind Cookie Run: Kingdom, hundreds of alerts fired at once for error rate, database connections and health checks, and the game could no longer be played normally. Emergency maintenance started at 16:52.

On part of the database nodes the filesystem became unstable, writes failed and the cluster isolated those nodes automatically. 16 of the 24 nodes were affected, more than the majority that CockroachDB's Raft consensus needs, so a node-level fault became a failure of the whole cluster. The whole DevOps team spent about 30 minutes reading shell history, CockroachDB logs, kernel logs and the CockroachDB source code before it was clear what had happened.

The first recovery attempts did not bring the cluster back. The affected partitions were copied to EBS volumes with dd and checked with fsck, which reported so many errors that the repair could not be trusted, and a cluster rebuilt from the fsck output still could not find certain SST files. At 23:02 one of the unaffected nodes went down with a Host Status Failure, leaving 7 healthy nodes. Starting a cluster from those 7 nodes failed because CockroachDB checks the Raft quorum at start-up and the records of the lost nodes kept it from reaching a healthy state; attempts to get around this went on until 5:00 the next morning.

After the game reopened, users who had kept checking for the reopening overloaded DevPlay, the Devsisters login platform, and its database failed in a cascade. Later, during the midnight peak, the new Kingdom database cluster showed errors from DB timeouts and some nodes at 100% CPU, and at 2:42 every query and request to the database failed, which forced a second emergency maintenance.

Symptom Service outageData lossDatabase overload

Root cause

Root cause Operator error

Component Database
Contributing factor No safeguard on destructive changeOverlapping failuresRetry storm
Trigger
On the Monday after the launch weekend, storage use was growing with the number of users and the cluster had about 36 hours left before its disks would fill. The team ran a prepared script to create ballast files on every node, as the official guide for CockroachDB 19.1.11 recommends. The ballast file path in the script was set wrong and pointed at a block device, and the command had no check for an existing file at the target path, so it overwrote the partition.

Detection

Metrics and alerts set up in advance fired hundreds of alerts on error rate, database connections and health checks right after the script ran.

Remediation

Action items

Lessons

Because a person's mistake caused the failure, Devsisters changed its process for infrastructure work rather than relying on individual care: even experienced engineers make mistakes, so the prevention has to be built into the system. The author credits the recovery to an incident policy that does not blame the individual and asks for detailed records, to past incidents that were written down so people without the experience knew what to do, and to having infrastructure operations and data analysis teams with the needed infrastructure and skills in one organisation.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.