Cases2 cases

1 to 2 of 2

1. IL-0565 Cookie Run: Kingdom's CockroachDB cluster failed when 16 of 24 nodes lost their filesystems, causing a 36-hour outage

   Devsisters2022

   Symptoms

   Right after a script ran on the CockroachDB nodes behind Cookie Run: Kingdom, hundreds of alerts fired at once for error rate, database connections and health checks, and the game could no longer be played normally. Emergency maintenance started at 16:52.

   Service outageData lossDatabase overload

   Trigger

   On the Monday after the launch weekend, storage use was growing with the number of users and the cluster had about 36 hours left before its disks would fill. The team ran a prepared script to create ballast files on every node, as the official guide for CockroachDB 19.1.11 recommends. The ballast file path in the script was set wrong and pointed at a block device, and the command had no check for an existing file at the target path, so it overwrote the partition.

   Root cause

   Operator error

   Component

   Database

   Contributing factor

   No safeguard on destructive changeOverlapping failuresRetry storm

   Detection

   Metrics and alerts set up in advance fired hundreds of alerts on error rate, database connections and health checks right after the script ran.

   Remediation

   • Started emergency maintenance of the game at 16:52
   • Wrote Spark code that extracts the contents of CockroachDB SST files to CSV, and modified the CockroachDB CLI to make it more than 10 times faster
   • Converted about 7.5TB of SST files from the 7 surviving nodes to CSV
   • Deduplicated the extracted rows and merged them into one large CSV file that could be imported
   • Imported the latest snapshot and the latest 300 journal entries per user into a new cluster with more storage
   • Restored the other tables from the latest backup and rebuilt the user table from snapshots and journals with server logic
   • Checked the restored data against a database restored from backup, against analytics logs for sampled users, and with snapshot replay for every user
   • Reopened the game at 23:37 on the second day, after 30 hours 50 minutes
   • Failed over the AWS RDS database of the DevPlay platform after the reopening traffic overloaded it
   • Built a third database cluster on more m5d instances and moved the data with the official backup and restore
   • Ended the maintenance at 8:30, after 36 hours 29 minutes

   Action items

   • Require at least two engineers for any work on production infrastructure, however small
   • Write the scripts and the scenario for each infrastructure task in advance and have the team review them

   Lessons

   Because a person's mistake caused the failure, Devsisters changed its process for infrastructure work rather than relying on individual care: even experienced engineers make mistakes, so the prevention has to be built into the system. The author credits the recovery to an incident policy that does not blame the individual and asks for detailed records, to past incidents that were written down so people without the experience knew what to do, and to having infrastructure operations and data analysis teams with the needed infrastructure and skills in one organisation.

   View case
2. IL-0566 Hot ranges after a Cookie Run: Kingdom guild update drove CockroachDB nodes to 100% CPU, forcing emergency maintenance

   Devsisters2021

   Symptoms

   About half an hour after the Cookie Run: Kingdom guild update maintenance ended at 11:00 on 6 March 2021, one CockroachDB node was using all 24 cores of its 24-core machine and server metrics began to show a significant number of errors. Soon two more nodes showed abnormal CPU use.

   Load imbalanceDatabase overloadHigh error rate

   Trigger

   During the guild update maintenance, new guild tables were added and pre-split across the cluster, and a new database node was added to replace one removed a few days earlier for a physical hardware issue. When the game reopened, all players at once wrote to existing tables whose primary-key prefixes were new or had changed in the update.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Unverified assumptionOverlapping failures

   Detection

   Monitoring after the maintenance showed one node's CPU at the ceiling together with errors in server metrics. Later a Slack alert reported a node failing its Host Status Check, and the abnormal-log channel reported filemap_fdatawait_range_keep_errors.

   Remediation

   • Blocked new logins without kicking players already in the game, using the maintenance switch as a makeshift access queue
   • Put the game into full emergency maintenance at 14:50 when several nodes failed health checks
   • Backed up the game data table by table after the full database backup stalled
   • Paused writing play records to reduce load on the database before reopening
   • Split the problem tables from about 18:30
   • Decommissioned the node that failed its Host Status Check
   • Reopened the game and used the login-block queue again when a node reached 100% CPU
   • Terminated the node that kept returning to 100% CPU

   Lessons

   In a single-server relational database a change to the primary-key prefix is usually harmless, but in CockroachDB's distributed design such a change on a production cluster needs great care. The team had studied this in CockroachDB documentation, but living through it taught far more than reading about it.

   View case

By root cause2