Incident Library

IL-0566 Devsisters · 2021-03-06databaseaws

Hot ranges after a Cookie Run: Kingdom guild update drove CockroachDB nodes to 100% CPU, forcing emergency maintenance

CockroachDB stores data in units called ranges and splits or merges them on its own by size and queries per second; by default a range stays between 128MB and 512MB and below 2500 queries per second. Ranges are cut from primary keys in sorted order, so similar keys land in the same or neighbouring ranges, and a range that receives most of the workload, a hot range, leaves its few nodes carrying the whole load. In the Cookie Run: Kingdom guild update, which ended at 11:00 on 6 March 2021, the new guild tables were split in advance, but the root cause was an index prefix structure that was added and changed without the team noticing. Analytics tables written by the game server use primary keys of the form <feature>-<uuid>; they had never been split and had caused no problem since launch, because the tutorial spread players' first writes over time. The guild feature had almost no such funnel, so right after the update every player wrote huge numbers of guild-<uuid> keys before the system could split the range. In a second table, a refactoring changed the primary-key format, so only a small part of its already split ranges was used. Both tables existed before, had no schema change and took no direct gameplay traffic, so nobody expected hot ranges.

CockroachDB's load-based rebalancing could have fixed this over time, but the node added during the maintenance made CockroachDB move ranges onto it, and moving ranges uses the same queue as splitting them. The replication queue had not drained by the reopening, so the hot ranges could only be spread after the new node had received its share. From about 13:00 the team used the game's login block without kicking players as a makeshift queue, since the game had no access queue; at 14:50, when several nodes failed health checks, it put the game into full emergency maintenance, and the cluster improved at once. Because full backups stalled, the team backed up the game data table by table, with help from CockroachDB support, paused writing of play records, and from about 18:30 split the problem tables while waiting for the replication queue to reach zero. A node that failed its Host Status Check at about 19:20 and logged filemap_fdatawait_range_keep_errors was decommissioned, after which no range was under-replicated and cluster metrics returned to normal.

The game reopened around 22:00 ahead of the midnight login-bonus rush. One node reached 100% CPU, was calmed with the login-block queue, and soon did it again; because it was always the same node, the problem was most likely the node itself, and terminating it ended the incident.

Severity
–
Scope
Cookie Run: Kingdom returned errors after the guild update, new logins were blocked for periods, and the game was in emergency maintenance from 14:50 until it reopened in the evening
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

About half an hour after the Cookie Run: Kingdom guild update maintenance ended at 11:00 on 6 March 2021, one CockroachDB node was using all 24 cores of its 24-core machine and server metrics began to show a significant number of errors. Soon two more nodes showed abnormal CPU use.

Right after the maintenance, database CPU had looked a little higher than expected, but with more players online than usual and a new interactive feature, the team judged it nothing to worry about. Once single nodes hit the ceiling, it was clear that load was not being spread evenly and that the overloaded nodes were hurting the stability of the whole database cluster.

Blocking new logins while letting players already in the game continue calmed the cluster for a while, but at 14:50 several nodes suddenly started failing health checks, more nodes than right after the maintenance. During the instability CockroachDB marked some ranges Unavailable, blocking access because it could not guarantee their consistency. A full database backup with the BACKUP statement stalled partway, and an export to CSV made the cluster less stable and was stopped.

Later, at about 19:20, a Slack alert reported that a database node had failed its Host Status Check, and the channel that watches for abnormal server logs reported filemap_fdatawait_range_keep_errors from the same instance. After the game reopened, one database node reached 100% CPU, and after it was calmed with the login block, the same node went back to 100% CPU again and again.

Symptom Load imbalanceDatabase overloadHigh error rate

Root cause

Root cause Faulty deployment

Component Database
Contributing factor Unverified assumptionOverlapping failures
Trigger
During the guild update maintenance, new guild tables were added and pre-split across the cluster, and a new database node was added to replace one removed a few days earlier for a physical hardware issue. When the game reopened, all players at once wrote to existing tables whose primary-key prefixes were new or had changed in the update.

Detection

Monitoring after the maintenance showed one node's CPU at the ceiling together with errors in server metrics. Later a Slack alert reported a node failing its Host Status Check, and the abnormal-log channel reported filemap_fdatawait_range_keep_errors.

Remediation

Action items

–

Lessons

In a single-server relational database a change to the primary-key prefix is usually harmless, but in CockroachDB's distributed design such a change on a production cluster needs great care. The team had studied this in CockroachDB documentation, but living through it taught far more than reading about it.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.