Incident Library
Applied filters SymptomHigh error rateOrganisationDevsistersClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0566 Hot ranges after a Cookie Run: Kingdom guild update drove CockroachDB nodes to 100% CPU, forcing emergency maintenance
    Devsisters2021
    Symptoms

    About half an hour after the Cookie Run: Kingdom guild update maintenance ended at 11:00 on 6 March 2021, one CockroachDB node was using all 24 cores of its 24-core machine and server metrics began to show a significant number of errors. Soon two more nodes showed abnormal CPU use.

    Load imbalanceDatabase overloadHigh error rate
    Trigger

    During the guild update maintenance, new guild tables were added and pre-split across the cluster, and a new database node was added to replace one removed a few days earlier for a physical hardware issue. When the game reopened, all players at once wrote to existing tables whose primary-key prefixes were new or had changed in the update.

    Root cause
    Faulty deployment
    Component
    Database
    Contributing factor
    Unverified assumptionOverlapping failures
    Detection

    Monitoring after the maintenance showed one node's CPU at the ceiling together with errors in server metrics. Later a Slack alert reported a node failing its Host Status Check, and the abnormal-log channel reported filemap_fdatawait_range_keep_errors.

    Remediation
    • Blocked new logins without kicking players already in the game, using the maintenance switch as a makeshift access queue
    • Put the game into full emergency maintenance at 14:50 when several nodes failed health checks
    • Backed up the game data table by table after the full database backup stalled
    • Paused writing play records to reduce load on the database before reopening
    • Split the problem tables from about 18:30
    • Decommissioned the node that failed its Host Status Check
    • Reopened the game and used the login-block queue again when a node reached 100% CPU
    • Terminated the node that kept returning to 100% CPU
    Lessons

    In a single-server relational database a change to the primary-key prefix is usually harmless, but in CockroachDB's distributed design such a change on a production cluster needs great care. The team had studied this in CockroachDB documentation, but living through it taught far more than reading about it.

    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
Database
Contributing factor
Unverified assumptionOverlapping failures
Organisation
Devsisters
  1. IL-0566 Hot ranges after a Cookie Run: Kingdom guild update drove CockroachDB nodes to 100% CPU, forcing emergency maintenance
    Devsisters2021