Incident Library
Applied filters Root causeCloud provider outageComponentHelmOrganisationDevsistersClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured
    Devsisters2021
    Symptoms

    During a team dinner, a Slack alert reported that one node of the CockroachDB cluster behind Cookie Run: Kingdom had gone down. The DevOps engineers were not worried at first, because CockroachDB is built to survive the loss of some nodes.

    Node NotReadyData lossService outage
    Trigger

    A cooling system failure in an AWS Tokyo region data centre overheated the servers in one availability zone, which held CockroachDB nodes.

    Root cause
    Cloud provider outage
    Component
    Application databaseHelm
    Contributing factor
    Unverified assumptionExcessive blast radius
    Detection

    A Slack alert reported that a database node had gone down.

    Remediation
    • copied the data outside the broken ranges into new tables with INSERT INTO ... SELECT queries that skipped the affected ranges
    • ran debug unsafe-remove-dead-replicas together with Cockroach Labs engineers, recovering one of the two lost data ranges
    • restored the last data range with the storage-layer recovery method from the previous incident
    Action items
    • Configure CockroachDB Locality so data is spread evenly across availability zones
    • Deploy with one chart and a separate values file per availability zone to set Locality
    • Replace nodes by adding Locality-configured nodes, confirming replicas in every zone, then retiring the old nodes
    Lessons

    A replication factor sized for random instance retirements does not protect against a whole availability zone failing; for a game of this scale a multi-AZ placement is required, not optional. Keeping one uniform Helm chart made zone topology hard to set, which is part of why it was left out. A repetitive chart with a values file per zone sets it reliably.

    View case
Check your cluster for this

This case involves Application database and Helm. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage1 case

Component
Application databaseHelm
Contributing factor
Unverified assumptionExcessive blast radius
Organisation
Devsisters
  1. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured
    Devsisters2021