Cases15 cases

6 to 10 of 15

1. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured

   Devsisters2021

   Symptoms

   During a team dinner, a Slack alert reported that one node of the CockroachDB cluster behind Cookie Run: Kingdom had gone down. The DevOps engineers were not worried at first, because CockroachDB is built to survive the loss of some nodes.

   Node NotReadyData lossService outage

   Trigger

   A cooling system failure in an AWS Tokyo region data centre overheated the servers in one availability zone, which held CockroachDB nodes.

   Root cause

   Cloud provider outage

   Component

   Application databaseHelm

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   A Slack alert reported that a database node had gone down.

   Remediation

   • copied the data outside the broken ranges into new tables with INSERT INTO ... SELECT queries that skipped the affected ranges
   • ran debug unsafe-remove-dead-replicas together with Cockroach Labs engineers, recovering one of the two lost data ranges
   • restored the last data range with the storage-layer recovery method from the previous incident

   Action items

   • Configure CockroachDB Locality so data is spread evenly across availability zones
   • Deploy with one chart and a separate values file per availability zone to set Locality
   • Replace nodes by adding Locality-configured nodes, confirming replicas in every zone, then retiring the old nodes

   Lessons

   A replication factor sized for random instance retirements does not protect against a whole availability zone failing; for a game of this scale a multi-AZ placement is required, not optional. Keeping one uniform Helm chart made zone topology hard to set, which is part of why it was left out. A repetitive chart with a values file per zone sets it reliably.

   View case
2. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub

   2i2c2020

   Symptoms

   At 08:52 PM IST a report came in from WER that many students could not access the JupyterHub and that it was frozen; pages were stuck and nothing was usable. A rise in activity was visible by 09:02 PM, but the usual fixes, such as opening an incognito window and restarting servers, did not help.

   Service outageSustained high node CPU

   Trigger

   WER students' pods used a good chunk of their memory limit (tests had shown they almost always use just under 1G of their 2G limit, against a 512M guarantee) on nodes shared with the hub's core pods.

   Root cause

   Capacity exhaustion

   Component

   Node poolContainer memory limit

   Contributing factor

   No resource requests or limitsUnverified assumptionExcessive blast radius

   Detection

   WER reported stuck pages and students unable to reach the hub in a GitHub issue (2i2c-org/docs#27).

   Remediation

   • removed memory overcommit for the WER students
   • raised resource guarantees for all core pods, which restarted them and moved some to a new node

   Action items

   • Put user pods in a separate pool so they do not create pressure on the core pods (infrastructure#89)
   • Set limits on the support infrastructure (Prometheus, Grafana, ingress) as well (infrastructure#90)
   • Document and think about overcommit ratios for memory usage (infrastructure#91)
   • Set up better Grafana dashboards to monitor resource usage (infrastructure#92)
   • Document how people can get kubectl access to the cluster so others can look into issues too (infrastructure#87)

   Lessons

   Core pods without resource guarantees get whatever is left on a node, so when user pods on the same nodes fill the CPU and memory, the hub that serves everyone freezes. The memory overcommit ratio has to come from measured use: WER students used just under 1G against a 512M guarantee, which a 4x overcommit could not hold.

   View case
3. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data

   Palark2020

   Symptoms

   Right after an updated RedisFailover resource was rolled out, the large Redis database lost all its data and the service went down; the data had to be restored from a backup taken just before the change.

   Data lossService outage

   Trigger

   A RedisFailover change that raised the number of Redis replicas from one to two and, in the same rollout, lowered the containers' memory requests and limits from 22 GB to 19 GB.

   Root cause

   Software defect

   Component

   Application databasePersistent volumes and stateful setsContainer memory limit

   Contributing factor

   No staged rolloutSingle point of failureUnverified assumption

   Remediation

   • restored the Redis data from the backup prepared before the rollout

   Action items

   • Report the readiness probe flaw to the Redis Operator project and test the developers' fix
   • Reconsider the use of the Spotahome Redis Operator
   • Clean up the Redis databases and fix the application code that created the excess keys
   • Consider Redis built-in clustering and sharding

   Lessons

   Changing the replica count and the pod resources in one rollout turned a seemingly harmless change into a rolling restart of the only pod with data while the new replica was still empty. A readiness probe that only checks that Redis answers a ping does not tell whether a replica has finished syncing. Operators for critical stateful components need to be evaluated for maturity before production use, and the dump.rdb file should be backed up before any change to the Redis cluster.

   View case
4. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage

   Grafana Labs2019

   Symptoms

   An alert fired because some Metrictank instances could not drain their write queue, which meant they could not write to Cassandra. Google had not yet published an incident, but a team member's Discord server, also hosted on GCP, was down, which pointed at GCP; Google confirmed an SSD persistent disk incident about 90 minutes after it began.

   High error rateNode NotReadyPods PendingService outage

   Trigger

   A Google Cloud incident in which SSD persistent disks performed so poorly that many I/O operations failed for an extended period.

   Root cause

   Cloud provider outage

   Component

   Persistent volumes and stateful setsApplication databaseNode root disk

   Contributing factor

   Overlapping failuresUnpatched known bugUnverified assumption

   Detection

   An alert that Metrictank write instances could not flush their write queue to Cassandra. Error rates alone rose too slowly to notice quickly, because most queries were served from memory.

   Remediation

   • Added compute nodes hoping new nodes could attach the disks, which did not work
   • Deleted and recreated the Cassandra StatefulSet with an updated pod management policy, so all pods but one started
   • Raised Kafka topic retention to avoid data loss during the long downtime
   • Temporarily added nodes so Metrictank restarts were not slowed by CPU starvation
   • Restarted all Metrictank instances, which took more than 20 hours, so they connected to the new Cassandra IPs

   Action items

   • Re-instantiate the gocql connection in Metrictank when it is detected dead, so it reconnects even when all Cassandra IPs change
   • Fix the Metrictank write-queue deadlock by updating an internal data structure with atomics instead of a lock

   Lessons

   An OrderedReady StatefulSet cannot recover past a single pod whose volume will not attach; when every pod is down, the ordering that protects normal start-up blocks the whole cluster. A client that learns peer addresses only from live peers needs a way to go back to the service name when all peers are replaced at once. A long-tolerated bug with a config workaround can surface at the worst time; this one turned out to have a simple fix. Writing incoming data to a Kafka write-ahead log first meant all data could still be ingested after recovery, and Kafka on non-SSD disks escaped the disk incident. With both Metrictank bugs fixed, the team expects a similar event to take less than 3 hours to recover from instead of 23.

   View case
5. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic

   Airbnb2019

   Symptoms

   An operation to upgrade the stats agent in Kubernetes was announced in advance, with a warning that metrics might be unavailable to services for a few minutes. A few minutes later a reply came in saying that all of the sender's Kubernetes services seemed to be down.

   Probe failuresService outage

   Trigger

   A scheduled upgrade of the stats agent sidecar, during which the agent container crashed.

   Root cause

   Faulty deployment

   Component

   Telemetry agentService discovery

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   A reply to the operation announcement, a few minutes later, said that all of the sender's Kubernetes services appeared to be down.

   Action items

   • Mark logging, tracing and stats agent containers as non-critical and leave them out of the service mesh's pod readiness check

   Lessons

   Every container in a pod affects the pod's readiness, so each sidecar needs a deliberate decision about how its health should count. A sidecar that crashes is not ready even when it has no probes, and any logic that requires all containers to be ready will take the pod out of traffic.

   View case

By root cause7