Cases11 cases

9 to 11 of 11

1. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured

   Devsisters2021

   Symptoms

   During a team dinner, a Slack alert reported that one node of the CockroachDB cluster behind Cookie Run: Kingdom had gone down. The DevOps engineers were not worried at first, because CockroachDB is built to survive the loss of some nodes.

   Node NotReadyData lossService outage

   Trigger

   A cooling system failure in an AWS Tokyo region data centre overheated the servers in one availability zone, which held CockroachDB nodes.

   Root cause

   Cloud provider outage

   Component

   DatabaseHelm

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   A Slack alert reported that a database node had gone down.

   Remediation

   • copied the data outside the broken ranges into new tables with INSERT INTO ... SELECT queries that skipped the affected ranges
   • ran debug unsafe-remove-dead-replicas together with Cockroach Labs engineers, recovering one of the two lost data ranges
   • restored the last data range with the storage-layer recovery method from the previous incident

   Action items

   • Configure CockroachDB Locality so data is spread evenly across availability zones
   • Deploy with one chart and a separate values file per availability zone to set Locality
   • Replace nodes by adding Locality-configured nodes, confirming replicas in every zone, then retiring the old nodes

   Lessons

   A replication factor sized for random instance retirements does not protect against a whole availability zone failing; for a game of this scale a multi-AZ placement is required, not optional. Keeping one uniform Helm chart made zone topology hard to set, which is part of why it was left out. A repetitive chart with a values file per zone sets it reliably.

   View case
2. IL-0518 gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage

   freedesktop.org2018

   Symptoms

   Shortly after 9am UTC on 17 July 2018, the disk space of the GitLab cluster behind gitlab.freedesktop.org was critical, and some API requests were failing because of the shortage. The administrator first learned of the failure when people mentioned it on IRC.

   Persistent volume fullHigh error rateService outage

   Trigger

   Many recently created forks of Mesa, each about 2.1GB, filled the disk space of the GitLab cluster.

   Root cause

   Capacity exhaustion

   Component

   Persistent volumes and stateful setsCloud block storageHelm

   Contributing factor

   Unverified assumptionUnrepresentative stagingMonitoring blind spotChange during peak hours

   Detection

   People mentioned the failure on IRC. Monitoring and alerting on disk usage and API error rate had not been set up yet.

   Remediation

   • Dropped all inbound traffic to GitLab at the firewall except from the administrator's IP
   • Took a manual backup by streaming tar over SSH to an encrypted partition on a local machine
   • Snapshotted all GCE disks behind the PersistentVolumes
   • Copied the data from the old disk snapshots to new enlarged volumes through a temporary GCE VM
   • Re-did the copy into new empty volumes with the correct directory structure and checked repositories, users and workflows
   • Restored public access to GitLab

   Action items

   • Set up monitoring, logging and alerts
   • Find ways to communicate outages much better; this one was announced only on IRC, and the lists were not told when it dragged on
   • Use the Cloud console clients instead of the slow Cloud Console web UI
   • Add more GitLab administrators, ideally someone familiar with Kubernetes deployments
   • Move from the single Omnibus container to GitLab's cloud native deployment with one container per service
   • Move storage to resizeable volumes or to Google Cloud Storage

   Lessons

   The backup step alone used up the outage window; backups had grown along with the disk problem, and a test run before announcing the window could have measured it. The changes had been tested on a scratch cluster, but not exactly as run, and tests of steps such as disk snapshots could have been more realistic. A single GitLab administrator who was new to Kubernetes was a single point of failure. Keeping at least three copies of the data from the first destructive step meant no data was lost.

   View case
3. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down

   Saltside2017

   Symptoms

   While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

   Pods PendingImage pull failuresNode NotReadyService outage

   Trigger

   A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

   Root cause

   Capacity exhaustion

   Component

   Node root diskkubeletContainer runtimeHelm

   Contributing factor

   Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure

   Detection

   The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

   Remediation

   • terminated the three worker nodes and let the auto scaling group replace them

   Action items

   • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
   • Increase the worker root EBS volumes from 20GB to 120GB
   • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
   • Configure different allocation limits for the different layers in Kubernetes
   • Discuss a highly available Tiller with the Helm team
   • Find a way to throttle the number of newly created pods

   Lessons

   An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

   View case

By root cause6