Cases13 cases

10 to 13 of 13

1. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect

   Webhook Relay2022

   Symptoms

   At 12:00 EST the health check system reported that the backend APIs, the frontend and the tunnels in the EU region were down. The team first took it for a transient error, such as an interrupted client request or a server evicted for memory or CPU pressure, but the main dashboard was offline as well.

   Service outageControl plane lagControl plane unresponsiveStale or inconsistent API readsCredentials unavailable

   Root cause

   Certificate or credential expiry

   Component

   GKE control planeKubernetes API serverNode poolRBAC and service accounts

   Contributing factor

   Limited debug accessSingle point of failure

   Detection

   The health check system sent a notification that the service was down; the team then found the main dashboard offline.

   Remediation

   • recreated the node pools by hand after the credential rotation failed to recreate them
   • duplicated the main cluster's services into a newly created GKE cluster
   • deleted the Kubernetes Service records in the old cluster so its load balancers could be attached to the new cluster
   • detached the StatefulSet from the old cluster and switched all writes to the main disk
   • exported the data from the temporary ClickHouse instance and inserted it into the main instance

   Action items

   • Change the deployment manifests so services can be switched between clusters quickly without a complicated persistent disk migration

   Lessons

   Recovery of existing infrastructure needs a time limit. Throughout the outage the old cluster looked about to recover, and the reluctance to give up on the salvage effort kept the team from starting over with a new cluster much sooner, which would have been the right call. On older GKE clusters, check the validity of the cluster CA certificate, for example by decoding masterAuth.clusterCaCertificate from gcloud container clusters describe and reading it with openssl x509. GKE's multiple availability zones made no difference in this failure.

   View case
2. IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hung

   Meta AI2021

   Symptoms

   Between the team's 17 November 2021 update and 3 December 2021, the 175B-parameter OPT training run went down and had to be restarted more than 40 times, for hardware, infrastructure and training stability reasons. A hardware fault could stop the run for hours at any time of day, and several did over the Thanksgiving break.

   Training job failureGPU unavailable

   Trigger

   Nodes in the cloud training cluster failed with ECC errors, and each failure took the training run down.

   Root cause

   Dependency failure

   Component

   GPU hardware and memoryNCCLInfiniBand fabricObject storage

   Contributing factor

   No headroomLimited debug accessNo runbook

   Detection

   During these two weeks the team added monitoring of train.log, GPU burn-in testing to detect memory errors, and automated InfiniBand testing.

   Remediation

   • replaced failed nodes, and later did it through a script
   • added GPU burn-in testing to detect memory errors
   • automated InfiniBand testing
   • added monitoring of train.log
   • debugged the cloud provider's Python API until checkpoint downloads from the blob store worked well enough

   Action items

   • Run an on-call rotation for the training run, with runbooks and tooling for the most common issues, and invite more people to join it

   Lessons

   At this scale cloud hardware failures are certain, and the team concluded that automating the handling of hardware failures is necessary. Without spare nodes and burn-in tests, each failure costs hours and a replacement can bring the same fault back. The team also concluded that cloud infrastructure instability is something to always prepare for, because the customer has no control over the underlying hardware. A checkpoint large enough to need hundreds of multi-gigabyte downloads makes the storage path part of recovery time.

   View case
3. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1

   GitLab2021

   Symptoms

   At 08:05 UTC on 2021-06-21 the API service's Apdex (the share of requests that finish within a target time) and Rails request queuing began trending the wrong way. At 09:01 the team found the API deployment unhealthy: it needed more pods for the day's growing traffic, but no new nodes could be added to give it pod capacity. Users saw slow responses; the review found no sign that error rates rose.

   Scale-up failureHigh latencyPods PendingDeploys failing

   Trigger

   Google Cloud ran out of SSD persistent disk stock in us-east1, the region GitLab.com runs in, while the API service needed more nodes for rising user traffic.

   Root cause

   Cloud provider outage

   Component

   Cluster autoscalerNode pool

   Contributing factor

   Monitoring blind spotLimited debug accessOverlapping failures

   Detection

   There was no alert on resource availability. Apdex and Rails queuing began trending the wrong way at 08:05 UTC and an engineer declared the incident in Slack at 08:43; finding the cause took a lot of log digging: cluster autoscaler errors about node pools in backoff, then ZONE_RESOURCE_POOL_EXHAUSTED at the instance-group level, and finally Google Cloud support naming SSD as the exhausted resource.

   Remediation

   • added a node pool that uses pd-standard disks
   • scaled the new node pool by hand across all three zones to relieve the API
   • brought more pd-standard node pools online for the other blocked workloads
   • stopped production deployments and opened a Google Cloud support case about the cluster autoscaler not working with the new node pools
   • changed node pools and nodeSelectors so the remaining workloads ran on pd-standard nodes
   • removed the SSD-backed node pools after a day without problems

   Action items

   • Move Kubernetes services to pd-standard disks (delivery#1837)
   • Capture resource exhaustion errors as a metric

   Lessons

   A cloud region can run out of one specific resource, here SSD persistent disks, and the only error may be a generic ZONE_RESOURCE_POOL_EXHAUSTED that does not name it. GitLab could neither monitor nor reserve resource availability, so detection took 22 hours and the cause came from Google Cloud support. The workloads ran fine on pd-standard disks. A second problem found on the way, a cluster autoscaler that does not fall back to another matching node pool when one cannot scale, slowed the move to the new node pools.

   View case
4. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop

   Fullstaq2019

   Symptoms

   Dex, the OIDC proxy that handled Kubernetes authentication, stored its data in the Kubernetes API (storage type kubernetes, inCluster: true) and had built up a huge number of AuthRequest objects. Listing all AuthRequests timed out, and without the list they could not be deleted.

   Control plane unresponsiveMonitoring outage

   Trigger

   The dex namespace was deleted so that Kubernetes would clean up the accumulated AuthRequest objects itself.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serveretcdCluster authentication provider

   Contributing factor

   Unreviewed defaultsHarmful manual actionRetry stormLimited debug accessIgnored early signal

   Detection

   The speaker says he had got hints of the problem earlier. While the masters were down, plenty of alerts arrived in Slack, but monitoring could not be reached because authentication and port-forwarding no longer worked.

   Remediation

   • removed the kubernetes finalizer from the dex namespace by calling the API directly (kill-kube-ns)
   • deployed Dex to a new namespace with sane defaults

   Lessons

   Dex with Kubernetes as its storage keeps every AuthRequest as an API object, and with a long TTL they pile up until the API cannot list them. Deleting a namespace that holds that many objects makes Kubernetes list them in a loop and can overload the masters. Monitoring that is reached through cluster authentication is lost when authentication is what broke.

   View case

By root cause8