Cases14 cases

6 to 10 of 14

1. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes

   incident.io2023

   Symptoms

   Shortly after incident.io moved its infrastructure fully into Google Cloud, the application saw a larger-than-normal number of Postgres connection timeouts, and new database connections per second were in the hundreds, spiking up and down. At the same time, the memcached instance used for transient data and faster API responses had sporadic connection timeouts.

   Intermittent timeoutsPacket loss

   Trigger

   After the move to Google Cloud, the application opened hundreds of new Postgres connections per second, and an accidental database join made it send thousands of duplicate calls to an external third party in parallel.

   Root cause

   Capacity exhaustion

   Component

   CNI pluginDatabaseIn-memory cache

   Contributing factor

   Unreviewed defaultsOverlapping failures

   Detection

   An incident was declared when the number of Postgres connection timeouts rose above normal. The remaining cache timeouts were traced through the logs of the Kubernetes node that served an affected request, the request's trace and the node's metrics.

   Remediation

   • Doubled the maximum Postgres connection lifespan from 15m to 30m
   • Made the Postgres connection pools static so they always keep N connections open
   • Raised the memcache client's MaxIdleConns from 2 to 20
   • Moved temporarily from Google's hosted memcached to a memcached inside the Kubernetes cluster
   • Shipped the fix for the accidental database join that made thousands of duplicate external calls

   Action items

   • Set more sensible concurrency limits for outbound network calls

   Lessons

   Network timeouts are not always local to one request or pod: when they appear, the whole activity of the node, including other pods on it, needs to be checked. Staying within a provider's rate limits does not keep outbound calls from affecting a node, so outbound network calls need more sensible concurrency limits.

   View case
2. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case
3. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded

   OpenAI2022

   Symptoms

   At 18:56 PDT on 23 June 2022 (01:56 UTC on 24 June) several models went offline at once: some fine-tuned curie and davinci models, the Codex beta models code-davinci-001 and code-cushman-001, the legacy GPT models curie, babbage and ada, and all Embeddings models.

   Service outageHigh error rateIntermittent timeouts

   Trigger

   A destructive operation on one of the three production clusters, caused by human error that the existing safeguards did not catch, took the whole cluster offline.

   Root cause

   Operator error

   Component

   Ingress controllerModel hub

   Contributing factor

   No safeguard on destructive changeNo headroomDeferred upgradesOverlapping failures

   Remediation

   • re-allocated the models from the lost cluster to the remaining two clusters
   • rebuilt the VPN link with Azure support after losing the connection to AKS during cluster setup
   • built workarounds with Azure support after losing the connection to Azure Private Link
   • tore down the cluster and rebuilt it from scratch after the upgraded Kubernetes version broke the ingress
   • moved traffic back into the rebuilt cluster between 05:30 and 06:30 PDT

   Action items

   • Add checks and locks so that this and similar production-critical infrastructure cannot be modified by accident
   • Add better visibility into the operations being performed and make parts of the system more lexicographically distinct
   • Keep working with Azure on the AKS network connectivity issues so they do not disrupt new cluster operations
   • Update playbooks with the new cluster failure modes and change the setup steps for new clusters and infrastructure
   • Upgrade Kubernetes systematically and set aside time so the clusters do not fall behind again
   • Shard fine-tuned models across multiple clusters and work towards more automatic cross-cluster replication
   • Distribute and replicate Azure Blob Storage so fine-tuned models load faster and more reliably
   • Finish rolling out node-to-node caching to relieve Azure Blob Storage bottlenecks
   • Change how model weights are stored so they are more efficient and faster to load

   Lessons

   Losing one of three clusters exhausted the GPU headroom, and GPUs could not be added on short notice, so full recovery depended on bringing the same cluster back. Rebuilding it exposed a deferred Kubernetes upgrade: the newer version broke the ingress, and because AKS cannot downgrade there was no way back except a full rebuild. Recovering many large models at once is limited by the bandwidth of the storage they load from, and the caches that normally hide this do not help when everything moves to a different cluster.

   View case
4. IL-0591 Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster release

   Gitpod2022

   Symptoms

   From April 11, 2022 at 13:17 UTC customers and Gitpod staff reported that Gitpod workspaces were intermittently becoming unresponsive. CPU usage of nodes and pods looked normal, with nothing maxed out, and at 13:58 UTC the team found that the workspaces were IO bound rather than CPU bound.

   Throughput degradationSustained high node CPUNode NotReadyData loss

   Trigger

   The ws39 generation of workspace clusters, released to all regional clusters on April 8th at 20:08 UTC and to the XL clusters on April 9th at 17:45 UTC, split the 1TB disk into smaller RAID0 disks, changed CPU limiting and workspace networking, and moved to faster machine types.

   Root cause

   Faulty deployment

   Component

   Node root diskContainer CPU limit

   Contributing factor

   No staged rolloutGap in automated testsUnrepresentative stagingUnknown limitOverlapping failuresIneffective rollbackHarmful manual actionLimited debug access

   Detection

   Customers and Gitpod staff reported on April 11th at 13:17 UTC that workspaces were intermittently unresponsive, and the team opened an incident in Slack through its incident.io integration. On April 17th the team noticed increased CPU load on the workspace nodes and opened an internal incident.

   Remediation

   • Shifted workspace starts back to the ws38 clusters on April 11th
   • Built ws39a clusters with c2d machines and local SSDs and moved traffic to them
   • Requested a limit increase from Let's Encrypt
   • Routed EU XL traffic to the regional EU cluster and built ws-eu39bxl to replace the crashed ws-eu39xl
   • Cleaned up workspaces stuck in a stopping state so users could start them again
   • Released ws40 with 1TB boot disks to get more IO bandwidth
   • Removed the workspaces that consumed excessive CPU time for crypto-mining
   • Blocked the fraudulent users behind the crypto-mining from using Gitpod
   • Rolled back the ws-daemon change that broke CPU limiting and shipped the fix in the ws34 clusters
   • Blocked ACME HTTP requests to workspace ports so Let's Encrypt rate limits would not be hit again
   • Rolled out IO limiting with the ws41xl and ws41 clusters and moved traffic to them step by step

   Action items

   • Invest in test automation so that test results stop releases like this one
   • Vary the interval between traffic shifts to new clusters, with more time for larger releases
   • Add a risk assessment stage, to be automated, that governs changeset size and rollout speed of workspace cluster rollouts
   • Reduce the impact teams have on each other
   • Add review criteria that find backwards incompatible changes before pull requests are merged, and split such pull requests
   • Recommend rollback as the primary means of mitigation in the incident procedures, and look at automating it
   • Make the status page show the workspace service as down when it is, and update the playbooks
   • Keep incident communication in a single Slack channel and Google Meet
   • Put 10% of the engineering staff on the on-call rotation from May 9th and use that time on observability, alerting and stability

   Lessons

   Gitpod concluded that it should have committed to rolling back on April 11th instead of building and shipping fixes, which kept customers on degraded workspaces longer than necessary. A large release so soon after the previous one needed a risk assessment and slower traffic shifts, and backwards incompatible changes that block a rollback need to be found before pull requests are merged. Failure-free operations require experience with failure.

   View case
5. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1

   GitLab2021

   Symptoms

   At 08:05 UTC on 2021-06-21 the API service's Apdex (the share of requests that finish within a target time) and Rails request queuing began trending the wrong way. At 09:01 the team found the API deployment unhealthy: it needed more pods for the day's growing traffic, but no new nodes could be added to give it pod capacity. Users saw slow responses; the review found no sign that error rates rose.

   Scale-up failureHigh latencyPods PendingDeploys failing

   Trigger

   Google Cloud ran out of SSD persistent disk stock in us-east1, the region GitLab.com runs in, while the API service needed more nodes for rising user traffic.

   Root cause

   Cloud provider outage

   Component

   Cluster autoscalerNode pool

   Contributing factor

   Monitoring blind spotLimited debug accessOverlapping failures

   Detection

   There was no alert on resource availability. Apdex and Rails queuing began trending the wrong way at 08:05 UTC and an engineer declared the incident in Slack at 08:43; finding the cause took a lot of log digging: cluster autoscaler errors about node pools in backoff, then ZONE_RESOURCE_POOL_EXHAUSTED at the instance-group level, and finally Google Cloud support naming SSD as the exhausted resource.

   Remediation

   • added a node pool that uses pd-standard disks
   • scaled the new node pool by hand across all three zones to relieve the API
   • brought more pd-standard node pools online for the other blocked workloads
   • stopped production deployments and opened a Google Cloud support case about the cluster autoscaler not working with the new node pools
   • changed node pools and nodeSelectors so the remaining workloads ran on pd-standard nodes
   • removed the SSD-backed node pools after a day without problems

   Action items

   • Move Kubernetes services to pd-standard disks (delivery#1837)
   • Capture resource exhaustion errors as a metric

   Lessons

   A cloud region can run out of one specific resource, here SSD persistent disks, and the only error may be a generic ZONE_RESOURCE_POOL_EXHAUSTED that does not name it. GitLab could neither monitor nor reserve resource availability, so detection took 22 hours and the cause came from Google Cloud support. The workloads ran fine on pd-standard disks. A second problem found on the way, a cluster autoscaler that does not fall back to another matching node pool when one cannot scale, slowed the move to the new node pools.

   View case

By root cause8