Cases7 cases

1 to 5 of 7

1. IL-0247 One in a million search requests took 100 times longer on Pinterest's Manas leaf nodes after moving to Kubernetes

   Pinterest2025

   Symptoms

   During performance testing of Pinterest's search system Manas on its new Kubernetes platform, a small but persistent share of search requests timed out. About one request in a million took 100 times longer than usual.

   High latencyIntermittent timeouts

   Root cause

   Software defect

   Component

   Telemetry agentLinux kernel

   Contributing factor

   Unreviewed defaults

   Detection

   Performance testing with traffic sent to a test cluster showed a small but persistent percentage of requests timing out.

   Remediation

   • Disabled cAdvisor's working set size estimation on all PinCompute nodes

   Lessons

   CPU shielding does not fully isolate a process from its neighbours. Showing early that the node image was not the cause narrowed the search and avoided dead ends. Simple blackbox debugging, breaking things until the system stops working, works as a way of bisecting the problem.

   View case
2. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days

   Barush Mendez2025

   Symptoms

   Every service in the homelab Kubernetes cluster stopped answering: the V-Rising server, Grafana, Jellyfin and Linkding were all unreachable, and the Kubernetes API was down, so the cluster could not be managed at all. The outage was found by hand, about 3 hours after it began, when someone tried to use the services.

   Control plane unresponsiveService outageMonitoring outage

   Trigger

   A power outage forced the Kubernetes master node to reboot.

   Root cause

   Software defect

   Component

   GPU driverLinux kernelKubernetes API server

   Contributing factor

   Ignored early signalChange with a delayed effectSingle point of failureMonitoring blind spot

   Detection

   Manually, when someone tried to access the services; there were no automated alerts.

   Remediation

   • Connected a monitor to the master node and read the boot logs, which showed the kernel panic
   • Booted the previous kernel version, which brought the system back
   • Removed the broken Nvidia packages with sudo apt-get remove --purge '^nvidia-.*' and ran sudo dpkg --configure -a
   • Found remaining broken packages with dpkg -l | grep -v ^ii and removed them by hand
   • Ran sudo apt-get install -f and reinstalled linux-headers-$(uname -r)
   • Rebooted and confirmed the kernel panic was gone

   Action items

   • Make sure GPU driver installations fully succeed, or revert immediately when they fail
   • Move the cluster to a high-availability control plane
   • Expose the Kubernetes API through a load balancer in the HA setup
   • Set up a ticketing system to record incidents and their resolution
   • Write predefined troubleshooting workflows to shorten response time
   • Add infrastructure monitoring such as Prometheus and Grafana
   • Add blackbox HTTP and TCP checks from outside the cluster for critical services

   Lessons

   A driver installation that fails half-way can leave kernel packages broken without any visible effect until the next reboot, which may come weeks later and at the worst time. A failed DKMS build should be fixed or reverted at once. With a single master node and no monitoring from outside the cluster, one unbootable machine meant a total outage that nobody was told about; the fix itself was simple once the boot logs were seen.

   View case
3. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case
4. IL-0462 Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacement

   Shopify2021

   Symptoms

   In 2021 Shopify's KateSQL, a database-as-a-service platform that runs several hundred production MySQL instances on GKE, found that some MySQL pods started and shut down much more slowly than similar pods with the same data set. Instance replacement, the procedure that creates a new MySQL replica pod and stops the old one, showed pod creation times ranging from 10 to 30 minutes, which held up maintenance such as configuration changes and upgrades.

   Slow pod startup or shutdownHigh latency

   Root cause

   Software defect

   Component

   Linux kernelNode operating system imageDatabase

   Contributing factor

   Unpatched known bug

   Detection

   KateSQL's instance replacement operations showed inconsistent MySQL pod creation times, and the team then measured the mysql-init container and MySQL startup times across pods.

   Remediation

   • Cordoned older nodes, moved MySQL pods to new nodes through instance replacement and drained the old nodes
   • Forced the kubepods memory cgroup to empty its dead cgroups on an affected node to test the fix
   • Created new GKE 1.18 clusters and started creating MySQL pods on them

   Lessons

   The investigation was one of the longest the Database Platform group had carried out, because of the nature of the problem, how hard it was to reproduce and the lack of an established method to follow; once the cause was known, there were several ways to fix it.

   View case
5. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit

   Buffer2020

   Symptoms

   Services on Buffer's kops-managed cluster on AWS (about 60 nodes and 1500 containers) answered more slowly than they should have. The container throttling metric showed that many containers were being throttled whatever their CPU usage was, including containers that never came close to their CPU limit.

   CPU throttlingHigh latencyNode NotReady

   Trigger

   CPU limits on user-facing services, enforced through CFS quota by a Linux kernel with a bug that throttled containers even when they were well under their limit.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelNode operating system image

   Contributing factor

   Unpatched known bugUnverified assumption

   Detection

   While reviewing the container throttling metric, the team noticed many containers being throttled even though their CPU usage stayed far below their limits.

   Remediation

   • removed CPU limits from all services directly or indirectly on the users' critical path
   • moved those services to dedicated tainted nodes and gave them matching tolerations
   • set each service's CPU request to its peak usage over a few months plus a margin of more than 20%, and set memory requests the same way

   Action items

   • Move nodes to a kernel that contains the CFS throttling fix (4.19 or later), which for Buffer means a kops version whose host image is Ubuntu 20.04
   • After the kernel upgrade, check whether throttling is fully gone and update the post

   Lessons

   A CPU limit can slow a service down even when the service uses a small part of it, because of how the kernel enforces CFS quota on older kernels; the throttling metric shows it, the CPU usage graph does not. Upgrading the kernel is the safer fix. Removing limits works for latency, but only together with honest CPU and memory requests, isolation of the unbounded services, and monitoring of node usage, since unbounded containers can starve the kubelet.

   View case

By root cause1