Cases7 cases

1 to 5 of 7

1. IL-0247 One in a million search requests took 100 times longer on Pinterest's Manas leaf nodes after moving to Kubernetes

   Pinterest2025

   Symptoms

   During performance testing of Pinterest's search system Manas on its new Kubernetes platform, a small but persistent share of search requests timed out. About one request in a million took 100 times longer than usual.

   High latencyIntermittent timeouts

   Root cause

   Software defect

   Component

   Telemetry agentLinux kernel

   Contributing factor

   Unreviewed defaults

   Detection

   Performance testing with traffic sent to a test cluster showed a small but persistent percentage of requests timing out.

   Remediation

   • Disabled cAdvisor's working set size estimation on all PinCompute nodes

   Lessons

   CPU shielding does not fully isolate a process from its neighbours. Showing early that the node image was not the cause narrowed the search and avoided dead ends. Simple blackbox debugging, breaking things until the system stops working, works as a way of bisecting the problem.

   View case
2. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS

   Datadog2022

   Symptoms

   From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

   High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing

   Trigger

   Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

   Root cause

   Configuration change

   Component

   conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library

   Contributing factor

   Change with a delayed effectRetry stormUnknown limitMonitoring blind spot

   Detection

   Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

   Remediation

   • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
   • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
   • Removed the custom gRPC reconnect parameters from the clients to restore the defaults

   Action items

   • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error

   Lessons

   The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

   View case
3. IL-0462 Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacement

   Shopify2021

   Symptoms

   In 2021 Shopify's KateSQL, a database-as-a-service platform that runs several hundred production MySQL instances on GKE, found that some MySQL pods started and shut down much more slowly than similar pods with the same data set. Instance replacement, the procedure that creates a new MySQL replica pod and stops the old one, showed pod creation times ranging from 10 to 30 minutes, which held up maintenance such as configuration changes and upgrades.

   Slow pod startup or shutdownHigh latency

   Root cause

   Software defect

   Component

   Linux kernelNode operating system imageDatabase

   Contributing factor

   Unpatched known bug

   Detection

   KateSQL's instance replacement operations showed inconsistent MySQL pod creation times, and the team then measured the mysql-init container and MySQL startup times across pods.

   Remediation

   • Cordoned older nodes, moved MySQL pods to new nodes through instance replacement and drained the old nodes
   • Forced the kubepods memory cgroup to empty its dead cgroups on an affected node to test the fix
   • Created new GKE 1.18 clusters and started creating MySQL pods on them

   Lessons

   The investigation was one of the longest the Database Platform group had carried out, because of the nature of the problem, how hard it was to reproduce and the lack of an established method to follow; once the cause was known, there were several ways to fix it.

   View case
4. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit

   Buffer2020

   Symptoms

   Services on Buffer's kops-managed cluster on AWS (about 60 nodes and 1500 containers) answered more slowly than they should have. The container throttling metric showed that many containers were being throttled whatever their CPU usage was, including containers that never came close to their CPU limit.

   CPU throttlingHigh latencyNode NotReady

   Trigger

   CPU limits on user-facing services, enforced through CFS quota by a Linux kernel with a bug that throttled containers even when they were well under their limit.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelNode operating system image

   Contributing factor

   Unpatched known bugUnverified assumption

   Detection

   While reviewing the container throttling metric, the team noticed many containers being throttled even though their CPU usage stayed far below their limits.

   Remediation

   • removed CPU limits from all services directly or indirectly on the users' critical path
   • moved those services to dedicated tainted nodes and gave them matching tolerations
   • set each service's CPU request to its peak usage over a few months plus a margin of more than 20%, and set memory requests the same way

   Action items

   • Move nodes to a kernel that contains the CFS throttling fix (4.19 or later), which for Buffer means a kops version whose host image is Ubuntu 20.04
   • After the kernel upgrade, check whether throttling is fully gone and update the post

   Lessons

   A CPU limit can slow a service down even when the service uses a small part of it, because of how the kernel enforces CFS quota on older kernels; the throttling metric shows it, the CPU usage graph does not. Upgrading the kernel is the safer fix. Removing limits works for latency, but only together with honest CPU and memory requests, isolation of the unbounded services, and monitoring of node usage, since unbounded containers can starve the kubelet.

   View case
5. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts

   Omio2020

   Symptoms

   For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

   CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures

   Trigger

   Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelkubelet

   Contributing factor

   Unknown limitMonitoring blind spotUnpatched known bug

   Detection

   Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

   Remediation

   • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters

   Action items

   • Deploy the kernel with the CFS quota bug fix across all clusters
   • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected

   Lessons

   A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

   View case

By root cause3