Cases2 cases

1 to 2 of 2

1. IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioning

   Amazon Web Services2023

   Symptoms

   From 11:49 AM PDT on 13 June 2023, Lambda function invocations in the Northern Virginia (us-east-1) Region returned more errors and took longer. Customers that triggered functions from asynchronous or streaming event sources saw their event backlogs grow because the events were not being processed.

   High error rateHigh latencyCredentials unavailable

   Trigger

   At 11:49 AM PDT, while adding capacity for normal daily traffic growth, the Lambda Frontend fleet in one cell crossed a capacity threshold that no single cell had reached before.

   Root cause

   Software defect

   Component

   Serverless function platform

   Contributing factor

   Unknown limit

   Detection

   Engineering teams were engaged immediately and began investigating; by 12:26 PM PDT they had identified the latent defect and its effect on provisioning compute capacity.

   Remediation

   • scaled the Lambda Frontend fleet down to a size that no longer triggered the latent defect

   Action items

   • Disable the Lambda Frontend scaling activities that triggered the event
   • Fix the latent bug and deploy the fix to all Regions
   • Complete action items addressing the immediate concern with cellular scaling
   • Bound all cells to a well-tested size to avoid unexpected scaling issues

   Lessons

   A fleet that scales automatically can grow into a size that was never tested, and a defect that only appears at that size stays hidden until ordinary daily growth reaches it. AWS's answer is to cap every cell at a well-tested size. For EKS users, the event shows that provisioning new clusters in us-east-1 depended on Lambda, while clusters that were already running were not affected.

   View case
2. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts

   Omio2020

   Symptoms

   For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

   CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures

   Trigger

   Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelkubelet

   Contributing factor

   Unknown limitMonitoring blind spotUnpatched known bug

   Detection

   Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

   Remediation

   • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters

   Action items

   • Deploy the kernel with the CFS quota bug fix across all clusters
   • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected

   Lessons

   A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

   View case

By root cause1