Cases13 cases

6 to 10 of 13

1. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services

   OpenAI2024

   Symptoms

   Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.

   Service outageHigh error rateControl plane unresponsiveDNS resolution failures

   Trigger

   At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.

   Root cause

   Configuration change

   Component

   Kubernetes API serverClient-side DNS resolver and cacheTelemetry agent

   Contributing factor

   Unrepresentative stagingNo staged rolloutCache-masked failureMonitoring blind spotLimited debug access

   Detection

   OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.

   Remediation

   • Scaled clusters down to reduce the total Kubernetes API load
   • Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
   • Scaled up the Kubernetes API servers
   • Removed the telemetry service once enough control was back
   • Moved traffic from impacted clusters to healthy ones
   • Intervened manually in clusters where simultaneous resource downloads saturated resource limits

   Action items

   • Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
   • Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
   • Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
   • Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
   • Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster

   Lessons

   Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.

   View case
2. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query

   OpenAI2024

   Symptoms

   On August 16, 2024 from 11:38 AM to 1:16 PM PT, the OpenAI primary API was unreliable: ChatGPT conversations succeeded less often, and logging in and creating accounts were affected. The status page called it elevated error rates for ChatGPT and the Platform API. The incident came in two waves, of 44 and 15 minutes.

   High error rateDatabase overload

   Trigger

   A scheduled maintenance combined with an upgrade to the ingress of OpenAI's user-facing clusters.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerApplication database

   Contributing factor

   No staged rolloutAutomation amplification

   Action items

   • Mitigate the networking control plane regression and validate that control plane restarts do not interfere with the clusters' data plane
   • Change service software to start faster and remove the first-start query, relieving the persistence layer
   • Deploy configuration changes that optimise the networking control plane's effect on the clusters' ability to handle traffic
   • Remove the expensive database query from critical startup paths
   • Add monitoring and alerting for networking control plane issues
   • Introduce staged rollouts with longer soak time for infrastructure changes
   • Audit systems for other slow queries that may affect service start time

   Lessons

   A brief connectivity loss turned into a long outage because recovery itself was expensive: the services that restarted overwhelmed one backend persistence store with a heavy first-start query. Keeping costly queries off startup paths shortens recovery, and staged rollouts with longer soak time limit how many systems an infrastructure regression reaches before it is caught.

   View case
3. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR

   Scaleway2024

   Symptoms

   At 01:04 CET on 13 February 2024, the Kubernetes API behind Scaleway's managed Kubernetes services Kapsule and Kosmos in the Paris region (FR-PAR) spiked in RAM and CPU use and ran out of memory. Nodes in FR-PAR could no longer authenticate to their control plane, and some of them went "Not Ready". Customers on both shared (mutualized) and dedicated environments were affected.

   Node NotReadyControl plane lagOOMKilled

   Trigger

   A new Kubernetes release was made available to all users on 12 February 2024, and during the night of 12 to 13 February many clusters triggered their auto-upgrade at the same time.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API server

   Contributing factor

   Retry stormAutomation amplificationNo staged rollout

   Detection

   The timeline's first entry, at 01:04 CET, records the Kubernetes API's RAM and CPU spikes and out-of-memory state, with nodes unable to authenticate to their control plane and some becoming "Not Ready". The first recorded action, 42 minutes later at 01:46, was to increase the resources allocated to the API.

   Remediation

   • increased the resources allocated to the Kubernetes API at 01:46 and again at 02:20
   • applied API traffic restrictions at 02:30 to smooth the incoming traffic
   • removed the traffic restrictions progressively in 50-100 batches from 03:20
   • ran a script at 03:40 to replace the failing nodes created during the congestion
   • rolled back to the former node authentication mechanism, first on a subset of clusters from 15:45 and on all clusters by 19:41 on 19 February
   • disabled cluster upgrade and configuration update features until 10:30 on 20 February

   Action items

   • Increase the resources allocated to the Kubernetes API
   • Roll back to the former node authentication mechanism
   • Implement a local authentication cache on nodes
   • Rework the resource allocation strategy to behave better under load spikes
   • Add new request queuing mechanisms for Kubernetes
   • Separate the different components to reduce side effects under high load or failure
   • Monitor selected metrics closely to identify anomalous patterns earlier
   • Refund customers who deployed nodes during the incident but could not use them normally

   Lessons

   Letting many clusters auto-upgrade at the same moment after a release turns routine upgrades into one coordinated load spike on the shared Kubernetes API. A node authentication mechanism that retries without letting up turns a slow API into an unavailable one, and auto-healing that replaces NotReady nodes amplifies the problem. The congestion pattern had already appeared three weeks earlier, and short-term mitigations did not stop it from recurring, so Scaleway rolled back the new authentication mechanism. Scaleway notes that as the number of customer clusters and nodes grows sharply, traffic control mechanisms have to improve to prevent congestion.

   View case
4. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers

   Datadog2023

   Symptoms

   Between 06:00 and 07:00 UTC on 8 March 2023, Kubernetes nodes in several Datadog regions lost network connectivity at the same time, on AWS, Google Cloud and Azure. All services across multiple regions were affected, including the regions that host Datadog's CI/CD and automation tools. Datadog's regions are not connected to each other and changes are rolled out one region at a time, so a simultaneous failure on several cloud providers looked implausible at first.

   Node NotReadyService outageThroughput degradation

   Trigger

   On 7 March 2023 Ubuntu published a systemd patch for a CVE (249.11-0ubuntu3.7). Ubuntu's default unattended upgrades installed it on the nodes between 06:00 and 07:00 UTC on 8 March, and installing it restarted systemd-networkd.

   Root cause

   Breaking change in an upgrade

   Component

   systemd-networkdCiliumNode operating system image

   Contributing factor

   Unreviewed defaultsChange with a delayed effectNo staged rolloutAutomation amplification

   Remediation

   • restarted thousands of disconnected instances on Google Cloud and Azure through the cloud APIs, keeping their local disks
   • rebuilt the data lost with terminated AWS instances' local disks through replication, reprocessing and restores from backup

   Lessons

   A change in a base operating system component can stay dormant for months until something restarts that component. Here the trigger was a security patch, and the change reached every region at once because automatic OS updates ran outside the progressive, region-by-region process used for every other change. Cloud providers react differently to an instance that loses its network. Google Cloud and Azure left the instances running, so a restart recovered them with their disks; AWS replaced them quickly, which brought stateless services back fast but destroyed the local-disk data the stateful services needed.

   View case
5. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case

By root cause4