Cases7 cases

1 to 5 of 7

1. IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUs

   Google Cloud2024

   Symptoms

   From 08:00 US/Pacific on 12 March 2024, GKE GPU nodes running Container-Optimized OS (COS) that were newly created or recreated came up with GPU drivers that did not work, so workloads that used the GPUs could not run. GKE showed elevated errors.

   GPU unavailableNode NotReady

   Root cause

   Dependency failure

   Component

   GPU driverNode operating system imageNode pool

   Contributing factor

   Automation amplification

   Detection

   The report does not say how Google detected the issue. Affected users saw "Failed to download GPU driver installer, status: 403 Forbidden" on the GPU node when installing the driver.

   Remediation

   • Halted automatic node recreations, which attempt GPU driver downloads, until the issue was mitigated
   • Worked with the driver partner to resolve access to the driver download bucket

   Lessons

   The status page report states no lessons or follow-up actions.

   View case
2. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking

   DigitalOcean2023

   Symptoms

   Customers of DigitalOcean Managed Kubernetes (DOKS) saw worker nodes go NotReady between 06:00 and 07:00 UTC and stay unavailable, taking down the services on them, until the nodes were replaced or rebooted; some nodes needed several replacements or reboots. The first sporadic reports on October 8 said replacing the node fixed it. The same thing happened again around 06:00 UTC on October 9, more widely and across many regions, and again on October 10. Only clusters on versions 1.27.6-do.0 and 1.28.2-do.0 were affected.

   Node NotReadyService outage

   Trigger

   A systemd timer in the DigitalOcean Debian image started apt upgrade daily at 06:00 UTC, with a random delay of up to 60 minutes.

   Root cause

   Configuration change

   Component

   Node operating system imageLinux kernelkubelet

   Contributing factor

   Unreviewed defaultsLimited debug accessChange with a delayed effect

   Detection

   Sporadic customer reports of failed worker nodes arrived at 07:43 UTC on October 8; internal incident response started at 07:10 UTC on October 9 after broader impact across regions.

   Remediation

   • Advised customers to replace or reboot affected worker nodes
   • Arranged SSH and console access to worker nodes while they were healthy, and live-debugged an affected node on October 10
   • Released a DaemonSet to all affected clusters that disables the apt upgrade systemd timer on every worker node
   • Released new node images without the auto-update process

   Action items

   • Disable the timer during node provisioning, then remove the DaemonSet from all clusters
   • Add a check to the internal conformance test suite for new cluster versions that the auto-upgrade process and other unwanted timers stay disabled
   • Support logging into worker nodes even when normal access paths are broken

   Lessons

   An operating system auto-update timer left enabled in a node image can change the kernel and systemd underneath Kubernetes, and a timeout that kills the upgrade half-way can leave a node without networking. Failures that start at the same time every day point to a scheduled job on the nodes, and systemd timers belong on that list next to cron jobs. When the failure takes out a node's network, an investigation that depends on logging into the node stalls; access paths that work while the node is broken shorten diagnosis.

   View case
3. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers

   Datadog2023

   Symptoms

   Between 06:00 and 07:00 UTC on 8 March 2023, Kubernetes nodes in several Datadog regions lost network connectivity at the same time, on AWS, Google Cloud and Azure. All services across multiple regions were affected, including the regions that host Datadog's CI/CD and automation tools. Datadog's regions are not connected to each other and changes are rolled out one region at a time, so a simultaneous failure on several cloud providers looked implausible at first.

   Node NotReadyService outageThroughput degradation

   Trigger

   On 7 March 2023 Ubuntu published a systemd patch for a CVE (249.11-0ubuntu3.7). Ubuntu's default unattended upgrades installed it on the nodes between 06:00 and 07:00 UTC on 8 March, and installing it restarted systemd-networkd.

   Root cause

   Breaking change in an upgrade

   Component

   systemd-networkdCiliumNode operating system image

   Contributing factor

   Unreviewed defaultsChange with a delayed effectNo staged rolloutAutomation amplification

   Remediation

   • restarted thousands of disconnected instances on Google Cloud and Azure through the cloud APIs, keeping their local disks
   • rebuilt the data lost with terminated AWS instances' local disks through replication, reprocessing and restores from backup

   Lessons

   A change in a base operating system component can stay dormant for months until something restarts that component. Here the trigger was a security patch, and the change reached every region at once because automatic OS updates ran outside the progressive, region-by-region process used for every other change. Cloud providers react differently to an instance that loses its network. Google Cloud and Azure left the instances running, so a restart recovered them with their disks; AWS replaced them quickly, which brought stateless services back fast but destroyed the local-disk data the stateful services needed.

   View case
4. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case
5. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable

   2i2c2022

   Symptoms

   On 30 August 2022 at 11:15 AM Pacific time, the UToronto JupyterHub at jupyter.utoronto.ca stopped responding and users could not reach it at all. The outage was reported through 2i2c's support desk.

   Service outageNode NotReady

   Trigger

   Microsoft Azure rolled out an Ubuntu update to all Ubuntu 18.04 nodes, which triggered a systemd bug that broke networking.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system image

   Contributing factor

   Limited debug access

   Detection

   The outage was reported at 11:15 AM through 2i2c's Freshdesk support desk because jupyter.utoronto.ca was not responding. At 11:25 AM the customer's representative forwarded the Microsoft advisory on the same ticket, which pointed the team to the cause.

   Remediation

   • Started an upgrade of the cluster's Kubernetes and node versions, which replaced all old nodes with new working ones

   Action items

   • Document how to SSH into the Kubernetes nodes
   • Find a way to be notified of Azure advisories

   Lessons

   Once the cause was known, a single action brought the hub back quickly. The team had no documented way to SSH into its Kubernetes nodes, and the customer's representative saw the Azure advisory and pointed it out on the support ticket, which saved a lot of time.

   View case

By root cause4