Cases4 cases

1 to 4 of 4

1. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days

   Barush Mendez2025

   Symptoms

   Every service in the homelab Kubernetes cluster stopped answering: the V-Rising server, Grafana, Jellyfin and Linkding were all unreachable, and the Kubernetes API was down, so the cluster could not be managed at all. The outage was found by hand, about 3 hours after it began, when someone tried to use the services.

   Control plane unresponsiveService outageMonitoring outage

   Trigger

   A power outage forced the Kubernetes master node to reboot.

   Root cause

   Software defect

   Component

   GPU driverLinux kernelKubernetes API server

   Contributing factor

   Ignored early signalChange with a delayed effectSingle point of failureMonitoring blind spot

   Detection

   Manually, when someone tried to access the services; there were no automated alerts.

   Remediation

   • Connected a monitor to the master node and read the boot logs, which showed the kernel panic
   • Booted the previous kernel version, which brought the system back
   • Removed the broken Nvidia packages with sudo apt-get remove --purge '^nvidia-.*' and ran sudo dpkg --configure -a
   • Found remaining broken packages with dpkg -l | grep -v ^ii and removed them by hand
   • Ran sudo apt-get install -f and reinstalled linux-headers-$(uname -r)
   • Rebooted and confirmed the kernel panic was gone

   Action items

   • Make sure GPU driver installations fully succeed, or revert immediately when they fail
   • Move the cluster to a high-availability control plane
   • Expose the Kubernetes API through a load balancer in the HA setup
   • Set up a ticketing system to record incidents and their resolution
   • Write predefined troubleshooting workflows to shorten response time
   • Add infrastructure monitoring such as Prometheus and Grafana
   • Add blackbox HTTP and TCP checks from outside the cluster for critical services

   Lessons

   A driver installation that fails half-way can leave kernel packages broken without any visible effect until the next reboot, which may come weeks later and at the worst time. A failed DKMS build should be fixed or reverted at once. With a single master node and no monitoring from outside the cluster, one unbootable machine meant a total outage that nobody was told about; the fix itself was simple once the boot logs were seen.

   View case
2. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case
3. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS

   Datadog2022

   Symptoms

   From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

   High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing

   Trigger

   Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

   Root cause

   Configuration change

   Component

   conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library

   Contributing factor

   Change with a delayed effectRetry stormUnknown limitMonitoring blind spot

   Detection

   Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

   Remediation

   • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
   • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
   • Removed the custom gRPC reconnect parameters from the clients to restore the defaults

   Action items

   • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error

   Lessons

   The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

   View case
4. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts

   Omio2020

   Symptoms

   For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

   CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures

   Trigger

   Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelkubelet

   Contributing factor

   Unknown limitMonitoring blind spotUnpatched known bug

   Detection

   Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

   Remediation

   • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters

   Action items

   • Deploy the kernel with the CFS quota bug fix across all clusters
   • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected

   Lessons

   A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

   View case

By root cause2