Cases3 cases

1 to 3 of 3

1. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days

   Barush Mendez2025

   Symptoms

   Every service in the homelab Kubernetes cluster stopped answering: the V-Rising server, Grafana, Jellyfin and Linkding were all unreachable, and the Kubernetes API was down, so the cluster could not be managed at all. The outage was found by hand, about 3 hours after it began, when someone tried to use the services.

   Control plane unresponsiveService outageMonitoring outage

   Trigger

   A power outage forced the Kubernetes master node to reboot.

   Root cause

   Software defect

   Component

   GPU driverLinux kernelKubernetes API server

   Contributing factor

   Ignored early signalChange with a delayed effectSingle point of failureMonitoring blind spot

   Detection

   Manually, when someone tried to access the services; there were no automated alerts.

   Remediation

   • Connected a monitor to the master node and read the boot logs, which showed the kernel panic
   • Booted the previous kernel version, which brought the system back
   • Removed the broken Nvidia packages with sudo apt-get remove --purge '^nvidia-.*' and ran sudo dpkg --configure -a
   • Found remaining broken packages with dpkg -l | grep -v ^ii and removed them by hand
   • Ran sudo apt-get install -f and reinstalled linux-headers-$(uname -r)
   • Rebooted and confirmed the kernel panic was gone

   Action items

   • Make sure GPU driver installations fully succeed, or revert immediately when they fail
   • Move the cluster to a high-availability control plane
   • Expose the Kubernetes API through a load balancer in the HA setup
   • Set up a ticketing system to record incidents and their resolution
   • Write predefined troubleshooting workflows to shorten response time
   • Add infrastructure monitoring such as Prometheus and Grafana
   • Add blackbox HTTP and TCP checks from outside the cluster for critical services

   Lessons

   A driver installation that fails half-way can leave kernel packages broken without any visible effect until the next reboot, which may come weeks later and at the worst time. A failed DKMS build should be fixed or reverted at once. With a single master node and no monitoring from outside the cluster, one unbootable machine meant a total outage that nobody was told about; the fix itself was simple once the boot logs were seen.

   View case
2. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data

   Palark2020

   Symptoms

   Right after an updated RedisFailover resource was rolled out, the large Redis database lost all its data and the service went down; the data had to be restored from a backup taken just before the change.

   Data lossService outage

   Trigger

   A RedisFailover change that raised the number of Redis replicas from one to two and, in the same rollout, lowered the containers' memory requests and limits from 22 GB to 19 GB.

   Root cause

   Software defect

   Component

   Application databasePersistent volumes and stateful setsContainer memory limit

   Contributing factor

   No staged rolloutSingle point of failureUnverified assumption

   Remediation

   • restored the Redis data from the backup prepared before the rollout

   Action items

   • Report the readiness probe flaw to the Redis Operator project and test the developers' fix
   • Reconsider the use of the Spotahome Redis Operator
   • Clean up the Redis databases and fix the application code that created the excess keys
   • Consider Redis built-in clustering and sharding

   Lessons

   Changing the replica count and the pod resources in one rollout turned a seemingly harmless change into a rolling restart of the only pod with data while the new replica was still empty. A readiness probe that only checks that Redis answers a ping does not tell whether a replica has finished syncing. Operators for critical stateful components need to be evaluated for maturity before production use, and the dump.rdb file should be backed up before any change to the Redis cluster.

   View case
3. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage

   Monzo2017

   Symptoms

   At 14:10 an engineer deployed a change to a service that Monzo's current account needs to process payments, and every request to that service started to fail. Current account customers saw payment failures; the prepaid card was not affected at first because it does not use that service. The change was rolled back at 14:12, but the errors continued.

   High error rateStale endpoint routingService outage

   Trigger

   At 14:10 an engineer deployed a change to a service needed to process current account payments. Two weeks earlier the etcd cluster had been upgraded and grown from three to nine nodes, and one day earlier another team had scaled a new service down to zero replicas while leaving its Kubernetes service in place.

   Root cause

   Software defect

   Component

   etcdKubernetes API serverLinkerdkubelet

   Contributing factor

   Ineffective rollbackHarmful manual actionOverlapping failuresMonitoring blind spotSingle point of failure

   Detection

   Payment failures began right after the 14:10 deploy. The team rolled back at 14:12 and declared an outage internally at 14:16. At 15:13, when no service could serve requests, practically all automated alerts fired.

   Remediation

   • Rolled back the payments service change (the errors continued)
   • Activated internal backup processes so most customers could still use their card
   • Restarted all linkerd instances (this made the outage worse)
   • Restarted the three Kubernetes API server processes so replacement linkerd instances could start
   • Began deploying a newer linkerd version with the fix for empty services
   • Deleted the Kubernetes service that had no endpoints, which let linkerd load service discovery again

   Action items

   • Fix the bug in Kubernetes that can trigger timeouts after a cluster reconfiguration
   • Roll out a new version of linkerd that fixes the parsing error
   • Create better health checks, dashboards and alerts for the affected components so they give clearer signals about what is wrong and prevent human error
   • Improve procedures to communicate outages internally and externally as clearly and quickly as possible

   Lessons

   A large failure in a distributed system can be very hard to understand, and well-meant human action can compound it, as restarting every linkerd instance did here by exposing a second incompatibility. Because Monzo deploys often and its network changes constantly, the deployment and request-routing systems have to be trustworthy. Monzo committed to publishing a similar post-mortem for any incident of this severity.

   View case

By root cause1