Cases2 cases

1 to 2 of 2

1. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days

   Barush Mendez2025

   Symptoms

   Every service in the homelab Kubernetes cluster stopped answering: the V-Rising server, Grafana, Jellyfin and Linkding were all unreachable, and the Kubernetes API was down, so the cluster could not be managed at all. The outage was found by hand, about 3 hours after it began, when someone tried to use the services.

   Control plane unresponsiveService outageMonitoring outage

   Trigger

   A power outage forced the Kubernetes master node to reboot.

   Root cause

   Software defect

   Component

   GPU driverLinux kernelKubernetes API server

   Contributing factor

   Ignored early signalChange with a delayed effectSingle point of failureMonitoring blind spot

   Detection

   Manually, when someone tried to access the services; there were no automated alerts.

   Remediation

   • Connected a monitor to the master node and read the boot logs, which showed the kernel panic
   • Booted the previous kernel version, which brought the system back
   • Removed the broken Nvidia packages with sudo apt-get remove --purge '^nvidia-.*' and ran sudo dpkg --configure -a
   • Found remaining broken packages with dpkg -l | grep -v ^ii and removed them by hand
   • Ran sudo apt-get install -f and reinstalled linux-headers-$(uname -r)
   • Rebooted and confirmed the kernel panic was gone

   Action items

   • Make sure GPU driver installations fully succeed, or revert immediately when they fail
   • Move the cluster to a high-availability control plane
   • Expose the Kubernetes API through a load balancer in the HA setup
   • Set up a ticketing system to record incidents and their resolution
   • Write predefined troubleshooting workflows to shorten response time
   • Add infrastructure monitoring such as Prometheus and Grafana
   • Add blackbox HTTP and TCP checks from outside the cluster for critical services

   Lessons

   A driver installation that fails half-way can leave kernel packages broken without any visible effect until the next reboot, which may come weeks later and at the worst time. A failed DKMS build should be fixed or reverted at once. With a single master node and no monitoring from outside the cluster, one unbootable machine meant a total outage that nobody was told about; the fix itself was simple once the boot logs were seen.

   View case
2. IL-0041 fluent-bit silently stopped shipping application logs from one node to Elasticsearch

   PrometheusKube2020

   Symptoms

   On a Tuesday morning a developer reported that some production logs were missing. Only the logs from a single node were gone; the other nodes kept shipping normally through fluent-bit, the log shipper the team had run for more than three years.

   Monitoring outage

   Root cause

   Software defect

   Component

   Telemetry agentElasticsearch or OpenSearch

   Contributing factor

   Monitoring blind spotUnpatched known bug

   Detection

   A developer reported missing production logs; no alert fired because fluent-bit's metrics showed no errors.

   Remediation

   • added CPU to the Elasticsearch virtual machines, which cleared the write queue rejections
   • rolled out Fluentd one node at a time, using node affinity to place Fluentd and anti-affinity to remove fluent-bit
   • lowered Fluentd's flush_thread_count to 1 and set flush_interval to 5 seconds so the backlog of old logs did not overflow the buffer
   • enabled Fluentd's Position DB so already-ingested data was skipped

   Action items

   • Build monitoring for Fluentd on its Prometheus integration, including a throughput alert with separate day and night thresholds or a black-box log check

   Lessons

   A log shipper can fail without an error message and with metrics that simply stop moving, so error-based alerts never fire; throughput alerts or end-to-end checks can catch it. Open source software can break in ways nobody expects, and sometimes the way out is to switch to a different shipper fast, node by node. A new shipper that starts from scratch replays every old log on the node and needs to be slowed down.

   View case

By root cause1