IL-0138 Barush Mendez · 2025-03-03kubernetes
The single master node of a homelab Kubernetes cluster has an Nvidia GPU for hardware acceleration in Jellyfin. Weeks before the incident, installing the nvidia-dkms-390 driver failed because it would not build against the installed kernel, and the kernel headers and other dependencies were left unconfigured. The failure was not dealt with, and the node kept running normally until a power outage on 2025-03-03 rebooted it. On boot the kernel panicked and the node could not start.
Because it was the only master node, the Kubernetes API was gone and every hosted application became unreachable, as services relied on the master node's private IP. There were no automated alerts, so the outage was found manually after 3 hours, and recovery took until 2025-03-09, six days in total. Once the kernel panic was seen on a monitor, booting an older kernel and repairing the broken packages fixed the node. No data was lost.
Every service in the homelab Kubernetes cluster stopped answering: the V-Rising server, Grafana, Jellyfin and Linkding were all unreachable, and the Kubernetes API was down, so the cluster could not be managed at all. The outage was found by hand, about 3 hours after it began, when someone tried to use the services.
The master node could not be reached remotely. With a monitor attached, its boot logs showed a kernel panic during boot. The node had been restarted by a power outage, and it was the only master node; the hosted services also depended on its private IP address, so its failure took down everything.
Symptom Control plane unresponsiveService outageMonitoring outage
Root cause Software defect
Manually, when someone tried to access the services; there were no automated alerts.
A driver installation that fails half-way can leave kernel packages broken without any visible effect until the next reboot, which may come weeks later and at the worst time. A failed DKMS build should be fixed or reverted at once.
With a single master node and no monitoring from outside the cluster, one unbootable machine meant a total outage that nobody was told about; the fix itself was simple once the boot logs were seen.
IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025same root cause · shared GPU driver
IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limitMarcin Cuber · 2024 · sev1same root cause · shared Kubernetes API server
IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022same root cause · shared Linux kernel
IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limitBuffer · 2020same root cause · shared Linux kernel
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Linux kernel
IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS imageGoogle Cloud · 2019same root cause · shared Linux kernel
IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELBZalando · 2019same root cause · shared Kubernetes API server
IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its stateNordstrom · 2017same root cause · shared Kubernetes API server
IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outageMonzo · 2017same root cause · shared Kubernetes API server