Incident Library
Applied filters Root causeSoftware defectFactorIgnored early signalDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
    Symptoms

    Every service in the homelab Kubernetes cluster stopped answering: the V-Rising server, Grafana, Jellyfin and Linkding were all unreachable, and the Kubernetes API was down, so the cluster could not be managed at all. The outage was found by hand, about 3 hours after it began, when someone tried to use the services.

    Control plane unresponsiveService outageMonitoring outage
    Trigger

    A power outage forced the Kubernetes master node to reboot.

    Root cause
    Software defect
    Component
    GPU driverLinux kernelKubernetes API server
    Contributing factor
    Ignored early signalChange with a delayed effectSingle point of failureMonitoring blind spot
    Detection

    Manually, when someone tried to access the services; there were no automated alerts.

    Remediation
    • Connected a monitor to the master node and read the boot logs, which showed the kernel panic
    • Booted the previous kernel version, which brought the system back
    • Removed the broken Nvidia packages with sudo apt-get remove --purge '^nvidia-.*' and ran sudo dpkg --configure -a
    • Found remaining broken packages with dpkg -l | grep -v ^ii and removed them by hand
    • Ran sudo apt-get install -f and reinstalled linux-headers-$(uname -r)
    • Rebooted and confirmed the kernel panic was gone
    Action items
    • Make sure GPU driver installations fully succeed, or revert immediately when they fail
    • Move the cluster to a high-availability control plane
    • Expose the Kubernetes API through a load balancer in the HA setup
    • Set up a ticketing system to record incidents and their resolution
    • Write predefined troubleshooting workflows to shorten response time
    • Add infrastructure monitoring such as Prometheus and Grafana
    • Add blackbox HTTP and TCP checks from outside the cluster for critical services
    Lessons

    A driver installation that fails half-way can leave kernel packages broken without any visible effect until the next reboot, which may come weeks later and at the worst time. A failed DKMS build should be fixed or reverted at once. With a single master node and no monitoring from outside the cluster, one unbootable machine meant a total outage that nobody was told about; the fix itself was simple once the boot logs were seen.

    View case
  2. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020
    Symptoms

    A production service at Airbnb showed a high error rate and could not be deployed. The deploy failed with a cryptic Kubernetes error about being unable to access invalid index 5.

    High error rateThroughput degradationDeploys failing
    Trigger

    A change to the mutating admission controller, which had been rolled out within the previous month, had gone in seven days before the incident surfaced; when it had to delete two items from an array in a spec, it generated an invalid JSON patch that removed index 2 and then index 3.

    Root cause
    Software defect
    Component
    Admission webhookHorizontal pod autoscaler
    Contributing factor
    Change with a delayed effectMonitoring blind spotIgnored early signal
    Detection

    The affected service reported a high error rate and its deploy failed; the slow degradation of other services over the previous seven days had gone unnoticed.

    Remediation
    • deleted the mutating admission controller from the affected cluster
    Action items
    • Create alerts from the existing metrics that would have shown the problem earlier
    Lessons

    Keep in mind that a mutating admission controller exists in the cluster; a recently added one is easy to forget during an incident. Search for the exact error message early instead of working on the visible symptoms. Page a teammate when progress stalls.

    View case
  3. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order
    Skyscanner2019
    Symptoms

    On a Sunday, a team using one of Skyscanner's Kubernetes clusters manually paged the out-of-hours Kubernetes support because their service showed elevated latency, worst at the p90 and maximum levels.

    High latencyLoad imbalanceOOMKilledIntermittent connection failures
    Trigger

    Upgrading the HAProxy ingress controller to v0.6 and going back from Service virtual IPs to individual pod IPs as backends, on the understanding that the sort-backends flag introduced in v0.5, false by default, meant backends were randomly shuffled.

    Root cause
    Software defect
    Component
    Ingress controllerHAProxyConfiguration templating
    Contributing factor
    Unrepresentative stagingUnverified assumptionIgnored early signal
    Detection

    A cluster user manually paged the out-of-hours Kubernetes support about elevated latency; Prometheus queries then showed the uneven request distribution across pods.

    Remediation
    • switched the affected service from pod IPs back to the Service virtual IP through a config change in the internal deployment tool Slingshot
    • patched the controller to keep each backend's servers in an insertion-ordered slice and changed the template to iterate over that slice
    • tested the patch in sandbox clusters and rolled it out to the production clusters
    Action items
    • Run the internal controller patch in production and return to pod IPs as the default backends
    • Raise the problem upstream with a GitHub issue and pull request
    • Test changes to cluster components against the specific behaviour and conditions that matter
    Lessons

    The team already knew that sorted backends cause hot-podding and still did not test the new flag well enough to see that it did not work; changes to cluster components need tests aimed at the exact behaviour that matters, under conditions like production. A keyword can behave differently in different parts of a language: range over a map is unordered in Go code but sorted in text/template. A small cluster component can do outsized harm to every user of the cluster and, in the end, to customers.

    View case
Check your cluster for this

These 3 cases involve Admission webhook and Configuration templating. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect3 cases

Component
Admission webhook1Configuration templating1GPU driver1HAProxy1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1Linux kernel1
Contributing factor
Ignored early signal3Change with a delayed effect2Monitoring blind spot2Unverified assumption1Single point of failure1Unrepresentative staging1
Organisation
Airbnb1Barush Mendez1Skyscanner1
  1. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  2. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020
  3. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order
    Skyscanner2019