Incident Library

Applied filters SymptomControl plane unresponsiveClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

5 results

Lit on the graph.

Barush Mendez, 2025
Render, 2024
OpenAI, 2024
OpenAI, 2024
DigitalOcean, 2024

Cases22 cases

5 to 9 of 22

IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
Barush Mendez2025
Symptoms
Every service in the homelab Kubernetes cluster stopped answering: the V-Rising server, Grafana, Jellyfin and Linkding were all unreachable, and the Kubernetes API was down, so the cluster could not be managed at all. The outage was found by hand, about 3 hours after it began, when someone tried to use the services.
Control plane unresponsive Service outage Monitoring outage
Trigger
A power outage forced the Kubernetes master node to reboot.
Root cause
Software defect
Component
GPU driver Linux kernel Kubernetes API server
Contributing factor
Ignored early signal Change with a delayed effect Single point of failure Monitoring blind spot
Detection
Manually, when someone tried to access the services; there were no automated alerts.
Remediation
Connected a monitor to the master node and read the boot logs, which showed the kernel panic
Booted the previous kernel version, which brought the system back
Removed the broken Nvidia packages with sudo apt-get remove --purge '^nvidia-.*' and ran sudo dpkg --configure -a
Found remaining broken packages with dpkg -l | grep -v ^ii and removed them by hand
Ran sudo apt-get install -f and reinstalled linux-headers-$(uname -r)
Rebooted and confirmed the kernel panic was gone
Action items
Make sure GPU driver installations fully succeed, or revert immediately when they fail
Move the cluster to a high-availability control plane
Expose the Kubernetes API through a load balancer in the HA setup
Set up a ticketing system to record incidents and their resolution
Write predefined troubleshooting workflows to shorten response time
Add infrastructure monitoring such as Prometheus and Grafana
Add blackbox HTTP and TCP checks from outside the cluster for critical services
Lessons
A driver installation that fails half-way can leave kernel packages broken without any visible effect until the next reboot, which may come weeks later and at the worst time. A failed DKMS build should be fixed or reverted at once. With a single master node and no monitoring from outside the cluster, one unbootable machine meant a total outage that nobody was told about; the fix itself was simple once the boot logs were seen.
View case
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
Render2024
Symptoms
In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.
Control plane unresponsive Deploys failing DNS resolution failures High error rate
Trigger
A memory spike in etcd overloaded the cluster's control plane.
Root cause
Capacity exhaustion
Component
etcd CoreDNS Kubernetes API server
Contributing factor
Unreviewed defaults Harmful manual action
Action items
Run CoreDNS on data plane nodes instead of control plane nodes
Run etcd on dedicated nodes
Lessons
CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running. The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.
View case
IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
OpenAI2024
Symptoms
Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.
Service outage High error rate Control plane unresponsive DNS resolution failures
Trigger
At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.
Root cause
Configuration change
Component
Kubernetes API server Client-side DNS resolver and cache Telemetry agent
Contributing factor
Unrepresentative staging No staged rollout Cache-masked failure Monitoring blind spot Limited debug access
Detection
OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.
Remediation
Scaled clusters down to reduce the total Kubernetes API load
Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
Scaled up the Kubernetes API servers
Removed the telemetry service once enough control was back
Moved traffic from impacted clusters to healthy ones
Intervened manually in clusters where simultaneous resource downloads saturated resource limits
Action items
Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster
Lessons
Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.
View case
IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
OpenAI2024
Symptoms
From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.
High error rate High latency Control plane unresponsive
Trigger
A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.
Root cause
Configuration change
Component
Kubernetes API server
Contributing factor
Excessive blast radius Limited debug access
Remediation
moved workloads out of the impacted clusters by hand
with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover
Action items
Lock namespace changes so this incident cannot recur
Freeze certain types of infrastructure deploys
Allowlist identities to prevent a recurrence of this class of incidents
Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
Improve change management for fleet-wide infrastructure changes
View case
IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycled
DigitalOcean2024
Symptoms
From 22:25 to 22:50 UTC every DigitalOcean service in the SFO3 region had network connectivity problems, as traffic between the datacenter and the backbone dropped. Traffic through the router was back by 22:30 and SFO3's control plane systems had reconnected by 22:50, and most services recovered then.
Node NotReady Control plane unresponsive Intermittent connection failures
Trigger
Most of the line cards on one SFO3 core router rebooted at the same time at 22:21 UTC because of hardware errors on the device.
Root cause
DNS or network fault
Component
Kubernetes API server kubelet
Contributing factor
–
Remediation
The crashed line cards came back online and traffic on the core router was restored
Manually recycled all affected nodes across the SFO3 DOKS fleet
Action items
Work with the router vendor's support team to find the root cause of the line card crash
Upgrade software on the core routers in SFO3
Raise thresholds for automated remediation so affected nodes are restored without manual action
Lessons
A short network fault can leave Kubernetes nodes broken long after the network itself is healthy, when the guest networking on the machines does not recover by itself. Checking node health and the API servers after the network returns, and having automated remediation that is allowed to replace many nodes at once, shortens that tail.
View case

Check your cluster for this

These 22 cases involve Kubernetes API server and etcd. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap