Incident Library

Applied filters ComponentKubernetes API serverClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

5 results

Lit on the graph.

Barush Mendez, 2025
Render, 2024
OpenAI, 2024
OpenAI, 2024
Namespace, 2024

Cases28 cases

3 to 7 of 28

IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
Barush Mendez2025
Symptoms
Every service in the homelab Kubernetes cluster stopped answering: the V-Rising server, Grafana, Jellyfin and Linkding were all unreachable, and the Kubernetes API was down, so the cluster could not be managed at all. The outage was found by hand, about 3 hours after it began, when someone tried to use the services.
Control plane unresponsive Service outage Monitoring outage
Trigger
A power outage forced the Kubernetes master node to reboot.
Root cause
Software defect
Component
GPU driver Linux kernel Kubernetes API server
Contributing factor
Ignored early signal Change with a delayed effect Single point of failure Monitoring blind spot
Detection
Manually, when someone tried to access the services; there were no automated alerts.
Remediation
Connected a monitor to the master node and read the boot logs, which showed the kernel panic
Booted the previous kernel version, which brought the system back
Removed the broken Nvidia packages with sudo apt-get remove --purge '^nvidia-.*' and ran sudo dpkg --configure -a
Found remaining broken packages with dpkg -l | grep -v ^ii and removed them by hand
Ran sudo apt-get install -f and reinstalled linux-headers-$(uname -r)
Rebooted and confirmed the kernel panic was gone
Action items
Make sure GPU driver installations fully succeed, or revert immediately when they fail
Move the cluster to a high-availability control plane
Expose the Kubernetes API through a load balancer in the HA setup
Set up a ticketing system to record incidents and their resolution
Write predefined troubleshooting workflows to shorten response time
Add infrastructure monitoring such as Prometheus and Grafana
Add blackbox HTTP and TCP checks from outside the cluster for critical services
Lessons
A driver installation that fails half-way can leave kernel packages broken without any visible effect until the next reboot, which may come weeks later and at the worst time. A failed DKMS build should be fixed or reverted at once. With a single master node and no monitoring from outside the cluster, one unbootable machine meant a total outage that nobody was told about; the fix itself was simple once the boot logs were seen.
View case
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
Render2024
Symptoms
In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.
Control plane unresponsive Deploys failing DNS resolution failures High error rate
Trigger
A memory spike in etcd overloaded the cluster's control plane.
Root cause
Capacity exhaustion
Component
etcd CoreDNS Kubernetes API server
Contributing factor
Unreviewed defaults Harmful manual action
Action items
Run CoreDNS on data plane nodes instead of control plane nodes
Run etcd on dedicated nodes
Lessons
CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running. The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.
View case
IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
OpenAI2024
Symptoms
Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.
Service outage High error rate Control plane unresponsive DNS resolution failures
Trigger
At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.
Root cause
Configuration change
Component
Kubernetes API server Client-side DNS resolver and cache Telemetry agent
Contributing factor
Unrepresentative staging No staged rollout Cache-masked failure Monitoring blind spot Limited debug access
Detection
OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.
Remediation
Scaled clusters down to reduce the total Kubernetes API load
Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
Scaled up the Kubernetes API servers
Removed the telemetry service once enough control was back
Moved traffic from impacted clusters to healthy ones
Intervened manually in clusters where simultaneous resource downloads saturated resource limits
Action items
Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster
Lessons
Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.
View case
IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
OpenAI2024
Symptoms
From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.
High error rate High latency Control plane unresponsive
Trigger
A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.
Root cause
Configuration change
Component
Kubernetes API server
Contributing factor
Excessive blast radius Limited debug access
Remediation
moved workloads out of the impacted clusters by hand
with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover
Action items
Lock namespace changes so this incident cannot recur
Freeze certain types of infrastructure deploys
Allowlist identities to prevent a recurrence of this class of incidents
Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
Improve change management for fleet-wide infrastructure changes
View case
IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU
Namespace2024
Symptoms
From 17 October 2024, packets through the vSwitch that joined Namespace's bare metal machines at one provider in the fra3 region were dropped at random. Retries in TCP, QUIC and the Namespace control plane hid most of it, but the SLO error budget for latency targets on key interactions began to burn.
Intermittent connection failures High latency Control plane lag Service outage
Trigger
Random packet drops on the bare metal provider's vSwitch from 17 October 2024; for the second outage, a rate of Kubernetes changes that the slow NVMe on one control plane node could no longer keep up with.
Root cause
Cloud provider outage
Component
Kubernetes API server
Contributing factor
No capacity planning No headroom Overlapping failures
Detection
SLO-based monitoring of latency targets showed the error budget burning while retries hid the individual packet drops.
Remediation
diverted all customer traffic from fra3 to other regions
connected each provider machine to the zrh1 cluster over Wireguard instead of the vSwitch
repaved every fra3 machine with a new identity, certificates and encryption-at-rest keys, dropping regional caches to recover faster
cloned existing datasets in a more distributed way to speed up repaving the machines
added machines back to the new region in batches and returned the region to the compute pool
redirected workloads away from zrh1 during the second outage
cloned the failing control plane node's key state to a higher-capacity node that took over its role and IP address
Action items
Credit all instances that ran during the outages back to customers
Build up backup regions and solutions so a similar event has options that do not affect customers
Lessons
Running hot with a delayed hardware delivery left no room to take a region out, so fixing a provider network fault turned into a global capacity shortage. After one outage the team was anchored on its cause and first looked for network problems in the second outage, which had a different cause: a control plane node whose disk could not keep up with the Kubernetes change rate.
View case

Check your cluster for this

These 28 cases involve Kubernetes API server and etcd. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap