Incident Library

Applied filters ComponentKubernetes API serverSymptomHigh error rateClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

5 results

Lit on the graph.

Render, 2024
OpenAI, 2024
OpenAI, 2024
Reddit, 2023
Zalando, 2019

Cases8 cases

1 to 5 of 8

IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
Render2024
Symptoms
In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.
Control plane unresponsive Deploys failing DNS resolution failures High error rate
Trigger
A memory spike in etcd overloaded the cluster's control plane.
Root cause
Capacity exhaustion
Component
etcd CoreDNS Kubernetes API server
Contributing factor
Unreviewed defaults Harmful manual action
Action items
Run CoreDNS on data plane nodes instead of control plane nodes
Run etcd on dedicated nodes
Lessons
CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running. The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.
View case
IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
OpenAI2024
Symptoms
Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.
Service outage High error rate Control plane unresponsive DNS resolution failures
Trigger
At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.
Root cause
Configuration change
Component
Kubernetes API server Client-side DNS resolver and cache Telemetry agent
Contributing factor
Unrepresentative staging No staged rollout Cache-masked failure Monitoring blind spot Limited debug access
Detection
OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.
Remediation
Scaled clusters down to reduce the total Kubernetes API load
Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
Scaled up the Kubernetes API servers
Removed the telemetry service once enough control was back
Moved traffic from impacted clusters to healthy ones
Intervened manually in clusters where simultaneous resource downloads saturated resource limits
Action items
Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster
Lessons
Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.
View case
IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
OpenAI2024
Symptoms
From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.
High error rate High latency Control plane unresponsive
Trigger
A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.
Root cause
Configuration change
Component
Kubernetes API server
Contributing factor
Excessive blast radius Limited debug access
Remediation
moved workloads out of the impacted clusters by hand
with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover
Action items
Lock namespace changes so this incident cannot recur
Freeze certain types of infrastructure deploys
Allowlist identities to prevent a recurrence of this class of incidents
Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
Improve change management for fleet-wide infrastructure changes
View case
IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
Reddit2023
Symptoms
About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.
Service outage High error rate DNS resolution failures Monitoring outage Pods stuck in ContainerCreating Control plane lag
Trigger
An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.
Root cause
Breaking change in an upgrade
Component
Calico Kubernetes API server Admission webhook etcd
Contributing factor
Configuration not in code Unrepresentative staging Untested backup Monitoring blind spot Single point of failure
Detection
The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.
Remediation
deleted the Open Policy Agent webhook configurations to stop API server timeouts
terminated all worker nodes of the cluster
restored the control plane from the pre-upgrade backup onto the first control plane node
switched off the cluster autoscaler until the control plane was stable
generated new TLS certificates so replacement control plane nodes could join etcd
brought traffic back in steps from 1% to 100%
restarted legacy services that had been stopped by hand
Action items
Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
Standardise components across clusters and replace bespoke configurations
Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
Add pre-production clusters that can be tested with confidence
Improve the backup restore process and make cluster upgrades safe
Lessons
Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.
View case
IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
Zalando2019
Symptoms
Builds on Zalando's in-house Continuous Delivery Platform (CDP), which runs CI/CD on Kubernetes, timed out and failed for almost an hour on 11 June 2019. The Kubernetes builder pods could not get AWS IAM credentials.
High error rate Credentials unavailable Control plane lag
Trigger
An unlucky rescaling of the CDP Kubernetes cluster left only one node available for builder pods. Builder pods were created and deleted on that node so quickly that kubelet fell behind.
Root cause
Capacity exhaustion
Component
kubelet Credential broker for pods Kubernetes API server CI system
Contributing factor
Unreviewed defaults Single point of failure
Remediation
Scaled the CDP cluster up by hand so builder pods spread over more than one node
Action items
Raise the kubelet QPS limit from the default 5 to 50 in the global cluster configuration
View case

Check your cluster for this

These 8 cases involve Kubernetes API server and etcd. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Graph

Cases8 cases

By root cause4

Capacity exhaustion4 cases

Configuration change2 cases

Breaking change in an upgrade1 case

Software defect1 case