IL-0005 OpenAI · 2024-12-11kubernetes
OpenAI runs hundreds of Kubernetes clusters around the world. As part of a reliability effort it deployed a new telemetry service to collect detailed Kubernetes control plane metrics. The service's configuration made every node in each cluster run resource-intensive Kubernetes API operations whose cost grew with the size of the cluster. With thousands of nodes doing this at once, the API servers were overwhelmed and the control plane went down in most large clusters, which also broke DNS-based service discovery.
The change had been tested in a staging cluster on 10 December without problems, because the effect only appears in clusters above a certain size. Before the rollout the team checked CPU and memory utilization in every cluster so the new service would not disrupt running services, and tuned resource requests per cluster, but took no precautions about API server load; the rollout monitored service health but lacked sufficient cluster health monitoring. The DNS cache on each node delayed visible failures long enough for the change to reach the whole fleet.
Alerts fired a few minutes before customers were affected, but the fix, removing the service, needed the Kubernetes control plane, which the load had locked engineers out of. Teams worked in parallel: they scaled clusters down to cut the total API load, blocked network access to the Kubernetes admin APIs so that no new expensive requests arrived, and scaled up the API servers. Together this restored enough control to remove the service. Where possible, traffic was moved to healthy clusters; the first cluster recovered at 4:36 PM. Some clusters needed further manual work because many services downloading resources at once saturated resource limits.
The API recovered substantially at 5:36 PM and fully across all models at 7:38 PM PST. ChatGPT recovered substantially at 5:45 PM and fully at 7:01 PM, and Sora fully at 7:01 PM. OpenAI states that the outage was caused by this internal change, not by a security incident or a launch.
Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.
The Kubernetes API servers in most of OpenAI's large clusters were overwhelmed and the control plane went down. The data plane, which serves workloads such as model inference, can run largely without the control plane, but DNS-based service discovery depends on it, so services could no longer find each other.
The failure showed up gradually. The DNS cache on each node kept serving stale but working records, so services kept running at first while the rollout continued. As the cached records expired over the following 20 minutes, services that need live DNS resolution began to fail, and once the caches were empty the load on the DNS servers multiplied and added still more load to the control plane.
Engineers found the cause within minutes, but could not remove the offending service: that required the Kubernetes API, which was too overloaded to answer. Even after access came back, some clusters stayed degraded because many services tried to download resources at the same time and saturated resource limits.
Symptom Service outageHigh error rate
Root cause Configuration change
OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.
Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.
IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second burstsMUSINSA · 2026same root cause · shared Telemetry agent
IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clustersOpenAI · 2024same root cause · shared Kubernetes API server
IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1DigitalOcean · 2024same root cause · shared Kubernetes API server
IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSetAqua Security · 2023same root cause · shared Kubernetes API server
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Telemetry agent
IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time outDFDS · 2020same root cause · shared Kubernetes API server
IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every nodeJetstack · 2019same root cause · shared Kubernetes API server