IL-0118 OpenAI · 2024-11-25kubernetesai-infra
A global change to Kubernetes namespace labels, without anyone intending it, made the networking layer recompute metadata. The small clusters handled this safely, but in three of the largest GPU clusters the recomputation overwhelmed the Kubernetes control plane. That led to cascading failures, high latency and elevated error rates across several OpenAI products.
OpenAI first reduced the impact by moving workloads out of the affected clusters by hand, which the unresponsive control planes made harder. After the root cause was found, OpenAI worked with its cloud provider to scale up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover. Some major workloads have since moved back into those clusters with additional protections.
From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.
ChatGPT was degraded for Paid and Enterprise users from 10:15 am to 12 pm, with error rates peaking at 13% for Paid and 23% for Enterprise; Free users saw degraded performance until 1:20 pm. Search in ChatGPT was disabled between 11:15 am and 12:05 pm. In the API, gpt-4-turbo-preview, gpt-4-o125-preview and text-embedding-3-large were hit hardest, with higher latency and error rates until as late as 1 pm.
The control planes of three of the largest GPU clusters were unresponsive, which made moving workloads out of those clusters by hand more complicated.
Symptom High error rateHigh latencyControl plane unresponsive
Root cause Configuration change
–
–
IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI servicesOpenAI · 2024same root cause · shared Kubernetes API server
IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1DigitalOcean · 2024same root cause · shared Kubernetes API server
IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSetAqua Security · 2023same root cause · shared Kubernetes API server
IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time outDFDS · 2020same root cause · shared Kubernetes API server
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Kubernetes API server