IL-0117 OpenAI · 2024-08-16networking
OpenAI attributes the incident to a combination of factors. A scheduled maintenance and an upgrade to the ingress of the user-facing clusters introduced a regression in the networking control plane, which caused a short outage of the data plane. Because of the momentary loss of connectivity, a set of services became unhealthy and were restarted automatically.
The restarts took much longer than expected. Each service ran a heavy query on first start, and so many services starting together overwhelmed a backend persistence store, which then needed extra time to catch up and recover.
Afterwards OpenAI mitigated the control plane regression and checked that control plane restarts no longer disturb the data plane, removed the expensive query from critical startup paths, and changed configuration so that the networking control plane affects the clusters' ability to handle traffic less.
On August 16, 2024 from 11:38 AM to 1:16 PM PT, the OpenAI primary API was unreliable: ChatGPT conversations succeeded less often, and logging in and creating accounts were affected. The status page called it elevated error rates for ChatGPT and the Platform API. The incident came in two waves, of 44 and 15 minutes.
Underneath, connectivity was lost for a moment, a set of services became unhealthy and were restarted automatically, and the restarts took much longer than expected because the starting services overwhelmed a backend persistence store with a heavy query they ran on first start.
Symptom High error rateDatabase overload
Root cause Breaking change in an upgrade
–
–
A brief connectivity loss turned into a long outage because recovery itself was expensive: the services that restarted overwhelmed one backend persistence store with a heavy first-start query. Keeping costly queries off startup paths shortens recovery, and staged rollouts with longer soak time limit how many systems an infrastructure regression reaches before it is caught.
IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates2i2c · 2026same root cause · shared Ingress controller
IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applicationsUK Ministry of Justice · 2026same root cause · shared Ingress controller
IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replacedHoneycomb · 2024same root cause · shared Ingress controller
IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and changeUK Ministry of Justice · 2021same root cause · shared Ingress controller
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.