IL-0123 OpenAI · 2023-11-08ai-infra
OpenAI's routing layer had a chronic memory problem: its code allocated a new responseBuffer on every pass of a loop instead of reusing one, and garbage collection fell behind, especially under many requests. On a morning with dramatically more completions than any earlier day, routing nodes reached their memory limits and failed readiness checks, until too few nodes were left to carry the traffic and the service could not recover.
OpenAI ended the outage by limiting incoming traffic, redeploying the service en masse and then ramping traffic back up slowly. Afterwards the code was changed to pre-allocate the buffer and reuse it, which brought a 3X improvement in both memory and CPU usage. The memory limits were set to an appropriate level with significant headroom, rate limit controls were added to shed load more gracefully, and capacity was increased. Autoscaling had not been possible for this service because adding capacity could hurt an upstream service; that issue has been fixed and autoscaling is to be configured.
Between 5:42 AM and 7:16 AM PT on 8 November 2023 a large share of requests to OpenAI failed with 502 or 503 errors. Every model and API endpoint saw significant failures.
Nodes of the routing layer hit their memory limits and failed readiness checks. As more of them became unavailable, the remaining nodes did not have the capacity to serve incoming traffic, and the service could not recover on its own. That morning had dramatically more completions than any day before.
Root cause Software defect
–
–
IL-0022 Three overlapping infrastructure bugs (context-window misrouting, TPU output corruption and an XLA top-k miscompilation) degraded Claude's responsesAnthropic · 2025same root cause · shared Model request routing
IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutesGoogle Cloud · 2022same root cause · shared Model request routing
IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020same root cause · shared Container memory limit
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.