Incident Library

Organisation

OpenAI

OpenAI · openai

11 cases · 2022 to 2026

Root cause Configuration change4Software defect2Breaking change in an upgrade2Cloud provider outage1Faulty deployment1Operator error1
Component Application database2Ingress controller2Kubernetes API server2Automated capacity manager1CNI plugin1Client-side DNS resolver and cache1Container memory limit1Model hub1Model request routing1Node operating system image1systemd-networkd1Telemetry agent1
Contributing factor No staged rollout4Excessive blast radius3Limited debug access3Monitoring blind spot3No safeguard on destructive change3Automation amplification2No headroom2Unrepresentative staging2Change with a delayed effect1Cache-masked failure1No alert on limit1Overlapping failures1Health check tied to a dependency1Deferred upgrades1
Symptom High error rate10Service outage4Control plane unresponsive2Container crash loop2High latency2Probe failures2Database overload1DNS resolution failures1GPU unavailable1Intermittent timeouts1Throughput degradation1

Cases

  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error ratesOpenAI · 2026

    Root cause Software defect

  2. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error ratesOpenAI · 2025

    Root cause Configuration change

  3. IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managedOpenAI · 2025

    Root cause Faulty deployment

  4. IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the APIOpenAI · 2025

    Root cause Breaking change in an upgrade

  5. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layerOpenAI · 2025

    Root cause Cloud provider outage

  6. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI servicesOpenAI · 2024

    Root cause Configuration change

  7. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clustersOpenAI · 2024

    Root cause Configuration change

  8. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024

    Root cause Breaking change in an upgrade

  9. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all modelsOpenAI · 2023

    Root cause Software defect

  10. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully downOpenAI · 2023

    Root cause Configuration change

  11. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgradedOpenAI · 2022

    Root cause Operator error