Cases5 cases

1 to 5 of 5

1. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates

   OpenAI2025

   Symptoms

   From 7:43 PM to 8:38 PM PDT on July 15, 2025, a small percentage of ChatGPT users saw more errors in their conversations.

   Container crash loopHigh error rate

   Trigger

   An internal engineer applied a configuration change that contained an invalid value.

   Root cause

   Configuration change

   Component

   –

   Contributing factor

   Excessive blast radiusNo staged rollout

   Remediation

   • reverted the configuration to the correct value
   • started a rolling restart of the affected pods to force them to resynchronise

   Action items

   • Roll out future critical configuration changes for the service to a single cluster first, with automated health guards that stop the rollout if availability drops
   • Move to a service-specific configuration value instead of one shared between services
   • Improve deployment safeguards

   View case
2. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer

   OpenAI2025

   Symptoms

   From 3:30am to 7:08am PT, chatgpt.com and a small number of OpenAI APIs returned elevated error rates. The status page classed the incident as a full outage.

   High error rateContainer crash loopProbe failures

   Trigger

   A failure of Cosmos DB in ChatGPT, at OpenAI's service provider.

   Root cause

   Cloud provider outage

   Component

   Database

   Contributing factor

   Health check tied to a dependency

   Remediation

   • the service provider recovered the affected Cosmos DB databases at 4:25am PT
   • fixed the web server's health checking logic, which ended the pod crash loop

   Action items

   • Change the web systems to degrade more gracefully when they fail, including concurrency control and better load shedding
   • Make the database layer more resilient to service provider outages (described as already underway)

   Lessons

   The database came back at 4:25am PT, yet users saw errors until 7:08am PT: the web tier's own health checks kept removing pods after the dependency had recovered. The planned changes target the web tier's behaviour under failure, with concurrency control and load shedding, rather than the database alone.

   View case
3. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services

   OpenAI2024

   Symptoms

   Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.

   Service outageHigh error rateControl plane unresponsiveDNS resolution failures

   Trigger

   At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.

   Root cause

   Configuration change

   Component

   Kubernetes API serverClient-side DNS resolver and cacheTelemetry agent

   Contributing factor

   Unrepresentative stagingNo staged rolloutCache-masked failureMonitoring blind spotLimited debug access

   Detection

   OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.

   Remediation

   • Scaled clusters down to reduce the total Kubernetes API load
   • Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
   • Scaled up the Kubernetes API servers
   • Removed the telemetry service once enough control was back
   • Moved traffic from impacted clusters to healthy ones
   • Intervened manually in clusters where simultaneous resource downloads saturated resource limits

   Action items

   • Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
   • Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
   • Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
   • Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
   • Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster

   Lessons

   Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.

   View case
4. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters

   OpenAI2024

   Symptoms

   From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.

   High error rateHigh latencyControl plane unresponsive

   Trigger

   A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.

   Root cause

   Configuration change

   Component

   Kubernetes API server

   Contributing factor

   Excessive blast radiusLimited debug access

   Remediation

   • moved workloads out of the impacted clusters by hand
   • with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover

   Action items

   • Lock namespace changes so this incident cannot recur
   • Freeze certain types of infrastructure deploys
   • Allowlist identities to prevent a recurrence of this class of incidents
   • Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
   • Improve change management for fleet-wide infrastructure changes

   View case
5. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded

   OpenAI2022

   Symptoms

   At 18:56 PDT on 23 June 2022 (01:56 UTC on 24 June) several models went offline at once: some fine-tuned curie and davinci models, the Codex beta models code-davinci-001 and code-cushman-001, the legacy GPT models curie, babbage and ada, and all Embeddings models.

   Service outageHigh error rateIntermittent timeouts

   Trigger

   A destructive operation on one of the three production clusters, caused by human error that the existing safeguards did not catch, took the whole cluster offline.

   Root cause

   Operator error

   Component

   Ingress controllerModel hub

   Contributing factor

   No safeguard on destructive changeNo headroomDeferred upgradesOverlapping failures

   Remediation

   • re-allocated the models from the lost cluster to the remaining two clusters
   • rebuilt the VPN link with Azure support after losing the connection to AKS during cluster setup
   • built workarounds with Azure support after losing the connection to Azure Private Link
   • tore down the cluster and rebuilt it from scratch after the upgraded Kubernetes version broke the ingress
   • moved traffic back into the rebuilt cluster between 05:30 and 06:30 PDT

   Action items

   • Add checks and locks so that this and similar production-critical infrastructure cannot be modified by accident
   • Add better visibility into the operations being performed and make parts of the system more lexicographically distinct
   • Keep working with Azure on the AKS network connectivity issues so they do not disrupt new cluster operations
   • Update playbooks with the new cluster failure modes and change the setup steps for new clusters and infrastructure
   • Upgrade Kubernetes systematically and set aside time so the clusters do not fall behind again
   • Shard fine-tuned models across multiple clusters and work towards more automatic cross-cluster replication
   • Distribute and replicate Azure Blob Storage so fine-tuned models load faster and more reliably
   • Finish rolling out node-to-node caching to relieve Azure Blob Storage bottlenecks
   • Change how model weights are stored so they are more efficient and faster to load

   Lessons

   Losing one of three clusters exhausted the GPU headroom, and GPUs could not be added on short notice, so full recovery depended on bringing the same cluster back. Rebuilding it exposed a deferred Kubernetes upgrade: the newer version broke the ingress, and because AKS cannot downgrade there was no way back except a full rebuild. Recovering many large models at once is limited by the bandwidth of the storage they load from, and the caches that normally hide this do not help when everything moves to a different cluster.

   View case

By root cause3