Cases4 cases

1 to 4 of 4

1. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates

   OpenAI2025

   Symptoms

   From 7:43 PM to 8:38 PM PDT on July 15, 2025, a small percentage of ChatGPT users saw more errors in their conversations.

   Container crash loopHigh error rate

   Trigger

   An internal engineer applied a configuration change that contained an invalid value.

   Root cause

   Configuration change

   Component

   –

   Contributing factor

   Excessive blast radiusNo staged rollout

   Remediation

   • reverted the configuration to the correct value
   • started a rolling restart of the affected pods to force them to resynchronise

   Action items

   • Roll out future critical configuration changes for the service to a single cluster first, with automated health guards that stop the rollout if availability drops
   • Move to a service-specific configuration value instead of one shared between services
   • Improve deployment safeguards

   View case
2. IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the API

   OpenAI2025

   Symptoms

   At 11:36 PM PDT on 9 June, alerts fired across multiple OpenAI services as a large number of cloud-hosted GPU nodes lost network connectivity, and the engineering team was escalated to immediately. Losing those nodes cut the capacity available to serve ChatGPT and the API.

   High error rateGPU unavailable

   Trigger

   A routine daily scheduled update of the host operating system on the cloud-hosted GPU servers restarted systemd-networkd on the affected nodes.

   Root cause

   Breaking change in an upgrade

   Component

   systemd-networkdNode operating system image

   Contributing factor

   No staged rolloutLimited debug access

   Detection

   Alerts fired immediately across multiple services when the infrastructure began losing network connectivity at 11:36 PM PDT, and the engineering team was escalated to at once.

   Remediation

   • re-imaged the affected GPU virtual machines
   • deployed recovery automation to restart the remaining impacted nodes
   • halted the background update mechanisms on the GPU VMs

   Action items

   • Disable automatic daily updates on GPU VMs so updates run in a controlled, scheduled way
   • Update system configurations to prevent conflicts between systemd-networkd and the networking service
   • Audit VM configurations across the fleet to find and reduce similar risks
   • Improve recovery speed for critical infrastructure such as GPU VMs and clusters
   • Run regular disaster recovery drills

   Lessons

   An unattended operating system update that restarts systemd-networkd can wipe the routes another networking agent manages, and it hits every node that takes the update at about the same time. Without break-glass tooling to restore connectivity, nodes that lost their network had to be re-imaged, which stretched recovery over many hours.

   View case
3. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services

   OpenAI2024

   Symptoms

   Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.

   Service outageHigh error rateControl plane unresponsiveDNS resolution failures

   Trigger

   At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.

   Root cause

   Configuration change

   Component

   Kubernetes API serverClient-side DNS resolver and cacheTelemetry agent

   Contributing factor

   Unrepresentative stagingNo staged rolloutCache-masked failureMonitoring blind spotLimited debug access

   Detection

   OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.

   Remediation

   • Scaled clusters down to reduce the total Kubernetes API load
   • Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
   • Scaled up the Kubernetes API servers
   • Removed the telemetry service once enough control was back
   • Moved traffic from impacted clusters to healthy ones
   • Intervened manually in clusters where simultaneous resource downloads saturated resource limits

   Action items

   • Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
   • Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
   • Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
   • Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
   • Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster

   Lessons

   Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.

   View case
4. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query

   OpenAI2024

   Symptoms

   On August 16, 2024 from 11:38 AM to 1:16 PM PT, the OpenAI primary API was unreliable: ChatGPT conversations succeeded less often, and logging in and creating accounts were affected. The status page called it elevated error rates for ChatGPT and the Platform API. The incident came in two waves, of 44 and 15 minutes.

   High error rateDatabase overload

   Trigger

   A scheduled maintenance combined with an upgrade to the ingress of OpenAI's user-facing clusters.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerApplication database

   Contributing factor

   No staged rolloutAutomation amplification

   Action items

   • Mitigate the networking control plane regression and validate that control plane restarts do not interfere with the clusters' data plane
   • Change service software to start faster and remove the first-start query, relieving the persistence layer
   • Deploy configuration changes that optimise the networking control plane's effect on the clusters' ability to handle traffic
   • Remove the expensive database query from critical startup paths
   • Add monitoring and alerting for networking control plane issues
   • Introduce staged rollouts with longer soak time for infrastructure changes
   • Audit systems for other slow queries that may affect service start time

   Lessons

   A brief connectivity loss turned into a long outage because recovery itself was expensive: the services that restarted overwhelmed one backend persistence store with a heavy first-start query. Keeping costly queries off startup paths shortens recovery, and staged rollouts with longer soak time limit how many systems an infrastructure regression reaches before it is caught.

   View case

By root cause2