Incident Library
Applied filters Root causeBreaking change in an upgradeSymptomHigh error rateOrganisationOpenAIClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the API
    OpenAI2025
    Symptoms

    At 11:36 PM PDT on 9 June, alerts fired across multiple OpenAI services as a large number of cloud-hosted GPU nodes lost network connectivity, and the engineering team was escalated to immediately. Losing those nodes cut the capacity available to serve ChatGPT and the API.

    High error rateGPU unavailable
    Trigger

    A routine daily scheduled update of the host operating system on the cloud-hosted GPU servers restarted systemd-networkd on the affected nodes.

    Root cause
    Breaking change in an upgrade
    Component
    systemd-networkdNode operating system image
    Contributing factor
    No staged rolloutLimited debug access
    Detection

    Alerts fired immediately across multiple services when the infrastructure began losing network connectivity at 11:36 PM PDT, and the engineering team was escalated to at once.

    Remediation
    • re-imaged the affected GPU virtual machines
    • deployed recovery automation to restart the remaining impacted nodes
    • halted the background update mechanisms on the GPU VMs
    Action items
    • Disable automatic daily updates on GPU VMs so updates run in a controlled, scheduled way
    • Update system configurations to prevent conflicts between systemd-networkd and the networking service
    • Audit VM configurations across the fleet to find and reduce similar risks
    • Improve recovery speed for critical infrastructure such as GPU VMs and clusters
    • Run regular disaster recovery drills
    Lessons

    An unattended operating system update that restarts systemd-networkd can wipe the routes another networking agent manages, and it hits every node that takes the update at about the same time. Without break-glass tooling to restore connectivity, nodes that lost their network had to be re-imaged, which stretched recovery over many hours.

    View case
  2. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024
    Symptoms

    On August 16, 2024 from 11:38 AM to 1:16 PM PT, the OpenAI primary API was unreliable: ChatGPT conversations succeeded less often, and logging in and creating accounts were affected. The status page called it elevated error rates for ChatGPT and the Platform API. The incident came in two waves, of 44 and 15 minutes.

    High error rateDatabase overload
    Trigger

    A scheduled maintenance combined with an upgrade to the ingress of OpenAI's user-facing clusters.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerApplication database
    Contributing factor
    No staged rolloutAutomation amplification
    Action items
    • Mitigate the networking control plane regression and validate that control plane restarts do not interfere with the clusters' data plane
    • Change service software to start faster and remove the first-start query, relieving the persistence layer
    • Deploy configuration changes that optimise the networking control plane's effect on the clusters' ability to handle traffic
    • Remove the expensive database query from critical startup paths
    • Add monitoring and alerting for networking control plane issues
    • Introduce staged rollouts with longer soak time for infrastructure changes
    • Audit systems for other slow queries that may affect service start time
    Lessons

    A brief connectivity loss turned into a long outage because recovery itself was expensive: the services that restarted overwhelmed one backend persistence store with a heavy first-start query. Keeping costly queries off startup paths shortens recovery, and staged rollouts with longer soak time limit how many systems an infrastructure regression reaches before it is caught.

    View case
Check your cluster for this

These 2 cases involve Application database and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade2 cases

Component
Application database1Ingress controller1Node operating system image1systemd-networkd1
Contributing factor
No staged rollout2Automation amplification1Limited debug access1
Organisation
OpenAI2
  1. IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the API
    OpenAI2025
  2. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024