Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024
    Symptoms

    On August 16, 2024 from 11:38 AM to 1:16 PM PT, the OpenAI primary API was unreliable: ChatGPT conversations succeeded less often, and logging in and creating accounts were affected. The status page called it elevated error rates for ChatGPT and the Platform API. The incident came in two waves, of 44 and 15 minutes.

    High error rateDatabase overload
    Trigger

    A scheduled maintenance combined with an upgrade to the ingress of OpenAI's user-facing clusters.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerApplication database
    Contributing factor
    No staged rolloutAutomation amplification
    Action items
    • Mitigate the networking control plane regression and validate that control plane restarts do not interfere with the clusters' data plane
    • Change service software to start faster and remove the first-start query, relieving the persistence layer
    • Deploy configuration changes that optimise the networking control plane's effect on the clusters' ability to handle traffic
    • Remove the expensive database query from critical startup paths
    • Add monitoring and alerting for networking control plane issues
    • Introduce staged rollouts with longer soak time for infrastructure changes
    • Audit systems for other slow queries that may affect service start time
    Lessons

    A brief connectivity loss turned into a long outage because recovery itself was expensive: the services that restarted overwhelmed one backend persistence store with a heavy first-start query. Keeping costly queries off startup paths shortens recovery, and staged rollouts with longer soak time limit how many systems an infrastructure regression reaches before it is caught.

    View case
Check your cluster for this

This case involves Application database and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade1 case

Component
Application databaseIngress controller
Contributing factor
Automation amplificationNo staged rollout
Organisation
OpenAI
  1. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024