Incident Library
Applied filters SymptomContainer crash loopOrganisationOpenAIClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
    Symptoms

    From 7:43 PM to 8:38 PM PDT on July 15, 2025, a small percentage of ChatGPT users saw more errors in their conversations.

    Container crash loopHigh error rate
    Trigger

    An internal engineer applied a configuration change that contained an invalid value.

    Root cause
    Configuration change
    Component
    –
    Contributing factor
    Excessive blast radiusNo staged rollout
    Remediation
    • reverted the configuration to the correct value
    • started a rolling restart of the affected pods to force them to resynchronise
    Action items
    • Roll out future critical configuration changes for the service to a single cluster first, with automated health guards that stop the rollout if availability drops
    • Move to a service-specific configuration value instead of one shared between services
    • Improve deployment safeguards
    View case
  2. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer
    OpenAI2025
    Symptoms

    From 3:30am to 7:08am PT, chatgpt.com and a small number of OpenAI APIs returned elevated error rates. The status page classed the incident as a full outage.

    High error rateContainer crash loopProbe failures
    Trigger

    A failure of Cosmos DB in ChatGPT, at OpenAI's service provider.

    Root cause
    Cloud provider outage
    Component
    Application database
    Contributing factor
    Health check tied to a dependency
    Remediation
    • the service provider recovered the affected Cosmos DB databases at 4:25am PT
    • fixed the web server's health checking logic, which ended the pod crash loop
    Action items
    • Change the web systems to degrade more gracefully when they fail, including concurrency control and better load shedding
    • Make the database layer more resilient to service provider outages (described as already underway)
    Lessons

    The database came back at 4:25am PT, yet users saw errors until 7:08am PT: the web tier's own health checks kept removing pods after the dependency had recovered. The planned changes target the web tier's behaviour under failure, with concurrency control and load shedding, rather than the database alone.

    View case
Check your cluster for this

These 2 cases involve Application database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Configuration change1 case

Component
–
Contributing factor
Excessive blast radiusNo staged rollout
Organisation
OpenAI
  1. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025

Cloud provider outage1 case

Component
Application database
Contributing factor
Health check tied to a dependency
Organisation
OpenAI
  1. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer
    OpenAI2025