Incident Library
Applied filters FactorExcessive blast radiusOrganisationOpenAIClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
    Symptoms

    From 7:43 PM to 8:38 PM PDT on July 15, 2025, a small percentage of ChatGPT users saw more errors in their conversations.

    Container crash loopHigh error rate
    Trigger

    An internal engineer applied a configuration change that contained an invalid value.

    Root cause
    Configuration change
    Component
    –
    Contributing factor
    Excessive blast radiusNo staged rollout
    Remediation
    • reverted the configuration to the correct value
    • started a rolling restart of the affected pods to force them to resynchronise
    Action items
    • Roll out future critical configuration changes for the service to a single cluster first, with automated health guards that stop the rollout if availability drops
    • Move to a service-specific configuration value instead of one shared between services
    • Improve deployment safeguards
    View case
  2. IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed
    OpenAI2025
    Symptoms

    Between 2:10 PM and 3:05 PM PDT on July 9, 2025, success rates of the gpt-4.1-mini model in the API dropped for about 20 minutes, and ChatGPT search returned many more errors than usual for about 35 minutes.

    High error rate
    Trigger

    A faulty code change was deployed to a control plane cluster whose main job is managing low-priority workloads.

    Root cause
    Faulty deployment
    Component
    –
    Contributing factor
    Excessive blast radiusNo safeguard on destructive change
    Remediation
    • rolled back the faulty code change
    Action items
    • Add safeguards that prevent production inference workloads from being managed by the low-priority control plane cluster
    • Strengthen testing for deploying high-impact control plane changes
    • Move all existing production inference workloads under the production control plane cluster
    • Build a circuit breaker that stops large-scale unintended changes
    View case
  3. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
    Symptoms

    From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.

    High error rateHigh latencyControl plane unresponsive
    Trigger

    A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.

    Root cause
    Configuration change
    Component
    Kubernetes API server
    Contributing factor
    Excessive blast radiusLimited debug access
    Remediation
    • moved workloads out of the impacted clusters by hand
    • with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover
    Action items
    • Lock namespace changes so this incident cannot recur
    • Freeze certain types of infrastructure deploys
    • Allowlist identities to prevent a recurrence of this class of incidents
    • Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
    • Improve change management for fleet-wide infrastructure changes
    View case
Check your cluster for this

These 3 cases involve Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Configuration change2 cases

Component
Kubernetes API server1
Contributing factor
Excessive blast radius2Limited debug access1No staged rollout1
Organisation
OpenAI2
  1. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
  2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024

Faulty deployment1 case

Component
–
Contributing factor
Excessive blast radiusNo safeguard on destructive change
Organisation
OpenAI
  1. IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed
    OpenAI2025