Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
    Symptoms

    From 7:43 PM to 8:38 PM PDT on July 15, 2025, a small percentage of ChatGPT users saw more errors in their conversations.

    Container crash loopHigh error rate
    Trigger

    An internal engineer applied a configuration change that contained an invalid value.

    Root cause
    Configuration change
    Component
    –
    Contributing factor
    Excessive blast radiusNo staged rollout
    Remediation
    • reverted the configuration to the correct value
    • started a rolling restart of the affected pods to force them to resynchronise
    Action items
    • Roll out future critical configuration changes for the service to a single cluster first, with automated health guards that stop the rollout if availability drops
    • Move to a service-specific configuration value instead of one shared between services
    • Improve deployment safeguards
    View case
  2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
    Symptoms

    From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.

    High error rateHigh latencyControl plane unresponsive
    Trigger

    A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.

    Root cause
    Configuration change
    Component
    Kubernetes API server
    Contributing factor
    Excessive blast radiusLimited debug access
    Remediation
    • moved workloads out of the impacted clusters by hand
    • with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover
    Action items
    • Lock namespace changes so this incident cannot recur
    • Freeze certain types of infrastructure deploys
    • Allowlist identities to prevent a recurrence of this class of incidents
    • Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
    • Improve change management for fleet-wide infrastructure changes
    View case
Check your cluster for this

These 2 cases involve Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
Kubernetes API server1
Contributing factor
Excessive blast radius2Limited debug access1No staged rollout1
Organisation
OpenAI2
  1. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
  2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024