Incident Library
Applied filters SymptomControl plane unresponsiveFactorExcessive blast radiusOrganisationOpenAIClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
    Symptoms

    From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.

    High error rateHigh latencyControl plane unresponsive
    Trigger

    A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.

    Root cause
    Configuration change
    Component
    Kubernetes API server
    Contributing factor
    Excessive blast radiusLimited debug access
    Remediation
    • moved workloads out of the impacted clusters by hand
    • with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover
    Action items
    • Lock namespace changes so this incident cannot recur
    • Freeze certain types of infrastructure deploys
    • Allowlist identities to prevent a recurrence of this class of incidents
    • Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
    • Improve change management for fleet-wide infrastructure changes
    View case
Check your cluster for this

This case involves Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Kubernetes API server
Contributing factor
Excessive blast radiusLimited debug access
Organisation
OpenAI
  1. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024