Incident Library
Applied filters Root causeConfiguration changeFactorLimited debug accessClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
    Symptoms

    Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.

    Service outageHigh error rateControl plane unresponsiveDNS resolution failures
    Trigger

    At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.

    Root cause
    Configuration change
    Component
    Kubernetes API serverClient-side DNS resolver and cacheTelemetry agent
    Contributing factor
    Unrepresentative stagingNo staged rolloutCache-masked failureMonitoring blind spotLimited debug access
    Detection

    OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.

    Remediation
    • Scaled clusters down to reduce the total Kubernetes API load
    • Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
    • Scaled up the Kubernetes API servers
    • Removed the telemetry service once enough control was back
    • Moved traffic from impacted clusters to healthy ones
    • Intervened manually in clusters where simultaneous resource downloads saturated resource limits
    Action items
    • Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
    • Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
    • Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
    • Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
    • Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster
    Lessons

    Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.

    View case
  2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
    Symptoms

    From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.

    High error rateHigh latencyControl plane unresponsive
    Trigger

    A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.

    Root cause
    Configuration change
    Component
    Kubernetes API server
    Contributing factor
    Excessive blast radiusLimited debug access
    Remediation
    • moved workloads out of the impacted clusters by hand
    • with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover
    Action items
    • Lock namespace changes so this incident cannot recur
    • Freeze certain types of infrastructure deploys
    • Allowlist identities to prevent a recurrence of this class of incidents
    • Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
    • Improve change management for fleet-wide infrastructure changes
    View case
  3. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
    Symptoms

    Customers of DigitalOcean Managed Kubernetes (DOKS) saw worker nodes go NotReady between 06:00 and 07:00 UTC and stay unavailable, taking down the services on them, until the nodes were replaced or rebooted; some nodes needed several replacements or reboots. The first sporadic reports on October 8 said replacing the node fixed it. The same thing happened again around 06:00 UTC on October 9, more widely and across many regions, and again on October 10. Only clusters on versions 1.27.6-do.0 and 1.28.2-do.0 were affected.

    Node NotReadyService outage
    Trigger

    A systemd timer in the DigitalOcean Debian image started apt upgrade daily at 06:00 UTC, with a random delay of up to 60 minutes.

    Root cause
    Configuration change
    Component
    Node operating system imageLinux kernelkubelet
    Contributing factor
    Unreviewed defaultsLimited debug accessChange with a delayed effect
    Detection

    Sporadic customer reports of failed worker nodes arrived at 07:43 UTC on October 8; internal incident response started at 07:10 UTC on October 9 after broader impact across regions.

    Remediation
    • Advised customers to replace or reboot affected worker nodes
    • Arranged SSH and console access to worker nodes while they were healthy, and live-debugged an affected node on October 10
    • Released a DaemonSet to all affected clusters that disables the apt upgrade systemd timer on every worker node
    • Released new node images without the auto-update process
    Action items
    • Disable the timer during node provisioning, then remove the DaemonSet from all clusters
    • Add a check to the internal conformance test suite for new cluster versions that the auto-upgrade process and other unwanted timers stay disabled
    • Support logging into worker nodes even when normal access paths are broken
    Lessons

    An operating system auto-update timer left enabled in a node image can change the kernel and systemd underneath Kubernetes, and a timeout that kills the upgrade half-way can leave a node without networking. Failures that start at the same time every day point to a scheduled job on the nodes, and systemd timers belong on that list next to cron jobs. When the failure takes out a node's network, an investigation that depends on logging into the node stalls; access paths that work while the node is broken shorten diagnosis.

    View case
Check your cluster for this

These 3 cases involve Kubernetes API server and Client-side DNS resolver and cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change3 cases

Component
Kubernetes API server2Client-side DNS resolver and cache1kubelet1Linux kernel1Node operating system image1Telemetry agent1
Contributing factor
Limited debug access3Excessive blast radius1Unreviewed defaults1Change with a delayed effect1Cache-masked failure1Monitoring blind spot1No staged rollout1Unrepresentative staging1
Organisation
OpenAI2DigitalOcean1
  1. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  3. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023