Incident Library
Applied filters Root causeConfiguration changeComponentKubernetes API serverSymptomControl plane unresponsiveClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases4 cases

1 to 4 of 4

  1. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
    Symptoms

    Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.

    Service outageHigh error rateControl plane unresponsiveDNS resolution failures
    Trigger

    At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.

    Root cause
    Configuration change
    Component
    Kubernetes API serverClient-side DNS resolver and cacheTelemetry agent
    Contributing factor
    Unrepresentative stagingNo staged rolloutCache-masked failureMonitoring blind spotLimited debug access
    Detection

    OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.

    Remediation
    • Scaled clusters down to reduce the total Kubernetes API load
    • Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
    • Scaled up the Kubernetes API servers
    • Removed the telemetry service once enough control was back
    • Moved traffic from impacted clusters to healthy ones
    • Intervened manually in clusters where simultaneous resource downloads saturated resource limits
    Action items
    • Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
    • Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
    • Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
    • Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
    • Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster
    Lessons

    Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.

    View case
  2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
    Symptoms

    From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.

    High error rateHigh latencyControl plane unresponsive
    Trigger

    A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.

    Root cause
    Configuration change
    Component
    Kubernetes API server
    Contributing factor
    Excessive blast radiusLimited debug access
    Remediation
    • moved workloads out of the impacted clusters by hand
    • with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover
    Action items
    • Lock namespace changes so this incident cannot recur
    • Freeze certain types of infrastructure deploys
    • Allowlist identities to prevent a recurrence of this class of incidents
    • Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
    • Improve change management for fleet-wide infrastructure changes
    View case
  3. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
    Symptoms

    At 13:02 UTC on 2024-01-23 DigitalOcean posted that, as of 12:18 UTC, its engineers were investigating an issue with Kubernetes clusters in the FRA1 region and that users might see errors when communicating with their clusters. The control planes of the affected DOKS (DigitalOcean Kubernetes) clusters had become inaccessible.

    Resources deletedControl plane unresponsive
    Trigger

    An incorrect configuration was applied to an internal infrastructure cluster in FRA1 at 12:21 UTC during a maintenance operation.

    Root cause
    Configuration change
    Component
    etcdKubernetes API server
    Contributing factor
    Automation amplificationNo safeguard on destructive changeExcessive blast radius
    Detection

    Engineers were investigating errors with Kubernetes clusters in FRA1 as of 12:18 UTC and identified the incorrect configuration as the cause at 12:36.

    Remediation
    • restored each affected cluster's etcd from its latest twice-daily backup
    • restored the stateless control plane components after etcd
    • patched the recovery mechanism after unexpected problems during recovery
    • found and fixed an additional small bug during recovery
    • contacted affected customers directly with further instructions, since some might need to reconfigure Kubernetes resources
    Action items
    • Prevent automatic deletion of infrastructure cluster resources under any circumstances in the production environment
    Lessons

    When managed control planes run as resources in another cluster, one configuration mistake in that cluster can delete hundreds of control planes at once through ordinary automation. The etcd backups limited the damage, but their twice-daily schedule set the worst case at 12 hours of lost control plane state, and the recovery tooling itself needed patching during the incident.

    View case
  4. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
    Symptoms

    Jetstack was upgrading the master of a development cluster that many teams use to test their apps during the working day, a regional Google Kubernetes Engine (GKE) cluster in europe-west1. The upgrade ran through Jetstack's GKE Terraform pipeline and did not finish before the 20-minute Terraform timeout, while the GKE console still showed the cluster as upgrading.

    Upgrade stallControl plane unresponsiveNode recreation loopService outage
    Trigger

    A planned master upgrade of the shared development cluster. The same upgrade had been done on another pre-production environment earlier that day, and no API downtime was expected.

    Root cause
    Configuration change
    Component
    Admission webhookGKE control planeKubernetes API serverkubelet
    Contributing factor
    Excessive blast radiusNo health probesMonitoring blind spotAutomation amplification
    Detection

    The first sign was the master upgrade missing the 20-minute Terraform timeout; re-running the pipeline returned the kube-apiserver unhealthy error. The cause was identified with the help of Google Support.

    Remediation
    • Deleted the OPA ValidatingAdmissionWebhook resource during a window of API server access
    Action items
    • Limit the OPA webhook to the namespaces where the policy applies and development teams have access
    • Enable the webhook only for Ingress and Service resources, the only resources the policy validates
    • Add a liveness probe so OPA is restarted when it stops responding, and update the OPA documentation
    • Follow the open upstream Kubernetes issue about this behaviour
    Lessons

    With alerting on API server response times, the team might have seen CREATE and UPDATE requests slow down across the board as soon as the OPA-backed webhook was first deployed. Every workload needs probes: OPA looked so simple to deploy that the team skipped the Helm chart, which adds a livenessProbe for the admission controller containers among other adjustments to the tutorial configuration. Others had hit the same problem, and an upstream issue to improve it was still open.

    View case
Check your cluster for this

These 4 cases involve Kubernetes API server and Admission webhook. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change4 cases

Component
Kubernetes API server4Admission webhook1Client-side DNS resolver and cache1etcd1GKE control plane1kubelet1Telemetry agent1
Contributing factor
Excessive blast radius3Automation amplification2Limited debug access2Monitoring blind spot2Cache-masked failure1No health probes1No safeguard on destructive change1No staged rollout1Unrepresentative staging1
Organisation
OpenAI2DigitalOcean1Jetstack1
  1. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  3. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
  4. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019