Cases7 cases

1 to 5 of 7

1. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services

   OpenAI2024

   Symptoms

   Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.

   Service outageHigh error rateControl plane unresponsiveDNS resolution failures

   Trigger

   At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.

   Root cause

   Configuration change

   Component

   Kubernetes API serverClient-side DNS resolver and cacheTelemetry agent

   Contributing factor

   Unrepresentative stagingNo staged rolloutCache-masked failureMonitoring blind spotLimited debug access

   Detection

   OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.

   Remediation

   • Scaled clusters down to reduce the total Kubernetes API load
   • Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
   • Scaled up the Kubernetes API servers
   • Removed the telemetry service once enough control was back
   • Moved traffic from impacted clusters to healthy ones
   • Intervened manually in clusters where simultaneous resource downloads saturated resource limits

   Action items

   • Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
   • Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
   • Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
   • Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
   • Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster

   Lessons

   Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.

   View case
2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters

   OpenAI2024

   Symptoms

   From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.

   High error rateHigh latencyControl plane unresponsive

   Trigger

   A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.

   Root cause

   Configuration change

   Component

   Kubernetes API server

   Contributing factor

   Excessive blast radiusLimited debug access

   Remediation

   • moved workloads out of the impacted clusters by hand
   • with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover

   Action items

   • Lock namespace changes so this incident cannot recur
   • Freeze certain types of infrastructure deploys
   • Allowlist identities to prevent a recurrence of this class of incidents
   • Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
   • Improve change management for fleet-wide infrastructure changes

   View case
3. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1

   DigitalOcean2024

   Symptoms

   At 13:02 UTC on 2024-01-23 DigitalOcean posted that, as of 12:18 UTC, its engineers were investigating an issue with Kubernetes clusters in the FRA1 region and that users might see errors when communicating with their clusters. The control planes of the affected DOKS (DigitalOcean Kubernetes) clusters had become inaccessible.

   Resources deletedControl plane unresponsive

   Trigger

   An incorrect configuration was applied to an internal infrastructure cluster in FRA1 at 12:21 UTC during a maintenance operation.

   Root cause

   Configuration change

   Component

   etcdKubernetes API server

   Contributing factor

   Automation amplificationNo safeguard on destructive changeExcessive blast radius

   Detection

   Engineers were investigating errors with Kubernetes clusters in FRA1 as of 12:18 UTC and identified the incorrect configuration as the cause at 12:36.

   Remediation

   • restored each affected cluster's etcd from its latest twice-daily backup
   • restored the stateless control plane components after etcd
   • patched the recovery mechanism after unexpected problems during recovery
   • found and fixed an additional small bug during recovery
   • contacted affected customers directly with further instructions, since some might need to reconfigure Kubernetes resources

   Action items

   • Prevent automatic deletion of infrastructure cluster resources under any circumstances in the production environment

   Lessons

   When managed control planes run as resources in another cluster, one configuration mistake in that cluster can delete hundreds of control planes at once through ordinary automation. The etcd backups limited the damage, but their twice-daily schedule set the worst case at 12 hours of lost control plane state, and the recovery tooling itself needed patching during the incident.

   View case
4. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet

   Aqua Security2023

   Symptoms

   Aqua's research team Nautilus recorded the attack on one of its Kubernetes honeypots. What appeared in the cluster was a DaemonSet running the image kuberntesio/kube-controller:1.0.1 from Docker Hub on every node, using the nodes' CPU to mine Monero.

   Unauthorised access or a foreign workloadResources deleted

   Trigger

   An API server misconfigured to accept unauthenticated requests from anonymous users with privileges.

   Root cause

   Configuration change

   Component

   Kubernetes API serverRBAC and service accountsContainer registry and its credentials

   Contributing factor

   Monitoring blind spot

   Detection

   Aqua recorded the attacker's API requests on its honeypot; planted AWS access keys raised a beacon when the attacker tried to use them.

   Lessons

   Fixing an exposed API server is not enough once an attacker has been in: a ClusterRoleBinding to an attacker-created ServiceAccount keeps working after anonymous access is closed. Attackers name these objects and images after real Kubernetes components so they blend into the API audit logs and pass for legitimate deployments. Aqua recommends scanning clusters for vulnerabilities, exposed secrets and misconfigurations with a tool such as Trivy.

   View case
5. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out

   DFDS2020

   Symptoms

   At 14:41 CET on 4 November 2020 users reported timeouts when deploying to Hellman, the production Kubernetes cluster. The timeouts were confirmed at 15:04.

   Deploys failingControl plane lag

   Trigger

   During an Alcide workshop that began at 13:00 CET, an Alcide admission controller was applied to the production cluster at 13:13 in the belief that it would not disrupt anything.

   Root cause

   Configuration change

   Component

   Admission webhookKubernetes API server

   Contributing factor

   No configuration testsUnverified assumptionMonitoring blind spot

   Detection

   Users reported timeouts when deploying to the Hellman cluster at 14:41 CET; the timeouts were verified at 15:04.

   Remediation

   • toggled the Alcide admission controller to verify that it caused the timeouts
   • permanently deleted the Alcide admission controller

   Action items

   • Monitor Kubernetes deployments for timeouts
   • Monitor 5xx status codes from the Kubernetes API server

   Lessons

   New components are applied in sandbox clusters, not in production. The team needs enough Alcide knowledge to maintain the component until it is fully deprecated.

   View case

By root cause1