IL-0168 Scaleway · 2024-02-13kubernetes
Scaleway's managed Kubernetes services Kapsule and Kosmos failed in the Paris region (FR-PAR) on 13 February 2024 from 01:04 until 16:40 CET. A new Kubernetes release had become available to all users on 12 February, and overnight many clusters triggered their auto-upgrade simultaneously. The resulting load made the Kubernetes API slow and congested, and after RAM and CPU spikes it ran out of memory. Scaleway later named the root cause as insufficient resources allocated to the Kubernetes API server.
With the API unavailable, nodes failed to authenticate to their control plane and became NotReady. A new node authentication mechanism, deployed on 24 January 2024, failed and retried continuously, which deepened the congestion, and auto-healing replaced some nodes automatically, which amplified the problem. The same fail-and-retry congestion had caused an incident on 24 January 2024, and the short-term mitigations put in place since then had not prevented it from happening again.
Scaleway raised the API's resources twice, applied API traffic restrictions at 02:30 to smooth incoming traffic, and from 03:20 removed them progressively in 50-100 batches, after which nodes authenticated and became Ready. A script launched at 03:40 replaced the failing nodes created during the congestion, and service was stable by 05:30. At 15:45 Scaleway began rolling back to the former authentication mechanism on a subset of clusters; congestion returned at 16:00 and was handled the same way, and all nodes were Ready by 16:40. Cluster upgrade and configuration update features were disabled to avoid simultaneous node authentication. The rollback reached all clusters at 19:41 on 19 February, the features were re-enabled at 10:30 on 20 February, and the incident was closed at 17:56 that day.
At 01:04 CET on 13 February 2024, the Kubernetes API behind Scaleway's managed Kubernetes services Kapsule and Kosmos in the Paris region (FR-PAR) spiked in RAM and CPU use and ran out of memory. Nodes in FR-PAR could no longer authenticate to their control plane, and some of them went "Not Ready". Customers on both shared (mutualized) and dedicated environments were affected.
Kubernetes API response times rose until some requests timed out. The auto-healing feature automatically replaced some of the failing nodes, which made the problem bigger.
The first increase of the API's resources, at 01:46, seemed to help but the API ran out of memory again; a second increase at 02:20 brought improvement. Later in the day, when the rollback of the node authentication mechanism started at 15:45, API congestion returned at 16:00 and nodes again went Not Ready.
Root cause Capacity exhaustion
The timeline's first entry, at 01:04 CET, records the Kubernetes API's RAM and CPU spikes and out-of-memory state, with nodes unable to authenticate to their control plane and some becoming "Not Ready". The first recorded action, 42 minutes later at 01:46, was to increase the resources allocated to the API.
Letting many clusters auto-upgrade at the same moment after a release turns routine upgrades into one coordinated load spike on the shared Kubernetes API. A node authentication mechanism that retries without letting up turns a slow API into an unavailable one, and auto-healing that replaces NotReady nodes amplifies the problem.
The congestion pattern had already appeared three weeks earlier, and short-term mitigations did not stop it from recurring, so Scaleway rolled back the new authentication mechanism. Scaleway notes that as the number of customer clusters and nodes grows sharply, traffic control mechanisms have to improve to prevent congestion.
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared Kubernetes API server
IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agentsPalark · 2023same root cause · shared Kubernetes API server
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Kubernetes API server
IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023same root cause · shared Kubernetes API server
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Kubernetes API server
IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019same root cause · shared Kubernetes API server
IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019same root cause · shared Kubernetes API server
IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018same root cause · shared Kubernetes API server