IL-0169 Scaleway · 2024-01-24kubernetes
Simultaneous deployments in FR-PAR drove the Scaleway Kubernetes API, the service behind Kapsule and Kosmos, out of memory. This API was also handling node authentication at the time, so when it failed the kubelets on customer nodes could not renew their leases, the nodes went NotReady, and they retried indefinitely. Requests from these malfunctioning components flooded the API Gateway, which slowed down and timed out on some requests but did not go fully down; changes made after an earlier incident helped it absorb the spike.
On-call engineers took the incident at 00:00:10 and by 00:30 had identified the out-of-memory condition as the cause. Between 00:35 and 00:40 they raised the memory limit and garbage collection thresholds of the Kubernetes managed services and launched more replicas of key services. API Gateway requests began falling fast at 00:43, nodes came back from 00:54 while a heavy queue of instance actions was watched, and from 01:03 all FR-PAR clusters began to stabilise after further steps such as purging the registry cache. Cockpit metrics came back at 01:13.
At 23:48 CET on 24 January 2024 the Scaleway Kubernetes API for the FR-PAR region ran out of memory. At the same minute Scaleway saw requests to its API Gateway spike.
Nodes of customer clusters could not authenticate with their control plane, so their kubelets could not update their leases, and some nodes became NotReady and kept retrying without end. Response times of the API Gateway rose until some requests reached timeouts. Where it applied, the cluster auto-healing process started and replaced nodes automatically.
Customers of the managed Kubernetes services Kapsule and Kosmos were affected, in both mutualized and dedicated environments, along with some other products such as Cockpit, whose metrics were missing until the end of the incident.
Root cause Capacity exhaustion
Increased load on the API Gateway was detected at 23:48, the minute the incident began, and the incident was escalated to on-call engineers at 00:00:10.
Because the Kubernetes API also handled node authentication, its memory exhaustion made some nodes NotReady, and the endless retries of those nodes added load on the API Gateway. Scaleway plans a local authentication cache and further work on node authentication.
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared kubelet, Container memory limit
IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshotsOnkar Raghunath Shelke · 2026same root cause · shared kubelet
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Container memory limit
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Container memory limit
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Container memory limit
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Container memory limit
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Container memory limit
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Container memory limit
IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Container memory limit
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Container memory limit
IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020same root cause · shared kubelet
IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Container memory limit
IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019same root cause · shared kubelet
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared Container memory limit