IL-0026 Blue Matador · 2019-06-15kubernetesawsobservability
Two production nodes at Blue Matador, on two consecutive evenings in June 2019, reported SystemOOM events. In both cases the memory had been consumed by the fluentd-sumologic pods, the Ruby-based log shipper that forwards logs to Sumo Logic. Their manifest set no resource requests or limits, so nothing stopped them from growing until the whole node was short of memory. Kubernetes resolved each event by killing and restarting those pods, and no application pod was affected.
The first event was dismissed because the check for restarted pods looked only at the application namespace. After the second event the engineer restarted all sumologic pods so they started from low memory, then on Monday added the RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=0.9 environment variable suggested in a GitHub issue and set memory requests of 128Mi and limits of 1024Mi (CPU 100m and 250m) on the pods. Node memory stayed stable in the following days.
Each SystemOOM was preceded by heavy disk activity and a falling EBS Burst Balance on the node's root volume. Swap was not enabled, and the cause of that disk I/O was never found because the logs from before the events were no longer available.
On a Saturday at 5:12 pm, Blue Matador's own monitoring raised a SystemOOM event for one node in the production Kubernetes cluster. Four minutes later a second warning reported a low EBS Burst Balance on that node's root volume. CloudWatch data showed the burst balance had actually bottomed out at 5:02 pm, before the memory event; the warning arrived late because EBS metrics lag 10 to 15 minutes.
The engineer ran kubectl get pods and saw no application pod had died, and kubectl top nodes showed the node back at about 60% memory. With the node healthy and nothing restarted in the application namespace, the event was treated as a one-off and left for Monday.
The next evening, at 6:02 pm, a different node raised the same SystemOOM event, again followed by a low EBS Burst Balance warning; the burst balance had been falling for about 2.5 hours beforehand. Pods again looked fine and node memory sat just under 70%. Datadog memory graphs then showed that the node's memory had been taken by the fluentd-sumologic pods and dropped sharply at the moment of the SystemOOM: Kubernetes had killed and restarted those pods to reclaim memory. The restarts had been missed the day before because the fluentd-sumologic pods ran in a separate namespace that nobody checked.
Symptom OOMKilled
Root cause Capacity exhaustion
Blue Matador's own monitoring product alerted on the SystemOOM event, followed minutes later by a low EBS Burst Balance warning for the same node.
When checking for restarted pods, look in every namespace, not only the application one. Set resource requests and limits on every pod, third-party ones in particular, so that one workload cannot exhaust a node. Investigate side signals such as the disk I/O spike while the logs still exist.
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Container memory limit, Telemetry agent
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Container memory limit
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Container memory limit
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Container memory limit
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Telemetry agent
IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Container memory limit
IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024same root cause · shared Container memory limit
IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategyUK Ministry of Justice · 2023same root cause · shared Telemetry agent
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Container memory limit
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Container memory limit
IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023same root cause · shared Telemetry agent
IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Container memory limit
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Container memory limit
IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019same root cause · shared Telemetry agent
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared Container memory limit