IL-0110 DFDS · 2024-03-13kubernetesaws
DFDS runs dev and production workloads on the same Kubernetes cluster on AWS. On 2024-03-13 a Grafana Agent pod used a large amount of memory on one node, and Hangfire pods of a dev container were scheduled to the same node. Those pods failed on system OOM, restarted and failed again in a spiral of CrashLoopBackOff until the whole node stopped responding and went NotReady. Unlike most of the monitoring stack, the Grafana Agent did not run on a dedicated node group.
In the morning the team marked the EC2 instance unhealthy so a new node started, cordoned and force-drained the NotReady node and terminated the instance, and declared the incident resolved at 12:10 while investigations continued. They then found the Grafana Agent still down because of a newly applied LimitRange, removed the LimitRange and restarted the agent. They found a correlation between the tenfold rise in EC2 network traffic, the RDS problems and the dev container, scaled the container to 0 and then back to its original 3 replicas, and the replacement node went NotReady at 15:03.
The second time, the team scaled the dev container to 0 again, a new node was added, and the team that owns the container was emailed and confirmed at 15:52 that it was working on a fix. The incident was closed at 09:20 UTC on 2024-03-14. The post-mortem gives 4 minutes to detection and 399 minutes to recovery.
At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.
CloudWatch showed a tenfold increase in EC2 network traffic leading up to the incident, and the Grafana Agent pod on the unhealthy node had high memory use. After the node was replaced, the Grafana Agent stayed down until 13:02 because of a LimitRange newly applied to its namespace.
At 13:49 the team concluded that a dev container using Hangfire to trigger jobs was failing with OOM and going into CrashLoopBackOff with new OOMs. At 13:55 they found problems on the RDS instance used by that container and a correlation between the increased EC2 network traffic, the RDS instance and the container. Scaling the container to 0 replicas brought network traffic back to normal within a few minutes, but after it was scaled back up to its original 3 replicas, the replacement node entered NotReady at 15:03.
Root cause Capacity exhaustion
A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.
Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.
IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Application database, Container memory limit
IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Container memory limit, Telemetry agent
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Container memory limit
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Application database
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Container memory limit
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Container memory limit
IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk spaceUK Ministry of Justice · 2024same root cause · shared Telemetry agent
IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Container memory limit
IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024same root cause · shared Container memory limit
IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared Application database
IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategyUK Ministry of Justice · 2023same root cause · shared Telemetry agent
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Container memory limit
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Container memory limit
IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023same root cause · shared Telemetry agent
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Container memory limit
IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019same root cause · shared Application database
IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019same root cause · shared Application database
IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019same root cause · shared Telemetry agent
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared Container memory limit