Cases6 cases

1 to 5 of 6

1. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours

   Magic Pages2025

   Symptoms

   At around 09:33 UTC one of the three control plane nodes of the Kubernetes cluster that hosts Magic Pages went offline. Looking at the node showed that it had run out of memory.

   Service outageNode NotReadyCloud API rate limit exhausted

   Trigger

   One of the three control plane nodes ran out of memory and went offline at about 09:33 UTC; the attempts to replace it with hetzner-k3s failed while the cluster's own components were exhausting the Hetzner API rate limit.

   Root cause

   Capacity exhaustion

   Component

   Cloud controller managerPersistent volumes and stateful sets

   Contributing factor

   Unknown limitRetry stormMonitoring blind spotLimited debug accessSingle point of failureNo runbook

   Detection

   The founder found the control plane node offline at about 09:33 UTC. Monitoring alerts about unreachable customer websites started around the time the replacement attempts failed, and about half an hour of investigation identified the exhausted Hetzner API rate limit.

   Remediation

   • powered off all servers of the Kubernetes cluster to stop the API requests so the Hetzner rate limit could recover
   • waited until ratelimit-remaining had built up a buffer before starting recovery
   • restarted the control plane servers
   • started the essential Longhorn components by hand so Longhorn could heal and make storage volumes available
   • restarted the remaining websites by hand

   Action items

   • Monitor the Hetzner API call rate coming from the Kubernetes cluster nodes themselves to get early warning of high usage
   • Get a higher Hetzner API rate limit; Hetzner support was contacted
   • Investigate how to reduce the cluster's reliance on a single API, including alternatives to hetzner-k3s
   • Document and test a procedure for recovering Longhorn after a full cluster shutdown

   Lessons

   The running cluster, not only the deployment tooling, consumed the cloud provider's API quota, and in a failure it consumed it fast enough that the quota never recovered. The default limit of 3600 requests per hour was not enough for failure and recovery scenarios, and API usage from inside the cluster was not monitored. A tool that hides the provider's API errors slows diagnosis. After a full, non-graceful shutdown, Longhorn needed manual intervention to come back, which lengthened the outage.

   View case
2. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day

   DFDS2024

   Symptoms

   At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

   Node NotReadyContainer crash loopOOMKilledMonitoring outage

   Trigger

   Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limitDatabase

   Contributing factor

   Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot

   Detection

   A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

   Remediation

   • marked the EC2 instance unhealthy so a new node started
   • cordoned and force-drained the NotReady node and terminated its EC2 instance
   • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
   • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
   • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
   • emailed the team that owns the dev container, which confirmed it was working on a fix

   Action items

   • Move grafana-agent to the monitoring node group
   • Add alerts for when bandwidth across the cluster reaches a certain threshold
   • Investigate whether memory limits can be set on the grafana-agent Helm chart
   • Investigate grafana-agent memory usage
   • Ask Grafana support whether grafana-agent should use a PVC
   • Consider pushing node logs to Loki with limited retention
   • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes

   Lessons

   Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

   View case
3. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case
4. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics

   Kubernetes project2019

   Symptoms

   On 30 May 2019, in the run-up to the Kubernetes 1.15 code freeze, Prow test jobs in the k8s-prow-build cluster failed in surprising ways or were aborted partway through. Jobs were reported to GitHub as failed while Deck still showed them as running, because pods exited without uploading `finished.json`, the file most Prow tooling uses to tell a job's state.

   Node NotReadyNode kernel panicPods stuck in TerminatingThroughput degradation

   Trigger

   A spike in test load before the Kubernetes 1.15 code freeze pushed disk I/O on the build nodes far beyond the IOPS that GCP allows for 250 GB disks.

   Root cause

   Capacity exhaustion

   Component

   Cloud block storageNode root diskkubeletCI system

   Contributing factor

   Monitoring blind spotIgnored early signal

   Detection

   Several users reported jobs that GitHub showed as failed but Deck showed as still running, and the first report came on Slack at 13:26. Alerting for these failures did not exist or did not work.

   Remediation

   • Requested and received additional disk quota
   • Brought up 55 nodes with 500 GB disks in a new node pool
   • Migrated all nodes from the 250 GB disk pool to the 500 GB disk pool by draining old nodes, finishing at 23:55

   Action items

   • Create or fix monitoring so the oncall is paged when tide stops doing things
   • Set a process and guidelines for adding new members to the oncall rotation, with preparation and extra support

   Lessons

   Prow metrics were not recorded before the incident, which made it hard to understand, and alerting for these failures did not exist or did not work. After the cause was found and a fix devised, nothing happened for another two hours. The team counts it as luck that someone who recognised IOPS as the problem happened to be around to look.

   View case
5. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory

   NU.nl2019

   Symptoms

   The NU.nl development team got reports that several services on one of its Kubernetes clusters behaved erratically: occasional error pages, slow responses and timeouts. The affected services were mostly internal tools and dashboards.

   Node NotReadyHigh error rateHigh latencyIntermittent timeoutsMonitoring outageOOMKilled

   Trigger

   A new ElastAlert rule, deployed shortly before from the command line, that fired if the Cloudfront-* Elasticsearch indexes stopped receiving documents; those indexes take a few million requests per hour, and the rule made ElastAlert's memory grow without bound.

   Root cause

   Configuration change

   Component

   Container memory limitkubeletPrometheus

   Contributing factor

   No resource requests or limitsUnreviewed defaultsNo alert on limitMonitoring blind spotExcessive blast radius

   Detection

   Users reported error pages, slow responses and timeouts from several services; no alert fired.

   Remediation

   • terminated the NotReady nodes one by one so the auto-scaling group replaced them (the new nodes also went NotReady, but Prometheus and Grafana landed on a stable node)
   • found the ElastAlert pods' memory growth on the Grafana pod memory dashboard
   • fixed the ElastAlert problem that caused the memory growth

   Action items

   • Define alerts on resource usage such as CPU, memory and disk space
   • Keep researching an alerting strategy that combines Elasticsearch, Prometheus Alertmanager and New Relic
   • Consider exporting metrics out of the cluster and running Grafana outside it, on EC2 or a separate Kubernetes cluster
   • Either visualise the important metrics in the ELK stack or make Prometheus and Grafana more available, and improve the metrics strategy
   • Set resource limits for ElastAlert through the Helm values
   • Configure namespaces with default memory requests and limits
   • Isolate customer-facing workloads from build, logging and monitoring workloads (a dedicated node group, or later a separate tools cluster)
   • Deploy everything through automation such as Jenkins pipelines, and consider a GitOps approach
   • Deploy through a pipeline that includes a smoke test step
   • Make Kubernetes and cloud infrastructure work part of the team's sprints, picked up by all team members while pairing with more experienced members, and hold workshops that go deep into specific topics

   Lessons

   Fixing the root cause covers only a small part of what an incident can teach; the chain of smaller failures around it is where most of the learning is. A pod without a memory limit can take whole nodes down, so limits need defaults at the namespace level. Monitoring that runs inside the cluster it watches goes blind with it. Changes installed by hand from the command line leave the rest of the team without a record of what changed.

   View case

By root cause3