Cases12 cases

10 to 12 of 12

1. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub

   2i2c2020

   Symptoms

   At 08:52 PM IST a report came in from WER that many students could not access the JupyterHub and that it was frozen; pages were stuck and nothing was usable. A rise in activity was visible by 09:02 PM, but the usual fixes, such as opening an incognito window and restarting servers, did not help.

   Service outageSustained high node CPU

   Trigger

   WER students' pods used a good chunk of their memory limit (tests had shown they almost always use just under 1G of their 2G limit, against a 512M guarantee) on nodes shared with the hub's core pods.

   Root cause

   Capacity exhaustion

   Component

   Node poolContainer memory limit

   Contributing factor

   No resource requests or limitsUnverified assumptionExcessive blast radius

   Detection

   WER reported stuck pages and students unable to reach the hub in a GitHub issue (2i2c-org/docs#27).

   Remediation

   • removed memory overcommit for the WER students
   • raised resource guarantees for all core pods, which restarted them and moved some to a new node

   Action items

   • Put user pods in a separate pool so they do not create pressure on the core pods (infrastructure#89)
   • Set limits on the support infrastructure (Prometheus, Grafana, ingress) as well (infrastructure#90)
   • Document and think about overcommit ratios for memory usage (infrastructure#91)
   • Set up better Grafana dashboards to monitor resource usage (infrastructure#92)
   • Document how people can get kubectl access to the cluster so others can look into issues too (infrastructure#87)

   Lessons

   Core pods without resource guarantees get whatever is left on a node, so when user pods on the same nodes fill the CPU and memory, the hub that serves everyone freezes. The memory overcommit ratio has to come from measured use: WER students used just under 1G against a 512M guarantee, which a 4x overcommit could not hold.

   View case
2. IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limits

   Blue Matador2019

   Symptoms

   On a Saturday at 5:12 pm, Blue Matador's own monitoring raised a SystemOOM event for one node in the production Kubernetes cluster. Four minutes later a second warning reported a low EBS Burst Balance on that node's root volume. CloudWatch data showed the burst balance had actually bottomed out at 5:02 pm, before the memory event; the warning arrived late because EBS metrics lag 10 to 15 minutes.

   OOMKilled

   Trigger

   Memory use of the fluentd-sumologic log shipper pods grew until the node ran out of memory; log volume in Sumo Logic had been steady, so a burst of logs was not the cause.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limit

   Contributing factor

   No resource requests or limitsIgnored early signal

   Detection

   Blue Matador's own monitoring product alerted on the SystemOOM event, followed minutes later by a low EBS Burst Balance warning for the same node.

   Remediation

   • restarted all fluentd-sumologic pods so they started with low memory
   • set RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR to 0.9 on the fluentd-sumologic pods
   • added memory requests of 128Mi and limits of 1024Mi, and CPU requests of 100m and limits of 250m, to the fluentd-sumologic pods

   Action items

   • Apply default resource limits and resource quotas so that every pod runs with requests and limits

   Lessons

   When checking for restarted pods, look in every namespace, not only the application one. Set resource requests and limits on every pod, third-party ones in particular, so that one workload cannot exhaust a node. Investigate side signals such as the disk I/O spike while the logs still exist.

   View case
3. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries

   Zalando2019

   Symptoms

   Customers of the Zalando Fashion Store got a high rate of errors on all web product pages and outfit pages for more than an hour. The errors came from the main data aggregation layer service, which runs in one of Zalando's Kubernetes clusters.

   High error rateDNS resolution failuresOOMKilledMonitoring outage

   Trigger

   Calls from the aggregation layer to a single, not very important downstream service started timing out. The aggregation layer returned 404s for them, clients retried, and the resulting spike in requests became a spike in DNS queries to CoreDNS.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSContainer memory limitClient-side DNS resolver and cache

   Contributing factor

   Unknown limitNo cachingRetry stormUnreviewed defaultsExcessive blast radiusMonitoring blind spot

   Detection

   The on-call engineer for the aggregation layer phoned the on-call Kubernetes engineer. Automatic paging did not happen because the cluster's monitoring and alerting ran inside the cluster and failed along with DNS. The Kubernetes engineer then saw that the CoreDNS pods were being OOM-killed.

   Remediation

   • Raised the CoreDNS memory requests and limits by hand from 100Mi to 2000Mi

   Action items

   • Introduce a more resilient DNS setup: CoreDNS fronted by dnsmasq on every node
   • Improve high-level alerting from outside the clusters, so that a cluster problem is known before or at the same time as applications are affected
   • Improve incident response processes and tooling so that any on-call engineer can escalate directly to other teams when automatic paging fails, for example through a chat bot

   Lessons

   The Kubernetes DNS setup was far from resilient enough: a single application could take down DNS for everything in the cluster. The blast radius of one application sending more DNS queries should be limited to one node at most, ideally to single pods. Known Kubernetes DNS problems, such as Linux kernel races in conntrack handling and the ndots 5 problem, were familiar to the team and partly mitigated, but their combined effect on the DNS infrastructure was worse than the team had understood. Monitoring that runs inside the cluster was not robust enough: when the cluster's DNS broke, it could not page anyone, and recovery took longer than it should have.

   View case

By root cause1