Cases36 cases

31 to 35 of 36

1. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory

   NU.nl2019

   Symptoms

   The NU.nl development team got reports that several services on one of its Kubernetes clusters behaved erratically: occasional error pages, slow responses and timeouts. The affected services were mostly internal tools and dashboards.

   Node NotReadyHigh error rateHigh latencyIntermittent timeoutsMonitoring outageOOMKilled

   Trigger

   A new ElastAlert rule, deployed shortly before from the command line, that fired if the Cloudfront-* Elasticsearch indexes stopped receiving documents; those indexes take a few million requests per hour, and the rule made ElastAlert's memory grow without bound.

   Root cause

   Configuration change

   Component

   Container memory limitkubeletPrometheus

   Contributing factor

   No resource requests or limitsUnreviewed defaultsNo alert on limitMonitoring blind spotExcessive blast radius

   Detection

   Users reported error pages, slow responses and timeouts from several services; no alert fired.

   Remediation

   • terminated the NotReady nodes one by one so the auto-scaling group replaced them (the new nodes also went NotReady, but Prometheus and Grafana landed on a stable node)
   • found the ElastAlert pods' memory growth on the Grafana pod memory dashboard
   • fixed the ElastAlert problem that caused the memory growth

   Action items

   • Define alerts on resource usage such as CPU, memory and disk space
   • Keep researching an alerting strategy that combines Elasticsearch, Prometheus Alertmanager and New Relic
   • Consider exporting metrics out of the cluster and running Grafana outside it, on EC2 or a separate Kubernetes cluster
   • Either visualise the important metrics in the ELK stack or make Prometheus and Grafana more available, and improve the metrics strategy
   • Set resource limits for ElastAlert through the Helm values
   • Configure namespaces with default memory requests and limits
   • Isolate customer-facing workloads from build, logging and monitoring workloads (a dedicated node group, or later a separate tools cluster)
   • Deploy everything through automation such as Jenkins pipelines, and consider a GitOps approach
   • Deploy through a pipeline that includes a smoke test step
   • Make Kubernetes and cloud infrastructure work part of the team's sprints, picked up by all team members while pairing with more experienced members, and hold workshops that go deep into specific topics

   Lessons

   Fixing the root cause covers only a small part of what an incident can teach; the chain of smaller failures around it is where most of the learning is. A pod without a memory limit can take whole nodes down, so limits need defaults at the namespace level. Monitoring that runs inside the cluster it watches goes blind with it. Changes installed by hand from the command line leave the rest of the team without a record of what changed.

   View case
2. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh

   Target2019sev1

   Symptoms

   An upgrade of the network subsystem under Target's datacenter OpenStack, meant to cause only a brief disruption, ran into problems and left network connectivity disrupted for several hours. Kafka, which runs on that OpenStack, became intermittently available.

   Node NotReadyLoad imbalanceHigh latencyCredentials unavailable

   Trigger

   A network subsystem upgrade in the datacenter OpenStack infrastructure disrupted connectivity for several hours and made Kafka intermittently available.

   Root cause

   Dependency failure

   Component

   Container runtimeService discoveryTelemetry agentCredential broker for pods

   Contributing factor

   Excessive blast radiusAutomation amplificationSingle point of failureUnpatched known bug

   Remediation

   • enabled gossip encryption on Consul so messages from the poisoned mesh were rejected
   • stabilized the deployment engine and redeployed the development environment with encryption, terminating instances without it
   • upgraded Consul in place to 1.2.3 and then to 1.3.0

   Action items

   • Reduce how much of an outage a single Docker daemon can cause, together with running smaller and more numerous clusters

   Lessons

   Smaller and more numerous clusters limited the damage: the smaller development clusters and production were not hit the same way as the one large cluster, and dev and prod on one cluster would have been far worse. A Docker daemon shared by many workloads is a brittle failure point. Per-workload logging and metrics sidecars were judged the right design; a shared cluster-level agent would almost certainly have made the problem worse and harder to fix.

   View case
3. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries

   Zalando2019

   Symptoms

   Customers of the Zalando Fashion Store got a high rate of errors on all web product pages and outfit pages for more than an hour. The errors came from the main data aggregation layer service, which runs in one of Zalando's Kubernetes clusters.

   High error rateDNS resolution failuresOOMKilledMonitoring outage

   Trigger

   Calls from the aggregation layer to a single, not very important downstream service started timing out. The aggregation layer returned 404s for them, clients retried, and the resulting spike in requests became a spike in DNS queries to CoreDNS.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSContainer memory limitClient-side DNS resolver and cache

   Contributing factor

   Unknown limitNo cachingRetry stormUnreviewed defaultsExcessive blast radiusMonitoring blind spot

   Detection

   The on-call engineer for the aggregation layer phoned the on-call Kubernetes engineer. Automatic paging did not happen because the cluster's monitoring and alerting ran inside the cluster and failed along with DNS. The Kubernetes engineer then saw that the CoreDNS pods were being OOM-killed.

   Remediation

   • Raised the CoreDNS memory requests and limits by hand from 100Mi to 2000Mi

   Action items

   • Introduce a more resilient DNS setup: CoreDNS fronted by dnsmasq on every node
   • Improve high-level alerting from outside the clusters, so that a cluster problem is known before or at the same time as applications are affected
   • Improve incident response processes and tooling so that any on-call engineer can escalate directly to other teams when automatic paging fails, for example through a chat bot

   Lessons

   The Kubernetes DNS setup was far from resilient enough: a single application could take down DNS for everything in the cluster. The blast radius of one application sending more DNS queries should be limited to one node at most, ideally to single pods. Known Kubernetes DNS problems, such as Linux kernel races in conntrack handling and the ndots 5 problem, were familiar to the team and partly mitigated, but their combined effect on the DNS infrastructure was worse than the team had understood. Monitoring that runs inside the cluster was not robust enough: when the cluster's DNS broke, it could not page anyone, and recovery took longer than it should have.

   View case
4. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest

   Zalando2018

   Symptoms

   Customers were affected, and Zalando's IAM service answered requests with HTTP 404. The talk shows this on slides titled "Customer impact" and "IAM returning 404", followed by a slide on the number of pods in the cluster.

   High error rateControl plane unresponsiveOOMKilled

   Trigger

   A CronJob (apiVersion batch/v2alpha1, schedule "*/15 9-19 * * Mon-Fri") whose concurrencyPolicy: Forbid and job history limits sat inside the pod template spec instead of the CronJob spec. The talk shows it right after the slides on the number of pods and the API server being OOM-killed, and the lessons add a quota on the number of pods.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverIngress controller

   Contributing factor

   Single point of failureExcessive blast radius

   Remediation

   • fixed the CronJob manifest: moved concurrencyPolicy: Forbid and the job history limits to the CronJob spec, added activeDeadlineSeconds: 120 and changed the schedule to "7 8-18 * * Mon-Fri"

   Action items

   • Make the ingress stay healthy during API server problems
   • Make the ingress keep the last known set of routes
   • Use a ResourceQuota on the number of pods (the example sets pods: "1500")

   Lessons

   An AWS ALB routes traffic to all hosts when all hosts report unhealthy, so the ingress should stay healthy during API server problems. An ingress controller that reads routes from the API server should keep its last known set of routes. Use a quota for the number of pods.

   View case
5. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction

   Oath2018

   Symptoms

   During a rolling update of a kubelet change in one of Oath's on-premises production clusters, nodes started going NotReady, and soon all nodes in the cluster were NotReady.

   Node NotReadyPod evictionPods Pending

   Trigger

   A change to kubelet was rolled out across the nodes of one cluster; on restart, kubelet cleans up transient cgroup files, and a Kubernetes bug in that step left kubelet stuck instead of starting.

   Root cause

   Software defect

   Component

   kubeletKubernetes controller manager

   Contributing factor

   No health probesExcessive blast radius

   Detection

   Nodes reported NotReady and the eviction graphs climbed while the kubelet rollout was in progress.

   Remediation

   • failed the affected data center over so users were routed to the next available data center

   Action items

   • Run a functional test of kubelet after each upgrade batch and certify it before the next 30% of nodes are updated

   Lessons

   Check kubelet after upgrading it before moving to the next batch; a simple curl against the kubelet API would have caught this. The node controller's eviction throttling and full disruption mode, which the speakers note is not well documented, kept a cluster-wide kubelet failure from evicting every pod.

   View case

By root cause8