Cases6 cases

3 to 6 of 6

1. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents

   Palark2023

   Symptoms

   A tech support engineer called Palark to say that a customer's cluster, production environment included, had practically stopped working. On the control plane nodes the kube-apiserver processes had used up all the memory and were crashing and restarting over and over, so the Kubernetes API was unreachable and completely non-functional.

   Control plane unresponsiveOOMKilled

   Trigger

   Cilium agent pods on more than 200 nodes sent LIST requests to the Kubernetes API en masse at the same time.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverCilium

   Contributing factor

   Unreviewed defaults

   Detection

   A tech support engineer phoned the team to report that the customer's cluster was practically non-operational.

   Remediation

   • added CPU and memory to the control plane nodes in two steps, from 8 CPUs and 16 GB of RAM to 16 CPUs and 64 GB, until the API stabilised
   • restarted the Cilium agents in a test window agreed with the customer to reproduce the memory growth
   • deployed a FlowSchema and a PriorityLevelConfiguration named cilium-pods that limit and queue LIST requests from the Cilium service accounts to cilium.io resources
   • reduced the control plane nodes back to their original resources

   Action items

   • Make an API Priority and Fairness configuration a standard part of Palark's Kubernetes platform

   Lessons

   In a large cluster, many agents sending LIST requests at once can exhaust kube-apiserver memory, and with several API servers the crash moves from one to the next. APF can hold back such a client: a FlowSchema matches its requests and a PriorityLevelConfiguration gives them a small concurrency share and a queue, so the requests run more slowly instead of all at once. With APF enabled, the total concurrency it divides is the sum of --max-requests-inflight (default 400) and --max-mutating-requests-inflight (default 200), shared among priority levels in proportion to their assured concurrency shares, so raising one level's share lowers every other level's limit. Long-running requests such as logs and exec, and WATCH requests, are not subject to APF. The response headers X-Kubernetes-PF-FlowSchema-UID and X-Kubernetes-PF-PriorityLevel-UID show where a request landed, and the metrics Apiserver_flowcontrol_rejected_requests_total, Apiserver_current_inqueue_requests and Apiserver_flowcontrol_request_execution_seconds, with the debug endpoints /debug/api_priority_and_fairness/dump_priority_levels and /debug/api_priority_and_fairness/dump_queues, show how the queues behave.

   View case
2. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout

   Grafana Labs2022

   Symptoms

   During a rollout of the Grafana Loki distributor Deployment in one Grafana Cloud Logs environment, the write path went down for about 25 minutes. The newly started distributor pods were OOM-killed over and over, never passed their health and readiness checks, and the rollout stopped making progress.

   Service outageOOMKilledLoad imbalanceProbe failures

   Trigger

   Step 1 of moving the Loki ring off a separately deployed key-value store onto embedded memberlist added a new label, gossip_ring_member: 'true', to the distributor pods and Service, together with a small config change, and the change required a rollout of the distributors.

   Root cause

   Configuration change

   Component

   Configuration templatingService discoveryContainer memory limit

   Contributing factor

   Unreviewed defaultsIgnored early signalUnrepresentative staging

   Detection

   The first clue was kubectl get pods and the kube_deployment_status_replicas_updated metric showing that only 16 of 140 distributor pods had been updated.

   Remediation

   • scaled the distributor Deployment out horizontally so the load spread over more pods with the new label and the rollout could continue
   • reverted the migration step to end the outage
   • set service_ignored_labels:: [$._config.gossip_member_label] and passed it to k.util.serviceFor so the label is no longer copied into the Service selector

   Action items

   • Update the function that generates a Service from a Deployment or StatefulSet to handle edge cases, including this one

   Lessons

   A Service whose selector is built by copying every label of a workload changes which pods it matches as soon as a new label is added. During a rollout it then matches only the new pods, which receive all the traffic before there are enough of them. A few errors during a rollout in other environments hid the problem, because those environments gave distributors more memory and could absorb the shift. Manifest diffs kept for every pull request made it possible to see that the Service selector had changed.

   View case
3. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory

   NU.nl2019

   Symptoms

   The NU.nl development team got reports that several services on one of its Kubernetes clusters behaved erratically: occasional error pages, slow responses and timeouts. The affected services were mostly internal tools and dashboards.

   Node NotReadyHigh error rateHigh latencyIntermittent timeoutsMonitoring outageOOMKilled

   Trigger

   A new ElastAlert rule, deployed shortly before from the command line, that fired if the Cloudfront-* Elasticsearch indexes stopped receiving documents; those indexes take a few million requests per hour, and the rule made ElastAlert's memory grow without bound.

   Root cause

   Configuration change

   Component

   Container memory limitkubeletPrometheus

   Contributing factor

   No resource requests or limitsUnreviewed defaultsNo alert on limitMonitoring blind spotExcessive blast radius

   Detection

   Users reported error pages, slow responses and timeouts from several services; no alert fired.

   Remediation

   • terminated the NotReady nodes one by one so the auto-scaling group replaced them (the new nodes also went NotReady, but Prometheus and Grafana landed on a stable node)
   • found the ElastAlert pods' memory growth on the Grafana pod memory dashboard
   • fixed the ElastAlert problem that caused the memory growth

   Action items

   • Define alerts on resource usage such as CPU, memory and disk space
   • Keep researching an alerting strategy that combines Elasticsearch, Prometheus Alertmanager and New Relic
   • Consider exporting metrics out of the cluster and running Grafana outside it, on EC2 or a separate Kubernetes cluster
   • Either visualise the important metrics in the ELK stack or make Prometheus and Grafana more available, and improve the metrics strategy
   • Set resource limits for ElastAlert through the Helm values
   • Configure namespaces with default memory requests and limits
   • Isolate customer-facing workloads from build, logging and monitoring workloads (a dedicated node group, or later a separate tools cluster)
   • Deploy everything through automation such as Jenkins pipelines, and consider a GitOps approach
   • Deploy through a pipeline that includes a smoke test step
   • Make Kubernetes and cloud infrastructure work part of the team's sprints, picked up by all team members while pairing with more experienced members, and hold workshops that go deep into specific topics

   Lessons

   Fixing the root cause covers only a small part of what an incident can teach; the chain of smaller failures around it is where most of the learning is. A pod without a memory limit can take whole nodes down, so limits need defaults at the namespace level. Monitoring that runs inside the cluster it watches goes blind with it. Changes installed by hand from the command line leave the rest of the team without a record of what changed.

   View case
4. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries

   Zalando2019

   Symptoms

   Customers of the Zalando Fashion Store got a high rate of errors on all web product pages and outfit pages for more than an hour. The errors came from the main data aggregation layer service, which runs in one of Zalando's Kubernetes clusters.

   High error rateDNS resolution failuresOOMKilledMonitoring outage

   Trigger

   Calls from the aggregation layer to a single, not very important downstream service started timing out. The aggregation layer returned 404s for them, clients retried, and the resulting spike in requests became a spike in DNS queries to CoreDNS.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSContainer memory limitClient-side DNS resolver and cache

   Contributing factor

   Unknown limitNo cachingRetry stormUnreviewed defaultsExcessive blast radiusMonitoring blind spot

   Detection

   The on-call engineer for the aggregation layer phoned the on-call Kubernetes engineer. Automatic paging did not happen because the cluster's monitoring and alerting ran inside the cluster and failed along with DNS. The Kubernetes engineer then saw that the CoreDNS pods were being OOM-killed.

   Remediation

   • Raised the CoreDNS memory requests and limits by hand from 100Mi to 2000Mi

   Action items

   • Introduce a more resilient DNS setup: CoreDNS fronted by dnsmasq on every node
   • Improve high-level alerting from outside the clusters, so that a cluster problem is known before or at the same time as applications are affected
   • Improve incident response processes and tooling so that any on-call engineer can escalate directly to other teams when automatic paging fails, for example through a chat bot

   Lessons

   The Kubernetes DNS setup was far from resilient enough: a single application could take down DNS for everything in the cluster. The blast radius of one application sending more DNS queries should be limited to one node at most, ideally to single pods. Known Kubernetes DNS problems, such as Linux kernel races in conntrack handling and the ndots 5 problem, were familiar to the team and partly mitigated, but their combined effect on the DNS infrastructure was worse than the team had understood. Monitoring that runs inside the cluster was not robust enough: when the cluster's DNS broke, it could not page anyone, and recovery took longer than it should have.

   View case

By root cause3