IL-0075 Civis Analytics · 2019-01-16 (published)kubernetesawsobservability
Civis Analytics runs its Data Science Platform on Kubernetes: Jupyter notebooks, client services and, since autumn 2018, client scripts. When script workloads moved to Kubernetes, the cluster grew to about 290 EC2 instances and the three API servers, running on m4.large masters with no CPU limit, saturated CPU and memory and made the control plane unusable. Moving internal users back to the old infrastructure stabilised the cluster. The author names the main mistake as assuming that Kubernetes handles scale for free.
The team set a goal of 500 nodes, with 1,000 as a stretch, and ran load tests on a staging cluster by raising the size of the AWS autoscaling group. They added Prometheus through the CoreOS Prometheus operator for API metrics. Larger masters (three m4.4xlarge, 16 CPU and 64 GiB each, against the Kubernetes recommendation of c4.4xlarge for 251 to 500 nodes) helped, but the cluster still struggled before 500 nodes.
Load tests without the Datadog and Sumologic (fluentd-based) DaemonSets reached about 800 nodes before latency and 5xx errors rose; adding Sumologic back still reached more than 700, which pointed at Datadog. With DD_KUBERNETES_COLLECT_METADATA_TAGS, every Datadog agent, one per node, polled the API server for the names of all services so it could tag pod metrics, and those tags were not even useful to Civis. Datadog offers two fixes: turning that setting off, or running the Datadog Cluster Agent so only one agent talks to the API server. With the Datadog changes the cluster reached more than 700 EC2 instances easily, and Civis concluded it could support 700 nodes.
Two other changes stayed in place without making a large difference: max-requests-inflight for non-mutating requests went from 400 to 1200, which reduced 429 responses, and the etcd EBS volumes grew from 20 to 100 GB, raising IOPS from 100 to 300. The team also found kube2iam running out of memory because it kept every pod's metadata in memory, and notes the same pattern in the fluentd Kubernetes metadata filter. For the later DNS timeouts, scripts were switched from the ClusterFirst DNS policy to Default, which skips cluster DNS.
After Civis released its new script infrastructure on Kubernetes internally in September 2018, developers reported that kubectl commands were slow, and users intermittently could not start notebooks and services.
The serious failures came when the cluster was loaded with a large number of scripts and grew to about 290 EC2 instances; it had never run more than 80 before. CPU on the master nodes went high, and the three Kubernetes API server containers were pegged on CPU. The API servers had no CPU limit, so they starved the other control plane components on the same nodes and the cluster became unusable. The API servers also used up all the memory on their node before restarting.
The team could see master-node CPU in CloudWatch and per-container CPU and memory through Datadog agents, but not API request throughput, 5xx error rates or request latency. Once they deployed Prometheus, their load tests showed API response latency and 5xx error rates spiking at peak load, and the API server access logs contained several 429 responses.
After the infrastructure was released again, users reported scripts failing with DNS timeout errors. The team could not reproduce them reliably and only connected them to the larger cluster size.
Symptom Control plane lagControl plane unresponsiveHigh latency
Root cause Capacity exhaustion
Developer reports of slow kubectl commands and user reports of notebooks and services failing to start, then high CPU on the master nodes.
Kubernetes can reach thousands of nodes only after tuning, and third-party DaemonSets are a common cause of control plane load: an agent on every node that polls or watches the API server multiplies with the cluster size. Datadog, kube2iam and the fluentd metadata filter all showed this. A small configuration change often fixes it, but finding it took load testing and the API server access logs.
IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023same root cause · shared Kubernetes API server, Telemetry agent, etcd
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared Kubernetes API server, etcd
IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019same root cause · shared Kubernetes API server, etcd
IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk spaceUK Ministry of Justice · 2024same root cause · shared Telemetry agent
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Telemetry agent
IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024same root cause · shared Kubernetes API server
IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategyUK Ministry of Justice · 2023same root cause · shared Telemetry agent
IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agentsPalark · 2023same root cause · shared Kubernetes API server
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Kubernetes API server
IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Telemetry agent
IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019same root cause · shared Kubernetes API server
IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018same root cause · shared Kubernetes API server