Cases17 cases

10 to 14 of 17

1. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes

   Palark2023

   Symptoms

   One evening, right after an ordinary restart of kube-apiserver in a customer's production cluster, etcd memory consumption rose almost threefold. The sudden growth led to a cascading restart of all the master nodes.

   Control plane unresponsiveDatabase overload

   Trigger

   An ordinary restart of kube-apiserver, after which every pod of the log collector DaemonSet sent its list requests at the same time.

   Root cause

   Capacity exhaustion

   Component

   etcdKubernetes API serverTelemetry agent

   Contributing factor

   Unknown limitUnreviewed defaults

   Detection

   Memory graphs showed etcd memory on the master nodes almost tripling after a kube-apiserver restart; the audit logs then pointed to one DaemonSet's list requests.

   Remediation

   • created a PriorityLevelConfiguration named limit-list-custom that queues requests (assuredConcurrencyShares 5, 16 queues, handSize 4, queueLengthLimit 50)
   • created a FlowSchema that routes list and get requests on pods from the log collector's service account to that priority level

   Action items

   • Submit a pull request adding resource version support to the Kubernetes Rust library
   • Open a pull request to the Vector log collector to address the startup list requests
   • Apply the same flow-control approach to a similar problem found with the Cilium CNI

   Lessons

   A field selector reduces what kube-apiserver returns, not what it reads from etcd: listing the pods of one node still reads every pod in the cluster. A DaemonSet whose pods all list at startup multiplies that by the node count each time kube-apiserver restarts. Where exact freshness is not needed, resourceVersion=0 lets kube-apiserver answer from its cache; where the client cannot set it, API Priority and Fairness can queue the requests so they reach etcd gradually.

   View case
2. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes

   DFDS2023sev1

   Symptoms

   At 09:33 UTC on 14 February 2023 end users told the platform team that Kubernetes logs were not arriving in CloudWatch as expected. The investigation went through fluentd and found that the nodes were running containerd as their container runtime instead of docker-shim. A scheduled node rollover the day before, at 07:30 UTC on 13 February, had set containerd as the default runtime by accident.

   Service outageResources deletedMonitoring outage

   Trigger

   A scheduled node rollover had accidentally made containerd the default container runtime, which stopped log shipping to CloudWatch; the unscheduled rollover to revert it timed out, and the pipeline run that followed reduced the Auto Scaling group from 30 to 15 instances.

   Root cause

   Configuration change

   Component

   Container runtimeNode launch templateNode poolTerraformTelemetry agent

   Contributing factor

   Automation amplificationUnreviewed defaultsUnrepresentative stagingNo configuration testsNo runbookHarmful manual action

   Detection

   End users reported the missing CloudWatch logs that started the work. The postmortem lists 2 minutes as the time to detection of the node failure.

   Remediation

   • ran the node rollover script to add new nodes, which failed on a timeout from Docker Hub
   • ran the node rollover script a second time, which added new nodes that the load balancer marked healthy at 11:01 UTC

   Action items

   • Document the rollover process, especially what can be done when it goes wrong
   • Investigate options for improving the rollover process
   • Add fluentd log collection to the QA test suite so logs are covered end to end
   • Add alarms for fluentd when too little or no log data is sent
   • Implement a staging environment
   • Implement guards against selecting an invalid container runtime

   Lessons

   When a change goes wrong, rolling back to a known working configuration is safer than rushing a fix, which can make the damage worse. Launching new worker nodes depends on the cloud-init script not failing in any way. The QA configuration differed from production, there was no staging environment, the QA tests did not cover fluentd log collection, and the rollover process had no documentation for when it fails.

   View case
3. IL-0041 fluent-bit silently stopped shipping application logs from one node to Elasticsearch

   PrometheusKube2020

   Symptoms

   On a Tuesday morning a developer reported that some production logs were missing. Only the logs from a single node were gone; the other nodes kept shipping normally through fluent-bit, the log shipper the team had run for more than three years.

   Monitoring outage

   Root cause

   Software defect

   Component

   Telemetry agentElasticsearch or OpenSearch

   Contributing factor

   Monitoring blind spotUnpatched known bug

   Detection

   A developer reported missing production logs; no alert fired because fluent-bit's metrics showed no errors.

   Remediation

   • added CPU to the Elasticsearch virtual machines, which cleared the write queue rejections
   • rolled out Fluentd one node at a time, using node affinity to place Fluentd and anti-affinity to remove fluent-bit
   • lowered Fluentd's flush_thread_count to 1 and set flush_interval to 5 seconds so the backlog of old logs did not overflow the buffer
   • enabled Fluentd's Position DB so already-ingested data was skipped

   Action items

   • Build monitoring for Fluentd on its Prometheus integration, including a throughput alert with separate day and night thresholds or a black-box log check

   Lessons

   A log shipper can fail without an error message and with metrics that simply stop moving, so error-based alerts never fire; throughput alerts or end-to-end checks can catch it. Open source software can break in ways nobody expects, and sometimes the way out is to switch to a different shipper fast, node by node. A new shipper that starts from scratch replays every old log on the node and needs to be slowed down.

   View case
4. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic

   Airbnb2019

   Symptoms

   An operation to upgrade the stats agent in Kubernetes was announced in advance, with a warning that metrics might be unavailable to services for a few minutes. A few minutes later a reply came in saying that all of the sender's Kubernetes services seemed to be down.

   Probe failuresService outage

   Trigger

   A scheduled upgrade of the stats agent sidecar, during which the agent container crashed.

   Root cause

   Faulty deployment

   Component

   Telemetry agentService discovery

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   A reply to the operation announcement, a few minutes later, said that all of the sender's Kubernetes services appeared to be down.

   Action items

   • Mark logging, tracing and stats agent containers as non-critical and leave them out of the service mesh's pod readiness check

   Lessons

   Every container in a pod affects the pod's readiness, so each sidecar needs a deliberate decision about how its health should count. A sidecar that crashes is not ready even when it has no probes, and any logic that requires all containers to be ready will take the pod out of traffic.

   View case
5. IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limits

   Blue Matador2019

   Symptoms

   On a Saturday at 5:12 pm, Blue Matador's own monitoring raised a SystemOOM event for one node in the production Kubernetes cluster. Four minutes later a second warning reported a low EBS Burst Balance on that node's root volume. CloudWatch data showed the burst balance had actually bottomed out at 5:02 pm, before the memory event; the warning arrived late because EBS metrics lag 10 to 15 minutes.

   OOMKilled

   Trigger

   Memory use of the fluentd-sumologic log shipper pods grew until the node ran out of memory; log volume in Sumo Logic had been steady, so a burst of logs was not the cause.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limit

   Contributing factor

   No resource requests or limitsIgnored early signal

   Detection

   Blue Matador's own monitoring product alerted on the SystemOOM event, followed minutes later by a low EBS Burst Balance warning for the same node.

   Remediation

   • restarted all fluentd-sumologic pods so they started with low memory
   • set RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR to 0.9 on the fluentd-sumologic pods
   • added memory requests of 128Mi and limits of 1024Mi, and CPU requests of 100m and limits of 250m, to the fluentd-sumologic pods

   Action items

   • Apply default resource limits and resource quotas so that every pod runs with requests and limits

   Lessons

   When checking for restarted pods, look in every namespace, not only the application one. Set resource requests and limits on every pod, third-party ones in particular, so that one workload cannot exhaust a node. Investigate side signals such as the disk I/O spike while the logs still exist.

   View case

By root cause6