Cases4 cases

1 to 4 of 4

1. IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limits

   Blue Matador2019

   Symptoms

   On a Saturday at 5:12 pm, Blue Matador's own monitoring raised a SystemOOM event for one node in the production Kubernetes cluster. Four minutes later a second warning reported a low EBS Burst Balance on that node's root volume. CloudWatch data showed the burst balance had actually bottomed out at 5:02 pm, before the memory event; the warning arrived late because EBS metrics lag 10 to 15 minutes.

   OOMKilled

   Trigger

   Memory use of the fluentd-sumologic log shipper pods grew until the node ran out of memory; log volume in Sumo Logic had been steady, so a burst of logs was not the cause.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limit

   Contributing factor

   No resource requests or limitsIgnored early signal

   Detection

   Blue Matador's own monitoring product alerted on the SystemOOM event, followed minutes later by a low EBS Burst Balance warning for the same node.

   Remediation

   • restarted all fluentd-sumologic pods so they started with low memory
   • set RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR to 0.9 on the fluentd-sumologic pods
   • added memory requests of 128Mi and limits of 1024Mi, and CPU requests of 100m and limits of 250m, to the fluentd-sumologic pods

   Action items

   • Apply default resource limits and resource quotas so that every pod runs with requests and limits

   Lessons

   When checking for restarted pods, look in every namespace, not only the application one. Set resource requests and limits on every pod, third-party ones in particular, so that one workload cannot exhaust a node. Investigate side signals such as the disk I/O spike while the logs still exist.

   View case
2. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop

   Fullstaq2019

   Symptoms

   Dex, the OIDC proxy that handled Kubernetes authentication, stored its data in the Kubernetes API (storage type kubernetes, inCluster: true) and had built up a huge number of AuthRequest objects. Listing all AuthRequests timed out, and without the list they could not be deleted.

   Control plane unresponsiveMonitoring outage

   Trigger

   The dex namespace was deleted so that Kubernetes would clean up the accumulated AuthRequest objects itself.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serveretcdCluster authentication provider

   Contributing factor

   Unreviewed defaultsHarmful manual actionRetry stormLimited debug accessIgnored early signal

   Detection

   The speaker says he had got hints of the problem earlier. While the masters were down, plenty of alerts arrived in Slack, but monitoring could not be reached because authentication and port-forwarding no longer worked.

   Remediation

   • removed the kubernetes finalizer from the dex namespace by calling the API directly (kill-kube-ns)
   • deployed Dex to a new namespace with sane defaults

   Lessons

   Dex with Kubernetes as its storage keeps every AuthRequest as an API object, and with a long TTL they pile up until the API cannot list them. Deleting a namespace that holds that many objects makes Kubernetes list them in a loop and can overload the masters. Monitoring that is reached through cluster authentication is lost when authentication is what broke.

   View case
3. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics

   Kubernetes project2019

   Symptoms

   On 30 May 2019, in the run-up to the Kubernetes 1.15 code freeze, Prow test jobs in the k8s-prow-build cluster failed in surprising ways or were aborted partway through. Jobs were reported to GitHub as failed while Deck still showed them as running, because pods exited without uploading `finished.json`, the file most Prow tooling uses to tell a job's state.

   Node NotReadyNode kernel panicPods stuck in TerminatingThroughput degradation

   Trigger

   A spike in test load before the Kubernetes 1.15 code freeze pushed disk I/O on the build nodes far beyond the IOPS that GCP allows for 250 GB disks.

   Root cause

   Capacity exhaustion

   Component

   Cloud block storageNode root diskkubeletCI system

   Contributing factor

   Monitoring blind spotIgnored early signal

   Detection

   Several users reported jobs that GitHub showed as failed but Deck showed as still running, and the first report came on Slack at 13:26. Alerting for these failures did not exist or did not work.

   Remediation

   • Requested and received additional disk quota
   • Brought up 55 nodes with 500 GB disks in a new node pool
   • Migrated all nodes from the 250 GB disk pool to the 500 GB disk pool by draining old nodes, finishing at 23:55

   Action items

   • Create or fix monitoring so the oncall is paged when tide stops doing things
   • Set a process and guidelines for adding new members to the oncall rotation, with preparation and extra support

   Lessons

   Prow metrics were not recorded before the incident, which made it hard to understand, and alerting for these failures did not exist or did not work. After the cause was found and a fix devised, nothing happened for another two hours. The team counts it as luck that someone who recognised IOPS as the problem happened to be around to look.

   View case
4. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down

   Saltside2017

   Symptoms

   While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

   Pods PendingImage pull failuresNode NotReadyService outage

   Trigger

   A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

   Root cause

   Capacity exhaustion

   Component

   Node root diskkubeletContainer runtimeHelm

   Contributing factor

   Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure

   Detection

   The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

   Remediation

   • terminated the three worker nodes and let the auto scaling group replace them

   Action items

   • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
   • Increase the worker root EBS volumes from 20GB to 120GB
   • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
   • Configure different allocation limits for the different layers in Kubernetes
   • Discuss a highly available Tiller with the Helm team
   • Find a way to throttle the number of newly created pods

   Lessons

   An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

   View case

By root cause1