Cases19 cases

18 to 19 of 19

1. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down

   Saltside2017

   Symptoms

   While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

   Pods PendingImage pull failuresNode NotReadyService outage

   Trigger

   A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

   Root cause

   Capacity exhaustion

   Component

   Node root diskkubeletContainer runtimeHelm

   Contributing factor

   Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure

   Detection

   The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

   Remediation

   • terminated the three worker nodes and let the auto scaling group replace them

   Action items

   • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
   • Increase the worker root EBS volumes from 20GB to 120GB
   • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
   • Configure different allocation limits for the different layers in Kubernetes
   • Discuss a highly available Tiller with the Helm team
   • Find a way to throttle the number of newly created pods

   Lessons

   An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

   View case
2. IL-0468 New JupyterHub student pods stuck Pending on MaxVolumeCount after a memory limit cut with node size unchanged

   UC Berkeley2017

   Symptoms

   On 20 March 2017, from some time early in the day until about 13:00, some new student servers on UC Berkeley's JupyterHub stayed in Pending forever and the students got 500 errors that kept them from logging in.

   Pods PendingHigh error rate

   Trigger

   On 18 March at 16:30 the RAM per student was reduced from 2G to 1G to save resources while the nodes kept the same size of 26G RAM, and as a result the nodes hit the Google Cloud the limit on the number of disks per node.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitCloud block storageNode pool

   Contributing factor

   Unknown limit

   Detection

   New student servers reported a 500 error that kept students from logging in. At 12:44 this was judged widespread, and at 12:53 `kubectl describe pod` showed the pods Pending on MaxVolumeCount.

   Remediation

   • Copied the Google Container Engine instance template and changed it to spawn smaller n1-highmem-2 nodes
   • Switched the managed instance group to the new template
   • Grew the instance group by 30 new nodes
   • Ran populate.bash so student pods would start on time on the new nodes
   • Stopped the simple autoscaler
   • Stopped and started affected user pods from the control panel
   • Restarted the hub and proxy pods

   Action items

   • Follow the status of kubernetes/kubernetes issue 24317
   • Find and fix the cause of the 'too many redirects' error in JupyterHub
   • Delete the older, larger nodes that are no longer in use
   • Alert when any pods stay Pending for a non-negligible time

   View case

By root cause6