Cases2 cases

1 to 2 of 2

1. IL-0403 mybinder.org could not create pods for hours as the GKE gke-resource-quotas quota wrongly reported 15k pods in use

   Binder Project2022

   Symptoms

   On 2 June 2022 from 10:00 CET, mybinder.org stopped launching any new pods, and nobody on the team noticed until a user reported a Binder outage in the Matrix channel at 21:00.

   Deploys failingService outage

   Root cause

   Software defect

   Component

   GKE control plane

   Contributing factor

   Monitoring blind spot

   Detection

   A user reported a Binder outage in the Matrix channel at 21:00, and a team member who saw it found that pods had not been launching for several hours.

   Remediation

   • Deleted the gke-resource-quotas ResourceQuota, after which pods could be created again

   Action items

   • Address uptime monitoring and alerting

   Lessons

   It took 11 hours to realise there was a major outage. Once the right object was known, deleting it fixed the problem quickly, and the team was lucky that someone with the skills and permissions to make the change was at their computer at 11pm their time.

   View case
2. IL-0504 mybinder.org launches failed for about five hours as undeleted user pods piled up and filled the cluster

   Binder Project2018

   Symptoms

   On Saturday 8 July 2018 at about 18:30 CEST, the launch success rate of mybinder.org dropped to zero and stayed there. Suspicious behaviour was first reported on Gitter at 22:48, more than four hours later.

   Service outagePods PendingStale or inconsistent API reads

   Root cause

   Software defect

   Component

   JupyterHub

   Contributing factor

   No alert on limit

   Detection

   Suspicious behaviour was reported on Gitter at 22:48 CEST, more than four hours after launch success dropped to zero; no downtime notifications were set up.

   Remediation

   • Deleted user pods older than four hours with the delete-pods.py script
   • Culled pods of the most-used images, such as ipython-in-depth, more aggressively
   • Restarted the Hub pod after about 600 pods were culled, which brought launches back to normal
   • Cordoned some of the extra nodes so the autoscaler could reclaim them
   • Ran manual max-age culling the next morning, letting the cluster scale back to 3 nodes

   Action items

   • Set up notifications of downtime
   • Automate cluster scale-down
   • Document the downtime recovery steps so every team member knows them
   • Fix KubeSpawner's failure to delete pods when the pod reflector fails
   • Make KubeSpawner more robust to pod reflector failure
   • Refactor the event reflector into a single reflector instead of one per pod
   • Detect a likely unhealthy Hub state and restart the Hub without manual intervention

   Lessons

   The delete-pods.py script was useful for bulk-deleting old pods, and once the problem was noticed the cluster was healthy within a few minutes. The outage lasted five hours before anyone noticed because no downtime notifications existed, and scaling the cluster back down still took several manual steps of cordoning and draining nodes.

   View case

By root cause1