Incident Library

Organisation

UC Berkeley

UC Berkeley · uc-berkeley

4 cases · 2017 to 2019

Root cause Capacity exhaustion1Cloud provider outage1Operator error1unknown1
Component AKS control plane1Cloud block storage1Container runtime1Database1JupyterHub1Kubernetes API server1kubelet1Container memory limit1Node pool1Persistent volumes and stateful sets1
Contributing factor Unknown limit1Limited debug access1Harmful manual action1No safeguard on destructive change1Unsafe failure behaviour1
Symptom Service outage2Control plane unresponsive1Data loss1High error rate1Pods Pending1Pods stuck in Terminating1Misrouted requests1

Cases

  1. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreatedUC Berkeley · 2019

    Root cause Cloud provider outage

  2. IL-0507 Student pods on UC Berkeley DataHub got stuck in Terminating or would not start because Docker hung on two Azure nodesUC Berkeley · 2017

    Root cause unknown

  3. IL-0468 New JupyterHub student pods stuck Pending on MaxVolumeCount after a memory limit cut with node size unchangedUC Berkeley · 2017

    Root cause Capacity exhaustion

  4. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwrittenUC Berkeley · 2017

    Root cause Operator error