Cases14 cases

12 to 14 of 14

1. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated

   UC Berkeley2019

   Symptoms

   On 25 February 2019 the Kubernetes API server of the managed Azure cluster running the data100 hub became unreachable. kubectl failed for every operation and requests to create new resources failed, so when the hub pod stopped, no new one was created and users saw a proxy error message.

   Control plane unresponsiveService outage

   Root cause

   Cloud provider outage

   Component

   AKS control planeKubernetes API serverkubelet

   Contributing factor

   Limited debug access

   Detection

   Infrastructure staff were notified in Slack at 11:34 and found that kubectl failed for all operations while the hub proxy was up.

   Remediation

   • Created a new cluster and requested a new NFS server in its resource group
   • Detached the data volumes from the old NFS server and moved them to the new cluster's resource group
   • Imported the ZFS pool into the new NFS server
   • Created the staging hub with helm and updated DNS
   • Created the prod hub with helm and updated DNS

   Action items

   • Monitor the API server endpoint remotely and alert when it is down

   View case
2. IL-0505 kubectl logs failed on GKE with "No SSH tunnels currently open" and mybinder.org build logs stopped reaching users

   Binder Project2019

   Symptoms

   On 20 February 2019 at 08:03 CET, `kubectl logs -f <pod>` stopped working for every pod in the mybinder.org cluster on GKE. Because pod logs could not be read, build logs were no longer forwarded to users, although the builds themselves ran normally.

   Monitoring outageDeploys failing

   Root cause

   Cloud provider outage

   Component

   GKE control planeKubernetes API server

   Contributing factor

   Limited debug access

   Remediation

   • Cordoned node gke-prod-a-users-b921bb88-rd5d so it would be removed from the pool once empty
   • Drained the node and removed it from the pool, after which kubectl logs worked again

   Action items

   • Test that kubectl logs works and notify when it does not
   • Add the command for retrieving API server logs to the SRE guide
   • Describe in the SRE guide how to identify suspicious node networking status

   Lessons

   The root cause was very difficult to identify, partly because it appears to sit in the Kubernetes master or the GKE service itself. Capturing `kubectl describe node` for the suspicious node would have allowed a comparison with healthy nodes, and a node that reports no ExternalIP or InternalIP in its status was likely a direct symptom of the problem and can be used as a diagnostic in the future.

   View case
3. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services

   Kubernetes project2019

   Symptoms

   On 8 February 2019 the Kubernetes project's Prow instance at prow.k8s.io became very slow: its controllers took about five minutes to respond to the hook, bot commands such as the cat command stopped working, and the prow.k8s.io site was hard to reach. GitHub status was normal, and the Prow service nodes were found running above 100% of CPU and memory.

   Pods PendingSustained high node CPUHigh latency

   Trigger

   At 13:20 PST an auto-generated bump PR for Prow was merged and deployed, and Prow started scheduling workloads onto the service cluster.

   Root cause

   Faulty deployment

   Component

   CI system

   Contributing factor

   Unrepresentative stagingNo resource requests or limitsLimited debug access

   Detection

   While working on a PR, a maintainer noticed the bot's cat command failing, found GitHub status normal, learned on Slack that the Prow nodes were above 100% CPU and memory, and then found 600 pending pods in the test-pods namespace.

   Remediation

   • Rolled Prow back to v20190206-b01b288 and cleaned up the pods and ProwJobs in the service cluster
   • Applied the revert by hand with kubectl apply after the deployment permission was missing
   • Sent a fix, PR 11209, so the client no longer overwrites a provided literal default client
   • Rolled Prow back to v20190125-2aca69d, from before the bug was introduced
   • Cleaned up the wrongly scheduled pods and ProwJobs again and confirmed Prow was healthy

   Action items

   • Find out why the bug did not show deterministically
   • Be more careful with sweeping Prow changes, with more end-to-end tests or a real staging Prow

   Lessons

   The behaviour was not deterministic, and the change was deployed on a Friday afternoon. Prow instances are set up differently, for example in how build clusters are defined, so another project's Prow running a version fine does not show it is safe for the Kubernetes Prow, which needs a staging environment that looks like it.

   View case

By root cause7