Cases2 cases

1 to 2 of 2

1. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop

   Fullstaq2019

   Symptoms

   Dex, the OIDC proxy that handled Kubernetes authentication, stored its data in the Kubernetes API (storage type kubernetes, inCluster: true) and had built up a huge number of AuthRequest objects. Listing all AuthRequests timed out, and without the list they could not be deleted.

   Control plane unresponsiveMonitoring outage

   Trigger

   The dex namespace was deleted so that Kubernetes would clean up the accumulated AuthRequest objects itself.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serveretcdCluster authentication provider

   Contributing factor

   Unreviewed defaultsHarmful manual actionRetry stormLimited debug accessIgnored early signal

   Detection

   The speaker says he had got hints of the problem earlier. While the masters were down, plenty of alerts arrived in Slack, but monitoring could not be reached because authentication and port-forwarding no longer worked.

   Remediation

   • removed the kubernetes finalizer from the dex namespace by calling the API directly (kill-kube-ns)
   • deployed Dex to a new namespace with sane defaults

   Lessons

   Dex with Kubernetes as its storage keeps every AuthRequest as an API object, and with a long TTL they pile up until the API cannot list them. Deleting a namespace that holds that many objects makes Kubernetes list them in a loop and can overload the masters. Monitoring that is reached through cluster authentication is lost when authentication is what broke.

   View case
2. IL-0505 kubectl logs failed on GKE with "No SSH tunnels currently open" and mybinder.org build logs stopped reaching users

   Binder Project2019

   Symptoms

   On 20 February 2019 at 08:03 CET, `kubectl logs -f <pod>` stopped working for every pod in the mybinder.org cluster on GKE. Because pod logs could not be read, build logs were no longer forwarded to users, although the builds themselves ran normally.

   Monitoring outageDeploys failing

   Root cause

   Cloud provider outage

   Component

   GKE control planeKubernetes API server

   Contributing factor

   Limited debug access

   Remediation

   • Cordoned node gke-prod-a-users-b921bb88-rd5d so it would be removed from the pool once empty
   • Drained the node and removed it from the pool, after which kubectl logs worked again

   Action items

   • Test that kubectl logs works and notify when it does not
   • Add the command for retrieving API server logs to the SRE guide
   • Describe in the SRE guide how to identify suspicious node networking status

   Lessons

   The root cause was very difficult to identify, partly because it appears to sit in the Kubernetes master or the GKE service itself. Capturing `kubectl describe node` for the suspicious node would have allowed a comparison with healthy nodes, and a node that reports no ExternalIP or InternalIP in its status was likely a direct symptom of the problem and can be used as a diagnostic in the future.

   View case

By root cause2