Cases24 cases

22 to 24 of 24

1. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped

   Binder Project2017

   Symptoms

   On 30 November 2017 at 16:59 PST, mybinder.org operators noticed that Grafana looked wrong and that launching a new server failed with an error that a temporary user could not be created. Pods that were already running kept running, but no new ones were being created.

   Service outageContainer crash loopProbe failuresOOMKilled

   Trigger

   A build container, which ran without a memory limit, most likely used up all the memory on the node that ran the proxy pod.

   Root cause

   Capacity exhaustion

   Component

   Ingress controllerContainer memory limitJupyterHub

   Contributing factor

   No resource requests or limitsHarmful manual actionNo alert on limit

   Detection

   An operator happened to look at Grafana at 16:59 PST, saw that it looked wrong, and confirmed the error by launching a new server.

   Remediation

   • Cordoned the node that had run out of memory
   • Uncordoned the node once the proxy was working

   Action items

   • Limit the memory available to build pods
   • Change the JupyterHub configuration so user pods are not cleaned up when the jupyterhub pod restarts
   • Nominate one person to drive the response, possibly in a shared terminal, so only one person changes the cluster
   • Alert when something is wrong with the cluster, for example by counting 4xx responses on the jupyterhub pod

   View case
2. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage

   Monzo2017

   Symptoms

   At 14:10 an engineer deployed a change to a service that Monzo's current account needs to process payments, and every request to that service started to fail. Current account customers saw payment failures; the prepaid card was not affected at first because it does not use that service. The change was rolled back at 14:12, but the errors continued.

   High error rateStale endpoint routingService outage

   Trigger

   At 14:10 an engineer deployed a change to a service needed to process current account payments. Two weeks earlier the etcd cluster had been upgraded and grown from three to nine nodes, and one day earlier another team had scaled a new service down to zero replicas while leaving its Kubernetes service in place.

   Root cause

   Software defect

   Component

   etcdKubernetes API serverLinkerdkubelet

   Contributing factor

   Ineffective rollbackHarmful manual actionOverlapping failuresMonitoring blind spotSingle point of failure

   Detection

   Payment failures began right after the 14:10 deploy. The team rolled back at 14:12 and declared an outage internally at 14:16. At 15:13, when no service could serve requests, practically all automated alerts fired.

   Remediation

   • Rolled back the payments service change (the errors continued)
   • Activated internal backup processes so most customers could still use their card
   • Restarted all linkerd instances (this made the outage worse)
   • Restarted the three Kubernetes API server processes so replacement linkerd instances could start
   • Began deploying a newer linkerd version with the fix for empty services
   • Deleted the Kubernetes service that had no endpoints, which let linkerd load service discovery again

   Action items

   • Fix the bug in Kubernetes that can trigger timeouts after a cluster reconfiguration
   • Roll out a new version of linkerd that fixes the parsing error
   • Create better health checks, dashboards and alerts for the affected components so they give clearer signals about what is wrong and prevent human error
   • Improve procedures to communicate outages internally and externally as clearly and quickly as possible

   Lessons

   A large failure in a distributed system can be very hard to understand, and well-meant human action can compound it, as restarting every linkerd instance did here by exposing a second incompatibility. Because Monzo deploys often and its network changes constantly, the deployment and request-routing systems have to be trustworthy. Monzo committed to publishing a similar post-mortem for any incident of this severity.

   View case
3. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten

   UC Berkeley2017

   Symptoms

   At about 1 AM on 9 February 2017, UC Berkeley's DataHub was reported down: users trying to log in got a proxy error. The hub pod was running, but its log was full of sqlite errors.

   Service outageData lossMisrouted requests

   Trigger

   While debugging, an operator ran `.save jupyterhub.sqlite` in sqlite3 inside the hub pod without having attached the database, which replaced the JupyterHub database file with an empty one.

   Root cause

   Operator error

   Component

   DatabasePersistent volumes and stateful setsJupyterHub

   Contributing factor

   No safeguard on destructive changeUnsafe failure behaviourHarmful manual action

   Detection

   DataHub was reported down at about 1 AM, when users got a proxy error at login. At 1:12 AM a user reported that he could log in but all his notebooks were gone.

   Remediation

   • Scaled the hub to 0 replicas
   • Rebuilt the JupyterHub sqlite database by matching Kubernetes PVC names back to user ids
   • Set cookie_id to null instead of a dummy value and attached the rebuilt database to the hub
   • Changed the cookie secret in the hub pod and restarted it to log all users out
   • Deleted all user pods to log everyone out
   • Ran a script that filled in cookie_id for all users, restarted the hub and deleted all user pods again
   • Dropped the new users with a '.' in their name and renamed the older users from '-' to '.'

   Action items

   • Report upstream that JupyterHub uses only part of the cookie secret and discards the rest
   • Report upstream that the cookie_id column of the users table should have UNIQUE and NOT NULL constraints
   • Ask KubeSpawner to support username hashes in PVC and pod names so the names stay the same when the database is deleted
   • Ask OAuthenticator to support using the id from the Google authenticator as the user id, giving stable user ids
   • Switch to Google Cloud SQL, a hosted and managed MySQL database
   • Take regular and tested backups of the database
   • Start an operational FAQ of things to do and not to do
   • Set up better monitoring and paging
   • Document escalation procedures explicitly

   View case

By root cause11