Cases12 cases

9 to 12 of 12

1. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files

   thredUP2018

   Symptoms

   While thredUP's move to Kubernetes seemed to be going well, Datadog showed the number of open files on the nodes climbing steadily, with no obvious reason.

   Service outageFile descriptor exhaustion

   Trigger

   Containers starting on a node, which in Kubernetes happens constantly with every deploy and every pod that moves between nodes.

   Root cause

   Software defect

   Component

   Telemetry agent

   Contributing factor

   Unverified assumption

   Detection

   Datadog showed the number of open files on the nodes rising.

   Remediation

   • killed the affected node, brought up a new one and redeployed

   Action items

   • Move away from the log aggregation vendor

   Lessons

   Libraries and agents that worked with the EC2 deployment lifecycle may not cope with Kubernetes, where containers start, stop and move between nodes constantly. Each one needs to be checked against the new lifecycle instead of being trusted because it worked before.

   View case
2. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures

   AirMap2018

   Symptoms

   At about 15:00 UTC on 19 July 2018, monitoring alerted on rising 5xx responses from several APIs, most notably pilot and tiles. Logins failed first, because the pilot service fetches the user's profile, and from there all services became unresponsive. Every affected API ran on the production Kubernetes cluster.

   High error rateNode NotReadyDNS resolution failuresContainer crash loopService outage

   Root cause

   Software defect

   Component

   kubelet

   Contributing factor

   Unpatched known bug

   Detection

   Monitoring alerted on an increase in 5xx responses from several APIs, most notably pilot and tiles.

   Remediation

   • restarted the kubelets on the stuck worker nodes, which did not last
   • restarted all worker nodes in a staggered order starting at about 18:00 UTC, which took about 1.5 hours

   Action items

   • Restart nodes on a rolling schedule to keep nodes and kubelets healthy until the fixes land upstream and reach the cloud provider, Microsoft Azure

   Lessons

   When nodes keep falling back to NotReady after kubelet restarts and pods fail on network and DNS errors, a known upstream bug may be the cause; matching the symptoms to the upstream issue tracker, with help from the community, found the cause and the fix candidates.

   View case
3. IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its state

   Nordstrom2017

   Symptoms

   A user reported that pods were starting up and then vanishing. When the cluster operators looked, kubectl get pods and kubectl get nodes returned two different sets of data that alternated back and forth, as if there were two opinions of the cluster's state.

   Stale or inconsistent API readsStale endpoint routingService outage

   Trigger

   etcd write latency rose far above the healthy single-digit milliseconds; the speakers say trouble starts once the database sync time crosses double digits or several hundred milliseconds.

   Root cause

   Software defect

   Component

   etcdKubernetes API serverIngress controllerPersistent volumes and stateful sets

   Contributing factor

   Unreviewed defaults

   Detection

   A user reported pods starting up and vanishing.

   Remediation

   • restored service on a replacement cluster
   • freed persistent volumes from the old cluster one by one so they could bind in the new cluster
   • updated DNS records that pointed at the old cluster's load balancer names

   Action items

   • Make the Kubernetes API server request quorum reads from etcd

   Lessons

   etcd is extremely sensitive to write latency; once the database sync time crosses double digits of milliseconds, stale reads or badly degraded performance follow. Without quorum reads, a consistent store can still hand out stale data. Documentation for a fast-moving system changes, so read it again regularly. Moving workloads to a new cluster is hard when they hold persistent volumes or when DNS records point at generated load balancer names.

   View case
4. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage

   Monzo2017

   Symptoms

   At 14:10 an engineer deployed a change to a service that Monzo's current account needs to process payments, and every request to that service started to fail. Current account customers saw payment failures; the prepaid card was not affected at first because it does not use that service. The change was rolled back at 14:12, but the errors continued.

   High error rateStale endpoint routingService outage

   Trigger

   At 14:10 an engineer deployed a change to a service needed to process current account payments. Two weeks earlier the etcd cluster had been upgraded and grown from three to nine nodes, and one day earlier another team had scaled a new service down to zero replicas while leaving its Kubernetes service in place.

   Root cause

   Software defect

   Component

   etcdKubernetes API serverLinkerdkubelet

   Contributing factor

   Ineffective rollbackHarmful manual actionOverlapping failuresMonitoring blind spotSingle point of failure

   Detection

   Payment failures began right after the 14:10 deploy. The team rolled back at 14:12 and declared an outage internally at 14:16. At 15:13, when no service could serve requests, practically all automated alerts fired.

   Remediation

   • Rolled back the payments service change (the errors continued)
   • Activated internal backup processes so most customers could still use their card
   • Restarted all linkerd instances (this made the outage worse)
   • Restarted the three Kubernetes API server processes so replacement linkerd instances could start
   • Began deploying a newer linkerd version with the fix for empty services
   • Deleted the Kubernetes service that had no endpoints, which let linkerd load service discovery again

   Action items

   • Fix the bug in Kubernetes that can trigger timeouts after a cluster reconfiguration
   • Roll out a new version of linkerd that fixes the parsing error
   • Create better health checks, dashboards and alerts for the affected components so they give clearer signals about what is wrong and prevent human error
   • Improve procedures to communicate outages internally and externally as clearly and quickly as possible

   Lessons

   A large failure in a distributed system can be very hard to understand, and well-meant human action can compound it, as restarting every linkerd instance did here by exposing a second incompatibility. Because Monzo deploys often and its network changes constantly, the deployment and request-routing systems have to be trustworthy. Monzo committed to publishing a similar post-mortem for any incident of this severity.

   View case

By root cause1