Incident Library

Organisation

Monzo

Monzo · monzo

1 case · 2017

Root cause Software defect

Component etcd Kubernetes API server kubelet Linkerd

Contributing factor Harmful manual action Monitoring blind spot Overlapping failures Ineffective rollback Single point of failure

Symptom High error rate Service outage Stale endpoint routing

Cases

IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outageMonzo · 2017
Root cause Software defect