IL-0003 Monzo · 2017-10-27kubernetesaws
Two weeks before the outage, Monzo's Platform team upgraded etcd to a new version and grew the cluster from three nodes (one per zone) to nine (three per zone). The upgrade caused no downtime, but it set up a bug in Kubernetes and the etcd client that can make requests time out after this kind of cluster reconfiguration. Because of those timeouts, linkerd did not receive the update from Kubernetes about where the newly deployed payments service was running, and kept routing its requests to old addresses. Rolling back the deploy did not help.
At 14:26, assuming every linkerd instance had the same problem, engineers decided to restart all of them, several hundred in total. The replacements could not start until the three Kubernetes API server processes were restarted at 14:37. When all linkerd pods had restarted at 15:13 and loaded service discovery afresh, they hit a second, unrelated problem: the linkerd version in use threw a NullPointerException on Kubernetes services with no endpoints, such as the service scaled to zero replicas the day before. Routing stopped across the whole platform. Monzo later called the mass restart an unfortunate and poor decision that made the outage worse.
The Platform team began deploying a newer linkerd version, tested in staging for several weeks, that fixed the incompatibility. At 15:31, after reading that code change, engineers realised that deleting the Kubernetes services with no endpoints would avoid the parsing error. They deleted the service, linkerd loaded service discovery, traffic flowed between services again and payments worked. During the outage, engineers used internal backup processes so that most customers could still use their card. The current account and the prepaid product were affected for around 1½ hours.
At 14:10 an engineer deployed a change to a service that Monzo's current account needs to process payments, and every request to that service started to fail. Current account customers saw payment failures; the prepaid card was not affected at first because it does not use that service. The change was rolled back at 14:12, but the errors continued.
Engineers saw that linkerd, which routes and load-balances the calls between Monzo's several hundred microservices, was in an unhealthy state. It turns a logical name such as service.foo into an IP address and port, but it had not received the update from Kubernetes about where the new pods were running, so it kept sending requests to IP addresses where no process was running any more.
Replacement linkerd instances then could not start, because the kubelet on each node failed to fetch their configuration from the Kubernetes API servers. After the three API server processes were restarted the replacements came up, but at 15:13 services that normally handle thousands of requests per second were receiving no traffic. Customers could not refresh their feed or balance in the Monzo app, the internal Customer Operations (COps) tools stopped working, and practically all automated alerts fired.
At 15:27 engineers noticed linkerd logging NullPointerException while parsing the service discovery response from the Kubernetes API server. The running versions of Kubernetes and linkerd were incompatible: linkerd failed to parse services that had no endpoints.
Root cause Software defect
Payment failures began right after the 14:10 deploy. The team rolled back at 14:12 and declared an outage internally at 14:16. At 15:13, when no service could serve requests, practically all automated alerts fired.
A large failure in a distributed system can be very hard to understand, and well-meant human action can compound it, as restarting every linkerd instance did here by exposing a second incompatibility. Because Monzo deploys often and its network changes constantly, the deployment and request-routing systems have to be trustworthy. Monzo committed to publishing a similar post-mortem for any incident of this severity.
IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limitMarcin Cuber · 2024 · sev1same root cause · shared Kubernetes API server, etcd
IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its stateNordstrom · 2017same root cause · shared Kubernetes API server, etcd
IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six daysBarush Mendez · 2025same root cause · shared Kubernetes API server
IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022same root cause · shared kubelet
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared kubelet
IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutesGrafana Labs · 2020same root cause · shared etcd
IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020same root cause · shared kubelet
IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELBZalando · 2019same root cause · shared Kubernetes API server
IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failuresAirMap · 2018same root cause · shared kubelet
IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod evictionOath · 2018same root cause · shared kubelet