IL-0190 Grafana Labs · 2020-03-16kubernetesobservabilitynetworking
Grafana Cloud Hosted Prometheus is built on Cortex. To deduplicate samples from pairs of HA Prometheus replicas, the Cortex distributor records in etcd which replica is currently accepted; if no samples arrive from that replica for 15 seconds, the next replica that writes becomes the accepted one. On March 16 a scheduled Kubernetes upgrade terminated the node running the etcd leader. etcd elected a new leader within about 10 seconds, but two distributor replicas kept a TCP connection open to the old leader, because the node died without closing it. Their etcd reads timed out and the writes passing through them failed.
The etcd client in Cortex supports gRPC keepalive probes, but Cortex had not configured them, so the client never noticed the dead connection. The team reproduced the failure in dev by dropping packets with iptables between a distributor and its etcd instance, and showed that setting DialKeepAliveTime, DialKeepAliveTimeout and PermitWithoutStream in the clientv3 configuration prevents it.
The upgrade scripts also played a part: the etcd pods are managed by the etcd operator, so the scripts did not know how to move them gracefully, and the API call the scripts used to terminate an instance gave it only about 90 seconds to shut down. Prometheus remote_write treats the timeout as recoverable, so senders retried the failed writes against other distributor replicas and no data was lost; storage was only delayed.
About a minute after the etcd leader changed, two replicas of the Cortex distributor started logging "context deadline exceeded" when they tried to read keys from etcd. Writes from some customers that landed on those two replicas failed, while the other distributor replicas kept working, so the service saw an error rate of about 20% on writes to the London region of Hosted Prometheus.
The distributor reads etcd to know which replica of a customer's HA Prometheus pair is currently elected to have its writes accepted. Another etcd replica had become the leader about 10 seconds after the old one disappeared, but the two distributors were still holding TCP connections to the old leader, whose node had been terminated without closing them.
Root cause Software defect
An SLO-based alert paged the on-call engineer at 11:31, 11 minutes after the errors started. At a 20% error rate the monthly SLA would have taken about 18 hours to breach.
When a node dies abruptly, clients holding TCP connections to it are not told; without keepalive probes an etcd client can keep sending requests into a dead connection long after a new leader exists. A failover that works on the server side still needs clients that detect the dead peer.
Node replacement during cluster upgrades should drain pods gracefully, including pods owned by operators that the upgrade scripts do not know about, and should give instances enough time to shut down.
IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binariesMicrosoft · 2025same root cause · shared Node pool
IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspacesZenML · 2025same root cause · shared Node pool
IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limitMarcin Cuber · 2024 · sev1same root cause · shared etcd
IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodesLinode · 2024same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared etcd