Incident Library
Applied filters Root causeSoftware defectFactorUnreviewed defaultsOrganisationGrafana LabsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes
    Grafana Labs2020
    Symptoms

    About a minute after the etcd leader changed, two replicas of the Cortex distributor started logging "context deadline exceeded" when they tried to read keys from etcd. Writes from some customers that landed on those two replicas failed, while the other distributor replicas kept working, so the service saw an error rate of about 20% on writes to the London region of Hosted Prometheus.

    High error rateIntermittent timeouts
    Trigger

    A scheduled Kubernetes version upgrade abruptly terminated the node that ran the etcd leader at 11:19:00 UTC.

    Root cause
    Software defect
    Component
    etcdNode pool
    Contributing factor
    Unreviewed defaultsNo safeguard on destructive change
    Detection

    An SLO-based alert paged the on-call engineer at 11:31, 11 minutes after the errors started. At a 20% error rate the monthly SLA would have taken about 18 hours to breach.

    Remediation
    • Restarted the two affected Cortex distributor replicas at 11:32, which stopped the errors
    Action items
    • Enable gRPC keepalive probes in the Cortex etcd client (DialKeepAliveTime 10s, DialKeepAliveTimeout, PermitWithoutStream)
    • Make the Kubernetes upgrade process terminate pods and machines gracefully, including the etcd pods managed by the etcd operator
    Lessons

    When a node dies abruptly, clients holding TCP connections to it are not told; without keepalive probes an etcd client can keep sending requests into a dead connection long after a new leader exists. A failover that works on the server side still needs clients that detect the dead peer. Node replacement during cluster upgrades should drain pods gracefully, including pods owned by operators that the upgrade scripts do not know about, and should give instances enough time to shut down.

    View case
Check your cluster for this

This case involves etcd and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
etcdNode pool
Contributing factor
Unreviewed defaultsNo safeguard on destructive change
Organisation
Grafana Labs
  1. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes
    Grafana Labs2020