Incident Library
Applied filters SymptomHigh error rateFactorNo safeguard on destructive changeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes
    Grafana Labs2020
    Symptoms

    About a minute after the etcd leader changed, two replicas of the Cortex distributor started logging "context deadline exceeded" when they tried to read keys from etcd. Writes from some customers that landed on those two replicas failed, while the other distributor replicas kept working, so the service saw an error rate of about 20% on writes to the London region of Hosted Prometheus.

    High error rateIntermittent timeouts
    Trigger

    A scheduled Kubernetes version upgrade abruptly terminated the node that ran the etcd leader at 11:19:00 UTC.

    Root cause
    Software defect
    Component
    etcdNode pool
    Contributing factor
    Unreviewed defaultsNo safeguard on destructive change
    Detection

    An SLO-based alert paged the on-call engineer at 11:31, 11 minutes after the errors started. At a 20% error rate the monthly SLA would have taken about 18 hours to breach.

    Remediation
    • Restarted the two affected Cortex distributor replicas at 11:32, which stopped the errors
    Action items
    • Enable gRPC keepalive probes in the Cortex etcd client (DialKeepAliveTime 10s, DialKeepAliveTimeout, PermitWithoutStream)
    • Make the Kubernetes upgrade process terminate pods and machines gracefully, including the etcd pods managed by the etcd operator
    Lessons

    When a node dies abruptly, clients holding TCP connections to it are not told; without keepalive probes an etcd client can keep sending requests into a dead connection long after a new leader exists. A failover that works on the server side still needs clients that detect the dead peer. Node replacement during cluster upgrades should drain pods gracefully, including pods owned by operators that the upgrade scripts do not know about, and should give instances enough time to shut down.

    View case
Check your cluster for this

These 6 cases involve Argo CD and Automated capacity manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Software defect2 cases

Component
Automated capacity manager1etcd1Node pool1
Contributing factor
No safeguard on destructive change2Automation amplification1Unreviewed defaults1Change with a delayed effect1Monitoring blind spot1
Organisation
Grafana Labs1OpenAI1
  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
  2. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes
    Grafana Labs2020

Faulty deployment1 case

Component
–
Contributing factor
Excessive blast radiusNo safeguard on destructive change
Organisation
OpenAI
  1. IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed
    OpenAI2025

Operator error2 cases

Component
Application database1Ingress controller1Model hub1
Contributing factor
No safeguard on destructive change2No headroom1Overlapping failures1Deferred upgrades1
Organisation
GitLab1OpenAI1
  1. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
  2. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022

Configuration change1 case

Component
Argo CDConfiguration templatingPersistent volumes and stateful sets
Contributing factor
Automation amplificationMonitoring blind spotNo runbookNo safeguard on destructive changeMissed in review
Organisation
InfluxData
  1. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022