Incident Library

Organisation

Elastic

Elastic · elastic

2 cases · 2019

Root cause Software defect2
Component Coordination and key-value store2conntrack table1Container runtime1Elasticsearch or OpenSearch1Linux kernel1Service discovery1
Contributing factor Retry storm2Unpatched known bug1Harmful manual action1Monitoring blind spot1Overlapping failures1Health check tied to a dependency1
Symptom Service outage2Database overload1High error rate1Intermittent timeouts1Node kernel panic1OOMKilled1

Cases

  1. IL-0277 Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachableElastic · 2019

    Root cause Software defect

  2. IL-0587 Elastic Cloud proxies in AWS eu-west-1 flooded an overloaded ZooKeeper and ran out of memory, cutting cluster accessElastic · 2019

    Root cause Software defect