Incident Library

Organisation

Civis Analytics

Civis Analytics · civis-analytics

1 case · 2019

Root cause Capacity exhaustion
Component etcdKubernetes API serverTelemetry agent
Contributing factor Unverified assumptionUnknown limitMonitoring blind spotNo capacity planning
Symptom Control plane lagControl plane unresponsiveDNS resolution failuresHigh error rateHigh latency

Cases

  1. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019

    Root cause Capacity exhaustion