Incident Library

Organisation

Datadog

Datadog · datadog

2 cases · 2019 to 2023

Root cause Capacity exhaustion1Breaking change in an upgrade1
Component Admission webhook1Cilium1Application database1NAT gateway1Node operating system image1Container registry and its credentials1systemd-networkd1
Contributing factor Automation amplification2Excessive blast radius1Unreviewed defaults1Change with a delayed effect1Unknown limit1Harmful manual action1No staged rollout1Single point of failure1
Symptom Service outage2Image pull failures1Node NotReady1Throughput degradation1

Cases

  1. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providersDatadog · 2023

    Root cause Breaking change in an upgrade

  2. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019

    Root cause Capacity exhaustion