Incident Library
Applied filters ComponentNode operating system imageSymptomService outageOrganisationDatadogClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
    Symptoms

    Between 06:00 and 07:00 UTC on 8 March 2023, Kubernetes nodes in several Datadog regions lost network connectivity at the same time, on AWS, Google Cloud and Azure. All services across multiple regions were affected, including the regions that host Datadog's CI/CD and automation tools. Datadog's regions are not connected to each other and changes are rolled out one region at a time, so a simultaneous failure on several cloud providers looked implausible at first.

    Node NotReadyService outageThroughput degradation
    Trigger

    On 7 March 2023 Ubuntu published a systemd patch for a CVE (249.11-0ubuntu3.7). Ubuntu's default unattended upgrades installed it on the nodes between 06:00 and 07:00 UTC on 8 March, and installing it restarted systemd-networkd.

    Root cause
    Breaking change in an upgrade
    Component
    systemd-networkdCiliumNode operating system image
    Contributing factor
    Unreviewed defaultsChange with a delayed effectNo staged rolloutAutomation amplification
    Remediation
    • restarted thousands of disconnected instances on Google Cloud and Azure through the cloud APIs, keeping their local disks
    • rebuilt the data lost with terminated AWS instances' local disks through replication, reprocessing and restores from backup
    Lessons

    A change in a base operating system component can stay dormant for months until something restarts that component. Here the trigger was a security patch, and the change reached every region at once because automatic OS updates ran outside the progressive, region-by-region process used for every other change. Cloud providers react differently to an instance that loses its network. Google Cloud and Azure left the instances running, so a restart recovered them with their disks; AWS replaced them quickly, which brought stateless services back fast but destroyed the local-disk data the stateful services needed.

    View case
Check your cluster for this

This case involves Cilium and Node operating system image. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade1 case

Component
CiliumNode operating system imagesystemd-networkd
Contributing factor
Automation amplificationUnreviewed defaultsChange with a delayed effectNo staged rollout
Organisation
Datadog
  1. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023