Incident Library
Applied filters Root causeBreaking change in an upgradeSymptomService outageFactorChange with a delayed effectClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
    Symptoms

    Event ingestion and the Honeycomb UI in EU1 became unavailable at 22:00:10 UTC on April 9, 2024, and the status page reported that the EU1 instance was down. Both the ingestion service and the service behind the web UI stopped at roughly the same moment.

    Service outageStale endpoint routing
    Trigger

    A routine system update to the Kubernetes cluster, rolled out earlier on April 9, followed by a regular deployment that replaced pods one at a time.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerCloud load balancer
    Contributing factor
    Monitoring blind spotUnrepresentative stagingChange with a delayed effect
    Detection

    The first status post, saying the EU1 instance was down, went up at 15:08 PDT (22:08 UTC), about 8 minutes after the outage began. There was no alerting on the AWS Load Balancer Controller.

    Remediation
    • Restarted the AWS Load Balancer Controller
    • Rolled back the Kubernetes cluster system update
    Action items
    • Find the exact cause of the failure so the system update can be deployed safely
    • Add monitoring and alerting on the AWS Load Balancer Controller so on-call engineers learn of a failure quickly and can pause deployments
    Lessons

    A rolling deployment can look healthy while the component that wires new pods into the load balancer is broken; the failure only shows when the last old pod goes away. Honeycomb now treats the AWS Load Balancer Controller as a critical component that needs its own alerting, so deployments can be paused before they remove the last pods that still receive traffic. Extensive testing in development and staging did not reveal the bad interaction.

    View case
  2. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
    Symptoms

    Between 06:00 and 07:00 UTC on 8 March 2023, Kubernetes nodes in several Datadog regions lost network connectivity at the same time, on AWS, Google Cloud and Azure. All services across multiple regions were affected, including the regions that host Datadog's CI/CD and automation tools. Datadog's regions are not connected to each other and changes are rolled out one region at a time, so a simultaneous failure on several cloud providers looked implausible at first.

    Node NotReadyService outageThroughput degradation
    Trigger

    On 7 March 2023 Ubuntu published a systemd patch for a CVE (249.11-0ubuntu3.7). Ubuntu's default unattended upgrades installed it on the nodes between 06:00 and 07:00 UTC on 8 March, and installing it restarted systemd-networkd.

    Root cause
    Breaking change in an upgrade
    Component
    systemd-networkdCiliumNode operating system image
    Contributing factor
    Unreviewed defaultsChange with a delayed effectNo staged rolloutAutomation amplification
    Remediation
    • restarted thousands of disconnected instances on Google Cloud and Azure through the cloud APIs, keeping their local disks
    • rebuilt the data lost with terminated AWS instances' local disks through replication, reprocessing and restores from backup
    Lessons

    A change in a base operating system component can stay dormant for months until something restarts that component. Here the trigger was a security patch, and the change reached every region at once because automatic OS updates ran outside the progressive, region-by-region process used for every other change. Cloud providers react differently to an instance that loses its network. Google Cloud and Azure left the instances running, so a restart recovered them with their disks; AWS replaced them quickly, which brought stateless services back fast but destroyed the local-disk data the stateful services needed.

    View case
Check your cluster for this

These 2 cases involve Cilium and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade2 cases

Component
Cilium1Cloud load balancer1Ingress controller1Node operating system image1systemd-networkd1
Contributing factor
Change with a delayed effect2Automation amplification1Unreviewed defaults1Monitoring blind spot1No staged rollout1Unrepresentative staging1
Organisation
Datadog1Honeycomb1
  1. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  2. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023