Incident Library
Applied filters Root causeConfiguration changeSymptomService outageFactorChange with a delayed effectClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
    Symptoms

    Customers of DigitalOcean Managed Kubernetes (DOKS) saw worker nodes go NotReady between 06:00 and 07:00 UTC and stay unavailable, taking down the services on them, until the nodes were replaced or rebooted; some nodes needed several replacements or reboots. The first sporadic reports on October 8 said replacing the node fixed it. The same thing happened again around 06:00 UTC on October 9, more widely and across many regions, and again on October 10. Only clusters on versions 1.27.6-do.0 and 1.28.2-do.0 were affected.

    Node NotReadyService outage
    Trigger

    A systemd timer in the DigitalOcean Debian image started apt upgrade daily at 06:00 UTC, with a random delay of up to 60 minutes.

    Root cause
    Configuration change
    Component
    Node operating system imageLinux kernelkubelet
    Contributing factor
    Unreviewed defaultsLimited debug accessChange with a delayed effect
    Detection

    Sporadic customer reports of failed worker nodes arrived at 07:43 UTC on October 8; internal incident response started at 07:10 UTC on October 9 after broader impact across regions.

    Remediation
    • Advised customers to replace or reboot affected worker nodes
    • Arranged SSH and console access to worker nodes while they were healthy, and live-debugged an affected node on October 10
    • Released a DaemonSet to all affected clusters that disables the apt upgrade systemd timer on every worker node
    • Released new node images without the auto-update process
    Action items
    • Disable the timer during node provisioning, then remove the DaemonSet from all clusters
    • Add a check to the internal conformance test suite for new cluster versions that the auto-upgrade process and other unwanted timers stay disabled
    • Support logging into worker nodes even when normal access paths are broken
    Lessons

    An operating system auto-update timer left enabled in a node image can change the kernel and systemd underneath Kubernetes, and a timeout that kills the upgrade half-way can leave a node without networking. Failures that start at the same time every day point to a scheduled job on the nodes, and systemd timers belong on that list next to cron jobs. When the failure takes out a node's network, an investigation that depends on logging into the node stalls; access paths that work while the node is broken shorten diagnosis.

    View case
  2. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
    Symptoms

    During a cluster upgrade, all traffic to Pusher's products stopped. The speaker recalls that requests were failing with 500 errors.

    Service outageHigh error rate
    Trigger

    A cluster upgrade replaced nodes, so the bridge pods restarted and loaded a ConfigMap that had been changed three or four weeks earlier.

    Root cause
    Configuration change
    Component
    ConfigMaps and Secrets
    Contributing factor
    Change with a delayed effectUnverified assumption
    Action items
    • Build and run Wave so that changes to ConfigMaps and Secrets roll out new pods immediately
    Lessons

    A ConfigMap change does not restart the pods that use it, so for an application that cannot reload its configuration the change takes effect only when pods are next replaced, possibly weeks later during unrelated work. Tie configuration to the pod template so a bad change fails at the moment it is made.

    View case
Check your cluster for this

These 2 cases involve ConfigMaps and Secrets and kubelet. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
ConfigMaps and Secrets1kubelet1Linux kernel1Node operating system image1
Contributing factor
Change with a delayed effect2Unverified assumption1Unreviewed defaults1Limited debug access1
Organisation
DigitalOcean1Pusher1
  1. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  2. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019