Incident Library
Applied filters Root causeBreaking change in an upgradeFactorAutomation amplificationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024
    Symptoms

    On August 16, 2024 from 11:38 AM to 1:16 PM PT, the OpenAI primary API was unreliable: ChatGPT conversations succeeded less often, and logging in and creating accounts were affected. The status page called it elevated error rates for ChatGPT and the Platform API. The incident came in two waves, of 44 and 15 minutes.

    High error rateDatabase overload
    Trigger

    A scheduled maintenance combined with an upgrade to the ingress of OpenAI's user-facing clusters.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerApplication database
    Contributing factor
    No staged rolloutAutomation amplification
    Action items
    • Mitigate the networking control plane regression and validate that control plane restarts do not interfere with the clusters' data plane
    • Change service software to start faster and remove the first-start query, relieving the persistence layer
    • Deploy configuration changes that optimise the networking control plane's effect on the clusters' ability to handle traffic
    • Remove the expensive database query from critical startup paths
    • Add monitoring and alerting for networking control plane issues
    • Introduce staged rollouts with longer soak time for infrastructure changes
    • Audit systems for other slow queries that may affect service start time
    Lessons

    A brief connectivity loss turned into a long outage because recovery itself was expensive: the services that restarted overwhelmed one backend persistence store with a heavy first-start query. Keeping costly queries off startup paths shortens recovery, and staged rollouts with longer soak time limit how many systems an infrastructure regression reaches before it is caught.

    View case
  2. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
    Symptoms

    Between 06:00 and 07:00 UTC on 8 March 2023, Kubernetes nodes in several Datadog regions lost network connectivity at the same time, on AWS, Google Cloud and Azure. All services across multiple regions were affected, including the regions that host Datadog's CI/CD and automation tools. Datadog's regions are not connected to each other and changes are rolled out one region at a time, so a simultaneous failure on several cloud providers looked implausible at first.

    Node NotReadyService outageThroughput degradation
    Trigger

    On 7 March 2023 Ubuntu published a systemd patch for a CVE (249.11-0ubuntu3.7). Ubuntu's default unattended upgrades installed it on the nodes between 06:00 and 07:00 UTC on 8 March, and installing it restarted systemd-networkd.

    Root cause
    Breaking change in an upgrade
    Component
    systemd-networkdCiliumNode operating system image
    Contributing factor
    Unreviewed defaultsChange with a delayed effectNo staged rolloutAutomation amplification
    Remediation
    • restarted thousands of disconnected instances on Google Cloud and Azure through the cloud APIs, keeping their local disks
    • rebuilt the data lost with terminated AWS instances' local disks through replication, reprocessing and restores from backup
    Lessons

    A change in a base operating system component can stay dormant for months until something restarts that component. Here the trigger was a security patch, and the change reached every region at once because automatic OS updates ran outside the progressive, region-by-region process used for every other change. Cloud providers react differently to an instance that loses its network. Google Cloud and Azure left the instances running, so a restart recovered them with their disks; AWS replaced them quickly, which brought stateless services back fast but destroyed the local-disk data the stateful services needed.

    View case
  3. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022
    Symptoms

    From 06:00 UTC on 30 August 2022, Azure VMs running Ubuntu 18.04 (bionic) with unattended-upgrades enabled began failing DNS resolution and losing network connectivity. Several Azure teams, among them the AKS and Azure Container Apps teams, were alerted by production alerts shortly after the update packages were published.

    DNS resolution failuresImage pull failuresMonitoring outage
    Trigger

    Unattended-upgrades on Ubuntu 18.04 VMs installed a Canonical security update published at 06:00 UTC on 30 August 2022, including systemd version 237-3ubuntu10.54.

    Root cause
    Breaking change in an upgrade
    Component
    Node operating system imagekubeletContainer registry and its credentialsClient-side DNS resolver and cache
    Contributing factor
    No staged rolloutExcessive blast radiusAutomation amplification
    Detection

    Several Azure teams, including AKS and Azure Container Apps, were alerted by production alerts shortly after the packages were published; incoming support cases then confirmed which Ubuntu versions were affected.

    Remediation
    • pointed customers to Canonical's description of the systemd bug and its fix
    • recommended rebooting affected VMs or running a template script that repairs DNS resolution
    • built automatic detection and remediation for AKS nodes and rolled it out across all regions
    • mitigated the remaining AKS customers manually through support
    Action items
    • Review the process that lets critical security updates bypass Safe Deployment Practices
    • Improve monitoring of the AKS data plane to alert on an uptick of errors
    • Review AKS monitoring algorithms so that similar VM errors are detected and alerted on
    • Establish an improved escalation path to Canonical during outages and add it to internal Technical Service Guides
    • AKS takes full control of the node security patch mechanism, with additional testing and controlled release of patches (estimated December 2022)
    • Work with Canonical to run dedicated tests on proposed packages before they are published for Azure IaaS VMs
    • Establish a process with Canonical to close the testing gap for the upgrade scenario (estimated March 2023)
    • Give AKS customers maintenance window control and environment staging for these patches (estimated December 2023)
    Lessons

    Security patches applied automatically by unattended-upgrades, once a day by default, and outside Azure's Safe Deployment Practices reached Ubuntu 18.04 VMs in every Azure region, so one systemd regression became a DNS outage across Azure. On AKS, lost node DNS showed up as image pull failures such as ImagePullBackoff, because kubelet could not resolve the registry. Azure plans to put node security patching under AKS's control with testing and controlled release; customers can use automatic node image upgrade for AKS, or automatic VM guest patching for VMs, instead of unattended-upgrades.

    View case
Check your cluster for this

These 3 cases involve Node operating system image and Cilium. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade3 cases

Component
Node operating system image2Cilium1Application database1Client-side DNS resolver and cache1Ingress controller1kubelet1Container registry and its credentials1systemd-networkd1
Contributing factor
Automation amplification3No staged rollout3Excessive blast radius1Unreviewed defaults1Change with a delayed effect1
Organisation
Datadog1Microsoft Azure1OpenAI1
  1. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024
  2. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
  3. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022