Incident Library
Applied filters Root causeConfiguration changeComponentNode poolDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

1 to 5 of 8

  1. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
    Symptoms

    At 16:40 UTC on 19 April 2024 CoW Protocol's mainnet production environment stopped producing settlements; the outage officially began then.

    Pods PendingIP address exhaustionPod evictionService outage
    Trigger

    During a cluster capacity expansion, the Infrastructure as Code automation updated the AWS VPC CNI plugin, because the CNI version was not hardcoded in the IaC and the IaC library version had recently changed.

    Root cause
    Configuration change
    Component
    AWS VPC CNINode pool
    Contributing factor
    Unpinned versionOverlapping failures
    Detection

    The CNI version mismatch was identified as the cause at 17:03 UTC, 23 minutes after settlements stopped.

    Remediation
    • upgraded the Kubernetes cluster
    • set up new node groups with a new version of the networking plugin
    Lessons

    Versions in IaC scripts should be specified explicitly so an automation run cannot update a component such as the CNI plugin unintentionally. Changes in production need change management that is easy to revert.

    View case
  2. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device
    UK Ministry of Justice2023
    Symptoms

    At 13:42 the RootVolUtilisation-Critical alert fired in the high-priority alerts channel. At 14:03 a user reported image pull errors with the message "no space left on device".

    Node disk fullImage pull failuresPod evictionPods stuck in ContainerCreatingIP address exhaustion
    Trigger

    Upgrading the EKS Terraform module to version 18, which changed the node launch template, and draining workloads onto the new nodes it created.

    Root cause
    Configuration change
    Component
    Node root diskNode launch templateNode poolAWS VPC CNICoreDNS
    Contributing factor
    Missed in reviewNo configuration testsNo alert on limit
    Detection

    The RootVolUtilisation-Critical alert fired in the high-priority alerts channel at 13:42, and a user reported "no space left on device" image pull errors at 14:03.

    Remediation
    • Uncordoned the old default node group and drained the new nodes to move workloads back
    • Identified the launch template change that cut the root volume to 20 GB and applied a fix on the manager and live clusters
    • Drained the old node group onto the fixed nodes
    Action items
    • Update the runbook to compare launch template changes during an EKS module upgrade
    • Create a test setup that pulls images of different sizes, like live
    • Update the RootVolUtilisation alert runbook to check the disk space configuration
    • Scale CoreDNS dynamically based on the number of nodes
    • Investigate whether IPv6 can solve the IP prefix starvation problem
    • Add drift testing to identify when a terraform plan shows a change to the launch template
    • Set up logging for the CNI and ipamd logs and alerts on errors related to IP prefix starvation
    Lessons

    The write-up lists review actions rather than lessons. They aim at two gaps: the launch template changes brought by a module upgrade were not compared or drift-tested before rollout, and there were no CNI or ipamd logs or alerts to show the IP prefix starvation the team suspected during the drain.

    View case
  3. IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreating
    UK Ministry of Justice2023
    Symptoms

    At 08:15 a user posted in the #ask-cloud-platform channel that new deployments were not starting: their pods stayed in ContainerCreating.

    Pods stuck in ContainerCreatingIP address exhaustion
    Root cause
    Configuration change
    Component
    AWS VPC CNINode pool
    Contributing factor
    No configuration tests
    Detection

    A user reported new deployments stuck in ContainerCreating in the #ask-cloud-platform channel at 08:15; the incident was declared at 09:31.

    Remediation
    • Added new nodes and cordoned and drained the six affected nodes
    • Enabled prefix delegation in the VPC CNI on the live cluster after comparing its CNI settings with a 1.23 test cluster
    Action items
    • Add a test or check that confirms IP address allocation is working as expected (#4669)
    Lessons

    The write-up states no lessons beyond its review action, which is to add a check that IP address allocation works.

    View case
  4. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023
    Symptoms

    On 7 July 2023 from 11:05 to 15:25 UTC many public Jenkins infrastructure services were completely down, including get.jenkins.io, plugins.jenkins.io, accounts.jenkins.io, javadoc.jenkins.io and incrementals.jenkins.io, while ci.jenkins.io, issues.jenkins.io, repo.jenkins-ci.org and www.jenkins.io were partly or completely disrupted.

    Network unreachableResources deletedCluster or node provisioning stallService outage
    Trigger

    After the upgrade to Kubernetes 1.25 changed the range of the pods' originating IP and LDAP access was blocked, a fix (Azure PR 431) deployed at 10:55 UTC to set a proper set of pod IP ranges removed all node pools of the cluster.

    Root cause
    Configuration change
    Component
    Node poolCNI pluginCloud load balancer
    Contributing factor
    Unverified assumptionNo staged rolloutNo safeguard on destructive change
    Detection

    After the upgrade, at 10:30 UTC, the team noticed that services inside the cluster, such as accounts.jenkins.io, could not reach the LDAP service.

    Remediation
    • Created the cluster again from scratch with a fixed network setup
    • Set a custom pod CIDR in a range separate from the virtual network
    • Moved the services and their DNS records to new public IPs
    • Pushed a pull request that recorded the manual rebuild, including the IP setup, in code
    Action items
    • Protect the public IPs from deletion with a Management Lock
    • Keep the public IPs in a separate resource group and point the Kubernetes-managed load balancers to it with the service.beta.kubernetes.io/azure-load-balancer-resource-group annotation
    • Improve network diagrams and documentation so possible overlaps are visible when preparing operations
    • Avoid changing the configuration of all AKS node pools at once
    Lessons

    The team said that changing one node pool at a time would have shown the problem after the first pool and avoided a full outage, and that the public IPs should change as little as possible because users behind corporate firewalls with allow-lists depend on them. The team also noted that it could rebuild, in under 4 hours, the cluster that hosts almost all public services.

    View case
  5. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
    Symptoms

    At 10:23 on 6 June 2023 a user of the Cloud Platform reported that all of their production pods had been deleted at the same moment. Several users received Pingdom alerts that their applications were down for a few minutes.

    Service outage
    Trigger

    The team changed the instance type of the cluster's worker nodes through Terraform.

    Root cause
    Configuration change
    Component
    Node poolTerraform
    Contributing factor
    Excessive blast radiusNo runbook
    Detection

    Users reported that their production pods had been deleted at once and that Pingdom alerts showed their applications down; the incident was declared at 11:00.

    Action items
    • Ask AWS, through a support ticket, for the steps to change the node instance type without an outage
    • Plan how to make instance type changes through Terraform without downtime and add a runbook with the steps for changing the node instance type
    View case
Check your cluster for this

These 8 cases involve Node pool and AWS VPC CNI. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change8 cases

Component
Node pool8AWS VPC CNI3Node launch template2Terraform2Cloud load balancer1CNI plugin1Configuration templating1Container runtime1CoreDNS1Jobs and CronJobs1Karpenter1Node root disk1Telemetry agent1
Contributing factor
No configuration tests3Unverified assumption2Automation amplification2Unreviewed defaults2No runbook2Excessive blast radius1Change with a delayed effect1Ignored early signal1Harmful manual action1No alert on limit1No safeguard on destructive change1No staged rollout1Overlapping failures1Missed in review1Shared credential1Unrepresentative staging1Unpinned version1
Organisation
UK Ministry of Justice3CoW Protocol1DFDS1Jenkins project1Stytch1Target1
  1. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
  2. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device
    UK Ministry of Justice2023
  3. IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreating
    UK Ministry of Justice2023
  4. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023
  5. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
  6. IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down
    Stytch2023
  7. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  8. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018