IL-0109 DFDS · 2023-02-14kubernetesaws
A scheduled node rollover on 13 February 2023 set containerd as the default container runtime by mistake, and as a result Kubernetes logs stopped being shipped to CloudWatch. To fix the logs, the team started an unscheduled node rollover the next morning that swapped the runtime back. The value used in the eks-bootstrap script was believed to be docker-shim but needed to be dockerd, so the new nodes never joined the cluster and the rollover timed out with them still in the Auto Scaling group.
The team then ran the EKS pipeline without a rollover to apply a launch template with dockerd. Terraform saw the desired size of the Auto Scaling group still at the rollover value of 30 and set it back to 15. The group's default termination policy removes the instances that use the oldest launch template first, and the oldest ones were the healthy nodes, not the faulty new ones that had not been removed yet. All healthy worker nodes were terminated and the cluster was left with none.
A first run of the node rollover script at 10:49 failed on a timeout from Docker Hub; a second run at 10:52 added new nodes from 10:57, and at 11:01 the load balancer marked them healthy and traffic flowed again. Recovery took 13 minutes. A Sev1 ticket was raised at 11:07 and closed at 11:59.
At 09:33 UTC on 14 February 2023 end users told the platform team that Kubernetes logs were not arriving in CloudWatch as expected. The investigation went through fluentd and found that the nodes were running containerd as their container runtime instead of docker-shim. A scheduled node rollover the day before, at 07:30 UTC on 13 February, had set containerd as the default runtime by accident.
An unscheduled node rollover to switch the runtime back started at 10:29 and failed on a timeout at 10:39. The runtime value used was wrong, so the new nodes never joined the cluster and were left as "zombies".
At 10:45 the EKS pipeline was run again without a node rollover, to apply the new launch template with dockerd as the runtime. Because the failed rollover had never set the Auto Scaling group's desired size back, Terraform changed it from 30 to 15. At 10:46 all the healthy nodes were terminated, since the group's default termination policy removes the instances with the oldest launch configuration or template first. By 10:48 the cluster had no worker nodes.
Root cause Configuration change
End users reported the missing CloudWatch logs that started the work. The postmortem lists 2 minutes as the time to detection of the node failure.
When a change goes wrong, rolling back to a known working configuration is safer than rushing a fix, which can make the damage worse. Launching new worker nodes depends on the cloud-init script not failing in any way.
The QA configuration differed from production, there was no staging environment, the QA tests did not cover fluentd log collection, and the rollover process had no documentation for when it fails.
IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second burstsMUSINSA · 2026same root cause · shared Container runtime, Telemetry agent
IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023same root cause · shared Node launch template, Node pool
IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023same root cause · shared Node pool, Terraform
IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI servicesOpenAI · 2024same root cause · shared Telemetry agent
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node pool
IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreatingUK Ministry of Justice · 2023same root cause · shared Node pool
IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021same root cause · shared Terraform
IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second oneSpotify · 2019same root cause · shared Terraform
IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a clusterTarget · 2018same root cause · shared Node pool