Cases5 cases

1 to 5 of 5

1. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device

   UK Ministry of Justice2023

   Symptoms

   At 13:42 the RootVolUtilisation-Critical alert fired in the high-priority alerts channel. At 14:03 a user reported image pull errors with the message "no space left on device".

   Node disk fullImage pull failuresPod evictionPods stuck in ContainerCreatingIP address exhaustion

   Trigger

   Upgrading the EKS Terraform module to version 18, which changed the node launch template, and draining workloads onto the new nodes it created.

   Root cause

   Configuration change

   Component

   Node root diskNode launch templateNode poolAWS VPC CNICoreDNS

   Contributing factor

   Missed in reviewNo configuration testsNo alert on limit

   Detection

   The RootVolUtilisation-Critical alert fired in the high-priority alerts channel at 13:42, and a user reported "no space left on device" image pull errors at 14:03.

   Remediation

   • Uncordoned the old default node group and drained the new nodes to move workloads back
   • Identified the launch template change that cut the root volume to 20 GB and applied a fix on the manager and live clusters
   • Drained the old node group onto the fixed nodes

   Action items

   • Update the runbook to compare launch template changes during an EKS module upgrade
   • Create a test setup that pulls images of different sizes, like live
   • Update the RootVolUtilisation alert runbook to check the disk space configuration
   • Scale CoreDNS dynamically based on the number of nodes
   • Investigate whether IPv6 can solve the IP prefix starvation problem
   • Add drift testing to identify when a terraform plan shows a change to the launch template
   • Set up logging for the CNI and ipamd logs and alerts on errors related to IP prefix starvation

   Lessons

   The write-up lists review actions rather than lessons. They aim at two gaps: the launch template changes brought by a module upgrade were not compared or drift-tested before rollout, and there were no CNI or ipamd logs or alerts to show the IP prefix starvation the team suspected during the drain.

   View case
2. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes

   DFDS2023sev1

   Symptoms

   At 09:33 UTC on 14 February 2023 end users told the platform team that Kubernetes logs were not arriving in CloudWatch as expected. The investigation went through fluentd and found that the nodes were running containerd as their container runtime instead of docker-shim. A scheduled node rollover the day before, at 07:30 UTC on 13 February, had set containerd as the default runtime by accident.

   Service outageResources deletedMonitoring outage

   Trigger

   A scheduled node rollover had accidentally made containerd the default container runtime, which stopped log shipping to CloudWatch; the unscheduled rollover to revert it timed out, and the pipeline run that followed reduced the Auto Scaling group from 30 to 15 instances.

   Root cause

   Configuration change

   Component

   Container runtimeNode launch templateNode poolTerraformTelemetry agent

   Contributing factor

   Automation amplificationUnreviewed defaultsUnrepresentative stagingNo configuration testsNo runbookHarmful manual action

   Detection

   End users reported the missing CloudWatch logs that started the work. The postmortem lists 2 minutes as the time to detection of the node failure.

   Remediation

   • ran the node rollover script to add new nodes, which failed on a timeout from Docker Hub
   • ran the node rollover script a second time, which added new nodes that the load balancer marked healthy at 11:01 UTC

   Action items

   • Document the rollover process, especially what can be done when it goes wrong
   • Investigate options for improving the rollover process
   • Add fluentd log collection to the QA test suite so logs are covered end to end
   • Add alarms for fluentd when too little or no log data is sent
   • Implement a staging environment
   • Implement guards against selecting an invalid container runtime

   Lessons

   When a change goes wrong, rolling back to a known working configuration is safer than rushing a fix, which can make the damage worse. Launching new worker nodes depends on the cloud-init script not failing in any way. The QA configuration differed from production, there was no staging environment, the QA tests did not cover fluentd log collection, and the rollover process had no documentation for when it fails.

   View case
3. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images

   UK Ministry of Justice2023

   Symptoms

   At 14:22 a Cloud Platform team member noticed failed Docker Hub logins. Fetching an image returned: failed to fetch manifest: Head "https://registry-1.docker.io/v2/ministryofjustice/cloud-platform-tools/manifests/2.1": toomanyrequests: too many failed login attempts for username or IP address

   Image pull failures

   Trigger

   The Docker Hub password was changed as part of LastPass remediation activities while the EKS cluster nodes and Concourse still logged in with the previous password.

   Root cause

   Operator error

   Component

   Container registry and its credentialsNode launch templateNode poolCI system

   Contributing factor

   Credential in configurationShared credential

   Detection

   A Cloud Platform team member saw the failed Docker Hub login error at 14:22. The Kuberhealthy daemonset check showed at 14:52 that the cluster could not pull images, and a user reported ErrImagePull in #ask-cloud-platform at 14:54.

   Remediation

   • Changed the Docker Hub password back to the previous value still used by the EKS cluster nodes and Concourse
   • Confirmed with the reporting user that the pod deployed and that cluster image pulls worked again

   Action items

   • Switch from the Docker Hub username and password to a Docker Hub token created specifically for Cloud Platform
   • Rotate the password cleanly out of hours with a graceful recycle of the nodes

   Lessons

   The write-up states no separate lessons. Its review action replaces the Docker Hub username and password with a Docker Hub token created specifically for Cloud Platform.

   View case
4. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case
5. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service

   DFDS2020

   Symptoms

   On 9 June 2020 at 12:25 CET a CloudWatch alert fired on a high count of 5XX responses from the AWS Application Load Balancers (ALBs) of DFDS's Hellman Kubernetes cluster, and at 12:28 the SRE team verified that no ingress traffic was reaching the cluster. Every service running inside the cluster was unreachable from outside it, because both ALBs, the authenticated one and the one without authentication, no longer had any Kubernetes worker nodes attached.

   Service outage

   Trigger

   Planned maintenance to split the Hellman cluster's auto-scaling groups by availability zone applied Terraform changes at 12:20 CET that removed the ALB instance attachments.

   Root cause

   Configuration change

   Component

   TerraformCloud load balancerNode launch template

   Contributing factor

   Unrepresentative stagingSignal buried in noiseMonitoring blind spot

   Detection

   A CloudWatch alert detected a high count of 5XX responses for the ALBs, and the lack of ingress traffic was then verified.

   Remediation

   • Attached the worker node instances to the ALBs by hand
   • Implemented a permanent fix within the hour

   Action items

   • Implement alerts on healthy and unhealthy ALB targets for more precise and immediate notice
   • Reduce alert noise further
   • Implement host name resolution for Goldpinger

   Lessons

   General alert noise is still too high, there was no alert on healthy and unhealthy ALB targets, and test clusters should check ingress through the ALBs.

   View case

By root cause3