Cases2 cases

1 to 2 of 2

1. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java

   PortSwigger2024

   Symptoms

   On 24 October 2024 every BSEE SaaS customer instance showed the Environment Unavailable message, and the SaaS Uptime Over Time metric dropped sharply at 04:45. The Technology Team learned of the outage when the Growth team posted about it at 08:07 and a member of the Product team told the Technology Team verbally at 08:15.

   Container crash loopService outage

   Trigger

   AWS made Bottlerocket v1.26.0 the latest Bottlerocket AMI in eu-west-1 between 02:00 and 04:30, and at 04:00 Karpenter saw the new version and started replacing the EC2 nodes of the production EKS cluster with it.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system imageKarpenter

   Contributing factor

   Unpinned version

   Detection

   The Growth team reported the outage in the BSEE cloud onboarding issues channel at 08:07, and a member of the Product team told the Technology Team verbally at 08:15. The SaaS Uptime Over Time metric had already dropped at 04:45.

   Remediation

   • Reverted all EC2 instances in the SaaS production account to Bottlerocket v1.25.0
   • Deployed Bottlerocket 1.25.0 across all EKS clusters and rolled out new EKS nodes

   Action items

   • Change the Karpenter resources to deploy a specific versioned Bottlerocket AMI instead of the latest released AMI
   • Add the Bottlerocket release ATOM feed to an internal tooling release dashboard
   • Set up a public status page to communicate with customers

   View case
2. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours

   CoW Protocol2024

   Symptoms

   At 16:40 UTC on 19 April 2024 CoW Protocol's mainnet production environment stopped producing settlements; the outage officially began then.

   Pods PendingIP address exhaustionPod evictionService outage

   Trigger

   During a cluster capacity expansion, the Infrastructure as Code automation updated the AWS VPC CNI plugin, because the CNI version was not hardcoded in the IaC and the IaC library version had recently changed.

   Root cause

   Configuration change

   Component

   AWS VPC CNINode pool

   Contributing factor

   Unpinned versionOverlapping failures

   Detection

   The CNI version mismatch was identified as the cause at 17:03 UTC, 23 minutes after settlements stopped.

   Remediation

   • upgraded the Kubernetes cluster
   • set up new node groups with a new version of the networking plugin

   Lessons

   Versions in IaC scripts should be specified explicitly so an automation run cannot update a component such as the CNI plugin unintentionally. Changes in production need change management that is easy to revert.

   View case

By root cause2