Incident Library
Applied filters Root causeBreaking change in an upgradeComponentKarpenterDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024
    Symptoms

    On 24 October 2024 every BSEE SaaS customer instance showed the Environment Unavailable message, and the SaaS Uptime Over Time metric dropped sharply at 04:45. The Technology Team learned of the outage when the Growth team posted about it at 08:07 and a member of the Product team told the Technology Team verbally at 08:15.

    Container crash loopService outage
    Trigger

    AWS made Bottlerocket v1.26.0 the latest Bottlerocket AMI in eu-west-1 between 02:00 and 04:30, and at 04:00 Karpenter saw the new version and started replacing the EC2 nodes of the production EKS cluster with it.

    Root cause
    Breaking change in an upgrade
    Component
    Node operating system imageKarpenter
    Contributing factor
    Unpinned version
    Detection

    The Growth team reported the outage in the BSEE cloud onboarding issues channel at 08:07, and a member of the Product team told the Technology Team verbally at 08:15. The SaaS Uptime Over Time metric had already dropped at 04:45.

    Remediation
    • Reverted all EC2 instances in the SaaS production account to Bottlerocket v1.25.0
    • Deployed Bottlerocket 1.25.0 across all EKS clusters and rolled out new EKS nodes
    Action items
    • Change the Karpenter resources to deploy a specific versioned Bottlerocket AMI instead of the latest released AMI
    • Add the Bottlerocket release ATOM feed to an internal tooling release dashboard
    • Set up a public status page to communicate with customers
    View case
Check your cluster for this

This case involves Karpenter and Node operating system image. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade1 case

Component
KarpenterNode operating system image
Contributing factor
Unpinned version
Organisation
PortSwigger
  1. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024