IL-0404 PortSwigger · 2024-10-24kubernetesaws

All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java

Bottlerocket PR-158, merged on 16 October 2024, added a default security setting that blocks writable and executable memory for all services, `MemoryDenyWriteExecute=yes`, and Bottlerocket v1.26.0, released on 24 October at 02:00, included it. The systemd documentation for this option says it is incompatible with programs that generate code at runtime, including JIT execution engines, so on v1.26.0 applications that rely on JIT compilation, such as Java, could not run. PortSwigger's Karpenter resources used the latest released Bottlerocket AMI, so once AWS published v1.26.0 in eu-west-1, Karpenter rolled the production EKS nodes onto it and every BSEE SaaS customer instance failed.

The team was told of the outage by other teams at 08:07 and 08:15, found the crash-looping pods and the JVM error, saw that the nodes had been updated at 05:33 to v1.26.0, and found the matching Bottlerocket issue. The team reverted all EC2 instances in the SaaS production account to Bottlerocket v1.25.0, deployed v1.25.0 across all EKS clusters and rolled out new nodes, and customer instances came back from 09:15. The incident lasted 3h 45m, affected all customers, raised one support case and lost no customer data.

Severity: –
Scope: Every BSEE SaaS customer instance was unavailable and showed the Environment Unavailable message; one support case was raised and no customer data was lost
Users affected: all BSEE SaaS customers
Region: eu-west-1
Time to detect: –
Time to resolve: 3h 45m

Symptoms

On 24 October 2024 every BSEE SaaS customer instance showed the Environment Unavailable message, and the SaaS Uptime Over Time metric dropped sharply at 04:45. The Technology Team learned of the outage when the Growth team posted about it at 08:07 and a member of the Product team told the Technology Team verbally at 08:15.

All customer Kubernetes pods were in crash loop back off. The logs of the `init-enterprise-server-keystore` container, which runs `/usr/local/burpsuite_enterprise/bin/createKeystore`, showed `Error occurred during initialization of VM` followed by `Failed to mark memory page as executable - check if grsecurity/PaX is enabled`.

The error pointed the team at the operating system. The AWS EC2 console showed the EC2 nodes had been updated at 05:33 and the workloads had moved to the new nodes after that. The nodes ran Bottlerocket v1.26.0, and the Bottlerocket GitHub repository had Issue 4260 about compatibility problems of v1.26.0 with Java-based applications.

Symptom Container crash loop Service outage

Root cause

Root cause Breaking change in an upgrade

Component Node operating system image Karpenter

Contributing factor Unpinned version

Trigger: AWS made Bottlerocket v1.26.0 the latest Bottlerocket AMI in eu-west-1 between 02:00 and 04:30, and at 04:00 Karpenter saw the new version and started replacing the EC2 nodes of the production EKS cluster with it.

Detection

The Growth team reported the outage in the BSEE cloud onboarding issues channel at 08:07, and a member of the Product team told the Technology Team verbally at 08:15. The SaaS Uptime Over Time metric had already dropped at 04:45.

Remediation

Reverted all EC2 instances in the SaaS production account to Bottlerocket v1.25.0
Deployed Bottlerocket 1.25.0 across all EKS clusters and rolled out new EKS nodes

Action items

Change the Karpenter resources to deploy a specific versioned Bottlerocket AMI instead of the latest released AMI
Add the Bottlerocket release ATOM feed to an internal tooling release dashboard
Set up a public status page to communicate with customers

Lessons

–

Sources

https://github.com/portswigger-cloud/postmortems/blob/main/2024/10/saas-outage.md · github

References

Related cases

IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the APIOpenAI · 2025
same root cause · shared Node operating system image
IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress trafficAltinity · 2025
same root cause · shared Node operating system image
IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providersDatadog · 2023
same root cause · shared Node operating system image
IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable2i2c · 2022
same root cause · shared Node operating system image
IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOffMicrosoft Azure · 2022
same root cause · shared Node operating system image
IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusableEnigmo · 2020
same root cause · shared Node operating system image

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Node operating system image, Karpenter among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0404
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0404/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.