Cases12 cases

1 to 5 of 12

1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications

   UK Ministry of Justice2026

   Symptoms

   At 08:56 on 30 March 2026, about an hour and three quarters after the rollout of a new ingress-nginx to production began, the MoJ Cloud Platform team noticed messages in the #hmpps-auth-audit-registers Slack channel about ModSecurity unexpectedly blocking requests. Users in HMPPS also raised incidents with the MoJ service desk.

   High error rateService outage

   Trigger

   The Cloud Platform team began deploying ingress-nginx 1.14.3, on images maintained by Chainguard, to the three production ingresses at 07:11 on 30 March 2026.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerWeb application firewall rules

   Contributing factor

   Unrepresentative staging

   Detection

   At 08:56 the Cloud Platform team saw messages in the #hmpps-auth-audit-registers Slack channel showing ModSecurity blocking requests unexpectedly; HMPPS users also raised incidents with the MoJ service desk.

   Remediation

   • rolled the ingress-nginx deployment back to the previous version, from 09:14 to 09:43
   • HMPPS applied ModSecurity rule exclusions for the rules blocking traffic to HMPPS Auth
   • added a temporary global exclusion for the CRS rule triggered by the PostHog cookie
   • redeployed ingress-nginx 1.14.3 with both exclusions in place, completed at 14:00

   Action items

   • Rescope the PostHog cookie so it is no longer sent with requests to all *.justice.gov.uk services
   • Remove the temporary global exclusion once the PostHog cookie has expired from enough users' browsers
   • Remind service teams to run dev and test workloads through the non-production ingress controllers so future ModSecurity and CRS updates are tested
   • Run production and non-production workloads in separate clusters in Cloud Platform 3.0, so non-production workloads must use the non-production ingress

   Lessons

   An ingress controller upgrade that bundles a web application firewall also changes which requests are allowed, not only how they are routed. A beta or non-production ingress only catches false positives for traffic that actually passes through it, and teams that keep their test workloads on the production ingress miss them. A fix on the client side, such as rescoping a cookie, takes effect only when old cookies have expired in users' browsers, so the rule change has to wait for it or ship with an exclusion.

   View case
2. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down

   PortSwigger2026

   Symptoms

   From 10:00 UTC on 11 January 2026 every BSEE SaaS instance in PortSwigger's US and Canada regions was unavailable: customers could not open the Web UI or start scans. The platform team learned of it only when customers reported it through support at 09:52 UTC the next day.

   Service outageTLS certificate errors

   Trigger

   The upgrade of the AWS Load Balancer Controller from v1.14.1 to v1.16.0 on 28 November 2025 left an orphaned webhook TLS secret that cert-manager could not renew, and the certificate in it expired on 8 January 2026.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerAdmission webhookcert-managerHelmConfigMaps and SecretsCloud load balancer

   Contributing factor

   Change with a delayed effectMonitoring blind spot

   Detection

   Customers reported the outage through support at 09:52 UTC on 12 January 2026, almost 24 hours after impact began, and the platform team began investigating at 10:00 UTC.

   Remediation

   • Deleted the orphaned aws-load-balancer-tls secrets in the US and Canada clusters
   • Redeployed the AWS Load Balancer Controller configuration with Helm
   • Verified that new certificates were generated and the webhooks worked again

   Action items

   • Audit all clusters for orphaned secrets that have no owning Certificate resource and resolve them
   • Add high-level application access alerting for each SaaS cluster from an external probe

   View case
3. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours

   Wealthy2025sev1

   Symptoms

   Nodes that already existed kept running, but every new node, spot or standard, on either v1.32 or v1.33, failed to initialise and could not join the cluster. Spot nodes that were preempted and recreated never came back, so services degraded progressively: background workers on spot instances, reporting services and order allocation stopped completely, and the APIs on existing nodes kept answering but could not scale for load spikes.

   Scale-up failureContainer crash loopService outageDNS resolution failures

   Trigger

   Google upgraded the GKE control plane from v1.32.x to v1.33.2 at the version deprecation deadline, although auto-upgrade was disabled on the node pools.

   Root cause

   Breaking change in an upgrade

   Component

   GKE control planeNode poolClient-side DNS resolver and cachePersistent volumes and stateful setsConfigMaps and Secrets

   Contributing factor

   Deferred upgradesSingle point of failureIneffective rollbackNo headroom

   Remediation

   • Created a new GKE cluster at v1.33, the version of the forced upgrade
   • Upgraded Istio to 1.27.1 for Kubernetes 1.33 compatibility
   • Updated the cert-manager configuration for Kubernetes 1.33
   • Migrated all production workloads to the new cluster and verified pod scheduling, DNS resolution, PVC mounting and external integrations
   • Decommissioned the broken cluster

   Action items

   • Configure all clusters to use the GKE stable release channel
   • Implement a separate management cluster for Argo CD, monitoring and other control components (Q4 2025)
   • Review spot instance usage for critical workloads and add on-demand fallback pools
   • Document the SRE upgrade procedure: management cluster first, production after a monitoring period of several days
   • Document a Kubernetes, Istio and cert-manager compatibility matrix
   • Create an automated upgrade testing pipeline
   • Reserve static IPs for quick migrations
   • Pin versions of Istio, cert-manager, CSI and DNS addons
   • Stay current with stable releases and regularly check GKE alerts and announcements to avoid deprecation deadlines

   Lessons

   GKE upgrades the control plane at a version's deprecation deadline even when auto-upgrade is disabled, so staying inside the supported version window is how to avoid a forced upgrade. A control plane upgrade can leave existing nodes healthy while every new node fails, and spot nodes that cannot rejoin after preemption turn this into cascading failures. Relying heavily on spot instances for critical workers turned a node join failure into a progressive outage. Kubernetes upgrades have to be planned together with dependent components such as Istio and cert-manager, and without a standby cluster on the same provider there was no immediate failover; the only recovery left was building a new cluster.

   View case
4. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic

   Altinity2025

   Symptoms

   On a Saturday, an OpsGenie alert fired for a ClickHouse cluster in Altinity.Cloud that had gone silent: no queries, no metrics, and no logs from customers trying to connect. The pods were running and the nodes looked healthy, but no traffic reached the database. Other clusters soon showed the same thing, and external access was blocked across several GKE clusters.

   Service outage

   Trigger

   GKE automatically upgraded worker nodes to the ubuntu_containerd image 1.32.3-gke.1785003, which ships kernel 6.8.0-1022-gke.

   Root cause

   Breaking change in an upgrade

   Component

   kube-proxyNode operating system imageLinux kernelCloud load balancer

   Contributing factor

   Unpatched known bugNo staged rollout

   Detection

   An OpsGenie alert fired when one ClickHouse cluster stopped receiving queries, metrics and customer connection logs.

   Remediation

   • added maintenance exceptions to pause all upgrades on the GKE clusters that were not yet affected
   • changed the load balancer health check policy to ignore failed kube-proxy health checks, so traffic flowed to the nodes again
   • began migrating the affected clusters from the ubuntu_containerd to the cos_containerd node image by recreating nodes

   Action items

   • Expand image pre-testing pipelines to catch regressions earlier
   • Enable notifications for upcoming GKE auto-upgrades
   • Make GKE maintenance windows required across the fleet to control when upgrades roll out
   • Track Google's issue and wait for a patched Ubuntu image

   Lessons

   A managed node upgrade can break kube-proxy in a way that leaves pods and nodes looking healthy while the load balancer sends nothing, because the load balancer health check depends on kube-proxy. kube-proxy's own logs showed the cause. Altinity recommends that GKE users on ubuntu_containerd restore traffic by switching to externalTrafficPolicy: Cluster, which disables the kube-proxy health check in the load balancer, move production workloads to cos_containerd, and pause auto-upgrades where possible until a patched image has been confirmed and tested.

   View case
5. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java

   PortSwigger2024

   Symptoms

   On 24 October 2024 every BSEE SaaS customer instance showed the Environment Unavailable message, and the SaaS Uptime Over Time metric dropped sharply at 04:45. The Technology Team learned of the outage when the Growth team posted about it at 08:07 and a member of the Product team told the Technology Team verbally at 08:15.

   Container crash loopService outage

   Trigger

   AWS made Bottlerocket v1.26.0 the latest Bottlerocket AMI in eu-west-1 between 02:00 and 04:30, and at 04:00 Karpenter saw the new version and started replacing the EC2 nodes of the production EKS cluster with it.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system imageKarpenter

   Contributing factor

   Unpinned version

   Detection

   The Growth team reported the outage in the BSEE cloud onboarding issues channel at 08:07, and a member of the Product team told the Technology Team verbally at 08:15. The SaaS Uptime Over Time metric had already dropped at 04:45.

   Remediation

   • Reverted all EC2 instances in the SaaS production account to Bottlerocket v1.25.0
   • Deployed Bottlerocket 1.25.0 across all EKS clusters and rolled out new EKS nodes

   Action items

   • Change the Karpenter resources to deploy a specific versioned Bottlerocket AMI instead of the latest released AMI
   • Add the Bottlerocket release ATOM feed to an internal tooling release dashboard
   • Set up a public status page to communicate with customers

   View case

By root cause1