Cases5 cases

1 to 5 of 5

1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications

   UK Ministry of Justice2026

   Symptoms

   At 08:56 on 30 March 2026, about an hour and three quarters after the rollout of a new ingress-nginx to production began, the MoJ Cloud Platform team noticed messages in the #hmpps-auth-audit-registers Slack channel about ModSecurity unexpectedly blocking requests. Users in HMPPS also raised incidents with the MoJ service desk.

   High error rateService outage

   Trigger

   The Cloud Platform team began deploying ingress-nginx 1.14.3, on images maintained by Chainguard, to the three production ingresses at 07:11 on 30 March 2026.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerWeb application firewall rules

   Contributing factor

   Unrepresentative staging

   Detection

   At 08:56 the Cloud Platform team saw messages in the #hmpps-auth-audit-registers Slack channel showing ModSecurity blocking requests unexpectedly; HMPPS users also raised incidents with the MoJ service desk.

   Remediation

   • rolled the ingress-nginx deployment back to the previous version, from 09:14 to 09:43
   • HMPPS applied ModSecurity rule exclusions for the rules blocking traffic to HMPPS Auth
   • added a temporary global exclusion for the CRS rule triggered by the PostHog cookie
   • redeployed ingress-nginx 1.14.3 with both exclusions in place, completed at 14:00

   Action items

   • Rescope the PostHog cookie so it is no longer sent with requests to all *.justice.gov.uk services
   • Remove the temporary global exclusion once the PostHog cookie has expired from enough users' browsers
   • Remind service teams to run dev and test workloads through the non-production ingress controllers so future ModSecurity and CRS updates are tested
   • Run production and non-production workloads in separate clusters in Cloud Platform 3.0, so non-production workloads must use the non-production ingress

   Lessons

   An ingress controller upgrade that bundles a web application firewall also changes which requests are allowed, not only how they are routed. A beta or non-production ingress only catches false positives for traffic that actually passes through it, and teams that keep their test workloads on the production ingress miss them. A fix on the client side, such as rescoping a cookie, takes effect only when old cookies have expired in users' browsers, so the rule change has to wait for it or ship with an exclusion.

   View case
2. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced

   Honeycomb2024

   Symptoms

   Event ingestion and the Honeycomb UI in EU1 became unavailable at 22:00:10 UTC on April 9, 2024, and the status page reported that the EU1 instance was down. Both the ingestion service and the service behind the web UI stopped at roughly the same moment.

   Service outageStale endpoint routing

   Trigger

   A routine system update to the Kubernetes cluster, rolled out earlier on April 9, followed by a regular deployment that replaced pods one at a time.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Monitoring blind spotUnrepresentative stagingChange with a delayed effect

   Detection

   The first status post, saying the EU1 instance was down, went up at 15:08 PDT (22:08 UTC), about 8 minutes after the outage began. There was no alerting on the AWS Load Balancer Controller.

   Remediation

   • Restarted the AWS Load Balancer Controller
   • Rolled back the Kubernetes cluster system update

   Action items

   • Find the exact cause of the failure so the system update can be deployed safely
   • Add monitoring and alerting on the AWS Load Balancer Controller so on-call engineers learn of a failure quickly and can pause deployments

   Lessons

   A rolling deployment can look healthy while the component that wires new pods into the load balancer is broken; the failure only shows when the last old pod goes away. Honeycomb now treats the AWS Load Balancer Controller as a critical component that needs its own alerting, so deployments can be paused before they remove the last pods that still receive traffic. Extensive testing in development and staging did not reveal the bad interaction.

   View case
3. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change

   UK Ministry of Justice2021

   Symptoms

   At 13:15 on 9 June 2021 a user reported in the #ask-cloud-platform channel that deploying a UAT application failed on its Ingress resource.

   Deploys failing

   Trigger

   The Cloud Platform team upgraded the ModSec ingress controller module to v3.33.0; the deployment appeared to succeed.

   Root cause

   Breaking change in an upgrade

   Component

   Admission webhookIngress controller

   Contributing factor

   Unrepresentative staging

   Detection

   A user reported the deployment error in the #ask-cloud-platform Slack channel at 13:15.

   Remediation

   • rolled back the ModSec ingress controller module to v0.0.7

   Action items

   • Find out why the issue was not flagged in the test cluster and try to reproduce it, possibly with a new test (ticket #2972)
   • Add a smoke test that checks the alerts in Alertmanager (ticket #2973)
   • Add a helloworld app that uses the ModSec controller so smoke tests can check that traffic works (ticket #2974)
   • Make a new version of the ModSec module work on EKS for both live-1 and live; neither the old nor the new version works on live (ticket #2975)

   View case
4. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order

   Skyscanner2019

   Symptoms

   On a Sunday, a team using one of Skyscanner's Kubernetes clusters manually paged the out-of-hours Kubernetes support because their service showed elevated latency, worst at the p90 and maximum levels.

   High latencyLoad imbalanceOOMKilledIntermittent connection failures

   Trigger

   Upgrading the HAProxy ingress controller to v0.6 and going back from Service virtual IPs to individual pod IPs as backends, on the understanding that the sort-backends flag introduced in v0.5, false by default, meant backends were randomly shuffled.

   Root cause

   Software defect

   Component

   Ingress controllerHAProxyConfiguration templating

   Contributing factor

   Unrepresentative stagingUnverified assumptionIgnored early signal

   Detection

   A cluster user manually paged the out-of-hours Kubernetes support about elevated latency; Prometheus queries then showed the uneven request distribution across pods.

   Remediation

   • switched the affected service from pod IPs back to the Service virtual IP through a config change in the internal deployment tool Slingshot
   • patched the controller to keep each backend's servers in an insertion-ordered slice and changed the template to iterate over that slice
   • tested the patch in sandbox clusters and rolled it out to the production clusters

   Action items

   • Run the internal controller patch in production and return to pod IPs as the default backends
   • Raise the problem upstream with a GitHub issue and pull request
   • Test changes to cluster components against the specific behaviour and conditions that matter

   Lessons

   The team already knew that sorted backends cause hot-podding and still did not test the new flag well enough to see that it did not work; changes to cluster components need tests aimed at the exact behaviour that matters, under conditions like production. A keyword can behave differently in different parts of a language: range over a map is unordered in Go code but sorted in text/template. A small cluster component can do outsized harm to every user of the cluster and, in the end, to customers.

   View case
5. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA

   loveholidays2019

   Symptoms

   On the evening of a long-overdue upgrade of a loveholidays production GKE cluster from 1.10.X to 1.12.5-gke.10, planned to take 2 to 2.5 hours, the master upgrade and the first two 7-node NodePools went smoothly. After that the NodePool upgrades slowed to a crawl. Each node is cordoned, drained and upgraded in turn. Even a 14-node cluster at about 5 minutes per node took over an hour, and with about 100 more nodes to go, some of them taking over 15 minutes to drain, the sequential drains were too slow to finish at a reasonable hour.

   Upgrade stallPods PendingScale-up failureService outageAutoscaler misbehaviour

   Trigger

   A manual upgrade of the GKE masters and NodePools from 1.10.X to 1.12.5-gke.10, two versions at once, on 19 March 2019; GKE's release notes of 14 March had already said that version was no longer available for new clusters or master upgrades because masters showed elevated error rates, but it could still be selected.

   Root cause

   Cloud provider outage

   Component

   GKE control planeNode poolCluster autoscalerIngress controllerHorizontal pod autoscaler

   Contributing factor

   Unrepresentative stagingOverlapping failuresIgnored early signal

   Detection

   The slow drains, zombie nodes and stuck NodePool were seen by the engineer running the upgrade; the broken Ingress was noticed after the upgrade and explained by Google support; the HPA problem was noticed the next day.

   Remediation

   • drained the remaining NodePool nodes ahead of the upgrade with kubectl drain --ignore-daemonsets --delete-local-data
   • deleted the zombie instances from the managed instance group with gcloud compute instance-groups managed delete-instances
   • cancelled the stuck UPGRADE_NODES operation with gcloud beta container operations cancel
   • used kubectl port-forward in place of the broken internal Ingresses
   • scaled down the older of two active ReplicaSets, or triggered a dummy deployment, to bring the HPA back

   Action items

   • Alert when a HorizontalPodAutoscaler is at 70% and at 100% of its capacity
   • Alert when a deployment has had two active ReplicaSets for 30 minutes and for 60 minutes
   • Re-evaluate termination grace periods that were set to arbitrary large values
   • Read the GKE release notes before every cluster upgrade
   • Use auto-upgrade in non-production environments, and manual cluster upgrades with automatic node upgrades in production
   • Keep environments as identical as possible, including resource names

   Lessons

   Read the release notes before upgrading: the target version had been withdrawn five days earlier. Drain time depends on pod configuration, so emptyDir volumes, long grace periods and missing disruption budgets turn a NodePool upgrade into hours. Pre-production only catches problems like the ingress bug if it matches production down to resource names. Auto-upgrades in non-production surface problems early, and a mix of manual cluster upgrades and automatic node upgrades in production keeps clusters current without much effort.

   View case

By root cause3