Cases2 cases

1 to 2 of 2

1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications

   UK Ministry of Justice2026

   Symptoms

   At 08:56 on 30 March 2026, about an hour and three quarters after the rollout of a new ingress-nginx to production began, the MoJ Cloud Platform team noticed messages in the #hmpps-auth-audit-registers Slack channel about ModSecurity unexpectedly blocking requests. Users in HMPPS also raised incidents with the MoJ service desk.

   High error rateService outage

   Trigger

   The Cloud Platform team began deploying ingress-nginx 1.14.3, on images maintained by Chainguard, to the three production ingresses at 07:11 on 30 March 2026.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerWeb application firewall rules

   Contributing factor

   Unrepresentative staging

   Detection

   At 08:56 the Cloud Platform team saw messages in the #hmpps-auth-audit-registers Slack channel showing ModSecurity blocking requests unexpectedly; HMPPS users also raised incidents with the MoJ service desk.

   Remediation

   • rolled the ingress-nginx deployment back to the previous version, from 09:14 to 09:43
   • HMPPS applied ModSecurity rule exclusions for the rules blocking traffic to HMPPS Auth
   • added a temporary global exclusion for the CRS rule triggered by the PostHog cookie
   • redeployed ingress-nginx 1.14.3 with both exclusions in place, completed at 14:00

   Action items

   • Rescope the PostHog cookie so it is no longer sent with requests to all *.justice.gov.uk services
   • Remove the temporary global exclusion once the PostHog cookie has expired from enough users' browsers
   • Remind service teams to run dev and test workloads through the non-production ingress controllers so future ModSecurity and CRS updates are tested
   • Run production and non-production workloads in separate clusters in Cloud Platform 3.0, so non-production workloads must use the non-production ingress

   Lessons

   An ingress controller upgrade that bundles a web application firewall also changes which requests are allowed, not only how they are routed. A beta or non-production ingress only catches false positives for traffic that actually passes through it, and teams that keep their test workloads on the production ingress miss them. A fix on the client side, such as rescoping a cookie, takes effect only when old cookies have expired in users' browsers, so the rule change has to wait for it or ship with an exclusion.

   View case
2. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns

   UK Ministry of Justice2020

   Symptoms

   At 18:27 on 21 September 2020 the Cloud Platform team detected that some services running on the live-1 production cluster had gone down, and some users could not reach them. The incident was declared at 18:40.

   Service outageResources deletedMonitoring outageStale endpoint routing

   Trigger

   A script for deleting a test cluster was triggered while the kube context was pointing at the live-1 production cluster instead of the test cluster.

   Root cause

   Operator error

   Component

   Ingress controllerCredential broker for podsexternal-dnsCloud load balancerPrometheus

   Contributing factor

   No safeguard on destructive change

   Detection

   The first detection was at 18:27, and the incident was declared 13 minutes later, at 18:40.

   Remediation

   • restored the default ingress controller together with external-dns and kiam, which it depends on
   • corrected the owner field in the Formbuilder Route 53 TXT records so external-dns pointed their A records at the new NLB
   • restored the remaining components with a targeted apply
   • ran the apply pipeline to restore all certificates, ServiceMonitors and Prometheus rules from the environment repository

   View case

By root cause2