Incident Library
Applied filters Root causeBreaking change in an upgradeComponentIngress controllerOrganisationUK Ministry of JusticeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
    Symptoms

    At 08:56 on 30 March 2026, about an hour and three quarters after the rollout of a new ingress-nginx to production began, the MoJ Cloud Platform team noticed messages in the #hmpps-auth-audit-registers Slack channel about ModSecurity unexpectedly blocking requests. Users in HMPPS also raised incidents with the MoJ service desk.

    High error rateService outage
    Trigger

    The Cloud Platform team began deploying ingress-nginx 1.14.3, on images maintained by Chainguard, to the three production ingresses at 07:11 on 30 March 2026.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerWeb application firewall rules
    Contributing factor
    Unrepresentative staging
    Detection

    At 08:56 the Cloud Platform team saw messages in the #hmpps-auth-audit-registers Slack channel showing ModSecurity blocking requests unexpectedly; HMPPS users also raised incidents with the MoJ service desk.

    Remediation
    • rolled the ingress-nginx deployment back to the previous version, from 09:14 to 09:43
    • HMPPS applied ModSecurity rule exclusions for the rules blocking traffic to HMPPS Auth
    • added a temporary global exclusion for the CRS rule triggered by the PostHog cookie
    • redeployed ingress-nginx 1.14.3 with both exclusions in place, completed at 14:00
    Action items
    • Rescope the PostHog cookie so it is no longer sent with requests to all *.justice.gov.uk services
    • Remove the temporary global exclusion once the PostHog cookie has expired from enough users' browsers
    • Remind service teams to run dev and test workloads through the non-production ingress controllers so future ModSecurity and CRS updates are tested
    • Run production and non-production workloads in separate clusters in Cloud Platform 3.0, so non-production workloads must use the non-production ingress
    Lessons

    An ingress controller upgrade that bundles a web application firewall also changes which requests are allowed, not only how they are routed. A beta or non-production ingress only catches false positives for traffic that actually passes through it, and teams that keep their test workloads on the production ingress miss them. A fix on the client side, such as rescoping a cookie, takes effect only when old cookies have expired in users' browsers, so the rule change has to wait for it or ship with an exclusion.

    View case
  2. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021
    Symptoms

    At 13:15 on 9 June 2021 a user reported in the #ask-cloud-platform channel that deploying a UAT application failed on its Ingress resource.

    Deploys failing
    Trigger

    The Cloud Platform team upgraded the ModSec ingress controller module to v3.33.0; the deployment appeared to succeed.

    Root cause
    Breaking change in an upgrade
    Component
    Admission webhookIngress controller
    Contributing factor
    Unrepresentative staging
    Detection

    A user reported the deployment error in the #ask-cloud-platform Slack channel at 13:15.

    Remediation
    • rolled back the ModSec ingress controller module to v0.0.7
    Action items
    • Find out why the issue was not flagged in the test cluster and try to reproduce it, possibly with a new test (ticket #2972)
    • Add a smoke test that checks the alerts in Alertmanager (ticket #2973)
    • Add a helloworld app that uses the ModSec controller so smoke tests can check that traffic works (ticket #2974)
    • Make a new version of the ModSec module work on EKS for both live-1 and live; neither the old nor the new version works on live (ticket #2975)
    View case
Check your cluster for this

These 2 cases involve Ingress controller and Admission webhook. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade2 cases

Component
Ingress controller2Admission webhook1Web application firewall rules1
Contributing factor
Unrepresentative staging2
Organisation
UK Ministry of Justice2
  1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  2. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021