Organisation

UK Ministry of Justice

UK Ministry of Justice · uk-ministry-of-justice

15 cases · 2020 to 2026

Root cause Capacity exhaustion5 Configuration change3 Faulty deployment2 Operator error2 Breaking change in an upgrade2 unknown1

Contributing factor No alert on limit5 Unreviewed defaults2 No configuration tests2 Single point of failure2 Unrepresentative staging2 Excessive blast radius1 Credential in configuration1 Unknown limit1 Harmful manual action1 Monitoring blind spot1 No runbook1 No safeguard on destructive change1 No headroom1 Missed in review1 Shared credential1

Symptom Monitoring outage6 Service outage4 Deploys failing2 Image pull failures2 IP address exhaustion2 Pods stuck in ContainerCreating2 Pod eviction2 Probe failures2 Container crash loop1 High error rate1 Node disk full1 Node NotReady1 OOMKilled1 Pods Pending1 Resources deleted1 Stale endpoint routing1 Upgrade stall1

Cases

IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applicationsUK Ministry of Justice · 2026
Root cause Breaking change in an upgrade
IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went downUK Ministry of Justice · 2025
Root cause Faulty deployment
IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk spaceUK Ministry of Justice · 2024
Root cause Capacity exhaustion
IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowedUK Ministry of Justice · 2024
Root cause Capacity exhaustion
IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metricsUK Ministry of Justice · 2023
Root cause unknown
IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023
Root cause Configuration change
IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategyUK Ministry of Justice · 2023
Root cause Capacity exhaustion
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023
Root cause Capacity exhaustion
IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreatingUK Ministry of Justice · 2023
Root cause Configuration change
IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023
Root cause Configuration change
IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull imagesUK Ministry of Justice · 2023
Root cause Operator error
IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and changeUK Ministry of Justice · 2021
Root cause Breaking change in an upgrade
IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dnsUK Ministry of Justice · 2020
Root cause Operator error
IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 errorUK Ministry of Justice · 2020
Root cause Faulty deployment
IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memoryUK Ministry of Justice · 2020
Root cause Capacity exhaustion