Cases13 cases

1 to 5 of 13

1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications

   UK Ministry of Justice2026

   Symptoms

   At 08:56 on 30 March 2026, about an hour and three quarters after the rollout of a new ingress-nginx to production began, the MoJ Cloud Platform team noticed messages in the #hmpps-auth-audit-registers Slack channel about ModSecurity unexpectedly blocking requests. Users in HMPPS also raised incidents with the MoJ service desk.

   High error rateService outage

   Trigger

   The Cloud Platform team began deploying ingress-nginx 1.14.3, on images maintained by Chainguard, to the three production ingresses at 07:11 on 30 March 2026.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerWeb application firewall rules

   Contributing factor

   Unrepresentative staging

   Detection

   At 08:56 the Cloud Platform team saw messages in the #hmpps-auth-audit-registers Slack channel showing ModSecurity blocking requests unexpectedly; HMPPS users also raised incidents with the MoJ service desk.

   Remediation

   • rolled the ingress-nginx deployment back to the previous version, from 09:14 to 09:43
   • HMPPS applied ModSecurity rule exclusions for the rules blocking traffic to HMPPS Auth
   • added a temporary global exclusion for the CRS rule triggered by the PostHog cookie
   • redeployed ingress-nginx 1.14.3 with both exclusions in place, completed at 14:00

   Action items

   • Rescope the PostHog cookie so it is no longer sent with requests to all *.justice.gov.uk services
   • Remove the temporary global exclusion once the PostHog cookie has expired from enough users' browsers
   • Remind service teams to run dev and test workloads through the non-production ingress controllers so future ModSecurity and CRS updates are tested
   • Run production and non-production workloads in separate clusters in Cloud Platform 3.0, so non-production workloads must use the non-production ingress

   Lessons

   An ingress controller upgrade that bundles a web application firewall also changes which requests are allowed, not only how they are routed. A beta or non-production ingress only catches false positives for traffic that actually passes through it, and teams that keep their test workloads on the production ingress miss them. A fix on the client side, such as rescoping a cookie, takes effect only when old cookies have expired in users' browsers, so the rule change has to wait for it or ship with an exclusion.

   View case
2. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy

   Yonghyun Kim2025

   Symptoms

   Every rolling update of the team's microservices on Amazon EKS, behind an Application Load Balancer, was followed by intermittent 502 and 503 errors for users. A Slack message at 3 a.m. read "배포는 성공했는데, 서비스는 죽었다" (the deployment succeeded, but the service died). Every pod was Ready and Datadog was green while users saw errors.

   High error rateService outage

   Trigger

   A rolling update of a Deployment served through an ALB that registers pod IPs as targets, with no pod readiness gates.

   Root cause

   Faulty deployment

   Component

   Cloud load balancerIngress controllerPod termination

   Contributing factor

   Unverified assumptionUnreviewed defaultsMonitoring blind spot

   Detection

   A Slack message at 3 a.m. reported that the deployment had succeeded but the service was down; Datadog dashboards were green at the time.

   Remediation

   • labelled the namespace with elbv2.k8s.aws/pod-readiness-gate-inject=enabled so the AWS Load Balancer Controller injects pod readiness gates and Kubernetes waits for the ALB target to be healthy before deleting old pods

   Lessons

   A pod being Ready in Kubernetes does not mean the load balancer will send it traffic. Without a readiness gate, a rolling update can delete the old pods while the ALB still considers the new targets unhealthy, and dashboards that track pod status stay green during the errors. Making pods wait longer on shutdown reduced the errors but did not fix the start-up gap and made every deployment much slower; the readiness gate ties the pod's Ready state to the ALB's view of the target.

   View case
3. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour

   2i2c2025

   Symptoms

   At 5:00 PM BST on 12 August 2025 a 2i2c engineer declared an incident. While deploying a new cluster called temple and running into NFS issues, the engineer had deleted the prod namespace; the deletion hung, and listing the pods showed many users from a different cluster. The engineer's first message suggested that the kubeconfig had somehow been mutated or corrupted and that some other cluster was being torn down.

   Resources deletedService outage

   Trigger

   While working around a bug in the NFS deployment process for a new hub, an engineer ran a delete of the prod namespace with the kubeconfig context pointing at the LEAP cluster.

   Root cause

   Operator error

   Component

   Persistent volumes and stateful setsJobs and CronJobsIngress controllerTerraform

   Contributing factor

   No safeguard on destructive changeNo runbook

   Detection

   The engineer who ran the deletion noticed when it hung and the pod list showed users of another cluster, and declared the incident at 5:00 PM BST, before the uptime check on leap.2i2c.cloud failed at 5:32 PM.

   Remediation

   • deleted the single remaining user pod in prod so the namespace deletion completed
   • killed the nfs-share-creator job that was blocking the deployment
   • redeployed the prod hub with the updated NFS server IP address
   • deleted the prod ingress during the deployment so that users could not start servers mid-deployment

   Action items

   • Write an incident report
   • Write a short incident response document for the case "I deleted prod, what next?"
   • Upgrade jupyterhub-home-nfs and remove the nfs-share-creator job
   • Add opt-in hardening against nested kubeconfig contexts to the deployer system
   • Change the engineer's shell configuration to show distinct kubeconfig contexts more clearly

   Lessons

   Provisioning persistent storage outside Kubernetes with Terraform meant a deleted namespace cost an hour of downtime and no data. The engineer had no visible cue for which kubeconfig context was active, which turned a workaround on a new cluster into a deletion on a production one. The engineer knew the fix but did not feel confident redeploying without first confirming with the tech lead, in another time zone, that there was no risk of losing persistent data, and reaching the tech lead took time.

   View case
4. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services

   Backpackr2025

   Symptoms

   During a Lunar New Year event in January 2024, a developer posted a message in Slack suspecting an outage. While several people were checking it, the problem grew into a full outage of Backpackr's web and app services, and the company decided to stop the event.

   Service outage

   Trigger

   An unexpected traffic surge, more than three times normal, during the Lunar New Year event in January 2024.

   Root cause

   Capacity exhaustion

   Component

   conntrack tableIngress controllerIstio

   Contributing factor

   Single point of failureUnreviewed defaultsMonitoring blind spot

   Detection

   A developer reported a suspected outage in Slack; the outage became total while several people were still confirming it.

   Remediation

   • stopped the Lunar New Year event

   Action items

   • Raise net.netfilter.nf_conntrack_max on all nodes to memory bytes/16384/2, and three times that on Istio nodes, applied by the nodeOptimizer DaemonSet
   • Configure Istio mesh networking so sidecars communicate directly instead of through the ingressGateway
   • Upgrade the Istio nodes from c6.xlarge (12.5 Gbps) to n-type instances with up to 25 Gbps of network bandwidth
   • Strengthen monitoring with ENA network bandwidth and packet drops, detailed Istio ingressGateway and sidecar metrics, and CPU, memory and saturation

   Lessons

   When every request in a cluster goes through one ingress gateway node, that node's connection tracking table and network bandwidth set the limit for the whole service. The default nf_conntrack table size was too small for three times normal traffic, and once it filled, packets were dropped. After the outage Backpackr strengthened its monitoring of network bandwidth, packet drops and the Istio components.

   View case
5. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced

   Honeycomb2024

   Symptoms

   Event ingestion and the Honeycomb UI in EU1 became unavailable at 22:00:10 UTC on April 9, 2024, and the status page reported that the EU1 instance was down. Both the ingestion service and the service behind the web UI stopped at roughly the same moment.

   Service outageStale endpoint routing

   Trigger

   A routine system update to the Kubernetes cluster, rolled out earlier on April 9, followed by a regular deployment that replaced pods one at a time.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Monitoring blind spotUnrepresentative stagingChange with a delayed effect

   Detection

   The first status post, saying the EU1 instance was down, went up at 15:08 PDT (22:08 UTC), about 8 minutes after the outage began. There was no alerting on the AWS Load Balancer Controller.

   Remediation

   • Restarted the AWS Load Balancer Controller
   • Rolled back the Kubernetes cluster system update

   Action items

   • Find the exact cause of the failure so the system update can be deployed safely
   • Add monitoring and alerting on the AWS Load Balancer Controller so on-call engineers learn of a failure quickly and can pause deployments

   Lessons

   A rolling deployment can look healthy while the component that wires new pods into the load balancer is broken; the failure only shows when the last old pod goes away. Honeycomb now treats the AWS Load Balancer Controller as a critical component that needs its own alerting, so deployments can be paused before they remove the last pods that still receive traffic. Extensive testing in development and staging did not reveal the bad interaction.

   View case

By root cause7