Cases32 cases

6 to 10 of 32

1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour

   2i2c2025

   Symptoms

   At 5:00 PM BST on 12 August 2025 a 2i2c engineer declared an incident. While deploying a new cluster called temple and running into NFS issues, the engineer had deleted the prod namespace; the deletion hung, and listing the pods showed many users from a different cluster. The engineer's first message suggested that the kubeconfig had somehow been mutated or corrupted and that some other cluster was being torn down.

   Resources deletedService outage

   Trigger

   While working around a bug in the NFS deployment process for a new hub, an engineer ran a delete of the prod namespace with the kubeconfig context pointing at the LEAP cluster.

   Root cause

   Operator error

   Component

   Persistent volumes and stateful setsJobs and CronJobsIngress controllerTerraform

   Contributing factor

   No safeguard on destructive changeNo runbook

   Detection

   The engineer who ran the deletion noticed when it hung and the pod list showed users of another cluster, and declared the incident at 5:00 PM BST, before the uptime check on leap.2i2c.cloud failed at 5:32 PM.

   Remediation

   • deleted the single remaining user pod in prod so the namespace deletion completed
   • killed the nfs-share-creator job that was blocking the deployment
   • redeployed the prod hub with the updated NFS server IP address
   • deleted the prod ingress during the deployment so that users could not start servers mid-deployment

   Action items

   • Write an incident report
   • Write a short incident response document for the case "I deleted prod, what next?"
   • Upgrade jupyterhub-home-nfs and remove the nfs-share-creator job
   • Add opt-in hardening against nested kubeconfig contexts to the deployer system
   • Change the engineer's shell configuration to show distinct kubeconfig contexts more clearly

   Lessons

   Provisioning persistent storage outside Kubernetes with Terraform meant a deleted namespace cost an hour of downtime and no data. The engineer had no visible cue for which kubeconfig context was active, which turned a workaround on a new cluster into a deletion on a production one. The engineer knew the fix but did not feel confident redeploying without first confirming with the tech lead, in another time zone, that there was no risk of losing persistent data, and reaching the tech lead took time.

   View case
2. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services

   Backpackr2025

   Symptoms

   During a Lunar New Year event in January 2024, a developer posted a message in Slack suspecting an outage. While several people were checking it, the problem grew into a full outage of Backpackr's web and app services, and the company decided to stop the event.

   Service outage

   Trigger

   An unexpected traffic surge, more than three times normal, during the Lunar New Year event in January 2024.

   Root cause

   Capacity exhaustion

   Component

   conntrack tableIngress controllerIstio

   Contributing factor

   Single point of failureUnreviewed defaultsMonitoring blind spot

   Detection

   A developer reported a suspected outage in Slack; the outage became total while several people were still confirming it.

   Remediation

   • stopped the Lunar New Year event

   Action items

   • Raise net.netfilter.nf_conntrack_max on all nodes to memory bytes/16384/2, and three times that on Istio nodes, applied by the nodeOptimizer DaemonSet
   • Configure Istio mesh networking so sidecars communicate directly instead of through the ingressGateway
   • Upgrade the Istio nodes from c6.xlarge (12.5 Gbps) to n-type instances with up to 25 Gbps of network bandwidth
   • Strengthen monitoring with ENA network bandwidth and packet drops, detailed Istio ingressGateway and sidecar metrics, and CPU, memory and saturation

   Lessons

   When every request in a cluster goes through one ingress gateway node, that node's connection tracking table and network bandwidth set the limit for the whole service. The default nf_conntrack table size was too small for three times normal traffic, and once it filled, packets were dropped. After the outage Backpackr strengthened its monitoring of network bandwidth, packet drops and the Istio components.

   View case
3. IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)

   Wiz2025

   Symptoms

   This is a vulnerability report, not an outage, so nothing failed visibly. A cluster is exposed if it runs Ingress NGINX Controller older than 1.12.1 or 1.11.5 with its validating admission webhook enabled. Wiz suggests `kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx` to find it. The webhook accepts AdmissionReview requests over HTTP without authentication from any pod, and Wiz found more than 6,500 clusters exposing it to the public internet.

   Unauthorised access or a foreign workload

   Trigger

   Wiz Research reviewed the code of the Ingress NGINX admission controller and reported the first findings to Kubernetes on 31 December 2024.

   Root cause

   Software defect

   Component

   Ingress controllerAdmission webhook

   Contributing factor

   Unreviewed defaultsExcessive blast radius

   Detection

   Found by Wiz Research in a code review of the admission controller, not by an operator.

   Remediation

   • Kubernetes patched the injection flaws CVE-2025-1097, CVE-2025-1098 and CVE-2025-24514
   • Kubernetes removed NGINX configuration testing from the ingress-nginx admission controller, resolving CVE-2025-1974
   • released Ingress NGINX Controller 1.12.1 and 1.11.5

   Lessons

   Admission webhooks often run as web servers inside the cluster without authentication and with relatively high privileges, so any pod can reach them, and the flaws pair well with SSRF in an application. Wiz concludes that `nginx -t` should be considered harmful when it processes unsanitized user input. Wiz recommends upgrading, keeping the webhook off the internet and allowing only the API server to reach it with network policies. If upgrading must wait, disable the webhook (Helm: controller.admissionWebhooks.enabled=false; manual installs: delete the ValidatingWebhookConfiguration ingress-nginx-admission and remove --validating-webhook from the controller) and enable it again after the upgrade.

   View case
4. IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was running

   Cresta2025

   Symptoms

   During a deployment, traffic into the Kubernetes cluster started hitting dead ends: some API requests timed out and requests were slow and disrupted while the rollout ran.

   Intermittent timeoutsStale endpoint routingHigh latency

   Trigger

   A deployment rollout changed backend endpoints while a long-running NGINX full reload was in progress, with the enable-serial-reloads option enabled on the ingress-nginx controller.

   Root cause

   Software defect

   Component

   Ingress controller

   Contributing factor

   –

   Remediation

   • restarted the NGINX Ingress controller as an attempt to resolve the problem

   Action items

   • Open an issue in kubernetes/ingress-nginx describing how enable-serial-reloads blocks dynamic endpoint updates

   Lessons

   A flag meant only to serialise full reloads also delayed endpoint updates, which opened a window during rollouts in which the ingress routed to pods that no longer existed. Options that change how the ingress controller applies configuration need to be checked against what happens to endpoint updates during a long reload.

   View case
5. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic

   HubSpot2024

   Symptoms

   Between 14:20 and 15:16 UTC on 27 August 2024, some HubSpot customers could not log in to their accounts, reach their content or use HubSpot tools. Requests were slow or failed because HubSpot's internal load balancers, which run on Kubernetes and spread traffic across the backend services, were overloaded.

   Probe failuresHigh latencyHigh error rate

   Trigger

   Routine operation of a critical component of HubSpot's traffic management system, from about 14:08 UTC, ran path handling updates in the load balancer stack, and an issue in that stack made those updates block health check responses.

   Root cause

   Software defect

   Component

   Ingress controllerLiveness and readiness probes

   Contributing factor

   Automation amplificationNo headroom

   Detection

   Monitoring detected widespread performance degradation across the platform at 14:23 UTC and the incident response team was mobilised. HubSpot says it had identified the issue internally by 14:20 UTC. The status page was not updated until 14:44 UTC, which left customers with outdated information.

   Remediation

   • Tried to increase the capacity of the remaining load balancer instances
   • Brought new load balancer instances online, which relieved the pressure and let traffic return to normal

   Action items

   • Change the health check configuration so health checks do not time out falsely
   • Redesign the health check system so health checks get priority and keep running during operations such as configuration updates
   • Increase the capacity of the load balancers for sudden traffic spikes
   • Change the load balancing architecture to detect traffic surges earlier and distribute load dynamically in real time
   • Update the status page automatically for large-scale outages
   • Integrate the monitoring systems more tightly with the communication tools
   • Expand load testing to simulate high-traffic scenarios
   • Introduce chaos engineering to test platform resilience regularly
   • Add traffic pattern metrics, including the ratio of current connections to the maximum allowed
   • Add auto-scaling that adjusts resource capacity to traffic patterns

   Lessons

   HubSpot concluded that earlier detection of the abnormal traffic surge could have prevented the cascading failure, and that health checks have to keep running during operations such as configuration updates so that working load balancers are not removed together. The late status page update caused confusion for customers.

   View case

By root cause8