Cases7 cases

1 to 5 of 7

1. IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)

   Wiz2025

   Symptoms

   This is a vulnerability report, not an outage, so nothing failed visibly. A cluster is exposed if it runs Ingress NGINX Controller older than 1.12.1 or 1.11.5 with its validating admission webhook enabled. Wiz suggests `kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx` to find it. The webhook accepts AdmissionReview requests over HTTP without authentication from any pod, and Wiz found more than 6,500 clusters exposing it to the public internet.

   Unauthorised access or a foreign workload

   Trigger

   Wiz Research reviewed the code of the Ingress NGINX admission controller and reported the first findings to Kubernetes on 31 December 2024.

   Root cause

   Software defect

   Component

   Ingress controllerAdmission webhook

   Contributing factor

   Unreviewed defaultsExcessive blast radius

   Detection

   Found by Wiz Research in a code review of the admission controller, not by an operator.

   Remediation

   • Kubernetes patched the injection flaws CVE-2025-1097, CVE-2025-1098 and CVE-2025-24514
   • Kubernetes removed NGINX configuration testing from the ingress-nginx admission controller, resolving CVE-2025-1974
   • released Ingress NGINX Controller 1.12.1 and 1.11.5

   Lessons

   Admission webhooks often run as web servers inside the cluster without authentication and with relatively high privileges, so any pod can reach them, and the flaws pair well with SSRF in an application. Wiz concludes that `nginx -t` should be considered harmful when it processes unsanitized user input. Wiz recommends upgrading, keeping the webhook off the internet and allowing only the API server to reach it with network policies. If upgrading must wait, disable the webhook (Helm: controller.admissionWebhooks.enabled=false; manual installs: delete the ValidatingWebhookConfiguration ingress-nginx-admission and remove --validating-webhook from the controller) and enable it again after the upgrade.

   View case
2. IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was running

   Cresta2025

   Symptoms

   During a deployment, traffic into the Kubernetes cluster started hitting dead ends: some API requests timed out and requests were slow and disrupted while the rollout ran.

   Intermittent timeoutsStale endpoint routingHigh latency

   Trigger

   A deployment rollout changed backend endpoints while a long-running NGINX full reload was in progress, with the enable-serial-reloads option enabled on the ingress-nginx controller.

   Root cause

   Software defect

   Component

   Ingress controller

   Contributing factor

   –

   Remediation

   • restarted the NGINX Ingress controller as an attempt to resolve the problem

   Action items

   • Open an issue in kubernetes/ingress-nginx describing how enable-serial-reloads blocks dynamic endpoint updates

   Lessons

   A flag meant only to serialise full reloads also delayed endpoint updates, which opened a window during rollouts in which the ingress routed to pods that no longer existed. Options that change how the ingress controller applies configuration need to be checked against what happens to endpoint updates during a long reload.

   View case
3. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic

   HubSpot2024

   Symptoms

   Between 14:20 and 15:16 UTC on 27 August 2024, some HubSpot customers could not log in to their accounts, reach their content or use HubSpot tools. Requests were slow or failed because HubSpot's internal load balancers, which run on Kubernetes and spread traffic across the backend services, were overloaded.

   Probe failuresHigh latencyHigh error rate

   Trigger

   Routine operation of a critical component of HubSpot's traffic management system, from about 14:08 UTC, ran path handling updates in the load balancer stack, and an issue in that stack made those updates block health check responses.

   Root cause

   Software defect

   Component

   Ingress controllerLiveness and readiness probes

   Contributing factor

   Automation amplificationNo headroom

   Detection

   Monitoring detected widespread performance degradation across the platform at 14:23 UTC and the incident response team was mobilised. HubSpot says it had identified the issue internally by 14:20 UTC. The status page was not updated until 14:44 UTC, which left customers with outdated information.

   Remediation

   • Tried to increase the capacity of the remaining load balancer instances
   • Brought new load balancer instances online, which relieved the pressure and let traffic return to normal

   Action items

   • Change the health check configuration so health checks do not time out falsely
   • Redesign the health check system so health checks get priority and keep running during operations such as configuration updates
   • Increase the capacity of the load balancers for sudden traffic spikes
   • Change the load balancing architecture to detect traffic surges earlier and distribute load dynamically in real time
   • Update the status page automatically for large-scale outages
   • Integrate the monitoring systems more tightly with the communication tools
   • Expand load testing to simulate high-traffic scenarios
   • Introduce chaos engineering to test platform resilience regularly
   • Add traffic pattern metrics, including the ratio of current connections to the maximum allowed
   • Add auto-scaling that adjusts resource capacity to traffic patterns

   Lessons

   HubSpot concluded that earlier detection of the abnormal traffic surge could have prevented the cascading failure, and that health checks have to keep running during operations such as configuration updates so that working load balancers are not removed together. The late status page update caused confusion for customers.

   View case
4. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404

   DFDS2021

   Symptoms

   At 08:00 UTC on 2021-09-06 every service in DFDS's Kubernetes cluster answered with 404. Investigation started five minutes later.

   Service outageHigh error rateProbe failures

   Trigger

   The automated Kubernetes node rollover on 2021-09-06 restarted all the Traefik v2 pods, five days after a malformed IngressRoute manifest had been applied.

   Root cause

   Software defect

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Change with a delayed effectNo configuration testsExcessive blast radius

   Detection

   All services returned 404 from 08:00 UTC and root cause investigation started at 08:05. Time to detection was 5 minutes.

   Remediation

   • changed the ALB health check path, which made the targets healthy but did not stop the 404s
   • rolled back by hand to Traefik v1 as the main ingress controller
   • deleted the malformed IngressRoute by hand and confirmed the Traefik v2 bug
   • applied the changes upstream

   Action items

   • Upgrade Traefik to v2.5, where the bug is fixed, and verify that it removes the bug
   • Change the Traefik v2 health check path to something more robust
   • Set up an external liveness probe on the ingress controller
   • Consider workload segmentation to limit the blast zone
   • Look into monitoring the Traefik v2 API for rapid changes
   • Look into smoke testing during and after node rollover

   Lessons

   A malformed IngressRoute did nothing visible for five days and then took down all ingress when every Traefik v2 pod had restarted during a node rollover. Because Traefik v2 failed gracefully, the first signals pointed at the ALB health checks instead of at Traefik. On incident handling, the team concluded that appointing an incident manager could have helped, that the war room meeting should be recorded and a log book kept, that people not involved should only make suggestions in writing, and that a short first round of sharing thoughts and gut feelings is useful.

   View case
5. IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them traffic

   Ravelin2019

   Symptoms

   After Ravelin moved its API layer from VMs into its GKE cluster behind a Kubernetes Ingress, the integration tests occasionally received 502 errors.

   Intermittent connection failuresStale endpoint routing

   Trigger

   Pods behind the Ingress being removed by their controller, as happens during rolling deploys and scaling.

   Root cause

   Software defect

   Component

   Ingress controllerPod termination

   Contributing factor

   Unverified assumption

   Detection

   Integration tests occasionally received 502 errors.

   Lessons

   Removing a pod's endpoint and signalling the pod happen at the same time, and an ingress can take a long time to catch up. A pod behind an ingress should keep serving after SIGTERM until its grace period ends, and the grace period should be longer than the load balancer's reprogramming time.

   View case

By root cause1