Cases5 cases

1 to 5 of 5

1. IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)

   Wiz2025

   Symptoms

   This is a vulnerability report, not an outage, so nothing failed visibly. A cluster is exposed if it runs Ingress NGINX Controller older than 1.12.1 or 1.11.5 with its validating admission webhook enabled. Wiz suggests `kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx` to find it. The webhook accepts AdmissionReview requests over HTTP without authentication from any pod, and Wiz found more than 6,500 clusters exposing it to the public internet.

   Unauthorised access or a foreign workload

   Trigger

   Wiz Research reviewed the code of the Ingress NGINX admission controller and reported the first findings to Kubernetes on 31 December 2024.

   Root cause

   Software defect

   Component

   Ingress controllerAdmission webhook

   Contributing factor

   Unreviewed defaultsExcessive blast radius

   Detection

   Found by Wiz Research in a code review of the admission controller, not by an operator.

   Remediation

   • Kubernetes patched the injection flaws CVE-2025-1097, CVE-2025-1098 and CVE-2025-24514
   • Kubernetes removed NGINX configuration testing from the ingress-nginx admission controller, resolving CVE-2025-1974
   • released Ingress NGINX Controller 1.12.1 and 1.11.5

   Lessons

   Admission webhooks often run as web servers inside the cluster without authentication and with relatively high privileges, so any pod can reach them, and the flaws pair well with SSRF in an application. Wiz concludes that `nginx -t` should be considered harmful when it processes unsanitized user input. Wiz recommends upgrading, keeping the webhook off the internet and allowing only the API server to reach it with network policies. If upgrading must wait, disable the webhook (Helm: controller.admissionWebhooks.enabled=false; manual installs: delete the ValidatingWebhookConfiguration ingress-nginx-admission and remove --validating-webhook from the controller) and enable it again after the upgrade.

   View case
2. IL-0065 Any pod could take over the cluster by rewriting Argo CD's unauthenticated Redis cache (CVE-2024-31989)

   Cycode2024

   Symptoms

   This is a vulnerability report, not an outage, so nothing failed visibly. What an engineer can check is whether the Argo CD Redis server accepts connections without a password from a pod in another namespace, on port 6379 at the address kube-dns returns. In the default installation it did.

   Unauthorised access or a foreign workload

   Trigger

   Cycode Labs installed Argo CD with the default configuration on a new Amazon EKS cluster and connected to its Redis server from a low-privilege pod in another namespace, simulating a compromised pod.

   Root cause

   Software defect

   Component

   Argo CD

   Contributing factor

   Unreviewed defaultsExcessive blast radiusCredential in configuration

   Detection

   Found by Cycode Labs security research, not by an operator.

   Remediation

   • Argo CD released 2.11.1, 2.10.10, 2.9.15 and 2.8.19 with an initialized password for the Redis server

   Lessons

   The hash that checks the cached manifests is computed without a secret key, so anyone who can write to Redis can recompute it and make Argo CD deploy an altered manifest. Cycode recommends upgrading to the fixed versions, checking that the argocd-redis-network-policy NetworkPolicy exists and that a CNI enforces it, using controller-based secret management instead of injecting secrets through Argo CD plugins so secrets never reach Redis, and running Argo CD on a dedicated cluster.

   View case
3. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes

   Google Cloud2022

   Symptoms

   From 10:10 US/Pacific on 2 June 2022, Vertex AI Online Prediction returned errors for 30% up to 100% of requests per region, depending on how each customer used the service.

   High error rateResources deletedService outage

   Trigger

   A faulty resource cleanup process marked Vertex Prediction Endpoints as deleted, globally.

   Root cause

   Software defect

   Component

   Model request routing

   Contributing factor

   Automation amplificationExcessive blast radius

   Detection

   The first public status update was posted at 13:58 US/Pacific and gave the start as 10:20 US/Pacific; the later mini incident report gives 10:10.

   Remediation

   • restored the Vertex Prediction Endpoints that had been marked as deleted

   View case
4. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404

   DFDS2021

   Symptoms

   At 08:00 UTC on 2021-09-06 every service in DFDS's Kubernetes cluster answered with 404. Investigation started five minutes later.

   Service outageHigh error rateProbe failures

   Trigger

   The automated Kubernetes node rollover on 2021-09-06 restarted all the Traefik v2 pods, five days after a malformed IngressRoute manifest had been applied.

   Root cause

   Software defect

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Change with a delayed effectNo configuration testsExcessive blast radius

   Detection

   All services returned 404 from 08:00 UTC and root cause investigation started at 08:05. Time to detection was 5 minutes.

   Remediation

   • changed the ALB health check path, which made the targets healthy but did not stop the 404s
   • rolled back by hand to Traefik v1 as the main ingress controller
   • deleted the malformed IngressRoute by hand and confirmed the Traefik v2 bug
   • applied the changes upstream

   Action items

   • Upgrade Traefik to v2.5, where the bug is fixed, and verify that it removes the bug
   • Change the Traefik v2 health check path to something more robust
   • Set up an external liveness probe on the ingress controller
   • Consider workload segmentation to limit the blast zone
   • Look into monitoring the Traefik v2 API for rapid changes
   • Look into smoke testing during and after node rollover

   Lessons

   A malformed IngressRoute did nothing visible for five days and then took down all ingress when every Traefik v2 pod had restarted during a node rollover. Because Traefik v2 failed gracefully, the first signals pointed at the ALB health checks instead of at Traefik. On incident handling, the team concluded that appointing an incident manager could have helped, that the war room meeting should be recorded and a log book kept, that people not involved should only make suggestions in writing, and that a short first round of sharing thoughts and gut feelings is useful.

   View case
5. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction

   Oath2018

   Symptoms

   During a rolling update of a kubelet change in one of Oath's on-premises production clusters, nodes started going NotReady, and soon all nodes in the cluster were NotReady.

   Node NotReadyPod evictionPods Pending

   Trigger

   A change to kubelet was rolled out across the nodes of one cluster; on restart, kubelet cleans up transient cgroup files, and a Kubernetes bug in that step left kubelet stuck instead of starting.

   Root cause

   Software defect

   Component

   kubeletKubernetes controller manager

   Contributing factor

   No health probesExcessive blast radius

   Detection

   Nodes reported NotReady and the eviction graphs climbed while the kubelet rollout was in progress.

   Remediation

   • failed the affected data center over so users were routed to the next available data center

   Action items

   • Run a functional test of kubelet after each upgrade batch and certify it before the next 30% of nodes are updated

   Lessons

   Check kubelet after upgrading it before moving to the next batch; a simple curl against the kubelet API would have caught this. The node controller's eviction throttling and full disruption mode, which the speakers note is not well documented, kept a cluster-wide kubelet failure from evicting every pod.

   View case

By root cause1