Cases3 cases

1 to 3 of 3

1. IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)

   Wiz2025

   Symptoms

   This is a vulnerability report, not an outage, so nothing failed visibly. A cluster is exposed if it runs Ingress NGINX Controller older than 1.12.1 or 1.11.5 with its validating admission webhook enabled. Wiz suggests `kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx` to find it. The webhook accepts AdmissionReview requests over HTTP without authentication from any pod, and Wiz found more than 6,500 clusters exposing it to the public internet.

   Unauthorised access or a foreign workload

   Trigger

   Wiz Research reviewed the code of the Ingress NGINX admission controller and reported the first findings to Kubernetes on 31 December 2024.

   Root cause

   Software defect

   Component

   Ingress controllerAdmission webhook

   Contributing factor

   Unreviewed defaultsExcessive blast radius

   Detection

   Found by Wiz Research in a code review of the admission controller, not by an operator.

   Remediation

   • Kubernetes patched the injection flaws CVE-2025-1097, CVE-2025-1098 and CVE-2025-24514
   • Kubernetes removed NGINX configuration testing from the ingress-nginx admission controller, resolving CVE-2025-1974
   • released Ingress NGINX Controller 1.12.1 and 1.11.5

   Lessons

   Admission webhooks often run as web servers inside the cluster without authentication and with relatively high privileges, so any pod can reach them, and the flaws pair well with SSRF in an application. Wiz concludes that `nginx -t` should be considered harmful when it processes unsanitized user input. Wiz recommends upgrading, keeping the webhook off the internet and allowing only the API server to reach it with network policies. If upgrading must wait, disable the webhook (Helm: controller.admissionWebhooks.enabled=false; manual installs: delete the ValidatingWebhookConfiguration ingress-nginx-admission and remove --validating-webhook from the controller) and enable it again after the upgrade.

   View case
2. IL-0065 Any pod could take over the cluster by rewriting Argo CD's unauthenticated Redis cache (CVE-2024-31989)

   Cycode2024

   Symptoms

   This is a vulnerability report, not an outage, so nothing failed visibly. What an engineer can check is whether the Argo CD Redis server accepts connections without a password from a pod in another namespace, on port 6379 at the address kube-dns returns. In the default installation it did.

   Unauthorised access or a foreign workload

   Trigger

   Cycode Labs installed Argo CD with the default configuration on a new Amazon EKS cluster and connected to its Redis server from a low-privilege pod in another namespace, simulating a compromised pod.

   Root cause

   Software defect

   Component

   Argo CD

   Contributing factor

   Unreviewed defaultsExcessive blast radiusCredential in configuration

   Detection

   Found by Cycode Labs security research, not by an operator.

   Remediation

   • Argo CD released 2.11.1, 2.10.10, 2.9.15 and 2.8.19 with an initialized password for the Redis server

   Lessons

   The hash that checks the cached manifests is computed without a secret key, so anyone who can write to Redis can recompute it and make Argo CD deploy an altered manifest. Cycode recommends upgrading to the fixed versions, checking that the argocd-redis-network-policy NetworkPolicy exists and that a CNI enforces it, using controller-based secret management instead of injecting secrets through Argo CD plugins so secrets never reach Redis, and running Argo CD on a dedicated cluster.

   View case
3. IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboard

   JW Player2019

   Symptoms

   Datadog alerted the DevOps team to a high normalised load average, first on the staging cluster (Day 0, 21:06 EST) and the next day on the development cluster (Day 1, 16:53 EST). The load was first put down to a legitimate service that kept scaling up and down in both environments, and because of the volume of identical alerts they were muted until things settled.

   Sustained high node CPUUnauthorised access or a foreign workload

   Trigger

   An automated crawler found the Weave Scope dashboard on a public load balancer that required no authentication. Weave Scope can open a shell in running containers, and its scope-agent containers ran privileged, as root and with the host's root filesystem mounted, so a command started there could run on the host.

   Root cause

   Configuration change

   Component

   Cluster dashboardCloud load balancer

   Contributing factor

   Ignored early signalUnreviewed defaultsExcessive blast radiusMonitoring blind spot

   Detection

   Datadog alerts on high normalised load average, muted at first as noise from a legitimate service; on Day 3 an engineer ran top on a node and found the gcc process at 100% CPU.

   Remediation

   • stopped Weave Scope and removed its deployment from all Kubernetes clusters
   • checked access logs for unauthorised access to the instances
   • confirmed the miner's outbound connections went only to a mining pool
   • isolated one affected node for later investigation
   • rotated out every node in each cluster so all affected hosts were destroyed and rebuilt

   Action items

   • Audit all Kubernetes and deployment YAML files and security groups for services that are unintentionally public or misconfigured
   • Replace load-average alerts with more meaningful monitors and tune them to signal a real anomaly
   • Research tools such as falcon or sysdig for behavioural monitoring and anomaly and intrusion detection
   • Evaluate Istio and Linkerd for observing and controlling end-to-end network traffic
   • Analyse the cost of scanning Docker images and containers for known vulnerabilities
   • Set aside time in architecture discussions to revisit older choices such as running containers as root
   • Share knowledge of large projects, such as Kubernetes version migrations, across the whole team

   Lessons

   Load average was not a useful signal for spotting a problem in the clusters, and muted noisy alerts nearly hid the miner. A load balancer that Kubernetes creates belongs to Kubernetes: a manual security group edit in the AWS console is reverted, so internal exposure must be declared in the Kubernetes YAML. A dashboard that can open shells, in front of privileged agents that mount the host's root filesystem, gives whoever reaches it root on every node.

   View case

By root cause2