IL-0052 JW Player · 2019-03-19 (published)kubernetesawssecurity
JW Player's DevOps team found a Monero miner, named gcc, running on every node of its development and staging Kubernetes clusters. Production was not affected, no customer data was accessed and service was not interrupted. The clusters had recently been rebuilt on Kubernetes 1.10 with different instance sizes, so some unusual behaviour was expected, and noisy load alerts from a legitimate service delayed the investigation.
The miner's parent process was Weave Scope, a tool for watching a Kubernetes cluster in real time. No CVE applied and its image hash matched the published Docker Hub image. Rebuilding an isolated cluster with the same deployment method showed the entry point: the Weave Scope service was a LoadBalancer without the annotation that makes the load balancer internal, so its dashboard was reachable from the internet with no login. Weave Scope can open a command line in any container, and the scope-agent container, which runs on every node, was started with --privileged, mounted files from the host's root filesystem and ran as root. The team reproduced the attack by starting a script on a host from a scope-agent shell.
AWS CloudTrail explained why an internal dashboard was public. When Kubernetes created the load balancer, it attached a security group open to 0.0.0.0/0 on port 80. Someone then restricted it to allow-listed IPs by hand in the AWS console, but Kubernetes owns that load balancer and reverted the edit shortly afterwards.
The damage was one CPU core at 100% on each node in development and staging. Access logs showed no outside logins, the Kubernetes API was not used by the intruder, and RBAC limited Weave Scope to the weave namespace, so a request for secrets would have been denied. The team concluded the crawler's only purpose was mining.
Datadog alerted the DevOps team to a high normalised load average, first on the staging cluster (Day 0, 21:06 EST) and the next day on the development cluster (Day 1, 16:53 EST). The load was first put down to a legitimate service that kept scaling up and down in both environments, and because of the volume of identical alerts they were muted until things settled.
On Day 3 at 17:40 EST, after four days of raised load, an engineer logged onto a node over SSH and ran top. A process named gcc was using 100% of a CPU, and the same process was running on every machine in both clusters, master nodes included.
The first guess was a faulty container launched by the internal deployment tool, but that did not explain why master nodes were affected, and the infrastructure-as-code repositories contained nothing that ran gcc. Inspecting the process showed that its parent was a Weave Scope container, that the binary ran from the host's root directory rather than inside a container, and that it was in fact a cryptocurrency miner. Weave Scope's own view showed the process holding an outbound connection to a mining pool; a search for the name XMRig led to a Monero miner.
Symptom Sustained high node CPUUnauthorised access or a foreign workload
Root cause Configuration change
Datadog alerts on high normalised load average, muted at first as noise from a legitimate service; on Day 3 an engineer ran top on a node and found the gcc process at 100% CPU.
Load average was not a useful signal for spotting a problem in the clusters, and muted noisy alerts nearly hid the miner. A load balancer that Kubernetes creates belongs to Kubernetes: a manual security group edit in the AWS console is reverted, so internal exposure must be declared in the Kubernetes YAML. A dashboard that can open shells, in front of privileged agents that mount the host's root filesystem, gives whoever reaches it root on every node.
IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025same root cause · shared Cloud load balancer
IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022same root cause · shared Cloud load balancer
IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022same root cause · shared Cloud load balancer
IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hourDFDS · 2021same root cause · shared Cloud load balancer
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Cloud load balancer