Incident Library
Applied filters ComponentRBAC and service accountsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. Symptoms

    Inside the cluster the attacker left these traces: a new pod appeared in the production Kubernetes cluster that was built to expose its mounted service account token; requests authenticated as a broad management service account listed secrets and touched workloads in many namespaces; and a backdoor was placed in a running production pod.

    Trigger

    A spearphishing attack gave the attacker a foothold on a developer's workstation. The attacker used the developer's active, privileged cloud session to deploy a malicious pod to the production cluster.

    Root cause
    Component
    Contributing factor
    Lessons

    Unit 42 concludes that a single over-privileged identity turned one malicious pod into cluster-wide and then cloud-wide access. Its recommendations are least-privilege RBAC with the Restricted Pod Security Standard, short-lived projected service account tokens bound to the pod's lifetime, runtime monitoring of what containers actually do, and Kubernetes audit logging so that identity-driven changes can be seen and reconstructed.

  2. Symptoms

    Aqua's research team Nautilus recorded the attack on one of its Kubernetes honeypots. What appeared in the cluster was a DaemonSet running the image kuberntesio/kube-controller:1.0.1 from Docker Hub on every node, using the nodes' CPU to mine Monero.

    Trigger

    An API server misconfigured to accept unauthenticated requests from anonymous users with privileges.

    Root cause
    Component
    Contributing factor
    Detection

    Aqua recorded the attacker's API requests on its honeypot; planted AWS access keys raised a beacon when the attacker tried to use them.

    Lessons

    Fixing an exposed API server is not enough once an attacker has been in: a ClusterRoleBinding to an attacker-created ServiceAccount keeps working after anonymous access is closed. Attackers name these objects and images after real Kubernetes components so they blend into the API audit logs and pass for legitimate deployments. Aqua recommends scanning clusters for vulnerabilities, exposed secrets and misconfigurations with a tool such as Trivy.

  3. Symptoms

    At 12:00 EST the health check system reported that the backend APIs, the frontend and the tunnels in the EU region were down. The team first took it for a transient error, such as an interrupted client request or a server evicted for memory or CPU pressure, but the main dashboard was offline as well.

    Root cause
    Component
    Contributing factor
    Detection

    The health check system sent a notification that the service was down; the team then found the main dashboard offline.

    Remediation
    • recreated the node pools by hand after the credential rotation failed to recreate them
    • duplicated the main cluster's services into a newly created GKE cluster
    • deleted the Kubernetes Service records in the old cluster so its load balancers could be attached to the new cluster
    • detached the StatefulSet from the old cluster and switched all writes to the main disk
    • exported the data from the temporary ClickHouse instance and inserted it into the main instance
    Action items
    • Change the deployment manifests so services can be switched between clusters quickly without a complicated persistent disk migration
    Lessons

    Recovery of existing infrastructure needs a time limit. Throughout the outage the old cluster looked about to recover, and the reluctance to give up on the salvage effort kept the team from starting over with a new cluster much sooner, which would have been the right call. On older GKE clusters, check the validity of the cluster CA certificate, for example by decoding masterAuth.clusterCaCertificate from gcloud container clusters describe and reading it with openssl x509. GKE's multiple availability zones made no difference in this failure.

Check your cluster for this

These 3 cases involve Kubernetes API server and RBAC and service accounts. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

By root cause3