Cases6 cases

1 to 5 of 6

1. IL-0069 Attackers stole a high-privilege service account token from a production Kubernetes cluster and used it to reach a cryptocurrency exchange's financial systems

   Unnamed cryptocurrency exchange2026

   Symptoms

   Inside the cluster the attacker left these traces: a new pod appeared in the production Kubernetes cluster that was built to expose its mounted service account token; requests authenticated as a broad management service account listed secrets and touched workloads in many namespaces; and a backdoor was placed in a running production pod.

   Unauthorised access or a foreign workload

   Trigger

   A spearphishing attack gave the attacker a foothold on a developer's workstation. The attacker used the developer's active, privileged cloud session to deploy a malicious pod to the production cluster.

   Root cause

   Credential exposure

   Component

   RBAC and service accountsKubernetes API server

   Contributing factor

   Excessive blast radius

   Lessons

   Unit 42 concludes that a single over-privileged identity turned one malicious pod into cluster-wide and then cloud-wide access. Its recommendations are least-privilege RBAC with the Restricted Pod Security Standard, short-lived projected service account tokens bound to the pod's lifetime, runtime monitoring of what containers actually do, and Kubernetes audit logging so that identity-driven changes can be seen and reconstructed.

   View case
2. IL-0464 IONOS Managed Kubernetes control planes went unreachable for minutes at a time as their shared etcd grew and fragmented

   IONOS Cloud2026

   Symptoms

   From 14 to 16 March 2026 a subset of customer control planes on IONOS Managed Kubernetes (MK8s) became unavailable again and again, each time for several minutes. While a cluster was unreachable, operations that depend on the Kubernetes API, such as deployments, scaling and health checks, did not work reliably.

   Control plane unresponsiveImage pull failures

   Root cause

   Capacity exhaustion

   Component

   etcdKubernetes API serverManaged Kubernetes control plane and provider API

   Contributing factor

   Excessive blast radius

   Remediation

   • Defragmented the affected database instances, reclaiming about 3–4 GiB per instance
   • Shortened the database compaction intervals on the affected instances
   • Deployed a fix for the affected AI Model Hub database services
   • Applied final changes to all affected clusters

   Action items

   • Expand monitoring dashboards to show control plane health per database instance
   • Move the highest-volume clusters to dedicated database backends
   • Make incidents of this kind trigger immediate paging
   • Roll the updated compaction interval out to all shared control plane database instances
   • Run automated, scheduled defragmentation jobs on all instances
   • Rebalance clusters across database instances to reduce concentration
   • Contact customers whose clusters write disproportionate data, for example to store large security scan reports outside the control plane database
   • Introduce per-tenant resource quotas and key-count limits on the shared control plane database
   • Re-architect the shared control plane for stronger isolation, with dedicated database instances and better vertical scaling
   • Evaluate full database sharding for customers with consistently high data volumes
   • Hold regular drills on database resource exhaustion and recovery

   View case
3. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters

   OpenAI2024

   Symptoms

   From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.

   High error rateHigh latencyControl plane unresponsive

   Trigger

   A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.

   Root cause

   Configuration change

   Component

   Kubernetes API server

   Contributing factor

   Excessive blast radiusLimited debug access

   Remediation

   • moved workloads out of the impacted clusters by hand
   • with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover

   Action items

   • Lock namespace changes so this incident cannot recur
   • Freeze certain types of infrastructure deploys
   • Allowlist identities to prevent a recurrence of this class of incidents
   • Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
   • Improve change management for fleet-wide infrastructure changes

   View case
4. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1

   DigitalOcean2024

   Symptoms

   At 13:02 UTC on 2024-01-23 DigitalOcean posted that, as of 12:18 UTC, its engineers were investigating an issue with Kubernetes clusters in the FRA1 region and that users might see errors when communicating with their clusters. The control planes of the affected DOKS (DigitalOcean Kubernetes) clusters had become inaccessible.

   Resources deletedControl plane unresponsive

   Trigger

   An incorrect configuration was applied to an internal infrastructure cluster in FRA1 at 12:21 UTC during a maintenance operation.

   Root cause

   Configuration change

   Component

   etcdKubernetes API server

   Contributing factor

   Automation amplificationNo safeguard on destructive changeExcessive blast radius

   Detection

   Engineers were investigating errors with Kubernetes clusters in FRA1 as of 12:18 UTC and identified the incorrect configuration as the cause at 12:36.

   Remediation

   • restored each affected cluster's etcd from its latest twice-daily backup
   • restored the stateless control plane components after etcd
   • patched the recovery mechanism after unexpected problems during recovery
   • found and fixed an additional small bug during recovery
   • contacted affected customers directly with further instructions, since some might need to reconfigure Kubernetes resources

   Action items

   • Prevent automatic deletion of infrastructure cluster resources under any circumstances in the production environment

   Lessons

   When managed control planes run as resources in another cluster, one configuration mistake in that cluster can delete hundreds of control planes at once through ordinary automation. The etcd backups limited the damage, but their twice-daily schedule set the worst case at 12 hours of lost control plane state, and the recovery tooling itself needed patching during the incident.

   View case
5. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node

   Jetstack2019

   Symptoms

   Jetstack was upgrading the master of a development cluster that many teams use to test their apps during the working day, a regional Google Kubernetes Engine (GKE) cluster in europe-west1. The upgrade ran through Jetstack's GKE Terraform pipeline and did not finish before the 20-minute Terraform timeout, while the GKE console still showed the cluster as upgrading.

   Upgrade stallControl plane unresponsiveNode recreation loopService outage

   Trigger

   A planned master upgrade of the shared development cluster. The same upgrade had been done on another pre-production environment earlier that day, and no API downtime was expected.

   Root cause

   Configuration change

   Component

   Admission webhookGKE control planeKubernetes API serverkubelet

   Contributing factor

   Excessive blast radiusNo health probesMonitoring blind spotAutomation amplification

   Detection

   The first sign was the master upgrade missing the 20-minute Terraform timeout; re-running the pipeline returned the kube-apiserver unhealthy error. The cause was identified with the help of Google Support.

   Remediation

   • Deleted the OPA ValidatingAdmissionWebhook resource during a window of API server access

   Action items

   • Limit the OPA webhook to the namespaces where the policy applies and development teams have access
   • Enable the webhook only for Ingress and Service resources, the only resources the policy validates
   • Add a liveness probe so OPA is restarted when it stops responding, and update the OPA documentation
   • Follow the open upstream Kubernetes issue about this behaviour

   Lessons

   With alerting on API server response times, the team might have seen CREATE and UPDATE requests slow down across the board as soon as the OPA-backed webhook was first deployed. Every workload needs probes: OPA looked so simple to deploy that the team skipped the Helm chart, which adds a livenessProbe for the admission controller containers among other adjustments to the tutorial configuration. Others had hit the same problem, and an upstream issue to improve it was still open.

   View case

By root cause3