Cases36 cases

4 to 8 of 36

1. IL-0069 Attackers stole a high-privilege service account token from a production Kubernetes cluster and used it to reach a cryptocurrency exchange's financial systems

   Unnamed cryptocurrency exchange2026

   Symptoms

   Inside the cluster the attacker left these traces: a new pod appeared in the production Kubernetes cluster that was built to expose its mounted service account token; requests authenticated as a broad management service account listed secrets and touched workloads in many namespaces; and a backdoor was placed in a running production pod.

   Unauthorised access or a foreign workload

   Trigger

   A spearphishing attack gave the attacker a foothold on a developer's workstation. The attacker used the developer's active, privileged cloud session to deploy a malicious pod to the production cluster.

   Root cause

   Credential exposure

   Component

   RBAC and service accountsKubernetes API server

   Contributing factor

   Excessive blast radius

   Lessons

   Unit 42 concludes that a single over-privileged identity turned one malicious pod into cluster-wide and then cloud-wide access. Its recommendations are least-privilege RBAC with the Restricted Pod Security Standard, short-lived projected service account tokens bound to the pod's lifetime, runtime monitoring of what containers actually do, and Kubernetes audit logging so that identity-driven changes can be seen and reconstructed.

   View case
2. IL-0464 IONOS Managed Kubernetes control planes went unreachable for minutes at a time as their shared etcd grew and fragmented

   IONOS Cloud2026

   Symptoms

   From 14 to 16 March 2026 a subset of customer control planes on IONOS Managed Kubernetes (MK8s) became unavailable again and again, each time for several minutes. While a cluster was unreachable, operations that depend on the Kubernetes API, such as deployments, scaling and health checks, did not work reliably.

   Control plane unresponsiveImage pull failures

   Root cause

   Capacity exhaustion

   Component

   etcdKubernetes API serverManaged Kubernetes control plane and provider API

   Contributing factor

   Excessive blast radius

   Remediation

   • Defragmented the affected database instances, reclaiming about 3–4 GiB per instance
   • Shortened the database compaction intervals on the affected instances
   • Deployed a fix for the affected AI Model Hub database services
   • Applied final changes to all affected clusters

   Action items

   • Expand monitoring dashboards to show control plane health per database instance
   • Move the highest-volume clusters to dedicated database backends
   • Make incidents of this kind trigger immediate paging
   • Roll the updated compaction interval out to all shared control plane database instances
   • Run automated, scheduled defragmentation jobs on all instances
   • Rebalance clusters across database instances to reduce concentration
   • Contact customers whose clusters write disproportionate data, for example to store large security scan reports outside the control plane database
   • Introduce per-tenant resource quotas and key-count limits on the shared control plane database
   • Re-architect the shared control plane for stronger isolation, with dedicated database instances and better vertical scaling
   • Evaluate full database sharding for customers with consistently high data volumes
   • Hold regular drills on database resource exhaustion and recovery

   View case
3. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table

   IONOS Cloud2026

   Symptoms

   On 3 March 2026, during a scheduled maintenance to replace a faulty switch in a cluster in IONOS Cloud's TXL data center, monitoring alerts fired and the hosts in that cluster lost network connectivity. Customers saw intermittent network degradation or outages, and the Data Center Designer (DCD) web frontend and the AI Model Hub became unreachable.

   Network unreachableLink flappingService outage

   Trigger

   During a scheduled maintenance to replace a faulty switch, Topology 2 of 2 in the affected cluster was disabled as planned, and a link flap on a switch in Topology 1 then flooded the InfiniBand control plane with alerts and started a fabric resweep.

   Root cause

   Capacity exhaustion

   Component

   InfiniBand fabricNetwork control plane (SDN)BGP routingNIC, cable and switch port linkModel hubDatabase

   Contributing factor

   Automation amplificationExcessive blast radius

   Detection

   IONOS began investigating monitoring alerts in TXL; the first status update was posted at 11:36 UTC.

   Remediation

   • Switched from the affected control plane to its standby
   • Rebooted the locked-up gateways
   • Recovered Topology 1 from the scheduled maintenance
   • Paused provisioning in the cluster that had a backlog of queued jobs
   • Re-enabled provisioning once network connectivity improved and watched job execution
   • Re-established connectivity for the systems that support the DCD

   Action items

   • Adjusted the control-plane configuration to use less aggressive failover timers and reduced heavy sweep activity, lowering the load on the forwarding table
   • Reduced the number of multicast group memberships to relieve the forwarding table
   • Set up continuous monitoring of forwarding table use and control-plane memory with alerts before capacity limits are reached
   • Move the control plane to dedicated high-performance servers, separate from data-plane traffic
   • Audit IPv6 configurations and IPoIB driver settings
   • Modernise the interconnect fabric, including switches, gateways and drivers
   • Hardened IAM maintenance protocols so that critical identity services keep working when the primary Cloud APIs are temporarily unavailable
   • Move IAM to a dedicated platform without a dependency on the standard managed Kubernetes clusters
   • Introduce a dedicated provisioning provider for DBaaS and Managed Kubernetes
   • Add automated circuit breakers to the provisioning engine against overload and queue buildup from self-healing services
   • Improve visibility and control over job execution and prioritisation in the provisioning queue
   • Change the provisioning queue architecture to decouple services and reduce spillover

   Lessons

   IONOS identified the linking of services through the shared provisioning queue as a key reason the degradation lasted as long as it did. Alongside making each service more robust, it is decoupling the provisioning queue and addressing dependencies between services to reduce the blast radius of future incidents.

   View case
4. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop

   Yonghyun Kim2025

   Symptoms

   While the Spring Cloud Config Server was briefly down for an upgrade, every client application began to answer its liveness probe with a failure saying it could not reach the config server.

   Probe failuresContainer crash loopService outage

   Trigger

   A downtime upgrade of the Spring Cloud Config Server made it stop responding for a short time.

   Root cause

   Dependency failure

   Component

   Liveness and readiness probes

   Contributing factor

   Unreviewed defaultsAutomation amplificationExcessive blast radius

   Lessons

   A liveness probe that includes an external dependency turns a short outage of that dependency into a restart loop of every client. Keep external dependencies out of liveness checks and handle temporary problems with readiness probes. Libraries can add health indicators to Spring Boot's actuator health endpoint automatically, so check what the probe path actually reports. The author ran another Kubernetes environment for two years without liveness probes, relying on strict readiness probes and on the application terminating itself on unrecoverable faults.

   View case
5. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates

   OpenAI2025

   Symptoms

   From 7:43 PM to 8:38 PM PDT on July 15, 2025, a small percentage of ChatGPT users saw more errors in their conversations.

   Container crash loopHigh error rate

   Trigger

   An internal engineer applied a configuration change that contained an invalid value.

   Root cause

   Configuration change

   Component

   –

   Contributing factor

   Excessive blast radiusNo staged rollout

   Remediation

   • reverted the configuration to the correct value
   • started a rolling restart of the affected pods to force them to resynchronise

   Action items

   • Roll out future critical configuration changes for the service to a single cluster first, with automated health guards that stop the rollout if availability drops
   • Move to a service-specific configuration value instead of one shared between services
   • Improve deployment safeguards

   View case

By root cause8