Cases7 cases

1 to 5 of 7

1. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters

   IONOS Cloud2026

   Symptoms

   On 8 May 2026 IONOS Cloud reported "K8s Control Planes not available": some Managed Kubernetes customers in DE/TXL, DE/FRA and DE/FRA/2 had trouble connecting to the control plane and saw degraded Kubernetes functionality. Between 15:22 and 16:12 (UTC+2) applications meant for a new internal development cluster were installed on other customer clusters, which were also modified.

   Control plane unresponsiveImage pull failures

   Trigger

   A routine internal operation onboarded a new Managed Kubernetes development cluster. The scoping configuration file for the onboarding operator failed to reach the deployment branch because of a merge conflict, and at the same time a second change removed the operator from the exclusion list, which deployed it at once.

   Root cause

   Faulty deployment

   Component

   CI systemContainer registry and its credentials

   Contributing factor

   No configuration testsExcessive blast radiusMissed in review

   Remediation

   • Corrected the onboarding operator so it needs an explicit namespace configuration before acting on any cluster
   • Reconfigured the deployment system to fail deployments when required configuration files are missing
   • Rolled back the identified changes on the affected resources
   • Ran a recovery script, validated in staging, that removed all unintended installations and updates from the affected clusters

   Action items

   • Add a mandatory CI/CD step that confirms all required configuration files reached the target branch before any deployment starts
   • Require additional peer review and change management approval for every deployment that touches production, whatever its size
   • Give teams a way to inform each affected customer in time about the status of post-incident cleanup

   Lessons

   IONOS treats the unintended change to cluster configurations as a failure of the controls that should keep internal operations from affecting customer environments.

   View case
2. IL-0464 IONOS Managed Kubernetes control planes went unreachable for minutes at a time as their shared etcd grew and fragmented

   IONOS Cloud2026

   Symptoms

   From 14 to 16 March 2026 a subset of customer control planes on IONOS Managed Kubernetes (MK8s) became unavailable again and again, each time for several minutes. While a cluster was unreachable, operations that depend on the Kubernetes API, such as deployments, scaling and health checks, did not work reliably.

   Control plane unresponsiveImage pull failures

   Root cause

   Capacity exhaustion

   Component

   etcdKubernetes API serverManaged Kubernetes control plane and provider API

   Contributing factor

   Excessive blast radius

   Remediation

   • Defragmented the affected database instances, reclaiming about 3–4 GiB per instance
   • Shortened the database compaction intervals on the affected instances
   • Deployed a fix for the affected AI Model Hub database services
   • Applied final changes to all affected clusters

   Action items

   • Expand monitoring dashboards to show control plane health per database instance
   • Move the highest-volume clusters to dedicated database backends
   • Make incidents of this kind trigger immediate paging
   • Roll the updated compaction interval out to all shared control plane database instances
   • Run automated, scheduled defragmentation jobs on all instances
   • Rebalance clusters across database instances to reduce concentration
   • Contact customers whose clusters write disproportionate data, for example to store large security scan reports outside the control plane database
   • Introduce per-tenant resource quotas and key-count limits on the shared control plane database
   • Re-architect the shared control plane for stronger isolation, with dedicated database instances and better vertical scaling
   • Evaluate full database sharding for customers with consistently high data volumes
   • Hold regular drills on database resource exhaustion and recovery

   View case
3. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters

   OpenAI2024

   Symptoms

   From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.

   High error rateHigh latencyControl plane unresponsive

   Trigger

   A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.

   Root cause

   Configuration change

   Component

   Kubernetes API server

   Contributing factor

   Excessive blast radiusLimited debug access

   Remediation

   • moved workloads out of the impacted clusters by hand
   • with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover

   Action items

   • Lock namespace changes so this incident cannot recur
   • Freeze certain types of infrastructure deploys
   • Allowlist identities to prevent a recurrence of this class of incidents
   • Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
   • Improve change management for fleet-wide infrastructure changes

   View case
4. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1

   DigitalOcean2024

   Symptoms

   At 13:02 UTC on 2024-01-23 DigitalOcean posted that, as of 12:18 UTC, its engineers were investigating an issue with Kubernetes clusters in the FRA1 region and that users might see errors when communicating with their clusters. The control planes of the affected DOKS (DigitalOcean Kubernetes) clusters had become inaccessible.

   Resources deletedControl plane unresponsive

   Trigger

   An incorrect configuration was applied to an internal infrastructure cluster in FRA1 at 12:21 UTC during a maintenance operation.

   Root cause

   Configuration change

   Component

   etcdKubernetes API server

   Contributing factor

   Automation amplificationNo safeguard on destructive changeExcessive blast radius

   Detection

   Engineers were investigating errors with Kubernetes clusters in FRA1 as of 12:18 UTC and identified the incorrect configuration as the cause at 12:36.

   Remediation

   • restored each affected cluster's etcd from its latest twice-daily backup
   • restored the stateless control plane components after etcd
   • patched the recovery mechanism after unexpected problems during recovery
   • found and fixed an additional small bug during recovery
   • contacted affected customers directly with further instructions, since some might need to reconfigure Kubernetes resources

   Action items

   • Prevent automatic deletion of infrastructure cluster resources under any circumstances in the production environment

   Lessons

   When managed control planes run as resources in another cluster, one configuration mistake in that cluster can delete hundreds of control planes at once through ordinary automation. The etcd backups limited the damage, but their twice-daily schedule set the worst case at 12 hours of lost control plane state, and the recovery tooling itself needed patching during the incident.

   View case
5. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast

   Microsoft Azure2023

   Symptoms

   From about 10:30 UTC on 30 August 2023, customers had trouble accessing or using Azure, Microsoft 365 and Power Platform services in Australia East. The affected Azure services included Azure Kubernetes Service (AKS) along with storage, SQL Database, Cosmos DB, virtual machines and Azure Resource Manager.

   Control plane unresponsiveHigh error rateService outage

   Trigger

   At 08:41 UTC a lightning strike on electrical infrastructure about 18 miles away caused a utility voltage sag that shut down the cooling chillers of datacenters in one Australia East availability zone; 13 chillers failed to restart automatically and five could not be restarted, so two data halls overheated and their infrastructure was shut down.

   Root cause

   Cloud provider outage

   Component

   AKS control planeDatabase

   Contributing factor

   Single point of failureExcessive blast radius

   Detection

   Cosmos DB impact was detected by monitoring at 10:47 UTC, first automated communications went to Azure Service Health at 10:48 UTC, the public Azure Status page was updated at 11:30 UTC, and all subscriptions in Australia East were sent portal communications at 11:36 UTC.

   Remediation

   • shut down the remaining networking, compute and storage in the two affected data halls so the chilled water loop could cool down
   • restarted the five chillers manually at 12:12 UTC
   • restored power to all hardware by 15:10 UTC and brought infrastructure back in phases
   • restored AKS operations through the recovery of the SQL Database operation queue, with no other mitigation

   Action items

   • AKS: convert all operation queue SQL Database databases to be zone redundant (estimated September 2023)
   • AKS: replace all cross-region SQL Database queue usage with zone-redundant Service Bus queues (estimated September 2023)
   • Change the control timing logic of the Chiller Management System so chillers do not fail to restart after voltage fluctuations
   • Change the emergency procedure for manual restarts of simultaneous chiller failures from an adjacent sequence to a load-based sequence per data hall
   • Increase technician staffing at the datacenter as a temporary measure for manual chiller restarts
   • Storage: optimise the large-scale recovery process and update troubleshooting guides (estimated December 2023)
   • Azure Service Fabric: improve reliability of SQL Database tenant ring recovery (estimated December 2023)
   • SQL Database: review the trigger criteria of auto-failover groups (estimated October 2023)
   • SQL Database: upgrade internal tooling to enable mass migration of databases (estimated December 2023)
   • Cosmos DB: optimise Service Managed Failover for single-region-write accounts (estimated November 2023)
   • ARM: complete the migration of its storage layer to the zonally redundant architecture (estimated December 2023)
   • Incident management: harden readiness, process and playbook for power-down scenarios (estimated October 2023)

   Lessons

   A control plane spread across availability zones still fails with a zone when one of its dependencies, here the SQL Database operation queue, is not zone redundant. When a second region uses the same database, the failure crosses into that region as well. Workloads already running were not expected to be affected; what stopped was everything that needs the managed service's control plane, such as scaling and cluster changes.

   View case

By root cause4