Cases2 cases

1 to 2 of 2

1. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters

   IONOS Cloud2026

   Symptoms

   On 8 May 2026 IONOS Cloud reported "K8s Control Planes not available": some Managed Kubernetes customers in DE/TXL, DE/FRA and DE/FRA/2 had trouble connecting to the control plane and saw degraded Kubernetes functionality. Between 15:22 and 16:12 (UTC+2) applications meant for a new internal development cluster were installed on other customer clusters, which were also modified.

   Control plane unresponsiveImage pull failures

   Trigger

   A routine internal operation onboarded a new Managed Kubernetes development cluster. The scoping configuration file for the onboarding operator failed to reach the deployment branch because of a merge conflict, and at the same time a second change removed the operator from the exclusion list, which deployed it at once.

   Root cause

   Faulty deployment

   Component

   CI systemContainer registry and its credentials

   Contributing factor

   No configuration testsExcessive blast radiusMissed in review

   Remediation

   • Corrected the onboarding operator so it needs an explicit namespace configuration before acting on any cluster
   • Reconfigured the deployment system to fail deployments when required configuration files are missing
   • Rolled back the identified changes on the affected resources
   • Ran a recovery script, validated in staging, that removed all unintended installations and updates from the affected clusters

   Action items

   • Add a mandatory CI/CD step that confirms all required configuration files reached the target branch before any deployment starts
   • Require additional peer review and change management approval for every deployment that touches production, whatever its size
   • Give teams a way to inform each affected customer in time about the status of post-incident cleanup

   Lessons

   IONOS treats the unintended change to cluster configurations as a failure of the controls that should keep internal operations from affecting customer environments.

   View case
2. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one

   Spotify2019

   Symptoms

   Terraform, applied from the shared cluster configuration repository after a merge, deleted Spotify's production GKE cluster in Asia. While the team tried to recreate it, Terraform deleted a second production cluster, so two of the three production clusters were gone.

   Resources deleted

   Trigger

   A feature team's pull request that added three users to an existing RBAC role binding file was merged before the cluster operators' pull request that imported the Asia cluster into Terraform state. The review build of the import pull request had already written the cluster into the shared state file, and the merged configuration had no definition of the Asia cluster, so Terraform removed it.

   Root cause

   Configuration change

   Component

   TerraformCI systemGKE control plane

   Contributing factor

   No safeguard on destructive changeMissed in reviewUnverified assumption

   Detection

   Internal service discovery could not reach the deleted cluster and the cluster operator team was paged.

   Remediation

   • removed the deleted clusters from the service discovery configuration and restarted service discovery so clients fell back to the instances outside Kubernetes
   • one team that ran half on Kubernetes and half on VMs created more VMs outside Kubernetes to handle the load
   • recreated the clusters, using the hourly backups
   • updated the places where the old clusters' master IPs were hardcoded

   Action items

   • Post the Terraform dry-run output as a comment on each pull request
   • Require a status check that the branch is up to date with master, so the dry run reflects what will actually happen
   • Require an approving review on pull requests
   • Fail reviews automatically when certain keywords such as destroy appear; cluster deletions are done manually outside this repository
   • Add linters and validators for the cluster configuration
   • Schedule and announce disaster recovery tests that cover a cluster going away and smaller failures

   Lessons

   Plan for failure from the start and migrate large infrastructure gradually, so there is redundancy and a way back. A backup whose restore has never been tested is not a backup. Practising disaster recovery paid off: restoring the first deleted cluster took three and a quarter hours, and the team can now restore a larger cluster in one hour. What matters is not who made the mistake but what was learned from it.

   View case

By root cause2