Cases2 cases

1 to 2 of 2

1. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters

   IONOS Cloud2026

   Symptoms

   On 8 May 2026 IONOS Cloud reported "K8s Control Planes not available": some Managed Kubernetes customers in DE/TXL, DE/FRA and DE/FRA/2 had trouble connecting to the control plane and saw degraded Kubernetes functionality. Between 15:22 and 16:12 (UTC+2) applications meant for a new internal development cluster were installed on other customer clusters, which were also modified.

   Control plane unresponsiveImage pull failures

   Trigger

   A routine internal operation onboarded a new Managed Kubernetes development cluster. The scoping configuration file for the onboarding operator failed to reach the deployment branch because of a merge conflict, and at the same time a second change removed the operator from the exclusion list, which deployed it at once.

   Root cause

   Faulty deployment

   Component

   CI systemContainer registry and its credentials

   Contributing factor

   No configuration testsExcessive blast radiusMissed in review

   Remediation

   • Corrected the onboarding operator so it needs an explicit namespace configuration before acting on any cluster
   • Reconfigured the deployment system to fail deployments when required configuration files are missing
   • Rolled back the identified changes on the affected resources
   • Ran a recovery script, validated in staging, that removed all unintended installations and updates from the affected clusters

   Action items

   • Add a mandatory CI/CD step that confirms all required configuration files reached the target branch before any deployment starts
   • Require additional peer review and change management approval for every deployment that touches production, whatever its size
   • Give teams a way to inform each affected customer in time about the status of post-incident cleanup

   Lessons

   IONOS treats the unintended change to cluster configurations as a failure of the controls that should keep internal operations from affecting customer environments.

   View case
2. IL-0465 IONOS Container Registry image pushes and pulls failed with 504 Gateway Timeout

   IONOS Cloud2026

   Symptoms

   On 4 March 2026 customers of the IONOS Container Registry got 504 Gateway Timeout errors when pushing or pulling container images, and deployments that relied on the registry were blocked. The status page first reported an increased error count on the registry at 09:23 UTC, saying customers might be unable to pull images.

   Image pull failuresHigh error rate

   Trigger

   A migration combined with a temporary capacity constraint placed two critical control plane components on the same proxy instance, despite anti-affinity rules; the shared proxy then reached its concurrent connection limit.

   Root cause

   Capacity exhaustion

   Component

   Container registry and its credentialsManaged Kubernetes control plane and provider APIPod scheduling and placement

   Contributing factor

   No headroomSingle point of failureNo alert on limit

   Remediation

   • Rolled back a component of the Kubernetes control plane to an earlier version
   • Provisioned additional proxy capacity
   • Moved the affected control plane components onto separate proxy instances

   Action items

   • Redesign registry-to-API connectivity so each node uses its own local proxy
   • Adjust alert thresholds to warn before proxy connection use approaches capacity
   • Deploy more infrastructure clusters and redistribute registries so no cluster exceeds safe capacity, with automation that keeps balancing load

   View case

By root cause2