Cases5 cases

1 to 5 of 5

1. IL-0464 IONOS Managed Kubernetes control planes went unreachable for minutes at a time as their shared etcd grew and fragmented

   IONOS Cloud2026

   Symptoms

   From 14 to 16 March 2026 a subset of customer control planes on IONOS Managed Kubernetes (MK8s) became unavailable again and again, each time for several minutes. While a cluster was unreachable, operations that depend on the Kubernetes API, such as deployments, scaling and health checks, did not work reliably.

   Control plane unresponsiveImage pull failures

   Root cause

   Capacity exhaustion

   Component

   etcdKubernetes API serverManaged Kubernetes control plane and provider API

   Contributing factor

   Excessive blast radius

   Remediation

   • Defragmented the affected database instances, reclaiming about 3–4 GiB per instance
   • Shortened the database compaction intervals on the affected instances
   • Deployed a fix for the affected AI Model Hub database services
   • Applied final changes to all affected clusters

   Action items

   • Expand monitoring dashboards to show control plane health per database instance
   • Move the highest-volume clusters to dedicated database backends
   • Make incidents of this kind trigger immediate paging
   • Roll the updated compaction interval out to all shared control plane database instances
   • Run automated, scheduled defragmentation jobs on all instances
   • Rebalance clusters across database instances to reduce concentration
   • Contact customers whose clusters write disproportionate data, for example to store large security scan reports outside the control plane database
   • Introduce per-tenant resource quotas and key-count limits on the shared control plane database
   • Re-architect the shared control plane for stronger isolation, with dedicated database instances and better vertical scaling
   • Evaluate full database sharding for customers with consistently high data volumes
   • Hold regular drills on database resource exhaustion and recovery

   View case
2. IL-0167 A phased rollout of an LKE software release in Osaka stalled LKE-E cluster creation, version upgrades and control plane ACL changes

   Linode2026

   Symptoms

   From about 19:15 UTC on 12 March 2026, customers in the Osaka data center (JP-OSA) could not create Linode Kubernetes Engine for Enterprise (LKE-E) clusters. Cluster creation stalled indefinitely.

   Cluster or node provisioning stallUpgrade stall

   Trigger

   A phased rollout of the LKE software release reached the Osaka data center.

   Root cause

   Faulty deployment

   Component

   Managed Kubernetes control plane and provider API

   Contributing factor

   –

   Detection

   The problem began at about 19:15 UTC, and the first status post, about five and a half hours later at 00:45 UTC on 13 March, said the team was investigating an emerging issue with deploying new LKE Enterprise clusters in Osaka.

   Remediation

   • deployed a software fix to mitigate the impact

   Action items

   • Investigate the root cause further and take preventive actions

   Lessons

   The problem followed a phased rollout of an LKE software release in the Osaka data center. There, cluster creation and administrative tasks stalled for more than six hours, and the first status post came about five and a half hours after the start.

   View case
3. IL-0465 IONOS Container Registry image pushes and pulls failed with 504 Gateway Timeout

   IONOS Cloud2026

   Symptoms

   On 4 March 2026 customers of the IONOS Container Registry got 504 Gateway Timeout errors when pushing or pulling container images, and deployments that relied on the registry were blocked. The status page first reported an increased error count on the registry at 09:23 UTC, saying customers might be unable to pull images.

   Image pull failuresHigh error rate

   Trigger

   A migration combined with a temporary capacity constraint placed two critical control plane components on the same proxy instance, despite anti-affinity rules; the shared proxy then reached its concurrent connection limit.

   Root cause

   Capacity exhaustion

   Component

   Container registry and its credentialsManaged Kubernetes control plane and provider APIPod scheduling and placement

   Contributing factor

   No headroomSingle point of failureNo alert on limit

   Remediation

   • Rolled back a component of the Kubernetes control plane to an earlier version
   • Provisioned additional proxy capacity
   • Moved the affected control plane components onto separate proxy instances

   Action items

   • Redesign registry-to-API connectivity so each node uses its own local proxy
   • Adjust alert thresholds to warn before proxy connection use approaches capacity
   • Deploy more infrastructure clusters and redistribute registries so no cluster exceeds safe capacity, with automation that keeps balancing load

   View case
4. IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors

   Linode2025

   Symptoms

   From about 02:29 UTC on 24 September 2025, customers of Linode Kubernetes Engine (LKE) in US-Southeast (Atlanta) could not manage their clusters, and autoscaling was impacted. Internal logs showed 5xx errors whenever customers tried to manage LKE clusters associated with the affected cluster controller.

   High error rateAutoscaler misbehaviour

   Trigger

   Old certificates on the LKE cluster controllers were revoked at 02:29 UTC on 24 September 2025, after the expiring certificates had been rotated on 23 September.

   Root cause

   Certificate or credential expiry

   Component

   Managed Kubernetes control plane and provider APICluster autoscaler

   Contributing factor

   Change with a delayed effect

   Remediation

   • deployed a fix for the certificate key mismatch
   • restarted the API endpoints

   Action items

   • Continue investigating the root cause and take preventive actions

   View case
5. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck

   Scaleway2024

   Symptoms

   On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.

   Network unreachablePacket lossNode NotReady

   Trigger

   At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routingNetwork switchDatabaseManaged Kubernetes control plane and provider API

   Contributing factor

   Excessive blast radiusAutomation amplification

   Detection

   Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.

   Remediation

   • Fixed both VPC route reflectors by manual action

   Action items

   • Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
   • Run different software versions on the route reflectors so a single bug cannot stop several instances
   • Investigate the software issue that left the route reflectors in a stale state

   View case

By root cause4