Incident Library
Applied filters Root causeCapacity exhaustionFactorExcessive blast radiusOrganisationIONOS CloudClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0464 IONOS Managed Kubernetes control planes went unreachable for minutes at a time as their shared etcd grew and fragmented
    IONOS Cloud2026
    Symptoms

    From 14 to 16 March 2026 a subset of customer control planes on IONOS Managed Kubernetes (MK8s) became unavailable again and again, each time for several minutes. While a cluster was unreachable, operations that depend on the Kubernetes API, such as deployments, scaling and health checks, did not work reliably.

    Control plane unresponsiveImage pull failures
    Root cause
    Capacity exhaustion
    Component
    etcdKubernetes API serverManaged Kubernetes control plane and provider API
    Contributing factor
    Excessive blast radius
    Remediation
    • Defragmented the affected database instances, reclaiming about 3–4 GiB per instance
    • Shortened the database compaction intervals on the affected instances
    • Deployed a fix for the affected AI Model Hub database services
    • Applied final changes to all affected clusters
    Action items
    • Expand monitoring dashboards to show control plane health per database instance
    • Move the highest-volume clusters to dedicated database backends
    • Make incidents of this kind trigger immediate paging
    • Roll the updated compaction interval out to all shared control plane database instances
    • Run automated, scheduled defragmentation jobs on all instances
    • Rebalance clusters across database instances to reduce concentration
    • Contact customers whose clusters write disproportionate data, for example to store large security scan reports outside the control plane database
    • Introduce per-tenant resource quotas and key-count limits on the shared control plane database
    • Re-architect the shared control plane for stronger isolation, with dedicated database instances and better vertical scaling
    • Evaluate full database sharding for customers with consistently high data volumes
    • Hold regular drills on database resource exhaustion and recovery
    View case
  2. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026
    Symptoms

    On 3 March 2026, during a scheduled maintenance to replace a faulty switch in a cluster in IONOS Cloud's TXL data center, monitoring alerts fired and the hosts in that cluster lost network connectivity. Customers saw intermittent network degradation or outages, and the Data Center Designer (DCD) web frontend and the AI Model Hub became unreachable.

    Network unreachableLink flappingService outage
    Trigger

    During a scheduled maintenance to replace a faulty switch, Topology 2 of 2 in the affected cluster was disabled as planned, and a link flap on a switch in Topology 1 then flooded the InfiniBand control plane with alerts and started a fabric resweep.

    Root cause
    Capacity exhaustion
    Component
    InfiniBand fabricNetwork control plane (SDN)BGP routingNIC, cable and switch port linkModel hubDatabase
    Contributing factor
    Automation amplificationExcessive blast radius
    Detection

    IONOS began investigating monitoring alerts in TXL; the first status update was posted at 11:36 UTC.

    Remediation
    • Switched from the affected control plane to its standby
    • Rebooted the locked-up gateways
    • Recovered Topology 1 from the scheduled maintenance
    • Paused provisioning in the cluster that had a backlog of queued jobs
    • Re-enabled provisioning once network connectivity improved and watched job execution
    • Re-established connectivity for the systems that support the DCD
    Action items
    • Adjusted the control-plane configuration to use less aggressive failover timers and reduced heavy sweep activity, lowering the load on the forwarding table
    • Reduced the number of multicast group memberships to relieve the forwarding table
    • Set up continuous monitoring of forwarding table use and control-plane memory with alerts before capacity limits are reached
    • Move the control plane to dedicated high-performance servers, separate from data-plane traffic
    • Audit IPv6 configurations and IPoIB driver settings
    • Modernise the interconnect fabric, including switches, gateways and drivers
    • Hardened IAM maintenance protocols so that critical identity services keep working when the primary Cloud APIs are temporarily unavailable
    • Move IAM to a dedicated platform without a dependency on the standard managed Kubernetes clusters
    • Introduce a dedicated provisioning provider for DBaaS and Managed Kubernetes
    • Add automated circuit breakers to the provisioning engine against overload and queue buildup from self-healing services
    • Improve visibility and control over job execution and prioritisation in the provisioning queue
    • Change the provisioning queue architecture to decouple services and reduce spillover
    Lessons

    IONOS identified the linking of services through the shared provisioning queue as a key reason the degradation lasted as long as it did. Alongside making each service more robust, it is decoupling the provisioning queue and addressing dependencies between services to reduce the blast radius of future incidents.

    View case
Check your cluster for this

These 2 cases involve BGP routing and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
BGP routing1Database1etcd1InfiniBand fabric1Kubernetes API server1Managed Kubernetes control plane and provider API1Model hub1Network control plane (SDN)1NIC, cable and switch port link1
Contributing factor
Excessive blast radius2Automation amplification1
Organisation
IONOS Cloud2
  1. IL-0464 IONOS Managed Kubernetes control planes went unreachable for minutes at a time as their shared etcd grew and fragmented
    IONOS Cloud2026
  2. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026