Incident Library
Applied filters ComponentNIC, cable and switch port linkSymptomNetwork unreachableClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026
    Symptoms

    On 3 March 2026, during a scheduled maintenance to replace a faulty switch in a cluster in IONOS Cloud's TXL data center, monitoring alerts fired and the hosts in that cluster lost network connectivity. Customers saw intermittent network degradation or outages, and the Data Center Designer (DCD) web frontend and the AI Model Hub became unreachable.

    Network unreachableLink flappingService outage
    Trigger

    During a scheduled maintenance to replace a faulty switch, Topology 2 of 2 in the affected cluster was disabled as planned, and a link flap on a switch in Topology 1 then flooded the InfiniBand control plane with alerts and started a fabric resweep.

    Root cause
    Capacity exhaustion
    Component
    InfiniBand fabricNetwork control plane (SDN)BGP routingNIC, cable and switch port linkModel hubDatabase
    Contributing factor
    Automation amplificationExcessive blast radius
    Detection

    IONOS began investigating monitoring alerts in TXL; the first status update was posted at 11:36 UTC.

    Remediation
    • Switched from the affected control plane to its standby
    • Rebooted the locked-up gateways
    • Recovered Topology 1 from the scheduled maintenance
    • Paused provisioning in the cluster that had a backlog of queued jobs
    • Re-enabled provisioning once network connectivity improved and watched job execution
    • Re-established connectivity for the systems that support the DCD
    Action items
    • Adjusted the control-plane configuration to use less aggressive failover timers and reduced heavy sweep activity, lowering the load on the forwarding table
    • Reduced the number of multicast group memberships to relieve the forwarding table
    • Set up continuous monitoring of forwarding table use and control-plane memory with alerts before capacity limits are reached
    • Move the control plane to dedicated high-performance servers, separate from data-plane traffic
    • Audit IPv6 configurations and IPoIB driver settings
    • Modernise the interconnect fabric, including switches, gateways and drivers
    • Hardened IAM maintenance protocols so that critical identity services keep working when the primary Cloud APIs are temporarily unavailable
    • Move IAM to a dedicated platform without a dependency on the standard managed Kubernetes clusters
    • Introduce a dedicated provisioning provider for DBaaS and Managed Kubernetes
    • Add automated circuit breakers to the provisioning engine against overload and queue buildup from self-healing services
    • Improve visibility and control over job execution and prioritisation in the provisioning queue
    • Change the provisioning queue architecture to decouple services and reduce spillover
    Lessons

    IONOS identified the linking of services through the shared provisioning queue as a key reason the degradation lasted as long as it did. Alongside making each service more robust, it is decoupling the provisioning queue and addressing dependencies between services to reduce the blast radius of future incidents.

    View case
  2. IL-0344 Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failed
    Linode2023
    Symptoms

    On 12 February 2023, customers in Linode's US-West region could not reach their Linodes. Santa Clara customers lost Internet connectivity after a change made at 8:45 UTC until it was reverted at 10:13 UTC, and later in the day the connection between the Fremont and Santa Clara data centers was cut completely, leaving both with degraded service or a full outage until 16:25 UTC; customers could also have trouble provisioning, powering down or rebooting services.

    Network unreachable
    Trigger

    A single link of the A-side fibre circuit between Fremont and Santa Clara went down on 11 February, the whole B-side circuit followed on 12 February, and during troubleshooting with the hardware vendor a non-intrusive command triggered a rare bug that took down all A-side links.

    Root cause
    Software defect
    Component
    Backbone network (WAN)NIC, cable and switch port link
    Contributing factor
    Overlapping failuresHarmful manual action
    Detection

    The response team began investigating when the A-side link went down on 11 February. The status page opened at 10:22 UTC on 12 February to investigate reports of connectivity issues in the US-West region.

    Remediation
    • Reverted the change that removed internet traffic from the Santa Clara border routers
    • Performed an emergency code upgrade of both circuits, restoring connectivity between the sites
    • Restored the B-side path with the hardware vendor and fiber provider, regaining full redundancy
    View case
Check your cluster for this

These 2 cases involve NIC, cable and switch port link and Backbone network (WAN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Capacity exhaustion1 case

Component
BGP routingDatabaseInfiniBand fabricModel hubNetwork control plane (SDN)NIC, cable and switch port link
Contributing factor
Automation amplificationExcessive blast radius
Organisation
IONOS Cloud
  1. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026

Software defect1 case

Component
Backbone network (WAN)NIC, cable and switch port link
Contributing factor
Harmful manual actionOverlapping failures
Organisation
Linode
  1. IL-0344 Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failed
    Linode2023