Cases3 cases

1 to 3 of 3

1. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table

   IONOS Cloud2026

   Symptoms

   On 3 March 2026, during a scheduled maintenance to replace a faulty switch in a cluster in IONOS Cloud's TXL data center, monitoring alerts fired and the hosts in that cluster lost network connectivity. Customers saw intermittent network degradation or outages, and the Data Center Designer (DCD) web frontend and the AI Model Hub became unreachable.

   Network unreachableLink flappingService outage

   Trigger

   During a scheduled maintenance to replace a faulty switch, Topology 2 of 2 in the affected cluster was disabled as planned, and a link flap on a switch in Topology 1 then flooded the InfiniBand control plane with alerts and started a fabric resweep.

   Root cause

   Capacity exhaustion

   Component

   InfiniBand fabricNetwork control plane (SDN)BGP routingNIC, cable and switch port linkModel hubDatabase

   Contributing factor

   Automation amplificationExcessive blast radius

   Detection

   IONOS began investigating monitoring alerts in TXL; the first status update was posted at 11:36 UTC.

   Remediation

   • Switched from the affected control plane to its standby
   • Rebooted the locked-up gateways
   • Recovered Topology 1 from the scheduled maintenance
   • Paused provisioning in the cluster that had a backlog of queued jobs
   • Re-enabled provisioning once network connectivity improved and watched job execution
   • Re-established connectivity for the systems that support the DCD

   Action items

   • Adjusted the control-plane configuration to use less aggressive failover timers and reduced heavy sweep activity, lowering the load on the forwarding table
   • Reduced the number of multicast group memberships to relieve the forwarding table
   • Set up continuous monitoring of forwarding table use and control-plane memory with alerts before capacity limits are reached
   • Move the control plane to dedicated high-performance servers, separate from data-plane traffic
   • Audit IPv6 configurations and IPoIB driver settings
   • Modernise the interconnect fabric, including switches, gateways and drivers
   • Hardened IAM maintenance protocols so that critical identity services keep working when the primary Cloud APIs are temporarily unavailable
   • Move IAM to a dedicated platform without a dependency on the standard managed Kubernetes clusters
   • Introduce a dedicated provisioning provider for DBaaS and Managed Kubernetes
   • Add automated circuit breakers to the provisioning engine against overload and queue buildup from self-healing services
   • Improve visibility and control over job execution and prioritisation in the provisioning queue
   • Change the provisioning queue architecture to decouple services and reduce spillover

   Lessons

   IONOS identified the linking of services through the shared provisioning queue as a key reason the degradation lasted as long as it did. Alongside making each service more robust, it is decoupling the provisioning queue and addressing dependencies between services to reduce the blast radius of future incidents.

   View case
2. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas

   Cloudflare2020

   Symptoms

   On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

   High error rateHigh latencyDatabase overloadWrites rejectedPacket loss

   Trigger

   At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

   Root cause

   DNS or network fault

   Component

   NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication

   Contributing factor

   Automation amplificationNo headroomHarmful manual action

   Detection

   At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

   Remediation

   • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
   • Moved API traffic that could use read replicas to the secondary data center by hand
   • Sent authentication calls fully back to the primary data center when the dashboard got worse
   • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
   • Ramped the turned-down services back up so asynchronous processing could catch up

   Action items

   • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
   • Redesign the user sessions system so dashboard traffic can be steered between data centers
   • Change the auto-remediation configuration so a database replica is not promoted so quickly
   • Tune each system involved to be more resilient to failures of its dependencies

   Lessons

   Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

   View case
3. IL-0210 GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hours

   GitHub2018

   Symptoms

   At 22:54 UTC on 21 October 2018 GitHub's internal monitoring started alerting on many faults at once. By 23:02 UTC the first responders saw that the replication topologies of many MySQL clusters were in an unexpected state: the Orchestrator API showed topologies made only of servers in the US West Coast data center. When connectivity came back, the application tier had immediately started sending writes to the new West Coast primaries.

   Split brainFailover failureReplication lagHigh latency

   Trigger

   At 22:52 UTC on 21 October, routine maintenance to replace failing 100G optical equipment cut connectivity between GitHub's US East Coast network hub and its primary US East Coast data center for 43 seconds.

   Root cause

   DNS or network fault

   Component

   DatabaseDatabase failover managerDatabase replicationDatabase backups and snapshotsNIC, cable and switch port link

   Contributing factor

   Automation amplificationUnverified assumption

   Detection

   GitHub's internal monitoring raised alerts on numerous faults at 22:54 UTC, two minutes after the partition. By 23:02 UTC the first responders had found, by querying the Orchestrator API, that many database clusters had a topology with only US West Coast servers.

   Remediation

   • Locked the internal deployment tooling to stop further changes
   • Paused webhook delivery and GitHub Pages builds to stop jobs writing metadata
   • Restored the affected MySQL clusters from backups into the US East Coast data center and let them replicate from the West Coast
   • Found ways to restore directly from the West Coast to get around the throughput limits of downloading from off-site storage
   • Moved all database primaries back to the US East Coast data center
   • Spread read traffic over additional MySQL read replicas in the US East Coast public cloud so replication could catch up
   • Failed over to the original topology once the replicas were in sync
   • Processed the backlog of webhook events and Pages builds, pausing to raise an internal TTL after about 200,000 webhook payloads were dropped

   Action items

   • Reconcile the writes from the captured MySQL binary logs that were not replicated, automatically where possible and with user outreach otherwise
   • Make sure status updates can be published even when GitHub Pages builds are paused
   • Give more accurate public estimates of the time to repair
   • Configure Orchestrator so it cannot promote database primaries across regional boundaries
   • Move to a status reporting mechanism that shows the status of each component of the platform
   • Serve GitHub traffic from multiple data centers in an active/active/active design with N+1 redundancy at the facility level
   • Test assumptions about the system more proactively
   • Validate failure scenarios systematically, with investment in fault injection and chaos engineering tooling

   Lessons

   Orchestrator did what it was configured to do, but the application tier could not work with primaries on the other side of the country: leader election within one region is generally safe, while the sudden cross-country latency was a major contributing factor. The behaviour emerged from a partition larger than any GitHub had seen. Replication caught up along a power decay curve, not a straight line, so the linear estimates of recovery time were wrong. GitHub concluded that tighter operational controls and faster response are not enough on their own and that failure scenarios have to be validated before they happen.

   View case

By root cause2